Helping Moms make purchase decisions

zulily’s unique needs

“Something special every day”  is our motto at zulily. Thousands of new products go live every morning. Most of these products are available for 72 hours and a good number of them often sell out before the sales event ends! Many of our engaged customers visit us every single day to discover what’s new. We want to make sure our customers don’t miss out on any items or events that may be special to them, while also giving them more confidence in their purchase decisions. A traditional eCommerce ratings and reviews model could help, but is not the best fit for zulily’s unique business model which offers customers a daily assortment of new products, for a limited amount of time. We needed a different approach.

Our solution

Our specific business requirements demanded a more real time and community-oriented approach. We came up with a set of signals that highlighted the social proof and scarcity. Signals like “Only X left” and “Almost Gone” were designed to encourage users to act on a product that they are interested in before it is gone. Signals like “Popular”, “X people viewing” and “Y just sold” were intended to give users more confidence in their purchase decision. We were quickly able to bring these signals to life, thanks to our real time big data platform. These signals were shown on our product pages and the shopping cart.

Product page

Shopping cart


We tested the feature out on our web and m-web experiences. The results turned out to be better than our most optimistic expectations! It was interesting to note that the feature was almost twice as effective on the mobile device compared to desktop. In hindsight it made a lot of sense as our customers have a lot of small sessions on the mobile devices during the day and this information helped them make timely decisions. The social and scarcity signals turned out to be a perfect complement to zulily’s unique business model.

zulily’s Kubernetes launch presentation at OSCON

In July Steve Reed from zulily presented at O’Reilly’s Open Source Convention (OSCON). He spoke to zulily’s pre-launch experience with Kubernetes. It was an honor for zulily to be asked to speak as part of the Kubernetes customer showcase, given the success we have had with Kubernetes.

Kubernetes launch announcement:

Partnering with zulily – A Facebook Perspective

During F8, Facebook’s global developer conference (March 25-26, 2015), Facebook announced a new program with zulily as one of the launch partners: Messenger for Business. Just a month later, the service was live and in production! We on the zulily engineering team truly enjoyed the partnership with Facebook on this exciting product launch. It’s been a few months since the launch and we reached out to Rob Daniel, senior Product Manager at Messenger, to share his team’s experience on collaborating with zulily.

Here’s what Rob had to say:

How Messenger for Business Started

Messenger serves over 700 million active people with the mission of reinventing and improving everyday communication. We invest in performance, reliability, and the core aspects of a trusted and fast communication tool. Furthermore, we build features that allow people to better express themselves to their friends and contacts, including through stickers, free video and voice calling, GIFs, and other rich media content. We build the product to be as inclusive as possible – even supporting people that prefer to register for Messenger without a Facebook account – because we know that the most important feature of a messaging app is that all your friends and anyone you’d want to speak with is reachable, instantly.

In that vein, we found that some of people’s most important interactions are not just with friends and family, but with businesses they care about and interact with almost every day. However, until recently Messenger didn’t fully support a robust channel for communicating with these businesses. And when we looked at how people interact with businesses today – email, phone (typically…IVR phone trees), text – there were obvious drawbacks. Either the channels were nosiy (email), or lacked richeness (text), or were just really heavyweight and inefficient (voice).

Reinventing How People Communicate with Businesses

When we started talking with zulily about a partnership to reinvent how people communicate with businesses, under the mission of creating a delightful and high utility experience, we found we met an incredible match.

Why? Simply put, zulily thinks about Mom first. That’s the same people-centric approach to building products that’s at the heart of Messenger. And along the way, as we scoped out the interactions for launch, it was evident that we had the same mindset: build a product with incredible utility and value to people, establish a relationship and build trust, and over time build interactions that pay dividends via higher customer loyalty and lifetime value. And despite time pressures and the many opportunities to put business values over people values, we collectively held the line that people were first, business was second.

We also found that “zulily Speed” was no joke, and matched the Facebook mantra of “Move Fast.” Up and through our announcement at F8 (Facebook’s two-day developer conference) and launch later the next month, our two teams moved in sync at incredible speed. As an example, late one evening an issue was spotted by a Messenger engineer and by 10:00 am the next morning the problem had been identified, fixed and pushed to production. That kind of turn around time and coordination is just unheard of between companies at zulily and Facebook’s scale.

Speaking to the strength of the individual engineers on both sides, team sizes were small. This led to quick decision making and efficient, frequent communications between teams, forming a unique bond of trust and transparency. Despite the distance between Seattle and Menlo Park, we rarely had timing miscues and latency was incredibly low.

This Journey is 1% Finished

At Facebook we say “this journey is 1% finished,” highlighting that regardless of our past accomplishments we have a long journey ahead, and that we owe people that use our services so much more. And in that same spirit, we respect that Messenger is only beginning to provide zulily and other businesses the communication capabilities they need to form lasting, trusted, and valuable relationships with their customers. But we’re thrilled to be teamed up with a company, like zulily, that has the commitment and vision alignment to help shape the experience along the way and see the opportunity along the path ahead.

zulily hosted Women Who Code Event


A sample of zulily’s “women in tech”

zulily knows that women in tech rock! On January 28, 2015 zulily hosted the Women Who Code Seattle chapter’s kick-off meeting, bringing women in technology together to discuss their passions. zulily has built its business on providing great value to moms. Our female engineers are a big driving force in delivering this expectation, so we were very excited to partner with  Women Who Code.  

Women Who Code is a global nonprofit organization that supports women in technology, helping them advance their careers. They provide supportive environments where women can learn from each other while offering mentorship and networking. 

The night started with mingling in a variety of technology discussions ranging from software, robotics, data analysis and bio engineering! College students and seasoned professionals gathered to share experiences, answer questions and consume food and wine. The atmosphere was inspiring and empowering. 

Kristin Smith, CEO of Code Fellows, shared with us her story and career path from Amazon to zulily to CEO of Code Fellows. Kristin talked about how women should be persistent in constantly learning new things and trying new technologies to empower their skills and overcome their self-doubt. “Embrace the ambiguity. Embrace the fear. That’s the time when things are going to explode!” 


Kristin Smith, CEO of Code Fellows

We continued the night with interesting tech talks from women who work in zulily tech. Jaime Dughi is Sr. Product Manager of zulily’s Mobile Development team. She walked us through zulily’s unique business model and core development principles: fearless innovation and moving fast. She described the future of the mobile app platform in zulily’s business and how her team is advancing mobile development to tackle the demands of zulily customers while embracing the ever-changing mobile app landscape. (slides)


Jaime Dughi – Sr. Product Manager, zulily Mobile Development Team

Our next speaker was Sara Adineh, Software Engineer on the Personalization Team. Sara remarked on her passion in bringing more women in tech, and shared her past volunteer activities: helping run a club for female students at her school to support women in the tech and science fields. She dove deep into what the Personalization Team does at zulily. “Personalization is the heart of zulily’s business model, presenting something special every day for each member” said Sara. She described the mathematics and Machine Learning techniques that zulily uses to understand what zulily’s customers need, so we can provide them something fresh every day. Sara talked about one of the engineering principles of zulily’s tech culture: zulily had open-sourced some of its projects as well as contributed to open-source technologies (such as Go, Kubernetes, and NSQ) used by the company. (Slides)


Sara Adineh – Software Engineer, zulily Personalization Team

Our last speaker was Echo Li, Data Engineer on the Business Intelligence and Data Services team at zulily. She started with “I’m a woman, I’m a mom, I’m an engineer and I’m proud!” and the crowd went wild! She continued explaining her work as a data engineer at zulily, and spoke on how to understand data and get meaningful information out of it. She talked about different stacks of the data-processing pipeline and analytics here at zulily. She shared with us the growth of zulily’s big data platform from SQL server to Hadoop, Google Cloud and Big Query. (Slides)


Echo Li – Sr. Data Warehouse Engineer, BI and Data Services

It was a night of thoughtful questions, engaged attendees and new friendships made. You can find it on the comments. We are looking forward to hosting the next Women in Tech event! Follow us if you want to hear about our future events!




Facebook–zulily Joint Hack Day

zulily launches over 9,000 unique styles – more than the size of an average Costco – every day. This causes unique time and scale challenges for the Marketing team as they rapidly create, place and manage ads. Facebook and zulily’s Business and Technology teams have been working together to build a platform – the Acquisition Marketing Platform (AMP) – to automate the ad management process on Facebook.

During this partnership, we realized that both companies share a passion for enabling the engineering teams to move fast and put new, exciting features in the hands of customers. In that spirit, we hosted a joint hack day to see how we can further advance the AMP toolset.

Joint FB-ZU Hack day participants: David Whitney (FB), Leo Hu (ZU) and Tim Moon (ZU). Not in picture: Omar Zayat (FB), Justin Sadoski (FB), Mason Bryant (ZU) and Rahul Srivastava (ZU)

The key wins for the hack day:

  • End-to-end automation: Currently, Marketing uses AMP to create the ad assets and Facebook’s PowerEditor to target and place the ads. The joint team built a prototype, utilizing the FB Graph and Ads APIs, to automate the ad process entirely within AMP. This is a huge win for the team! Marketing no longer has to switch between tools and now has a central place to manage all FB ads. This feature will further enable Marketing to scale up their ad creation process. The AMP Engineering team is now working on building a production version of the prototype.
  • Ad objects hierarchy management: FB ads have a hierarchy of campaign, ad-set and ad. The joint team spent time diving deep into the API to manage this hierarchy. Building this understanding is critical to automating management of campaigns, ad sets and reporting on key metrics.
  • API-driven ad Targeting: There are many ways to define the target of an ad on FB: interest, website customer audience, custom audience, etc. The team used the Targeting API to dynamically change the target range of an ad. This enables zulily to build some really interesting scenarios that can leverage real-time data in its systems; as we track the cost and conversion of every ad that is launched, we can now automate the increase or decrease the targeting reach via the API calls.
  • Account management: In October, FB released a new model to manage business ad accounts. The team spent time revamping the existing account model. This was a huge help and removed numerous access- and account-related roadblocks to further development. It would have taken us days of back-and-forth over email and conference calls to adopt the new model. During the hack day, this was completed in hours!

We had a fun and highly productive day-and-a-half of hacking. It was amazing to see the cool ideas and products that materialized from this event. The AMP team is super excited (and the Marketing team even moreso!) to build all the features that are now on their backlog. We are looking forward to future FB-ZU hack days!

Expect: How a 20-Year-Old Tool Saved My Project

I joined zulily in August of 2010, and at the time the company as a whole consisted of only 35 people. One of my first projects involved integrating one of our systems with a system owned by a much larger, more established partner company. The details of what these systems did aren’t relevant, except that the integration mechanism was for our system to drop XML files on an SFTP server that the partner company owned and operated.

At the time we were a 100% PHP shop (this has since changed), so I implemented our side of the integration in PHP, and used an open source PHP library called phpseclib to handle the actual SFTP data transfer. The partner company didn’t have any clients who used PHP, and didn’t officially support this library, but it worked great throughout development and integration testing. The integration test phase of the project took approximately 3 months, and not once during that time did we ever have even the slightest hiccup when transferring data between systems.

However, once we went to full production, we started seeing file corruption — specifically, sometimes files we transferred to the partner’s SFTP server would be truncated. There was no discernible pattern to the truncation; it happened at different points every time, and often a file that failed once would work fine when it was retried.

Naturally, this caused some consternation, as our code hadn’t changed, and it had been working for months without fail. When I pointed the exact same code at their test server, and sent the exact same file content, everything worked flawlessly. When I pointed it back to their production server, the files were truncated.

Clearly, to my mind at least, the problem was on their end. After a couple of days of frenzied troubleshooting, we discovered that the version of the SFTP server software running on their test machines was newer than what they ran in production. Presumably, we were hitting some bug in that software that was fixed between the two versions.

Since they were a very large company with many other clients using these servers, they were unwilling to upgrade their SFTP software on our behalf. Also, given that we were already well into the go-live phase of the integration, rewriting our system in another language wasn’t an appealing option, especially since there was no guarantee that the new implementation would work any better.

One option that the parent company did officially support, though, was the sftp client included with OpenSSL. I tried manually transferring a few files this way…and the file truncation issue disappeared.

There were problems with this approach, though: for one, the SFTP server required authentication, and the partner company was unwilling to set up SSH keys for us to do non-interactive authentication. OpenSSL’s sftp client doesn’t support setting the authentication credentials via command line parameters, leaving us stuck with authenticating interactively. This obviously wasn’t an acceptable long-term solution, since these systems needed to communicate with each other without human intervention.

I don’t recall exactly when I stumbled across it, but somewhere in the midst of searching for a solution I came across Expect.

Quoting the introduction on the Expect homepage:

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial.

This sounded exactly like what I needed! I bought a copy of “Exploring Expect“, read enough of it on the train ride home to get started, and after a couple of hours I had whipped up a PHP script that did the following:

  1. Dump the XML data we wanted to transfer into a local temp file.
  2. Build an Expect script in memory by doing some variable interpolation in a PHP string.
  3. Write the resulting Expect script to another local temp file.
  4. Exec() the Expect script, capturing the process return code and anything the script wrote to stdout or stderr.
  5. Write the process output to our application log for later troubleshooting if anything went wrong.
  6. Finally, if the process return code was zero (indicating success), delete the two temp files. Otherwise, send an email to our notification alias so I could investigate.

Here’s the meat of the Expect script generation in all its glory:

$expectContents =
 '#!/usr/bin/expect' . "\n"
 . 'set timeout ' . $this->m_timeout . "\n"
 . 'spawn sftp -o Port=' . $this->m_port . ' ' . $this->m_user . '@' . $this->m_host . "\n"
 . 'expect {' . "\n"
 . ' default {exit 1}' . "\n"
 . ' "Connecting to ' . $this->m_host . '..."' . "\n"
 . '}' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 2}' . "\n"
 . ' "continue connecting (yes/no)?" {send "yes\n"; exp_continue}' . "\n"
 . ' "ssword"' . "\n"
 . '}' . "\n"
 . 'send "' . $this->m_pass . '\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 3}' . "\n"
 . ' "ermission denied" {exit 4}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "cd /inbound\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 5}' . "\n"
 . ' "not found" {exit 6}' . "\n"
 . ' "No such file" {exit 7}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "put ' . $filename . '\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 8}' . "\n"
 . ' "not found" {exit 9}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "quit\n"' . "\n"
 . 'exit 0' . "\n";

Yes, I’m aware that there are more elegant ways to interpolate strings in PHP than this, but at the time I was still fairly new to PHP and under a ton of pressure to get something — anything — working.

Coming from a background in statically typed compiled languages, this just felt wrong somehow. Even my stints in the land of Perl didn’t feel this hacky. It was an ugly, nasty, weird abomination…but it worked.

And it kept working, without fail, for the entire lifespan of this system. Over time, we encountered bugs in many other areas, as is inevitable with any complex system, but this little corner of the codebase never had a single issue.

What this taught me is that sometimes you should go slowly, be methodical, and take the time necessary to create a simple, elegant solution to your problem.

And sometimes you just need to break out the duct tape.

welcome to the zulily engineering blog!

It has been just over four and a half years now since Darrell and Mark (our two co-founders) came up with the original idea for zulily.  And from the beginning we’ve focused on building software to power a new way of shopping online.  We call it discovery-based shopping.

Here at zulily our tech team is at the core of the business and involved in the entire life-cycle of both our vendors and our customers.  Whether it’s building internal tools for our merchandizing and studio teams, launching new features on our vendor portal or vendor data exchange or delivering a new personalized experience on our mobile or site experience, we are always focused on challenging ourselves to build world-class solutions which exceed expectations.

We are a build shop and big supporters of the open source community.  We believe in the power of the community and feel we have an obligation to give back to the projects that have helped us get to where we are today.  As we continue our transition from small, frenetic start-up, expect to see us continue to be more active in the community.

At our core we have 10 values we try to live by on a daily basis.  These have served us well over the past 4+ years as we’ve tried new things and experienced major wins… and a number of “well, that was a bad idea” moments.

  1. “No” is not in our vocabulary — we strive to find creative solutions and the path to “yeah, we’ll give it a go”.
  2. We believe in speed of innovation and taking agile development to the extreme.
  3. We embrace a customer-centric view to delivering technology solutions — always start with the customer.
  4. Mistakes are expected and encouraged — we learn from them and move on.
  5. We empower our engineers to solve business problems and tailor our process accordingly.
  6. Engineers write production code and own it from start to finish.
  7. We are defensive in nature: we assume things will break and plan for it.
  8. We believe in “just-in-time” software with an eye towards capacity and scalability.
  9. We value full transparency and continuous communication.
  10. We strive to find the simple solution in anything we do.

In the end we’re all about building an amazing team, passionate about building awesome software and technology solutions.  We love to move fast and take risks.  And we’re big believers in the idea of continuous improvement.

Thanks for taking a few minutes out of your busy day to read our tech blog.   We hope you enjoy it!


Meet a zulily Developer: Trevor

Processed with VSCOcamEach month zulily will talk with a developer and learn about a day in the life of a zulily engineer.

Who are you, and what do you do at zulily?

I’m Trevor, a developer on the Relevancy team. Prior to that, I worked on our fulfillment and warehouse management systems.

When did you join zulily?

I started in August of 2010, so it’s been 4 years now.

What was it like in the early days? Tell us a crazy story.

Oh man, where to start….

  • My first desk was the classic startup cliché: a door blank on top of two filing cabinets. (We have proper desks now.)
  • My second day on the job, the director in charge of the Supply Chain team stopped by my desk and introduced herself like so: “Hi, I’m Lys. I hear you’re traveling with me to our vendor site next week?” At that point my manager leaned over and said, “Oh, uh, heh, I meant to ask you: can you go to our vendor next week?”
  • The following week consisted of Lys and me in a conference room with 10 folks in suits from the supply chain logistics company with whom we were gearing up to integrate. I had never worked on anything remotely related to supply chain logistics before, and couldn’t have told you what “GOH” stood for if my life depended on it (“garment on hanger”, if you’re curious). I spent most of that week furiously scribbling notes and wondering what in the world I’d gotten myself into.
  • About a year later, we needed to build our own fulfillment center in Reno. My understanding at the time was that a typical FC startup project took 6 to 9 months. We had 10 weeks to go from an empty building to shipping packages — and we got it done. To me that was a testament to what a small, tightly focused, extremely motivated group of people can do. It was a lot of work, with not a lot of sleep, but in the end it was worth it.

How is that different from now?

Things are much, much less hectic nowadays. We still move fast and set aggressive goals, but we don’t have to burn ourselves out to achieve them. The team is also bigger now, so there are a lot more hands to help carry the load.

What’s a typical day like for you?

I usually get into the office at around 10 am. First off, I usually grab a cup of coffee and check email. Then I give our API monitoring charts a look to make sure everything’s healthy.

99% of the code I work on nowadays is in Java, so once I’ve confirmed that everything’s humming along I’ll fire up IntelliJ and get to coding. Somewhere between 11 a.m and 1 p.m. I’ll take a break for lunch, then back to coding for a few more hours before our daily afternoon standup meeting. After that, more coding, till around 7 p.m. when I head home.

We’re definitely fans of the “ship early, ship often” mantra. It’s not at all unusual for me to push 3 or 4 different builds to production over the course of a day. Of course, there are also plenty of days where I’m heads-down working on larger changes, but we try to keep our changes small enough, and the barrier to releasing new code low enough, that we don’t go dark for long stretches of time.

What gets you excited about working at zulily?

There are so many things:

  • The team is absolutely top-notch. I’m surrounded by smart, talented people, both on my immediate team and across the entire organization. I learn something new from my coworkers every day.
  • We move fast and try new things. Sometimes they work, sometimes they don’t, but every time we learn something new.
  • The Relevancy team’s mandate is to figure out how to quickly and accurately surface the most engaging content for our members. We’re continually searching for ways to improve our systems, either by trying new and novel recommendation algorithms, or by increasing our capacity and reducing the time it takes our recommendations to update in response to user behavior. It’s a fascinating space that combines machine learning with hard-core engineering for scale. I love it.
  • I’ve worked at places building packaged software with 9-to-12-month release cycles. It’s disheartening to put that much effort into a project, just to see it languish on a shelf somewhere because the customer can’t (or won’t) deploy it. Our team is the polar opposite of that-we push new code to production several times a day. This creates an incredible virtuous cycle. The barrier to pushing code live is low, which means you do it more often, which means each change is small, which means it’s both easy to verify and easy to roll back if something goes sideways. With such low friction, we’re constantly pushing forward, constantly improving our service, creating a much richer, more engaging experience for our members.

zulily Hack Day Video

Four times a year, all of zulily tech stows away our current project work and we engineers get to hack on whatever we want all day Friday (though, sometimes teams have so much fun they roll through the weekend).  Come Monday, we present our projects to the entire company and someone is crowned the Hack Day Champion by our CEO (along with shiny, shiny trophy bling to decorate the winners desk).

Over the last four years, in fact, many of of the zulily features on our site today originated as zulily Hack Day ideas.  Go engineers!

This last Hack Day, we took a video camera around to capture some of the zulily culture and the fun we have!

Interested in learning more about Hack Day and joining our team?


Technical History of zulily

When I started at zulily in late 2010, the technology team was small but growing. As you can imagine by looking back at our growth over the past few years, zulily was not just your typical startup. In 2010 the business was starting to ramp up and we were figuring things out as we went.

Early technology and trade-offs

Before I joined, zulily had chosen a full-stack implementation of a commonly used open-source e-commerce platform and many of our processes were run out of spreadsheets or online documents. Also, the philosophy of the company regarding software development was already in place (more on this later).

As with any startup you need to make the technology choices that will enable growth, but not over-design to the point that technology becomes a bottleneck. From the beginning, the technology decisions had been made in order to facilitate growing the business very quickly. However, by the time I joined the team was starting to scale out of some of these early decisions and had to think of new ways of doing things.

Continue reading