How Booking.com Uses 1000’s of Experiments To Build a Culture of Scientists
The Company
Booking.com is the world’s largest price comparison site for travel expenses such as flights, hotels, car rentals, excursions, and more, offering discounts and rewards for consistent usage.
The Experiment Pt. 1
According to Lukas Vermeer, Director of Experimentation, they run over 1000 experiments at a time, at all times. More impressive, they run tests for each individual website visitor, in order to get specific learnings. This means, assuming each test is an A/B test, they have 2¹⁰⁰⁰ experiments active right now!
Rather than putting all their tests in one basket, they diversify by setting up sub-tests for each to find the perfect sequence of listings for each user, figuring out their preferences along the way.
The Experiment Pt. 2
In order to maintain such a high number of experiments, Booking.com’s team needs to have the right mindset. Stuart Frisby, Booking’s Ex-Director of Design, once set a few guidelines for their culture:
- No HIPPOs (highest paid person’s opinions)
- Every decision is a democracy, but test every decision
- Trust your tools
In addition to having a designer on every development team and a forced role change every 10–12 months, employees understand the business from a holistic perspective, making them more able to come up with new ideas, hypotheses, and, of course, tests. Then it’s all up to your software partners to run experiments i.e. VWO, Optimizely, Mailchimp, or Marpipe ;D
The Results
Booking.com was born in 1996. In 2005, Priceline bought them for $133mm. By 2011, they had net profits over $1.1 Billion. Today they make nearly $5 Billion in profit and publicly trade as BKNG, after acquiring most of their largest competitors (Kayak.com, OpenTable, momondo, etc.).
Their results are clear. Experimentation breeds success in business.
The Application
While adding a designer to every dev team and avoiding centralized executive decision-making may be hard for companies with fewer employees than Booking’s 26,000+ strength staff, one guideline is easy to follow; it’s something we say all the time at Marpipe:
Test Everything!
