Microsoft. Netflix. Airbnb. Google. All run 10K+ eXperiments every year. The rest? 1-2 eXperiments per week with lots of manual effort. Welcome to the eXperimentation Gap. Here’s how you can jump over it and outperform your competition with well-oiled eXperimentation programs that deliver results over and over. Microsoft and the rest of FAANG don’t only lead in design, innovation, and marketing. They also lead in eXperimentation, running over 10,000 eXperiments per year. This lets them precisely quantify the impact of 80% of ideas. In lots of cases, they even get profound results. Bing ran a simple eXperiment of re-adjusting the ad header layout. And increased its annual revenue by 12%. Yet, the rest of the companies live in a grim reality. Rarely eXperiment. Have to enter lots of manual effort. Got bottlenecked processes. Old-school, less relevant statistical methods. The disarray of third-party tools. Colliding eXperiments. Buggy eXperiments. Data interoperability. eXperimentation that doesn’t scale. eXperimentaiton that isn’t mature. How can you fix this? 🔷 Develop transparent eXperimentation platforms where everyone in your organization can design eXperiments and review the ones running. Include eXperiment design, outcome, and hypothesis. 🔷 Make sharing eXperiments and their results easier in Slack or email. We at Speero got an automatic channel for this. 🔷 Incorporate systematized workflows that everyone understands. At Speero, we got SOPs for 90% of our processes. This is how you jump over the eXperimentation gap and develop a well-oiled, mature eXperimentation program that lets you quantify the effect of every eXperiments whether for revenue or learning. The eXperimentation Gap is one of the grimmest divides in business. The original article is from Davis Treybig. Find the link for reading in comments.
Experimentation is currently just the domain of data nerds and technicians. I can also bet it’s the same in the companies that are hailed as the “holy grail” to some extent. The reasons the rest of the businesses haven’t caught up is they’re stuck in a vicious cycle. How experimentation (or rather CRO ) is brought in to an Organization and how it never escapes it because of a ‘faulty’ setup that is heavily skewed towards practitioners and their siloes.
Any stats on the percentage of their “experiments” that are merely automated monitoring after code elevations for error detection vs experiments meant to inform a future decision point?
There's a massive challenge to getting to experimentation at scale at highly regulated companies - it can be done, but it requires intentional, locked-arms collaboration between many groups to achieve a run state. I've done it previously and I'm in the middle of that challenge now in my current role. It's one of the hardest things I've done - pulling the operational part through (i.e., how do we enable experimentation at scale WHILE ALSO staying compliant, maintaining audit-ability, and creating the operating run rules around all that) is a herculean effort.
You'd have to add Booking.com to that list wouldn't you? Apparently they have automated testing for every single release, even bug fixes
🔷 Make sharing eXperiments and their results easier in Slack or email. We at Speero got an automatic channel for this. This needs a system built for this purpose and that is Datatinga (with integrations into Slack and email). And it's not the experiments themselves, it's the insights that we gain from them. Combine that w/ simplicity and insights from other types of initiatives and you've suddenly got an insights forum that stretches farther and wider across an organization, where all insights are consistently laid out, and easy to interpret fast. If it ain't easy, it doesn't happen.
It's not so much the number of experiments that set them apart, but how product leadership and an organisation approach product development. Companies focussing on outcomes and the impact they are driving rather than outputs, get into an experimental mindset much easier.
Link to article: https://towardsdatascience.com/the-experimentation-gap-3f5d374d354c
Adobe has an extremely rich experimentation culture & a lot of strategic initiatives are first tested before they go out in the real world. And yea we run 1000s of tests across the board without necessarily automating everything as sometimes insights are worth a lot more than just the primary kpi result.
These large companies also have a lot of teams running experiments so maybe the total number of experiments is not the right metric to use to compare. At Booking we have over 200 teams running experiments..... 😀 And wait, what. Why is Booking not in you list of companies running 10K+ experiments? This is continentism ;-)
Optimising Experimentation: Industry leading Expertise, Coaching and Mentorship
1yTrue. There’s a graph to be made here that shows most people are only testing enough to support their software vendors, not actually help grow their companies. The average testing may not even be as high as you suggest (1-2 a week) as I think this is generous 🤣