Speero’s Post

Problem and insights: Customer surveys and exit polls showed that website promos are important. But too many will ruin the experience. Heatmaps showed no interaction with promos. Hypothesis: By relocating the promotions tile, we will reduce banner blindness and products are shown faster on the page. A new placement of the promotion stands out more from products and this will lead to more conversions. Outcome/Takeaway: Winner +1.73% in Transactions. Quite noticeable on mobile, where users could now see products faster (without the banner at the beginning). Next steps: Implement & Iterate. An iteration would be to try another position that works more positively for both devices.

For Experimentation Live + Unite ‘24, the agenda revolves around table discussions, not keynotes. Curated convos where you’re grouped with people with the same issues and goals. Keynotes are there to start conversations, not steal the limelight. We’ll give you a hand-curated list of who to meet. We’ll ask you about your issues, goals, and priorities. We’ll find people who have them as well. You all get together and discuss them. Find solutions and mingle. Of course, you’ll still hear great talks: Lukas Vermeer, Luke Mortensen, Diana Jerman, Chad Sanderson, Melissa Weiss, Sidhant Bakhru, Stewart Ehoff, Shanelle Mullin, and more. We’re also running workshops on the day before: 1 — Ronny Kohavi: Advanced topics in practical A/B testing (P-values misconceptions, power and false positive risk, why give positive results a haircut, when and how to correctly peek). 2 — Various practitioners: Application of AI in experimentation (how to use AI for ideation, hypothesis, problem statement and exploration, discovery). Experimentation Live + Unite 2024. October 10-11, 2024. Optional workshop day on Oct 9. By Speero and Kameleeon. See you there. (Link in comments)

We've transformed Speero’s A/B Test Calculator into a handy Chrome plugin. Now you can calculate your MDE straight from a URL, do quick manual calcs without flipping tabs, and even sync it up with your GA4 account. And nope, we’re not snooping into your data - that’s not our game. Ever find yourself puzzling over questions like: What should my MDE be? How large a sample size do I need? What’s the ideal duration for my test? Well, the Speero A/B Test Calculator plugin is here to solve those riddles, right in your browser. And guess what? It's totally free. But wait, there's more on the horizon. We’re gearing up to evolve this plugin into a complete version of Speero's original A/B Test Calculator. Think of the one on our website, but with added powers like scraping capabilities for your testing tools, and a whole lot more. Link in comments!

— Experimentation maturity hasn’t changed in 3 years. — Only 19% of companies have role-specific goals that can be correlated with the organization's main goals. — Core subject-matter-expert teams aren’t always needed. — 60% of brands don’t have a well-resourced set of research methods and sources to feed the ideation process. — Only 11% of businesses strongly agree they have program-level metrics in place to assess the health of their experimentation program. Where did we pick this? From our Experimentation Benchmark Maturity Report 2024 (with 206 respondents). If you want to find out more (18 findings!), download the report yourself. Link in comments.

How many times did you have trouble adding user research to your experimentation program (on an ongoing basis)? But categorize all research initiatives into one of these 3 categories, and you can plan your research better. Speero’s Research Objectives Blueprint™ speaks about different research types you can (ideally should) incorporate into your experimentation programs on an ongoing basis. Also, help you plan those kinds of research more effectively. At its core, this blueprint maps out three main research types. 1) Exploratory research—evaluating and exploring the overall user experience. Its goal is to identify barriers to conversion and key areas of opportunity and generate new ideas. 2) Focused research—digging deeper into specific issues, themes, topics, or opportunities. Sometimes you need to understand the problem better before you can fix it. This is where focused research helps. 3) Validation research—gathering data to validate concepts or solutions. As you can see from the blueprint, A/B testing can serve as a validation method, but you can use other validation methods as well. In this blueprint we’ve mapped out typical research methods for each research type or objective. But… You need to keep in mind some research methods like usability studies are quite flexible, and you can use them effectively for different types of research objectives. We’ve also mapped out the typical cadence of research. Now the real cadence is going to depend on how quickly your company can action outputs from recommendations to research. This might be outside of your hands, with factors like design and dev perspective, traffic for A/B tests, etc. But this is typically what Speero would expect of 1 or 2 large exploratory research projects every year. You can download the Research Objectives Blueprint from our website (link in comments). Send a DM on LinkedIn to Emma Travis, Martin P., or Ben Labay for any feedback, suggestions, or questions about the blueprint. Thanks to the blueprint owner, Emma Travis!

We run four types of workshops for spotting issues, generating ideas, and improving the client’s program. Here’s why they’re vital and how to run them. Test Ideation Workshop — for creating a broad group of test ideas for a specific problem. For this workshop, you need to prepare the info and problem, brainstorm ideas (we do this in Miro), and evaluate/write hypotheses. Focus on a specific problem, spend 2-3hr on the workshop, and bring people from other teams to this to nurture experimentation culture, buy-in, and awareness. Goal Tree Mapping Workshops let you build accountability, reporting, and ROI tools for your program. You map out all the relevant metrics in this workshop, then use the results for further OKR-ing or strategy mapping. For the workshop itself, discuss with the team: — What’s the company’s bigger goal for this year? — Map out 2-4 main goals. — Now ask yourself: how can I move the needle on them? — Identify engagement and tactical metrics that can contribute to the main KPIs Experimentation program advisory clinic lets you improve your program maturity, and increase velocity and impact with the same headcount. Experienced agency and in-house industry consultants assess a program and provide "hard-earned, 'lived'' advice. In the end, our clients end up with structured advice on where/how to scale their program. UX heuristic audits let you find issues and friction in the online customer journey. The UX heuristic audit workshop assesses the user experience against a set of industry standards, or “UX principles”. For this kind of workshop, you’ll need somewhere between 12 to 15 hours to: — Assess the customer journey for issues — Create a list of opportunities and problems to address — Create a list of insights and themes to address with tests or JDIs

"Running A/A tests is such an eye-opening exercise for product and marketing managers. In a nutshell, an A/A test involves setting up an A/B test where the variants are identical, showing no differences. And even though they're recommended for technical reasons (validating that the instrumentation setup is accurate) I find them so valuable for educational purposes. It’s quite common to observe large, even statistically significant, differences in the early days after launch. The kind of differences that can cause premature excitement and lead to declaring "imaginary winners." However, when you let them run long enough, these differences eventually even out (as long as there are no underlying issues) and every possible explanation about why one of the variants was better just vanishes. Which is a great exercise to understand how often we attribute outcomes to specific features, tactics, or behavioral principles when, in fact, they may simply be the result of random chance. Like it or not, they make you realize your ideas and changes might not be that impactful after all. Go run an A/A test on your website, ad creatives, or emails. It will make you see things differently the next time you try to determine causality." By Carlos Trujillo. In this week's Speero's best personal post — where we celebrate the opinions and innovations from experts in data and experimentation.

The customers coudln't decide if the product is right for them. We tested placing the most helpful positive review so they get more information and complete more transactions. Problem and insights: Customers are struggling to find/decide on suitable products for them. The review content is quite down the page. Hypothesis: If we place the most helpful positive review in a more prominent location then more visitors are likely to see the information and decide if the product is for them. Outcome/Takeaway: Winner - Implement on Desktop, Iterate on Mobile The uplift in key performance metrics reflected in positive revenue metrics was visible only on desktop devices. On mobile, the situation was reversed, probably because this new PDP layout forced the product info further down the page.

What do you want to discuss with your peers? What are some challenges/priorities you're working on right now? These are the two most important questions we ask for attending EXL + Unite ‘24. For a reason. This way, we can create our famous table discussions for the event. Small groups of people talking and mingling, ‘matchmaked’ by sharing the same challenges and priorities. You all sit at small tables. Talk about the issues and priorities troubling you specifically with people who also have them. Find solutions and meet each other. Keynotes are rocking as well: Lukas Vermeer, Luke Mortensen, Diana Jerman, Chad Sanderson, Melissa Weiss, Sidhant Bakhru, Stewart Ehoff, Shanelle Mullin, and more. Experimentation Live + Unite 2024. October 10-11, 2024. Optional workshop day on Oct 9. By Speero and Kameleeon. See you there. (Link in comments)

Sunk cost fallacy —a tendency to continue investing in a decision because we have already invested time or effort into it. A type of psychological principle used in different parts of digital experience: — Multi-step forms (easier to fill the rest once you’ve already filled a part) — Interstitial pages (provide a small piece of information or request an action without taking the user to another page) — Free trials, demos, or downloadable links (that lead to the ‘big’ conversion) — Foot-in-door tactic (make a modest request or sell the cheaper product, then follow up later with a larger request to increase the chances of succeeding with the larger request) You can also find countless examples of sunk cost fallacy in real life: — We stay in romantic relationships simply because we’ve spent so much time and energy in them. — We watch boring movies until they end simply because we already watched a part of them. — We decide to go to the concert sick because we paid the ticket $50, even though by going there we wouldn’t recover the investment. We base our decisions on past investments rather than on future benefits or drawbacks. Micro-commitments can often lead to a big commitment/conversion.