Ben Labay’s Post

Attention AB testing geeks! EXL+Unite conference in October will feature one of the, if not the most, advanced workshop on AB testing you'll ever see. I'm excited for this workshop from Ron Kohavi. He's spoken at CXL Live in the past, this year we get him to dive deep into: "Advanced topics in practical AB testing", a half day, interactive workshop. The course covers topics not usually covered in introductory courses, including Ronny's Maven class on A/B Testing and his book. These advanced topics include: - P-values misconceptions, - Power, and False Positive Risk - Why you should always give positive results a haircut  - Success rate estimation, and success of ideas vs. experiments - The cost of false positives - TOST and early stopping – when and how to correctly peek - Bayesian and Frequentist approaches - CUPED – variance reduction - Risks of unequal variants and rare events - Long-term holdouts - Simpson’s paradox - Randomization strategies: front door or at trigger point, ramp-up - Cultural issues: can I test that? OFAT or major redesigns Now is the best time to get tickets, if you know you're coming, go get tickets using 'Earlybird35' as the coupon code to get 35% off.

Speero Scorecard Series: Sometimes a test is not about 'conversion'. Search is all about 'product discovery', Carlos Trujillo has it in the Ecom testing goal pillar of 'Merchandising'. Quite straight forward, and an easy test if you're not doing this already. Perhaps a JDI (just do it) as there's not too much risk there right?

Experimentation Live + UNITE conference table moderator list is growing! Here are newly added experimentation leaders: Liona To from lululemon Mike Grabowski from Boats Group (ha, we have cars.com and boats.com people, my inner child is quite excited) Divya AL from Target Robbie Williams from Dr. Squatch Cory Peterson from LED Lighting Inc. Soumya Gujjar from Tipalti Marc Miller from NRG Energy Alex Rolleigh from International Rescue Committee Mike De Boer from Xfinity Kendra Bones from YETI and Travis Brooks from Netflix We have the core set of moderators, but need about 10 more I think, but now we'll be quite picky as we try to round out roles and industries to have a nice balance of folks from different perspectives. This month is the time to get tickets, the cheapest they will be with 'Earlybird35' coupon code, 35% off.

Research can be so much more than just research. It can be a direction, a strategy. "Strategic UX Research is a force multiplier. Organizations get more done with the same number of people. It delivers superpowers to the senior executives, enabling their organization to be a visionary market leader because the entire team has a better picture of how their product improves people’s lives." - Jared Spool Speero's 'ResearchXL' service is so much more than just research. I've always hated the name, it does such a disservice. It is a process and method to action. It develops a customer- and business-backed strategy for improving experiences and making more money. The tactical research is just the data and info gathering. It's how it all gets actioned that is where the power is. We've been experimenting with 'reskins' of RXL: - Aquisition RXL - for aligning PPC teams with CRO teams - ICP Research - for message/copy alignment - SaaS pricing research - for tackling the heart of the product strategy for SaaS - Ecom product page research CRO teams and agencies don't do enough research generally. And what is done doesn't get actioned enough.

We talk about button colors a lot, but what about product packaging color? We tested this via a Fake Door test, with a beauty & cosmetic brand's product, changing the packaging with the image only. Customers were to receive the white packaging even if purchased the variant. No difference in measured transaction or RVP. I like these types of tests. Easy, and in this case you could say not impactful, right? Well, we did save a TON of investment into new packaging ;)

The world of AB testing is undergoing a dramatic transformation. Here's a look at the two biggest trends shaping its future: Big change #1. From Tactics to Strategy: It's like AB Testing gets a rebranding AB testing is no longer just about tweaking buttons and colors. It's becoming more strategic, with a focus on experimentation. We see this a ton in CRO teams of 1 growing to departments and functions embedded into other teams across an org. This means testing with different teams (think SEO, product, and sales!), across new channels, and for reasons beyond just small-scale improvements. It's a more innovative and operationalized approach. This shift from service to function within an organization isn't without its challenges. It can clash with existing company cultures built around rapid product releases. Here's how to navigate this transition: - Collaborate across teams: Break down silos and involve more departments in the testing process. - Embrace decentralization: Move away from a centralized testing approach. - Standardize for success: Create a standardized testing process and governance framework. - **Befriend failure: Develop a culture that learns from failures and uses them to improve future efforts. Big change #2. A Farewell to Cookies: The Rise of First(&Zero)-Party Data With the decline of third-party cookies due to privacy concerns, the way we use data in AB testing is changing. The focus is now on user experience across the customer journey, including navigation, progressive profiling, surveys, and preference centers. (think 'choose your journey' type of experiences) This empowers customers with control over their data and provides marketers with richer data for personalization and recommendations. Here's how I think to adapt to this 'data-centric' future: - Gather more user feedback: Utilize surveys and qualitative data collection methods, perhaps even leveraging AI for analysis at scale. - Understand your customers: Focus on customer intent and preference classification. - Invest in a modern data architecture: Build a data infrastructure that uses data warehouses and warehouse-native tools. - Align testing with business goals: Ensure your testing strategy aligns with core business metrics like pricing models and subscription types.

You have a testing tool, you have analytics.... but do you have an "experimentation management and analysis platform"? Here's my POV on how to set one up: (btw, no, it's not just 'Airtable' nor is it your testing tool + BigQuery) Step 1. Benchmark the program maturity and capabilities across teams. - mainly needs to have A to Z questions, to identify gaps and critical drivers. (e.g., where are you now vs where you want to be...). This is the heart of usefulness of Speero's Program Maturity Benchmark Audit. - capabilities are the key here. Esp ones that speed the flywheel of feedback and insight generation. (e.g., Leadership Buy-In, Evangelism, Collaboration, Personalization, Targeting, Strategy, Test Complexity, Tool Investment, Tracking and KPIs, Research Knowledge base, Hypothesis Generation, Prioritization Framework, Roadmapping, Quality Assurance, Pre-Test Analysis, Communication, etc., etc, etc) Step 2. Plan against a north star for the program - yes have your goal tree map, and Goal vs Driver vs Guardrail metrics (input/output/outcome metrics too) - but also have a 'remit' for your org on what the experimentation program means to you and the team. Purpose & Motivation, Mission & Tenants, Mechanisms to invest in (tied to the capabilities audit above). Step 3. Document language (the heart of standardization) - Nouns and verbs, types and categories... a data dictionary or semantic library for your org. - The MVP is two columns of data, no joke. And it's what's lacking at 90% of orgs and is the 80/20 of good uses of AI btw. - This is the connective tissue across the teams and touchpoints and such that the testing program touches. It's what creates the leverage points for automations and AI to grab hold of. - This is also what Chad Sanderson won't stop talking about, for good reason. You can't align people if you don't have a shared language. Step 4. Standardize artifacts and rituals - If the data language documentation are the notes, the rituals and artifacts (tools) you use are the music that you play. - This is the 'drumbeat' of your org. How your team finds a rhythm. - Speero blueprints are ALL about this step 4. - Not all these tools and cultural practices will make it into the 'platform', but you *shouldn't* build that platform without having this figured out. Step 4. Standardize platform architecture - how the data flows - who sees what, when, and how - what happens, when, and why - tables, Interfaces, and Actions - This step is the actual building of the bespoke 'platform' that is your testing program. Step 5. Test&Learn, with Experimentation Consultants to advise - Like an annual medical check, vitals being velocity and impact, and ultimately your organization's ability to adapt. - this is where you bring in me, Speero to check your work - this is where you bring in consultants like Lukas Vermeer, Craig Sullivan, Iqbal Ali, Simon Jackson, Liam Furman, Roderik Koenders to check your work - do it ;)

CRO and Experimentation teams need to make leadership do their jobs. Leadership should provide core strategic 'problems' for teams to solve. New approaches to customers that the business believes will provide growth. New channels, new products, new approaches with the existing channels and products. The typical ‘CRO’ world: Most CRO testing work is not integrated into business strategy with the customer front and center, it will wreck your brand and turn your website into a casino. It’s short sighted and not impactful. - Tactics and hacks  - Lack of systems  - Siloed work - Little accountability - Poor data management, trust issues Speero’s #ELG (Experimentation Led Growth) POV Experimentation is a core operational model for the business, a method of growth and learning that allows teams to ship fast with certainty, and take ownership and accountability. - Explicit connections to product/brand strategy - Research and test learning systems - Artifact and Ritual systems to build a drumbeat of consistency and trust - Education-first: Training, Workshops, Wiki’s, knowledge base - Change Management consulting for implementing a Center-of-Excellence It's so so much easier to sell and 'read out' on a test program if it has the companies customer north start metric as it's goal post. We need to test to solve strategic problems. A bad conversion rate is not a strategic problem (I define strategy as something where the opposite is not stupid).

"Christmas tree approach to experimentation" ? I just saw this image from MAT DO (@ItsMatDo) on X. His context was from the book "Mastery", by Robert Greene. Mat was talking about one's own use of experimentation, and alluding to the fact that it fails unless there's a strong sense of purpose. Same can be true for an organization. The strength of mission/purpose, the strength of OKR and operational 'drumbeat' systems can determine how a testing function works or doesn't. replace "life's task" with OKR, or Rocks, or 'Strategic activity agenda' and you can have a testing program make a TON more sense.