Most experimentation teams can’t answer a simple question. Their VP asks “what’s the return on this testing tool?” and the team either deflects and talks in vague terms about the lessons learnt, or they share a number so clearly inflated it destroys their credibility in the room.
Neither approach works. And the question is fair. These days, the cost of an enterprise experimentation platform runs anywhere from £40k to £300k a year once you add licences, engineering time, and headcount. That’s real money. Leadership is allowed to ask where it goes.
This article is for the people who need to answer that question properly. Not with a story. With a calculation. One that holds up when someone presses on it.
Why Most Experimentation ROI Cases Fall Apart
The standard approach goes like this. Someone pulls together a list of winning tests from the last 12 months, adds up the projected revenue lifts from each one, and presents the total as proof. “We ran 40 experiments and generated £2.3 million in uplift.” The number sounds impressive until someone in finance asks how the projection was validated, whether the wins were ever checked for durability, and why the losses aren’t in the calculation… At that point the case collapses.
There are two problems with that approach. The first is attribution. When a test wins and you ship the change, the subsequent revenue improvement is driven by dozens of factors:
- seasonality
- marketing spend
- product changes
- price shifts
Crediting all of it to the experiment is bold af! The second problem is selection bias. Showing only wins and ignoring the losses produces a number that nobody with financial literacy will trust. A real ROI case has to account for the L’s as well as the W’s.
The harder truth is that most CRO programs confuse activity with value. Running 40 experiments isn’t evidence of ROI. It’s evidence of activity. The question is whether those experiments produced decisions that wouldn’t have been made otherwise, and whether those decisions moved a metric that connects to revenue.
What ROI Actually Means in an Experimentation Context
ROI is a ratio. Return divided by investment, expressed as a percentage or multiplier. If you spend £100k on something and it generates £400k in value, your ROI is 300% or a 4x return. Simple. The complexity in experimentation comes from defining “return” in a way that’s both honest and complete.
There are two categories of return worth accounting for. The first is direct revenue impact from shipped winning experiments. The second is cost avoidance, which is the money you didn’t spend building things that would have failed. Both are real. Both belong in the calculation. Most teams only talk about the first one, which means they’re undervaluing their program and making a weaker case than they need to.
Let’s take cost avoidance seriously for a moment. A mid-size e-commerce company runs a test on a proposed checkout redesign before a full engineering build. The test loses. The full build would have cost £80k in engineering time and taken three months. That loss just saved the business £80k and a quarter of a year’s engineering capacity. That saving is ROI. It doesn’t show up in revenue figures but it absolutely belongs in your calculation, and it’s one of the most powerful arguments you can make to a product or engineering leader.
The Actual Calculation: How to Build It
Here’s the framework. You need four inputs:
- the annualised revenue impact of shipped winners
- the estimated cost avoidance from prevented bad builds
- the total cost of your experimentation programme
- …and a credibility discount rate.
That last one is the piece most teams skip, and it’s the piece that makes the difference between a number that gets challenged and a number that lands.
A credibility discount is a deliberate reduction you apply to your revenue projections to account for the fact that not all uplift is durable, not all measurement is perfect, and some attribution is uncertain. A reasonable discount rate sits between 30% and 50% depending on how mature your measurement setup is. If you have solid holdout groups and long-term tracking, go 30%. If your analytics are less reliable, go 50%. Applying a discount makes your case more conservative and therefore more credible. It’s a counter-intuitive move. Showing a smaller number on purpose, with your reasoning visible, will generate more trust than an inflated number ever will.
The formula looks like this. Take your total projected revenue from winning experiments over 12 months. Apply your credibility discount. Add your cost avoidance figure. Subtract your total programme cost. Divide the result by the total programme cost. Multiply by 100 to get a percentage. That’s your ROI.
Here’s an example with real numbers. Imagine an experimentation team runs 30 experiments over the year. Twelve of those win and get shipped. The combined projected annual revenue uplift from those twelve winners is £600k. You apply a 40% credibility discount, which brings that figure to £360k. You also identify three large builds that were tested and killed before full development, saving an estimated £120k in engineering cost. Your total return is £480k. Your programme costs for the year are £180k, covering the platform licence, one dedicated CRO hire, and an allocated portion of engineering time for implementation. Your ROI is: (£480k minus £180k) divided by £180k, multiplied by 100. That gives you a 167% return, or roughly a 2.7x multiple on investment.
That number is defensible. It’s been discounted. It includes cost avoidance. It has a clear methodology. You can walk someone through every input. Much more credible!
The Metrics That Build the Case
Revenue impact and cost avoidance are the headline figures, but there are supporting metrics that make the story more complete and give you answers to the follow-up questions leadership will ask.
Win rate matters because it shows your team is prioritising well. A 30-40% win rate on properly controlled experiments is solid. Below 20% suggests either poor hypothesis generation or weak prioritisation. Above 60% usually means you’re testing safe, marginal changes and avoiding the experiments that could generate meaningful learning. Neither extreme is good. Win rate alone says nothing about ROI, but in the context of your ROI case it shows whether the programme is being run with discipline.
Experiment velocity is the number of experiments completed per quarter. But track this carefully. High velocity with low quality is worse than lower velocity with rigorous design. What you actually want is something closer to validated decisions per quarter… the number of experiments that reached statistical significance and produced a clear, documented decision.
Test-to-ship ratio tells you what proportion of winning experiments actually got implemented. This is a critical metric that almost nobody tracks. If you win 15 experiments but only ship 6 of them, you are leaving most of your ROI on the table and the fault is almost certainly organisational, not in the testing programme itself. A low test-to-ship ratio is the sign of a culture problem or a resourcing bottleneck. It’s worth surfacing because it shows leadership exactly where the return is being lost.
Average revenue impact per shipped winner is useful for benchmarking over time. If that number is growing, your team is getting better at identifying high-value opportunities. If it’s shrinking, you may be running out of obvious optimisations on your core funnels and need to expand scope.
Presenting This to Leadership
The biggest mistake I see when teams present experimentation ROI is that they present a number without a story and then a story without a number. You need both, in the right order.
Lead with the number. Then immediately show the methodology. Then walk through two or three specific experiments as concrete examples:
- One win with a clear revenue tie
- One loss that saved significant build cost
- …and one ambiguous test that produced important learning even without a clear winner.
That last one’s important because it demonstrates that your team understands why you experiment in the first place. You run experiments because you don’t know the answer. The value is in finding out. Sometimes finding out means learning your hypothesis was wrong, and that is still a return on investment.
The second thing to do differently is to make the comparison explicit. What would happen if you didn’t have the programme? If leadership decides not to renew the platform, the business doesn’t stop making product decisions. It just makes them without evidence. Show what that alternative looks like. A major feature built without testing, launched to the full audience, performing 15% below expectations, costs more than your annual programme budget in lost revenue alone. The question is never “is this programme worth the money?”. The question is “what’s the cost of making decisions blind?”
Frame your ROI presentation around annual budget cycles. The worst time to make this case is when you’re already in a cost-cutting conversation. Build the case proactively, quarterly if you can, so there’s a documented record of returns before anyone starts questioning the line item.
Common Mistakes That Undermine Your Credibility
Projecting without decay curves is one of the most common errors in experimentation ROI. When a test wins and you project the revenue uplift forward, you need to account for the fact that the impact of most conversion changes decays over time as the market, competition, and customer behaviour shifts. Projecting a win as if the uplift is permanent and stable overstates value significantly. A more honest projection applies either a decay rate or caps the projection at 6 to 12 months.
Counting learnings from losses as “soft ROI” without quantifying them is another pattern that erodes credibility. It’s not that the learning isn’t real, it’s that saying “we learned a lot” without connecting that learning to a subsequent decision is not an argument. If a losing test produced insight that shaped a winning one three months later, you can make that connection explicit and defensible. Otherwise, leave it out of the ROI case and put it in a separate narrative about programme quality.
Padding the cost avoidance estimate is a temptation worth resisting. It’s easy to inflate the engineering cost of hypothetical builds that were killed by a test. Use your engineering team’s actual estimates, or use actuals from comparable past projects. If you can’t ground the number, don’t use it. An unverified cost avoidance figure will get challenged and the challenge will contaminate confidence in your whole calculation.
When the Numbers Don’t Justify the Platform
This is the conversation most CRO consultants won’t have with clients, so I’ll have it here.
Sometimes the ROI genuinely doesn’t justify the current platform spend.
If you’re on a £200k per year enterprise tool and running 12 experiments a year with a team that doesn’t have the bandwidth or the traffic to reach significance quickly, the numbers will not add up. That doesn’t mean you should stop experimenting. It means you should probably be on a different tool, at a different price point, and expanding capacity before you expand the platform.
The calculation is useful here because it’s clarifying. If you do the honest version of the ROI calculation and the number is below 1x, that tells you something real. Either the programme is underperforming and needs a structural fix, or the organisation isn’t ready for the investment level it’s currently carrying. Both are answerable problems. But you can only answer them once you’ve stopped pretending the number looks better than it does.
The goal isn’t to justify the programme. The goal is to understand its actual return so you can either defend it accurately or improve it deliberately.
Where to Start
If you haven’t run this calculation before, the first step is an audit. Pull every experiment from the last 12 months. Categorise each one as:
- won and shipped
- won and not shipped
- lost
- …or inconclusive.
For the shipped winners, pull the actual revenue data from the period post-implementation and compare it to a comparable prior period with appropriate adjustments for seasonality. For the losses, work with engineering or product to estimate the build cost of the change you decided not to make. That gives you the raw inputs. Apply your credibility discount. Add cost avoidance. Run the formula.
The second step is to start scoring your experiments by business impact before you run them, not after. This is where most programmes leak value. Teams prioritise by what’s easiest to test or what someone has a strong opinion about, rather than by what could move a metric that actually connects to business outcomes. Building a scoring system that forces you to estimate business impact upfront means your ROI case gets stronger over time as the programme naturally gravitates towards higher-value work.
That’s exactly what the Impact Scorecard is built for. It helps you score and compare experiments by business impact before you commit resource to them, so the experiments that reach the top of your backlog are the ones most likely to generate returns worth reporting. It also has a simple way to track metrics that leadership care about so when your ROI review comes around, you’re not scrambling to retrospectively justify what you worked on. The case builds itself because you built it into how you prioritise from the start.
