Usually, once a company gets serious about the idea of experimentation, they hit the same fork in the road. Do we run it in-house, or bring in an agency? It sounds like a procurement decision but I assure you it isn’t. Get it wrong and you’ll spend 18 months and a significant budget learning that lesson the hard way.
Here’s my honest read on both sides.
The case for building in-house
The strongest experimentation programs in the world are internal. Think Netflix & Booking.com… these aren’t companies running experiments through an agency on retainer. They built capability, embedded it into how decisions get made, and the results compound over time because the institutional knowledge stays inside the business.
That last part is the real argument for in-house. Insight accumulates. You run an experiment, it loses, and you learn something specific about how your customers behave. Then the next test is shaped by that. Then the next one. After a year of disciplined experimentation you have a picture of your customer that nobody outside the business can replicate, because they weren’t there for all the tests that didn’t make it into a case study.
When that knowledge lives with an agency, it leaves with them. You might get a report but you don’t get the nuance. You don’t get the researcher’s memory of the edge case in segment three that informed the hypothesis. You get a slide.
Ownership of insight is the strongest argument for keeping this internal. If your experimentation program is genuinely going to inform product decisions, UX direction, and commercial strategy then the people running it need to be embedded in the business, attending the planning meetings, sitting with the product teams, knowing where the bodies are buried.
The case for going agency
The problem with building in-house is that it takes time to do properly, and most businesses don’t have that runway.
A competent agency brings something you can’t hire overnight:
- A team with range.
- A strategist.
- A statistician.
- A developer.
- A researcher.
You’re not relying on one person who might leave, get sick, or simply doesn’t know what they don’t know yet. That redundancy matters more than people admit when they’re building out a program for the first time.
Speed to competence is real. If you need results in six months, building from scratch internally is a harder path. An experienced agency has seen the mistakes before. They have frameworks, they have tools, they’ve already learned from the kind of errors that would cost you a full quarter to discover on your own.
The challenge is the incentive structure. An agency that runs your experiments, owns your insights, and invoices you monthly has a financial reason to stay. Not necessarily a sinister one, but the retainer model doesn’t naturally push toward building your capability. It pushes toward maintaining its own. That’s not an accusation. It’s just how the economics work.
The trap nobody warns you about
I’ve seen this play out directly. Not in CRO specifically, but in SEO, which has almost identical dynamics when it comes to the build vs. buy question.
A restaurant SaaS company wasn’t sure whether to go agency or hire internally. They decided to save money by bringing in a couple of junior people. And the problem wasn’t effort, everyone was trying. The problem was that they only knew what they knew. Junior practitioners have limits that aren’t visible until they’re already constraining the program. They optimise within the boundaries of their training. They don’t know what questions they’re not asking.
Worse, the person doing the hiring had the same limitation. They couldn’t assess whether the people they’d brought in were genuinely strong, because they didn’t have the depth to evaluate it. You end up in a situation where nobody in the room can tell the difference between good work and adequate work. And adequate work, in CRO, produces marginal results at best.
Hiring junior to save budget isn’t the cheap option. It’s the expensive option with a delayed invoice.
The enable-and-leave model
There’s a third path worth knowing about although I don’t come across it often. It revolves around a specific premise:
CRO should be owned in-house, and if you’re working with an external partner for too long, that partner has failed you.
The model is:
- Come in
- Run the first experiments
- Build the internal capability
- Leave.
The goal is to make the engagement short on purpose. If they’re still working with you in two years, something’s gone wrong.
That framing cuts against how most agencies operate, and it’s honest about why. Retainer agencies benefit from your continued dependence. They own the workflow, they own the insights, they run the tests. You get the outputs but you don’t accumulate the capability.
I find the third model genuinely interesting because it aligns the incentives correctly. The measure of success isn’t how long the engagement runs. It’s how fast the internal team can fly without support.
The honest verdict
There’s no universal right answer here. Anyone who tells you there is, is selling something.
Where you are in the maturity journey determines everything. A business that has never run a structured experiment, has no statistical literacy in-house, and no dedicated resource is not ready to build an internal program from scratch. The ground isn’t prepared. Go agency, but go in with a clear exit plan. What does internal capability look like, and what’s the trigger point for the transition?
A business that’s been experimenting for two years, has a functioning roadmap, and a team that understands the methodology doesn’t necessarily need an agency. They need to build, protect the institutional knowledge, and stop paying for the privilege of someone else owning their insights.
The worst outcome isn’t choosing agency or choosing in-house. The worst outcome is choosing the cheap option at the wrong stage. Staying on an agency retainer when you have the internal capability to own it costs you margin and the compounding value of institutional knowledge that should have been yours.
The question isn’t agency or in-house. The question is where are you right now, and what does the next stage actually require?
If you’re not sure how to answer that, start there. The Experimentation Maturity Quiz takes about five minutes and gives you a clear read on where your program currently sits. And if you have a team but want them to be able to confidently own experimentation, we have a 12-week accelerator program designed to raise the standard of experimentation across the board.
