4 rules for your growth experimentation roadmap
Finding the ideal mix between radical and incremental experiments
My week started with this question:
“How do you define the ideal mix between radical and cosmetic experiments?”
It was a fellow product professional, wondering which were my principles.
This led me to analyse my experimentation roadmap building process. And identify four factors that guide my decisions.
But let’s start from definitions.
Radical experiments
Radical experiments are high risk and high reward tests.
They involve trying something innovative that could potentially lead to a significant increase in performance.
Given that they diverge greatly from the current product experience, they also have comparatively higher chances of failure or imply higher effort.
Indeed, radical experiments typically change multiple factors at once, like product messaging and flow, or channel.
An example of a radical experiment is a totally new acquisition funnel, for instance for a different audience or based on a new logic.
Incremental experiments
Cosmetic experiments, or, rather, incremental experiments, are low risk and low reward ones.
They constitute a step-by-step improvement from the current experience, that builds on the top of what works already. As a consequence, they have typically lower effort and lower impact.
These experiments normally change one factor at the time.
An example is a change in a call-to-action position or messaging in an acquisition funnel, or a new order of its steps.
The mix
It is always difficult to define a good mix of growth experiments, and the risk of not getting it right has multiple implications.
For instance, if you try too many radical experiments, and many of them fail, you do not only miss your goals, but also affect management trust and team morale.
As a consequence, I consider these four factors when building an experimentation roadmap:
Baseline (growth) experiment mix
Business goals (the math)
Growth stage (and saturation)
Team motivation (tactical)
1. Baseline experiment mix
This is where everything begins.
I start with a 70-30 rule of thumb, aiming to deliver 70% of incremental experiments and 30% of radical experiments in a given timeframe.
What matters here is the delivery of initiatives, not the related effort.
It is important to separate the two elements, since radical experiments are typically longer to define (analysis, research, design) and to implement (engineering), while incremental ones are quicker.
As a consequence, the 70-30 delivery mix can translate in a 50-50 resource split, if not 40-60 (or more).
2. Business goals
Your target informs the growth experiment mix further.
Here it is a matter of doing some math.
Given the resources allocated and the expected impact (and probability of success) of each experiment, it is possible to define where you can get, and tweak your mix accordingly.
It is a bit like venture capital investing. Investors know that they are going to have, let’s say, 30% of companies failing, 40% doing decently, 25% doing good and 5% doing great. Or, at least, that’s what they hope for!
3. Growth stage and saturation
This factor is crucial and often misunderstood.
Products (and companies) go through phases of growth, which demand different tactics.
To understand your stage you need to identify the ceiling, which is the maximum value your KPI can get to.
When the level of your KPI is far from the ceiling, improvement potential is big and radical bets are less needed. On the other hand, when you are closer to saturation, you might want to try bolder experiments, to start a new phase of growth.
For instance, we can imagine you are responsible for an acquisition funnel which currently converts at 60% into registered users.
You could argue that the improvement potential is 40 percentage points (which seems a lot!).
The truth is that your ceiling is most likely lower, let’s say 70% or 80%. Indeed, it is not that realistic to convert all the traffic going through the funnel, especially as volumes grow, marketing channel saturate and lower quality users get in touch with your product.
Moreover, as you get closer to your ceiling, you hit decreasing returns on product actions, meaning that more and more effort will be required to achieve a given impact. This creates a better argument for radical actions.
4. Team motivation
Given that radical experiments have less chances of success, doing too many of them can result in decreasing team morale and affect management support.
Teams can get frustrated by trying new approaches that consistently do not work and lose trust in their product managers and leaders.
Despite members of growth teams (engineers, designers, analysts, product managers) are ideally individuals with relatively high risk tolerance, success is still important for motivation.
This might lead you to prioritise, even just tactically, some incremental experiments, aiming for quick wins that will reinvigorate the team morale.
The 70-30 baseline experiment mix, business goals, growth stage and team motivation are the factors that I consider when building an experimentation roadmap, in search for the ideal risk/ reward ratio.
I hope it will help you make more informed decisions.
Please comment below in case you have further questions. And share if you care ;)
Have a good (growth) journey,
—Livio
07/02/2021