2023 does not look like the year the chaos is going to stabilize into the ultimate post-pandemic “normal”. Disruptions will continue to abound. With plunging savings, rising debt, and the overall economic pessimism that’s coloring predictions, consumer behavior will change in significant ways.
In the past, organizations could afford to bludgeon prospects and buyers into a state of submission with overwhelmingly loud marketing tactics. Ubiquitousness was synonymous with being “the market leader”.
Why?
‘Cause businesses and their VCs had money to spare.
If you can be everywhere… with vague messaging and a 2012 playbook, why would you innovate?
Buyer reluctance and funding reluctance have phased out this story. VCs quite literally slammed the brakes on aggressive funding in 2022. According to a Crunchbase report, Q4 2022 saw North American startup funding drop by 63% compared to Q4 2021, which was a glorious time in startup funding.
And the trend will continue in 2023 while inspiring behavioral changes at the same time.
Reshma Sohoni, Founding Partner of Seedcamp shared this prediction with Forbes: “…the market will take its ‘truth serum’ and re-emerge from the valley of despair (hopefully with some lessons learned)…”
To make it through and out of the valley in one piece, businesses have to try harder. Period.
Build buyer feedback and intelligence loops (Where they activate customer insights and establish a clear understanding of what’s in their control and can drive more purchases, more value, more business)
Throw away outdated playbooks and find new ones. This cannot be achieved without innovation in some form.
Yes, incremental improvements are underrated. They compound to create the “big wins”. But when the status quo comes crumbling down, you have to zag if you’ve been zigging.
The combo of optimization and experimentation is the ideal poise in today’s business landscape. When it became easier to set up businesses, it became more difficult to grow.
In 2023, the big risks, the ones that fundamentally alter the way organizations do business and carve out new playing grounds, will be the order of the day.
Here’s a short discussion between Rishi Rawat and Lorenzo Carreri that sets the tone for what’s to come:
This is also where the idea of A/B testing is a game-changer. ‘Cause:
Lots of failed tests for a new strategy can be a red flag. It means it’s time to question the validity of the strategy (top-down approach). However, documented learning and data from past tests can nudge and inspire new concepts (bottom-up approach).
The whole concept of a “test run” is less “risk” prone and more capital conservative than blowing your entire quarterly budget on developing something for the full ecosystem of buyers and prospects, only to have it fail — despite good intentions and research.
You don’t need a safety net when you climb the stairs to go from one floor to the next.
But you just can’t do without one when the building’s on fire… and you have to jump out the window and into the unknown. That’s when a safety net is NON-NEGOTIABLE.
Right from what needs to change to what the change can look like and the impact of the change in the real world, experimentation is capable of supporting businesses as they redefine what success means for them.
Understanding Disruption & Innovation
Disruption and innovation tango with each other in a never-ending loop.
If you ask Google about disruption, one of the explanations is: “radical change to an existing industry or market due to technological innovation.”
In short: existing market, new technology.
However, in this blog, we want to expand on the scope of both innovation and disruption. How their ripple effect travels… and why right now is the best time to double and triple down on experimentation.
Take a look at this quadrant. It is familiar to most CEOs and innovation seekers.
When you introduce new technology into an existing market, you disrupt it.
There is nuance to the science of disruption and innovation.
In “Shaping Strategy In a World of Constant Disruption”, the authors put forth the idea that disruptions also push market players and competitors to innovate at a faster pace, simply to protect what they already have.
They may try to continue to make their products better (labeled incremental innovation in the quadrant and also referred to as “innovation to sustain”), especially if they are the incumbent. This isn’t the best course of action though.
Winners step up to the plate and help move the entire industry forward through their own Shaping Views (we know it as a “strategic narrative”) and Shaping Acts (confidence in the future demonstrated by the entity that prescribes to the Shaping View. You can say the Metaverse is a Shaping Act… whether it will pay off or not is a different story).
Then there’s radical innovation or innovation to pivot where players carve out a new niche and market with better technology. This rarely happens but when it does, it takes everyone by storm, flipping tables in the air, which might be good for some and catastrophic for others.
So, what triggers this cycle of innovation – disruption – innovation?
Often businesses will disrupt themselves. This is the recommended type of disruption. Because here innovation is proactive instead of reactive. Innovation defines and drives disruption, either challenging leaders in existing markets or branching off into new markets.
The former seems alien because reactive innovation has been the norm for the better part of humanity’s existence. It is reflected in the saying “Necessity is the mother of invention”. Though we prefer Plato’s take: Our need will be the real creator!
Well, in 2023, human beings and companies have enough data and analysis ability to pre-empt future needs and innovate.
We’ve spoken at length about where we are right now in the business world. Both flavors of innovation will be important.
The “Need”, in today’s story arc, is that the changing macro conditions that don’t have much to do with “technology” and a whole lot to do with what’s transpired over the last 3 years WILL disrupt (as in disturb).
This disruption or disturbance will force reactive innovation from everyone involved. And not just technological innovation, but innovation in terms of growth motions and levers too.
Those who have seen the disruption coming have already begun innovating.
The rest will struggle to catch up. But guess what… they can’t blindly follow the self-disruptor’s model (Shaping View).
Consumers won’t fall for poorly differentiated value, wrapped in low pricing. And thanks to the lack of funding, toppling those who come in first is going to get even harder (it wasn’t easy, to begin with).
Consider the iPhone for a minute. In The One Device: The Secret History of the iPhone, the author Brian Merchant told the story of how Steve Jobs just wanted to create a cell phone that didn’t drop calls (a common issue with regular cell phones back then) and, with strong emphasis, fit perfectly in your hands. This is why the iPhone was limited to a 4-inch screen size until 2012.
Jobs had a Shaping View. He chose to proactively innovate and disrupt the existing mobile devices market with the iPhone. But the consequences were far reaching. His innovation and its subsequent successful adoption disrupted the camera film industry. It was a radical innovation that wasn’t even meant for the space and the incumbents it annihilated.
John has his own way of putting innovation on a spectrum. Efficiency leans towards incremental innovation (in our quadrant). Sustaining is another play on it. While Transformative is an outright disruptive innovation.
All of this is to say that reactive innovation can be stressful. It is like summoning genius in short order. It can lead to an identity crisis for brands.
There’s a calmer path; one that when part of your organization’s culture can keep you thriving in your best form, especially in the face of uncertainty. And that is…
Experimentation. It has to be the mantra in today’s economic climate.
Why?
It fosters proactive innovation (we’ll show you how) and renders the path for reactive innovation a lot smoother, less riddled with doubt and unnecessary high stakes.
...Leading by experiment gives teams the safety and permission to truly innovate. When failure is cheap and the capability to try any idea is always available, you will discover so much more opportunity than your competition will. You go beyond incremental change to what exists today and get to ask the “What if?” questions that might shift the entire strategy.
Leading by experiment also elevates team culture over time.
With enough surprising experiment results, we learn that nobody is an infallible genius – and nobody is incapable of having brilliant ideas either.
Ryan Lucht
This ties in with the strategy of businesses disrupting themselves by choice. Proactive innovation.
Experimentation is one of the best (if not THE best) ways to use your team members (those who come up with the hypotheses) and your customers (who validate or invalidate the hypothesis) as sensors to gauge changes on the horizon and help you translate them into meaningful business impact, thereby laying the foundation for true innovation.
Okay, If It’s a No-Brainer… Why Is the Silence Deafening?
Don’t get us wrong!
There are people in the experimentation community shouting themselves hoarse about the importance of taking more risks, delivering greater value, and leveraging A/B testing to innovate.
We are using A/B testing and experimentation interchangeably here. However, they are not the same. A/B testing is a randomized controlled experiment. Experimentation on the other hand is a way of life. Yes, it is that deep.
However, the response is often crickets.
David Mannheim wrote a whole piece articulating the frustration we feel.
In it, he states reasons from the best minds in the space as to why experimentation is overshadowed by the most blinkered type of CRO (Conversion Rate Optimization) and how the validation of ideas and learning gets lost in the bustle.
To properly court innovation with A/B testing, the 50,000-foot view is crucial. You won’t get that if you keep obsessing over a slice of the “funnel” or vaunted metrics.
How to Use Experimentation to Innovate? 5 Professionals Chime In
There is so much hidden gold in the community. And mature businesses and agencies are starting to rally around the right things.
A culture of experimentation is a culture of science within the company.
Meaning that when we are making a claim about the world, we challenge ourselves to point to some evidence, usually statistical evidence, that backs up those claims.
A culture of innovation is a culture of seeking risks and taking chances.
Meaning that we are constantly pushing boundaries and enabling groups of people to solve problems in counterintuitive ways.
The will and desire to innovate are fickle. You need determination and discipline and a whole lot of nurturing to help it take root. Otherwise, the most formidable team of A-players can fall victim to groupthink, the status quo and vanity results.
Paul Randall from Speero talks about this as well:
Keep things fresh by feeding it newly discovered insights and opportunities.
Refresh your research if you’re still using insights from 12 months ago.
Conduct regular exploratory analytical research, discovering new pages, new experiences or big changes.
Most important is a healthy appetite to learn and discover new things! If you are getting too comfortable it’s time to step out of your comfort zone and try something new.
This could mean trying out new technologies, new ideas, and taking calculated risks to push the boundaries of what is possible.
It’s tempting to keep iterating on the same hypotheses, but over time these can have reduced effects.
We categorize experiments as iterative, substantial and disruptive. Classifying your experiments means you can define a mix of substantial or disruptive test ideas.
Paul Randall
Jeremy Epperson of Conversion Advocates believes that a lot of time should be spent helping the company in question and everyone involved grasp why and how something like optimization, when done right, can further their understanding of customers, build up a repository of causal learning, discover new segments, and basically do the mise en place for innovation.
CRO is not about the technical setup of some a/b tests. Unfortunately that is the dominant mental model that exists in our industry right now. The perspective should be shifted to innovation. Innovation in strategy, processes, market positioning, customer experience, and other aspects.
Experimentation helps with organizational innovation like sharing insights across the business, building cross function growth teams, and breaking down business and agency silos.
It drives innovation in decision making including gaining an understanding of customers, becoming data driven and customer focused in all decision making, and building a culture of growth.
Even in the standard definition of innovation, CRO can unlock new customer segments, new products or services, new marketing channels. CRO is a catalyst for transformation and building the capabilities to scale.
Our definition of Conversion Rate Optimization:
a comprehensive process
for strategic research
and iterative testing
that measurably drives growth”
Jeremy Epperson
Tinker (Yes, Innovation May Have Humble Beginnings)
Innovation is a heavyweight. And it comes with expectations.
We’ll borrow words from Jakub Linowski again. You can’t really come at innovation with the mindset of “I want to discover X, hence I’ll discover X”.
You most likely will find dozens of drafts that aren’t X, before you get to something even remotely resembling X!
Innovation is as much a result of clear thinking, problem articulation and dogged perseverance, as it is of inquisitiveness and a desire to find connections between seemingly unrelated events.
If innovating as a part of the job description sounds like a tough nut to crack, turn to experimentation.
Run A/B tests to optimize (based on your research)
The by-product is learning and improvements. One underutilized way to accelerate the number of insights per test is to leverage the Heterogeneous Treatment Effect by drilling an uplift down into relevant and logical segments.
The difference between a CRO agency that prioritizes metric optimization and a business interested in self-disrupting (proactive innovation) is that the latter will probe causal relationships. It will expand the scope of the hypotheses.
You may be one A/B test away from striking innovation gold — even if the idea that inspired you was meant to offer incremental uplifts.
Once you have this framework, the more playful tinkering is easier. For example, David Allen proposes the organized confines of his Getting Things Done® system just so you can “make a mess” and get creative.
But this division is unnecessary. Instead of being one-way, the relationship between science and invention is reciprocal. Research and tinkering are both essential parts of the innovation process. While science and past knowledge provide a framework and guide the experimentation process in a meaningful way, tinkering at the boundaries of scientific knowledge can provide solutions outside the realm of current understanding.
David Stepien
Acknowledge Exploitation, Then Introduce Exploration
It’s not as bad as it sounds. We’re talking about exploitation in A/B testing. This cheatsheet by Jakub Linowski will help you see what we mean:
Why you run an experiment matters so much.
We’ve said it again and again. Intent and people make or break a process or a resource. Experimentation is no different.
You can choose to keep “exploiting” an insight that’s working for you, either by iterating on the hypothesis or by combining several changes with positive tendencies together.
Or you can choose to go beyond the definition of “pass/fail/inconclusive” to actively seek the maximum surprise. That’s where the good stuff happens.
Why do you need to acknowledge the “exploitation” though?
By doing so, you are not taking away the reliable and frankly quite successful playbook of metric optimization from your team. Think of it as a comforter. It’s reassuring to know you can always improve.
Innovation then seems to build on “exploitation”. It’s unlocking the next challenge. Instead of the scary prospect of seeking disruption using an unfamiliar process and then coping with the aftermath of a “win”. What do you do when you stumble upon the shape of an innovative idea? Radical change intimidates most people more than the thought of stagnation.
Ryan Lucht of CRO Metrics also advises a balanced approach:
If Peter Drucker wrote in 1954 that a business only has two fundamental functions – marketing and innovation – then we’d do well to remember that experimentation as a strategy can lead to more effective execution of each. That’s why I find the term “conversion rate optimization” so narrowly focused – it completely neglects the innovation side.
When I look at the skyline outside my window in Cambridge, MA, I’m looking primarily at life sciences companies, currently engaged in thousands of experiments; seeking to find innovation in vaccine development, medication for neurodegenerative diseases, or treatments for rare blood disorders. In their business, the biggest value is derived from finding the next product, patent, or other breakthrough. Marketers can think of experimentation for innovation the same way.
Some experiments help us get the most out of our current strategy (optimization), others show us entirely new opportunities via novel solutions or unmet customer needs (innovation). Sometimes we will intentionally form hypotheses that seek to explore potential innovations, other times we’ll see particularly surprising results from what we thought was a simple optimization test that will end up encouraging us to ask more fundamental, innovative questions about our business and customers. Keep an eye out for both opportunities!
Ryan Lucht
Build That Backbone
Ben Labay and the entire Speero team have posted at length about their ExperimentationOS mindset.
They’ve put in a tremendous amount of work over the last nine months standardizing and refining each aspect of running an A/B test to the point where they can visualize and encapsulate complex concepts into a series of easy-to-digest blueprints.
These blueprints are, in a sense, the basis of the system that powers the innovation and improvement machines Speero sets up for its clients.
It makes them more consistent when running testing and research programs and brings alignment across the whole experimentation journey, for everyone to see how it all fits together. Everyone gets the big picture, plus the motivation and focus to go after the desired outcomes.
In our blog post tackling A/B testing metrics, we explained how the experimentation machine chugs along if you ensure high-quality inputs (quantified by metrics) for desired outcomes.
In fact, every professional in the field we have spoken to since has stressed the importance of having a process in place.
Simon Girardin believes that a lack of defined phases and processes leads to reactionary “CRO”. The kind where you are forever putting out fires and have little time to spend on the passionate parts of improving a business or a product—the parts that are the most likely to yield fodder for innovation.
Finally, Samuel Hess of Drip Agency did an interview with Convert where he iterated that viewing experimentation as just a performance channel is a myth that needs to go away. Experimentation (when supported by a proper process) is the start of innovation and invention.
Think in Terms of Risk/Reward Ratio
To win some you definitely have to lose some.
A/B testing is a phenomenal way to manage risk and give you some validation before the big bad real world sinks its teeth into your solutions to pressing problems. But where innovation is concerned, you would benefit from keeping the concept of risk/reward ratio in your back pocket.
We’ll let Georgi explain why (because he does it better than we could):
Please understand that you do not have to chase the mythical 95% statistical significance. It is mostly a guiding reference.
Yet, it is also important that you get what 80% significance means for your experiment. If you can capitalize on a lucrative opportunity, then maybe an 80% significance is what you should be okay with since the cost of losing the potential breakthrough is far riskier than not realizing all the “positive change” that you see in the controlled setting of your experiment.
Let’s try a simple practical illustration with some numbers: You can interpret 80% stat sig to mean 8 times out of 10 you’re certain to get the same outcome from your experiment. According to the 95% stat sig guideline, that’s not good enough.
But if the potential benefit of going with the test result is an additional $10,000 to your bottom line while the risk of everything going sideways (in that 2 out of 10 instances) is losing $2,000, isn’t it worth it to take the risk?
You could lose the opportunity of making $10,000 while protecting $2,000 — that’s what we’re saying. You’ll have to compute this for your specific scenario to find how low you can reasonably afford to go with your statistical significance.
This is where the innovation mindset comes in handy. The will to innovate and swing big. While being cautious about not stumbling upon roadblocks that could have been avoided with all the validation tools you have available.
Start Climbing the Mountain
Innovation doesn’t stop at the big idea. As with everything, you have to see it through to completion.
Tinkering, research, and being risk positive (wisely and fearlessly) get you to the outline of your self-disruption blueprint. You still have to scale the mountain.
Make your innovation as adaptable as possible. This is a road often traveled with A/B testing for most businesses.
We spoke to Evie Brockwell, a stellar Digital Product Consultant, and Johann von Tonder of AWA Digital.
They both gave sage advice and shared how they go about supporting innovation’s complete maturity arc with A/B testing.
This is Evie’s view:
AB testing – does it stifle or aid innovation? The reality is, that it depends on how you choose to execute this. The main reasons that companies want to innovate are usually to: Attract a new target audience. Gain a large competitive advantage. Change trajectory of the company. Work with new markets / products.
AB testing, when used properly, will help the success of every single one of these aims. However, if you are just using AB tests to assess conversion, and you’re expecting a new launch or change to perform just as well as a product that existed previously – you will be setting yourself up for failure.
The key ways to use AB testing to your advantage are to: Learn through micro-changes about the best way to create your product, before releasing a major change. Monitor and understand the performance of this change. Continue to optimize with features post-launch.
If innovating on a current product, it isn’t uncommon for conversion to dip before you can optimize your way to a new local maximum (better than the conversion rates you experienced previously).
This is often true when you’re attracting a new customer type to your product (where really your measure for an AB test should then be incremental new customers, not conversion). In these cases – a non-inferiority test can be used for launch to keep conversion in a range in which your business desires.
When you’re launching into a new product or market, then AB testing can be a huge asset. It will allow you to keep adding changes that you know are going to be successful, as opposed to pumping too much time and money into something that just isn’t working.
The most important factors to get right when using AB testing with innovation, are to: Be crystal clear on what you’re trying to achieve through innovating.
Think about the way to approach this with your test set-up, learning what to include & launching. To have hypotheses and success criteria that accurately reflect what you want to achieve as a business.
Evie Brockwell
And Johann’s:
Most of what I see paraded on Twitter as CRO is simply moving stuff around the page. Tiny adjustments. Not quite button colour testing, but not much more adventurous than that.
Nothing wrong with this skew towards simpler tests, but you will never discover new sources of hidden value with small moves like that. The most successful companies play on both sides of a spectrum from optimisation (iterative tests) to innovation (radical tests).
With A/B-testing, there’s no need to play it safe. It allows us to push the envelope. If it doesn’t go the way you expected, just turn off the test! Mostly it won’t go your way, and that’s the point of it all. Limiting yourself to iterative improvements means climbing the local maxima hill, unaware of the real opportunity if you were to push beyond.
Source: Forbes
Here’s an example from my own experience as a consumer. While doing a course on Datacamp, I noticed this new “Discussion” module.
Clicking the “View Discussion” button presented this dialogue box: “We’re working on this feature … Please let us know if you’d be interested in (it)”
More than a year later, I’m still a subscriber and I haven’t seen this roll out. My assumption is that this was an MVP and the feedback from the test wasn’t all that encouraging.
While I have no inside information, here’s how it may have worked. Half of all users see the “View Discussion” element as part of an A/B test. The business can then compare key metrics between users in the test and those who saw the control (“Hints”). Percentage of clicks on the “View Discussion” CTA could serve as a proxy for demand for this feature. While there may be an element of curiosity bias or novelty effect, it’s possible to account for those.
This type of test is often combined with qualitative data, as you can see in this example. My guess is that not enough people clicked on “I’m Interested” which would explain why I still haven’t seen this live. Assuming that’s true, this would have been a neat way to test an idea before investing in development only to find out that there isn’t enough demand for it.
Another example, this time from my own agency. This is one of the largest retailers in South Africa with more than 700 stores across the country and a significant online presence. They sell a wide range of clothing, general merchandise and food products.
Like every online store, they ask customers to enter their shipping address during the checkout process. At the time, this was done manually.
That meant customers in a hurry, or those with fat finger syndrome, could enter their address wrongly. As a result, the retailer lost money every time a driver was unable to locate a customer’s address.
To improve address accuracy, our client considered implementing an enterprise address lookup solution. It promised to do everything they needed and save them money.
Given this retailer’s customer-first philosophy, they wanted to check that it would work well for both customers and the business.
Our approach is always to test the core assumption, the one that will sink the ship if it’s wrong. Their original wireframes were based on the assumption that the tech worked perfectly. But what if it didn’t?
To validate that assumption, we built a web-based checkout prototype outside the live shop environment, integrating this new address lookup technology. This enabled us to test both the user interface and how the system would function with real-world data. It yielded insights no other method could have done.
Findings from the test:
A third of users in the test had difficulty recognising their home address in the system – with multiple, confusing options.
Nearly half abandoned their purchase when they didn’t get an address that matched.
One participant, when the system didn’t find his address, looked at the page for eleven seconds before concluding that the retailer didn’t deliver to him. (It did.)
Without this thorough testing on real users, they could have launched expensive technology only to discover it wasn’t fit for the real world. The standard approach of testing interactive wireframes would not have unearthed this insight, as the wireframes were based on the assumption that the technology works as advertised.
The Optimal Approach to Business Growth: Combining Optimization and Experimentation
Since the economic climate is swinging unfavorably, businesses will have to replace outdated playbooks with innovation. That’s the only way they thrive and remain relevant and competitive in the midst of radical changes.
1. The faster things change, the more valuable it is to ask the right questions vs. have the right answers
2. To build a culture where innovation and customer-centricity can thrive, leaders should hold strong opinions but constantly update them by seeking new data.
3. Testing an idea that fails is cheaper than ignoring a potential game-changer. The former incurs a small cost of time, the latter can doom entire empires.
When in doubt, default to trying the idea. (aka the “Curiosity over Confidence” heuristic)
4. There’s little advantage available where “best practices” are clear. Market share is won by exploring uncertainty.
(e.g. by seeking ideas that buck, or at least tweak, the present state and consensus.)
So, how do businesses go about innovating for growth and sustainability?
Get in the habit of thinking innovatively, powered by asking questions and conducting regular exploratory analytical research and experimentation.
Encourage playful tinkering where you explore causal relationships and expand the scope of your hypotheses. Inquisitiveness often unravels unbeaten paths that might lead to radical ideas.
Develop a structure for consistent innovation. You don’t have to reinvent the wheel, you can use models and blueprints already developed by those who’ve gone before you to discover opportunities.
Revisit how much risk you’re willing to take, with the reward clearly in the picture. Don’t shy away from taking big swings. Calculate your moves and decide if you want to break past limitations you previously held on to.
Avoid the trap of getting limited by experimentation. That is when you’re only using it to make iterative improvements. Take bigger risks. It’s a test. If it doesn’t work, you can turn it off. Real amazing opportunities abound beyond where it feels safe.
Originally published February 13, 2023 - Updated November 23, 2023
Mobile reading?Scan this QR code and take this blog with you, wherever you go.
Authors
Trina Moitra
Trina Moitra is the head of marketing at Convert.
Uwemedimo Usa
Conversion copywriter helping B2B SaaS companies grow.
Editors
Carmen Apostu
In her role as Head of Content at Convert, Carmen is dedicated to delivering top-notch content that people can’t help but read through. Connect with Carmen on LinkedIn for any inquiries or requests.
A unique summit designed to help you network, learn, and grow in experimentation. Free & open to all. Save Your Spot
