The Double Diamond vs. Rapid Experimentation

Traditional, upfront user research doesn’t fit every project, no matter how hard you squeeze.

Rather than squeezing a user research phase into a packed project, you might need to consider a new research model. Not every project needs upfront research, and upfront research won’t fit some fast-moving organizations.

Cyclical models like rapid experimentation take a different approach to upfront, traditional research. Let’s look at the traditional design research model first.

Traditional Design Research Process a.k.a. Learn to Build

[image by Jeff Humble]

Process models like Design Thinking and the Double Diamond make product people think that discovery always has to come first. This popularity speaks to the success of the Double Diamond Design Process Model, which made this idea of doing research before designing visible.

This research model is all about learning first and building after. By “learning to build,” you fall in love with the problem before jumping to solutions. It’s the dominant idea behind user experience (UX) design, service design, and product design.

But upfront, traditional discovery research isn’t the only way to do research. You can evolve beyond this problem-then-solution process. Your life will be much easier if you let go of the rigid concept that research always precedes design.

Force-fitting a research phase at the beginning of every project won’t earn you a trophy. In fact, it may show your managers that you don’t get how your company learns.

Most tech startups learn through experimentation, not big research presentations. Now, I’m not saying experimentation is better than in-depth qualitative research. I’m just saying that most companies will prioritize quantitative learning from things they’ve built over qualitative learning from something that isn’t built. Unfortunately, most designers are only taught the latter type of research, but that doesn’t mean it’s always the right approach.

If you take a “research-first” mentality at all costs, you might be causing yourself unnecessary anxiety. If you think research has to come first, you might think your only option is to hide time for research in your projects. Sure, there are ways to squeeze discovery in (see continuous research), but you can’t always find the answer upfront in new domains where you’re the first to try an innovative idea.

Apple, Spotify, and Upfront Research on Subscriptions

Sometimes, users aren’t imaginative enough to provide helpful feedback. When an idea is new, upfront research can often steer you in the wrong direction.

When Apple first released the iTunes store in 2003, Steve Jobs was vehemently against charging a subscription fee. Their research told them that users didn’t want to rent music.

“People have told us over and over again, they don’t want to rent their music…they don’t want subscriptions…that just doesn’t fly with customers. They don’t want subscriptions.”

[from a Steve Jobs presentation in 2003]

“These services treat you like a criminal. And they are subscription-based based and we think subscriptions are the wrong path.” In a Rolling Stone interview, he went even further, railing against the subscription model:

“The subscription model of buying music is bankrupt. I think you could make available the Second Coming in a subscription model and it might not be successful.” -Steve Jobs in 2003

Of course, he had done his research, and he was right to a point. Apple customers did not want to rent their music when the iPod was king. But Apple customers couldn’t imagine anything different because renting music wasn’t the prevailing paradigm, buying albums at a store was.

But three years later, in 2006, a company called Spotify started experimenting with music as a subscription. Spotify (a company known for experimentation) tested its way to an idea that was essentially users renting their music.

By 2009, an all-out war was developing on the iPhone over whether users would choose iTunes or Spotify, which continues to this day.

In the end, Spotify got a huge headstart, grabbing market share from Apple. Apple eventually copied its model in 2015 with Apple Music. But even today, Apple Music struggles to keep up with users of Spotify.

[image from PrioriData article on Spotify performance, 2024]

Ironically, Steve Jobs got stuck on something he learned in upfront market research when he was very vocal about the shortcomings of such research. Another company that wasn’t against letting users rent their music came along, and they found a way to make it work where Apple wouldn’t.

This David and Goliath story shows us that even innovation giants like Apple can get stuck on insights from upfront research, not realizing that a paradigm shift is just around the corner.

Let’s learn more about a research process that can help you uncover that new paradigm quickly.

Rapid Experimentation Process a.k.a. Build to Learn

[image by Jeff Humble]

Definition: Rapid experimentation is a cyclical process for testing ideas early and often. Running lean experiments in frequent testing cycles helps you validate your ideas before you waste effort on full builds.

This research model is all about building something to learn as quickly as possible. By “building to learn,” you try to test ideas in as-near-to-real-as-possible environments so you don’t waste time building something fully featured without testing its value. It’s one of the dominant ideas behind lean startup, growth hacking, and agile methodology.

Rapid experimentation brings a quantitative, prototyping mentality to research.

Rapid experimentation is an agile approach to the product development process. With this approach, frequent experiments are deployed in an attempt to discover new, innovative ideas. Experiments can range in severity, from simple A/B tests to larger field experiments.

-Product Plan

Rapid experimentation is about placing lots of bets. You place hundreds of tiny bets on a long project rather than all your chips. The goal is to test as many ideas as possible in testing environments that mimic the real world. Experiment results will often surprise you. By testing many ideas, you open yourself up to more positive surprises.

Here is a year’s worth of experiments and how it might feel:

[image by Jeff Humble]

Traditional research means you don’t measure as often. Experimentation forces you to take a measure with every single experiment. That means you will notice sooner if your product ideas suck.

*“72% of all new products flop.” -*Simon, Kucher & Partners

When you face reality, most ideas fail. Rapid experimentation ensures that you can detect the failure as soon as possible. It’s a very lean way to research through tests.

Rapid Experimentation at Dropbox

For an example of rapid experimentation, look at Dropbox. In the early days, its value proposition was built around magically syncing files between devices when they started. While syncing is standard today, it was a radical concept at the time. Before they built it, they did a lot of upfront research to validate their idea.

However, since Dropbox was new, the upfront research involved users who had never experienced the product. The company received mixed reviews, and it didn’t seem like the customers were as excited by the idea as the founders were. What do you do when all the upfront research tells you something, but you know your idea is good?

They decided that since their idea was so new, they needed to fake it to gauge the user’s reaction objectively. They made a fake door test where they faked the experience of using the app in a video, which went viral.

They followed up the video with a few more videos, testing their way towards a working product. Eventually, they knew that their idea was de-risked enough to build it.

These experiments showed interest in the idea and were far more effective than a focus group or a series of user interviews. Dropbox knew they could devote more time to building the experience they showed in their videos.

Innovation projects, in particular, don’t pair well with upfront research. It’s hard to do research when nothing like your solution exists anywhere.

But it is possible to research as you deliver. Let’s look at another way to do research.

When Rapid Experimentation makes sense

Testing first doesn’t make sense in all situations. Here are some scenarios where rapid experimentation can shine:

• The idea you’re working on is completely new
• It’s a new domain where there isn’t a lot of upfront research available
• Company culture prioritizes quantitative insights over qualitative insights
• You need to act fast and don’t have time for a long research project
• The team prefers to learn from building
• You don’t have in-house qualitative research expertise

Crucially, the experiment doesn’t have to be coded or release-ready. It can be a scrappy prototypethat tests the value behind your product ideas. The important thing is to test the value, according to the customer, before you jump to implementation details.

Here is what might be new in Rapid Experimentation:

• You need to know how to write a good hypothesis
• You need to test a lot of prototypes quickly
• You need a way to measure the prototypes you’re testing
• You need to analyze the results of your test
• You need to form insights from experiment data

I teach these skills in Designing Product Experiments, and I’ve seen dozens of designers turn their design skills into experimentation skills.

Setting up an experiment takes time, but it’s much faster than doing 20 rounds of upfront user interviews on an idea nobody understands. You can still supplement experiments with more traditional qualitative UX research to understand the “why” behind the experiment numbers.

Remember, when an idea is new, sometimes the only way to learn about it is to put it in customers’ hands and watch.

Remember, everything is an assumption until it's in the hands of a user.

You can find ways to test your assumptions early and often. Testing can offer more realistic insights than traditional research can provide.