Experiment-Driven Design Process

Posted on by in Design, Process

So you want to grow your product? That’s super awesome. Growth is often a goal that startups rush towards. 

“We need 100,000 monthly active users yesterday.” – random startup person 

Growth can mean a lot of things. Maybe you’re trying to grow the number of users, or increase time spent using your product. Whatever it is, growth usually means moving metrics. But meeting your growth goal takes a very meticulous and strategic design process. You need to try out ideas and see what actually works. In this article, I am going to share how to set up a successful experiment-driven design process that can help you identify the features and changes to grow your product. Let’s grow!

Create a Designated Growth Team

First, you need to think about how to structure your team. If you are just starting to think about growth, then chances are you don’t have anyone at your company dedicated to this role. Here is how to look at growth design in your design organization and how this process can help that team differentiate themselves from others. Angel Steger has a wonderful talk that explains this concept — you can watch it here.

Think about having separate teams, with distinct roles: 

Core Product Team – focuses on conceiving and maintaining features that improve the core experience of the product. This team creates value.

Brand Team – focuses on how your product feels, sounds and looks in the wild; they also help new users find and relate to your product. This team expresses value. 

Growth Team – focuses on finding new features and optimizing core features, that can make your product more competitive and meet goals. This team expands value.

*if your team is not big enough to be separated like this, Lex Roman, @calexity, has some great advice and I would read her blog post about growth designers.

The Build, Measure, Learn Cycle

Now that you know you want to grow your product and expand the value it offers to users, and you’ve designated a team to be in charge of growth, let’s talk about how to incorporate growth into a design process. A great frame of reference for what I am talking about it is the Build, Measure, Learn Cycle. Check out this simple explanation of it. It’s a cycle commonly used in product development. The idea is you Build (your product), Measure (to see if it meets your goals), and Learn (about how to make it meet those goals).

Phase 1: Understand & Strategize 

Step one: Create an MVP & ship to users.

This might seem obvious, but I think the best way to start this process is to get something in code as fast as possible to publicly launch it. If you want to grow, you, need to have real people using your product, right? The goal of this step is to get your best working hypothesis out in the world. For this scenario, and for lots of startups, the hypothesis is the product.

If you are having a hard time starting this process I suggest you read Carbon Five’s Guide to User Research. That guide will help you define what you are building and make sure your hypothesis is good enough to ship. Note: it is super important to have your metrics set up correctly, so you can figure out whether you’re successful.

Step two: Create an Experience Map.

An experience map is a physical representation of your customer’s journey. Think of your experience map not as a way to solve a problem but as a way to create a skeleton of your product that you will then start to ideate on. Carbon Five has a deep dive on experience maps here

When you are done creating your experience map, you should have something that looks like this:

Keep your experience map in your project room; you will want to refer to it throughout the experiment process. I suggest mounting it on a large piece of foam core so that you can move your post-its or index cards around to different rooms.

Step three: Create an idea library 

With your experience map in hand, now it is time to start thinking of how you can change your customer’s journey to increase growth. Take each high-level feature or epic of your product and brainstorm different ways you could adjust the experience to meet your goal. We often facilitate group brainstorming sessions to come up with these ideas. I would recommend any brainstorming activity on this list. Note that it is up to the Product Owner to prioritize these ideas once you come up with them. Once your idea library is created feel free to add to it, edit it, and icebox it anytime, it is not made of stone. 

Feel free to use this template to create your experiment library. 

Perhaps one of the most important parts of this template is the success metric. Each one of your ideas to have success metrics; you need to know what target you are hitting before you take aim. It’s a huge deal because you won’t measure the results until much later in your experiment process. Finding the right thing to measure might take a minute, and you might discover whole new things to measure later, but understanding why you are doing the experiment first is super helpful. For example, if you are trying to measure sign up flows you might want to see how many people compete for your sign up flow.

Phase 2: Create & Iterate 

Step Four: Edit your experience map & iterate your product

This is the most laborious but straightforward part of the whole process. At Carbon Five, we follow an iterative product development and design process, and we think it’s pretty handy for this part of the process. At the bare minimum, we think you have should iteration or sprint planning meetings, design reviews, reflections, and lots and lots of cross-functional pairing.

The key to this part of the process vs a less growth-oriented product development process is working fast and not caring about the details too much. Think about every feature you ship as a prototype. You might chuck it all away in the end, the purpose is to learn and discover what customers and users need to find value in your product. So don’t spend too much time nitpicking every detail like you might with more traditional design projects. If the feature becomes useful and gains traction, then you will want to make some enhancements before releasing to the greater public. 

Step Five: Ship your experiment & measure the results

Great! Users now have your experiment in their hands and you can see how they are doing. It’s important to remember the success metrics of the feature you are experimenting with when you are analyzing the results of your experiments. Also, the experiment results will most likely not be what you expect. So keep your eye out for hidden value hidden in data: maybe your users are not doing what you expected but are behaving in all new ways that can inform your product and design decisions.

Phase 3: Reflect and Repeat

Step Six: Reflect

Alright, after several key steps, including brainstorming, creating and analyzing, it’s time to either embrace or retire your experiment. If you choose to retire your idea make sure to document what you learned. Lots of people will just throw this information in a wiki of some sort where it goes to die. Try and make it part of your product team’s design principles. On one project I did we posted up all the failed experiments in a graveyard in our project room, so we were always reminded of what we had done in the past, and it also it acted as a really easy reference!

Step Seven: Repeat

This process is repeatable and, depending on how big your growth team is, you can run multiple experiments at the same time. In my experience, a successful growth team is not too busy, so it should consist of 1 product manager, 1 designer, and 2 developers. That’s not including other disciplines that might need to help out, such as content and customer support. 

At Carbon Five, we do reflections to after every experiment so we can discuss what we learned about the product and about how our teams work. Here are some recommendations on how to run an effective reflection with your team.

Netflix is the king of experimentation. I highly suggest reading their blog posts about it (see the additional resources section, below).  

A word of warning 

The need and desire to grow can make for sketchy product choices that might make your platform less than desirable to users. I recently saw a great talk by Sheryl Cababa of Artefact at the Seattle Interactive Conference. When growing your product, please consider not only your user’s engagement but also they’re well being. They are the reason your product can grow at all. Respect your users and respect your product. 

Think about annoying modals that pop up and demand you share this article, or ask you to subscribe. These types of UI patterns do the opposite of what growth is meant to do which is to expand value. These features are simply trying to move a metric and not expand the value of your product. While moving a metric could be your goal, expanding the value of your product and design should do much more than hit a metric, it should actually solve real human problems.

Additional Resources 

Netflix Growth articles:

Journey Mapping:

Experimentation:

Growth Design: