High Tempo Testing

Almost three years ago, I wrote a post about the importance of iteration and I told the story of how the Gossamer Condor team successfully claimed the Kremer Prize for manned flight after many other teams had failed.  Their secret: they could assemble a prototype, test it, and learn from it many times faster than anyone else.

I was reminded of that story when I attended a brilliant presentation by Sean Ellis at last week’s San Francisco Agile Marketing meetup. You can find Sean’s slides at his growthhackers site and a video of the presentation should soon be available at the San Francisco Agile Marketer archives,

High Tempo Testing Drives Growth

Sean has one of the most impressive resumes in Silicon Valley helping companies scale, having served in a growth role at companies like LogMeIn, DropBox, EventBrite and Lookout.  He’s also met with some of the top people at Facebook, LinkedIn, Uber and Pinterest to compare strategies.  His main finding is simple:

High Tempo Testing Drives Growth

Here’s his slide describing the impact that Satya Patel had on Twitter’s growth when he came in and insisted that the Twitter team move from roughly one test every other week to 10 tests per week.

One of Sean’s more interesting off-the-cuff remarks was that it wasn’t enough to tell his team, “test more”.  He had to give them a specific goal, “run three tests a week”.  Depending on your resources, start small until you learn how to do 2 tests per week, than advance to 3 and so forth.

Sean also described two kinds of tests:

• Tests to discover (pings) when you’re trying out a new channel, or discovering if their is a new audience out there for your product, or testing out a new feature.  He compared this to the game of Battleship.
• Tests to optimize (A vs B) when you’re trying to optimize the effectiveness of a particular message or presentation to a particular audience in a particular channel.

High Tempo Testing Growth Team

The building of the high tempo testing growth team begins with the Growth Master.  In some ways, the growth master’s role is like that of the scrum master – they are responsible for keeping and grooming the test backlog, scheduling and holding the weekly meeting (see below) and removing any obstacles that are preventing the team from moving forward. But in other ways, I think Sean sees this position as something more.  He also called this position the Product Manager of Growth, which suggests a more strategic role.

I see the Growth Master as ultimately accountable for growth. Certainly not the person with all of the ideas on how to stimulate growth (more below when I describe the process), but the person who drives growth forward, ensuring that the team executes on the desired tempo of testing, and that the team is testing the right things. Obviously, this is a critical position for any startup or growth company, and one that is in high demand. If you needed any evidence of the demand, just read the comments on the San Francisco Agile Marketing meetup site after the meeting; half the comments are people looking for growth masters for their company.

The remaining members of the team are specialists, depending on what is needed to run the tests.  You will almost certainly have some design and/or web coding talent (if your tests involve interactions over the web).  You might have specialists in SEO, or search engine marketing, or social media, or email marketing, depending on the tests you’re running.  You probably also want an analyst, who can crunch the numbers and tell you if your results are statistically significant.

Sean recommended, and I think this makes sense, that you contract with temporary specialists until you figure out if the channel works for you and if the person works for you.  No use hiring a full time social media specialist only to find out that social media isn’t very effective for your product or brand, for example.

High Tempo Testing Growth Process

Sean’s process has some roots in Agile, but with some interesting differences.

He begins with what he calls “unbridled ideation”.  You could also describe it as brainstorming ideas for tests.  Each test is written as a hypothesis: if we do X, then Y will happen.  For example: If we move the sign up box from the bottom of the page to the top of the page, signups will increase significantly. It is very important that there be a “so what”, a result from the hypothesis.  It has to be provable.

Sean suggested that you invite everyone in the company to participate in this unbridled ideation.  People from different backgrounds and with different skill sets may bring a different perspective to what should be and even can be tested. It also relieves the growth master from the burden of coming up with all of the tests.

He also suggested that you encourage ideas across all of the vectors as he called them: Acquisition, Activation, Retention, Referral, Revenue and Resurrection.  These seem to be a variation of Dave McClure’s Pirate metrics (AARRR). If you’re a business to business marketer, like me, you can also think of these as the stages of the funnel.

The writeup of the idea, which might also include any research that you did to support the hypothesis, and a description of the target lever to be changed in the test, could be put on a Trello card.  Sean and his team are also developing a tool they call a canvas to track high tempo testing.  You can learn more about this tool at https://canvas.growthhackers.com/

Prioritizing the Backlog

The process for prioritizing the growth test backlog is somewhat similar to a Sprint planning meeting, but it has enough differences to make it important to describe.

First, every idea is ranked along three dimensions

• I equals Impact (what do you expect the impact of the change to be on the metric that you’re measuring)
• C equals Confidence (how confident are you that the change is going to have the impact you describe)
• E equals Easy (how easy or hard is it to perform the test)

Each idea is given a score from 1 to 10 along each of these dimensions, and a composite score is calculated as the average of the three. The best ideas obviously have high impact, high confidence and are easy to implement, but of course no idea is probably high on all dimensions. Sean also stressed that it is often very hard to determine in advance which ideas are going to have high impact.  One of their ideas which was given something like a 4 on a scale of 1 to 10 for impact ended up having an impact of almost 700%.  This is one of the reasons that you want to drive tempo in your testing.  More testing means more opportunities to discover and optimize your product offer and your sales process.

At the weekly growth team meeting, growth team members nominate ideas based on these scores, and the team decides which tests to prioritize for the coming week.  Each test is then assigned to a project manager, who is responsible for making sure that test gets run, and the whole company is notified of each test release.

The growth meeting itself is time-boxed to an hour, and it includes quite a few critical individuals.  At his company, GrowthHackers, it includes the CEO, CTO, VP of Product, head of growth, head of design, a growth engineer and an analyst.  That’s an expensive meeting.

You can see the agenda for the meeting in his slides; I won’t repeat it here.  What’s important is that they have a very rigorous agenda, and they establish a rhythm of making decisions quickly and consistently.

Potential Issues

I love the methodology and I’m totally bought in on the concept that increasing the tempo of testing can itself lead to growth.  The only issue that I could see would be unintended interactions between tests. For example, let’s say you were running two tests on the same week that both impacted your home page, and both are measuring lift in signups.  How do you know which test, or was is the combination of tests, contributed to the lift?

I asked Sean about this, and he admitted that this could become a problem, but basically said they were careful to try to avoid overlapping tests.  I took this to mean that they probably didn’t run two tests on the same page at the same time, and this makes sense.

Thanks to Sean for a great presentation, and thanks to San Francisco Agile Marketing meetup for hosting the meeting.