Category: Experiments

How Do You Plan Your Cold Email Experiments?

Getting a steady flow of leads from your outbound campaigns requires a lot of experimentation.

It’s a lot like the scientific method: You need to hypothesize how to grow, run that experiment, track how it performed, and move forward with winning experiments that reached statistical significance.

Most importantly, documenting what you have learned during the entire experiment – did we get more opens with this experiment, more clicks, more traffic to the website, more positive responses, etc.. This information will then inform your future experiments.

Miro, the Visual Collaboration Tool, had this to say about the Experiment Trackers. I adapted their process for us.


  1. Log all ideas you can think of. This usually starts with “We believe if we do X, we will achieve Y”.
  2. Prioritize them by ease of implementation, which ones can you test right away without significant resources.
  3. Decide WHAT you want to test with WHOM and HOW.
  4. Define which data you want to measure and which targets you need to meet to call it a success.
  5. Take time to craft the experiment and run it.
  6. Run the campaign and periodically update your findings. Distill key insights out of it. Any surprises?
  7. Define how you want to continue: Kill this idea right away, give it another chance with some adjustments (pivot) or continue like is (persevere).
  8. Enjoy the journey

Common challenges and mistakes to avoid.

  • Start with more than one experiment and loose focus
  • Overestimate the success of their solution, keep in mind: If you don’t reach the target you might consider to kill the idea
  • Set the bar too low. A good benchmark for cold email replies are between 1% and 5%. If you need more benchmarks, here’s a list of cold email statistics to use.
  • Are thinking too big creating the first experiment. What is the minimum thing you need to do to learn?

Why Experimenting with your Cold Email Campaigns?

Outbound marketing experimentation is essential. It enables you to evaluate your different methods and make adjustments based on what’s happening. This usually requires getting a pulse on what campaigns were initially done.

Generally, folks would experiment or run AB tests on messaging, subject lines, or even something as broad as how they’re packaging different things.

This is great, but the thing is, it generally not recorded anywhere. All information is stored in the outbound automation tool, which just has the data.

It wouldn’t have enough information on the learnings, what we should do or do not do in the future, and if there was any bottlenecks or unexpected issues arising during the campaigns.

So how do you organize, keep track and learn the things you need to learn in an effective way? In this post we discuss how to organize outbound marketing experimentation with the use of an experiment ledger.

The experiment ledger is typically on an excel sheet (there are also a few tools out there that can do this). You need to be able to track the specific details of the experiment in order to ensure its success.

For example, you need to be able to track what experiments you have already done, what experiments are coming up in the future, and what success looks like for each.

For instance, an experimentation idea can be going after Director of IT with a certain type of messaging and shooting for a 10% response rate and a 2% meeting rate. Getting into these level of details allows you to know for sure what is working and what is not.

Even though we also want to see our stats on opens and clicks, we also have to place attention on the kinds of responses we are getting, who we are actually engaging, what the specific value proposition used, and determine if there are any learnings there.

All of that will be documented in the experiment ledger. This will allow us to suggest a clear path forward for connecting with prospects, at scale, include nuanced insights gleaned from our various experiments, as well as tailored channel strategies to increase response rates.

Why you should not copy email templates online

It’s easy when creating an outbound campaign to rely on our own judgment or other people’s insights into what worked in the past.

We might even fall back on templates found online, “the email that got 300-400 appointments in one day” or “the sequence that saved my company from the brink of disaster and delivered $3 Trillion dollars in sales, download it now!” or whatever Breakthrough Email or Predictable Revenue’s Aaron Ross is “working” now – copy it word-for-word, then wonder why we’re not getting the same amazing results that that company got. The problem is that everyone else is also using that same template.

The truth is, when you’re creating your next campaign, you shouldn’t look at the campaigns of others and do what they do. Instead, you need to do a test and find out what works for you.

You’re unique, special, and no one knows your journey better than you, no one knows the ins and outs, challenges and solutions your company solves, and trying to copy the successful campaigns of others is no guarantee they’ll work for you.

Fortunately, there’s a solution. A solution that requires some work, creativity, and a lot of patience.


Treating each campaign as a test or an experiment. Trying different variations and measuring the results against a predetermined hypothesis.

It would require a lab coat, clipboard, clicky pens, pensive looks, and lots of hmm and aaah. Hmm, Sales Directors w/ 15 years in this game knows the importance of a CRM, lets do a shorter, to the point, discovery messaging to that audience. Aah, more positive replies compared to other campaigns. Pensive look. Que clicky pen and clipboard and measure the number of meetings or positive responses received against your other campaigns and determine if it’s worthwhile to continue that campaign.

Running outbound experiments is a fantastic way to find out what exactly works best for your business, rather than relying on wishy-washy ‘gut feeling’, short is better, long is doom, first line personalizations fad, and general ‘best practice.’