Meaningful Testing

Date Published
Leanne Barkley

I love to test. I enjoy the challenge that comes with developing new ideas that aim to lift response, increase the average gift, and add more donors to the constituent file. In fact, I wish I could do it more often. But the truth is sometimes it can be difficult to set up a test that will result in meaningful insights.

What makes a meaningful test?

In a perfect world, a meaningful test is one that generates a statistically significant result. Statistical significance is a calculation that tells us if the outcome of a test (i.e., a better response rate or larger average gift) is repeatable and unlikely to be caused by chance.

To help ensure that your next test will generate a significant result, you need to be mindful of two factors: mail volume and expected outcome.

Mail Volume Matters

When developing a test, you must first create a hypothesis about what your outcome may be. If you are conducting an A/B split in a mailing of 50,000 pieces of acquisition, you might — in this example —hypothesize that the test will work to lift response by 10%.

If we assume that that control cell will generate a 1% response rate, then this would result in a total of 250 gifts. If your test works to produce your desired 10% lift, then this would result in 275 gifts. Unfortunately, in this instance the difference of only 25 gifts between the control and the test is not large enough to generate a statistically significant result.

However, if you replay this same scenario but assign 100,000 pieces to each acquisition test cell, a 10% lift for the test translates into 100 more total gifts compared to the control. Statistics tell us that this 100-gift difference is large enough to generate a significant result. We can now have confidence in this outcome being repeated in the future.

The key difference here between these two examples is the mail volume. For large sample sizes, the difference between the control and test outcome does not need to be as great to mathematically prove that the result was not a coincidence.

Your Expected Outcome Matters Just as Much

Based on the size of your organization, and on the scenario above, you might think that your mail volume may be too small to ever generate a significant result. However, that’s simply not true.

Let’s take our original 50,000-piece scenario mentioned earlier. If we predict a 20% lift instead of a 10% lift in response, then our result suddenly becomes statistically significant. With the control generating 250 gifts and our test now generating 300 gifts, the difference in performance between the two groups is much larger and therefore we can have confidence that this outcome is not happenstance.

The bottom line is that if you expect a test to make a considerable impact (increasing response by a large percentage, for example) then you can get by with smaller mail volumes to generate a significant result.

Free Online Tools Can Set You Up for Success

Not every test is going to result in statistical significance — and, that’s okay. There are still insights to be garnered from those efforts to guide your way forward. Thankfully though, there are several free and easy-to-use online tools that can help predict whether your next test could be statistically significant. You can find a resource that works best for you by using search terms like statistical significance calculator.

By using these tools — and keeping mail volume and expected outcome at the forefront of your mind — you’ll be prepared to avoid common pitfalls, such as testing for the sake of testing rather than testing to gain actionable insights that make a difference.

About the Author:
Leanne Barkley
Account Manager, Agency Services

My role is to work closely with our clients to create and execute successful fundraising campaigns. Some of my responsibilities include developing annual budgets, providing creative and strategic recommendations, analyzing campaign results reports, and ensuring that my clients have everything they need to maximize their program’s fundraising potential.

Years of experience in the industry


What excites you about your work at AFG?

Our clients are the best! I feel incredibly lucky to work with several outstanding organizations across that country that have a very real and important impact in their local communities.

What are your hobbies/interests outside AFG?

Spending time with my family outdoors – we love kayaking, going on hikes, doing yard work (admittedly, I might be the only one who likes the yard work), camping, etc. Just get me out in the fresh air and I’m happy.

Get the fundraising solutions you need to fuel your mission.

Get Started

Finally, an End-to-End Fundraising Solution Accessible to all Nonprofits

DMW Fundraising and Member Benefits are now part of Allegiance Fundraising Group

Allegiance Fundraising Group brings together three well-established fundraising teams to create an end-to-end fundraising solution that makes it easier for you to reach your mission. Today you can turn to a single team for seamless fundraising strategy and implementation, donor management software, and membership loyalty programs.

Learn more about Allegiance Fundraising