Welcome to Episode Two of the What’s Working Now podcast, which focuses on experimentation in marketing. We are getting meta in this episode — we’re talking about the process of running experiments.
I’m very excited to bring Rebecca Flora to you, who is the digital marketing manager at Network for Good, a certified B Corp that offers a SaaS fundraising and donor management solution for nonprofits. Rebecca and her team have made some changes in their approach to marketing over the last year or so. And this approach has been guided primarily by experimentation. I know you’ll come away from this episode with plenty of ideas for your own marketing experiment.
Key Takeaways
(7:22) Create a standalone, single moment of consumption.
The campaign creative helped increase our impressions by 314% month over month. The new creative assets gave us the opportunity to offer a standalone single moment of consumption. What I mean by that is, I was able to start a story and end the story in the platform where prospects were viewing the content, as opposed to starting a story in an ad, and then finishing the story on the landing page. That also resulted in a 69% reduction in our CPM.
(12:00) Define an experimentation matrix and create a template.
We defined an experimentation matrix that fell in line with the purpose of what we were doing as a collective. And we took that experimentation matrix and we templatized it so that going into every experiment, we used that documentation as our North Star, and did not move away from that at any point.
(21:33) Identify what’s converting and remove all friction from those user paths.
You have customers because you have a great product. And you’ve encouraged people to really lean into that. People aren’t becoming your customers because it’s their last resort, or there isn’t any competition out there. So I really encourage SaaS companies to lean into what’s working, identify those paths from prospect to customer, and then eliminate all of the friction from those paths.
(23:39) Define the goal of any experiment before launch.
I want to talk about defining the goal of any initiative before launch. Defining your experiment prior to launch allows you to best understand where your audience exists, which platforms to launch in, and what objectives to use. Also, this allows us to use certain objectives to build into other objectives.
Transcript
(Laura) Welcome to Episode Two of What’s Working Now! This podcast focuses on experimentation in marketing, and we are getting meta in this episode — we’re talking about the process of running experiments.
I’m very excited to bring Rebecca Flora to you, who is the digital marketing manager at Network for Good, a certified B Corp that offers a SaaS fundraising and donor management solution for nonprofits. Rebecca and her team have made some changes in their approach to marketing over the last year or so. And this approach has been guided primarily by experimentation. I know you’ll come away from this episode with plenty of ideas for your own marketing experiment.
(Laura 1:38) Rebecca, thanks so much for coming on today.
(Rebecca 1:41) Hello, thank you for having me.
(Laura 1:44) I can’t wait to hear what you have to share with us. But before we dive in, tell us a little bit about Network for Good and the role that you play there.
(Rebecca 1:53) Absolutely. So as you mentioned, I’m the digital marketing manager here at Network for Good. As an organization, we help small nonprofits advance their mission by providing simple and smart fundraising software that enables them to build strong relationships with their supporters and donors. Since 2001, we’ve advised and supported over 400,000 small nonprofits. So when it comes to fundraising technology, we definitely know what nonprofits need to get the job done. As a digital marketing manager, I’m primarily focused on creating and maintaining our paid media strategy, ultimately securing prospects that, through a sequence of very intricate nurture streams, become customers.
(Laura 2:42) So I’d love to start first with a fun campaign you recently ran called “Donors Seeking Nonprofit” that was quite successful. Can you describe the campaign and your objectives for it?
(Rebecca 2:59) Absolutely. So the campaign goal was to deliver a bold and creative messaging that captured the attention of small under-resourced nonprofits and triggered their desire to invest in Network for Good. The messaging took the framework of, donors are seeking relationships with the causes they care about, and they’re ready to give now, even in this COVID-ridden reality. Our software helps build those better relationships. And that was really our hook — that you can successfully build impactful relationships by leaning further into technology.
The two main objectives for our omni-channel campaign were: 1) introduce the campaign messaging to both new and current prospects, and 2) accelerate our pipeline to achieve bottom line goals.
So circling back on our first main objective, the campaign creative was an adventurous step outside of our traditional brand aesthetic. So the creative team got together and decided to take this bold direction as they were aiming to deliver a message that captured the attention of small nonprofits and trigger their desire to invest in Network for Good software. With this, and really any new campaign creative, it was my role as digital marketing manager to introduce the narrative to all audiences both new and returning in order to generate awareness.
(Laura 4:29) Okay, nice.
(Rebecca 4:31) The second objective, accelerate the pipeline, was defined because this bold messaging really served to empower small nonprofits and give them the confidence to cultivate impactful relationships. The first rollout, which we called phase one, captured the audience’s attention through storytelling. I worked with the creative team to develop videos of different donors, with each video delivering an effective and captivating message. These videos were paired with static images and gifs, and all of this creative was used to launch the campaign messaging across all of our paid platforms. The goals here, in Phase 1, were impressions, video views, and landing page visits. So really rolling up into that more general awareness bucket that marketers define. Then in Phase 2, after pumping up each audience with this new creative and excitement, the goal is to convert these prospects into requesting a demonstration of our product. And as you can surmise, these leads convert into customers at a much higher rate.
(Laura 5:40) I love that, with this campaign, you’re really demonstrating that you understand your customers’ audience. So you’re helping them by saying, “This is what the donors are looking for. This is what their desires are. And here’s how you can connect with that.”
(Rebecca 6:00) Yeah, and it’s proved to be really effective, which is awesome. And I think the most effective headline about the campaign was that it did take a different angle. So we as a company, one of our brand pillars is to instill confidence in our customers, because we’re here to support nonprofits, and really help them grow. And it was just a different messaging that I think captured their attention a bit more and helped them understand that Network for Good and the technology we offer can help them grow their good.
(Laura 6:35) And so fun with the “Donor Seeking Nonprofit” theme.
(Rebecca 6:40) There were a few different characters that we curated. And then those characters were used in the videos we launched in social media. And that was a really fun thing to watch come to life. Each video took on its own unique little storyline, which was cool to see the creative team build out.
(Laura 7:02) We should link to some of those in the show notes so people can see visually what that actually ended up looking like.
(Rebecca 7:11) Absolutely. I’ll follow up with all those videos so everyone can enjoy.
(Laura 7:15) Awesome. So now tell us about the results that you got from that campaign.
(Rebecca 7:22) Yeah, absolutely. So we actually saw huge results in both creative objectives and our pipeline acceleration objective. The campaign creative helped increase our impressions by 314% month over month. And that’s because the new creative assets gave us the opportunity to offer a standalone single moment of consumption.
And so I’m going to pause. And what I mean by that is, I was able to go to market and start a story and end the story in the platform that they were viewing the content, as opposed to starting a story in an ad, and then finishing the story on the landing page. Which was really awesome.
Because that allowed and actually resulted in a 69% reduction in our CPM. Because we have this new creative, and we had our new objectives. We were able to go after and achieve a much lower CPM.
(Laura) Oh, nice.
(Rebecca) Yeah, it was a big headline there. And it also resulted in a 66% reduction in our Facebook cost per through play objective. And the reason I highlight that is because back in Q4 2019, we actually launched a video campaign that featured a series of customer videos. So the creative team took the learnings from that Q4 campaign and injected them into this Q3-Q4 2020 campaign. And it was nice to see the learnings come to fruition and then also results in a reduction of cost per through play, which was really cool.
And two other major headlines, we saw a 63% increase in the number of demo requests. So we captivated an interested audience with our awareness objective. And then we actually, in our pipeline acceleration objective, amplified our unique features through a sequence of video animations. So we were able to captivate audiences by addressing their pain points that our software solved. And ultimately, all of that influenced $75,000 in revenue. So we were pretty excited with those results.
(Laura 9:35) That is awesome. And I love how you really connected the campaign back to your product’s UVP because, you know, especially with content marketing, we’re really focused on education, which is great, but sometimes we can lose sight of what we’re ultimately trying to accomplish.
(Rebecca 9:58) Absolutely, and I think you’ve hit the nail on the head. It is hard to build that bridge between, needing to educate my audiences and tell them how to build lasting relationships and show them how to do things. And then how do you parlay that into a big sales pitch? You know, people feel a little awkward and they’re kind of opposing objectives, really. But this campaign did a great job of instilling confidence, but without overselling.
(Laura 10:33) Yeah, and it sounds like this campaign really did a good job of finding that balance. Now, offline, you mentioned that with this campaign, you were able to build new audiences that you were then able to retarget. Can you tell us a little bit about that?
(Rebecca 10:49) Absolutely. Our awareness campaign. So with this big rollout, we actually had an increase in our budget. So we had an additional spend in our awareness bucket. And our awareness campaigns, combined with that additional spend allowed us to introduce two new objectives to our current strategy. And those two objectives were landing page visits and video views. So with these Facebook objectives, we all know, traditionally, they come at a higher reach and impression volume, then, you know, their custom conversion objectives, which Network for Good is traditionally using, which means we were able to cultivate these hungry audiences at a much lower cost.
(Laura 11:31) That is super smart. So let’s talk about experimentation. I know you guys have been doing a lot of this, and you have a very systematized process that I think people would really benefit from hearing about. So two questions. First of all, how do you and your team come up with ideas to test? And second, how do you actually build out the process for running each experiment?
(Rebecca 12:00) Absolutely. So I’m going to tackle the first question, how do you come up with ideas to test, and I will say I was fortunate enough and Network for Good was fortunate enough to be able to start an internal growth team. So I joined a team that consisted of a product lead, a dev resource, a UX resource, and a marketing resource. And what we did collectively was spend time evaluating Network for Good. So we went through all of our processes with a fine-toothed comb. And we were just searching for ideas for lead acquisition, because that was our main objective. And when I say that we were searching for ideas for well-performing lead acquisition, what we were trying to find were ways to add fuel to that fire, and really accelerate those great ideas.
Heading into the second question, how do you build on a process for running each experiment? Immediately, when we formed that team, we defined an experimentation matrix that fell in line with the purpose of what we were doing as a collective. And one important thing that we did was, we took that experimentation matrix and we templatized it so that going into every experiment, we used that documentation as our North Star, and did not move away from that at any point.
Now, the two key parts of that matrix were 1) an experimentation question checklist. And 2) the learning plan. For example, what’s our hypothesis? What exactly does the ad look like if we’re running an ads-based experiment? What exactly does the user experience look like as they run through the experiment? Defining all that prior to launch, and then also defining our learning plan — what questions are we going to ask if we achieve our hypothesis? What questions are we going to ask if we fail at achieving our hypothesis?
By addressing all of those outliers and uncertainties and defining your North Star prior to launching each experiment, it allows for an unbiased experimentation tracking, and that’s really unique. Because I don’t know about everyone else, but in the digital marketing space, I consistently have my optimization brain on — so if I see something launched, and I know there’s a quick way to tweak it and improve results, I move immediately taking that action.
But as we all know, in experimentation, that would completely throw off everything. Now we’re no longer running a full, trustworthy experiment, where the data is trustworthy and unbiased, but rather, I’ve tweaked it to achieve my results. So that’s all to say, by defining that experimentation matrix and using that template going into each one, it really allowed for us to have the smoothest run process for every experiment that we launched,
(Laura 15:11) That is super smart to have a template that you follow every time. I know as creatives, we’re always coming up with new ideas that we just want to get in there and try this, try that out, tweak this tweak that. And with experimentation, it’s important, like you said, to make sure that everything is consistent and done the same way every time so that you get accurate results.
(Rebecca 15:35) Exactly, exactly.
(Laura 15:40) So I know you recently ran an interesting experiment looking at an idea for boosting sign-ups through generating interest in the video creation feature of your software. And your hypothesis ultimately failed. But you had a great process for evaluating the failure and learning from it to improve the results of your next experiment. So can you tell us a little bit about that to give us an idea of what this process would look like?
(Rebecca 16:12) Yeah, absolutely. So I’m actually going to do a quick step back and share the hypothesis and then I’ll go into an evaluation of the failure. We had observed in previous experiments that prospects fill out marketing lead generation forms at the rate of 30-40%, and we knew that our customers were more likely to create a video thanking donors after being shown an example of their peers creating these videos and thanking their donors. So we hypothesized that users and prospects searching for fundraising content, donor thank you content, and nonprofit video content, if they were provided relevant videos to watch and gained inspiration from that, then 30% of them would watch a video and click to create their own.
So, that’s just an example to show that we really do, at Network for Good, use learnings from all of our experimentation to parlay into building and creating a new experiment. And as you mentioned, unfortunately, it failed — we did not achieve our 30% goal. However, one of the questions we had asked in our learning plan was, “Is there an obvious point of failure?” And here we identified the obvious point of failure.
But circling back to why that learning plan is so important, is because we evaluated what worked in the experiment, we evaluated what didn’t work by asking “What was the obvious point of failure?” Then we used those two things and that entire process as a springboard into a new brainstorming session as a way to help improve and correct our next experiment. So, we’re evaluating what worked — great, let’s keep doing that. And we’re evaluating what failed —maybe some major points of failure or minor points of failure. And then we’re optimizing and curating and changing all of that into our next experiment and building on it. What our goal is, is that we want to experiment quickly, learn and move and grow into this ultimate lead generation machine.
(Laura 18:26) So, I’m curious, you mentioned some of the points of failure are obvious. But, I know that you’re also looking at all of these points of failure that are possible. How do you go about identifying what those possible points of failure are? If that makes sense?
(Rebecca 18:48) Yeah, so in the experimentation matrix, we actually do quite a bit of extensive research both on our prospects and our customers before filling that out. So we identify and estimate possible conversion rates, and put those pen-to-paper before launch. As I had mentioned before, we’re identifying the entire user experience. And instead of just writing it out, we’re also adding numbers to it. So that allows us to get a little more granular with our evaluation.
So, let’s use email events as an example. Let’s say we sent an email for this experiment. And we wanted to see how many people got to the end of the experiment. You know, we’re defining our email, open rate or click rate, our landing page view rate, our landing page bounce rate, and then ultimately our form conversion rate. And that’s through research into, as I mentioned before, both our customers and our prospects. And sometimes it’s a blended average. Sometimes, if it’s just our prospects, we use our prospect North Star. And by defining those steps and applying numbers to them, it holds the experiment more accountable, and ultimately allows us to identify points of failure earlier and quickly.
(Laura 20:13) Okay, so for example, if you see that the email has a low open rate, you know that the problem is probably with the subject line. Or if it’s with the click-through rate, then you know, the CTA is probably having a problem.
(Rebecca 20:28) Exactly. That’s thanks for that summation. And one other thing to mention, in relation directly to that video creation experiment, our hypothesis was that users were more likely to see a video and then create their own. And as we applied our conversion metrics to each step of the user experience, we saw that 35% of users actually watched the videos. And you know, as our hypothesis was that all of those people would then create their video, but only 14% of users actually clicked to create their video. That’s how we identified that obvious point of failure. So we achieved the metric in one area, and then it dropped off in the next, identifying that as a point of failure.
(Laura 21:16) Got it. Okay, that makes sense. So, I’m wondering if you have any advice for other SaaS companies who are looking to either start running experiments or improve the effectiveness of the experiments they’re currently doing?
(Rebecca 21:33) Yes. So my advice to other SaaS companies is to lean into what is working for your organization. Oftentimes, the mindset is, “New equals better.” And I would say, from both a user experience, and an ROI, this is not necessarily the case.
You have customers because you have a great product. And I encourage people to really lean into that. People aren’t becoming your customers because it’s their last resort, or there isn’t any competition out there. So I really encourage SaaS companies to lean into what’s working, identify those paths from prospect to customer, and then eliminate all of the friction from those paths. And really, by making sure that those paths have the least amount of friction, that’s how you’ll gain the most amount of customers. And I will add the caveat that sometimes new things need to be built to help support that friction reduction, but it is most important to stick to your company’s core competencies.
(Laura 22:44) I love that focus on the user experience. And reducing friction, because it really is about figuring out what your prospects are trying to accomplish. And then how you can facilitate that to make it easier for them. Because that’s where you’re really going to see success.
(Rebecca 23:05) Absolutely. I always find it fascinating to read into the UX experimentation as I grow in my career. Because the faster you can get a user from knowing that they need a solution and finding that solution with your organization, I feel as though those are lifetime customers.
(Laura 23:26) Yes, I agree 100%. So last question, is there anything that I didn’t ask you about that you would like to share?
(Rebecca 23:39) There is one thing I thought about. I want to talk about defining the goal of any initiative before launch. And that’s one thing I want to share and really circle back on because it seems so simple. Defining your initiative of that campaign prior to launch allows you to best understand where your audience exists, which platforms to launch in, and what objectives to use. Also, this allows us to use certain objectives to build into other objectives.
And the reason I mentioned that is because I have a really great example for our listeners. So let’s say your campaign launches in LinkedIn, Google, Facebook, and Bing, and you’re launching the campaign as an impression campaign. And all of the creative and the assets are built around this impression base campaign because we’ve identified that as our goal. Now, one of your teammates, or maybe your boss or manager comes in later after launch and says this campaign isn’t delivering. Let’s use demo requests as an example. You know where all the demo requests are coming from. We’re spending all this money on the campaign. I’m looking for that objective. But you’ve defined the campaign goal as impressions and that’s what it was built for.
So what I encourage users to do, or what I encourage our audience to do, is launch another campaign that satisfies that demo request goal. Because that’s a new objective that your manager or boss team has introduced to your campaign strategy. And you can parlay that current campaign objective into the new campaign objective, and we’re really building on each other there.
So, one real life example, let’s say, we launched in LinkedIn and/or we have a campaign objective of demo requests. Let’s use that Network for Good option there. So we have a campaign for demo requests, and we launched on LinkedIn, and it’s just not satisfying our need, the cost per form fill is really high. And maybe the requests aren’t always qualified, meaning they don’t really fit our ideal customer profile. But we do know that there is a wealth of audience and business professionals in that platform that could satisfy our ideal customer profile, ultimately becoming fantastic, longtime customers.
So what you can do is launch a click objective in LinkedIn, pulling audiences and users off of that platform and to your website. And then use that website data to retarget in Facebook, Google, and Bing, where we know there’s a much lower cost per acquisition. So that’s just an example how to really be direct with your goal when launching each campaign. And don’t be afraid to launch a new campaign and build off of the objectives that are already in market.
(Laura 26:42) That’s really good, and I love the focus on flexibility. Because so often, you’re right, people come in, they’re like, oh, we should do this, or we should have thought about this. Let’s add that. And being able to do that, while maintaining the integrity of the experiment is super important. So that’s a really good approach to take, and a good reminder.
(Rebecca 27:06) Yeah, exactly. And I even find that I’m sometimes the one leading those adjustment conversations where I’m like, oh, it wasn’t exactly the way I wanted it to go, I need to switch my focus to this. And that’s where your teammates come in and that’s where that upfront “defining the goal” comes in. Because now, rather than thinking of that initiative as a failure, you’re seeing it for its objective. You’re saying, “Okay, I am actually achieving that impression. But now, how do I get those impressions or those website clicks to help me achieve a new goal, and really build off that?” Because I swear, I’m sometimes maybe the worst offender, I have a very tight optimization brain where I’m always defining new goals. And we’re always working toward that new objective to help us and it worked to specifically reach our sales goals.
(Laura 28:08) Well, it sounds like that approach is definitely working for you! It’s been really exciting to watch you guys and see how you’ve been able to really boost your success so dramatically over the last year.
(Rebecca 28:21) Yeah, we’re excited. We’re heading into the busiest time of year. And all of this planning and brainstorming has started to pay off. So we’re really energized as we head into this giving season.
(Laura 28:39) This has been fantastic. I really appreciate you coming on and sharing and giving us the behind-the-scenes look of how you’re running these experiments, and all of the tips that you’ve offered to help people optimize their experiments. And I know that people are really going to appreciate what they’ve heard.
(Rebecca 29:02)Yeah, you’re welcome. Thank you so much for having me. And I look forward to following up with those videos. I hope that everyone enjoys them.
(Laura 29:11) Yes, I’m looking forward to seeing them as well!
Learn more about Rebecca here, and check out the resources (and videos!) she mentioned in the show: