Blog · November 3, 2026 · 5 min read

Fix, then test the fix: why one round of changes isn't enough

By Pavan, Founder, FirstReaction · Updated

In short

Fixing everything a test finds doesn't guarantee a better ad. Sahajan's first re-cut fixed every listed miss and stayed flat (58.7 to 57.7), because it won two buyer types and lost two. The second round, based on testing the first, reached 66.8. And an unchanged ad swung from 77 to 69 between runs, so test each fix more than once.

The usual way to improve an ad goes: find what's wrong, fix it, launch. Two of the ads we simulated for What's Wrong With This Ad? show why that last step is missing something. A fix can leave an ad flat, or quietly trade one group of buyers for another. And a single test can be off by more than the change you're trying to measure.

The fix that did nothing

Sahajan's ad is a 46-second creator video for a $41 cleansing oil. We simulated it against US women 22–40 who wear makeup and SPF, three runs per version. The simulation's list of misses was clear: no price or value for a cold audience, a before-and-after that was implied rather than shown, and ingredients never mentioned.

So we fixed exactly those. The first re-cut was 24 seconds, with a clear before-and-after, the ingredients on screen and a price end card. Likelihood to act went from 58.7 to 57.7. Flat.

The average hid what actually happened:

Buyer typeOriginalFirst re-cutSecond re-cut
Ingredient Investigator536266
Practical Purist455162
Time-Strapped Seeker575562
Skincare Curious675969
Socially Influenced Explorer726276
Average58.757.766.8

The first re-cut won the buyers who wanted ingredients and value, and lost the ones who'd liked the original's creator energy: two buyer types fell by 8 and 10 points. Net result: nothing.

The fix that worked came from testing the first fix

We didn't get the second re-cut from the original list. We got it by simulating the first re-cut and reading its new feedback, which asked for social proof and a clearer reason to trust a brand the viewer didn't know.

The second re-cut (27.6 seconds) added a real customer's review card ("worth the purchase!", 4.9★ from 257 reviews), framed the oil as step one of a double cleanse, and kept the creator's own ending on camera. Likelihood to act rose to 66.8, and every buyer type finished above where it started.

The lesson: a list of fixes is a hypothesis. Test the fixed version, and expect the second round of feedback to matter as much as the first.

The same ad scored 77, then 69

The other surprise was noise. For Snitch's plus-size jeans carousel, we simulated a finished version three times and got an average likelihood to act of 77. We changed one punctuation mark in the primary text, simulated it three more times, and got 69.

Nothing meaningful changed. That eight-point swing is how much an AI simulation can move between sets of runs. Even inside one set, Sahajan's original video scored 62, 66 and 48 on its three runs.

That's why we pooled both Snitch sets into six runs per version before publishing anything (61 to 73), and why a single run of anything, simulated or live, can mislead you.

A loop that works

  1. Fix what the first test found, using only claims you can back.
  2. Test the fixed version, at least three times, against the same audience.
  3. Compare per buyer type, not just the average. Look for who you won and who you lost.
  4. Read the new feedback and make a second round of changes.
  5. Treat small differences as ties. For close calls, run more times (we use six) before deciding.
  6. Then take the winner into a live test.

For the hook side of this story (the Sahajan video's opening was already strong, which is why changing it didn't help), see our guide to testing Facebook ad hooks. For the method behind these numbers, see how FirstReaction simulations work.

Sources

  1. Meta Ad Library