Blog · November 3, 2026 · 5 min read
Fix, then test the fix: why one round of changes isn't enough
By Pavan, Founder, FirstReaction · Updated
In short
Fixing everything a test finds doesn't guarantee a better ad. Sahajan's first re-cut fixed every listed miss and stayed flat (58.7 to 57.7), because it won two buyer types and lost two. The second round, based on testing the first, reached 66.8. And an unchanged ad swung from 77 to 69 between runs, so test each fix more than once.
The usual way to improve an ad goes: find what's wrong, fix it, launch. Two of the ads we simulated for What's Wrong With This Ad? show why that last step is missing something. A fix can leave an ad flat, or quietly trade one group of buyers for another. And a single test can be off by more than the change you're trying to measure.
The fix that did nothing
Sahajan's ad is a 46-second creator video for a $41 cleansing oil. We simulated it against US women 22–40 who wear makeup and SPF, three runs per version. The simulation's list of misses was clear: no price or value for a cold audience, a before-and-after that was implied rather than shown, and ingredients never mentioned.
So we fixed exactly those. The first re-cut was 24 seconds, with a clear before-and-after, the ingredients on screen and a price end card. Likelihood to act went from 58.7 to 57.7. Flat.
The average hid what actually happened:
| Buyer type | Original | First re-cut | Second re-cut |
|---|---|---|---|
| Ingredient Investigator | 53 | 62 | 66 |
| Practical Purist | 45 | 51 | 62 |
| Time-Strapped Seeker | 57 | 55 | 62 |
| Skincare Curious | 67 | 59 | 69 |
| Socially Influenced Explorer | 72 | 62 | 76 |
| Average | 58.7 | 57.7 | 66.8 |
The first re-cut won the buyers who wanted ingredients and value, and lost the ones who'd liked the original's creator energy: two buyer types fell by 8 and 10 points. Net result: nothing.
The fix that worked came from testing the first fix
We didn't get the second re-cut from the original list. We got it by simulating the first re-cut and reading its new feedback, which asked for social proof and a clearer reason to trust a brand the viewer didn't know.
The second re-cut (27.6 seconds) added a real customer's review card ("worth the purchase!", 4.9★ from 257 reviews), framed the oil as step one of a double cleanse, and kept the creator's own ending on camera. Likelihood to act rose to 66.8, and every buyer type finished above where it started.
The lesson: a list of fixes is a hypothesis. Test the fixed version, and expect the second round of feedback to matter as much as the first.
The same ad scored 77, then 69
The other surprise was noise. For Snitch's plus-size jeans carousel, we simulated a finished version three times and got an average likelihood to act of 77. We changed one punctuation mark in the primary text, simulated it three more times, and got 69.
Nothing meaningful changed. That eight-point swing is how much an AI simulation can move between sets of runs. Even inside one set, Sahajan's original video scored 62, 66 and 48 on its three runs.
That's why we pooled both Snitch sets into six runs per version before publishing anything (61 to 73), and why a single run of anything, simulated or live, can mislead you.
A loop that works
- Fix what the first test found, using only claims you can back.
- Test the fixed version, at least three times, against the same audience.
- Compare per buyer type, not just the average. Look for who you won and who you lost.
- Read the new feedback and make a second round of changes.
- Treat small differences as ties. For close calls, run more times (we use six) before deciding.
- Then take the winner into a live test.
For the hook side of this story (the Sahajan video's opening was already strong, which is why changing it didn't help), see our guide to testing Facebook ad hooks. For the method behind these numbers, see how FirstReaction simulations work.
Sources
Related guides
- How to test Facebook ad hooksHow to test the first seconds of a Facebook or Instagram ad: what counts as a hook, hook rate and hold rate, nine hook types, live and pre-launch testing, and a real example where the hook wasn't the problem.
- FirstReaction.ai vs A/B testing for Facebook and Instagram adsFirstReaction.ai simulates how your audience reacts to an ad before launch; an A/B test measures real results after you pay to run it. What each tells you, what each costs, and how to use them together.
- What is AI audience simulation in advertising?How AI audience simulation works for Facebook and Instagram ads, what the research says about simulated buyers, a real example with five buyer types, and where it falls short.