How I tested AI quality at Waymark
Waymark uses AI to make TV and online video ads for small businesses. You give it a website, and it writes the script, picks footage and images, applies the brand's colors and builds the ad. Most of the people making these ads were salespeople at local TV stations, pitching businesses like the pizza place or the roofer. If the first ad they made had a green screen clip or an animation pop up in the middle, a lot of them didn't come back. When I took over the quality work from our CEO, the hard part was knowing whether a change made the ads better or just different.
So I built a set of testing tools, mostly in Bolt, an AI app builder, one for each part of building an ad: footage, scripts, color and images, plus a review station for finished ads. They shared the same sign-in, model picker and prompt version manager, so each new tool took less time than the last. By the end I could build a new testing tool suite in a day and a half instead of a week. My other two pages cover what changed in the product. This one covers how I tested it.
Starting with the failures
Before any tooling, I watched a lot of bad ads and wrote down what was wrong with them. Those notes grouped into nine categories: footage, color, images, text, script, logo, template, cut-off faces and respect for the brand. At the start, scoring against them was my own judgment, since the team was mostly engineers and judging creative quality wasn't their job.
How the footage pipeline worked
The old pipeline matched footage mostly to the voiceover line, so in an ad for a sock company, a line like "a perfect fit" could pull a clip of a guy trying on a suit.
The new one worked in stages. For each line of the script, a model wrote several stock searches: one for the business, one for the business plus the voiceover line, and one for the voiceover line alone. A keyword and category filter then dropped clips like green screen and animation, and the rest were ranked on fit to both the brand and the script.
A vision model checked the thumbnails of a short list from the top for what the tags missed, so the slower check only ran on a few clips, and a model picked the final clip without repeating one already in the ad. Engineering built it into the product while I tested which models should run each step.
COMPANY
Waymark
ROLE
Director of Product
BUILT WITH
Bolt, Replit
FOCUS
LLM evaluation
Human review design
Model selection
AI-assisted development
Old against new, on the same inputs
Every change was tested against production, the version customers were using, on the same inputs.
The footage tester ran the production pipeline and the new one on the same business and script, and showed each side's search queries and chosen footage, with separate prompts and models on each side. I ran over 300 of these, because reading the actual searches and clips side by side was the fastest way to see what was going wrong and why. The batch runs below then measured it across many ads.
For a wider read, I had engineering pull 100 real ads out of the production database, and I ran that same set through both pipelines. A model scored every chosen clip from 1 to 10 against the business, and a 6 or better counted as a pass.
An early batch had the new pipeline only slightly ahead, with no fewer bad clips. The query and filter testing below came after it. By the time the new pipeline shipped, it had cut disqualifying footage to zero and off-brand picks by 50 to 60%.
Getting the searches right was its own struggle, since a clip can only be picked if a search brings it back. A generic search returned a huge number of varied clips, mostly unusable. A very specific one might return nothing. And beyond the words, each search could carry filters and settings, such as a stock category, that changed what came back.
The prompt writing the searches only knew what we could pass it about the business, and it had to work for every kind of business, from a country club to a plumber.
How many searches to write for each line was a big tradeoff. More searches cast a wider net and protected against one bad search, but each one meant another call, more processing and more results to sift through.
Production wrote one search for each line. I tested new prompts that wrote one, two or three, aimed at the business, the voiceover line and the two together, and had a model score every clip each side's searches brought back.
Across 25 saved comparisons and about 2,500 scored clips, the new prompts averaged about 5.4 out of 10 against 3.8 for production, and the share of clips scoring 6 or better went from about a third to more than half.
The script prompt's job was to write the best voiceover script for the business. The script tester put three versions side by side for each of 100 real requests: the script production actually shipped, the production prompt run again, and the new prompt. Running the production prompt again showed how much a script changes just from asking twice, so we could tell that apart from a real improvement.
I ran it 11 times, about 780 three-way comparisons, and the output went to a spreadsheet for myself and a very experienced script reviewer to score until we agreed that the improvements were significant enough to ship. The new prompt shipped.
Which model for which step
Three of the pipeline's stages call a model on every line: writing the searches, checking thumbnails and picking the clip. I tested models for each of them separately.
A speed benchmark timed seven models from OpenAI, Anthropic and Google on each one, on the same inputs. It randomized the order of the models each run and threw away a warm-up call, so the first model didn't pay for a cold start. A different model was fastest at each step, but speed was only half the question.
The whole process of building an ad took two to three minutes. Individual steps could get longer, as long as they didn't extend the overall time.
The new footage pipeline did more work on every line of the script and was running several times slower than production. One of the engineers built a graph showing the overall time and the time taken by each step of building an ad, so we could see which steps were eating it. With a few optimizations the new pipeline stayed inside the overall time.
For quality, a side-by-side test ran two vision models over the same 50 stock images and showed which images each passed or rejected, and how long each took. The model picker covered about a dozen models across the three companies, and every scoring and comparison test could run on any of them.
Cost came into it too, since each model has a different price per call. Where a higher-priced model only did marginally better, I didn't pay for it.
Checking the checker
To know how often the automated filters were right, I built a labeling screen into the footage tester and labeled clips from real ads myself, each clip marked bad or not and off-brand or not, with a note where it helped. Next to each label, the tool showed what the production filter did and what the new filter did.
From those labels the tool built a simple grid: bad clips the filter caught, bad clips it missed, good clips it wrongly removed, and good clips it left alone. It also listed every filter term with its correct catches and its false alarms, and it could put production and new side by side on the same labels.
The first check showed where the new filters still needed work. On one set of 94 clips, production's filters caught none of the 12 bad clips and none of the 15 off-brand ones. The new filters caught 3 of the bad clips and 8 of the off-brand ones, but they also threw out 18 good clips.
The per-term view showed where the false alarms came from. "Close up" sounds like a reasonable term to block, but plenty of close-up shots are usable footage: it removed seven clips, and all seven were good. Without counting each term against my own labels, it would have stayed on the list.
I used that view to tune the filters, and by the time the new pipeline shipped, disqualifying footage was down to zero.
Prompt versions
I introduced prompt versioning. Every prompt in the tools had saved versions, and every test ran on whichever version was applied, so each comparison above was also a comparison of prompt versions. In the footage tester alone, I saved 35 versions in under three weeks. The names read like a change log: "avoid repeats," "reject sideways," "added brand/vertical match," "more strongly reject non-footage."
Versioning also made quick fixes safe. When client services saw a new kind of bad clip, I could figure out the addition to the prompt and push it, and the fix was live in a day or two.
Color and images
An AI can pick the right brand colors and still put them in the wrong places. I had each template's color slots mapped and audited, with names for the failures we kept seeing. "Ghost element" was text that disappeared because it got the same color as the bar behind it. "Vibrancy theft" was a bright, high-contrast pair that turned into two similar dark shades.
Across 19 templates, the audit found 24 named failures in 7 of them, mostly these two kinds, and the maps carried rules to prevent them, such as forcing two slots to stay different colors. The brand colors themselves were picked by a model, and I tested several models against the prompt to find the best combination of the two.
For images, the point was to understand where the best images for an ad came from, and use that to make better selections. My hypothesis was that images from the brand's own website were better for a commercial than images pulled from Facebook.
Across 20 brands and about 1,600 scored images, that turned out to be true, partly because Facebook had more images of things that had nothing to do with the brand, like the CEO running a marathon or a company onsite, while the website images more often reflected the brand as a whole.
Scoring finished ads as a team
Partway through the work, we reorganized into mission-driven teams, which brought designers and client services people into the quality work. That's when I formalized the nine categories and tested them against an overall score.
Reviewers gave each ad a score from 1 to 4, tied to how long a person would need to fix it: 1, ready to air as is; 2, needs a little work, under 10 minutes; 3, needs a lot of work, 10 to 20 minutes; 4, better to regenerate than fix.
We started with five points. I ran several rounds with the team to check that everyone was scoring the same way, and those rounds are what moved us to four.
This started in a spreadsheet and grew into its own app, a review station that spread the ads across a panel of reviewers, including designers and client services people, with two reviewers on each ad. Each one picked the three biggest problems, rated how severe each was, and wrote a note on each plus an overall note. A typical pair: "The logo needs to be upscaled," and "I think it's shippable if you upscale the logo."
In the March runs, seven reviewers left 91 reviews and 174 written notes. The results showed which categories came up most and how severe they were, and that set the order of the next round of work. They also showed that some reviewers scored more leniently than others, which matters when you read any one person's scores.
Footage problems dropped after the new footage pipeline shipped, and color and logo came out on top, so that's what we took on next.
Since then
Since then I've taken Hamel Husain and Shreya Shankar's course, AI Evals For Engineers & PMs, and look forward to bringing what I learned there into the next AI product I work on.