A week ago I wrote that Fable 5 felt worse at coding. I inspected the raw requests and found that a setting labelled high was using a much smaller reasoning budget than I expected. The post drew 545 reactions and 127 comments. I cannot prove that every weaker answer came from that setting. I can prove that trusting a label left me without a baseline when the behaviour changed.
That experience is why every company using AI needs an LLM regression testing workflow. Models change. Prompts change. Tool permissions, retrieval sources and hidden provider settings change too. A workflow that passed last Friday can fail on Monday while still returning fluent text. The practical job is to detect a meaningful quality drop before a customer, employee or campaign becomes the test environment.
in
“Fable 5 got silently nerfed. I have proof. Here are the 7 things I dug up.”
Start the LLM Regression Testing Workflow With Real Work
Begin with the tasks that create business consequences. For Growth Cab that could mean researching an account, classifying a reply, drafting a personalized message or updating a client artifact. Public benchmarks may describe general capability. They do not tell me whether the system preserved a source, respected an exclusion rule or produced a claim that a salesperson can safely send.
Collect twenty to fifty representative cases before chasing hundreds. Include normal inputs, awkward inputs and the failures that already cost time. Each case needs the original input, the expected behaviour and the reason the case matters. A compact set built from production pain usually reveals more than a large synthetic set designed around what the system already handles well.
Protect the baseline. Save the model identifier, prompt, system instructions, temperature, tool definitions, retrieval context and any reasoning or effort control available to you. Record the date and provider response metadata. If several variables move together, a failed comparison tells you that something changed while giving you little evidence about which component caused it.
Build Checks in Two Layers
The first layer is deterministic. Validate schemas, required fields, citations, tool routes, forbidden phrases, length limits and permission boundaries with code. These checks are cheap, repeatable and easy to explain. A response that calls the wrong tool or invents a required identifier should fail immediately instead of receiving a sophisticated quality score.
The second layer handles graded quality. Write a short rubric for correctness, completeness, relevance, tone and evidence. Score each dimension separately. Human review is the strongest reference for high-impact work. An evaluator model can help at larger volume when you calibrate it against human decisions and keep the rubric stable across the comparison.
Avoid a single average score. A candidate can improve eighteen easy cases and break two critical ones while the average rises. Compare every case with the baseline, mark material regressions and group failures by task. Set hard gates for behaviours that must never degrade, then use aggregate scores to understand the rest of the distribution.
Run the Candidate Beside the Baseline
When a model, prompt or tool configuration changes, run the same test set through both versions. Store the raw outputs and evaluation results. Review disagreements instead of only reading a pass rate. The most useful question is specific: which business task became less reliable, under which input pattern, and what changed in the execution path?
Repeat noisy evaluations. Creative output and evaluator models can vary between runs, so one sample can create false confidence or a false alarm. Use several runs for high-variance tasks, keep deterministic settings where appropriate and investigate unstable cases. Stability itself is a quality signal when the workflow supports repeatable operational decisions.
Turn every confirmed production failure into a permanent case. Save the smallest input that reproduces it, add the expected behaviour and link the case to the incident. This makes the suite compound. The next provider update, prompt edit or tool migration has to prove that it did not reopen a failure your business has already paid to understand.
Monitor Changes You Did Not Make
A release test covers changes you control. Scheduled regression runs cover changes you do not. Providers can update routing, safety behaviour or configuration defaults without changing your application code. Run the critical set daily or weekly against the live route. Alert on case-level failures and metadata changes, then preserve enough evidence to reproduce the comparison.
For important workflows, send a small share of live traffic to a candidate while the baseline remains available. Compare acceptance rate, corrections, latency, cost and downstream outcomes. Define the rollback condition before the test begins. A canary is useful because an offline set cannot contain every strange input that real users will eventually produce.
Keep humans on irreversible decisions. Regression tests can approve a draft, extraction or classification pipeline. They should not silently expand authority to send, publish, pay or alter customer data. Quality and permission are separate gates. A system can produce an excellent answer and still be the wrong actor to execute it.
Where LLM Regression Testing Breaks
The first failure is a convenient test set. If every example comes from successful demos, the suite protects the demo. Add edge cases, adversarial inputs, missing context and real failures. Review the set whenever the product or customer base changes. A frozen benchmark becomes less useful when the work around it evolves.
The second failure is treating an evaluator score as truth. Model judges can prefer longer answers, familiar wording or outputs that resemble their own style. Calibrate them with human labels, inspect disagreements and keep the rubric visible. The judge should reduce review work. It should never make the quality definition impossible to challenge.
The third failure is testing the model in isolation. Most operational regressions come from the full system: a prompt, a retrieval source, a tool schema, a permission or a parser. Test components when diagnosing a problem. Approve releases on the end-to-end route that produces the business output, because that is where customers experience quality.
A Small Regression Suite Beats a Large Assumption
You do not need a research lab to start. Choose one valuable workflow, collect thirty cases, freeze the baseline, add deterministic checks and score the parts that require judgment. Run the suite before every material change and on a schedule against the live provider route. When something fails, keep the case forever.
My original post began with a suspicion that a model felt worse. The useful lesson was not the verdict about one model. It was the absence of a controlled comparison. An LLM regression testing workflow replaces memory and mood with cases, evidence and rollback rules. That is how an AI system becomes dependable enough to operate inside a company.
Every Thursday, AI Frontier gives you one signal, my read on it, and one practical play from the AI and go to market systems we run inside Growth Cab, all in under five minutes. The original post and discussion are on LinkedIn. If your AI workflow has changed without a clear release, send me the case. I want to know what your baseline caught.
