Building Regression Tests for an AI Support Workflow

One of the most important parts of AI support reliability work is regression testing. If an AI support agent answers customer questions, there needs to be a way to check whether previously correct behavior still holds after changes to knowledge, product or configuration.

Whether the system relies on five knowledge sources or hundreds, the principle is the same. A change intended to improve one answer shouldn't silently break another part of the workflow.

For this evaluation, I used Fin's Batch Tests and focused on one billing workflow:

failed payment → 14-day grace period → expiry with read-only → recovery

I kept the knowledge base small with only three short articles describing the grace period, what happens after expiry and a provider-specific recovery rule. This kept the expected behavior easy to inspect and made it easier to judge whether an answer was actually supported by the available knowledge.

Why random FAQ questions were not enough

A regression suite shouldn't be a collection of common questions paraphrased a few times. For this workflow, I designed the test suite to cover the following:

• normal customer paths
• different customer states
• missing information
• ambiguous wording
• unsupported questions
• previously observed risky behavior.

The initial suite contained 25 cases. Two questions could sound similar while representing two different customer states. For example, "My payment failed 5 days ago. Can I still use the service?" and "It's been 13 days since my payment failed. Do I still have access?" are both questions about the grace period. However, they are not equivalent. The second customer is much closer to expiry so slightly wrong recovery guidance could have a more immediate consequence.

That's why a good regression suite needs to represent states and conditions rather than only variations in wording.

The first baseline

After running the suite and reviewing the answers, the first run produced 14 Good answers, 8 Acceptable answers and 3 Poor answers.

All three Poor answers belonged to the same failure pattern. One of the sources I created had the following statement:

A successful payment during the 14-day grace period restores the subscription.

In several answers, Fin weakened this requirement into guidance such as “update your payment before the grace period ends” or “update your payment to keep going.”

These statements are close to the source but updating payment details doesn't mean the payment succeeded. A customer could change their card and assume the problem was resolved then later discover the payment failed and their subscription expired.

Not every imperfect answer was a failure

Not every unsupported sentence should automatically become a failed test. Some answers were operationally correct while still containing plausible but unsupported details.

Fin occasionally added payment processing language, suggested checking places such as a billing dashboard or used stronger wording about data availability than the source supported. These answers were imperfect but the customer still received the correct core guidance.

I separated material failures from lower quality issues based on whether the answer could change the customer's action or outcome. If the core guidance remained correct and the unsupported wording didn't misdirect the customer, I treated the answer differently from one which removed a critical condition or could lead to the wrong action.

This calibration was important because an evaluation could become strict in ways which look rigorous without reflecting actual customer risk.

Making one controlled change

The repeated payment-condition failure gave me one specific intervention to test. I updated the statement in the related article.

The original fact stated that a successful payment during the grace period restores the subscription. I added the following statement to it:

Updating payment details alone does not restore the subscription. The payment must complete successfully during the grace period.

I left the other knowledge unchanged because changing several things at once would make the result much harder to interpret.

Targeted validation before full regression

I reran the previously failing cases together with a few risk cases and controls. The failure disappeared. Fin preserved the successful payment requirement in the cases where it previously weakened it. One answer still used slightly loose wording but it no longer replaced the payment success requirement with a payment method update.

The controls also remained correct after the rerun. Once the targeted validation showed the intended improvement, I ran the whole regression suite again to check whether the knowledge change damaged another part of the workflow.

The original failure cases continued to preserve the successful payment condition and I didn't observe a new failure elsewhere.

Conclusion

A good regression suite represents customer states, conditions, ambiguity and rules which must remain stable. Multiple failing prompts may reveal one behavioral pattern rather than several separate problems. An answer can also look broadly correct while weakening the one condition which actually determines the customer's outcome.

When a material failure is fixed, the test which exposed it should become part of the permanent regression suite so the same behavior doesn't silently return after the next knowledge, configuration or product change.