Testing recurring payment retry guidance against the documented workflow

Context

I tested a payments platform's public AI support assistant against its documented recurring subscription workflow. The broader assessment covered failed payments, retries and cancellation.

This case study focuses on one finding: how the assistant explained what happens after a recurring payment fails.

I phrased the questions from the perspective of a merchant trying to understand what would happen to a customer's subscription after a failed payment.

Expected behavior

The documentation had the following failed payment rules:

• After a subscription payment fails, the payment may enter a retry cycle depending on the failure reason.
• Another collection attempt can happen on a later day. The subscription is only cancelled after the applicable retries have been exhausted.

So a failed payment isn't always the final state. Another collection attempt may still happen before the subscription is cancelled.

Observed behavior

I tested the retry rule using three differently worded questions. The first asked what happens after a subscription payment fails. The second asked whether a failed recurring payment would be retried automatically. The third asked whether another payment attempt could happen the following day.

In all three responses, the assistant said that failed subscription payments weren't retried. This directly contradicted the documented workflow.

Why it matters

A merchant relying on this answer could assume that the first failed subscription payment is final even when another attempt was still possible.

This could affect how the merchant handles failed renewals in their own systems or communicates about them to customers.

I classify this as a high severity finding because the incorrect guidance concerns payment recovery and the interpretation of subscription state.

Investigating the cause

Identifying the cause requires access to the internal system.

The first step is checking whether the documented retry guidance was retrieved and included in the context used to generate each answer:

• If it was absent, investigate source coverage, retrieval and context construction.
• If it was present, investigate why the generated answer contradicted it.

Once the failure point is located, make one change at a time and rerun the affected cases. Depending on where the problem was introduced, this could mean changing source content, retrieval configuration or generation instructions.

Regression test

Add this finding as a regression case for the subscription retry workflow.

One case should cover a failed subscription payment where another payment attempt may still be possible. An answer claiming that failed payments are never retried should fail the test.

A paired case should cover a failed payment that won't be retried. The answer should explain why no retry will happen without implying that this applies to all failed subscription payments.

Retain the three original question formulations as part of the regression coverage. The contradiction appeared across all three so the wording variation is worth keeping.