Technical note

Testing Help-Center Retrieval Before Generation

Update: This article describes the first version of the project. I later expanded it into a retrieval evaluation workbench with more test questions, BM25, grouped diagnostics by category/slice/difficulty and question-level inspection. The current repo and live demo reflect the newer version.

AI support assistants can fail before answer generation even starts. If the retrieval step brings back the wrong help-center article, the generated answer may still sound confident but it is grounded in the wrong source.

For this project, I wanted to evaluate that retrieval step using different strategies.

I built a small retrieval evaluation dashboard for a fake SaaS help center called ExampleOps. The dataset includes fake help-center docs, 16 realistic support questions, expected source documents for each question and script-generated retrieval results across four strategies.

Retrieval strategies

I compared four retrieval strategies against the same set of expected source documents:

Strategy	Description
`tfidf`	Keyword-based retrieval using TF-IDF and cosine similarity
`embedding-minilm`	Local semantic retrieval using `sentence-transformers/all-MiniLM-L6-v2`
`embedding-mpnet`	Local semantic retrieval using `sentence-transformers/all-mpnet-base-v2`
`hybrid-tfidf50-mpnet50`	50/50 weighted hybrid of normalized TF-IDF and MPNet scores

Metrics

I used the following metrics to evaluate each strategy’s ranking quality:

Metric	Meaning
`Hit@1`	Whether at least one expected source appears at rank 1
`Hit@3`	Whether at least one expected source appears in the top 3
`Hit@5`	Whether at least one expected source appears in the top 5
`Recall@3`	How many expected source docs appear in the top 3
`Recall@5`	How many expected source docs appear in the top 5
`MRR`	Mean reciprocal rank of the first correct source

Hit metrics show whether retrieval found at least one useful source
Recall metrics show whether retrieval covered all expected sources

Main findings

Aggregated results:

Strategy	Mean Hit@1	Mean Hit@3	Mean Hit@5	Mean Recall@3	Mean Recall@5	MRR
`tfidf`	0.875	1.000	1.000	0.781	0.927	0.927
`embedding-minilm`	0.813	1.000	1.000	0.875	0.948	0.896
`embedding-mpnet`	1.000	1.000	1.000	0.917	0.948	1.000
`hybrid-tfidf50-mpnet50`	0.938	1.000	1.000	0.844	1.000	0.958

TF-IDF can over-rank exact terms with the wrong intent

TF-IDF performs well when the question contains exact terms, such as invoice, payment, refund or export. However, it can struggle when the question uses language that also matches the wrong sources, as in q010:

Why do I still have paid features after cancelling?

TF-IDF got distracted by the billing and cancellation language. The expected sources were cancellation and end-of-period access but refund-policy was ranked first. That produced a missed Hit@1 even though one of the expected sources still appeared later in the ranking.

In such cases, embedding retrieval can improve semantic matching when the query describes a product rule indirectly such as continued access after cancellation.

Results of different strategies for q010

MiniLM improved some semantic cases but is not consistently better

all-MiniLM-L6-v2 improved semantic coverage in some cases but it can also over-rank semantically related but incomplete sources.

q007:

The old billing admin left the company and nobody can update the payment method.

MiniLM over-focused on the admin-left-company part of the question and ranked the admin-role source above the billing-permissions source.

Switching to all-mpnet-base-v2 handled the intent better and ranked the billing-permissions source first.

MPNet is the strongest current retrieval baseline

all-mpnet-base-v2 performs best overall on this dataset. It improved top-ranked source quality and semantic source coverage compared with both TF-IDF and MiniLM.

However, it still misses some supporting sources in multi-source questions which is why Recall@k remains important.

50/50 hybrid retrieval weakened early ranking

The 50/50 TF-IDF + MPNet hybrid improved full top-5 coverage but hurt early ranking compared with pure MPNet.

For cases that use exact wording from existing sources but have a different intent, equal weight distribution between TF-IDF and embeddings can hurt the ranking.

One case that didn't perform well with the 50/50 hybrid strategy was q004:

The invoice says paid but my team still can't access the workspace.

Looking at the results below, TF-IDF struggled to rank the expected sources early because the question mixes invoice/payment wording with workspace-access intent. Running MPNet alone ranked the access-related source higher while the 50/50 hybrid was pulled back toward literal invoice and team wording because the lexical signal had significant weight (50%).

Results of different strategies for q004

Note: I only tested a simple 50/50 hybrid baseline. Different weights may perform better but I didn't tune weights because the goal was to compare retrieval behavior rather than optimize a small synthetic dataset.

What I would improve next

The next version would need a larger dataset, more varied source docs and more failure cases before making stronger claims. I would also test embedding-heavy hybrid weights, reranking, query rewriting and a clearer failure taxonomy for source-coverage issues.