Your Forty Answers Do Not Need a Vector Database
A salon has maybe forty things a caller ever asks. Prices, opening hours, parking, whether you do gel removal, how long a full set takes, what happens if they are late. That content fits in a prompt with room to spare.
Put it behind retrieval instead and you have added a network hop to every turn, a similarity threshold you cannot interpret, and a failure mode where the agent finds nothing and answers anyway. In a chat window a bad retrieval is survivable, because the user sees three sources and scans them. On a phone call there are no sources, no scanning, and one spoken answer that is either right or a confident invention.
This piece is about when that trade is worth making. Sometimes it genuinely is. For most UK small businesses, on the documented evidence below, it is not.
Key takeaways
- Retell documents that retrieval runs automatically before every single response, including turns where the caller said "thanks".
- No platform ships a strict refusal by default. Retell's own answer is a prompt snippet you paste in yourself.
- Similarity thresholds are not comparable across vendors. Azure's score for a cosine query is 1/(1+distance), so 0.5 means orthogonal, not half-relevant.
- Semantic chunking does not reliably beat naive fixed-size chunking, per a NAACL 2025 paper whose author list includes a RAG vendor.
- Price lists and opening-hours tables are the content retrieval handles worst, and they are most of what a small business needs answered.
What does a knowledge base actually do on each platform?
Wildly different things, and only one platform tells you enough to reason about it. All figures below were read on 4 September 2026, and defaults in this space change without notice.
| Platform | What is documented |
|---|---|
| Retell | Vector store, retrieval on every turn, top-k 3, similarity threshold 0.6, chunk size 2,000 characters and immutable after creation |
| ElevenLabs | max_vector_distance 0.6, max_documents_length 50,000, max_retrieved_rag_chunks_count 20; index size capped by plan tier, from 1MB on Free to 1GB on Business |
| Vapi | Its knowledge-base page publishes no chunking, embedding, top-k or threshold information at all |
| Synthflow | Contradicts itself on the same page about whether retrieval is a model-chosen tool call or an automatic parallel process |
Retell's transparency is a real credit to it and it is also what makes the costs legible. Its documentation states that when a knowledge base is linked, "the agent will always try to retrieve information from the knowledge base before responding" with "no need to change your prompt for it to trigger". Every turn. Including the ones where the caller says "yes, that's fine".
One more Retell detail worth knowing before you debug anything: the search query is not what the caller said. Retell "condenses the recent conversation into a short, standalone search query", and the agent's own prompt is excluded from it. When a retrieval inexplicably misses, the thing that missed is a paraphrase you never see unless you open the call log.
Two numbers that cannot be compared
Retell says knowledge-base retrieval "should generally be under 100ms of latency impact". ElevenLabs publishes around 250ms. Neither figure carries a percentile, an index size, a date or a method, so the 2.5x gap between them tells you nothing about which is faster. Treat both as vendor claims and measure your own.
For a sense of where the budget goes, one vendor publishes an actual breakdown: query rewrite 80ms p50 against a 150ms ceiling, vector search 60ms p50 against 120ms, reranking 80ms p50 against 150ms, inside an end-to-end first-audio target of 600 to 800ms. Note the p50s. Half your turns are expected to exceed them, and the end-to-end budget is the subject of the latency guide.
What happens when retrieval finds nothing?
The agent answers from its own weights, and nothing in the transcript marks that it did. This is the finding that should decide the architecture for most small businesses.
Retell's documentation addresses it in an FAQ, and the answer is a prompt you write yourself: instruct the model to "Only answer using the information in ## Related Knowledge Base Contexts" and, when that section is missing or irrelevant, to say so. That is good advice. It is also prompt work you own on every platform, not a safety net any of them ships enabled.
So the refusal path is the deliverable, not the retrieval. Which raises the obvious question: if you are writing a strict answer-only-from-approved-content instruction regardless, what is the retrieval adding for forty answers that already fit in the prompt?
Your threshold is not a percentage
Practitioners tune a similarity threshold as if 0.6 meant sixty per cent confident. It does not, and the scale is not even the same between vendors.
Azure AI Search is the clearest documented example: for a cosine vector query, @search.score is not the cosine value but 1/(1+cosine_distance), bounded between 0.333 and 1.00. A score of 0.5 implies a cosine distance of 1.0, which is a similarity of zero. Two orthogonal, entirely unrelated vectors. Anyone who set a 0.5 floor believing it was a midpoint set it below "no relationship at all".
Retell's 0.6 and ElevenLabs' max_vector_distance of 0.6 are numbers on different scales measuring different things in opposite directions. Copying a threshold between platforms is meaningless, and so is reasoning about one without reading how that vendor computes it.
Why price lists are the worst case
Because embeddings are weak on exactly the content a small business most needs answered, and because chunking is structure-blind.
Retell's chunker works in characters, defaults to 2,000 with a 400-character merge floor, and is immutable after the knowledge base is created. A weekly opening-hours table can be cut mid-week. A price row can be separated from its heading. Fixing it means rebuilding the knowledge base rather than changing a setting.
What the research says about tables
TableRAG, published at EMNLP 2025, measured naive RAG at 34.54 against its table-aware approach at 44.19 on the HeteQA benchmark with a Claude 3.5 backbone, though note HeteQA is 304 examples and the paper's headline "at least 10%" is a relative claim. The direction is unambiguous: generic chunk-and-embed handles tabular content badly, and a price list is a table.
And the fix people reach for does not reliably help. A NAACL 2025 paper found that semantic chunking does not consistently beat naive fixed-size chunking. Its author list includes someone affiliated with a RAG vendor, which makes the result more interesting rather than less, since it cuts against the seller's interest.
For reference, OpenAI's file_search defaults to 800-token chunks with 400-token overlap, which is fifty per cent overlap, and returns 10 results by default.
Opening hours are a data problem
Bank holidays are the case that exposes retrieval as the wrong tool. UK bank holidays are published as a free official JSON feed with three separate divisions, and the divergence is not cosmetic: the summer bank holiday falls on 3 August in Scotland and 31 August in England and Wales in the same year. A Scottish business running English hours is open on a day it should be shut, and shut on a day it should be open.
No amount of chunking a PDF fixes that. It is a structured-data problem with an authoritative free source, and the answer is to compute the response rather than retrieve it.
While we are here, a caution about trusting authoritative-looking specs blindly: Google's structured-data documentation for local businesses encodes "closed" and "open 24 hours" as sentinel time values rather than a boolean, and the property name in one of its own examples is misspelled. A model ingesting that page inherits the error.
When retrieval does earn its place
Three conditions, and you want at least two of them.
Volume. Hundreds of distinct answers rather than dozens. A single-site salon is not this. A multi-branch group with different prices, staff and hours per branch might be.
Volatility. Content that changes weekly and is maintained by someone who will not touch a prompt. A live product catalogue qualifies. A price list revised twice a year does not.
Long-tail questions. Callers asking things nobody anticipated, where the value is coverage rather than accuracy on the top forty.
Even then, know what you are buying. On Retell, retrieved chunks are appended to the prompt under a named header and count toward your prompt tokens, so retrieval does not shrink the context, it makes it variable. Billing follows the same shape: the first ten knowledge bases are free, then $8 per month each, plus $0.005 per minute for every call with a knowledge base enabled, charged whether retrieval helped or not, because it runs regardless.
Brittleness worth knowing about
Retrieval can fail for reasons that look like nothing. In one published comparison, rewording a single question dropped a RAG pipeline's accuracy from 80.34% to 4.58% while a long-context baseline stayed flat at 69.32%. That figure comes from PassKey, a synthetic needle-in-a-haystack task where a literal string match carries the retrieval, so it overstates what you would see on real business content. It is still the right shape of warning: retrieval quality is sensitive to phrasing in ways a prompt is not, and your callers will not phrase things the way your documents do.
If you do run it, add reranking
Reranking is the highest-value addition you can make. On the BEIR benchmark, cross-encoder reranking gave roughly +11% average nDCG@10 over BM25 across 18 datasets, winning 16 of 18. The latency figures from that work are on 2021-era hardware and should be treated as a ratio rather than a number: reranking is roughly an order of magnitude more expensive than the retrieval it improves, which is why it sits on a 150ms ceiling in the budget above.
One number to distrust if you meet it: the widely circulated "79% accuracy at position one falling to 62% mid-context" attributed to the lost-in-the-middle paper does not appear in that paper. The effect is real and well documented. Those specific figures are not in the source.
What we do instead
For a business with content that fits in a prompt, which is most of them:
- A curated answer set the owner signs off, written as spoken answers rather than extracted from a PDF. For a nail salon that is one page covering services, timings and the late-arrival policy. If the answer is wrong, one person changed one line, and you can see which.
- Computed answers for anything with structure: opening hours from the bank-holiday feed and a rota, not from prose.
- A hard refusal path, in the wording of the guardrails guide: if it is not in the approved set, the agent says it will find out and takes a message.
- A log of every question that hit the refusal. That list is the actual product. After a fortnight it tells you the ten answers you were missing, and you add them.
That last point is the one that matters. Retrieval promises coverage you have not thought about yet. A refusal log gives you the same information, in your callers' own words, with none of the failure modes above, and it turns the gap into a decision instead of a guess.
Frequently asked questions
How much content is too much for a prompt? There is no clean threshold, and anyone quoting one is guessing. The practical test is not size but maintenance: if one person can read the whole answer set in a sitting and tell you whether it is correct, keep it in the prompt. Note that platforms have their own ceilings, and at least one warns that very long prompts begin hitting caching limits well before the model's context window is full, so check yours rather than assuming the model's advertised context applies.
Does putting everything in the prompt cost more? Not necessarily, and on at least one platform retrieval is the more expensive option. Retell appends retrieved chunks to the prompt and bills them as prompt tokens on top of a per-minute knowledge-base fee, so you pay for the retrieval infrastructure and then pay again for the tokens it injects. A static prompt is cacheable in a way a per-turn variable prompt is not.
Can I just point it at my website? You can, and it is the fastest way to get a knowledge base full of navigation menus, cookie banners and last year's prices. If you do, crawl deliberately rather than broadly, check what was actually ingested, and remember that the page you edited is not the index the agent reads until it is re-indexed.
What is the one thing to test before going live? Ask it three questions your documents do not answer, and listen. A correct system says it does not know and offers to take a message. If it improvises a plausible answer to any of the three, the refusal path is not working, and no amount of retrieval tuning fixes that. The rest of the pre-launch battery is in the twelve test calls.