How the voice sales assistant checks its answers
The assistant separates language generation from data access, then checks generated answers against query results. These notes explain that design, the recorded latency and load tests, and the integration faults found during evaluation. Client details remain confidential.
← Back to the case studyThe answer pipeline, stage by stageView diagramHide diagram
The constraint that shaped everything
A sales rep repeats what the assistant says in front of a customer. If it misquotes a revenue figure, the rep finds out mid-conversation, and the product is finished for that rep. The design goal was to validate generated figures against the database results before speaking them.
That rules out the default architecture, where a language model reads some context and writes the answer. The model handles question classification and reply wording. Everything between those two points is deterministic code that can be tested, measured and audited.
Seven stages, two model calls
Every question, typed or spoken, runs the same pipeline. Classify asks the model what the rep wants and maps it onto a fixed set of intents; anything off-topic is refused here. Resolve turns names into records, asking a clarifying question when a name is ambiguous rather than guessing. Permission pins the rep's scope to the session before any data is fetched. The query tool runs a parameterised query our code wrote. Feed strip removes any field the rep is not entitled to see. Answer has the model phrase the result, and validate checks that phrasing against the data that was actually served, falling back to a deterministic template if a single figure fails to match.
The two model calls are the only non-deterministic stages, and each is fenced: classify can only choose from known intents, and answer can only rephrase what the tools returned. Classification can still select the wrong topic. Source validation checks the facts returned for that topic; live-model evaluation separately checks whether the question was routed correctly.
Grounding, checked rather than hoped for
The validation gate
Numbers, dates and entity names in a generated answer are checked against the tool results they must have come from. A mismatch means the phrasing is discarded and a deterministic template serves the same facts instead, so the returned facts can still be presented.
Failure rate as a metric
The grounding failure rate is exported from a metrics endpoint with an alert threshold, so drift in model behaviour shows up in monitoring rather than in a customer conversation.
Sanity checks on served data
Figures are also checked for internal consistency before serving: a value that cannot be true, whatever its source, is flagged and logged rather than passed through on trust.
Honest gaps
Topics the system knows it does not cover are declared in a registry. Ask about one and the answer says so plainly instead of improvising, and compound questions name the part they did not answer and offer it.
Data access and permissions
The database connection is read-only and every query is parameterised; there is no code path by which model output becomes SQL. Each rep's scope, which accounts they may see and which data categories they may ask about, is resolved at sign-in from the client's own user records and pinned to the session server-side. A request cannot widen its own scope, whatever the question says.
Stripping happens before caching, and the cache key includes a hash of the scope it was stripped for, so cached answers are separated by permission scope. Sign-in verifies against the client's existing user store and password hashes; no accounts were migrated and no parallel credential store was created.
Voice engineering
Real-time speech
Speech-to-speech over WebRTC, with the browser holding an ephemeral session key issued by our server. Tool calls made by the voice model route through the same pipeline as typed questions, so the grounding rules hold on both paths.
Barge-in
The rep can talk over an answer, say stop, or tap to interrupt, and the assistant yields. The interaction rules live in a pure state machine that is tested without any audio at all.
Spoken answers fit speech
A list that reads fine on screen is unbearable read aloud. Spoken answers lead with the few figures that matter and offer the rest; asking for more continues from held state without another model call.
An offline simulator
The full voice flow, including barge-in, runs against a simulator with no API key and no audio hardware, so voice behaviour is exercised in CI rather than only by hand.
Measured, with the method stated
Latency was measured by running the question set through the real pipeline against the live database: median under three seconds, and the slowest question around eleven. The slow tail is the questions that make a second model call to write the phrasing; the deterministic template could remove it, and keeping the better phrasing was a deliberate trade made with the client.
Load was ramped from 10 to 100 concurrent users against the live database with the model stubbed, so the test measured our ceiling rather than the model provider's. No requests failed in that recorded test; requests queue when demand exceeds the pool, and the pool size was chosen from that curve rather than guessed.
Evaluation, offline and live
Three layers. Over 700 automated tests run with no key and no database, covering the pipeline, the permission rules, the voice state machine and the validation gate. A golden suite locks the exact answers for a set of reference questions, so a change that alters an answer fails a test rather than shipping silently. To check behaviour beyond the fixtures, a routing evaluation replays the full approved question set against the real model, and has to come back clean on consecutive runs before release.
The live layer earns its keep: its first run exposed three faults the offline suite could not see, all fixed before any user met them.
Faults the process caught
On the voice path, the browser forwarded only two of the arguments the model sent with a tool call, and every server-side test stayed green because the loss happened before the server. Comparing what the model sent with what the server received found it. Both ends are logged now.
Two different questions about the same account shared a cache entry, so asking what a customer had stopped buying could return the previous answer about top sellers. Caught by reading answers end to end rather than trusting that a served result meant a correct one.
One path reported answered even when there was no data behind the answer, which would have poisoned the accuracy figures we report. Found by auditing outcomes against the data, fixed, and covered by tests that assert the failure is recorded as a failure.
Deployment and operations
The system ships as a hardened multi-stage Docker image, served over HTTPS behind nginx on the client's infrastructure. Removing the package manager and build tooling from the runtime image cut the reported CVE count from twelve to four, and what remains is documented rather than ignored.
Every request carries a correlation id through per-stage timing spans, and every conversation is kept as a transcript: what was asked, what was served, what was spoken. When a question about an answer comes up weeks later, the evidence exists.