How the voice sales assistant checks its answers
The assistant separates language generation from data access, then checks generated answers against query results. These notes explain that design, the recorded latency and load tests, and the integration faults found during evaluation. Client details remain confidential.
← Back to the case studyThe answer pipeline, stage by stageView diagramHide diagram
The constraint that shaped everything
A sales rep repeats what the assistant says in front of a customer. If it misquotes a revenue figure, the rep finds out mid-conversation, and the product is finished for that rep. The design goal was to keep the language model away from the numbers: every figure comes from the database, and every figure in an answer is checked against what was served.
Most account answers use fixed templates. Model calls handle routing, select from data-backed options or supply wording, while application code controls queries, calculations and permissions. Each part can be tested separately, then exercised together against the real models.
Seven stages, narrow model calls
Every question, typed or spoken, runs the same pipeline. Classify asks the model what the rep wants and maps it onto a fixed set of intents; anything off-topic is refused here. Resolve turns names into records, asking a clarifying question when a name is ambiguous rather than guessing. Permission checks use the rep's scope from the client's user records before data is returned. The query tool runs a parameterised query our code wrote. Feed strip removes any field the rep is not entitled to see. Answer writes the reply, from a fixed template for most account questions or with a model where the wording should vary, and validate checks it against the data that was actually served, falling back to a deterministic template if a figure a model wrote fails to match.
Model calls are the only non-deterministic steps, and each has a narrow job: classify can only choose from known intents, other calls choose from lists the data supplies or add wording around returned figures, and on account answers any model text carrying a figure it was not given is dropped or replaced. Classification can still select the wrong topic. Source validation checks the facts returned for that topic; live-model evaluation separately checks whether the question was routed correctly.
Grounding, checked rather than hoped for
The validation gate
On the typed account path, numeric tokens and the account name are checked against the tool results. Where a model wrote the wording, a failed check replaces it with a template. This check does not establish that every figure has the right label or catch numbers written as words.
Failure rate as a metric
Spoken figures are checked after speech and mismatches contribute to a monitored failure rate with an alert threshold. This helps identify changes in model behaviour; it detects spoken errors rather than preventing them.
Sanity checks on served data
Figures are also checked for internal consistency before serving: a value that cannot be true, whatever its source, is flagged and logged rather than passed through on trust.
Honest gaps
Topics the system knows it does not cover are declared in a registry. Ask about one and the answer says so plainly instead of improvising, and compound questions name the part they did not answer and offer it.
Data access and permissions
Client business data is read through fixed, parameterised queries; no model output becomes SQL. The application writes conversation history and operational records to its own tables. Each rep's scope, which accounts they may see and which data categories they may ask about, is checked on each request using the client's own user records, cached for up to five minutes, so a change on the client's side applies without signing in again. The signed session token carries only who is asking and which data set they're looking at. A request cannot widen its own scope, whatever the question says.
Stripping happens before caching, and the cache key includes a hash of the scope it was stripped for, so cached answers are separated by permission scope. Sign-in verifies against the client's existing user store and password hashes; no accounts were migrated and no parallel credential store was created.
Voice engineering
Real-time speech
Speech-to-speech over WebRTC, with the browser holding an ephemeral session key issued by our server. Tool calls made by the voice model run through the same pipeline as typed questions, under the same permissions. The voice model words the answer itself, and every figure it speaks is checked against what was served.
Barge-in
The rep can talk over an answer, say stop, or tap to interrupt, and the assistant yields. Session states are a pure state machine, and the stop and echo rules are pure functions, all tested without audio.
Spoken answers fit speech
A list that reads fine on screen is unbearable read aloud. Spoken answers lead with the few figures that matter and offer the rest; asking for more continues from held state without another model call.
An offline simulator
A simulator runs the full voice flow with no API key, for development. In CI, the state machine, the barge-in rules and the browser voice controller are tested with audio and network mocked.
Measured, with the method stated
On 14 October 2026, fifteen representative account questions were each run twice through the pipeline on a development machine, using the live model and the client's development database. The cold-cache median was 2.7 seconds; the warm-cache median was 1.8 seconds. The slowest cold run took 12.1 seconds. These are in-process measurements, excluding browser, HTTP and sign-in overhead; they do not measure spoken latency. Web research questions were outside this set and can take 20 seconds or more.
Load was tested on 14 October 2026 in bursts of 10 to 100 simultaneous in-process questions from a development machine against the development database, with the model stubbed. This measures the application pipeline and database, not concurrent voice sessions or production capacity with real model traffic. No request failed in two runs, and at 100 every answer came back within 6.5 seconds. Requests queue when demand exceeds the connection pool, and the pool size was chosen from that curve rather than guessed.
Evaluation, offline and live
Three layers. More than 2,400 automated tests run with no key and no database, covering the pipeline, the permission rules, the voice state machine and the validation gate. A golden suite locks the exact answers for a set of reference questions, so a change that alters an answer fails a test rather than shipping silently. To check behaviour beyond the fixtures, a routing evaluation replays the approved questions against the real model and is run three times after any change to routing, with each case checked against its own baseline, and a spoken evaluation runs against the client's development database before each deployment.
The live layer earns its keep: its first run exposed three faults the offline suite could not see, all fixed before any user met them.
Faults the process caught
On the voice path, the browser forwarded only two of the arguments the model sent with a tool call, and every server-side test stayed green because the loss happened before the server. Comparing what the model sent with what the server received found it. Both ends are logged now.
Two different questions about the same account shared a cache entry, so asking what a customer had stopped buying could return the previous answer about top sellers. Caught by reading answers end to end rather than trusting that a served result meant a correct one.
One path reported answered even when there was no data behind the answer, which would have poisoned the accuracy figures we report. Found by auditing outcomes against the data, fixed, and covered by tests that assert the failure is recorded as a failure.
Deployment and operations
The system ships as a hardened multi-stage Docker image, served over HTTPS behind nginx on the client's infrastructure. Removing the package manager and build tooling from the runtime image cut the critical and high findings in an image scan from twelve to four (August 2026), and what remains is documented rather than ignored.
Every request carries a correlation id through per-stage timing spans, and every question and its outcome are logged, with the answer as written and as spoken. When a question about an answer comes up weeks later, the evidence exists.