How the voice sales assistant actually works
The case study says what was built. This sheet says how: the pipeline that makes invented figures structurally impossible, the way the system was measured, and the faults the process caught before users could. The client is under NDA, so nothing here identifies them; every figure on this page is our own measurement of our own build.
← Back to sheet 1, the case studyThe constraint that shaped everything
A sales rep repeats what the assistant says in front of a customer. If it misquotes a revenue figure, the rep finds out mid-conversation, and the product is finished for that rep. So the design goal was never "usually right". It was that a figure the system cannot trace back to the database does not get spoken at all.
That rules out the default architecture, where a language model reads some context and writes the answer. Instead the model is confined to two jobs it is genuinely good at: working out what the rep is asking, and phrasing a reply. Everything between those two points is deterministic code that can be tested, measured and audited.
- The model never writes SQL; it picks from approved, parameterised query tools
- Figures, dates and names are immutable; phrasing is free
- Restricted fields are stripped before the model or any cache sees them
- Every answer carries its source, and voice answers say what they could not cover
Seven stages, two model calls
Every question, typed or spoken, runs the same pipeline. Classify asks the model what the rep wants and maps it onto a fixed set of intents; anything off-topic is refused here. Resolve turns names into records, asking a clarifying question when a name is ambiguous rather than guessing. Permission pins the rep's scope to the session before any data is fetched. The query tool runs a parameterised query our code wrote. Feed strip removes any field the rep is not entitled to see. Answer has the model phrase the result, and validate checks that phrasing against the data that was actually served, falling back to a deterministic template if a single figure fails to match.
The two model calls are the only non-deterministic stages, and each is fenced: classify can only choose from known intents, and answer can only rephrase what the tools returned. A wrong classification produces a wrong-but-grounded answer or a clarifying question, never an invented figure.
Grounding, checked rather than hoped for
Numbers, dates and entity names in a generated answer are checked against the tool results they must have come from. A mismatch means the phrasing is discarded and a deterministic template serves the same facts instead, so the correct answer still goes out.
The grounding failure rate is exported from a metrics endpoint with an alert threshold, so drift in model behaviour shows up in monitoring rather than in a customer conversation.
Figures are also checked for internal consistency before serving: a value that cannot be true, whatever its source, is flagged and logged rather than passed through on trust.
Topics the system knows it does not cover are declared in a registry. Ask about one and the answer says so plainly instead of improvising, and compound questions name the part they did not answer and offer it.
Data access and permissions
The database connection is read-only and every query is parameterised; there is no code path by which model output becomes SQL. Each rep's scope, which accounts they may see and which data categories they may ask about, is resolved at sign-in from the client's own user records and pinned to the session server-side. A request cannot widen its own scope, whatever the question says.
Stripping happens before caching, and the cache key includes a hash of the scope it was stripped for, so one rep's answer can never be served to another with different permissions. Sign-in verifies against the client's existing user store and password hashes; no accounts were migrated and no parallel credential store was created.
- Read-only, parameterised queries only
- Scope pinned per session at sign-in
- Restricted fields stripped before model and cache
- Scope hash in every cache key
- Per-user history, isolated by account
Voice engineering
Speech-to-speech over WebRTC, with the browser holding an ephemeral session key issued by our server. Tool calls made by the voice model route through the same pipeline as typed questions, so the grounding rules hold on both paths.
The rep can talk over an answer, say stop, or tap to interrupt, and the assistant yields. The interaction rules live in a pure state machine that is tested without any audio at all.
A list that reads fine on screen is unbearable read aloud. Spoken answers lead with the few figures that matter and offer the rest; asking for more continues from held state without another model call.
The full voice flow, including barge-in, runs against a simulator with no API key and no audio hardware, so voice behaviour is exercised in CI rather than only by hand.
Measured, with the method stated
Latency was measured by running the question set through the real pipeline against the live database: median under three seconds, and the slowest question around eleven. The slow tail is the questions that make a second model call to write the phrasing; the deterministic template could remove it, and keeping the better phrasing was a deliberate trade made with the client.
Load was ramped from 10 to 100 concurrent users against the live database with the model stubbed, so the test measured our ceiling rather than the model provider's. Nothing failed at any level; requests queue when demand exceeds the pool, and the pool size was chosen from that curve rather than guessed.
Evaluation, offline and live
Three layers. Over 700 automated tests run with no key and no database, covering the pipeline, the permission rules, the voice state machine and the validation gate. A golden suite locks the exact answers for a set of reference questions, so a change that alters an answer fails a test rather than shipping silently. And because a green fixture suite proves nothing about a live model, a routing evaluation replays the full approved question set against the real model, and has to come back clean on consecutive runs before a ship.
The live layer earns its keep: its first run exposed three faults the offline suite could not see, all fixed before any user met them.
Faults the process caught
On the voice path, the browser forwarded only two of the arguments the model sent with a tool call, and every server-side test stayed green because the loss happened before the server. Comparing what the model sent with what the server received found it. Both ends are logged now.
Two different questions about the same account shared a cache entry, so asking what a customer had stopped buying could return the previous answer about top sellers. Caught by reading answers end to end rather than trusting that a served result meant a correct one.
One path reported answered even when there was no data behind the answer, which would have poisoned the accuracy figures we report. Found by auditing outcomes against the data, fixed, and covered by tests that assert the failure is recorded as a failure.
Deployment and operations
The system ships as a hardened multi-stage Docker image, served over HTTPS behind nginx on the client's infrastructure. Removing the package manager and build tooling from the runtime image cut the reported CVE count from twelve to four, and what remains is documented rather than ignored.
Every request carries a correlation id through per-stage timing spans, and every conversation is kept as a transcript: what was asked, what was served, what was spoken. When a question about an answer comes up weeks later, the evidence exists.
- Multi-stage Docker build, minimal runtime image
- HTTPS behind nginx, secrets via environment
- Metrics endpoint with alert thresholds
- Correlation ids and per-stage timings
- Full transcripts, retained per user