How the estimation engine runs and checks its outputs
These notes explain how QPOI parallelised the estimation pipeline, checked it against the sequential baseline and kept reviewed totals in application code. They also describe the recorded test method and the faults caught during validation. The client and platform remain confidential.
← Back to the case studyThe analysis pipeline, stage by stageView diagramHide diagram
The constraint that shaped everything
The estimate this pipeline produces feeds quotes, bid packages and a generated contract. So when the client asked for the analysis to be faster, there was one non-negotiable: faster execution had to preserve the scope and numerical consistency of the analysis. Runtime alone was not enough to accept the change.
That constraint set the acceptance gate before any code changed: every iteration of the parallel pipeline re-ran the same real blueprint and was compared with the sequential baseline, stage by stage and trade by trade. More than one iteration failed that gate and never shipped. Section 6.0 covers what the failures taught us.
Untangling a stateful pipeline
The pipeline began life as one long stateful conversation: each stage read everything before it, which made the output coherent and the whole thing impossible to parallelise naively. The untangling started by asking, per stage, what it actually needs. The opening stages depend on each other, so they stay sequential. The sixteen trade analyses each need the project context and the building's measurements, but not each other, so the measurements were promoted into an explicit quantity basis stage: computed once, handed to every trade as authoritative context, and launched as soon as the initial analysis exists so the quantity calculation overlaps with the remaining opening stages. Every trade receives the same measurement basis, giving reviewers a common reference for checking its output.
The trades then run in parallel under a bounded concurrency cap, and the tail, the stages that read the assembled report, runs in six dependency-ordered waves. Two anchors are enforced in code rather than asked of the model: the cost rollup is summed from the persisted trade outputs, and the executive summary is locked to that rollup, so no stage can invent a second total. Stored stage results let reviewers check which trades completed and investigate gaps in a run.
Keeping money deterministic
Parse structure, not labels
Language models intermittently drop forced markers and section labels, and under parallel load they drop them more. Every parser that keyed on a literal label eventually broke on some trades, so the rule is now uniform: find a table by its headers, find a section by its shape, and treat labels as decoration.
Numbers are arithmetic
Once an estimate exists, its figures live in a canonical store where recalculation is a fixed formula, not a model call. Editing a line item is maths. AI is reserved for the edges, suggesting a replacement product or regenerating a narrative report, and the reviewer decides what is applied.
One figure, many disguises
The same total appears across documents as a raw number, a grouped figure and a truncated abbreviation. When an edit changes it, substitution handles each shape, applies the longest match first so a fragment cannot corrupt a fuller figure, and reproduces the document's own style.
Guarded find-and-replace
Substitution into prose and contract text is fenced: a percentage is only rewritten when the rate it belongs to is named alongside it, abbreviations that could collide with a product name are skipped, and an ambiguous match is suppressed entirely. A safe miss is acceptable; a corruption is not.
Measured, with the method stated
Our analysis-time comparison uses stored timestamps from job creation to the first executive summary: 19 completed sequential runs and 20 parallel runs in the development and QA environment, July to September 2026. Median duration fell from 24.0 to 16.3 minutes. The runs used different blueprints, and prompt changes accompanied the engine redesign. This measures processing time before human review, rather than time saved by customers.
A separate validation run produced a material total within 0.1% of its sequential baseline. That result applies to material costs in that run, not overall estimating accuracy or equivalence across runs. The platform figures were measured in July 2026. Backend query counts come from logging comparable, manually driven sessions before and after the change; the API-call counts come from a scripted browser session on the same job, with the pre-fix code restored from source control so the comparison is honest. One lesson is worth stating: on a small, far-away development database, milliseconds are mostly network noise, so we count queries rather than seconds. The query counts describe those sessions; they are not a claim about response time for every production user.
Tested against real artefacts
The dangerous parsers never meet production first. Before a parser changes, the new logic is replayed in simulation over stored analysis reports alongside the old logic, and the comparison is row counts and totals, not opinion. One such replay showed the old path recovering a single trade section from a report where the new path recovered all sixteen, plus general conditions; only with that evidence did the change ship.
The substitution logic that edits contract prose has its own test harness: fourteen checks covering the cases that make find-and-replace dangerous, such as two items sharing a rating where only one was edited, or a manufacturer name that reads like an abbreviated number. The harness compiles the production code directly rather than a copy, so the tests cannot silently drift from what ships.
Faults the process caught
The first parallel runs produced a material total of $128,466 against a $467,415 baseline. Not a costing error: under parallel load the model had stopped emitting the section labels the cost rollup keyed on, so whole trades dropped out of the sum. The A/B gate caught it before anything shipped. The fix parses tables by structure instead of labels, and the validation rerun landed within 0.1% of the baseline.
Under the parallel burst, shared document references began failing, and per-trade error isolation did its job too well: a run reported success with 7 of 16 trades absent. It was caught because runs were checked against the full expected stage list in the database, not judged by whether they finished. The fix sends documents inline on every call, and the stage-by-stage record keeps any gap visible.
Two overlapping first-loads of the same job each decided it was unseeded, and the takeoff total doubled. The application-level guard could not have fired: it read state once, before the race began. The durable fix is a unique database index that makes the duplicate impossible, backed by recovery that serves the winning rows and request de-duplication in the client. The guard that matters is the one no race can slip past.
Operations for long-running AI
An analysis is a long-running background operation, so nothing about it depends on a browser staying open. Runs execute as queued background jobs on Azure, progress is broadcast over SignalR and written to the database at every sequential seam, and a page refresh resumes from the persisted state rather than guessing. Concurrency is capped and configurable, set with the AI provider's quotas in mind rather than optimism.
The prompts that drive the pipeline live in blob storage per environment, not baked into the build, so prompt changes deploy independently of code. And because every stage's output is persisted individually, a support question weeks later starts from evidence: what ran, when, and what it produced.