How Build IG's estimation engine runs and checks its outputs
These notes explain how QPOI parallelised Build IG's estimation pipeline, checked it against the sequential baseline and kept reviewed totals in application code. They also describe the recorded test method and the faults caught during validation.
← Back to the case studyThe analysis pipeline, stage by stageView diagramHide diagram
The constraint that shaped everything
The estimate this pipeline produces feeds quotes, bid packages and a generated contract. So when the client asked for the analysis to be faster, there was one non-negotiable: faster execution had to preserve the scope and numerical consistency of the analysis. Runtime alone was not enough to accept the change.
That constraint set the acceptance gate before any code changed: every iteration of the parallel pipeline re-ran the same real blueprint and was compared with the sequential baseline, stage by stage and trade by trade. More than one iteration failed that gate and never shipped. Section 6.0 covers what the failures taught us.
Untangling a stateful pipeline
The pipeline began life as one long stateful conversation: each stage read everything before it, which made the output coherent and the whole thing impossible to parallelise naively. The untangling started by asking, per stage, what it actually needs. The opening stages depend on each other, so they stay sequential. The sixteen trade analyses each need the project context and the building's measurements, but not each other, so the measurements were promoted into an explicit quantity basis stage: computed once, handed to every trade as authoritative context, and launched alongside the head so the quantity calculation overlaps with the opening stages. Because every trade prices from the same basis, the trades cannot quietly disagree about the building they are pricing.
The trades then run in parallel under a bounded concurrency cap, and the tail, the stages that read the assembled report, runs in six dependency-ordered waves. Two anchors are enforced in code rather than asked of the model: the cost rollup is summed from the persisted trade outputs, and the executive summary is locked to that rollup, so no stage can invent a second total. Each stage persists its result as it completes, which is what makes a partial failure something the system can detect instead of something a reader has to notice.
Keeping money deterministic
Parse structure, not labels
Language models intermittently drop forced markers and section labels, and under parallel load they drop them more. Every parser that keyed on a literal label eventually broke on some trades, so the rule is now uniform: find a table by its headers, find a section by its shape, and treat labels as decoration.
Numbers are arithmetic
Once an estimate exists, its figures live in a canonical store where recalculation is a fixed formula, not a model call. Editing a line item is maths. AI is reserved for the edges, suggesting a replacement product or regenerating a narrative report, and never recomputes a figure a user has touched.
One figure, many disguises
The same total appears across documents as a raw number, a grouped figure and a truncated abbreviation. When an edit changes it, substitution handles each shape, applies the longest match first so a fragment cannot corrupt a fuller figure, and reproduces the document's own style.
Guarded find-and-replace
Substitution into prose and contract text is fenced: a percentage is only rewritten when the sentence names the rate it belongs to, abbreviations that could collide with a product name are skipped, and an ambiguous match is suppressed entirely. A safe miss is acceptable; a corruption is not.
Measured, with the method stated
The pipeline numbers come from persisted evidence, not stopwatches. Every stage writes its result to the database as it completes, so a run's duration is the span from its first persisted stage to its last, and the same query measures any run, old or new. Speed was never accepted on its own: the validated parallel run carried all 32 stages and all 16 trades, and its material total landed within 0.1% of the sequential baseline.
The platform figures use the same discipline. Backend query counts come from replaying an identical session against the logs before and after the change; the API-call counts come from a scripted browser session on the same job, with the pre-fix code restored from source control so the comparison is honest. One lesson is worth stating: on a small, far-away development database, milliseconds are mostly network noise, so we count queries rather than seconds. The query counts describe those replayed sessions; they are not a claim about response time for every production user.
Tested against real artefacts
The dangerous parsers never meet production first. Before a parser changes, the new logic is replayed in simulation over stored production reports alongside the old logic, and the comparison is row counts and totals, not opinion. One such replay showed the old path recovering a single trade section from a report where the new path recovered seventeen; only with that evidence did the change ship.
The substitution logic that edits contract prose has a permanent test suite: fourteen checks covering the cases that make find-and-replace dangerous, such as two items sharing a rating where only one was edited, or a manufacturer name that reads like an abbreviated number. The suite links the production code directly rather than a copy, so the tests cannot silently drift from what ships.
Faults the process caught
The first parallel runs produced a material total of $128,466 against a $467,415 baseline. Not a costing error: under parallel load the model had stopped emitting the section labels the cost rollup keyed on, so 11 of 16 trades summed to zero. The A/B gate caught it before anything shipped. The fix parses tables by structure instead of labels, and the rerun landed within 0.1% of the baseline.
Under the parallel burst, shared document references began failing, and per-trade error isolation did its job too well: a run completed with 7 of 16 trades absent. Caught because every run is verified against the full expected stage list in the database, never judged by whether it finished. The fix falls back to inlining documents on the first failure and latches for the rest of the run, so degradation is graceful instead of silent.
Two overlapping first-loads of the same job each decided it was unseeded, and the takeoff total doubled. The application-level guard could not have fired: it read state once, before the race began. The durable fix is a unique database index that makes the duplicate impossible, backed by recovery that serves the winning rows and request de-duplication in the client. The guard that matters is the one no race can slip past.
Operations for long-running AI
An analysis is a long-running background operation, so nothing about it depends on a browser staying open. Runs execute as queued background jobs on Azure, progress is broadcast over SignalR and written to the database at every sequential seam, and a page refresh resumes from the persisted state rather than guessing. Concurrency is capped and configurable, sized to the AI provider's quotas rather than to optimism.
The prompts that drive the pipeline are versioned in blob storage per environment, not baked into the build, so prompt changes deploy and roll back independently of code. And because every stage's output is persisted individually, a support question weeks later starts from evidence: what ran, when, and what it produced.