Some notes on a class of system I’ve spent time on: platforms that automate document-driven business workflows — intake, extraction, routing, review, approval — in settings where getting it wrong is expensive enough that you can’t take the human out.
I want to describe the shape of the problem and the decisions that mattered, because the interesting parts weren’t where I expected them to be.
The obvious problem, and the real one#
Document workflows fail predictably. The documents are semi-structured: the same information, arranged differently by every source. Rules-based extraction handles common layouts well and then falls apart on the long tail. The usual response is more rules, and past some threshold nobody can predict what the system does with an unfamiliar document.
That’s the problem everyone sets out to solve. It’s not the expensive one.
The expensive one showed up after extraction was working: reviewers were still re-reading documents the system had already parsed correctly. Nothing distinguished a confident extraction from a lucky one, so the safe behaviour was to check everything. We’d reduced typing but not reading, and reading was where the time went.
Reframing around that changed what we built. The goal stopped being accurate extraction and became extraction annotated well enough that a reviewer can safely skip most of it.
Confidence as a routing input#
The decision the rest of the design rests on: route documents on extraction confidence rather than document type.
Extractions above a threshold proceed automatically. Uncertain ones escalate to a human, with the specific low-confidence fields flagged rather than the whole document handed over. This turns a binary automate-or-don’t choice into a graduated one, and makes review effort track genuine ambiguity instead of staying constant.
Two things I underestimated:
The threshold has to be configurable per workflow. The tradeoff curve isn’t symmetric — raising it sends more work to humans, dropping throughput and raising quality — and the right point depends entirely on what a wrong answer costs in that specific workflow. We initially tuned it globally and were wrong in both directions at once: too conservative for low-stakes workflows, too aggressive for high-stakes ones.
And confidence is only useful if you’re honest about what it measures. A model can be confidently wrong on a document format it has never seen. Confidence routing reduces review load; it doesn’t bound the error rate. Treating it as a safety guarantee rather than a prioritisation signal is the failure mode I’d warn someone about.
The review interface is an architectural concern#
This is the part I’d argue hardest for, and the part that’s easiest to under-resource.
We display each extracted value adjacent to its source region in the document, pre-focus the doubtful fields, and make confirmation a single action. That sounds like UI work. It isn’t — it requires the extraction layer to emit source coordinates alongside every value, which changes its output contract and couples extraction to presentation in a way that makes both harder to evolve.
I’d make that trade again. Without source coordinates the reviewer has to find the field in the document themselves, and review time roughly doubles — that’s from watching reviewers work before and after, not from a controlled measurement, but the difference wasn’t subtle.
The general point: review sits on the critical path of the slowest stage in the system. Treating it as internal tooling and staffing it accordingly is a mistake I’ve watched teams make, including mine.
Recording disagreement#
We capture corrections as structured events — field, prior value, corrected value, document region, extractor version — rather than overwriting the extracted record.
This costs write-path complexity and storage for data with no immediate consumer. It’s a bet that you’ll want to evaluate and improve later, and it has to be made before you have evidence justifying it, which is why it usually doesn’t get made.
It’s worth it because a reviewer overriding the system is the most informative event the platform produces. When reviewers systematically disagree on a class of document, that’s a defect report arriving on a channel nobody reads unless you instrument it. Versioning the extraction logic and recording which version produced each result is what makes the stream usable — without it you can’t distinguish a genuine improvement from a redistribution of errors.
We captured this data for a long time before doing anything useful with it. The retraining loop is still unfinished, and it has its own problem: a model trained on corrections inherits reviewer bias, and we don’t have a good handle on how much of the apparent model error is actually disagreement between reviewers.
What operating it taught me#
The operational question isn’t whether the system is up. It’s whether it’s still making good decisions, and those degrade quietly.
The most useful signal turned out to be the distribution of extraction confidence, watched over time. A shift in it is the earliest warning that inputs have changed — a new document format, an upstream source change, a model regression — and unlike accuracy, it’s available immediately, without waiting for labels.
Correction rate segmented by field and extractor version is the other one. Aggregate accuracy is the number everyone asks for and the one that least informs a decision; a rising correction rate on one field is specific enough to act on.
Two operational things I’d tell anyone building this: separate processing latency from time-spent-waiting, because one is an engineering problem and the other is a staffing problem and averaging them hides which is degrading. And alert on the age of the oldest item awaiting review rather than queue depth — extraction scales horizontally, review does not, so without bounding this the queue grows and the oldest items quietly age out of relevance.
The rest is unglamorous and non-negotiable: idempotent stage handlers, because documents get reprocessed after every bug and model update; workflow state in a transactional store rather than process memory, because a deploy will always land mid-flight for some document; immutable original documents with derived data referencing them; and bounded retries with a quarantine a human can inspect, so one malformed file can’t occupy a worker forever.
Where I landed#
Human-in-the-loop is a design stance rather than a limitation. Systems built assuming eventual full automation treat the human path as a degraded mode, and you can feel it in the ergonomics. Designing for permanent collaboration produced a better system — and better automation, because the correction data was better.
The thing I’d tell myself at the start: the hard problem isn’t extraction accuracy. It’s giving a reviewer enough information to trust the output without re-deriving it.