Tracefield
A trust runtime for AI document extraction. Every value proves where it came from on the page — or is flagged as a claim nothing supports. There is no third state.
The rule that makes it a product rather than a model wrapper is four lines of SQL: a field is backed by evidence or it is flagged unverified. It lives in Postgres, so no code path can forget it — and everything else in the system exists to serve it.
Accuracy plateaus. Proof does not.
Hand a document to a vision model and it returns clean, confident JSON. Some of it is wrong, and nothing in the output tells you which part. Teams respond the only way they can — a person opens every document and checks every field, which is exactly the work the tool was supposed to remove.
The thesis is not “be more accurate”. Accuracy is a model property and it plateaus. It is that every value should carry the reason to believe it, so a person reviews the handful that cannot prove themselves instead of all of them.
Built solo across two languages — a TypeScript runtime and Next.js app, a Python/FastAPI extraction service — and running on a real Kubernetes cluster.
every value arrives holding the words it came from
02 — Extract
It reads the document, not the vibes.
Every value is pulled from the words on the page — and the model must hand back the exact words it read, so there is nothing to take on trust.
- OCR word stream with coordinates for every word
- The model returns each value with its verbatim quote
- Quotes are matched back against the page, not assumed
- A quote that isn’t on the page fails the field
03 — Investigate
Red claims get a detective.
When a value fails verification, the Review Agent works the case before a human sees it — searches the word stream, reads the page around the label, cross-checks the document’s own arithmetic, ranks the reference data — then recommends.
- A bounded tool loop — every call lands in a persisted trace
- One agent, two engines: raw loop and LangGraph, evaluated 4/4 each
- Spend is checked before every model call, not reported after
- It recommends; a person decides, under their own name
The model produced this. The page does not contain it.
a person decides — and the decision is recorded by name
04 — Review
What it can’t prove, it won’t pass.
A value with no source on the page is routed to your review queue — clearly marked — instead of flowing silently into your ERP. That is the whole product.
- Unverified never auto-passes — the database refuses the row
- A person approves, corrects, or rejects, by name
- Every decision lands in the audit trail
- Corrections feed back into future extractions
| DOCUMENT | SUPPLIER | TOTAL | PURCHASE_ORDER |
|---|---|---|---|
| invoice-0912.pdf | Acme Trading Ltd | 1,200.00 | PO-4471 |
| invoice-0913.pdf | Nordwind AB | 845.50 | PO-4472 |
| invoice-0914.pdf | Ostsee Handel KG | 2,310.00 |
every blank is named — nothing is quietly missing
05 — Export
Only what it can prove leaves the building.
The spreadsheet your team actually works from is filled by verified values and human decisions — and by nothing else. A cell nobody could stand behind exports empty, with the reason beside it.
- A cell is an auto-verified claim or a person’s decision
- Withheld cells are counted, named, and announced in the response
- Fields marked
redactnever leave, in any format - CSV, JSON, or straight into an agent over MCP
06 — The rule
A constraint, not a convention.
The whole argument of the product is four lines of SQL. A field is backed by evidence or it is flagged unverified, and because that lives in Postgres there is no code path that can forget it — not a new endpoint, not a refactor, not a hurried Friday.
claims.verdict is NOT NULL with no default, and a trigger freezes step outputs once terminal, so a resumed run re-reads rather than regenerates. Fifteen migrations carry the schema, the constraints, the triggers and row-level security.
- The rule is enforced by the database, not by discipline
- Cross-tenant probes answer 404, never 403 — existence is not leaked
- Credits charged atomically before work; a dead job refunds exactly once, by unique index
- A claim belongs to a run, not a document — so re-extraction cannot double fields
07 — The agent
It investigates. It cannot decide.
Every failed claim is investigated before a person sees it. The Review Agent searches the word stream, reads the region around the label, cross-checks the document’s own arithmetic and ranks reference data — pure tools over a bundle the worker assembled, bounded by step count and by a spend ceiling checked before each model call.
And the line that matters: it recommends, it has no path to claims.verdict. Two engines were built and graded against ground-truth scenarios — hand-rolled and LangGraph — both 4/4 correct, at $0.37 and $0.57 a run.
- Bounded by steps and by a spend ceiling checked before each call
- Reads only a bundle the worker assembled — no open-ended access
- Recommends a verdict; writing one stays a human’s job
- Both engines graded against ground truth, and the cheaper one won
08 — Defence
Tests that can actually fail.
A test asserting “the stranger gets 404” passes just as happily when the URL is wrong and everyone gets 404. So every negative check is paired with a positive control — each isolation probe runs twice, once as the owner, who must succeed. That pairing has already caught a probe that proved nothing, and a refund proof that would have passed while quietly keeping a customer’s credit.
The eval suite grades two deliberately broken extractors alongside the real one and refuses to report unless both fail as badly as they should: one that reports nothing, and one that returns correct answers with no source. The first catches a suite that rewards abstaining; the second catches one that has stopped measuring provenance.
- Every negative check paired with a positive control
- Two broken baselines gate the eval — if they pass, the suite is wrong
- Invariant metrics are SQL run at scrape time, not counters a code path can forget
- CI re-derives the landing page’s highlight coordinates and fails on 0.002 drift
Shipped on real infrastructure
Helm chart and Terraform modules for EKS, RDS, ElastiCache, S3 and ECR, running in eu-north-1. A document has crossed the full path on a live cluster — web to queue to worker to object storage to review UI — with S3 reached through an IAM role assumed via IRSA rather than static credentials sitting in the cluster.
The bill taught the sharpest lesson. Running cost disagreed with the resource list for two days, and the gap was the story: the cluster had been created on a Kubernetes version that had left standard support, and EKS bills an extended-support control plane at six times the rate. The surcharge appears on no resource list and in no price lookup — only as a usage-type row on the bill. Upgrading stopped it accruing.
Storage sits behind one adapter with no silent fallback: a worker writing to its own ephemeral disk because credentials were missing would look healthy while producing evidence nobody can load.
Stack
Runtime
- TypeScript 5.9 · Node 22
- Next.js 16 · React 19
- PostgreSQL · RLS
- Redis · BullMQ
- 15 migrations
Extraction
- Python · FastAPI
- Anthropic Claude
- Tesseract OCR
- PDF text layer
- LangGraph & hand-rolled agents
Infrastructure
- AWS EKS · RDS · ElastiCache
- Terraform modules
- Helm chart
- S3 via IRSA
- Vercel (marketing)
Operations
- OpenTelemetry traces
- Prometheus · 12 alert rules
- MCP server
- Paddle billing
- 22 acceptance scripts
Outcome
Working end to end, locally and on AWS: multi-tenant auth, credits with dead-job refunds, document types and reference catalogues managed from the UI, batch upload, verified-only export, re-extraction on schema change, an MCP server, and Paddle billing that moves a plan on a signed event.
Thirty-nine numbered decision records carry what was decided and what it cost — including the ones that were wrong first. A retrieval experiment that moved nothing is written up as a clean negative result rather than quietly dropped.
Every value proves where it came from — or says that it cannot.