Work / 04
An AI contract reader that has to prove every claim it makes
Redline reads a contract before a freelancer signs it, flags the clauses that could hurt them, and drafts a reply they can send. It’s my own product, it’s still in build, and it’s where I’m learning the parts of AI product work that don’t show up in a demo.
- 100%of risk flags must quote a real sentence from the document, enforced in CI
- 10architecture decision records, written before the code
- 7tests that define “good”, one build gate and six launch gates
The user
A US freelancer or small-business owner looking at a contract the other side drafted. They suspect a clause could hurt them and can’t tell which one. A lawyer’s review costs around $670 on average (ContractsCounsel), so most people just sign.
Before building anything I collected 18 sourced pain findings and looked at 11 competitors, including free tools that already explain contracts.
The real problem
A language model that paraphrases a contract can sound right and be wrong, and the reader has no way to tell the difference. For this user, a confident mistake is worse than no answer. So the product question wasn’t “can a model read a contract?” It was “how does the user check what the model says?”
Decisions
Every flag cites its sentence, exactly
The server splits the stored text into sentences with fixed positions. The model cites a sentence ID and repeats the quote, and the quote has to match the stored text byte for byte or the whole analysis fails. I dropped character offsets early because models are bad at counting characters. Nothing is fuzzy-matched.
Quiet is a feature
Redline flags only what is uncapped or inescapable. A fair contract returns zero risk flags and a checklist of what was checked. A tool that finds something in every document is a horoscope.
Absences are kept separate
Missing protections, like a contract with no payment deadline, go in their own list. They cite nothing, because they claim nothing about the text.
Confidence follows provenance
How firmly Redline says something depends on what the statement rests on, never on how sure the model sounds. Where the answer depends on facts it doesn’t have, like jurisdiction or leverage, it says nothing.
Say no to OCR
Scanned documents are refused. A citation is worthless if it points at misread text.
Explain, never advise
Redline says what the document says and does. It doesn’t tell anyone what to do legally. DoNotPay’s $193K FTC settlement came from exactly that claim.
How it’s built
Research first, then a PRD with seven tests, then ten decision records, then tickets. Coding agents built the tickets in an unattended run. Every ticket was checked with typecheck, the full test suite, a production build and a scan for stubs and mocks before it was committed. Wherever an agent had to decide something without me, the decision went into a build report I review.
Honest status
Where it stands, September 2026. 17 of 20 build tickets are done and CI passes. What isn’t done:
- No output has been verified against the real model yet. An upstream rate limit blocked every attempt, so only the stubbed pipeline is proven.
- The real test corpora, including a set of fair contracts that must come back clean, are still to be assembled by hand. I don’t want an agent-written “fair” contract grading the model.
- A lawyer’s review of the severity ranking and a direct test against a free competitor are both still open.
- Who pays, and how, is undecided.
I’m publishing the gaps on purpose. Eval design, failure behaviour and deciding what not to claim are the job.
What carries over
Evaluation as a product deliverable. Designing for the times the model is wrong. Keeping cost, latency and quality in view together. And the same instinct as in regulated fintech: if users can’t check it, they shouldn’t have to trust it.