Proof
I built one. You can inspect it.
This site argues that the hard part of AI is getting it working safely inside a real business. That would be a weak argument if nothing here were actually running — so the contact form on this site is read by a system I built, and the rest of this page is how it works, what it is allowed to decide, and where it falls short.
In one paragraph
Send me a message and a system reads it, works out what you want, scores how good a fit the work is, and drafts a reply in my voice. Then it stops. It is not allowed to send anything. I read every message myself, and if a reply goes out it is because I edited that draft and pressed send.
Its report card is public, including the 2 tests it fails. Most people would have quietly adjusted the test until the score looked better. That is exactly what you should assume when someone shows you a perfect one.
This is a small job on purpose. The claim isn't “look how big” — it's “look how carefully,” and careful is what you should be shopping for.
01 · The job
What work this actually replaces
Messages arrive through one form, where the sender picks a category for themselves. They split roughly four ways: people who want to hire me, people who want to work with me, automated junk, and genuine notes that are none of those.
Those self-picked categories are wrong often enough to be useless — spam never files itself honestly. That is exactly the sort of small untruth about a process that sinks projects, and it is why the system treats the category as a hint rather than a fact.
Done by hand, the job is: read it, decide which pile it belongs in, decide whether it deserves a reply today or Friday, then write the reply. Small, real, and — like all back-office work — difficult mostly because of the exceptions.
02 · The boundary
Which decisions the AI is allowed to make
The most important decision in the build, and the one most failed projects get wrong by handing everything to the AI.
Here is every decision this system makes, and what makes it. Notice how little of it is the AI.
| The decision | Made by | Why |
|---|---|---|
| Is the email address real, is anything missing, is this a bot? | Ordinary softwaredeterministic checks | These have exactly one right answer. AI could only add mistakes. |
| What does this person actually want? | AI judgmentLLM classification | Free-form human writing, where the rules can't be listed in advance. This is where AI genuinely earns its place. |
| How good a fit is this work? | Ordinary softwareweighted score | The AI answers five yes/no questions; a fixed formula turns those into a score. The same answers always give the same number, and you can read the formula. |
| Is the AI sure enough to be trusted here? | Ordinary softwareconfidence threshold | A comparison against a fixed number. Below 0.6, its answer is recorded but not acted on, and the message comes to me instead. |
| Writing the reply | AI judgmentLLM generation | Open-ended writing that has to sound like a person. |
| Actually sending it | A personhuman review | Can't be undone, and it's my reputation. I read the draft, edit it, and press send. Never delegated. |
Two of six decisions are the AI's. That ratio is normal in systems that work, and rare in systems that get sold. The general rule behind it is on the method.
03 · The test
How I know whether it works
I wrote 14 realistic messages where I already know the right answer, and the system is checked against all of them whenever anything changes.
They are chosen the way I would choose them for a client: mostly ordinary messages, plus the awkward ones that break things. Someone who ticked the wrong category. A message with almost no detail in it. An attempt to trick the AI with hidden instructions. And one deliberately unclear message where the correct behaviour is for the system to admit it doesn't know and hand it to me.
All 14 are written by me. None are real messages from real people, and nothing on this site pretends otherwise.
Scoring is split into six separate measures rather than one average, because a system that always sorts correctly but is wildly overconfident has a completely different problem from one that is well-judged and occasionally misfiles. An average would hide both.
2 of them fail, and they stay on the page. Both come down to the same thing: on very short or vague messages the AI reports itself as only about 55% sure, just under the line I set for trusting it — so it refuses to guess and passes the message to me. In at least one case that is arguably the system behaving correctly and my test being too strict.
04 · How it breaks
Every failure has a name
“It's 92% accurate” is a nearly useless sentence. It doesn't tell you what the other 8% looks like, or whether it is all one fixable problem.
So every run that doesn't succeed is filed under exactly one of these, and the counts are published. That is what turns “it's a bit flaky” into a to-do list.
| What went wrong | What it means |
|---|---|
| The AI returned something the wrong shapeschema_violation | The AI answered in a format the system couldn't read, and kept doing it after being asked again. |
| The AI wasn't sure enough to be trustedlow_confidence | The AI wasn't sure enough about its own answer, so the system refused to act on it and passed the message to a person. Working as intended. |
| Something required came back emptymissing_field | Something the decision depends on came back empty, so there was nothing to act on. |
| The AI took too long to answertimeout | The AI service took longer to answer than the system was willing to wait. |
| The AI service was down or unreachableprovider_error | The AI service returned an error or couldn't be reached at all. |
| The message genuinely doesn't fit any categoryambiguous_intent | The message genuinely doesn't fit any of the categories — a real gap in how I've defined them, not the AI getting it wrong. |
| Not a real enquiry — junk or automatedout_of_scope | Not a real enquiry: automated, promotional, or entirely unrelated. Also working as intended. |
Two of these aren't bugs at all. Correctly identifying junk is the system working, and refusing to guess when it isn't sure is the behaviour I most want it to have.
05 · Trust
How much freedom it has earned
Software gets trusted the way a new hire does — one step at a time, and each step has to be paid for with evidence. These are the five steps, from practising on old work to acting alone.
| Step | What it's allowed to do | Where this system is |
|---|---|---|
| Practises on old workoffline replay | Runs against work that already happened, where the right answer is already known. It never sees anything live. | passed |
| Shadows the real work, unseenshadow mode | Sees the same live work your team does and records what it would have done — but shows that answer to nobody. Your team works exactly as if it were not there. | here now |
| Suggests — your team still does the workassist | Now its answer is visible. Your team sees a draft before they start, and is free to use it, change it, or ignore it completely. They still do the job. | not earned |
| Does the work — a person approves itsupervised autonomy | The system does the work now, not your team. Someone reviews and signs off before anything lands. The default has flipped: things happen unless a person stops them. | not earned |
| Acts alone, inside limitsautonomous | Acts without waiting for anyone, inside boundaries that cap how much a single mistake can cost. Checked afterwards by sampling rather than beforehand. | not earned |
It sits on the second step and stays there. Moving up would mean showing its drafts to a real reviewer, and the evidence for that is whether it agrees with a human on real messages rather than practice ones — which needs more traffic than this site has yet.
Saying so is more useful than claiming a level it hasn't earned. Every run also keeps a step-by-step record, so any single decision can be reconstructed months later without re-running it.
06 · What exists
What's built, and what isn't
The same list of deliverables I'd hand a client, from the method, marked honestly against this build.
| Stage | What exists | State |
|---|---|---|
| DiscoveryUnderstanding how the work really happens today | a map of how the work actually flows, exceptions included · every place it hurts, ranked · what it costs you today, in hours and errors · what happens if it goes wrong, and how fast you'd know | built |
| ArchitectureDeciding what gets built, and who is allowed to do what | a picture of how the pieces connect · exactly what information moves between them · what the system may touch, and what it may not · how someone could abuse it, and what stops them | built |
| EvalsProving it works, with numbers instead of impressions | real cases where the right answer is already known · an agreed written definition of “correct” · every distinct way it breaks, named and counted · what each run costs and how long it takes | built |
| DeploymentTurning it on carefully, and being able to turn it off | the staged rollout, with what has to be true at each step · alerts when it starts behaving differently · the conditions that shut it off automatically · written instructions so your team can run it without me | partial |
Deployment is marked partial, not done: there is no automatic shut-off, because a system that takes no action has nothing to shut off. It gets built the day it is allowed to act, and not before.
Reading this because you're hiring? Everything above is what I'd do inside your customers' businesses instead of my own contact form. Here's how I'd run the first 90 days.