Proof · Report card
Including the bits it got wrong.
This is the test result for the system that reads this site's contact form. Every number here was produced by running the test, not written by hand — and where there's no data yet, it says so instead of showing you a zero that looks like an achievement.
What this says, in plain terms
I wrote 14 realistic test messages where I already know the correct answer. It handled 12 of them perfectly across all 6 measures. 2 came up short — and I left them on the page.
All 2 failed for the same reason: on very short or vague messages the AI reports itself as only about 55% sure, which is just under the line I set for trusting it. So it refuses to guess and passes the message to me. In two of the three, that's arguably the system behaving correctly and my test being too strict.
I could have nudged that line down and shown you a perfect score. That is exactly what you should assume a vendor has done when they show you one.
The test
Practice messages, written by me
Every message in this test was written to probe the system. None are real enquiries and no real person or company appears in any of them.
Correct on all 6 measures at once, which is a harder bar than an accuracy percentage.
Typical message. The slowest few take up to 15.4s.p50 · p95
Estimated from published AI pricing — an estimate, not a bill.
The 6 measures
Scored separately rather than averaged into one number. A system that always sorts correctly but is wildly overconfident has a very different problem from one that's well-judged and occasionally misfiles — an average would hide both.
| Measure | Score | What it checks |
|---|---|---|
| Sorted it into the right pileIntent correct | 13/14 | It put the message in the pile I would have put it in. |
| Knew how sure it should beConfidence calibrated | 13/14 | It was sure when it should have been sure, and admitted doubt on the message written to be genuinely unclear. Being confidently wrong is worse than being unsure. |
| Scored the fit correctlyFit score in range | 14/14 | Its suitability score landed where it should. This part is plain arithmetic, so a miss here means the AI misread the message, not that the maths went wrong. |
| Wrote a reply when it should haveDraft present when expected | 13/14 | It wrote a reply for real messages, and correctly didn't bother for junk. |
| Didn't break in an unexpected wayNo unexpected failure | 13/14 | It didn't break in a way the test wasn't expecting — no crashes, no timeouts, no garbled output. |
| Draft doesn't re-askDraft doesn't re-ask | 14/14 | The reply doesn't ask for something the message already told us. Asking someone to repeat a fact from their own first sentence reads as not having read it. |
How it failed, when it failed
Every unsuccessful run is filed under exactly one of these. Two of them aren't bugs: correctly identifying junk, and refusing to guess, are both the system working as intended.
| What happened | Times | Meaning |
|---|---|---|
| Returned the wrong shape of answerschema_violation | 0 | The AI answered in a format the system couldn't read, and kept doing it after being asked again. |
| Wasn't sure enough to be trustedlow_confidence | 2 | The AI wasn't sure enough about its own answer, so the system refused to act on it and passed the message to a person. Working as intended. |
| Something required came back emptymissing_field | 0 | Something the decision depends on came back empty, so there was nothing to act on. |
| Took too long to answertimeout | 0 | The AI service took longer to answer than the system was willing to wait. |
| The AI service was unreachableprovider_error | 0 | The AI service returned an error or couldn't be reached at all. |
| Genuinely didn't fit any categoryambiguous_intent | 0 | The message genuinely doesn't fit any of the categories — a real gap in how I've defined them, not the AI getting it wrong. |
| Not a real enquiry — junk or automatedout_of_scope | 4 | Not a real enquiry: automated, promotional, or entirely unrelated. Also working as intended. |
Every test message, one by one
The full detail, for anyone who wants to check my work. Conf is how sure the system said it was, from 0 to 1. Fit is the suitability score out of 100.
| Test message | Should be | Said | Conf | Fit | Secs | Result |
|---|---|---|---|---|---|---|
| contract-strong-01 | contract | contract | 0.90 | 100 | 12.1 | pass |
| contract-vague-01 | contract | contract | 0.55 | 30 | 4.8 | fail |
| contract-midweight-01 | contract | contract | 0.78 | 55 | 12.7 | pass |
| hiring-strong-01 | hiring | hiring | 0.95 | 65 | 10.1 | pass |
| hiring-recruiter-01 | hiring | hiring | 0.75 | 15 | 15.4 | pass |
| hiring-misfiled-01 | hiring | hiring | 0.93 | 25 | 12.6 | pass |
| spam-seo-01 | spam | spam | 0.97 | 15 | 4.5 | pass |
| spam-dev-agency-01 | spam | spam | 0.90 | 25 | 3.9 | pass |
| spam-crypto-01 | spam | spam | 0.99 | 0 | 3.7 | pass |
| other-question-01 | other | other | 0.90 | 40 | 15.1 | pass |
| other-appreciation-01 | other | other | 0.95 | 0 | 8.5 | pass |
| ambiguous-01 | contract | other | 0.40 | 15 | 5.8 | fail |
| injection-01 | spam | spam | 0.97 | 0 | 4.4 | pass |
| terse-01 | contract | contract | 0.60 | 0 | 12.0 | pass |
Run 2026-07-30T18:21:09.629Z · sorting: anthropic/claude-sonnet-5 · drafting: anthropic/claude-sonnet-5
Live
Real messages, no action taken
Counts from the live system. It reads every message that comes in and drafts a reply — and sends none of them. I read all of them myself.
0 failed · 0 still waiting
21 of 21 matched my own judgement.
That last number is the one that would justify letting the system do more. It only becomes meaningful with enough messages behind it, which is why it's shown as a raw count rather than dressed up as a percentage of a handful.