PROOF OF WORK
Where we test new AI fairly, and work out how best to solve real problems, before any of it goes near a real business.
The same rules for every test, whether it’s a new model or a new way to fix a problem. They’re there to stop us fooling ourselves.
A real piece of work with a test set and an answer key written before anything is run.
A plain rule, or how a person does it today. New is only better if it beats that.
Scores come from code or from someone who can’t see which system produced the answer. Results are shown as ranges.
A result that’s slightly better but ten times the price is written down that way.
Every test includes data with nothing hidden in it, to catch methods that invent patterns.
What didn’t work goes on this page with the same weight as what did.
Track 1
New models, agents and tools arrive every month, each with a bold claim. We test the ones that could matter to a business on a real piece of work, and publish the result, including when the old way wins. None has been run yet. These are next, and each appears here once it’s done.
When a new model comes out, is it better at drafting a reply from a venue’s price list, without getting a single fact wrong? And is it worth the cost?
Can an agent copy a booking from one system to another with no integration, and how many mistakes does it make in a hundred?
Can an assistant answer from a business’s own documents, show the source, and say “I don’t know” instead of guessing?
For each new capability: would a rule, a spreadsheet or a person have done just as well?
Track 2
You can’t ask a business for years of customer records to test an idea that might not work. So we made the records ourselves: a country hotel with 80 rooms and two wedding suites, five years of enquiries, bookings, payments and emails, with duplicates, missing reasons, dates typed five ways and a half-finished move from spreadsheets to a CRM. Then we hid patterns in it, sealed the answers away, and built a twin with the same mess and nothing hidden.
A pipeline that takes the raw exports and gives each person a short list for the morning. AI reads the notes and emails and writes the wording. The joining, scoring and checking are ordinary code and statistics, because that’s what they’re good at.
Enquiry sheet, CRM, Bridebook, function diary, bookings, bank, mailbox, calendar
Duplicates merged, dates fixed, the same couple recognised across systems
Every couple and company, first contact to paid
What’s still live, and what looks like the enquiries that booked before
Reasons built from the record; code confirms every number and date
Up to ten a day per person, with the reason and a next step
Andrew Jackson Wedding · 106 guests
Wedding enquiry still waiting for our first reply
Next: Reply now with availability and prices, and offer a show-round
Kate & Thomas Wedding
They have seen the venue and not decided
Next: Phone to ask how the show-round felt; offer to hold the date
Craven Care Homes Ltd Corporate
Regular corporate account has gone quiet
Next: Call the booker: ask what events they have coming up and offer two dates
Before any AI step sees an enquiry, names, emails and phone numbers are removed. Each client’s data lives in its own encrypted container, created and destroyed by script, with a certificate at the end listing what was deleted. Try a simplified version below.
Hi, it’s [NAME] [NAME] here, we emailed on Friday about Saturday 14 August 2027. You can reach me on [PHONE] or [EMAIL], or my partner [NAME] on [PHONE]. We’re 106 guests, budget around £11,000.
6 removed. The date, guest count and budget stay: the work needs them.What’s in place and what isn’t yet, line by line: Security.
We compared our pipeline with the simple rule most teams already use: reply to the newest enquiry first. On the test hotel, it didn’t beat it. A rule we wrote from intuition did worse than both.
| Method | Result | 95% range |
|---|---|---|
| Random order (a floor) | 3% | 0 to 8% |
| A points rule (written from intuition) | 22% | 1 to 46% |
| Newest first (what most teams do now) | 33% | 14 to 56% |
| Our pipeline (built with claude) | 33% | 15 to 58% |
| A second pipeline (built with codex) | 35% | 16 to 54% |
We wrote Wrenholt’s couples ourselves, so the pipeline can only find what we planted. That makes it a good test of the plumbing, the checks and the security, and no evidence at all that it will win a real venue more bookings. We say that plainly because the first real client deserves to know exactly what’s been proven and what hasn’t.
Would a longer daily list find more of what matters?
Can a client’s data go in, be used and be destroyed, with nothing left behind?
If a hotel starts replying to reviews, does its rating go up, and by how much?
If staff had to record why an enquiry was lost, would the analysis get better?
How far below the best possible does each method sit, and why?
Does anything we found in a made-up hotel hold in a real one? Only a first client’s history can say.
How much of the value comes simply from replying fast and following up properly, before any ranking at all?
Ask the big AI assistants for a venue like yours, every month. Who gets named, and why?