Case study
Quote-Intake Agent
An AI intake agent for a home-services contractor. It reads inbound quote requests, routes them, and drafts replies for a dispatcher to approve, without ever stating a price, promising an arrival time, or missing a gas leak.
TypeScript, Node.js, Claude Opus 5.5 with structured outputs, Zod · View on GitHub
- Held-out routes correct
- 20/20
- Held-out safety cases escalated
- 4/4
- Injected model mistakes kept from the customer
- 232/232
- Dev-set routes correct
- 43/43
The problem
A trades business gets quote requests as run-on texts and emails: “furnace, 15 years old, 14615, faint gas smell.” Someone has to work out what the job is, what's missing, whether it's an emergency, and reply quickly. A language model is good at the reading and the writing. It shouldn't be the last line of defense on what a job costs or whether a customer should leave the house. The business is a fictional Rochester heating, plumbing and electrical company with its own price book, service area and policies.
The design: the model reads and writes, code decides
The agent makes one structured-output call to Claude, then passes the result through guardrails written in plain code before a dispatcher sees the draft.
| Decision | Made by |
|---|---|
| What the customer wants, and drafting the reply | Claude, constrained to a JSON schema |
| Route (safety, urgent, human, decline, ask, estimate) | Code: priority order is policy, not judgment |
| Prices | The price book, inserted by code. The model never writes a dollar figure |
| What information is still missing | The price book's requirements for each service |
| Hazards | Keyword rules that override the model, plus the model's own flag |
| ZIPs, phone numbers, photos, equipment age | Discarded unless they appear in the customer's text |
| Model refused, errored or returned bad JSON | Fails closed to a person |
| Sending | A person. Every reply is a draft |
How I measured it
- A held-out set written blind. A separate session wrote 20 messy inquiries from the business rules alone, without seeing the code, prompt or dev cases: texting style, Spanish, two jobs in one message, hazards mentioned in passing, prompt injections and discount fishing. On the first run, before any fixes, it routed all 20 correctly and escalated all 4 safety cases.
- A labelled dev set. 43 hand-written cases covering complete requests, missing information, 7 safety cases, emergencies, out-of-area jobs, spam and injections. I treat it as a regression suite, since I wrote it alongside the prompt.
- Fault injection. Strong models pass these cases easily, so the more useful question is what happens when the model gets it wrong. I took the recorded outputs, injected one mistake at a time (a missed gas leak, a made-up price, a promised arrival time, an invented phone number) and replayed them through the guardrails. After fixes, all 232 injected mistakes were caught.
What the evals caught
- A bug I introduced: a check that a fixture count appears in the text only understood English, so “2 lámparas de techo” was dropped and the agent would have re-asked a question the customer had answered.
- The keyword backstop was narrower than the dev set suggested. With the model's own safety flag removed, a Spanish burnt-outlet message and “the puddle is running under the electrical panel” slipped through. I added patterns for each, with regression tests.
- My own grader was wrong. A one-prompt baseline looked unsafe until I found every flag was a false positive, such as correct arithmetic on published prices. That changed the conclusion: the agent's edge isn't that today's model misbehaves. It's structured output a dispatcher can act on, guarantees that hold when the model changes, and evidence to re-run after every change.
Limits and next steps
- All 63 inquiries are synthetic. Real inquiries with personal details removed are the next step.
- The held-out set has been used for fixes, so it needs a fresh blind set.
- Hazard keywords cover English and some Spanish. Other languages rely on the model alone.
- Live cost and latency aren't measured yet. The published runs used recorded outputs.
Get in touch if you'd like to talk through how I'd build something like this for your team.