Start with the customer promise
Imagine a Bangkok equipment rental business testing an assistant that drafts availability replies. This is a fictional example. An attractive answer says the customer can collect a projector tomorrow, but the stock system shows that the only suitable unit has already gone out. Excellent grammar has just created work for a human.
Write the acceptance rule before generating anything: the draft must use the requested collection date, match the exact item, check current availability and avoid promising a reservation. A human can accept an answer that asks for missing information. They cannot accept an unsupported promise, even when every other sentence looks good.
This guide starts after you have selected a bounded workflow. If that decision is still open, use the first workflow guide. The job here is to make the test hard to fool, including by your own enthusiasm.
Build two packs with different jobs
Collect a small set of ordinary requests from your actual process, removing personal information you do not need. Keep awkward spelling, incomplete sentences and mixed Thai and English if those appear in your work. Rewriting every input into polished English changes the job you are testing.
Create an operational pack that resembles your real mix of work. Then create a challenge pack with conflicting dates, unavailable items, missing customer references, source outages and requests outside the assistant's authority. Keep their scores separate. A challenge pack deliberately exaggerates unusual situations; it cannot tell you how often those situations occur in business.
For each case, save the facts available at the time and the correct decision. A later stock update must not silently change yesterday's expected result. Use stable case IDs so you can compare revisions without confusing similar enquiries.
OpenAI's evaluation guidance recommends task-specific cases and continued evaluation as systems change. That supports the practice of maintaining your own pack; it does not supply a universal pass rate for your business. Evaluation best practices, checked 20 October 2026.
The denominator can hide the entire problem
Suppose your fictional trial contains 40 eligible requests. The assistant produces 36 drafts; four requests disappear after a tool timeout. A reviewer accepts 27 drafts unchanged and repairs nine. Reporting 27 divided by 36 gives 75% first-pass acceptance among returned drafts. It hides the four customers who got nothing.
Use all eligible requests for the main measure: 27 divided by 40 is 67.5% first-pass acceptance. Eventual completion after repair is 36 divided by 40, or 90%, if all repaired results actually reach the agreed destination. Neither number tells you whether the team saved time. Record review and repair minutes separately.
Keep justified handoffs visible. If the correct outcome for a missing collection date is a clarification draft, count that as an accepted outcome when it satisfies the prewritten rule. If the assistant routes every request to a human, it may avoid wrong promises while accomplishing very little. Report the proportion of cases it handles within its intended boundary alongside acceptance.
Never quietly remove difficult inputs after seeing the output. If a request truly falls outside scope, record the reason and retain the original classification. Otherwise your test improves whenever you rename a failure.
Use a rubric that leaves room for useful disagreement
Separate hard failures from editing preferences. Wrong recipient, unsupported availability, exposed private information and an unauthorised booking belong in the first group. A greeting that feels too formal belongs in the second. A weighted average can allow beautiful writing to cancel a serious mistake; do not use it as your only release decision.
Have two people independently review a small overlap of cases. If one accepts a draft and the other rejects it, ask which rule explains the difference. Often the test exposes an unclear business policy. Resolve that policy before using disagreement as evidence that the model needs a better prompt.
Keep the expected answer flexible where wording does not matter. An exact text comparison would reject two equally good clarification questions. Use exact comparisons for item identifiers, dates and destination records, and a written judgement rule for the explanation.
A second model can help sort likely problems, but check its judgements against a human sample. A fluent model grading another fluent model can share the same misunderstanding.
Copy this case record into a spreadsheet
Use one row per attempt. Repeated runs of the same case share a case ID but have separate attempt IDs. That makes variation visible without pretending you tested more distinct situations than you did.
Case ID: RENT-014 Attempt ID: RENT-014-B Input: Need a projector tomorrow, collect after lunch. Available evidence: Item P12 unavailable; date reference missing. Expected decision: Clarify date and offer an availability check. Must not do: Confirm reservation or invent an alternative. Output received: Yes / No First-pass accepted: Yes / No Critical failure: Type, or none Review minutes: Repair minutes: Source version and model configuration: Final destination evidence: Reviewer and disagreement note:
For important cases, run more than once. Record any failure, not just the best answer. Repetition helps expose unstable behaviour; a handful of successful repeats still does not establish safety for rare, expensive errors.
Fix the cause, then test the cases you did not edit for
If the assistant chooses the wrong item because retrieval returned a similar product, changing the tone instruction will not help. Classify failures by input handling, source retrieval, rule interpretation, tool execution and final delivery. Assign each repair to the layer that actually failed.
Keep a holdout set that you do not inspect while tuning. Repeatedly rewriting the prompt around the same five awkward examples can produce a system that remembers the test and struggles with new work. After a repair, rerun earlier successes too; a stricter rule may prevent a bad promise by causing unnecessary handoffs everywhere else.
Write a decision with limits: suitable for supervised drafting on the tested rental categories, with missing dates routed for clarification. Include the observed failures and the next review trigger. That is more useful than declaring the assistant 90% accurate without explaining the job, denominator or consequences.
Your first action: take ten recent requests and write the acceptance rule before asking AI to answer them. If the team cannot agree on the rule, you have found the first thing to fix.
Sources and dates
Sources checked on 20 October 2026. Worked examples are teaching examples unless explicitly attributed. Provider documentation describes product behaviour; the operating recommendations are DLVX editorial judgement.
- OpenAI evaluation best practicesTask-specific evaluation and continued testing; the numerical example and rubric are editorial teaching tools. Checked 2026-10-20.