The fluent answer often wins the wrong contest
Two assistants answer the same customer question. One is warm, concise and confident. The other pauses because the source policy does not cover the requested date. A reviewer prefers the first because it feels more professional.
If the first answer invented a delivery promise, the review rewarded the defect. Comparing outputs requires a definition of better that matches the business job. Tone matters, but only after the answer respects the source and the authority available.
Start with a narrow question: which output gives our reviewer a more usable draft for this task? That is easier to answer than which AI is best. The example below is fictional and uses no measured vendor performance. It shows how to compare results without allowing model reputation or fluent writing to decide the result.
Give both candidates an equivalent opportunity
Use the same source pack, task instruction and intended output. Record the product, model if visible, date, settings and tool access. A connected assistant that can search your records has a different setup from a chat window with only a pasted sentence. Compare the complete routes if that is your buying decision, but describe the difference.
Include the approved policy and known customer facts. Do not give one candidate a hidden correction learned during the other's trial. If you improve the brief, rerun both with the revised version and preserve the earlier results as history.
Choose representative normal and awkward cases. Keep some cases out of prompt development so the comparison does not merely reward memorized examples. Use permissioned or synthetic inputs appropriate for the accounts being tested. No comparison needs unnecessary customer identifiers merely to feel realistic.
A promise hidden inside a good sentence
The fictional source says standard delivery takes three business days after dispatch. The request arrived on Thursday. Dispatch has not been confirmed. The customer asks: “Can you guarantee it will arrive by Monday?” The assistant may draft a reply but cannot approve an exception.
Output A: “Absolutely, we can have your order with you by Monday. I'll make sure the team prioritizes it.”
Output B: “I cannot confirm Monday from the information available. Standard delivery is three business days after dispatch, and dispatch is not yet confirmed. I can ask the team to check the earliest available date.”
A is shorter and warmer. It invents both a guarantee and priority handling. B preserves the relevant rule and identifies the next check. It could be improved stylistically, but it has not made a promise outside the evidence.
The correct evaluation does not award A enough tone points to cancel the unsupported commitment. It fails the hard condition.
Use a short rubric with anchors
| Dimension | Weak | Usable | Strong |
|---|---|---|---|
| Source fidelity | Changes or omits material facts | Preserves the essential facts | Preserves facts and exposes uncertainty |
| Task completion | Does not deliver the requested artifact | Delivers a usable draft | Delivers it with the next action clear |
| Review effort | Requires reconstruction | Needs limited corrections | Evidence and open questions are easy to inspect |
| Communication | Confusing or inappropriate | Clear and suitable | Concise, natural and audience-aware |
Use a small score scale if useful, but define the anchors before reviewing. Mark hard failures separately: invented price, unauthorized commitment, wrong recipient, unrelated customer data or fabricated source. The relevant list depends on the task.
Measure correction time when practical. A beautifully formatted output that takes fifteen minutes to verify may be less useful than a plain draft whose evidence can be checked in two minutes.
Hide the names and vary the order
Label the outputs A and B without vendor branding. Give reviewers the source pack and the rubric. Alternate which candidate appears first across cases so a consistent ordering does not become an accidental advantage.
Ask reviewers to make their initial judgments independently. Then discuss disagreements using source evidence. “I like this one” is a preference. “This sentence introduces a service promise absent from the policy” is a testable defect.
Keep the raw outputs. Do not clean up formatting for one candidate or silently fix its facts before scoring. If presentation is not relevant to the task, normalize it equally and say so. If preserving a spreadsheet or document layout is part of the task, include that in the evaluation rather than stripping it away.
A blind review reduces one source of bias. It does not make a small sample representative of all your work.
A disagreement can reveal a missing business rule
Suppose one reviewer says output B is too cautious and another says it is appropriately precise. Ask what the team is authorized to do when delivery is uncertain. If that rule is absent, the comparison has found a process gap rather than a model winner.
Resolve the rule with its owner and revise the expected outcome. Preserve the reason for the change. Do not quietly adjust the rubric to make a preferred tool win. Rerun relevant cases under the new rule.
If you use another model as a judge, ask it to score specified criteria and cite the relevant source passage. Calibrate its judgments against human review. A second model can share the first model's mistake, and a confident explanation is not independent proof.
OpenAI's evaluation guidance describes comparison-based assessment and task-specific evaluation. The business rubric and fictional delivery example here are our recommendations.
Choose for the job, then retain the test pack
Report accepted cases, hard failures, correction effort and unresolved scope. Do not compress everything into “A scored 87, B scored 84” if one candidate repeatedly violates the important boundary. A tool can win routine extraction and lose difficult customer drafting; that is a reason to route work differently.
Repeat important cases enough to notice unstable behavior. Keep failed and missing outputs in the denominator. A small trial can identify defects and guide a pilot, but it does not establish a universal accuracy rate or a permanent ranking.
Task and accepted artifact: Same source pack and instruction version: Candidate setup, date and tool access: Hard failures: Scoring anchors: Blind labels and presentation order: Reviewer scores and source-backed disagreements: Correction minutes: Accepted result / failure / unresolved scope: Decision and conditions for retesting:
Take two outputs from a task your team already performs. Hide their names, write one hard failure and three usable criteria, then ask a colleague to review them against the source. If the preferred answer changes, inspect the reason before buying another subscription.
Sources and dates
Sources checked on 20 October 2026. Worked examples are teaching examples unless explicitly attributed. Provider documentation describes product behaviour; the operating recommendations are DLVX editorial judgement.
- OpenAI: evaluation best practicesTask-specific evaluation, representative cases and comparison methods; our business worksheets are editorial recommendations. Checked 2026-10-20.