The failure can be an improvement in the wrong direction
A fictional Thai service company upgrades its enquiry assistant. The new model writes friendlier replies and adds helpful context, but it also starts treating approximate dates as confirmed appointments. Staff like the tone until a customer arrives for a slot that nobody booked.
The supplier may accurately describe the model as better on its evaluations. Your workflow still needs its own test. Different prompts, tools, languages and business rules create a different operating environment.
Keep the decision narrow: does this candidate complete our defined job at least as reliably, within our limits for review effort, response time and cost? You are evaluating a complete configuration, not awarding a prize to a model name. If the upgrade changes tool arguments or handoff behaviour, include those effects in the comparison.
Record what is actually running
Capture the model identifier or snapshot where the provider offers one, instructions, tool definitions, generation settings, source selection rules and application version. A model alias can change underneath a workflow; record the returned model information when available and monitor the provider's notices.
OpenAI's deprecation register shows that model and endpoint availability can change. Deprecations, checked 20 October 2026. Keeping an old configuration is useful only while its dependencies remain available. The ultimate fallback may be supervised or manual work rather than the retired model.
Do not update the prompt, retrieval rules and model simultaneously unless the release genuinely requires all of them. Separate changes make failures easier to locate. If you must change several parts together, compare the full candidate bundle and save the previous bundle as the rollback unit.
Protect the evaluation environment too. Use the same source snapshots for the initial comparison, or a changing stock record can make one model appear wrong simply because it ran later.
Use the same cases, then inspect what changed
Run the current configuration and candidate against the same test cases. Preserve both outputs and blind reviewers to which one is new where practical. Include cases that previously failed, ordinary work and high-consequence edge cases. Repeat important cases to expose variability.
OpenAI's evaluation guidance recommends continued evaluation, including as systems change. Evaluation best practices. The release thresholds below are examples of business decisions, not provider guarantees or universal statistical rules.
Suppose a fictional comparison has 50 cases. The old workflow passes 40; the new one passes 44. That looks attractive, but paired review shows six repaired failures and two new failures. One new failure creates an unauthorised appointment. The total improved by four while the release became unacceptable for booking work.
Record wins, regressions and unchanged failures. Break the results down by relevant categories, such as Thai enquiries, mixed-language inputs, missing dates and tool outages. A high overall score can hide a specific group that now receives worse service.
Agree on the stop conditions before seeing the result
Write hard conditions separately from preference scores. For this fictional assistant: no unauthorised appointment creation, no invented availability and no customer record crossing. These conditions block the release when observed. A finite test with no such failure does not prove the true failure probability is zero.
Then define acceptable changes in first-pass acceptance, review minutes, latency and actual usage cost. A faster model that requires more human correction may be a poor trade. A slower model may still be useful for overnight document preparation and unsuitable for a live customer conversation.
For numerical extraction or tool calls, inspect the saved fields and arguments. Do not rely on the final narrative saying it followed the rules. For a drafting workflow, ask reviewers whether they would send the answer unchanged and why not. Keep the failure reason attached to the case.
When the candidate fails, diagnose the cause before tuning. If a changed output format broke your parser, repair and test the integration. If stricter instructions fix one case, rerun the earlier successes and the untouched holdout set.
A shadow trial should not create real side effects
In a shadow trial, the candidate receives appropriate inputs and proposes outputs while the current workflow remains responsible for actions. Disable the candidate's write and send path. Otherwise a supposedly observational test can create duplicate customer messages or records.
After offline checks pass, start with a limited supervised group of eligible work. Choose a group whose boundaries are easy to enforce, such as one internal drafting category. Watch actual correction effort and missing outputs as well as successful examples.
Set the observation window and escalation owner before starting. A low-volume workflow needs enough real cases to learn anything; two quiet hours do not establish reliability. Keep the owner available during the initial release instead of upgrading immediately before a holiday or shift handover.
If a serious regression appears, pause the affected action path first. Restoring an old model does not undo a message already sent or a record already changed. Reconcile the affected work and assign any customer correction to the responsible person.
Keep a rollback card someone else can follow
Model release card Current configuration ID: Candidate configuration ID: Source snapshot and test-pack version: Paired results: wins, regressions, unchanged failures Hard-condition failures: Review time, latency and actual usage observations: Approved limited rollout boundary: Monitor owner and stop trigger: Rollback Pause new consequential actions. Restore tested prior configuration if still available. Reconcile in-progress and unknown-outcome actions. Run smoke cases and inspect destination state. If prior dependencies are unavailable, use manual fallback. Record affected work and correction owner.
Include prompt and tool versions in the restoration step. Switching only the model while leaving a new parser or instruction bundle in place may produce a combination you have never tested.
The release decision can be partial: use the candidate for internal summaries but retain the current configuration for customer commitments. That is a sensible outcome when the evidence supports different boundaries. The point of the comparison is to choose where the change helps, not to justify an upgrade that has already become inevitable.
Sources and dates
Sources checked on 20 October 2026. Worked examples are teaching examples unless explicitly attributed. Provider documentation describes product behaviour; the operating recommendations are DLVX editorial judgement.
- OpenAI evaluation best practicesContinued evaluation as systems change; paired examples and thresholds are editorial. Checked 2026-10-20.
- OpenAI deprecationsModel and endpoint availability changes; rollback requires available dependencies. Checked 2026-10-20.