A demo should resolve uncertainty
The vendor's assistant finds a customer record, drafts a reply and updates a dashboard. It is smooth. Your team applauds. A week later nobody can say whether it handles your shared inbox, your missing customer references or the approval step that matters most.
The demonstration answered “Can this look convincing?” Your buying question was “Will this particular work finish correctly with less effort in our environment?” Prepare the second question before joining the call.
Choose one recurring job and one accepted result. For example, a service enquiry should become a correctly assigned draft case with the source message and unresolved facts attached. Keep automatic sending outside this trial unless sending is the specific authorized capability you intend to evaluate. The scope should be concrete enough that a colleague can call a result incomplete.
Give the vendor the rules and a representative practice set
Prepare six to ten synthetic or appropriately permissioned cases from the shapes of work your team sees. Include ordinary cases, missing information, a correction later in the thread, a similar customer name, and a request outside your service. Supply the approved policy and the fields the destination needs.
Tell the vendor how you will judge the demonstration. This is not a trick examination. They should know which errors are unacceptable and what evidence you need. Give them several practice cases so they can configure the workflow properly.
Keep a few cases back to test whether the workflow generalizes beyond its rehearsed inputs. Do not reveal the expected answer inside the case itself. If the test file says “this is the duplicate customer case,” you are measuring the system's ability to follow the label rather than identify the ambiguity.
Use safe records. A sales demonstration rarely needs your entire live customer database, private contracts or unrestricted production access.
One awkward enquiry is worth several perfect slides
Consider this fictional enquiry: “Please book the same package as last time for Friday. My colleague sent the address yesterday.” The supplied records contain two customers with the same first name, no verified link to either one, and a later message changing the delivery area.
A polished failure says: “All confirmed for Friday at your previous address.” It sounds helpful and creates a commitment the evidence cannot support.
The accepted result is a draft case that preserves the source, flags identity and address uncertainty, identifies the requested date as needing clarification if the context is insufficient, and routes the question to the responsible colleague. It must not expose either customer's private record while guessing which one is right.
Ask the vendor to show the actual saved case. A generated summary on their demonstration screen is not evidence that your team will find the correct fields in the destination system.
Write pass conditions before seeing the answer
| Condition | Pass evidence | Failure |
|---|---|---|
| Source retained | Reviewer can open the input | Untraceable summary |
| Identity handled | Unknown remains unknown | Unrelated customer exposed |
| Current request used | Latest correction is visible | Old address treated as current |
| Ownership assigned | Named queue accepts the case | Record exists without follow-up |
| Commitment controlled | Draft stays pending | Unapproved confirmation |
Separate hard failures from improvements. A slightly awkward sentence may need editing. Revealing another customer's address should fail the case. Do not average them into one score that allows excellent tone to offset a serious defect.
Have the people who will review the output measure their effort. Builder speed and operator effort are different quantities. Record the time needed to find the source and decide whether the result is safe to use.
Ask to see an ordinary failure
Request a controlled test in which the destination is unavailable. The vendor should explain what remains saved, who sees the failure and how work resumes. Ask what happens if a retry arrives twice. A successful workflow needs a recoverable state, not just a happy-path recording.
Then change one approved fact, such as a service area or response template, through the intended maintenance route. Observe whether the next run uses it. If the vendor edits a hidden prompt manually during the demo, record that as configuration work rather than pretending the business user completed the update.
Ask what happens when the account owner leaves, access expires or the model changes. You do not need every incident simulated in one call. You do need a specific operating answer and an owner for the parts the demonstration cannot establish.
Make the quotation match the tested workflow
After the demo, ask for a proposal that names the tested version, required accounts, included steps, exclusions, support responsibilities and export method. Separate the one-time setup from recurring charges and variable usage. Do not let an attractive prototype silently become a much broader production promise.
Mark untested claims as untested. If the demo used English only, Thai support remains a separate acceptance task. If it processed one attachment, complex multi-attachment threads have not passed. If the builder operated every control, usability for your staff remains unknown.
A useful outcome can be “do not buy yet.” Perhaps a form with clearer required fields solves the problem, or the current office suite already handles the narrow task. The demo has value when it changes a decision, even when that decision is to avoid a purchase.
Send a short acceptance brief before the meeting
Business job and destination result: Current source of truth: Who will operate and review it: Practice cases supplied: Held-back cases for the session: Hard failures that disqualify a case: Required evidence in the destination system: Review time to record: One recovery scenario to demonstrate: Untested scope to label explicitly: Commercial decision this demo should support:
OpenAI's evaluation guidance discusses task-specific examples and evaluation design. This procurement worksheet is an editorial application of those ideas, not a certification scheme.
Before your next vendor call, pick the awkward enquiry your team most often has to rescue. Convert it into a safe test case and write the expected disposition. Send the rule, keep the exact case for the session, and ask the future operator to score the result.
Sources and dates
Sources checked on 20 October 2026. Worked examples are teaching examples unless explicitly attributed. Provider documentation describes product behaviour; the operating recommendations are DLVX editorial judgement.
- OpenAI: evaluation best practicesTask-specific evaluation, representative cases and comparison methods; our business worksheets are editorial recommendations. Checked 2026-10-20.