Why Did Working AI Features Break After a Model Change?

1. Record Changes Before Editing Production Prompts

Preserve one failed task with inputs, expected and observed outcomes, time and ID. Record provider and model version, settings, prompts, index, tools and application commit. Check whether aliases or managed services changed. Sanitize logs and keep credentials private. Retain previous configuration for comparison.

Build a timeline of model, document, chunking, prompt, API and access changes. Reconstruct comparable configurations in test before isolating variables. Do not repeatedly write production data. Pause risky actions when work is affected, retaining safe queries or human drafts and an assigned recovery owner.

2. Compare Fixed Tasks, Not a Few Conversations

Use authorized, sanitized tasks including frequent work and rare costly exceptions. Define fields, allowed evidence, actions and escalation conditions. Business owners approve expected outcomes; engineers make runs reproducible. Model grading is only an aid, not a substitute for field or access checks. Resolve ambiguous examples first.

Repeat sensitive or unstable tasks according to an agreed plan and retain all outcomes rather than the best screenshot. Compare refusals, access, tool calls, latency, edits and cost as well as quality. Results from different environments are not directly comparable. Passing evidence covers tested conditions, not every future input.

3. An Illustrative Contract Extraction Regression

This is a design example, not a measured client case. A contract workbench extracts parties, amount, expiry and renewal terms for draft reminders. Test normal contracts, poor scans, amendments, absent expiry and denied access. Misreading an amendment date as expiry is a serious defect even if average accuracy improves. Show evidence and confirm before creating reminders.

For illustration, 18 correct results out of 20 describe those 20 tests only. A tenant data leak blocks release regardless of a 90% average. Record repetition, sample makeup and configuration. These numbers explain measurement, not a client result or guarantee. Agree severity and thresholds from actual business impact.

On narrow screens, scroll horizontally to see all columns.

4. Stage Releases with Stop and Recovery Controls

Compare in test or a non-writing shadow setup, then use an authorized small cohort. Shadow runs still create cost and logs and require access approval. Assign scope, reviewers, stop criteria and follow-up. Show draft status, required confirmation and fallback processes so users understand responsibility.

Separate recovery of code, models, indexes and business data. A retired model may not be recoverable, and reverting it cannot undo sent reminders. Stop intake, classify active, completed and uncertain tasks, and reconcile each appropriately. Retest affected examples and tell users which results need review before reopening.

5. What to Inspect After an Employee Reports a Failure

Let employees flag a task and failure type without copying entire conversations. Inspect input changes, source validity, retrieved clauses, model output and tool results. A wrong contract date may arise in extraction, interpretation or timezone conversion. Show evidence, versions and edits; employees report business mismatches rather than diagnose implementation.

Record investigation status, affected users, temporary handling, owner and recheck conditions. Fix missing evidence, ambiguous rules or API errors at the relevant layer. Keep unexplained incidents open for verification rather than inventing a cause. Add authorized sanitized regression examples and check similar tasks with retention and access controls.

6. Compare Costs per Completed Business Task

Lower request prices do not establish lower task costs. Include failed attempts, retries, retrieval, tools and human checks. Compare identical scope and samples, reporting first-pass completion, retries, escalation and unresolved work without dropping failures. Measure human effort explicitly or mark it unmeasured; generated text volume is not labor savings.

Longer outputs or additional tool iterations can offset lower model prices. Budget experiments and production separately, with limits, alerts and over-limit behavior. Report trial costs without guaranteeing future monthly bills. Assess completed outcomes within agreed risk and time constraints before expanding to more teams.

7. Scope Costs, Maintenance and Handover

Quote diagnosis, task-set preparation, adaptation, staged release and ongoing maintenance separately. Missing baselines, sources or API documentation require discovery first. Separate development from model, test infrastructure and subscription costs. Define inspectable scope before promising remediation of an unknown system.

Deliver version differences, tasks, item-level results, failures, fixes, release and recovery steps and limitations. Distinguish provider changes, source updates, new requirements and defects under agreed responsibilities. Maintainers should rerun tests and locate active configuration. Begin inquiries with symptoms, timing and sanitized examples, not production access.

Official information and scope of verification

Reference check date: 2026-10-06. Platform capabilities change with the version, the package, the area and the authority; information is used to describe technical capabilities and does not represent search volumes, the results of the customer in Sino-China or the original cooperative qualifications.

• Anthropic: Success Criteria and Evaluation Design (External Reference)