ADR 0020: Golden-set AI evaluation harness

Status

Accepted

Context

Duckbill™ cannot treat classification or extraction as production-ready without a reproducible evaluation gate. MAIL-36 / MAIL-57 require a checked-in golden corpus and a runner that fails the suite on unexplained critical regression. Stream B must not call mail provider adapters. Classify and extract production APIs are not on preview yet (MAIL-61 / MAIL-77). Stream A already recorded ADR 0019 for untrusted input and the policy gateway.

Source: 01 AI §§10–11, §14; 03 NFR §12 / §16; Test Strategy AI evaluation dataset and golden fixtures.

Decision

1. Keep a versioned synthetic/redacted corpus in tests/ai/golden/corpus.json. Categories cover personal, work, notifications, newsletters, receipts, threads, forwards, quoted history, ambiguous requests, deadlines, attachments, and prompt injection. No live mailbox tokens or production mail.
2. Score classification and extraction through a small AiEvalPort in @inbox-agent/ai. The checked-in fixture-classifier is a deterministic double that looks up golden labels and treats injection text as untrusted. It is not a second production inference stack. MAIL-61 / MAIL-77 can later implement the same port.
3. The runner in packages/application/src/ai-eval.ts is deterministic from schema / taxonomy / prompt / model / corpus versions. It records schema validity, critical-field regressions, high-impact false positives, latency, and cost. Invalid JSON is not coerced. The process exits non-zero on unexplained critical regression. CI runs npm run ai:eval in addition to the Vitest self-test.

Alternatives

• Wait for MAIL-61 / MAIL-77 before any harness — rejected; MAIL-114 must run now, and the story requires a gate before material AI changes can be called done.
• Invent a heuristic production classifier in this ticket — rejected; that is a second inference stack and belongs to classify/extract issues.
• Call live provider adapters to fetch evaluation mail — rejected; AI must not call adapters. Fixtures and recorded mail only.

Consequences

npm test and npm run ai:eval fail when golden labels, schema validity, injection handling, or high-impact false-positive rules regress. Production classify/extract/persist remain out of scope. MAIL-8 stays open. ADR 0019 remains the Stream A security decision.