NFR and evaluation release gates
Duckbill™ is not releasable until the service-quality gates in 03_NFR §16 have evidence. This page is the Stream B (MAIL-51 / MAIL-109 / MAIL-167) checklist. It is not the full MVP exit matrix (MAIL-49 / MAIL-62) and it is not the live-mailbox E2E (MAIL-50 / MAIL-103).
Recorded: 2026-09-26T19:52:01.969Z. Environment: local worktree feature/MAIL-109-nfr from origin/preview @ b886b6b, Node 26, ephemeral PGlite, 24 timed samples unless noted. ENABLE_SEND and ENABLE_DELETE stayed false. AI evaluation used the fixture classifier only — it never called a mail adapter.
Owners: agent = this Stream B change. host = Jason / human approval, live mailbox, or production promotion.
Checklist
| Gate | Result | Owner | Evidence |
|---|---|---|---|
| Replay / idempotency / out-of-order / cursor | pass | agent | tests/application/initial-sync.test.ts, message-identity.test.ts, incremental-sync.test.ts, cursor-recovery.test.ts, tests/domain/ownership.test.ts, tests/persistence/operations.test.ts, tests/application/command-service.test.ts, tests/application/webhook-ingress.test.ts |
| Reconciliation | pass | agent | tests/application/reconciliation.test.ts (3 tests) |
| Write-back convergence | pass | agent | tests/application/writeback-convergence.test.ts, tests/application/command-service.test.ts |
| Privacy / logging (redaction + scan) | pass | agent | tests/privacy/redaction.test.ts, tests/privacy/token-secret-redaction.test.ts, tests/nfr/privacy-log-scan.test.ts |
| Restore exercise | pass | agent | tests/ops/backup-restore.test.ts, fixtures/ops/restore-rehearsal-checklist.json |
| AI golden-set | pass | agent | npm run ai:eval, tests/ai/golden-set.test.ts |
| Performance p95 measured (local synthetic) | pass | agent | tests/nfr/performance-sample.test.ts vs 03_NFR §4 |
| No known data-loss defect in automated suites | pass | agent | Replay, identity, reconciliation, and restore suites created no duplicate or wiped canonical rows |
| Live-provider p95 under a real mailbox | fail | host | Not measured in this stream. Follow-up MAIL-217. Blocked by MAIL-103. |
Free-text log fields (lastError) vs ciphertext | pass | agent | MAIL-218: redactString now strips the project ciphertext prefix from free text. tests/privacy/redaction.test.ts, tests/privacy/token-secret-redaction.test.ts, tests/nfr/privacy-log-scan.test.ts. |
| Full release-docs matrix | not in scope | host | MAIL-49 / MAIL-62. Cite this page; do not treat it as the matrix. |
Failures become linked Jira issues. There are no silent waivers.
Replay, idempotency, cursor, reconciliation, write-back
Local Vitest on 2026-09-26: all listed files passed.
- initial-sync page replay without duplicate instances
- MailMessage identity replay (
existing_mapping) - incremental delta cursor advance only after durable processing; stale / out-of-order provider versions ignored
- expired-cursor recovery without wiping canonical rows
- webhook duplicate suppression (no second AI enqueue)
- ProviderMutation idempotency keys
- command retry with the same idempotency key
- three-level reconciliation (missing, orphan, stale, duplicate)
- write-back persist-before-provider, denied send/delete, echo suppression
AI never calls adapters in these suites.
Privacy and logging
Redaction is always on. The composite scan in tests/nfr/privacy-log-scan.test.ts feeds a synthetic body, OAuth token, token ciphertext, client secret, and JWT through sync, provider, write-back, and ops loggers plus redactHealthDocument and buildModelPrompt. Named sensitive fields are [REDACTED]. The sink did not contain those needles.
Safe metadata (message id, folder, subject) may remain. Production mailbox content is forbidden in fixtures/.
MAIL-218: redactString treats a bare project ciphertext prefix (v1.aes256gcm. / v1.aesgcm.) as a secret when it appears in a non-sensitive field such as lastError. Named-field redaction is unchanged.
Restore exercise
fixtures/ops/restore-rehearsal-checklist.json is the disposable PGlite rehearsal artifact from MAIL-56 / MAIL-112. The suite re-ran the export/restore path on 2026-09-26:
- environment:
pglite-disposable - RPO 24 h / RTO 4 h recorded
- recovery mode entered; mutations paused
- classifications, rules, and audit restored (
classificationsRestored: 1,rulesRestored: 1,auditRestored: 1) productionMailInFixture: false- provider tokens and body caches skipped
This is not a production-mail restore.
AI golden-set
npm run ai:eval ran scripts/run-ai-eval.ts against tests/ai/golden/corpus.json with createFixtureClassifier: 12/12 passed, 0 schema-invalid, 0 critical regressions, 0 high-impact false positives, fixture cost $0. The Vitest self-test (tests/ai/golden-set.test.ts, 10 tests) passed. This is the 01_AI §11 / §14 gate, not a live-model latency test.
Performance sample vs 03_NFR §4
Isolated sample 2026-09-26T19:52:01.969Z; contended re-run during full npm test 2026-09-26T19:52:32.385Z (numbers in parentheses). Inclusive p95 via @inbox-agent/observability percentile. Do not read local pass as live-mailbox pass.
| NFR target | p95 measured | Result | Environment |
|---|---|---|---|
| API metadata read < 500 ms | 1.51 ms (5.78 ms) n=24 | pass | PGlite getById |
| Interactive list/search < 1.5 s | 4.67 ms (16.91 ms) n=48 | pass | PGlite listByAccount + in-memory triage home/search (80 conversations) |
| UI state-change ack < 1 s | 0.42 ms (9.51 ms) n=24 | pass | in-memory submitCommand (MarkRead; send/delete off) |
| Background event pickup < 30 s | 1.23 ms (2.72 ms) n=24 | pass | PGlite outbox insert + publishOutboxBatch |
| New-mail canonical availability < 2 min | 16.15 ms (88.15 ms) n=24 | pass (synthetic only) | mock adapter + PGlite runIncrementalSync |
| AI enrichment < 5 min | 0–1 ms n=12 cases | pass (fixture only) | fixture classifier case latencies |
Live Microsoft Graph discovery-to-canonical p95 and live-model enrichment p95 were not measured. That row stays fail for representative production conditions.
Observability
Critical API and sync metrics already exist (MAIL-75): api.duration_ms p95, sync lag, queue depth/age, DLQ, reconciliation drift. Health documents are redacted before they leave the process. See operational health and privacy and logging.
Kill switches and flags
ENABLE_SEND and ENABLE_DELETE remain false. ENABLE_MICROSOFT_WRITES is only true inside write-back unit tests. PAUSE_AUTOMATION and PAUSE_AI stay default-off configuration keys.
Follow-up
Opened from this gate run:
- MAIL-217 — measure live-provider p95 for new-mail canonical availability and live-model AI enrichment (moved to MAIL Sprint 08; needs a hosted classify worker)
- MAIL-218 — done: redact token ciphertext when it appears in free-text fields such as
lastError
File another linked issue when a later gate fails. Do not mark the live-provider p95 row pass because the local sample passed.