From de9e66636adc951eb9a1467339e4c56ce2601d42 Mon Sep 17 00:00:00 2001 From: John O'Keefe Date: Sat, 17 Oct 2026 12:39:59 -0400 Subject: [PATCH] =?UTF-8?q?docs(agents):=20test-failure=20triage=20discipl?= =?UTF-8?q?ine=20=E2=80=94=20suspect=20the=20test=20first,=20but=20test=20?= =?UTF-8?q?edits=20stay=20asymmetric?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2026-10-17 lessons from the hardening session: most test/bruno failures were harness/schema/environment drift (drifted body keys, truncated payloads, container-only paths), found by triaging the test before the server. Codified as: triage order (test schema -> environment -> cheap probe), with the guard against "writing tests to work" — test/contract edits are asymmetric with code fixes: only harness plumbing fixes, additive pin widenings with a changelog entry, or deletions of tests of removed behavior. Pins were green at a known commit; git history arbitrates stale-vs-changed. A right-looking test that fails is prima facie a server bug: stop and flag. contract_pins.md rule 4: narrowing/removing a pinned key requires the user's explicit sign-off — a pin freezing buggy behavior is a ledger, not a license. --- AGENTS.md | 23 +++++++++++++++++++++++ docs/contract_pins.md | 3 +++ 2 files changed, 26 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 60819108..2eca4e97 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -67,6 +67,29 @@ were name-only across ALL libraries. Fixed; do not regress. - Analytics day-buckets must bucket in the user's timezone (`CreatedAt.In(loc)`), and date-range end bounds must be end-of-day. +## Debug discipline (test-failure triage) — 2026-10-17 + +When a test or bruno request fails, suspect the TEST FIRST — most +failures in this repo are harness/schema/environment drift, not server +bugs (2026-10-17 bruno sweep: drifted body keys, bodies pre-truncated +at their first `{{var}}`, missing `/api` prefixes; host runs hitting +container-only paths/hostnames). Triage order: (1) read the +test/fixture/payload schema and diff against a known-good sibling +(bruno: the `NewDevDBSetup` files are the reference for `body.data` +form); (2) check environment assumptions (host vs container, +harness cleanup, live-DB traffic); (3) cheap probe (curl, one SQL +query) before any deep dive. + +But test changes are ASYMMETRIC with code changes: server code may be +fixed freely when it is the bug; test/contract edits may only +(a) fix harness plumbing, (b) WIDEN pins additively with a +`docs/contract_pins.md` changelog entry, (c) delete tests of deleted +behavior — never narrow an assertion or flip an expectation to match +observed behavior. Contract pins were green at a known commit, so git +history arbitrates "test stale vs code changed." If a test looks right +but fails and no harness bug explains it, that is prima facie evidence +of a server bug: stop and flag the user. + ## Known gotchas (do not relearn) 1. **templ does NOT interpolate `{ expr }` inside quoted attribute diff --git a/docs/contract_pins.md b/docs/contract_pins.md index fe5b01fd..43da2f50 100644 --- a/docs/contract_pins.md +++ b/docs/contract_pins.md @@ -15,6 +15,9 @@ These are the response SHAPES (JSON key sets, not values) frozen by the 3. KOReader / Kobo / progress surfaces are the hard freeze: the KOReader plugin is an external client whose handlers keep response shapes byte-identical (only their internals may route through shared funnels). +4. A behavior fix that would require narrowing or REMOVING a pinned key + needs the user's explicit sign-off before the pin is touched — a pin + freezing buggy behavior is a known-bugs ledger, not a license. ## Pinned surfaces