docs(agents): test-failure triage discipline — suspect the test first, but test edits stay asymmetric
2026-10-17 lessons from the hardening session: most test/bruno failures were harness/schema/environment drift (drifted body keys, truncated payloads, container-only paths), found by triaging the test before the server. Codified as: triage order (test schema -> environment -> cheap probe), with the guard against "writing tests to work" — test/contract edits are asymmetric with code fixes: only harness plumbing fixes, additive pin widenings with a changelog entry, or deletions of tests of removed behavior. Pins were green at a known commit; git history arbitrates stale-vs-changed. A right-looking test that fails is prima facie a server bug: stop and flag. contract_pins.md rule 4: narrowing/removing a pinned key requires the user's explicit sign-off — a pin freezing buggy behavior is a ledger, not a license.
This commit is contained in:
1 parent
14cdbe70eb
commit
de9e66636a
2 files changed
+26
No files matched your search
@@ -67,6 +67,29 @@ were name-only across ALL libraries. Fixed; do not regress.
|
|||||||
- Analytics day-buckets must bucket in the user's timezone
|
- Analytics day-buckets must bucket in the user's timezone
|
||||||
(`CreatedAt.In(loc)`), and date-range end bounds must be end-of-day.
|
(`CreatedAt.In(loc)`), and date-range end bounds must be end-of-day.
|
||||||
|
|
||||||
|
## Debug discipline (test-failure triage) — 2026-10-17
|
||||||
|
|
||||||
|
When a test or bruno request fails, suspect the TEST FIRST — most
|
||||||
|
failures in this repo are harness/schema/environment drift, not server
|
||||||
|
bugs (2026-10-17 bruno sweep: drifted body keys, bodies pre-truncated
|
||||||
|
at their first `{{var}}`, missing `/api` prefixes; host runs hitting
|
||||||
|
container-only paths/hostnames). Triage order: (1) read the
|
||||||
|
test/fixture/payload schema and diff against a known-good sibling
|
||||||
|
(bruno: the `NewDevDBSetup` files are the reference for `body.data`
|
||||||
|
form); (2) check environment assumptions (host vs container,
|
||||||
|
harness cleanup, live-DB traffic); (3) cheap probe (curl, one SQL
|
||||||
|
query) before any deep dive.
|
||||||
|
|
||||||
|
But test changes are ASYMMETRIC with code changes: server code may be
|
||||||
|
fixed freely when it is the bug; test/contract edits may only
|
||||||
|
(a) fix harness plumbing, (b) WIDEN pins additively with a
|
||||||
|
`docs/contract_pins.md` changelog entry, (c) delete tests of deleted
|
||||||
|
behavior — never narrow an assertion or flip an expectation to match
|
||||||
|
observed behavior. Contract pins were green at a known commit, so git
|
||||||
|
history arbitrates "test stale vs code changed." If a test looks right
|
||||||
|
but fails and no harness bug explains it, that is prima facie evidence
|
||||||
|
of a server bug: stop and flag the user.
|
||||||
|
|
||||||
## Known gotchas (do not relearn)
|
## Known gotchas (do not relearn)
|
||||||
|
|
||||||
1. **templ does NOT interpolate `{ expr }` inside quoted attribute
|
1. **templ does NOT interpolate `{ expr }` inside quoted attribute
|
||||||
|
|||||||
@@ -15,6 +15,9 @@ These are the response SHAPES (JSON key sets, not values) frozen by the
|
|||||||
3. KOReader / Kobo / progress surfaces are the hard freeze: the KOReader
|
3. KOReader / Kobo / progress surfaces are the hard freeze: the KOReader
|
||||||
plugin is an external client whose handlers keep response shapes
|
plugin is an external client whose handlers keep response shapes
|
||||||
byte-identical (only their internals may route through shared funnels).
|
byte-identical (only their internals may route through shared funnels).
|
||||||
|
4. A behavior fix that would require narrowing or REMOVING a pinned key
|
||||||
|
needs the user's explicit sign-off before the pin is touched — a pin
|
||||||
|
freezing buggy behavior is a known-bugs ledger, not a license.
|
||||||
|
|
||||||
## Pinned surfaces
|
## Pinned surfaces
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user