H3 leftover: the annotation handlers route everything through the sync
service (dedup/LWW/tombstones), leaving the plain Create/Update/Delete
queries for media_notes/media_highlights/media_bookmarks with zero
callers repo-wide (handlers, services, tests). Queries.sql drops them
so H4's row work doesn't carry dead weight; sqlc regen compiles clean,
proving nothing referenced the generated stubs.
Also removes the literal duplicate `AND mi.missing_scan_count = 0`
line in the library-stats query (harmless but confusing).
Caught by the live round-trip sweep: a dedup-key re-save (a device echo
or a client re-POSTing a bookmark it already had) carried no locator
fields, and the full-column sync update wrote zeroes — which
pgFloat8/pgInt4 turn into NULLs — erasing the stored
percentage/page/cfi on every echo. The highlight stack had string
coalescing for exactly this reason (its locatorRefresher doctrine
comment); the numeric columns and the two cloned stacks never got it.
One primitive now carries the whole doctrine: coalesce[T comparable]
(zero value -> stored value), applied at the single write point of
each LWW apply — highlight (percentages, color, note_text,
chapter_reference added to its existing string coalescing), bookmark
(page/chapter/percentage/epubcfi/chapter_reference added to its
cfi/position pair), note (all locators). The redundant pre-coalescing
in the id-targeted resolution blocks is removed — the apply is the one
coalesce point, so dedup-path and id-path edits behave identically.
Verified live on the rebuilt container: dedup re-save and legacy
3-field PUTs preserve every stored locator on all three stacks; an
explicitly-sent field still lands. Pins, H1/H3 tests, unit suites
green.
The consolidation half of H3 — shared LOGIC with thin per-type remains:
- lwwDecision: the three compareIncoming* functions were the same
skeleton (client timestamp -> strictly-newer wins; no timestamp ->
any content difference wins) drifting apart only in per-type
equality and the schema-driven fallback timestamp (bookmarks carry
no updated_at). The decision now lives in one function; each compare
keeps only its equality expression.
- idTargetOwnership + tombstoneWins: the id-targeted resolution blocks
(HighlightID/BookmarkID/NoteID) shared the ownership check and the
deleted-row gate — now one gate each instead of three copies of the
resurrect semantics.
- queue.go: syncNote/syncHighlight/syncBookmark decoded device payloads
through three 10-key if-type-assertion chains; typed map accessors
(mapStr/mapF64/mapI32/mapDeviceSyncData) replace them — same
missing-key-is-zero semantics, declarative field lists. The progress
decoder is untouched (H1 territory, separately tested).
Per-type remains are the things that genuinely cannot merge: request
structs, dedup triples, the per-table sqlc param builders, and the
KOReader/Kobo payload adapters. Pins, H1/H3 tests, both unit suites
green.
H3 census bugs #2 and #3: both web PUTs used plain queries that
bypassed the sync-aware layer entirely — no LWW, no source bookkeeping,
and locator fields the client couldn't even send. The highlight PUT was
the only one already routed (via HighlightID); the other two now mirror
it:
- SaveBookmarkRequest gains BookmarkID, SaveNoteRequest gains NoteID:
id-targeted resolution reads the exact row, verifies ownership,
LWWs under its stored dedup key, and resurrects tombstones when the
edit is newer (same doctrine and comment structure as the highlight
path).
- Fields the PUT omits are coalesced from the stored row before the
full-column sync update — the legacy web PUT sends only
title/notes/position, and zeroing cfi/percentage/page on every edit
would have been data loss. Verified both directions by
TestH3_BookmarkPUTPreservesLocators (omitted survives, explicit
lands).
- The bookmark PUT's wire DTO gains the create endpoint's locator keys
as optional fields (documented additive request keys in
contract_pins.md); the note PUT keeps its content+position wire.
UpdateMediaBookmark/UpdateMediaNote plain queries are now uncalled
(the generated queries stay until the H4 query sweep).
H1's doctrine applied to the annotation service, which had been left on
setters: NewMediaHandler/NewKoboHandler/NewKOReaderHandler/
NewSyncQueueProcessor(WithConfig) now take it as a parameter; the four
SetAnnotationService setters are deleted; wiring updated (main.go,
router/sync.go, test harness).
With the service unskippable, the 18 nil-guards and their plain-query
fallbacks die:
- media.go: the create fallbacks for notes/highlights/bookmarks (the
bookmark one was the H3 census's Bug #1 — it dropped
percentage/chapter_reference the sync path carries), the highlight
PUT fallback, and the three delete-path plain deletes (tombstone is
now the only delete semantics).
- kobo.go: both markup-ingest annotation loops (the bookmark fallback
wrote the text into media_notes — a category error the fallback
never should have had).
- koreader.go: all three push-stack guards.
- queue.go: the three "annotation service not available" errors in
syncNote/syncHighlight/syncBookmark.
Behavior note: the deleted highlight create/PUT fallbacks were the only
paths passing note_id to the plain query — the sync service has no
NoteID field, so the web's highlight→note link was already being
dropped on the live path; this commit makes that pre-existing
limitation explicit rather than papering it over (surfacing note_id on
SaveHighlightRequest is left for a behavior commit that needs it).
Pins (now 20 incl. the 3 annotation CRUD suites) + H1 tests + both
unit suites green.
Pin-first batch for H3: freeze web annotation CRUD response shapes
(create 201 / update 200 / delete 204; full sqlc row models) before
the stack consolidation touches anything.
The new pins immediately caught a live server bug the 28 unit tests
could never see (annotations_test.go is pure-unit: keys and compare
logic, no DB): applyNoteLWW built UpdateMediaNoteForSyncParams without
UserID, so the query's WHERE id=$1 AND user_id=$13 compared against
zero — every dedup-hit note re-save (same content+position) returned
500 "update note: no rows in result set" and the update never landed.
The bookmark twin passes UserID and works; the note stack is the clone
that dropped the field.
Fix: pass UserID in the params (one line). Pins go green and become
the regression net; changelog records the new surfaces.
The last behavior gap in the H2 batch: has_conflict was part of the
frozen card shape but stamped false-forever on these three JSON
surfaces (only dashboard sections, search, and the SSR pages had it),
so every conflict-driven card action — the resolve-conflict CTA in
BookCard — never fired there. MarkActiveConflicts now runs after the
builder on GET /api/collections/:id, series books, and smart-collection
preview (all user-scoped; the endpoint already authenticates).
Key set is unchanged — has_conflict was always emitted, this only
makes its value truthful. All 15 contract pins + H1 tests green; the
pin's bookInfoKeys freeze unaffected.
All five branch shapes of GET /collections/:id now build cards via the
cards.go adapters: smart (BookCardsFromMediaItems), system-query (same
— gains format_group/badge it previously dropped), scoped
(BookCardsFromCollectionDashRows with the Excluded filter kept), and
the all-libraries branch whose rows distinguish ID vs MediaItemID
(BookCardsFromCollectionItemRows). LibraryTypeMap hoisted to once per
page; both []BookInfo{} error seeds die with the adapters' non-nil
empties.
The ExecuteSearch-driven bookshelf page fed SearchMediaItemsUnifiedRow
— which HAS FormatGroup and LibraryTypeName — into a bare 4-field
literal that dropped both, so bookshelf cards lost the media-type
badge the shelf once had. BookCardsFromUnified emits badge/format on
this surface (card-marking rides on; the SQL column strategy means
zero extra queries). audiobook shelves regain the corner glyph.
Both BrowseDetail surfaces hand-rolled bare 4-field literals (badges
dropped) and patched the nil-slice with []BookInfo{} seeds after.
BookCardsFromMediaItems feeds GetSeriesBooks/GetBooksByTag rows —
cards gain format_group + badge on the series and tag pages, and the
two seed blocks die with the adapter's non-nil empties. Conflict
stamping at these sites was already adjacent and is preserved.
router/search.go (the templ BooksGrid fragment surface) now uses
BookCardsFromUnified — the unified row's LibraryTypeName column gives
the badge with no LibraryTypeMap query; also drops that site's
needless pgtype.UUID byte-roundtrip on the cover URL.
The dashboard section builder — feeding both the dashboard JSON pin
surface and the SSR dashboard — now calls BookCardsFromMediaItems on
ds.Items. libTypes already arrives as a BuildSections parameter
(once per request); the literal only duplicated the badge lookup.
GET /api/series/detail?name=… books array via BookCardsFromMediaItems.
Series had the third LibraryTypeMap call INSIDE its per-item loop —
ListLibraries ran once per book on the page; now once per request.
POST preview-feed had the ladder's 14th hand-rolled literal. Now
BookCardsFromMediaItems + one LibraryTypeMap per request; the response
carries format_group/badge values for the first time (all seven keys
were already emitted as zero-value JSON, so this is a value change on
a pin-frozen shape, not a key change).
GET /api/collections/:id books — all four branches now build cards via
the cards.go adapters. LibraryTypeMap hoisted to once per handler (it
was 1 query per item in the smart and no-library branches). The system
branch — which before stamped cards BARE (no format_group, no badge —
the ladder's only JSON gap) — now emits the same card shape as its
siblings by feeding section.Items through BookCardsFromMediaItems,
matching what BuildSections produces for the identical data. The
collection-error seed dies via the adapters' non-nil empty slices;
the no-matching-section nil result is preserved as-is (that branch
never seeded — today's 'items': null stays today's 'items': null).
Pins: CollectionsJSON, DashboardSections, MediaListBothBranches green.
BookInfo, LibraryTypeMap, and MediaBadgeKind move verbatim out of
collections.go into handlers/cards.go (the card package surface, same
package: no import-graph or templ changes), joined by the H2 builder:
- buildBookCard (unexported): the ONLY place BookInfo is constructed,
fed by cardSource; cover URL via ResolveMediaURL, badge via
MediaBadgeKind — both inside the builder, not per call site.
- Four exported adapters normalize the four real feed shapes:
database.MediaItems (sections/smart/series/tags/preview),
GetCollectionItemsForDashboardRow (collection dashboard branch),
GetCollectionItemsRow (collection-items joins — MediaItemID field),
SearchMediaItemsUnifiedRow (badge rides its LibraryTypeName column,
so no LibraryTypeMap query at all).
- Adapters return non-nil empty slices, which retires the per-site
[]BookInfo{} empty-seed assignments when call sites migrate.
No call site changes in this commit — the 20 literals and 5 seeds are
migrated one surface per commit next, each with the H0 pin suite.
Verified: build, vet (only the two pre-existing flagged saveCoverImage
warnings), and 14/14 contract pins — the key freeze is byte-identical.
2026-10-17 lessons from the hardening session: most test/bruno failures
were harness/schema/environment drift (drifted body keys, truncated
payloads, container-only paths), found by triaging the test before the
server. Codified as: triage order (test schema -> environment -> cheap
probe), with the guard against "writing tests to work" — test/contract
edits are asymmetric with code fixes: only harness plumbing fixes,
additive pin widenings with a changelog entry, or deletions of tests of
removed behavior. Pins were green at a known commit; git history
arbitrates stale-vs-changed. A right-looking test that fails is prima
facie a server bug: stop and flag.
contract_pins.md rule 4: narrowing/removing a pinned key requires the
user's explicit sign-off — a pin freezing buggy behavior is a ledger,
not a license.
Book-side twin of the audio-survival test, closing H1's verification
list. Scenario: web push carries percentage + epubcfi + context_text;
a koreader-style percentage-only push from a second source within the
window detects a conflict; the user resolves with the CFI-carrying
side. The funneled resolution must land the winner's ENTIRE locator
pair — percentage 0.5, epubcfi, context_text together (the old direct
writer NULLed context_text and character_offset, breaking the coherent
pair the far-anchor work established) — and must not spawn a duplicate
pending conflict.
Follow-up to the sweep route audit, correcting its diagnosis: the CLI
DOES read .yml bodies — via OpenCollection's `body.data` string form
(the NewDevDBSetup script always worked because it used it). The
failing scenario POSTs used `body.json` (structured) / `body.jsonBody`
(string) — keys the CLI silently ignores, sending Content-Length: 0
with form-urlencoded content-type, so every server bind failed.
- 81 bodies converted to data-form (json maps serialized; jsonBody
strings kept verbatim, {{vars}} intact).
- 6 pre-existing TRUNCATED payloads repaired: the original generator
cut bodies at the first {{var}} (every SHA-256-identified scenario
plus Sync Progress/Sync Bookmarks shipped mid-JSON and 400'd from
day one). Rewritten whole, faithful to each file's documented
purpose; book_sha256/book_uuid added to the environment.
- environments: opds_base_url correction from the sweep retained.
Verified live against the rebuilt dev container: the full koreader
scenario tree runs green through `bru run` — 14/14 (202s across
immediate/checkpoint/SHA-256 progress, 200s bookmarks, highlights,
notes; SHA-256 book identification exercised against a real
file_sha256).
Audited every request (247 .yml files) against the routes the server
actually registers (extracted from internal/router, group prefixes
resolved). Results:
- 44 stale requests: deleted where the endpoint no longer exists
(universal-progress GET/PUT /api/progress/:id — routes never existed,
the handlers were deleted in H1; queue stats/clear-failed/process;
admin profile page; auth/account) and FIXED where the endpoint exists
under a different path (kobo token-in-path scheme: every
/api/sync/kobo/<seg> and /api/v1/kobo/<seg> URL now carries
{{kobo_device_token}}; /sync/* missing their /api prefix; koreader
/api/v1/koreader/sync/* -> /api/sync/koreader/*; link-book and
unlinked-books moved to /api/devices/:id/sync/*).
- koreader Sync Highlights/Sync Notes pointed at endpoints that never
existed with a body shape to match — rewritten against POST
/api/sync/koreader/bookmarks (the annotations funnel carries
highlights and notes arrays).
- 9 files the GUI refused to load: 4 had bruno-invalid script type
"tests" (converted to after-response), 4 were info/type:collection
doc files with bogus http blocks (converted to folder.yml folder
docs), 1 had a dead pre-YAML schema and no route (deleted).
- devices/koreader requests now carry request-level bearer auth
({{koreader_device_token}}) — they inherited the collection's user-JWT
bearer, which device endpoints rightly reject — and their bodies
reference {{media_item_id}} instead of dead placeholder uuids.
- environments: opds_base_url was empty (every OPDS request missed the
/opds prefix); set to http://localhost:8765/opds and un-doubled the
prefix in opds/scenarios.
Known limitation, for the handoff: bru CLI cannot execute .yml request
bodies (the collection is GUI OpenCollection format; the CLI runner
reads bodies from .bru files only). A .bru migration is the remaining
bruno work if CLI runs are wanted.
Found while verifying H1's "app offline-play -> flush -> position lands"
with the emulator path: the flush has been dead code in two independent
ways, invisible because the first break made the second unreachable.
1. ListPendingSyncQueueItems matched `device_id = $1` with a NULL $1
(the processor passes DeviceID{Valid:false} to enumerate all devices;
uuid = NULL matches no rows). The ticker therefore processed nothing,
ever. The WHERE now treats NULL as "every device" (the same
narg::uuid IS NULL idiom the visibility queries use; EXPLAIN'd — seq
scan on a bounded table, idx_sync_queue_status_priority_created
fits). The processor also dropped its fake-pagination loop, which
re-fetched the same first batch forever once the backlog hit the
batch size (no OFFSET in the query).
2. dbItemToSyncQueueItem dropped SyncData — even with a working
enumerator, every flushed item arrived with an empty payload and
failed JSON unmarshal. Dead code harboring silent breakage, exactly
the plan's thesis.
Callers updated for the renamed RowLimit param (queue handler, offline
detector). TestH1_SyncQueueFlush_LandsAndPreservesAudio drives the real
flow: web PUT seeds audio 900s, a pending checkpoint-style queue row
applies percentage 0.8 through SaveProgress, audio survives, queue row
completes.
Also fixes dashboard_test.go's stale BuildSections signature (the
package's test binary has been unbuildable) — user-approved: tests
designed against old shapes are not worth keeping.
The cleanup in setupSyncTestDB ran DELETE FROM sync_queue WHERE true —
a table-wide wipe that discards REAL devices' pending offline entries
on every suite run. Same class as the 2026-10-14 reading_history
incident the standing rules ban. The test's own queue items cascade
from its 'Test %'-named device, so the wipe is redundant for cleanup —
scoped it to those devices (defense in depth ahead of the cascade).
Flagged during H1 verification: the line executed during an in-
container suite run this session; any device with pending offline
queue entries at that moment should re-push on its next sync.
Batch H1 step 4:
- Handler GetUniversalProgress (progress.go) deleted — no route mounts
it; only /api/progress/:id/history (GetProgressHistory) and the SSR
/progress page exist. Note: the bruno universal-progress suite
documents GET/PUT /api/progress/:id + /api/progress/device/:device_id
— those routes do NOT exist server-side; that suite is stale
documentation, flagged in the plan.
- Handler GetAllProgress deleted — unregistered near-copy of
GetAllProgressData (which the /progress SSR page uses and stays).
- series.go: dropped the `_ = libTypes` no-op (libTypes IS used by the
Badge line right below — the assignment stays).
- router/helpers.go: visibleLibraryIDsFor's signature and first
statement were fused onto one line; split for readability. No
behavior change.
Batch H1 step 3b — THE data fix. Conflict resolution was the one live
erasure writer: applyProgressResolution and its verbatim twin
applyResolution both wrote the winner via a raw UpdateUniversalProgress
whose param list omits audio_offset_seconds/context_text/viewports, and
the SQL's DO UPDATE SET ... = EXCLUDED... NULLed exactly those. The web
UI resolves through POST /api/conflicts/:id/resolve and
/api/conflicts/bulk-resolve (main.js resolveConflict/bulkResolve), and
the reader's mark-as-read fires a resolution first when the reader SSR
stamps data-conflict-id/-winner — so every resolution today erased the
audiobook position and reader context.
Now: ProgressService is a required ConflictHandler dependency; BOTH
call sites (single resolve + bulk resolve) share one
applyProgressResolution that funnels through SaveProgress (the verbatim
twin is deleted). Deliberate choices, per the plan:
- SkipConflictDetection on the resolution write: both flows apply
BEFORE the conflict row is marked resolved, so the funnel's LWW
detector would otherwise file a duplicate pending conflict
(HasRecentConflictResolution is false at apply time).
- Source "conflict_resolution" — the row's last-write attribution; the
winner stays durably recorded in resolution_data.winner.
- Broadcast: true — live devices jump to the winning position.
- The winner snapshot maps percentage/epubcfi/context_text/chapter/
character/page/total_pages; a winner without a CFI clears the locator
pair (ClearLocator) instead of re-pairing with the loser's stale
anchor. Everything the snapshot doesn't carry SURVIVES via the LWW
seed: audio_offset_seconds above all. chapter_progress keeps its
stored value (was overwritten with the book percentage), an absent
percentage survives (was written as 0%).
- Accepted: the resolution captures one reading_history session
(attributed to conflict_resolution; the 2h idle ceiling applies).
Verification: TestH1_ConflictResolutionPreservesAudio drives the real
flow — web push with audio 300s, diverging push from a second source
detects a conflict, resolve with the second source as winner, GET shows
percentage 0.3 AND audio_offset_seconds 300 (previously NULLed), and no
duplicate pending conflict remains. Full contract pin suite green.
The host-run TestConflict*/TestProgress* failures are environmental
(container-only /app/uploads path + db hostname; fail identically at
baseline b4ae423) — the same suites pass in the test-runner container.
Batch H1 step 3a. The ceiling census (exhaustive UpdateUniversalProgress
grep) found 9 nil-guarded `if progressSvc != nil { funnel } else { raw
erase-write }` fallbacks — the survey's "four" under-counted kobo.go,
which has FOUR sites (Markup ReadingSync, Markup last-read-place,
AnalyticsGettests, SyncFromServer), plus one each in media.go, queue.go
and koreader.go. All unreachable: the service is created unconditionally
at main.go:127 and wired into every handler and the queue processor; the
test harness wires them too.
The erasure pattern is now unrepresentable:
- NewMediaHandler / NewKOReaderHandler / NewKoboHandler / both
NewSyncQueueProcessor flavors take the ProgressService as a REQUIRED
constructor arg; the SetProgressService setters are gone.
- Every fallback body (raw UpdateUniversalProgress with explicit-NULL
pgtypes for viewport/scroll/panel/reading_mode/zoom and no
audio_offset_seconds/context_text) is deleted; each write funnels
through SaveProgress with the LWW seed, so fields the writer doesn't
know survive.
- Removed the now-dead koreader pg-type helpers (textPtrToPgText,
intPtrToPgInt4, int64PtrToPgInt8) and their self-tests.
Behavior deltas (stored-row content, no wire shape changes —
KOReader/Kobo response pins stayed green):
- KOReader fallback used to write explicit NULL viewports and drop the
context_text the koplugin sends; the funneled mainline already maps
book.ContextText.
- Queue flushes no longer overwrite chapter_progress with the book
percentage (bogus data; preserved via the seed).
- A web PUT without percentage no longer writes 0% via the deleted
fallback; percentage seeds from the stored row.
Contract suite: KOReader×4 + Kobo×3 pins green.
Batch H1 steps 1-2 of the hardening plan. Wire contracts frozen; the
pinned shapes are unchanged modulo the whitelisted additive GET keys
(docs/contract_pins.md).
1. One canonical request struct. The web PUT binds SaveProgressRequest
directly (identity/attribution fields are json:"-" so they stay
server-owned) and copies its server fields once — the hand-rolled DTO
and the 20-line field-copy block are gone. Parse set for existing
clients is IDENTICAL to the old DTO (verified in web/src: the ebook
reader sends percentage/current_page/total_pages/reading_mode/epubcfi/
context_text/chapter/zoom_level — reader.ts:3113-3143; the audiobook
reader sends percentage/chapter/audio_offset_seconds/elapsed_seconds/
reading_mode — audio.ts:518-520).
2. GET /api/media-items/:id/progress now serializes through one
row-struct-driven builder instead of a hand-map, so a row column
cannot be silently forgotten again (the audio_offset_seconds
omission). Existing keys are verbatim; the row already carried
audio_offset_seconds/reading_mode/viewport_x-y/zoom_level/
scroll_position_x-y/panel_number and they surface as WHITELISTED
additive keys. The web audiobook reader already reads
audio_offset_seconds (audio.ts:219) — with the key absent,
durationless books resumed at chapter start (offset always 0); they
now resume correctly. The conditional context_text bolt-on and the
server-derived css_selector/anchor_href block are unchanged —
restore precedence stays percentage-first, never re-derived.
Batch H0 of the hardening plan: freeze the JSON key SETS of every
load-bearing surface so H1-H7 can only change them additively.
- cmd/server/tests/contract_pins_test.go: 14 tests / ~30 subtests pinning
progress GET+PUT, the KOReader plugin surfaces (progress push/pull,
library, bookmarks), the Kobo adapter shapes (initialization, markup,
sync-from-server), collections detail all four branches + preview,
series JSON, search rows, dashboard sections, media list BOTH branches
(raw sqlc rows vs resolved 38-key map), and SSR smoke for
bookshelf/collections/series/tags.
- docs/contract_pins.md: the pinned-shape inventory + the rule that a
batch widening a shape must widen the pin and this doc in one commit.
Suite is green on the unmodified tree at b4ae423. KOReader/Kobo/progress
pins are the hard freeze (the plugin is an external client).
The daily charts' x-axis labels were bare MM-DD ('10-15', '10-16') — on
the Listening per Day chart (y in hours) those read as hour values, which
is exactly the '10-16 hours a day' misread that prompted this. Labels now
spell the month ('Oct 15') and every chart's tooltip carries its unit
(min / pages / hrs). main.js?v=5
The /stats page gains a second tab (Reading | Library) implementing the
Audiobookshelf-style library dashboard (HANDOFF item 9): media_items
INVENTORY aggregates — items, hours/pages, distinct authors, size,
distinct narrators, Top Genres / Top Authors bars, Longest / Largest
items — for one library or the All-libraries aggregate.
- GET /api/stats/library-usage?library_id=<uuid|omit>; family lens
enforced: a library_id outside the requester's visible set is a 403,
never a silent pass (AGENTS.md standing rule; 431fec0 history)
- six sqlc queries with the visibility pattern + healthy-item filter
(archived_at IS NULL AND missing_scan_count = 0); hours_mode lets the
client swap the Hours/Pages card and the Longest ranking per scope
- Reading tab unit audit (HANDOFF item 3, settled 2026-10-16): minutes
are the merged currency everywhere honest; only the PAGES unit splits
— Pages card goes mode-aware (Pages · text / Listening Time in an
audio-only pool) and the charts row gains Listening per Day (hrs)
when audio minutes exist in the range; summary daily buckets gain
audio_minutes split by the item's library type
- tab state in the URL (?tab=library&library_id=…); selector fed by the
existing /api/libraries/visibility endpoint; main.js?v=4
- integration tests: family-lens 403 pin, hidden-library exclusion
(differential), seeded text+audio aggregate correctness, auth gate
- docs/developer/api/stats/get_library_usage.md
Known-remaining suite failures are environmental: TestConflictList_Empty
+ TestWebSocketPingPong collide with the concurrently-running app session
(live emulator pushes + scans on the shared dev DB), and
TestAdminCreateUser's timezone expectation drifted from live system
settings — all three fail on the clean tree too.
A2 made narrators/narrators_search NOT NULL DEFAULT '{}' (86f0af6), but an
explicit NULL in an INSERT/UPDATE column list bypasses the column default
— the API media-create path (POST /api/media-items without narrators)
500'd on every call, taking the whole integration suite down with it
(ISBN tests, capture seeds, all direct-DB seeds).
- set_library_type_name() trigger now coalesces NULL narrators/
narrators_search to '{}' at the row level, and its update trigger
widens from BEFORE UPDATE OF library_id to BEFORE UPDATE so every
write path is covered
- defense in depth at the callers: handler create/update normalize nil
to empty slices (normalizeNarrators), scanner seeds the same
(nilToEmpty) for files that carry no narrator credits
- analytics device-usage tests move to the dedicated per-test user
(the shared admin's device list accrues live rows mid-suite — the
documented testuser flapping class)
Reader fullscreen: the topbar gains a fullscreen toggle (icon state + f
shortcut + help row; f is guarded so Ctrl/Cmd+F stays find-in-page).
Enter requests the Fullscreen API with fallbacks (desktop/Android get a
true full display; iOS Safari has no page-fullscreen API and falls back
to the soft visual mode only). Margins: the foliate paginator's closed
shadow root only yields through its observed 'margin' attribute, so
fullscreen fits a comfortable ratio (6% of the short side, floored at
48px) and floors it by the measured env insets — Android reports true
cutout insets in fullscreen, so portrait phones keep text clear of the
camera band; iOS letterboxes are already env-padded chrome. Fixed-layout
renderers skip it (fxl/webtoon pack pages edge-to-edge by design).
Exits the user makes outside the toggle (Esc, browser UI) sync via
fullscreenchange(+webkit) and restore the ordinary 48px margin; a
position-keeping re-render follows the ResizeObserver re-entry path.
Verified on-device: enter -> true fullscreen + icon swap + camera band
clear, exit -> chrome back, margins restored, position kept.
Volume: revert the stacked panel — the popup returns to the liked shape
(mute button left, horizontal dragger right, dialogue above the strip);
one-line flex restore of the pre-89665fb layout.
Manga badges: MediaBadgeKind makes the library type authoritative for
the comic/manga split — manga volumes ship as fixed_layout scans as
often as comic_archive (and manga_type is unreliable across sources:
unknown/NULL on real rows), so the old comic_archive+manga mapping left
the scan volumes unbadged. Anything in a manga library reads manga;
anything in a comics library reads comic. Verified: /dashboard lands 24
title="Manga" cover badges for the Manga library and the sections API
reports badge=manga for a fixed_layout item.
Ghost sidebar: the >=1024px media block applied body
padding-left:var(--sidebar-w) unconditionally, so pages that never
render the sidebar (logged-out pages, the audio player, the reader
itself) carried a phantom 16rem left margin that pushed content off
center on desktop widths. The body:has(.app-sidebar) rule below was
already the correct conditional source — it is now the only one.
Verified in the built stylesheet: exactly one such rule remains.
Infra: main.js ?v=3 stamp bump (reader.ts changed; hashed chunks churn
their URLs themselves).
Annotations banner: audio/text bookmarks join notes and highlights —
third row kind (bookmark glyph indicator, H:MM:SS / Page N / Chapter N
friendly labels), edit-note/delete actions branch by kind, and the
banner's export now covers all annotations. The archived ledger
(ListDeletedAnnotationsForBook) already tombstoned bookmarks equally
and Restore/Purge take annotation_type=bookmark — verified live:
delete → restore brings the row back with its note intact.
Volume: the dragger sits BELOW the mute icon in the panel (stacked
flyout shape) with the horizontal slider from the last round.
Badges: the real fix for comics/manga — format_group only knows
'comic_archive' (manga_type is unreliable across sources), so the
comic/manga split is resolved from the LIBRARY type server-side.
handlers.MediaBadgeKind + BookInfo.Badge at every construction site
(dashboard SSR + sections API, collections, series, and the bookshelf
search grid — which was also missing FormatGroup entirely, so no card
ever badged there). Both renderers (templ + dashboard.ts) switch on
the resolved badge. Verified: All-Books grid renders 2 Comic + 3
Manga badges, dashboard the audiobook one.
Infra: header.templ main.js src gains ?v=<stamp> — browsers keyed an
intermediate bundle by URL and served it across rebuilds (the stale
'Add note' the user saw); bump the stamp when JS stale-shadows.
Bookmarks: notes are markdown now — capture and drawer edit are
textareas (raw source), display renders through the reader's
markdown-it + DOMPurify pipeline (renderNoteHtml) as x-html; the
edit textarea drops the enter-to-save handler so newlines survive.
Volume: the panel dragger is horizontal (mute button beside it,
7.5rem slider) per user feedback — vertical orientation read as a
scrubber clone at a glance.
Play/pause: unicode glyphs replaced with SVG — the triangle is
optically centered (2px right-bias) and the pause bars are longer
and thicker rounded rects (4.2×15 at rx 1.7).
Dashboard: the client-side card renderer (dashboard.ts — the path the
JSON sections actually render through) now mirrors the templ BookCard:
cover-corner badges for audiobook/comic/manga (books unbadged) and the
hover action shows headphones with 'Listen to …' labels for audiobooks.
BookInfo type + Go struct carry format_group end to end.
Verified on-device: pause glyph centered/thicker, horizontal volume
panel, markdown note ('**bold**' + list) rendered in the bookmarks
drawer, badges on dashboard cards, sections JSON carrying
format_group.
Transport row is now the five-button audiobook shape the user asked
for: previous-chapter · circular-skip-back (seconds displayed inside
the ring) · play/pause · circular-skip-forward · next-chapter — the
ring arrows are stroke SVGs (rotate-ccw/rotate-cw) with the number as
an overlay so it stays upright and follows the skip setting.
Settings strip collapses to four controls: Skip dropdown, Speed
dropdown, Bookmark (capture popover unchanged), Volume. Volume opens
a panel with a vertical dragger (writing-mode:rtl slider) and a
mute/unmute button; the strip icon is dynamic — 1–3 arcs by level (low
<34%, mid <67%, high) and crossed when muted or zero. Volume/mute
persist in localStorage and restore with the element; unmute floors
zero-volume to 50% so unmuting is audible.
Verified on-device: dropdowns open upward with current value
highlighted, mute flips both icons to the crossed state, slider
drag changes arc count, chapter buttons jump (pill → 15/34) and the
caption tracks the drawer's chapter labels.
A2a (metadata round): narrators TEXT[] + narrators_search (tags
pattern: accumulator + GIN pairs) + abridged bool on media_items;
scanner fills from the Libation/ABS sidecar (narrators verbatim,
abridged flag) with audio-payload promotion; editorial fields ride
the real columns while audio_metadata stays scan-derived (duration/
codec/chapters). Detail/edit/save paths carry the fields; the web
edit dialog grows an Audio section (Narrator(s) comma-input, Abridged
toggle, scanned duration read-only); search weights + narrator filter
(word_similarity over narrators_search) + narrator autocomplete added.
A2c (per-media-type wording): copy helpers keyed on format_group —
Listen Now, Mark Complete/Not Complete, Listening Progress, Your
Listening Stats, Time Listening, Time listened (pages tile swapped for
audio), Position H:MM:SS / total H:MM:SS on detail + progress pages
(duration parsed from audio_metadata). Dashboard: system collections
swap to Continue Listening / Recently Listened at DISPLAY time when
the audiobooks library is selected (stored names unchanged; restores
keep working); week card = Listening Time / Sessions / Books Touched;
cover-corner badges identify audiobook (headphones) / comic / manga
cards on mixed grids — books stay unbadged; hover action icon follows
the media type.
A2b (player): the scrubber binds value to the playhead with an explicit
scrub guard (the handle previously froze at its initial max); header
row = reader-style back arrow + Chapter n/total launcher; chapters
open in a floating top-down drawer (current chapter highlighted, page
never shifts); Skip/Speed rows share one label-left layout; honest
wall-clock session accounting (playing-only deltas, paused flush).
A2d (bookmarks): note-bearing time bookmarks ride the existing
media_bookmarks substrate (notes column + sync-aware service already
there) — audio locator convention audio:<seconds>, chapter_number
filled, auto-title '<chapter> — H:MM:SS', capture popover with optional
note, drawer with jump/edit-note/delete. Zero server changes needed.
Verified live on the Libation sample: narrators backfilled by rescan;
PUT edit persisted + overrides detected (narrators, abridged); force
rescan preserved the user edit; reset-to-scanned-of-defaults restored
truth; narrator filter/autocomplete scoped by library; dashboard
listsening wording + badges render; bookmark save with note deduped to
one row.
Phase 1 (foundation): 'audiobooks' library type seeded (schema + DB);
audio_metadata JSONB on media_items (the kobo_metadata precedent —
only audio paths read it); reading_progress.audio_offset_seconds for
the honest {chapter, offset} locator; extensions/mimes registered for
ALL breadth formats (m4b/m4a/mp3/flac — deliberate per the DOC/LIT
doctrine, m4b scans richly first); classifyFormatGroup 'audiobook'
case (gotcha 39 recomputes on rescan); Go mime table extended at boot
so media elements accept the streams; ffmpeg/ffprobe in the image
(8.1.2 — also the xHE-AAC identification tripwire).
Phase 2 (ingest): sidecar-first (Libation/ABS metadata.json —
narrators/ASIN/curated chapters), ffprobe fallback/filler (duration,
bitrate, codec/profile, embedded chapters). Fixed three real
parser brittlenesses the Libation sidecar exposed: series accepts
string OR array-of-objects, publishedYear accepts number OR string,
ffprobe chapter timings accept number OR string (flexFloat). Folder-
structure fallback now runs on the UPDATE path too (rescans were
clearing sidecar-absent series/author); audio facts re-derive on
every rescan (gotcha 39).
Phase 3 (playback + sync): GET /api/media-items/:id/chapters (the
format-abstraction list); streaming rides the existing cookie-authed
Range /uploads path; PUT progress carries audio_offset_seconds +
elapsed_seconds (honest wall-clock playing time) — the reading-
history capture prefers player-reported elapsed over the inter-push
delta, same 2h ceiling; reading_mode='audio' marks the row.
Phase 4 (web player): dedicated audio reader page (foliate not
involved) — cover, chapter title, whole-book scrubber, skip ± (chips
1/5/10/15/30s, default 10s), speed (0.75×–2×), chapter list with
jump-to, Media Session lockscreen/notification transport, periodic
playing-only progress pushes, position restore on open.
Verified live: ingest of the Libation sample (Gardens of the Moon —
title/author/series/cover/date from sidecar+folders, 34 chapters,
26h duration, aac/LC); chapters endpoint; the reader route serving
the player; playback with position restore (0:57 round-trip);
progress rows in audio mode; wall-clock session rows in
reading_history.
Completes the reader-chrome UX batch (handoff item 7): the gitea
mirror now carries fork 0.5.4, so the pin replaces the temporary
node_modules patch the verification build used. Rebuilt and re-verified
live (chrome starts hidden; flick-down reveals).
The chrome previously woke on ANY activity: pointermove (mouse drift
and, worse, the touch drag of a page turn itself), every keypress, and
every touch tap toggled it. Now reveal requires explicit UI intent and
reading actions suppress:
- Desktop: cursor into the top 10% / bottom 8% band raises the chrome
(screenY-based so content iframes classify identically); content
pointer movement never does. Mouse center-clicks no longer toggle.
- Touch: a vertical flick (dominant + fast) reveals; while visible,
flick down OR up dismisses. Plain taps are fully inert — a stray tap
can never flash the UI.
- Keyboard: page keys never poke; turning (keys, mouse edge zones, or
a touch swipe via the snap-relocate) actively dismisses visible
chrome.
- Highlight saves return to a clean screen: the selection flow owns
the screen and the post-save pointer/keyboard pokes are gone.
- Chrome starts hidden; always-visible pin and auto-hide timer remain
(bands use a shorter 1.2s fade).
Pairs with the foliate-js 0.5.4 paginator change (scrolled layouts
scroll manually so the flick gesture is observable in webtoon); the
fork pin bump follows once the gitea mirror shows f622841.
Extends the device-authed koreader surface with the handoff item 4
Batch B contract:
- GET/POST /api/sync/koreader/rating — the device mirrors the .sdr
summary's 5-star rating against the 1-10 server scale (POST 0 clears).
- GET/POST /api/sync/koreader/review — raw-markdown review mirror
(summary.note); empty body clears, matching the first-party PUT.
- POST /api/sync/koreader/stats — the device pushes its CUMULATIVE
total reading seconds; the server watermark (highest total ever
converted, read from the koreader sessions' own device_metadata)
converts only the delta into a reading_history session attributed
to the koreader device. Reinstalls cannot double-count; sub-minute
deltas ride forward (no session, no watermark movement); a lagging
device clamps to zero instead of dragging the total back.
Book identity resolves through the existing priority chain (uuid →
sha256 → alias → title/author). GET binding uses query tags; POST
endpoints bind the ref+payload once (an earlier resolve-then-bind
draft drained the body).
Live-verified against the dev server with the koreader device token:
rating set/get/clear, review mirror, and the stats sequence
900→session, 960→delta session, 970→rides forward, 500→clamps; the
book's stats card and the By Device panel both pick up the split.
media_reviews table mirrors media_ratings' shape (unique per user+item,
cascade deletes, updated_at for LWW) but standalone: a review without a
rating must work. Body is RAW MARKDOWN — storage and every payload carry
the source; rendering happens only at display sites (same doctrine as
notes, same sanitization pipeline: markdown-it + DOMPurify).
API: GET/PUT/DELETE /api/media-items/:id/review (PUT upserts; an empty
body deletes — clearing via an editor is a delete, not an empty row).
Book-detail page payload now carries the review beside the rating.
Page: the review renders above the synopsis as a scrolling card (same
max-height box as the summary), hydrated client-side through
renderNoteHtml; 'Write review'/'Edit review' sits next to the rating
stars and opens a raw-markdown editor dialog (save / delete / cancel).
sqlc regenerated; schema applied to the dev DB. Live-verified: upsert,
get, empty-delete, page markup, and the app round-trip against the dev
server.
Bug: the admin 'Scanning Libraries' widget fed the server's 0-1
progress straight into percent strings and bar widths, so the overall
and per-library percentages capped at '1%' and the bars never filled.
Both now scale by 100 (matching the setup page, which was already
correct).
Bug: scan state only reached clients via websocket events, so a page
opened (or refreshed) mid-scan showed nothing until the next tick —
and the admin widget never re-attached at all. The worker now exposes
active jobs (with their library_id so rows keep their names), served
as GET /api/scanner/active (admin, re-attaches the widget + polling)
and GET /api/scanner/active-summary (any user, restores the sidebar
indicator on load). Idle hides both, exactly as before.
Annoyance: the sidebar's 'Scanning… N%' was a block element between
the logo and the nav, shoving every menu entry down while a scan ran.
The percent now lives inside the fixed-height logo row next to the
spinner (spinner last, so growing digits extend leftward); nothing
below it moves.
The htmx self-refresh was bound to an element that was itself being
destroyed: the widget rendered inside #collections-container, and the
switcher's renderDashboardCollections replaces that container's whole
innerHTML on every library switch — listener and all — before any
refresh could fire. Hence 'widget disappears instead of updating'.
The widget now renders in its own strip between the switcher subbar
and the sections container, which the switcher never touches. The
week-refresh htmx re-fetch (with the current #library-select value)
then has a living target and re-scopes the numbers per library.
Verified: the subbar's refresh button drives the identical
re-render path and the widget persists with its data.
sync_integration_test's cleanup ran DELETE FROM reading_history WHERE
true — harmless for years while the table sat empty, and a silent
stats-data destroyer the moment session capture started feeding it:
every suite run erased the shared dev accounts' captured reading
(the user's manual flip-through test data included).
The cleanup is now scoped: the sync tests' sessions belong to their
test-sync users, and deleting those users cascades their rows. Any
future table-wide cleanup DELETE is off the table (AGENTS.md test
rules updated).
Three dashboard complaints, one build:
- The widget collapses to a single summary row (This Week · minutes ·
pages · sessions · books · streak · Full stats) on every screen size
and expands to the full grid via a details/summary toggle — the
carousel sections start immediately below it again.
- The 'Your Library' heading moved into the library switcher subbar
(LibrarySwitcher gains an optional title param; collections/series
keep the plain 'Library' label). The standalone heading block is
gone.
- Root-caused the JS-switch vanishing: renderDashboardCollections
replaces ALL of #collections-container's innerHTML, so anything
server-rendered inside it (title, widget) died on the first library
switch. The widget now re-fetches ITSELF: a /dashboard/week-widget
fragment (same resolveLibrary family-lens rules as the page) is
pulled by htmx whenever the switcher dispatches week-refresh —
carrying the current #library-select value — so the widget both
survives switches and RE-SCOPES to the switched library. Initial SSR
renders the same scope the sections use.
ThisWeek (and the streak calendar) gained an optional library filter.
Tests: TestReadingStatsWeekLibraryScope — per-library totals, scoped
streaks, invisible-library isolation, malformed id 400. Full suite
PASS.
The bucket name read as confusing — 'unsynced' implied data loss when
the push is always captured; it only lacks a device credential. The
bucket is just 'Web' now (service fallback, stats.ts fallback, panel
copy, tests).
Also: analytics_test exact-zero assertions (EmptyHistory,
NoReadingHistory) ran against testuser, which is a LIVE reader — real
sessions accrued on it mid-suite and both flapped. They now run under
the fresh regular user, like the stats tests; the 'testuser is a live
reader' rule is recorded in AGENTS.md.
- Attribution: registered clients may now send their device token in
X-Device-Token on PUT /api/media-items/:id/progress (the user JWT
stays primary). The token must map to a device OWNED BY the caller —
foreign/invalid/absent tokens soft-fail to plain web attribution so a
stats decoration can never break a sync. Attributed pushes capture
device_id (FK-safe), device_metadata.source = the registered device
name ('Pixel 9', 'Kindle Paperwhite'), and last_sync surfaces show
that device. Side effect worth having: app-vs-web pushes used to be
LWW-invisible (both sources read 'web'); they now surface as
legitimate cross-source progress conflicts when they diverge.
- /stats gains a By Device panel alongside By Library (grid 3-wide):
reading minutes + sessions + books per registered device, plus one
Web / unsynced bucket for device-less pushes so the panel always
reconciles with the headline totals. Range-aware, family-lens
scoped; renderBars shared between panels in stats.ts.
- The device identity IS the registered platform: devices rows already
carry the name/type the client registered (app sends Build.MODEL +
'mobile' at /api/devices/register, KOReader/Kobo likewise) — no new
identity plumbing on the server.
Tests: capture attributed to the device (row device_id + source
metadata + last_sync_source), foreign/nonexistent tokens soft-fail,
device buckets reconcile (80 attributed + 75 unsynced = 155 total).
Full suite PASS.
The design parked in the handoff (multi-tier stats off reading_history;
bookorbit-style in data, not looks) is in. The real prerequisite work
was data: reading_history existed but NOTHING wrote to it — the whole
stats feature was sitting on an empty table.
- CAPTURE (no client changes, no repo-boundary crossings): every
successful progress write through SaveProgress (KOReader + app + web
+ Kobo + offline queue — the one chokepoint) now records one
reading_history session row: span = the interval since the PREVIOUS
write instant, pages = positive page delta when both sides know the
page index, marker rows for first contact. Spans over 2h are kept as
zero-second markers (overnight suspend→resume, offline flushes must
not inflate totals) — the honest 'sync cadence, not a stopwatch'
caveat stays. device_id only when a REAL device row matches (the
web/app progress path historically stuffs the USER id into DeviceID
— probe instead of trusting it; FK violation would silently drop
every capture). Source preserved in device_metadata.
- API: /api/stats/summary?range=7|30|90|year (totals, zero-filled
daily arrays, streaks, most-active day/weekday, top books, library
breakdown — one call), /api/stats/this-week (widget: rolling 7d vs
previous + streak), /api/stats/books/:id (lifetime + recent
sessions). All bucketed in the user's timezone; finished ≡ any push
≥0.995; visible-library lens enforced at source. Legacy
/api/analytics/* kept untouched. Range windows anchor end-of-day to
the START of today (blasting the clock time into tomorrow resolved
'today' a calendar day late — found by the streak test).
- /stats page (sidebar Stats between Progress and Devices, previously
/analytics was an orphan): range selector, five stat cards, two
Chart.js v4 charts (daily minutes bar, pages/day line — chunked via
dynamic import so main.js stays untouched), per-library breakdown,
most-read list, empty-state explainer. /analytics now 301s to /stats.
The orphaned analytics.templ + analytics.ts removed (their containers
and the page JSON never matched; the page was astub).
- Dashboard 'This Week' widget (server-rendered, suppressed on service
failure) and the book detail 'Your Reading Stats' card (lifetime
totals of the viewer's own sessions) complete the tiers: per-user
default, per-library breakdown inline, per-book on the page,
per-system/admin aggregate parked as before.
Tests: statistics_test.go (capture semantics at the chokepoint,
tz/day buckets, zero-fill, streak math, finished detection, range
validation, auth gate, widget, per-book lifetime). Full suite PASS.
From the smart-collections build: templ emits quoted attribute values
literally (whole-attribute expressions required — the rules page
shipped '/api/collections/{ collection.ID }' until proven via request
logs); SettingsRegistry only reloads through the settings API (direct
SQL writes look ignored); and user collections must never route through
the dashboard-section path (query_type='filter' is a schema DEFAULT,
not a query type — the 7ce7321 static-detail regression).
Type is DERIVED, not stored: a saved ruleset makes a collection SMART,
no rules = STATIC (stored collection_items links, unchanged behavior).
No schema change — collections.auto_assign_rules JSONB already carried
the config; user's decision 2026-10-14 (handoff: live-at-render, not
refresh-on-scan; create-time Static/Smart toggle; v1 NO hybrid).
- Rule gains an enabled flag (absent = enabled, legacy rules unaffected).
- Rule CRUD: POST/PUT(DELETE) /api/collections/:id/rules[/:ruleId] — the
survivor dead effort of the rules-builder UI finally has real
endpoints (rules were never editable after creation; nothing ever
re-evaluated them). First rule converts a user collection to smart;
removing the last reverts it to static. UpdateCollectionRulesJSON
edits ONLY the rules doc so rule CRUD cannot clobber metadata.
- Live evaluation: candidates from GetVisibleLiveMediaForRules —
visible-library set (family lens enforced at source) + archived IS
NULL AND missing_scan_count = 0 — any enabled rule matches, title
order. Zero refresh hooks by design: always current by construction.
- Membership mutations (add/remove/bulk-add) are 400 on smart — v1 no
hybrid; collection_items.excluded stays the reserved future hook.
- Rules builder page (/collections/:id/rules) first time ever routable;
templ whole-attribute interpolation ("{ expr }" inside a quoted attr
value is literal text, as theme-{ user.Theme } silently proved).
- UI: create modal Static/⚡Smart cards (smart create redirects into the
rules builder), ⚡ badge + real per-card counts on the grid (smart =
live matches, static = stored-link count), smart detail renders
matched books with no hand-pick actions + an Edit Rules button.
- API: GET /:id gains is_smart + live books; GET /: keeps the Android
app's default exclusion, include_auto=true surfaces smart rows with
live counts; TestRules previews against the SAME candidates the live
render uses (visible + live, was raw first-1000).
- FIXED pre-existing regression from 7ce7321: static collection detail
with NO library_id returned zero books — QueryType != '' caught user
rows ('filter' default) and routed them to dashboard sections that
never contain them; now only real system query types go there
(handlers + SSR). App-visible symptom: /api/collections/:id without
library_id came back books:[] for every static collection.
Tests: collections_smart_test.go (conversion, disabled rules, mutation
guards, revert-to-static, validation, system-collection prohibition,
include_auto semantics). Full offline integration suite PASS.
GET /api/auth/password-rules (public, pre-auth) exposes the ACTIVE
complexity config from the settings registry; password_validation.ts
now fetches it and generates the checklist rows per active rule —
inactive requirements no longer linger as fake tasks, and the min
length mirrors the admin's setting instead of a hardcoded 8.
Surfaces: register, setup, admin create-user (Users page), profile
password change (self + admin variants, now gated on checklist pass;
misleading minlength=6 dropped — the server was always the enforcer).
Verified live: flip min_length=12 / number+special off via the
settings API → endpoint + register/profile/admin checklists re-render
row-for-row; enforcement follows (short rejected, rule-valid accepted
without digits/specials); restore defaults → checklists return.
The sed that removed the verify-guidelines lines also deleted the
.PHONY line (it contained those words), so 'make release' saw the
./release wrapper script as an up-to-date target and silently did
nothing. Restored with the verifier targets omitted.
Three changes from the network-drive audit:
1. Delete-ordering fix. Deleting a media item cascades its annotations,
so the delete must never proceed while the item's library storage is
unreachable — the DB row died, the file survived as an orphan, and
the next scan re-imported it bare (annotations gone forever).
resolveMediaStorage stats the library's folders: reachable → delete
proceeds and the file is removed by ABSOLUTE path (the old code
passed a library-relative path to os.Remove, which never resolved —
file deletion has been a silent no-op); every folder unreachable →
409 with 'storage unreachable', row and annotations intact.
2. Latency: resolveLibrary stats every library folder on every SSR
render; folder reachability is now cached with a 30s TTL.
3. README: Network Storage section documenting the outage hysteresis
(mark → archive → auto-restore), the 90-day retention purge
(ARCHIVE_RETENTION_DAYS, 0 = never), refused deletes while storage
is down, and watch mode not firing on network mounts.
Integration tests: storage-down bulk delete refuses with the row
intact; real-file delete removes the file from disk.
PROJECT_GUIDELINES.md had drifted out of reality (podman-only rule
while the stack runs docker; verifier scripts wired to Makefile targets
nobody runs). Trashed it, its two verifier scripts, and the Makefile
targets; the rules that still matched reality (docker runtime, schema
in place until release, server binary only via Docker, services layer,
integration-test pool rules) are absorbed into the new AGENTS.md.
The library-visibility family-lens rule now has a real home:
- AGENTS.md: the standing rule agents auto-discover (new media
endpoints MUST thread the visible-library set; 403, never silently
pass; history of the 2026-10-13 leak).
- docs/developer/library-visibility.md: the full explanation (helpers,
SQL pattern, per-item content gates, series per-library), linked
from the developer docs index.
Stale doc links updated (docs index, contributing, development).
Field-by-field audit of the profile surface (set → save → reopen →
read back, driven through the emulator browser via CDP):
- getTemplateUserWithTheme (every SSR page + the /profile self-edit
form) never mapped FirstName/LastName/CreatedAt — stored names
rendered as empty inputs. Now mapped (timezone was already mapped).
- Admin profile-modal Timezone mapping fixed earlier this round held:
all seven fields verified set → 200 + HX-Refresh → reopen → correct
values, DB cross-checked.
- Role promote/demote via the card select verified against the DB (the
PUT answers HX-Refresh, so the page reload shows the new role).
- Admin password reset verified end-to-end: new password logs in, old
one 401s, original restored.
- admin_library folder-remove hx-vals was string-concatenated JSON —
a folder path containing a quote or backslash made it invalid and
the remove button silently died under htmx 2 (the same class as the
profile-save bug). Now built with json.Marshal behind the js:
prefix. Verified the generated attribute on the live page.
- Other hx-vals/hx-headers sites audited: constants and uuids only
(safe charset), left alone.
The admin profile-modal route hand-rolled its templates.User without
the Timezone field, so every selected?={ user.Timezone == … } in the
timezone select was false and the first option (UTC) rendered selected
for every user on every open. Saves worked fine — the modal just never
displayed what was stored, which read as 'not saving'. The route's
GetUser query already returns the column; it just wasn't mapped.
Verified end-to-end in the emulator browser: modal opens showing the
stored zone, save → HX-Refresh → reopen shows the new zone.
The single-line card squeezed identity, created date, role select, and
two buttons into one row — on a phone the username truncated to t…,
the email to te…, and the created date wrapped vertically one fragment
per line (10- / 13- / 2026 / …). Restructured into stacked full-width
rows: identity (avatar + username + email), created on its own line
(a full date never wraps), a labeled Role row under a divider, and
Edit/Delete as side-by-side flex-1 buttons (thumb-sized on mobile,
right-aligned sm+). Last-admin protections render identically in both
controls. Verified on the emulator (Pixel 9, Brave): full names/emails
visible, no wrapping, modal + role controls functional.
Reproduced on the emulator via CDP: clicking Save Changes fired the
submit event, something preventDefault-ed it, and htmx aborted while
parsing hx-headers — 'Bearer ' + localStorage.getItem('token') is a JS
expression, and htmx 2 requires valid JSON (or a js: prefix) there.
The PUT never left the browser: zero htmx lifecycle events, no console
error, no toast. My earlier HX-Refresh fix was correct but unreachable
— the request died before it could ever be answered.
The header was htmx-1.x legacy and redundant: the browser already sends
the token cookie, which the auth middleware accepts. Removed hx-headers
from the profile form and both password-reset buttons.
Verified end-to-end in the emulator browser via CDP: Edit → change
timezone → Save → PUT 200, HX-Refresh reload, DB updated. The tab
reload mid-probe is the fix working.
The admin/user profile modal PUT succeeded but the response was a bare
JSON body swapped into a small div at the bottom of the still-open
modal — no close, no toast, nothing moved. It read as 'the button did
nothing' even though the save landed. HX-Request calls now answer with
HX-Refresh (re-render shows the saved values everywhere and closes the
modal) plus a visible confirmation. API callers keep the JSON body.
Silent-failure display already exists site-wide (toast.ts listens for
htmx:responseError), so failures were never the silent case.
The middleware's per-request user lookup (ghost-session guard) fetched
the full row and discarded it, then built the context stub without a
timezone — so every server-side conversion via the context user fell
back to UTC no matter what the handlers did. The stub now carries
Timezone from that same lookup (fetchLiveUser).
Live-verified: progress page renders 02:03 PM for a 14:03 New York
instant; an analytics session at 02:00 UTC buckets to 10-12 (10pm NY
the previous day) instead of 10-13.
The audit showed only 5 of ~11 time-display surfaces converted to the
user's timezone; everything else rendered pgx-decoded UTC. The server
owns conversion — clients (web/app/KOReader) receive display-ready
strings or machine-readable instants and never convert.
- Progress page: LastUpdated formatted in the requesting user's tz
(was raw UTC).
- Analytics: date-range defaults, day buckets, most-active-day, device
last-sync, and popular-book last-read all computed in the user's tz —
evening sessions no longer land on the next UTC day. Also fixed the
end-date boundary: the query bounds with <= and the end bound was
midnight, so the end date's own sessions (including 'today') never
appeared.
- Queue page: new created_at_display field (admin tz) rendered by the
template; created_at stays RFC3339 for API consumers.
- Admin hash-conflicts SSR bridge: detected date rendered in the
admin's tz.
- OPDS: .UTC() before the Z-suffixed feed timestamps (honesty fix —
correct today only because the session zone happens to be UTC).
- KOReader annotation datetimes deliberately untouched (device owns its
display).
default_timezone existed as a system setting (admin-editable, exposed
via SettingsRegistry.DefaultTimezone) but nothing read it at user
creation — the CreateUser query didn't even carry the column, so every
new user got the DB default UTC regardless. Both creation paths
(public register + admin create) now seed the user's timezone from the
setting, validated with time.LoadLocation and falling back to UTC when
unwired/empty/invalid.
Test-infra fixes surfaced by the run: TestUpdateProfileSelfEdit was
self-editing the SHARED admin (renaming it past the harness cleanup's
preserve-list, leaking a second admin row that broke the last-admin
tests) — it now edits a dedicated user with a unique name; the
last-admin tests get an ensureSingleAdmin precondition helper; the
websocket collection name is run-unique (non-test names persist in the
shared DB and UNIQUE(user_id,name) 409s on rerun).
The /register checklist was dead markup: initPasswordValidation() was
only ever called on the setup page, so the requirement rows never
updated and the Register button stayed disabled forever — web
self-registration was effectively broken. The register Alpine component
now initializes it.
The validator is parameterized (field/submit ids, confirm + username/
email checks optional) so the admin Create User form reuses it: confirm
field, live requirement rows, and the Create button gated on all checks
passing. Requirements mirror the server's passwordcomplex rules
(8+ chars, upper/lower/number/special — the hardcoded defaults; server-
tuned rule values are a future nicety, the server remains the enforcer
either way).
The md:-flipped table/card split still broke at tablet widths and with
the nav sidebar open — a viewport breakpoint cannot see the sidebar
eating content width. Replaced the table entirely with the same fluid
card list admin_library uses (flex + min-w-0/truncate identity block,
fixed action cluster, flex-wrap): it reflows continuously at every
width instead of stepping between two layouts. The duplicate-DOM-id
suffixing from the previous two-rendering approach is gone with it.
- The desktop-on-Android bug: 8 page templates (admin, admin_settings,
admin_users, admin_library, admin_archived, admin_hash_conflicts,
admin_processing_issues, profile) lacked the viewport meta, so mobile
browsers fell back to a ~980px layout viewport and rendered the
desktop layout. Added the same meta the other 21 pages already had —
partials without a head were skipped deliberately.
- Admin Users page: the create-user form no longer dominates the page —
it is hidden behind an explicit '+ Create User' button (Alpine
x-show/x-cloak, the site-wide convention) with a Close control.
- The user list is responsive: table on md+, stacked per-user cards
below. Role/delete controls are shared partials with DOM-id suffixes
so both renderings keep unique swap targets (last-admin protections
preserved in both).
The family-lens hardening: library_visibility was only enforced where
file bytes flowed (download/reader/OPDS) — metadata surfaces leaked
hidden libraries to any authenticated account, and registration was
open by default with no admin-side user creation.
- Series are grouped PER LIBRARY now (same name in two libraries =
two entries: manga adaptations / LN splits no longer merge);
/api/series aggregates only the requester's VISIBLE libraries and
/api/series/books REQUIRES a visible library_id (breaking contract;
web + app clients updated in the same change).
- /api/media-items, item detail, and the reader content endpoints
(chapters/panels/outline/THUMBNAILS — page images leaked unguarded
next to a gated GetPage) all enforce the visible set; resolveLibrary
no longer honors a hand-typed library_id outside the visible set.
- Annotation/note/bookmark mutations (update/delete/tombstone, sync
LWW paths incl. conflict resolution) carry ownership predicates:
another account's bare row id is a 404, not a silent delete.
- Admin user creation: POST /api/auth/users (AdminMiddleware) + a
Create User card in the admin Users UI — accounts stay creatable
with self-service registration OFF, which is now the schema default
(fresh installs; the first-admin bootstrap exception is unchanged).
- Tests: series contract re-pinned (per-library, visibility 403s),
new ownership + admin-create integration suites, OPDS all=1 test
now passes an explicit page size (shared-DB growth made the fresh
books fall off page 1), device-cap helper switched to the admin
endpoint.
The export/ directory holds database extracts (annotations, reading
positions, book metadata) pulled from production for debugging. The
contents are personal data and must never be committed.
scripts/build_torture_epub.py deterministically builds a small EPUB
exercising every CRE-xpointer to CFI conversion hazard observed in the
wild, one chapter per hazard class: multi-text-node paragraphs (verse
spans, inline markup), whitespace-only inline spans and comments,
drop caps and footnote anchors, br-separated poetry with speaker and
line-number spans, nested blockquotes/lists/tables/definition lists,
surrogate pairs and ZWJ emoji, Hebrew niqqud, Arabic RTL, combining
diacritics, invisible characters, HTML entities, CJK ruby, a 400-
paragraph chapter with heavy phrase repetition for search
disambiguation, and edge constructs (span-first paragraphs, empty
spans, adjacent same-tag spans, deep same-tag nesting).
Default output lands in the Ebooks library for real-device testing
via OPDS; the script takes an output path argument for use as a test
fixture.
TestManualCleanRepro replays a real device highlight export through the
same conversion path the sync handler uses, diffs the output against
the CFIs stored at ingest time, and computes a ground-truth oracle by
locating each selection in the flattened document itself. Classifies
every row as healthy / diff-base / backwards for both the converter
and the stored data. Skips automatically when fixture files are not
present on the machine (manual test, like cfi_verify_manual_test.go).
REPRO_VERBOSE=1 prints per-row detail; REPRO_EMIT_SQL=<path> emits
idempotent UPDATE statements that heal database rows written by the
old converter from the preserved raw device positions - used to
backfill dev and validated against production data.
toRangeCfi required stored start and end CFIs to share a base path and
otherwise fell back to a bare point CFI, which resolves to a collapsed
range and paints nothing - or painted from the wrong anchor after end
repair. A legitimate selection crossing inline elements anchors at two
different text nodes by definition, so requiring a shared base path
discarded correct endpoints.
Build the range parent from the endpoints' longest shared step prefix
(down to the bare spine step when needed) and emit each divergent
remainder per the EPUB CFI range grammar, epubcfi(parent,start,end).
Degenerate-end repair (missing end, or end before start) now only
applies when both sides terminate on the SAME node - equal remaining
step count with different step values is a valid two-node range, not a
degenerate pair; document order is verified for those instead.
Verified with foliate's own parser against the real chapter XHTML:
ranges resolve to exactly the selected text for same-node, two-node,
and repaired-degenerate cases.
KOReader highlights converted to wrong or unresolvable web CFIs
whenever a selection crossed text-node boundaries. Three compounding
defects: ParseCREXPointer discarded the text()[N] node index and
applied .offset as a global block offset; the end anchor converted
with an empty context (blind structural landing, zero verification);
and the block-containment check accepted wrong-node landings because
a selection's text always occurs somewhere in its own block. pos0 and
pos1 were converted independently, yielding mismatched start/end base
paths and backwards ranges - highlights stored but never painted, or
painted at the wrong location, in the web reader.
Add ConvertHighlightRange: both endpoints are anchored by locating the
selection text in the flattened document text (whitespace-normalized,
crossing block boundaries), with the device locators demoted to
disambiguation hints. A single document match yields a same-spine,
ordered start/end pair by construction. Falls back to the legacy
per-endpoint path when the text cannot be located, so behavior is
never worse than before.
Also fix the shared flatten stream: zero-width format characters
(ZWSP/ZWNJ/BOM) and whitespace-only inline spans (<span> </span>)
now act as word separators instead of fusing adjacent words, which
prevented text matching in real books.
Verified against 28 real device highlights from production: converter
output now matches the text-search oracle on all of them (previously
8/28 correct, 18 with mismatched base paths, 2 backwards).
The "full 2D" commit (5a0d9e0) added the tip aim but never removed the
y-band candidate filter from pdfAdjustHandleMove — vertical handle
drags still snapped to the boundary rect's own line, so up/down
movement was impossible (the user's original report). With the filter
gone: the end handle drags down a line and the extent extends with
clean text — verified with real touch on the emulator (down one line,
release, PUT applied: 6 per-line rects).
User report after the lifted-handle change: "I cannot go up or down,
just side to side" — a consequence of the line-band pin added with the
lift (the floating finger sits above the text; a band-constrained scan
ignored vertical movement entirely).
The endpoint drag is now full 2D: the finger position (plus the
~15px body-center-to-tip offset) maps to page fractions and the
nearest glyph re-anchors the grabbed end — up/down crosses lines,
side-to-side moves along them, clamped so the extent never inverts.
The boundary rect's own band constraint is gone; sliver rects remain
structurally impossible (rects rebuild from the char-table span as
per-line unions).
Direct-fraction probes verified: (0.4, 0.098) → mid line 1; (0.2,
0.117) → line 2 head; (0.72, 0.117) → line 2 mid — spans cluster into
the right per-line boxes. The dot-listener plumbing (pixels → map →
fractions → scan) verified by instrumented probe on the same load.
Final desktop finger pass pending (emulator input pipeline wedged).
User report: the teardrop tip touched the letters. Raised the handle
boxes 8px (tip now rests at the line-box top, ~4px above the glyph
tops) — matches the Android handle spacing the app leg shipped.
User report: the handles covered the highlighted text. The dots now
hang ABOVE the extent's boundary corners (teardrop tip down at the
first rect's top-left / last rect's top-right), so the highlighted
line stays readable.
Same treatment as the app leg: the nearest-char scan in the endpoint
drag is band-constrained to the boundary rect's own line (the lifted
finger floats ABOVE the text — a whole-table scan re-anchored onto the
line above). Cross-line changes go through re-selection as before.
User report (desktop Brave, fresh private window): the endpoint handle
"jumped several lines down and stuck". The instrumented drag exposed
the chain:
1. THE bug: the endpoint handleMove API takes PAGE FRACTIONS, but the
dot's pointermove listener passed raw e.clientX/clientY PIXELS.
Scanning for the nearest glyph to (296, 170) in a 0–1 fraction space
picks the bottom-right-most glyph — the extent leaped to the page's
end and never moved again (stuck). Engine-independent: failed
identically on Chrome/Brave/Firefox desktops; my synthetic tests
masked it by passing pre-converted fractions. handleMove now maps
host px → fractions internally (one contract).
2. pdfPageMapping mixed coordinate spaces: the iframe-internal canvas
rect with the host iframe rect. Whether that combination is valid
depends on the engine's cross-document getBoundingClientRect
semantics. Replaced with the page WRAPPER element (the iframe's
parent — the same box the fork's rect-annotation overlay fills),
measured in a single host context. Handles now align with the
painted highlights by construction.
3. Char-table hygiene: per-char boxes are LINE-HEIGHT boxes (~one line
gap tall). Dedupe at a full box height ate the line-below char in
the same x-column (silent character loss); clustering at full box
height never split lines (merged 0.45-tall "line" boxes). Dedupe
now tolerates 0.75 of the SHORTER box; clustering breaks at 0.6 of
the char's own height (inter-line gap ≈ 1.0× breaks, baseline
variance ≈ 0.1× stays).
Verified end-to-end (CDP synthetic pointer path): drag to (0.72, 0.117)
→ span extends line 1 → line 2, 2 per-line rects saved, clean
multi-line text. Direct-fraction probes and the pointer path now agree
on the same load.
User report (desktop): the endpoint handles wouldn't move.
renderPdfAdjustHandles rebuilt the handle layer via innerHTML="" on
every preview frame — the first drag move triggered a repaint that
DESTROYED the dot being dragged. A removed element loses pointer
capture, so real mouse/touch drags died silently after one pixel; the
pointerup never reached the detached dot either, so nothing saved and
the session preview reverted — the same mechanism behind the earlier
"resets to the original selection" report. CDP tests masked it because
synthetic events dispatched directly at the detached node still ran its
listeners.
The two dots are now created once per session and every later call only
repositions them (style updates move nothing the listeners live on).
Grip-era code had this reuse pattern; the endpoint rewrite regressed it.
Real-touch verified on the emulator (adb-injected drag): the handle
tracks, the release saves a multi-line extent with clean recomputed
text (3 rects, PUT applied).
The static handler sent no Cache-Control, so browsers fell back to
heuristic caching (10% of age since Last-Modified) and kept serving a
stale reader.js for hours after a deploy — fixed web bugs reported as
"not fixed". no-cache forces revalidation every load; the filesystem
Last-Modified/If-Modified-Since pair keeps unchanged loads cheap.
Found while closing out the PDF adjust v2 session: the user's Chrome
was running a pre-endpoint bundle, which read as "the web has not
been fixed".
Multi-line partial extents ("line 1's tail + the first words of line
2") were inexpressible with one-rect-per-drag grips, and the vertical
drags produced sliver rects whose text recompute swept fragments. The
web's re-selection uses what creation already uses: a selection made
during a rect-edge session is intercepted in checkPDFSelectionFor —
rects rebuild from the selection's per-line client rects, text re-reads
via the deduped glyph boxes, saved immediately (grip release-to-save
parity), and the create popover is suppressed for that gesture.
Vertical grip drags clamp to the rect's OWN line box (band ±quarter
height) — multi-line changes go through re-selection, slivers are gone.
Also verifies clean under the duplicated text-layer span sets (the
fork's pdf.js render never clears .textLayer — open bug): the text
recompute dedupes by glyph overlap.
CDP-verified end-to-end: session → multi-line selection → 6-rect
extent + clean text persisted (PUT 200); create popover correctly
suppressed during the session.
User report: extending a highlight doubled letters into the recomputed
selection text ("aauuthenticcity off ... ruunss iinnto"). The exact-
position dedupe key (toFixed(3)) only collapsed duplicate glyphs whose
re-render copies landed inside the same rounding bucket — pdf.js
re-render copies can sit SUB-PIXEL SHIFTED, so per character the jitter
sometimes straddled a boundary and both copies survived, partially.
Dedupe by overlap instead: a glyph is a duplicate when a kept glyph on
the same line (y0 within half a glyph height) overlaps more than half
the narrower glyph's width. Distinct glyphs — including tight pairs —
never reach that; shifted duplicate copies always do. The snap tables
share the source, so their boundary lists are clean too.
The root cause remains the fork's pdf.js render() appending a fresh
span set without clearing .textLayer (open bug, fork-cycle fix); this
is the defensive layer that makes the adjust feature correct regardless.
Repaired the affected highlight's stored text with the fixed recompute
(verified: the same live duplicated DOM now yields the exact line text
with zero doubling).
fetchBannerAnnotations published the rebuilt rows unconditionally, with
a failed endpoint's list defaulted to empty — so one failing request
(a single handler 500, an auth blip on one call) silently wiped that
annotation kind from the banner while the other kind stayed, and the
empty state could claim "No notes or highlights on this book yet" over
real data. A full outage was already safe (the thrown fetch takes the
catch path, which keeps previous rows and toasts) — the partial case
was the hole, spotted while mirroring the app's fetch-wipe fix
(bookhoard-app 07bbb4c) onto the web.
Both responses must now succeed before the rows are published;
otherwise the previous rows stay and the next banner open retries
(bannerLoaded only sets on success). Verified on the emulator: banner
renders 3 rows, a full outage keeps them with bannerError recorded,
and a post-recovery refetch re-renders from live data.
The left-edge drag kept the rect width instead of anchoring the opposite
edge, so shrinking from the left silently dragged the right edge along
and the recomputed text grew words at the far end (found by the Android
device test of the app leg). Same fix for the top edge / bottom pin.
The end-edge branches were already correct.
The reflowable adjust (objective 9) had no fixed-layout counterpart: the
popover's Adjust button relied on adjustContext, which is only stashed by
the reflowable show-annotation path, so PDF rows showed a button that
silently did nothing. PDF highlights persist as page-fraction rect
payloads ({v:1,page,rects}), so their adjustment is geometry, not range
re-anchoring.
- startAnyAdjust routes the popover button by format; startPdfAdjust
runs a rect-edge session: 4 grips per rect (all edges, one rect per
drag), live preview through the renderer's own addRectAnnotation
(replace-by-key — no fork change), commit via saveHighlightChanges.
- Snap vocabulary mirrors snapBoundary: maximal letter runs are atomic
tokens (start edge takes the word start, end edge the word end), each
punctuation glyph is its own token with both edges grabbable (a period
or line-break hyphen can be taken or shed precisely). Horizontal edges
snap to line top/bottom bands (own line + neighbours).
- selection_text is recomputed from the glyph boxes inside the final
rects (reading order, lines joined with spaces) so the stored text
stays honest with the extent.
- Cancel paths mirror the reflowable session: page turn (renderer
relocate) discards and restores; a committed session skips the restore.
- getContents() from the fixed-layout renderer carries no index — page
iframe docs are mapped through a WeakMap filled by the view's load
events. Pointer mapping goes through the canvas's post-transform rect
(the dpr denominator lesson) via the fork's own convertPoint ratios.
- pdfCharBoxes dedupes by box key: pdf.js re-renders append a second
span set into the same .textLayer without clearing, and undeduped
glyphs doubled the recomputed text.
Device-verified (emulator Chrome + CDP): popover route, grip placement,
word-atomic shrink ("But this" -> "this"), extension to the exact
line-break hyphen boundary (0.8777), far-drag clamp at the last token,
page-turn cancel, and the reflowable adjust still routing to handles.
The guard returned c.JSON(401, ...), whose nil error told echo the
SuccessHandler succeeded — the pipeline then continued into the route
handler, which type-asserted the never-set "user" context value and
panic-recovered as a 500 appended after the 401 body. The status code
happened to survive (first write wins), which is why the pin test
passed, but every ghost request also executed the target handler with
no user — the exact unauthenticated-execution class the guard exists to
prevent.
Return echo.NewHTTPError(401, ...) instead: non-nil aborts the chain
and renders {"message":"user_not_found: ..."}; the pin test now asserts
that shape. Full suite: 219 groups, zero failures.
The JWT middleware validated signature + expiry and fabricated the
request user purely from token claims, so a signature-valid token for a
DELETED user kept full API access until token expiry — observed live on
hardware (a phone session survived the test suite's user-table
recreation and kept serving its true empty sets; open handoff bug 1).
An ensureUserExistsMiddleware already existed but was only wired on the
frontend SSR group (frontend.go) — every /api group used bare
createJWTMiddleware. The existence check now lives INSIDE
createJWTMiddleware's SuccessHandler so every protected group (SSR and
API alike) gets it:
- After claim validation, GetUser by the token's user_id; a missing row
clears the token cookie and answers 401 user_not_found for API calls
(the app's TokenAuthenticator then re-logins with stored credentials)
or redirects to /login?session=invalid for page requests — same
shape-aware split the error handler already uses.
- Cost: one primary-key lookup per authenticated request (~0.2ms local);
instant revocation, no TTL semantics. The redundant SSR-only
middleware is deleted (its redirect behavior is preserved inside the
guard).
The guard exposed two processing-issues tests that were unknowingly
RELYING on ghost semantics: they nested a second setupTestServer whose
bootstrap deleted the first setup's regular user, then asserted 403
from the dead token. De-nested: a createLibraryDirect helper inserts
the library under the live setup's user, no second bootstrap.
New TestDeletedUserTokenRejected pins the contract: token works while
the user exists -> 401 user_not_found after deletion. Full suite: 219
groups, zero failures.
Handler.UpdateUniversalProgress (handlers/progress.go) had no route
registration anywhere (router/progress.go only wires the history route
to the scanner handler) and no callers — a legacy progress-write path
that bypassed the verify/heal pipeline entirely. Deleting it removes
the last bypass-shaped writer that could poison progress rows without
verification; conflict-resolution apply remains the only deliberate
direct write (user-approved winner rows).
Pure deletion: 115 lines, no route, no callers, no behavior change.
The verify/heal pipeline verified only INBOUND context-bearing writes;
stored-row coherence was never enforced. SaveProgress seeds missing
fields from the existing row, so every percentage-only write (KOReader
low-confidence conversion, KOReader no-pointer payloads, Kobo markup
ReadingSync / analytics / sync-from-server, locator-less queue entries)
silently re-paired its NEW percentage with the PREVIOUS writer's stale
CFI+context — the incoherent row shape that fed the client-side
far-anchor restore floor (and, via the read-side anchor handles, could
even hand a first-party client the OLD position's scroll address).
KOReader's converter already declared the right intent ("storing
percentage only — the row's percentage restores approximately"); the
seeding betrayed it. SaveProgressRequest gains ClearLocator, set by the
percentage-only call sites, and the seeding rule honors it: stale
epubcfi/context/character_offset clear (the upsert assigns columns
directly, so NULL sticks) unless the request carries its own. A
request-carried context survives so the verify/heal pass can still mint
a fresh anchor from it; the HTTP merge path (field-absent PUTs
preserve) and all anchor-bearing writes are untouched. The fixed-layout
branch already cleared locators unconditionally — unchanged.
Consumer audit: percentage — the only thing every sync round-trip
exchanges — is untouched on all paths; KOReader pulls on locator-less
rows already support the nil-CFI shape (builder guards
progress.Epubcfi.Valid); first-party restores go percentage-first
instead of floor-fighting a stale anchor.
Tests: TestProgressWeb_LocatorCoherence pins clear-on-declared,
request-carried-context survival, and merge-path preservation. Full
suite with the change: 218 groups, zero failures.
Closes the remaining integration-suite failures (handoff open bug 3),
line by line:
- Fixture family (the upsert equivalence): tags_filter_test
("Science Fiction" found 1 of 3; "Fantasy" 404'd on zero results),
media_item_isbn_test (normalization read back the first case's ISBN
everywhere; edge cases saw a nil ISBN; pagination collapsed 5
creates onto 1 row), search_unified_test (its 3 books were actually
1 row + two no-op creates) — all POSTed a constant file_path into
one library while CreateMediaItem upserts ON CONFLICT
(library_id, file_path). Every fixture now uses unique per-item
paths, the suite's universal convention elsewhere. TestTagsFilter
confirmed same-family: counts came right with no second bug.
- New TestMediaItemCreateDuplicatePath pins the honest-create
contract from da13898: 201 for the first create; 200 + duplicate
flag + the EXISTING row (never a 201 lie), second payload not
overwritten.
- library_browse_test: the admin subtest browsed the container's
/tmp and required it non-empty — env-coupled (empty /tmp → "was
[]"). Now browses a self-created t.TempDir with two subdirs and a
decoy file, asserting ONLY directories are listed.
- series_integration_test: three stale assertions re-anchored to the
shipped contracts, each with a ground-truth comment: /dashboard/
sections key identity on the collection NAME (BuildSections ID =
CollectionName; the web/app reorder + hidden prefs persist those
strings) — the kebab query_type is the storage key and is not
exposed; /api/series/books is name-ONLY (the app client and the
web SSR page both send name only; library_id is ignored, never
validated); /api/series is optional-scope BY DESIGN (GetDistinct-
Series: narg IS NULL OR library_id = narg aggregates across
libraries when omitted). The former 400-on-missing-library_id
expectations asserted validation that never shipped.
- gofmt rider on three untouched-in-this-arc test files
(dashboard_integration, search, test_helpers).
Full suite with -parallel=1: 217 groups pass, zero failures.
The create endpoint shared the scanner's upsert (ON CONFLICT
(library_id, file_path) DO UPDATE SET updated_at = NOW() RETURNING *) —
so a POST naming an already-registered file returned 201 CREATED with
the OLD row while silently discarding everything the caller submitted.
Nothing depended on that lie: the app only GETs/PUTs media items, the
web only searches/purges/deletes, KOReader never calls it — the
endpoint's create path had zero production callers.
Split the API path from the scanner's idempotent upsert, which is kept
verbatim in CreateMediaItem (re-scan dedup remains its safety net):
- CreateMediaItemIfNew: same insert with ON CONFLICT DO NOTHING — the
loser of a create race writes nothing and returns no row.
- GetMediaItemByLibraryAndPath: fetch the existing row for the
duplicate response.
- Handler branch: on pgx.ErrNoRows → 200 OK with
{"duplicate": true, "message": ..., "item": <existing row>} instead
of a 201 pretending it created something. Real creates still answer
201 with the item as before.
Bulk-import shape (the reason for 200+flag over 409): a re-run after a
mid-import failure is harmless and countable — each response tells the
caller exactly which of the two things happened.
Verified by the full integration suite: 217 groups pass, zero failures,
including a new pin of the duplicate contract (same file_path twice in
one library → 201 then 200/duplicate:true, second payload not
overwritten).
Born dead in e7ce0cb (2026-04-03, backend reader infrastructure):
the "Phase 2.5" plan assumed MOBI/AZW3/DOCX/RTF would need server-side
parsing, so ParseEbook was written as the endpoint that would do it —
but it only ever returned 501 "not yet implemented", and no route in
router/reader.go ever pointed at it. No caller in Go, templates, or
web JS/TS ever existed either.
The eventual architecture never needed it: the web reader hands the
raw file to foliate-js and parses client-side (MOBI renders via the
fork's mobi.js — verified live; DOCX via mammoth; RTF/PDB shipped).
By now it was also stale — it 501'd DOCX/RTF as "not yet implemented"
while both work, and offered an AZW3 path that was dropped entirely.
Pure deletion: no route registration, no references, no tests, no
behavior change. go build ./... clean.
TestProgressWeb_KoboIntegration 500s'd on every markup with a
last-read-place bookmark: the panic-recovered stack was a nil pointer
dereference in LibraryService.ResolveMediaPath (s.db deref), reached
from KoboHandler.convertKoboCFIToStandard during kepub CFI conversion.
Two-layer bug:
1. Test harness (root cause): the router.Config built in
test_helpers_test.go omitted LibraryService, so router/sync.go's
SetLibraryService(cfg.LibraryService) stored a nil
*services.LibraryService in the interface field. Production wiring
(main.go:246) was always correct.
2. Latent trap (hardening): `h.libraryService != nil` at the
convertKoboCFIToStandard call site can never catch a typed nil —
an interface holding a nil *LibraryService is non-nil. ResolveMediaPath
now degrades with a "library service not configured" error instead
of panicking on a nil receiver; callers already handle the error
by skipping CFI conversion.
Harness-only fix, zero production behavior change: the endpoint now
returns a clean 200 with the position saved wherever kepub conversion
is unavailable instead of a panic-recovered 500.
Verified: TestProgressWeb_KoboIntegration passes all three subtests
(401 unauthenticated, ReadingSync→last-read-place percentage
preservation, last-read-place without prior ReadingSync) and on
immediate rerun — previously it failed every run.
The whole-body "should not display Manga Type / Scan Info" assertions
could never pass: every book-detail page for an admin carries the
edit modal, whose form labels ("Manga Type", "Scan Information") and
option texts ("Yes (Right to Left)") share vocabulary with the
metadata display block. This was pre-existing rot — the test never
ran to completion before today because an earlier panic killed the
process first.
Replaced with value-level checks that only the display block could
produce ("yes and right to left", "B&W"): absent for plain ebooks and
unknown-manga-type items, matching the templ guards in
book_detail.templ. Emoji-icon asserts dropped from the RTL/Story-Arc
positive steps — the redesigned comic badge renders text chips
without icons.
Symptom: repeated full-suite runs poisoned the dev database with
leaked rows (duplicate Prefs Scope libraries, colliding device names,
saved-filter name conflicts on rerun) because tests created rows but
never deleted them, and stale-binary runs left rows the new cleanup
could not match.
- test_helpers: library creation now registers t.Cleanup DELETE (the
leak root cause — createTestLibraryWithFolder never cleaned up).
Suite-wide cleanup passes for libraries/collections/devices/
saved-filters run both at setup (pre-pass against legacy residue)
and in t.Cleanup (no per-run residue). Preserve the admin's real
libraries; delete only convention-named test rows.
- setup seeds canonical system collections every run keyed on
query_type (idempotent — heals renames/stale tombstones), moved out
of the preserved-admin branch so it runs for every suite start.
- dashboard_integration: scope-testing library renamed into the
test-row convention; ordering assertions on the restored
"Want to Read" collection preserved.
- filters_test: filter names moved into the convention (rerun-safe —
name-uniqueness 409 no longer trips second runs); five .(string)
type assertions panic-proofed with require.True guards (a panic here
used to kill the whole test process mid-run, which is why a stack of
"never ran" failures surfaced at once later).
- device_test: device names moved into the convention (ILIKE %test%
hit them; duplicate-device-name conflicts gone).
The old model hard-deleted system collections and re-created them at
the tail, resurrecting stale dedups and breaking ordering. Redesign
(user-directed B-final):
- schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER —
the whole schema is re-applied on every boot via executeSchema).
- queries.sql: every collection list/read query filters
deleted_at IS NULL; new GetSystemCollectionByQueryType :one
(unfiltered, sees tombstones), TombstoneSystemCollection :exec,
ClearCollectionItems :exec, RestoreSystemCollectionRow :one
(in-place metadata reset + un-tombstone; collections have no
updated_at column — ResetUpdatedAt was a 500 caught by tests).
sqlc regenerated.
- collection_service.DeleteCollection: system collections are
tombstoned (preserving position/ownership); user collections keep
the hard delete. No ErrNoRows to the client (delete is idempotent).
- dashboard_service.RestoreSystemCollection: resolve by query_type;
found → in-place restore (ClearCollectionItems + metadata reset,
un-tombstoned); missing → provision fresh. "Want to Read"/etc.
stay at their home positions forever.
- handlers/collections.CreateCollection: name-taken is 409, not 500.
- handlers/auth.CreateDefaultCollectionsForUser: idempotent by
(user_id, query_type) — update-in-place when a row for that query
type exists (heals renames/tombstones), insert when missing; this
also fixes the duplicate-system-collections regression at source.
The drawer was the only top-bar tool without a keyboard shortcut, so the
shortcuts & gestures help modal had to describe it as '📝 button'. Bind
plain 'a' to toggleBookmarks(), following the same pattern as the
existing t/s/b letter shortcuts.
- reader.ts: new 'a' branch in handleKeydown, guarded against
Ctrl/Cmd/Alt so Select All (a common reader action when selecting
text to highlight/copy) is never intercepted; typing guard already
protects note textareas, search box, and selects. Works in both the
host page and book-content iframes since both keydown listeners route
through handleKeydown.
- reader.templ: help modal row now shows <kbd>a</kbd> like the other
letter shortcuts; 📝 button tooltip gains '(a)' to match its
siblings (b), (/), (s), (?).
- reader_templ.go: regenerated.
John O'Keefe