7537063cb71efedbb7b6b902a7100cf1d4a55a8a
230
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
7537063cb7 |
feat(filters+nav): author navigation end-to-end — arrays on every media response, deep-linkable bookshelf filters, clickable authors
Completes the A2c acceptance scenario: click an author on a book detail page -> land on the bookshelf filtered to everything that author wrote (including their share of co-authored books). - Media detail + unscoped list resolved maps expose subtitle, explicit, authors, publishers (the library-scoped raw branch and the search endpoint already did — raw rows follow the SELECT). - SearchMediaItemsUnified SELECT carries the new columns; the search endpoint serializes the row directly, so clients get them there too. - book_detail.templ renders each author as a link to /bookshelf?author_filter=<name>. - Bookshelf honors filter params in the URL on load (author_filter, series_filter, genre_filter, publisher_filter, tags_filter, narrator_filter, language_filter): pre-fills the form, applies it to the first grid load. Comma-joined values are multi-select via the server-side string_to_array split. Filtered views are shareable. Pins widened: search rows (+7 keys), resolved keys (+4); changelog entries in docs/contract_pins.md. Android follow-up plan written to ~/.opencode/plan/bookhoard-app-v2-compat.md. |
||
|
John O'Keefe
|
1dbb0ce406 |
feat(collections): playlist auto-creation, position ordering, hide_finished, Up Next, dashboard eligibility
Phase 2b + 3 of the metadata overhaul (plan §2.4, §3.1-3.7). Auto-playlist collections (§2.4): the scanner turns obviously-playlist sidecar series (explicit '(Reading Order)'/chronological/publication/ in-order/timeline markers only — precision over recall) into curated static collections: upsert by view_settings.imported_from_series then name (rename-safe), position set ONLY on insert (clean integers; 1.5/1a stay unpositioned rather than guessing), membership add-if-missing, never removed or re-ordered by rescans, provenance in description + view_settings, show_on_dashboard true (dashboard-eligible by design), 'scanner.auto_playlist_collections' setting (default on, Settings Registry + schema seed). Ordering (§3.1-3.2): collection listings order position ASC NULLS LAST then added_at ASC (was added_at DESC — walked playlists backwards); PUT /api/collections/:id/items/order applies full-order rewrites (owner-only, service-gated). hide_finished + Up Next (§3.3-3.4): GetCollectionItemsForDashboard gains viewer_user_id + hide_finished (LEFT JOIN reading_progress, COALESCE percentage < 0.999) and viewer_percentage rides the row for card graying; GetCollectionUpNext serves the collection's reading-queue head with the family-lens visible-library array enforced. Dashboard (§3.5): user collections are dashboard-ELIGIBLE BY DEFAULT — the show_on_dashboard gate (never writable for user rows since the column shipped) is gone from GetUserCollectionsForDashboard; hiding stays dashboard-prefs (hidden_collections) territory; hide_finished filtered-everything now emits a Complete section instead of silently vanishing; sections JSON carries is_playlist/complete. Series page ordering hardening (§3.6): GetSeriesBooks tiebreaks by title ASC — NULL/tied numbers previously returned heap (scan) order, which is what the user saw. Frontend custom-section-builder note: the builder has sent show_on_dashboard: true since its creation; the API never bound it. User-row visibility is now intentionally prefs-governed, so the field stays unbound by design (documented in code). Pins widened (docs/contract_pins.md): collection detail + up_next; dashboard sections + is_playlist/complete; new reorder endpoint recorded. Verified: TestPlaylistCollectionsEndToEnd (new, full e2e: sidecar series split, collection auto-creation with position 7, rescan idempotency, owner-only reorder, finished-filter + completion marker + Up Next drain) plus the full affected integration sweep green. |
||
|
John O'Keefe
|
646477afad |
feat(schema): subtitle/explicit columns, authors+publishers arrays, float series numbers
Phase 1 of the series/collections/metadata overhaul
(~/.opencode/plan/series-collections-metadata-overhaul.md).
Schema (in place, idempotent):
- media_items.series_number INTEGER -> DOUBLE PRECISION: real-world
ordinals are decimals (0.5, 1.5 novellas); INTEGER silently truncated
them (calibre 2.5 pinned as 2 in tests).
- + subtitle TEXT, explicit BOOLEAN (sidecar ingest; ABS explicit /
Audible isAdult).
- + authors/authors_search/publishers/publishers_search TEXT[] NOT NULL
(narrators A2a precedent). Arrays are write truth; author/publisher
varchars become trigger-regenerated display strings
(sync_author_publisher_display) so every varchar consumer (trgm
search, facets, API, OPDS/Kobo) keeps working unchanged.
- Row-level NULL normalization for the new arrays in
set_library_type_name (A2 lesson: Go nil -> explicit NULL -> 23502).
- Backfills ARRAY[author]/ARRAY[publisher] — deliberately no
comma-splitting ('Tolkien, J.R.R.' would corrupt; scanner kept only
authors[0] historically).
- collection_items.position INTEGER + (collection_id, position) index
for playlist ordering.
Go: sqlc regen; MediaMetadata/handler params gain Subtitle/Explicit/
Authors/Publishers (legacy singles still accepted, arrays win);
SeriesNumber int32 -> float64 end to end; metadata_overrides adapted
with legacy override keys ('author'/'publisher') preserved so existing
user overrides survive; bulk author edit writes the array.
Pins widened (docs/contract_pins.md): raw sqlc row key set + the six
new columns; changelog entry added.
Verified: unit suite green; integration sweep (bulk/smart collections/
scanner/stats/sync/queue/calibre/pins/audiobooks) green against the
shared dev DB after in-place migration. Pre-existing unrelated red:
TestOwnershipAndAdminTestSuite/TestAdminCreateUser timezone inheritance
fails identically at the pre-change commit (live shared state).
|
||
|
John O'Keefe
|
220d97cfe0 |
test(series): pin the SSR pagination contract under tied timestamps
New integration test mirroring a prod bulk import: a dedicated test library with 30 series whose items share ONE identical created_at (single-transaction NOW()), the exact data shape that made prod pagination look broken. Pins, via the real SSR /series route: - page 1 renders exactly 24 deterministic series (01-24) with chip "Page 1 of 2", no Previous, and a Next href carrying library_id and page (templ escapes & as & inside attributes) - page 2 renders the remaining 6, disjoint from page 1, chip "Page 2 of 2", no Next, Previous href correct - missing page param defaults to page 1; page=0/-3/two clamp to page 1 Guards both the SQL tiebreaker and the /series handler contract; would fail on any return of the stomp or an unstable sort. |
||
|
John O'Keefe
|
6216bd9bcf |
feat(audiobooks): multi-file books (folder-of-tracks + zip) + exhaustive metadata ingestion
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
|
||
|
John O'Keefe
|
24228e7147 |
test(scan): cancellation endpoint semantics and mid-run safety
- endpoint semantics: non-admin cancel is 403, unknown job is 404 - mid-scan safety: start a scan of 1200 unparseable .epub files and cancel it, then pin the invariant that no media item is marked missing or archived. The canary item's file is lexically LAST in the folder, so an abort that (incorrectly) ran the archive pass would treat it as unvisited and trip the assertion. Assertions hold under every interleaving - whether the cancel lands mid-walk or the scan already finished - so the test cannot flap against timing. - re-cancelling a terminal job is a 409 The test library is created through the shared createTestLibraryWithFolder helper, which registers API-delete cleanup so runs leave no residue in the dev database. Worker-level unit tests live in internal/services/worker_cancel_test.go (previous commit). |
||
|
John O'Keefe
|
9e831085ef |
fix(ui): derive Library Type dropdowns from library_types — audiobooks was missing
The audiobook library type shipped backend-complete (LibraryTypeAudiobooks constant, allowed extensions, MIME map, schema seed row, GET /api/libraries/types) but neither create-library form offered it: the admin Create Library modal and the setup wizard's add-library step both hardcoded their <option> lists to the original three types (ebooks/comics/manga), so an audiobook library could not be created from the UI. Both dropdowns now render from the same DB-backed library_types table the create API resolves types against — a future type appears without template edits (the exact drift the rendering-ownership rule warns about): - templates/utils.go: OrderedLibraryTypes pins the historical dropdown order (ebooks, comics, manga, audiobooks) and appends unknown future types by name; LibraryTypeDisplayName renders labels with a capitalize-the-name fallback for types added without a label - templates/admin_library.templ: new LibraryTypeOptions fragment used by the modal's select; AdminLibrary takes the types list - templates/setup.templ: Setup takes the types list, same fragment - internal/router/helpers.go: libraryTypeOptionsFor fetches via LibraryService.GetLibraryTypes, falling back to the built-in type names on DB failure so the forms stay usable (CreateLibrary still resolves the name against the DB, so the fallback is never authoritative) - internal/router/frontend.go, internal/router/setup.go: pass the list into the two pages Tests: - templates/utils_test.go: pure unit tests for ordering + labels - cmd/server/tests/admin_library_page_test.go: SSR regression — /admin/library must offer all four library types, audiobooks exactly once (options come from the DB loop, not a hardcode) |
||
|
John O'Keefe
|
45b8542e25 |
fix(api): honor the request's origin as the writer label — app rows stop wearing "web"
Release / build-and-push (push) Successful in 3m36s
Every web-API write endpoint hardcoded Source:"web", so the Android app's own pushes landed with last_sync_source/last_modified_source "web" — the app was indistinguishable from the web SPA in progress conflicts, device_sync_data, and annotation bookkeeping. The app CAN say who it is (bookmark create already carried origin → origin_source); the server just ignored it for labeling. All six write sites (progress PUT, bookmark/note/highlight create+PUT) now parse an ADDITIVE optional "origin" key, normalized through one gate: "android"/"ios" pass through, anything absent/unknown stays "web" (the label keys conflict claims and device_sync_data — free-form client strings would pollute them). Precedence on progress: a paired X-Device-Token still outranks origin. Bookmark origin_source now rides the same gate. The web SPA sends no origin — its rows are unchanged; KOReader/Kobo handlers keep their own labels. Wire discipline: additive request keys only (rule 2), response shapes untouched, pins changelog widened in-commit, locked by TestOriginWriterAttribution (android lands, default stays web, unknown normalizes). Full battery: only the two known pre-existing failures (TestOPDSLibraryFolders rot, TestWebSocketPingPong — both reproduce with the change stashed). |
||
|
John O'Keefe
|
010c0cb287 |
test(contracts): whitelist the unified row's two new keys on search rows
The H4 row-completion commit made abridged/audio_metadata emit on the search wire (the endpoint serializes the row directly) — exactly what the pin exists to surface. Whitelisted per the additive rule with the changelog entry. |
||
|
John O'Keefe
|
188ae076ab |
refactor(utils): one PgTextString for the four pgtype.Text unwrappers
handlers textToString, router textToString, router getText, and utils normText were four copies of the same NULL-to-empty unwrap (99 call sites). One exported utils.PgTextString replaces them all; the private normText (comparison normalizer) stays but is now the odd one out. Harness fix found on the way: setupTestServer built a full wiring literal for route registration and then RETURNED a second, sparser literal that never copied AnnotationService — setup.AnnotationService was always nil on the struct every test receives. Nobody noticed until cross_library_move_test became the first caller (it previously used the now-retired plain CreateMediaHighlight query); it now exercises the real SaveHighlight path through the wired service. |
||
|
John O'Keefe
|
7d698ea0d8 |
fix(annotations): bookmark and note PUTs land through the sync service (LWW by id)
H3 census bugs #2 and #3: both web PUTs used plain queries that bypassed the sync-aware layer entirely — no LWW, no source bookkeeping, and locator fields the client couldn't even send. The highlight PUT was the only one already routed (via HighlightID); the other two now mirror it: - SaveBookmarkRequest gains BookmarkID, SaveNoteRequest gains NoteID: id-targeted resolution reads the exact row, verifies ownership, LWWs under its stored dedup key, and resurrects tombstones when the edit is newer (same doctrine and comment structure as the highlight path). - Fields the PUT omits are coalesced from the stored row before the full-column sync update — the legacy web PUT sends only title/notes/position, and zeroing cfi/percentage/page on every edit would have been data loss. Verified both directions by TestH3_BookmarkPUTPreservesLocators (omitted survives, explicit lands). - The bookmark PUT's wire DTO gains the create endpoint's locator keys as optional fields (documented additive request keys in contract_pins.md); the note PUT keeps its content+position wire. UpdateMediaBookmark/UpdateMediaNote plain queries are now uncalled (the generated queries stay until the H4 query sweep). |
||
|
John O'Keefe
|
c579179fee |
refactor(annotations): AnnotationService is a required constructor dependency
H1's doctrine applied to the annotation service, which had been left on setters: NewMediaHandler/NewKoboHandler/NewKOReaderHandler/ NewSyncQueueProcessor(WithConfig) now take it as a parameter; the four SetAnnotationService setters are deleted; wiring updated (main.go, router/sync.go, test harness). With the service unskippable, the 18 nil-guards and their plain-query fallbacks die: - media.go: the create fallbacks for notes/highlights/bookmarks (the bookmark one was the H3 census's Bug #1 — it dropped percentage/chapter_reference the sync path carries), the highlight PUT fallback, and the three delete-path plain deletes (tombstone is now the only delete semantics). - kobo.go: both markup-ingest annotation loops (the bookmark fallback wrote the text into media_notes — a category error the fallback never should have had). - koreader.go: all three push-stack guards. - queue.go: the three "annotation service not available" errors in syncNote/syncHighlight/syncBookmark. Behavior note: the deleted highlight create/PUT fallbacks were the only paths passing note_id to the plain query — the sync service has no NoteID field, so the web's highlight→note link was already being dropped on the live path; this commit makes that pre-existing limitation explicit rather than papering it over (surfacing note_id on SaveHighlightRequest is left for a behavior commit that needs it). Pins (now 20 incl. the 3 annotation CRUD suites) + H1 tests + both unit suites green. |
||
|
John O'Keefe
|
ff029f1ad5 |
fix(annotations): note LWW re-save 500'd — UserID missing from the sync update params
Pin-first batch for H3: freeze web annotation CRUD response shapes (create 201 / update 200 / delete 204; full sqlc row models) before the stack consolidation touches anything. The new pins immediately caught a live server bug the 28 unit tests could never see (annotations_test.go is pure-unit: keys and compare logic, no DB): applyNoteLWW built UpdateMediaNoteForSyncParams without UserID, so the query's WHERE id=$1 AND user_id=$13 compared against zero — every dedup-hit note re-save (same content+position) returned 500 "update note: no rows in result set" and the update never landed. The bookmark twin passes UserID and works; the note stack is the clone that dropped the field. Fix: pass UserID in the params (one line). Pins go green and become the regression net; changelog records the new surfaces. |
||
|
John O'Keefe
|
14cdbe70eb |
test(contracts): CFI/context winner must land intact through conflict resolution
Book-side twin of the audio-survival test, closing H1's verification list. Scenario: web push carries percentage + epubcfi + context_text; a koreader-style percentage-only push from a second source within the window detects a conflict; the user resolves with the CFI-carrying side. The funneled resolution must land the winner's ENTIRE locator pair — percentage 0.5, epubcfi, context_text together (the old direct writer NULLed context_text and character_offset, breaking the coherent pair the far-anchor work established) — and must not spawn a duplicate pending conflict. |
||
|
John O'Keefe
|
6808c1babb |
fix(sync): the offline queue flush never processed anything — fix the enumerator + the dropped payload
Found while verifying H1's "app offline-play -> flush -> position lands"
with the emulator path: the flush has been dead code in two independent
ways, invisible because the first break made the second unreachable.
1. ListPendingSyncQueueItems matched `device_id = $1` with a NULL $1
(the processor passes DeviceID{Valid:false} to enumerate all devices;
uuid = NULL matches no rows). The ticker therefore processed nothing,
ever. The WHERE now treats NULL as "every device" (the same
narg::uuid IS NULL idiom the visibility queries use; EXPLAIN'd — seq
scan on a bounded table, idx_sync_queue_status_priority_created
fits). The processor also dropped its fake-pagination loop, which
re-fetched the same first batch forever once the backlog hit the
batch size (no OFFSET in the query).
2. dbItemToSyncQueueItem dropped SyncData — even with a working
enumerator, every flushed item arrived with an empty payload and
failed JSON unmarshal. Dead code harboring silent breakage, exactly
the plan's thesis.
Callers updated for the renamed RowLimit param (queue handler, offline
detector). TestH1_SyncQueueFlush_LandsAndPreservesAudio drives the real
flow: web PUT seeds audio 900s, a pending checkpoint-style queue row
applies percentage 0.8 through SaveProgress, audio survives, queue row
completes.
Also fixes dashboard_test.go's stale BuildSections signature (the
package's test binary has been unbuildable) — user-approved: tests
designed against old shapes are not worth keeping.
|
||
|
John O'Keefe
|
d16ca4acbe |
fix(tests): scope the sync_queue cleanup wipe to the test devices
The cleanup in setupSyncTestDB ran DELETE FROM sync_queue WHERE true — a table-wide wipe that discards REAL devices' pending offline entries on every suite run. Same class as the 2026-10-14 reading_history incident the standing rules ban. The test's own queue items cascade from its 'Test %'-named device, so the wipe is redundant for cleanup — scoped it to those devices (defense in depth ahead of the cascade). Flagged during H1 verification: the line executed during an in- container suite run this session; any device with pending offline queue entries at that moment should re-push on its next sync. |
||
|
John O'Keefe
|
8fc74c908a |
fix(conflicts): resolution writes funnel through SaveProgress — audio position survives
Batch H1 step 3b — THE data fix. Conflict resolution was the one live
erasure writer: applyProgressResolution and its verbatim twin
applyResolution both wrote the winner via a raw UpdateUniversalProgress
whose param list omits audio_offset_seconds/context_text/viewports, and
the SQL's DO UPDATE SET ... = EXCLUDED... NULLed exactly those. The web
UI resolves through POST /api/conflicts/:id/resolve and
/api/conflicts/bulk-resolve (main.js resolveConflict/bulkResolve), and
the reader's mark-as-read fires a resolution first when the reader SSR
stamps data-conflict-id/-winner — so every resolution today erased the
audiobook position and reader context.
Now: ProgressService is a required ConflictHandler dependency; BOTH
call sites (single resolve + bulk resolve) share one
applyProgressResolution that funnels through SaveProgress (the verbatim
twin is deleted). Deliberate choices, per the plan:
- SkipConflictDetection on the resolution write: both flows apply
BEFORE the conflict row is marked resolved, so the funnel's LWW
detector would otherwise file a duplicate pending conflict
(HasRecentConflictResolution is false at apply time).
- Source "conflict_resolution" — the row's last-write attribution; the
winner stays durably recorded in resolution_data.winner.
- Broadcast: true — live devices jump to the winning position.
- The winner snapshot maps percentage/epubcfi/context_text/chapter/
character/page/total_pages; a winner without a CFI clears the locator
pair (ClearLocator) instead of re-pairing with the loser's stale
anchor. Everything the snapshot doesn't carry SURVIVES via the LWW
seed: audio_offset_seconds above all. chapter_progress keeps its
stored value (was overwritten with the book percentage), an absent
percentage survives (was written as 0%).
- Accepted: the resolution captures one reading_history session
(attributed to conflict_resolution; the 2h idle ceiling applies).
Verification: TestH1_ConflictResolutionPreservesAudio drives the real
flow — web push with audio 300s, diverging push from a second source
detects a conflict, resolve with the second source as winner, GET shows
percentage 0.3 AND audio_offset_seconds 300 (previously NULLed), and no
duplicate pending conflict remains. Full contract pin suite green.
The host-run TestConflict*/TestProgress* failures are environmental
(container-only /app/uploads path + db hostname; fail identically at
baseline
|
||
|
John O'Keefe
|
e83c00f869 |
refactor(progress): ProgressService required at construction; delete the 9 unreachable erase-fallbacks
Batch H1 step 3a. The ceiling census (exhaustive UpdateUniversalProgress
grep) found 9 nil-guarded `if progressSvc != nil { funnel } else { raw
erase-write }` fallbacks — the survey's "four" under-counted kobo.go,
which has FOUR sites (Markup ReadingSync, Markup last-read-place,
AnalyticsGettests, SyncFromServer), plus one each in media.go, queue.go
and koreader.go. All unreachable: the service is created unconditionally
at main.go:127 and wired into every handler and the queue processor; the
test harness wires them too.
The erasure pattern is now unrepresentable:
- NewMediaHandler / NewKOReaderHandler / NewKoboHandler / both
NewSyncQueueProcessor flavors take the ProgressService as a REQUIRED
constructor arg; the SetProgressService setters are gone.
- Every fallback body (raw UpdateUniversalProgress with explicit-NULL
pgtypes for viewport/scroll/panel/reading_mode/zoom and no
audio_offset_seconds/context_text) is deleted; each write funnels
through SaveProgress with the LWW seed, so fields the writer doesn't
know survive.
- Removed the now-dead koreader pg-type helpers (textPtrToPgText,
intPtrToPgInt4, int64PtrToPgInt8) and their self-tests.
Behavior deltas (stored-row content, no wire shape changes —
KOReader/Kobo response pins stayed green):
- KOReader fallback used to write explicit NULL viewports and drop the
context_text the koplugin sends; the funneled mainline already maps
book.ContextText.
- Queue flushes no longer overwrite chapter_progress with the book
percentage (bogus data; preserved via the seed).
- A web PUT without percentage no longer writes 0% via the deleted
fallback; percentage seeds from the stored row.
Contract suite: KOReader×4 + Kobo×3 pins green.
|
||
|
John O'Keefe
|
a2ea34232c |
test(contracts): pin wire shapes before the canonical-types migrations
Batch H0 of the hardening plan: freeze the JSON key SETS of every
load-bearing surface so H1-H7 can only change them additively.
- cmd/server/tests/contract_pins_test.go: 14 tests / ~30 subtests pinning
progress GET+PUT, the KOReader plugin surfaces (progress push/pull,
library, bookmarks), the Kobo adapter shapes (initialization, markup,
sync-from-server), collections detail all four branches + preview,
series JSON, search rows, dashboard sections, media list BOTH branches
(raw sqlc rows vs resolved 38-key map), and SSR smoke for
bookshelf/collections/series/tags.
- docs/contract_pins.md: the pinned-shape inventory + the rule that a
batch widening a shape must widen the pin and this doc in one commit.
Suite is green on the unmodified tree at
|
||
|
John O'Keefe
|
ce366b1604 |
feat(stats): Library tab — per-library inventory stats + audio-aware Reading tab
The /stats page gains a second tab (Reading | Library) implementing the
Audiobookshelf-style library dashboard (HANDOFF item 9): media_items
INVENTORY aggregates — items, hours/pages, distinct authors, size,
distinct narrators, Top Genres / Top Authors bars, Longest / Largest
items — for one library or the All-libraries aggregate.
- GET /api/stats/library-usage?library_id=<uuid|omit>; family lens
enforced: a library_id outside the requester's visible set is a 403,
never a silent pass (AGENTS.md standing rule;
|
||
|
John O'Keefe
|
a222ef5c27 |
fix(media-items): normalize NOT NULL narrators on every write path
A2 made narrators/narrators_search NOT NULL DEFAULT '{}' (
|
||
|
John O'Keefe
|
199d82c613 |
fix(tests): sync-integration cleanup wiped the shared reading_history table
sync_integration_test's cleanup ran DELETE FROM reading_history WHERE true — harmless for years while the table sat empty, and a silent stats-data destroyer the moment session capture started feeding it: every suite run erased the shared dev accounts' captured reading (the user's manual flip-through test data included). The cleanup is now scoped: the sync tests' sessions belong to their test-sync users, and deleting those users cascades their rows. Any future table-wide cleanup DELETE is off the table (AGENTS.md test rules updated). |
||
|
John O'Keefe
|
0914c935d9 |
feat(dashboard): collapsible This Week widget, library-scoped + switch-surviving; title into the subbar
Three dashboard complaints, one build: - The widget collapses to a single summary row (This Week · minutes · pages · sessions · books · streak · Full stats) on every screen size and expands to the full grid via a details/summary toggle — the carousel sections start immediately below it again. - The 'Your Library' heading moved into the library switcher subbar (LibrarySwitcher gains an optional title param; collections/series keep the plain 'Library' label). The standalone heading block is gone. - Root-caused the JS-switch vanishing: renderDashboardCollections replaces ALL of #collections-container's innerHTML, so anything server-rendered inside it (title, widget) died on the first library switch. The widget now re-fetches ITSELF: a /dashboard/week-widget fragment (same resolveLibrary family-lens rules as the page) is pulled by htmx whenever the switcher dispatches week-refresh — carrying the current #library-select value — so the widget both survives switches and RE-SCOPES to the switched library. Initial SSR renders the same scope the sections use. ThisWeek (and the streak calendar) gained an optional library filter. Tests: TestReadingStatsWeekLibraryScope — per-library totals, scoped streaks, invisible-library isolation, malformed id 400. Full suite PASS. |
||
|
John O'Keefe
|
5a2c1585ae |
refactor(stats): 'Web / unsynced' bucket label → 'Web'; analytics tests isolated from the live reader
The bucket name read as confusing — 'unsynced' implied data loss when the push is always captured; it only lacks a device credential. The bucket is just 'Web' now (service fallback, stats.ts fallback, panel copy, tests). Also: analytics_test exact-zero assertions (EmptyHistory, NoReadingHistory) ran against testuser, which is a LIVE reader — real sessions accrued on it mid-suite and both flapped. They now run under the fresh regular user, like the stats tests; the 'testuser is a live reader' rule is recorded in AGENTS.md. |
||
|
John O'Keefe
|
7a604c2fec |
feat(stats): per-device reading breakdown + device attribution on progress syncs
- Attribution: registered clients may now send their device token in
X-Device-Token on PUT /api/media-items/:id/progress (the user JWT
stays primary). The token must map to a device OWNED BY the caller —
foreign/invalid/absent tokens soft-fail to plain web attribution so a
stats decoration can never break a sync. Attributed pushes capture
device_id (FK-safe), device_metadata.source = the registered device
name ('Pixel 9', 'Kindle Paperwhite'), and last_sync surfaces show
that device. Side effect worth having: app-vs-web pushes used to be
LWW-invisible (both sources read 'web'); they now surface as
legitimate cross-source progress conflicts when they diverge.
- /stats gains a By Device panel alongside By Library (grid 3-wide):
reading minutes + sessions + books per registered device, plus one
Web / unsynced bucket for device-less pushes so the panel always
reconciles with the headline totals. Range-aware, family-lens
scoped; renderBars shared between panels in stats.ts.
- The device identity IS the registered platform: devices rows already
carry the name/type the client registered (app sends Build.MODEL +
'mobile' at /api/devices/register, KOReader/Kobo likewise) — no new
identity plumbing on the server.
Tests: capture attributed to the device (row device_id + source
metadata + last_sync_source), foreign/nonexistent tokens soft-fail,
device buckets reconcile (80 attributed + 75 unsynced = 155 total).
Full suite PASS.
|
||
|
John O'Keefe
|
6457fbaf72 |
feat(stats): reading statistics — session capture wired + /stats page + widget + per-book card
The design parked in the handoff (multi-tier stats off reading_history; bookorbit-style in data, not looks) is in. The real prerequisite work was data: reading_history existed but NOTHING wrote to it — the whole stats feature was sitting on an empty table. - CAPTURE (no client changes, no repo-boundary crossings): every successful progress write through SaveProgress (KOReader + app + web + Kobo + offline queue — the one chokepoint) now records one reading_history session row: span = the interval since the PREVIOUS write instant, pages = positive page delta when both sides know the page index, marker rows for first contact. Spans over 2h are kept as zero-second markers (overnight suspend→resume, offline flushes must not inflate totals) — the honest 'sync cadence, not a stopwatch' caveat stays. device_id only when a REAL device row matches (the web/app progress path historically stuffs the USER id into DeviceID — probe instead of trusting it; FK violation would silently drop every capture). Source preserved in device_metadata. - API: /api/stats/summary?range=7|30|90|year (totals, zero-filled daily arrays, streaks, most-active day/weekday, top books, library breakdown — one call), /api/stats/this-week (widget: rolling 7d vs previous + streak), /api/stats/books/:id (lifetime + recent sessions). All bucketed in the user's timezone; finished ≡ any push ≥0.995; visible-library lens enforced at source. Legacy /api/analytics/* kept untouched. Range windows anchor end-of-day to the START of today (blasting the clock time into tomorrow resolved 'today' a calendar day late — found by the streak test). - /stats page (sidebar Stats between Progress and Devices, previously /analytics was an orphan): range selector, five stat cards, two Chart.js v4 charts (daily minutes bar, pages/day line — chunked via dynamic import so main.js stays untouched), per-library breakdown, most-read list, empty-state explainer. /analytics now 301s to /stats. The orphaned analytics.templ + analytics.ts removed (their containers and the page JSON never matched; the page was astub). - Dashboard 'This Week' widget (server-rendered, suppressed on service failure) and the book detail 'Your Reading Stats' card (lifetime totals of the viewer's own sessions) complete the tiers: per-user default, per-library breakdown inline, per-book on the page, per-system/admin aggregate parked as before. Tests: statistics_test.go (capture semantics at the chokepoint, tz/day buckets, zero-fill, streak math, finished detection, range validation, auth gate, widget, per-book lifetime). Full suite PASS. |
||
|
John O'Keefe
|
0a7fbd6f48 |
feat(collections): smart (dynamic) collections — rules evaluated live at render
Type is DERIVED, not stored: a saved ruleset makes a collection SMART,
no rules = STATIC (stored collection_items links, unchanged behavior).
No schema change — collections.auto_assign_rules JSONB already carried
the config; user's decision 2026-10-14 (handoff: live-at-render, not
refresh-on-scan; create-time Static/Smart toggle; v1 NO hybrid).
- Rule gains an enabled flag (absent = enabled, legacy rules unaffected).
- Rule CRUD: POST/PUT(DELETE) /api/collections/:id/rules[/:ruleId] — the
survivor dead effort of the rules-builder UI finally has real
endpoints (rules were never editable after creation; nothing ever
re-evaluated them). First rule converts a user collection to smart;
removing the last reverts it to static. UpdateCollectionRulesJSON
edits ONLY the rules doc so rule CRUD cannot clobber metadata.
- Live evaluation: candidates from GetVisibleLiveMediaForRules —
visible-library set (family lens enforced at source) + archived IS
NULL AND missing_scan_count = 0 — any enabled rule matches, title
order. Zero refresh hooks by design: always current by construction.
- Membership mutations (add/remove/bulk-add) are 400 on smart — v1 no
hybrid; collection_items.excluded stays the reserved future hook.
- Rules builder page (/collections/:id/rules) first time ever routable;
templ whole-attribute interpolation ("{ expr }" inside a quoted attr
value is literal text, as theme-{ user.Theme } silently proved).
- UI: create modal Static/⚡Smart cards (smart create redirects into the
rules builder), ⚡ badge + real per-card counts on the grid (smart =
live matches, static = stored-link count), smart detail renders
matched books with no hand-pick actions + an Edit Rules button.
- API: GET /:id gains is_smart + live books; GET /: keeps the Android
app's default exclusion, include_auto=true surfaces smart rows with
live counts; TestRules previews against the SAME candidates the live
render uses (visible + live, was raw first-1000).
- FIXED pre-existing regression from
|
||
|
John O'Keefe
|
17274bb868 |
fix(nas): refuse deletes while library storage is unreachable; real file removal; stat cache
Three changes from the network-drive audit: 1. Delete-ordering fix. Deleting a media item cascades its annotations, so the delete must never proceed while the item's library storage is unreachable — the DB row died, the file survived as an orphan, and the next scan re-imported it bare (annotations gone forever). resolveMediaStorage stats the library's folders: reachable → delete proceeds and the file is removed by ABSOLUTE path (the old code passed a library-relative path to os.Remove, which never resolved — file deletion has been a silent no-op); every folder unreachable → 409 with 'storage unreachable', row and annotations intact. 2. Latency: resolveLibrary stats every library folder on every SSR render; folder reachability is now cached with a 30s TTL. 3. README: Network Storage section documenting the outage hysteresis (mark → archive → auto-restore), the 90-day retention purge (ARCHIVE_RETENTION_DAYS, 0 = never), refused deletes while storage is down, and watch mode not firing on network mounts. Integration tests: storage-down bulk delete refuses with the row intact; real-file delete removes the file from disk. |
||
|
John O'Keefe
|
740213f0a4 |
fix(auth): new users inherit the system-wide default timezone
default_timezone existed as a system setting (admin-editable, exposed via SettingsRegistry.DefaultTimezone) but nothing read it at user creation — the CreateUser query didn't even carry the column, so every new user got the DB default UTC regardless. Both creation paths (public register + admin create) now seed the user's timezone from the setting, validated with time.LoadLocation and falling back to UTC when unwired/empty/invalid. Test-infra fixes surfaced by the run: TestUpdateProfileSelfEdit was self-editing the SHARED admin (renaming it past the harness cleanup's preserve-list, leaking a second admin row that broke the last-admin tests) — it now edits a dedicated user with a unique name; the last-admin tests get an ensureSingleAdmin precondition helper; the websocket collection name is run-unique (non-test names persist in the shared DB and UNIQUE(user_id,name) 409s on rerun). |
||
|
John O'Keefe
|
431fec05ad |
feat(security): library visibility enforced at the source; per-library series; annotation ownership; admin user creation
The family-lens hardening: library_visibility was only enforced where file bytes flowed (download/reader/OPDS) — metadata surfaces leaked hidden libraries to any authenticated account, and registration was open by default with no admin-side user creation. - Series are grouped PER LIBRARY now (same name in two libraries = two entries: manga adaptations / LN splits no longer merge); /api/series aggregates only the requester's VISIBLE libraries and /api/series/books REQUIRES a visible library_id (breaking contract; web + app clients updated in the same change). - /api/media-items, item detail, and the reader content endpoints (chapters/panels/outline/THUMBNAILS — page images leaked unguarded next to a gated GetPage) all enforce the visible set; resolveLibrary no longer honors a hand-typed library_id outside the visible set. - Annotation/note/bookmark mutations (update/delete/tombstone, sync LWW paths incl. conflict resolution) carry ownership predicates: another account's bare row id is a 404, not a silent delete. - Admin user creation: POST /api/auth/users (AdminMiddleware) + a Create User card in the admin Users UI — accounts stay creatable with self-service registration OFF, which is now the schema default (fresh installs; the first-admin bootstrap exception is unchanged). - Tests: series contract re-pinned (per-library, visibility 403s), new ownership + admin-create integration suites, OPDS all=1 test now passes an explicit page size (shared-DB growth made the fresh books fall off page 1), device-cap helper switched to the admin endpoint. |
||
|
John O'Keefe
|
679e0f9116 |
fix(server): ghost 401 must abort the chain — c.JSON's nil error let the request continue
Release / build-and-push (push) Successful in 3m21s
The guard returned c.JSON(401, ...), whose nil error told echo the
SuccessHandler succeeded — the pipeline then continued into the route
handler, which type-asserted the never-set "user" context value and
panic-recovered as a 500 appended after the 401 body. The status code
happened to survive (first write wins), which is why the pin test
passed, but every ghost request also executed the target handler with
no user — the exact unauthenticated-execution class the guard exists to
prevent.
Return echo.NewHTTPError(401, ...) instead: non-nil aborts the chain
and renders {"message":"user_not_found: ..."}; the pin test now asserts
that shape. Full suite: 219 groups, zero failures.
|
||
|
John O'Keefe
|
9cd6356892 |
feat(server): ghost-session guard — JWTs for deleted users stop authenticating
The JWT middleware validated signature + expiry and fabricated the request user purely from token claims, so a signature-valid token for a DELETED user kept full API access until token expiry — observed live on hardware (a phone session survived the test suite's user-table recreation and kept serving its true empty sets; open handoff bug 1). An ensureUserExistsMiddleware already existed but was only wired on the frontend SSR group (frontend.go) — every /api group used bare createJWTMiddleware. The existence check now lives INSIDE createJWTMiddleware's SuccessHandler so every protected group (SSR and API alike) gets it: - After claim validation, GetUser by the token's user_id; a missing row clears the token cookie and answers 401 user_not_found for API calls (the app's TokenAuthenticator then re-logins with stored credentials) or redirects to /login?session=invalid for page requests — same shape-aware split the error handler already uses. - Cost: one primary-key lookup per authenticated request (~0.2ms local); instant revocation, no TTL semantics. The redundant SSR-only middleware is deleted (its redirect behavior is preserved inside the guard). The guard exposed two processing-issues tests that were unknowingly RELYING on ghost semantics: they nested a second setupTestServer whose bootstrap deleted the first setup's regular user, then asserted 403 from the dead token. De-nested: a createLibraryDirect helper inserts the library under the live setup's user, no second bootstrap. New TestDeletedUserTokenRejected pins the contract: token works while the user exists -> 401 user_not_found after deletion. Full suite: 219 groups, zero failures. |
||
|
John O'Keefe
|
69516ae165 |
feat(server): progress locator coherence — percentage-only writes clear stale anchors
The verify/heal pipeline verified only INBOUND context-bearing writes;
stored-row coherence was never enforced. SaveProgress seeds missing
fields from the existing row, so every percentage-only write (KOReader
low-confidence conversion, KOReader no-pointer payloads, Kobo markup
ReadingSync / analytics / sync-from-server, locator-less queue entries)
silently re-paired its NEW percentage with the PREVIOUS writer's stale
CFI+context — the incoherent row shape that fed the client-side
far-anchor restore floor (and, via the read-side anchor handles, could
even hand a first-party client the OLD position's scroll address).
KOReader's converter already declared the right intent ("storing
percentage only — the row's percentage restores approximately"); the
seeding betrayed it. SaveProgressRequest gains ClearLocator, set by the
percentage-only call sites, and the seeding rule honors it: stale
epubcfi/context/character_offset clear (the upsert assigns columns
directly, so NULL sticks) unless the request carries its own. A
request-carried context survives so the verify/heal pass can still mint
a fresh anchor from it; the HTTP merge path (field-absent PUTs
preserve) and all anchor-bearing writes are untouched. The fixed-layout
branch already cleared locators unconditionally — unchanged.
Consumer audit: percentage — the only thing every sync round-trip
exchanges — is untouched on all paths; KOReader pulls on locator-less
rows already support the nil-CFI shape (builder guards
progress.Epubcfi.Valid); first-party restores go percentage-first
instead of floor-fighting a stale anchor.
Tests: TestProgressWeb_LocatorCoherence pins clear-on-declared,
request-carried-context survival, and merge-path preservation. Full
suite with the change: 218 groups, zero failures.
|
||
|
John O'Keefe
|
f24a2d6c22 |
test(server): fixture collisions fixed, stale contract asserts re-anchored, gofmt — suite now fully green
Closes the remaining integration-suite failures (handoff open bug 3),
line by line:
- Fixture family (the upsert equivalence): tags_filter_test
("Science Fiction" found 1 of 3; "Fantasy" 404'd on zero results),
media_item_isbn_test (normalization read back the first case's ISBN
everywhere; edge cases saw a nil ISBN; pagination collapsed 5
creates onto 1 row), search_unified_test (its 3 books were actually
1 row + two no-op creates) — all POSTed a constant file_path into
one library while CreateMediaItem upserts ON CONFLICT
(library_id, file_path). Every fixture now uses unique per-item
paths, the suite's universal convention elsewhere. TestTagsFilter
confirmed same-family: counts came right with no second bug.
- New TestMediaItemCreateDuplicatePath pins the honest-create
contract from
|
||
|
John O'Keefe
|
208fc2c4d6 |
fix(server): kobo markup panic — wire LibraryService into the test harness, guard typed-nil resolver
TestProgressWeb_KoboIntegration 500s'd on every markup with a last-read-place bookmark: the panic-recovered stack was a nil pointer dereference in LibraryService.ResolveMediaPath (s.db deref), reached from KoboHandler.convertKoboCFIToStandard during kepub CFI conversion. Two-layer bug: 1. Test harness (root cause): the router.Config built in test_helpers_test.go omitted LibraryService, so router/sync.go's SetLibraryService(cfg.LibraryService) stored a nil *services.LibraryService in the interface field. Production wiring (main.go:246) was always correct. 2. Latent trap (hardening): `h.libraryService != nil` at the convertKoboCFIToStandard call site can never catch a typed nil — an interface holding a nil *LibraryService is non-nil. ResolveMediaPath now degrades with a "library service not configured" error instead of panicking on a nil receiver; callers already handle the error by skipping CFI conversion. Harness-only fix, zero production behavior change: the endpoint now returns a clean 200 with the position saved wherever kepub conversion is unavailable instead of a panic-recovered 500. Verified: TestProgressWeb_KoboIntegration passes all three subtests (401 unauthenticated, ReadingSync→last-read-place percentage preservation, last-read-place without prior ReadingSync) and on immediate rerun — previously it failed every run. |
||
|
John O'Keefe
|
18969a0f01 |
test(server): comic negative-case assertions target display values, not admin-modal labels
The whole-body "should not display Manga Type / Scan Info" assertions
could never pass: every book-detail page for an admin carries the
edit modal, whose form labels ("Manga Type", "Scan Information") and
option texts ("Yes (Right to Left)") share vocabulary with the
metadata display block. This was pre-existing rot — the test never
ran to completion before today because an earlier panic killed the
process first.
Replaced with value-level checks that only the display block could
produce ("yes and right to left", "B&W"): absent for plain ebooks and
unknown-manga-type items, matching the templ guards in
book_detail.templ. Emoji-icon asserts dropped from the RTL/Story-Arc
positive steps — the redesigned comic badge renders text chips
without icons.
|
||
|
John O'Keefe
|
843f35bbfc |
test(server): runs leave no residue — auto-cleanup for every resource the suite creates
Symptom: repeated full-suite runs poisoned the dev database with leaked rows (duplicate Prefs Scope libraries, colliding device names, saved-filter name conflicts on rerun) because tests created rows but never deleted them, and stale-binary runs left rows the new cleanup could not match. - test_helpers: library creation now registers t.Cleanup DELETE (the leak root cause — createTestLibraryWithFolder never cleaned up). Suite-wide cleanup passes for libraries/collections/devices/ saved-filters run both at setup (pre-pass against legacy residue) and in t.Cleanup (no per-run residue). Preserve the admin's real libraries; delete only convention-named test rows. - setup seeds canonical system collections every run keyed on query_type (idempotent — heals renames/stale tombstones), moved out of the preserved-admin branch so it runs for every suite start. - dashboard_integration: scope-testing library renamed into the test-row convention; ordering assertions on the restored "Want to Read" collection preserved. - filters_test: filter names moved into the convention (rerun-safe — name-uniqueness 409 no longer trips second runs); five .(string) type assertions panic-proofed with require.True guards (a panic here used to kill the whole test process mid-run, which is why a stack of "never ran" failures surfaced at once later). - device_test: device names moved into the convention (ILIKE %test% hit them; duplicate-device-name conflicts gone). |
||
|
John O'Keefe
|
e7ccd7b222 |
test(dashboard): align test seed naming with the canonical Title-case set + fix stale expectations
The test helper createDefaultCollectionsForUser carried its own kebab-case copy of the default system collections — drifted from the canonical Title-case list the registration seed and RestoreSystemCollection use. Every suite run against the shared dev DB re-created a kebab set beside the canonical one (the duplicate sections on the customize screens); the dev DB is cleaned and the helper now uses the canonical list. Two stale expectations corrected while re-running the suite: TestGetSections_EndToEndFlow asserted the kebab ids (now asserts the Title-case names — section ids ARE collection names), and TestGetSections_MissingLibraryID expected a 400 the handler never implemented — absent library_id is the All Libraries view by design (now asserts 200 + aggregated sections). |
||
|
John O'Keefe
|
25d44452c0 |
feat(dashboard): global (All Libraries) preferences scope
The dashboard's All Libraries view could render sections across every library but could not persist its own preferences: the PUT required a UUID library_id, and the prefs GET-without-library_id matched an arbitrary library's row through a loose NULL OR-clause. Storage: a dedicated user_dashboard_preferences_global table (one row per user, user_id PK → users CASCADE) instead of a NULL/sentinel library_id — chosen over both because (a) the column's FK to libraries rules out a sentinel UUID without dropping the constraint or minting a fake library row, and (b) intent becomes structural: only the explicit global path writes that table, so a buggy or outside-the-API write can never masquerade as global. System and user collections are user-scoped, so the order/hidden arrays are library-independent and valid in this scope. API: PUT /api/dashboard/preferences binds library_id as *string — JSON null/absent = global, empty string = 400 (a lost value must stay loud, not silently clobber the global scope), valid UUID = per-library (unchanged). GET without library_id now returns the global row instead of an arbitrary one; the per-library query's dead NULL OR-arm is removed. GET /sections keeps requiring library_id for per-library views and aggregates across libraries when absent (unchanged). Tests: global save/update-dedupe/read round-trip, empty-string 400, global-vs-per-library independence. Two suite failures are the handoff's documented pre-existing rot (duplicate system collections in the dev DB; a sections test expecting a 400 the handler never implemented) — reproduced and confirmed unrelated to this change. Live-verified on the dev stack: null save 200 x2 with a single row updated, empty string 400, global read carries the second save. |
||
|
John O'Keefe
|
1205526422 |
feat(notes): raise annotation text limit from 10,000 to 100,000 chars
The 10k validator cap rejected legitimate long-form notes — a scholarly reading note with quoted passages and footnotes lands around 10.1k chars and failed at save with the raw validator error. Raise the cap to 100,000 on all six annotation text fields in the media handler request structs: - CreateMediaNoteRequest / UpdateMediaNoteRequest Content - CreateMediaHighlightRequest / UpdateMediaHighlightRequest NoteText - CreateMediaBookmarkRequest / UpdateMediaBookmarkRequest Notes No other layer changes: the media_notes / media_highlights columns are unbounded TEXT, and the KOReader + websocket sync paths never had a length cap, so the REST API now matches the rest of the system instead of being the strictest gate. New integration test pins the boundary: 50k and 100k-char notes return 201, 100,001 chars returns 400. |
||
|
John O'Keefe
|
7b0b2793a6 |
test(infra): self-healing admin password; cross-library move/merge suite
setupTestServer reuses the shared dev admin (testuser@tests.bookhoard.internal) instead of re-inserting it, but never reset its password — once any test mutated the admin's password, every later test in the run failed to log in with 401s until the database was manually wiped. The seeding step now resets the password hash to the known test constant on reuse, so full integration runs are repeatable against an existing database. Adds cross_library_move_test.go, seven integration tests covering the duplicate-content features end to end: - TestCrossLibraryMovePreservesHistory: a book moved between two same-type libraries keeps its row ID — progress and annotations survive, no duplicate, no archived ghost, library_type_name stays truthful. - TestCrossLibraryCopyStaysIndependent: deliberate copies in two libraries stay independent rows with isolated progress. - TestCrossLibraryMoveRejectedForTypeMismatch: a reflowable EPUB is not repointed between manga libraries; a library_type_mismatch processing issue is recorded instead. - TestListHiddenMediaItemsMatchesActiveTwin: archived rows expose their active same-SHA twin (and rows without one report no match). - TestMergeArchivedItemIntoActiveTwin: merging moves progress and annotations onto the active copy and removes the archived row. - TestMergeArchivedItemRejections: non-admin 403, active source, hash mismatch, missing target_id, self-merge, and archived target all fail closed. - TestLibraryTypeNameTriggerOnLibraryChange: the UPDATE trigger refreshes library_type_name when a row changes libraries. Tests create their own temp-dir library folders (host runs don't have /app/uploads), use unique device identifiers (device_identifier is UNIQUE and leftover rows broke reruns), and clean up via an explicit defer that runs while the pool is still open — a t.Cleanup registered for the same purpose silently no-ops because it executes after setup.Close has closed the pool. |
||
|
John O'Keefe
|
5da97b2e5d |
feat(opds): library folder navigation and library-scoped search
The device catalog flattened every visible library into one list, so
duplicate copies of the same book each got an entry and search had no
library context. The root feed is now a standard OPDS 1.1 navigation feed
that mirrors the web UI's library model:
- Root /catalog: an "All Books" entry first (the previous flat cross-
library behavior, also still served at ?all=1 for clients that want
the single flat list), followed by one folder entry per visible
library with live book counts from GetVisibleLibraryMediaCounts.
- New GET /library/:libraryId/catalog: acquisition feed scoped to one
visible library (403 when the device owner cannot see it), paginated,
with an up-link back to the root.
- Scoped search: each library feed's rel="search" OpenSearch template
pins &library_id=<id>, so opening search from inside a library folder
searches only that library — clients substitute only {searchTerms},
so no client-side changes are required. Root search stays global.
- Global search entries now carry the owning library as a category (and
as a fallback summary when the book has no description), so duplicate
copies are distinguishable in unscoped result lists.
The acquisition entry builder is extracted into addAcquisitionEntries and
shared by the all-books and per-library feeds. bookhoard.koplugin needs
no changes: it only registers the root URL, and KOReader's stock OPDS
client renders navigation feeds natively.
Tests: TestOPDSLibraryFolders covers the nav-feed shape, flat ?all=1
mode, scoped catalog isolation, scoped/global search behavior, and 403s
for libraries outside the device's visibility. Library names avoid the
word "test" on purpose — setupDeviceTest re-runs setupTestServer's
setup-time cleanup, which deletes every library whose name contains it.
|
||
|
John O'Keefe
|
1002cbca93 |
fix(schema): refresh library_type_name on library change; make trigger block re-runnable
schema.sql is re-executed in full on every application startup, so every statement in it must be idempotent. Two related problems introduced with the cross-library move feature, plus their fix: Problem 1: media_items.library_type_name was only populated by a BEFORE INSERT trigger. When the scanner repoints an existing row to a different library (cross-library move detection), the denormalized library_type_name went stale, mislabeling the item's type for validation and display. Fix: add a second trigger, set_library_type_name_on_library_change, that fires BEFORE UPDATE OF library_id and re-runs the same population function. Problem 2 (outage): the new trigger EXECUTE FUNCTIONs set_library_type_name(), but the pre-existing idempotency dance drops that function on every startup before recreating it. On the second and later boots, DROP FUNCTION failed with SQLSTATE 2BP01 (function still depended on by the trigger created by the previous boot), aborting the whole schema transaction and crash-looping the container. Fix: drop BOTH triggers before dropping the function, and create both after it. The ordering now survives any number of restarts on any database state. Adds TestSchemaInitializationIsIdempotent, which replays the production database.Initialize twice against the same database so this class of "second startup" regression fails in CI instead of in production. Note the test uses a dedicated pgxpool: Initialize holds the advisory lock on one connection while executing the schema on another, which deadlocks against the shared test pool's MaxConns=1. |
||
|
John O'Keefe
|
4faa217822 |
test: repair integration harness — wire SettingsRegistry, reuse preserved dev admin
Two pre-existing harness breaks found while running the device tests: - RegisterRoutes dereferences cfg.Settings (auth rate limit) but setupTestServer never set it — every integration test nil-panicked at route registration. Wire database.NewSettingsRegistry(queries) the same way main.go does. - The cleanup deliberately PRESERVES testuser@tests.bookhoard.internal (shared dev admin), but setup then blindly re-INSERTed that user, so every run after the first failed on users_email_key. Reuse the user when it exists; default collections are created only for a NEW user (the preserved admin already has its set). Also documented the offline runner used on slow links: build the test binary on the host (CGO_ENABLED=0 go test -c) and execute it inside the bookhoard-tests image against the compose network — no in-container module downloads, no image rebuild. |
||
|
John O'Keefe
|
1cd8557b58 |
fix(devices): re-approving a known device rotates its token instead of 500
App reinstalls that preserve data (Android Studio installDebug over an existing install) re-register with the same device_identifier, but the devices row from the previous install still exists — device_identifier is UNIQUE, so ApproveDevice's blind INSERT failed with a unique violation and returned 500 'failed to create device' (reproduced via curl: second approve with the same identifier = instant 500; the ~98s in the original report was app-side retry/polling, not server wait). ApproveDevice is now idempotent: look the device up by identifier first; a row owned by the approving user gets its auth token rotated via UpdateDeviceAuthToken (row id unchanged, so synced highlights/ bookmarks/progress anchored to it stay valid; fresh install = fresh credentials, old token invalidated); a row owned by another user gets 409; unknown identifiers INSERT as before, with the 23505 race falling through to the rotate path. DB failures are logged (they were silent). Also guard the in-memory pendingRegistrations map with a mutex — register/approve/reject/status/list all touch it from HTTP goroutines, and a racing write is a Go runtime fatal, not an error. The approver's credential publication and the status poller's approved-branch snapshot now run under the lock so the token can never be read half-written. Regression test: TestApproveDeviceReapprovalRotatesToken — register → approve → re-register same identifier → approve (must be 200) → token rotated, exactly one devices row, row carries the new token. |
||
 john-okeefe
|
59d5de3607 |
fix(scanner): eliminate fsnotify watcher leak and harden worker against panics
The bookhoard container crashed with 'panic: Failed to create file
watcher: too many open files' (media_scanner.go) after running for a few
hours, preceded by floods of 'no space left on device' from watcher.Add.
Root cause: every scan job called NewMediaScanner(), which eagerly
created an fsnotify watcher. SetFolders() then walked the entire library
tree and registered one inotify watch per directory (~3,000+ across the
libraries), and ScanFolders() registered them again during its walk. The
worker never called scanner.Close() on these ephemeral per-job scanners,
and the worker loop had no recover(), so:
1. Leaked watchers accumulated until the kernel inotify watch cap was
hit (ENOSPC -> 'no space left on device'), then
2. the process fd limit (ulimit -n 1024) was exhausted, causing
fsnotify.NewWatcher() to fail with EMFILE, and
3. NewMediaScanner panicked on that error, taking down the whole
process (exit code 2). With no restart policy the container stayed
down.
The scan jobs run frequently (scan_poll_interval), so the leak built up
within hours. Note this was NOT a disk-space issue; df showed plenty free.
Fix:
- media_scanner.go: NewMediaScanner no longer creates a watcher eagerly
(s.watcher starts nil), which removes the panic site entirely -- there
is nothing to fail at construction. The watcher is created lazily only
when needed.
- media_scanner.go: SetFolders gains a [?1049h[22;0;0t[1;24r(B[m[4l[?7h[?25l[H[2JEvery 2.0s: bool[1;37Hgaruda-ser8: Fri 31 Jul 2026 10:45:41 AM EDT[2;66Hin 0.002s (127)[2;80H
[3dsh: line 1: bool: command not found
[4d[24;1H[?12l[?25h[?1049l[23;0;0t
[?1l> parameter. It creates
and populates a watcher (returning an error instead of panicking) only
when watch=true; otherwise it skips all watcher.Add calls. ScanFolders
guards its watcher.Add with a nil check, and the WatchChanges event
loop exits cleanly when there is no watcher (polling still runs).
- worker.go: the worker() loop now wraps each job in defer/recover() so a
panicking job is recorded as failed and can never kill the process.
- worker.go: the three ephemeral scan handlers (processScanJob,
processSetFoldersJob, processDirectoryScanJob) now defer scanner.Close()
and call SetFolders(..., false), so scan jobs allocate zero watchers and
zero inotify watches. Any pre-existing leak is also bounded by Close().
- handlers/scanner.go: the long-lived watch-mode scanners (StartScanner
and StartWatchModeForLibrary) pass watch=true since they actually read
watcher.Events for live change detection.
- calibre_integration_test.go: updated to the new SetFolders signature
(watch=false, matching one-off scan usage).
Auto-add is fully preserved: new files are still detected by the periodic
poller (startBackupScan), which is independent of fsnotify and unaffected
by these changes. The watch-mode event loop remains as bonus responsiveness
when inotify is available; through Docker bind mounts where inotify is
unreliable, polling is what catches new books.
|
||
|
john-okeefe
|
75b33fdae6 |
feat(sync): wire annotation sync into all device and web handlers
Complete the annotation sync pipeline across all ingest and serve paths.
Previously, annotations sent inline with KOReader progress pushes were
silently discarded, and no annotations were ever served back to devices.
INGEST (device → server):
KOReader (koreader.go):
- Add processBookAnnotations helper that processes inline highlights,
notes, and bookmarks from every progress push (immediate + checkpoint)
- Highlights get CRE→CFI position conversion before SaveHighlight
- KOReader 'notes' (text + notes) stored as highlights with NoteText
to ensure correct round-trip classification
- Bookmarks routed through SaveBookmark with device sync data
- Called from both updateProgressForBook and handleCheckpointSync
Kobo (kobo.go):
- Markup handler: annotations and bookmarks route through
AnnotationService (SaveHighlight/SaveBookmark)
- Bookmark handler: same routing with device sync data
- SyncFromServer handler: same routing
- All handlers fall back to direct DB calls when annotationSvc == nil
Web reader (media.go):
- CreateMediaHighlight → SaveHighlight (Source="web", ModifiedAt=now)
- CreateMediaNote → SaveNote (Source="web")
- DeleteMediaHighlight → TombstoneHighlightByID
- DeleteMediaNote → TombstoneNoteByID (was hard delete, now tombstone)
- All fall back to old behavior when annotationSvc == nil
SERVE (server → device):
KOReader GetMetadata (koreader.go):
- Query and serve bookmarks from media_bookmarks table (was missing)
- Serve deleted_highlights and deleted_bookmarks arrays containing
device_sync_data + dedup_key for client-side deletion
- Highlights/notes already served with reverse CFI conversion
Kobo Markup handler (kobo.go):
- Track processed books during sync
- Query tombstones per book, extract bookmark_id from device_sync_data
- Return DeletedAnnotations array in KoboSyncStatus response
Conflict resolution (conflicts.go):
- Enable annotation conflict types in ResolveConflict handler
- Add applyAnnotationResolution dispatching to:
applyHighlightResolution / applyBookmarkResolution / applyNoteResolution
- Each looks up by dedup_key and applies winner's fields
- Allow manual override of auto_resolved conflicts
(changed check from != "unresolved" to == "user_resolved")
Infrastructure:
- AnnotationService field + SetAnnotationService in router Config
- Inject AnnotationService into KOReader, Kobo, Media handlers
- Start tombstone purger goroutine in main.go (24h interval)
- Test helpers: construct AnnotationService in test setup
|
||
|
john-okeefe
|
73fc609d7b |
fix(tests): protect dev admin from test cleanup, use isolated test names
Tests were deleting the development admin user, causing ON DELETE SET NULL to cascade and set created_by_admin_id to NULL on all libraries. - test_helpers: skip deletion of testuser@tests.bookhoard.internal - sync_integration_test: use test-sync% prefix for isolated test data |
||
|
john-okeefe
|
a57694b738 |
fix(tests): URL-encode series name in special characters test
The TestGetSeries_SpecialCharactersInName test was failing with a 400 status because the series name 'Series: Book & Other (Vol. 1)' was interpolated directly into the URL without encoding. The ampersand was parsed as a query parameter delimiter, corrupting the request. Use url.QueryEscape() to properly encode the name parameter. |
||
|
john-okeefe
|
cce7ad4907 |
test(series): add unit and integration tests for series feature
Unit tests: - series_service_test.go: test continueSeriesRowToMediaItems conversion with valid fields, null fields, and comprehensive field mapping - series_test.go: test handler initialization and textToString helper Integration tests (series_integration_test.go): - GET /api/series: requires library_id, rejects invalid UUID, returns empty array for empty library, pagination params, limit clamped to 100, response structure validation, special characters in names - GET /api/series/books: requires library_id and name, handles nonexistent series, unauthorized access - Restore Continue Series system collection - Dashboard sections include all 5 collections (including continue-series) Update test helpers: - Add SeriesHandler to setupTestServer router config - Add 5th Continue Series collection to createDefaultCollectionsForUser - Update dashboard integration test for 5 collections |