VisibleLibrarySet becomes a package-level function taking the queries
(db-only services like statistics reach it without new wiring); the
LibraryService method stays as the ergonomic entry point. Statistics'
ids-shaper and the reader pages' three hand-rolled membership loops
(empty-rows fallback then linear scans) now delegate; the gates keep
their 403-on-invisible semantics (>null set membership, never silent
pass).
Census addendum from this pass: two OPDS sites call
GetUserVisibleLibraries directly but consume FULL ROWS (per-library
folder enumeration) — a third legitimate pattern alongside set/ids and
resolveLibrary's active-selection; documented, untouched. The library
LIST endpoint and its SSR data helper are row consumers too, by design.
One core for "which libraries can this user see": family-lens default-
allow stays where it always was (the GetUserVisibleLibraries SQL);
the core adds the shared set+ids shaping that four endpoints each
hand-rolled. Media's visibleLibrarySet is now a thin identity adapter
(JWT user object -> resolved id) and the download-permission gate uses
the set instead of a manual membership loop.
Semantic diff (H4.5 step 1, in the plan): zero family-lens divergence
found across all six sites — the rule lives in one query; the variation
was shape-level (identity adapter, set-vs-ids, UUID idiom).
SELECT mi.*, ci.excluded forced every caller to filter tombstone
exclusions in Go over a 69-field row, then a 45-field hand converter
(getCollectionItemsRowToMediaItems) copied it field-by-field into
MediaItems for BuildSections. The query now filters
(ci.excluded IS NOT TRUE) itself and returns sqlc.embed(mi) — callers
consume .MediaItems directly, the Go filter loops die in both
collection-detail ladders (JSON + SSR), and the converter is deleted.
Pins, H1/H3 tests, unit suites green.
checkRulesAgainstBook/evaluateRule and the service's EvaluateRules
existed in two parallel universes: one over database.MediaItems (what
every query actually returns) and one over ListMediaItemsRow (what two
hand-written 45-field mediaItemsToListMediaItemsRow converters
produced — verbatim twins in the handler and the dashboard service,
with a URL-resolution side effect baked into a "converter"). The rule
fields (genre/series/author/language/publisher/copyright_year/tags)
all exist on MediaItems, so the evaluators now take the real row and
the converters, their three call-site loops, and the dead
ApplyAutoAssignRules entry point (zero callers) are gone.
The rules surface is wire-covered by the collections rules scenarios;
handlers+services suites and the full pin/H1/H3 battery green.
The /stats page gains a second tab (Reading | Library) implementing the
Audiobookshelf-style library dashboard (HANDOFF item 9): media_items
INVENTORY aggregates — items, hours/pages, distinct authors, size,
distinct narrators, Top Genres / Top Authors bars, Longest / Largest
items — for one library or the All-libraries aggregate.
- GET /api/stats/library-usage?library_id=<uuid|omit>; family lens
enforced: a library_id outside the requester's visible set is a 403,
never a silent pass (AGENTS.md standing rule; 431fec0 history)
- six sqlc queries with the visibility pattern + healthy-item filter
(archived_at IS NULL AND missing_scan_count = 0); hours_mode lets the
client swap the Hours/Pages card and the Longest ranking per scope
- Reading tab unit audit (HANDOFF item 3, settled 2026-10-16): minutes
are the merged currency everywhere honest; only the PAGES unit splits
— Pages card goes mode-aware (Pages · text / Listening Time in an
audio-only pool) and the charts row gains Listening per Day (hrs)
when audio minutes exist in the range; summary daily buckets gain
audio_minutes split by the item's library type
- tab state in the URL (?tab=library&library_id=…); selector fed by the
existing /api/libraries/visibility endpoint; main.js?v=4
- integration tests: family-lens 403 pin, hidden-library exclusion
(differential), seeded text+audio aggregate correctness, auth gate
- docs/developer/api/stats/get_library_usage.md
Known-remaining suite failures are environmental: TestConflictList_Empty
+ TestWebSocketPingPong collide with the concurrently-running app session
(live emulator pushes + scans on the shared dev DB), and
TestAdminCreateUser's timezone expectation drifted from live system
settings — all three fail on the clean tree too.
A2 made narrators/narrators_search NOT NULL DEFAULT '{}' (86f0af6), but an
explicit NULL in an INSERT/UPDATE column list bypasses the column default
— the API media-create path (POST /api/media-items without narrators)
500'd on every call, taking the whole integration suite down with it
(ISBN tests, capture seeds, all direct-DB seeds).
- set_library_type_name() trigger now coalesces NULL narrators/
narrators_search to '{}' at the row level, and its update trigger
widens from BEFORE UPDATE OF library_id to BEFORE UPDATE so every
write path is covered
- defense in depth at the callers: handler create/update normalize nil
to empty slices (normalizeNarrators), scanner seeds the same
(nilToEmpty) for files that carry no narrator credits
- analytics device-usage tests move to the dedicated per-test user
(the shared admin's device list accrues live rows mid-suite — the
documented testuser flapping class)
A2a (metadata round): narrators TEXT[] + narrators_search (tags
pattern: accumulator + GIN pairs) + abridged bool on media_items;
scanner fills from the Libation/ABS sidecar (narrators verbatim,
abridged flag) with audio-payload promotion; editorial fields ride
the real columns while audio_metadata stays scan-derived (duration/
codec/chapters). Detail/edit/save paths carry the fields; the web
edit dialog grows an Audio section (Narrator(s) comma-input, Abridged
toggle, scanned duration read-only); search weights + narrator filter
(word_similarity over narrators_search) + narrator autocomplete added.
A2c (per-media-type wording): copy helpers keyed on format_group —
Listen Now, Mark Complete/Not Complete, Listening Progress, Your
Listening Stats, Time Listening, Time listened (pages tile swapped for
audio), Position H:MM:SS / total H:MM:SS on detail + progress pages
(duration parsed from audio_metadata). Dashboard: system collections
swap to Continue Listening / Recently Listened at DISPLAY time when
the audiobooks library is selected (stored names unchanged; restores
keep working); week card = Listening Time / Sessions / Books Touched;
cover-corner badges identify audiobook (headphones) / comic / manga
cards on mixed grids — books stay unbadged; hover action icon follows
the media type.
A2b (player): the scrubber binds value to the playhead with an explicit
scrub guard (the handle previously froze at its initial max); header
row = reader-style back arrow + Chapter n/total launcher; chapters
open in a floating top-down drawer (current chapter highlighted, page
never shifts); Skip/Speed rows share one label-left layout; honest
wall-clock session accounting (playing-only deltas, paused flush).
A2d (bookmarks): note-bearing time bookmarks ride the existing
media_bookmarks substrate (notes column + sync-aware service already
there) — audio locator convention audio:<seconds>, chapter_number
filled, auto-title '<chapter> — H:MM:SS', capture popover with optional
note, drawer with jump/edit-note/delete. Zero server changes needed.
Verified live on the Libation sample: narrators backfilled by rescan;
PUT edit persisted + overrides detected (narrators, abridged); force
rescan preserved the user edit; reset-to-scanned-of-defaults restored
truth; narrator filter/autocomplete scoped by library; dashboard
listsening wording + badges render; bookmark save with note deduped to
one row.
Phase 1 (foundation): 'audiobooks' library type seeded (schema + DB);
audio_metadata JSONB on media_items (the kobo_metadata precedent —
only audio paths read it); reading_progress.audio_offset_seconds for
the honest {chapter, offset} locator; extensions/mimes registered for
ALL breadth formats (m4b/m4a/mp3/flac — deliberate per the DOC/LIT
doctrine, m4b scans richly first); classifyFormatGroup 'audiobook'
case (gotcha 39 recomputes on rescan); Go mime table extended at boot
so media elements accept the streams; ffmpeg/ffprobe in the image
(8.1.2 — also the xHE-AAC identification tripwire).
Phase 2 (ingest): sidecar-first (Libation/ABS metadata.json —
narrators/ASIN/curated chapters), ffprobe fallback/filler (duration,
bitrate, codec/profile, embedded chapters). Fixed three real
parser brittlenesses the Libation sidecar exposed: series accepts
string OR array-of-objects, publishedYear accepts number OR string,
ffprobe chapter timings accept number OR string (flexFloat). Folder-
structure fallback now runs on the UPDATE path too (rescans were
clearing sidecar-absent series/author); audio facts re-derive on
every rescan (gotcha 39).
Phase 3 (playback + sync): GET /api/media-items/:id/chapters (the
format-abstraction list); streaming rides the existing cookie-authed
Range /uploads path; PUT progress carries audio_offset_seconds +
elapsed_seconds (honest wall-clock playing time) — the reading-
history capture prefers player-reported elapsed over the inter-push
delta, same 2h ceiling; reading_mode='audio' marks the row.
Phase 4 (web player): dedicated audio reader page (foliate not
involved) — cover, chapter title, whole-book scrubber, skip ± (chips
1/5/10/15/30s, default 10s), speed (0.75×–2×), chapter list with
jump-to, Media Session lockscreen/notification transport, periodic
playing-only progress pushes, position restore on open.
Verified live: ingest of the Libation sample (Gardens of the Moon —
title/author/series/cover/date from sidecar+folders, 34 chapters,
26h duration, aac/LC); chapters endpoint; the reader route serving
the player; playback with position restore (0:57 round-trip);
progress rows in audio mode; wall-clock session rows in
reading_history.
Bug: the admin 'Scanning Libraries' widget fed the server's 0-1
progress straight into percent strings and bar widths, so the overall
and per-library percentages capped at '1%' and the bars never filled.
Both now scale by 100 (matching the setup page, which was already
correct).
Bug: scan state only reached clients via websocket events, so a page
opened (or refreshed) mid-scan showed nothing until the next tick —
and the admin widget never re-attached at all. The worker now exposes
active jobs (with their library_id so rows keep their names), served
as GET /api/scanner/active (admin, re-attaches the widget + polling)
and GET /api/scanner/active-summary (any user, restores the sidebar
indicator on load). Idle hides both, exactly as before.
Annoyance: the sidebar's 'Scanning… N%' was a block element between
the logo and the nav, shoving every menu entry down while a scan ran.
The percent now lives inside the fixed-height logo row next to the
spinner (spinner last, so growing digits extend leftward); nothing
below it moves.
Three dashboard complaints, one build:
- The widget collapses to a single summary row (This Week · minutes ·
pages · sessions · books · streak · Full stats) on every screen size
and expands to the full grid via a details/summary toggle — the
carousel sections start immediately below it again.
- The 'Your Library' heading moved into the library switcher subbar
(LibrarySwitcher gains an optional title param; collections/series
keep the plain 'Library' label). The standalone heading block is
gone.
- Root-caused the JS-switch vanishing: renderDashboardCollections
replaces ALL of #collections-container's innerHTML, so anything
server-rendered inside it (title, widget) died on the first library
switch. The widget now re-fetches ITSELF: a /dashboard/week-widget
fragment (same resolveLibrary family-lens rules as the page) is
pulled by htmx whenever the switcher dispatches week-refresh —
carrying the current #library-select value — so the widget both
survives switches and RE-SCOPES to the switched library. Initial SSR
renders the same scope the sections use.
ThisWeek (and the streak calendar) gained an optional library filter.
Tests: TestReadingStatsWeekLibraryScope — per-library totals, scoped
streaks, invisible-library isolation, malformed id 400. Full suite
PASS.
The bucket name read as confusing — 'unsynced' implied data loss when
the push is always captured; it only lacks a device credential. The
bucket is just 'Web' now (service fallback, stats.ts fallback, panel
copy, tests).
Also: analytics_test exact-zero assertions (EmptyHistory,
NoReadingHistory) ran against testuser, which is a LIVE reader — real
sessions accrued on it mid-suite and both flapped. They now run under
the fresh regular user, like the stats tests; the 'testuser is a live
reader' rule is recorded in AGENTS.md.
- Attribution: registered clients may now send their device token in
X-Device-Token on PUT /api/media-items/:id/progress (the user JWT
stays primary). The token must map to a device OWNED BY the caller —
foreign/invalid/absent tokens soft-fail to plain web attribution so a
stats decoration can never break a sync. Attributed pushes capture
device_id (FK-safe), device_metadata.source = the registered device
name ('Pixel 9', 'Kindle Paperwhite'), and last_sync surfaces show
that device. Side effect worth having: app-vs-web pushes used to be
LWW-invisible (both sources read 'web'); they now surface as
legitimate cross-source progress conflicts when they diverge.
- /stats gains a By Device panel alongside By Library (grid 3-wide):
reading minutes + sessions + books per registered device, plus one
Web / unsynced bucket for device-less pushes so the panel always
reconciles with the headline totals. Range-aware, family-lens
scoped; renderBars shared between panels in stats.ts.
- The device identity IS the registered platform: devices rows already
carry the name/type the client registered (app sends Build.MODEL +
'mobile' at /api/devices/register, KOReader/Kobo likewise) — no new
identity plumbing on the server.
Tests: capture attributed to the device (row device_id + source
metadata + last_sync_source), foreign/nonexistent tokens soft-fail,
device buckets reconcile (80 attributed + 75 unsynced = 155 total).
Full suite PASS.
The design parked in the handoff (multi-tier stats off reading_history;
bookorbit-style in data, not looks) is in. The real prerequisite work
was data: reading_history existed but NOTHING wrote to it — the whole
stats feature was sitting on an empty table.
- CAPTURE (no client changes, no repo-boundary crossings): every
successful progress write through SaveProgress (KOReader + app + web
+ Kobo + offline queue — the one chokepoint) now records one
reading_history session row: span = the interval since the PREVIOUS
write instant, pages = positive page delta when both sides know the
page index, marker rows for first contact. Spans over 2h are kept as
zero-second markers (overnight suspend→resume, offline flushes must
not inflate totals) — the honest 'sync cadence, not a stopwatch'
caveat stays. device_id only when a REAL device row matches (the
web/app progress path historically stuffs the USER id into DeviceID
— probe instead of trusting it; FK violation would silently drop
every capture). Source preserved in device_metadata.
- API: /api/stats/summary?range=7|30|90|year (totals, zero-filled
daily arrays, streaks, most-active day/weekday, top books, library
breakdown — one call), /api/stats/this-week (widget: rolling 7d vs
previous + streak), /api/stats/books/:id (lifetime + recent
sessions). All bucketed in the user's timezone; finished ≡ any push
≥0.995; visible-library lens enforced at source. Legacy
/api/analytics/* kept untouched. Range windows anchor end-of-day to
the START of today (blasting the clock time into tomorrow resolved
'today' a calendar day late — found by the streak test).
- /stats page (sidebar Stats between Progress and Devices, previously
/analytics was an orphan): range selector, five stat cards, two
Chart.js v4 charts (daily minutes bar, pages/day line — chunked via
dynamic import so main.js stays untouched), per-library breakdown,
most-read list, empty-state explainer. /analytics now 301s to /stats.
The orphaned analytics.templ + analytics.ts removed (their containers
and the page JSON never matched; the page was astub).
- Dashboard 'This Week' widget (server-rendered, suppressed on service
failure) and the book detail 'Your Reading Stats' card (lifetime
totals of the viewer's own sessions) complete the tiers: per-user
default, per-library breakdown inline, per-book on the page,
per-system/admin aggregate parked as before.
Tests: statistics_test.go (capture semantics at the chokepoint,
tz/day buckets, zero-fill, streak math, finished detection, range
validation, auth gate, widget, per-book lifetime). Full suite PASS.
Type is DERIVED, not stored: a saved ruleset makes a collection SMART,
no rules = STATIC (stored collection_items links, unchanged behavior).
No schema change — collections.auto_assign_rules JSONB already carried
the config; user's decision 2026-10-14 (handoff: live-at-render, not
refresh-on-scan; create-time Static/Smart toggle; v1 NO hybrid).
- Rule gains an enabled flag (absent = enabled, legacy rules unaffected).
- Rule CRUD: POST/PUT(DELETE) /api/collections/:id/rules[/:ruleId] — the
survivor dead effort of the rules-builder UI finally has real
endpoints (rules were never editable after creation; nothing ever
re-evaluated them). First rule converts a user collection to smart;
removing the last reverts it to static. UpdateCollectionRulesJSON
edits ONLY the rules doc so rule CRUD cannot clobber metadata.
- Live evaluation: candidates from GetVisibleLiveMediaForRules —
visible-library set (family lens enforced at source) + archived IS
NULL AND missing_scan_count = 0 — any enabled rule matches, title
order. Zero refresh hooks by design: always current by construction.
- Membership mutations (add/remove/bulk-add) are 400 on smart — v1 no
hybrid; collection_items.excluded stays the reserved future hook.
- Rules builder page (/collections/:id/rules) first time ever routable;
templ whole-attribute interpolation ("{ expr }" inside a quoted attr
value is literal text, as theme-{ user.Theme } silently proved).
- UI: create modal Static/⚡Smart cards (smart create redirects into the
rules builder), ⚡ badge + real per-card counts on the grid (smart =
live matches, static = stored-link count), smart detail renders
matched books with no hand-pick actions + an Edit Rules button.
- API: GET /:id gains is_smart + live books; GET /: keeps the Android
app's default exclusion, include_auto=true surfaces smart rows with
live counts; TestRules previews against the SAME candidates the live
render uses (visible + live, was raw first-1000).
- FIXED pre-existing regression from 7ce7321: static collection detail
with NO library_id returned zero books — QueryType != '' caught user
rows ('filter' default) and routed them to dashboard sections that
never contain them; now only real system query types go there
(handlers + SSR). App-visible symptom: /api/collections/:id without
library_id came back books:[] for every static collection.
Tests: collections_smart_test.go (conversion, disabled rules, mutation
guards, revert-to-static, validation, system-collection prohibition,
include_auto semantics). Full offline integration suite PASS.
The family-lens hardening: library_visibility was only enforced where
file bytes flowed (download/reader/OPDS) — metadata surfaces leaked
hidden libraries to any authenticated account, and registration was
open by default with no admin-side user creation.
- Series are grouped PER LIBRARY now (same name in two libraries =
two entries: manga adaptations / LN splits no longer merge);
/api/series aggregates only the requester's VISIBLE libraries and
/api/series/books REQUIRES a visible library_id (breaking contract;
web + app clients updated in the same change).
- /api/media-items, item detail, and the reader content endpoints
(chapters/panels/outline/THUMBNAILS — page images leaked unguarded
next to a gated GetPage) all enforce the visible set; resolveLibrary
no longer honors a hand-typed library_id outside the visible set.
- Annotation/note/bookmark mutations (update/delete/tombstone, sync
LWW paths incl. conflict resolution) carry ownership predicates:
another account's bare row id is a 404, not a silent delete.
- Admin user creation: POST /api/auth/users (AdminMiddleware) + a
Create User card in the admin Users UI — accounts stay creatable
with self-service registration OFF, which is now the schema default
(fresh installs; the first-admin bootstrap exception is unchanged).
- Tests: series contract re-pinned (per-library, visibility 403s),
new ownership + admin-create integration suites, OPDS all=1 test
now passes an explicit page size (shared-DB growth made the fresh
books fall off page 1), device-cap helper switched to the admin
endpoint.
TestProgressWeb_KoboIntegration 500s'd on every markup with a
last-read-place bookmark: the panic-recovered stack was a nil pointer
dereference in LibraryService.ResolveMediaPath (s.db deref), reached
from KoboHandler.convertKoboCFIToStandard during kepub CFI conversion.
Two-layer bug:
1. Test harness (root cause): the router.Config built in
test_helpers_test.go omitted LibraryService, so router/sync.go's
SetLibraryService(cfg.LibraryService) stored a nil
*services.LibraryService in the interface field. Production wiring
(main.go:246) was always correct.
2. Latent trap (hardening): `h.libraryService != nil` at the
convertKoboCFIToStandard call site can never catch a typed nil —
an interface holding a nil *LibraryService is non-nil. ResolveMediaPath
now degrades with a "library service not configured" error instead
of panicking on a nil receiver; callers already handle the error
by skipping CFI conversion.
Harness-only fix, zero production behavior change: the endpoint now
returns a clean 200 with the position saved wherever kepub conversion
is unavailable instead of a panic-recovered 500.
Verified: TestProgressWeb_KoboIntegration passes all three subtests
(401 unauthenticated, ReadingSync→last-read-place percentage
preservation, last-read-place without prior ReadingSync) and on
immediate rerun — previously it failed every run.
The old model hard-deleted system collections and re-created them at
the tail, resurrecting stale dedups and breaking ordering. Redesign
(user-directed B-final):
- schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER —
the whole schema is re-applied on every boot via executeSchema).
- queries.sql: every collection list/read query filters
deleted_at IS NULL; new GetSystemCollectionByQueryType :one
(unfiltered, sees tombstones), TombstoneSystemCollection :exec,
ClearCollectionItems :exec, RestoreSystemCollectionRow :one
(in-place metadata reset + un-tombstone; collections have no
updated_at column — ResetUpdatedAt was a 500 caught by tests).
sqlc regenerated.
- collection_service.DeleteCollection: system collections are
tombstoned (preserving position/ownership); user collections keep
the hard delete. No ErrNoRows to the client (delete is idempotent).
- dashboard_service.RestoreSystemCollection: resolve by query_type;
found → in-place restore (ClearCollectionItems + metadata reset,
un-tombstoned); missing → provision fresh. "Want to Read"/etc.
stay at their home positions forever.
- handlers/collections.CreateCollection: name-taken is 409, not 500.
- handlers/auth.CreateDefaultCollectionsForUser: idempotent by
(user_id, query_type) — update-in-place when a row for that query
type exists (heals renames/tombstones), insert when missing; this
also fixes the duplicate-system-collections regression at source.
The dashboard's All Libraries view could render sections across every
library but could not persist its own preferences: the PUT required a
UUID library_id, and the prefs GET-without-library_id matched an
arbitrary library's row through a loose NULL OR-clause.
Storage: a dedicated user_dashboard_preferences_global table (one row
per user, user_id PK → users CASCADE) instead of a NULL/sentinel
library_id — chosen over both because (a) the column's FK to libraries
rules out a sentinel UUID without dropping the constraint or minting a
fake library row, and (b) intent becomes structural: only the explicit
global path writes that table, so a buggy or outside-the-API write can
never masquerade as global. System and user collections are
user-scoped, so the order/hidden arrays are library-independent and
valid in this scope.
API: PUT /api/dashboard/preferences binds library_id as *string —
JSON null/absent = global, empty string = 400 (a lost value must stay
loud, not silently clobber the global scope), valid UUID = per-library
(unchanged). GET without library_id now returns the global row instead
of an arbitrary one; the per-library query's dead NULL OR-arm is
removed. GET /sections keeps requiring library_id for per-library
views and aggregates across libraries when absent (unchanged).
Tests: global save/update-dedupe/read round-trip, empty-string 400,
global-vs-per-library independence. Two suite failures are the
handoff's documented pre-existing rot (duplicate system collections in
the dev DB; a sections test expecting a 400 the handler never
implemented) — reproduced and confirmed unrelated to this change.
Live-verified on the dev stack: null save 200 x2 with a single row
updated, empty string 400, global read carries the second save.
.IMAGE — extensions removed from AllowedExtensions (scan gate),
bookExtensions, MimeTypes and sync/format.go maps: .doc/.lit files
are no longer scanned or indexed. classifyFormatGroup gains the
RTF and PDB reflowable arms that were missing when those rungs
shipped (rows only reclassify on creation). Scanner docs updated.
DOC + LIT have no credible JS tooling (mammoth is docx-only; LIT
needs LZX and its DRM variants are dead) and text-extraction-only
support would misrepresent what the reader can do. Verified: probe
.doc/.lit files watched but never scanned; probe .txt control
scanned; TestClassifyFormatGroup extended and passing.
Docs change granted in-session by the user (extensions + format
docs).
.docx is accepted for ingest but the format-group switch had no case
for it, so every DOCX row stored format_group 'unknown' with
is_reflowable=false. Extract the classification switch into a pure
classifyFormatGroup helper with a table test, and classify .docx
reflowable like the other reflowable text formats (.rtf/.doc stay out
until they are readable client-side). Existing rows: dev alice (DOCX)
reclassified by hand; production rows reclassify on rescan.
Adds publisher_filter to SearchMediaItems/SearchMediaItemsUnified
(fuzzy word_similarity against mi.publisher, mirroring genre_filter),
plumbs it through services.SearchParams and the search handler, and
extends the GREATEST relevance ranking to include publisher matches.
Serves the app's author/publisher/genre/tag click-through browses
(UX pass item 5 — publisher was the only facet without a server
filter).
Previously the scanner's SHA-256 dedup was library-scoped: moving a book
between libraries created a duplicate row (new ID) while the old row went
missing and archived after two scans, orphaning reading progress and
annotations from the file that users still see.
processMediaFile now falls through to cross-library detection when the
same-library hash lookup misses:
- GetMediaItemsBySHA256AndSameLibraryType returns candidates in other
same-type libraries; each candidate's file is stat'd through ITS OWN
library's folders (the old existence check stat'd against the current
scan's folder tree, which is meaningless across libraries).
- File gone -> it is a move: the candidate must pass the target library's
type rules (ValidateMediaItemForLibrary), then the row is repointed via
MoveMediaItemToLibrary with the recomputed relative path. Reading
history, annotations, and collections follow automatically because the
row keeps its ID. Incompatible formats (e.g. a reflowable EPUB into a
manga library) are rejected with a processing issue instead of being
force-imported; the upsert on (media_item_id, issue_type) keeps
repeated scans from spamming duplicates.
- File still present -> deliberate multi-library copy: fall through to
normal import so both libraries keep independent rows.
Candidate selection is factored into selectMoveCandidate (pure function,
unit-tested in media_scanner_move_test.go): input arrives pre-ordered
(archived first, then most missing scans, then oldest) and only
candidates whose file is verifiably gone qualify, so deliberate copies
are never repointed.
Fixed-layout EPUBs lean on the reading_direction column (the web reader
forces book.dir = rtl from it when the file didn't set direction
itself), but the scanner never populated it for EPUBs - only ComicInfo
fed it. Meanwhile real Japanese EPUBs declare page-progression-direction
on the OPF spine, which foliate reads client-side but nothing stored.
Read the spine attribute in the structured OPF parser and map it into
ReadingDirection in parseOPFContent (EPUB2/3, case-insensitive, plus
'right-to-left'/'left-to-right' spellings); undeclared stays empty
rather than forcing ltr, preserving the editor's Auto default. Sidecar
OPFs are metadata-only documents without spines, so the Calibre path is
a no-op. The merge gap-fill copies an embedded-only direction into a
blank sidecar field, and hand-set values keep winning through the
existing OverrideReadingDirection protection.
Tests: declared rtl/RTL/ltr, undeclared and unknown values staying
empty, plus sidecar-wins vs embedded-fills merge cases. Existing manga
EPUBs declaring rtl (verified live in-library) pick the value up on
their next scan, feeding the API and reader config mobile clients
consume.
Same gap-fill rule as the EPUB path, against the PDF's embedded Info
dictionary via a readPDFInfoDict helper reusing extractPDFMetadata's
field conventions (creator falls back to author, subject maps to
description, producer to publisher, keywords to tags, plus page count).
Sidecar values always win; unopenable PDFs skip silently.
TestMergeMetadataPDFGapFill uses an Info-bearing hand-built PDF fixture
(shared with the sidecar-cover test) and asserts the sidecar title is
kept while author/description/publisher/tags/page count fill in.
A sparse metadata.opf/metadata.json (title only, no description) left
books thin even when the file itself carried rich data: the embedded
extractors only ran when no sidecar existed at all. Now mergeMetadata
fills blanks from the book's own OPF - title, author, description,
publisher, language, ISBN, ASIN, series/number, publish date, tags,
contributors - while sidecar values always win and unparseable files
skip silently. Also covers .kepub, which the merge previously ignored
while the extractor already supported it.
Adds TestMergeMetadataEPUBGapFill asserting both directions: sidecar
title/author survive, embedded description/publisher/language fill in.
When the SHA-256 dedup found identical content already in the library,
the scan skipped the file as a duplicate - and after files moved
between folders the old row kept its stale path, cycled missing ->
archived, and the new path never took. The archive feature turned the
old destructive move behavior into a stuck move instead.
Now the dedup branch stats the old location: if it is gone, the book
was MOVED, so the row is repointed (file_path, file_size) with archive
state cleared and reading history intact. 'Skip as duplicate' only
applies when the old path still exists (a true copy). Verified live: a
moved EPUB kept its single row, followed the file, and never entered
the missing/archive cycle.
A full rescan wiped cover_image_path for every PDF/EPUB living in a
metadata.json (or metadata.opf-less) folder with no cover.jpg next to
the book: the sidecar branches returned early after findSidecarCover
missed, and mergeMetadata has no PDF/EPUB cover logic of its own. Four
books lost their thumbnails while their {file}.cover.jpg files still sat
on disk - most visibly the Audiobookshelf-managed No Starch titles.
Both sidecar branches now fall through to an embedded-cover fallback
(PDF via extractPDFCover, EPUB/KEPUB via extractEPUBCover) whenever no
sidecar cover file exists. Comics are untouched: mergeMetadata already
extracts their covers from the archive.
Adds TestSidecarCoverFallback with a hand-built one-page PDF carrying a
JPEG XObject plus a metadata.json sidecar, asserting the sidecar title
wins while the cover still comes from the file. Verified live: rescans
restored all four dereferenced covers with no leftover rows.
Reset to Scanned cleared the overrides row (successfully) but then set
the in-memory copy to nil before handing it to updateMediaItem. pgx
encodes a nil []string parameter as SQL NULL, so the follow-up UPDATE
wrote metadata_overrides = NULL into the column's NOT NULL constraint
and the whole rescan failed with:
failed to update media item: ERROR: null value in column
"metadata_overrides" of relation "media_items" violates not-null
constraint (SQLSTATE 23502)
Two changes:
- RescanMediaItem's reset path assigns []string{} instead of nil, with a
comment explaining the pgx nil-to-NULL encoding trap.
- updateMediaItem routes the override set through utils.MergeOverrides,
whose contract guarantees a non-nil slice, so no caller can write
NULL into that column again (verified against pgx v5.9.2 source: a
scanned '{}' round-trips as non-nil in both directions; the nil could
only come from our own assignment).
The plain Rescan path never hit this - only Reset did. Worse, the reset
is the remedy when a book's cover_image_path override pins an empty
cover, so the crash also blocked the way out of that state. After this
fix, a plain rescan on an already-reset book repopulates scanned
metadata and extracts the cover.
Adopt Calibre's reading conventions for the Dublin Core metadata that
parseOPFContent now pulls from the structured OPF parse:
- Titles: EPUB3 title-type selection (prefer 'main', join a distinct
subtitle with ': ' exactly as Calibre stores it). There is no separate
subtitle column by design - Calibre-sidecar books arrive pre-joined,
so a column would stay empty for most libraries and force every client
to reimplement concatenation.
- Genre: first dc:subject, mirroring the existing processGenresAndTags
behavior of the Calibre-sidecar path; the embedded path never
populated Genre before. Subjects stay one-element-one-tag - Library
of Congress headings legitimately contain commas ("Holmes, Sherlock
(Fictitious character) -- Fiction") and must not be split.
- Identifiers: urn:isbn:/urn:asin: prefixed values parse in addition to
opf:scheme attributes, and the scheme-less fallback now requires an
ISBN-shaped value (10/13 digits, optional separators/trailing X) so
URIs like the Gutenberg identifiers cannot masquerade as ISBNs -
observed live on 'A Study in Scarlet'.
- Series: EPUB3 belongs-to-collection with collection-type=series and
group-position refines, ahead of the classic calibre:series metas.
- Audiobookshelf metadata.json sidecars join their subtitle field into
the title the same way.
Tests cover title-type main+subtitle joining, belongs-to-collection
series with fractional group-position, urn:isbn extraction, genre/tag
parity, and comma preservation inside subject headings.
The cover lookup scraped the OPF with attribute-order-sensitive regexes.
Real books serialize attributes in any order - Grand Central's '3 Days to
Live' puts href before id on manifest items and content before name on
the cover meta - so all three regex paths missed and the book fell
through to filename guessing, extracting no cover at all. Attribute
order is meaningless in XML; the regexes were never safe.
Replace them with a structured parse (encoding/xml, namespace and
attribute-order agnostic; see the new media_scanner_opf.go) and follow
Calibre's read_raster_cover resolution order:
1. manifest item with properties=cover-image (non-(X)HTML media only)
2. <meta name=cover> resolved through the manifest, same media guard
3. first spine item that is itself a raster image (store manga)
4. NEW cover-page fallback: books declaring no raster cover at all -
the classic EPUB2/Adobe cover.xhtml wrapper - are mined for
<img src> / SVG <image xlink:href> references (Calibre renders the
page with Qt; extracting the referenced image covers the practical
cases without a rendering engine)
5. existing zip filename guessing stays as the last resort, and the old
regex chain survives as findCoverInOPFLegacy for OPFs too malformed
for a real XML parse.
Hrefs are now URL-decoded and posix-normalized against the OPF's own
path (path.Join semantics), so '../art/cover.jpg' from a nested cover
page and %20-encoded names resolve correctly.
Tests: attribute-order chaos modeled on the failing Patterson book,
SVG-wrapped cover pages via guide references, image-first spines, and
path resolution edge cases. Verified live against the real
'3 Days to Live' EPUB, which previously produced no cover.
Replace the silent hard-delete orphan cleanup (which logged only through
ScannerLogger file logs and whose failure paths left rows undetected)
with an archive lifecycle that preserves reading history:
- A file missing in one scan is marked (missing_scan_count = 1); missing
in a second consecutive scan archives it (archived_at, hidden from
browsing, progress/notes/highlights survive). Every branch logs to
stdout with an [ARCHIVE] prefix so skips are always visible.
- When a file reappears - same path, or identical content at a new path
via the SHA-256 dedup match - the archived state clears automatically
and the item returns with its history intact.
- Archived rows older than ARCHIVE_RETENTION_DAYS are hard-purged at
scan time (cascading deletes); 0 disables auto-purge for manual-only
management. Retention is read from the environment in NewMediaScanner.
Libraries managed by Audiobookshelf keep a metadata.json next to each
book (title, authors, series+sequence, genres/tags, publisher,
description, isbn/asin, language, published year/date) - and no
metadata.opf. The scanner silently ignored those files: deleting them
changed nothing, and their data never reached the database.
Parse them as a first-class sidecar in extractMetadata, priority
metadata.opf -> metadata.json -> embedded media. Only fields with a
matching media_items column are mapped; narrators, subtitle, explicit,
abridged, and chapters are deliberately skipped.
Cover handling is unchanged: the existing findSidecarCover priority
(cover.jpg / folder.jpg / {basename}.jpg) applies to the sidecar branch
exactly as it does for Calibre.
go-epub's ReadBook parses every spine chapter and fails the entire call
if any single chapter (or the TOC) is malformed, discarding already-parsed
OPF metadata. For books like Pragmatic's 'A Common-Sense Guide' the OPF
holds good title/author/publisher/ISBN metadata that rescans then wrote
as blanks - success toast, no (visible) change.
Refactor to parse the EPUB's own OPF document with the same Dublin Core
machinery used for Calibre sidecars: parseCalibreMetadataOPF is now a thin
file wrapper around a reusable parseOPFContent([]byte), and the OPF lookup
previously inline in extractEPUBCover is shared via findOPFPathInZip. Since
metadata never touches chapter bodies, chapter damage cannot blank it.
Also picked up along the way: dc:language mapping and scheme-less
dc:identifier values that normalize to a valid ISBN (EPUB3 style).
Tests cover a Pragmatic-style EPUB (dc namespace on <metadata>, no
identifier scheme, deliberately malformed chapter) that must still yield
full metadata, plus series/date/subject OPF parsing.
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
pdftoppm defaults to rasterizing the MediaBox, while PDF viewers (pdf.js
in the reader, and every other viewer) display the CropBox. For PDFs
whose page 1 is the full print cover wrap (back cover + spine + front
cover in one landscape page) with a CropBox covering only the front
cover - e.g. No Starch's XeTeX-built 'Algorithmic Thinking' - the
fallback stored the entire spread as a squashed landscape cover, while
the reader correctly showed just the front cover.
Pass -cropbox so the rendered cover always matches what the reader
displays. Poppler falls back to the MediaBox when a PDF defines no
CropBox, so PDFs with identical boxes (the common case) render exactly
as before.
extractPDFCover previously only saved embedded raster images from page 1,
so vector/text-first-page PDFs (e.g. InDesign exports like Data Structures
the Fun Way) ended up with no cover and a dashboard placeholder. When no
embedded image is found it now falls back to rendering page 1 with
pdftoppm (poppler-utils), saving the same {pdf}.cover.jpg sidecar.
Also adds MediaScanner.RescanMediaItem, which re-extracts metadata for a
single media item (resolving its on-disk path from library folders) so
previously imported books can backfill covers without a full force rescan.
Dockerfile installs poppler-utils in the final and test-runner stages.
Phase 4 of the reader redesign (foliate-js ea268df):
- Webtoon mode for comics: continuous vertical scroll of all pages
(900px centered column on wide screens), lazy-loaded with a 150%
IntersectionObserver margin, far pages unloaded to bound memory
with stable aspect-ratio placeholders so the scrollbar never jumps.
Chosen per book (Paged | Webtoon segmented control in Settings →
Layout & Display; stored in localStorage per media item since a
webtoon title and a paged manga volume want different flows).
Toggling reloads the reader — the renderer is chosen at open time —
and progress restores from the saved page. Relocate events flow
through the same pipeline, so the slider, progress saving, back
stack, tap zones, and edge zones all work unchanged. Zoom/fit/
magnifier/spread controls hide in webtoon (natural-width scroll).
- Display filters for fixed-layout: brightness (30-130%) and
contrast (70-130%) sliders with live preview, plus Night Mode
(invert) — also a quick row in the ⋯ tools menu. One --fx-filter
CSS var drives everything: ::part(filter) on foliate-view iframes
(forwarded via the new exportparts attribute) and the webtoon
page images alike. Persisted as fx_brightness/fx_contrast/fx_invert
(types + defaults both sides); Restore Defaults resets them.
Phase 2 of the reader redesign:
- Fixed-layout touch engine (foliate-js e9e61d8): pinch-zoom around
the midpoint, two-finger pan, single-finger pan while zoomed,
horizontal swipe page-turn at fit (RTL-aware via next()/prev()),
and double-tap to zoom 2.5x / reset. Touch events forwarded from
page iframes with converted coordinates; preventDefault only when
the engine consumes the gesture, so PDF text selection and native
taps stay intact. touch-action: none on the host and in comic/pdf
page documents keeps the browser from fighting the engine.
- Tap zones (Kindle-style) for touch devices: tap the outer margins
to page, center to toggle chrome. Size configurable (10-50%) via
the revived tap_zone_size setting; toggle via new tap_zones_enabled
(Behavior section of the settings drawer). Pointer-based + passive
so drags/swipes/selection never trigger; attached both to the
viewport and inside every page document (iframe events don't
bubble); debounced 280ms so double-tap zoom doesn't also page; no
zone actions while a fixed-layout page is zoomed.
- Drawers become full-width sheets on screens <= 640px.
Phase 1 of the reader redesign:
- Reading surface is edge-to-edge; top/bottom bars overlay
translucently (backdrop-blur) instead of reserving insets, killing
the inset-coordination bug class entirely. Chrome auto-hides after
2.5s of pointer inactivity (chrome_behavior setting finally wired:
auto-hide / always-visible; legacy values map to auto-hide). Pointer
activity inside page iframes keeps it awake; Esc toggles.
- TOC / Settings / Bookmarks become slide-over drawers with a scrim
(z-50, full-height, safe-area aware), replacing the dockable-panel
system and its window-shade headers. Only one drawer opens at a
time; Esc or scrim click closes.
- Bottom bar is contextual: reflowable keeps nav/slider/progress/TOC;
fixed-layout row adds Fit Page/Width select, zoom cluster,
magnifier (now shows active state), Double Page Spread toggle, and
a Smart | Pan | Text segmented control replacing the cryptic
two-state icon. Smart = text-aware drag; Text = selection-only
(manual smart-detect off); Pan = force pan. Choice persists via
pdf_interaction_mode (new setting + foliate 29bc958 'text' mode).
- Settings drawer: Behavior (chrome, progress mode), Appearance with
18 Kindle-style theme swatches (single source of truth from
THEME_COLORS), Typography, Layout — each scoped by format.
- Keyboard: t/s/b open TOC/settings/bookmark, Esc closes drawers
before toggling chrome, shortcuts skip form inputs; both slider
rows tracked correctly (no duplicate-ID lookups).
- Topbar: Back, title, add-bookmark, bookmarks drawer, Aa settings;
chrome follows user theme.
Complete the SHA-256 lifecycle for preexisting databases: items
imported before hashing existed get hashed automatically, and any
content duplicates discovered in the process land on the new admin
Hash Conflicts page for an explicit keep/merge decision.
HashBackfillService (runs once 30s after startup, independent of
auto-scan):
- hashes every media_items row where file_sha256 IS NULL, resolving
each path through LibraryService; per-item failures are logged and
skipped so one unreadable file cannot block the pass
- no-op once everything is hashed (logged and skipped)
- finishes with a conflict sweep flagging every content-duplicate
group via FindHashConflictGroups + CreateHashConflict; the sweep
runs after the per-item pass because a preexisting pair only
becomes detectable once both sides have their hash
API (admin-only):
- GET /api/admin/hash-conflicts - pending groups with member items
and usage counts
- POST /api/admin/hash-conflicts/:id/resolve - action=keep_all, or
action=keep with keep_uuid: validates the uuid belongs to the
group, re-parents every other copy's child rows onto the kept item
(reparent_media_item_children), deletes the losers, and records
the resolution + resolving admin; accepts form or JSON bodies and
returns the htmx resolved fragment
Page route /admin/hash-conflicts (admin-only) renders the template
with hydrated conflict data; HashConflictsHandler wired into the
router Config and constructed in main.
Verified end-to-end against the live database: duplicate detection,
pending listing, keep_all resolution, merge path (re-parent +
delete), and - critically - a resolved group is not re-flagged by a
later sweep (upsert no-op). Database restored afterward.
Force rescan was metadata-only: updateMediaItem never touched the
hash identifiers, so a force scan could not backfill file_sha256 for
items imported before hashing existed (or where extraction originally
failed). Those items were invisible to content dedup and SHA-256
device matching with no way to fix short of delete + re-import.
processMediaFile now refreshes hash identifiers in three cases:
- force rescan (the admin Scan button becomes the backfill tool)
- file size change (stored hash is stale - the bytes changed)
- unchanged file with no stored hash (ordinary scans self-heal the
legacy backlog incrementally, no admin action required)
Each recompute runs recordHashConflictIfAny: when the freshly stored
hash is now shared by more than one item in the library, the group is
upserted into hash_conflicts for the admin Hash Conflicts page. The
upsert is a no-op for already-tracked groups, so resolved 'keep both'
decisions stick.
Also extract a package-level computeFileSHA256 (the scanner method
now delegates to it) so the startup backfill service can hash files
without a scanner instance.
The platform had three duplicated, divergent book resolvers (koreader,
kobo, BookMatchingService) and none of them consulted
media_item_formats.file_sha256 - per-format hashes for converted files
(KEPUB, PDF) are computed and stored at import/conversion time but were
never used for lookup. GetMediaItemFormatBySHA256 existed with zero
callers. Any client holding a converted file could never match by
hash.
Add internal/services/book_resolver.go: a single shared resolution
path from client-supplied identifier to media_item.
ResolveBySHA256 checks media_items.file_sha256 first (indexed
GetMediaItemBySHA256), then falls back to media_item_formats.
file_sha256 (indexed GetMediaItemFormatBySHA256, first caller) so a
converted format matches with equal confidence. The import-time
SHA-256 is the canonical identifier shared by every interface.
Wire two of the existing resolvers through it:
- BookMatchingService.matchBySHA256: replaces the in-memory
ListMediaItems scan of up to 1000 rows with the resolver's indexed
lookups, and gains format awareness for the link/auto-link UI.
MatchMethod now reports sha256_sha256 or sha256_sha256_format
- KoboHandler.mapContentIdToBookhoardUUID: the SHA-256 heuristic
branch (ContentId that looks like a 64-char hash) now resolves
format-aware too. Kobo's entitlement_id wire identity is untouched;
only the opportunistic hash branch changed
A read-then-write race in processMediaFile allowed the same file to be
imported twice: two concurrent scan jobs (startup scan, fsnotify dirty-
directory scan, periodic backup poll, or a manual scan each run on
separate worker goroutines with separate MediaScanner instances) could
both SELECT 'not found' and both INSERT. There was no transaction, no
row lock, no unique constraint on (library_id, file_path), and no
ON CONFLICT clause, so nothing stopped the double insert. Observed in
production as two identical 'Head First SQL' rows created in the same
second (same sha256, size, path, library).
Database enforcement:
- schema.sql: add UNIQUE(library_id, file_path) constraint, guarded so
re-runs don't error
- schema.sql: add self-healing migration that runs on every startup -
dedup_media_items_by_path() collapses existing path-duplicates and
reparent_media_item_children() moves all child rows (progress,
highlights, bookmarks, notes, collections, formats, aliases, kobo
entitlements, etc.) onto a survivor before deleting losers, so the
constraint applies cleanly on already-duplicated servers without
losing reading history. Survivor picks the row with the most user
data, ties broken by lowest id
- CreateMediaItem: upsert via ON CONFLICT (library_id, file_path) DO
UPDATE so concurrent inserts collapse to one row and return it
- CreateMediaItemFormat: upsert via ON CONFLICT (media_item_id,
format_type), closing the same race on format rows
Application-level guards:
- media_scanner processMediaFile: after computing the file hash, check
GetMediaItemBySHA256AndLibrary (new query) and treat the file as
existing when identical content is already in the library under a
different path (content dedup, library-scoped so multi-library
setups still work)
Ops tooling:
- scripts/dedup_media_items.sql: standalone idempotent maintenance
script with a dry-run report (path + content duplicate groups, child
row counts) and transactional cleanup, for servers that prefer to
dedup manually before upgrading
Verified against the live database: the duplicate pair was collapsed
(reading_progress preserved on the survivor), schema.sql re-runs are a
no-op, and the constraint is in place with 62 unique books remaining.
The double_page_spread checkbox in the reader settings panel was inert:
it had no Alpine binding, no apply logic, and no persistence. Default
was also inconsistent (false in settings-manager, absent from server
defaults).
- Add doublePageSpread state to the reader Alpine component, loaded
from saved settings (default true)
- Add applyDoublePageSpread() which sets the renderer's 'spread'
attribute to auto/none and persists the setting via saveSettings
- Apply the spread attribute during fixed-layout renderer init
- Bind the settings checkbox with x-model and @change
- Add double_page_spread: true to ReaderService server defaults so
new users get the same starting value the client expects
- Also improve the PDF pan/select toolbar button: distinct smart-
select vs pan icons, highlighted state while pan mode is active,
and dynamic tooltips/aria-labels explaining each mode
Split each constructor into a default-args wrapper and a config-accepting
variant so the sync queue interval/batch size and the worker pool size/
queue cap can be sourced from the settings registry at startup. These
values are constructed once at boot, so they are tagged requires_restart
in the admin UI.
queue.go:
- NewSyncQueueProcessorWithConfig(db, interval, batchSize) takes the
flush interval and batch size as parameters; NewSyncQueueProcessor
becomes a thin wrapper with the historical 5s / 50 defaults.
worker.go:
- NewWorkerWithConfig(numWorkers, queueCap, connManager) takes the
queue capacity as a parameter; NewWorker becomes a thin wrapper with
the historical cap of 100.
No behavior change for existing callers; main.go will switch to the
config-accepting variants in a follow-up wiring commit.
Move three more hardcoded values behind the settings registry. All
apply immediately on the next request (no restart needed).
device_auth.go:
- DeviceAuthMiddleware reads per-route device rate limits (sync /
progress / metadata per minute) from the registry on each
authenticated request via a rateLimitConfig() helper, falling back to
the Default* constants when no registry is wired.
- The X-RateLimit-Limit response header previously hardcoded "60" for
every request type; it now reflects the actual configured limit for
the request type via rateLimitForRequestType().
opds.go:
- Default (50) and maximum (200) OPDS page sizes come from the
registry's OpdsDefaultPageSize()/OpdsMaxPageSize() instead of inline
literals, so catalog pagination can be tuned without a redeploy.
conversion_service.go:
- The 24h kepub cache lifetime is read from the registry via a
cacheTTL() helper (was a bare 24 * time.Hour literal in the
constructor). The field default is retained for tests that construct
the service directly.
- conversion_service_test.go updated to assert both the field default
and the cacheTTL() accessor return 24h.
The bookhoard container crashed with 'panic: Failed to create file
watcher: too many open files' (media_scanner.go) after running for a few
hours, preceded by floods of 'no space left on device' from watcher.Add.
Root cause: every scan job called NewMediaScanner(), which eagerly
created an fsnotify watcher. SetFolders() then walked the entire library
tree and registered one inotify watch per directory (~3,000+ across the
libraries), and ScanFolders() registered them again during its walk. The
worker never called scanner.Close() on these ephemeral per-job scanners,
and the worker loop had no recover(), so:
1. Leaked watchers accumulated until the kernel inotify watch cap was
hit (ENOSPC -> 'no space left on device'), then
2. the process fd limit (ulimit -n 1024) was exhausted, causing
fsnotify.NewWatcher() to fail with EMFILE, and
3. NewMediaScanner panicked on that error, taking down the whole
process (exit code 2). With no restart policy the container stayed
down.
The scan jobs run frequently (scan_poll_interval), so the leak built up
within hours. Note this was NOT a disk-space issue; df showed plenty free.
Fix:
- media_scanner.go: NewMediaScanner no longer creates a watcher eagerly
(s.watcher starts nil), which removes the panic site entirely -- there
is nothing to fail at construction. The watcher is created lazily only
when needed.
- media_scanner.go: SetFolders gains a [?1049h[22;0;0t[1;24r(B[m[4l[?7h[?25l[H[2JEvery 2.0s: bool[1;37Hgaruda-ser8: Fri 31 Jul 2026 10:45:41 AM EDT[2;66Hin 0.002s (127)[2;80H
[3dsh: line 1: bool: command not found
[4d[24;1H[?12l[?25h[?1049l[23;0;0t
[?1l> parameter. It creates
and populates a watcher (returning an error instead of panicking) only
when watch=true; otherwise it skips all watcher.Add calls. ScanFolders
guards its watcher.Add with a nil check, and the WatchChanges event
loop exits cleanly when there is no watcher (polling still runs).
- worker.go: the worker() loop now wraps each job in defer/recover() so a
panicking job is recorded as failed and can never kill the process.
- worker.go: the three ephemeral scan handlers (processScanJob,
processSetFoldersJob, processDirectoryScanJob) now defer scanner.Close()
and call SetFolders(..., false), so scan jobs allocate zero watchers and
zero inotify watches. Any pre-existing leak is also bounded by Close().
- handlers/scanner.go: the long-lived watch-mode scanners (StartScanner
and StartWatchModeForLibrary) pass watch=true since they actually read
watcher.Events for live change detection.
- calibre_integration_test.go: updated to the new SetFolders signature
(watch=false, matching one-off scan usage).
Auto-add is fully preserved: new files are still detected by the periodic
poller (startBackupScan), which is independent of fsnotify and unaffected
by these changes. The watch-mode event loop remains as bonus responsiveness
when inotify is available; through Docker bind mounts where inotify is
unreliable, polling is what catches new books.
- dashboard_service.go: Change libraryID parameter from uuid.UUID to
pgtype.UUID across GetDashboardSections, GetDashboardPreferences,
and all helper methods. pgtype.UUID{Valid: false} now signals
"no library filter" (All Libraries), which gets passed through
to sqlc.narg() in the SQL layer.
- series_service.go: Drop libraryID parameter from GetSeriesBooks
entirely. Series are not library-specific — all books in a series
are shown regardless of which library they belong to.
The MessageTypeScanComplete constant existed but was never actually sent by
the worker. This meant the frontend had no way to know when a scan finished.
- After a JobTypeScan completes, broadcast scan_complete to the job's user
via WebSocket ConnectionManager
- Includes job_id, files_scanned, new_items, and errors in the payload
- Only broadcasts for JobTypeScan (not other job types) when connManager
is available and job.UserID is set
John O'Keefe
john-okeefe