2c5ad014c009487464f0ea95c4fa24047d51e99f
168
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
2c5ad014c0 |
feat(scanner): multifile folder/zip sidecar detection (faceted plan Phase 6)
Reverses the Phase 4 scope cut: for EXISTING folder/zip audiobooks a
sidecar edit was invisible to everything, including force rescan —
updateAudioBookItem writes only scan-derived fields, and these formats
often have no useful embedded tags, so the sidecar is the only source.
- Change fingerprint gains the manifest compare in both processors:
changed := size differs || track payload differs || manifest drifted.
The ladder runs against the bulk-loaded map entry (stat pairs only,
zero extra queries); NULL stored manifest backfills without
re-extraction (deploy-day rule, same as single-file).
- Manifest-changed applies the once-extracted book metadata to the
editorial columns (updateMediaItemWithMetadata, overrides protected)
and re-runs the playlist hook. The apply runs BEFORE the focused
payload write: the generic update's format-group pass derives mime
from the sentinel (none) and chapters from metadata.ChapterCount
(unset) — the payload write must win (rescanMultiFileAudioBook's
ordering). Track-only changes keep the existing editorial-keep
update; force rescan keeps it too.
- Manifests seeded at multifile create (folder names derive from the
sentinel — the exact representative the extractor reads — via
detectSidecarChangeWithRepresentative, so manifest and extractor can
never disagree; a directory-basename cover is ignored by the
extractor and therefore not tracked).
- Zip parity fixes: the fast path now lifts archive state like the
folder path always did (a returned zip book stayed archived
forever), and the change branch handles the unchanged-archived case.
- Single-file sidecar change path: extract-failure guard — the old
non-nil empty &MediaMetadata{} fallback bypassed the update path's
filename-title guard (preExtracted == nil only) and would blank the
title on a failed extraction; the row is now left untouched and the
manifest refreshed. Empty-success extraction (sidecar removed from a
tag-less book) now passes nil so the create-equivalent chain (tags →
filename) re-derives instead of blanking.
- Multifile changes now count in the scan summary (changedItems);
they were previously uncounted.
Tests: sentinel-name derivation unit pin; folder sidecar edit
(title + collection sync + fresh manifest + steady-state skip),
track-only change keeps editorial, zip sidecar edit + archive
restore, sidecar removal falls back to filename (the wipe-guard's
testable sibling). Full container suite: green except the documented
pre-existing TestAdminCreateUser red.
|
||
|
John O'Keefe
|
5245f292ca |
feat(scanner): quiet scans, enqueue coalescing, watcher probe, sibling cache
Part 2 of the faceted plan, Phase 5 (steps 5-6): the scan that used to print ~4 lines per unchanged file now prints ONE summary line, and a poll tick landing mid-scan no longer stacks a second full scan. - Quiet scans (§4): the walk loop is silent — inserts, changes, and errors log inside processMediaFile; the duplicate 'Successfully processed' branch (both branches printed it) and the per-file 'processing / already exists / same size, skipping' chatter are gone. Summary carries the trigger and a changed counter: 'Scan of <folders> (trigger: poll): N files, N changed, N new, Xs, N errors'. Triggers threaded through every enqueue site: poll (startBackupScan), watcher (dirty-dirs flush), admin (scan API), initial (startup scan). - Enqueue coalescing (§5): Worker.HasActiveScanForLibrary (JobResult gains additive job_type so only SCAN jobs coalesce — an import sharing the library id never blocks one). enqueueLibraryScan skips silently; the admin scan API answers 409 with the active job id. Enforces the one-scan-per-library invariant the bulk scan-state map leans on. - Watcher capability probe (§4): ONE Add on a root folder; failure disables the watcher with a single startup line (replaces the per-directory warning spam on inotify-broken deployments). The watcher stays an optional latency reducer over scheduled scans. - Per-directory sibling cache (§6): hasSiblingBookFileCached caches the ReadDir per directory for one scan — loose-image manga re-stats thousands of images sharing a handful of directories. Tests: TestHasActiveScanForLibrary (unit — running/pending coalesce, terminal jobs don't, non-scan jobs don't, empty library never, nil worker safe). Full scanner suite green in the container. |
||
|
John O'Keefe
|
6750e9e0dc |
feat(scanner): manifest-gated sidecar change detection + playlist sync on change path
Part 2 of the faceted plan, Phase 4 (steps 3-4). Size-only detection
was blind to metadata-sidecar edits; a size-stable book is changed by
EDITING its metadata.opf/metadata.json/cover — previously invisible
until a force rescan.
- Sidecar manifest fingerprints (media_scanner_manifest.go): the
manifest stats exactly the names the extractors consume (metadata.opf,
metadata.json, findSidecarCover patterns incl. <basename>.<ext>), so
manifest and extractors can never disagree. Shape v1:
{v, entries:[{n, s, m (unix nanos — second granularity would hide
edits landing in the backfill scan's own second), h (sha256 — covers
included: the repoint ladder needs a baseline hash for any stat-
drifted sidecar; a hashless cover turns harmless mtime drift into a
re-extraction, the exact flood this design kills)]}.
- Evaluation ladder in cost order: stat pairs identical → skip with
ZERO bytes read; stat drift → hash that sidecar (equal = host repoint
reconciliation, stat pair refreshed, no re-extraction; differing =
changed); NULL stored manifest → backfill only, NO re-extraction
(deploy-day rule: the first scan after release must not look like an
mtime flood).
- Change path: re-read full row, extractMetadata once and pass it
through updateMediaItemWithMetadata (preExtracted — metadata_overrides
stay protected), run syncPlaylistCollectionsAfterCreate (Correction 2:
a reading-order entry added to an EXISTING book now creates its
collection; add-if-missing is idempotent), store a fresh full
manifest. Creates seed their manifest at insert time.
- Multifile folder/zip books intentionally keep Phase 3 behavior: their
change path does not consume sidecar extractors, so gating them would
flip books 'changed' without picking anything up (plan correction —
the plan's §3 multifile note assumed otherwise).
- UpsertMediaScanState + GetMediaScanState queries.
Tests: media_scanner_manifest_test.go (ladder unit tests — fast path,
repoint, same-size drift, appear/vanish, unhashable legacy entries);
TestScanManifestLifecycle (seeding, unchanged-skip stability, repoint
reconciliation with stat-pair refresh assertion, same-size edit
consumed, playlist sync on the change path + idempotency, no-leak
guard). Container runner.
|
||
|
John O'Keefe
|
693053a9f5 |
perf(scanner): bulk scan-state load — one query per library scan
Part 2 of the faceted plan, Phase 3 (steps 1-2). The periodic scan
issued one GetMediaItemByFilePath per scannable file (~3,094 queries
per scan in prod, nearly all answering 'unchanged, skip') plus one
GetLibraryByFolder per file for folder resolution.
- NEW media_scan_state table: the scanner's per-item fingerprint
cache. Deliberately NOT a media_items column — the library-scoped
list emits raw sqlc rows as JSON (
|
||
|
John O'Keefe
|
7e586df48a |
feat(facets): facet registry, Batch A server params, genre model v2, facets endpoint
Phase 1 + backend of Phase 2 of the faceted-navigation plan. One
declarative facet registry (services/facet_registry.go) drives the
bookshelf filter system; a drift-guard unit test reflects over the
generated SearchMediaItemsUnifiedParams so a registry entry and its
sqlc.narg block can never disagree silently.
- Nine new filter params on SearchMediaItemsUnified: story_arc,
imprint, edition (scalar-fuzzy), age_rating, format_group,
manga_type, reading_direction (exact), explicit, abridged
(presence-bound bools). Comma = OR-within multi-select on every
text facet; AND-across-facets unchanged.
- Genre model v2 (user rule: a subgenre is not a genre, it's a tag):
genre_lexicon.go rebuilt as shelf set + subgenre->shelf map;
pickSmartGenre picks direct shelf > mapped subgenre > Fiction floor
> empty (unknown-only stays tags-only). Type-words (Manga, Comics,
Graphic Novel) join the demote list. Both 'adventure' and 'action &
adventure' are shelves — provider data uses both and the verified
traces pin them.
- Language filter UX: comma elements normalize through the ingest
language map, so display names ('English') match rows stored as ISO
codes ('en'). The language autocomplete now serves display names.
- NEW GET /api/media-items/facets?facet=X&library_id=... — registry-
driven value+count listing (14 static per-column queries, family-lens
join); language values carry display names; bool/range facets 400.
- Family-lens gates: an explicit hidden-library library_id is now a
403 on the search endpoints (was a silent empty set) — JSON branch
and HTMX grid partial both.
- handlers.ParseMediaSearchParams is the SINGLE param parser for the
search JSON endpoint and the HTMX grid partial; the HTML branch
previously dropped publisher_filter and narrator_filter silently.
Pins: docs/contract_pins.md changelog entries (additive request
params; new facets endpoint). Tests: facet_registry_test.go (shape +
SQL drift guard + library-type union), series_parsing_test.go genre-v2
cases, facets_integration_test.go (param filtering e2e, counts,
language display, 403s) — container runner.
|
||
|
John O'Keefe
|
f94e7472b4 |
feat(filters+ui): per-element author/publisher filters, playlist toggles, Up Next API, reorder UI, dashboard badges
Phases 4 + 5 of the metadata overhaul (plan §4, §5) plus §3 handler wiring. Filters (§4.1): author_filter/publisher_filter now match per-element against the authors[]/publishers[] arrays — a co-authored book matches any one of its authors — with comma-joined params acting as OR-within-facet multi-select (string_to_array + cross unnest). Legacy single-value params unchanged on the wire; series/genre/language filters stay on their varchar columns; narrator filter already per-element. Collections API (§3.3-3.5): collection detail carries viewer + hide_finished into the listing query and exposes up_next (visible- library-enforced reading-queue head); PUT /api/collections/:id/items/order (owner-only, full-order rewrite); UpdateCollectionRequest gains hide_finished/show_up_next ([] string last-value-wins for the hidden-false-sibling checkbox pattern) merged server-side OVER stored view_settings so imported_from_series provenance survives edits; EditCollectionModal renders the toggles from view_settings. Dashboard (§5.5): sections JSON carries is_playlist/complete (dashboard.ts is the AGENTS-documented JSON-render exception; first paint stays templ). Fully-read playlists render '✓ Playlist complete' instead of an empty rail; imported playlists get a 📖 Playlist badge. Frontend: book detail shows subtitle + full authors + Explicit badge; edit modal gains subtitle, comma-separated authors/publishers (client- split to arrays exactly like narrators), step=any series number, explicit checkbox; collection book cards get playlist move ↑/↓ buttons persisting the full order via the reorder endpoint (templ partial re-render on failure — DOM order is client-local, API order is truth). AGENTS.md gotcha #3 amended per plan §3.7: user collections are dashboard sections by default; the show_on_dashboard premise is retired. Verified: full affected integration sweep green (pins widened for up_next/is_playlist/complete — changelog entries in docs/contract_pins.md); unit suite green; go build + templ + vite green. |
||
|
John O'Keefe
|
1dbb0ce406 |
feat(collections): playlist auto-creation, position ordering, hide_finished, Up Next, dashboard eligibility
Phase 2b + 3 of the metadata overhaul (plan §2.4, §3.1-3.7). Auto-playlist collections (§2.4): the scanner turns obviously-playlist sidecar series (explicit '(Reading Order)'/chronological/publication/ in-order/timeline markers only — precision over recall) into curated static collections: upsert by view_settings.imported_from_series then name (rename-safe), position set ONLY on insert (clean integers; 1.5/1a stay unpositioned rather than guessing), membership add-if-missing, never removed or re-ordered by rescans, provenance in description + view_settings, show_on_dashboard true (dashboard-eligible by design), 'scanner.auto_playlist_collections' setting (default on, Settings Registry + schema seed). Ordering (§3.1-3.2): collection listings order position ASC NULLS LAST then added_at ASC (was added_at DESC — walked playlists backwards); PUT /api/collections/:id/items/order applies full-order rewrites (owner-only, service-gated). hide_finished + Up Next (§3.3-3.4): GetCollectionItemsForDashboard gains viewer_user_id + hide_finished (LEFT JOIN reading_progress, COALESCE percentage < 0.999) and viewer_percentage rides the row for card graying; GetCollectionUpNext serves the collection's reading-queue head with the family-lens visible-library array enforced. Dashboard (§3.5): user collections are dashboard-ELIGIBLE BY DEFAULT — the show_on_dashboard gate (never writable for user rows since the column shipped) is gone from GetUserCollectionsForDashboard; hiding stays dashboard-prefs (hidden_collections) territory; hide_finished filtered-everything now emits a Complete section instead of silently vanishing; sections JSON carries is_playlist/complete. Series page ordering hardening (§3.6): GetSeriesBooks tiebreaks by title ASC — NULL/tied numbers previously returned heap (scan) order, which is what the user saw. Frontend custom-section-builder note: the builder has sent show_on_dashboard: true since its creation; the API never bound it. User-row visibility is now intentionally prefs-governed, so the field stays unbound by design (documented in code). Pins widened (docs/contract_pins.md): collection detail + up_next; dashboard sections + is_playlist/complete; new reorder endpoint recorded. Verified: TestPlaylistCollectionsEndToEnd (new, full e2e: sidecar series split, collection auto-creation with position 7, rescan idempotency, owner-only reorder, finished-filter + completion marker + Up Next drain) plus the full affected integration sweep green. |
||
|
John O'Keefe
|
7e4c97f639 |
feat(scanner): fix ABS sidecar series parsing; three-way series mapping; smart genre; language codes; title-derived series
Phase 2a of the metadata overhaul (plan §2.1-2.3, §2.5-2.9). The headline bug: ABS writes sidecar series as an array of STRINGS (["The Witcher #3", ...]); the parser only accepted {series, sequence} OBJECT arrays, so every ABS-matched book silently lost its series (found live: 9 sidecars, zero series ingested; only folder-structure Malazan survived). parseSeriesCandidates now normalizes string arrays, object arrays and Libation plain strings. Three-way series assignment (plan §2.2): - primary bibliographic series -> series/series_number columns - additional real series (Drizzt: Icewind Dale #1 = Legend of Drizzt #4) -> alternate_info JSONB, comic-established shape - ordering constructs ('The Witcher (Reading Order)') -> PlaylistSeries candidates; NEVER the series column. Detection is explicit-marker-only ((reading order|chronological|publication order|in order|timeline) as parenthetical/suffix) — precision over recall; a series merely containing 'timeline' mid-name stays a series. Sequence policy: leading-number extraction ('Book 2' -> 2, '2, Dramatized Adaptation' -> 2, '1a' -> 1); lossy extractions preserve the raw text in alternate_info.series_sequence_raw. Smart genre (deterministic lexicon): compound Audible blobs split ('Epic, Action & Adventure, Military, Sword & Sorcery' -> 4 candidates), alias-normalized, format noise demoted ('Audiobook' never wins), most-specific canonical genre takes the genre column, all candidates stay in tags. Language: display names normalize to ISO 639-1 on ingest; ROOT-CAUSED the update-path drop — CreateMediaItem call site never set Language (explicit NULL over the column default), the update path did; create now carries it. Title-derived series fallback (§2.7): 'Baptism of Fire: The Witcher, Book 3' 2-level folders split conservatively (only when the post-colon tail matches ', Book N'), subtitle parsed for series+number, blanks-only precedence, matching-name number fill, zero display mutation. Works with no sidecar and no network. Ingest fixes: subtitle/explicit land in their columns (audio tag subtitle stops joining into title); full authors[] array (was [0]- only); sidecar descriptions sanitized via bluemonday UGC policy (vendored). metadata_overrides: verified scan-update path already applies ApplyMetadataOverrides; subtitle/explicit/authors covered by the Phase 1 override adaptation. Tests: new series_parsing_test.go (string-array sidecar e2e, Drizzt alternate_info, lossy raw preservation, playlist classification, title-derived fallback incl. no-mutation + blanks-only, genre lexicon, language map); audio-tag subtitle pin updated to column behavior; calibre/ABS 2.5-series_index pins widened per float change (changelog'd in phase 1). Deferred to 2b: auto-created playlist collections (§2.4) — settings toggle, collection upsert, ownership. |
||
|
John O'Keefe
|
646477afad |
feat(schema): subtitle/explicit columns, authors+publishers arrays, float series numbers
Phase 1 of the series/collections/metadata overhaul
(~/.opencode/plan/series-collections-metadata-overhaul.md).
Schema (in place, idempotent):
- media_items.series_number INTEGER -> DOUBLE PRECISION: real-world
ordinals are decimals (0.5, 1.5 novellas); INTEGER silently truncated
them (calibre 2.5 pinned as 2 in tests).
- + subtitle TEXT, explicit BOOLEAN (sidecar ingest; ABS explicit /
Audible isAdult).
- + authors/authors_search/publishers/publishers_search TEXT[] NOT NULL
(narrators A2a precedent). Arrays are write truth; author/publisher
varchars become trigger-regenerated display strings
(sync_author_publisher_display) so every varchar consumer (trgm
search, facets, API, OPDS/Kobo) keeps working unchanged.
- Row-level NULL normalization for the new arrays in
set_library_type_name (A2 lesson: Go nil -> explicit NULL -> 23502).
- Backfills ARRAY[author]/ARRAY[publisher] — deliberately no
comma-splitting ('Tolkien, J.R.R.' would corrupt; scanner kept only
authors[0] historically).
- collection_items.position INTEGER + (collection_id, position) index
for playlist ordering.
Go: sqlc regen; MediaMetadata/handler params gain Subtitle/Explicit/
Authors/Publishers (legacy singles still accepted, arrays win);
SeriesNumber int32 -> float64 end to end; metadata_overrides adapted
with legacy override keys ('author'/'publisher') preserved so existing
user overrides survive; bulk author edit writes the array.
Pins widened (docs/contract_pins.md): raw sqlc row key set + the six
new columns; changelog entry added.
Verified: unit suite green; integration sweep (bulk/smart collections/
scanner/stats/sync/queue/calibre/pins/audiobooks) green against the
shared dev DB after in-place migration. Pre-existing unrelated red:
TestOwnershipAndAdminTestSuite/TestAdminCreateUser timezone inheritance
fails identically at the pre-change commit (live shared state).
|
||
|
John O'Keefe
|
6216bd9bcf |
feat(audiobooks): multi-file books (folder-of-tracks + zip) + exhaustive metadata ingestion
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
|
||
|
John O'Keefe
|
30ecdd4bf3 |
feat(scanner): ScanFolders honors its context; cancelling never archives
ScanFolders never checked its context, so a cancelled job's walk ran to completion no matter what the worker's cancel did. - ctx checks at every stage boundary (file-counting pass, per folder, before and during the archive pass) and inside the walk callback (fs.SkipAll): the file currently being processed finishes, then the scan aborts with context.Canceled, which processJob classifies as a cancelled job rather than a failed one - the archive/purge pass is skipped entirely when the scan was cancelled. It only visits what the walk visited, so after a partial walk the unvisited (but on-disk) files would be treated as missing, archived, and eventually purged. Cancelling is a UI convenience and must never destroy library data. |
||
|
John O'Keefe
|
b2220756e2 |
feat(worker): real per-job cancellation for the background job pool
CancelJob existed but was dead code: it only flipped a status in the in-memory results map - it never cancelled anything, and processJob's unconditional completion write overwrote the marker anyway. - derive a per-job context in processJob (parent is the handler ctx for manual scans, Background for auto-scans) and register its cancel func under the job id; previously every manual scan shared the handler's process-lifetime context, so no targeted cancel was possible without killing everything - skip execution at dequeue for jobs cancelled while still queued - record a pending JobResult on BOTH enqueue paths (Enqueue used by auto-scans, EnqueueJob used by the API): queued jobs were invisible to status polling (404 until a worker picked them up) and could not be cancelled before execution started - cancelled jobs keep their last real progress instead of a fake 100% and drop the meaningless "context canceled" error text - broadcast the new scan_cancelled websocket event for cancelled scan jobs instead of scan_complete, so clients reset their scanning indicators rather than reading the abort as a finished scan - sentinel errors (ErrJobNotFound, ErrJobNotCancellable) so HTTP handlers map outcomes to 404/409 without string matching Unit tests (worker_cancel_test.go) pin: queued-cancel never executes, running-cancel invokes the context, error mapping, pending recording on both enqueue paths. |
||
|
John O'Keefe
|
327fc2c0f9 |
feat(matching): match results as a scoped templ partial + enriched matches
- BookMatch now carries title/author/cover (the fields the old client renderer always assumed but the API never sent — cards would have shown 'undefined'); filled during the family-lens pass in QueryBooks - new partial route GET /unlinked-books/matches renders the scoped match cascade server-side; searchMatches swaps it via htmx.ajax instead of rebuilding cards from JSON - NOTE: the unlinked-books surface is currently dark end-to-end — the templ page has no route, the live linking.ts renderer targets a container no template has, and its Link button calls a window.showMatchModal that does not exist. This commit ships the scoped backend + partial; wiring the UI is a product decision. |
||
|
John O'Keefe
|
040c10ac76 |
feat(matching): family-lens scoping for book matching and linking
/api/books/match returned media metadata from ALL libraries regardless
of visibility: the resolver lookups (GetMediaItemBySHA256 and friends)
are global, so match results leaked hidden-library titles, covers and
ids to any authenticated user, and LinkBook accepted a link target from
a hidden library. Both now enforce the family-lens rule:
- QueryBooks post-filters the cascade by the requester's visible-library
set (absent-from-results — a hidden library's book simply never
matches), before determineAction so the suggested action reflects the
filtered set. BookQueryRequest gains a server-side-only
VisibleLibraryIDs field; nil keeps legacy callers unfiltered.
- LinkBook verifies the device belongs to the caller
(DeviceExistsForUser) and the target item's library is visible —
403 otherwise. Linking was previously unscoped on both axes.
- GetUnlinkedBooks likewise verifies device ownership (its service is
still a stub, but the route no longer answers for other users'
devices).
Also: the collections books-grid partial uses a light loader —
requested library validated against the visible set directly instead of
the page-only library resolution (folder reachability, counts,
dropdown payload), removing the extra queries that made library
switching hold the spinner longer.
And the dashboard library switch now folds the This Week widget's
htmx refresh behind the spinner (refreshWeekWidget awaits the widget's
swap, with a safety timeout) instead of letting it pop in after the
fade-in — the two-stage update introduced with the widget's
library-scoped re-fetch.
Guardrail: the rendering lint now counts all innerHTML assignments (not
just template-literal ones) — the old pattern undercounted 20 files at
80 sites, including .join("") builders like renderDashboardCollections.
Baseline rebalanced accordingly.
|
||
|
John O'Keefe
|
454938c31f |
refactor(vis): H4.5 steps 4-5 — statistics and the three reader gates on the canonical resolver; core goes package-level
VisibleLibrarySet becomes a package-level function taking the queries (db-only services like statistics reach it without new wiring); the LibraryService method stays as the ergonomic entry point. Statistics' ids-shaper and the reader pages' three hand-rolled membership loops (empty-rows fallback then linear scans) now delegate; the gates keep their 403-on-invisible semantics (>null set membership, never silent pass). Census addendum from this pass: two OPDS sites call GetUserVisibleLibraries directly but consume FULL ROWS (per-library folder enumeration) — a third legitimate pattern alongside set/ids and resolveLibrary's active-selection; documented, untouched. The library LIST endpoint and its SSR data helper are row consumers too, by design. |
||
|
John O'Keefe
|
1d12a64915 |
refactor(vis): H4.5 step 1 — LibraryService.VisibleLibrarySet, the canonical resolver; media endpoints migrated
One core for "which libraries can this user see": family-lens default- allow stays where it always was (the GetUserVisibleLibraries SQL); the core adds the shared set+ids shaping that four endpoints each hand-rolled. Media's visibleLibrarySet is now a thin identity adapter (JWT user object -> resolved id) and the download-permission gate uses the set instead of a manual membership loop. Semantic diff (H4.5 step 1, in the plan): zero family-lens divergence found across all six sites — the rule lives in one query; the variation was shape-level (identity adapter, set-vs-ids, UUID idiom). |
||
|
John O'Keefe
|
77cf0b827b |
refactor(db): GetCollectionItemsForDashboard embeds the media row; excluded filter moves into SQL; third converter dies
SELECT mi.*, ci.excluded forced every caller to filter tombstone exclusions in Go over a 69-field row, then a 45-field hand converter (getCollectionItemsRowToMediaItems) copied it field-by-field into MediaItems for BuildSections. The query now filters (ci.excluded IS NOT TRUE) itself and returns sqlc.embed(mi) — callers consume .MediaItems directly, the Go filter loops die in both collection-detail ladders (JSON + SSR), and the converter is deleted. Pins, H1/H3 tests, unit suites green. |
||
|
John O'Keefe
|
18727fef93 |
refactor(collections): rule evaluators take the real row; both 45-field converter twins deleted
checkRulesAgainstBook/evaluateRule and the service's EvaluateRules existed in two parallel universes: one over database.MediaItems (what every query actually returns) and one over ListMediaItemsRow (what two hand-written 45-field mediaItemsToListMediaItemsRow converters produced — verbatim twins in the handler and the dashboard service, with a URL-resolution side effect baked into a "converter"). The rule fields (genre/series/author/language/publisher/copyright_year/tags) all exist on MediaItems, so the evaluators now take the real row and the converters, their three call-site loops, and the dead ApplyAutoAssignRules entry point (zero callers) are gone. The rules surface is wire-covered by the collections rules scenarios; handlers+services suites and the full pin/H1/H3 battery green. |
||
|
John O'Keefe
|
ce366b1604 |
feat(stats): Library tab — per-library inventory stats + audio-aware Reading tab
The /stats page gains a second tab (Reading | Library) implementing the
Audiobookshelf-style library dashboard (HANDOFF item 9): media_items
INVENTORY aggregates — items, hours/pages, distinct authors, size,
distinct narrators, Top Genres / Top Authors bars, Longest / Largest
items — for one library or the All-libraries aggregate.
- GET /api/stats/library-usage?library_id=<uuid|omit>; family lens
enforced: a library_id outside the requester's visible set is a 403,
never a silent pass (AGENTS.md standing rule;
|
||
|
John O'Keefe
|
a222ef5c27 |
fix(media-items): normalize NOT NULL narrators on every write path
A2 made narrators/narrators_search NOT NULL DEFAULT '{}' (
|
||
|
John O'Keefe
|
86f0af632e |
feat(audiobooks): A2 — narrator metadata, audio-aware UX, player polish, bookmarks
A2a (metadata round): narrators TEXT[] + narrators_search (tags pattern: accumulator + GIN pairs) + abridged bool on media_items; scanner fills from the Libation/ABS sidecar (narrators verbatim, abridged flag) with audio-payload promotion; editorial fields ride the real columns while audio_metadata stays scan-derived (duration/ codec/chapters). Detail/edit/save paths carry the fields; the web edit dialog grows an Audio section (Narrator(s) comma-input, Abridged toggle, scanned duration read-only); search weights + narrator filter (word_similarity over narrators_search) + narrator autocomplete added. A2c (per-media-type wording): copy helpers keyed on format_group — Listen Now, Mark Complete/Not Complete, Listening Progress, Your Listening Stats, Time Listening, Time listened (pages tile swapped for audio), Position H:MM:SS / total H:MM:SS on detail + progress pages (duration parsed from audio_metadata). Dashboard: system collections swap to Continue Listening / Recently Listened at DISPLAY time when the audiobooks library is selected (stored names unchanged; restores keep working); week card = Listening Time / Sessions / Books Touched; cover-corner badges identify audiobook (headphones) / comic / manga cards on mixed grids — books stay unbadged; hover action icon follows the media type. A2b (player): the scrubber binds value to the playhead with an explicit scrub guard (the handle previously froze at its initial max); header row = reader-style back arrow + Chapter n/total launcher; chapters open in a floating top-down drawer (current chapter highlighted, page never shifts); Skip/Speed rows share one label-left layout; honest wall-clock session accounting (playing-only deltas, paused flush). A2d (bookmarks): note-bearing time bookmarks ride the existing media_bookmarks substrate (notes column + sync-aware service already there) — audio locator convention audio:<seconds>, chapter_number filled, auto-title '<chapter> — H:MM:SS', capture popover with optional note, drawer with jump/edit-note/delete. Zero server changes needed. Verified live on the Libation sample: narrators backfilled by rescan; PUT edit persisted + overrides detected (narrators, abridged); force rescan preserved the user edit; reset-to-scanned-of-defaults restored truth; narrator filter/autocomplete scoped by library; dashboard listsening wording + badges render; bookmark save with note deduped to one row. |
||
|
John O'Keefe
|
902cbe5915 |
feat(audiobooks): Batch A — foundation, m4b ingest, playback sync, web player
Phase 1 (foundation): 'audiobooks' library type seeded (schema + DB);
audio_metadata JSONB on media_items (the kobo_metadata precedent —
only audio paths read it); reading_progress.audio_offset_seconds for
the honest {chapter, offset} locator; extensions/mimes registered for
ALL breadth formats (m4b/m4a/mp3/flac — deliberate per the DOC/LIT
doctrine, m4b scans richly first); classifyFormatGroup 'audiobook'
case (gotcha 39 recomputes on rescan); Go mime table extended at boot
so media elements accept the streams; ffmpeg/ffprobe in the image
(8.1.2 — also the xHE-AAC identification tripwire).
Phase 2 (ingest): sidecar-first (Libation/ABS metadata.json —
narrators/ASIN/curated chapters), ffprobe fallback/filler (duration,
bitrate, codec/profile, embedded chapters). Fixed three real
parser brittlenesses the Libation sidecar exposed: series accepts
string OR array-of-objects, publishedYear accepts number OR string,
ffprobe chapter timings accept number OR string (flexFloat). Folder-
structure fallback now runs on the UPDATE path too (rescans were
clearing sidecar-absent series/author); audio facts re-derive on
every rescan (gotcha 39).
Phase 3 (playback + sync): GET /api/media-items/:id/chapters (the
format-abstraction list); streaming rides the existing cookie-authed
Range /uploads path; PUT progress carries audio_offset_seconds +
elapsed_seconds (honest wall-clock playing time) — the reading-
history capture prefers player-reported elapsed over the inter-push
delta, same 2h ceiling; reading_mode='audio' marks the row.
Phase 4 (web player): dedicated audio reader page (foliate not
involved) — cover, chapter title, whole-book scrubber, skip ± (chips
1/5/10/15/30s, default 10s), speed (0.75×–2×), chapter list with
jump-to, Media Session lockscreen/notification transport, periodic
playing-only progress pushes, position restore on open.
Verified live: ingest of the Libation sample (Gardens of the Moon —
title/author/series/cover/date from sidecar+folders, 34 chapters,
26h duration, aac/LC); chapters endpoint; the reader route serving
the player; playback with position restore (0:57 round-trip);
progress rows in audio mode; wall-clock session rows in
reading_history.
|
||
|
John O'Keefe
|
fd6f0e3353 |
fix(web): scan progress — correct percentages, no layout shift, refresh-survivable
Bug: the admin 'Scanning Libraries' widget fed the server's 0-1 progress straight into percent strings and bar widths, so the overall and per-library percentages capped at '1%' and the bars never filled. Both now scale by 100 (matching the setup page, which was already correct). Bug: scan state only reached clients via websocket events, so a page opened (or refreshed) mid-scan showed nothing until the next tick — and the admin widget never re-attached at all. The worker now exposes active jobs (with their library_id so rows keep their names), served as GET /api/scanner/active (admin, re-attaches the widget + polling) and GET /api/scanner/active-summary (any user, restores the sidebar indicator on load). Idle hides both, exactly as before. Annoyance: the sidebar's 'Scanning… N%' was a block element between the logo and the nav, shoving every menu entry down while a scan ran. The percent now lives inside the fixed-height logo row next to the spinner (spinner last, so growing digits extend leftward); nothing below it moves. |
||
|
John O'Keefe
|
0914c935d9 |
feat(dashboard): collapsible This Week widget, library-scoped + switch-surviving; title into the subbar
Three dashboard complaints, one build: - The widget collapses to a single summary row (This Week · minutes · pages · sessions · books · streak · Full stats) on every screen size and expands to the full grid via a details/summary toggle — the carousel sections start immediately below it again. - The 'Your Library' heading moved into the library switcher subbar (LibrarySwitcher gains an optional title param; collections/series keep the plain 'Library' label). The standalone heading block is gone. - Root-caused the JS-switch vanishing: renderDashboardCollections replaces ALL of #collections-container's innerHTML, so anything server-rendered inside it (title, widget) died on the first library switch. The widget now re-fetches ITSELF: a /dashboard/week-widget fragment (same resolveLibrary family-lens rules as the page) is pulled by htmx whenever the switcher dispatches week-refresh — carrying the current #library-select value — so the widget both survives switches and RE-SCOPES to the switched library. Initial SSR renders the same scope the sections use. ThisWeek (and the streak calendar) gained an optional library filter. Tests: TestReadingStatsWeekLibraryScope — per-library totals, scoped streaks, invisible-library isolation, malformed id 400. Full suite PASS. |
||
|
John O'Keefe
|
5a2c1585ae |
refactor(stats): 'Web / unsynced' bucket label → 'Web'; analytics tests isolated from the live reader
The bucket name read as confusing — 'unsynced' implied data loss when the push is always captured; it only lacks a device credential. The bucket is just 'Web' now (service fallback, stats.ts fallback, panel copy, tests). Also: analytics_test exact-zero assertions (EmptyHistory, NoReadingHistory) ran against testuser, which is a LIVE reader — real sessions accrued on it mid-suite and both flapped. They now run under the fresh regular user, like the stats tests; the 'testuser is a live reader' rule is recorded in AGENTS.md. |
||
|
John O'Keefe
|
7a604c2fec |
feat(stats): per-device reading breakdown + device attribution on progress syncs
- Attribution: registered clients may now send their device token in
X-Device-Token on PUT /api/media-items/:id/progress (the user JWT
stays primary). The token must map to a device OWNED BY the caller —
foreign/invalid/absent tokens soft-fail to plain web attribution so a
stats decoration can never break a sync. Attributed pushes capture
device_id (FK-safe), device_metadata.source = the registered device
name ('Pixel 9', 'Kindle Paperwhite'), and last_sync surfaces show
that device. Side effect worth having: app-vs-web pushes used to be
LWW-invisible (both sources read 'web'); they now surface as
legitimate cross-source progress conflicts when they diverge.
- /stats gains a By Device panel alongside By Library (grid 3-wide):
reading minutes + sessions + books per registered device, plus one
Web / unsynced bucket for device-less pushes so the panel always
reconciles with the headline totals. Range-aware, family-lens
scoped; renderBars shared between panels in stats.ts.
- The device identity IS the registered platform: devices rows already
carry the name/type the client registered (app sends Build.MODEL +
'mobile' at /api/devices/register, KOReader/Kobo likewise) — no new
identity plumbing on the server.
Tests: capture attributed to the device (row device_id + source
metadata + last_sync_source), foreign/nonexistent tokens soft-fail,
device buckets reconcile (80 attributed + 75 unsynced = 155 total).
Full suite PASS.
|
||
|
John O'Keefe
|
6457fbaf72 |
feat(stats): reading statistics — session capture wired + /stats page + widget + per-book card
The design parked in the handoff (multi-tier stats off reading_history; bookorbit-style in data, not looks) is in. The real prerequisite work was data: reading_history existed but NOTHING wrote to it — the whole stats feature was sitting on an empty table. - CAPTURE (no client changes, no repo-boundary crossings): every successful progress write through SaveProgress (KOReader + app + web + Kobo + offline queue — the one chokepoint) now records one reading_history session row: span = the interval since the PREVIOUS write instant, pages = positive page delta when both sides know the page index, marker rows for first contact. Spans over 2h are kept as zero-second markers (overnight suspend→resume, offline flushes must not inflate totals) — the honest 'sync cadence, not a stopwatch' caveat stays. device_id only when a REAL device row matches (the web/app progress path historically stuffs the USER id into DeviceID — probe instead of trusting it; FK violation would silently drop every capture). Source preserved in device_metadata. - API: /api/stats/summary?range=7|30|90|year (totals, zero-filled daily arrays, streaks, most-active day/weekday, top books, library breakdown — one call), /api/stats/this-week (widget: rolling 7d vs previous + streak), /api/stats/books/:id (lifetime + recent sessions). All bucketed in the user's timezone; finished ≡ any push ≥0.995; visible-library lens enforced at source. Legacy /api/analytics/* kept untouched. Range windows anchor end-of-day to the START of today (blasting the clock time into tomorrow resolved 'today' a calendar day late — found by the streak test). - /stats page (sidebar Stats between Progress and Devices, previously /analytics was an orphan): range selector, five stat cards, two Chart.js v4 charts (daily minutes bar, pages/day line — chunked via dynamic import so main.js stays untouched), per-library breakdown, most-read list, empty-state explainer. /analytics now 301s to /stats. The orphaned analytics.templ + analytics.ts removed (their containers and the page JSON never matched; the page was astub). - Dashboard 'This Week' widget (server-rendered, suppressed on service failure) and the book detail 'Your Reading Stats' card (lifetime totals of the viewer's own sessions) complete the tiers: per-user default, per-library breakdown inline, per-book on the page, per-system/admin aggregate parked as before. Tests: statistics_test.go (capture semantics at the chokepoint, tz/day buckets, zero-fill, streak math, finished detection, range validation, auth gate, widget, per-book lifetime). Full suite PASS. |
||
|
John O'Keefe
|
0a7fbd6f48 |
feat(collections): smart (dynamic) collections — rules evaluated live at render
Type is DERIVED, not stored: a saved ruleset makes a collection SMART,
no rules = STATIC (stored collection_items links, unchanged behavior).
No schema change — collections.auto_assign_rules JSONB already carried
the config; user's decision 2026-10-14 (handoff: live-at-render, not
refresh-on-scan; create-time Static/Smart toggle; v1 NO hybrid).
- Rule gains an enabled flag (absent = enabled, legacy rules unaffected).
- Rule CRUD: POST/PUT(DELETE) /api/collections/:id/rules[/:ruleId] — the
survivor dead effort of the rules-builder UI finally has real
endpoints (rules were never editable after creation; nothing ever
re-evaluated them). First rule converts a user collection to smart;
removing the last reverts it to static. UpdateCollectionRulesJSON
edits ONLY the rules doc so rule CRUD cannot clobber metadata.
- Live evaluation: candidates from GetVisibleLiveMediaForRules —
visible-library set (family lens enforced at source) + archived IS
NULL AND missing_scan_count = 0 — any enabled rule matches, title
order. Zero refresh hooks by design: always current by construction.
- Membership mutations (add/remove/bulk-add) are 400 on smart — v1 no
hybrid; collection_items.excluded stays the reserved future hook.
- Rules builder page (/collections/:id/rules) first time ever routable;
templ whole-attribute interpolation ("{ expr }" inside a quoted attr
value is literal text, as theme-{ user.Theme } silently proved).
- UI: create modal Static/⚡Smart cards (smart create redirects into the
rules builder), ⚡ badge + real per-card counts on the grid (smart =
live matches, static = stored-link count), smart detail renders
matched books with no hand-pick actions + an Edit Rules button.
- API: GET /:id gains is_smart + live books; GET /: keeps the Android
app's default exclusion, include_auto=true surfaces smart rows with
live counts; TestRules previews against the SAME candidates the live
render uses (visible + live, was raw first-1000).
- FIXED pre-existing regression from
|
||
|
John O'Keefe
|
431fec05ad |
feat(security): library visibility enforced at the source; per-library series; annotation ownership; admin user creation
The family-lens hardening: library_visibility was only enforced where file bytes flowed (download/reader/OPDS) — metadata surfaces leaked hidden libraries to any authenticated account, and registration was open by default with no admin-side user creation. - Series are grouped PER LIBRARY now (same name in two libraries = two entries: manga adaptations / LN splits no longer merge); /api/series aggregates only the requester's VISIBLE libraries and /api/series/books REQUIRES a visible library_id (breaking contract; web + app clients updated in the same change). - /api/media-items, item detail, and the reader content endpoints (chapters/panels/outline/THUMBNAILS — page images leaked unguarded next to a gated GetPage) all enforce the visible set; resolveLibrary no longer honors a hand-typed library_id outside the visible set. - Annotation/note/bookmark mutations (update/delete/tombstone, sync LWW paths incl. conflict resolution) carry ownership predicates: another account's bare row id is a 404, not a silent delete. - Admin user creation: POST /api/auth/users (AdminMiddleware) + a Create User card in the admin Users UI — accounts stay creatable with self-service registration OFF, which is now the schema default (fresh installs; the first-admin bootstrap exception is unchanged). - Tests: series contract re-pinned (per-library, visibility 403s), new ownership + admin-create integration suites, OPDS all=1 test now passes an explicit page size (shared-DB growth made the fresh books fall off page 1), device-cap helper switched to the admin endpoint. |
||
|
John O'Keefe
|
208fc2c4d6 |
fix(server): kobo markup panic — wire LibraryService into the test harness, guard typed-nil resolver
TestProgressWeb_KoboIntegration 500s'd on every markup with a last-read-place bookmark: the panic-recovered stack was a nil pointer dereference in LibraryService.ResolveMediaPath (s.db deref), reached from KoboHandler.convertKoboCFIToStandard during kepub CFI conversion. Two-layer bug: 1. Test harness (root cause): the router.Config built in test_helpers_test.go omitted LibraryService, so router/sync.go's SetLibraryService(cfg.LibraryService) stored a nil *services.LibraryService in the interface field. Production wiring (main.go:246) was always correct. 2. Latent trap (hardening): `h.libraryService != nil` at the convertKoboCFIToStandard call site can never catch a typed nil — an interface holding a nil *LibraryService is non-nil. ResolveMediaPath now degrades with a "library service not configured" error instead of panicking on a nil receiver; callers already handle the error by skipping CFI conversion. Harness-only fix, zero production behavior change: the endpoint now returns a clean 200 with the position saved wherever kepub conversion is unavailable instead of a panic-recovered 500. Verified: TestProgressWeb_KoboIntegration passes all three subtests (401 unauthenticated, ReadingSync→last-read-place percentage preservation, last-read-place without prior ReadingSync) and on immediate rerun — previously it failed every run. |
||
|
John O'Keefe
|
7ce73213ae |
feat(server): system collections become permanent — soft-delete + in-place restore + query_type-keyed provisioning
The old model hard-deleted system collections and re-created them at the tail, resurrecting stale dedups and breaking ordering. Redesign (user-directed B-final): - schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER — the whole schema is re-applied on every boot via executeSchema). - queries.sql: every collection list/read query filters deleted_at IS NULL; new GetSystemCollectionByQueryType :one (unfiltered, sees tombstones), TombstoneSystemCollection :exec, ClearCollectionItems :exec, RestoreSystemCollectionRow :one (in-place metadata reset + un-tombstone; collections have no updated_at column — ResetUpdatedAt was a 500 caught by tests). sqlc regenerated. - collection_service.DeleteCollection: system collections are tombstoned (preserving position/ownership); user collections keep the hard delete. No ErrNoRows to the client (delete is idempotent). - dashboard_service.RestoreSystemCollection: resolve by query_type; found → in-place restore (ClearCollectionItems + metadata reset, un-tombstoned); missing → provision fresh. "Want to Read"/etc. stay at their home positions forever. - handlers/collections.CreateCollection: name-taken is 409, not 500. - handlers/auth.CreateDefaultCollectionsForUser: idempotent by (user_id, query_type) — update-in-place when a row for that query type exists (heals renames/tombstones), insert when missing; this also fixes the duplicate-system-collections regression at source. |
||
|
John O'Keefe
|
25d44452c0 |
feat(dashboard): global (All Libraries) preferences scope
The dashboard's All Libraries view could render sections across every library but could not persist its own preferences: the PUT required a UUID library_id, and the prefs GET-without-library_id matched an arbitrary library's row through a loose NULL OR-clause. Storage: a dedicated user_dashboard_preferences_global table (one row per user, user_id PK → users CASCADE) instead of a NULL/sentinel library_id — chosen over both because (a) the column's FK to libraries rules out a sentinel UUID without dropping the constraint or minting a fake library row, and (b) intent becomes structural: only the explicit global path writes that table, so a buggy or outside-the-API write can never masquerade as global. System and user collections are user-scoped, so the order/hidden arrays are library-independent and valid in this scope. API: PUT /api/dashboard/preferences binds library_id as *string — JSON null/absent = global, empty string = 400 (a lost value must stay loud, not silently clobber the global scope), valid UUID = per-library (unchanged). GET without library_id now returns the global row instead of an arbitrary one; the per-library query's dead NULL OR-arm is removed. GET /sections keeps requiring library_id for per-library views and aggregates across libraries when absent (unchanged). Tests: global save/update-dedupe/read round-trip, empty-string 400, global-vs-per-library independence. Two suite failures are the handoff's documented pre-existing rot (duplicate system collections in the dev DB; a sections test expecting a 400 the handler never implemented) — reproduced and confirmed unrelated to this change. Live-verified on the dev stack: null save 200 x2 with a single row updated, empty string 400, global read carries the second save. |
||
|
John O'Keefe
|
fb21d5249c | fix(rescan): updateMediaItem re-derives format flags — force rescan reclassifies rows | ||
|
John O'Keefe
|
72839921f2 |
drop DOC and LIT from the pipeline (deliberately unsupported)
.IMAGE — extensions removed from AllowedExtensions (scan gate), bookExtensions, MimeTypes and sync/format.go maps: .doc/.lit files are no longer scanned or indexed. classifyFormatGroup gains the RTF and PDB reflowable arms that were missing when those rungs shipped (rows only reclassify on creation). Scanner docs updated. DOC + LIT have no credible JS tooling (mammoth is docx-only; LIT needs LZX and its DRM variants are dead) and text-extraction-only support would misrepresent what the reader can do. Verified: probe .doc/.lit files watched but never scanned; probe .txt control scanned; TestClassifyFormatGroup extended and passing. Docs change granted in-session by the user (extensions + format docs). |
||
|
John O'Keefe
|
f2629942e4 |
fix(scanner): classify .docx as reflowable — was landing format_group unknown
.docx is accepted for ingest but the format-group switch had no case for it, so every DOCX row stored format_group 'unknown' with is_reflowable=false. Extract the classification switch into a pure classifyFormatGroup helper with a table test, and classify .docx reflowable like the other reflowable text formats (.rtf/.doc stay out until they are readable client-side). Existing rows: dev alice (DOCX) reclassified by hand; production rows reclassify on rescan. |
||
|
John O'Keefe
|
3d88f0b960 |
feat(search): publisher_filter for the media search endpoint
Adds publisher_filter to SearchMediaItems/SearchMediaItemsUnified (fuzzy word_similarity against mi.publisher, mirroring genre_filter), plumbs it through services.SearchParams and the search handler, and extends the GREATEST relevance ranking to include publisher matches. Serves the app's author/publisher/genre/tag click-through browses (UX pass item 5 — publisher was the only facet without a server filter). |
||
|
John O'Keefe
|
64ef691d64 |
feat(scanner): repoint rows on cross-library moves of identical content
Previously the scanner's SHA-256 dedup was library-scoped: moving a book between libraries created a duplicate row (new ID) while the old row went missing and archived after two scans, orphaning reading progress and annotations from the file that users still see. processMediaFile now falls through to cross-library detection when the same-library hash lookup misses: - GetMediaItemsBySHA256AndSameLibraryType returns candidates in other same-type libraries; each candidate's file is stat'd through ITS OWN library's folders (the old existence check stat'd against the current scan's folder tree, which is meaningless across libraries). - File gone -> it is a move: the candidate must pass the target library's type rules (ValidateMediaItemForLibrary), then the row is repointed via MoveMediaItemToLibrary with the recomputed relative path. Reading history, annotations, and collections follow automatically because the row keeps its ID. Incompatible formats (e.g. a reflowable EPUB into a manga library) are rejected with a processing issue instead of being force-imported; the upsert on (media_item_id, issue_type) keeps repeated scans from spamming duplicates. - File still present -> deliberate multi-library copy: fall through to normal import so both libraries keep independent rows. Candidate selection is factored into selectMoveCandidate (pure function, unit-tested in media_scanner_move_test.go): input arrives pre-ordered (archived first, then most missing scans, then oldest) and only candidates whose file is verifiably gone qualify, so deliberate copies are never repointed. |
||
|
John O'Keefe
|
3ab294bf81 |
feat(scanner): store EPUB reading direction at scan time
Release / build-and-push (push) Successful in 2m33s
Fixed-layout EPUBs lean on the reading_direction column (the web reader forces book.dir = rtl from it when the file didn't set direction itself), but the scanner never populated it for EPUBs - only ComicInfo fed it. Meanwhile real Japanese EPUBs declare page-progression-direction on the OPF spine, which foliate reads client-side but nothing stored. Read the spine attribute in the structured OPF parser and map it into ReadingDirection in parseOPFContent (EPUB2/3, case-insensitive, plus 'right-to-left'/'left-to-right' spellings); undeclared stays empty rather than forcing ltr, preserving the editor's Auto default. Sidecar OPFs are metadata-only documents without spines, so the Calibre path is a no-op. The merge gap-fill copies an embedded-only direction into a blank sidecar field, and hand-set values keep winning through the existing OverrideReadingDirection protection. Tests: declared rtl/RTL/ltr, undeclared and unknown values staying empty, plus sidecar-wins vs embedded-fills merge cases. Existing manga EPUBs declaring rtl (verified live in-library) pick the value up on their next scan, feeding the API and reader config mobile clients consume. |
||
|
John O'Keefe
|
e944f11415 |
feat(scanner): fill sparse sidecars from the PDF Info dictionary
Same gap-fill rule as the EPUB path, against the PDF's embedded Info dictionary via a readPDFInfoDict helper reusing extractPDFMetadata's field conventions (creator falls back to author, subject maps to description, producer to publisher, keywords to tags, plus page count). Sidecar values always win; unopenable PDFs skip silently. TestMergeMetadataPDFGapFill uses an Info-bearing hand-built PDF fixture (shared with the sidecar-cover test) and asserts the sidecar title is kept while author/description/publisher/tags/page count fill in. |
||
|
John O'Keefe
|
708598687e |
feat(scanner): fill sparse sidecars from embedded EPUB metadata
A sparse metadata.opf/metadata.json (title only, no description) left books thin even when the file itself carried rich data: the embedded extractors only ran when no sidecar existed at all. Now mergeMetadata fills blanks from the book's own OPF - title, author, description, publisher, language, ISBN, ASIN, series/number, publish date, tags, contributors - while sidecar values always win and unparseable files skip silently. Also covers .kepub, which the merge previously ignored while the extractor already supported it. Adds TestMergeMetadataEPUBGapFill asserting both directions: sidecar title/author survive, embedded description/publisher/language fill in. |
||
|
John O'Keefe
|
249b1dfe93 |
feat(scanner): treat reappeared content as a move, not a duplicate
When the SHA-256 dedup found identical content already in the library, the scan skipped the file as a duplicate - and after files moved between folders the old row kept its stale path, cycled missing -> archived, and the new path never took. The archive feature turned the old destructive move behavior into a stuck move instead. Now the dedup branch stats the old location: if it is gone, the book was MOVED, so the row is repointed (file_path, file_size) with archive state cleared and reading history intact. 'Skip as duplicate' only applies when the old path still exists (a true copy). Verified live: a moved EPUB kept its single row, followed the file, and never entered the missing/archive cycle. |
||
|
John O'Keefe
|
841db91a29 |
fix(scanner): extract embedded covers for sidecar-managed books
A full rescan wiped cover_image_path for every PDF/EPUB living in a
metadata.json (or metadata.opf-less) folder with no cover.jpg next to
the book: the sidecar branches returned early after findSidecarCover
missed, and mergeMetadata has no PDF/EPUB cover logic of its own. Four
books lost their thumbnails while their {file}.cover.jpg files still sat
on disk - most visibly the Audiobookshelf-managed No Starch titles.
Both sidecar branches now fall through to an embedded-cover fallback
(PDF via extractPDFCover, EPUB/KEPUB via extractEPUBCover) whenever no
sidecar cover file exists. Comics are untouched: mergeMetadata already
extracts their covers from the archive.
Adds TestSidecarCoverFallback with a hand-built one-page PDF carrying a
JPEG XObject plus a metadata.json sidecar, asserting the sidecar title
wins while the cover still comes from the file. Verified live: rescans
restored all four dereferenced covers with no leftover rows.
|
||
|
John O'Keefe
|
1eb9c92d6a |
fix(scanner): nil metadata_overrides crashed Reset to Scanned with SQLSTATE 23502
Release / build-and-push (push) Successful in 2m30s
Reset to Scanned cleared the overrides row (successfully) but then set
the in-memory copy to nil before handing it to updateMediaItem. pgx
encodes a nil []string parameter as SQL NULL, so the follow-up UPDATE
wrote metadata_overrides = NULL into the column's NOT NULL constraint
and the whole rescan failed with:
failed to update media item: ERROR: null value in column
"metadata_overrides" of relation "media_items" violates not-null
constraint (SQLSTATE 23502)
Two changes:
- RescanMediaItem's reset path assigns []string{} instead of nil, with a
comment explaining the pgx nil-to-NULL encoding trap.
- updateMediaItem routes the override set through utils.MergeOverrides,
whose contract guarantees a non-nil slice, so no caller can write
NULL into that column again (verified against pgx v5.9.2 source: a
scanned '{}' round-trips as non-nil in both directions; the nil could
only come from our own assignment).
The plain Rescan path never hit this - only Reset did. Worse, the reset
is the remedy when a book's cover_image_path override pins an empty
cover, so the crash also blocked the way out of that state. After this
fix, a plain rescan on an already-reset book repopulates scanned
metadata and extracts the cover.
|
||
|
John O'Keefe
|
1fa8ee3a59 |
feat(scanner): Calibre-aligned OPF metadata extraction
Release / build-and-push (push) Successful in 3m5s
Adopt Calibre's reading conventions for the Dublin Core metadata that
parseOPFContent now pulls from the structured OPF parse:
- Titles: EPUB3 title-type selection (prefer 'main', join a distinct
subtitle with ': ' exactly as Calibre stores it). There is no separate
subtitle column by design - Calibre-sidecar books arrive pre-joined,
so a column would stay empty for most libraries and force every client
to reimplement concatenation.
- Genre: first dc:subject, mirroring the existing processGenresAndTags
behavior of the Calibre-sidecar path; the embedded path never
populated Genre before. Subjects stay one-element-one-tag - Library
of Congress headings legitimately contain commas ("Holmes, Sherlock
(Fictitious character) -- Fiction") and must not be split.
- Identifiers: urn:isbn:/urn:asin: prefixed values parse in addition to
opf:scheme attributes, and the scheme-less fallback now requires an
ISBN-shaped value (10/13 digits, optional separators/trailing X) so
URIs like the Gutenberg identifiers cannot masquerade as ISBNs -
observed live on 'A Study in Scarlet'.
- Series: EPUB3 belongs-to-collection with collection-type=series and
group-position refines, ahead of the classic calibre:series metas.
- Audiobookshelf metadata.json sidecars join their subtitle field into
the title the same way.
Tests cover title-type main+subtitle joining, belongs-to-collection
series with fractional group-position, urn:isbn extraction, genre/tag
parity, and comma preservation inside subject headings.
|
||
|
John O'Keefe
|
72d167005f |
fix(scanner): resolve EPUB covers via structured OPF parsing, Calibre chain
The cover lookup scraped the OPF with attribute-order-sensitive regexes. Real books serialize attributes in any order - Grand Central's '3 Days to Live' puts href before id on manifest items and content before name on the cover meta - so all three regex paths missed and the book fell through to filename guessing, extracting no cover at all. Attribute order is meaningless in XML; the regexes were never safe. Replace them with a structured parse (encoding/xml, namespace and attribute-order agnostic; see the new media_scanner_opf.go) and follow Calibre's read_raster_cover resolution order: 1. manifest item with properties=cover-image (non-(X)HTML media only) 2. <meta name=cover> resolved through the manifest, same media guard 3. first spine item that is itself a raster image (store manga) 4. NEW cover-page fallback: books declaring no raster cover at all - the classic EPUB2/Adobe cover.xhtml wrapper - are mined for <img src> / SVG <image xlink:href> references (Calibre renders the page with Qt; extracting the referenced image covers the practical cases without a rendering engine) 5. existing zip filename guessing stays as the last resort, and the old regex chain survives as findCoverInOPFLegacy for OPFs too malformed for a real XML parse. Hrefs are now URL-decoded and posix-normalized against the OPF's own path (path.Join semantics), so '../art/cover.jpg' from a nested cover page and %20-encoded names resolve correctly. Tests: attribute-order chaos modeled on the failing Patterson book, SVG-wrapped cover pages via guide references, image-first spines, and path resolution edge cases. Verified live against the real '3 Days to Live' EPUB, which previously produced no cover. |
||
|
John O'Keefe
|
14445a7c3f |
feat(scanner): archive-at-two-scans lifecycle for files missing from disk
Replace the silent hard-delete orphan cleanup (which logged only through ScannerLogger file logs and whose failure paths left rows undetected) with an archive lifecycle that preserves reading history: - A file missing in one scan is marked (missing_scan_count = 1); missing in a second consecutive scan archives it (archived_at, hidden from browsing, progress/notes/highlights survive). Every branch logs to stdout with an [ARCHIVE] prefix so skips are always visible. - When a file reappears - same path, or identical content at a new path via the SHA-256 dedup match - the archived state clears automatically and the item returns with its history intact. - Archived rows older than ARCHIVE_RETENTION_DAYS are hard-purged at scan time (cascading deletes); 0 disables auto-purge for manual-only management. Retention is read from the environment in NewMediaScanner. |
||
|
John O'Keefe
|
44b98f3fc3 |
feat(scanner): read Audiobookshelf metadata.json sidecars
Libraries managed by Audiobookshelf keep a metadata.json next to each
book (title, authors, series+sequence, genres/tags, publisher,
description, isbn/asin, language, published year/date) - and no
metadata.opf. The scanner silently ignored those files: deleting them
changed nothing, and their data never reached the database.
Parse them as a first-class sidecar in extractMetadata, priority
metadata.opf -> metadata.json -> embedded media. Only fields with a
matching media_items column are mapped; narrators, subtitle, explicit,
abridged, and chapters are deliberately skipped.
Cover handling is unchanged: the existing findSidecarCover priority
(cover.jpg / folder.jpg / {basename}.jpg) applies to the sidecar branch
exactly as it does for Calibre.
|
||
|
John O'Keefe
|
61681aac23 |
feat(scanner): extract EPUB metadata from the embedded OPF directly
go-epub's ReadBook parses every spine chapter and fails the entire call if any single chapter (or the TOC) is malformed, discarding already-parsed OPF metadata. For books like Pragmatic's 'A Common-Sense Guide' the OPF holds good title/author/publisher/ISBN metadata that rescans then wrote as blanks - success toast, no (visible) change. Refactor to parse the EPUB's own OPF document with the same Dublin Core machinery used for Calibre sidecars: parseCalibreMetadataOPF is now a thin file wrapper around a reusable parseOPFContent([]byte), and the OPF lookup previously inline in extractEPUBCover is shared via findOPFPathInZip. Since metadata never touches chapter bodies, chapter damage cannot blank it. Also picked up along the way: dc:language mapping and scheme-less dc:identifier values that normalize to a valid ISBN (EPUB3 style). Tests cover a Pragmatic-style EPUB (dc namespace on <metadata>, no identifier scheme, deliberately malformed chapter) that must still yield full metadata, plus series/date/subject OPF parsing. |
||
|
John O'Keefe
|
7f64b92b9d |
feat(metadata): per-field user overrides that survive library rescans
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
|
||
|
John O'Keefe
|
9c8337d0a8 |
fix(scanner): render the PDF CropBox, not the MediaBox, in the cover fallback
pdftoppm defaults to rasterizing the MediaBox, while PDF viewers (pdf.js in the reader, and every other viewer) display the CropBox. For PDFs whose page 1 is the full print cover wrap (back cover + spine + front cover in one landscape page) with a CropBox covering only the front cover - e.g. No Starch's XeTeX-built 'Algorithmic Thinking' - the fallback stored the entire spread as a squashed landscape cover, while the reader correctly showed just the front cover. Pass -cropbox so the rendered cover always matches what the reader displays. Poppler falls back to the MediaBox when a PDF defines no CropBox, so PDFs with identical boxes (the common case) render exactly as before. |