a42c6d4f75b09dc5a7046e9d7acfa1bd9e6bf02b
245
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
a42c6d4f75 |
feat(scanner): force rescan = original scan; feat(filters): the manually-edited filter (faceted plan Phase 13)
Release / build-and-push (push) Successful in 3m7s
Force rescan contract change (owner-directed): the editorial-keep skip
is retired. Force rescan now re-derives EVERY field on EVERY book from
all sources — multi-file books included (the folder/zip flows ran the
full chain on create and on manifest-flagged edits, but skipped the
sidecar apply under force). metadata_overrides is the only protection:
ApplyMetadataOverrides re-applies manual values inside the writer, so
manual edits survive; everything else re-derives.
- A force pass also STAMPS each re-derived book's manifest with the
current parser version (success-only — same truthfulness rule as
Phase 10) and closes its open processing issues (Phase 11 self-heal).
Force rescan and the ev-bump trickle now converge: either leaves the
library fully current; the deploy order is bump-first, then Rescan
All (documented in the plan and contract_pins).
- Single-file force rescan already ran the full chain — unchanged,
now pinned.
NEW filter field manually_edited ("Manually edited", boolean):
cardinality(mi.metadata_overrides) > 0 / = 0. Composed-engine (qf)
only — the builder's Book facts group gains it with is true/false;
no value menu, no suggestion surface; family lens inherited.
Tests: multifile force applies sidecar metadata + stamps + override
protection; single-file pin; the filter both directions scoped to the
test library; adjacent suites (Audiobook, Calibre, ExtractorVersion,
CrossLibraryMove, ProcessingIssues, FilterBuilder, SchemaIdempotency)
green. The extractor mechanism itself is untouched — the owner bumps
scan_extractor_version manually via the admin UI (plan Phase 13 §3).
|
||
|
John O'Keefe
|
9b5defcb31 |
feat(issues): the honest processing-issues ledger — row-less import failures, self-healing resolution, family lens (faceted plan Phase 11)
processing_issues only recorded cross-library move rejections; files that failed to import vanished into an error counter and stdout. The ledger now records every failure class and heals itself: Schema (in place, pre-release boundary): - media_item_id NULLABLE + file_path TEXT (deliberately unbounded — an over-long path is itself a failure mode; a VARCHAR(500) ledger column died on the exact 22001 it existed to report — caught by the new test). Row-less failures key on (library, file_path); item-backed on (item, type). Widened dedup = COALESCE-expression unique index; the old table UNIQUE stays (NULLs pairwise-distinct for row-less). - Conflict-target upsert: repeat failures UPDATE the same row and REOPEN resolved ones — never duplicates. Wirings: - CreateMediaItem failure (single-file + multifile createAudioBookItem) → row-less import_failed (error). - Extraction hard-fail with fallback import → item-backed metadata_extraction_failed (warning). - Change-path re-extraction failure (the ev/sidecar guard) → item-backed metadata_extraction_failed; LogProcessingIssue keeps its signature, a row-less helper joins it. Auto-resolution: - Successful import → ResolveOpenProcessingIssuesForPath (library, file_path). - Successful (re)extraction → ResolveOpenProcessingIssuesForItem (single-file change path, multifile editorial apply). - Unchanged-skip scans do NOT resolve: extraction never re-ran, so the degraded metadata is still degraded — the page heals exactly when the book does. Family lens on ALL issues surfaces (list, stats, resolve, delete, SSR page): explicit out-of-set library id → 403, never silent. This tightens one pre-lens behavior — a nonexistent library id returned 200-empty, now 403 (pinned, changelog entry in docs/contract_pins.md). List is LEFT-JOINed so row-less rows appear (media_item_id "", title falls back to the path). Tests: full lifecycle state machine through real scans (fail → appears, repeat → no dups, success-at-path → resolved, LEFT-JOIN list shape); family lens 403s with a fresh non-shared admin; all 7 pre-existing issue suites green (two expectations widened to the lens, changelogged); adjacent suites (CrossLibraryMove, Audiobook, Calibre, ExtractorVersion, SchemaIdempotency) green; unit suites green. |
||
|
John O'Keefe
|
19b9c7c9b8 |
fix(tests): wire settings registry into the test auth handler — unstick TestAdminCreateUser
setupTestServer built the settings registry (and wired it into
systemSettingsHandler) but never called authHandler.SetSettings, unlike
main.go:112. The auth handler therefore ran with a nil registry and
defaultTimezone() took its nil-fallback branch, returning "UTC" for every
Register/AdminCreateUser in tests regardless of the system default_timezone
setting — so the
|
||
|
John O'Keefe
|
91b0c45dd4 |
feat(scanner): extractor-version (ev) mechanism — parser fixes reach old rows without a force rescan (faceted plan Phase 10)
Parser fixes (
|
||
|
John O'Keefe
|
ab713e2feb |
fix(filters): blank-typed browse 500 on id-valued fields (library/collection/added_by)
The browse-on-click blank sentinel was trimmed to empty (correct), but the typed query had already been bound unconditionally — the id-source suggestion query has no score CASE to reference it, leaving an unused parameter Postgres rejects (42P18). Every click of an empty library / collection / added-by box answered 'No matching values' via the error path. - ComposedFieldValues binds the typed query ONLY in branches that reference it and only when non-blank; the score expression is '0 AS score' for browse (no param) and the similarity CASE when typing. composedIDSourceValues takes the raw typed string and binds it only when the name filter is built. userBind already followed this discipline; audited every bind site in the file (fuzzy helpers, composed search, count) — all interpolate immediately. - Tests: library browse under rules (exactly the surviving library, count pinned), series browse blank, added_by browse blank (no error), collection browse blank (offered with count) and under a zero-match rule (never offered). Verified end-to-end against the live dev data: author+Sapkowski AND narrator+Kenny → Audiobooks (8). |
||
|
John O'Keefe
|
9c006a9a2f |
fix(filters): browse with no rules works, popup under the input, no sticky headers (user report)
- Blank typed queries rode the STATIC autocomplete path for the classic six fields, where a single space similarity-matches into nothing — clicking an empty value box with no other filters said 'No matching values' in a library full of values. Blank = browse now always rides the composed engine (which trims and answers the top values); static keeps its pinned contract for real typed queries. Regression-pinned: browse with no rules lists the field's values. - The value popup now drops BELOW the input (top: 100%), not over it. - Field-menu group headers lost position:sticky — the search box alone stays pinned. |
||
|
John O'Keefe
|
c747f0a76c |
feat(filters): per-row AND/OR (Excel-style chips) + browse-on-click value menus (user report)
- Per-rule connectives: the chip before each rule flips that rule between AND and OR — OR-joined neighbors merge into an invisible OR-block, AND pulls them back out. The tree stays the single source of truth (implicit blocks are a client marker the server ignores), so deep links, saved filters, and the composed engine are untouched. Group-mode pills are gone; explicit groups remain as pure structure (+ group). Compile limit, same as Excel: flat chips express AND-of-OR-groups; a top-level OR of AND-blocks still needs nesting. - Browse-on-click: clicking an empty value box now fetches the field's top values UNDER the other rules (blank typed = browse, not a similarity filter) — 'assign an author, click Publisher, see their publishers' without typing. Server: the composed suggestion engine trims the typed query; client: the >=2-char guard no longer blocks the empty query, browse limit 15 vs 8 while typing. - Tests: the browse case pinned (blank sentinel + rules → the field's values under those rules, zero-match values absent). |
||
|
John O'Keefe
|
4ee24e7152 |
feat(filters): the rule builder — composed engines over the field registry (faceted plan Phase 9)
- Field registry (services/filter_registry.go): ~55 typed fields — every book column, file/lifecycle facts, the three HOLDING STRUCTURES (library, collections with excluded-aware membership, series with FULL primary+alternate_info membership), and the whole user-relative class (status, progress, last read, my rating, review, notes/highlights presence, note/highlight text, time spent, pages read, collections, added by) reading the VIEWER's rows. Operators are typed constants; negation is an operator. - Composed engines (services/filter_query.go): one Go clause builder over raw pgx feeds the search (identical SearchMediaItemsUnifiedRow shape), the field-aware suggestions, and a true-COUNT live count. Mandatory WHERE head everywhere (family lens + integrity facts). Values are always bound parameters — no user string reaches a field/operator position, ever. - Wire: the qf JSON param (validated strictly — unknown fields/operators/malformed values are 400); qf wins over flat params on hand-crafted URLs; flat deep links keep working unchanged. - Endpoints: GET /api/media-items/filter-count (live count); field-values autocomplete upgraded in place — any registry field key, qf binds the OTHER rules (symmetric under-filter suggestions, zero-match values never offered); classic six without a tree keep their static pinned contract. - UI: the drawer becomes the rule builder — the ONLY filter UI (templ markup, Alpine tree state, sanctioned island; grouped field menus, operator menus with red negations, field-aware value menus, live count, chips per rule with red negations, URL rehydration, smart serialization flat-when-simple else qf). - Tests: composed semantics (AND/OR/negation/nesting/enums/bools/ numbers/dates/empty), every user-relative field class with the my-data weld (two readers, different worlds), note-text substring pin (catharsis ≠ Carthage), family lens (hidden libraries nowhere, 403s), suggestion-under-rules exactness, qf validation + precedence, flat-vs-tree equivalence; registry completeness unit test. |
||
|
John O'Keefe
|
ebb7f4a5b0 |
feat(filters): panel removal + All-Books field-values fix (faceted plan Phase 8)
- Remove the Phase 7 facet-panel UI: SSR route, facet_panel.templ, FacetCountsForView, drawer panel block, panel TS + .facet-* CSS. The ENGINE stays (FacetCountsUnderFilters, all-facets endpoint, drift guard) — it becomes the Phase 9 builder's machinery. - Fix the All-Books bug: field-values autocomplete (author/genre/ tags/narrator/series/language) treats library_id as OPTIONAL — absent aggregates every VISIBLE library (family lens in-query); explicit out-of-set library_id is now 403, never a silent pass. The four scalar Search*Values queries get the nullable shape. - toast.ts: suggestion fetches are exempt from error toasts (console-only) — a failed background hint never interrupts. - TestFacetPanelPartial deleted with the behavior; TestFieldValuesAll- Libraries pins the fix (200 + cross-library aggregation, family lens, 403). Contract-pins changelog entry added. |
||
|
John O'Keefe
|
f52bb6f073 |
feat(facets): all-facets panel with under-filter counts (faceted plan Phase 7)
The 9 Batch A facets could not be SET from the web UI, the facets
endpoint had no in-app consumer, and the registry's libraryTypes hints
rendered nowhere. This phase closes all three.
- ONE UNION-ALL sqlc query (FacetCountsUnderFilters) computes all 14
listable facets in a single round trip: each arm omits its OWN
facet's filter block and carries every other registered facet's
block (copied mechanically from SearchMediaItemsUnified) — classic
faceted semantics: with filters active each facet counts under the
others; with none, blocks are neutral and the counts are the v1
global-per-library ones. One query serves both cases. The 14
per-column Batch C queries are deleted.
- Endpoint: no facet param → the new all-facets shape
({facets: {param: [{value, display?, count}]}}), filters bound via
the shared ParseMediaSearchParams, library_id optional (the
All-Libraries view, family lens still enforced); facet=X keeps the
pinned single-facet shape (now routed through the same query with
neutral filters).
- SSR partial GET /media-items/facet-panel: templ renders the drawer's
value lists — registry order, Batch D library-type scoping
(FacetsForLibraryType: manga_type on manga, reading_direction on
manga+comics, narrator on audiobooks; All view = union), prettified
genre labels + language display names, active values marked. Family
lens: out-of-set explicit library_id is 403.
- Drawer wiring: the panel fetches on drawer open and after every
apply with the CURRENT filter query string; a value click toggles
that facet's comma-joined filter (hidden input where the drawer has
no field — same as deep links) and re-applies the form; chips and
URL stay in sync. Values are server-rendered (rendering doctrine);
bookshelf.ts owns only the click/fetch orchestration.
- prettifyGenre moves to internal/utils as the canonical
implementation (services' display enrichment shares it; templ wraps
it; the TS port in bookshelf.ts is unchanged).
- Drift guard extended: FacetCountsUnderFiltersParams must carry every
registry facet's param, the bools as pgtype.Bool.
Tests: under-filter exactness (own-block omission, OR-within,
AND-across), family lens (hidden values absent; 403 on both shapes),
all-shape + pinned-shape coexistence, partial rendering with Batch D
scoping + active marking + 403. Full container suite green except the
documented pre-existing TestAdminCreateUser red.
|
||
|
John O'Keefe
|
6152137156 |
fix(tests): repair stale pins surfaced by the Phase 6 full-suite run
Two suite failures, neither caused by scanner code (the Phase 6 diff
touches no handler):
- search_test invalid_library_id expected the pre-family-lens 404;
|
||
|
John O'Keefe
|
2c5ad014c0 |
feat(scanner): multifile folder/zip sidecar detection (faceted plan Phase 6)
Reverses the Phase 4 scope cut: for EXISTING folder/zip audiobooks a
sidecar edit was invisible to everything, including force rescan —
updateAudioBookItem writes only scan-derived fields, and these formats
often have no useful embedded tags, so the sidecar is the only source.
- Change fingerprint gains the manifest compare in both processors:
changed := size differs || track payload differs || manifest drifted.
The ladder runs against the bulk-loaded map entry (stat pairs only,
zero extra queries); NULL stored manifest backfills without
re-extraction (deploy-day rule, same as single-file).
- Manifest-changed applies the once-extracted book metadata to the
editorial columns (updateMediaItemWithMetadata, overrides protected)
and re-runs the playlist hook. The apply runs BEFORE the focused
payload write: the generic update's format-group pass derives mime
from the sentinel (none) and chapters from metadata.ChapterCount
(unset) — the payload write must win (rescanMultiFileAudioBook's
ordering). Track-only changes keep the existing editorial-keep
update; force rescan keeps it too.
- Manifests seeded at multifile create (folder names derive from the
sentinel — the exact representative the extractor reads — via
detectSidecarChangeWithRepresentative, so manifest and extractor can
never disagree; a directory-basename cover is ignored by the
extractor and therefore not tracked).
- Zip parity fixes: the fast path now lifts archive state like the
folder path always did (a returned zip book stayed archived
forever), and the change branch handles the unchanged-archived case.
- Single-file sidecar change path: extract-failure guard — the old
non-nil empty &MediaMetadata{} fallback bypassed the update path's
filename-title guard (preExtracted == nil only) and would blank the
title on a failed extraction; the row is now left untouched and the
manifest refreshed. Empty-success extraction (sidecar removed from a
tag-less book) now passes nil so the create-equivalent chain (tags →
filename) re-derives instead of blanking.
- Multifile changes now count in the scan summary (changedItems);
they were previously uncounted.
Tests: sentinel-name derivation unit pin; folder sidecar edit
(title + collection sync + fresh manifest + steady-state skip),
track-only change keeps editorial, zip sidecar edit + archive
restore, sidecar removal falls back to filename (the wipe-guard's
testable sibling). Full container suite: green except the documented
pre-existing TestAdminCreateUser red.
|
||
|
John O'Keefe
|
6750e9e0dc |
feat(scanner): manifest-gated sidecar change detection + playlist sync on change path
Part 2 of the faceted plan, Phase 4 (steps 3-4). Size-only detection
was blind to metadata-sidecar edits; a size-stable book is changed by
EDITING its metadata.opf/metadata.json/cover — previously invisible
until a force rescan.
- Sidecar manifest fingerprints (media_scanner_manifest.go): the
manifest stats exactly the names the extractors consume (metadata.opf,
metadata.json, findSidecarCover patterns incl. <basename>.<ext>), so
manifest and extractors can never disagree. Shape v1:
{v, entries:[{n, s, m (unix nanos — second granularity would hide
edits landing in the backfill scan's own second), h (sha256 — covers
included: the repoint ladder needs a baseline hash for any stat-
drifted sidecar; a hashless cover turns harmless mtime drift into a
re-extraction, the exact flood this design kills)]}.
- Evaluation ladder in cost order: stat pairs identical → skip with
ZERO bytes read; stat drift → hash that sidecar (equal = host repoint
reconciliation, stat pair refreshed, no re-extraction; differing =
changed); NULL stored manifest → backfill only, NO re-extraction
(deploy-day rule: the first scan after release must not look like an
mtime flood).
- Change path: re-read full row, extractMetadata once and pass it
through updateMediaItemWithMetadata (preExtracted — metadata_overrides
stay protected), run syncPlaylistCollectionsAfterCreate (Correction 2:
a reading-order entry added to an EXISTING book now creates its
collection; add-if-missing is idempotent), store a fresh full
manifest. Creates seed their manifest at insert time.
- Multifile folder/zip books intentionally keep Phase 3 behavior: their
change path does not consume sidecar extractors, so gating them would
flip books 'changed' without picking anything up (plan correction —
the plan's §3 multifile note assumed otherwise).
- UpsertMediaScanState + GetMediaScanState queries.
Tests: media_scanner_manifest_test.go (ladder unit tests — fast path,
repoint, same-size drift, appear/vanish, unhashable legacy entries);
TestScanManifestLifecycle (seeding, unchanged-skip stability, repoint
reconciliation with stat-pair refresh assertion, same-size edit
consumed, playlist sync on the change path + idempotency, no-leak
guard). Container runner.
|
||
|
John O'Keefe
|
693053a9f5 |
perf(scanner): bulk scan-state load — one query per library scan
Part 2 of the faceted plan, Phase 3 (steps 1-2). The periodic scan
issued one GetMediaItemByFilePath per scannable file (~3,094 queries
per scan in prod, nearly all answering 'unchanged, skip') plus one
GetLibraryByFolder per file for folder resolution.
- NEW media_scan_state table: the scanner's per-item fingerprint
cache. Deliberately NOT a media_items column — the library-scoped
list emits raw sqlc rows as JSON (
|
||
|
John O'Keefe
|
7e586df48a |
feat(facets): facet registry, Batch A server params, genre model v2, facets endpoint
Phase 1 + backend of Phase 2 of the faceted-navigation plan. One
declarative facet registry (services/facet_registry.go) drives the
bookshelf filter system; a drift-guard unit test reflects over the
generated SearchMediaItemsUnifiedParams so a registry entry and its
sqlc.narg block can never disagree silently.
- Nine new filter params on SearchMediaItemsUnified: story_arc,
imprint, edition (scalar-fuzzy), age_rating, format_group,
manga_type, reading_direction (exact), explicit, abridged
(presence-bound bools). Comma = OR-within multi-select on every
text facet; AND-across-facets unchanged.
- Genre model v2 (user rule: a subgenre is not a genre, it's a tag):
genre_lexicon.go rebuilt as shelf set + subgenre->shelf map;
pickSmartGenre picks direct shelf > mapped subgenre > Fiction floor
> empty (unknown-only stays tags-only). Type-words (Manga, Comics,
Graphic Novel) join the demote list. Both 'adventure' and 'action &
adventure' are shelves — provider data uses both and the verified
traces pin them.
- Language filter UX: comma elements normalize through the ingest
language map, so display names ('English') match rows stored as ISO
codes ('en'). The language autocomplete now serves display names.
- NEW GET /api/media-items/facets?facet=X&library_id=... — registry-
driven value+count listing (14 static per-column queries, family-lens
join); language values carry display names; bool/range facets 400.
- Family-lens gates: an explicit hidden-library library_id is now a
403 on the search endpoints (was a silent empty set) — JSON branch
and HTMX grid partial both.
- handlers.ParseMediaSearchParams is the SINGLE param parser for the
search JSON endpoint and the HTMX grid partial; the HTML branch
previously dropped publisher_filter and narrator_filter silently.
Pins: docs/contract_pins.md changelog entries (additive request
params; new facets endpoint). Tests: facet_registry_test.go (shape +
SQL drift guard + library-type union), series_parsing_test.go genre-v2
cases, facets_integration_test.go (param filtering e2e, counts,
language display, 403s) — container runner.
|
||
|
John O'Keefe
|
7537063cb7 |
feat(filters+nav): author navigation end-to-end — arrays on every media response, deep-linkable bookshelf filters, clickable authors
Completes the A2c acceptance scenario: click an author on a book detail page -> land on the bookshelf filtered to everything that author wrote (including their share of co-authored books). - Media detail + unscoped list resolved maps expose subtitle, explicit, authors, publishers (the library-scoped raw branch and the search endpoint already did — raw rows follow the SELECT). - SearchMediaItemsUnified SELECT carries the new columns; the search endpoint serializes the row directly, so clients get them there too. - book_detail.templ renders each author as a link to /bookshelf?author_filter=<name>. - Bookshelf honors filter params in the URL on load (author_filter, series_filter, genre_filter, publisher_filter, tags_filter, narrator_filter, language_filter): pre-fills the form, applies it to the first grid load. Comma-joined values are multi-select via the server-side string_to_array split. Filtered views are shareable. Pins widened: search rows (+7 keys), resolved keys (+4); changelog entries in docs/contract_pins.md. Android follow-up plan written to ~/.opencode/plan/bookhoard-app-v2-compat.md. |
||
|
John O'Keefe
|
1dbb0ce406 |
feat(collections): playlist auto-creation, position ordering, hide_finished, Up Next, dashboard eligibility
Phase 2b + 3 of the metadata overhaul (plan §2.4, §3.1-3.7). Auto-playlist collections (§2.4): the scanner turns obviously-playlist sidecar series (explicit '(Reading Order)'/chronological/publication/ in-order/timeline markers only — precision over recall) into curated static collections: upsert by view_settings.imported_from_series then name (rename-safe), position set ONLY on insert (clean integers; 1.5/1a stay unpositioned rather than guessing), membership add-if-missing, never removed or re-ordered by rescans, provenance in description + view_settings, show_on_dashboard true (dashboard-eligible by design), 'scanner.auto_playlist_collections' setting (default on, Settings Registry + schema seed). Ordering (§3.1-3.2): collection listings order position ASC NULLS LAST then added_at ASC (was added_at DESC — walked playlists backwards); PUT /api/collections/:id/items/order applies full-order rewrites (owner-only, service-gated). hide_finished + Up Next (§3.3-3.4): GetCollectionItemsForDashboard gains viewer_user_id + hide_finished (LEFT JOIN reading_progress, COALESCE percentage < 0.999) and viewer_percentage rides the row for card graying; GetCollectionUpNext serves the collection's reading-queue head with the family-lens visible-library array enforced. Dashboard (§3.5): user collections are dashboard-ELIGIBLE BY DEFAULT — the show_on_dashboard gate (never writable for user rows since the column shipped) is gone from GetUserCollectionsForDashboard; hiding stays dashboard-prefs (hidden_collections) territory; hide_finished filtered-everything now emits a Complete section instead of silently vanishing; sections JSON carries is_playlist/complete. Series page ordering hardening (§3.6): GetSeriesBooks tiebreaks by title ASC — NULL/tied numbers previously returned heap (scan) order, which is what the user saw. Frontend custom-section-builder note: the builder has sent show_on_dashboard: true since its creation; the API never bound it. User-row visibility is now intentionally prefs-governed, so the field stays unbound by design (documented in code). Pins widened (docs/contract_pins.md): collection detail + up_next; dashboard sections + is_playlist/complete; new reorder endpoint recorded. Verified: TestPlaylistCollectionsEndToEnd (new, full e2e: sidecar series split, collection auto-creation with position 7, rescan idempotency, owner-only reorder, finished-filter + completion marker + Up Next drain) plus the full affected integration sweep green. |
||
|
John O'Keefe
|
646477afad |
feat(schema): subtitle/explicit columns, authors+publishers arrays, float series numbers
Phase 1 of the series/collections/metadata overhaul
(~/.opencode/plan/series-collections-metadata-overhaul.md).
Schema (in place, idempotent):
- media_items.series_number INTEGER -> DOUBLE PRECISION: real-world
ordinals are decimals (0.5, 1.5 novellas); INTEGER silently truncated
them (calibre 2.5 pinned as 2 in tests).
- + subtitle TEXT, explicit BOOLEAN (sidecar ingest; ABS explicit /
Audible isAdult).
- + authors/authors_search/publishers/publishers_search TEXT[] NOT NULL
(narrators A2a precedent). Arrays are write truth; author/publisher
varchars become trigger-regenerated display strings
(sync_author_publisher_display) so every varchar consumer (trgm
search, facets, API, OPDS/Kobo) keeps working unchanged.
- Row-level NULL normalization for the new arrays in
set_library_type_name (A2 lesson: Go nil -> explicit NULL -> 23502).
- Backfills ARRAY[author]/ARRAY[publisher] — deliberately no
comma-splitting ('Tolkien, J.R.R.' would corrupt; scanner kept only
authors[0] historically).
- collection_items.position INTEGER + (collection_id, position) index
for playlist ordering.
Go: sqlc regen; MediaMetadata/handler params gain Subtitle/Explicit/
Authors/Publishers (legacy singles still accepted, arrays win);
SeriesNumber int32 -> float64 end to end; metadata_overrides adapted
with legacy override keys ('author'/'publisher') preserved so existing
user overrides survive; bulk author edit writes the array.
Pins widened (docs/contract_pins.md): raw sqlc row key set + the six
new columns; changelog entry added.
Verified: unit suite green; integration sweep (bulk/smart collections/
scanner/stats/sync/queue/calibre/pins/audiobooks) green against the
shared dev DB after in-place migration. Pre-existing unrelated red:
TestOwnershipAndAdminTestSuite/TestAdminCreateUser timezone inheritance
fails identically at the pre-change commit (live shared state).
|
||
|
John O'Keefe
|
220d97cfe0 |
test(series): pin the SSR pagination contract under tied timestamps
New integration test mirroring a prod bulk import: a dedicated test library with 30 series whose items share ONE identical created_at (single-transaction NOW()), the exact data shape that made prod pagination look broken. Pins, via the real SSR /series route: - page 1 renders exactly 24 deterministic series (01-24) with chip "Page 1 of 2", no Previous, and a Next href carrying library_id and page (templ escapes & as & inside attributes) - page 2 renders the remaining 6, disjoint from page 1, chip "Page 2 of 2", no Next, Previous href correct - missing page param defaults to page 1; page=0/-3/two clamp to page 1 Guards both the SQL tiebreaker and the /series handler contract; would fail on any return of the stomp or an unstable sort. |
||
|
John O'Keefe
|
6216bd9bcf |
feat(audiobooks): multi-file books (folder-of-tracks + zip) + exhaustive metadata ingestion
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
|
||
|
John O'Keefe
|
24228e7147 |
test(scan): cancellation endpoint semantics and mid-run safety
- endpoint semantics: non-admin cancel is 403, unknown job is 404 - mid-scan safety: start a scan of 1200 unparseable .epub files and cancel it, then pin the invariant that no media item is marked missing or archived. The canary item's file is lexically LAST in the folder, so an abort that (incorrectly) ran the archive pass would treat it as unvisited and trip the assertion. Assertions hold under every interleaving - whether the cancel lands mid-walk or the scan already finished - so the test cannot flap against timing. - re-cancelling a terminal job is a 409 The test library is created through the shared createTestLibraryWithFolder helper, which registers API-delete cleanup so runs leave no residue in the dev database. Worker-level unit tests live in internal/services/worker_cancel_test.go (previous commit). |
||
|
John O'Keefe
|
9e831085ef |
fix(ui): derive Library Type dropdowns from library_types — audiobooks was missing
The audiobook library type shipped backend-complete (LibraryTypeAudiobooks constant, allowed extensions, MIME map, schema seed row, GET /api/libraries/types) but neither create-library form offered it: the admin Create Library modal and the setup wizard's add-library step both hardcoded their <option> lists to the original three types (ebooks/comics/manga), so an audiobook library could not be created from the UI. Both dropdowns now render from the same DB-backed library_types table the create API resolves types against — a future type appears without template edits (the exact drift the rendering-ownership rule warns about): - templates/utils.go: OrderedLibraryTypes pins the historical dropdown order (ebooks, comics, manga, audiobooks) and appends unknown future types by name; LibraryTypeDisplayName renders labels with a capitalize-the-name fallback for types added without a label - templates/admin_library.templ: new LibraryTypeOptions fragment used by the modal's select; AdminLibrary takes the types list - templates/setup.templ: Setup takes the types list, same fragment - internal/router/helpers.go: libraryTypeOptionsFor fetches via LibraryService.GetLibraryTypes, falling back to the built-in type names on DB failure so the forms stay usable (CreateLibrary still resolves the name against the DB, so the fallback is never authoritative) - internal/router/frontend.go, internal/router/setup.go: pass the list into the two pages Tests: - templates/utils_test.go: pure unit tests for ordering + labels - cmd/server/tests/admin_library_page_test.go: SSR regression — /admin/library must offer all four library types, audiobooks exactly once (options come from the DB loop, not a hardcode) |
||
|
John O'Keefe
|
45b8542e25 |
fix(api): honor the request's origin as the writer label — app rows stop wearing "web"
Release / build-and-push (push) Successful in 3m36s
Every web-API write endpoint hardcoded Source:"web", so the Android app's own pushes landed with last_sync_source/last_modified_source "web" — the app was indistinguishable from the web SPA in progress conflicts, device_sync_data, and annotation bookkeeping. The app CAN say who it is (bookmark create already carried origin → origin_source); the server just ignored it for labeling. All six write sites (progress PUT, bookmark/note/highlight create+PUT) now parse an ADDITIVE optional "origin" key, normalized through one gate: "android"/"ios" pass through, anything absent/unknown stays "web" (the label keys conflict claims and device_sync_data — free-form client strings would pollute them). Precedence on progress: a paired X-Device-Token still outranks origin. Bookmark origin_source now rides the same gate. The web SPA sends no origin — its rows are unchanged; KOReader/Kobo handlers keep their own labels. Wire discipline: additive request keys only (rule 2), response shapes untouched, pins changelog widened in-commit, locked by TestOriginWriterAttribution (android lands, default stays web, unknown normalizes). Full battery: only the two known pre-existing failures (TestOPDSLibraryFolders rot, TestWebSocketPingPong — both reproduce with the change stashed). |
||
|
John O'Keefe
|
010c0cb287 |
test(contracts): whitelist the unified row's two new keys on search rows
The H4 row-completion commit made abridged/audio_metadata emit on the search wire (the endpoint serializes the row directly) — exactly what the pin exists to surface. Whitelisted per the additive rule with the changelog entry. |
||
|
John O'Keefe
|
188ae076ab |
refactor(utils): one PgTextString for the four pgtype.Text unwrappers
handlers textToString, router textToString, router getText, and utils normText were four copies of the same NULL-to-empty unwrap (99 call sites). One exported utils.PgTextString replaces them all; the private normText (comparison normalizer) stays but is now the odd one out. Harness fix found on the way: setupTestServer built a full wiring literal for route registration and then RETURNED a second, sparser literal that never copied AnnotationService — setup.AnnotationService was always nil on the struct every test receives. Nobody noticed until cross_library_move_test became the first caller (it previously used the now-retired plain CreateMediaHighlight query); it now exercises the real SaveHighlight path through the wired service. |
||
|
John O'Keefe
|
7d698ea0d8 |
fix(annotations): bookmark and note PUTs land through the sync service (LWW by id)
H3 census bugs #2 and #3: both web PUTs used plain queries that bypassed the sync-aware layer entirely — no LWW, no source bookkeeping, and locator fields the client couldn't even send. The highlight PUT was the only one already routed (via HighlightID); the other two now mirror it: - SaveBookmarkRequest gains BookmarkID, SaveNoteRequest gains NoteID: id-targeted resolution reads the exact row, verifies ownership, LWWs under its stored dedup key, and resurrects tombstones when the edit is newer (same doctrine and comment structure as the highlight path). - Fields the PUT omits are coalesced from the stored row before the full-column sync update — the legacy web PUT sends only title/notes/position, and zeroing cfi/percentage/page on every edit would have been data loss. Verified both directions by TestH3_BookmarkPUTPreservesLocators (omitted survives, explicit lands). - The bookmark PUT's wire DTO gains the create endpoint's locator keys as optional fields (documented additive request keys in contract_pins.md); the note PUT keeps its content+position wire. UpdateMediaBookmark/UpdateMediaNote plain queries are now uncalled (the generated queries stay until the H4 query sweep). |
||
|
John O'Keefe
|
c579179fee |
refactor(annotations): AnnotationService is a required constructor dependency
H1's doctrine applied to the annotation service, which had been left on setters: NewMediaHandler/NewKoboHandler/NewKOReaderHandler/ NewSyncQueueProcessor(WithConfig) now take it as a parameter; the four SetAnnotationService setters are deleted; wiring updated (main.go, router/sync.go, test harness). With the service unskippable, the 18 nil-guards and their plain-query fallbacks die: - media.go: the create fallbacks for notes/highlights/bookmarks (the bookmark one was the H3 census's Bug #1 — it dropped percentage/chapter_reference the sync path carries), the highlight PUT fallback, and the three delete-path plain deletes (tombstone is now the only delete semantics). - kobo.go: both markup-ingest annotation loops (the bookmark fallback wrote the text into media_notes — a category error the fallback never should have had). - koreader.go: all three push-stack guards. - queue.go: the three "annotation service not available" errors in syncNote/syncHighlight/syncBookmark. Behavior note: the deleted highlight create/PUT fallbacks were the only paths passing note_id to the plain query — the sync service has no NoteID field, so the web's highlight→note link was already being dropped on the live path; this commit makes that pre-existing limitation explicit rather than papering it over (surfacing note_id on SaveHighlightRequest is left for a behavior commit that needs it). Pins (now 20 incl. the 3 annotation CRUD suites) + H1 tests + both unit suites green. |
||
|
John O'Keefe
|
ff029f1ad5 |
fix(annotations): note LWW re-save 500'd — UserID missing from the sync update params
Pin-first batch for H3: freeze web annotation CRUD response shapes (create 201 / update 200 / delete 204; full sqlc row models) before the stack consolidation touches anything. The new pins immediately caught a live server bug the 28 unit tests could never see (annotations_test.go is pure-unit: keys and compare logic, no DB): applyNoteLWW built UpdateMediaNoteForSyncParams without UserID, so the query's WHERE id=$1 AND user_id=$13 compared against zero — every dedup-hit note re-save (same content+position) returned 500 "update note: no rows in result set" and the update never landed. The bookmark twin passes UserID and works; the note stack is the clone that dropped the field. Fix: pass UserID in the params (one line). Pins go green and become the regression net; changelog records the new surfaces. |
||
|
John O'Keefe
|
14cdbe70eb |
test(contracts): CFI/context winner must land intact through conflict resolution
Book-side twin of the audio-survival test, closing H1's verification list. Scenario: web push carries percentage + epubcfi + context_text; a koreader-style percentage-only push from a second source within the window detects a conflict; the user resolves with the CFI-carrying side. The funneled resolution must land the winner's ENTIRE locator pair — percentage 0.5, epubcfi, context_text together (the old direct writer NULLed context_text and character_offset, breaking the coherent pair the far-anchor work established) — and must not spawn a duplicate pending conflict. |
||
|
John O'Keefe
|
6808c1babb |
fix(sync): the offline queue flush never processed anything — fix the enumerator + the dropped payload
Found while verifying H1's "app offline-play -> flush -> position lands"
with the emulator path: the flush has been dead code in two independent
ways, invisible because the first break made the second unreachable.
1. ListPendingSyncQueueItems matched `device_id = $1` with a NULL $1
(the processor passes DeviceID{Valid:false} to enumerate all devices;
uuid = NULL matches no rows). The ticker therefore processed nothing,
ever. The WHERE now treats NULL as "every device" (the same
narg::uuid IS NULL idiom the visibility queries use; EXPLAIN'd — seq
scan on a bounded table, idx_sync_queue_status_priority_created
fits). The processor also dropped its fake-pagination loop, which
re-fetched the same first batch forever once the backlog hit the
batch size (no OFFSET in the query).
2. dbItemToSyncQueueItem dropped SyncData — even with a working
enumerator, every flushed item arrived with an empty payload and
failed JSON unmarshal. Dead code harboring silent breakage, exactly
the plan's thesis.
Callers updated for the renamed RowLimit param (queue handler, offline
detector). TestH1_SyncQueueFlush_LandsAndPreservesAudio drives the real
flow: web PUT seeds audio 900s, a pending checkpoint-style queue row
applies percentage 0.8 through SaveProgress, audio survives, queue row
completes.
Also fixes dashboard_test.go's stale BuildSections signature (the
package's test binary has been unbuildable) — user-approved: tests
designed against old shapes are not worth keeping.
|
||
|
John O'Keefe
|
d16ca4acbe |
fix(tests): scope the sync_queue cleanup wipe to the test devices
The cleanup in setupSyncTestDB ran DELETE FROM sync_queue WHERE true — a table-wide wipe that discards REAL devices' pending offline entries on every suite run. Same class as the 2026-10-14 reading_history incident the standing rules ban. The test's own queue items cascade from its 'Test %'-named device, so the wipe is redundant for cleanup — scoped it to those devices (defense in depth ahead of the cascade). Flagged during H1 verification: the line executed during an in- container suite run this session; any device with pending offline queue entries at that moment should re-push on its next sync. |
||
|
John O'Keefe
|
8fc74c908a |
fix(conflicts): resolution writes funnel through SaveProgress — audio position survives
Batch H1 step 3b — THE data fix. Conflict resolution was the one live
erasure writer: applyProgressResolution and its verbatim twin
applyResolution both wrote the winner via a raw UpdateUniversalProgress
whose param list omits audio_offset_seconds/context_text/viewports, and
the SQL's DO UPDATE SET ... = EXCLUDED... NULLed exactly those. The web
UI resolves through POST /api/conflicts/:id/resolve and
/api/conflicts/bulk-resolve (main.js resolveConflict/bulkResolve), and
the reader's mark-as-read fires a resolution first when the reader SSR
stamps data-conflict-id/-winner — so every resolution today erased the
audiobook position and reader context.
Now: ProgressService is a required ConflictHandler dependency; BOTH
call sites (single resolve + bulk resolve) share one
applyProgressResolution that funnels through SaveProgress (the verbatim
twin is deleted). Deliberate choices, per the plan:
- SkipConflictDetection on the resolution write: both flows apply
BEFORE the conflict row is marked resolved, so the funnel's LWW
detector would otherwise file a duplicate pending conflict
(HasRecentConflictResolution is false at apply time).
- Source "conflict_resolution" — the row's last-write attribution; the
winner stays durably recorded in resolution_data.winner.
- Broadcast: true — live devices jump to the winning position.
- The winner snapshot maps percentage/epubcfi/context_text/chapter/
character/page/total_pages; a winner without a CFI clears the locator
pair (ClearLocator) instead of re-pairing with the loser's stale
anchor. Everything the snapshot doesn't carry SURVIVES via the LWW
seed: audio_offset_seconds above all. chapter_progress keeps its
stored value (was overwritten with the book percentage), an absent
percentage survives (was written as 0%).
- Accepted: the resolution captures one reading_history session
(attributed to conflict_resolution; the 2h idle ceiling applies).
Verification: TestH1_ConflictResolutionPreservesAudio drives the real
flow — web push with audio 300s, diverging push from a second source
detects a conflict, resolve with the second source as winner, GET shows
percentage 0.3 AND audio_offset_seconds 300 (previously NULLed), and no
duplicate pending conflict remains. Full contract pin suite green.
The host-run TestConflict*/TestProgress* failures are environmental
(container-only /app/uploads path + db hostname; fail identically at
baseline
|
||
|
John O'Keefe
|
e83c00f869 |
refactor(progress): ProgressService required at construction; delete the 9 unreachable erase-fallbacks
Batch H1 step 3a. The ceiling census (exhaustive UpdateUniversalProgress
grep) found 9 nil-guarded `if progressSvc != nil { funnel } else { raw
erase-write }` fallbacks — the survey's "four" under-counted kobo.go,
which has FOUR sites (Markup ReadingSync, Markup last-read-place,
AnalyticsGettests, SyncFromServer), plus one each in media.go, queue.go
and koreader.go. All unreachable: the service is created unconditionally
at main.go:127 and wired into every handler and the queue processor; the
test harness wires them too.
The erasure pattern is now unrepresentable:
- NewMediaHandler / NewKOReaderHandler / NewKoboHandler / both
NewSyncQueueProcessor flavors take the ProgressService as a REQUIRED
constructor arg; the SetProgressService setters are gone.
- Every fallback body (raw UpdateUniversalProgress with explicit-NULL
pgtypes for viewport/scroll/panel/reading_mode/zoom and no
audio_offset_seconds/context_text) is deleted; each write funnels
through SaveProgress with the LWW seed, so fields the writer doesn't
know survive.
- Removed the now-dead koreader pg-type helpers (textPtrToPgText,
intPtrToPgInt4, int64PtrToPgInt8) and their self-tests.
Behavior deltas (stored-row content, no wire shape changes —
KOReader/Kobo response pins stayed green):
- KOReader fallback used to write explicit NULL viewports and drop the
context_text the koplugin sends; the funneled mainline already maps
book.ContextText.
- Queue flushes no longer overwrite chapter_progress with the book
percentage (bogus data; preserved via the seed).
- A web PUT without percentage no longer writes 0% via the deleted
fallback; percentage seeds from the stored row.
Contract suite: KOReader×4 + Kobo×3 pins green.
|
||
|
John O'Keefe
|
a2ea34232c |
test(contracts): pin wire shapes before the canonical-types migrations
Batch H0 of the hardening plan: freeze the JSON key SETS of every
load-bearing surface so H1-H7 can only change them additively.
- cmd/server/tests/contract_pins_test.go: 14 tests / ~30 subtests pinning
progress GET+PUT, the KOReader plugin surfaces (progress push/pull,
library, bookmarks), the Kobo adapter shapes (initialization, markup,
sync-from-server), collections detail all four branches + preview,
series JSON, search rows, dashboard sections, media list BOTH branches
(raw sqlc rows vs resolved 38-key map), and SSR smoke for
bookshelf/collections/series/tags.
- docs/contract_pins.md: the pinned-shape inventory + the rule that a
batch widening a shape must widen the pin and this doc in one commit.
Suite is green on the unmodified tree at
|
||
|
John O'Keefe
|
ce366b1604 |
feat(stats): Library tab — per-library inventory stats + audio-aware Reading tab
The /stats page gains a second tab (Reading | Library) implementing the
Audiobookshelf-style library dashboard (HANDOFF item 9): media_items
INVENTORY aggregates — items, hours/pages, distinct authors, size,
distinct narrators, Top Genres / Top Authors bars, Longest / Largest
items — for one library or the All-libraries aggregate.
- GET /api/stats/library-usage?library_id=<uuid|omit>; family lens
enforced: a library_id outside the requester's visible set is a 403,
never a silent pass (AGENTS.md standing rule;
|
||
|
John O'Keefe
|
a222ef5c27 |
fix(media-items): normalize NOT NULL narrators on every write path
A2 made narrators/narrators_search NOT NULL DEFAULT '{}' (
|
||
|
John O'Keefe
|
199d82c613 |
fix(tests): sync-integration cleanup wiped the shared reading_history table
sync_integration_test's cleanup ran DELETE FROM reading_history WHERE true — harmless for years while the table sat empty, and a silent stats-data destroyer the moment session capture started feeding it: every suite run erased the shared dev accounts' captured reading (the user's manual flip-through test data included). The cleanup is now scoped: the sync tests' sessions belong to their test-sync users, and deleting those users cascades their rows. Any future table-wide cleanup DELETE is off the table (AGENTS.md test rules updated). |
||
|
John O'Keefe
|
0914c935d9 |
feat(dashboard): collapsible This Week widget, library-scoped + switch-surviving; title into the subbar
Three dashboard complaints, one build: - The widget collapses to a single summary row (This Week · minutes · pages · sessions · books · streak · Full stats) on every screen size and expands to the full grid via a details/summary toggle — the carousel sections start immediately below it again. - The 'Your Library' heading moved into the library switcher subbar (LibrarySwitcher gains an optional title param; collections/series keep the plain 'Library' label). The standalone heading block is gone. - Root-caused the JS-switch vanishing: renderDashboardCollections replaces ALL of #collections-container's innerHTML, so anything server-rendered inside it (title, widget) died on the first library switch. The widget now re-fetches ITSELF: a /dashboard/week-widget fragment (same resolveLibrary family-lens rules as the page) is pulled by htmx whenever the switcher dispatches week-refresh — carrying the current #library-select value — so the widget both survives switches and RE-SCOPES to the switched library. Initial SSR renders the same scope the sections use. ThisWeek (and the streak calendar) gained an optional library filter. Tests: TestReadingStatsWeekLibraryScope — per-library totals, scoped streaks, invisible-library isolation, malformed id 400. Full suite PASS. |
||
|
John O'Keefe
|
5a2c1585ae |
refactor(stats): 'Web / unsynced' bucket label → 'Web'; analytics tests isolated from the live reader
The bucket name read as confusing — 'unsynced' implied data loss when the push is always captured; it only lacks a device credential. The bucket is just 'Web' now (service fallback, stats.ts fallback, panel copy, tests). Also: analytics_test exact-zero assertions (EmptyHistory, NoReadingHistory) ran against testuser, which is a LIVE reader — real sessions accrued on it mid-suite and both flapped. They now run under the fresh regular user, like the stats tests; the 'testuser is a live reader' rule is recorded in AGENTS.md. |
||
|
John O'Keefe
|
7a604c2fec |
feat(stats): per-device reading breakdown + device attribution on progress syncs
- Attribution: registered clients may now send their device token in
X-Device-Token on PUT /api/media-items/:id/progress (the user JWT
stays primary). The token must map to a device OWNED BY the caller —
foreign/invalid/absent tokens soft-fail to plain web attribution so a
stats decoration can never break a sync. Attributed pushes capture
device_id (FK-safe), device_metadata.source = the registered device
name ('Pixel 9', 'Kindle Paperwhite'), and last_sync surfaces show
that device. Side effect worth having: app-vs-web pushes used to be
LWW-invisible (both sources read 'web'); they now surface as
legitimate cross-source progress conflicts when they diverge.
- /stats gains a By Device panel alongside By Library (grid 3-wide):
reading minutes + sessions + books per registered device, plus one
Web / unsynced bucket for device-less pushes so the panel always
reconciles with the headline totals. Range-aware, family-lens
scoped; renderBars shared between panels in stats.ts.
- The device identity IS the registered platform: devices rows already
carry the name/type the client registered (app sends Build.MODEL +
'mobile' at /api/devices/register, KOReader/Kobo likewise) — no new
identity plumbing on the server.
Tests: capture attributed to the device (row device_id + source
metadata + last_sync_source), foreign/nonexistent tokens soft-fail,
device buckets reconcile (80 attributed + 75 unsynced = 155 total).
Full suite PASS.
|
||
|
John O'Keefe
|
6457fbaf72 |
feat(stats): reading statistics — session capture wired + /stats page + widget + per-book card
The design parked in the handoff (multi-tier stats off reading_history; bookorbit-style in data, not looks) is in. The real prerequisite work was data: reading_history existed but NOTHING wrote to it — the whole stats feature was sitting on an empty table. - CAPTURE (no client changes, no repo-boundary crossings): every successful progress write through SaveProgress (KOReader + app + web + Kobo + offline queue — the one chokepoint) now records one reading_history session row: span = the interval since the PREVIOUS write instant, pages = positive page delta when both sides know the page index, marker rows for first contact. Spans over 2h are kept as zero-second markers (overnight suspend→resume, offline flushes must not inflate totals) — the honest 'sync cadence, not a stopwatch' caveat stays. device_id only when a REAL device row matches (the web/app progress path historically stuffs the USER id into DeviceID — probe instead of trusting it; FK violation would silently drop every capture). Source preserved in device_metadata. - API: /api/stats/summary?range=7|30|90|year (totals, zero-filled daily arrays, streaks, most-active day/weekday, top books, library breakdown — one call), /api/stats/this-week (widget: rolling 7d vs previous + streak), /api/stats/books/:id (lifetime + recent sessions). All bucketed in the user's timezone; finished ≡ any push ≥0.995; visible-library lens enforced at source. Legacy /api/analytics/* kept untouched. Range windows anchor end-of-day to the START of today (blasting the clock time into tomorrow resolved 'today' a calendar day late — found by the streak test). - /stats page (sidebar Stats between Progress and Devices, previously /analytics was an orphan): range selector, five stat cards, two Chart.js v4 charts (daily minutes bar, pages/day line — chunked via dynamic import so main.js stays untouched), per-library breakdown, most-read list, empty-state explainer. /analytics now 301s to /stats. The orphaned analytics.templ + analytics.ts removed (their containers and the page JSON never matched; the page was astub). - Dashboard 'This Week' widget (server-rendered, suppressed on service failure) and the book detail 'Your Reading Stats' card (lifetime totals of the viewer's own sessions) complete the tiers: per-user default, per-library breakdown inline, per-book on the page, per-system/admin aggregate parked as before. Tests: statistics_test.go (capture semantics at the chokepoint, tz/day buckets, zero-fill, streak math, finished detection, range validation, auth gate, widget, per-book lifetime). Full suite PASS. |
||
|
John O'Keefe
|
0a7fbd6f48 |
feat(collections): smart (dynamic) collections — rules evaluated live at render
Type is DERIVED, not stored: a saved ruleset makes a collection SMART,
no rules = STATIC (stored collection_items links, unchanged behavior).
No schema change — collections.auto_assign_rules JSONB already carried
the config; user's decision 2026-10-14 (handoff: live-at-render, not
refresh-on-scan; create-time Static/Smart toggle; v1 NO hybrid).
- Rule gains an enabled flag (absent = enabled, legacy rules unaffected).
- Rule CRUD: POST/PUT(DELETE) /api/collections/:id/rules[/:ruleId] — the
survivor dead effort of the rules-builder UI finally has real
endpoints (rules were never editable after creation; nothing ever
re-evaluated them). First rule converts a user collection to smart;
removing the last reverts it to static. UpdateCollectionRulesJSON
edits ONLY the rules doc so rule CRUD cannot clobber metadata.
- Live evaluation: candidates from GetVisibleLiveMediaForRules —
visible-library set (family lens enforced at source) + archived IS
NULL AND missing_scan_count = 0 — any enabled rule matches, title
order. Zero refresh hooks by design: always current by construction.
- Membership mutations (add/remove/bulk-add) are 400 on smart — v1 no
hybrid; collection_items.excluded stays the reserved future hook.
- Rules builder page (/collections/:id/rules) first time ever routable;
templ whole-attribute interpolation ("{ expr }" inside a quoted attr
value is literal text, as theme-{ user.Theme } silently proved).
- UI: create modal Static/⚡Smart cards (smart create redirects into the
rules builder), ⚡ badge + real per-card counts on the grid (smart =
live matches, static = stored-link count), smart detail renders
matched books with no hand-pick actions + an Edit Rules button.
- API: GET /:id gains is_smart + live books; GET /: keeps the Android
app's default exclusion, include_auto=true surfaces smart rows with
live counts; TestRules previews against the SAME candidates the live
render uses (visible + live, was raw first-1000).
- FIXED pre-existing regression from
|
||
|
John O'Keefe
|
17274bb868 |
fix(nas): refuse deletes while library storage is unreachable; real file removal; stat cache
Three changes from the network-drive audit: 1. Delete-ordering fix. Deleting a media item cascades its annotations, so the delete must never proceed while the item's library storage is unreachable — the DB row died, the file survived as an orphan, and the next scan re-imported it bare (annotations gone forever). resolveMediaStorage stats the library's folders: reachable → delete proceeds and the file is removed by ABSOLUTE path (the old code passed a library-relative path to os.Remove, which never resolved — file deletion has been a silent no-op); every folder unreachable → 409 with 'storage unreachable', row and annotations intact. 2. Latency: resolveLibrary stats every library folder on every SSR render; folder reachability is now cached with a 30s TTL. 3. README: Network Storage section documenting the outage hysteresis (mark → archive → auto-restore), the 90-day retention purge (ARCHIVE_RETENTION_DAYS, 0 = never), refused deletes while storage is down, and watch mode not firing on network mounts. Integration tests: storage-down bulk delete refuses with the row intact; real-file delete removes the file from disk. |
||
|
John O'Keefe
|
740213f0a4 |
fix(auth): new users inherit the system-wide default timezone
default_timezone existed as a system setting (admin-editable, exposed via SettingsRegistry.DefaultTimezone) but nothing read it at user creation — the CreateUser query didn't even carry the column, so every new user got the DB default UTC regardless. Both creation paths (public register + admin create) now seed the user's timezone from the setting, validated with time.LoadLocation and falling back to UTC when unwired/empty/invalid. Test-infra fixes surfaced by the run: TestUpdateProfileSelfEdit was self-editing the SHARED admin (renaming it past the harness cleanup's preserve-list, leaking a second admin row that broke the last-admin tests) — it now edits a dedicated user with a unique name; the last-admin tests get an ensureSingleAdmin precondition helper; the websocket collection name is run-unique (non-test names persist in the shared DB and UNIQUE(user_id,name) 409s on rerun). |
||
|
John O'Keefe
|
431fec05ad |
feat(security): library visibility enforced at the source; per-library series; annotation ownership; admin user creation
The family-lens hardening: library_visibility was only enforced where file bytes flowed (download/reader/OPDS) — metadata surfaces leaked hidden libraries to any authenticated account, and registration was open by default with no admin-side user creation. - Series are grouped PER LIBRARY now (same name in two libraries = two entries: manga adaptations / LN splits no longer merge); /api/series aggregates only the requester's VISIBLE libraries and /api/series/books REQUIRES a visible library_id (breaking contract; web + app clients updated in the same change). - /api/media-items, item detail, and the reader content endpoints (chapters/panels/outline/THUMBNAILS — page images leaked unguarded next to a gated GetPage) all enforce the visible set; resolveLibrary no longer honors a hand-typed library_id outside the visible set. - Annotation/note/bookmark mutations (update/delete/tombstone, sync LWW paths incl. conflict resolution) carry ownership predicates: another account's bare row id is a 404, not a silent delete. - Admin user creation: POST /api/auth/users (AdminMiddleware) + a Create User card in the admin Users UI — accounts stay creatable with self-service registration OFF, which is now the schema default (fresh installs; the first-admin bootstrap exception is unchanged). - Tests: series contract re-pinned (per-library, visibility 403s), new ownership + admin-create integration suites, OPDS all=1 test now passes an explicit page size (shared-DB growth made the fresh books fall off page 1), device-cap helper switched to the admin endpoint. |
||
|
John O'Keefe
|
679e0f9116 |
fix(server): ghost 401 must abort the chain — c.JSON's nil error let the request continue
Release / build-and-push (push) Successful in 3m21s
The guard returned c.JSON(401, ...), whose nil error told echo the
SuccessHandler succeeded — the pipeline then continued into the route
handler, which type-asserted the never-set "user" context value and
panic-recovered as a 500 appended after the 401 body. The status code
happened to survive (first write wins), which is why the pin test
passed, but every ghost request also executed the target handler with
no user — the exact unauthenticated-execution class the guard exists to
prevent.
Return echo.NewHTTPError(401, ...) instead: non-nil aborts the chain
and renders {"message":"user_not_found: ..."}; the pin test now asserts
that shape. Full suite: 219 groups, zero failures.
|
||
|
John O'Keefe
|
9cd6356892 |
feat(server): ghost-session guard — JWTs for deleted users stop authenticating
The JWT middleware validated signature + expiry and fabricated the request user purely from token claims, so a signature-valid token for a DELETED user kept full API access until token expiry — observed live on hardware (a phone session survived the test suite's user-table recreation and kept serving its true empty sets; open handoff bug 1). An ensureUserExistsMiddleware already existed but was only wired on the frontend SSR group (frontend.go) — every /api group used bare createJWTMiddleware. The existence check now lives INSIDE createJWTMiddleware's SuccessHandler so every protected group (SSR and API alike) gets it: - After claim validation, GetUser by the token's user_id; a missing row clears the token cookie and answers 401 user_not_found for API calls (the app's TokenAuthenticator then re-logins with stored credentials) or redirects to /login?session=invalid for page requests — same shape-aware split the error handler already uses. - Cost: one primary-key lookup per authenticated request (~0.2ms local); instant revocation, no TTL semantics. The redundant SSR-only middleware is deleted (its redirect behavior is preserved inside the guard). The guard exposed two processing-issues tests that were unknowingly RELYING on ghost semantics: they nested a second setupTestServer whose bootstrap deleted the first setup's regular user, then asserted 403 from the dead token. De-nested: a createLibraryDirect helper inserts the library under the live setup's user, no second bootstrap. New TestDeletedUserTokenRejected pins the contract: token works while the user exists -> 401 user_not_found after deletion. Full suite: 219 groups, zero failures. |
||
|
John O'Keefe
|
69516ae165 |
feat(server): progress locator coherence — percentage-only writes clear stale anchors
The verify/heal pipeline verified only INBOUND context-bearing writes;
stored-row coherence was never enforced. SaveProgress seeds missing
fields from the existing row, so every percentage-only write (KOReader
low-confidence conversion, KOReader no-pointer payloads, Kobo markup
ReadingSync / analytics / sync-from-server, locator-less queue entries)
silently re-paired its NEW percentage with the PREVIOUS writer's stale
CFI+context — the incoherent row shape that fed the client-side
far-anchor restore floor (and, via the read-side anchor handles, could
even hand a first-party client the OLD position's scroll address).
KOReader's converter already declared the right intent ("storing
percentage only — the row's percentage restores approximately"); the
seeding betrayed it. SaveProgressRequest gains ClearLocator, set by the
percentage-only call sites, and the seeding rule honors it: stale
epubcfi/context/character_offset clear (the upsert assigns columns
directly, so NULL sticks) unless the request carries its own. A
request-carried context survives so the verify/heal pass can still mint
a fresh anchor from it; the HTTP merge path (field-absent PUTs
preserve) and all anchor-bearing writes are untouched. The fixed-layout
branch already cleared locators unconditionally — unchanged.
Consumer audit: percentage — the only thing every sync round-trip
exchanges — is untouched on all paths; KOReader pulls on locator-less
rows already support the nil-CFI shape (builder guards
progress.Epubcfi.Valid); first-party restores go percentage-first
instead of floor-fighting a stale anchor.
Tests: TestProgressWeb_LocatorCoherence pins clear-on-declared,
request-carried-context survival, and merge-path preservation. Full
suite with the change: 218 groups, zero failures.
|
||
|
John O'Keefe
|
f24a2d6c22 |
test(server): fixture collisions fixed, stale contract asserts re-anchored, gofmt — suite now fully green
Closes the remaining integration-suite failures (handoff open bug 3),
line by line:
- Fixture family (the upsert equivalence): tags_filter_test
("Science Fiction" found 1 of 3; "Fantasy" 404'd on zero results),
media_item_isbn_test (normalization read back the first case's ISBN
everywhere; edge cases saw a nil ISBN; pagination collapsed 5
creates onto 1 row), search_unified_test (its 3 books were actually
1 row + two no-op creates) — all POSTed a constant file_path into
one library while CreateMediaItem upserts ON CONFLICT
(library_id, file_path). Every fixture now uses unique per-item
paths, the suite's universal convention elsewhere. TestTagsFilter
confirmed same-family: counts came right with no second bug.
- New TestMediaItemCreateDuplicatePath pins the honest-create
contract from
|
||
|
John O'Keefe
|
208fc2c4d6 |
fix(server): kobo markup panic — wire LibraryService into the test harness, guard typed-nil resolver
TestProgressWeb_KoboIntegration 500s'd on every markup with a last-read-place bookmark: the panic-recovered stack was a nil pointer dereference in LibraryService.ResolveMediaPath (s.db deref), reached from KoboHandler.convertKoboCFIToStandard during kepub CFI conversion. Two-layer bug: 1. Test harness (root cause): the router.Config built in test_helpers_test.go omitted LibraryService, so router/sync.go's SetLibraryService(cfg.LibraryService) stored a nil *services.LibraryService in the interface field. Production wiring (main.go:246) was always correct. 2. Latent trap (hardening): `h.libraryService != nil` at the convertKoboCFIToStandard call site can never catch a typed nil — an interface holding a nil *LibraryService is non-nil. ResolveMediaPath now degrades with a "library service not configured" error instead of panicking on a nil receiver; callers already handle the error by skipping CFI conversion. Harness-only fix, zero production behavior change: the endpoint now returns a clean 200 with the position saved wherever kepub conversion is unavailable instead of a panic-recovered 500. Verified: TestProgressWeb_KoboIntegration passes all three subtests (401 unauthenticated, ReadingSync→last-read-place percentage preservation, last-read-place without prior ReadingSync) and on immediate rerun — previously it failed every run. |