77cd268f4e417d6c8c11cf68251d4dc9b6fb2211
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
a42c6d4f75 |
feat(scanner): force rescan = original scan; feat(filters): the manually-edited filter (faceted plan Phase 13)
Release / build-and-push (push) Successful in 3m7s
Force rescan contract change (owner-directed): the editorial-keep skip
is retired. Force rescan now re-derives EVERY field on EVERY book from
all sources — multi-file books included (the folder/zip flows ran the
full chain on create and on manifest-flagged edits, but skipped the
sidecar apply under force). metadata_overrides is the only protection:
ApplyMetadataOverrides re-applies manual values inside the writer, so
manual edits survive; everything else re-derives.
- A force pass also STAMPS each re-derived book's manifest with the
current parser version (success-only — same truthfulness rule as
Phase 10) and closes its open processing issues (Phase 11 self-heal).
Force rescan and the ev-bump trickle now converge: either leaves the
library fully current; the deploy order is bump-first, then Rescan
All (documented in the plan and contract_pins).
- Single-file force rescan already ran the full chain — unchanged,
now pinned.
NEW filter field manually_edited ("Manually edited", boolean):
cardinality(mi.metadata_overrides) > 0 / = 0. Composed-engine (qf)
only — the builder's Book facts group gains it with is true/false;
no value menu, no suggestion surface; family lens inherited.
Tests: multifile force applies sidecar metadata + stamps + override
protection; single-file pin; the filter both directions scoped to the
test library; adjacent suites (Audiobook, Calibre, ExtractorVersion,
CrossLibraryMove, ProcessingIssues, FilterBuilder, SchemaIdempotency)
green. The extractor mechanism itself is untouched — the owner bumps
scan_extractor_version manually via the admin UI (plan Phase 13 §3).
|
||
|
John O'Keefe
|
9b5defcb31 |
feat(issues): the honest processing-issues ledger — row-less import failures, self-healing resolution, family lens (faceted plan Phase 11)
processing_issues only recorded cross-library move rejections; files that failed to import vanished into an error counter and stdout. The ledger now records every failure class and heals itself: Schema (in place, pre-release boundary): - media_item_id NULLABLE + file_path TEXT (deliberately unbounded — an over-long path is itself a failure mode; a VARCHAR(500) ledger column died on the exact 22001 it existed to report — caught by the new test). Row-less failures key on (library, file_path); item-backed on (item, type). Widened dedup = COALESCE-expression unique index; the old table UNIQUE stays (NULLs pairwise-distinct for row-less). - Conflict-target upsert: repeat failures UPDATE the same row and REOPEN resolved ones — never duplicates. Wirings: - CreateMediaItem failure (single-file + multifile createAudioBookItem) → row-less import_failed (error). - Extraction hard-fail with fallback import → item-backed metadata_extraction_failed (warning). - Change-path re-extraction failure (the ev/sidecar guard) → item-backed metadata_extraction_failed; LogProcessingIssue keeps its signature, a row-less helper joins it. Auto-resolution: - Successful import → ResolveOpenProcessingIssuesForPath (library, file_path). - Successful (re)extraction → ResolveOpenProcessingIssuesForItem (single-file change path, multifile editorial apply). - Unchanged-skip scans do NOT resolve: extraction never re-ran, so the degraded metadata is still degraded — the page heals exactly when the book does. Family lens on ALL issues surfaces (list, stats, resolve, delete, SSR page): explicit out-of-set library id → 403, never silent. This tightens one pre-lens behavior — a nonexistent library id returned 200-empty, now 403 (pinned, changelog entry in docs/contract_pins.md). List is LEFT-JOINed so row-less rows appear (media_item_id "", title falls back to the path). Tests: full lifecycle state machine through real scans (fail → appears, repeat → no dups, success-at-path → resolved, LEFT-JOIN list shape); family lens 403s with a fresh non-shared admin; all 7 pre-existing issue suites green (two expectations widened to the lens, changelogged); adjacent suites (CrossLibraryMove, Audiobook, Calibre, ExtractorVersion, SchemaIdempotency) green; unit suites green. |
||
|
John O'Keefe
|
91b0c45dd4 |
feat(scanner): extractor-version (ev) mechanism — parser fixes reach old rows without a force rescan (faceted plan Phase 10)
Parser fixes (
|
||
|
John O'Keefe
|
4ee24e7152 |
feat(filters): the rule builder — composed engines over the field registry (faceted plan Phase 9)
- Field registry (services/filter_registry.go): ~55 typed fields — every book column, file/lifecycle facts, the three HOLDING STRUCTURES (library, collections with excluded-aware membership, series with FULL primary+alternate_info membership), and the whole user-relative class (status, progress, last read, my rating, review, notes/highlights presence, note/highlight text, time spent, pages read, collections, added by) reading the VIEWER's rows. Operators are typed constants; negation is an operator. - Composed engines (services/filter_query.go): one Go clause builder over raw pgx feeds the search (identical SearchMediaItemsUnifiedRow shape), the field-aware suggestions, and a true-COUNT live count. Mandatory WHERE head everywhere (family lens + integrity facts). Values are always bound parameters — no user string reaches a field/operator position, ever. - Wire: the qf JSON param (validated strictly — unknown fields/operators/malformed values are 400); qf wins over flat params on hand-crafted URLs; flat deep links keep working unchanged. - Endpoints: GET /api/media-items/filter-count (live count); field-values autocomplete upgraded in place — any registry field key, qf binds the OTHER rules (symmetric under-filter suggestions, zero-match values never offered); classic six without a tree keep their static pinned contract. - UI: the drawer becomes the rule builder — the ONLY filter UI (templ markup, Alpine tree state, sanctioned island; grouped field menus, operator menus with red negations, field-aware value menus, live count, chips per rule with red negations, URL rehydration, smart serialization flat-when-simple else qf). - Tests: composed semantics (AND/OR/negation/nesting/enums/bools/ numbers/dates/empty), every user-relative field class with the my-data weld (two readers, different worlds), note-text substring pin (catharsis ≠ Carthage), family lens (hidden libraries nowhere, 403s), suggestion-under-rules exactness, qf validation + precedence, flat-vs-tree equivalence; registry completeness unit test. |
||
|
John O'Keefe
|
ebb7f4a5b0 |
feat(filters): panel removal + All-Books field-values fix (faceted plan Phase 8)
- Remove the Phase 7 facet-panel UI: SSR route, facet_panel.templ, FacetCountsForView, drawer panel block, panel TS + .facet-* CSS. The ENGINE stays (FacetCountsUnderFilters, all-facets endpoint, drift guard) — it becomes the Phase 9 builder's machinery. - Fix the All-Books bug: field-values autocomplete (author/genre/ tags/narrator/series/language) treats library_id as OPTIONAL — absent aggregates every VISIBLE library (family lens in-query); explicit out-of-set library_id is now 403, never a silent pass. The four scalar Search*Values queries get the nullable shape. - toast.ts: suggestion fetches are exempt from error toasts (console-only) — a failed background hint never interrupts. - TestFacetPanelPartial deleted with the behavior; TestFieldValuesAll- Libraries pins the fix (200 + cross-library aggregation, family lens, 403). Contract-pins changelog entry added. |
||
|
John O'Keefe
|
f52bb6f073 |
feat(facets): all-facets panel with under-filter counts (faceted plan Phase 7)
The 9 Batch A facets could not be SET from the web UI, the facets
endpoint had no in-app consumer, and the registry's libraryTypes hints
rendered nowhere. This phase closes all three.
- ONE UNION-ALL sqlc query (FacetCountsUnderFilters) computes all 14
listable facets in a single round trip: each arm omits its OWN
facet's filter block and carries every other registered facet's
block (copied mechanically from SearchMediaItemsUnified) — classic
faceted semantics: with filters active each facet counts under the
others; with none, blocks are neutral and the counts are the v1
global-per-library ones. One query serves both cases. The 14
per-column Batch C queries are deleted.
- Endpoint: no facet param → the new all-facets shape
({facets: {param: [{value, display?, count}]}}), filters bound via
the shared ParseMediaSearchParams, library_id optional (the
All-Libraries view, family lens still enforced); facet=X keeps the
pinned single-facet shape (now routed through the same query with
neutral filters).
- SSR partial GET /media-items/facet-panel: templ renders the drawer's
value lists — registry order, Batch D library-type scoping
(FacetsForLibraryType: manga_type on manga, reading_direction on
manga+comics, narrator on audiobooks; All view = union), prettified
genre labels + language display names, active values marked. Family
lens: out-of-set explicit library_id is 403.
- Drawer wiring: the panel fetches on drawer open and after every
apply with the CURRENT filter query string; a value click toggles
that facet's comma-joined filter (hidden input where the drawer has
no field — same as deep links) and re-applies the form; chips and
URL stay in sync. Values are server-rendered (rendering doctrine);
bookshelf.ts owns only the click/fetch orchestration.
- prettifyGenre moves to internal/utils as the canonical
implementation (services' display enrichment shares it; templ wraps
it; the TS port in bookshelf.ts is unchanged).
- Drift guard extended: FacetCountsUnderFiltersParams must carry every
registry facet's param, the bools as pgtype.Bool.
Tests: under-filter exactness (own-block omission, OR-within,
AND-across), family lens (hidden values absent; 403 on both shapes),
all-shape + pinned-shape coexistence, partial rendering with Batch D
scoping + active marking + 403. Full container suite green except the
documented pre-existing TestAdminCreateUser red.
|
||
|
John O'Keefe
|
6152137156 |
fix(tests): repair stale pins surfaced by the Phase 6 full-suite run
Two suite failures, neither caused by scanner code (the Phase 6 diff
touches no handler):
- search_test invalid_library_id expected the pre-family-lens 404;
|
||
|
John O'Keefe
|
5245f292ca |
feat(scanner): quiet scans, enqueue coalescing, watcher probe, sibling cache
Part 2 of the faceted plan, Phase 5 (steps 5-6): the scan that used to print ~4 lines per unchanged file now prints ONE summary line, and a poll tick landing mid-scan no longer stacks a second full scan. - Quiet scans (§4): the walk loop is silent — inserts, changes, and errors log inside processMediaFile; the duplicate 'Successfully processed' branch (both branches printed it) and the per-file 'processing / already exists / same size, skipping' chatter are gone. Summary carries the trigger and a changed counter: 'Scan of <folders> (trigger: poll): N files, N changed, N new, Xs, N errors'. Triggers threaded through every enqueue site: poll (startBackupScan), watcher (dirty-dirs flush), admin (scan API), initial (startup scan). - Enqueue coalescing (§5): Worker.HasActiveScanForLibrary (JobResult gains additive job_type so only SCAN jobs coalesce — an import sharing the library id never blocks one). enqueueLibraryScan skips silently; the admin scan API answers 409 with the active job id. Enforces the one-scan-per-library invariant the bulk scan-state map leans on. - Watcher capability probe (§4): ONE Add on a root folder; failure disables the watcher with a single startup line (replaces the per-directory warning spam on inotify-broken deployments). The watcher stays an optional latency reducer over scheduled scans. - Per-directory sibling cache (§6): hasSiblingBookFileCached caches the ReadDir per directory for one scan — loose-image manga re-stats thousands of images sharing a handful of directories. Tests: TestHasActiveScanForLibrary (unit — running/pending coalesce, terminal jobs don't, non-scan jobs don't, empty library never, nil worker safe). Full scanner suite green in the container. |
||
|
John O'Keefe
|
6750e9e0dc |
feat(scanner): manifest-gated sidecar change detection + playlist sync on change path
Part 2 of the faceted plan, Phase 4 (steps 3-4). Size-only detection
was blind to metadata-sidecar edits; a size-stable book is changed by
EDITING its metadata.opf/metadata.json/cover — previously invisible
until a force rescan.
- Sidecar manifest fingerprints (media_scanner_manifest.go): the
manifest stats exactly the names the extractors consume (metadata.opf,
metadata.json, findSidecarCover patterns incl. <basename>.<ext>), so
manifest and extractors can never disagree. Shape v1:
{v, entries:[{n, s, m (unix nanos — second granularity would hide
edits landing in the backfill scan's own second), h (sha256 — covers
included: the repoint ladder needs a baseline hash for any stat-
drifted sidecar; a hashless cover turns harmless mtime drift into a
re-extraction, the exact flood this design kills)]}.
- Evaluation ladder in cost order: stat pairs identical → skip with
ZERO bytes read; stat drift → hash that sidecar (equal = host repoint
reconciliation, stat pair refreshed, no re-extraction; differing =
changed); NULL stored manifest → backfill only, NO re-extraction
(deploy-day rule: the first scan after release must not look like an
mtime flood).
- Change path: re-read full row, extractMetadata once and pass it
through updateMediaItemWithMetadata (preExtracted — metadata_overrides
stay protected), run syncPlaylistCollectionsAfterCreate (Correction 2:
a reading-order entry added to an EXISTING book now creates its
collection; add-if-missing is idempotent), store a fresh full
manifest. Creates seed their manifest at insert time.
- Multifile folder/zip books intentionally keep Phase 3 behavior: their
change path does not consume sidecar extractors, so gating them would
flip books 'changed' without picking anything up (plan correction —
the plan's §3 multifile note assumed otherwise).
- UpsertMediaScanState + GetMediaScanState queries.
Tests: media_scanner_manifest_test.go (ladder unit tests — fast path,
repoint, same-size drift, appear/vanish, unhashable legacy entries);
TestScanManifestLifecycle (seeding, unchanged-skip stability, repoint
reconciliation with stat-pair refresh assertion, same-size edit
consumed, playlist sync on the change path + idempotency, no-leak
guard). Container runner.
|
||
|
John O'Keefe
|
bbedbfb13f |
feat(facets+ui): clickable facet values everywhere, genre prettify, active-filter chips
Frontend half of the faceted-navigation Phase 2. 'Everything filterable should be clickable': every displayed facet value links to the bookshelf pre-filtered by its raw stored value; display forms are prettified at render, never in the DB. - templates/facet_links.templ: FacetLink (inline links) + FacetBadge (badge links) to /bookshelf?<facet>_filter=<value>; the raw value rides the param while the shown text is prettified, so matching and display can never disagree. Series/tags keep their dedicated pages by design; the card media-type icon stays unlinked (a derived kind, not a column value — deep-linking it would send wrong values). - book_detail.templ: publisher, language, edition, genre, imprint cells + narrators line + age-rating/story-arc badges are now links. - prettifyGenre (templates/utils.go + unit tests): title-cases only entirely-lowercase values; small words stay down except first position; & preserved. TS port for chip labels documents the twin. - bookshelf.templ + bookshelf.ts: active-filter chips bar — Alpine renders chips from client-local form state (sanctioned island); one bubbled htmx:afterRequest hook keeps chips + shareable URL in sync; removeFilterChip clears the facet and re-applies. URL deep links now accept all 18 facet params (hidden inputs where the drawer has no field). Pins changelog: Batch B/C entry in docs/contract_pins.md. Rendering ownership guard (make lint-rendering) green. |
||
|
John O'Keefe
|
7e586df48a |
feat(facets): facet registry, Batch A server params, genre model v2, facets endpoint
Phase 1 + backend of Phase 2 of the faceted-navigation plan. One
declarative facet registry (services/facet_registry.go) drives the
bookshelf filter system; a drift-guard unit test reflects over the
generated SearchMediaItemsUnifiedParams so a registry entry and its
sqlc.narg block can never disagree silently.
- Nine new filter params on SearchMediaItemsUnified: story_arc,
imprint, edition (scalar-fuzzy), age_rating, format_group,
manga_type, reading_direction (exact), explicit, abridged
(presence-bound bools). Comma = OR-within multi-select on every
text facet; AND-across-facets unchanged.
- Genre model v2 (user rule: a subgenre is not a genre, it's a tag):
genre_lexicon.go rebuilt as shelf set + subgenre->shelf map;
pickSmartGenre picks direct shelf > mapped subgenre > Fiction floor
> empty (unknown-only stays tags-only). Type-words (Manga, Comics,
Graphic Novel) join the demote list. Both 'adventure' and 'action &
adventure' are shelves — provider data uses both and the verified
traces pin them.
- Language filter UX: comma elements normalize through the ingest
language map, so display names ('English') match rows stored as ISO
codes ('en'). The language autocomplete now serves display names.
- NEW GET /api/media-items/facets?facet=X&library_id=... — registry-
driven value+count listing (14 static per-column queries, family-lens
join); language values carry display names; bool/range facets 400.
- Family-lens gates: an explicit hidden-library library_id is now a
403 on the search endpoints (was a silent empty set) — JSON branch
and HTMX grid partial both.
- handlers.ParseMediaSearchParams is the SINGLE param parser for the
search JSON endpoint and the HTMX grid partial; the HTML branch
previously dropped publisher_filter and narrator_filter silently.
Pins: docs/contract_pins.md changelog entries (additive request
params; new facets endpoint). Tests: facet_registry_test.go (shape +
SQL drift guard + library-type union), series_parsing_test.go genre-v2
cases, facets_integration_test.go (param filtering e2e, counts,
language display, 403s) — container runner.
|
||
|
John O'Keefe
|
7537063cb7 |
feat(filters+nav): author navigation end-to-end — arrays on every media response, deep-linkable bookshelf filters, clickable authors
Completes the A2c acceptance scenario: click an author on a book detail page -> land on the bookshelf filtered to everything that author wrote (including their share of co-authored books). - Media detail + unscoped list resolved maps expose subtitle, explicit, authors, publishers (the library-scoped raw branch and the search endpoint already did — raw rows follow the SELECT). - SearchMediaItemsUnified SELECT carries the new columns; the search endpoint serializes the row directly, so clients get them there too. - book_detail.templ renders each author as a link to /bookshelf?author_filter=<name>. - Bookshelf honors filter params in the URL on load (author_filter, series_filter, genre_filter, publisher_filter, tags_filter, narrator_filter, language_filter): pre-fills the form, applies it to the first grid load. Comma-joined values are multi-select via the server-side string_to_array split. Filtered views are shareable. Pins widened: search rows (+7 keys), resolved keys (+4); changelog entries in docs/contract_pins.md. Android follow-up plan written to ~/.opencode/plan/bookhoard-app-v2-compat.md. |
||
|
John O'Keefe
|
1dbb0ce406 |
feat(collections): playlist auto-creation, position ordering, hide_finished, Up Next, dashboard eligibility
Phase 2b + 3 of the metadata overhaul (plan §2.4, §3.1-3.7). Auto-playlist collections (§2.4): the scanner turns obviously-playlist sidecar series (explicit '(Reading Order)'/chronological/publication/ in-order/timeline markers only — precision over recall) into curated static collections: upsert by view_settings.imported_from_series then name (rename-safe), position set ONLY on insert (clean integers; 1.5/1a stay unpositioned rather than guessing), membership add-if-missing, never removed or re-ordered by rescans, provenance in description + view_settings, show_on_dashboard true (dashboard-eligible by design), 'scanner.auto_playlist_collections' setting (default on, Settings Registry + schema seed). Ordering (§3.1-3.2): collection listings order position ASC NULLS LAST then added_at ASC (was added_at DESC — walked playlists backwards); PUT /api/collections/:id/items/order applies full-order rewrites (owner-only, service-gated). hide_finished + Up Next (§3.3-3.4): GetCollectionItemsForDashboard gains viewer_user_id + hide_finished (LEFT JOIN reading_progress, COALESCE percentage < 0.999) and viewer_percentage rides the row for card graying; GetCollectionUpNext serves the collection's reading-queue head with the family-lens visible-library array enforced. Dashboard (§3.5): user collections are dashboard-ELIGIBLE BY DEFAULT — the show_on_dashboard gate (never writable for user rows since the column shipped) is gone from GetUserCollectionsForDashboard; hiding stays dashboard-prefs (hidden_collections) territory; hide_finished filtered-everything now emits a Complete section instead of silently vanishing; sections JSON carries is_playlist/complete. Series page ordering hardening (§3.6): GetSeriesBooks tiebreaks by title ASC — NULL/tied numbers previously returned heap (scan) order, which is what the user saw. Frontend custom-section-builder note: the builder has sent show_on_dashboard: true since its creation; the API never bound it. User-row visibility is now intentionally prefs-governed, so the field stays unbound by design (documented in code). Pins widened (docs/contract_pins.md): collection detail + up_next; dashboard sections + is_playlist/complete; new reorder endpoint recorded. Verified: TestPlaylistCollectionsEndToEnd (new, full e2e: sidecar series split, collection auto-creation with position 7, rescan idempotency, owner-only reorder, finished-filter + completion marker + Up Next drain) plus the full affected integration sweep green. |
||
|
John O'Keefe
|
646477afad |
feat(schema): subtitle/explicit columns, authors+publishers arrays, float series numbers
Phase 1 of the series/collections/metadata overhaul
(~/.opencode/plan/series-collections-metadata-overhaul.md).
Schema (in place, idempotent):
- media_items.series_number INTEGER -> DOUBLE PRECISION: real-world
ordinals are decimals (0.5, 1.5 novellas); INTEGER silently truncated
them (calibre 2.5 pinned as 2 in tests).
- + subtitle TEXT, explicit BOOLEAN (sidecar ingest; ABS explicit /
Audible isAdult).
- + authors/authors_search/publishers/publishers_search TEXT[] NOT NULL
(narrators A2a precedent). Arrays are write truth; author/publisher
varchars become trigger-regenerated display strings
(sync_author_publisher_display) so every varchar consumer (trgm
search, facets, API, OPDS/Kobo) keeps working unchanged.
- Row-level NULL normalization for the new arrays in
set_library_type_name (A2 lesson: Go nil -> explicit NULL -> 23502).
- Backfills ARRAY[author]/ARRAY[publisher] — deliberately no
comma-splitting ('Tolkien, J.R.R.' would corrupt; scanner kept only
authors[0] historically).
- collection_items.position INTEGER + (collection_id, position) index
for playlist ordering.
Go: sqlc regen; MediaMetadata/handler params gain Subtitle/Explicit/
Authors/Publishers (legacy singles still accepted, arrays win);
SeriesNumber int32 -> float64 end to end; metadata_overrides adapted
with legacy override keys ('author'/'publisher') preserved so existing
user overrides survive; bulk author edit writes the array.
Pins widened (docs/contract_pins.md): raw sqlc row key set + the six
new columns; changelog entry added.
Verified: unit suite green; integration sweep (bulk/smart collections/
scanner/stats/sync/queue/calibre/pins/audiobooks) green against the
shared dev DB after in-place migration. Pre-existing unrelated red:
TestOwnershipAndAdminTestSuite/TestAdminCreateUser timezone inheritance
fails identically at the pre-change commit (live shared state).
|
||
|
John O'Keefe
|
6216bd9bcf |
feat(audiobooks): multi-file books (folder-of-tracks + zip) + exhaustive metadata ingestion
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
|
||
|
John O'Keefe
|
45b8542e25 |
fix(api): honor the request's origin as the writer label — app rows stop wearing "web"
Release / build-and-push (push) Successful in 3m36s
Every web-API write endpoint hardcoded Source:"web", so the Android app's own pushes landed with last_sync_source/last_modified_source "web" — the app was indistinguishable from the web SPA in progress conflicts, device_sync_data, and annotation bookkeeping. The app CAN say who it is (bookmark create already carried origin → origin_source); the server just ignored it for labeling. All six write sites (progress PUT, bookmark/note/highlight create+PUT) now parse an ADDITIVE optional "origin" key, normalized through one gate: "android"/"ios" pass through, anything absent/unknown stays "web" (the label keys conflict claims and device_sync_data — free-form client strings would pollute them). Precedence on progress: a paired X-Device-Token still outranks origin. Bookmark origin_source now rides the same gate. The web SPA sends no origin — its rows are unchanged; KOReader/Kobo handlers keep their own labels. Wire discipline: additive request keys only (rule 2), response shapes untouched, pins changelog widened in-commit, locked by TestOriginWriterAttribution (android lands, default stays web, unknown normalizes). Full battery: only the two known pre-existing failures (TestOPDSLibraryFolders rot, TestWebSocketPingPong — both reproduce with the change stashed). |
||
|
John O'Keefe
|
010c0cb287 |
test(contracts): whitelist the unified row's two new keys on search rows
The H4 row-completion commit made abridged/audio_metadata emit on the search wire (the endpoint serializes the row directly) — exactly what the pin exists to surface. Whitelisted per the additive rule with the changelog entry. |
||
|
John O'Keefe
|
7d698ea0d8 |
fix(annotations): bookmark and note PUTs land through the sync service (LWW by id)
H3 census bugs #2 and #3: both web PUTs used plain queries that bypassed the sync-aware layer entirely — no LWW, no source bookkeeping, and locator fields the client couldn't even send. The highlight PUT was the only one already routed (via HighlightID); the other two now mirror it: - SaveBookmarkRequest gains BookmarkID, SaveNoteRequest gains NoteID: id-targeted resolution reads the exact row, verifies ownership, LWWs under its stored dedup key, and resurrects tombstones when the edit is newer (same doctrine and comment structure as the highlight path). - Fields the PUT omits are coalesced from the stored row before the full-column sync update — the legacy web PUT sends only title/notes/position, and zeroing cfi/percentage/page on every edit would have been data loss. Verified both directions by TestH3_BookmarkPUTPreservesLocators (omitted survives, explicit lands). - The bookmark PUT's wire DTO gains the create endpoint's locator keys as optional fields (documented additive request keys in contract_pins.md); the note PUT keeps its content+position wire. UpdateMediaBookmark/UpdateMediaNote plain queries are now uncalled (the generated queries stay until the H4 query sweep). |
||
|
John O'Keefe
|
ff029f1ad5 |
fix(annotations): note LWW re-save 500'd — UserID missing from the sync update params
Pin-first batch for H3: freeze web annotation CRUD response shapes (create 201 / update 200 / delete 204; full sqlc row models) before the stack consolidation touches anything. The new pins immediately caught a live server bug the 28 unit tests could never see (annotations_test.go is pure-unit: keys and compare logic, no DB): applyNoteLWW built UpdateMediaNoteForSyncParams without UserID, so the query's WHERE id=$1 AND user_id=$13 compared against zero — every dedup-hit note re-save (same content+position) returned 500 "update note: no rows in result set" and the update never landed. The bookmark twin passes UserID and works; the note stack is the clone that dropped the field. Fix: pass UserID in the params (one line). Pins go green and become the regression net; changelog records the new surfaces. |
||
|
John O'Keefe
|
de9e66636a |
docs(agents): test-failure triage discipline — suspect the test first, but test edits stay asymmetric
2026-10-17 lessons from the hardening session: most test/bruno failures were harness/schema/environment drift (drifted body keys, truncated payloads, container-only paths), found by triaging the test before the server. Codified as: triage order (test schema -> environment -> cheap probe), with the guard against "writing tests to work" — test/contract edits are asymmetric with code fixes: only harness plumbing fixes, additive pin widenings with a changelog entry, or deletions of tests of removed behavior. Pins were green at a known commit; git history arbitrates stale-vs-changed. A right-looking test that fails is prima facie a server bug: stop and flag. contract_pins.md rule 4: narrowing/removing a pinned key requires the user's explicit sign-off — a pin freezing buggy behavior is a ledger, not a license. |
||
|
John O'Keefe
|
28bd323505 |
refactor(progress): SaveProgressRequest becomes the canonical PUT payload; GET serializes from the row
Batch H1 steps 1-2 of the hardening plan. Wire contracts frozen; the pinned shapes are unchanged modulo the whitelisted additive GET keys (docs/contract_pins.md). 1. One canonical request struct. The web PUT binds SaveProgressRequest directly (identity/attribution fields are json:"-" so they stay server-owned) and copies its server fields once — the hand-rolled DTO and the 20-line field-copy block are gone. Parse set for existing clients is IDENTICAL to the old DTO (verified in web/src: the ebook reader sends percentage/current_page/total_pages/reading_mode/epubcfi/ context_text/chapter/zoom_level — reader.ts:3113-3143; the audiobook reader sends percentage/chapter/audio_offset_seconds/elapsed_seconds/ reading_mode — audio.ts:518-520). 2. GET /api/media-items/:id/progress now serializes through one row-struct-driven builder instead of a hand-map, so a row column cannot be silently forgotten again (the audio_offset_seconds omission). Existing keys are verbatim; the row already carried audio_offset_seconds/reading_mode/viewport_x-y/zoom_level/ scroll_position_x-y/panel_number and they surface as WHITELISTED additive keys. The web audiobook reader already reads audio_offset_seconds (audio.ts:219) — with the key absent, durationless books resumed at chapter start (offset always 0); they now resume correctly. The conditional context_text bolt-on and the server-derived css_selector/anchor_href block are unchanged — restore precedence stays percentage-first, never re-derived. |
||
|
John O'Keefe
|
a2ea34232c |
test(contracts): pin wire shapes before the canonical-types migrations
Batch H0 of the hardening plan: freeze the JSON key SETS of every
load-bearing surface so H1-H7 can only change them additively.
- cmd/server/tests/contract_pins_test.go: 14 tests / ~30 subtests pinning
progress GET+PUT, the KOReader plugin surfaces (progress push/pull,
library, bookmarks), the Kobo adapter shapes (initialization, markup,
sync-from-server), collections detail all four branches + preview,
series JSON, search rows, dashboard sections, media list BOTH branches
(raw sqlc rows vs resolved 38-key map), and SSR smoke for
bookshelf/collections/series/tags.
- docs/contract_pins.md: the pinned-shape inventory + the rule that a
batch widening a shape must widen the pin and this doc in one commit.
Suite is green on the unmodified tree at
|