178b1f30c5ed2456650105a3459efef5fc7f8585
221
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
cce488b76f |
fix(filter-builder): multi-pick id values matched nothing; hint the multi boxes
Comma-joined multi-picks matched ZERO: clauseFeed/clauseLibrary/ clauseCollection parsed each rule value as ONE uuid, but the builder's pick loop stores several picks as a single comma-joined value (["id1,id2"]) — uuid.Parse failed on the joined string, the id list came back empty, and "feeds in [A, B]" matched no episodes (one pick worked, which is why the briefing recipe survived). parseUUIDValues now splits every value on commas (trimming) before parsing, accepting both the builder's joined shape and hand-written separate elements — same leniency the suggest-side id lookup already has. The multi-value cells now say so: id-kind value inputs get the placeholder "pick one or more" (the only boxes on the page that accept several values; every other cell replaces its value). Pins: comma_joined_multi_feed_matches_both — a tree collection built with the exact comma-joined wire the client sends, asserting BOTH feeds' episodes match. Filter-builder vitest 35/35, container filter + episode-tree slices green. |
||
|
John O'Keefe
|
09f8f0875f |
fix(filter-builder): id labels survive Apply — stored ids resolve to names
The library cell reverted to the raw uuid after Apply: the apply flow reloads the page, the SSR-preloaded tree re-ingests, and idLabels — populated only by suggestion PICKS — starts empty, so valueCellText falls back to the id. Same on any fresh load of a stored id-rule. Two halves: - Server: the id-source suggest query's typed filter (library/ collection/added_by/feed) now also exact-matches a comma-separated ID list alongside the name ILIKE — so a suggest call with the stored ids returns their labeled rows. No new endpoint; the menu and the resolver share one surface. - Client: fetchSuggestions gains a QUIET mode (fills idLabels, touches no menu state), id rows fall back to their stored ids as the typed lookup when no draft is present, and init() calls resolveIdLabels() after tree ingest — one quiet fetch per id-kind rule resolves every stored id to its name from the first paint after reload. Pins: the Go side (suggest with a library's UUID returns the row with the library NAME as display), the client side (resolveIdLabels fills labels, the cell renders the name, no menu opens). 35 filter-builder vitest cases green; container filter slice green. |
||
|
John O'Keefe
|
7c7e282e79 |
feat(dashboard): custom-section builder saves again — the flagged tree migration
Archaeology, prompted by the user asking when it broke: the builder's save NEVER worked. The page was born (2026-02-19, |
||
|
John O'Keefe
|
a86bdac87e |
refactor(collections): retire the legacy rules surface — tree-only storage
The pre-tree rules machinery is deleted: no row ever carried a rules array or a non-default match mode (verified in the dev DB), the web never read the envelope keys, and the Android app consumes none of it. Dead code gets no documentation. Removed: - PUT /api/collections/:id/match-mode — endpoint, route, and the UpdateCollectionMatchMode query. The tree's root mode owns any|all. - auto_assign_rules / rule_match_mode from every envelope (create/list/detail/update), from the create/update request shapes, and from the schema: both columns DROPPED (schema.sql carries idempotent DROP COLUMN IF EXISTS heals for existing DBs; sqlc regenerated). - The rules CRUD service layer (AddRule/SetRuleEnabled/RemoveRule + validateRule + UpdateCollectionRulesJSON), ParseRules / EnabledRules / IsSmartCollection, RulesToFilterTree (+ the SSR preload branch), the legacy branch of EvaluateSmartMembership, and SmartEvalOptions.MatchMode. IsCollectionSmart = tree presence; create-path isSmart = collection_type=smart OR a filter_tree. Kept (live consumers): POST /api/collections/preview still evaluates rules payloads in memory (the custom-section builder's preview), and include_auto stays on the list (the Android app sends it). The custom-section builder's SAVE was already broken — its stringified auto_assign_rules payload failed request binding (probed live: 400 invalid request). Create now rejects that payload loudly with a 400 naming the retirement, instead of silently creating a hollow static section; custom-section-builder.md carries the flag. The builder's tree migration remains the flagged follow-up. Test-side (asymmetry rule — deleted behavior, deleted tests): TestPodcastSmartCollections rewritten onto trees (mode lives in the tree root), match-mode subtests and tree_wins_over_legacy_rules deleted, TestRulesToFilterTree deleted, contract-pin envelope keys retracted per the contract_pins 2026-10-31 retirement entry. Mirrors hand-synced: openapi.yaml, bruno create/update, docs/api/*.md, docs/schema.go, api.d.ts. Full container suite green (155s). |
||
|
John O'Keefe
|
ab98015190 |
feat(collections): RLS ownership enforcement at the DB layer
collections + collection_items join the RLS-governed tables, same doctrine as the media_items family lens (2026-10-29). Policies route through two PURE predicate functions (app_can_read_collection / app_can_write_collection) that never query collections — no policy recursion under FORCE RLS — so a future sharing model is one EXISTS branch inside the functions, not a schema re-touch. Reads: owner or admin; writes: owner, admin escape for SYSTEM collections only. Why: the write path had no app-level ownership gate on Update/Delete/ AddBooks — any authenticated user could mutate another user's collection by id. The DB is now the enforcement of record; existing app-level gates (match-mode, reorder, view-settings) stay as defense-in-depth. Behavior changes (contract_pins 2026-10-31): - cross-user GET detail: 200 -> 404 (row invisible) - cross-user PUT update: silent success -> 404 (ErrNoRows mapping; service wrap fixed %v -> %w so errors.Is sees pgx.ErrNoRows) - cross-user POST books: silent success -> 404 - cross-user DELETE: 204, now a true DB no-op - anonymous sees nothing; admin persona unchanged Bootstrap: registration seeds system collections BEFORE the JWT exists — CreateDefaultCollectionsForUser declares the new user's identity (rls.WithUser), the same fix mirrored in the test harness (createDefaultCollectionsForUser owner-scoped; collection sweeps run under the owner's identity). Tests: new TestCollectionsRLS pins the cross-user matrix end-to-end (404 detail/update/add/view-settings, 204 no-op delete, admin escape, owner positive control, registration seeding under RLS). Harness fixture writes to non-system collections now run as the row's OWNER (scopeExecAs/ownerSeedCtx) — the admin write-escape covers system rows only, and bare-ctx pool reads of collections see nothing. Filter/playlist/audiobook/scan-state fixtures converted. Full container suite green (159s). |
||
|
John O'Keefe
|
fd716a7dab |
fix(filters): the episodes-scope vocabulary buried the episode fields under book-only metadata
The scope door's first live test showed the episodes editor leading its field menu with People (Author/Narrator/Contributor) and Publishing — book-only fields that can never match an episode row — while the fields the scope exists for (Feed, Downloaded, Played) sat below the fold. Root cause: the registry's book fields carry no Scopes restriction (nil = all scopes), a leftover from before the episode fields existed. 35 unambiguous book-only fields (people, publishing metadata, topics, series/comics, identifiers, cover, reading-specific statuses) are now tagged Scopes:books. Title, description, published dates, language, file facts, explicit, the my-reading group (listening progress is real), and collection stay dual-scope. Parse continues to accept the hidden fields everywhere (their composed clauses simply match nothing on episode rows) — this governs the AUTHORING surface, not the grammar. TestFilterParseValidation's empty-root rejection updated to the new contract (empty root legal — the seed; nested empty groups still invalid), and TestEpisodesScopeHidesBookOnlyFields pins both scopes' vocabulary. |
||
|
John O'Keefe
|
cc0e9d7f91 |
feat(podcasts): the episodes authoring door + curated chain order
"Newest episodes from my selected podcasts" is now buildable in the UI, end to end — the user could not find ANY podcast-related filter surface because none existed: the New Collection modal could only create book-scope smart collections, and the engine's episodes scope (item 3.1) had no authoring door. The scope door: - The create modal's Smart card gains a Rules-scope segmented control (📚 Books / 🎧 Podcast episodes); the copy no longer claims smart means "match books by rules". Choosing Episodes seeds an EMPTY episodes tree ({"scope":"episodes","mode":"all","children":[]}) so the collection is born episode-scoped: the editor opens with the episode vocabulary (feed/downloaded/played/published date) and membership runs the episode engine from the first rule. - ParseFilterTree now accepts an EMPTY ROOT (nested empty groups stay invalid) — the seed's legal shape. The root arrives at depth 1 by call convention, hence the depth>1 exemption. - The composers define the empty-tree semantics: zero children matches NOTHING (ComposedLiveMedia/ComposedCount guards), mirroring EvaluateSmartCollection's zero-enabled-rules convention. Without the guard the composed query degrades to the bare scope head and a fresh collection would swallow every episode in scope — pinned by TestEpisodeCollectionsEmptySeedMatchesNothing (envelope book_count 0) and TestEmptyRootTreeSeedsEpisodesScope. The curated chain order (plan §10 R2, user need: the briefing plays NEWEST-first): - view_settings.chain_order ("native"/"newest"/"oldest"/"reverse") stored on the collection — curated content, not a player-side knob. - PUT /api/collections/:id/view-settings merges the key into the stored JSONB (a blind write would clobber the playlist toggles). - buildSmartCollectionSequenceContext routes OrderEpisodes through handlers.CollectionChainOrder; the detail header's Play-as-Sequence gains the order select next to it (PlayCollectionButton), which PUTs and reloads. Pinned by chain_follows_the_curated_order (reverse flips the chain indexes and hops). Caught by the new test, a latent clobber fixed: UpdateCollection treated a NIL view_settings payload as "write NULL" — any edit that carried no settings dropped every stored key (chain_order, playlist toggles from API clients, imported provenance). Nil now means untouched, the same convention filterTree already had. Composer guards also cover ComposedCount, so the builder's live count reads the same semantics. Tests: collectionChainOrder unit matrix, empty-root parse unit pin, two integration pins (seed matches nothing; curated order drives the chain and survives whole-map edits). |
||
|
John O'Keefe
|
eb68672390 |
feat(filters): one vocabulary, server-rendered — the TS mirror retires (item 3.3)
"One filter system to maintain" now holds at the presentation layer too: the field/op list the builder renders comes from services.FilterFieldsPayload(scope) — the Go registry's wire face, scope-filtered and group-ordered — rendered into the bookshelf mount as a data attribute (templates.JSONArg), ingested once by the Alpine component. The registry carries operator presentation (labels, negation, arity) in FilterOpWire alongside the grammar; per-field value-cell semantics (valueKind) derive from the existing Kind. The hardcoded TS mirror (OPS, op(), F(), TEXT_OPS/NUM_OPS/DATE_OPS, FIELD_GROUPS, its known drift surface) is deleted; filter_builder.ts now ingests the wire (ingestFilterFields — malformed payloads leave empty menus rather than client-invented rows), and the field menu's episode fields are HIDDEN on the All-Books page for free (the books scope payload excludes them — the plan's "hide podcasts from the rules on the all books page" is now structural, not enforced by a second hand-kept list). Unit tests ingest a minimal wire fixture (the payload shape is the contract, not the mirror); vitest 167 green, tsc + vp clean. |
||
|
John O'Keefe
|
6b2125a45a |
feat(podcasts): smart collections ride the unified filter engine — filter_tree column (item 3.2)
collections gains filter_tree JSONB (idempotent in-place; same qf DSL as saved filters — the scope key included). IsCollectionSmart is the ONE detection (tree OR legacy rules), and EvaluateSmartMembership is the ONE evaluator: tree-bearing collections run the composed engine via ComposedLiveMedia (full media rows, so the collection surfaces keep their episode-card stamping, feed positions for the replay chain, and the family lens rides the mandatory head), while legacy rule arrays keep per-item FieldMatches until rewritten — one call surface, two storage epochs. All smart surfaces (detail JSON, counts, detail page, reader chain) dispatch through it; the search service is injected into the collection subsystem (variadic — main.go builds one shared instance; the test harness mirrors it). Write path: CreateCollection/UpdateCollection accept the tree, validated against the engine grammar at the HANDLER (an invalid tree is a 400, never a stored render-time 500) and exposed in the envelope (pin widened in TestContractPin_CollectionsJSON + changelog). Update semantics: an edit form that doesn't send a tree keeps the stored one. RulesToFilterTree converts legacy rules + match mode at upgrade time: episode rules map mechanically (feed→in-list, played/downloaded→ booleans, published_within→published_date+within_days), tree root mode absorbs rule_match_mode, episode rules force the episodes scope, and legacy starts_with/ends_with fall back to contains (documented). Read-compat by design: nothing is migrated at boot — legacy arrays evaluate exactly as before until the collection is rewritten. Pinned by TestRulesToFilterTree (unit conversion) + TestEpisodeCollectionsFilterTree (a tree collection evaluates through the engine, the chain follows the tree's live membership, invalid trees 400 at write, the tree wins when legacy rules coexist, and an edit without a tree keeps it). |
||
|
John O'Keefe
|
56871b65ed |
feat(filters): scopes + episode fields — the engine becomes the single query system (item 3.1)
The qf tree gains an optional top-level `scope` ("books" default |
"episodes" — absent = books, so every existing saved tree and legacy
client stays valid). The row head is scope-aware AND this closes a
real leak: the composed engine's head had NO podcasts exclusion, so
episodes already surfaced on All-Books qf queries (plan §9 violation —
the sqlc flat head excluded them, the composed head didn't). Books
scope now says library_type_name <> 'podcasts'; episodes scope is
source = 'podcast_episode'.
The registry fields declare their scopes (the plan's "reusable
template with conditionals" = scope-driven rendering in 3.3): episode
fields declare exactly the episodes scope and the books scope REJECTS
them at parse (the "hide podcasts from the rules on the all books
page" rule enforced server-side); books fields inside episodes trees
are allowed and honestly no-op on NULLs.
New fields (the item-2.4 rule vocabulary expressed in SQL):
- feed — KindIdentifier over podcast_feeds (RLS-governed suggestion
source and match; unsubscribed feeds match nothing, by the engine)
- downloaded — is_true/is_false on download_state = 'downloaded'
- played — THE canonical finished predicate (viewer progress >= 0.999,
the hide_finished definition shared with item 2.4), reading_progress
join per viewer
- published_date — clauseDateFactory like added_at, within_days
default op (the published_within rule's SQL form)
Engine hygiene: the stray ComposedSearch SQL debug printf is gone.
Pinned by TestFilterScopeValidation (unit: scope grammar, books×
episode-field 400s, legacy no-scope trees) and
TestFilterScopesAndEpisodeFields (integration: books head excludes
podcasts including the 404-no-results contract, episodes head is
episodes only, feed-in/downloaded/played/published-date evaluate in
SQL against a subscribed-feed fixture with a played viewer).
|
||
|
John O'Keefe
|
abb7c84833 |
feat(podcasts): smart episode collections — ALL/ANY match modes + episode rule fields (item 2.4)
The daily briefing needs AND: today's rules engine was OR-only. The
match semantics become a per-collection column
(collections.rule_match_mode, 'any' legacy default | 'all'), flipped on
its own endpoint (PUT /api/collections/:id/match-mode, owner-only —
nothing else on the row changes) and surfaced in the rules page as an
ANY/ALL toggle with matching copy.
Episode rule fields join the vocabulary (plan item 2.4): feed (the
multi-select rides the new `in` operator — comma-separated feed ids),
played (the canonical finished predicate — viewer progress ≥ 0.999,
same as hide_finished), downloaded, published_within (N days), plus
title/description contains. The rules candidate query now carries the
viewer's progress (LEFT JOIN reading_progress) so played/unplayed is
per-user, and TestRules/preview speak the SAME evaluator: the handler's
duplicated field switch is deleted and delegates to
CollectionService.FieldMatches — one vocabulary, one semantics.
Legitimate semantics note: in ANY mode `played:false` matches unplayed
BOOKS too (everything with no progress row) — correct per-rule
behavior, and exactly why the briefing mode exists.
Backward compatibility: every legacy row defaults to 'any' (the old
OR), UpdateCollection keeps the stored mode when the form doesn't send
one, and the evaluator treats absent data honestly (no date → no
match; no progress → unplayed).
Pinned by TestPodcastSmartCollections: the AND-mode briefing
(feed∈{A} ∧ unplayed ∧ <7d = exactly the new episode), OR-mode
widening, owner-only flips, the preview honoring the mode, and
downloaded/contains rules. Also fixes a latent bug the new query
surfaced: UpdateCollectionViewSettings referenced a nonexistent
updated_at column (never exercised until now — the new sibling query
was written without it).
|
||
|
John O'Keefe
|
0741e98ea5 |
feat(admin): all-podcasts view — disk-first maintenance for every feed
§0 item 8 (Phase-4 item 1 pulled forward, user directive: maintenance
tooling ships WITH the features it maintains):
- /admin/podcasts (AdminMiddleware): every feed from EVERY user via the
RLS admin escape (first-admin identity, poller/fetcher contract) —
feed + host, library, subscriber count/names, downloaded/total, DISK
USAGE, last fetch. Sorted disk DESC ('what's eating the disk'),
totals header (feeds · episodes · bytes on disk).
- Per-feed actions: Refresh · Delete files (rows → pending, disk freed;
confirms spell out the re-materialization contract) · Remove feed
(rows + files for everyone — the same admin delete as the detail
page). All confirms ride the global confirm modal.
- New sqlc: ListAllPodcastFeedsAdmin (correlated aggregates),
ResetPodcastEpisodeFileDeleted / ResetPodcastFeedFilesDeleted —
downloaded rows reset to pending with download-derived facts cleared
(audio_metadata, chapter_count, cover, size, mime); file_path KEPT
(fetch-maintained layout — a re-download lands on the same path).
file deletion via the existing DeleteFeedFiles file-removal pattern
(episode files + .cover.jpg siblings).
- Poll upsert never touches download_state, so deleted files STAY
deleted — a feed refresh re-lists metadata but re-materializes
nothing (pinned by TestAdminDeleteEpisodeFiles).
- Per-episode admin delete-file: POST /feeds/:id/episodes/:eid/
delete-file (admin) — one episode's file gone, row pending, siblings
untouched; HX swaps the episode list. (Found while testing: the
handler first read c.Param("id") — the FEED's uuid — instead of
:eid; the 404 'episode not found' trail led straight to it.)
- Tests: admin sees other users' feeds + subscriber names; regular
users 403 on page AND actions; single + feed-level deletion reset
rows/clear facts/delete files; refresh leaves pending rows pending.
- Admin nav gains 'Podcasts' under Administration.
Wrong-gotcha note for future agents: parseUUID on c.Param('id') inside
a /feeds/:id/episodes/:eid route resolves the FEED, not the episode.
|
||
|
John O'Keefe
|
b574df346a |
chore(podcasts): drop the dead QueueFeedDownloads wrapper
All callers moved to QueueFeedDownloadsAs in the item-5 commit; the no-override shim had no remaining callers. |
||
|
John O'Keefe
|
5c545534c2 |
feat(podcasts): Download All queues in the LISTENING order + add-modal order dropdown
Download All previously queued in feed-native order — 'getting the last chapter first doesn't help me'. Now the queue follows the feed's effective listening order (§0 item 5b): QueueFeedDownloadsAs reads the CALLER's order override and runs episodes through services.OrderEpisodes (same helper as the sequence/reader/page surfaces — the consolidation from item 5a landed with the reverse-order commit). Queue tickets (SEQUENCE-stamped, item 4) make that enqueue order the claim order, so chapters land in the order you'll actually listen. Add modal (§0 item 5c): with 'Download all episodes now' checked, a 'Download order' select appears (pure-CSS :checked sibling reveal — no JS island, CSP-clean). Its choice is the feed's INITIAL order override (reader_settings upsert; 'feed' = no row) AND the queue order for the initial batch. Per-book by design: the override IS the download order. Test TestPodcastAddModalDownloadOrder: add with download_all+reverse → the override row exists AND persisted tickets run position 3→1. Tickets are asserted (not live queued rows) because the shared dev server's worker legitimately races this test's queue — tickets survive every state flip. |
||
|
John O'Keefe
|
0f3c4a8cec |
feat(podcasts): concurrent download pool — fair share, tickets, admin tunables
Replaces the sequential worker with a download-manager-style pool (§0 item 4). Model: the user's connection is the limit; semantics are fair-share ACROSS feeds, not goroutine-per-book (a global FIFO would make Book B wait behind Book A's 100 files). - Atomic claim: GetNextQueuedEpisode + SetPodcastEpisodeDownloading was a read-then-write race under concurrency; ClaimQueuedPodcastEpisode converts queued→downloading in ONE guarded statement (two claimants, one winner — the loser gets ErrNoRows). SetPodcastEpisodeDownloading retired. - Queue tickets: download_queue_pos BIGINT fed by a Postgres SEQUENCE stamped at enqueue — the enqueue loop's ordering IS the claim order (the foundation §0 item 5 builds on; previously all rows in a batch shared one updated_at, so within-feed order was arbitrary). - Pool: up to podcast_download_concurrency (default 4, live range 1–16) claim-loop goroutines per drain round; the setting re-reads every round so admin changes take effect without restart. - Fair share: claim prefers the feed least-recently-served (pickFeedToServe; never-served first), earliest ticket within a feed — books proceed in parallel while chapters land in order (start listening while the rest download). - Bandwidth: shared token bucket across all workers (podcast_download_speed_mbps, default 0 = unlimited); take() reserves bandwidth and callers sleep outside the lock. - Settings: new float setting type end to end (registry defaults + clamping, save validation, admin UI number-with-step input) + the two schema seeds; worker wires the registry via SetSettings (tests run on registered defaults). - Tests: unit coverage for the bucket reservation math + fair-share pick; integration TestPodcastDownloadPool pins ticket order = enqueue order, claim exclusivity, and a full 3-episode pool drain. Dev-DB note: the column/sequence/settings were applied live (idempotent DDL, same statements the schema runs at next boot) since the running container predates them. |
||
|
John O'Keefe
|
e3675536aa |
feat(podcasts): reverse order — position DESC, date-agnostic; single order helper
The 'Oldest first' order sorts by PUB DATE, which breaks on Scribl's migrated podiobooks feeds (messy/all-equal dates). New order value 'reverse' = feed position DESC with a date-DESC tiebreak — dates are not involved, so nothing can break. - services.OrderEpisodes: THE single order source of truth (§0 items 3+5a). The four surfaces (sequence JSON, reader chain context, SSR page list) previously carried three near-identical handwritten comparators — all now route through one generic helper. The SSR page list gains a SortDate key (lexical 2006-01-02) so the pub-date tiebreak is actually chronological; the old comparator compared 'Jan 2, 2006' display strings alphabetically (latent drift bug). - Dropdown becomes four options: Feed order / Feed order (reversed) / Oldest first (pub date) / Newest first (pub date). - Serial suggestion button renamed 'Reverse feed order' (sets reverse, not oldest) and its response out-of-band-swaps the order select (PodcastOrderSelectOOB — same mechanism as Download All's policy select), so the dropdown mirrors the server state immediately. - Validation: oneof=native reverse oldest newest everywhere. - Tests: reverse binds via the form wire; messy-dates sequence returns 3,2,1; existing reorder-safety stays green. |
||
|
John O'Keefe
|
253d8d0e8e |
feat(podcasts+ops): Apple launcher-scheme feeds + visible build stamp
Release / build-and-push (push) Successful in 3m29s
Apple scheme normalization (follow-up to the Scribl discussion): Add Feed accepts the launcher spellings podcast pages ship next to plain RSS — itpc://, podcasts://, feed:// — and rewrites them to the https feed URL they wrap (scheme match case-insensitive). http stays and the guard still rejects it. Unit-pinned. Build stamp (stale-container drift guard): the 2026-10-29 podcast session lost an hour to a container running a build that predated the feature. version.Commit rides ldflags from a GIT_COMMIT build arg; the dev compose passes git rev-parse --short HEAD via the Makefile; the app logs 'Bookhoard <ver> (commit <hash>)' at boot and /api/health reports it — the UI and the repo can no longer silently diverge. |
||
|
John O'Keefe
|
30766020c8 |
feat(podcasts): real-time episode list — byte progress bars, direct swaps (issues #1/#3)
WHY: downloads needed manual refreshes to see progress; the 204 + HX-Trigger event indirection missed its listener, and the worker buffered whole enclosures so no progress existed to show. - Worker streams enclosures to disk through a counting loop (64KiB chunks, scheme/redirect/size guards preserved) with an in-memory per-episode progress snapshot (received/total; total from Content-Length, seeded by the feed's declared length). Deliberately transient — the DB row state remains the durable truth. - Episode-list renderer seam: the router injects a closure into PodcastHandler (handlers can't import templates — MediaDetail cycle) so htmx download/retry/Download All/refresh/order/policy actions answer with the REFRESHED partial directly (200, hx-target= #episode-list) — no events to miss. Non-HX callers keep JSON; tests without a renderer keep 204+trigger. - Polling tightens to 2s while downloads are active; downloading rows render a percentage bar + MB counter (quote-free inline style per templ gotcha #8) or an indeterminate bar when the server sends no length. - Issue #3: Download All (button AND the add-feed checkbox) flips the CALLER's subscription policy to 'all' — per-user, never global (D6: files shared, policies not) — and the auto-download select out-of-band-swaps so the dropdown shows what the user is getting. They can turn it down afterwards; the flip is honest communication, not coercion (user-approved 2026-10-29). PINs: Download All with HX answers 200 + episode-list + oob select and flips newest → all; add-feed download_all sets policy all. |
||
|
John O'Keefe
|
8d9fb0046c |
refactor(podcasts): drop the dead downloadRoot param on upsertEpisode
FilePath is library-relative (the pinned layout resolves at download time via the library folders); the parameter never carried a value. |
||
|
John O'Keefe
|
49b0b6eeb9 |
test(podcasts): integration suite + RLS auditor + contract widening (task 1.10)
cmd/server/tests/podcast_test.go — six suites, the shared-dev-DB doctrine (test*-named libraries, cascading cleanup, no binary fixtures): - TestPodcastRLSSurfaces: THE AUDITOR (invariant 13) — non-subscriber 404 on feed detail + SSR page (existence leak), episodes absent from generic media reads, admin escape, subscriber visibility, then the hidden-library flip hiding an already-subscribed feed (the feed policy's library-visibility join). Seeding runs as the FIRST ADMIN — the seed identity has neither subscription nor users row, so podcast INSERT..RETURNING (SELECT policy under FORCE) 42501s for it; production pods run as exactly that admin (invariant 12). All direct row probes read through the admin scope for the same reason. - TestPodcastAddFeedFlow: member add + subscribe + metadata-only upsert; duplicate-URL dedupe (no second row); allow_member_feeds gate (member 403 / admin 201 / subscribe-still-allowed). - TestPodcastShrunkenFeedRetention: feed purges an episode from the XML, both rows persist (upsert-without-delete pin, D14). - TestPodcastEditSegregation: generic edit 400 + write-to-file refusal for episodes; control book edit unaffected. - TestPodcastDownloadWorker (ffmpeg-generated tone at runtime): file lands at <download-root>/<feed-slug>/NNNN-…, audio_metadata probed; re-download of a downloaded episode never re-fetches (idempotency pin, enclosure-hit counted); dead enclosure → failed + persisted reason + metadata intact. Fetch/SSRF tests swap the client via SetPodcastHTTPClientForTests (the production client refuses loopback httptest servers by design — the real client gets its own negative test in services). - TestPodcastScannerExclusion: SetFolders on a podcasts library root yields zero scanner folders (invariant 4 pin) — MediaScanner .FolderCount() accessor added. Unit: internal/services/podcast_rss_test.go — namespace-aware parsing fixture (three XML namespaces, fallback chains, hosts/guests), duration spellings, RFC-822 variants, explicit variants, slug/path layout, enclosure-extension selection, stripHTML, and the SSRF matrix (private/loopback/link-local/multicast/mapped-v6 blocked; a loopback httptest server PROVES the production client refuses it). suites green: full integration run 161s / 0 failures; vitest 12 files 163 tests. Contract widening: TestContractPin_MediaListBothBranches' raw-row allowlist gains the 15 podcast episode columns (additive; documented in docs/contract_pins.md — inert defaults on scan rows). Bruno: bruno/ podcasts/ mirrors the §8 endpoints (12 requests) + Bookhoard env vars. tests/ui/COVERAGE.md rows 18-23: podcasts pages/lifecycle/RLS/chain — Playwright sweep pending. |
||
|
John O'Keefe
|
54d7d6e7a1 |
feat(podcasts): edit segregation — feed metadata is fetcher-owned (task 1.8)
D9/D10: users never edit podcast episode metadata. Three gates: - Generic edit endpoint (UpdateMediaItem) rejects source= 'podcast_episode' items with 400 — the next fetch refresh would clobber any edit anyway, and the fetcher is the sole metadata writer. - Write-to-file (MetadataWriter.WriteMetadataToFile) refuses podcast episodes for the same reason (tag write-back for downloaded episodes is a named Phase-4 item needing its own iTunes-tag mapping). - The book-detail page hides the admin Edit affordance for podcasts-type items (direct-URL visits), and the metadata-editor modal's field list has no podcast fields by construction (fixed templ field list) — episodes render read-only via /podcasts surfaces. |
||
|
John O'Keefe
|
a6723e6ac8 |
feat(podcasts): feed poller + podcast_poll_interval_seconds setting (task 1.6)
One ticker re-reading its interval from the settings registry each cycle (live tunable — the cache stays honest because writes go through the settings API path, AGENTS.md gotcha #2). Per tick: list feeds (first-admin identity, invariant 12), skip is_complete feeds (back off; explicit refresh still works), poll each (conditional GET via stored validators), and apply the subscriber auto-download policy union after each successful fetch so new episodes queue without waiting a tick. Schema/registry: podcast_poll_interval_seconds (default 3600s, floor 300s, category podcasts) — seeded with ON CONFLICT DO NOTHING per the idempotency rule. |
||
|
John O'Keefe
|
4288a86b2d |
feat(podcasts): download worker — state machine, idempotency, policy union (task 1.5)
One sequential worker (day one) over a DB-backed queue (download_state='queued' IS the worklist — the episode list always renders truth, D8/D19). State machine: pending → queued → downloading → downloaded | failed (+ download_error + download_attempts). Idempotency: downloaded re-request is a no-op; a file already on disk that probes cleanly re-marks the row without re-fetching (crash-between-move-and-update reconciliation). Retry: manual path resets attempts (human override), auto policies re-queue up to maxAutoDownloadAttempts=5 then stay failed until a human retries. Pipeline: enclosure fetch (same SSRF/size guards as the fetcher) → .part temp → declared-size pre-check → MIME/size validation (empty / over-cap / content-sniff refuses non-audio) → ffprobe reconcile + embedded-cover extraction (scanner machinery reuse: runFFProbe, extractEmbeddedAudioCover, AudioMetadataPayload) → atomic rename → SUCCESS-ONLY row update (invariant 7: a failure never clears existing episode data). Policy union (D6): subscriber policies per feed — 'all' → everything missing; 'newest N' → the N tail positions; 'off' contributes nothing; auto-retry budget respected at queue time. DeleteFeedFiles for the admin-only removal path (episode files + cover siblings + artwork). |
||
|
John O'Keefe
|
85a9873a75 |
feat(podcasts): RSS fetcher — parse, SSRF guards, upsert-without-delete (task 1.4)
The fetcher is the SOLE writer of feed/episode metadata (D9). Three files, no new deps (stdlib encoding/xml + net/http): podcast_rss.go — RSS 2.0 + iTunes + Podcasting-2.0 structs. iTunes fields are namespace-qualified in tags (local names collide with core RSS: title/subtitle/author/summary/explicit/image/duration); podcast:* fields match on local name (collision-free, tolerant of both declared namespace URIs). Description/author fallback chains per §5; itunes:duration parses seconds/HH:MM:SS/MM:SS/prose; pubDate accepts the RFC-822/1123 spellings feeds actually emit. podcast_http.go — the security boundary for user-supplied URLs: https-only on every hop (CheckRedirect), private/loopback/link-local/ multicast IP ranges blocked at DIAL time (Dialer.Control sees the RESOLVED address — DNS rebinding into 127.0.0.1 is refused at the socket), 10-redirect cap, hard body caps (25MiB feeds / 20MiB artwork / 1GiB enclosures), no proxy. podcast_service.go — fetch → parse → upsert. Episodes upsert metadata-only keyed on (source_feed_id, episode_guid); rows for items that vanish from the XML are NEVER deleted (D14 search covers historical episodes). itunes:new-feed-url + permanent HTTP redirects rewrite feed_url in place (row/subscriptions/episodes stay put); podcast:guid dedupe reuses the canonical row instead of forking. Artwork downloads feed-level under <feed-slug>/artwork.<ext>. AddFeed authorizes ONLY on explicit arguments (library visible + allow_member_feeds for non-admins — the D20 scoping model); detached work runs under the first-admin identity via adminCtx (invariant 12). Feed layout: <download-root>/<feed-slug>/<pos:04d>-<slug>.<ext> so ServeFile/Range streaming resolve unchanged. |
||
|
John O'Keefe
|
9a148c2344 |
feat(podcasts): generic-surface exclusions — podcasts never render as books (task 1.3)
§9 / D3 / D13 / D15 / D16: podcasts-type libraries are excluded from every generic book surface. Under RLS these are pure UX/type filters (engine already hides other users' episodes); their job is preventing episode grids in surfaces designed for books. The discriminator is the denormalized media_items.library_type_name (IS DISTINCT FROM 'podcasts' — NULL rows predate the column and stay visible). SQL (queries.sql, marked '-- podcasts exclusion (plan §9)'): ListMediaItems/ListMediaItemsSorted (All Books) · SearchMediaItems + SearchMediaItemsUnified (generic search) · FacetCountsUnderFilters (all 14 arms) + the six Search*Values facet-suggestion queries · GetDistinctSeries + Count (series pages) · GetLibraryItems (collections picker) · dashboard system sections (Continue Reading / Recently Added / Recently Read / Not Started) · GetPopularBooks + GetTopBooksByMinutes (book rankings must not rank episodes). Go: statistics summary counts episodes' sessions into time-spent (D15 keeps that) but skips them in BooksActive/BooksFinished — an episode is not a book. OPDS filters podcasts libraries via one helper at all 7 library-walk sites (D16). Templ: LibrarySwitcher skips podcasts entries and TotalMediaCount excludes them — presentation-layer filter so every switcher consumer (dashboard, collections, custom-section builder) is correct at once (D13); podcasts surface via /podcasts only. |
||
|
John O'Keefe
|
c0e6993273 |
feat(podcasts): scanner exclusion — podcasts libraries are feed-managed (task 1.2)
WHY: podcast download roots hold fetcher-written episode files; the FS
walker must never discover them (invariant 4) — a walk would double-
ingest episodes as scanner items and fight the fetcher's sole-writer
rule (D9). Every scan/watch path funnels through MediaScanner.SetFolders
(StartScanner, ScanLibrary jobs incl. scoped single-folder scans,
watch mode), so the filter lives there once: folders resolving to a
podcasts-type library are dropped before registration, with a log line.
Belt and braces: ScanLibrary's per-folder scope refuses purpose='podcast'
folders explicitly (400) — the panel-level path that bypasses type
resolution. AddLibraryFolder now derives purpose from the library type
('podcast' for podcasts libraries, enforced single download root;
'scan' default elsewhere), so folder purpose and library type cannot
diverge through the API.
|
||
|
John O'Keefe
|
0b7ad0f376 |
feat(podcasts): schema, RLS policy extension, sqlc queries — feed core foundations
Podcast integration task 1.1 (plan: podcast-integration.md §6/§8, D20).
WHY: podcasts are a new first-class media domain — RSS feeds users
subscribe to personally, with episodes as media_items rows so progress,
stats and the audio player work unchanged. Visibility is per-user
(strict: only your subscriptions), which after the RLS migration
(2026-10-29) belongs in the POLICIES, not Go helpers.
Schema (idempotent — schema.sql re-executes every boot):
- 'podcasts' library type seeded (ON CONFLICT DO NOTHING, ARRAY[]::TEXT[]
cast required — bare ARRAY[] 42802s under ON_ERROR_STOP)
- podcast_feeds: per-library feed entity (UNIQUE(library_id, feed_url)),
full RSS/iTunes/Podcasting-2.0 metadata inventory, conditional-GET
validators, is_complete poll backoff
- podcast_subscriptions: the ONLY user-writable podcast state (display
name, auto-download policy); deliberately UNGOVERNED — it IS the read
policy's lookup table
- media_items episode columns: source/source_feed_id/episode_guid/
feed_position/season/episode/enclosure_url/declared_duration/
transcript_url/chapters_url/hosts/guests + download state machine
(pending|queued|downloading|downloaded|failed + error + attempts),
all partial-indexed so scan rows never touch them
- libraries.allow_member_feeds (first per-library knob); library_folders
.purpose ('podcast' download roots, scanner-skipped)
RLS (inside the labeled policy block):
- p_media_items_visible gains the podcast-episode subscription predicate
+ admin escape (source IS DISTINCT FROM 'podcast_episode' guard keeps
scan rows on the cheap path); unset identity still fails loud
- podcast_feeds governed identically: library-visibility join AND
subscriber-or-admin SELECT; permissive write stand-ins (mandatory at
ENABLE time; fetcher/worker declare first-admin, add-feed upsert uses
ON CONFLICT DO UPDATE RETURNING so creators never need SELECT on
foreign rows)
- Verified against dev DB: migration identity sees nothing, admin sees
all, member sees feeds/episodes only post-subscribe (probed in a
rolled-back tx — no rows persisted)
sqlc: 30 new queries (feed upsert/CRUD, subscriptions, episode
upsert-with-write-after-success-guard, download state transitions,
poller worklist, subscription-scoped search) — no visibility clauses
anywhere, per the D20 scoping model.
|
||
|
John O'Keefe
|
bb2cc26e9d |
feat(scanner): scanner entrypoints + generic jobs carry RLS identity; SeedDB harness fallback
The full suite enumerated the remaining triage class: tests drive the scanner directly (ScanFolders(context.Background())) and read seeded media via Queries with bare contexts — every governed read raised. - MediaScanner.ScanFolders self-declares its configured admin (SetAdminID — set by main AND the test helpers): a scan is server maintenance, product-correct regardless of caller. - ScopedDB gains a Fallback identity, exposed as NewSeedDB for the TEST HARNESS ONLY: setup.DB falls back to rls.SeedIdentity when the context carries none, so direct seeding/verification helpers work uniformly instead of per-site context threading. rls.SeedIdentity is the shared constant (no library_visibility rows => full visibility). - generic job creation (jobs.go) stamps the requesting admin into the job Context, same as the scan endpoint. The sync queue processor stays identity-free: sync_queue is not an RLS-governed table. |
||
|
John O'Keefe
|
385a5d9dba |
feat(scanner): background jobs declare acting identity for RLS
Phase 4 of the RLS plan: detached contexts (context.Background()) that reach media_items now carry identities — with the policies live, a bare context made governed reads raise (fresh connections: 42704 unrecognized parameter; reused pool connections: 22P02 empty-string cast after the LOCAL GUC reverted — the plan's A2 signature). - scanner jobs (API-triggered, mtime-poll, initial scan): the job's Context now carries the acting admin's identity — the library owner, first-admin fallback (the job's UserID already encoded this trust decision; the Context just needed to carry it). performInitialScan uses the scanner's configured admin. - websocket initial-sync read (GetUserMediaItemsForSync): runs as the connection's own user. - hash backfill: runs as first admin (server self-heal over all media). Verified on the dev stack: forced rescan of the 1199-file Audiobooks library completed with 0 errors under RLS (previously every SHA lookup and create raised); hash-backfill refusals gone. The sync queue processor needs no identity — sync_queue is not an RLS-governed table. Known remaining detached reads (conflicts/analytics handlers using context.Background()) are HTTP-path triage, enumerated by the full suite next. |
||
|
John O'Keefe
|
795225f097 |
fix(scanner): reconcile archive pass per library, not per folder
Release / build-and-push (push) Successful in 3m5s
The archive pass looped over the scan's folders and, inside each
iteration, compared the ENTIRE library's item list against that one
folder's walk. With two registered folders every item living under the
other folder was flagged "missing (1/2)" on every reconciling scan —
prod's audiobook library (the only multi-folder one) showed 743/743
flagged — and nothing reached the two-consecutive-miss threshold only
because each scan's walk clears missing_scan_count before the pass
re-flags it. With three or more folders the defect stops bouncing: a
later iteration sees an earlier iteration's count-1 marks on items its
own walk lacks and archives them mid-pass — one registered folder away
from archiving a third of a library per scan. Single-folder libraries
were unaffected, which is why comics/ebooks/manga showed zero flags.
Folder-book presence was suspected alongside it (audioBookRoots read
during the comparison), but the map is written for every >=2-track
folder-book root on every walk (processAudioBookFile, since
|
||
|
John O'Keefe
|
7b9e8e4fc2 |
fix(scanner): match library folders only at directory boundaries
A bare string-prefix test (strings.CutPrefix / strings.HasPrefix) let a registered folder whose path was a string prefix of a sibling claim the sibling's files: "/app/audiobooks" matched "/app/audiobookssd/Brian Jacques/...", so getRelativePath sliced the remnant "sd/" into every stored relative path for that library (prod scan log: "[AUDIO] Created folder book 'Taggerung' (41 tracks) at sd/Brian Jacques/Redwall/Book 14"). Scanner lookups built their keys with the same function (getMediaItemByFilePath, scanStateLookup), so writes and lookups were consistently poisoned and no scan ever corrected them; everything outside the scanner resolves paths by joining back to the roots (ResolveMediaPath), so cover/track URLs 404ed for files sitting one directory away. The same unchecked HasPrefix pattern also drove five folder-membership sites, so a sibling-prefix pair misattributed more than paths: owning-library resolution (processMediaFile, maybeProcessAudioBook), per-library-type extension allowlisting (isScannableFile), and folder-structure metadata extraction on both create and update paths — a file could be scanned under the wrong library's rules entirely. folderContains(folder, path) is now the single boundary-aware primitive: path must equal the folder or continue with a filepath.Separator (after Clean). All seven sites go through it, so no folder name — however similar to another — can interfere with any other's files again, in any order of registration; trailing slashes on registered folders are normalized. Registration order still decides for genuinely overlapping (nested) registered roots: pre-existing behavior, deliberately unchanged. Regression tests (media_scanner_path_test.go, pure unit — no DB) pin the prod trap (audiobooks/audiobookssd), order-independence, trailing slashes, mid-word siblings, and the helper itself. Repair of already-poisoned rows is deliberately separate and safe only under this fix: a rescan repoints them via the SHA-256 move path (MoveMediaItemFilePath) once lookups stop producing poisoned keys. |
||
|
John O'Keefe
|
fd9292e6f2 |
fix(series): Date Added respects the direction toggle; bump bundle stamp
Two follow-ups from live testing: - "recent" is BOTH a base toolbar key and a full registry key (where it means desc). ResolveSeriesSort's full-key passthrough ran first, so sort=recent&dir=asc silently resolved back to desc — the direction chevron flipped but Date Added could never sort ascending. "recent" now composes with the direction before the passthrough, unknown bases fall to the default entirely (a hostile dir can't ride an unknown sort), and the composition table is unit-pinned plus an HTTP-level row in the series filter test. - The header search "still broken" report was a stale-cache artifact: the fixed code shipped but main.js kept its ?v=7 stamp, so browsers with a cached copy never refetched (fresh-profile e2e passed while the user's browser didn't — heuristically cached module). Stamp bumped to ?v=8 in header + docs; the e2e suite re-verified against the rebuilt server. |
||
|
John O'Keefe
|
ffbcc53083 |
test(e2e): drive write-to-file per format in the real UI; fix sidecar perms
Per the owner's testing bar: a Playwright spec registers an admin, seeds one fixture per write-capable type (EPUB, PDF, CBZ, MP3, FLAC, M4A, M4B — generated at runtime, no binary media in git), drives the REAL metadata-editor flow (Edit → change title → in-app dialog → Save & write) for each, and then reads every artifact back INDEPENDENTLY of the app — EPUB/CBZ/sidecar via python zip+XML parse, audio via ffprobe — asserting the written title landed in the FILE and the old one is gone. Writing the e2e exposed a real bug: atomicWrite left server-written sidecars at 0600 root (os.CreateTemp's default), so host-side tools sharing the volume — Calibre reading metadata.opf — were locked out. The temp file is now chmod'd 0644 before the rename, matching the scanner's cover-file convention; the e2e's host-side reads prove it. |
||
|
John O'Keefe
|
fb1645c90c |
fix(metadata): m4b text-track fallback; in-app confirm; save-first write
Three defects from real-world use: - The m4b rewrite assumed every stream survives -map 0 -c copy; files carrying text-ish tracks (the movenc 'text' fourcc) fail the ipod muxer with "Tag text incompatible with output codec id" — exactly the error a write attempt hit in the field. The rewrite now attempts full fidelity first, then retries mapping audio + attached pictures only (-dn -sn): chapters live in the container header (-map_chapters 0) so they survive; every attempt still ffprobe-validates before the atomic rename and the original is never touched on total failure. - The browser confirm() is replaced by an in-app dialog (WriteToFileConfirmDialog, Alpine + templ — no native dialogs). - "Write to file" now SAVES the pending editor edits first (persisted through the same PUT the Save button uses), then writes the file from those DB values, and closes the editor + refreshes the page exactly like Save — previously the endpoint wrote the LAST-SAVED DB values, so edits sitting in the modal never reached the file, and the page didn't visibly update. Test coverage per the owner's bar: EVERY single-file container (MP3, FLAC, M4A, M4B) gets its own runtime-generated fixture, gets written, and the FILE's metadata is then verified independently — ffprobe reads the rewritten container and every mapped key is asserted per format, including the subtitle round-trip split (MP3/FLAC yes, M4A/B no) — and the EPUB/CBZ/sidecar assertions now PARSE the written XML (dc titles/ creators, ComicInfo fields, calibre meta) instead of substring matching. A smart-genre unit test also pins the mixed-blob contract: a recognized shelf wins the genre column even when demoted format words ride along in the same blob. |
||
|
John O'Keefe
|
96c1228805 |
feat(series): multi-field search, sort-direction toggle, apply placement
Toolbar feedback round: the series search box now matches whatever the main search matches for a group — series NAME, any book's AUTHOR, and any book's TITLE inside the series (WHERE EXISTS per group-row, so counts stay true); the separate author input is gone from the toolbar (the author= param stays live for the JSON API and deep links). The sort control becomes a base-key select (Date added/Name/Author/Books/ Progress/Last read) plus a chevron direction toggle that flips and resubmits; sort/dir compose through ResolveSeriesSort (full registry keys keep working for back-compat, unknown base keys fall to recent). The search/apply button moved next to the search box as requested, and default URLs stay clean (sort/dir only emitted once the user chooses). |
||
|
John O'Keefe
|
59eddc1386 |
feat(metadata): audio tag rewrite via ffmpeg + CBZ ComicInfo injection
Phases 5b and 5c of the metadata writer. 5b — single-file audio (MP3/FLAC/M4A/M4B): the container's tags are rewritten with ffmpeg -c copy (no re-encode; -map 0 keeps cover art, -map_chapters 0 keeps chapters). The key set is exactly what the scanner's applyAudioTagFallbacks reads back — title+album, subtitle, artist/album_artist, composer (narrators, Audible's convention), grouping + iTunes series-part, date, genre — so a write + rescan round-trips. Output is ffprobe-VALIDATED (duration within 1s/0.5% of the source) before the atomic rename; a lossy container rewrite never replaces a good book. Fields the format can't carry per our own read chain are deliberately not written, and fields_written reports only what really round-trips (subtitle rides MP3/FLAC — this image's mp4 muxer drops subtitle keys, discovered empirically and pinned by test). 5c — CBZ: ComicInfo.xml injected (or replaced) with the same atomic zip rewrite as the EPUB path, extended with an append-if-missing mode since a comic may not carry one yet. Field map mirrors the scanner's comic block: Title/Series/Number/Count/Volume, Writer (authors), Summary, Publisher, Year/Month/Day, LanguageISO, Web, Genre+Tags; fractional series numbers are omitted rather than truncated to a lie. CBR/CB7 stay on the sidecar path (RAR/7z can't be rewritten in place). Integration tests generate media at test runtime with ffmpeg (m4b) and in-memory zips (cbz) — no binary fixtures — pinning the tag round-trip through ffprobe, post-write rescan titles, the ComicInfo injection and the validation-gate behavior. |
||
|
John O'Keefe
|
eaf1b2c8dd |
feat(metadata): write-to-file — embedded EPUB OPF + Calibre sidecars
Phase 5a of the metadata-writer feature: the book editor gains a "Write to file" action (admin) that persists the item's CURRENT database metadata into the artifact on disk, so the library stays portable to Calibre and other apps — the thing Save alone can't do. Dispatch (services/metadata_writer.go): EPUB/KEPUB gets its embedded OPF spliced in place (fresh dc:* block generated inside the ORIGINAL metadata open tag; manifest/spine/guide untouched; every other zip entry copied byte-identical in original order so mimetype stays first and STORED; original dc:identifiers preserved — the package's unique-identifier must keep resolving). Everything else takes a Calibre-style sidecar: metadata.opf when the book is ALONE in its directory (Calibre's own convention — round-trips through Calibre too), <basename>.opf when siblings share the folder, so one book's sidecar can never poison dir-mates' scans (the scanner now reads both, basename first). Safety contract: temp file + atomic rename — the original is never truncated on failure; the generated OPF is unit-tested to ROUND-TRIP through the scanner's own parser (write side and read side can never disagree); metadata_overrides are deliberately kept (file and DB agree after the write; overrides still guard against stale sidecars). The endpoint re-extracts inline (RescanMediaItem) and reports rescan failure separately — a failed rescan doesn't undo the write. SHA-256 changes are inherent and documented for users (Kobo/KOReader re-key). UI: the metadata editor modal carries the file path as a data attribute; the button confirms before mutating, labels the target (embedded vs sidecar) and toasts the outcome. Integration tests pin the 403, missing-file error, EPUB embed + post-write rescan title, the lone-book metadata.opf case, and the shared-dir <basename>.opf case with no dir-scoped litter. |
||
|
John O'Keefe
|
d1f9ad3b25 |
feat(series): sort and filter the series page
The /series page rendered its grid in one fixed order (newest first)
with no way to find a series beyond eyeballing cards. The page now has
a toolbar — series-name search, author filter, viewer-progress filter
and a 12-key sort select — wired as a plain GET form (the page's
existing model: the URL is the single source of truth, and the library
switcher now carries q/author/status/sort along on switch).
Extensibility is deliberate (user request): sorts live in a registry
(services.ResolveSeriesSort) mirrored by a CASE cascade in
GetDistinctSeries and a <select> in series.templ — a new sort is one
entry in each. The sort value is a BOUND PARAM compared inside the
cascade, never interpolated; unknown keys resolve to the default.
Filters: series-name matches in WHERE (the group key, so counts stay
true); author (BOOL_OR any-book match) and status (finished/reading/
unstarted vs the viewer's progress, same 0.999 threshold as the
filter_query status arms) in HAVING so COUNT(*) keeps reporting the
whole series. GetDistinctSeriesCount shares the JOIN + HAVING so
pagination totals match. The reading_progress join can't multiply rows
(the table is UNIQUE per media_item+user).
Cards gain the viewer's finished-count ("X of N read" + progress bar),
and the empty state distinguishes "no series at all" from "no matches —
clear filters". The JSON /api/series endpoint takes the same params and
gains read_count/last_read_at additively.
Integration test pins every sort key against a three-series fixture
with distinct created/author/count/progress answers, the filter matrix,
composition, unknown-sort fallback, card read-counts and the JSON
contract — all as the fresh regular user so the live reader's data is
untouched.
|
||
|
John O'Keefe
|
6f37be59fc |
feat(issues): retry a processing issue through the individual scanner
POST /api/libraries/:id/issues/:issueId/retry (admin, family-lens gated via the existing gateIssue) re-runs the work that produced an open issue — deliberately NOT a library-wide rescan. Two shapes mirror the two ledger entry kinds: - item-backed issues rescan their media item through the book-detail RescanMediaItem mechanism (self-resolving folders, multi-file audiobook routing, write-after-success guards) and then close the issue (RetryMediaItemIssue); - row-less import_failed issues re-import the single file through the watcher's standalone processMediaFile path (RetryFailedImport: folders resolved RescanMediaItem-style so paths stay library-root-relative), whose create-path self-heal plus an explicit path resolve closes the issue even when the content was already imported by a prior attempt. Failures return the REAL underlying error so the UI can show why the issue still exists; a vanished file is a 409 (ErrRetryFileMissing); already-resolved issues reject re-retry. Docs + bruno mirrors regenerated for the new route (the generator's seq renumbering touches three unrelated bruno files as fallout; openapi gains the path and the date bump), and the docs schema registry gains the retry entry so TestDocsMatchRoutes passes. |
||
|
John O'Keefe
|
fb117635f8 |
feat(scanner): ledger failed metadata updates; guard partial-scope archive passes
Two scanner-integrity changes supporting the Processing Issues retry work: - The size-change and sidecar-change update paths only printf'd their failures, so a book whose metadata update kept failing (the Crusader genre-22001 loop) degraded silently on every scan with nothing on the Processing Issues page. Both paths now record an item-backed metadata_extraction_failed issue — the same type and shape the sidecar-extraction guard already used — and resolveIssuesForItem clears it when an update finally succeeds. - The archive reconciliation pass compares every library item against what one scan saw on disk. The new per-folder scan scope runs a scan covering a single folder, which would mark every item under the library's OTHER folders missing and archive them on a later pass. The pass now skips a library whose registered folders the scan only partially covered (GetLibraryFolders vs the scan's folder set), logging the skip. |
||
|
John O'Keefe
|
885d614092 |
fix(scanner): route the audio-tag genre through the smart-genre pipeline
applyAudioTagFallbacks captured the ffprobe genre atom verbatim into
media_items.genre — no splitting, no shelf resolution, no cap. That raw
capture is what fed 100+-char Audible category blobs into the column
(the 2026-10 Crusader import/update failures).
The ABS sidecar path already has the correct doctrine (
|
||
|
John O'Keefe
|
ab2a5568ae |
fix(scanner): split comma-joined ABS sidecar genres into separate tags
Audiobookshelf-style exports join genres into one comma-string
(["Epic, Action & Adventure, Military, Sword & Sorcery"]). The
smart-genre picker already split candidates for the genre column,
but the tags assembly used the raw unsplit array — so the whole
string landed as ONE element of media_items.tags and rendered as a
single tag everywhere.
Use pickSmartGenre's split candidates for tags too, exactly as the
function's own comment documents ('split parts included stay in
tags'). Commas inside a single subject heading remain legal within
one tag (EPUB dc:subject path unchanged).
|
||
|
John O'Keefe
|
a716b386a1 |
fix: wire security headers, graceful shutdown, and settings invalidation
Lifecycle (internal/app + main.go): - App.OnShutdown hooks run after the HTTP server drains, in registration order: Worker.Shutdown (background jobs now drain instead of dying on SIGTERM), DeviceAuthMiddleware.Stop (releases the rate-limiter ticker), Handler.CloseScanner (fsnotify watcher and scan log files) Security headers: - SecurityHeadersMiddleware (survivor of middleware/security.go, renamed security_headers.go) is now on the echo stack: nosniff, frame-deny, HSTS, CSP (unsafe-inline required by the sanctioned theme flash guard + Alpine), referrer and permissions policy Scanner settings cache: - settings saves (UpdateScanSettings + ApplySetting for the three scan keys) now invalidate MediaScanner's 30s settings cache via SystemSettingsHandler.SetScanSettingsChanged, so changes apply on the next poll instead of after the TTL - MediaScanner.Close closes the scan logger, fixing a file-descriptor leak on every per-item rescan (fresh scanners never closed logs) Client fix: - custom-section-builder's add/remove-selection buttons used inline onclick attributes referencing module-scoped functions, which are not global in a bundled ES module: the buttons threw ReferenceError. Functions restored and exposed on window via declare global Also fixes a pre-existing vet error (saveCoverImage copied echo.Context by value, mutex included) and drops the per-key SettingsCache.InvalidateKey variant left callerless by the full-flush invalidation wiring. |
||
|
John O'Keefe
|
8631aa00bf |
refactor: remove dead code across server, templ pages, and client modules
Deadcode-verified removals (RTA reachability from main, cross-checked with -test and textual xrefs): Go server: - whole files: middleware/transaction.go, middleware/error_handler.go, middleware/password_validator.go (live registration machinery kept), handlers/sync.go, sync/offline.go - unrouted endpoints: auth UpdateUsername/UpdateEmail, timezone settings handler, OPDS RegisterOPDS, Kobo LibrarySync handler, ReaderHandler.ShowReader, JSON-data twins on collections/library - duplicate book-matching handlers (book_matching.go Handler twins, matching.go MatchingHandler twins), superseded helpers (router.parseUUID, handlers.ValidateDeviceToken, normalizePath, userExists, generateLockID) Templ/client: - never-routed pages: api_explorer.templ, queue.templ and their client modules api-explorer.ts, api-explorer-docs.ts, queue.ts - fully orphaned wrapper modules events.ts, dom.ts - broken clearSyncQueue (called a nonexistent endpoint) and the hidden, never-populated sync-queue section on the devices page - 166+ unused exports demoted to private (knip-verified); ~85 truly dead declarations removed via tsc noUnusedLocals fixpoint - stale 'interactive API Explorer' claims dropped from docs The docs drift-gate package (internal/docs), templ-bound modules (dashboard, stats, header, index, login, profile*, bookshelf, collections, toast-error), sqlc-generated code, and web/static are intentionally untouched. |
||
|
John O'Keefe
|
8733e83f1f |
style: apply gofmt across Go sources
Repo-wide gofmt sweep: const-block alignment, doc-comment spacing, statement-after-brace fix in media.go, trailing blank lines. No semantic changes — whitespace-insensitive diff is empty for every file in this commit. |
||
|
John O'Keefe
|
80bfc1a77b |
fix(scanner): re-extraction is write-after-success — no NULL audio_metadata
The 2026-10-21 wipe: a re-extraction pass NULLed audio_metadata on the m4b audiobooks — every affected book dropped its chapters, track manifest and duration and displayed as 1 second long (the reader also lost its manifest, since single-file books synthesize it from the payload). Guards on both writers of that column: - updateMediaItemWithMetadata (generic full-row pass): when the new extraction carries no audio payload, keep existing.AudioMetadata. Single-file m4b books route here, and a failed/absent extraction leaves AudioMeta nil. - updateAudioBookItem (focused payload write): an empty payload (no tracks, no chapters, no duration) over a populated row is a failed extraction — refuse the write with a loud warning and let the next scan retry. audioPayloadIsEmpty() pins the empty-payload predicate (unit-tested): the column is only ever replaced by fresh data from a real audio extraction, never by absence of one. |
||
|
John O'Keefe
|
a42c6d4f75 |
feat(scanner): force rescan = original scan; feat(filters): the manually-edited filter (faceted plan Phase 13)
Release / build-and-push (push) Successful in 3m7s
Force rescan contract change (owner-directed): the editorial-keep skip
is retired. Force rescan now re-derives EVERY field on EVERY book from
all sources — multi-file books included (the folder/zip flows ran the
full chain on create and on manifest-flagged edits, but skipped the
sidecar apply under force). metadata_overrides is the only protection:
ApplyMetadataOverrides re-applies manual values inside the writer, so
manual edits survive; everything else re-derives.
- A force pass also STAMPS each re-derived book's manifest with the
current parser version (success-only — same truthfulness rule as
Phase 10) and closes its open processing issues (Phase 11 self-heal).
Force rescan and the ev-bump trickle now converge: either leaves the
library fully current; the deploy order is bump-first, then Rescan
All (documented in the plan and contract_pins).
- Single-file force rescan already ran the full chain — unchanged,
now pinned.
NEW filter field manually_edited ("Manually edited", boolean):
cardinality(mi.metadata_overrides) > 0 / = 0. Composed-engine (qf)
only — the builder's Book facts group gains it with is true/false;
no value menu, no suggestion surface; family lens inherited.
Tests: multifile force applies sidecar metadata + stamps + override
protection; single-file pin; the filter both directions scoped to the
test library; adjacent suites (Audiobook, Calibre, ExtractorVersion,
CrossLibraryMove, ProcessingIssues, FilterBuilder, SchemaIdempotency)
green. The extractor mechanism itself is untouched — the owner bumps
scan_extractor_version manually via the admin UI (plan Phase 13 §3).
|
||
|
John O'Keefe
|
9b5defcb31 |
feat(issues): the honest processing-issues ledger — row-less import failures, self-healing resolution, family lens (faceted plan Phase 11)
processing_issues only recorded cross-library move rejections; files that failed to import vanished into an error counter and stdout. The ledger now records every failure class and heals itself: Schema (in place, pre-release boundary): - media_item_id NULLABLE + file_path TEXT (deliberately unbounded — an over-long path is itself a failure mode; a VARCHAR(500) ledger column died on the exact 22001 it existed to report — caught by the new test). Row-less failures key on (library, file_path); item-backed on (item, type). Widened dedup = COALESCE-expression unique index; the old table UNIQUE stays (NULLs pairwise-distinct for row-less). - Conflict-target upsert: repeat failures UPDATE the same row and REOPEN resolved ones — never duplicates. Wirings: - CreateMediaItem failure (single-file + multifile createAudioBookItem) → row-less import_failed (error). - Extraction hard-fail with fallback import → item-backed metadata_extraction_failed (warning). - Change-path re-extraction failure (the ev/sidecar guard) → item-backed metadata_extraction_failed; LogProcessingIssue keeps its signature, a row-less helper joins it. Auto-resolution: - Successful import → ResolveOpenProcessingIssuesForPath (library, file_path). - Successful (re)extraction → ResolveOpenProcessingIssuesForItem (single-file change path, multifile editorial apply). - Unchanged-skip scans do NOT resolve: extraction never re-ran, so the degraded metadata is still degraded — the page heals exactly when the book does. Family lens on ALL issues surfaces (list, stats, resolve, delete, SSR page): explicit out-of-set library id → 403, never silent. This tightens one pre-lens behavior — a nonexistent library id returned 200-empty, now 403 (pinned, changelog entry in docs/contract_pins.md). List is LEFT-JOINed so row-less rows appear (media_item_id "", title falls back to the path). Tests: full lifecycle state machine through real scans (fail → appears, repeat → no dups, success-at-path → resolved, LEFT-JOIN list shape); family lens 403s with a fresh non-shared admin; all 7 pre-existing issue suites green (two expectations widened to the lens, changelogged); adjacent suites (CrossLibraryMove, Audiobook, Calibre, ExtractorVersion, SchemaIdempotency) green; unit suites green. |
||
|
John O'Keefe
|
91b0c45dd4 |
feat(scanner): extractor-version (ev) mechanism — parser fixes reach old rows without a force rescan (faceted plan Phase 10)
Parser fixes (
|
||
|
John O'Keefe
|
ab713e2feb |
fix(filters): blank-typed browse 500 on id-valued fields (library/collection/added_by)
The browse-on-click blank sentinel was trimmed to empty (correct), but the typed query had already been bound unconditionally — the id-source suggestion query has no score CASE to reference it, leaving an unused parameter Postgres rejects (42P18). Every click of an empty library / collection / added-by box answered 'No matching values' via the error path. - ComposedFieldValues binds the typed query ONLY in branches that reference it and only when non-blank; the score expression is '0 AS score' for browse (no param) and the similarity CASE when typing. composedIDSourceValues takes the raw typed string and binds it only when the name filter is built. userBind already followed this discipline; audited every bind site in the file (fuzzy helpers, composed search, count) — all interpolate immediately. - Tests: library browse under rules (exactly the surviving library, count pinned), series browse blank, added_by browse blank (no error), collection browse blank (offered with count) and under a zero-match rule (never offered). Verified end-to-end against the live dev data: author+Sapkowski AND narrator+Kenny → Audiobooks (8). |