main
147
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
9b5defcb31 |
feat(issues): the honest processing-issues ledger — row-less import failures, self-healing resolution, family lens (faceted plan Phase 11)
processing_issues only recorded cross-library move rejections; files that failed to import vanished into an error counter and stdout. The ledger now records every failure class and heals itself: Schema (in place, pre-release boundary): - media_item_id NULLABLE + file_path TEXT (deliberately unbounded — an over-long path is itself a failure mode; a VARCHAR(500) ledger column died on the exact 22001 it existed to report — caught by the new test). Row-less failures key on (library, file_path); item-backed on (item, type). Widened dedup = COALESCE-expression unique index; the old table UNIQUE stays (NULLs pairwise-distinct for row-less). - Conflict-target upsert: repeat failures UPDATE the same row and REOPEN resolved ones — never duplicates. Wirings: - CreateMediaItem failure (single-file + multifile createAudioBookItem) → row-less import_failed (error). - Extraction hard-fail with fallback import → item-backed metadata_extraction_failed (warning). - Change-path re-extraction failure (the ev/sidecar guard) → item-backed metadata_extraction_failed; LogProcessingIssue keeps its signature, a row-less helper joins it. Auto-resolution: - Successful import → ResolveOpenProcessingIssuesForPath (library, file_path). - Successful (re)extraction → ResolveOpenProcessingIssuesForItem (single-file change path, multifile editorial apply). - Unchanged-skip scans do NOT resolve: extraction never re-ran, so the degraded metadata is still degraded — the page heals exactly when the book does. Family lens on ALL issues surfaces (list, stats, resolve, delete, SSR page): explicit out-of-set library id → 403, never silent. This tightens one pre-lens behavior — a nonexistent library id returned 200-empty, now 403 (pinned, changelog entry in docs/contract_pins.md). List is LEFT-JOINed so row-less rows appear (media_item_id "", title falls back to the path). Tests: full lifecycle state machine through real scans (fail → appears, repeat → no dups, success-at-path → resolved, LEFT-JOIN list shape); family lens 403s with a fresh non-shared admin; all 7 pre-existing issue suites green (two expectations widened to the lens, changelogged); adjacent suites (CrossLibraryMove, Audiobook, Calibre, ExtractorVersion, SchemaIdempotency) green; unit suites green. |
||
|
John O'Keefe
|
91b0c45dd4 |
feat(scanner): extractor-version (ev) mechanism — parser fixes reach old rows without a force rescan (faceted plan Phase 10)
Parser fixes (
|
||
|
John O'Keefe
|
ebb7f4a5b0 |
feat(filters): panel removal + All-Books field-values fix (faceted plan Phase 8)
- Remove the Phase 7 facet-panel UI: SSR route, facet_panel.templ, FacetCountsForView, drawer panel block, panel TS + .facet-* CSS. The ENGINE stays (FacetCountsUnderFilters, all-facets endpoint, drift guard) — it becomes the Phase 9 builder's machinery. - Fix the All-Books bug: field-values autocomplete (author/genre/ tags/narrator/series/language) treats library_id as OPTIONAL — absent aggregates every VISIBLE library (family lens in-query); explicit out-of-set library_id is now 403, never a silent pass. The four scalar Search*Values queries get the nullable shape. - toast.ts: suggestion fetches are exempt from error toasts (console-only) — a failed background hint never interrupts. - TestFacetPanelPartial deleted with the behavior; TestFieldValuesAll- Libraries pins the fix (200 + cross-library aggregation, family lens, 403). Contract-pins changelog entry added. |
||
|
John O'Keefe
|
f52bb6f073 |
feat(facets): all-facets panel with under-filter counts (faceted plan Phase 7)
The 9 Batch A facets could not be SET from the web UI, the facets
endpoint had no in-app consumer, and the registry's libraryTypes hints
rendered nowhere. This phase closes all three.
- ONE UNION-ALL sqlc query (FacetCountsUnderFilters) computes all 14
listable facets in a single round trip: each arm omits its OWN
facet's filter block and carries every other registered facet's
block (copied mechanically from SearchMediaItemsUnified) — classic
faceted semantics: with filters active each facet counts under the
others; with none, blocks are neutral and the counts are the v1
global-per-library ones. One query serves both cases. The 14
per-column Batch C queries are deleted.
- Endpoint: no facet param → the new all-facets shape
({facets: {param: [{value, display?, count}]}}), filters bound via
the shared ParseMediaSearchParams, library_id optional (the
All-Libraries view, family lens still enforced); facet=X keeps the
pinned single-facet shape (now routed through the same query with
neutral filters).
- SSR partial GET /media-items/facet-panel: templ renders the drawer's
value lists — registry order, Batch D library-type scoping
(FacetsForLibraryType: manga_type on manga, reading_direction on
manga+comics, narrator on audiobooks; All view = union), prettified
genre labels + language display names, active values marked. Family
lens: out-of-set explicit library_id is 403.
- Drawer wiring: the panel fetches on drawer open and after every
apply with the CURRENT filter query string; a value click toggles
that facet's comma-joined filter (hidden input where the drawer has
no field — same as deep links) and re-applies the form; chips and
URL stay in sync. Values are server-rendered (rendering doctrine);
bookshelf.ts owns only the click/fetch orchestration.
- prettifyGenre moves to internal/utils as the canonical
implementation (services' display enrichment shares it; templ wraps
it; the TS port in bookshelf.ts is unchanged).
- Drift guard extended: FacetCountsUnderFiltersParams must carry every
registry facet's param, the bools as pgtype.Bool.
Tests: under-filter exactness (own-block omission, OR-within,
AND-across), family lens (hidden values absent; 403 on both shapes),
all-shape + pinned-shape coexistence, partial rendering with Batch D
scoping + active marking + 403. Full container suite green except the
documented pre-existing TestAdminCreateUser red.
|
||
|
John O'Keefe
|
6750e9e0dc |
feat(scanner): manifest-gated sidecar change detection + playlist sync on change path
Part 2 of the faceted plan, Phase 4 (steps 3-4). Size-only detection
was blind to metadata-sidecar edits; a size-stable book is changed by
EDITING its metadata.opf/metadata.json/cover — previously invisible
until a force rescan.
- Sidecar manifest fingerprints (media_scanner_manifest.go): the
manifest stats exactly the names the extractors consume (metadata.opf,
metadata.json, findSidecarCover patterns incl. <basename>.<ext>), so
manifest and extractors can never disagree. Shape v1:
{v, entries:[{n, s, m (unix nanos — second granularity would hide
edits landing in the backfill scan's own second), h (sha256 — covers
included: the repoint ladder needs a baseline hash for any stat-
drifted sidecar; a hashless cover turns harmless mtime drift into a
re-extraction, the exact flood this design kills)]}.
- Evaluation ladder in cost order: stat pairs identical → skip with
ZERO bytes read; stat drift → hash that sidecar (equal = host repoint
reconciliation, stat pair refreshed, no re-extraction; differing =
changed); NULL stored manifest → backfill only, NO re-extraction
(deploy-day rule: the first scan after release must not look like an
mtime flood).
- Change path: re-read full row, extractMetadata once and pass it
through updateMediaItemWithMetadata (preExtracted — metadata_overrides
stay protected), run syncPlaylistCollectionsAfterCreate (Correction 2:
a reading-order entry added to an EXISTING book now creates its
collection; add-if-missing is idempotent), store a fresh full
manifest. Creates seed their manifest at insert time.
- Multifile folder/zip books intentionally keep Phase 3 behavior: their
change path does not consume sidecar extractors, so gating them would
flip books 'changed' without picking anything up (plan correction —
the plan's §3 multifile note assumed otherwise).
- UpsertMediaScanState + GetMediaScanState queries.
Tests: media_scanner_manifest_test.go (ladder unit tests — fast path,
repoint, same-size drift, appear/vanish, unhashable legacy entries);
TestScanManifestLifecycle (seeding, unchanged-skip stability, repoint
reconciliation with stat-pair refresh assertion, same-size edit
consumed, playlist sync on the change path + idempotency, no-leak
guard). Container runner.
|
||
|
John O'Keefe
|
693053a9f5 |
perf(scanner): bulk scan-state load — one query per library scan
Part 2 of the faceted plan, Phase 3 (steps 1-2). The periodic scan
issued one GetMediaItemByFilePath per scannable file (~3,094 queries
per scan in prod, nearly all answering 'unchanged, skip') plus one
GetLibraryByFolder per file for folder resolution.
- NEW media_scan_state table: the scanner's per-item fingerprint
cache. Deliberately NOT a media_items column — the library-scoped
list emits raw sqlc rows as JSON (
|
||
|
John O'Keefe
|
7e586df48a |
feat(facets): facet registry, Batch A server params, genre model v2, facets endpoint
Phase 1 + backend of Phase 2 of the faceted-navigation plan. One
declarative facet registry (services/facet_registry.go) drives the
bookshelf filter system; a drift-guard unit test reflects over the
generated SearchMediaItemsUnifiedParams so a registry entry and its
sqlc.narg block can never disagree silently.
- Nine new filter params on SearchMediaItemsUnified: story_arc,
imprint, edition (scalar-fuzzy), age_rating, format_group,
manga_type, reading_direction (exact), explicit, abridged
(presence-bound bools). Comma = OR-within multi-select on every
text facet; AND-across-facets unchanged.
- Genre model v2 (user rule: a subgenre is not a genre, it's a tag):
genre_lexicon.go rebuilt as shelf set + subgenre->shelf map;
pickSmartGenre picks direct shelf > mapped subgenre > Fiction floor
> empty (unknown-only stays tags-only). Type-words (Manga, Comics,
Graphic Novel) join the demote list. Both 'adventure' and 'action &
adventure' are shelves — provider data uses both and the verified
traces pin them.
- Language filter UX: comma elements normalize through the ingest
language map, so display names ('English') match rows stored as ISO
codes ('en'). The language autocomplete now serves display names.
- NEW GET /api/media-items/facets?facet=X&library_id=... — registry-
driven value+count listing (14 static per-column queries, family-lens
join); language values carry display names; bool/range facets 400.
- Family-lens gates: an explicit hidden-library library_id is now a
403 on the search endpoints (was a silent empty set) — JSON branch
and HTMX grid partial both.
- handlers.ParseMediaSearchParams is the SINGLE param parser for the
search JSON endpoint and the HTMX grid partial; the HTML branch
previously dropped publisher_filter and narrator_filter silently.
Pins: docs/contract_pins.md changelog entries (additive request
params; new facets endpoint). Tests: facet_registry_test.go (shape +
SQL drift guard + library-type union), series_parsing_test.go genre-v2
cases, facets_integration_test.go (param filtering e2e, counts,
language display, 403s) — container runner.
|
||
|
John O'Keefe
|
7537063cb7 |
feat(filters+nav): author navigation end-to-end — arrays on every media response, deep-linkable bookshelf filters, clickable authors
Completes the A2c acceptance scenario: click an author on a book detail page -> land on the bookshelf filtered to everything that author wrote (including their share of co-authored books). - Media detail + unscoped list resolved maps expose subtitle, explicit, authors, publishers (the library-scoped raw branch and the search endpoint already did — raw rows follow the SELECT). - SearchMediaItemsUnified SELECT carries the new columns; the search endpoint serializes the row directly, so clients get them there too. - book_detail.templ renders each author as a link to /bookshelf?author_filter=<name>. - Bookshelf honors filter params in the URL on load (author_filter, series_filter, genre_filter, publisher_filter, tags_filter, narrator_filter, language_filter): pre-fills the form, applies it to the first grid load. Comma-joined values are multi-select via the server-side string_to_array split. Filtered views are shareable. Pins widened: search rows (+7 keys), resolved keys (+4); changelog entries in docs/contract_pins.md. Android follow-up plan written to ~/.opencode/plan/bookhoard-app-v2-compat.md. |
||
|
John O'Keefe
|
f94e7472b4 |
feat(filters+ui): per-element author/publisher filters, playlist toggles, Up Next API, reorder UI, dashboard badges
Phases 4 + 5 of the metadata overhaul (plan §4, §5) plus §3 handler wiring. Filters (§4.1): author_filter/publisher_filter now match per-element against the authors[]/publishers[] arrays — a co-authored book matches any one of its authors — with comma-joined params acting as OR-within-facet multi-select (string_to_array + cross unnest). Legacy single-value params unchanged on the wire; series/genre/language filters stay on their varchar columns; narrator filter already per-element. Collections API (§3.3-3.5): collection detail carries viewer + hide_finished into the listing query and exposes up_next (visible- library-enforced reading-queue head); PUT /api/collections/:id/items/order (owner-only, full-order rewrite); UpdateCollectionRequest gains hide_finished/show_up_next ([] string last-value-wins for the hidden-false-sibling checkbox pattern) merged server-side OVER stored view_settings so imported_from_series provenance survives edits; EditCollectionModal renders the toggles from view_settings. Dashboard (§5.5): sections JSON carries is_playlist/complete (dashboard.ts is the AGENTS-documented JSON-render exception; first paint stays templ). Fully-read playlists render '✓ Playlist complete' instead of an empty rail; imported playlists get a 📖 Playlist badge. Frontend: book detail shows subtitle + full authors + Explicit badge; edit modal gains subtitle, comma-separated authors/publishers (client- split to arrays exactly like narrators), step=any series number, explicit checkbox; collection book cards get playlist move ↑/↓ buttons persisting the full order via the reorder endpoint (templ partial re-render on failure — DOM order is client-local, API order is truth). AGENTS.md gotcha #3 amended per plan §3.7: user collections are dashboard sections by default; the show_on_dashboard premise is retired. Verified: full affected integration sweep green (pins widened for up_next/is_playlist/complete — changelog entries in docs/contract_pins.md); unit suite green; go build + templ + vite green. |
||
|
John O'Keefe
|
1dbb0ce406 |
feat(collections): playlist auto-creation, position ordering, hide_finished, Up Next, dashboard eligibility
Phase 2b + 3 of the metadata overhaul (plan §2.4, §3.1-3.7). Auto-playlist collections (§2.4): the scanner turns obviously-playlist sidecar series (explicit '(Reading Order)'/chronological/publication/ in-order/timeline markers only — precision over recall) into curated static collections: upsert by view_settings.imported_from_series then name (rename-safe), position set ONLY on insert (clean integers; 1.5/1a stay unpositioned rather than guessing), membership add-if-missing, never removed or re-ordered by rescans, provenance in description + view_settings, show_on_dashboard true (dashboard-eligible by design), 'scanner.auto_playlist_collections' setting (default on, Settings Registry + schema seed). Ordering (§3.1-3.2): collection listings order position ASC NULLS LAST then added_at ASC (was added_at DESC — walked playlists backwards); PUT /api/collections/:id/items/order applies full-order rewrites (owner-only, service-gated). hide_finished + Up Next (§3.3-3.4): GetCollectionItemsForDashboard gains viewer_user_id + hide_finished (LEFT JOIN reading_progress, COALESCE percentage < 0.999) and viewer_percentage rides the row for card graying; GetCollectionUpNext serves the collection's reading-queue head with the family-lens visible-library array enforced. Dashboard (§3.5): user collections are dashboard-ELIGIBLE BY DEFAULT — the show_on_dashboard gate (never writable for user rows since the column shipped) is gone from GetUserCollectionsForDashboard; hiding stays dashboard-prefs (hidden_collections) territory; hide_finished filtered-everything now emits a Complete section instead of silently vanishing; sections JSON carries is_playlist/complete. Series page ordering hardening (§3.6): GetSeriesBooks tiebreaks by title ASC — NULL/tied numbers previously returned heap (scan) order, which is what the user saw. Frontend custom-section-builder note: the builder has sent show_on_dashboard: true since its creation; the API never bound it. User-row visibility is now intentionally prefs-governed, so the field stays unbound by design (documented in code). Pins widened (docs/contract_pins.md): collection detail + up_next; dashboard sections + is_playlist/complete; new reorder endpoint recorded. Verified: TestPlaylistCollectionsEndToEnd (new, full e2e: sidecar series split, collection auto-creation with position 7, rescan idempotency, owner-only reorder, finished-filter + completion marker + Up Next drain) plus the full affected integration sweep green. |
||
|
John O'Keefe
|
646477afad |
feat(schema): subtitle/explicit columns, authors+publishers arrays, float series numbers
Phase 1 of the series/collections/metadata overhaul
(~/.opencode/plan/series-collections-metadata-overhaul.md).
Schema (in place, idempotent):
- media_items.series_number INTEGER -> DOUBLE PRECISION: real-world
ordinals are decimals (0.5, 1.5 novellas); INTEGER silently truncated
them (calibre 2.5 pinned as 2 in tests).
- + subtitle TEXT, explicit BOOLEAN (sidecar ingest; ABS explicit /
Audible isAdult).
- + authors/authors_search/publishers/publishers_search TEXT[] NOT NULL
(narrators A2a precedent). Arrays are write truth; author/publisher
varchars become trigger-regenerated display strings
(sync_author_publisher_display) so every varchar consumer (trgm
search, facets, API, OPDS/Kobo) keeps working unchanged.
- Row-level NULL normalization for the new arrays in
set_library_type_name (A2 lesson: Go nil -> explicit NULL -> 23502).
- Backfills ARRAY[author]/ARRAY[publisher] — deliberately no
comma-splitting ('Tolkien, J.R.R.' would corrupt; scanner kept only
authors[0] historically).
- collection_items.position INTEGER + (collection_id, position) index
for playlist ordering.
Go: sqlc regen; MediaMetadata/handler params gain Subtitle/Explicit/
Authors/Publishers (legacy singles still accepted, arrays win);
SeriesNumber int32 -> float64 end to end; metadata_overrides adapted
with legacy override keys ('author'/'publisher') preserved so existing
user overrides survive; bulk author edit writes the array.
Pins widened (docs/contract_pins.md): raw sqlc row key set + the six
new columns; changelog entry added.
Verified: unit suite green; integration sweep (bulk/smart collections/
scanner/stats/sync/queue/calibre/pins/audiobooks) green against the
shared dev DB after in-place migration. Pre-existing unrelated red:
TestOwnershipAndAdminTestSuite/TestAdminCreateUser timezone inheritance
fails identically at the pre-change commit (live shared state).
|
||
|
John O'Keefe
|
38efbff7e1 |
fix(db): total ordering for series OFFSET pagination
GetDistinctSeries ordered by MAX(created_at) DESC alone. Bulk imports inserted in a single transaction share one NOW() timestamp, leaving large numbers of (library_id, series) groups tied on MAX(created_at); Postgres does not guarantee tie order between executions, so LIMIT/OFFSET pages could repeat or drop series across page loads. Add the GROUP BY keys (library_id, series) as tiebreakers so the sort is total and deterministic. Comment documents why the tiebreaker is required so it is not dropped as noise. sqlc regenerated (interface unchanged; the comment flows into querier.go docs). |
||
|
John O'Keefe
|
6216bd9bcf |
feat(audiobooks): multi-file books (folder-of-tracks + zip) + exhaustive metadata ingestion
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
|
||
|
John O'Keefe
|
d74db03cf3 |
feat(sync): unlinked-books queue — ingest, page route, dashboard banner
Release / build-and-push (push) Successful in 3m7s
The identification queue for device files the sync cannot match is now live end-to-end. Previously the whole feature was dark: unmatched device files were silently dropped (losing their reading progress), the queue table sat empty by construction, and the page template had no route. - Ingest: the KOReader sync loop, on an unresolvable file, upserts unlinked_books (one row per device + content id; re-syncs bump last_seen_at and refresh device-reported metadata) instead of dropping the file. Schema gains the (device_id, content_id) unique index the upsert needs. - Page: GET /unlinked-books routes the existing page template, listing the caller's own unresolved rows (ListUnresolvedUnlinkedBooksForUser — devices.user_id scoping; no cross-user leakage). - Dashboard banner: a warning card above the shelves, rendered only when the caller's queue is non-empty (hidden-unless-needed), linking to the page. - Resolve flow unchanged and already gated: MatchingHandler.LinkBook verifies device ownership and target-item visibility (403s) and creates the device file alias, so the next sync matches by path. SyncHandler.LinkBook and Handler.LinkBook are unrouted dead handlers — triage candidates, not part of the live flow. |
||
|
John O'Keefe
|
4f913da0b1 |
chore(db): schema.go searches three levels up — replaces the tests-dir symlink
Release / build-and-push (push) Successful in 3m8s
Host-side test runs use CWD=cmd/server/tests, three levels below the repo root; the location search only went up two, so it never found database/schema/schema.sql from there and init() fatal-exited. That gap was being papered over by an untracked local symlink (cmd/server/tests/database -> ../../../database) hidden behind a bare `database` line in .git/info/exclude — which also broke `git add` under database/. Fix the search list instead and drop the filesystem trick: no symlink to commit, no exclude hack, works on any platform. Verified: contract pins green from cmd/server/tests with no symlink present (init resolves via the new entry); parse test green. |
||
|
John O'Keefe
|
c538efe722 |
feat(db): SearchMediaItemsUnifiedRow carries abridged + audio_metadata
The unified search row lagged the table (survey overstated it — narrators was already there). Both columns come from the already-scanned media_items side of the joins: EXPLAIN before/after is plan-identical (same Nested Loop shape, no new scans). Row-completeness only — no client surface changes; the hand maps select their own keys. |
||
|
John O'Keefe
|
77cf0b827b |
refactor(db): GetCollectionItemsForDashboard embeds the media row; excluded filter moves into SQL; third converter dies
SELECT mi.*, ci.excluded forced every caller to filter tombstone exclusions in Go over a 69-field row, then a 45-field hand converter (getCollectionItemsRowToMediaItems) copied it field-by-field into MediaItems for BuildSections. The query now filters (ci.excluded IS NOT TRUE) itself and returns sqlc.embed(mi) — callers consume .MediaItems directly, the Go filter loops die in both collection-detail ladders (JSON + SSR), and the converter is deleted. Pins, H1/H3 tests, unit suites green. |
||
|
John O'Keefe
|
082a3e0c3c |
fix(sync): offline detector wiped device name, auto_sync, and metadata on every transition
setOfflineMode/setOnlineMode only mean to flip sync_enabled, but they called the full-row UpdateDevice with Go zero values — which wrote DeviceName: "" and NULLed auto_sync/sync_frequency_minutes/ device_metadata on the device row at every offline->online flip. New targeted SetDeviceSyncEnabled query touches exactly that column; the real settings update keeps using UpdateDevice. Caught as part of the H4 helper-census pass (plan §5.4 erasure list). |
||
|
John O'Keefe
|
00305ffc7b |
chore(db): retire the nine dead annotation plain queries + duplicate SQL line
H3 leftover: the annotation handlers route everything through the sync service (dedup/LWW/tombstones), leaving the plain Create/Update/Delete queries for media_notes/media_highlights/media_bookmarks with zero callers repo-wide (handlers, services, tests). Queries.sql drops them so H4's row work doesn't carry dead weight; sqlc regen compiles clean, proving nothing referenced the generated stubs. Also removes the literal duplicate `AND mi.missing_scan_count = 0` line in the library-stats query (harmless but confusing). |
||
|
John O'Keefe
|
63a20dcc95 | chore(db): sqlc regen — querier interface comment for the flush-enumerator fix | ||
|
John O'Keefe
|
6808c1babb |
fix(sync): the offline queue flush never processed anything — fix the enumerator + the dropped payload
Found while verifying H1's "app offline-play -> flush -> position lands"
with the emulator path: the flush has been dead code in two independent
ways, invisible because the first break made the second unreachable.
1. ListPendingSyncQueueItems matched `device_id = $1` with a NULL $1
(the processor passes DeviceID{Valid:false} to enumerate all devices;
uuid = NULL matches no rows). The ticker therefore processed nothing,
ever. The WHERE now treats NULL as "every device" (the same
narg::uuid IS NULL idiom the visibility queries use; EXPLAIN'd — seq
scan on a bounded table, idx_sync_queue_status_priority_created
fits). The processor also dropped its fake-pagination loop, which
re-fetched the same first batch forever once the backlog hit the
batch size (no OFFSET in the query).
2. dbItemToSyncQueueItem dropped SyncData — even with a working
enumerator, every flushed item arrived with an empty payload and
failed JSON unmarshal. Dead code harboring silent breakage, exactly
the plan's thesis.
Callers updated for the renamed RowLimit param (queue handler, offline
detector). TestH1_SyncQueueFlush_LandsAndPreservesAudio drives the real
flow: web PUT seeds audio 900s, a pending checkpoint-style queue row
applies percentage 0.8 through SaveProgress, audio survives, queue row
completes.
Also fixes dashboard_test.go's stale BuildSections signature (the
package's test binary has been unbuildable) — user-approved: tests
designed against old shapes are not worth keeping.
|
||
|
John O'Keefe
|
ce366b1604 |
feat(stats): Library tab — per-library inventory stats + audio-aware Reading tab
The /stats page gains a second tab (Reading | Library) implementing the
Audiobookshelf-style library dashboard (HANDOFF item 9): media_items
INVENTORY aggregates — items, hours/pages, distinct authors, size,
distinct narrators, Top Genres / Top Authors bars, Longest / Largest
items — for one library or the All-libraries aggregate.
- GET /api/stats/library-usage?library_id=<uuid|omit>; family lens
enforced: a library_id outside the requester's visible set is a 403,
never a silent pass (AGENTS.md standing rule;
|
||
|
John O'Keefe
|
86f0af632e |
feat(audiobooks): A2 — narrator metadata, audio-aware UX, player polish, bookmarks
A2a (metadata round): narrators TEXT[] + narrators_search (tags pattern: accumulator + GIN pairs) + abridged bool on media_items; scanner fills from the Libation/ABS sidecar (narrators verbatim, abridged flag) with audio-payload promotion; editorial fields ride the real columns while audio_metadata stays scan-derived (duration/ codec/chapters). Detail/edit/save paths carry the fields; the web edit dialog grows an Audio section (Narrator(s) comma-input, Abridged toggle, scanned duration read-only); search weights + narrator filter (word_similarity over narrators_search) + narrator autocomplete added. A2c (per-media-type wording): copy helpers keyed on format_group — Listen Now, Mark Complete/Not Complete, Listening Progress, Your Listening Stats, Time Listening, Time listened (pages tile swapped for audio), Position H:MM:SS / total H:MM:SS on detail + progress pages (duration parsed from audio_metadata). Dashboard: system collections swap to Continue Listening / Recently Listened at DISPLAY time when the audiobooks library is selected (stored names unchanged; restores keep working); week card = Listening Time / Sessions / Books Touched; cover-corner badges identify audiobook (headphones) / comic / manga cards on mixed grids — books stay unbadged; hover action icon follows the media type. A2b (player): the scrubber binds value to the playhead with an explicit scrub guard (the handle previously froze at its initial max); header row = reader-style back arrow + Chapter n/total launcher; chapters open in a floating top-down drawer (current chapter highlighted, page never shifts); Skip/Speed rows share one label-left layout; honest wall-clock session accounting (playing-only deltas, paused flush). A2d (bookmarks): note-bearing time bookmarks ride the existing media_bookmarks substrate (notes column + sync-aware service already there) — audio locator convention audio:<seconds>, chapter_number filled, auto-title '<chapter> — H:MM:SS', capture popover with optional note, drawer with jump/edit-note/delete. Zero server changes needed. Verified live on the Libation sample: narrators backfilled by rescan; PUT edit persisted + overrides detected (narrators, abridged); force rescan preserved the user edit; reset-to-scanned-of-defaults restored truth; narrator filter/autocomplete scoped by library; dashboard listsening wording + badges render; bookmark save with note deduped to one row. |
||
|
John O'Keefe
|
902cbe5915 |
feat(audiobooks): Batch A — foundation, m4b ingest, playback sync, web player
Phase 1 (foundation): 'audiobooks' library type seeded (schema + DB);
audio_metadata JSONB on media_items (the kobo_metadata precedent —
only audio paths read it); reading_progress.audio_offset_seconds for
the honest {chapter, offset} locator; extensions/mimes registered for
ALL breadth formats (m4b/m4a/mp3/flac — deliberate per the DOC/LIT
doctrine, m4b scans richly first); classifyFormatGroup 'audiobook'
case (gotcha 39 recomputes on rescan); Go mime table extended at boot
so media elements accept the streams; ffmpeg/ffprobe in the image
(8.1.2 — also the xHE-AAC identification tripwire).
Phase 2 (ingest): sidecar-first (Libation/ABS metadata.json —
narrators/ASIN/curated chapters), ffprobe fallback/filler (duration,
bitrate, codec/profile, embedded chapters). Fixed three real
parser brittlenesses the Libation sidecar exposed: series accepts
string OR array-of-objects, publishedYear accepts number OR string,
ffprobe chapter timings accept number OR string (flexFloat). Folder-
structure fallback now runs on the UPDATE path too (rescans were
clearing sidecar-absent series/author); audio facts re-derive on
every rescan (gotcha 39).
Phase 3 (playback + sync): GET /api/media-items/:id/chapters (the
format-abstraction list); streaming rides the existing cookie-authed
Range /uploads path; PUT progress carries audio_offset_seconds +
elapsed_seconds (honest wall-clock playing time) — the reading-
history capture prefers player-reported elapsed over the inter-push
delta, same 2h ceiling; reading_mode='audio' marks the row.
Phase 4 (web player): dedicated audio reader page (foliate not
involved) — cover, chapter title, whole-book scrubber, skip ± (chips
1/5/10/15/30s, default 10s), speed (0.75×–2×), chapter list with
jump-to, Media Session lockscreen/notification transport, periodic
playing-only progress pushes, position restore on open.
Verified live: ingest of the Libation sample (Gardens of the Moon —
title/author/series/cover/date from sidecar+folders, 34 chapters,
26h duration, aac/LC); chapters endpoint; the reader route serving
the player; playback with position restore (0:57 round-trip);
progress rows in audio mode; wall-clock session rows in
reading_history.
|
||
|
John O'Keefe
|
2517baedea |
feat(koreader): device bridge for ratings, reviews and reading-time stats
Release / build-and-push (push) Successful in 3m4s
Extends the device-authed koreader surface with the handoff item 4 Batch B contract: - GET/POST /api/sync/koreader/rating — the device mirrors the .sdr summary's 5-star rating against the 1-10 server scale (POST 0 clears). - GET/POST /api/sync/koreader/review — raw-markdown review mirror (summary.note); empty body clears, matching the first-party PUT. - POST /api/sync/koreader/stats — the device pushes its CUMULATIVE total reading seconds; the server watermark (highest total ever converted, read from the koreader sessions' own device_metadata) converts only the delta into a reading_history session attributed to the koreader device. Reinstalls cannot double-count; sub-minute deltas ride forward (no session, no watermark movement); a lagging device clamps to zero instead of dragging the total back. Book identity resolves through the existing priority chain (uuid → sha256 → alias → title/author). GET binding uses query tags; POST endpoints bind the ref+payload once (an earlier resolve-then-bind draft drained the body). Live-verified against the dev server with the koreader device token: rating set/get/clear, review mirror, and the stats sequence 900→session, 960→delta session, 970→rides forward, 500→clamps; the book's stats card and the By Device panel both pick up the split. |
||
|
John O'Keefe
|
2117ec6181 |
feat(web): book reviews — markdown, sibling of ratings, full API + page surface
media_reviews table mirrors media_ratings' shape (unique per user+item, cascade deletes, updated_at for LWW) but standalone: a review without a rating must work. Body is RAW MARKDOWN — storage and every payload carry the source; rendering happens only at display sites (same doctrine as notes, same sanitization pipeline: markdown-it + DOMPurify). API: GET/PUT/DELETE /api/media-items/:id/review (PUT upserts; an empty body deletes — clearing via an editor is a delete, not an empty row). Book-detail page payload now carries the review beside the rating. Page: the review renders above the synopsis as a scrolling card (same max-height box as the summary), hydrated client-side through renderNoteHtml; 'Write review'/'Edit review' sits next to the rating stars and opens a raw-markdown editor dialog (save / delete / cancel). sqlc regenerated; schema applied to the dev DB. Live-verified: upsert, get, empty-delete, page markup, and the app round-trip against the dev server. |
||
|
John O'Keefe
|
0914c935d9 |
feat(dashboard): collapsible This Week widget, library-scoped + switch-surviving; title into the subbar
Three dashboard complaints, one build: - The widget collapses to a single summary row (This Week · minutes · pages · sessions · books · streak · Full stats) on every screen size and expands to the full grid via a details/summary toggle — the carousel sections start immediately below it again. - The 'Your Library' heading moved into the library switcher subbar (LibrarySwitcher gains an optional title param; collections/series keep the plain 'Library' label). The standalone heading block is gone. - Root-caused the JS-switch vanishing: renderDashboardCollections replaces ALL of #collections-container's innerHTML, so anything server-rendered inside it (title, widget) died on the first library switch. The widget now re-fetches ITSELF: a /dashboard/week-widget fragment (same resolveLibrary family-lens rules as the page) is pulled by htmx whenever the switcher dispatches week-refresh — carrying the current #library-select value — so the widget both survives switches and RE-SCOPES to the switched library. Initial SSR renders the same scope the sections use. ThisWeek (and the streak calendar) gained an optional library filter. Tests: TestReadingStatsWeekLibraryScope — per-library totals, scoped streaks, invisible-library isolation, malformed id 400. Full suite PASS. |
||
|
John O'Keefe
|
7a604c2fec |
feat(stats): per-device reading breakdown + device attribution on progress syncs
- Attribution: registered clients may now send their device token in
X-Device-Token on PUT /api/media-items/:id/progress (the user JWT
stays primary). The token must map to a device OWNED BY the caller —
foreign/invalid/absent tokens soft-fail to plain web attribution so a
stats decoration can never break a sync. Attributed pushes capture
device_id (FK-safe), device_metadata.source = the registered device
name ('Pixel 9', 'Kindle Paperwhite'), and last_sync surfaces show
that device. Side effect worth having: app-vs-web pushes used to be
LWW-invisible (both sources read 'web'); they now surface as
legitimate cross-source progress conflicts when they diverge.
- /stats gains a By Device panel alongside By Library (grid 3-wide):
reading minutes + sessions + books per registered device, plus one
Web / unsynced bucket for device-less pushes so the panel always
reconciles with the headline totals. Range-aware, family-lens
scoped; renderBars shared between panels in stats.ts.
- The device identity IS the registered platform: devices rows already
carry the name/type the client registered (app sends Build.MODEL +
'mobile' at /api/devices/register, KOReader/Kobo likewise) — no new
identity plumbing on the server.
Tests: capture attributed to the device (row device_id + source
metadata + last_sync_source), foreign/nonexistent tokens soft-fail,
device buckets reconcile (80 attributed + 75 unsynced = 155 total).
Full suite PASS.
|
||
|
John O'Keefe
|
6457fbaf72 |
feat(stats): reading statistics — session capture wired + /stats page + widget + per-book card
The design parked in the handoff (multi-tier stats off reading_history; bookorbit-style in data, not looks) is in. The real prerequisite work was data: reading_history existed but NOTHING wrote to it — the whole stats feature was sitting on an empty table. - CAPTURE (no client changes, no repo-boundary crossings): every successful progress write through SaveProgress (KOReader + app + web + Kobo + offline queue — the one chokepoint) now records one reading_history session row: span = the interval since the PREVIOUS write instant, pages = positive page delta when both sides know the page index, marker rows for first contact. Spans over 2h are kept as zero-second markers (overnight suspend→resume, offline flushes must not inflate totals) — the honest 'sync cadence, not a stopwatch' caveat stays. device_id only when a REAL device row matches (the web/app progress path historically stuffs the USER id into DeviceID — probe instead of trusting it; FK violation would silently drop every capture). Source preserved in device_metadata. - API: /api/stats/summary?range=7|30|90|year (totals, zero-filled daily arrays, streaks, most-active day/weekday, top books, library breakdown — one call), /api/stats/this-week (widget: rolling 7d vs previous + streak), /api/stats/books/:id (lifetime + recent sessions). All bucketed in the user's timezone; finished ≡ any push ≥0.995; visible-library lens enforced at source. Legacy /api/analytics/* kept untouched. Range windows anchor end-of-day to the START of today (blasting the clock time into tomorrow resolved 'today' a calendar day late — found by the streak test). - /stats page (sidebar Stats between Progress and Devices, previously /analytics was an orphan): range selector, five stat cards, two Chart.js v4 charts (daily minutes bar, pages/day line — chunked via dynamic import so main.js stays untouched), per-library breakdown, most-read list, empty-state explainer. /analytics now 301s to /stats. The orphaned analytics.templ + analytics.ts removed (their containers and the page JSON never matched; the page was astub). - Dashboard 'This Week' widget (server-rendered, suppressed on service failure) and the book detail 'Your Reading Stats' card (lifetime totals of the viewer's own sessions) complete the tiers: per-user default, per-library breakdown inline, per-book on the page, per-system/admin aggregate parked as before. Tests: statistics_test.go (capture semantics at the chokepoint, tz/day buckets, zero-fill, streak math, finished detection, range validation, auth gate, widget, per-book lifetime). Full suite PASS. |
||
|
John O'Keefe
|
0a7fbd6f48 |
feat(collections): smart (dynamic) collections — rules evaluated live at render
Type is DERIVED, not stored: a saved ruleset makes a collection SMART,
no rules = STATIC (stored collection_items links, unchanged behavior).
No schema change — collections.auto_assign_rules JSONB already carried
the config; user's decision 2026-10-14 (handoff: live-at-render, not
refresh-on-scan; create-time Static/Smart toggle; v1 NO hybrid).
- Rule gains an enabled flag (absent = enabled, legacy rules unaffected).
- Rule CRUD: POST/PUT(DELETE) /api/collections/:id/rules[/:ruleId] — the
survivor dead effort of the rules-builder UI finally has real
endpoints (rules were never editable after creation; nothing ever
re-evaluated them). First rule converts a user collection to smart;
removing the last reverts it to static. UpdateCollectionRulesJSON
edits ONLY the rules doc so rule CRUD cannot clobber metadata.
- Live evaluation: candidates from GetVisibleLiveMediaForRules —
visible-library set (family lens enforced at source) + archived IS
NULL AND missing_scan_count = 0 — any enabled rule matches, title
order. Zero refresh hooks by design: always current by construction.
- Membership mutations (add/remove/bulk-add) are 400 on smart — v1 no
hybrid; collection_items.excluded stays the reserved future hook.
- Rules builder page (/collections/:id/rules) first time ever routable;
templ whole-attribute interpolation ("{ expr }" inside a quoted attr
value is literal text, as theme-{ user.Theme } silently proved).
- UI: create modal Static/⚡Smart cards (smart create redirects into the
rules builder), ⚡ badge + real per-card counts on the grid (smart =
live matches, static = stored-link count), smart detail renders
matched books with no hand-pick actions + an Edit Rules button.
- API: GET /:id gains is_smart + live books; GET /: keeps the Android
app's default exclusion, include_auto=true surfaces smart rows with
live counts; TestRules previews against the SAME candidates the live
render uses (visible + live, was raw first-1000).
- FIXED pre-existing regression from
|
||
|
John O'Keefe
|
740213f0a4 |
fix(auth): new users inherit the system-wide default timezone
default_timezone existed as a system setting (admin-editable, exposed via SettingsRegistry.DefaultTimezone) but nothing read it at user creation — the CreateUser query didn't even carry the column, so every new user got the DB default UTC regardless. Both creation paths (public register + admin create) now seed the user's timezone from the setting, validated with time.LoadLocation and falling back to UTC when unwired/empty/invalid. Test-infra fixes surfaced by the run: TestUpdateProfileSelfEdit was self-editing the SHARED admin (renaming it past the harness cleanup's preserve-list, leaking a second admin row that broke the last-admin tests) — it now edits a dedicated user with a unique name; the last-admin tests get an ensureSingleAdmin precondition helper; the websocket collection name is run-unique (non-test names persist in the shared DB and UNIQUE(user_id,name) 409s on rerun). |
||
|
John O'Keefe
|
431fec05ad |
feat(security): library visibility enforced at the source; per-library series; annotation ownership; admin user creation
The family-lens hardening: library_visibility was only enforced where file bytes flowed (download/reader/OPDS) — metadata surfaces leaked hidden libraries to any authenticated account, and registration was open by default with no admin-side user creation. - Series are grouped PER LIBRARY now (same name in two libraries = two entries: manga adaptations / LN splits no longer merge); /api/series aggregates only the requester's VISIBLE libraries and /api/series/books REQUIRES a visible library_id (breaking contract; web + app clients updated in the same change). - /api/media-items, item detail, and the reader content endpoints (chapters/panels/outline/THUMBNAILS — page images leaked unguarded next to a gated GetPage) all enforce the visible set; resolveLibrary no longer honors a hand-typed library_id outside the visible set. - Annotation/note/bookmark mutations (update/delete/tombstone, sync LWW paths incl. conflict resolution) carry ownership predicates: another account's bare row id is a 404, not a silent delete. - Admin user creation: POST /api/auth/users (AdminMiddleware) + a Create User card in the admin Users UI — accounts stay creatable with self-service registration OFF, which is now the schema default (fresh installs; the first-admin bootstrap exception is unchanged). - Tests: series contract re-pinned (per-library, visibility 403s), new ownership + admin-create integration suites, OPDS all=1 test now passes an explicit page size (shared-DB growth made the fresh books fall off page 1), device-cap helper switched to the admin endpoint. |
||
|
John O'Keefe
|
da13898046 |
feat(server): POST /api/media-items answers honestly on a duplicate file_path
The create endpoint shared the scanner's upsert (ON CONFLICT
(library_id, file_path) DO UPDATE SET updated_at = NOW() RETURNING *) —
so a POST naming an already-registered file returned 201 CREATED with
the OLD row while silently discarding everything the caller submitted.
Nothing depended on that lie: the app only GETs/PUTs media items, the
web only searches/purges/deletes, KOReader never calls it — the
endpoint's create path had zero production callers.
Split the API path from the scanner's idempotent upsert, which is kept
verbatim in CreateMediaItem (re-scan dedup remains its safety net):
- CreateMediaItemIfNew: same insert with ON CONFLICT DO NOTHING — the
loser of a create race writes nothing and returns no row.
- GetMediaItemByLibraryAndPath: fetch the existing row for the
duplicate response.
- Handler branch: on pgx.ErrNoRows → 200 OK with
{"duplicate": true, "message": ..., "item": <existing row>} instead
of a 201 pretending it created something. Real creates still answer
201 with the item as before.
Bulk-import shape (the reason for 200+flag over 409): a re-run after a
mid-import failure is harmless and countable — each response tells the
caller exactly which of the two things happened.
Verified by the full integration suite: 217 groups pass, zero failures,
including a new pin of the duplicate contract (same file_path twice in
one library → 201 then 200/duplicate:true, second payload not
overwritten).
|
||
|
John O'Keefe
|
7ce73213ae |
feat(server): system collections become permanent — soft-delete + in-place restore + query_type-keyed provisioning
The old model hard-deleted system collections and re-created them at the tail, resurrecting stale dedups and breaking ordering. Redesign (user-directed B-final): - schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER — the whole schema is re-applied on every boot via executeSchema). - queries.sql: every collection list/read query filters deleted_at IS NULL; new GetSystemCollectionByQueryType :one (unfiltered, sees tombstones), TombstoneSystemCollection :exec, ClearCollectionItems :exec, RestoreSystemCollectionRow :one (in-place metadata reset + un-tombstone; collections have no updated_at column — ResetUpdatedAt was a 500 caught by tests). sqlc regenerated. - collection_service.DeleteCollection: system collections are tombstoned (preserving position/ownership); user collections keep the hard delete. No ErrNoRows to the client (delete is idempotent). - dashboard_service.RestoreSystemCollection: resolve by query_type; found → in-place restore (ClearCollectionItems + metadata reset, un-tombstoned); missing → provision fresh. "Want to Read"/etc. stay at their home positions forever. - handlers/collections.CreateCollection: name-taken is 409, not 500. - handlers/auth.CreateDefaultCollectionsForUser: idempotent by (user_id, query_type) — update-in-place when a row for that query type exists (heals renames/tombstones), insert when missing; this also fixes the duplicate-system-collections regression at source. |
||
|
John O'Keefe
|
25d44452c0 |
feat(dashboard): global (All Libraries) preferences scope
The dashboard's All Libraries view could render sections across every library but could not persist its own preferences: the PUT required a UUID library_id, and the prefs GET-without-library_id matched an arbitrary library's row through a loose NULL OR-clause. Storage: a dedicated user_dashboard_preferences_global table (one row per user, user_id PK → users CASCADE) instead of a NULL/sentinel library_id — chosen over both because (a) the column's FK to libraries rules out a sentinel UUID without dropping the constraint or minting a fake library row, and (b) intent becomes structural: only the explicit global path writes that table, so a buggy or outside-the-API write can never masquerade as global. System and user collections are user-scoped, so the order/hidden arrays are library-independent and valid in this scope. API: PUT /api/dashboard/preferences binds library_id as *string — JSON null/absent = global, empty string = 400 (a lost value must stay loud, not silently clobber the global scope), valid UUID = per-library (unchanged). GET without library_id now returns the global row instead of an arbitrary one; the per-library query's dead NULL OR-arm is removed. GET /sections keeps requiring library_id for per-library views and aggregates across libraries when absent (unchanged). Tests: global save/update-dedupe/read round-trip, empty-string 400, global-vs-per-library independence. Two suite failures are the handoff's documented pre-existing rot (duplicate system collections in the dev DB; a sections test expecting a 400 the handler never implemented) — reproduced and confirmed unrelated to this change. Live-verified on the dev stack: null save 200 x2 with a single row updated, empty string 400, global read carries the second save. |
||
|
John O'Keefe
|
b9f797789f |
feat: registration_enabled admin setting — hides signup surfaces, gates the API
Release / build-and-push (push) Successful in 3m9s
One boolean (Security > Public Registration) switches the whole public signup lifecycle: front page + login links, the logged-out sidebar's Create-an-account (via a package-level templates hook so shared page templates keep their signatures), GET /register -> 302 /login, and an early 403 on POST /api/auth/register. First-user exception preserved: zero admins keeps every route and link reachable for bootstrapping — same 'users exist' reasoning as the setup gate. Admin user creation is unaffected by design. Live E2E verified: flag off hides all surfaces and blocks the POST (403), flag on restores them (201); the admin UI renders the row automatically. |
||
|
John O'Keefe
|
3d88f0b960 |
feat(search): publisher_filter for the media search endpoint
Adds publisher_filter to SearchMediaItems/SearchMediaItemsUnified (fuzzy word_similarity against mi.publisher, mirroring genre_filter), plumbs it through services.SearchParams and the search handler, and extends the GREATEST relevance ranking to include publisher matches. Serves the app's author/publisher/genre/tag click-through browses (UX pass item 5 — publisher was the only facet without a server filter). |
||
|
John O'Keefe
|
a4dbe79d0b |
feat(data): duplicate-content queries — archive filters, cross-library moves, archived-twin matching
Three related query groups in queries.sql (sqlc regenerated; no generated
signatures changed, so no caller edits were needed):
1. Dashboard/collection archive filters. The smart-section and collection
queries returned archived and missing items (visible only as broken
covers once their files vanished). Added the codebase-standard
"AND archived_at IS NULL AND missing_scan_count = 0" predicate to:
GetContinueReadingItems, GetRecentlyAddedItems, GetRecentlyReadItems,
GetNotStartedItems, GetCollectionItemsForDashboard, GetLibraryItems.
Matches the existing convention in ListMediaItems, SearchMediaItems,
the library counts, and GetContinueSeriesItems.
2. Cross-library SHA move detection support.
- GetMediaItemsBySHA256AndSameLibraryType: finds identical content
(same file_sha256) registered in a DIFFERENT library of the SAME
library type, ordered archived-first, most-missing, oldest. Type
scoping keeps ebooks from merging into manga/comics libraries.
- MoveMediaItemToLibrary: repoints an existing row to the new library
(library_id + recomputed file_path + file_size) and clears
archive/missing state. The row keeps its ID, so reading progress,
annotations, collections, and kobo shelves follow the book
automatically; the new set_library_type_name_on_library_change
trigger refreshes the denormalized type column.
3. Archived-twin matching. ListHiddenMediaItems (admin archived-items
page) now LEFT JOIN LATERALs the best active same-SHA twin per hidden
row — preferring same library type, then same library, then most
reading progress, then oldest — exposing match_id / match_title /
match_library_name so the UI can offer merging the archived row's
reading data into its active duplicate. COALESCE on match_title keeps
the no-match case NULL-safe.
|
||
|
John O'Keefe
|
17216e9cc7 |
feat(db): archive lifecycle queries, immediate-hide filters, move support
Storage behind the scan lifecycle fixes: - MoveMediaItemFilePath: repoints a row (path, size) and clears archive state when identical content reappears at a new location. - ListHiddenMediaItems: archived OR missing rows for the admin archived-items page; ListMediaItemsByLibraryIncludingArchived (prior commit) already fed the scanner. - All user-facing listings now hide missing items immediately, not just archived ones: ListMediaItems, ListMediaItemsByLibrary, ListMediaItemsSorted, SearchMediaItems, SearchMediaItemsUnified, the next_books CTE, library media counts, and the five search autocomplete value queries gained AND mi.missing_scan_count = 0 next to the archived_at filter. Purge timing is unchanged (archived_at + ARCHIVE_RETENTION_DAYS), so the grace period keeps protecting data while the UI reflects removals on the first scan. - Detail lookups by id/path/sha stay unfiltered on purpose. |
||
|
John O'Keefe
|
e6aceae0da |
feat(db): archive lifecycle schema and queries for missing media
Add the storage behind the archive-instead-of-delete lifecycle: - media_items.missing_scan_count (INT, default 0) and archived_at (TIMESTAMPTZ, partial index), both as idempotent ADD COLUMN IF NOT EXISTS backfills for existing installs. - MarkMediaItemMissing / ArchiveMediaItem / ClearMediaItemArchive plus PurgeExpiredArchivedMediaItems (retention cutoff) and PurgeAllArchivedMediaItems (manual bulk), with CountArchivedMediaItems for the admin UI. - Archived items are hidden from every user-facing listing: ListMediaItems, ListMediaItemsByLibrary, ListMediaItemsSorted, SearchMediaItems, SearchMediaItemsUnified, the next_books CTE, the library media counts, and the search autocomplete value lists. Detail lookups by id/path/sha are intentionally unfiltered, and a dedicated ListMediaItemsByLibraryIncludingArchived feeds the scanner so the lifecycle pass can see and restore archived rows. |
||
|
John O'Keefe
|
7f64b92b9d |
feat(metadata): per-field user overrides that survive library rescans
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
|
||
|
John O'Keefe
|
b9645752f5 |
fix(db): skip SQL comment lines when parsing schema table names
parseTableNames() regex-scanned every line of schema.sql, comments
included, with 'CREATE TABLE (?:IF NOT EXISTS )?(?:\w+\.)?(\w+)'. A doc
comment containing that phrase in prose - e.g. 'declared in CREATE TABLE
above' - registered a phantom table ('above'), and startup verification
then failed with 'missing tables: above', crash-looping the app
container on every restart.
Skip lines whose trimmed form starts with '--' so comments can never
contribute table names, and add a regression test asserting every parsed
table maps back to a real CREATE TABLE statement.
|
||
 john-okeefe
|
7b1c809ae3 |
feat(bookmarks): location identity, origin provenance, KOReader-style labels
- Drop UNIQUE(media_item_id,user_id,title): titles are display labels shared verbatim across clients; same-title bookmarks on different pages now coexist instead of 500ing (deleting over a tombstone no longer blocks future creates with that title) - Add origin_source column recording the creating client (android/web/ koreader), set once at insert, exposed in API responses - Web reader auto-title mirrors KOReader's 'in <chapter>' convention, falling back to 'Bookmark'; adds bookmark rename in the drawer |
||
|
john-okeefe
|
b1eda696f1 |
Revert "fix(bookmarks): upsert on title conflict so position upgrades don't 500"
This reverts commit
|
||
|
john-okeefe
|
27b3dcb69f |
fix(bookmarks): upsert on title conflict so position upgrades don't 500
Bookmark dedup is keyed on hash(title + position bucket), but the table also enforces UNIQUE(media_item_id, user_id, title). When a client re- saves the same bookmark title with a changed position form - e.g. the Android app upgrading a percentage-only row to an EPUB CFI, or a web and app bookmark landing on the same 'Bookmark at 44%' title - the dedup-key lookup misses and the INSERT violates the title constraint, returning HTTP 500 and failing the sync. A title collision on the same (user, item) is by definition the same bookmark slot, so take the LWW semantics all the way: ON CONFLICT DO UPDATE replaces position/cfi_position/page/chapter/percentage, refreshes dedup_key and timestamps, merges device_sync_data, and - matching UpdateMediaBookmarkForSync - clears deleted/deleted_at so a re-create resurrects a tombstoned title slot instead of leaving an invisible row holding it. Device sync flows are unaffected: KOReader/Kobo pushes that carry their own dedup-key echoes never reach the INSERT, and same-key saves still go through applyBookmarkLWW with its tombstone freshness checks. |
||
|
john-okeefe
|
acbb6c7981 |
feat(db): queries for the deleted-annotation history
ListDeletedAnnotationsForBook unions tombstoned highlights, notes, and bookmarks for a user+book regardless of the sync TTL cutoff (the history must show everything still restorable, not just recent deletes), with display text, secondary text, color, and both timestamps. Restore queries clear deleted/deleted_at (lossless — the row was soft- deleted, never removed) and are scoped to the owning user and media item so a restore can never touch another user's annotation. Purge queries hard-delete an already-tombstoned row: the user-driven counterpart of the TTL maintenance sweep, for explicit 'delete permanently' actions from the history. All six write queries are :execrows so callers can distinguish 'restored' from 'nothing matched' without a follow-up read. |
||
|
john-okeefe
|
0670d904a0 |
feat(db): locator columns for tombstoned annotations
GetTombstonedAnnotationsForBook now also returns each tombstone's start_position/end_position and epubcfi_start/end (note: position/ epubcfi_location, bookmark: position/cfi_position), so serving code can resolve a device-native locator for deletions of web-created annotations, whose device_sync_data carries no pos0. |
||
|
john-okeefe
|
a962342ee0 |
fix(sync): resurrect tombstoned annotations when a newer save re-creates them
Deleting a bookmark/highlight/note and then re-adding the same content at the same position (same dedup key — e.g. the reader's auto-titled 'Bookmark at X%') was silently swallowed: the save hit the tombstone branch, returned 201 with the deleted row, and the list (which filters deleted) stayed empty. Bookmarks were further blocked by the UNIQUE(media_item_id, user_id, title) slot the tombstoned row holds, and notes had no TTL escape at all. Tombstones now only block saves that predate them (stale replays from a device that still has the annotation). A save whose modification time is newer than max(deleted_at, last_modified_at) — a deliberate re-create from the web or a device — resurrects the row via the LWW update queries, which now clear deleted/deleted_at. |
||
|
john-okeefe
|
ba95cc3e8b |
fix(reader): stabilize chrome panels, bookmarks end-to-end, dead UI removal
Phase 0 of the reader redesign: - Panels no longer render under the top/bottom bars: sidebars get measured insets (same resize/safe-area mechanism as the viewport); panel max-height now derives from the bounded sidebar instead of a 100vh guess; right-side border targets the actual sidebar. - Bookmarks work end-to-end for the first time: REST CRUD under /api/media-items/:id/bookmarks (create/delete route through AnnotationService for dedup/LWW/tombstones), fix UpdateMediaBookmark referencing nonexistent updated_at column, frontend posts to the real API with per-format position (CFI vs page), live list with jump + delete instead of SSR-only snapshot. - Fix chapter matching in progress saves: boundaries were compared by a nonexistent tocItem property, so chapter was never persisted. - Remove dead UI: Navigator panel stub, empty dictionary popup shell, unwired Chrome Behavior select; purge 160 stale build artifacts. - Reader chrome now follows the user's app theme instead of hardcoded theme-tokyo-night. |
||
|
john-okeefe
|
0c39e04e4a |
feat(db): hash_conflicts table and backfill/conflict queries
Content duplicates (same library + file_sha256 at different paths, e.g. the same book imported twice under two names on a preexisting database) cannot be auto-collapsed the way path duplicates were: keeping both copies may be intentional. Surface them for an explicit admin decision instead. Schema: - new hash_conflicts table keyed (library_id, file_sha256) with a status/resolution lifecycle: 'pending' until an admin resolves via 'keep_all' or 'kept:<uuid>' (which copy was kept after merging) - resolution is VARCHAR(50) - 'kept:<uuid>' is 41 chars; include a widening ALTER for databases created with the initial 30-char width - resolution/resolved_by/resolved_at record who decided what and when Queries: - ListMediaItemsMissingHash: items imported before hashing existed (file_sha256 IS NULL), ordered oldest-first for the backfill pass - FindHashConflictGroups: the content-duplicate group detection (GROUP BY library_id, file_sha256 HAVING COUNT(*) > 1) - ListMediaItemsBySHA256AndLibrary: full membership of one group - CreateHashConflict: upsert with DO NOTHING so already-tracked groups are untouched - critical behavior: a group an admin resolved as 'keep both' is never re-flagged by later sweeps - ListPendingHashConflicts: admin listing with library name and live item counts (items may have been deleted since flagging) - GetHashConflict / ResolveHashConflict: lifecycle - GetMediaItemUsageCounts: per-item progress/highlight/bookmark/note/ collection counts so the admin can make an informed keep choice - ReparentMediaItemChildren: sqlc binding for the existing reparent_media_item_children() migration function, used to merge a losing copy's child rows into the kept copy |