c0e699327309fa7ae8ce4523c107ab5a4954c0c1
90
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
0b7ad0f376 |
feat(podcasts): schema, RLS policy extension, sqlc queries — feed core foundations
Podcast integration task 1.1 (plan: podcast-integration.md §6/§8, D20).
WHY: podcasts are a new first-class media domain — RSS feeds users
subscribe to personally, with episodes as media_items rows so progress,
stats and the audio player work unchanged. Visibility is per-user
(strict: only your subscriptions), which after the RLS migration
(2026-10-29) belongs in the POLICIES, not Go helpers.
Schema (idempotent — schema.sql re-executes every boot):
- 'podcasts' library type seeded (ON CONFLICT DO NOTHING, ARRAY[]::TEXT[]
cast required — bare ARRAY[] 42802s under ON_ERROR_STOP)
- podcast_feeds: per-library feed entity (UNIQUE(library_id, feed_url)),
full RSS/iTunes/Podcasting-2.0 metadata inventory, conditional-GET
validators, is_complete poll backoff
- podcast_subscriptions: the ONLY user-writable podcast state (display
name, auto-download policy); deliberately UNGOVERNED — it IS the read
policy's lookup table
- media_items episode columns: source/source_feed_id/episode_guid/
feed_position/season/episode/enclosure_url/declared_duration/
transcript_url/chapters_url/hosts/guests + download state machine
(pending|queued|downloading|downloaded|failed + error + attempts),
all partial-indexed so scan rows never touch them
- libraries.allow_member_feeds (first per-library knob); library_folders
.purpose ('podcast' download roots, scanner-skipped)
RLS (inside the labeled policy block):
- p_media_items_visible gains the podcast-episode subscription predicate
+ admin escape (source IS DISTINCT FROM 'podcast_episode' guard keeps
scan rows on the cheap path); unset identity still fails loud
- podcast_feeds governed identically: library-visibility join AND
subscriber-or-admin SELECT; permissive write stand-ins (mandatory at
ENABLE time; fetcher/worker declare first-admin, add-feed upsert uses
ON CONFLICT DO UPDATE RETURNING so creators never need SELECT on
foreign rows)
- Verified against dev DB: migration identity sees nothing, admin sees
all, member sees feeds/episodes only post-subscribe (probed in a
rolled-back tx — no rows persisted)
sqlc: 30 new queries (feed upsert/CRUD, subscriptions, episode
upsert-with-write-after-success-guard, download state transitions,
poller worklist, subscription-scoped search) — no visibility clauses
anywhere, per the D20 scoping model.
|
||
|
John O'Keefe
|
304ea9599f |
feat(deploy): inline bootstrap into db-init — compose is the single source; DB_PORT everywhere
z_rls_role.sh is gone. The db-init service carries the bootstrap SQL
inline (the script's docker-exec/local-socket mode was its only reason
to exist as a file, and the no-clone deployment path could not use it:
docker bind-mounts of missing host paths auto-create empty DIRECTORIES,
so a compose-only install got '/z_rls_role.sh: no such file').
The inline form surfaced two silent bugs the script had masked:
- psql defaulted its DATABASE to the username (POSTGRES_DB is not a
libpq variable) — the ownership loop ran against 'postgres' and
no-opped on existing volumes. Fixed with PGDATABASE: bookhoard.
- PGPORT was never propagated: with a custom DB_PORT, db-init dialed
5432 while Postgres listened on the custom port. Fixed with
PGPORT: ${DB_PORT:-5432} — DB_PORT is now a variable in every
connection path (db listener, host mappings, app, db-init).
Also removed the db service's initdb.d mount: the app executes the full
schema as the runtime-role owner on every boot incl. first (pg_trgm is
a trusted extension; the db owner can create it), so the bootstrap SQL
was its only content.
Verified: dev two-boot against the existing 40-table volume (owners
converged, 0 off-owner tables/functions), custom-port throwaway
(DB_PORT=15432: db-init + app both on the custom port, healthy),
default-port throwaway (fresh volume, healthy), full Go suite green.
Manual repair path: docker compose run --rm db-init.
|
||
|
John O'Keefe
|
313268a7ae |
feat(deploy): db-init one-shot service — paste-and-go bootstrap for the runtime role
The role split previously required a one-time manual script run on existing volumes (initdb.d only fires on fresh ones). This folds the bootstrap into the compose file the way split-account deployments standardly do it: a one-shot db-init service runs z_rls_role.sh (idempotent, network mode via PGHOST=db + PGPASSWORD from DBPASS) on every 'compose up', and the app gates on db-init: condition: service_completed_successfully. - Zero manual steps for fresh AND existing deployments: update compose, add DBSRVPASS to .env, 'compose pull && up -d'. - Security posture improves: the superuser password is now held ONLY by the init service — the app container connects exclusively as the non-superuser runtime role. - z_rls_role.sh gains dual connection modes (local socket for docker exec, network for the init service) with the DBPASS requirement enforced only in network mode. - Verified on the dev stack: init runs and exits clean, app gates on it and comes up healthy, second 'up' re-converges idempotently, invariant test passes through the new dependency chain. The initdb.d mount stays as-is (harmless double coverage on fresh volumes; both paths are idempotent). |
||
|
John O'Keefe
|
f4b581872f |
wip(rls): ScopedDB + rls package + schema policies + echo bridge [pin compiling state]
Phase 3 of the RLS plan, pinned mid-flight because a parallel writer
was actively editing cmd/server/tests/rls_visibility_test.go (A14):
the file flip-flopped between two implementations and gained a compile
error after the previous commit. This snapshot makes the whole tree
compile; functional verification (two-boot gate, invariant green,
full suite) lands in the next commit(s).
- internal/rls: typed ctx keys — user identity (bridged from echo by
the router) and an INTERNAL marker whose semantics grant nothing:
no GUC is declared, so policy'd tables still raise at the engine;
only RLS-ungoverned tables (system_settings, users counts) work for
user-less startup paths (settings registry Load, CountAdmins).
- internal/database/scoped.go: DBTX decorator — per-statement tx,
commit-on-Close (:many) / commit-on-Scan (:one), error paths always
terminate the tx, refusal before SQL when no identity and not
internal.
- database/schema/schema.sql: idempotent policy block (DROP IF EXISTS
+ CREATE — schema.sql re-runs every boot): ENABLE + FORCE, read
policy exactly mirroring COALESCE(is_visible,true), permissive
per-command write stand-ins (FOR ALL would OR into SELECT).
- router.go: RLS bridge at BOTH c.Set("user") sites (JWT + device
auth) — echo-local storage never reaches the stdlib ctx.
- main.go + test harness: database.New over ScopedDB at both wiring
sites; startup reads use rls.WithInternal.
|
||
|
John O'Keefe
|
cb258b6d20 |
feat(db): dedicated non-superuser runtime role for RLS
The app connected as 'postgres' — the bootstrap SUPERUSER — which bypasses row-level security unconditionally (FORCE RLS does not apply to superusers), making any future policy a silent no-op. This is the prerequisite commit for the RLS visibility work (plan ~/.opencode/plan/server-rls-library-visibility.md REV 2, §4.0). What changed: - database/schema/z_rls_role.sh: creates the bookhoard_srv role (LOGIN, non-superuser, non-bypassrls) and transfers ownership of every public table/sequence/function + schema + database to it. Ownership (not mere grants) because executeSchema re-runs the whole schema.sql on EVERY boot and needs owner DDL. Idempotent, catalog- only — no user data is read or written. - docker-compose.yml app service + docker-compose.dev.yml tests service: DATABASE_USER postgres -> bookhoard_srv, DATABASE_PASSWORD DBPASS -> DBSRVPASS (new .env key, generated locally). - POSTGRES_USER: postgres stays as the container bootstrap account; on FRESH volumes z_rls_role.sh runs automatically via docker-entrypoint-initdb.d AFTER schema.sql (alphabetical: z > s). Verified on the dev stack: role has rolsuper=f, rolbypassrls=f, owns all 40 tables and every public function; single boot as bookhoard_srv executed schema.sql cleanly (CREATE EXTENSION IF NOT EXISTS pg_trgm skips fine as non-superuser when the extension exists); health + authenticated smoke pass. Behavior otherwise unchanged — no policies yet. |
||
|
John O'Keefe
|
d42c78f58c |
fix(schema): widen media_items.genre to TEXT
The genre column was VARCHAR(100), and the audio tag fallback stores the
file's raw genre atom verbatim. Libation writes Audible's category ladder
as one comma-joined string ("Politicians, Presidents & Heads of State,
United States, Russia, Communism & Socialism, International Relations" —
111 chars), so any such book failed its import or metadata update with
SQLSTATE 22001 (value too long), taking the whole row down with it.
VARCHAR -> TEXT is a metadata-only change in Postgres and safe to re-run;
the in-place ALTER follows the media_highlights position-widening
precedent so existing databases converge on restart (schema.sql runs on
every boot).
|
||
|
John O'Keefe
|
f6a73ab5aa |
feat(unlinked-books): dismiss (ignore) or delete queue entries
Release / build-and-push (push) Successful in 3m15s
Unidentified sync files used to be stuck: every KOReader re-sync re-queues them, and there was no way to say "this isn't a book" or "I don't want this one on the server". Ignore is the durable dismiss: a new ignored_at column hides the row from the page list, dashboard badge, and per-device API while the sync keeps refreshing its metadata (the upsert never clears the column). A collapsible "Ignored" section lists dismissed rows with Restore and Delete so a misclick is recoverable. Delete is the hard exit — the row is gone, and honestly re-queued if the device syncs that file again (the confirm dialog says so). Actions are page-scoped SSR routes: htmx swaps in the re-rendered section so counts and buckets can never drift from the DB; ownership is row → device → user (admins may act on any row). Integration tests pin the lifecycle (hide everywhere, survive the sync upsert, restore, delete), foreign-user 403s, and the render surface. |
||
|
John O'Keefe
|
c645b8e207 |
feat(profile): keep_screen_awake user preference (default on)
New users column surfaced as an On/Off dropdown in the profile form. The form and API carry "true"/"false" strings with the theme-style ""-guard, so JSON clients that omit the field can't clobber it; anything else is rejected. GET/PUT /api/auth/profile expose keep_screen_awake alongside the other profile fields, and the reader route threads the preference into its template metadata. Integration tests pin the schema default, both values, omission semantics, and rejection — dedicated test user, never testuser. |
||
|
John O'Keefe
|
3576a4415a |
feat(audio-player): per-book playback speed + user default (plan §8)
Speed becomes a per-USER-per-BOOK preference that syncs across devices, with a user-level default for books not yet played: - new audio_playback_speed table (mirrors reading_speed's shape; a preference, NOT a measurement and NOT a reading_progress column) + users.default_audio_speed, schema in place - chapters endpoint resolves per-book row → user default → 1.0 and ships playback_speed + default_playback_speed (zero extra round trips; default powers the dropdown marker) - PUT /api/media-items/:id/audio-speed upserts (range-checked; the client SPEEDS list stays the real gate; library visibility gates writes) - the user default rides the profile path like theme/timezone: profile form gains a 'Default Playback Speed' select (server-side SpeedOptions mirrors the web SPEEDS list), GET /profile carries it - web: restorePosition applies the resolved server speed (localStorage audio_speed demotes to a first-paint cache); setSpeed PUTs fire-and-forget with a failure toast; the Speed dropdown gains a '(default)' row marker and a 'Make N× my default' action with a success toast ALSO fixes a pre-existing profile-page bug surfaced by the §8 form verification: the password section's current_password input was required INSIDE the shared form, so HTML5 validation blocked every Save Changes submission (dead since the shell migration). The password flow keeps its own gates (disabled-button checkers + server 400). Integration test TestAudioPlaybackSpeed pins the full contract (unset → 1x, upsert resolution, range rejection, default fallback for row-less books, default updates re-satisfying unset books). Verified live: 9 Playwright checks. |
||
|
John O'Keefe
|
9b5defcb31 |
feat(issues): the honest processing-issues ledger — row-less import failures, self-healing resolution, family lens (faceted plan Phase 11)
processing_issues only recorded cross-library move rejections; files that failed to import vanished into an error counter and stdout. The ledger now records every failure class and heals itself: Schema (in place, pre-release boundary): - media_item_id NULLABLE + file_path TEXT (deliberately unbounded — an over-long path is itself a failure mode; a VARCHAR(500) ledger column died on the exact 22001 it existed to report — caught by the new test). Row-less failures key on (library, file_path); item-backed on (item, type). Widened dedup = COALESCE-expression unique index; the old table UNIQUE stays (NULLs pairwise-distinct for row-less). - Conflict-target upsert: repeat failures UPDATE the same row and REOPEN resolved ones — never duplicates. Wirings: - CreateMediaItem failure (single-file + multifile createAudioBookItem) → row-less import_failed (error). - Extraction hard-fail with fallback import → item-backed metadata_extraction_failed (warning). - Change-path re-extraction failure (the ev/sidecar guard) → item-backed metadata_extraction_failed; LogProcessingIssue keeps its signature, a row-less helper joins it. Auto-resolution: - Successful import → ResolveOpenProcessingIssuesForPath (library, file_path). - Successful (re)extraction → ResolveOpenProcessingIssuesForItem (single-file change path, multifile editorial apply). - Unchanged-skip scans do NOT resolve: extraction never re-ran, so the degraded metadata is still degraded — the page heals exactly when the book does. Family lens on ALL issues surfaces (list, stats, resolve, delete, SSR page): explicit out-of-set library id → 403, never silent. This tightens one pre-lens behavior — a nonexistent library id returned 200-empty, now 403 (pinned, changelog entry in docs/contract_pins.md). List is LEFT-JOINed so row-less rows appear (media_item_id "", title falls back to the path). Tests: full lifecycle state machine through real scans (fail → appears, repeat → no dups, success-at-path → resolved, LEFT-JOIN list shape); family lens 403s with a fresh non-shared admin; all 7 pre-existing issue suites green (two expectations widened to the lens, changelogged); adjacent suites (CrossLibraryMove, Audiobook, Calibre, ExtractorVersion, SchemaIdempotency) green; unit suites green. |
||
|
John O'Keefe
|
91b0c45dd4 |
feat(scanner): extractor-version (ev) mechanism — parser fixes reach old rows without a force rescan (faceted plan Phase 10)
Parser fixes (
|
||
|
John O'Keefe
|
6750e9e0dc |
feat(scanner): manifest-gated sidecar change detection + playlist sync on change path
Part 2 of the faceted plan, Phase 4 (steps 3-4). Size-only detection
was blind to metadata-sidecar edits; a size-stable book is changed by
EDITING its metadata.opf/metadata.json/cover — previously invisible
until a force rescan.
- Sidecar manifest fingerprints (media_scanner_manifest.go): the
manifest stats exactly the names the extractors consume (metadata.opf,
metadata.json, findSidecarCover patterns incl. <basename>.<ext>), so
manifest and extractors can never disagree. Shape v1:
{v, entries:[{n, s, m (unix nanos — second granularity would hide
edits landing in the backfill scan's own second), h (sha256 — covers
included: the repoint ladder needs a baseline hash for any stat-
drifted sidecar; a hashless cover turns harmless mtime drift into a
re-extraction, the exact flood this design kills)]}.
- Evaluation ladder in cost order: stat pairs identical → skip with
ZERO bytes read; stat drift → hash that sidecar (equal = host repoint
reconciliation, stat pair refreshed, no re-extraction; differing =
changed); NULL stored manifest → backfill only, NO re-extraction
(deploy-day rule: the first scan after release must not look like an
mtime flood).
- Change path: re-read full row, extractMetadata once and pass it
through updateMediaItemWithMetadata (preExtracted — metadata_overrides
stay protected), run syncPlaylistCollectionsAfterCreate (Correction 2:
a reading-order entry added to an EXISTING book now creates its
collection; add-if-missing is idempotent), store a fresh full
manifest. Creates seed their manifest at insert time.
- Multifile folder/zip books intentionally keep Phase 3 behavior: their
change path does not consume sidecar extractors, so gating them would
flip books 'changed' without picking anything up (plan correction —
the plan's §3 multifile note assumed otherwise).
- UpsertMediaScanState + GetMediaScanState queries.
Tests: media_scanner_manifest_test.go (ladder unit tests — fast path,
repoint, same-size drift, appear/vanish, unhashable legacy entries);
TestScanManifestLifecycle (seeding, unchanged-skip stability, repoint
reconciliation with stat-pair refresh assertion, same-size edit
consumed, playlist sync on the change path + idempotency, no-leak
guard). Container runner.
|
||
|
John O'Keefe
|
693053a9f5 |
perf(scanner): bulk scan-state load — one query per library scan
Part 2 of the faceted plan, Phase 3 (steps 1-2). The periodic scan
issued one GetMediaItemByFilePath per scannable file (~3,094 queries
per scan in prod, nearly all answering 'unchanged, skip') plus one
GetLibraryByFolder per file for folder resolution.
- NEW media_scan_state table: the scanner's per-item fingerprint
cache. Deliberately NOT a media_items column — the library-scoped
list emits raw sqlc rows as JSON (
|
||
|
John O'Keefe
|
1dbb0ce406 |
feat(collections): playlist auto-creation, position ordering, hide_finished, Up Next, dashboard eligibility
Phase 2b + 3 of the metadata overhaul (plan §2.4, §3.1-3.7). Auto-playlist collections (§2.4): the scanner turns obviously-playlist sidecar series (explicit '(Reading Order)'/chronological/publication/ in-order/timeline markers only — precision over recall) into curated static collections: upsert by view_settings.imported_from_series then name (rename-safe), position set ONLY on insert (clean integers; 1.5/1a stay unpositioned rather than guessing), membership add-if-missing, never removed or re-ordered by rescans, provenance in description + view_settings, show_on_dashboard true (dashboard-eligible by design), 'scanner.auto_playlist_collections' setting (default on, Settings Registry + schema seed). Ordering (§3.1-3.2): collection listings order position ASC NULLS LAST then added_at ASC (was added_at DESC — walked playlists backwards); PUT /api/collections/:id/items/order applies full-order rewrites (owner-only, service-gated). hide_finished + Up Next (§3.3-3.4): GetCollectionItemsForDashboard gains viewer_user_id + hide_finished (LEFT JOIN reading_progress, COALESCE percentage < 0.999) and viewer_percentage rides the row for card graying; GetCollectionUpNext serves the collection's reading-queue head with the family-lens visible-library array enforced. Dashboard (§3.5): user collections are dashboard-ELIGIBLE BY DEFAULT — the show_on_dashboard gate (never writable for user rows since the column shipped) is gone from GetUserCollectionsForDashboard; hiding stays dashboard-prefs (hidden_collections) territory; hide_finished filtered-everything now emits a Complete section instead of silently vanishing; sections JSON carries is_playlist/complete. Series page ordering hardening (§3.6): GetSeriesBooks tiebreaks by title ASC — NULL/tied numbers previously returned heap (scan) order, which is what the user saw. Frontend custom-section-builder note: the builder has sent show_on_dashboard: true since its creation; the API never bound it. User-row visibility is now intentionally prefs-governed, so the field stays unbound by design (documented in code). Pins widened (docs/contract_pins.md): collection detail + up_next; dashboard sections + is_playlist/complete; new reorder endpoint recorded. Verified: TestPlaylistCollectionsEndToEnd (new, full e2e: sidecar series split, collection auto-creation with position 7, rescan idempotency, owner-only reorder, finished-filter + completion marker + Up Next drain) plus the full affected integration sweep green. |
||
|
John O'Keefe
|
646477afad |
feat(schema): subtitle/explicit columns, authors+publishers arrays, float series numbers
Phase 1 of the series/collections/metadata overhaul
(~/.opencode/plan/series-collections-metadata-overhaul.md).
Schema (in place, idempotent):
- media_items.series_number INTEGER -> DOUBLE PRECISION: real-world
ordinals are decimals (0.5, 1.5 novellas); INTEGER silently truncated
them (calibre 2.5 pinned as 2 in tests).
- + subtitle TEXT, explicit BOOLEAN (sidecar ingest; ABS explicit /
Audible isAdult).
- + authors/authors_search/publishers/publishers_search TEXT[] NOT NULL
(narrators A2a precedent). Arrays are write truth; author/publisher
varchars become trigger-regenerated display strings
(sync_author_publisher_display) so every varchar consumer (trgm
search, facets, API, OPDS/Kobo) keeps working unchanged.
- Row-level NULL normalization for the new arrays in
set_library_type_name (A2 lesson: Go nil -> explicit NULL -> 23502).
- Backfills ARRAY[author]/ARRAY[publisher] — deliberately no
comma-splitting ('Tolkien, J.R.R.' would corrupt; scanner kept only
authors[0] historically).
- collection_items.position INTEGER + (collection_id, position) index
for playlist ordering.
Go: sqlc regen; MediaMetadata/handler params gain Subtitle/Explicit/
Authors/Publishers (legacy singles still accepted, arrays win);
SeriesNumber int32 -> float64 end to end; metadata_overrides adapted
with legacy override keys ('author'/'publisher') preserved so existing
user overrides survive; bulk author edit writes the array.
Pins widened (docs/contract_pins.md): raw sqlc row key set + the six
new columns; changelog entry added.
Verified: unit suite green; integration sweep (bulk/smart collections/
scanner/stats/sync/queue/calibre/pins/audiobooks) green against the
shared dev DB after in-place migration. Pre-existing unrelated red:
TestOwnershipAndAdminTestSuite/TestAdminCreateUser timezone inheritance
fails identically at the pre-change commit (live shared state).
|
||
|
John O'Keefe
|
6216bd9bcf |
feat(audiobooks): multi-file books (folder-of-tracks + zip) + exhaustive metadata ingestion
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
|
||
|
John O'Keefe
|
d74db03cf3 |
feat(sync): unlinked-books queue — ingest, page route, dashboard banner
Release / build-and-push (push) Successful in 3m7s
The identification queue for device files the sync cannot match is now live end-to-end. Previously the whole feature was dark: unmatched device files were silently dropped (losing their reading progress), the queue table sat empty by construction, and the page template had no route. - Ingest: the KOReader sync loop, on an unresolvable file, upserts unlinked_books (one row per device + content id; re-syncs bump last_seen_at and refresh device-reported metadata) instead of dropping the file. Schema gains the (device_id, content_id) unique index the upsert needs. - Page: GET /unlinked-books routes the existing page template, listing the caller's own unresolved rows (ListUnresolvedUnlinkedBooksForUser — devices.user_id scoping; no cross-user leakage). - Dashboard banner: a warning card above the shelves, rendered only when the caller's queue is non-empty (hidden-unless-needed), linking to the page. - Resolve flow unchanged and already gated: MatchingHandler.LinkBook verifies device ownership and target-item visibility (403s) and creates the device file alias, so the next sync matches by path. SyncHandler.LinkBook and Handler.LinkBook are unrouted dead handlers — triage candidates, not part of the live flow. |
||
|
John O'Keefe
|
0799cca3e6 |
fix(schema): ship the two Batch-A upgrade ALTERs that 1.15.0 missed
audio_metadata (media_items) and audio_offset_seconds (reading_progress) were declared only inside their CREATE TABLE blocks — fresh installs got them, upgraded installs never did, because columns reach existing databases only via the idempotent ALTER block that schema.sql re-runs on every boot. On prod (v1.15.0) this 42703'd every media-existence check and the archive pass (1817/1817 scan errors), and every progress save, since the shared progress upsert names audio_offset_seconds. Both ALTERs match the declared shapes exactly (nullable JSONB; INTEGER CHECK (audio_offset_seconds >= 0)); idempotent, applied on every boot. Verified against a scratch database built to the v1.14.0 shape: overlay the patched schema -> both columns created; re-run (every boot) skips cleanly; probes resolve reading_progress.audio_offset_seconds integer and media_items.audio_metadata jsonb. Parse test and a real container boot (38 tables, 6 functions verified) green. |
||
|
John O'Keefe
|
a222ef5c27 |
fix(media-items): normalize NOT NULL narrators on every write path
A2 made narrators/narrators_search NOT NULL DEFAULT '{}' (
|
||
|
John O'Keefe
|
86f0af632e |
feat(audiobooks): A2 — narrator metadata, audio-aware UX, player polish, bookmarks
A2a (metadata round): narrators TEXT[] + narrators_search (tags pattern: accumulator + GIN pairs) + abridged bool on media_items; scanner fills from the Libation/ABS sidecar (narrators verbatim, abridged flag) with audio-payload promotion; editorial fields ride the real columns while audio_metadata stays scan-derived (duration/ codec/chapters). Detail/edit/save paths carry the fields; the web edit dialog grows an Audio section (Narrator(s) comma-input, Abridged toggle, scanned duration read-only); search weights + narrator filter (word_similarity over narrators_search) + narrator autocomplete added. A2c (per-media-type wording): copy helpers keyed on format_group — Listen Now, Mark Complete/Not Complete, Listening Progress, Your Listening Stats, Time Listening, Time listened (pages tile swapped for audio), Position H:MM:SS / total H:MM:SS on detail + progress pages (duration parsed from audio_metadata). Dashboard: system collections swap to Continue Listening / Recently Listened at DISPLAY time when the audiobooks library is selected (stored names unchanged; restores keep working); week card = Listening Time / Sessions / Books Touched; cover-corner badges identify audiobook (headphones) / comic / manga cards on mixed grids — books stay unbadged; hover action icon follows the media type. A2b (player): the scrubber binds value to the playhead with an explicit scrub guard (the handle previously froze at its initial max); header row = reader-style back arrow + Chapter n/total launcher; chapters open in a floating top-down drawer (current chapter highlighted, page never shifts); Skip/Speed rows share one label-left layout; honest wall-clock session accounting (playing-only deltas, paused flush). A2d (bookmarks): note-bearing time bookmarks ride the existing media_bookmarks substrate (notes column + sync-aware service already there) — audio locator convention audio:<seconds>, chapter_number filled, auto-title '<chapter> — H:MM:SS', capture popover with optional note, drawer with jump/edit-note/delete. Zero server changes needed. Verified live on the Libation sample: narrators backfilled by rescan; PUT edit persisted + overrides detected (narrators, abridged); force rescan preserved the user edit; reset-to-scanned-of-defaults restored truth; narrator filter/autocomplete scoped by library; dashboard listsening wording + badges render; bookmark save with note deduped to one row. |
||
|
John O'Keefe
|
902cbe5915 |
feat(audiobooks): Batch A — foundation, m4b ingest, playback sync, web player
Phase 1 (foundation): 'audiobooks' library type seeded (schema + DB);
audio_metadata JSONB on media_items (the kobo_metadata precedent —
only audio paths read it); reading_progress.audio_offset_seconds for
the honest {chapter, offset} locator; extensions/mimes registered for
ALL breadth formats (m4b/m4a/mp3/flac — deliberate per the DOC/LIT
doctrine, m4b scans richly first); classifyFormatGroup 'audiobook'
case (gotcha 39 recomputes on rescan); Go mime table extended at boot
so media elements accept the streams; ffmpeg/ffprobe in the image
(8.1.2 — also the xHE-AAC identification tripwire).
Phase 2 (ingest): sidecar-first (Libation/ABS metadata.json —
narrators/ASIN/curated chapters), ffprobe fallback/filler (duration,
bitrate, codec/profile, embedded chapters). Fixed three real
parser brittlenesses the Libation sidecar exposed: series accepts
string OR array-of-objects, publishedYear accepts number OR string,
ffprobe chapter timings accept number OR string (flexFloat). Folder-
structure fallback now runs on the UPDATE path too (rescans were
clearing sidecar-absent series/author); audio facts re-derive on
every rescan (gotcha 39).
Phase 3 (playback + sync): GET /api/media-items/:id/chapters (the
format-abstraction list); streaming rides the existing cookie-authed
Range /uploads path; PUT progress carries audio_offset_seconds +
elapsed_seconds (honest wall-clock playing time) — the reading-
history capture prefers player-reported elapsed over the inter-push
delta, same 2h ceiling; reading_mode='audio' marks the row.
Phase 4 (web player): dedicated audio reader page (foliate not
involved) — cover, chapter title, whole-book scrubber, skip ± (chips
1/5/10/15/30s, default 10s), speed (0.75×–2×), chapter list with
jump-to, Media Session lockscreen/notification transport, periodic
playing-only progress pushes, position restore on open.
Verified live: ingest of the Libation sample (Gardens of the Moon —
title/author/series/cover/date from sidecar+folders, 34 chapters,
26h duration, aac/LC); chapters endpoint; the reader route serving
the player; playback with position restore (0:57 round-trip);
progress rows in audio mode; wall-clock session rows in
reading_history.
|
||
|
John O'Keefe
|
2117ec6181 |
feat(web): book reviews — markdown, sibling of ratings, full API + page surface
media_reviews table mirrors media_ratings' shape (unique per user+item, cascade deletes, updated_at for LWW) but standalone: a review without a rating must work. Body is RAW MARKDOWN — storage and every payload carry the source; rendering happens only at display sites (same doctrine as notes, same sanitization pipeline: markdown-it + DOMPurify). API: GET/PUT/DELETE /api/media-items/:id/review (PUT upserts; an empty body deletes — clearing via an editor is a delete, not an empty row). Book-detail page payload now carries the review beside the rating. Page: the review renders above the synopsis as a scrolling card (same max-height box as the summary), hydrated client-side through renderNoteHtml; 'Write review'/'Edit review' sits next to the rating stars and opens a raw-markdown editor dialog (save / delete / cancel). sqlc regenerated; schema applied to the dev DB. Live-verified: upsert, get, empty-delete, page markup, and the app round-trip against the dev server. |
||
|
John O'Keefe
|
431fec05ad |
feat(security): library visibility enforced at the source; per-library series; annotation ownership; admin user creation
The family-lens hardening: library_visibility was only enforced where file bytes flowed (download/reader/OPDS) — metadata surfaces leaked hidden libraries to any authenticated account, and registration was open by default with no admin-side user creation. - Series are grouped PER LIBRARY now (same name in two libraries = two entries: manga adaptations / LN splits no longer merge); /api/series aggregates only the requester's VISIBLE libraries and /api/series/books REQUIRES a visible library_id (breaking contract; web + app clients updated in the same change). - /api/media-items, item detail, and the reader content endpoints (chapters/panels/outline/THUMBNAILS — page images leaked unguarded next to a gated GetPage) all enforce the visible set; resolveLibrary no longer honors a hand-typed library_id outside the visible set. - Annotation/note/bookmark mutations (update/delete/tombstone, sync LWW paths incl. conflict resolution) carry ownership predicates: another account's bare row id is a 404, not a silent delete. - Admin user creation: POST /api/auth/users (AdminMiddleware) + a Create User card in the admin Users UI — accounts stay creatable with self-service registration OFF, which is now the schema default (fresh installs; the first-admin bootstrap exception is unchanged). - Tests: series contract re-pinned (per-library, visibility 403s), new ownership + admin-create integration suites, OPDS all=1 test now passes an explicit page size (shared-DB growth made the fresh books fall off page 1), device-cap helper switched to the admin endpoint. |
||
|
John O'Keefe
|
7ce73213ae |
feat(server): system collections become permanent — soft-delete + in-place restore + query_type-keyed provisioning
The old model hard-deleted system collections and re-created them at the tail, resurrecting stale dedups and breaking ordering. Redesign (user-directed B-final): - schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER — the whole schema is re-applied on every boot via executeSchema). - queries.sql: every collection list/read query filters deleted_at IS NULL; new GetSystemCollectionByQueryType :one (unfiltered, sees tombstones), TombstoneSystemCollection :exec, ClearCollectionItems :exec, RestoreSystemCollectionRow :one (in-place metadata reset + un-tombstone; collections have no updated_at column — ResetUpdatedAt was a 500 caught by tests). sqlc regenerated. - collection_service.DeleteCollection: system collections are tombstoned (preserving position/ownership); user collections keep the hard delete. No ErrNoRows to the client (delete is idempotent). - dashboard_service.RestoreSystemCollection: resolve by query_type; found → in-place restore (ClearCollectionItems + metadata reset, un-tombstoned); missing → provision fresh. "Want to Read"/etc. stay at their home positions forever. - handlers/collections.CreateCollection: name-taken is 409, not 500. - handlers/auth.CreateDefaultCollectionsForUser: idempotent by (user_id, query_type) — update-in-place when a row for that query type exists (heals renames/tombstones), insert when missing; this also fixes the duplicate-system-collections regression at source. |
||
|
John O'Keefe
|
25d44452c0 |
feat(dashboard): global (All Libraries) preferences scope
The dashboard's All Libraries view could render sections across every library but could not persist its own preferences: the PUT required a UUID library_id, and the prefs GET-without-library_id matched an arbitrary library's row through a loose NULL OR-clause. Storage: a dedicated user_dashboard_preferences_global table (one row per user, user_id PK → users CASCADE) instead of a NULL/sentinel library_id — chosen over both because (a) the column's FK to libraries rules out a sentinel UUID without dropping the constraint or minting a fake library row, and (b) intent becomes structural: only the explicit global path writes that table, so a buggy or outside-the-API write can never masquerade as global. System and user collections are user-scoped, so the order/hidden arrays are library-independent and valid in this scope. API: PUT /api/dashboard/preferences binds library_id as *string — JSON null/absent = global, empty string = 400 (a lost value must stay loud, not silently clobber the global scope), valid UUID = per-library (unchanged). GET without library_id now returns the global row instead of an arbitrary one; the per-library query's dead NULL OR-arm is removed. GET /sections keeps requiring library_id for per-library views and aggregates across libraries when absent (unchanged). Tests: global save/update-dedupe/read round-trip, empty-string 400, global-vs-per-library independence. Two suite failures are the handoff's documented pre-existing rot (duplicate system collections in the dev DB; a sections test expecting a 400 the handler never implemented) — reproduced and confirmed unrelated to this change. Live-verified on the dev stack: null save 200 x2 with a single row updated, empty string 400, global read carries the second save. |
||
|
John O'Keefe
|
b9f797789f |
feat: registration_enabled admin setting — hides signup surfaces, gates the API
Release / build-and-push (push) Successful in 3m9s
One boolean (Security > Public Registration) switches the whole public signup lifecycle: front page + login links, the logged-out sidebar's Create-an-account (via a package-level templates hook so shared page templates keep their signatures), GET /register -> 302 /login, and an early 403 on POST /api/auth/register. First-user exception preserved: zero admins keeps every route and link reachable for bootstrapping — same 'users exist' reasoning as the setup gate. Admin user creation is unaffected by design. Live E2E verified: flag off hides all surfaces and blocks the POST (403), flag on restores them (201); the admin UI renders the row automatically. |
||
|
John O'Keefe
|
1002cbca93 |
fix(schema): refresh library_type_name on library change; make trigger block re-runnable
schema.sql is re-executed in full on every application startup, so every statement in it must be idempotent. Two related problems introduced with the cross-library move feature, plus their fix: Problem 1: media_items.library_type_name was only populated by a BEFORE INSERT trigger. When the scanner repoints an existing row to a different library (cross-library move detection), the denormalized library_type_name went stale, mislabeling the item's type for validation and display. Fix: add a second trigger, set_library_type_name_on_library_change, that fires BEFORE UPDATE OF library_id and re-runs the same population function. Problem 2 (outage): the new trigger EXECUTE FUNCTIONs set_library_type_name(), but the pre-existing idempotency dance drops that function on every startup before recreating it. On the second and later boots, DROP FUNCTION failed with SQLSTATE 2BP01 (function still depended on by the trigger created by the previous boot), aborting the whole schema transaction and crash-looping the container. Fix: drop BOTH triggers before dropping the function, and create both after it. The ordering now survives any number of restarts on any database state. Adds TestSchemaInitializationIsIdempotent, which replays the production database.Initialize twice against the same database so this class of "second startup" regression fails in CI instead of in production. Note the test uses a dedicated pgxpool: Initialize holds the advisory lock on one connection while executing the schema on another, which deadlocks against the shared test pool's MaxConns=1. |
||
|
John O'Keefe
|
e6aceae0da |
feat(db): archive lifecycle schema and queries for missing media
Add the storage behind the archive-instead-of-delete lifecycle: - media_items.missing_scan_count (INT, default 0) and archived_at (TIMESTAMPTZ, partial index), both as idempotent ADD COLUMN IF NOT EXISTS backfills for existing installs. - MarkMediaItemMissing / ArchiveMediaItem / ClearMediaItemArchive plus PurgeExpiredArchivedMediaItems (retention cutoff) and PurgeAllArchivedMediaItems (manual bulk), with CountArchivedMediaItems for the admin UI. - Archived items are hidden from every user-facing listing: ListMediaItems, ListMediaItemsByLibrary, ListMediaItemsSorted, SearchMediaItems, SearchMediaItemsUnified, the next_books CTE, the library media counts, and the search autocomplete value lists. Detail lookups by id/path/sha are intentionally unfiltered, and a dedicated ListMediaItemsByLibraryIncludingArchived feeds the scanner so the lifecycle pass can see and restore archived rows. |
||
|
John O'Keefe
|
7f64b92b9d |
feat(metadata): per-field user overrides that survive library rescans
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
|
||
 john-okeefe
|
7b1c809ae3 |
feat(bookmarks): location identity, origin provenance, KOReader-style labels
- Drop UNIQUE(media_item_id,user_id,title): titles are display labels shared verbatim across clients; same-title bookmarks on different pages now coexist instead of 500ing (deleting over a tombstone no longer blocks future creates with that title) - Add origin_source column recording the creating client (android/web/ koreader), set once at insert, exposed in API responses - Web reader auto-title mirrors KOReader's 'in <chapter>' convention, falling back to 'Bookmark'; adds bookmark rename in the drawer |
||
|
john-okeefe
|
5c5593644d |
fix(media): gate file serving by library visibility; proper download URL
ServeFile previously authenticated only ("any logged-in user") and never
checked that the user can actually see the library owning the file, so
knowing a library UUID + path was enough to fetch content from hidden
libraries. Library visibility is the permission model - the library is
what grants access to its media.
- ServeFile now resolves two URL forms through one flow:
/uploads/library-{id}/{path} (covers, reader files)
/api/media-items/{id}/download (explicit book download, new)
The item form looks up the media item, derives its library and file
path, and adds a Content-Disposition attachment header.
- Both forms enforce GetUserVisibleLibraries for the authenticated
user, mirroring the OPDS download handler (403 when not visible).
- Deleted the dead MediaHandler.DownloadBook handler (never routed).
Also widen media_highlights.start_position/end_position from
VARCHAR(100) to TEXT: the API handlers validate up to 1000 characters
(full Readium locators, KOReader CRE xpointers) but the column rejected
anything longer at the database layer. Metadata-only change applied
idempotently at startup; existing rows are untouched.
Verified against the running server: download 200 + attachment headers
+ epub bytes, unauthenticated 401, user hidden from the library 403 on
both URL forms, visible user 200, covers unchanged, and a 334-char
locator JSON now round-trips through the highlights API.
|
||
|
john-okeefe
|
0c39e04e4a |
feat(db): hash_conflicts table and backfill/conflict queries
Content duplicates (same library + file_sha256 at different paths, e.g. the same book imported twice under two names on a preexisting database) cannot be auto-collapsed the way path duplicates were: keeping both copies may be intentional. Surface them for an explicit admin decision instead. Schema: - new hash_conflicts table keyed (library_id, file_sha256) with a status/resolution lifecycle: 'pending' until an admin resolves via 'keep_all' or 'kept:<uuid>' (which copy was kept after merging) - resolution is VARCHAR(50) - 'kept:<uuid>' is 41 chars; include a widening ALTER for databases created with the initial 30-char width - resolution/resolved_by/resolved_at record who decided what and when Queries: - ListMediaItemsMissingHash: items imported before hashing existed (file_sha256 IS NULL), ordered oldest-first for the backfill pass - FindHashConflictGroups: the content-duplicate group detection (GROUP BY library_id, file_sha256 HAVING COUNT(*) > 1) - ListMediaItemsBySHA256AndLibrary: full membership of one group - CreateHashConflict: upsert with DO NOTHING so already-tracked groups are untouched - critical behavior: a group an admin resolved as 'keep both' is never re-flagged by later sweeps - ListPendingHashConflicts: admin listing with library name and live item counts (items may have been deleted since flagging) - GetHashConflict / ResolveHashConflict: lifecycle - GetMediaItemUsageCounts: per-item progress/highlight/bookmark/note/ collection counts so the admin can make an informed keep choice - ReparentMediaItemChildren: sqlc binding for the existing reparent_media_item_children() migration function, used to merge a losing copy's child rows into the kept copy |
||
|
john-okeefe
|
9b171a0060 |
fix(scanner): prevent duplicate media item imports
A read-then-write race in processMediaFile allowed the same file to be imported twice: two concurrent scan jobs (startup scan, fsnotify dirty- directory scan, periodic backup poll, or a manual scan each run on separate worker goroutines with separate MediaScanner instances) could both SELECT 'not found' and both INSERT. There was no transaction, no row lock, no unique constraint on (library_id, file_path), and no ON CONFLICT clause, so nothing stopped the double insert. Observed in production as two identical 'Head First SQL' rows created in the same second (same sha256, size, path, library). Database enforcement: - schema.sql: add UNIQUE(library_id, file_path) constraint, guarded so re-runs don't error - schema.sql: add self-healing migration that runs on every startup - dedup_media_items_by_path() collapses existing path-duplicates and reparent_media_item_children() moves all child rows (progress, highlights, bookmarks, notes, collections, formats, aliases, kobo entitlements, etc.) onto a survivor before deleting losers, so the constraint applies cleanly on already-duplicated servers without losing reading history. Survivor picks the row with the most user data, ties broken by lowest id - CreateMediaItem: upsert via ON CONFLICT (library_id, file_path) DO UPDATE so concurrent inserts collapse to one row and return it - CreateMediaItemFormat: upsert via ON CONFLICT (media_item_id, format_type), closing the same race on format rows Application-level guards: - media_scanner processMediaFile: after computing the file hash, check GetMediaItemBySHA256AndLibrary (new query) and treat the file as existing when identical content is already in the library under a different path (content dedup, library-scoped so multi-library setups still work) Ops tooling: - scripts/dedup_media_items.sql: standalone idempotent maintenance script with a dry-run report (path + content duplicate groups, child row counts) and transactional cleanup, for servers that prefer to dedup manually before upgrading Verified against the live database: the duplicate pair was collapsed (reading_progress preserved on the survivor), schema.sql re-runs are a no-op, and the constraint is in place with 62 unique books remaining. |
||
|
john-okeefe
|
bc47450653 |
feat(db): typed tunable system settings + SettingsRegistry
Add a typed, cached registry over the system_settings table so that values which used to be hardcoded Go literals can be changed at runtime. Schema (database/schema/schema.sql): - Extend system_settings with setting_type, min_value, max_value, requires_restart, and category columns (all ADD COLUMN IF NOT EXISTS, nullable for backward compat with the original three rows). - Seed rows for every tunable: session duration, password rules, login lockout, auth/device rate limits, OPDS page size, tombstone TTL, conversion cache TTL, sync queue interval/batch, and worker pool size/cap. Seed values equal the previous hardcoded literals, so behavior is unchanged on upgrade. ON CONFLICT DO NOTHING preserves any admin-modified values. Queries (queries.sql): - Add UpsertSystemSetting (RETURNING *) so new keys without a seed row can still be written through the API. - Add GetSystemSettingFull + GetAllSystemSettingsFull returning the full typed row. - Refactor CleanupExpiredRefreshTokens to take the retention window as a parameter (make_interval(secs => $1)) instead of the INTERVAL '7 days' literal, so it can follow a configurable session duration. Registry (internal/database/settings_registry.go): - SettingsRegistry holds an in-memory cache of all known settings, populated by Load at startup and refreshed by Reload on writes. - Typed domain getters (SessionDuration, PasswordRules, DeviceRateLimits, TombstoneTTL, OpdsPageSize, ConversionCacheTTL, SyncQueueConfig, WorkerPoolConfig, LoginLockout, AuthRateLimit, ...) with compiled-in fallback defaults and min/max clamping, so a corrupt or missing row can never break the app. - SettingDefaults is the single source of truth for keys, types, bounds, and human descriptions; All() exposes metadata + current values for the admin UI/API. The registry lives in the database package (rather than its own internal/settings package) because a quirk in this custom go1.26.5 toolchain prevented the large handlers package from importing any newly-created package; every consumer already imports database. Tests: settings_registry_test.go covers default validity per type, int clamping at both bounds, garbage-value fallback, and unknown-key lookup. |
||
|
john-okeefe
|
4716790564 |
fix: OPDS base_url placeholder bug + setup gate requires base_url
Three bugs fixed: 1. Schema seeded base_url with fake placeholder 'bookhoard.example.com'. Removed seed; startup now seeds from BASE_URL env var only if DB row is empty (admin changes persist across restarts). One-time UPDATE clears the placeholder in existing installs. 2. config.GetBaseURL() had a broken type assertion (local SystemConfigRow vs database.SystemConfig) that always failed, returning . Admin panel showed env var fallback instead of actual DB value. Fixed with a function-type getter that properly wraps the DB query. 3. OPDS handler read base_url only from DB with no fallback. When DB had the placeholder, all feed links pointed to an unreachable domain, breaking KOReader search/download. Added deriveBaseURL() helper that falls back to the request Host/scheme when DB value is empty. Setup gate improvements: - isSetupComplete now requires both admin user AND non-empty base_url - Setup middleware no longer exempts all /api/ routes; only allows /api/auth/register, /api/auth/login, /api/system/config before setup is complete. All other API routes get 503. - Cache invalidated when base_url is saved via admin settings Dev workflow: - New bruno/NewDevDBSetup/SetBaseUrl.yml for dev DB setup - NewDB.sh runs SetBaseUrl between RegisterUser and CreateEbookLibrary |
||
|
john-okeefe
|
2a15effc3e |
feat(db): add annotation sync schema, queries, and tombstone support
Add migration columns to media_highlights, media_notes, and media_bookmarks for cross-device annotation sync: - dedup_key: SHA-1 of normalized selection text + bucketed position, used as the stable cross-device identity for annotations - last_modified_at / last_modified_source: edit clock for LWW resolution and cross-source conflict detection - deleted / deleted_at: sticky tombstone columns for delete-wins semantics with a 30-day TTL before rows are physically purged - note_text on highlights: stores attached notes from KOReader entries that have both selected text and a user note - Location columns on bookmarks (cfi_position, percentage_location, epubcfi_location, chapter_reference, paragraph_reference) - device_sync_data JSONB on all three tables: stores per-device native identifiers (e.g. KOReader pos0/datetime, Kobo bookmark_id) so each device can locate and manipulate its own copy of an annotation Add partial unique indexes on (user_id, media_item_id, dedup_key) where deleted = FALSE to enforce one active annotation per dedup key. Add tombstone purge indexes on (deleted, deleted_at) for efficient GC. New queries: - GetByDedupKey for all three tables (returns active or most-recent tombstone) - CreateFull / UpdateForSync for all three tables (populate sync columns) - TombstoneByDedupKey / TombstoneByID for all three tables - PurgeExpired* for all three tables (GC past TTL) - GetActiveAnnotationsForBook (filtered union of highlights + notes) - GetTombstonedAnnotationsForBook (union of all 3 deleted within TTL) - GetMediaBookmarks with deleted filter - GetMediaBookmark (singular) with deleted filter - Added deleted=FALSE filter to GetMediaHighlights, GetAnnotationsForBook - CreateAutoResolvedSyncConflict (INSERT with resolution_status='auto_resolved') |
||
|
john-okeefe
|
980aaee0d9 |
refactor(setup): derive setup-complete status from admin user count
Setup completion was previously tracked by a manually-flipped setup_complete row in system_settings, written via a JWT-protected PUT /api/setup/complete endpoint. This meant any admin user created outside the setup wizard (future CLI, seed scripts, direct DB inserts) would not flip the switch, leaving the app stuck redirecting to /setup.
The trigger is now derived from real data: setup is complete iff at least one admin user exists. This is self-correcting regardless of how users are created, and re-engages setup automatically if all admins are ever removed.
Changes:
- Add internal/setupstatus package with IsSetupComplete() (queries CountAdmins, 10s in-memory cache, fails open on DB error) and Invalidate() to clear the cache. Uses an AdminCounter interface to avoid importing the database package.
- Add CountAdmins sqlc query (SELECT COUNT(*) FROM users WHERE role = 'admin') and regenerate.
- Rewire router/setup.go isSetupComplete() to delegate to setupstatus; drop the old setup_complete setting read, cache vars, and the PUT /api/setup/complete route.
- Call setupstatus.Invalidate() in the auth handler after CreateUser, UpdateUserRole, and DeleteUser so the cache reflects admin-count changes immediately.
- Align first-user promotion in Register to key off !adminExists instead of len(users) == 0, so the two checks cannot diverge.
- Remove the now-dead SetSetupComplete/GetSetupStatus handlers.
- Drop the setup_complete seed row from schema.sql.
- Remove the apiPut('/setup/complete') call from the setup wizard finishSetup(); the admin account created in submitAdmin already marks setup complete server-side.
|
||
|
john-okeefe
|
cea8e64da2 |
feat(api): add setup_complete system setting and status handlers
Add 'setup_complete' boolean to the system_settings seed data (defaults
to false) so fresh databases start in the unconfigured state.
Add two new handlers to SystemSettingsHandler:
- SetSetupComplete: marks setup_complete=true in the database
- GetSetupStatus: reads the current setup_complete value, returns
{setup_complete: bool} JSON response, defaults to false if the
setting row is missing or unparseable
|
||
|
john-okeefe
|
c82f20c3f2 |
feat(db): add context_text column to reading_progress
Stores surrounding text (~100 chars) at the reader's current position. Used as fallback for CFI resolution when converting between epubcfi and CREngine XPointer formats. Updates: - schema.sql: add context_text TEXT column, update stored procedure - queries.sql: add context_text to GetUniversalProgress and UpdateUniversalProgress queries - Regenerate sqlc Go code (models.go, querier.go, queries.sql.go) |
||
|
john-okeefe
|
d0460885ff |
feat(db): add imported_at column to media_items for accurate "Recently Added" sorting
The created_at column stores file modification time (intentional for preserving original metadata), but this makes 'Recently Added' sorting unreliable for imported files. Add imported_at column that records the actual database insert timestamp. Changes: - Add imported_at TIMESTAMPTZ column to media_items (nullable) - Update GetRecentlyAddedItems to sort by imported_at DESC NULLS LAST first - Add ReassignLibraries and ReassignMediaItems queries for admin deletion handover - Add SyncLibraryTypeExtensions query for startup extension sync - Update all media_items SELECT queries to include imported_at column |
||
|
john-okeefe
|
607ce8ce3a | Merge branch 'main' of ssh://git.linuxhg.com:2222/Bookhoard/bookhoard | ||
|
john-okeefe
|
ff0517d038 |
fix(library): sync allowed extensions across service, schema, and tests
Add .epub and .pdf to comics type, add .pdf to manga type, and ensure avif/tiff/tif extensions are consistently included in manga across all layers. Update test fixtures to match the canonical extension lists. |
||
|
john-okeefe
|
caf50ade31 |
Add timezone support to database schema and queries
- Add timezone column (VARCHAR(50) DEFAULT 'UTC') to users table - Add default_timezone row to system_settings seed data - Add idx_users_timezone index for user timezone lookups - Add UpdateUserTimezone and GetSystemTimezone queries - Regenerate sqlc code (models, querier, queries.sql.go) - Reuse existing UpdateSystemSetting for system timezone updates instead of creating a redundant UpdateSystemTimezone query |
||
|
john-okeefe
|
b912a037ff |
fix(database): correct mismatched parentheses in detect_fixed_layout_epub
The string_to_array call in detect_fixed_layout_epub() had an extra closing parenthesis after the '<img' delimiter, causing a SQL syntax error that prevented the database container from initializing: IF array_length(string_to_array(opf_content, '<img')), 1) - 1 > 50 Fixed to: IF array_length(string_to_array(opf_content, '<img'), 1) - 1 > 50 |
||
|
john-okeefe
|
7ac86dafa9 |
feat(schema): Add processing_issues table for tracking media validation problems
- Add processing_issues table to track media items that cannot be properly processed in their assigned library - Include fields for issue type, description, severity, and resolution status - Add indexes for efficient querying by library and severity - Support tracking format mismatches and other processing problems - Unique constraint on media_item_id and issue_type to prevent duplicates |
||
|
john-okeefe
|
cf8c690474 |
Add library_type_name column to media_items with auto-population trigger
- Add library_type_name VARCHAR(50) column to media_items table in schema.sql - Create PostgreSQL trigger 'set_library_type_name_on_insert' that automatically populates library_type_name by joining libraries table with library_types on insert - Add library_type_name parameter to CreateMediaItem SQL query - Update Go models (models.go) to include LibraryTypeName field - Add UpdateMediaItemChapterMetadata query method to querier.go - Regenerate queries.sql.go with sqlc This allows media items to store their library type (e.g., 'Books', 'Comics', 'Manga') at the database level, enabling filtering and display without needing additional joins. |
||
|
john-okeefe
|
dd621956b9 |
database: change reading_speed columns from DECIMAL to REAL
- Changed words_per_minute, pages_per_minute, total_reading_minutes from DECIMAL(6,2) / DECIMAL(8,2) to REAL - REAL (pgtype.Float4) is sufficient for reading statistics and simplifies Go code - No precision loss for typical reading speed values (200-400 wpm, 0.5-3.0 pages/min) |
||
|
john-okeefe
|
0667cad8a9 |
feat: add reader infrastructure - Phase 0 database schema and queries
Implement Phase 0 prerequisites for reader functionality including database schema, SQL queries, and frontend dependencies. ## Database Schema (5 New Tables + 1 Column Addition) ### New Tables Added: 1. **panel_data** - Comic/manga panel detection results - Stores detected panel boundaries (x, y, width, height) - Supports grid, ML, and manual detection methods - JSONB storage for flexible panel structures 2. **reading_speed** - User reading speed statistics - Tracks pages per minute and total reading time - Per-user per-media-item tracking - Enables progress estimation and analytics 3. **dictionary_cache** - Offline dictionary word definitions - Caches external dictionary lookups - Reduces API calls and improves performance - Supports offline reading functionality 4. **reader_settings** - User reader preferences (per-user) - Stores typography, theme, and display settings - JSONB storage for flexible configuration - Per-user customization (fonts, margins, themes) 5. **media_bookmarks** - Enhanced bookmarks with chapter/CFI support - Unified bookmarking for ebooks, comics, manga, PDFs - Supports page_number, chapter_number, and epubcfi_position - Includes notes field for annotations - Unique constraint on (media_item_id, user_id, title) ### Column Addition: - **media_items.chapter_metadata** (JSONB) - Caches detected chapter structure - Stores TOC/chapter detection results - Prevents re-parsing files on every read - Populated by ReaderService.DetectChapters() ## Database Queries (12 New Queries) Added queries for all reader functionality: - Panel data: GetPanelData, UpsertPanelData - Reading speed: GetReadingSpeed, CreateReadingSpeed, UpdateReadingSpeed - Dictionary: GetDictionaryEntry, CreateDictionaryEntry, UpdateDictionaryAccessed - Settings: GetReaderSettings, UpsertReaderSettings - Bookmarks: GetMediaBookmarks, CreateMediaBookmark, DeleteMediaBookmark, UpdateMediaBookmark ## Frontend Dependencies Added to package.json: - jszip@^3.10.1 - EPUB/comic archive parsing (client-side) - pdfjs-dist@^3.11.174 - PDF rendering library (Mozilla PDF.js) ## Generated Code Ran `sqlc generate` to regenerate: - models.go - Go structs for new tables (55 lines added) - querier.go - Database query methods (14 lines added) - queries.sql.go - Compiled SQL queries (504 lines added) ## Implementation Status Phase 0 prerequisites now complete: ✅ Database schema (5 tables + 1 column) ✅ SQL queries (12 queries) ✅ Frontend dependencies (2 packages) ✅ Generated Go code (sqlc) ✅ Database recreated with new schema Ready for Phase 1: Infrastructure & Basic Reader implementation. Related to: Universal web reader for ebooks, comics, manga, PDFs |
||
|
john-okeefe
|
54559a6e91 |
schema: change community_rating to DOUBLE PRECISION for simpler type mapping
Change community_rating from DECIMAL(3,1) to DOUBLE PRECISION to: - Eliminate awkward pgtype.Numeric conversion in Go code - Enable direct pgtype.Float8 mapping from ComicInfo.xml float64 - Simplify code by matching natural types (XML float64 → PostgreSQL DOUBLE PRECISION → Go pgtype.Float8) - Remove need for string formatting and Scan() method calls The floating-point precision error (< 0.00001%) is negligible for 0-10 rating scale. This simplifies Phase 4 implementation significantly. Column comment updated to reflect DOUBLE PRECISION type. Relates to: Phase 1 database schema changes for comic metadata support |
||
|
john-okeefe
|
2ffd8e032b |
schema: add comic metadata and reading direction support
Add 14 new columns to media_items table for comprehensive comic and manga metadata support, including reading direction fields and universal metadata that applies to all media formats. New Columns: - Reading direction: manga_type (raw ComicInfo.xml field), reading_direction (computed) - Universal series: series_count, volume (apply to ebooks, audiobooks, comics) - Publisher info: imprint, age_rating (all formats) - Comic-specific: story_arc, is_black_and_white, alternate_info, scan_information, summary - Additional metadata: metadata_notes, community_rating, web_url Constraints: - manga_type CHECK: unknown, no, yes, yes_and_right_to_left - reading_direction CHECK: auto, ltr, rtl, vertical Indexes (8 new): - idx_media_items_reading_direction, idx_media_items_manga_type - idx_media_items_story_arc, idx_media_items_imprint - idx_media_items_age_rating, idx_media_items_series_count, idx_media_items_volume - idx_media_items_alternate_info_gin (GIN index for JSONB queries) Documentation: - Added COMMENT ON COLUMN for all 14 new fields - Distinctions between comic-specific and universal fields clearly documented This supports the ComicInfo.xml v2.0 standard with 29 fields and enables proper reading direction detection for manga, webtoons, and Western comics. Part of Phase 1: Database Schema Changes Implementation: IMPLEMENTATION_PLAN_MERGE_METADATA_READING_DIRECTION.md |