processing_issues only recorded cross-library move rejections; files
that failed to import vanished into an error counter and stdout. The
ledger now records every failure class and heals itself:
Schema (in place, pre-release boundary):
- media_item_id NULLABLE + file_path TEXT (deliberately unbounded —
an over-long path is itself a failure mode; a VARCHAR(500) ledger
column died on the exact 22001 it existed to report — caught by the
new test). Row-less failures key on (library, file_path); item-backed
on (item, type). Widened dedup = COALESCE-expression unique index;
the old table UNIQUE stays (NULLs pairwise-distinct for row-less).
- Conflict-target upsert: repeat failures UPDATE the same row and
REOPEN resolved ones — never duplicates.
Wirings:
- CreateMediaItem failure (single-file + multifile createAudioBookItem)
→ row-less import_failed (error).
- Extraction hard-fail with fallback import → item-backed
metadata_extraction_failed (warning).
- Change-path re-extraction failure (the ev/sidecar guard) → item-backed
metadata_extraction_failed; LogProcessingIssue keeps its signature, a
row-less helper joins it.
Auto-resolution:
- Successful import → ResolveOpenProcessingIssuesForPath (library,
file_path).
- Successful (re)extraction → ResolveOpenProcessingIssuesForItem
(single-file change path, multifile editorial apply).
- Unchanged-skip scans do NOT resolve: extraction never re-ran, so the
degraded metadata is still degraded — the page heals exactly when the
book does.
Family lens on ALL issues surfaces (list, stats, resolve, delete, SSR
page): explicit out-of-set library id → 403, never silent. This
tightens one pre-lens behavior — a nonexistent library id returned
200-empty, now 403 (pinned, changelog entry in docs/contract_pins.md).
List is LEFT-JOINed so row-less rows appear (media_item_id "",
title falls back to the path).
Tests: full lifecycle state machine through real scans (fail → appears,
repeat → no dups, success-at-path → resolved, LEFT-JOIN list shape);
family lens 403s with a fresh non-shared admin; all 7 pre-existing
issue suites green (two expectations widened to the lens, changelogged);
adjacent suites (CrossLibraryMove, Audiobook, Calibre,
ExtractorVersion, SchemaIdempotency) green; unit suites green.
Part 2 of the faceted plan, Phase 3 (steps 1-2). The periodic scan
issued one GetMediaItemByFilePath per scannable file (~3,094 queries
per scan in prod, nearly all answering 'unchanged, skip') plus one
GetLibraryByFolder per file for folder resolution.
- NEW media_scan_state table: the scanner's per-item fingerprint
cache. Deliberately NOT a media_items column — the library-scoped
list emits raw sqlc rows as JSON (7537063c), so scanner bookkeeping
must never leak into API payloads nor churn the model or pins.
Manifest JSONB shape v1 documented in the schema (n/s/m/h; ev
reserved for the Phase 4+ extractor-version mechanism).
- GetScanStateByLibrary: lean bulk projection (id, file_path,
file_size, archived_at, missing_scan_count, file_sha256, manifest).
archived/hash columns ride along so the skip path's archive-restore
and hash self-heal checks stay DB-free too (refines the plan's
4-column sketch; same goal, narrower invariant).
- ScanFolders: per-run reset, per-folder library cache, one bulk load
per distinct library. processMediaFile: map hit answers existence +
size + flags with zero DB reads; force-rescan and size-change
branches re-read the full row (the change path may be expensive);
write-through on create/size-change keeps later passes honest.
- Multifile folder/zip books keep their full-row lookup: it is one
query per BOOK ROOT (not per track) and needs audio_metadata for
the duration-probe check. They join the map in Phase 4, when the
manifest compare replaces their fingerprint check.
- nil map (watcher's scanDirectory path, bulk-load failure) falls
back to per-file lookups — behavior-preserving; the failure case
logged once per scan.
Test: TestScanStateBulkLoad (unchanged rescan leaves updated_at
stable; API-created rows are seen by the bulk load; size changes
still update in place). Container runner.
Phase 1 of the series/collections/metadata overhaul
(~/.opencode/plan/series-collections-metadata-overhaul.md).
Schema (in place, idempotent):
- media_items.series_number INTEGER -> DOUBLE PRECISION: real-world
ordinals are decimals (0.5, 1.5 novellas); INTEGER silently truncated
them (calibre 2.5 pinned as 2 in tests).
- + subtitle TEXT, explicit BOOLEAN (sidecar ingest; ABS explicit /
Audible isAdult).
- + authors/authors_search/publishers/publishers_search TEXT[] NOT NULL
(narrators A2a precedent). Arrays are write truth; author/publisher
varchars become trigger-regenerated display strings
(sync_author_publisher_display) so every varchar consumer (trgm
search, facets, API, OPDS/Kobo) keeps working unchanged.
- Row-level NULL normalization for the new arrays in
set_library_type_name (A2 lesson: Go nil -> explicit NULL -> 23502).
- Backfills ARRAY[author]/ARRAY[publisher] — deliberately no
comma-splitting ('Tolkien, J.R.R.' would corrupt; scanner kept only
authors[0] historically).
- collection_items.position INTEGER + (collection_id, position) index
for playlist ordering.
Go: sqlc regen; MediaMetadata/handler params gain Subtitle/Explicit/
Authors/Publishers (legacy singles still accepted, arrays win);
SeriesNumber int32 -> float64 end to end; metadata_overrides adapted
with legacy override keys ('author'/'publisher') preserved so existing
user overrides survive; bulk author edit writes the array.
Pins widened (docs/contract_pins.md): raw sqlc row key set + the six
new columns; changelog entry added.
Verified: unit suite green; integration sweep (bulk/smart collections/
scanner/stats/sync/queue/calibre/pins/audiobooks) green against the
shared dev DB after in-place migration. Pre-existing unrelated red:
TestOwnershipAndAdminTestSuite/TestAdminCreateUser timezone inheritance
fails identically at the pre-change commit (live shared state).
A2a (metadata round): narrators TEXT[] + narrators_search (tags
pattern: accumulator + GIN pairs) + abridged bool on media_items;
scanner fills from the Libation/ABS sidecar (narrators verbatim,
abridged flag) with audio-payload promotion; editorial fields ride
the real columns while audio_metadata stays scan-derived (duration/
codec/chapters). Detail/edit/save paths carry the fields; the web
edit dialog grows an Audio section (Narrator(s) comma-input, Abridged
toggle, scanned duration read-only); search weights + narrator filter
(word_similarity over narrators_search) + narrator autocomplete added.
A2c (per-media-type wording): copy helpers keyed on format_group —
Listen Now, Mark Complete/Not Complete, Listening Progress, Your
Listening Stats, Time Listening, Time listened (pages tile swapped for
audio), Position H:MM:SS / total H:MM:SS on detail + progress pages
(duration parsed from audio_metadata). Dashboard: system collections
swap to Continue Listening / Recently Listened at DISPLAY time when
the audiobooks library is selected (stored names unchanged; restores
keep working); week card = Listening Time / Sessions / Books Touched;
cover-corner badges identify audiobook (headphones) / comic / manga
cards on mixed grids — books stay unbadged; hover action icon follows
the media type.
A2b (player): the scrubber binds value to the playhead with an explicit
scrub guard (the handle previously froze at its initial max); header
row = reader-style back arrow + Chapter n/total launcher; chapters
open in a floating top-down drawer (current chapter highlighted, page
never shifts); Skip/Speed rows share one label-left layout; honest
wall-clock session accounting (playing-only deltas, paused flush).
A2d (bookmarks): note-bearing time bookmarks ride the existing
media_bookmarks substrate (notes column + sync-aware service already
there) — audio locator convention audio:<seconds>, chapter_number
filled, auto-title '<chapter> — H:MM:SS', capture popover with optional
note, drawer with jump/edit-note/delete. Zero server changes needed.
Verified live on the Libation sample: narrators backfilled by rescan;
PUT edit persisted + overrides detected (narrators, abridged); force
rescan preserved the user edit; reset-to-scanned-of-defaults restored
truth; narrator filter/autocomplete scoped by library; dashboard
listsening wording + badges render; bookmark save with note deduped to
one row.
Phase 1 (foundation): 'audiobooks' library type seeded (schema + DB);
audio_metadata JSONB on media_items (the kobo_metadata precedent —
only audio paths read it); reading_progress.audio_offset_seconds for
the honest {chapter, offset} locator; extensions/mimes registered for
ALL breadth formats (m4b/m4a/mp3/flac — deliberate per the DOC/LIT
doctrine, m4b scans richly first); classifyFormatGroup 'audiobook'
case (gotcha 39 recomputes on rescan); Go mime table extended at boot
so media elements accept the streams; ffmpeg/ffprobe in the image
(8.1.2 — also the xHE-AAC identification tripwire).
Phase 2 (ingest): sidecar-first (Libation/ABS metadata.json —
narrators/ASIN/curated chapters), ffprobe fallback/filler (duration,
bitrate, codec/profile, embedded chapters). Fixed three real
parser brittlenesses the Libation sidecar exposed: series accepts
string OR array-of-objects, publishedYear accepts number OR string,
ffprobe chapter timings accept number OR string (flexFloat). Folder-
structure fallback now runs on the UPDATE path too (rescans were
clearing sidecar-absent series/author); audio facts re-derive on
every rescan (gotcha 39).
Phase 3 (playback + sync): GET /api/media-items/:id/chapters (the
format-abstraction list); streaming rides the existing cookie-authed
Range /uploads path; PUT progress carries audio_offset_seconds +
elapsed_seconds (honest wall-clock playing time) — the reading-
history capture prefers player-reported elapsed over the inter-push
delta, same 2h ceiling; reading_mode='audio' marks the row.
Phase 4 (web player): dedicated audio reader page (foliate not
involved) — cover, chapter title, whole-book scrubber, skip ± (chips
1/5/10/15/30s, default 10s), speed (0.75×–2×), chapter list with
jump-to, Media Session lockscreen/notification transport, periodic
playing-only progress pushes, position restore on open.
Verified live: ingest of the Libation sample (Gardens of the Moon —
title/author/series/cover/date from sidecar+folders, 34 chapters,
26h duration, aac/LC); chapters endpoint; the reader route serving
the player; playback with position restore (0:57 round-trip);
progress rows in audio mode; wall-clock session rows in
reading_history.
media_reviews table mirrors media_ratings' shape (unique per user+item,
cascade deletes, updated_at for LWW) but standalone: a review without a
rating must work. Body is RAW MARKDOWN — storage and every payload carry
the source; rendering happens only at display sites (same doctrine as
notes, same sanitization pipeline: markdown-it + DOMPurify).
API: GET/PUT/DELETE /api/media-items/:id/review (PUT upserts; an empty
body deletes — clearing via an editor is a delete, not an empty row).
Book-detail page payload now carries the review beside the rating.
Page: the review renders above the synopsis as a scrolling card (same
max-height box as the summary), hydrated client-side through
renderNoteHtml; 'Write review'/'Edit review' sits next to the rating
stars and opens a raw-markdown editor dialog (save / delete / cancel).
sqlc regenerated; schema applied to the dev DB. Live-verified: upsert,
get, empty-delete, page markup, and the app round-trip against the dev
server.
The family-lens hardening: library_visibility was only enforced where
file bytes flowed (download/reader/OPDS) — metadata surfaces leaked
hidden libraries to any authenticated account, and registration was
open by default with no admin-side user creation.
- Series are grouped PER LIBRARY now (same name in two libraries =
two entries: manga adaptations / LN splits no longer merge);
/api/series aggregates only the requester's VISIBLE libraries and
/api/series/books REQUIRES a visible library_id (breaking contract;
web + app clients updated in the same change).
- /api/media-items, item detail, and the reader content endpoints
(chapters/panels/outline/THUMBNAILS — page images leaked unguarded
next to a gated GetPage) all enforce the visible set; resolveLibrary
no longer honors a hand-typed library_id outside the visible set.
- Annotation/note/bookmark mutations (update/delete/tombstone, sync
LWW paths incl. conflict resolution) carry ownership predicates:
another account's bare row id is a 404, not a silent delete.
- Admin user creation: POST /api/auth/users (AdminMiddleware) + a
Create User card in the admin Users UI — accounts stay creatable
with self-service registration OFF, which is now the schema default
(fresh installs; the first-admin bootstrap exception is unchanged).
- Tests: series contract re-pinned (per-library, visibility 403s),
new ownership + admin-create integration suites, OPDS all=1 test
now passes an explicit page size (shared-DB growth made the fresh
books fall off page 1), device-cap helper switched to the admin
endpoint.
The old model hard-deleted system collections and re-created them at
the tail, resurrecting stale dedups and breaking ordering. Redesign
(user-directed B-final):
- schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER —
the whole schema is re-applied on every boot via executeSchema).
- queries.sql: every collection list/read query filters
deleted_at IS NULL; new GetSystemCollectionByQueryType :one
(unfiltered, sees tombstones), TombstoneSystemCollection :exec,
ClearCollectionItems :exec, RestoreSystemCollectionRow :one
(in-place metadata reset + un-tombstone; collections have no
updated_at column — ResetUpdatedAt was a 500 caught by tests).
sqlc regenerated.
- collection_service.DeleteCollection: system collections are
tombstoned (preserving position/ownership); user collections keep
the hard delete. No ErrNoRows to the client (delete is idempotent).
- dashboard_service.RestoreSystemCollection: resolve by query_type;
found → in-place restore (ClearCollectionItems + metadata reset,
un-tombstoned); missing → provision fresh. "Want to Read"/etc.
stay at their home positions forever.
- handlers/collections.CreateCollection: name-taken is 409, not 500.
- handlers/auth.CreateDefaultCollectionsForUser: idempotent by
(user_id, query_type) — update-in-place when a row for that query
type exists (heals renames/tombstones), insert when missing; this
also fixes the duplicate-system-collections regression at source.
The dashboard's All Libraries view could render sections across every
library but could not persist its own preferences: the PUT required a
UUID library_id, and the prefs GET-without-library_id matched an
arbitrary library's row through a loose NULL OR-clause.
Storage: a dedicated user_dashboard_preferences_global table (one row
per user, user_id PK → users CASCADE) instead of a NULL/sentinel
library_id — chosen over both because (a) the column's FK to libraries
rules out a sentinel UUID without dropping the constraint or minting a
fake library row, and (b) intent becomes structural: only the explicit
global path writes that table, so a buggy or outside-the-API write can
never masquerade as global. System and user collections are
user-scoped, so the order/hidden arrays are library-independent and
valid in this scope.
API: PUT /api/dashboard/preferences binds library_id as *string —
JSON null/absent = global, empty string = 400 (a lost value must stay
loud, not silently clobber the global scope), valid UUID = per-library
(unchanged). GET without library_id now returns the global row instead
of an arbitrary one; the per-library query's dead NULL OR-arm is
removed. GET /sections keeps requiring library_id for per-library
views and aggregates across libraries when absent (unchanged).
Tests: global save/update-dedupe/read round-trip, empty-string 400,
global-vs-per-library independence. Two suite failures are the
handoff's documented pre-existing rot (duplicate system collections in
the dev DB; a sections test expecting a 400 the handler never
implemented) — reproduced and confirmed unrelated to this change.
Live-verified on the dev stack: null save 200 x2 with a single row
updated, empty string 400, global read carries the second save.
Add the storage behind the archive-instead-of-delete lifecycle:
- media_items.missing_scan_count (INT, default 0) and archived_at
(TIMESTAMPTZ, partial index), both as idempotent ADD COLUMN IF NOT
EXISTS backfills for existing installs.
- MarkMediaItemMissing / ArchiveMediaItem / ClearMediaItemArchive plus
PurgeExpiredArchivedMediaItems (retention cutoff) and
PurgeAllArchivedMediaItems (manual bulk), with CountArchivedMediaItems
for the admin UI.
- Archived items are hidden from every user-facing listing:
ListMediaItems, ListMediaItemsByLibrary, ListMediaItemsSorted,
SearchMediaItems, SearchMediaItemsUnified, the next_books CTE, the
library media counts, and the search autocomplete value lists. Detail
lookups by id/path/sha are intentionally unfiltered, and a dedicated
ListMediaItemsByLibraryIncludingArchived feeds the scanner so the
lifecycle pass can see and restore archived rows.
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
- Drop UNIQUE(media_item_id,user_id,title): titles are display labels
shared verbatim across clients; same-title bookmarks on different pages
now coexist instead of 500ing (deleting over a tombstone no longer
blocks future creates with that title)
- Add origin_source column recording the creating client (android/web/
koreader), set once at insert, exposed in API responses
- Web reader auto-title mirrors KOReader's 'in <chapter>' convention,
falling back to 'Bookmark'; adds bookmark rename in the drawer
Bookmark dedup is keyed on hash(title + position bucket), but the table
also enforces UNIQUE(media_item_id, user_id, title). When a client re-
saves the same bookmark title with a changed position form - e.g. the
Android app upgrading a percentage-only row to an EPUB CFI, or a web and
app bookmark landing on the same 'Bookmark at 44%' title - the dedup-key
lookup misses and the INSERT violates the title constraint, returning
HTTP 500 and failing the sync.
A title collision on the same (user, item) is by definition the same
bookmark slot, so take the LWW semantics all the way: ON CONFLICT DO
UPDATE replaces position/cfi_position/page/chapter/percentage, refreshes
dedup_key and timestamps, merges device_sync_data, and - matching
UpdateMediaBookmarkForSync - clears deleted/deleted_at so a re-create
resurrects a tombstoned title slot instead of leaving an invisible row
holding it.
Device sync flows are unaffected: KOReader/Kobo pushes that carry their
own dedup-key echoes never reach the INSERT, and same-key saves still go
through applyBookmarkLWW with its tombstone freshness checks.
Phase 0 of the reader redesign:
- Panels no longer render under the top/bottom bars: sidebars get
measured insets (same resize/safe-area mechanism as the viewport);
panel max-height now derives from the bounded sidebar instead of a
100vh guess; right-side border targets the actual sidebar.
- Bookmarks work end-to-end for the first time: REST CRUD under
/api/media-items/:id/bookmarks (create/delete route through
AnnotationService for dedup/LWW/tombstones), fix UpdateMediaBookmark
referencing nonexistent updated_at column, frontend posts to the
real API with per-format position (CFI vs page), live list with
jump + delete instead of SSR-only snapshot.
- Fix chapter matching in progress saves: boundaries were compared by
a nonexistent tocItem property, so chapter was never persisted.
- Remove dead UI: Navigator panel stub, empty dictionary popup shell,
unwired Chrome Behavior select; purge 160 stale build artifacts.
- Reader chrome now follows the user's app theme instead of hardcoded
theme-tokyo-night.
Content duplicates (same library + file_sha256 at different paths,
e.g. the same book imported twice under two names on a preexisting
database) cannot be auto-collapsed the way path duplicates were:
keeping both copies may be intentional. Surface them for an explicit
admin decision instead.
Schema:
- new hash_conflicts table keyed (library_id, file_sha256) with a
status/resolution lifecycle: 'pending' until an admin resolves via
'keep_all' or 'kept:<uuid>' (which copy was kept after merging)
- resolution is VARCHAR(50) - 'kept:<uuid>' is 41 chars; include a
widening ALTER for databases created with the initial 30-char width
- resolution/resolved_by/resolved_at record who decided what and when
Queries:
- ListMediaItemsMissingHash: items imported before hashing existed
(file_sha256 IS NULL), ordered oldest-first for the backfill pass
- FindHashConflictGroups: the content-duplicate group detection
(GROUP BY library_id, file_sha256 HAVING COUNT(*) > 1)
- ListMediaItemsBySHA256AndLibrary: full membership of one group
- CreateHashConflict: upsert with DO NOTHING so already-tracked groups
are untouched - critical behavior: a group an admin resolved as
'keep both' is never re-flagged by later sweeps
- ListPendingHashConflicts: admin listing with library name and live
item counts (items may have been deleted since flagging)
- GetHashConflict / ResolveHashConflict: lifecycle
- GetMediaItemUsageCounts: per-item progress/highlight/bookmark/note/
collection counts so the admin can make an informed keep choice
- ReparentMediaItemChildren: sqlc binding for the existing
reparent_media_item_children() migration function, used to merge a
losing copy's child rows into the kept copy
A read-then-write race in processMediaFile allowed the same file to be
imported twice: two concurrent scan jobs (startup scan, fsnotify dirty-
directory scan, periodic backup poll, or a manual scan each run on
separate worker goroutines with separate MediaScanner instances) could
both SELECT 'not found' and both INSERT. There was no transaction, no
row lock, no unique constraint on (library_id, file_path), and no
ON CONFLICT clause, so nothing stopped the double insert. Observed in
production as two identical 'Head First SQL' rows created in the same
second (same sha256, size, path, library).
Database enforcement:
- schema.sql: add UNIQUE(library_id, file_path) constraint, guarded so
re-runs don't error
- schema.sql: add self-healing migration that runs on every startup -
dedup_media_items_by_path() collapses existing path-duplicates and
reparent_media_item_children() moves all child rows (progress,
highlights, bookmarks, notes, collections, formats, aliases, kobo
entitlements, etc.) onto a survivor before deleting losers, so the
constraint applies cleanly on already-duplicated servers without
losing reading history. Survivor picks the row with the most user
data, ties broken by lowest id
- CreateMediaItem: upsert via ON CONFLICT (library_id, file_path) DO
UPDATE so concurrent inserts collapse to one row and return it
- CreateMediaItemFormat: upsert via ON CONFLICT (media_item_id,
format_type), closing the same race on format rows
Application-level guards:
- media_scanner processMediaFile: after computing the file hash, check
GetMediaItemBySHA256AndLibrary (new query) and treat the file as
existing when identical content is already in the library under a
different path (content dedup, library-scoped so multi-library
setups still work)
Ops tooling:
- scripts/dedup_media_items.sql: standalone idempotent maintenance
script with a dry-run report (path + content duplicate groups, child
row counts) and transactional cleanup, for servers that prefer to
dedup manually before upgrading
Verified against the live database: the duplicate pair was collapsed
(reading_progress preserved on the survivor), schema.sql re-runs are a
no-op, and the constraint is in place with 62 unique books remaining.
Add a typed, cached registry over the system_settings table so that
values which used to be hardcoded Go literals can be changed at runtime.
Schema (database/schema/schema.sql):
- Extend system_settings with setting_type, min_value, max_value,
requires_restart, and category columns (all ADD COLUMN IF NOT EXISTS,
nullable for backward compat with the original three rows).
- Seed rows for every tunable: session duration, password rules,
login lockout, auth/device rate limits, OPDS page size, tombstone TTL,
conversion cache TTL, sync queue interval/batch, and worker pool
size/cap. Seed values equal the previous hardcoded literals, so
behavior is unchanged on upgrade. ON CONFLICT DO NOTHING preserves
any admin-modified values.
Queries (queries.sql):
- Add UpsertSystemSetting (RETURNING *) so new keys without a seed row
can still be written through the API.
- Add GetSystemSettingFull + GetAllSystemSettingsFull returning the
full typed row.
- Refactor CleanupExpiredRefreshTokens to take the retention window as
a parameter (make_interval(secs => $1)) instead of the INTERVAL '7
days' literal, so it can follow a configurable session duration.
Registry (internal/database/settings_registry.go):
- SettingsRegistry holds an in-memory cache of all known settings,
populated by Load at startup and refreshed by Reload on writes.
- Typed domain getters (SessionDuration, PasswordRules, DeviceRateLimits,
TombstoneTTL, OpdsPageSize, ConversionCacheTTL, SyncQueueConfig,
WorkerPoolConfig, LoginLockout, AuthRateLimit, ...) with compiled-in
fallback defaults and min/max clamping, so a corrupt or missing row
can never break the app.
- SettingDefaults is the single source of truth for keys, types, bounds,
and human descriptions; All() exposes metadata + current values for
the admin UI/API.
The registry lives in the database package (rather than its own
internal/settings package) because a quirk in this custom go1.26.5
toolchain prevented the large handlers package from importing any
newly-created package; every consumer already imports database.
Tests: settings_registry_test.go covers default validity per type,
int clamping at both bounds, garbage-value fallback, and unknown-key
lookup.
Add migration columns to media_highlights, media_notes, and media_bookmarks
for cross-device annotation sync:
- dedup_key: SHA-1 of normalized selection text + bucketed position, used
as the stable cross-device identity for annotations
- last_modified_at / last_modified_source: edit clock for LWW resolution
and cross-source conflict detection
- deleted / deleted_at: sticky tombstone columns for delete-wins semantics
with a 30-day TTL before rows are physically purged
- note_text on highlights: stores attached notes from KOReader entries that
have both selected text and a user note
- Location columns on bookmarks (cfi_position, percentage_location,
epubcfi_location, chapter_reference, paragraph_reference)
- device_sync_data JSONB on all three tables: stores per-device native
identifiers (e.g. KOReader pos0/datetime, Kobo bookmark_id) so each
device can locate and manipulate its own copy of an annotation
Add partial unique indexes on (user_id, media_item_id, dedup_key) where
deleted = FALSE to enforce one active annotation per dedup key.
Add tombstone purge indexes on (deleted, deleted_at) for efficient GC.
New queries:
- GetByDedupKey for all three tables (returns active or most-recent tombstone)
- CreateFull / UpdateForSync for all three tables (populate sync columns)
- TombstoneByDedupKey / TombstoneByID for all three tables
- PurgeExpired* for all three tables (GC past TTL)
- GetActiveAnnotationsForBook (filtered union of highlights + notes)
- GetTombstonedAnnotationsForBook (union of all 3 deleted within TTL)
- GetMediaBookmarks with deleted filter
- GetMediaBookmark (singular) with deleted filter
- Added deleted=FALSE filter to GetMediaHighlights, GetAnnotationsForBook
- CreateAutoResolvedSyncConflict (INSERT with resolution_status='auto_resolved')
Stores surrounding text (~100 chars) at the reader's current position.
Used as fallback for CFI resolution when converting between epubcfi
and CREngine XPointer formats.
Updates:
- schema.sql: add context_text TEXT column, update stored procedure
- queries.sql: add context_text to GetUniversalProgress and
UpdateUniversalProgress queries
- Regenerate sqlc Go code (models.go, querier.go, queries.sql.go)
Regenerated database code after sqlc version upgrade from v1.30.0 to
v1.31.1. No functional changes — only the version header in generated
files was updated.
Files: db.go, models.go, querier.go, queries.sql.go
The created_at column stores file modification time (intentional for preserving
original metadata), but this makes 'Recently Added' sorting unreliable for
imported files. Add imported_at column that records the actual database insert
timestamp.
Changes:
- Add imported_at TIMESTAMPTZ column to media_items (nullable)
- Update GetRecentlyAddedItems to sort by imported_at DESC NULLS LAST first
- Add ReassignLibraries and ReassignMediaItems queries for admin deletion handover
- Add SyncLibraryTypeExtensions query for startup extension sync
- Update all media_items SELECT queries to include imported_at column
- Add timezone column (VARCHAR(50) DEFAULT 'UTC') to users table
- Add default_timezone row to system_settings seed data
- Add idx_users_timezone index for user timezone lookups
- Add UpdateUserTimezone and GetSystemTimezone queries
- Regenerate sqlc code (models, querier, queries.sql.go)
- Reuse existing UpdateSystemSetting for system timezone updates
instead of creating a redundant UpdateSystemTimezone query
- Add ProcessingIssues model struct
- Update Querier interface with processing issues methods
- Add generated query implementations for CreateProcessingIssue, ListProcessingIssuesByLibrary, GetProcessingIssueStats, ResolveProcessingIssue, DeleteProcessingIssue
- Add GetLibraryWithType query for fetching library with type information
- Add library_type_name VARCHAR(50) column to media_items table in schema.sql
- Create PostgreSQL trigger 'set_library_type_name_on_insert' that automatically
populates library_type_name by joining libraries table with library_types on insert
- Add library_type_name parameter to CreateMediaItem SQL query
- Update Go models (models.go) to include LibraryTypeName field
- Add UpdateMediaItemChapterMetadata query method to querier.go
- Regenerate queries.sql.go with sqlc
This allows media items to store their library type (e.g., 'Books', 'Comics', 'Manga')
at the database level, enabling filtering and display without needing additional joins.
Add database schema and SQL queries for generic saved filters system
that allows users to save custom filter presets for any resource type.
Database Schema:
- Add saved_filters table with user_id, name, resource_type, filters (JSONB)
- Create composite index on (user_id, resource_type) for efficient lookups
- Create index on (user_id, name) for future name search feature
- Add update_updated_at_column() trigger to auto-update timestamps
- Make trigger creation idempotent with DROP TRIGGER IF EXISTS
SQL Queries (5 new queries):
- GetSavedFilters: List all filters for user + resource type
- GetSavedFilterByID: Retrieve single filter by ID
- CreateSavedFilter: Create new saved filter
- UpdateSavedFilter: Update filter name/criteria
- DeleteSavedFilter: Remove saved filter
Design Decisions:
- Generic resource_type field supports any resource (media-items, collections, devices)
- JSONB filters field allows flexible schema without migrations
- User-scoped via JWT (user_id foreign key with CASCADE delete)
- Automatic updated_at timestamp via database trigger
Generated Code:
- database.SavedFilters model (10 fields including JSONB filters)
- All 5 CRUD query functions with proper parameter types
- pgtype.UUID wrappers for UUID parameters
Part of: Saved Filters Implementation (Phase 1: Database)
Related: #saved-filters-feature
Add support for Carousel-style dashboard with unified collections architecture:
Database Schema Changes:
- Add user_dashboard_preferences table:
- hidden_collections: TEXT[] for managing section visibility
- collection_order: TEXT[] for custom ordering
- items_per_section: INT for limiting items per section
- Update collections table:
- user_id: Make nullable to support system-owned collections (NULL = system)
- show_on_dashboard: BOOLEAN for controlling visibility
- query_type: TEXT for different query types (continue-reading, recently-added, etc.)
- priority: INT for display order (lower = higher priority)
- is_system_collection: BOOLEAN for flagging system defaults
- Update collection_items table:
- Add excluded BOOLEAN for user overrides of auto-assigned items
Indexes:
- idx_collections_dashboard: (user_id, show_on_dashboard, priority) WHERE show_on_dashboard = true
- idx_dashboard_prefs_user_library: (user_id, library_id)
- idx_collection_items_excluded: (collection_id, excluded) WHERE excluded = true
System Collections (pre-seeded defaults):
- continue-reading: Books with 0 < progress < 1
- recently-added: Newly added items to library
- recently-read: Books with progress >= 1
- not-started: Books with progress = 0 or no record
This implements Phase 1 of the Carousel-style dashboard redesign plan.
Auto-generated changes from running 'sqlc generate' after query updates:
- models.go: updated with SystemSettings struct, removed scan fields from Users
- querier.go: updated interface with new system settings methods
- queries.sql.go: regenerated with new query methods
Generated via: cd internal/database && sqlc generate
Schema changes:
- Add tags_search TEXT[] column for case-insensitive, punctuation-free search
- Add contributors_search TEXT[] column for case-insensitive, punctuation-free search
- Create GIN indexes for fast array searches on both search fields
Query updates:
- CreateMediaItem: Include tags_search and contributors_search parameters
- UpdateMediaItem: Include tags_search and contributors_search parameters
- SearchMediaItems: Search against tags_search instead of tags
- SearchMediaItems: Search against contributors_search instead of contributors
- SearchMediaItemsFuzzy: Use tags_search and contributors_search for fuzzy matching
- Update ranking and priority logic to use search fields
Benefits:
- Case-insensitive search: "acme corp" finds "ACME CORP."
- Punctuation-agnostic search: "oreilly" finds "O'Reilly Media"
- Better UX: Users don't need to match exact casing or punctuation
- Improved performance with dedicated GIN indexes
Relates to Tags & Contributors Migration Phases 5 & 8
Convert tags and contributors columns from comma-separated strings to PostgreSQL
TEXT[] arrays for better data normalization and query performance.
Database Changes:
- schema.sql: Change tags/contributors from TEXT to TEXT[]
- schema.sql: Add GIN indexes for fast array searches
- queries.sql: Update search queries to use ANY() operator
- queries.sql: Update fuzzy search with unnest() for arrays
Generated Code (sqlc):
- models.go: Auto-generated with []string types for tags/contributors
- queries.sql.go: Auto-generated with proper array handling
Handler Changes:
- media.go: Update request structs to use []string for tags/contributors
- media.go: Remove pgtype.Text wrapping, use direct array assignment
- media.go: Add tag normalization in CreateMediaItemHandler
- collections.go: Update tags evaluation to join arrays for comparison
- collections.go: Add strings import for Join() function
Service Changes:
- ebook_scanner.go: Update EbookMetadata struct to use []string
- ebook_scanner.go: Remove string Join(), assign arrays directly
- collection_service.go: Update tags rule evaluation to join arrays
- collection_service.go: Add strings import
New Utilities:
- internal/utils/tags.go: Create NormalizeTags(), JoinTags(), SplitTags()
- Normalizes tags by trimming, lowercasing, removing duplicates/empties
API Documentation:
- bruno/media-items/Create Media Item.bru: Update examples to use arrays
- bruno/media-items/Update Media Item.bru: Update examples to use arrays
- Update docs: tags/contributors now array of string
Breaking Change:
- JSON format changes from "tags": "tag1,tag2" to "tags": ["tag1", "tag2"]
- Tests already use array format (no changes needed)
Benefits:
- GIN indexes enable faster array searches
- Normalization prevents data quality issues (case, duplicates)
- Array operations use PostgreSQL native operators (ANY, &&, unnest)
- Better separation of concerns (no string parsing in application)
Complete the rename by updating:
- DeviceCatalogs struct field: BookmannUuid → BookhoardUuid (models.go)
- Generated queries: Update all references (queries.sql.go)
- Local variables: bookmannUUID → bookhoardUUID (kobo.go)
- Struct field access: catalog.BookmannUuid → catalog.BookhoardUuid
All "bookmann" and "BOOKMANN" references are now eliminated from the codebase.
Part of project rename to Bookhoard.
- Regenerate queries.sql.go with refresh token queries
- Update models.go with RefreshTokens type
- Update querier.go with new query methods
- Update db.go with generated code
- Add MediaNotes and MediaHighlights model structs with pgx v5 types
- Add EbookNotes and EbookHighlights for backward compatibility
- Add complete CRUD SQL queries for notes and highlights
- Add database connection pool function using pgx v5
- Generate sqlc code for new annotation functionality
- Add library_types table with ebooks, comics, manga types
- Add libraries table for multiple library support
- Add library_folders table for multi-folder libraries
- Add library_visibility table for user access control
- Add media_items table replacing ebooks for broader media support
- Create backward compatibility views for existing API
- Implement library service with type validation and file extension handling
- Support modular extension for future media types
Manga type includes cbz/cbr archives as requested
- Add role column to users table with admin/user constraint
- Add added_by_admin_id column to ebooks table for tracking
- Add constraint to ensure only admins can manage folders
- Update all SQL queries to include role field
- Regenerate database models with new schema
- Add admin dashboard at /admin route consolidating all user settings
- Move user preferences (username, email, password, theme) from separate page
- Add ebook library preferences section with folder management
- Add scan settings (frequency and auto-scan toggle) with database persistence
- Create database migration for scan_frequency_minutes and auto_scan_enabled columns
- Add API endpoints for scan settings management:
- PUT /api/library/scan-settings - Update scan preferences
- GET /api/library/scan-settings - Get current scan settings
- Update dashboard navigation to link to admin dashboard
- Remove old preferences.html template (functionality moved to admin)
- Create Bruno API testing files for library endpoints
- Add real-time folder loading and management in admin interface
- Implement scan settings persistence and retrieval from database
- Rename project from 'bookmann' to 'shelf'
- Move all backend/ contents to root level (flatten structure)
- Update Go module name from 'bookmann' to 'shelf'
- Update all import paths to use new 'shelf' module
- Update Dockerfile to work without backend/ subdirectory
- Update docker-compose.yml to use new structure and rename containers
- Update .gitignore for new file paths
- Update README.md with new project name and structure
- Regenerate database code with new module imports
John O'Keefe
john-okeefe