Reset to Scanned cleared the overrides row (successfully) but then set
the in-memory copy to nil before handing it to updateMediaItem. pgx
encodes a nil []string parameter as SQL NULL, so the follow-up UPDATE
wrote metadata_overrides = NULL into the column's NOT NULL constraint
and the whole rescan failed with:
failed to update media item: ERROR: null value in column
"metadata_overrides" of relation "media_items" violates not-null
constraint (SQLSTATE 23502)
Two changes:
- RescanMediaItem's reset path assigns []string{} instead of nil, with a
comment explaining the pgx nil-to-NULL encoding trap.
- updateMediaItem routes the override set through utils.MergeOverrides,
whose contract guarantees a non-nil slice, so no caller can write
NULL into that column again (verified against pgx v5.9.2 source: a
scanned '{}' round-trips as non-nil in both directions; the nil could
only come from our own assignment).
The plain Rescan path never hit this - only Reset did. Worse, the reset
is the remedy when a book's cover_image_path override pins an empty
cover, so the crash also blocked the way out of that state. After this
fix, a plain rescan on an already-reset book repopulates scanned
metadata and extracts the cover.
Adopt Calibre's reading conventions for the Dublin Core metadata that
parseOPFContent now pulls from the structured OPF parse:
- Titles: EPUB3 title-type selection (prefer 'main', join a distinct
subtitle with ': ' exactly as Calibre stores it). There is no separate
subtitle column by design - Calibre-sidecar books arrive pre-joined,
so a column would stay empty for most libraries and force every client
to reimplement concatenation.
- Genre: first dc:subject, mirroring the existing processGenresAndTags
behavior of the Calibre-sidecar path; the embedded path never
populated Genre before. Subjects stay one-element-one-tag - Library
of Congress headings legitimately contain commas ("Holmes, Sherlock
(Fictitious character) -- Fiction") and must not be split.
- Identifiers: urn:isbn:/urn:asin: prefixed values parse in addition to
opf:scheme attributes, and the scheme-less fallback now requires an
ISBN-shaped value (10/13 digits, optional separators/trailing X) so
URIs like the Gutenberg identifiers cannot masquerade as ISBNs -
observed live on 'A Study in Scarlet'.
- Series: EPUB3 belongs-to-collection with collection-type=series and
group-position refines, ahead of the classic calibre:series metas.
- Audiobookshelf metadata.json sidecars join their subtitle field into
the title the same way.
Tests cover title-type main+subtitle joining, belongs-to-collection
series with fractional group-position, urn:isbn extraction, genre/tag
parity, and comma preservation inside subject headings.
The cover lookup scraped the OPF with attribute-order-sensitive regexes.
Real books serialize attributes in any order - Grand Central's '3 Days to
Live' puts href before id on manifest items and content before name on
the cover meta - so all three regex paths missed and the book fell
through to filename guessing, extracting no cover at all. Attribute
order is meaningless in XML; the regexes were never safe.
Replace them with a structured parse (encoding/xml, namespace and
attribute-order agnostic; see the new media_scanner_opf.go) and follow
Calibre's read_raster_cover resolution order:
1. manifest item with properties=cover-image (non-(X)HTML media only)
2. <meta name=cover> resolved through the manifest, same media guard
3. first spine item that is itself a raster image (store manga)
4. NEW cover-page fallback: books declaring no raster cover at all -
the classic EPUB2/Adobe cover.xhtml wrapper - are mined for
<img src> / SVG <image xlink:href> references (Calibre renders the
page with Qt; extracting the referenced image covers the practical
cases without a rendering engine)
5. existing zip filename guessing stays as the last resort, and the old
regex chain survives as findCoverInOPFLegacy for OPFs too malformed
for a real XML parse.
Hrefs are now URL-decoded and posix-normalized against the OPF's own
path (path.Join semantics), so '../art/cover.jpg' from a nested cover
page and %20-encoded names resolve correctly.
Tests: attribute-order chaos modeled on the failing Patterson book,
SVG-wrapped cover pages via guide references, image-first spines, and
path resolution edge cases. Verified live against the real
'3 Days to Live' EPUB, which previously produced no cover.
Days an archived item is kept with its reading history before library
scans purge it for good (default 90). Set 0 to keep archived items until
purged manually from the library admin page.
Bulk escape hatch for archived rows (files missing from disk for 2+
scans) so a mass external deletion never has to wait out the retention
window or be clicked away row by row:
- POST /api/media-items/purge-archived (admin only) hard-deletes all
archived items and returns the purged count; reading history goes with
the rows, so the call is confirmed in the UI first.
- The library admin page shows an 'Archived items: N' card (only when
non-zero) with a Purge Archived Now button that calls the endpoint,
toasts the result, and reloads.
- Frontend admin JS exposes window.purgeArchivedItems following the
existing localStorage-bearer-token pattern.
Replace the silent hard-delete orphan cleanup (which logged only through
ScannerLogger file logs and whose failure paths left rows undetected)
with an archive lifecycle that preserves reading history:
- A file missing in one scan is marked (missing_scan_count = 1); missing
in a second consecutive scan archives it (archived_at, hidden from
browsing, progress/notes/highlights survive). Every branch logs to
stdout with an [ARCHIVE] prefix so skips are always visible.
- When a file reappears - same path, or identical content at a new path
via the SHA-256 dedup match - the archived state clears automatically
and the item returns with its history intact.
- Archived rows older than ARCHIVE_RETENTION_DAYS are hard-purged at
scan time (cascading deletes); 0 disables auto-purge for manual-only
management. Retention is read from the environment in NewMediaScanner.
Add the storage behind the archive-instead-of-delete lifecycle:
- media_items.missing_scan_count (INT, default 0) and archived_at
(TIMESTAMPTZ, partial index), both as idempotent ADD COLUMN IF NOT
EXISTS backfills for existing installs.
- MarkMediaItemMissing / ArchiveMediaItem / ClearMediaItemArchive plus
PurgeExpiredArchivedMediaItems (retention cutoff) and
PurgeAllArchivedMediaItems (manual bulk), with CountArchivedMediaItems
for the admin UI.
- Archived items are hidden from every user-facing listing:
ListMediaItems, ListMediaItemsByLibrary, ListMediaItemsSorted,
SearchMediaItems, SearchMediaItemsUnified, the next_books CTE, the
library media counts, and the search autocomplete value lists. Detail
lookups by id/path/sha are intentionally unfiltered, and a dedicated
ListMediaItemsByLibraryIncludingArchived feeds the scanner so the
lifecycle pass can see and restore archived rows.
Libraries managed by Audiobookshelf keep a metadata.json next to each
book (title, authors, series+sequence, genres/tags, publisher,
description, isbn/asin, language, published year/date) - and no
metadata.opf. The scanner silently ignored those files: deleting them
changed nothing, and their data never reached the database.
Parse them as a first-class sidecar in extractMetadata, priority
metadata.opf -> metadata.json -> embedded media. Only fields with a
matching media_items column are mapped; narrators, subtitle, explicit,
abridged, and chapters are deliberately skipped.
Cover handling is unchanged: the existing findSidecarCover priority
(cover.jpg / folder.jpg / {basename}.jpg) applies to the sidecar branch
exactly as it does for Calibre.
go-epub's ReadBook parses every spine chapter and fails the entire call
if any single chapter (or the TOC) is malformed, discarding already-parsed
OPF metadata. For books like Pragmatic's 'A Common-Sense Guide' the OPF
holds good title/author/publisher/ISBN metadata that rescans then wrote
as blanks - success toast, no (visible) change.
Refactor to parse the EPUB's own OPF document with the same Dublin Core
machinery used for Calibre sidecars: parseCalibreMetadataOPF is now a thin
file wrapper around a reusable parseOPFContent([]byte), and the OPF lookup
previously inline in extractEPUBCover is shared via findOPFPathInZip. Since
metadata never touches chapter bodies, chapter damage cannot blank it.
Also picked up along the way: dc:language mapping and scheme-less
dc:identifier values that normalize to a valid ISBN (EPUB3 style).
Tests cover a Pragmatic-style EPUB (dc namespace on <metadata>, no
identifier scheme, deliberately malformed chapter) that must still yield
full metadata, plus series/date/subject OPF parsing.
Complete the metadata editor modal to match the backend override work:
- File Path: read-only, monospace field showing the resolved on-disk
location (MediaDetail.FileLocation), next to Format and File Size, so
admins can see exactly which file backs the record without leaving the
editor. Renders empty for non-admins, who never receive the path.
- Reset to Scanned: new button beside Rescan. It confirms, then calls
POST /api/media-items/:id/rescan?reset_overrides=true to drop all
per-field user overrides and re-extract scanned defaults. Rescan alone
keeps customizations; Reset discards them.
Generate Cover has never worked: it fetched the book file using a URL
scraped from the cover preview <img> tag (so it downloaded either the
existing cover JPEG or, when no cover existed, the detail page HTML),
then handed it to foliate-js, which rejects both. Its fixed-layout path
also called view.renderer.renderPage(), a method that does not exist in
the pinned foliate fork. Every click ended in the same generic 'Cover
generation failed' toast.
The working alternative already exists server-side: the scanner's
PDF/EPUB cover extraction plus the per-book Rescan button, now that the
rasterizer renders the CropBox. Users who want a specific image can
upload one.
Delete web/src/cover-generator.ts, the modal buttons, and the dead
generateCover()/coverGenerating/fileUrl plumbing in book-detail.ts.
The book detail page exposed server internals and unusable controls to
everyday users: it now computes and renders the book's absolute on-disk
location, and gates all of it behind the admin role.
- The page handler resolves library folder + relative path (verified
with os.Stat; falls back to the relative path when the file is not
found on disk) into the new MediaDetail.FileLocation field - only for
admins, so the absolute path never leaves the server for regular
users. This also makes it possible to locate sparse entries whose
metadata rows are largely blank.
- The Metadata grid gains a full-width, monospace, click-selectable
Location row (admins only).
- The Edit button and the MetadataEditorModal markup render only for
admins. The modal drives admin-only endpoints (metadata PUT, rescan,
reset), so non-admins previously saw a button and a form that could
only ever fail with 403s.
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
parseTableNames() regex-scanned every line of schema.sql, comments
included, with 'CREATE TABLE (?:IF NOT EXISTS )?(?:\w+\.)?(\w+)'. A doc
comment containing that phrase in prose - e.g. 'declared in CREATE TABLE
above' - registered a phantom table ('above'), and startup verification
then failed with 'missing tables: above', crash-looping the app
container on every restart.
Skip lines whose trimmed form starts with '--' so comments can never
contribute table names, and add a regression test asserting every parsed
table maps back to a real CREATE TABLE statement.
pdftoppm defaults to rasterizing the MediaBox, while PDF viewers (pdf.js
in the reader, and every other viewer) display the CropBox. For PDFs
whose page 1 is the full print cover wrap (back cover + spine + front
cover in one landscape page) with a CropBox covering only the front
cover - e.g. No Starch's XeTeX-built 'Algorithmic Thinking' - the
fallback stored the entire spread as a squashed landscape cover, while
the reader correctly showed just the front cover.
Pass -cropbox so the rendered cover always matches what the reader
displays. Poppler falls back to the MediaBox when a PDF defines no
CropBox, so PDFs with identical boxes (the common case) render exactly
as before.
John O'Keefe