Fixed-layout EPUBs lean on the reading_direction column (the web reader
forces book.dir = rtl from it when the file didn't set direction
itself), but the scanner never populated it for EPUBs - only ComicInfo
fed it. Meanwhile real Japanese EPUBs declare page-progression-direction
on the OPF spine, which foliate reads client-side but nothing stored.
Read the spine attribute in the structured OPF parser and map it into
ReadingDirection in parseOPFContent (EPUB2/3, case-insensitive, plus
'right-to-left'/'left-to-right' spellings); undeclared stays empty
rather than forcing ltr, preserving the editor's Auto default. Sidecar
OPFs are metadata-only documents without spines, so the Calibre path is
a no-op. The merge gap-fill copies an embedded-only direction into a
blank sidecar field, and hand-set values keep winning through the
existing OverrideReadingDirection protection.
Tests: declared rtl/RTL/ltr, undeclared and unknown values staying
empty, plus sidecar-wins vs embedded-fills merge cases. Existing manga
EPUBs declaring rtl (verified live in-library) pick the value up on
their next scan, feeding the API and reader config mobile clients
consume.
Same gap-fill rule as the EPUB path, against the PDF's embedded Info
dictionary via a readPDFInfoDict helper reusing extractPDFMetadata's
field conventions (creator falls back to author, subject maps to
description, producer to publisher, keywords to tags, plus page count).
Sidecar values always win; unopenable PDFs skip silently.
TestMergeMetadataPDFGapFill uses an Info-bearing hand-built PDF fixture
(shared with the sidecar-cover test) and asserts the sidecar title is
kept while author/description/publisher/tags/page count fill in.
A sparse metadata.opf/metadata.json (title only, no description) left
books thin even when the file itself carried rich data: the embedded
extractors only ran when no sidecar existed at all. Now mergeMetadata
fills blanks from the book's own OPF - title, author, description,
publisher, language, ISBN, ASIN, series/number, publish date, tags,
contributors - while sidecar values always win and unparseable files
skip silently. Also covers .kepub, which the merge previously ignored
while the extractor already supported it.
Adds TestMergeMetadataEPUBGapFill asserting both directions: sidecar
title/author survive, embedded description/publisher/language fill in.
Admins need to see what the archive lifecycle is holding: a dedicated
/admin/archived page listing every hidden item (archived or still in
the missing grace window) with library, file path, status, and - when
retention is enabled - the exact date it will be permanently deleted
(archived_at + ARCHIVE_RETENTION_DAYS).
Per row: Restore (POST /api/media-items/:id/unarchive, clears the
archive state so it reappears; if the file is still gone the next scan
hides it again) and Delete (existing DELETE endpoint for single-row
purge with its reading history). Purge All Archived reuses the existing
bulk button. The library admin banner links to the page and keeps its
purge button; row actions use data attributes with delegated listeners
in admin.ts since this templ version has no JSFunctionCall helper.
Verified live: page 200 with purge dates shown, unarchive returned 204
and reset the row, per-item delete and bulk purge ({purged:1}) both
removed their rows with no leftovers.
Two visibility fixes so the UI reflects the shelf's real state on the
next scan instead of only after the archive gate:
- Missing items disappear at once: the user-facing filters already hid
archived rows; the same listings now also require missing_scan_count =
0. A deleted or moved-then-not-yet-repointed file vanishes from the
UI immediately, while purge timing stays gated on archived_at plus the
retention window - grace protects data, not visibility. Restored
automatically when the file returns.
- Steam Deck SD-card model for unmounted storage: resolveLibrary stats
each library's folder roots and flags libraries with no live folder
as Offline (LibraryData gains the field, computed at request time so
mounts/unmounts react instantly). The bookshelf shows an empty shelf
plus a 'storage is not connected' notice for an offline selected
library, and both the shared LibrarySwitcher and the bookshelf's
inline select label offline libraries with their true holding counts.
Nothing is marked or purged while offline.
- TotalMediaCount skips offline libraries, so the 'All Books/Libraries'
totals match what is actually visible.
Verified live: renaming uploads/Manga away produced the notice, an
empty shelf, and the offline dropdown label with a corrected total;
renaming it back restored all 38 cards with zero dirty rows.
When the SHA-256 dedup found identical content already in the library,
the scan skipped the file as a duplicate - and after files moved
between folders the old row kept its stale path, cycled missing ->
archived, and the new path never took. The archive feature turned the
old destructive move behavior into a stuck move instead.
Now the dedup branch stats the old location: if it is gone, the book
was MOVED, so the row is repointed (file_path, file_size) with archive
state cleared and reading history intact. 'Skip as duplicate' only
applies when the old path still exists (a true copy). Verified live: a
moved EPUB kept its single row, followed the file, and never entered
the missing/archive cycle.
Storage behind the scan lifecycle fixes:
- MoveMediaItemFilePath: repoints a row (path, size) and clears archive
state when identical content reappears at a new location.
- ListHiddenMediaItems: archived OR missing rows for the admin
archived-items page; ListMediaItemsByLibraryIncludingArchived (prior
commit) already fed the scanner.
- All user-facing listings now hide missing items immediately, not just
archived ones: ListMediaItems, ListMediaItemsByLibrary,
ListMediaItemsSorted, SearchMediaItems, SearchMediaItemsUnified, the
next_books CTE, library media counts, and the five search autocomplete
value queries gained AND mi.missing_scan_count = 0 next to the
archived_at filter. Purge timing is unchanged (archived_at +
ARCHIVE_RETENTION_DAYS), so the grace period keeps protecting data
while the UI reflects removals on the first scan.
- Detail lookups by id/path/sha stay unfiltered on purpose.
A full rescan wiped cover_image_path for every PDF/EPUB living in a
metadata.json (or metadata.opf-less) folder with no cover.jpg next to
the book: the sidecar branches returned early after findSidecarCover
missed, and mergeMetadata has no PDF/EPUB cover logic of its own. Four
books lost their thumbnails while their {file}.cover.jpg files still sat
on disk - most visibly the Audiobookshelf-managed No Starch titles.
Both sidecar branches now fall through to an embedded-cover fallback
(PDF via extractPDFCover, EPUB/KEPUB via extractEPUBCover) whenever no
sidecar cover file exists. Comics are untouched: mergeMetadata already
extracts their covers from the archive.
Adds TestSidecarCoverFallback with a hand-built one-page PDF carrying a
JPEG XObject plus a metadata.json sidecar, asserting the sidecar title
wins while the cover still comes from the file. Verified live: rescans
restored all four dereferenced covers with no leftover rows.
Reset to Scanned cleared the overrides row (successfully) but then set
the in-memory copy to nil before handing it to updateMediaItem. pgx
encodes a nil []string parameter as SQL NULL, so the follow-up UPDATE
wrote metadata_overrides = NULL into the column's NOT NULL constraint
and the whole rescan failed with:
failed to update media item: ERROR: null value in column
"metadata_overrides" of relation "media_items" violates not-null
constraint (SQLSTATE 23502)
Two changes:
- RescanMediaItem's reset path assigns []string{} instead of nil, with a
comment explaining the pgx nil-to-NULL encoding trap.
- updateMediaItem routes the override set through utils.MergeOverrides,
whose contract guarantees a non-nil slice, so no caller can write
NULL into that column again (verified against pgx v5.9.2 source: a
scanned '{}' round-trips as non-nil in both directions; the nil could
only come from our own assignment).
The plain Rescan path never hit this - only Reset did. Worse, the reset
is the remedy when a book's cover_image_path override pins an empty
cover, so the crash also blocked the way out of that state. After this
fix, a plain rescan on an already-reset book repopulates scanned
metadata and extracts the cover.
Adopt Calibre's reading conventions for the Dublin Core metadata that
parseOPFContent now pulls from the structured OPF parse:
- Titles: EPUB3 title-type selection (prefer 'main', join a distinct
subtitle with ': ' exactly as Calibre stores it). There is no separate
subtitle column by design - Calibre-sidecar books arrive pre-joined,
so a column would stay empty for most libraries and force every client
to reimplement concatenation.
- Genre: first dc:subject, mirroring the existing processGenresAndTags
behavior of the Calibre-sidecar path; the embedded path never
populated Genre before. Subjects stay one-element-one-tag - Library
of Congress headings legitimately contain commas ("Holmes, Sherlock
(Fictitious character) -- Fiction") and must not be split.
- Identifiers: urn:isbn:/urn:asin: prefixed values parse in addition to
opf:scheme attributes, and the scheme-less fallback now requires an
ISBN-shaped value (10/13 digits, optional separators/trailing X) so
URIs like the Gutenberg identifiers cannot masquerade as ISBNs -
observed live on 'A Study in Scarlet'.
- Series: EPUB3 belongs-to-collection with collection-type=series and
group-position refines, ahead of the classic calibre:series metas.
- Audiobookshelf metadata.json sidecars join their subtitle field into
the title the same way.
Tests cover title-type main+subtitle joining, belongs-to-collection
series with fractional group-position, urn:isbn extraction, genre/tag
parity, and comma preservation inside subject headings.
The cover lookup scraped the OPF with attribute-order-sensitive regexes.
Real books serialize attributes in any order - Grand Central's '3 Days to
Live' puts href before id on manifest items and content before name on
the cover meta - so all three regex paths missed and the book fell
through to filename guessing, extracting no cover at all. Attribute
order is meaningless in XML; the regexes were never safe.
Replace them with a structured parse (encoding/xml, namespace and
attribute-order agnostic; see the new media_scanner_opf.go) and follow
Calibre's read_raster_cover resolution order:
1. manifest item with properties=cover-image (non-(X)HTML media only)
2. <meta name=cover> resolved through the manifest, same media guard
3. first spine item that is itself a raster image (store manga)
4. NEW cover-page fallback: books declaring no raster cover at all -
the classic EPUB2/Adobe cover.xhtml wrapper - are mined for
<img src> / SVG <image xlink:href> references (Calibre renders the
page with Qt; extracting the referenced image covers the practical
cases without a rendering engine)
5. existing zip filename guessing stays as the last resort, and the old
regex chain survives as findCoverInOPFLegacy for OPFs too malformed
for a real XML parse.
Hrefs are now URL-decoded and posix-normalized against the OPF's own
path (path.Join semantics), so '../art/cover.jpg' from a nested cover
page and %20-encoded names resolve correctly.
Tests: attribute-order chaos modeled on the failing Patterson book,
SVG-wrapped cover pages via guide references, image-first spines, and
path resolution edge cases. Verified live against the real
'3 Days to Live' EPUB, which previously produced no cover.
Bulk escape hatch for archived rows (files missing from disk for 2+
scans) so a mass external deletion never has to wait out the retention
window or be clicked away row by row:
- POST /api/media-items/purge-archived (admin only) hard-deletes all
archived items and returns the purged count; reading history goes with
the rows, so the call is confirmed in the UI first.
- The library admin page shows an 'Archived items: N' card (only when
non-zero) with a Purge Archived Now button that calls the endpoint,
toasts the result, and reloads.
- Frontend admin JS exposes window.purgeArchivedItems following the
existing localStorage-bearer-token pattern.
Replace the silent hard-delete orphan cleanup (which logged only through
ScannerLogger file logs and whose failure paths left rows undetected)
with an archive lifecycle that preserves reading history:
- A file missing in one scan is marked (missing_scan_count = 1); missing
in a second consecutive scan archives it (archived_at, hidden from
browsing, progress/notes/highlights survive). Every branch logs to
stdout with an [ARCHIVE] prefix so skips are always visible.
- When a file reappears - same path, or identical content at a new path
via the SHA-256 dedup match - the archived state clears automatically
and the item returns with its history intact.
- Archived rows older than ARCHIVE_RETENTION_DAYS are hard-purged at
scan time (cascading deletes); 0 disables auto-purge for manual-only
management. Retention is read from the environment in NewMediaScanner.
Add the storage behind the archive-instead-of-delete lifecycle:
- media_items.missing_scan_count (INT, default 0) and archived_at
(TIMESTAMPTZ, partial index), both as idempotent ADD COLUMN IF NOT
EXISTS backfills for existing installs.
- MarkMediaItemMissing / ArchiveMediaItem / ClearMediaItemArchive plus
PurgeExpiredArchivedMediaItems (retention cutoff) and
PurgeAllArchivedMediaItems (manual bulk), with CountArchivedMediaItems
for the admin UI.
- Archived items are hidden from every user-facing listing:
ListMediaItems, ListMediaItemsByLibrary, ListMediaItemsSorted,
SearchMediaItems, SearchMediaItemsUnified, the next_books CTE, the
library media counts, and the search autocomplete value lists. Detail
lookups by id/path/sha are intentionally unfiltered, and a dedicated
ListMediaItemsByLibraryIncludingArchived feeds the scanner so the
lifecycle pass can see and restore archived rows.
Libraries managed by Audiobookshelf keep a metadata.json next to each
book (title, authors, series+sequence, genres/tags, publisher,
description, isbn/asin, language, published year/date) - and no
metadata.opf. The scanner silently ignored those files: deleting them
changed nothing, and their data never reached the database.
Parse them as a first-class sidecar in extractMetadata, priority
metadata.opf -> metadata.json -> embedded media. Only fields with a
matching media_items column are mapped; narrators, subtitle, explicit,
abridged, and chapters are deliberately skipped.
Cover handling is unchanged: the existing findSidecarCover priority
(cover.jpg / folder.jpg / {basename}.jpg) applies to the sidecar branch
exactly as it does for Calibre.
go-epub's ReadBook parses every spine chapter and fails the entire call
if any single chapter (or the TOC) is malformed, discarding already-parsed
OPF metadata. For books like Pragmatic's 'A Common-Sense Guide' the OPF
holds good title/author/publisher/ISBN metadata that rescans then wrote
as blanks - success toast, no (visible) change.
Refactor to parse the EPUB's own OPF document with the same Dublin Core
machinery used for Calibre sidecars: parseCalibreMetadataOPF is now a thin
file wrapper around a reusable parseOPFContent([]byte), and the OPF lookup
previously inline in extractEPUBCover is shared via findOPFPathInZip. Since
metadata never touches chapter bodies, chapter damage cannot blank it.
Also picked up along the way: dc:language mapping and scheme-less
dc:identifier values that normalize to a valid ISBN (EPUB3 style).
Tests cover a Pragmatic-style EPUB (dc namespace on <metadata>, no
identifier scheme, deliberately malformed chapter) that must still yield
full metadata, plus series/date/subject OPF parsing.
The book detail page exposed server internals and unusable controls to
everyday users: it now computes and renders the book's absolute on-disk
location, and gates all of it behind the admin role.
- The page handler resolves library folder + relative path (verified
with os.Stat; falls back to the relative path when the file is not
found on disk) into the new MediaDetail.FileLocation field - only for
admins, so the absolute path never leaves the server for regular
users. This also makes it possible to locate sparse entries whose
metadata rows are largely blank.
- The Metadata grid gains a full-width, monospace, click-selectable
Location row (admins only).
- The Edit button and the MetadataEditorModal markup render only for
admins. The modal drives admin-only endpoints (metadata PUT, rescan,
reset), so non-admins previously saw a button and a form that could
only ever fail with 403s.
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
parseTableNames() regex-scanned every line of schema.sql, comments
included, with 'CREATE TABLE (?:IF NOT EXISTS )?(?:\w+\.)?(\w+)'. A doc
comment containing that phrase in prose - e.g. 'declared in CREATE TABLE
above' - registered a phantom table ('above'), and startup verification
then failed with 'missing tables: above', crash-looping the app
container on every restart.
Skip lines whose trimmed form starts with '--' so comments can never
contribute table names, and add a regression test asserting every parsed
table maps back to a real CREATE TABLE statement.
pdftoppm defaults to rasterizing the MediaBox, while PDF viewers (pdf.js
in the reader, and every other viewer) display the CropBox. For PDFs
whose page 1 is the full print cover wrap (back cover + spine + front
cover in one landscape page) with a CropBox covering only the front
cover - e.g. No Starch's XeTeX-built 'Algorithmic Thinking' - the
fallback stored the entire spread as a squashed landscape cover, while
the reader correctly showed just the front cover.
Pass -cropbox so the rendered cover always matches what the reader
displays. Poppler falls back to the MediaBox when a PDF defines no
CropBox, so PDFs with identical boxes (the common case) render exactly
as before.
Admin-only endpoint that re-extracts metadata and cover art for a single
book from the file on disk via MediaScanner.RescanMediaItem and returns
the updated media item. Normal library scans skip unchanged files, so
this gives a targeted way to backfill covers for previously imported
books. Includes a Bruno request alongside the existing media-items
collection.
extractPDFCover previously only saved embedded raster images from page 1,
so vector/text-first-page PDFs (e.g. InDesign exports like Data Structures
the Fun Way) ended up with no cover and a dashboard placeholder. When no
embedded image is found it now falls back to rendering page 1 with
pdftoppm (poppler-utils), saving the same {pdf}.cover.jpg sidecar.
Also adds MediaScanner.RescanMediaItem, which re-extracts metadata for a
single media item (resolving its on-disk path from library folders) so
previously imported books can backfill covers without a full force rescan.
Dockerfile installs poppler-utils in the final and test-runner stages.
The LWW skip path compared only annotation content (text, color, note,
percentages) — locator columns were not part of 'changed'. A device echo
with identical content therefore resolved to skip, and the freshly
re-derived canonical CFIs were discarded in the same request that
computed them: a highlight whose stored start anchor had been corrupted
by the old percentage-exact bug could never heal, because every
subsequent echo carried the same text and was skipped before the
locator columns were written. Observed live: a push converted the
cross-block highlight's anchors correctly (structural, heading to
paragraph) yet the row kept its garbage start CFI and the highlight
stayed unpaintable on the web.
Echo saves now treat a non-empty incoming locator that differs from the
stored one as a change (locatorRefresher): empty locators still coalesce
(no drift), and once healed the echo produces identical CFIs, so the
steady state remains skip — no write churn. Highlights check
epubcfi_start/end; bookmarks check cfi_position/position.
applyBookmarkLWW also gains the empty-locator coalescing the highlight
path already had: web bookmark edits carry no device locators, and a
title/note edit must not wipe the stored device-native position.
Tonight's failures all traced to one blind spot: the converter could
only reason about text within a single block. A position at a chapter
heading sends walk-up context (heading + the paragraphs below, joined by
the plugin's block capture); a selection can span several paragraphs.
Neither shape could be verified (containment compared one block against
a multi-block quote, so the CORRECT structural landing at the heading
was rejected) nor matched by text search (it never crossed block
boundaries). The ladder then fell to the percentage rung — which labeled
its char-count guess Precision "exact" — and that confidently-wrong CFI
was stored: reading positions reopened paragraphs away from the true
spot, and a highlight echo overwrote the row's good web CFIs with a
garbage start anchor that made the highlight unpaintable ("disappeared").
Four changes, all in the forward converter and its consumers:
- Quote verification: after the structural walk lands, read the
whitespace-normalized document text forward from the landing point
(crossing block boundaries; inline spans join directly so drop-cap
splits still read as one word). A usable context must be a prefix of
that stream — which is exactly what device captures are: the text from
the position onward, or the selection between two anchors. The old
single-block containment checks remain as secondary acceptance.
- Cross-block text search: the search rung matches against the whole
document flattened in reading order, with every rune mapped back to
its source node and offset. A context spanning blocks now matches, and
the matched extent yields a true range end (EndEPUBCFI) that
highlights use as their end anchor, threaded through the facade as
CanonicalLocator.EndCFI.
- Honest labels: the percentage rung returns Precision "percentage" —
a char-count estimate must never masquerade as an exact anchor.
- Confident-only storage: progress adopts a converted locator solely at
structural/exact precision (section hrefs keep their legacy handling;
anything lower stores percentage only), and highlight conversion
returns CFIs only at structural/exact precision — a low-confidence
echo yields empty, which applyLWW coalescing turns into preservation
of the row's existing web CFIs instead of clobbering them.
Tests: walk-up context at a heading verifies structurally and lands in
the heading; a block-spanning context is found by search with a range
end landing in the following paragraph; the percentage rung is honestly
labeled; all drop-cap guards stay green.
The reader page embedded a snapshot of reading state (position,
bookmarks) server-side at render time. Browsers may reuse that HTML
(heuristic caching, bfcache), so opening a book could restore a stale
position — and worse, the restore's relocate auto-saved it back,
overwriting a newer device push minutes later. A KOReader sync followed
by opening the web reader would silently revert the row to the old web
position; the row's source and the rendered page disagreed.
The web reader is intrinsically tied to the server, so it has no business
preserving reading state client-side:
- The rendered page now carries only immutable book metadata. The reader
fetches progress fresh (cache: no-store) from the existing progress
API at open and restores with the same priority as before (page for
fixed-layout, CFI, percentage, fresh start); a failed fetch opens at
the start and writes nothing. Initial bookmarks likewise come from
their endpoint instead of the embed; annotations already did.
- Progress saves are gated on deliberate navigation only (page turns,
keys, slider, search/TOC/bookmark/back-stack jumps, tap zones — each
marks the session as user-moved). Restores and section-load
relocations never write, so displaying a position can no longer
clobber a newer one. A bfcache-resurrected page resets the flag and
cannot write its frozen position either. This replaces the old
five-second post-init suppression, which a stale page bypassed.
- The server-rendered initial progress badges render a neutral
placeholder until the first relocate fills them (sub-second).
No API, schema, or sync-engine changes. Normal reading saves exactly as
before — the first save now simply waits for the first real page turn.
PUT /highlights/:id parsed the row id from the URL and then dropped it:
the sync-aware path routed through SaveHighlight's content-derived dedup
key, on the assumption that the same text + CFI always resolves to the
same key. That assumption breaks in practice — the stored epubcfi_start
drifts from foliate's range shape to the converter's point shape after a
device echo rewrites the row (bucketPosition cuts at the last colon, so
'…/6,/1:367,…' and '…/6/1:367' bucket differently), and the user can
edit the selection text. The recomputed key then misses the row being
edited and createHighlight mints a second one: the edited row (with
note, no device pos0) beside the original — served to KOReader as two
highlights, one noted and one not. Editing a selection's text would hit
the same trap.
SaveHighlightRequest gains an optional HighlightID. When set, the save
resolves the row by id (ownership-checked), LWWs against it under its
stored dedup key, and never re-derives identity from content. The PUT
handler passes the already-parsed id. Device pushes, Kobo, and the sync
queue send no id and keep the identity-based flow untouched.
applyLWW also stops wiping stored locators on web edits: the web reader
sends empty start/end positions (it never had a CRE xpointer), so a
note/color edit now keeps the device-native positions and CFIs instead
of blanking them — round-trip serve-back for device-created highlights
survives web-side edits.
Progress, highlights, notes, and bookmarks entered position conversion
through three different doors: progress converted inline with an
uncached converter, annotations through the facade, bookmarks not at
all (the raw xpointer was stored verbatim, cfi_position stayed empty,
and the web drawer's goToBookmark silently no-ops on cfi-less entries).
Unify on the facade (ConvertToCanonical/ConvertFromCanonical):
- annotationEpub context resolved once per push: media item + EPUB path
shared by every annotation instead of re-fetched per entry
- progress forward: the inline block becomes one facade call;
non-reflowable formats pass through unchanged, and the cached
converter stops re-parsing the book on every sync
- progress reverse: convertCFIToXPointer delegates to reverseConvertCFI,
keeping the stored percentage in play for the fallback ladder
- bookmarks (bulk progress and /sync-bookmarks): pos0 resolves
structural-only — bookmark text is a display label, never book text,
so no context is supplied; webUsableCFI stores the result only for
structural/exact epubcfi landings, discarding href/percentage results
rather than storing dead drawer links. Also records percentage_location
and origin_source on the legacy endpoint.
- percentages thread through: highlights/notes/bookmarks pass the device
percentage or the derived section percentage instead of a hardcoded 0,
so the last-resort fallback lands near the true position instead of
the document start
- extendCFIByLength end-derivation now also fires on structural starts
(it had silently stopped matching when the structural rung began
landing starts with precision 'structural' rather than 'exact')
Tests: the drop-cap xpointer through the facade with empty context (the
bookmark scenario) must land structurally, not doc-start; webUsableCFI
table covers the store/discard gate.
Per-annotation percentage derivation (deriveAnnotationPercentage) built a
fresh CFIConverter for every highlight/note/bookmark, re-reading and
re-parsing the whole EPUB each time. Export SectionPercentageCached so
handlers reach the same bounded cache ConvertToCanonical already uses
(8 books, insertion-order eviction): one parse per book per push instead
of one per annotation.
- Drop UNIQUE(media_item_id,user_id,title): titles are display labels
shared verbatim across clients; same-title bookmarks on different pages
now coexist instead of 500ing (deleting over a tombstone no longer
blocks future creates with that title)
- Add origin_source column recording the creating client (android/web/
koreader), set once at insert, exposed in API responses
- Web reader auto-title mirrors KOReader's 'in <chapter>' convention,
falling back to 'Bookmark'; adds bookmark rename in the drawer
Drop-cap markup like <p><span>C</span>onvergence of Heaven and Earth</p>
made getTextFromXPointer return just 'C'. The text search then matched
the first 'C' in the chapter and stored doc-start (/4/2/1:0) with
'precision: exact', so the web reader reopened at the chapter start
while the percentage looked mid-chapter.
- Add convertByStructuralPath: walk the parsed CRE ElementPath against
the raw XHTML (same-tag 1-based indexing, mirroring buildCREXPointer),
map CharOffset into the target element's text, and build the CFI.
Usable device text verifies the landing; disagreement falls through
instead of storing a confident-but-wrong CFI.
- Add usableContextText guard (>=8 runes, >=2 words): single chars can
never claim an exact text-search hit in either direction
(ConvertCREToStandard and reverseByTextSearch).
- Add drop-cap regression fixtures plus usable-context unit tests.
- Verified against the real book: DocFragment[26]/p[12]/span header now
converts to epubcfi(/6/52!/4/28/2/1:0) structural both with 'C' and the
full header, and round-trips back to DocFragment[26].
Bookmark dedup is keyed on hash(title + position bucket), but the table
also enforces UNIQUE(media_item_id, user_id, title). When a client re-
saves the same bookmark title with a changed position form - e.g. the
Android app upgrading a percentage-only row to an EPUB CFI, or a web and
app bookmark landing on the same 'Bookmark at 44%' title - the dedup-key
lookup misses and the INSERT violates the title constraint, returning
HTTP 500 and failing the sync.
A title collision on the same (user, item) is by definition the same
bookmark slot, so take the LWW semantics all the way: ON CONFLICT DO
UPDATE replaces position/cfi_position/page/chapter/percentage, refreshes
dedup_key and timestamps, merges device_sync_data, and - matching
UpdateMediaBookmarkForSync - clears deleted/deleted_at so a re-create
resurrects a tombstoned title slot instead of leaving an invisible row
holding it.
Device sync flows are unaffected: KOReader/Kobo pushes that carry their
own dedup-key echoes never reach the INSERT, and same-key saves still go
through applyBookmarkLWW with its tombstone freshness checks.
ServeFile previously authenticated only ("any logged-in user") and never
checked that the user can actually see the library owning the file, so
knowing a library UUID + path was enough to fetch content from hidden
libraries. Library visibility is the permission model - the library is
what grants access to its media.
- ServeFile now resolves two URL forms through one flow:
/uploads/library-{id}/{path} (covers, reader files)
/api/media-items/{id}/download (explicit book download, new)
The item form looks up the media item, derives its library and file
path, and adds a Content-Disposition attachment header.
- Both forms enforce GetUserVisibleLibraries for the authenticated
user, mirroring the OPDS download handler (403 when not visible).
- Deleted the dead MediaHandler.DownloadBook handler (never routed).
Also widen media_highlights.start_position/end_position from
VARCHAR(100) to TEXT: the API handlers validate up to 1000 characters
(full Readium locators, KOReader CRE xpointers) but the column rejected
anything longer at the database layer. Metadata-only change applied
idempotently at startup; existing rows are untouched.
Verified against the running server: download 200 + attachment headers
+ epub bytes, unauthenticated 401, user hidden from the library 403 on
both URL forms, visible user 200, covers unchanged, and a 334-char
locator JSON now round-trips through the highlights API.
KOReader push (processBookAnnotations) accepts deleted_highlights and
deleted_bookmarks arrays of dedup keys and tombstones the matching rows,
after the upserts so a key present in both lists resolves to 'deleted'
(the newer intent). Deletions remain soft: rows stay restorable from the
history and echo to other devices as tombstones on their next pull. A
stale device replay of the annotation cannot resurrect the tombstone —
device pushes carry no modification timestamp, so the save loses to the
delete. Absence from these arrays is never a delete, keeping category
toggles safe.
New annotation-history endpoints (annotation_history.go, media.go):
GET /api/media-items/:id/annotations/deleted
POST /api/media-items/:id/annotations/:annotationId/restore
DELETE /api/media-items/:id/annotations/:annotationId
All scoped to the authenticated user and the route's book; the DELETE is
the permanent purge (annotation_type required in query or body).
MediaDetail gains DeletedAnnotations, populated by the book page route
via the shared DeletedAnnotationsForBook builder, so the server-rendered
history ships with the page instead of requiring a client round-trip.
Binding tests cover the plugin's exact wire shape and the legacy
plugin case (arrays omitted -> empty).
RestoreAnnotationByID and PurgeAnnotationByID dispatch on annotation kind
(highlight/note/bookmark) to the new queries, broadcasting an annotation
update on restore so connected web sessions refresh. Both report whether
a row actually changed.
TombstoneBookmarkByDedupKey mirrors the existing TombstoneHighlight for
bookmarks: devices report deletions by dedup key (they have no row IDs),
and until now only highlights had a key-based tombstone path — device
bookmark deletions had nowhere to land.
ValidAnnotationKind centralizes the kind check the HTTP handlers share.
ListDeletedAnnotationsForBook unions tombstoned highlights, notes, and
bookmarks for a user+book regardless of the sync TTL cutoff (the history
must show everything still restorable, not just recent deletes), with
display text, secondary text, color, and both timestamps.
Restore queries clear deleted/deleted_at (lossless — the row was soft-
deleted, never removed) and are scoped to the owning user and media item
so a restore can never touch another user's annotation.
Purge queries hard-delete an already-tombstoned row: the user-driven
counterpart of the TTL maintenance sweep, for explicit 'delete
permanently' actions from the history.
All six write queries are :execrows so callers can distinguish 'restored'
from 'nothing matched' without a follow-up read.
GET /api/sync/koreader/resolve?sha256={hash} maps a file content hash to
the book's UUID through the shared format-aware BookResolver (primary
media_items hash, then per-format hashes so converted KEPUB/PDF files
match) without touching any progress state.
Devices need the UUID to pull metadata, but a freshly downloaded book has
none cached. The old way of learning it was to push once, which
transmitted the device's first-page position and manufactured a progress
conflict for books already mid-read from another source. A read-only
lookup lets clients link (and pull) without ever pushing bootstrap
progress: resolve, then pull, then push.
Returns 200 {book_uuid, sha256, title, author}, 400 for a missing or
malformed hash, 404 when no library item matches.
Six converter tests pointed at absolute paths for 1984 and Crime and
Punishment under uploads/ — books that don't exist on most checkouts
(CI included), so the suite shipped with 5 permanently failing tests
(and a sixth passing only by accident: the percentage-fallback path
triggered by the missing file is the outcome it asserts).
A writeTestEPUB helper now builds a minimal deterministic EPUB in
t.TempDir() (zip → container.xml → OPF → 6-doc spine), so the tests
exercise the real zip/OPF/spine/document pipeline with no external
dependencies. The xpointer→CFI conversion, fragment-ID conversion,
both round-trips (bare and context-text-anchored), and the text-search
and percentage fallbacks all keep their original assertions, now
against known document content. internal/sync is green for the first
time on this machine.
Reverses the earlier "no colors to the device" decision now that the
echo machinery makes it safe: GetMetadata maps the stored web hex to
KOReader's fixed color names (#ce93d8→purple, #90caf9→blue,
#a5d6a7→green, #ffd54f→yellow; pink maps to purple as the closest —
round-trip drift is prevented on the device by echo suppression, and
a device edit still wins). mapColorToKOReader restored for serving;
ingest (name→hex, preserve-on-echo) unchanged.
Echo duplication: devices push their full annotation list on every
sync, and an echo of a web-created annotation computed a different
dedup key than the original (device locators differ from web locators)
— every pull→push cycle minted a duplicate row, and cleaning those up
on the web tombstoned them back to the device, deleting the
just-applied copies. That was the "web highlights never appear on
KOReader" experience. GetMetadata now serves each annotation's
dedup_key; the device stores it on the applied entry and echoes it in
pushes; SaveHighlight/SaveBookmark/SaveNote accept a DedupKey
override so echoes converge onto the original row (verified: pull →
echo push creates no rows, LWW skips identical content).
Color semantics (per user preference): devices render their own
default and cannot round-trip web colors, so GetMetadata no longer
serves colors at all — every highlight syncs regardless of its web
color and the device draws its default. An echo carries no color;
ingest then PRESERVES the stored web color (existingHighlightColor
lookup by dedup key) so round-trips never change it. A non-empty
device color means the user edited the highlight there: it maps
name→hex (green→#a5d6a7, default yellow) and wins. Verified: echo
kept #ffd54f; a simulated device edit with "green" updated the web
row to #a5d6a7.
Classification: KOReader auto-fills text="in Chapter X" on page
bookmarks (ReaderAnnotation:updateItemByXPointer), so the plugin's
text-presence classification turned every echoed bookmark into a junk
highlight on the web. v2 classification now keys off the drawer field
(present = highlight/note, absent = bookmark with its label in note).
Both directions synced data but rendered nothing:
- Web reader <- devices: highlights painted no overlay. Device pushes
resolve their start xpointer exactly (text-search anchored by the
selection) but the end conversion carries no context and fell back
to a document-start CFI (epubcfi .../1:0) — a garbage range end.
When the start resolved exactly, the end is now derived from it:
same node, character offset advanced by the selection's UTF-16
length (extendCFIByLength). Same repair when SERVING to devices,
where old web highlights (no end anchor) and converted range CFIs
both collapsed pos1 onto pos0 (extendXPointerByLength on the
xpointer form) — KOReader drew zero-width highlights.
- Colors: KOReader paints from a fixed name set (Blitbuffer
HIGHLIGHT_COLORS), the web uses hex swatches; neither understood
the other, so device colors fell back to defaults and web hex drew
nothing useful on devices. Both boundaries now translate: ingest
maps names to hex (default #ffd54f), GetMetadata maps hex to names
(default yellow) — per-datatype edits re-push with the editing
side's color, which LWW then propagates. SyncBookmarks endpoint
aligned to the same mapping and default.
ConvertToCanonical/ConvertFromCanonical built a fresh CFIConverter
per call, and each annotation converts twice (pos0+pos1) — a book
with 200 highlights re-opened and re-parsed the EPUB 400+ times per
sync, and again per metadata pull. A bounded 8-entry cache keyed by
path now shares converters (the parsing work belongs on the server;
clients stay thin). CFIConverter gained a mutex around its lazily
built spine/doc caches since instances are now shared between
concurrent requests.
Adds CFIConverter.SectionPercentage: book-wide percentage for a CRE
xpointer from the spine char distribution (midpoint of its document)
— the server-side counterpart to dropping per-annotation
getPageFromXPointer lookups from the plugin.
Two blockers, diagnosed by simulating the plugin against the live
server with real library books:
1. Every KOReader progress push carrying annotations failed the JSON
bind with 400 ('cannot unmarshal string into ... chapter/page of
type int') — the plugin sends chapter:'', page:'30', and for CRE
documents page:'/body/...' — so annotation sync AND progress sync
failed together. KOReader annotation chapter/page now use FlexInt,
which accepts numbers, numeric strings, empty strings, and
non-numeric strings (decoding to 0). The server is deliberately
liberal here so thin clients can send raw bookmark data.
2. GetMetadata served locators KOReader cannot place, so pulled items
were junk: web bookmarks leaked 'cfi:epubcfi(...)' positions, web
PDF highlights had empty pos0 (skipped by the plugin, invisible),
and web deletions carried no pos0 so tombstones never matched.
New koreaderPos0 resolver handles every source: device-native
xpointers pass through untouched (round-trip identical, verified),
web PDF JSON anchors map to their page number, EPUB CFIs convert
to CRE xpointers (selection text passed as text-search context for
exact anchoring), 'page:N' positions strip to the bare number.
Unresolvable annotations are skipped with a log line instead of
poisoning devices; tombstones get pos0 injected from the new
locator columns.
Also: thin clients omit per-annotation percentages (paging docs still
send arithmetic page/total); the server derives them — section
midpoint from the spine char distribution for CRE documents, page/
page-count for fixed formats.
GetTombstonedAnnotationsForBook now also returns each tombstone's
start_position/end_position and epubcfi_start/end (note: position/
epubcfi_location, bookmark: position/cfi_position), so serving code
can resolve a device-native locator for deletions of web-created
annotations, whose device_sync_data carries no pos0.
Phase 4 of the reader redesign (foliate-js ea268df):
- Webtoon mode for comics: continuous vertical scroll of all pages
(900px centered column on wide screens), lazy-loaded with a 150%
IntersectionObserver margin, far pages unloaded to bound memory
with stable aspect-ratio placeholders so the scrollbar never jumps.
Chosen per book (Paged | Webtoon segmented control in Settings →
Layout & Display; stored in localStorage per media item since a
webtoon title and a paged manga volume want different flows).
Toggling reloads the reader — the renderer is chosen at open time —
and progress restores from the saved page. Relocate events flow
through the same pipeline, so the slider, progress saving, back
stack, tap zones, and edge zones all work unchanged. Zoom/fit/
magnifier/spread controls hide in webtoon (natural-width scroll).
- Display filters for fixed-layout: brightness (30-130%) and
contrast (70-130%) sliders with live preview, plus Night Mode
(invert) — also a quick row in the ⋯ tools menu. One --fx-filter
CSS var drives everything: ::part(filter) on foliate-view iframes
(forwarded via the new exportparts attribute) and the webtoon
page images alike. Persisted as fx_brightness/fx_contrast/fx_invert
(types + defaults both sides); Restore Defaults resets them.
Phase 3 (EPUB half) of the reader redesign:
- Select text in a reflowable book → floating glass popover at the
selection (5 colors, note, copy). Clicking a color creates the
highlight via POST /api/media-items/:id/highlights, anchored by the
foliate range CFI (epubcfi_start) with percentage position.
- Highlights render through foliate's overlayer pipeline: draw-
annotation draws Overlayer.highlight with the stored color,
create-overlay re-adds persisted highlights as sections load,
show-annotation opens the edit popover when a highlight is clicked
(recolor, edit note, copy, delete).
- Backend: highlight create/update accept epubcfi_start/end,
note_text, and percentage fields; position validation relaxed
(CFIs exceed the old 100-char cap); PUT routes through
AnnotationService.SaveHighlight so edits get dedup/LWW treatment
and actually persist note_text (the plain query can't).
- Bookmarks drawer becomes the Annotations drawer with tabs:
Highlights (color-bar list, note previews, jump/edit/delete),
Notes (add note at current position, list, delete — backed by the
existing notes API), and Bookmarks (unchanged behavior).
- Popover dismissed on outside click, collapsed selection, page
navigation, or Esc (new top-priority Esc branch).
Phase 2 of the reader redesign:
- Fixed-layout touch engine (foliate-js e9e61d8): pinch-zoom around
the midpoint, two-finger pan, single-finger pan while zoomed,
horizontal swipe page-turn at fit (RTL-aware via next()/prev()),
and double-tap to zoom 2.5x / reset. Touch events forwarded from
page iframes with converted coordinates; preventDefault only when
the engine consumes the gesture, so PDF text selection and native
taps stay intact. touch-action: none on the host and in comic/pdf
page documents keeps the browser from fighting the engine.
- Tap zones (Kindle-style) for touch devices: tap the outer margins
to page, center to toggle chrome. Size configurable (10-50%) via
the revived tap_zone_size setting; toggle via new tap_zones_enabled
(Behavior section of the settings drawer). Pointer-based + passive
so drags/swipes/selection never trigger; attached both to the
viewport and inside every page document (iframe events don't
bubble); debounced 280ms so double-tap zoom doesn't also page; no
zone actions while a fixed-layout page is zoomed.
- Drawers become full-width sheets on screens <= 640px.
Deleting a bookmark/highlight/note and then re-adding the same content
at the same position (same dedup key — e.g. the reader's auto-titled
'Bookmark at X%') was silently swallowed: the save hit the tombstone
branch, returned 201 with the deleted row, and the list (which filters
deleted) stayed empty. Bookmarks were further blocked by the
UNIQUE(media_item_id, user_id, title) slot the tombstoned row holds,
and notes had no TTL escape at all.
Tombstones now only block saves that predate them (stale replays from
a device that still has the annotation). A save whose modification
time is newer than max(deleted_at, last_modified_at) — a deliberate
re-create from the web or a device — resurrects the row via the LWW
update queries, which now clear deleted/deleted_at.
Phase 1 of the reader redesign:
- Reading surface is edge-to-edge; top/bottom bars overlay
translucently (backdrop-blur) instead of reserving insets, killing
the inset-coordination bug class entirely. Chrome auto-hides after
2.5s of pointer inactivity (chrome_behavior setting finally wired:
auto-hide / always-visible; legacy values map to auto-hide). Pointer
activity inside page iframes keeps it awake; Esc toggles.
- TOC / Settings / Bookmarks become slide-over drawers with a scrim
(z-50, full-height, safe-area aware), replacing the dockable-panel
system and its window-shade headers. Only one drawer opens at a
time; Esc or scrim click closes.
- Bottom bar is contextual: reflowable keeps nav/slider/progress/TOC;
fixed-layout row adds Fit Page/Width select, zoom cluster,
magnifier (now shows active state), Double Page Spread toggle, and
a Smart | Pan | Text segmented control replacing the cryptic
two-state icon. Smart = text-aware drag; Text = selection-only
(manual smart-detect off); Pan = force pan. Choice persists via
pdf_interaction_mode (new setting + foliate 29bc958 'text' mode).
- Settings drawer: Behavior (chrome, progress mode), Appearance with
18 Kindle-style theme swatches (single source of truth from
THEME_COLORS), Typography, Layout — each scoped by format.
- Keyboard: t/s/b open TOC/settings/bookmark, Esc closes drawers
before toggling chrome, shortcuts skip form inputs; both slider
rows tracked correctly (no duplicate-ID lookups).
- Topbar: Back, title, add-bookmark, bookmarks drawer, Aa settings;
chrome follows user theme.
Phase 0 of the reader redesign:
- Panels no longer render under the top/bottom bars: sidebars get
measured insets (same resize/safe-area mechanism as the viewport);
panel max-height now derives from the bounded sidebar instead of a
100vh guess; right-side border targets the actual sidebar.
- Bookmarks work end-to-end for the first time: REST CRUD under
/api/media-items/:id/bookmarks (create/delete route through
AnnotationService for dedup/LWW/tombstones), fix UpdateMediaBookmark
referencing nonexistent updated_at column, frontend posts to the
real API with per-format position (CFI vs page), live list with
jump + delete instead of SSR-only snapshot.
- Fix chapter matching in progress saves: boundaries were compared by
a nonexistent tocItem property, so chapter was never persisted.
- Remove dead UI: Navigator panel stub, empty dictionary popup shell,
unwired Chrome Behavior select; purge 160 stale build artifacts.
- Reader chrome now follows the user's app theme instead of hardcoded
theme-tokyo-night.
John O'Keefe
john-okeefe