Closes the remaining integration-suite failures (handoff open bug 3),
line by line:
- Fixture family (the upsert equivalence): tags_filter_test
("Science Fiction" found 1 of 3; "Fantasy" 404'd on zero results),
media_item_isbn_test (normalization read back the first case's ISBN
everywhere; edge cases saw a nil ISBN; pagination collapsed 5
creates onto 1 row), search_unified_test (its 3 books were actually
1 row + two no-op creates) — all POSTed a constant file_path into
one library while CreateMediaItem upserts ON CONFLICT
(library_id, file_path). Every fixture now uses unique per-item
paths, the suite's universal convention elsewhere. TestTagsFilter
confirmed same-family: counts came right with no second bug.
- New TestMediaItemCreateDuplicatePath pins the honest-create
contract from da13898: 201 for the first create; 200 + duplicate
flag + the EXISTING row (never a 201 lie), second payload not
overwritten.
- library_browse_test: the admin subtest browsed the container's
/tmp and required it non-empty — env-coupled (empty /tmp → "was
[]"). Now browses a self-created t.TempDir with two subdirs and a
decoy file, asserting ONLY directories are listed.
- series_integration_test: three stale assertions re-anchored to the
shipped contracts, each with a ground-truth comment: /dashboard/
sections key identity on the collection NAME (BuildSections ID =
CollectionName; the web/app reorder + hidden prefs persist those
strings) — the kebab query_type is the storage key and is not
exposed; /api/series/books is name-ONLY (the app client and the
web SSR page both send name only; library_id is ignored, never
validated); /api/series is optional-scope BY DESIGN (GetDistinct-
Series: narg IS NULL OR library_id = narg aggregates across
libraries when omitted). The former 400-on-missing-library_id
expectations asserted validation that never shipped.
- gofmt rider on three untouched-in-this-arc test files
(dashboard_integration, search, test_helpers).
Full suite with -parallel=1: 217 groups pass, zero failures.
The create endpoint shared the scanner's upsert (ON CONFLICT
(library_id, file_path) DO UPDATE SET updated_at = NOW() RETURNING *) —
so a POST naming an already-registered file returned 201 CREATED with
the OLD row while silently discarding everything the caller submitted.
Nothing depended on that lie: the app only GETs/PUTs media items, the
web only searches/purges/deletes, KOReader never calls it — the
endpoint's create path had zero production callers.
Split the API path from the scanner's idempotent upsert, which is kept
verbatim in CreateMediaItem (re-scan dedup remains its safety net):
- CreateMediaItemIfNew: same insert with ON CONFLICT DO NOTHING — the
loser of a create race writes nothing and returns no row.
- GetMediaItemByLibraryAndPath: fetch the existing row for the
duplicate response.
- Handler branch: on pgx.ErrNoRows → 200 OK with
{"duplicate": true, "message": ..., "item": <existing row>} instead
of a 201 pretending it created something. Real creates still answer
201 with the item as before.
Bulk-import shape (the reason for 200+flag over 409): a re-run after a
mid-import failure is harmless and countable — each response tells the
caller exactly which of the two things happened.
Verified by the full integration suite: 217 groups pass, zero failures,
including a new pin of the duplicate contract (same file_path twice in
one library → 201 then 200/duplicate:true, second payload not
overwritten).
Born dead in e7ce0cb (2026-04-03, backend reader infrastructure):
the "Phase 2.5" plan assumed MOBI/AZW3/DOCX/RTF would need server-side
parsing, so ParseEbook was written as the endpoint that would do it —
but it only ever returned 501 "not yet implemented", and no route in
router/reader.go ever pointed at it. No caller in Go, templates, or
web JS/TS ever existed either.
The eventual architecture never needed it: the web reader hands the
raw file to foliate-js and parses client-side (MOBI renders via the
fork's mobi.js — verified live; DOCX via mammoth; RTF/PDB shipped).
By now it was also stale — it 501'd DOCX/RTF as "not yet implemented"
while both work, and offered an AZW3 path that was dropped entirely.
Pure deletion: no route registration, no references, no tests, no
behavior change. go build ./... clean.
TestProgressWeb_KoboIntegration 500s'd on every markup with a
last-read-place bookmark: the panic-recovered stack was a nil pointer
dereference in LibraryService.ResolveMediaPath (s.db deref), reached
from KoboHandler.convertKoboCFIToStandard during kepub CFI conversion.
Two-layer bug:
1. Test harness (root cause): the router.Config built in
test_helpers_test.go omitted LibraryService, so router/sync.go's
SetLibraryService(cfg.LibraryService) stored a nil
*services.LibraryService in the interface field. Production wiring
(main.go:246) was always correct.
2. Latent trap (hardening): `h.libraryService != nil` at the
convertKoboCFIToStandard call site can never catch a typed nil —
an interface holding a nil *LibraryService is non-nil. ResolveMediaPath
now degrades with a "library service not configured" error instead
of panicking on a nil receiver; callers already handle the error
by skipping CFI conversion.
Harness-only fix, zero production behavior change: the endpoint now
returns a clean 200 with the position saved wherever kepub conversion
is unavailable instead of a panic-recovered 500.
Verified: TestProgressWeb_KoboIntegration passes all three subtests
(401 unauthenticated, ReadingSync→last-read-place percentage
preservation, last-read-place without prior ReadingSync) and on
immediate rerun — previously it failed every run.
The whole-body "should not display Manga Type / Scan Info" assertions
could never pass: every book-detail page for an admin carries the
edit modal, whose form labels ("Manga Type", "Scan Information") and
option texts ("Yes (Right to Left)") share vocabulary with the
metadata display block. This was pre-existing rot — the test never
ran to completion before today because an earlier panic killed the
process first.
Replaced with value-level checks that only the display block could
produce ("yes and right to left", "B&W"): absent for plain ebooks and
unknown-manga-type items, matching the templ guards in
book_detail.templ. Emoji-icon asserts dropped from the RTL/Story-Arc
positive steps — the redesigned comic badge renders text chips
without icons.
Symptom: repeated full-suite runs poisoned the dev database with
leaked rows (duplicate Prefs Scope libraries, colliding device names,
saved-filter name conflicts on rerun) because tests created rows but
never deleted them, and stale-binary runs left rows the new cleanup
could not match.
- test_helpers: library creation now registers t.Cleanup DELETE (the
leak root cause — createTestLibraryWithFolder never cleaned up).
Suite-wide cleanup passes for libraries/collections/devices/
saved-filters run both at setup (pre-pass against legacy residue)
and in t.Cleanup (no per-run residue). Preserve the admin's real
libraries; delete only convention-named test rows.
- setup seeds canonical system collections every run keyed on
query_type (idempotent — heals renames/stale tombstones), moved out
of the preserved-admin branch so it runs for every suite start.
- dashboard_integration: scope-testing library renamed into the
test-row convention; ordering assertions on the restored
"Want to Read" collection preserved.
- filters_test: filter names moved into the convention (rerun-safe —
name-uniqueness 409 no longer trips second runs); five .(string)
type assertions panic-proofed with require.True guards (a panic here
used to kill the whole test process mid-run, which is why a stack of
"never ran" failures surfaced at once later).
- device_test: device names moved into the convention (ILIKE %test%
hit them; duplicate-device-name conflicts gone).
The old model hard-deleted system collections and re-created them at
the tail, resurrecting stale dedups and breaking ordering. Redesign
(user-directed B-final):
- schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER —
the whole schema is re-applied on every boot via executeSchema).
- queries.sql: every collection list/read query filters
deleted_at IS NULL; new GetSystemCollectionByQueryType :one
(unfiltered, sees tombstones), TombstoneSystemCollection :exec,
ClearCollectionItems :exec, RestoreSystemCollectionRow :one
(in-place metadata reset + un-tombstone; collections have no
updated_at column — ResetUpdatedAt was a 500 caught by tests).
sqlc regenerated.
- collection_service.DeleteCollection: system collections are
tombstoned (preserving position/ownership); user collections keep
the hard delete. No ErrNoRows to the client (delete is idempotent).
- dashboard_service.RestoreSystemCollection: resolve by query_type;
found → in-place restore (ClearCollectionItems + metadata reset,
un-tombstoned); missing → provision fresh. "Want to Read"/etc.
stay at their home positions forever.
- handlers/collections.CreateCollection: name-taken is 409, not 500.
- handlers/auth.CreateDefaultCollectionsForUser: idempotent by
(user_id, query_type) — update-in-place when a row for that query
type exists (heals renames/tombstones), insert when missing; this
also fixes the duplicate-system-collections regression at source.
The drawer was the only top-bar tool without a keyboard shortcut, so the
shortcuts & gestures help modal had to describe it as '📝 button'. Bind
plain 'a' to toggleBookmarks(), following the same pattern as the
existing t/s/b letter shortcuts.
- reader.ts: new 'a' branch in handleKeydown, guarded against
Ctrl/Cmd/Alt so Select All (a common reader action when selecting
text to highlight/copy) is never intercepted; typing guard already
protects note textareas, search box, and selects. Works in both the
host page and book-content iframes since both keydown listeners route
through handleKeydown.
- reader.templ: help modal row now shows <kbd>a</kbd> like the other
letter shortcuts; 📝 button tooltip gains '(a)' to match its
siblings (b), (/), (s), (?).
- reader_templ.go: regenerated.
- BUG: startBannerEdit extracted the note by splitting row.rich — the
RENDERED HTML — on \n\n, which the concatenated markup never
contains, so highlight-note dialogs always opened empty. Both row
kinds carry the raw note on bannerSources; use that lookup.
- Expanded body now splits at the highlight→note boundary: the
selection paragraph (left border in the row's highlight color), a
divider + pencil NOTE label, then the note markdown — instead of the
two texts flowing together.
- The note-indicator pencil (corner badge) now shows on ALL note rows,
not just highlights with attached notes.
Verified in headless Chromium: badge on note and highlight rows;
expanded highlight shows the bordered selection, NOTE label, and note
markdown; the highlight edit dialog prefills with the note text.
The test helper createDefaultCollectionsForUser carried its own
kebab-case copy of the default system collections — drifted from the
canonical Title-case list the registration seed and
RestoreSystemCollection use. Every suite run against the shared dev DB
re-created a kebab set beside the canonical one (the duplicate sections
on the customize screens); the dev DB is cleaned and the helper now
uses the canonical list.
Two stale expectations corrected while re-running the suite:
TestGetSections_EndToEndFlow asserted the kebab ids (now asserts the
Title-case names — section ids ARE collection names), and
TestGetSections_MissingLibraryID expected a 400 the handler never
implemented — absent library_id is the All Libraries view by design
(now asserts 200 + aggregated sections).
The All Libraries entry in the library switcher could render the
cross-library dashboard but rejected its own settings: the modal
refused to open without a library ('Select a specific library...') and
a save sent library_id: '' (server 400). The server now stores a
global scope (separate table; null/absent library_id, empty string
still 400), so:
- openDashboardSettings allows the All view and fetches prefs without
the library param (reads the global row; previously a loose NULL
OR-clause matched an arbitrary library's row).
- saveDashboardSettings sends library_id: null from All Libraries.
- The settings modal gains a scope label (#dashboard-scope-label)
showing which library — or All Libraries — a save will customize
(user requirement: always know the scope).
Verified end-to-end in headless Chromium: reorder on All Libraries →
save (null) → reload → modal re-opens with the persisted order and the
All Libraries scope label; per-library save lands in its own row
without touching global (SQL-checked).
The dashboard's All Libraries view could render sections across every
library but could not persist its own preferences: the PUT required a
UUID library_id, and the prefs GET-without-library_id matched an
arbitrary library's row through a loose NULL OR-clause.
Storage: a dedicated user_dashboard_preferences_global table (one row
per user, user_id PK → users CASCADE) instead of a NULL/sentinel
library_id — chosen over both because (a) the column's FK to libraries
rules out a sentinel UUID without dropping the constraint or minting a
fake library row, and (b) intent becomes structural: only the explicit
global path writes that table, so a buggy or outside-the-API write can
never masquerade as global. System and user collections are
user-scoped, so the order/hidden arrays are library-independent and
valid in this scope.
API: PUT /api/dashboard/preferences binds library_id as *string —
JSON null/absent = global, empty string = 400 (a lost value must stay
loud, not silently clobber the global scope), valid UUID = per-library
(unchanged). GET without library_id now returns the global row instead
of an arbitrary one; the per-library query's dead NULL OR-arm is
removed. GET /sections keeps requiring library_id for per-library
views and aggregates across libraries when absent (unchanged).
Tests: global save/update-dedupe/read round-trip, empty-string 400,
global-vs-per-library independence. Two suite failures are the
handoff's documented pre-existing rot (duplicate system collections in
the dev DB; a sections test expecting a 400 the handler never
implemented) — reproduced and confirmed unrelated to this change.
Live-verified on the dev stack: null save 200 x2 with a single row
updated, empty string 400, global read carries the second save.
Highlights with an attached note get a small edit-note stroke SVG
(same note-pencil path as the selection popover's Note affordance) in
the card's upper-right corner — themed via var(--text-secondary),
pointer-events: none so taps pass through, header gains pr-8 on those
rows so wrapped text never runs under it. Placement matches the
Android banner's corner badge (user preference: never left of text).
Regenerated annotations_banner_templ.go from the repo root.
Tailwind preflight zeroes p/blockquote/list margins globally, so notes
rendered through renderNoteHtml collapsed flat: blank lines between
paragraphs vanished, blockquotes were indistinguishable from body text,
lists had no bullets/indent, code blocks had no padding. The parser was
always correct — the loss was CSS at the x-html display sites.
One shared scoped block (.note-rich, in @layer components so it
outranks base preflight): p+p margin-top, blockquote bar + indent +
secondary color, list markers + padding, pre/code padding + themed bg,
modest heading sizes, underlined links, table borders, em-based so it
scales with each site's text-xs/text-sm context. Applied at all three
display sites: reader notes-drawer rows (replaces the unused
reader-note-rich hook), the 📝 highlight note line, and the banner's
expanded body.
The regenerated _templ.go files also catch up two stale .templ changes
that had shipped without regeneration (the selection popover's Adjust
button, the banner delete ternary) — the Docker build regenerates at
image-build, so the served site always matched the sources.
Verified live (headless Chromium against the dev stack): .note-rich in
the served stylesheet; p+p computed marginTop 9.6px, blockquote 3px
solid border-left, ul disc + 22.4px indent, pre padded + themed bg.
Objective 9 (web half). The selection popover's edit mode gains an
Adjust button: the overlay's show-annotation event stashes the live DOM
range, start/end teardrop handles render over it (host layer above the
iframe), and pointer drags map into the iframe's caretRangeFromPoint.
Snapping: land next to punctuation → the boundary sits exactly there;
land inside a word → the word's edge in the drag direction; whitespace
trims toward the adjacent word. A live preview repaints through the
overlayer under a scratch key; relocation (page turn) cancels safely.
Release recomputes the range CFI + collapsed-end CFI (view.getCFI) and
saves through the popover's own saveHighlightChanges path (PUT +
remove-old/add-new overlay dance), keeping selection_text and the
anchors in lockstep.
Verified live E2E in Brave: extent extended by one word with the
comma/space snap behavior, saved to the server row, handles cleaned;
all snap paths unit-exercised in-page (word end/start, punctuation
exact on both sides, whitespace trim); the test highlight was restored
to its original extent through the same adjust flow (punctuation-precision
drag back to the comma).
The book-detail banner's highlight rows gain Delete (tombstoned
server-side, restorable from the Deleted modal — same as the reader
drawers already offered). The deleted-annotations modal now says the
retention in plain language — 'kept for 30 days' from the
annotation_tombstone_ttl_days tunable (threaded through BookDetail →
NotesHighlightsModal; 1-day singular and retention-disabled variants
handled) — instead of referencing a settings name a reader cannot see.
Objective 8a (web half). Reader notes-drawer rows gain an inline editor
(mirroring the bookmark-rename pattern) that PUTs the new content — the
stored position rides along because the server's note PUT hard-sets the
position column (a probe note proved an omitted position NULLs the
anchor). Book-detail banner rows gain Edit/Delete for location notes
and Edit-note/Add-note for the note ATTACHED to a highlight (full-replace
highlight PUT; empty content clears the note; the highlight itself is
never edited from the banner). The edit dialog is centered with the
markdown caption.
The book-detail 'Notes' button becomes 'Deleted' (badge =
deleted-annotation count) and the modal title becomes 'Deleted
annotations' — that modal only ever showed removed annotations.
Verified live in Brave: drawer inline edit round-trips content and
position; banner note edit, highlight-note edit, and delete (tombstone
appears in the Deleted modal); rename renders with the count badge.
Objective 7 (web half). The 'Annotations' button slides a right-hand
banner (annotations_banner.templ + annotations-banner.ts) fed by the
existing per-book GET endpoints: friendly labels (Section N parsed from
the CFI spine / Page N from the PDF envelope / NN% + first-words
fallbacks), capped highlight previews, rich x-html expansion through the
objective-6 markdown pipeline, and per-row 'Open reader here' jumps
riding /readers/:uuid?loc= (consumeLocJump: jump outranks stored
progress for that open; URL stripped before sync can echo it). WEB-ONLY
export: single .md or one-file-per-item zip via fflate — item format
identical both modes ('# friendly label' / '# selection text' headers,
raw markdown bodies, no raw CFIs anywhere).
Verified live in Brave: 12 restored highlights + a markdown note
labeled/expanded correctly; the jump navigated, landed on the right
chapter and consumed ?loc=; export produced the 13-item .md (friendly
headers only) and a valid 13-entry zip. Book-detail's existing
recently-deleted modal stays untouched (banner is additive).
Objective 6 (web half). Storage/sync stay raw markdown (KOReader shows
literal source, accepted). Notes-grade syntax: headings, emphasis/
strikethrough, code, links, lists, blockquotes, GFM tables, reference
links, footnotes (markdown-it-footnote); images/raw HTML/math excluded
(html:false posture + FORBID img/style/form + default safe-scheme URI
check). Display sites: annotations-drawer note rows and highlight note
lines render pre-sanitized HTML (x-html); render-time sanitize only.
Rich paste: both note textareas intercept paste; a text/html clipboard
flavor converts via turndown (gfm tables incl. a headingless-table rule
emitting pipe syntax; images blanked) and inserts at the caret; plain
pastics fall through to the default paste unchanged.
Verified live in Brave: bold/italic/strike/code/lists/quote/table/
footnote-ref/reference-link render; <script> renders inert literal
text with no alert; javascript: hrefs absent from the DOM; paste of
rich HTML lands markdown at the caret; plain pastes untouched.
One boolean (Security > Public Registration) switches the whole public
signup lifecycle: front page + login links, the logged-out sidebar's
Create-an-account (via a package-level templates hook so shared page
templates keep their signatures), GET /register -> 302 /login, and an
early 403 on POST /api/auth/register. First-user exception preserved:
zero admins keeps every route and link reachable for bootstrapping —
same 'users exist' reasoning as the setup gate. Admin user creation is
unaffected by design.
Live E2E verified: flag off hides all surfaces and blocks the POST
(403), flag on restores them (201); the admin UI renders the row
automatically.
The unqualified 0.0.0.0 publish put Postgres on every interface, and
Docker delivers published ports through PREROUTING DNAT into the
FORWARD path — ufw's default deny incoming never sees those packets.
Result: the database was reachable from whatever network the laptop
joined (home, guest Wi-Fi, hotel), not just from the host.
Bind to 127.0.0.1/[::1] instead: with no DNAT matching LAN-destined
packets, they fall back to INPUT where the firewall actually applies.
Host-side tools keep working over the loopback publish (both families
bound because localhost may resolve to ::1 first); app↔db and tests↔db
are untouched — they use the db service name on the compose network,
which never traverses iptables on this host (br_netfilter not loaded).
If LAN access to the DB is ever wanted again, revert to an unqualified
publish and rely on DOCKER-USER home-subnet scoping instead of an
open binding.
.IMAGE — extensions removed from AllowedExtensions (scan gate),
bookExtensions, MimeTypes and sync/format.go maps: .doc/.lit files
are no longer scanned or indexed. classifyFormatGroup gains the
RTF and PDB reflowable arms that were missing when those rungs
shipped (rows only reclassify on creation). Scanner docs updated.
DOC + LIT have no credible JS tooling (mammoth is docx-only; LIT
needs LZX and its DRM variants are dead) and text-extraction-only
support would misrepresent what the reader can do. Verified: probe
.doc/.lit files watched but never scanned; probe .txt control
scanned; TestClassifyFormatGroup extended and passing.
Docs change granted in-session by the user (extensions + format
docs).
.docx is accepted for ingest but the format-group switch had no case
for it, so every DOCX row stored format_group 'unknown' with
is_reflowable=false. Extract the classification switch into a pure
classifyFormatGroup helper with a table test, and classify .docx
reflowable like the other reflowable text formats (.rtf/.doc stay out
until they are readable client-side). Existing rows: dev alice (DOCX)
reclassified by hand; production rows reclassify on rescan.
Oversized notes previously failed only at save time, surfacing the
raw backend validator error to the reader. The reader now handles
the limit inline and keeps oversized input editable:
- Live character counter under both note textareas (annotations
drawer "Add a note" and the selection popover note editor),
formatted "12,345/100,000" — muted normally, red once over
- Pasting is never truncated; the full text stays in the textarea
so the reader can shrink it however they see fit
- The three save actions (Add Note, Highlight with note, Save
note) disable while over the limit, with a toast fallback in
addNote / createHighlight / saveHighlightChanges so the guard
holds even outside the disabled-button path
- noteMaxLength mirrors the server-side cap (100,000, see
internal/handlers/media.go) so client and server stay linked
Also truncate note rows in the drawer list — highlight rows
already truncated, but a long note previously stretched the
drawer body.
The resize handle needed no work: .reader-note-input already
ships resize: vertical in the compiled CSS; style.css is rebuilt
only for the new disabled:* utility classes.
The 10k validator cap rejected legitimate long-form notes — a
scholarly reading note with quoted passages and footnotes lands
around 10.1k chars and failed at save with the raw validator error.
Raise the cap to 100,000 on all six annotation text fields in the
media handler request structs:
- CreateMediaNoteRequest / UpdateMediaNoteRequest Content
- CreateMediaHighlightRequest / UpdateMediaHighlightRequest NoteText
- CreateMediaBookmarkRequest / UpdateMediaBookmarkRequest Notes
No other layer changes: the media_notes / media_highlights columns
are unbounded TEXT, and the KOReader + websocket sync paths never
had a length cap, so the REST API now matches the rest of the
system instead of being the strictest gate.
New integration test pins the boundary: 50k and 100k-char notes
return 201, 100,001 chars returns 400.
AZW3/KF8 is a proprietary legacy Amazon format (the pipeline moved to
KFX in 2015; KDP dropped MOBI-family uploads in 2022) and does not
render in the web reader. The scanner still ingests .azw/.azw3 files
(code unchanged) — they are simply no longer documented as a supported
expectation.
makeTextBook: UTF-8 decode, blank-line paragraph split, hard-wrapped
lines joined, HTML-escaped, ~150KB section cap at paragraph boundaries
(Part N TOC). Enables the web reader for .txt (previously ingested but
unreadable client-side). Positions for txt books are percentage-based
(sections get index-derived fake CFIs).
Android 16+ gates per-app access to local networks behind a runtime
permission; without it the app's LAN logins time out with zero packets
leaving the phone (browser works — it is not per-app-gated). The new
device guide covers the sideload install, first-run server URL, the
permission prompt (and manual re-grant path), and a triage table for
the login screen's error line. Per the user: overlay-VPN addresses are
not a consideration and stay undocumented.
GetProfile returned the theme from the JWT-claims context stub, which
carries only id/email/username/role — user.Theme was always empty, so
the profile response omitted the field and the app's account-theme
read-back (SERVER sync mode) could never see the stored theme. GetProfile
now loads the full user row by id (h.db.GetUser).
The Android app's app-chrome theme (UX pass item 6) reads the account
theme from GET /api/auth/profile and re-applies it at startup in
SERVER sync mode. PUT /api/auth/theme already persisted the value;
this completes the round trip on the profile read.
Adds publisher_filter to SearchMediaItems/SearchMediaItemsUnified
(fuzzy word_similarity against mi.publisher, mirroring genre_filter),
plumbs it through services.SearchParams and the search handler, and
extends the GREATEST relevance ranking to include publisher matches.
Serves the app's author/publisher/genre/tag click-through browses
(UX pass item 5 — publisher was the only facet without a server
filter).
countTextCharsBefore started its sibling walk at the target node itself
and recursed after counting the current node's subtree, so every
ancestor level re-counted everything accumulated so far — a node at
depth 4 in the single-document 1984 epub reported 2.38M chars before it
in a 589k-char document, producing healed percentages of 4.04 (>1) and
a 500 on the reading_progress percentage check constraint. Every
healed progress save from a fresh client failed; only exact-context
matches (no heal) stored.
The walk now starts at target.PrevSibling: strictly the characters
before the node, per the contract all three callers already assume
(healed book offsets, CRE convert percentage, kepub offsets).
Found by the real-phone validation pass: the phone's first healed
submission 500ed where the emulator's had matched context exactly and
never taken the heal path.
The progress documentation described GET/POST /api/progress/:id —
endpoints that do not exist in the router — while the endpoint every
client actually uses (GET/PUT/DELETE /api/media-items/:id/progress) had
no field-level docs at all.
New:
- progress/position-contract.md: the canonical position model — server
as position authority, the three-tier submission (percentage /
context_text / epubcfi), ingest verification and healing, the restore
handles, the OPF spine numbering hazard (canonical CFI spine steps
include linear="no" items; clients resolve documents by anchor_href
and land by css_selector + char_offset, never by spine step), the
offset currencies (UTF-16 at the wire, runes internal), and
context_text rules.
- progress/get_media_progress.md and update_media_progress.md: the real
endpoints with full field tables, conditionality of the restore
handles, the first-page anti-clobber guard, and healed-response
semantics.
- progress/delete_media_progress.md: the real DELETE route.
- koreader/sync_progress.md and koreader/get_metadata.md rewritten to
the actual payloads: the plugin sends a single-book array whose
"epubcfi" field is a CRE xpointer; the metadata response navigates
via koreader_xpointer (canonical CFI converted back to CRE), with
page as the canonical locator for fixed-layout books.
Removed: the five files documenting the nonexistent /api/progress/:id
GET/POST/DELETE endpoints. Kept get_progress_history.md (that route
exists). The legacy developer/api-reference.md and the indexed
api/api-reference.md progress sections now match the wire and link the
new docs; the duplicate "Universal Progress" section points at Reading
Progress.
Offset currency policy, now explicit: EPUB CFI terminals, CRE text()
offsets and the served char_offset handle are UTF-16 code units (the
EPUB CFI spec, and what foliate/readium/KOReader/Kobo clients actually
observe), while internal arithmetic — the book-wide character_offset
column and percentage fractions — stays rune-based, consistent with
TotalCharacters. For all-BMP books the currencies are identical, so no
stored value changes; astral-plane text (emoji, rare CJK) no longer
drifts.
Boundaries converted: resolveCFIToNode interprets incoming CFI terminal
offsets as UTF-16; textNodeAtUTF16Offset (née textNodeAtRuneOffset)
interprets CRE text() offsets as UTF-16; buildCFI and buildCREXPointer
emit UTF-16 terminals; blockCharOffset (the served char_offset) is
UTF-16.
Also fixes two character_offset column defects: heals wrote a BLOCK-
relative offset into the book-wide column, and verified-but-unhealed
saves (e.g. KOReader pushes) never refreshed it, leaving it stale
behind the anchor. VerifyProgressAnchor now returns the verified book-
wide rune offset and SaveProgress refreshes the column on every
verified save.
Tests: astral currency round trip (offset after an emoji must shift by
one unit between currencies, in both heal and exact-verify directions)
and book-offset ordering. The cmd/server/tests integration harness
failures under docker (library folder 400 during setup) reproduce on
the pre-change tree and are unrelated.
Progress submissions now carry (percentage, context_text, epubcfi) and
the server becomes the position authority:
- VerifyProgressAnchor resolves the submitted standard CFI against the
book's own XHTML, extracts the text at the anchor, and cross-checks it
with the submitted context_text. A mismatch heals the anchor by text
search (percentage disambiguates repeats) instead of storing a bad
position.
- The anchor's block element is derived as a cssSelector plus a block-
relative character offset, and served on progress GET alongside the
anchor document's href — readium-native handles that let clients
re-open a book without parsing CFIs themselves.
- context_text-only submissions (no CFI — the dumb-client tier) are
anchored structurally from the context text.
Motivation: cross-client progress sync (web foliate CFIs, KOReader CRE
xpointers, readium-native apps) previously trusted each client's own
locator math; the app's EPUB restore drifted ±pages because readium's
paginator does not lay out far-from-viewport columns and the foliate-
ported CFI walk ran against readium's mutated WebView DOM. Server-side
verification heals both classes at ingest.
The version tag previously existed only in git and the Docker image tag —
the running app had no way to report what build it was. Version is now
injected at build time via -ldflags into internal/version.Version (defaults
to "dev" for local builds), passed by the release workflow as the
APP_VERSION Docker build arg from the pushed tag.
Surfaced in Admin → Settings → About (extensible card for future rows like
disk space) and in the /health JSON response, so deployments can be
verified with curl alone.
Regenerates all *_templ.go files after the templ generate pass (version
stamp updates across the board; functional changes only in
admin_archived_templ.go and admin_library_templ.go, which were committed
with their features). web/static/style.css and web/static/htmx.min.js are
refreshed outputs of npm run build (tailwind pass over the current
templates; htmx copied from the current node_modules version).
setupTestServer reuses the shared dev admin (testuser@tests.bookhoard.internal)
instead of re-inserting it, but never reset its password — once any test
mutated the admin's password, every later test in the run failed to log in
with 401s until the database was manually wiped. The seeding step now
resets the password hash to the known test constant on reuse, so full
integration runs are repeatable against an existing database.
Adds cross_library_move_test.go, seven integration tests covering the
duplicate-content features end to end:
- TestCrossLibraryMovePreservesHistory: a book moved between two same-type
libraries keeps its row ID — progress and annotations survive, no
duplicate, no archived ghost, library_type_name stays truthful.
- TestCrossLibraryCopyStaysIndependent: deliberate copies in two libraries
stay independent rows with isolated progress.
- TestCrossLibraryMoveRejectedForTypeMismatch: a reflowable EPUB is not
repointed between manga libraries; a library_type_mismatch processing
issue is recorded instead.
- TestListHiddenMediaItemsMatchesActiveTwin: archived rows expose their
active same-SHA twin (and rows without one report no match).
- TestMergeArchivedItemIntoActiveTwin: merging moves progress and
annotations onto the active copy and removes the archived row.
- TestMergeArchivedItemRejections: non-admin 403, active source, hash
mismatch, missing target_id, self-merge, and archived target all fail
closed.
- TestLibraryTypeNameTriggerOnLibraryChange: the UPDATE trigger refreshes
library_type_name when a row changes libraries.
Tests create their own temp-dir library folders (host runs don't have
/app/uploads), use unique device identifiers (device_identifier is
UNIQUE and leftover rows broke reruns), and clean up via an explicit
defer that runs while the pool is still open — a t.Cleanup registered
for the same purpose silently no-ops because it executes after
setup.Close has closed the pool.
The test-runner stage copied only /app from the builder, but the Go module
cache lives in /go/pkg/mod (GOMODCACHE) — the cache mounts used by the
builder target /root/go/pkg/mod, which the go tool ignores, so those mounts
never held anything. Every `compose run tests` / make test-integration
invocation therefore re-downloaded all dependencies from the network,
making integration runs slow and timeout-prone.
The builder now materializes /go/pkg/mod into an image layer
(go-module-cache) after the binary build, and the test-runner stage
restores it to /go/pkg/mod before running tests. Test-only stage; the
production final stage is unaffected.
Three fixes to the Delete Library flow on /admin/library:
1. Honest warning copy. The old text ("Media files will not be deleted")
read as reassurance while deleting a library actually cascade-destroys
every book record with it: reading progress, highlights, notes,
bookmarks, and ratings are gone with no archive window and no undo.
The modal now states this as a scannable list:
- Permanently removed, no archive or undo: the library, its folder
mappings, and all book records — with their reading progress,
highlights, notes, bookmarks, and ratings.
- Not touched: media files on disk.
Admins skim danger dialogs; the irreversible part now leads.
2. The dialog stayed open after confirming. The confirm button swaps
#libraries-container via htmx (so the deleted row vanished) but the
modal lives outside the swapped container and nothing closed it. An
htmx:afterRequest listener now hides the modal on successful deletes
and leaves it open on errors.
3. Latent ReferenceError in openFolderBrowser: the function parameter is
targetInputId but the body referenced an undefined targetInput, so
opening the folder browser threw and the picker never loaded.
The device catalog flattened every visible library into one list, so
duplicate copies of the same book each got an entry and search had no
library context. The root feed is now a standard OPDS 1.1 navigation feed
that mirrors the web UI's library model:
- Root /catalog: an "All Books" entry first (the previous flat cross-
library behavior, also still served at ?all=1 for clients that want
the single flat list), followed by one folder entry per visible
library with live book counts from GetVisibleLibraryMediaCounts.
- New GET /library/:libraryId/catalog: acquisition feed scoped to one
visible library (403 when the device owner cannot see it), paginated,
with an up-link back to the root.
- Scoped search: each library feed's rel="search" OpenSearch template
pins &library_id=<id>, so opening search from inside a library folder
searches only that library — clients substitute only {searchTerms},
so no client-side changes are required. Root search stays global.
- Global search entries now carry the owning library as a category (and
as a fallback summary when the book has no description), so duplicate
copies are distinguishable in unscoped result lists.
The acquisition entry builder is extracted into addAcquisitionEntries and
shared by the all-books and per-library feeds. bookhoard.koplugin needs
no changes: it only registers the root URL, and KOReader's stock OPDS
client renders navigation feeds natively.
Tests: TestOPDSLibraryFolders covers the nav-feed shape, flat ?all=1
mode, scoped catalog isolation, scoped/global search behavior, and 403s
for libraries outside the device's visibility. Library names avoid the
word "test" on purpose — setupDeviceTest re-runs setupTestServer's
setup-time cleanup, which deletes every library whose name contains it.
Covers the copy-then-delete-later workflow: a user copies books to a new
library, deletes the originals, and ends up with an active copy in the new
library plus an archived twin holding the real reading history. Until now
those twins could only be purged (destroying the history) or restored
(showing a permanently broken entry).
- POST /api/media-items/:id/merge (admin only, body {target_id}):
validates the source is archived/missing, the target is active, and
both share the same file_sha256; then re-parents every child row onto
the target via the existing reparent_media_item_children function (the
same machinery as hash-conflict resolution) and deletes the source row.
Per-user collisions keep the active copy's data, mirroring that flow.
Affected data: reading progress, speed, ratings, highlights, notes,
bookmarks (including tombstoned deleted-annotation history), formats,
collections, kobo shelves/entitlements, sync rows, panel data,
processing issues, and device aliases.
- GET /admin/archived: ListHiddenMediaItems' match_* columns surface each
row's best active twin; rows with a twin get a confirm-guarded "Merge"
button (data-merge-source/-target) next to Restore/Delete, wired in
web/src/admin.ts like the existing unarchive/delete handlers.
- templates.ArchivedItem gains MatchID/MatchTitle/MatchLibraryName;
frontend.go populates them from the listing row.
After a merge the archived row is gone, so the retention purge can never
destroy the merged data. Deleted-annotation tombstones carry over and
remain restorable from the target book's "recently deleted" history.
Previously the scanner's SHA-256 dedup was library-scoped: moving a book
between libraries created a duplicate row (new ID) while the old row went
missing and archived after two scans, orphaning reading progress and
annotations from the file that users still see.
processMediaFile now falls through to cross-library detection when the
same-library hash lookup misses:
- GetMediaItemsBySHA256AndSameLibraryType returns candidates in other
same-type libraries; each candidate's file is stat'd through ITS OWN
library's folders (the old existence check stat'd against the current
scan's folder tree, which is meaningless across libraries).
- File gone -> it is a move: the candidate must pass the target library's
type rules (ValidateMediaItemForLibrary), then the row is repointed via
MoveMediaItemToLibrary with the recomputed relative path. Reading
history, annotations, and collections follow automatically because the
row keeps its ID. Incompatible formats (e.g. a reflowable EPUB into a
manga library) are rejected with a processing issue instead of being
force-imported; the upsert on (media_item_id, issue_type) keeps
repeated scans from spamming duplicates.
- File still present -> deliberate multi-library copy: fall through to
normal import so both libraries keep independent rows.
Candidate selection is factored into selectMoveCandidate (pure function,
unit-tested in media_scanner_move_test.go): input arrives pre-ordered
(archived first, then most missing scans, then oldest) and only
candidates whose file is verifiably gone qualify, so deliberate copies
are never repointed.
Three related query groups in queries.sql (sqlc regenerated; no generated
signatures changed, so no caller edits were needed):
1. Dashboard/collection archive filters. The smart-section and collection
queries returned archived and missing items (visible only as broken
covers once their files vanished). Added the codebase-standard
"AND archived_at IS NULL AND missing_scan_count = 0" predicate to:
GetContinueReadingItems, GetRecentlyAddedItems, GetRecentlyReadItems,
GetNotStartedItems, GetCollectionItemsForDashboard, GetLibraryItems.
Matches the existing convention in ListMediaItems, SearchMediaItems,
the library counts, and GetContinueSeriesItems.
2. Cross-library SHA move detection support.
- GetMediaItemsBySHA256AndSameLibraryType: finds identical content
(same file_sha256) registered in a DIFFERENT library of the SAME
library type, ordered archived-first, most-missing, oldest. Type
scoping keeps ebooks from merging into manga/comics libraries.
- MoveMediaItemToLibrary: repoints an existing row to the new library
(library_id + recomputed file_path + file_size) and clears
archive/missing state. The row keeps its ID, so reading progress,
annotations, collections, and kobo shelves follow the book
automatically; the new set_library_type_name_on_library_change
trigger refreshes the denormalized type column.
3. Archived-twin matching. ListHiddenMediaItems (admin archived-items
page) now LEFT JOIN LATERALs the best active same-SHA twin per hidden
row — preferring same library type, then same library, then most
reading progress, then oldest — exposing match_id / match_title /
match_library_name so the UI can offer merging the archived row's
reading data into its active duplicate. COALESCE on match_title keeps
the no-match case NULL-safe.
schema.sql is re-executed in full on every application startup, so every
statement in it must be idempotent. Two related problems introduced with the
cross-library move feature, plus their fix:
Problem 1: media_items.library_type_name was only populated by a BEFORE
INSERT trigger. When the scanner repoints an existing row to a different
library (cross-library move detection), the denormalized library_type_name
went stale, mislabeling the item's type for validation and display.
Fix: add a second trigger, set_library_type_name_on_library_change, that
fires BEFORE UPDATE OF library_id and re-runs the same population function.
Problem 2 (outage): the new trigger EXECUTE FUNCTIONs set_library_type_name(),
but the pre-existing idempotency dance drops that function on every startup
before recreating it. On the second and later boots, DROP FUNCTION failed
with SQLSTATE 2BP01 (function still depended on by the trigger created by
the previous boot), aborting the whole schema transaction and crash-looping
the container.
Fix: drop BOTH triggers before dropping the function, and create both after
it. The ordering now survives any number of restarts on any database state.
Adds TestSchemaInitializationIsIdempotent, which replays the production
database.Initialize twice against the same database so this class of
"second startup" regression fails in CI instead of in production. Note the
test uses a dedicated pgxpool: Initialize holds the advisory lock on one
connection while executing the schema on another, which deadlocks against
the shared test pool's MaxConns=1.
go mod download all added sums for modules only reachable from test
builds (chroma, bluemonday, compress families); without them the
integration-test binary build fetches missing sums at compile time.
Two pre-existing harness breaks found while running the device tests:
- RegisterRoutes dereferences cfg.Settings (auth rate limit) but
setupTestServer never set it — every integration test nil-panicked at
route registration. Wire database.NewSettingsRegistry(queries) the
same way main.go does.
- The cleanup deliberately PRESERVES testuser@tests.bookhoard.internal
(shared dev admin), but setup then blindly re-INSERTed that user, so
every run after the first failed on users_email_key. Reuse the user
when it exists; default collections are created only for a NEW user
(the preserved admin already has its set).
Also documented the offline runner used on slow links: build the test
binary on the host (CGO_ENABLED=0 go test -c) and execute it inside the
bookhoard-tests image against the compose network — no in-container
module downloads, no image rebuild.
The mutex hardening added a fall-through 'pending' response path that
returned while still holding pendingMu — one status poll before approval
leaked the lock and every later register/approve/status request hung
forever (caught by TestDeviceRegistrationFlow's approve step hanging).
App reinstalls that preserve data (Android Studio installDebug over an
existing install) re-register with the same device_identifier, but the
devices row from the previous install still exists — device_identifier
is UNIQUE, so ApproveDevice's blind INSERT failed with a unique
violation and returned 500 'failed to create device' (reproduced via
curl: second approve with the same identifier = instant 500; the ~98s
in the original report was app-side retry/polling, not server wait).
ApproveDevice is now idempotent: look the device up by identifier
first; a row owned by the approving user gets its auth token rotated
via UpdateDeviceAuthToken (row id unchanged, so synced highlights/
bookmarks/progress anchored to it stay valid; fresh install = fresh
credentials, old token invalidated); a row owned by another user gets
409; unknown identifiers INSERT as before, with the 23505 race falling
through to the rotate path. DB failures are logged (they were silent).
Also guard the in-memory pendingRegistrations map with a mutex —
register/approve/reject/status/list all touch it from HTTP goroutines,
and a racing write is a Go runtime fatal, not an error. The approver's
credential publication and the status poller's approved-branch snapshot
now run under the lock so the token can never be read half-written.
Regression test: TestApproveDeviceReapprovalRotatesToken — register →
approve → re-register same identifier → approve (must be 200) → token
rotated, exactly one devices row, row carries the new token.
User-facing documentation for the rebuilt web reader (§6.5 of the app
handoff): swipe-to-page / tap-for-menu touch navigation, long-press word
selection with drag extension and handles, the selection popover (color
dots, notes with save-together-on-create, copy incl. plain-HTTP LAN,
edit/delete by tapping a painted highlight), below-the-selection popover
placement rationale, PDF/comic behavior, and annotations drawer.
Linked from the user documentation portal and the docs index quick
links / quick-find tables.
The lockfile is gitignored (each machine keeps its own), but
.dockerignore did not exclude it, so any stale local lock rode into
every docker build via `COPY package*.json`. Because the forked
foliate-js declares "version": "0.0.0" on every commit, npm treats
the git pin as already satisfied by name@version and never
re-resolves the new commit hash — silently installing and bundling
the old code. This bit both the host npm cache mount (documented at
Dockerfile:18-20) and, today, `make rebuild-app-force`: a fresh
no-cache image was built with the pre-feature 1305a52 foliate-js
(chunk fixed-layout-B8-qRQLl.js) despite package.json pinning
e16530a, while the Gitea runner (fresh checkout, no lockfile, cold
cache) built correctly.
With no lockfile in the context, npm install resolves git pins
fresh from package.json each build (tarballs are cached by
commit-specific URLs), so the persistent npm cache mount cannot
serve old commits across pin bumps. Local lockfiles can no longer
poison builds even if regenerated on the host.
Verified: after evicting the poisoned cache mounts
(docker builder prune --filter type=exec.cachemount) and rebuilding,
the container serves fixed-layout-BE0KdOql.js with both dblclick
handlers present, matching the reference build.
Cancel only collapsed the editor section, and edit-mode saves only set
noteOpen=false — the minimized popover (colors/pencil/copy/delete) stayed
up. Explicit Save and Cancel now call hideSelectionPopover; color tweaks
with the editor open still keep it open.
- The in-book pointerdown dismiss listener lived in the reflowable-only
block, so PDF taps never closed the selection popover — hoist it so
every content doc (EPUB, PDF, comics) dismisses on tap.
- createHighlight hardcoded note_text: '' — 'Highlight with note' saved
the highlight but silently dropped the note. Send p.note (and pass it
to the EPUB overlayer).
- saveHighlightChanges collapsed the note editor on every save; color
dots in create mode created a highlight outright. With the editor
open, color dots now just recolor (create) or recolor-and-save with
the editor kept open (edit).
Chromium's touch selection takeover swallows pointerup in the fx iframe,
so the PDF popover's only trigger never fired on real phones — it only
appeared when timing happened to deliver the event. Mirror the EPUB
path: a selectionchange debounce (400ms settle) opens the popover, with
the pointerup path kept for desktop plus the quick-tap word-select guard.
Also place the popover BELOW the selection on coarse pointers (+90px
clearing the native selection menu and drag handles), matching the
reflowable path; desktop keeps above-placement.
GitHub-pinned builds silently reused a stale foliate-js tarball from
Docker's npm cache mount (name@version never changed from 0.0.0);
every build since the GitHub pin shipped pre-94bb384 code. Repin to
the mirror and bust the cache mount.
The fx renderer re-rendered the PDF on every viewport resize;
mobile browser toolbar transitions (7-17% height change) during a
text selection drag caused the old canvas to be cleared for
re-rendering while the async pdf.js render raced with the next
resize, blanking the page. Now gated by the same 25% threshold
as the paginator.
The fx renderer's gesture classifier only checked whether the touch
started on a .textLayer span; Chromium's long-press selects the
nearest word even when the finger landed between spans, so the
classifier saw a non-selectable target and classified the drag as
swipe/pan — preventDefault on the iframe's touchmove then cancelled
the native selection extension mid-drag and could blank the PDF
canvas. If any frame already has a non-collapsed selection, the
drag is now classified as native.
pokeChrome's anySelection gate only prevents the chrome from being
raised — if it was already visible when the selection began, it
stayed up. noteSelectionActivity now actively hides it the moment
a selection appears.
- Custom drag handles (start/end) for post-lift selection adjustment:
the native handles are disabled with the rest of the native touch
selection controller; these are positioned at the selection's
boundary carets and driven through the same clamped caret mapping
- Copy button falls back to execCommand via a transient textarea on
plain HTTP (navigator.clipboard is unavailable on LAN addresses);
both paths clear the selection and dismiss the popover
- Only dismiss the selection popover on actual position changes in
relocate (detail.section is a fresh object on every relocate, so
reference comparison always saw a move); a tap's no-op relocate
was closing the just-opened highlight edit popover
- Pin foliate-js f872a01 (synthetic click dispatch on quick taps)
and 422e8e0 (don't snap taps that never panned)
- tap zones: map in-iframe taps into the visible page slice (the
iframe is laid out at full section width; every tap previously
computed as the left margin) and shrink the default zone size to 12%
- hyphens: manual on coarse pointers: Chrome's touch word-selection
walks hyphen fragments, re-anchoring line-start drags and
overshooting line ends into the next column
- selection popover: opens for settled touch selections (the gesture
takeover swallows pointerup), positioned below the selection with
on-screen clamping; creating a highlight clears the selection so
the browser's own menu follows
- settle-time recovery: return the view to the selection's anchor
page and clamp the selection to the visible page after the
browser's selection auto-scroll wanders off the page grid
- overscroll-behavior: none on the reader page (pull-to-refresh
during downward drags)
Non-touch gate on selection auto-paging (drag-selecting no longer
pages away mid-gesture) and small height-only viewport resizes no
longer re-wrap the book (mobile browser toolbar transitions).
Fixed-layout EPUBs lean on the reading_direction column (the web reader
forces book.dir = rtl from it when the file didn't set direction
itself), but the scanner never populated it for EPUBs - only ComicInfo
fed it. Meanwhile real Japanese EPUBs declare page-progression-direction
on the OPF spine, which foliate reads client-side but nothing stored.
Read the spine attribute in the structured OPF parser and map it into
ReadingDirection in parseOPFContent (EPUB2/3, case-insensitive, plus
'right-to-left'/'left-to-right' spellings); undeclared stays empty
rather than forcing ltr, preserving the editor's Auto default. Sidecar
OPFs are metadata-only documents without spines, so the Calibre path is
a no-op. The merge gap-fill copies an embedded-only direction into a
blank sidecar field, and hand-set values keep winning through the
existing OverrideReadingDirection protection.
Tests: declared rtl/RTL/ltr, undeclared and unknown values staying
empty, plus sidecar-wins vs embedded-fills merge cases. Existing manga
EPUBs declaring rtl (verified live in-library) pick the value up on
their next scan, feeding the API and reader config mobile clients
consume.
Same gap-fill rule as the EPUB path, against the PDF's embedded Info
dictionary via a readPDFInfoDict helper reusing extractPDFMetadata's
field conventions (creator falls back to author, subject maps to
description, producer to publisher, keywords to tags, plus page count).
Sidecar values always win; unopenable PDFs skip silently.
TestMergeMetadataPDFGapFill uses an Info-bearing hand-built PDF fixture
(shared with the sidecar-cover test) and asserts the sidecar title is
kept while author/description/publisher/tags/page count fill in.
A sparse metadata.opf/metadata.json (title only, no description) left
books thin even when the file itself carried rich data: the embedded
extractors only ran when no sidecar existed at all. Now mergeMetadata
fills blanks from the book's own OPF - title, author, description,
publisher, language, ISBN, ASIN, series/number, publish date, tags,
contributors - while sidecar values always win and unparseable files
skip silently. Also covers .kepub, which the merge previously ignored
while the extractor already supported it.
Adds TestMergeMetadataEPUBGapFill asserting both directions: sidecar
title/author survive, embedded description/publisher/language fill in.
Admins need to see what the archive lifecycle is holding: a dedicated
/admin/archived page listing every hidden item (archived or still in
the missing grace window) with library, file path, status, and - when
retention is enabled - the exact date it will be permanently deleted
(archived_at + ARCHIVE_RETENTION_DAYS).
Per row: Restore (POST /api/media-items/:id/unarchive, clears the
archive state so it reappears; if the file is still gone the next scan
hides it again) and Delete (existing DELETE endpoint for single-row
purge with its reading history). Purge All Archived reuses the existing
bulk button. The library admin banner links to the page and keeps its
purge button; row actions use data attributes with delegated listeners
in admin.ts since this templ version has no JSFunctionCall helper.
Verified live: page 200 with purge dates shown, unarchive returned 204
and reset the row, per-item delete and bulk purge ({purged:1}) both
removed their rows with no leftovers.
Two visibility fixes so the UI reflects the shelf's real state on the
next scan instead of only after the archive gate:
- Missing items disappear at once: the user-facing filters already hid
archived rows; the same listings now also require missing_scan_count =
0. A deleted or moved-then-not-yet-repointed file vanishes from the
UI immediately, while purge timing stays gated on archived_at plus the
retention window - grace protects data, not visibility. Restored
automatically when the file returns.
- Steam Deck SD-card model for unmounted storage: resolveLibrary stats
each library's folder roots and flags libraries with no live folder
as Offline (LibraryData gains the field, computed at request time so
mounts/unmounts react instantly). The bookshelf shows an empty shelf
plus a 'storage is not connected' notice for an offline selected
library, and both the shared LibrarySwitcher and the bookshelf's
inline select label offline libraries with their true holding counts.
Nothing is marked or purged while offline.
- TotalMediaCount skips offline libraries, so the 'All Books/Libraries'
totals match what is actually visible.
Verified live: renaming uploads/Manga away produced the notice, an
empty shelf, and the offline dropdown label with a corrected total;
renaming it back restored all 38 cards with zero dirty rows.
When the SHA-256 dedup found identical content already in the library,
the scan skipped the file as a duplicate - and after files moved
between folders the old row kept its stale path, cycled missing ->
archived, and the new path never took. The archive feature turned the
old destructive move behavior into a stuck move instead.
Now the dedup branch stats the old location: if it is gone, the book
was MOVED, so the row is repointed (file_path, file_size) with archive
state cleared and reading history intact. 'Skip as duplicate' only
applies when the old path still exists (a true copy). Verified live: a
moved EPUB kept its single row, followed the file, and never entered
the missing/archive cycle.
Storage behind the scan lifecycle fixes:
- MoveMediaItemFilePath: repoints a row (path, size) and clears archive
state when identical content reappears at a new location.
- ListHiddenMediaItems: archived OR missing rows for the admin
archived-items page; ListMediaItemsByLibraryIncludingArchived (prior
commit) already fed the scanner.
- All user-facing listings now hide missing items immediately, not just
archived ones: ListMediaItems, ListMediaItemsByLibrary,
ListMediaItemsSorted, SearchMediaItems, SearchMediaItemsUnified, the
next_books CTE, library media counts, and the five search autocomplete
value queries gained AND mi.missing_scan_count = 0 next to the
archived_at filter. Purge timing is unchanged (archived_at +
ARCHIVE_RETENTION_DAYS), so the grace period keeps protecting data
while the UI reflects removals on the first scan.
- Detail lookups by id/path/sha stay unfiltered on purpose.
A full rescan wiped cover_image_path for every PDF/EPUB living in a
metadata.json (or metadata.opf-less) folder with no cover.jpg next to
the book: the sidecar branches returned early after findSidecarCover
missed, and mergeMetadata has no PDF/EPUB cover logic of its own. Four
books lost their thumbnails while their {file}.cover.jpg files still sat
on disk - most visibly the Audiobookshelf-managed No Starch titles.
Both sidecar branches now fall through to an embedded-cover fallback
(PDF via extractPDFCover, EPUB/KEPUB via extractEPUBCover) whenever no
sidecar cover file exists. Comics are untouched: mergeMetadata already
extracts their covers from the archive.
Adds TestSidecarCoverFallback with a hand-built one-page PDF carrying a
JPEG XObject plus a metadata.json sidecar, asserting the sidecar title
wins while the cover still comes from the file. Verified live: rescans
restored all four dereferenced covers with no leftover rows.
Reset to Scanned cleared the overrides row (successfully) but then set
the in-memory copy to nil before handing it to updateMediaItem. pgx
encodes a nil []string parameter as SQL NULL, so the follow-up UPDATE
wrote metadata_overrides = NULL into the column's NOT NULL constraint
and the whole rescan failed with:
failed to update media item: ERROR: null value in column
"metadata_overrides" of relation "media_items" violates not-null
constraint (SQLSTATE 23502)
Two changes:
- RescanMediaItem's reset path assigns []string{} instead of nil, with a
comment explaining the pgx nil-to-NULL encoding trap.
- updateMediaItem routes the override set through utils.MergeOverrides,
whose contract guarantees a non-nil slice, so no caller can write
NULL into that column again (verified against pgx v5.9.2 source: a
scanned '{}' round-trips as non-nil in both directions; the nil could
only come from our own assignment).
The plain Rescan path never hit this - only Reset did. Worse, the reset
is the remedy when a book's cover_image_path override pins an empty
cover, so the crash also blocked the way out of that state. After this
fix, a plain rescan on an already-reset book repopulates scanned
metadata and extracts the cover.
Adopt Calibre's reading conventions for the Dublin Core metadata that
parseOPFContent now pulls from the structured OPF parse:
- Titles: EPUB3 title-type selection (prefer 'main', join a distinct
subtitle with ': ' exactly as Calibre stores it). There is no separate
subtitle column by design - Calibre-sidecar books arrive pre-joined,
so a column would stay empty for most libraries and force every client
to reimplement concatenation.
- Genre: first dc:subject, mirroring the existing processGenresAndTags
behavior of the Calibre-sidecar path; the embedded path never
populated Genre before. Subjects stay one-element-one-tag - Library
of Congress headings legitimately contain commas ("Holmes, Sherlock
(Fictitious character) -- Fiction") and must not be split.
- Identifiers: urn:isbn:/urn:asin: prefixed values parse in addition to
opf:scheme attributes, and the scheme-less fallback now requires an
ISBN-shaped value (10/13 digits, optional separators/trailing X) so
URIs like the Gutenberg identifiers cannot masquerade as ISBNs -
observed live on 'A Study in Scarlet'.
- Series: EPUB3 belongs-to-collection with collection-type=series and
group-position refines, ahead of the classic calibre:series metas.
- Audiobookshelf metadata.json sidecars join their subtitle field into
the title the same way.
Tests cover title-type main+subtitle joining, belongs-to-collection
series with fractional group-position, urn:isbn extraction, genre/tag
parity, and comma preservation inside subject headings.
The cover lookup scraped the OPF with attribute-order-sensitive regexes.
Real books serialize attributes in any order - Grand Central's '3 Days to
Live' puts href before id on manifest items and content before name on
the cover meta - so all three regex paths missed and the book fell
through to filename guessing, extracting no cover at all. Attribute
order is meaningless in XML; the regexes were never safe.
Replace them with a structured parse (encoding/xml, namespace and
attribute-order agnostic; see the new media_scanner_opf.go) and follow
Calibre's read_raster_cover resolution order:
1. manifest item with properties=cover-image (non-(X)HTML media only)
2. <meta name=cover> resolved through the manifest, same media guard
3. first spine item that is itself a raster image (store manga)
4. NEW cover-page fallback: books declaring no raster cover at all -
the classic EPUB2/Adobe cover.xhtml wrapper - are mined for
<img src> / SVG <image xlink:href> references (Calibre renders the
page with Qt; extracting the referenced image covers the practical
cases without a rendering engine)
5. existing zip filename guessing stays as the last resort, and the old
regex chain survives as findCoverInOPFLegacy for OPFs too malformed
for a real XML parse.
Hrefs are now URL-decoded and posix-normalized against the OPF's own
path (path.Join semantics), so '../art/cover.jpg' from a nested cover
page and %20-encoded names resolve correctly.
Tests: attribute-order chaos modeled on the failing Patterson book,
SVG-wrapped cover pages via guide references, image-first spines, and
path resolution edge cases. Verified live against the real
'3 Days to Live' EPUB, which previously produced no cover.
Days an archived item is kept with its reading history before library
scans purge it for good (default 90). Set 0 to keep archived items until
purged manually from the library admin page.
Bulk escape hatch for archived rows (files missing from disk for 2+
scans) so a mass external deletion never has to wait out the retention
window or be clicked away row by row:
- POST /api/media-items/purge-archived (admin only) hard-deletes all
archived items and returns the purged count; reading history goes with
the rows, so the call is confirmed in the UI first.
- The library admin page shows an 'Archived items: N' card (only when
non-zero) with a Purge Archived Now button that calls the endpoint,
toasts the result, and reloads.
- Frontend admin JS exposes window.purgeArchivedItems following the
existing localStorage-bearer-token pattern.
Replace the silent hard-delete orphan cleanup (which logged only through
ScannerLogger file logs and whose failure paths left rows undetected)
with an archive lifecycle that preserves reading history:
- A file missing in one scan is marked (missing_scan_count = 1); missing
in a second consecutive scan archives it (archived_at, hidden from
browsing, progress/notes/highlights survive). Every branch logs to
stdout with an [ARCHIVE] prefix so skips are always visible.
- When a file reappears - same path, or identical content at a new path
via the SHA-256 dedup match - the archived state clears automatically
and the item returns with its history intact.
- Archived rows older than ARCHIVE_RETENTION_DAYS are hard-purged at
scan time (cascading deletes); 0 disables auto-purge for manual-only
management. Retention is read from the environment in NewMediaScanner.
Add the storage behind the archive-instead-of-delete lifecycle:
- media_items.missing_scan_count (INT, default 0) and archived_at
(TIMESTAMPTZ, partial index), both as idempotent ADD COLUMN IF NOT
EXISTS backfills for existing installs.
- MarkMediaItemMissing / ArchiveMediaItem / ClearMediaItemArchive plus
PurgeExpiredArchivedMediaItems (retention cutoff) and
PurgeAllArchivedMediaItems (manual bulk), with CountArchivedMediaItems
for the admin UI.
- Archived items are hidden from every user-facing listing:
ListMediaItems, ListMediaItemsByLibrary, ListMediaItemsSorted,
SearchMediaItems, SearchMediaItemsUnified, the next_books CTE, the
library media counts, and the search autocomplete value lists. Detail
lookups by id/path/sha are intentionally unfiltered, and a dedicated
ListMediaItemsByLibraryIncludingArchived feeds the scanner so the
lifecycle pass can see and restore archived rows.
Libraries managed by Audiobookshelf keep a metadata.json next to each
book (title, authors, series+sequence, genres/tags, publisher,
description, isbn/asin, language, published year/date) - and no
metadata.opf. The scanner silently ignored those files: deleting them
changed nothing, and their data never reached the database.
Parse them as a first-class sidecar in extractMetadata, priority
metadata.opf -> metadata.json -> embedded media. Only fields with a
matching media_items column are mapped; narrators, subtitle, explicit,
abridged, and chapters are deliberately skipped.
Cover handling is unchanged: the existing findSidecarCover priority
(cover.jpg / folder.jpg / {basename}.jpg) applies to the sidecar branch
exactly as it does for Calibre.
go-epub's ReadBook parses every spine chapter and fails the entire call
if any single chapter (or the TOC) is malformed, discarding already-parsed
OPF metadata. For books like Pragmatic's 'A Common-Sense Guide' the OPF
holds good title/author/publisher/ISBN metadata that rescans then wrote
as blanks - success toast, no (visible) change.
Refactor to parse the EPUB's own OPF document with the same Dublin Core
machinery used for Calibre sidecars: parseCalibreMetadataOPF is now a thin
file wrapper around a reusable parseOPFContent([]byte), and the OPF lookup
previously inline in extractEPUBCover is shared via findOPFPathInZip. Since
metadata never touches chapter bodies, chapter damage cannot blank it.
Also picked up along the way: dc:language mapping and scheme-less
dc:identifier values that normalize to a valid ISBN (EPUB3 style).
Tests cover a Pragmatic-style EPUB (dc namespace on <metadata>, no
identifier scheme, deliberately malformed chapter) that must still yield
full metadata, plus series/date/subject OPF parsing.
Complete the metadata editor modal to match the backend override work:
- File Path: read-only, monospace field showing the resolved on-disk
location (MediaDetail.FileLocation), next to Format and File Size, so
admins can see exactly which file backs the record without leaving the
editor. Renders empty for non-admins, who never receive the path.
- Reset to Scanned: new button beside Rescan. It confirms, then calls
POST /api/media-items/:id/rescan?reset_overrides=true to drop all
per-field user overrides and re-extract scanned defaults. Rescan alone
keeps customizations; Reset discards them.
Generate Cover has never worked: it fetched the book file using a URL
scraped from the cover preview <img> tag (so it downloaded either the
existing cover JPEG or, when no cover existed, the detail page HTML),
then handed it to foliate-js, which rejects both. Its fixed-layout path
also called view.renderer.renderPage(), a method that does not exist in
the pinned foliate fork. Every click ended in the same generic 'Cover
generation failed' toast.
The working alternative already exists server-side: the scanner's
PDF/EPUB cover extraction plus the per-book Rescan button, now that the
rasterizer renders the CropBox. Users who want a specific image can
upload one.
Delete web/src/cover-generator.ts, the modal buttons, and the dead
generateCover()/coverGenerating/fileUrl plumbing in book-detail.ts.
The book detail page exposed server internals and unusable controls to
everyday users: it now computes and renders the book's absolute on-disk
location, and gates all of it behind the admin role.
- The page handler resolves library folder + relative path (verified
with os.Stat; falls back to the relative path when the file is not
found on disk) into the new MediaDetail.FileLocation field - only for
admins, so the absolute path never leaves the server for regular
users. This also makes it possible to locate sparse entries whose
metadata rows are largely blank.
- The Metadata grid gains a full-width, monospace, click-selectable
Location row (admins only).
- The Edit button and the MetadataEditorModal markup render only for
admins. The modal drives admin-only endpoints (metadata PUT, rescan,
reset), so non-admins previously saw a button and a form that could
only ever fail with 403s.
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
parseTableNames() regex-scanned every line of schema.sql, comments
included, with 'CREATE TABLE (?:IF NOT EXISTS )?(?:\w+\.)?(\w+)'. A doc
comment containing that phrase in prose - e.g. 'declared in CREATE TABLE
above' - registered a phantom table ('above'), and startup verification
then failed with 'missing tables: above', crash-looping the app
container on every restart.
Skip lines whose trimmed form starts with '--' so comments can never
contribute table names, and add a regression test asserting every parsed
table maps back to a real CREATE TABLE statement.
pdftoppm defaults to rasterizing the MediaBox, while PDF viewers (pdf.js
in the reader, and every other viewer) display the CropBox. For PDFs
whose page 1 is the full print cover wrap (back cover + spine + front
cover in one landscape page) with a CropBox covering only the front
cover - e.g. No Starch's XeTeX-built 'Algorithmic Thinking' - the
fallback stored the entire spread as a squashed landscape cover, while
the reader correctly showed just the front cover.
Pass -cropbox so the rendered cover always matches what the reader
displays. Poppler falls back to the MediaBox when a PDF defines no
CropBox, so PDFs with identical boxes (the common case) render exactly
as before.
John O'Keefe