KOReader highlights converted to wrong or unresolvable web CFIs
whenever a selection crossed text-node boundaries. Three compounding
defects: ParseCREXPointer discarded the text()[N] node index and
applied .offset as a global block offset; the end anchor converted
with an empty context (blind structural landing, zero verification);
and the block-containment check accepted wrong-node landings because
a selection's text always occurs somewhere in its own block. pos0 and
pos1 were converted independently, yielding mismatched start/end base
paths and backwards ranges - highlights stored but never painted, or
painted at the wrong location, in the web reader.
Add ConvertHighlightRange: both endpoints are anchored by locating the
selection text in the flattened document text (whitespace-normalized,
crossing block boundaries), with the device locators demoted to
disambiguation hints. A single document match yields a same-spine,
ordered start/end pair by construction. Falls back to the legacy
per-endpoint path when the text cannot be located, so behavior is
never worse than before.
Also fix the shared flatten stream: zero-width format characters
(ZWSP/ZWNJ/BOM) and whitespace-only inline spans (<span> </span>)
now act as word separators instead of fusing adjacent words, which
prevented text matching in real books.
Verified against 28 real device highlights from production: converter
output now matches the text-search oracle on all of them (previously
8/28 correct, 18 with mismatched base paths, 2 backwards).
Handler.UpdateUniversalProgress (handlers/progress.go) had no route
registration anywhere (router/progress.go only wires the history route
to the scanner handler) and no callers — a legacy progress-write path
that bypassed the verify/heal pipeline entirely. Deleting it removes
the last bypass-shaped writer that could poison progress rows without
verification; conflict-resolution apply remains the only deliberate
direct write (user-approved winner rows).
Pure deletion: 115 lines, no route, no callers, no behavior change.
The verify/heal pipeline verified only INBOUND context-bearing writes;
stored-row coherence was never enforced. SaveProgress seeds missing
fields from the existing row, so every percentage-only write (KOReader
low-confidence conversion, KOReader no-pointer payloads, Kobo markup
ReadingSync / analytics / sync-from-server, locator-less queue entries)
silently re-paired its NEW percentage with the PREVIOUS writer's stale
CFI+context — the incoherent row shape that fed the client-side
far-anchor restore floor (and, via the read-side anchor handles, could
even hand a first-party client the OLD position's scroll address).
KOReader's converter already declared the right intent ("storing
percentage only — the row's percentage restores approximately"); the
seeding betrayed it. SaveProgressRequest gains ClearLocator, set by the
percentage-only call sites, and the seeding rule honors it: stale
epubcfi/context/character_offset clear (the upsert assigns columns
directly, so NULL sticks) unless the request carries its own. A
request-carried context survives so the verify/heal pass can still mint
a fresh anchor from it; the HTTP merge path (field-absent PUTs
preserve) and all anchor-bearing writes are untouched. The fixed-layout
branch already cleared locators unconditionally — unchanged.
Consumer audit: percentage — the only thing every sync round-trip
exchanges — is untouched on all paths; KOReader pulls on locator-less
rows already support the nil-CFI shape (builder guards
progress.Epubcfi.Valid); first-party restores go percentage-first
instead of floor-fighting a stale anchor.
Tests: TestProgressWeb_LocatorCoherence pins clear-on-declared,
request-carried-context survival, and merge-path preservation. Full
suite with the change: 218 groups, zero failures.
The create endpoint shared the scanner's upsert (ON CONFLICT
(library_id, file_path) DO UPDATE SET updated_at = NOW() RETURNING *) —
so a POST naming an already-registered file returned 201 CREATED with
the OLD row while silently discarding everything the caller submitted.
Nothing depended on that lie: the app only GETs/PUTs media items, the
web only searches/purges/deletes, KOReader never calls it — the
endpoint's create path had zero production callers.
Split the API path from the scanner's idempotent upsert, which is kept
verbatim in CreateMediaItem (re-scan dedup remains its safety net):
- CreateMediaItemIfNew: same insert with ON CONFLICT DO NOTHING — the
loser of a create race writes nothing and returns no row.
- GetMediaItemByLibraryAndPath: fetch the existing row for the
duplicate response.
- Handler branch: on pgx.ErrNoRows → 200 OK with
{"duplicate": true, "message": ..., "item": <existing row>} instead
of a 201 pretending it created something. Real creates still answer
201 with the item as before.
Bulk-import shape (the reason for 200+flag over 409): a re-run after a
mid-import failure is harmless and countable — each response tells the
caller exactly which of the two things happened.
Verified by the full integration suite: 217 groups pass, zero failures,
including a new pin of the duplicate contract (same file_path twice in
one library → 201 then 200/duplicate:true, second payload not
overwritten).
Born dead in e7ce0cb (2026-04-03, backend reader infrastructure):
the "Phase 2.5" plan assumed MOBI/AZW3/DOCX/RTF would need server-side
parsing, so ParseEbook was written as the endpoint that would do it —
but it only ever returned 501 "not yet implemented", and no route in
router/reader.go ever pointed at it. No caller in Go, templates, or
web JS/TS ever existed either.
The eventual architecture never needed it: the web reader hands the
raw file to foliate-js and parses client-side (MOBI renders via the
fork's mobi.js — verified live; DOCX via mammoth; RTF/PDB shipped).
By now it was also stale — it 501'd DOCX/RTF as "not yet implemented"
while both work, and offered an AZW3 path that was dropped entirely.
Pure deletion: no route registration, no references, no tests, no
behavior change. go build ./... clean.
The old model hard-deleted system collections and re-created them at
the tail, resurrecting stale dedups and breaking ordering. Redesign
(user-directed B-final):
- schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER —
the whole schema is re-applied on every boot via executeSchema).
- queries.sql: every collection list/read query filters
deleted_at IS NULL; new GetSystemCollectionByQueryType :one
(unfiltered, sees tombstones), TombstoneSystemCollection :exec,
ClearCollectionItems :exec, RestoreSystemCollectionRow :one
(in-place metadata reset + un-tombstone; collections have no
updated_at column — ResetUpdatedAt was a 500 caught by tests).
sqlc regenerated.
- collection_service.DeleteCollection: system collections are
tombstoned (preserving position/ownership); user collections keep
the hard delete. No ErrNoRows to the client (delete is idempotent).
- dashboard_service.RestoreSystemCollection: resolve by query_type;
found → in-place restore (ClearCollectionItems + metadata reset,
un-tombstoned); missing → provision fresh. "Want to Read"/etc.
stay at their home positions forever.
- handlers/collections.CreateCollection: name-taken is 409, not 500.
- handlers/auth.CreateDefaultCollectionsForUser: idempotent by
(user_id, query_type) — update-in-place when a row for that query
type exists (heals renames/tombstones), insert when missing; this
also fixes the duplicate-system-collections regression at source.
The dashboard's All Libraries view could render sections across every
library but could not persist its own preferences: the PUT required a
UUID library_id, and the prefs GET-without-library_id matched an
arbitrary library's row through a loose NULL OR-clause.
Storage: a dedicated user_dashboard_preferences_global table (one row
per user, user_id PK → users CASCADE) instead of a NULL/sentinel
library_id — chosen over both because (a) the column's FK to libraries
rules out a sentinel UUID without dropping the constraint or minting a
fake library row, and (b) intent becomes structural: only the explicit
global path writes that table, so a buggy or outside-the-API write can
never masquerade as global. System and user collections are
user-scoped, so the order/hidden arrays are library-independent and
valid in this scope.
API: PUT /api/dashboard/preferences binds library_id as *string —
JSON null/absent = global, empty string = 400 (a lost value must stay
loud, not silently clobber the global scope), valid UUID = per-library
(unchanged). GET without library_id now returns the global row instead
of an arbitrary one; the per-library query's dead NULL OR-arm is
removed. GET /sections keeps requiring library_id for per-library
views and aggregates across libraries when absent (unchanged).
Tests: global save/update-dedupe/read round-trip, empty-string 400,
global-vs-per-library independence. Two suite failures are the
handoff's documented pre-existing rot (duplicate system collections in
the dev DB; a sections test expecting a 400 the handler never
implemented) — reproduced and confirmed unrelated to this change.
Live-verified on the dev stack: null save 200 x2 with a single row
updated, empty string 400, global read carries the second save.
One boolean (Security > Public Registration) switches the whole public
signup lifecycle: front page + login links, the logged-out sidebar's
Create-an-account (via a package-level templates hook so shared page
templates keep their signatures), GET /register -> 302 /login, and an
early 403 on POST /api/auth/register. First-user exception preserved:
zero admins keeps every route and link reachable for bootstrapping —
same 'users exist' reasoning as the setup gate. Admin user creation is
unaffected by design.
Live E2E verified: flag off hides all surfaces and blocks the POST
(403), flag on restores them (201); the admin UI renders the row
automatically.
The 10k validator cap rejected legitimate long-form notes — a
scholarly reading note with quoted passages and footnotes lands
around 10.1k chars and failed at save with the raw validator error.
Raise the cap to 100,000 on all six annotation text fields in the
media handler request structs:
- CreateMediaNoteRequest / UpdateMediaNoteRequest Content
- CreateMediaHighlightRequest / UpdateMediaHighlightRequest NoteText
- CreateMediaBookmarkRequest / UpdateMediaBookmarkRequest Notes
No other layer changes: the media_notes / media_highlights columns
are unbounded TEXT, and the KOReader + websocket sync paths never
had a length cap, so the REST API now matches the rest of the
system instead of being the strictest gate.
New integration test pins the boundary: 50k and 100k-char notes
return 201, 100,001 chars returns 400.
GetProfile returned the theme from the JWT-claims context stub, which
carries only id/email/username/role — user.Theme was always empty, so
the profile response omitted the field and the app's account-theme
read-back (SERVER sync mode) could never see the stored theme. GetProfile
now loads the full user row by id (h.db.GetUser).
The Android app's app-chrome theme (UX pass item 6) reads the account
theme from GET /api/auth/profile and re-applies it at startup in
SERVER sync mode. PUT /api/auth/theme already persisted the value;
this completes the round trip on the profile read.
Adds publisher_filter to SearchMediaItems/SearchMediaItemsUnified
(fuzzy word_similarity against mi.publisher, mirroring genre_filter),
plumbs it through services.SearchParams and the search handler, and
extends the GREATEST relevance ranking to include publisher matches.
Serves the app's author/publisher/genre/tag click-through browses
(UX pass item 5 — publisher was the only facet without a server
filter).
Offset currency policy, now explicit: EPUB CFI terminals, CRE text()
offsets and the served char_offset handle are UTF-16 code units (the
EPUB CFI spec, and what foliate/readium/KOReader/Kobo clients actually
observe), while internal arithmetic — the book-wide character_offset
column and percentage fractions — stays rune-based, consistent with
TotalCharacters. For all-BMP books the currencies are identical, so no
stored value changes; astral-plane text (emoji, rare CJK) no longer
drifts.
Boundaries converted: resolveCFIToNode interprets incoming CFI terminal
offsets as UTF-16; textNodeAtUTF16Offset (née textNodeAtRuneOffset)
interprets CRE text() offsets as UTF-16; buildCFI and buildCREXPointer
emit UTF-16 terminals; blockCharOffset (the served char_offset) is
UTF-16.
Also fixes two character_offset column defects: heals wrote a BLOCK-
relative offset into the book-wide column, and verified-but-unhealed
saves (e.g. KOReader pushes) never refreshed it, leaving it stale
behind the anchor. VerifyProgressAnchor now returns the verified book-
wide rune offset and SaveProgress refreshes the column on every
verified save.
Tests: astral currency round trip (offset after an emoji must shift by
one unit between currencies, in both heal and exact-verify directions)
and book-offset ordering. The cmd/server/tests integration harness
failures under docker (library folder 400 during setup) reproduce on
the pre-change tree and are unrelated.
Progress submissions now carry (percentage, context_text, epubcfi) and
the server becomes the position authority:
- VerifyProgressAnchor resolves the submitted standard CFI against the
book's own XHTML, extracts the text at the anchor, and cross-checks it
with the submitted context_text. A mismatch heals the anchor by text
search (percentage disambiguates repeats) instead of storing a bad
position.
- The anchor's block element is derived as a cssSelector plus a block-
relative character offset, and served on progress GET alongside the
anchor document's href — readium-native handles that let clients
re-open a book without parsing CFIs themselves.
- context_text-only submissions (no CFI — the dumb-client tier) are
anchored structurally from the context text.
Motivation: cross-client progress sync (web foliate CFIs, KOReader CRE
xpointers, readium-native apps) previously trusted each client's own
locator math; the app's EPUB restore drifted ±pages because readium's
paginator does not lay out far-from-viewport columns and the foliate-
ported CFI walk ran against readium's mutated WebView DOM. Server-side
verification heals both classes at ingest.
The device catalog flattened every visible library into one list, so
duplicate copies of the same book each got an entry and search had no
library context. The root feed is now a standard OPDS 1.1 navigation feed
that mirrors the web UI's library model:
- Root /catalog: an "All Books" entry first (the previous flat cross-
library behavior, also still served at ?all=1 for clients that want
the single flat list), followed by one folder entry per visible
library with live book counts from GetVisibleLibraryMediaCounts.
- New GET /library/:libraryId/catalog: acquisition feed scoped to one
visible library (403 when the device owner cannot see it), paginated,
with an up-link back to the root.
- Scoped search: each library feed's rel="search" OpenSearch template
pins &library_id=<id>, so opening search from inside a library folder
searches only that library — clients substitute only {searchTerms},
so no client-side changes are required. Root search stays global.
- Global search entries now carry the owning library as a category (and
as a fallback summary when the book has no description), so duplicate
copies are distinguishable in unscoped result lists.
The acquisition entry builder is extracted into addAcquisitionEntries and
shared by the all-books and per-library feeds. bookhoard.koplugin needs
no changes: it only registers the root URL, and KOReader's stock OPDS
client renders navigation feeds natively.
Tests: TestOPDSLibraryFolders covers the nav-feed shape, flat ?all=1
mode, scoped catalog isolation, scoped/global search behavior, and 403s
for libraries outside the device's visibility. Library names avoid the
word "test" on purpose — setupDeviceTest re-runs setupTestServer's
setup-time cleanup, which deletes every library whose name contains it.
Covers the copy-then-delete-later workflow: a user copies books to a new
library, deletes the originals, and ends up with an active copy in the new
library plus an archived twin holding the real reading history. Until now
those twins could only be purged (destroying the history) or restored
(showing a permanently broken entry).
- POST /api/media-items/:id/merge (admin only, body {target_id}):
validates the source is archived/missing, the target is active, and
both share the same file_sha256; then re-parents every child row onto
the target via the existing reparent_media_item_children function (the
same machinery as hash-conflict resolution) and deletes the source row.
Per-user collisions keep the active copy's data, mirroring that flow.
Affected data: reading progress, speed, ratings, highlights, notes,
bookmarks (including tombstoned deleted-annotation history), formats,
collections, kobo shelves/entitlements, sync rows, panel data,
processing issues, and device aliases.
- GET /admin/archived: ListHiddenMediaItems' match_* columns surface each
row's best active twin; rows with a twin get a confirm-guarded "Merge"
button (data-merge-source/-target) next to Restore/Delete, wired in
web/src/admin.ts like the existing unarchive/delete handlers.
- templates.ArchivedItem gains MatchID/MatchTitle/MatchLibraryName;
frontend.go populates them from the listing row.
After a merge the archived row is gone, so the retention purge can never
destroy the merged data. Deleted-annotation tombstones carry over and
remain restorable from the target book's "recently deleted" history.
The mutex hardening added a fall-through 'pending' response path that
returned while still holding pendingMu — one status poll before approval
leaked the lock and every later register/approve/status request hung
forever (caught by TestDeviceRegistrationFlow's approve step hanging).
App reinstalls that preserve data (Android Studio installDebug over an
existing install) re-register with the same device_identifier, but the
devices row from the previous install still exists — device_identifier
is UNIQUE, so ApproveDevice's blind INSERT failed with a unique
violation and returned 500 'failed to create device' (reproduced via
curl: second approve with the same identifier = instant 500; the ~98s
in the original report was app-side retry/polling, not server wait).
ApproveDevice is now idempotent: look the device up by identifier
first; a row owned by the approving user gets its auth token rotated
via UpdateDeviceAuthToken (row id unchanged, so synced highlights/
bookmarks/progress anchored to it stay valid; fresh install = fresh
credentials, old token invalidated); a row owned by another user gets
409; unknown identifiers INSERT as before, with the 23505 race falling
through to the rotate path. DB failures are logged (they were silent).
Also guard the in-memory pendingRegistrations map with a mutex —
register/approve/reject/status/list all touch it from HTTP goroutines,
and a racing write is a Go runtime fatal, not an error. The approver's
credential publication and the status poller's approved-branch snapshot
now run under the lock so the token can never be read half-written.
Regression test: TestApproveDeviceReapprovalRotatesToken — register →
approve → re-register same identifier → approve (must be 200) → token
rotated, exactly one devices row, row carries the new token.
PDFs shelved in comics or manga libraries (scanned manga, official manga
PDFs with text layers) now get comic reader treatment: bookmarks only —
no text-selection highlights, no annotations, no in-book search. Items
in ebooks libraries are completely unaffected, and EPUBs keep their
existing behavior everywhere.
- handlers: new exported ShouldTreatAsComic(libraryType, formatGroup) —
true for fixed_layout/comic_archive in manga/comics libraries; the
reader JSON API also exposes it as treat_as_comic
- router: the SSR reader page fetches the library type and passes
TreatAsComic through ReaderMetadata into the init config
- reader: when treatAsComic is set, the PDF textLayer selection listener
(mouse and touch paths) never attaches so no highlight popover can
open; pointer mode is forced to pan (Smart/Text segment hidden);
search button and toggleSearch are disabled; previously-created PDF
highlights stop rendering
- bookmarks are unchanged (already page-index based for fixed layout),
and device sync / OPDS / format classification are untouched since
format_group stays fixed_layout
Admins need to see what the archive lifecycle is holding: a dedicated
/admin/archived page listing every hidden item (archived or still in
the missing grace window) with library, file path, status, and - when
retention is enabled - the exact date it will be permanently deleted
(archived_at + ARCHIVE_RETENTION_DAYS).
Per row: Restore (POST /api/media-items/:id/unarchive, clears the
archive state so it reappears; if the file is still gone the next scan
hides it again) and Delete (existing DELETE endpoint for single-row
purge with its reading history). Purge All Archived reuses the existing
bulk button. The library admin banner links to the page and keeps its
purge button; row actions use data attributes with delegated listeners
in admin.ts since this templ version has no JSFunctionCall helper.
Verified live: page 200 with purge dates shown, unarchive returned 204
and reset the row, per-item delete and bulk purge ({purged:1}) both
removed their rows with no leftovers.
Bulk escape hatch for archived rows (files missing from disk for 2+
scans) so a mass external deletion never has to wait out the retention
window or be clicked away row by row:
- POST /api/media-items/purge-archived (admin only) hard-deletes all
archived items and returns the purged count; reading history goes with
the rows, so the call is confirmed in the UI first.
- The library admin page shows an 'Archived items: N' card (only when
non-zero) with a Purge Archived Now button that calls the endpoint,
toasts the result, and reloads.
- Frontend admin JS exposes window.purgeArchivedItems following the
existing localStorage-bearer-token pattern.
The book detail page exposed server internals and unusable controls to
everyday users: it now computes and renders the book's absolute on-disk
location, and gates all of it behind the admin role.
- The page handler resolves library folder + relative path (verified
with os.Stat; falls back to the relative path when the file is not
found on disk) into the new MediaDetail.FileLocation field - only for
admins, so the absolute path never leaves the server for regular
users. This also makes it possible to locate sparse entries whose
metadata rows are largely blank.
- The Metadata grid gains a full-width, monospace, click-selectable
Location row (admins only).
- The Edit button and the MetadataEditorModal markup render only for
admins. The modal drives admin-only endpoints (metadata PUT, rescan,
reset), so non-admins previously saw a button and a form that could
only ever fail with 403s.
Previously both the metadata editor (PUT /api/media-items/:id) and the
scanner (library scans, force rescans, per-book rescans) wrote through
the same unconditional UPDATE media_items query, so any rescan wiped
user-written descriptions, tags, and uploaded covers. Custom and scanned
values were indistinguishable, and custom cover uploads even wrote to
the same {file}.cover.jpg sidecar path the scanner generates, so each
side silently clobbered the other.
Introduce metadata_overrides, a TEXT[] column on media_items listing the
column names the user has customized:
- Saving metadata records overrides per field by diffing the submitted
values against the stored row (an untouched save records nothing);
overrides accumulate until an explicit reset. Cover upload/removal
always marks cover_image_path. Bulk updates mark each applied field.
- The scanner merges: updateMediaItem() now takes the existing row and
restores every overridden column (including derived *_search arrays)
before writing, and preserves the override set itself.
- Uploaded covers move to a dedicated {file}.custom_cover.{jpg|png|webp}
sidecar so the scanner can never overwrite a user cover on disk.
- RescanMediaItem gains resetOverrides: POST /api/media-items/:id/rescan?
reset_overrides=true clears the set first, returning the item to pure
scanned defaults.
Shared detection/restore helpers live in internal/utils
(metadata_overrides.go) with unit tests covering detection, accumulation,
unset-form equality, and restore-with-derived-fields. Schema change is
an idempotent ADD COLUMN IF NOT EXISTS applied on startup. Also includes
incidental gofmt of NewMediaScanner literals in media_scanner.go.
Admin-only endpoint that re-extracts metadata and cover art for a single
book from the file on disk via MediaScanner.RescanMediaItem and returns
the updated media item. Normal library scans skip unchanged files, so
this gives a targeted way to backfill covers for previously imported
books. Includes a Bruno request alongside the existing media-items
collection.
Tonight's failures all traced to one blind spot: the converter could
only reason about text within a single block. A position at a chapter
heading sends walk-up context (heading + the paragraphs below, joined by
the plugin's block capture); a selection can span several paragraphs.
Neither shape could be verified (containment compared one block against
a multi-block quote, so the CORRECT structural landing at the heading
was rejected) nor matched by text search (it never crossed block
boundaries). The ladder then fell to the percentage rung — which labeled
its char-count guess Precision "exact" — and that confidently-wrong CFI
was stored: reading positions reopened paragraphs away from the true
spot, and a highlight echo overwrote the row's good web CFIs with a
garbage start anchor that made the highlight unpaintable ("disappeared").
Four changes, all in the forward converter and its consumers:
- Quote verification: after the structural walk lands, read the
whitespace-normalized document text forward from the landing point
(crossing block boundaries; inline spans join directly so drop-cap
splits still read as one word). A usable context must be a prefix of
that stream — which is exactly what device captures are: the text from
the position onward, or the selection between two anchors. The old
single-block containment checks remain as secondary acceptance.
- Cross-block text search: the search rung matches against the whole
document flattened in reading order, with every rune mapped back to
its source node and offset. A context spanning blocks now matches, and
the matched extent yields a true range end (EndEPUBCFI) that
highlights use as their end anchor, threaded through the facade as
CanonicalLocator.EndCFI.
- Honest labels: the percentage rung returns Precision "percentage" —
a char-count estimate must never masquerade as an exact anchor.
- Confident-only storage: progress adopts a converted locator solely at
structural/exact precision (section hrefs keep their legacy handling;
anything lower stores percentage only), and highlight conversion
returns CFIs only at structural/exact precision — a low-confidence
echo yields empty, which applyLWW coalescing turns into preservation
of the row's existing web CFIs instead of clobbering them.
Tests: walk-up context at a heading verifies structurally and lands in
the heading; a block-spanning context is found by search with a range
end landing in the following paragraph; the percentage rung is honestly
labeled; all drop-cap guards stay green.
PUT /highlights/:id parsed the row id from the URL and then dropped it:
the sync-aware path routed through SaveHighlight's content-derived dedup
key, on the assumption that the same text + CFI always resolves to the
same key. That assumption breaks in practice — the stored epubcfi_start
drifts from foliate's range shape to the converter's point shape after a
device echo rewrites the row (bucketPosition cuts at the last colon, so
'…/6,/1:367,…' and '…/6/1:367' bucket differently), and the user can
edit the selection text. The recomputed key then misses the row being
edited and createHighlight mints a second one: the edited row (with
note, no device pos0) beside the original — served to KOReader as two
highlights, one noted and one not. Editing a selection's text would hit
the same trap.
SaveHighlightRequest gains an optional HighlightID. When set, the save
resolves the row by id (ownership-checked), LWWs against it under its
stored dedup key, and never re-derives identity from content. The PUT
handler passes the already-parsed id. Device pushes, Kobo, and the sync
queue send no id and keep the identity-based flow untouched.
applyLWW also stops wiping stored locators on web edits: the web reader
sends empty start/end positions (it never had a CRE xpointer), so a
note/color edit now keeps the device-native positions and CFIs instead
of blanking them — round-trip serve-back for device-created highlights
survives web-side edits.
Progress, highlights, notes, and bookmarks entered position conversion
through three different doors: progress converted inline with an
uncached converter, annotations through the facade, bookmarks not at
all (the raw xpointer was stored verbatim, cfi_position stayed empty,
and the web drawer's goToBookmark silently no-ops on cfi-less entries).
Unify on the facade (ConvertToCanonical/ConvertFromCanonical):
- annotationEpub context resolved once per push: media item + EPUB path
shared by every annotation instead of re-fetched per entry
- progress forward: the inline block becomes one facade call;
non-reflowable formats pass through unchanged, and the cached
converter stops re-parsing the book on every sync
- progress reverse: convertCFIToXPointer delegates to reverseConvertCFI,
keeping the stored percentage in play for the fallback ladder
- bookmarks (bulk progress and /sync-bookmarks): pos0 resolves
structural-only — bookmark text is a display label, never book text,
so no context is supplied; webUsableCFI stores the result only for
structural/exact epubcfi landings, discarding href/percentage results
rather than storing dead drawer links. Also records percentage_location
and origin_source on the legacy endpoint.
- percentages thread through: highlights/notes/bookmarks pass the device
percentage or the derived section percentage instead of a hardcoded 0,
so the last-resort fallback lands near the true position instead of
the document start
- extendCFIByLength end-derivation now also fires on structural starts
(it had silently stopped matching when the structural rung began
landing starts with precision 'structural' rather than 'exact')
Tests: the drop-cap xpointer through the facade with empty context (the
bookmark scenario) must land structurally, not doc-start; webUsableCFI
table covers the store/discard gate.
- Drop UNIQUE(media_item_id,user_id,title): titles are display labels
shared verbatim across clients; same-title bookmarks on different pages
now coexist instead of 500ing (deleting over a tombstone no longer
blocks future creates with that title)
- Add origin_source column recording the creating client (android/web/
koreader), set once at insert, exposed in API responses
- Web reader auto-title mirrors KOReader's 'in <chapter>' convention,
falling back to 'Bookmark'; adds bookmark rename in the drawer
ServeFile previously authenticated only ("any logged-in user") and never
checked that the user can actually see the library owning the file, so
knowing a library UUID + path was enough to fetch content from hidden
libraries. Library visibility is the permission model - the library is
what grants access to its media.
- ServeFile now resolves two URL forms through one flow:
/uploads/library-{id}/{path} (covers, reader files)
/api/media-items/{id}/download (explicit book download, new)
The item form looks up the media item, derives its library and file
path, and adds a Content-Disposition attachment header.
- Both forms enforce GetUserVisibleLibraries for the authenticated
user, mirroring the OPDS download handler (403 when not visible).
- Deleted the dead MediaHandler.DownloadBook handler (never routed).
Also widen media_highlights.start_position/end_position from
VARCHAR(100) to TEXT: the API handlers validate up to 1000 characters
(full Readium locators, KOReader CRE xpointers) but the column rejected
anything longer at the database layer. Metadata-only change applied
idempotently at startup; existing rows are untouched.
Verified against the running server: download 200 + attachment headers
+ epub bytes, unauthenticated 401, user hidden from the library 403 on
both URL forms, visible user 200, covers unchanged, and a 334-char
locator JSON now round-trips through the highlights API.
KOReader push (processBookAnnotations) accepts deleted_highlights and
deleted_bookmarks arrays of dedup keys and tombstones the matching rows,
after the upserts so a key present in both lists resolves to 'deleted'
(the newer intent). Deletions remain soft: rows stay restorable from the
history and echo to other devices as tombstones on their next pull. A
stale device replay of the annotation cannot resurrect the tombstone —
device pushes carry no modification timestamp, so the save loses to the
delete. Absence from these arrays is never a delete, keeping category
toggles safe.
New annotation-history endpoints (annotation_history.go, media.go):
GET /api/media-items/:id/annotations/deleted
POST /api/media-items/:id/annotations/:annotationId/restore
DELETE /api/media-items/:id/annotations/:annotationId
All scoped to the authenticated user and the route's book; the DELETE is
the permanent purge (annotation_type required in query or body).
MediaDetail gains DeletedAnnotations, populated by the book page route
via the shared DeletedAnnotationsForBook builder, so the server-rendered
history ships with the page instead of requiring a client round-trip.
Binding tests cover the plugin's exact wire shape and the legacy
plugin case (arrays omitted -> empty).
GET /api/sync/koreader/resolve?sha256={hash} maps a file content hash to
the book's UUID through the shared format-aware BookResolver (primary
media_items hash, then per-format hashes so converted KEPUB/PDF files
match) without touching any progress state.
Devices need the UUID to pull metadata, but a freshly downloaded book has
none cached. The old way of learning it was to push once, which
transmitted the device's first-page position and manufactured a progress
conflict for books already mid-read from another source. A read-only
lookup lets clients link (and pull) without ever pushing bootstrap
progress: resolve, then pull, then push.
Returns 200 {book_uuid, sha256, title, author}, 400 for a missing or
malformed hash, 404 when no library item matches.
Reverses the earlier "no colors to the device" decision now that the
echo machinery makes it safe: GetMetadata maps the stored web hex to
KOReader's fixed color names (#ce93d8→purple, #90caf9→blue,
#a5d6a7→green, #ffd54f→yellow; pink maps to purple as the closest —
round-trip drift is prevented on the device by echo suppression, and
a device edit still wins). mapColorToKOReader restored for serving;
ingest (name→hex, preserve-on-echo) unchanged.
Echo duplication: devices push their full annotation list on every
sync, and an echo of a web-created annotation computed a different
dedup key than the original (device locators differ from web locators)
— every pull→push cycle minted a duplicate row, and cleaning those up
on the web tombstoned them back to the device, deleting the
just-applied copies. That was the "web highlights never appear on
KOReader" experience. GetMetadata now serves each annotation's
dedup_key; the device stores it on the applied entry and echoes it in
pushes; SaveHighlight/SaveBookmark/SaveNote accept a DedupKey
override so echoes converge onto the original row (verified: pull →
echo push creates no rows, LWW skips identical content).
Color semantics (per user preference): devices render their own
default and cannot round-trip web colors, so GetMetadata no longer
serves colors at all — every highlight syncs regardless of its web
color and the device draws its default. An echo carries no color;
ingest then PRESERVES the stored web color (existingHighlightColor
lookup by dedup key) so round-trips never change it. A non-empty
device color means the user edited the highlight there: it maps
name→hex (green→#a5d6a7, default yellow) and wins. Verified: echo
kept #ffd54f; a simulated device edit with "green" updated the web
row to #a5d6a7.
Classification: KOReader auto-fills text="in Chapter X" on page
bookmarks (ReaderAnnotation:updateItemByXPointer), so the plugin's
text-presence classification turned every echoed bookmark into a junk
highlight on the web. v2 classification now keys off the drawer field
(present = highlight/note, absent = bookmark with its label in note).
Both directions synced data but rendered nothing:
- Web reader <- devices: highlights painted no overlay. Device pushes
resolve their start xpointer exactly (text-search anchored by the
selection) but the end conversion carries no context and fell back
to a document-start CFI (epubcfi .../1:0) — a garbage range end.
When the start resolved exactly, the end is now derived from it:
same node, character offset advanced by the selection's UTF-16
length (extendCFIByLength). Same repair when SERVING to devices,
where old web highlights (no end anchor) and converted range CFIs
both collapsed pos1 onto pos0 (extendXPointerByLength on the
xpointer form) — KOReader drew zero-width highlights.
- Colors: KOReader paints from a fixed name set (Blitbuffer
HIGHLIGHT_COLORS), the web uses hex swatches; neither understood
the other, so device colors fell back to defaults and web hex drew
nothing useful on devices. Both boundaries now translate: ingest
maps names to hex (default #ffd54f), GetMetadata maps hex to names
(default yellow) — per-datatype edits re-push with the editing
side's color, which LWW then propagates. SyncBookmarks endpoint
aligned to the same mapping and default.
Two blockers, diagnosed by simulating the plugin against the live
server with real library books:
1. Every KOReader progress push carrying annotations failed the JSON
bind with 400 ('cannot unmarshal string into ... chapter/page of
type int') — the plugin sends chapter:'', page:'30', and for CRE
documents page:'/body/...' — so annotation sync AND progress sync
failed together. KOReader annotation chapter/page now use FlexInt,
which accepts numbers, numeric strings, empty strings, and
non-numeric strings (decoding to 0). The server is deliberately
liberal here so thin clients can send raw bookmark data.
2. GetMetadata served locators KOReader cannot place, so pulled items
were junk: web bookmarks leaked 'cfi:epubcfi(...)' positions, web
PDF highlights had empty pos0 (skipped by the plugin, invisible),
and web deletions carried no pos0 so tombstones never matched.
New koreaderPos0 resolver handles every source: device-native
xpointers pass through untouched (round-trip identical, verified),
web PDF JSON anchors map to their page number, EPUB CFIs convert
to CRE xpointers (selection text passed as text-search context for
exact anchoring), 'page:N' positions strip to the bare number.
Unresolvable annotations are skipped with a log line instead of
poisoning devices; tombstones get pos0 injected from the new
locator columns.
Also: thin clients omit per-annotation percentages (paging docs still
send arithmetic page/total); the server derives them — section
midpoint from the spine char distribution for CRE documents, page/
page-count for fixed formats.
Phase 3 (EPUB half) of the reader redesign:
- Select text in a reflowable book → floating glass popover at the
selection (5 colors, note, copy). Clicking a color creates the
highlight via POST /api/media-items/:id/highlights, anchored by the
foliate range CFI (epubcfi_start) with percentage position.
- Highlights render through foliate's overlayer pipeline: draw-
annotation draws Overlayer.highlight with the stored color,
create-overlay re-adds persisted highlights as sections load,
show-annotation opens the edit popover when a highlight is clicked
(recolor, edit note, copy, delete).
- Backend: highlight create/update accept epubcfi_start/end,
note_text, and percentage fields; position validation relaxed
(CFIs exceed the old 100-char cap); PUT routes through
AnnotationService.SaveHighlight so edits get dedup/LWW treatment
and actually persist note_text (the plain query can't).
- Bookmarks drawer becomes the Annotations drawer with tabs:
Highlights (color-bar list, note previews, jump/edit/delete),
Notes (add note at current position, list, delete — backed by the
existing notes API), and Bookmarks (unchanged behavior).
- Popover dismissed on outside click, collapsed selection, page
navigation, or Esc (new top-priority Esc branch).
Phase 0 of the reader redesign:
- Panels no longer render under the top/bottom bars: sidebars get
measured insets (same resize/safe-area mechanism as the viewport);
panel max-height now derives from the bounded sidebar instead of a
100vh guess; right-side border targets the actual sidebar.
- Bookmarks work end-to-end for the first time: REST CRUD under
/api/media-items/:id/bookmarks (create/delete route through
AnnotationService for dedup/LWW/tombstones), fix UpdateMediaBookmark
referencing nonexistent updated_at column, frontend posts to the
real API with per-format position (CFI vs page), live list with
jump + delete instead of SSR-only snapshot.
- Fix chapter matching in progress saves: boundaries were compared by
a nonexistent tocItem property, so chapter was never persisted.
- Remove dead UI: Navigator panel stub, empty dictionary popup shell,
unwired Chrome Behavior select; purge 160 stale build artifacts.
- Reader chrome now follows the user's app theme instead of hardcoded
theme-tokyo-night.
Complete the SHA-256 lifecycle for preexisting databases: items
imported before hashing existed get hashed automatically, and any
content duplicates discovered in the process land on the new admin
Hash Conflicts page for an explicit keep/merge decision.
HashBackfillService (runs once 30s after startup, independent of
auto-scan):
- hashes every media_items row where file_sha256 IS NULL, resolving
each path through LibraryService; per-item failures are logged and
skipped so one unreadable file cannot block the pass
- no-op once everything is hashed (logged and skipped)
- finishes with a conflict sweep flagging every content-duplicate
group via FindHashConflictGroups + CreateHashConflict; the sweep
runs after the per-item pass because a preexisting pair only
becomes detectable once both sides have their hash
API (admin-only):
- GET /api/admin/hash-conflicts - pending groups with member items
and usage counts
- POST /api/admin/hash-conflicts/:id/resolve - action=keep_all, or
action=keep with keep_uuid: validates the uuid belongs to the
group, re-parents every other copy's child rows onto the kept item
(reparent_media_item_children), deletes the losers, and records
the resolution + resolving admin; accepts form or JSON bodies and
returns the htmx resolved fragment
Page route /admin/hash-conflicts (admin-only) renders the template
with hydrated conflict data; HashConflictsHandler wired into the
router Config and constructed in main.
Verified end-to-end against the live database: duplicate detection,
pending listing, keep_all resolution, merge path (re-parent +
delete), and - critically - a resolved group is not re-flagged by a
later sweep (upsert no-op). Database restored afterward.
The sidecar config's books map is keyed by the primary SHA-256 (UUID
fallback). A device holding a converted format (KEPUB/PDF) whose hash
lives only in media_item_formats could not resolve its file through
the sidecar.
After inserting the primary-keyed entry, also register the same entry
under each per-format hash from media_item_formats (first write wins,
so a primary hash is never shadowed). Devices now resolve converted
files via the sidecar the same way the server's BookResolver does.
Applied to both the GET and download sidecar builders.
DownloadBook populated fileSha256 only for the kepub and pdf format
branches, so the default EPUB download never emitted the
X-Bookhoard-SHA256 response header - the hash was only available in
the feed metadata, not on the download response itself.
Populate it from mediaItem.FileSha256 in the default branch so every
download response carries the canonical primary-format hash. Clients
that capture response headers at download time now learn the hash
regardless of which format they requested.
Fixes the 'cannot push until pulling first' wall on books downloaded
via OPDS. Root cause chain: the bookhoard koreader plugin only learns
the book UUID from a successful push response, but the first push had
to match by SHA-256 alone - and that match consulted only
media_items.file_sha256, missing converted formats. When the hash
missed, no UUID was returned, so pull stayed blocked (it requires the
UUID) and the book could not sync at all.
Resolution side - route all five SHA-256 match sites through the shared
BookResolver so they are format-aware:
- resolveBookToMediaItem priority 2
- SyncBookmarks book-level lookup
- per-bookmark, per-note, and per-highlight override lookups
Exposure side - return the canonical hash so clients can learn and
cache it from a pull regardless of how the book was obtained:
- KOReaderMetadata gains sha256, populated from mediaItem.FileSha256
- KOReaderLibraryBook gains sha256, populated the same way, so the
library list endpoint carries it for every book
Together with the plugin-side UUID bootstrap (bookhoard.koplugin),
push and pull now work in either order on any format.
The platform had three duplicated, divergent book resolvers (koreader,
kobo, BookMatchingService) and none of them consulted
media_item_formats.file_sha256 - per-format hashes for converted files
(KEPUB, PDF) are computed and stored at import/conversion time but were
never used for lookup. GetMediaItemFormatBySHA256 existed with zero
callers. Any client holding a converted file could never match by
hash.
Add internal/services/book_resolver.go: a single shared resolution
path from client-supplied identifier to media_item.
ResolveBySHA256 checks media_items.file_sha256 first (indexed
GetMediaItemBySHA256), then falls back to media_item_formats.
file_sha256 (indexed GetMediaItemFormatBySHA256, first caller) so a
converted format matches with equal confidence. The import-time
SHA-256 is the canonical identifier shared by every interface.
Wire two of the existing resolvers through it:
- BookMatchingService.matchBySHA256: replaces the in-memory
ListMediaItems scan of up to 1000 rows with the resolver's indexed
lookups, and gains format awareness for the link/auto-link UI.
MatchMethod now reports sha256_sha256 or sha256_sha256_format
- KoboHandler.mapContentIdToBookhoardUUID: the SHA-256 heuristic
branch (ContentId that looks like a 64-char hash) now resolves
format-aware too. Kobo's entitlement_id wire identity is untouched;
only the opportunistic hash branch changed
Add a single pair of admin-only endpoints that supersede the scattered
scan-settings JSON routes as the canonical way to read and write
tunable system settings. Existing legacy routes are kept working for
backward compatibility and now refresh the registry cache on write.
system_settings.go:
- GET /api/system/settings returns every known setting with full
metadata (value, type, min, max, requires_restart, category, group,
description, is_default) via SettingsRegistry.All().
- PUT /api/system/settings accepts {key, value}; ApplySetting() looks
up the compiled Default for the key, runs type-aware validation
(int range, bool parse, non-empty string, timezone via
time.LoadLocation), upserts via UpsertSystemSetting, reloads the
registry, and reports whether a restart is needed for the change to
take full effect. Shared by the JSON endpoint and the HTMX endpoint.
- Legacy UpdateScanSettings / GetScanSettings / UpdateTimezoneSettings
now reload the registry after writing and prefer the registry when
reading, so the cache stays consistent regardless of entry point.
sidecar.go:
- SidecarHandler gains an optional registry; the timezone branch of
UpdateSystemConfiguration (PUT /api/system/config) calls
settings.Reload() after the write so the new value is visible
immediately. base_url handling is unchanged.
system.go:
- Register GET/PUT /api/system/settings under the existing admin
/api/system group.
The 30-day retention window for soft-deleted annotations was a package
const; move it behind the registry so it can be tuned live.
annotations.go:
- AnnotationService gains an optional *database.SettingsRegistry and a
tombstoneTTL() helper. The skip-resurrect checks and the purge cutoff
now call it instead of reading the TombstoneTTL const directly.
- Add ActiveTombstoneTTL() so callers outside the sync package can
compute cutoffs consistently with the service.
- The package-level TombstoneTTL const is retained as the fallback for
tests / unwired code paths.
kobo.go, koreader.go:
- The per-book tombstone sweep cutoff now uses
h.annotationSvc.ActiveTombstoneTTL() instead of the wsync.TombstoneTTL
const, so both the service and the handlers honor the configured TTL.
Move three more hardcoded values behind the settings registry. All
apply immediately on the next request (no restart needed).
device_auth.go:
- DeviceAuthMiddleware reads per-route device rate limits (sync /
progress / metadata per minute) from the registry on each
authenticated request via a rateLimitConfig() helper, falling back to
the Default* constants when no registry is wired.
- The X-RateLimit-Limit response header previously hardcoded "60" for
every request type; it now reflects the actual configured limit for
the request type via rateLimitForRequestType().
opds.go:
- Default (50) and maximum (200) OPDS page sizes come from the
registry's OpdsDefaultPageSize()/OpdsMaxPageSize() instead of inline
literals, so catalog pagination can be tuned without a redeploy.
conversion_service.go:
- The 24h kepub cache lifetime is read from the registry via a
cacheTTL() helper (was a bare 24 * time.Hour literal in the
constructor). The field default is retained for tests that construct
the service directly.
- conversion_service_test.go updated to assert both the field default
and the cacheTTL() accessor return 24h.
Replace the hardcoded 7-day session lifetime and fixed password
complexity rules with registry-backed accessors so they can be tuned
from the admin UI without a code change.
auth.go:
- Drop the SessionDuration const; keep DefaultSessionDuration (7 days)
as the fallback used when no registry is wired (e.g. in tests).
- AuthHandler gains an optional *database.SettingsRegistry and a
sessionDuration() helper that reads the registry, falling back to
DefaultSessionDuration.
- Cookie MaxAge, JWT exp claim, and ExpiresIn responses now derive from
sessionDuration() instead of the package-level SessionDurationSec, so
a settings change takes effect on the next login.
refresh_token.go:
- Refresh-token lifetime follows sessionDuration() via a new
refreshTokenTTL() helper (was a separate refreshTokenExpiration const
that silently had to be kept in sync with the session duration).
password_validator.go:
- PasswordValidator now reads min length and the upper/lower/number/
special toggles from the registry at validation time, so rule
changes apply immediately. The special-character regex is compiled
once and reused (sync.Once).
- GetPasswordRequirements() and ValidatePassword() reflect the active
configured rules instead of a static list.
- Add SetDefaultPasswordSettings() so the package-level default
validator (used by echo's struct-tag validator) follows live config.
All paths degrade gracefully to the historical defaults when no
registry is wired.
GetAllProgressData (SSR handler) was missing LastUpdated, DeviceIcon,
DeviceName, DeviceType, and EpubCFI fields that the template expects.
All showed blank. Now matches the API handler's field population.
Three bugs fixed:
1. Schema seeded base_url with fake placeholder 'bookhoard.example.com'.
Removed seed; startup now seeds from BASE_URL env var only if DB row
is empty (admin changes persist across restarts). One-time UPDATE
clears the placeholder in existing installs.
2. config.GetBaseURL() had a broken type assertion (local SystemConfigRow
vs database.SystemConfig) that always failed, returning . Admin panel
showed env var fallback instead of actual DB value. Fixed with a
function-type getter that properly wraps the DB query.
3. OPDS handler read base_url only from DB with no fallback. When DB had
the placeholder, all feed links pointed to an unreachable domain,
breaking KOReader search/download. Added deriveBaseURL() helper that
falls back to the request Host/scheme when DB value is empty.
Setup gate improvements:
- isSetupComplete now requires both admin user AND non-empty base_url
- Setup middleware no longer exempts all /api/ routes; only allows
/api/auth/register, /api/auth/login, /api/system/config before setup
is complete. All other API routes get 503.
- Cache invalidated when base_url is saved via admin settings
Dev workflow:
- New bruno/NewDevDBSetup/SetBaseUrl.yml for dev DB setup
- NewDB.sh runs SetBaseUrl between RegisterUser and CreateEbookLibrary
The dashboard re-renders its sections client-side (library switch,
refresh, saving settings) via renderBookCard in dashboard.ts, which was
still the old markup with no .book-card-action overlay. So the read
button appeared on the server-rendered cards but vanished as soon as the
dashboard re-rendered, while the bookshelf (always templ-rendered) kept
working.
- Rewrite renderBookCard to match the templ BookCard: detail link plus
the play/read action overlay, routing to the reader or the detail page
when the book has an active conflict.
- Add has_conflict to the BookInfo TS type and stamp it in the dashboard
sections API (GetSections) so client-rendered cards can route correctly.
- Add pointer-events-none / group-hover:pointer-events-auto to the
client-rendered carousel nav buttons so they no longer swallow hover
over edge cards, matching the templ fix.
John O'Keefe
john-okeefe