Device-synced highlights stored POINT CFIs (epubcfi(.../8/1:1)); the
overlayer resolves those to a collapsed range and paints nothing, so
KOReader-made highlights were listed in the drawer but invisible on
the page. mapHighlightRow now builds a renderCfi: a proper RANGE CFI
(epubcfi(base,/start,/end)) synthesized from the stored start/end
points. It also repairs stale rows: missing ends (old web highlights)
and degenerate document-start ends (the old converter fallback) are
derived from the start offset plus the selection text's UTF-16
length. All overlay drawing, navigation (showAnnotation), and the
post-create/post-edit re-adds use renderCfi. Verified in-browser
against live device-synced rows: the paginator's overlayer paints
the highlight rects after the fix.
Both directions synced data but rendered nothing:
- Web reader <- devices: highlights painted no overlay. Device pushes
resolve their start xpointer exactly (text-search anchored by the
selection) but the end conversion carries no context and fell back
to a document-start CFI (epubcfi .../1:0) — a garbage range end.
When the start resolved exactly, the end is now derived from it:
same node, character offset advanced by the selection's UTF-16
length (extendCFIByLength). Same repair when SERVING to devices,
where old web highlights (no end anchor) and converted range CFIs
both collapsed pos1 onto pos0 (extendXPointerByLength on the
xpointer form) — KOReader drew zero-width highlights.
- Colors: KOReader paints from a fixed name set (Blitbuffer
HIGHLIGHT_COLORS), the web uses hex swatches; neither understood
the other, so device colors fell back to defaults and web hex drew
nothing useful on devices. Both boundaries now translate: ingest
maps names to hex (default #ffd54f), GetMetadata maps hex to names
(default yellow) — per-datatype edits re-push with the editing
side's color, which LWW then propagates. SyncBookmarks endpoint
aligned to the same mapping and default.
Web highlights stored only epubcfi_start, so devices received
degenerate pos0 == pos1 (zero-length) highlight ranges. The reader
now collapses the selection range to its end point for a second CFI
and stores it as epubcfi_end (PDF rect anchors reuse the JSON anchor
for both ends).
ConvertToCanonical/ConvertFromCanonical built a fresh CFIConverter
per call, and each annotation converts twice (pos0+pos1) — a book
with 200 highlights re-opened and re-parsed the EPUB 400+ times per
sync, and again per metadata pull. A bounded 8-entry cache keyed by
path now shares converters (the parsing work belongs on the server;
clients stay thin). CFIConverter gained a mutex around its lazily
built spine/doc caches since instances are now shared between
concurrent requests.
Adds CFIConverter.SectionPercentage: book-wide percentage for a CRE
xpointer from the spine char distribution (midpoint of its document)
— the server-side counterpart to dropping per-annotation
getPageFromXPointer lookups from the plugin.
Two blockers, diagnosed by simulating the plugin against the live
server with real library books:
1. Every KOReader progress push carrying annotations failed the JSON
bind with 400 ('cannot unmarshal string into ... chapter/page of
type int') — the plugin sends chapter:'', page:'30', and for CRE
documents page:'/body/...' — so annotation sync AND progress sync
failed together. KOReader annotation chapter/page now use FlexInt,
which accepts numbers, numeric strings, empty strings, and
non-numeric strings (decoding to 0). The server is deliberately
liberal here so thin clients can send raw bookmark data.
2. GetMetadata served locators KOReader cannot place, so pulled items
were junk: web bookmarks leaked 'cfi:epubcfi(...)' positions, web
PDF highlights had empty pos0 (skipped by the plugin, invisible),
and web deletions carried no pos0 so tombstones never matched.
New koreaderPos0 resolver handles every source: device-native
xpointers pass through untouched (round-trip identical, verified),
web PDF JSON anchors map to their page number, EPUB CFIs convert
to CRE xpointers (selection text passed as text-search context for
exact anchoring), 'page:N' positions strip to the bare number.
Unresolvable annotations are skipped with a log line instead of
poisoning devices; tombstones get pos0 injected from the new
locator columns.
Also: thin clients omit per-annotation percentages (paging docs still
send arithmetic page/total); the server derives them — section
midpoint from the spine char distribution for CRE documents, page/
page-count for fixed formats.
GetTombstonedAnnotationsForBook now also returns each tombstone's
start_position/end_position and epubcfi_start/end (note: position/
epubcfi_location, bookmark: position/cfi_position), so serving code
can resolve a device-native locator for deletions of web-created
annotations, whose device_sync_data carries no pos0.
Full reader redesign across 22 commits (with the foliate-js fork's
zoom-control engine work pinned per release):
- Phase 0: panel/chrome stabilization, bookmarks end-to-end (REST CRUD
via AnnotationService), dead UI removal, tombstone resurrection fix
- Phase 1: edge-to-edge glass chrome with auto-hide, slide-over drawers,
tri-state PDF pointer mode (Smart/Pan/Text), Kindle-style theme swatches
- Phase 2: touch gesture engine (pinch/pan/swipe/double-tap), tap zones,
mobile sheets + compact toolbar with overflow menu
- Phase 3: EPUB highlights & notes (selection popover, overlayer
rendering, annotations drawer), PDF text highlights (fraction-rect
overlays), in-book search for both EPUB and PDF, back-to-location
stack, page thumbnails, shortcuts help modal, desktop edge zones
- Phase 4: webtoon (vertical-scroll) mode for comics, brightness/
contrast/night filters, bookmark toast feedback
- Build hygiene: vite stale-chunk cleanup, browser-verified fixes for
Alpine proxy/dpr/duplicate-key classes of bugs along the way
The initial webtoon commit's IntersectionObserver (shadow-host root)
never delivered intersections in Chromium, leaving pages blank.
Scroll-driven loading in e448d36 fixes it; verified end-to-end in a
real browser: pages render (content-rich screenshots), deep scroll
advances the reading position (7/10) and progress readout, filters
visibly change both webtoon images and PDF pages via ::part(filter)
(brightness 5% -> 57% smaller screenshot), paged comics still use
foliate-fxl, and webtoon UI gating (zoom/spread hidden) works.
Phase 4 of the reader redesign (foliate-js ea268df):
- Webtoon mode for comics: continuous vertical scroll of all pages
(900px centered column on wide screens), lazy-loaded with a 150%
IntersectionObserver margin, far pages unloaded to bound memory
with stable aspect-ratio placeholders so the scrollbar never jumps.
Chosen per book (Paged | Webtoon segmented control in Settings →
Layout & Display; stored in localStorage per media item since a
webtoon title and a paged manga volume want different flows).
Toggling reloads the reader — the renderer is chosen at open time —
and progress restores from the saved page. Relocate events flow
through the same pipeline, so the slider, progress saving, back
stack, tap zones, and edge zones all work unchanged. Zoom/fit/
magnifier/spread controls hide in webtoon (natural-width scroll).
- Display filters for fixed-layout: brightness (30-130%) and
contrast (70-130%) sliders with live preview, plus Night Mode
(invert) — also a quick row in the ⋯ tools menu. One --fx-filter
CSS var drives everything: ::part(filter) on foliate-view iframes
(forwarded via the new exportparts attribute) and the webtoon
page images alike. Persisted as fx_brightness/fx_contrast/fx_invert
(types + defaults both sides); Restore Defaults resets them.
Help menu (the reader had a growing shortcut/gesture vocabulary with
no discoverability): a ? topbar button, the '?' key, and F1 open a
glass modal listing navigation, zoom/pan, highlight, and touch
gesture reference — format-aware (fixed-layout/PDF rows appear only
where they apply), Esc closes it first in the dismiss chain.
Desktop edge zones: clickable page-turn strips on the left/right
viewport edges (8% width, 44-72px), desktop only (hover+fine-pointer
media query — touch devices use tap zones, avoiding double paging).
Hovering reveals a chevron arrow and a subtle edge gradient. Zones
disable (pointer-events pass-through) while a fixed-layout page is
zoomed so edge clicks belong to content: panning, selection,
highlight editing. fxZoomed tracks zoom state via the renderer zoom
event, reset/fit actions, and init.
The 🏷️ bookmark button (and the 'b' shortcut) saved silently — an
accidental click gave no reaction at all. addBookmark() now shows a
short success toast ('Bookmark added — <progress>') using the
existing toast system, which the reader bundle hadn't been importing.
Importing it also activates the shared fetch interceptor, so failed
reader API calls (incl. bookmark saves) surface error toasts instead
of being swallowed.
Diagnosed in a real browser (playwright/chromium against the running
app + Head First SQL): the engine's book.toc held all 18 entries with
correct labels/hrefs and the tab counter even showed 380, yet zero
links rendered while the console flooded with 'Alpine Warning:
Duplicate key on x-for'.
Root cause: the drawer keyed TOC rows by item.href. PDF outline
entries frequently share the same destination (e.g. the printed TOC
page is targeted by several bookmark entries), so flattened items
carried duplicate keys — and Alpine's x-for renders NOTHING for a
duplicated key, not even the unique ones. EPUB TOCs never collided
because their hrefs are unique file paths, which is why this only
surfaced on PDFs.
Key is now href + row index (the list is static once loaded, so
positional keys are safe). Verified end-to-end in the browser: 18
entries render and the drawer populates.
Investigation: the contents drawer read book.toc, which makePDF
builds from pdf.getOutline() — verified against the real library PDF
(Head First SQL) through the exact vendored pdf.js build AND the exact
range transport the browser uses: 18 chapter entries come back. So
the source is right; manga-scan PDFs and CBZs simply have no embedded
outline, which made Contents look broken exactly where users expect
page-based navigation.
- TOC now populates eagerly right after the book opens (toggle-time
lazy population removed), so an existing outline can never silently
miss due to timing; the drawer keeps the honest empty-state text
for books without outlines.
- New 'Pages' tab in the contents drawer for fixed-layout books:
a Kavita-style thumbnail grid (3-up, current page highlighted and
scrolled into view, click to jump — recorded on the back-to-
location stack). Thumbnails render client-side: PDFs via the
in-memory pdf.js document (small viewport render, Alpine.raw
unwrap); comics via the page's image blob drawn down to a 110px
canvas, then unloading the full-size blob so thumbnailling doesn't
hoard page images. Lazy via IntersectionObserver scoped to the
drawer's scroll container (200px margin), canvases cached at module
level so revisits are instant; failures warn in console and allow
retry. The backend /readers/thumbnails endpoint turned out to be an
empty stub, so nothing server-side was worth wiring.
The display:none for the full toolbar lived in @layer components while
the div also carried Tailwind's flex utility (@layer utilities). Layer
order beats specificity, so the utilities layer always won and the
full bar never hid below the breakpoint (the compact row only worked
because it had no display utility of its own).
Switch to Tailwind's own responsive utilities in the markup — full
toolbar 'hidden md:flex', compact row 'flex md:hidden' — and delete
the custom rules; responsive display now resolves inside a single
layer where source order (responsive variants after base) guarantees
the right winner.
Wrapping alone isn't how polished mobile readers work. Adopt the
standard pattern (Kindle/Apple Books/Mihon) responsively:
- >= 768px: the full fixed-layout toolbar stays (wrap still absorbs
mid-size widths) — power users keep one-click zoom/fit/spread.
- < 768px: single-line compact row — page back, back-to-location pin,
slider, page forward, progress, and a ⋯ overflow button. No
wrapping, no horizontal scroll.
- ⋯ opens a glass menu anchored above the bar with LABELED rows
(Zoom −/%/+, Fit, Page position/Recenter, Magnifier, Pointer
Smart/Pan/Text, Double page, Contents) — labels beat mystery icons
on touch. Pointer row hides for comics; menu scrolls if tall.
- Dismissal: Esc, outside click (⋯ button exempt so it re-toggles
cleanly), opening any drawer or TOC closes it; hides with the
chrome. Compact slider registered in progressSliders() so all
three stay in sync with relocate events.
- Back-to-location moves from the topbar (where it sat between Back
and the title, too subtle and disconnected from navigation) into
both bottom-bar rows, beside the page-back arrow — the natural
'go back' cluster. New icon: a location pin, clearly distinct from
the back arrow and page controls. Appears only when the stack has
a return target; Alt+← unchanged.
- New recenter button in the fixed-layout row (crosshair icon, next
to zoom): resets pan offsets while keeping the current zoom —
backed by foliate's new recenter() (1c812e8), which zeroes the
wrapper translate and re-syncs the spread side.
- Both bottom-bar rows wrap gracefully on narrow windows instead of
overflowing/h-scrolling: controls are grouped (paging+back | slider |
fit+zoom+magnifier+recenter | pointer mode | spread | progress+TOC)
so groups flow to a second line at small widths; the slider shrinks
first (grow + min-width), everything else stays whole. Fixed-layout
row drops its overflow-x-auto.
Highlights landed on the right line but shifted right and oversized
on any display with devicePixelRatio != 1. Cause: selection fractions
divided the textLayer span rects by documentElement's screen rect,
but pdf.js scales the iframe's <html> by 1/dpr — that rect is dpr×
smaller than the visible page, inflating every x/w fraction by dpr
(on a 2× display a highlight started twice as far right and was twice
as wide). dpr=1 displays were coincidentally correct, which is why
the geometry looked sound when written.
The denominator is now the rendered canvas (#canvas canvas), whose
post-transform rect IS the visible page and shares the textLayer's
transform space — the dpr scaling cancels exactly. Comics keep the
img denominator; a viewport fallback covers any page without either.
The popover-placement scale factors (frame/denominator) become 1 for
PDFs as a side effect, fixing popover drift too. The fork's click
hit-test (86e234d) gets the same canvas-aware denominator so clicking
highlights opens the editor at the right spot.
Highlights saved before this fix stored dpr-inflated fractions and
will still render misplaced — delete and re-create them.
PDF search diagnosis: extraction and matching were proven correct
against the real 609-page library PDF (pdfjs 5.5.207, incl. the exact
range-transport setup makePDF uses — 841 hits for 'SELECT'), and the
served bundle had every piece. The failure was Alpine's reactivity:
this.book is a plain object, so reading .pdf through component state
returns a reactive Proxy around the PDFDocumentProxy — and pdf.js
v5 uses #private fields, so getPage() through the proxy throws
'cannot read private member', which the empty catch rendered as a
silent empty result set. runPdfSearch now unwraps via Alpine.raw
(falls back to the raw read), and search failures surface in the
drawer ('Search failed — see console') plus console.warn instead of
masquerading as 'No matches'.
Back-to-location stack (research/footnote workflow): the current
position is recorded before every programmatic jump — search-result
clicks, TOC entries, bookmark and highlight jumps — and on every
internal link click (footnotes, cross-references) via foliate's
'link' event. A ↩ button appears in the topbar once a return target
exists; Alt+← works everywhere. Ordinary paging never pollutes the
stack (max depth 50, consecutive duplicates collapse).
PDFs have fully searchable text (pdf.js text layer) — the previous
reflowable-only gate existed only because foliate's generic search
needs DOM documents that PDF sections don't provide. This adds a PDF
pipeline alongside it:
- Fork d065495 exposes the pdf.js document proxy as book.pdf so the
host can drive text extraction directly.
- New web/src/reader/pdf-search.ts: extractPdfPages() pulls each
page's textContent with item geometry (progress-reported, cached
after first search). PDF text items often omit inter-word spaces
(gaps are positional), so pages are joined gap-aware — baseline
changes, hasEOL, or horizontal gaps past a font-size threshold
become spaces — recording a char→item map. searchPdfPages() does
case-insensitive matching over the joined text and maps each hit
back to the page-fraction rects of the items it spans, with
ellipsized pre/match/post excerpts. Pure functions, unit-sanity
checked (cross-item 'brave new' → two rects).
- runSearch branches: EPUB keeps foliate's DOM search; PDFs search
the extracted pages, group hits per page ('Page 12'), and render
on-page hit rectangles through the existing fraction-rect overlay
(addRectAnnotation) — which re-render automatically when pages
revisit, same as highlights. Clearing the query removes them.
- Results navigate by page index; the 🔍 button and '/' shortcut now
appear for PDFs too (comics remain without searchable text).
Wires foliate's search engine into the new drawer system:
- 🔍 topbar button (reflowable-only; PDF/comic sections have no
searchable text documents) and the '/' keyboard shortcut open a
Search drawer: query input (Enter to run), live progress while
scanning (per-section percent), match count, and results grouped
by section with TOC labels.
- Each result shows pre/match/post excerpt rendered as three text
nodes (no x-html — book content never enters the DOM as markup);
the match is styled with a translucent <mark>. Clicking jumps to
the hit's CFI and closes the drawer.
- Hits are drawn on the page through foliate's overlayer (outline
style) and persist across page turns — the engine re-applies
search results when a section's overlay is created. Clearing the
query removes the outlines.
- A generation counter discards results and progress from superseded
searches (rapid re-query), and starting a new search clears the
previous one server-side via view.clearSearch().
- Search integrates with the drawer system: scrim, Esc-to-close,
one-drawer-at-a-time, / focuses the input via .
Two bugs broke the Phase 3b PDF highlight flow end to end:
1. Selection capture never attached: reader.ts read renderer.isPDF
before view.init() rendered the first spread, but the renderer
only sets that flag once frames exist (PDF frames carry pdf.js
onZoom). The stale undefined copy gated the pointerup selection
listener off, so selecting PDF text did nothing. The listener now
gates structurally on the loaded document having a .textLayer
(true for every PDF page, false for comics), and isPDF is re-read
after init — which also finally makes the Smart|Pan|Text control
and the saved pointer mode apply on PDFs.
2. Highlights rendered invisibly: the overlay SVG lived inside the
page iframe, whose <html> pdf.js scales by 1/devicePixelRatio —
shrinking the overlay into the top-left corner on any dpr != 1
display. The fork (1c0ebf3) now renders annotation rects
host-side, inside the frame wrapper element, positioned in
percentages of the visible page box — immune to the html
transform, zoom re-renders, comic iframe scaling, and pan/zoom.
Phase 3b of the reader redesign — highlighting for fixed-layout PDFs:
- Select text on a PDF page → same glass popover as EPUBs (colors,
note, copy). The selection's client rects are normalized to
page-fraction quads using a transform-inclusive denominator so
pdf.js's devicePixelRatio scaling on <html> cancels out, then
stored as a JSON anchor {page, rects} in epubcfi_start.
- Rendering goes through the fork's new rect-annotation pipeline
(foliate-js aba68d8): a full-bleed viewBox-0-100 SVG inside the
page iframe, so highlights stay aligned through pan/zoom, iframe
CSS-scaling, and PDF hi-res re-renders with zero re-anchoring.
Frames carry their page index and re-render annotations when
recreated on spread changes.
- Clicking an existing highlight hit-tests in fraction space and
opens the edit popover (recolor, note, copy, delete) at the
host-space click position; drag-selecting text never triggers it.
- Annotations drawer: PDF highlights jump by page index; notes and
recolors round-trip through the same LWW/dedup sync path as EPUBs
(same dedup key derivation on the JSON anchor).
- Comics keep bookmark-only highlighting (no text layer) by design.
Phase 3 (EPUB half) of the reader redesign:
- Select text in a reflowable book → floating glass popover at the
selection (5 colors, note, copy). Clicking a color creates the
highlight via POST /api/media-items/:id/highlights, anchored by the
foliate range CFI (epubcfi_start) with percentage position.
- Highlights render through foliate's overlayer pipeline: draw-
annotation draws Overlayer.highlight with the stored color,
create-overlay re-adds persisted highlights as sections load,
show-annotation opens the edit popover when a highlight is clicked
(recolor, edit note, copy, delete).
- Backend: highlight create/update accept epubcfi_start/end,
note_text, and percentage fields; position validation relaxed
(CFIs exceed the old 100-char cap); PUT routes through
AnnotationService.SaveHighlight so edits get dedup/LWW treatment
and actually persist note_text (the plain query can't).
- Bookmarks drawer becomes the Annotations drawer with tabs:
Highlights (color-bar list, note previews, jump/edit/delete),
Notes (add note at current position, list, delete — backed by the
existing notes API), and Bookmarks (unchanged behavior).
- Popover dismissed on outside click, collapsed selection, page
navigation, or Esc (new top-priority Esc branch).
emptyOutDir is false because web/static also holds tracked assets,
so *-<hash>.js chunks from every previous build accumulated
indefinitely and leaked into Docker images via the build context
(the reader serves whichever chunk the import chain names, so the
orphans are pure confusion + bloat). A closeBundle plugin now
deletes any hashed chunk this build did not produce.
Phase 2 of the reader redesign:
- Fixed-layout touch engine (foliate-js e9e61d8): pinch-zoom around
the midpoint, two-finger pan, single-finger pan while zoomed,
horizontal swipe page-turn at fit (RTL-aware via next()/prev()),
and double-tap to zoom 2.5x / reset. Touch events forwarded from
page iframes with converted coordinates; preventDefault only when
the engine consumes the gesture, so PDF text selection and native
taps stay intact. touch-action: none on the host and in comic/pdf
page documents keeps the browser from fighting the engine.
- Tap zones (Kindle-style) for touch devices: tap the outer margins
to page, center to toggle chrome. Size configurable (10-50%) via
the revived tap_zone_size setting; toggle via new tap_zones_enabled
(Behavior section of the settings drawer). Pointer-based + passive
so drags/swipes/selection never trigger; attached both to the
viewport and inside every page document (iframe events don't
bubble); debounced 280ms so double-tap zoom doesn't also page; no
zone actions while a fixed-layout page is zoomed.
- Drawers become full-width sheets on screens <= 640px.
Deleting a bookmark/highlight/note and then re-adding the same content
at the same position (same dedup key — e.g. the reader's auto-titled
'Bookmark at X%') was silently swallowed: the save hit the tombstone
branch, returned 201 with the deleted row, and the list (which filters
deleted) stayed empty. Bookmarks were further blocked by the
UNIQUE(media_item_id, user_id, title) slot the tombstoned row holds,
and notes had no TTL escape at all.
Tombstones now only block saves that predate them (stale replays from
a device that still has the annotation). A save whose modification
time is newer than max(deleted_at, last_modified_at) — a deliberate
re-create from the web or a device — resurrects the row via the LWW
update queries, which now clear deleted/deleted_at.
Modernize the reader chrome bars without touching the drawer system:
- Bars become theme-tinted glass: 70% bg-primary translucency over
the edge-to-edge page, 18px backdrop blur + saturation, hairline
translucent borders, soft directional shadows (single .reader-glass
class owns the effect; replaces solid opaque backgrounds and the
tailwind backdrop-blur that would override it).
- Chrome hide/show now slides the bars off-screen (translateY) in
addition to the opacity fade, via .chrome-hidden on #reader-chrome.
- Theme-aware hover pills (translucent currentColor tint) replace
hard-coded gray-700 hovers; focus-visible rings added.
- Progress slider gets a custom thin rounded track with a floating
white thumb (webkit + gecko), replacing native range styling.
- Separators and the fit-mode select match the glass language
(.reader-sep, .reader-select).
Phase 1 of the reader redesign:
- Reading surface is edge-to-edge; top/bottom bars overlay
translucently (backdrop-blur) instead of reserving insets, killing
the inset-coordination bug class entirely. Chrome auto-hides after
2.5s of pointer inactivity (chrome_behavior setting finally wired:
auto-hide / always-visible; legacy values map to auto-hide). Pointer
activity inside page iframes keeps it awake; Esc toggles.
- TOC / Settings / Bookmarks become slide-over drawers with a scrim
(z-50, full-height, safe-area aware), replacing the dockable-panel
system and its window-shade headers. Only one drawer opens at a
time; Esc or scrim click closes.
- Bottom bar is contextual: reflowable keeps nav/slider/progress/TOC;
fixed-layout row adds Fit Page/Width select, zoom cluster,
magnifier (now shows active state), Double Page Spread toggle, and
a Smart | Pan | Text segmented control replacing the cryptic
two-state icon. Smart = text-aware drag; Text = selection-only
(manual smart-detect off); Pan = force pan. Choice persists via
pdf_interaction_mode (new setting + foliate 29bc958 'text' mode).
- Settings drawer: Behavior (chrome, progress mode), Appearance with
18 Kindle-style theme swatches (single source of truth from
THEME_COLORS), Typography, Layout — each scoped by format.
- Keyboard: t/s/b open TOC/settings/bookmark, Esc closes drawers
before toggling chrome, shortcuts skip form inputs; both slider
rows tracked correctly (no duplicate-ID lookups).
- Topbar: Back, title, add-bookmark, bookmarks drawer, Aa settings;
chrome follows user theme.
Phase 0 of the reader redesign:
- Panels no longer render under the top/bottom bars: sidebars get
measured insets (same resize/safe-area mechanism as the viewport);
panel max-height now derives from the bounded sidebar instead of a
100vh guess; right-side border targets the actual sidebar.
- Bookmarks work end-to-end for the first time: REST CRUD under
/api/media-items/:id/bookmarks (create/delete route through
AnnotationService for dedup/LWW/tombstones), fix UpdateMediaBookmark
referencing nonexistent updated_at column, frontend posts to the
real API with per-format position (CFI vs page), live list with
jump + delete instead of SSR-only snapshot.
- Fix chapter matching in progress saves: boundaries were compared by
a nonexistent tocItem property, so chapter was never persisted.
- Remove dead UI: Navigator panel stub, empty dictionary popup shell,
unwired Chrome Behavior select; purge 160 stale build artifacts.
- Reader chrome now follows the user's app theme instead of hardcoded
theme-tokyo-night.
Complete the SHA-256 lifecycle for preexisting databases: items
imported before hashing existed get hashed automatically, and any
content duplicates discovered in the process land on the new admin
Hash Conflicts page for an explicit keep/merge decision.
HashBackfillService (runs once 30s after startup, independent of
auto-scan):
- hashes every media_items row where file_sha256 IS NULL, resolving
each path through LibraryService; per-item failures are logged and
skipped so one unreadable file cannot block the pass
- no-op once everything is hashed (logged and skipped)
- finishes with a conflict sweep flagging every content-duplicate
group via FindHashConflictGroups + CreateHashConflict; the sweep
runs after the per-item pass because a preexisting pair only
becomes detectable once both sides have their hash
API (admin-only):
- GET /api/admin/hash-conflicts - pending groups with member items
and usage counts
- POST /api/admin/hash-conflicts/:id/resolve - action=keep_all, or
action=keep with keep_uuid: validates the uuid belongs to the
group, re-parents every other copy's child rows onto the kept item
(reparent_media_item_children), deletes the losers, and records
the resolution + resolving admin; accepts form or JSON bodies and
returns the htmx resolved fragment
Page route /admin/hash-conflicts (admin-only) renders the template
with hydrated conflict data; HashConflictsHandler wired into the
router Config and constructed in main.
Verified end-to-end against the live database: duplicate detection,
pending listing, keep_all resolution, merge path (re-parent +
delete), and - critically - a resolved group is not re-flagged by a
later sweep (upsert no-op). Database restored afterward.
New /admin/hash-conflicts page (admin-only) listing pending
content-duplicate groups. Each group card shows the library, a
shortened SHA-256, and one row per copy with title, author, path,
size, and per-copy reading-data counts (progress, highlights,
bookmarks, notes, collections) - copies that own user data are
highlighted so the keep choice is informed.
Per copy: 'Keep this copy' merges the other copies' child rows into
it and deletes them. Per group: 'Keep both' for intentional
duplicates. Both confirm first, resolve via htmx POST, and swap the
card for a resolved confirmation inline. The confirmation fragment is
built inline in the handler rather than the templates package
(templates imports handlers; a back-import would be a cycle).
Empty state shown when no conflicts are pending. Adds a 'Hash
Conflicts' entry to the admin sidebar section between Libraries and
Users.
Force rescan was metadata-only: updateMediaItem never touched the
hash identifiers, so a force scan could not backfill file_sha256 for
items imported before hashing existed (or where extraction originally
failed). Those items were invisible to content dedup and SHA-256
device matching with no way to fix short of delete + re-import.
processMediaFile now refreshes hash identifiers in three cases:
- force rescan (the admin Scan button becomes the backfill tool)
- file size change (stored hash is stale - the bytes changed)
- unchanged file with no stored hash (ordinary scans self-heal the
legacy backlog incrementally, no admin action required)
Each recompute runs recordHashConflictIfAny: when the freshly stored
hash is now shared by more than one item in the library, the group is
upserted into hash_conflicts for the admin Hash Conflicts page. The
upsert is a no-op for already-tracked groups, so resolved 'keep both'
decisions stick.
Also extract a package-level computeFileSHA256 (the scanner method
now delegates to it) so the startup backfill service can hash files
without a scanner instance.
Content duplicates (same library + file_sha256 at different paths,
e.g. the same book imported twice under two names on a preexisting
database) cannot be auto-collapsed the way path duplicates were:
keeping both copies may be intentional. Surface them for an explicit
admin decision instead.
Schema:
- new hash_conflicts table keyed (library_id, file_sha256) with a
status/resolution lifecycle: 'pending' until an admin resolves via
'keep_all' or 'kept:<uuid>' (which copy was kept after merging)
- resolution is VARCHAR(50) - 'kept:<uuid>' is 41 chars; include a
widening ALTER for databases created with the initial 30-char width
- resolution/resolved_by/resolved_at record who decided what and when
Queries:
- ListMediaItemsMissingHash: items imported before hashing existed
(file_sha256 IS NULL), ordered oldest-first for the backfill pass
- FindHashConflictGroups: the content-duplicate group detection
(GROUP BY library_id, file_sha256 HAVING COUNT(*) > 1)
- ListMediaItemsBySHA256AndLibrary: full membership of one group
- CreateHashConflict: upsert with DO NOTHING so already-tracked groups
are untouched - critical behavior: a group an admin resolved as
'keep both' is never re-flagged by later sweeps
- ListPendingHashConflicts: admin listing with library name and live
item counts (items may have been deleted since flagging)
- GetHashConflict / ResolveHashConflict: lifecycle
- GetMediaItemUsageCounts: per-item progress/highlight/bookmark/note/
collection counts so the admin can make an informed keep choice
- ReparentMediaItemChildren: sqlc binding for the existing
reparent_media_item_children() migration function, used to merge a
losing copy's child rows into the kept copy
Bring the koreader protocol doc in line with the hash-sharing work:
- Request table: uuid is no longer required (it is absent on the first
sync of a newly downloaded book); document sha256 and file_path and
the resolution priority uuid -> sha256 -> file_path alias ->
title/author
- Note that SHA-256 matching is format-aware (media_items hash first,
media_item_formats fallback) so converted KEPUB/PDF downloads match
- Document the sha256 field returned by the metadata and library
endpoints
- Add a 'Book identification' section pointing current and future
clients (koreader, kobo, OPDS, device-link UI, mobile apps) at the
shared BookResolver as the single resolution path
The sidecar config's books map is keyed by the primary SHA-256 (UUID
fallback). A device holding a converted format (KEPUB/PDF) whose hash
lives only in media_item_formats could not resolve its file through
the sidecar.
After inserting the primary-keyed entry, also register the same entry
under each per-format hash from media_item_formats (first write wins,
so a primary hash is never shadowed). Devices now resolve converted
files via the sidecar the same way the server's BookResolver does.
Applied to both the GET and download sidecar builders.
DownloadBook populated fileSha256 only for the kepub and pdf format
branches, so the default EPUB download never emitted the
X-Bookhoard-SHA256 response header - the hash was only available in
the feed metadata, not on the download response itself.
Populate it from mediaItem.FileSha256 in the default branch so every
download response carries the canonical primary-format hash. Clients
that capture response headers at download time now learn the hash
regardless of which format they requested.
Fixes the 'cannot push until pulling first' wall on books downloaded
via OPDS. Root cause chain: the bookhoard koreader plugin only learns
the book UUID from a successful push response, but the first push had
to match by SHA-256 alone - and that match consulted only
media_items.file_sha256, missing converted formats. When the hash
missed, no UUID was returned, so pull stayed blocked (it requires the
UUID) and the book could not sync at all.
Resolution side - route all five SHA-256 match sites through the shared
BookResolver so they are format-aware:
- resolveBookToMediaItem priority 2
- SyncBookmarks book-level lookup
- per-bookmark, per-note, and per-highlight override lookups
Exposure side - return the canonical hash so clients can learn and
cache it from a pull regardless of how the book was obtained:
- KOReaderMetadata gains sha256, populated from mediaItem.FileSha256
- KOReaderLibraryBook gains sha256, populated the same way, so the
library list endpoint carries it for every book
Together with the plugin-side UUID bootstrap (bookhoard.koplugin),
push and pull now work in either order on any format.
The platform had three duplicated, divergent book resolvers (koreader,
kobo, BookMatchingService) and none of them consulted
media_item_formats.file_sha256 - per-format hashes for converted files
(KEPUB, PDF) are computed and stored at import/conversion time but were
never used for lookup. GetMediaItemFormatBySHA256 existed with zero
callers. Any client holding a converted file could never match by
hash.
Add internal/services/book_resolver.go: a single shared resolution
path from client-supplied identifier to media_item.
ResolveBySHA256 checks media_items.file_sha256 first (indexed
GetMediaItemBySHA256), then falls back to media_item_formats.
file_sha256 (indexed GetMediaItemFormatBySHA256, first caller) so a
converted format matches with equal confidence. The import-time
SHA-256 is the canonical identifier shared by every interface.
Wire two of the existing resolvers through it:
- BookMatchingService.matchBySHA256: replaces the in-memory
ListMediaItems scan of up to 1000 rows with the resolver's indexed
lookups, and gains format awareness for the link/auto-link UI.
MatchMethod now reports sha256_sha256 or sha256_sha256_format
- KoboHandler.mapContentIdToBookhoardUUID: the SHA-256 heuristic
branch (ContentId that looks like a 64-char hash) now resolves
format-aware too. Kobo's entitlement_id wire identity is untouched;
only the opportunistic hash branch changed
A read-then-write race in processMediaFile allowed the same file to be
imported twice: two concurrent scan jobs (startup scan, fsnotify dirty-
directory scan, periodic backup poll, or a manual scan each run on
separate worker goroutines with separate MediaScanner instances) could
both SELECT 'not found' and both INSERT. There was no transaction, no
row lock, no unique constraint on (library_id, file_path), and no
ON CONFLICT clause, so nothing stopped the double insert. Observed in
production as two identical 'Head First SQL' rows created in the same
second (same sha256, size, path, library).
Database enforcement:
- schema.sql: add UNIQUE(library_id, file_path) constraint, guarded so
re-runs don't error
- schema.sql: add self-healing migration that runs on every startup -
dedup_media_items_by_path() collapses existing path-duplicates and
reparent_media_item_children() moves all child rows (progress,
highlights, bookmarks, notes, collections, formats, aliases, kobo
entitlements, etc.) onto a survivor before deleting losers, so the
constraint applies cleanly on already-duplicated servers without
losing reading history. Survivor picks the row with the most user
data, ties broken by lowest id
- CreateMediaItem: upsert via ON CONFLICT (library_id, file_path) DO
UPDATE so concurrent inserts collapse to one row and return it
- CreateMediaItemFormat: upsert via ON CONFLICT (media_item_id,
format_type), closing the same race on format rows
Application-level guards:
- media_scanner processMediaFile: after computing the file hash, check
GetMediaItemBySHA256AndLibrary (new query) and treat the file as
existing when identical content is already in the library under a
different path (content dedup, library-scoped so multi-library
setups still work)
Ops tooling:
- scripts/dedup_media_items.sql: standalone idempotent maintenance
script with a dry-run report (path + content duplicate groups, child
row counts) and transactional cleanup, for servers that prefer to
dedup manually before upgrading
Verified against the live database: the duplicate pair was collapsed
(reading_progress preserved on the survivor), schema.sql re-runs are a
no-op, and the constraint is in place with 62 unique books remaining.
Switch the @bookhoard/foliate-js dependency from the github: shorthand
(d164d6f) to the explicit git+https URL form (d4d87a9). The newer
revision is required by the double-page-spread support (renderer
'spread' attribute) and the explicit URL form resolves more reliably
across npm/podman builds.
The double_page_spread checkbox in the reader settings panel was inert:
it had no Alpine binding, no apply logic, and no persistence. Default
was also inconsistent (false in settings-manager, absent from server
defaults).
- Add doublePageSpread state to the reader Alpine component, loaded
from saved settings (default true)
- Add applyDoublePageSpread() which sets the renderer's 'spread'
attribute to auto/none and persists the setting via saveSettings
- Apply the spread attribute during fixed-layout renderer init
- Bind the settings checkbox with x-model and @change
- Add double_page_spread: true to ReaderService server defaults so
new users get the same starting value the client expects
- Also improve the PDF pan/select toolbar button: distinct smart-
select vs pan icons, highlighted state while pan mode is active,
and dynamic tooltips/aria-labels explaining each mode
The scan-complete handler in dashboard.ts attempted to deduplicate book
cards by querying [data-media-item-id], but neither the client-side
renderBookCard nor the server-side BookCard template ever set that
attribute. As a result the dedup Set was always empty, every item from
the API response was treated as new, and all items were prepended via
insertAdjacentHTML('afterbegin', ...) on every 5-minute scan — causing
visible duplication (doubling, tripling) that only cleared on page
refresh.
Fix by replacing the fragile dedup-and-prepend logic with a per-track
full innerHTML replace. This is simpler, correctly handles items that
should be removed after a scan (the old code never removed anything),
and also removes stale sections no longer returned by the API.
Additional hardening:
- Add data-media-item-id to both renderBookCard (dashboard.ts) and the
server-side card wrapper (dashboard.templ) so server-rendered and
JS-rendered cards are structurally identical.
- Guard the bookhoard:scan-complete listener registration with a
module-level boolean (scanListenerRegistered) so the handler cannot
accumulate if Alpine ever re-inits the body subtree.
- Remove debug console.log statements from the scan handler.
Sidebar appearance menu improvements:
Accordion behavior:
- Lift panel open/close state to a shared 'openPanel' variable on the
parent container so only one sidebar panel (User, Appearance, Admin,
Sign In) can be open at a time; all can be closed.
- Admin panel still auto-opens on /admin/* pages via initial state.
- Add chevron rotation to User and Appearance panels (previously only
Admin rotated); add a chevron to the Sign In panel for consistency.
Wood texture previews:
- Generate 48x48 WebP thumbnails (~200 bytes each) from the full-size
PNG textures (873 KB – 1.9 MB) so the bookshelf option circles show
the actual wood grain instead of a flat grey dot.
- Use unquoted url() in the inline style to avoid templ's double-HTML
escaping of single quotes (SanitizeStyleAttributeValues + EscapeString
turned url('...') into url('...) which is invalid CSS).
- 'None' keeps the flat neutral circle.
CSS resilience:
- Move the wood background-image: url() rules from the compiled
style.css into input.css (the Tailwind source) so they survive CSS
rebuilds instead of being silently lost.
The checkmark in the Appearance theme menu was rendered server-side
(if user.Theme == opt.Name), so it never moved after switching themes
in the browser.
- Always render the check for every theme option, hidden by default,
using a new themeCheckClass(name, current) helper that returns the
hidden class unless the option is the active theme.
- Give each theme button a data-theme attribute and add
updateThemeIndicators() to web/src/theme.ts, which reads the applied
theme from the body class (theme-<name>) and toggles the hidden class
on each check accordingly.
- Call updateThemeIndicators() from changeTheme (before the async save
and on failure), initializeTheme, and loadUserTheme so the menu stays
in sync with the applied theme.
- Add unit test for themeCheckClass (templates/utils_test.go) and
regenerate templ output.
Bring the tighten-ui brand treatment into new-ui:
- Replace the emoji book (📚) in the sidebar header with the book-open
icon rendered in the theme accent color, matching the tighten-ui
header brand (templates/header.templ).
- Add web/static/favicon.svg (book-open glyph, tokyo-night accent
#7aa2f7 stroke) and reference it from the <head> of all 27 page
templates, so the favicon is present on login/setup/error pages too.
- Regenerate templ output for all affected templates.
CleanupExpiredRefreshTokens and CleanupExpiredOpdsTokens were generated
by sqlc but never invoked anywhere in the codebase, so expired/revoked
tokens accumulated in the database indefinitely. The refresh-token query
was parameterized in the settings-registry work specifically so its
retention window could follow the configurable session duration, but the
periodic caller was never wired up.
annotations.go:
- Rename StartTombstonePurger to StartDailyMaintenance, which now runs
all periodic cleanup tasks from a single 24h-tick goroutine.
- Add runDailyMaintenance helper: tombstones, then OPDS tokens, then
refresh tokens, each logging independently so one failure never skips
the others.
- Refresh-token retention is read from the registry (SessionDuration)
on every tick so live admin edits are honored; guarded on the registry
being wired so unwired test paths simply skip cleanup.
- All three queries only delete rows that are already expired or
revoked, so active sessions are never logged out.
main.go:
- Update the call site: tombstonePurgerCancel becomes maintenanceCancel
and calls StartDailyMaintenance.
Net footprint: still one goroutine and one ticker; the cleanup adds one
DELETE per table per day.
Replace the read-only "System Information" card (which listed hardcoded
values) with editable HTMX forms, organized so the live vs restart
distinction and related settings are visually clear.
admin_settings.templ:
- AdminSettings signature now takes liveGroups and restartGroups
([]SettingGroup) instead of a flat entry list.
- Remove the static System Information list. Render two cards: "Live"
(green, applies immediately) and "Restart Required" (warning header,
saved but only takes effect after restart).
- Within each card, TunableSettingsSection clusters entries into
labeled sub-sections by Group (e.g. "Password Quality", "Device Rate
Limits", "Login Lockout", "Worker Pool") with uppercase tracked
sub-headers.
- TunableSettingRow renders an inline HTMX form per setting: a Yes/No
select for bools, a number input with min/max for ints, text
otherwise, posting to /admin/settings/tunable. Rows show "modified
from default" when the value differs from the compiled default.
types.go:
- Add SettingEntry (template-local mirror of database.SettingEntry,
keeps templates from importing database) and SettingGroup.
utils.go:
- Add GroupTunableSettings: splits a flat, group-sorted entry list into
live and restart []SettingGroup buckets preserving source order.
utils_test.go covers the multi-group + empty cases.
frontend.go:
- The /admin/settings page handler now loads entries from the registry,
drops the three keys that have dedicated UI cards (default_timezone
dropdown, scan_poll_interval_seconds, auto_scan_enabled) so they are
not listed twice, groups the rest, and passes liveGroups/restartGroups
into the template.
Construct the SettingsRegistry at boot, load it, and thread it through
every consumer so the configurable values take effect and stay cached.
cmd/server/main.go:
- Build the registry from the Queries handle and Load() it right after
schema init; a load failure logs and continues (getters fall back to
compiled defaults, so startup is never blocked).
- Wire the registry into the package-level password validator
(SetDefaultPasswordSettings) and call SetSettings on every handler/
service that reads tunables: AuthHandler, DeviceAuthMiddleware,
OPDSHandler, SidecarHandler, SystemSettingsHandler,
AnnotationService, ConversionService.
- Source the restart-time values from the registry: login lockout
(max attempts + duration) feeds NewLoginAttemptTracker, and the new
NewSyncQueueProcessorWithConfig / NewWorkerWithConfig take the sync
queue and worker pool configs.
router.go:
- Config gains a Settings *database.SettingsRegistry field.
- The global auth rate limiter now reads RequestsPerMinute from
registry.AuthRateLimit() (env stays as the enabled/disabled switch
and as the fallback if the registry is unset).
admin_library.go:
- The HTMX scan-settings save endpoint reloads the registry after
writing so the change is visible without a page reload.
- Add PUT /admin/settings/tunable: a small HTMX endpoint that calls
SystemSettingsHandler.ApplySetting and returns a colored status
snippet ("Saved" or "Saved — restart required") for the admin UI's
per-row forms.
Add a single pair of admin-only endpoints that supersede the scattered
scan-settings JSON routes as the canonical way to read and write
tunable system settings. Existing legacy routes are kept working for
backward compatibility and now refresh the registry cache on write.
system_settings.go:
- GET /api/system/settings returns every known setting with full
metadata (value, type, min, max, requires_restart, category, group,
description, is_default) via SettingsRegistry.All().
- PUT /api/system/settings accepts {key, value}; ApplySetting() looks
up the compiled Default for the key, runs type-aware validation
(int range, bool parse, non-empty string, timezone via
time.LoadLocation), upserts via UpsertSystemSetting, reloads the
registry, and reports whether a restart is needed for the change to
take full effect. Shared by the JSON endpoint and the HTMX endpoint.
- Legacy UpdateScanSettings / GetScanSettings / UpdateTimezoneSettings
now reload the registry after writing and prefer the registry when
reading, so the cache stays consistent regardless of entry point.
sidecar.go:
- SidecarHandler gains an optional registry; the timezone branch of
UpdateSystemConfiguration (PUT /api/system/config) calls
settings.Reload() after the write so the new value is visible
immediately. base_url handling is unchanged.
system.go:
- Register GET/PUT /api/system/settings under the existing admin
/api/system group.
john-okeefe