Cover the admin API added in 03cb4c7 for duplicate-content decisions:
- GET /api/admin/hash-conflicts — pending conflict groups with member
items and per-item usage counts (progress, highlights, bookmarks,
notes, collections)
- POST /api/admin/hash-conflicts/:id/resolve — action=keep (merge child
rows into keep_uuid, delete losers) vs action=keep_all (dismiss);
JSON and form-encoded bodies, error codes including 409 for already
resolved
- When conflicts are created (startup backfill, rescans) and the
guarantee that files on disk are never deleted
The scattered scan-settings JSON routes are superseded by the new
admin-only /api/system/settings pair backed by the SettingsRegistry
(introduced in 885f6d8 / bc47450).
- Rewrite system/settings.md around GET/PUT /api/system/settings:
SettingEntry metadata shape (type, min/max, requires_restart,
category, group, is_default), type-aware validation rules, and the
full tunable-setting catalog (scanner, general, security, api, sync,
performance) with defaults, ranges, and restart requirements
- Note the legacy /api/libraries/scan-settings routes as back-compat
only (they now refresh the registry cache on write)
- Add system/config.md for GET/PUT /api/system/config: raw key/value
system configuration (e.g. base_url), including validation notes and
guidance to prefer the typed settings endpoint for registry keys
Admin:
- Admin pages live in the sidebar's Administration panel (Dashboard,
Libraries, Hash Conflicts, Users, Settings)
- Library creation is the Create Library modal (Library Name,
Description, Library Type); folders are added afterwards by expanding
the library row and using the Folders section's path input + Browse +
Add — the old Add Library modal with folder and 'Scan on save' fields
no longer exists
- Scanning is via the Scanner API or watch mode (File Watcher status on
the admin dashboard); remove references to the removed per-library
Rescan button and 'Force Rescan' option
Collections:
- Fill in the empty creating/managing placeholders with the real flow:
New Collection button, modal fields (name, description, icon grid,
color swatches), per-collection edit/delete icon buttons, Restore
System button, and dashboard-section visibility via Customize
Dashboard
Dashboard:
- Customize Dashboard is opened from the icon button at the right end
of the Library bar (next to Refresh), and requires a specific library
selected rather than 'All Libraries'
- Library switching uses the Library dropdown in the bar below the top
bar (includes 'All Libraries' with counts)
- Collection sections are shown/hidden from the Customize Dashboard
toggles — the per-collection 'Show on Dashboard' setting is gone
- Mention the hover chevrons for scrolling carousels
Bookshelf:
- Saved filters: document the new toolbar buttons — Filters (opens the
filter drawer with Apply Filters, Esc, and overlay-click close), Save,
Load (Saved Filters dropdown with trash-icon delete), and Clear —
replacing the emoji-labelled Save Filter / Saved Filters buttons
- Tag filtering: filters now live behind the Filters drawer on the All
Books page
- Theme switching lives in the sidebar's Appearance panel (palette
icon): swatch list with a checkmark on the active theme, and the
'Bookshelf' section below it for wood textures; on small screens the
sidebar opens via the top-bar menu button
- Profile: account menu is the username accordion at the bottom of the
sidebar (not top-right); save button is 'Save Changes'
- Remove Wood Light/Dark/Mahogany from the Available Themes list (they
are bookshelf backgrounds, not color themes) and consolidate the
Catppuccin variants
The new-ui redesign replaced the top header with a sidebar and removed
the Settings pages.
- Replace all 'Settings → Devices' paths with the Devices page in the
sidebar
- Conflicts are resolved from the book detail page's Sync Progress
button or the Conflicts page (/conflicts); drop the nonexistent
'Settings → Conflicts' path
- Queue status and unlinked-book references no longer invent per-device
button paths that don't exist on the Devices page
- Reading history now points to the Progress page / book detail
- Export FAQ no longer references a Settings → Export flow that isn't
in the UI
- Device setup and quick-find entries now lead with the KOReader guide
and label native Kobo sync as coming soon
- Kobo protocol/API listings tagged as a coming-soon feature
- Fix seven pre-existing broken links: contributing/Development.md had
the wrong case (development.md), and PROJECT_GUIDELINES.md links were
missing the ../ prefix to reach the repo root
- Refresh last-updated stamp
- Supported-devices table: Kobo moves from 'full support' to 'coming
soon, use KOReader on Kobo today'; mobile apps 'coming later' with no
speculative date
- Universal-sync pitch now states what actually syncs (position,
bookmarks, highlights, notes) between KOReader and the web
- Frame KEPUB conversion and collection shelf mappings as groundwork
for upcoming native Kobo support
- Fix dead links: docs/DEVELOPMENT.md → docs/developer/development.md
and docs/contributing/DEVELOPMENT.md → actual path
- sync-guide: only Web and KOReader are fully supported; move Kobo to
coming soon, drop fake Q2-Q4 2026 release dates for mobile/Kindle/
Remarkable, and describe the plugin + server-approval registration
flow instead of QR-code/URL approval
- sync-guide: remove cellular/mobile-app advice from battery and
best-practice sections, correct the Calibre compatibility FAQ, update
the changelog to reflect shipped vs. pending sync features, and fix
the license header (GPL-3.0, not MIT)
- user-guide: lead device setup with KOReader; mark the Kobo guide as
coming soon
- calibre-integration: OPDS client list no longer implies native Kobo
support
- auth overview: label the mobile-application token guidance as
'coming later' since no mobile apps exist yet
Native Kobo sync is implemented server-side but not yet supported on
real devices, so stop documenting it as a working feature.
- Rewrite kobo-setup.md as a coming-soon stub: point users to KOReader
(which runs on Kobo hardware) as the supported path today, and list
what native sync will deliver when released
- Add 'Coming Soon' status banners to the Kobo protocol spec, all five
Kobo endpoint docs, and both API references, noting the endpoints are
under active development and may change
- Tag the device shelf endpoints as pending native Kobo support
Replace the outdated Calibre-wireless/Basic-Auth instructions with the
actual current flow: install the bookhoard.koplugin plugin, enter the
server URL in the plugin menu, then approve the pending registration
from Settings → Devices. Registration tokens are delivered to the
plugin automatically after approval (5-minute expiry), so no
credentials are ever typed on the device.
Also document bidirectional sync of position, bookmarks, highlights
(colors mapped between web and KOReader palettes), and notes, plus
format-aware SHA-256 book matching, OPDS delivery, and trimmed
troubleshooting sections covering the new registration flow.
Six converter tests pointed at absolute paths for 1984 and Crime and
Punishment under uploads/ — books that don't exist on most checkouts
(CI included), so the suite shipped with 5 permanently failing tests
(and a sixth passing only by accident: the percentage-fallback path
triggered by the missing file is the outcome it asserts).
A writeTestEPUB helper now builds a minimal deterministic EPUB in
t.TempDir() (zip → container.xml → OPF → 6-doc spine), so the tests
exercise the real zip/OPF/spine/document pipeline with no external
dependencies. The xpointer→CFI conversion, fragment-ID conversion,
both round-trips (bare and context-text-anchored), and the text-search
and percentage fallbacks all keep their original assertions, now
against known document content. internal/sync is green for the first
time on this machine.
Server-side (8 commits): web annotations finally reach KOReader and
vice versa. Fixed the 400 bind failures on every annotation-carrying
push (loose client types), resolved device-native pos0 locators for
every source (device xpointers pass through round-trip identical,
web CFIs convert to CRE xpointers with text-search anchoring, PDF
anchors map to pages), derived degenerate range ends from selection
length, echo-deduplication via served dedup keys (pull→push cycles
converge instead of minting duplicates), web↔device color mapping at
both boundaries with echo suppression (web colors flow to devices,
round-trips never drift them, device edits win), drawer-based
annotation classification, and tombstone propagation that can't
cross-delete. Perf: parsed-EPUB converter cache (bounded, locked).
Plugin-side (bookhoard.koplugin @ 4ea3966): dual-model annotation
store (KOReader 2024.07+ v2 ui.annotation + legacy v1), thin-client
collection (no per-annotation CRE lookups), dedup-key identity
matching, device-default coloring for applied highlights with
datetime_updated-based echo suppression, and native-shaped
AnnotationsModified dispatches (fixes a ReaderThumbnail crash and
paints immediately instead of after restart).
Reverses the earlier "no colors to the device" decision now that the
echo machinery makes it safe: GetMetadata maps the stored web hex to
KOReader's fixed color names (#ce93d8→purple, #90caf9→blue,
#a5d6a7→green, #ffd54f→yellow; pink maps to purple as the closest —
round-trip drift is prevented on the device by echo suppression, and
a device edit still wins). mapColorToKOReader restored for serving;
ingest (name→hex, preserve-on-echo) unchanged.
Echo duplication: devices push their full annotation list on every
sync, and an echo of a web-created annotation computed a different
dedup key than the original (device locators differ from web locators)
— every pull→push cycle minted a duplicate row, and cleaning those up
on the web tombstoned them back to the device, deleting the
just-applied copies. That was the "web highlights never appear on
KOReader" experience. GetMetadata now serves each annotation's
dedup_key; the device stores it on the applied entry and echoes it in
pushes; SaveHighlight/SaveBookmark/SaveNote accept a DedupKey
override so echoes converge onto the original row (verified: pull →
echo push creates no rows, LWW skips identical content).
Color semantics (per user preference): devices render their own
default and cannot round-trip web colors, so GetMetadata no longer
serves colors at all — every highlight syncs regardless of its web
color and the device draws its default. An echo carries no color;
ingest then PRESERVES the stored web color (existingHighlightColor
lookup by dedup key) so round-trips never change it. A non-empty
device color means the user edited the highlight there: it maps
name→hex (green→#a5d6a7, default yellow) and wins. Verified: echo
kept #ffd54f; a simulated device edit with "green" updated the web
row to #a5d6a7.
Classification: KOReader auto-fills text="in Chapter X" on page
bookmarks (ReaderAnnotation:updateItemByXPointer), so the plugin's
text-presence classification turned every echoed bookmark into a junk
highlight on the web. v2 classification now keys off the drawer field
(present = highlight/note, absent = bookmark with its label in note).
Device-synced highlights stored POINT CFIs (epubcfi(.../8/1:1)); the
overlayer resolves those to a collapsed range and paints nothing, so
KOReader-made highlights were listed in the drawer but invisible on
the page. mapHighlightRow now builds a renderCfi: a proper RANGE CFI
(epubcfi(base,/start,/end)) synthesized from the stored start/end
points. It also repairs stale rows: missing ends (old web highlights)
and degenerate document-start ends (the old converter fallback) are
derived from the start offset plus the selection text's UTF-16
length. All overlay drawing, navigation (showAnnotation), and the
post-create/post-edit re-adds use renderCfi. Verified in-browser
against live device-synced rows: the paginator's overlayer paints
the highlight rects after the fix.
Both directions synced data but rendered nothing:
- Web reader <- devices: highlights painted no overlay. Device pushes
resolve their start xpointer exactly (text-search anchored by the
selection) but the end conversion carries no context and fell back
to a document-start CFI (epubcfi .../1:0) — a garbage range end.
When the start resolved exactly, the end is now derived from it:
same node, character offset advanced by the selection's UTF-16
length (extendCFIByLength). Same repair when SERVING to devices,
where old web highlights (no end anchor) and converted range CFIs
both collapsed pos1 onto pos0 (extendXPointerByLength on the
xpointer form) — KOReader drew zero-width highlights.
- Colors: KOReader paints from a fixed name set (Blitbuffer
HIGHLIGHT_COLORS), the web uses hex swatches; neither understood
the other, so device colors fell back to defaults and web hex drew
nothing useful on devices. Both boundaries now translate: ingest
maps names to hex (default #ffd54f), GetMetadata maps hex to names
(default yellow) — per-datatype edits re-push with the editing
side's color, which LWW then propagates. SyncBookmarks endpoint
aligned to the same mapping and default.
Web highlights stored only epubcfi_start, so devices received
degenerate pos0 == pos1 (zero-length) highlight ranges. The reader
now collapses the selection range to its end point for a second CFI
and stores it as epubcfi_end (PDF rect anchors reuse the JSON anchor
for both ends).
ConvertToCanonical/ConvertFromCanonical built a fresh CFIConverter
per call, and each annotation converts twice (pos0+pos1) — a book
with 200 highlights re-opened and re-parsed the EPUB 400+ times per
sync, and again per metadata pull. A bounded 8-entry cache keyed by
path now shares converters (the parsing work belongs on the server;
clients stay thin). CFIConverter gained a mutex around its lazily
built spine/doc caches since instances are now shared between
concurrent requests.
Adds CFIConverter.SectionPercentage: book-wide percentage for a CRE
xpointer from the spine char distribution (midpoint of its document)
— the server-side counterpart to dropping per-annotation
getPageFromXPointer lookups from the plugin.
Two blockers, diagnosed by simulating the plugin against the live
server with real library books:
1. Every KOReader progress push carrying annotations failed the JSON
bind with 400 ('cannot unmarshal string into ... chapter/page of
type int') — the plugin sends chapter:'', page:'30', and for CRE
documents page:'/body/...' — so annotation sync AND progress sync
failed together. KOReader annotation chapter/page now use FlexInt,
which accepts numbers, numeric strings, empty strings, and
non-numeric strings (decoding to 0). The server is deliberately
liberal here so thin clients can send raw bookmark data.
2. GetMetadata served locators KOReader cannot place, so pulled items
were junk: web bookmarks leaked 'cfi:epubcfi(...)' positions, web
PDF highlights had empty pos0 (skipped by the plugin, invisible),
and web deletions carried no pos0 so tombstones never matched.
New koreaderPos0 resolver handles every source: device-native
xpointers pass through untouched (round-trip identical, verified),
web PDF JSON anchors map to their page number, EPUB CFIs convert
to CRE xpointers (selection text passed as text-search context for
exact anchoring), 'page:N' positions strip to the bare number.
Unresolvable annotations are skipped with a log line instead of
poisoning devices; tombstones get pos0 injected from the new
locator columns.
Also: thin clients omit per-annotation percentages (paging docs still
send arithmetic page/total); the server derives them — section
midpoint from the spine char distribution for CRE documents, page/
page-count for fixed formats.
GetTombstonedAnnotationsForBook now also returns each tombstone's
start_position/end_position and epubcfi_start/end (note: position/
epubcfi_location, bookmark: position/cfi_position), so serving code
can resolve a device-native locator for deletions of web-created
annotations, whose device_sync_data carries no pos0.
Full reader redesign across 22 commits (with the foliate-js fork's
zoom-control engine work pinned per release):
- Phase 0: panel/chrome stabilization, bookmarks end-to-end (REST CRUD
via AnnotationService), dead UI removal, tombstone resurrection fix
- Phase 1: edge-to-edge glass chrome with auto-hide, slide-over drawers,
tri-state PDF pointer mode (Smart/Pan/Text), Kindle-style theme swatches
- Phase 2: touch gesture engine (pinch/pan/swipe/double-tap), tap zones,
mobile sheets + compact toolbar with overflow menu
- Phase 3: EPUB highlights & notes (selection popover, overlayer
rendering, annotations drawer), PDF text highlights (fraction-rect
overlays), in-book search for both EPUB and PDF, back-to-location
stack, page thumbnails, shortcuts help modal, desktop edge zones
- Phase 4: webtoon (vertical-scroll) mode for comics, brightness/
contrast/night filters, bookmark toast feedback
- Build hygiene: vite stale-chunk cleanup, browser-verified fixes for
Alpine proxy/dpr/duplicate-key classes of bugs along the way
The initial webtoon commit's IntersectionObserver (shadow-host root)
never delivered intersections in Chromium, leaving pages blank.
Scroll-driven loading in e448d36 fixes it; verified end-to-end in a
real browser: pages render (content-rich screenshots), deep scroll
advances the reading position (7/10) and progress readout, filters
visibly change both webtoon images and PDF pages via ::part(filter)
(brightness 5% -> 57% smaller screenshot), paged comics still use
foliate-fxl, and webtoon UI gating (zoom/spread hidden) works.
Phase 4 of the reader redesign (foliate-js ea268df):
- Webtoon mode for comics: continuous vertical scroll of all pages
(900px centered column on wide screens), lazy-loaded with a 150%
IntersectionObserver margin, far pages unloaded to bound memory
with stable aspect-ratio placeholders so the scrollbar never jumps.
Chosen per book (Paged | Webtoon segmented control in Settings →
Layout & Display; stored in localStorage per media item since a
webtoon title and a paged manga volume want different flows).
Toggling reloads the reader — the renderer is chosen at open time —
and progress restores from the saved page. Relocate events flow
through the same pipeline, so the slider, progress saving, back
stack, tap zones, and edge zones all work unchanged. Zoom/fit/
magnifier/spread controls hide in webtoon (natural-width scroll).
- Display filters for fixed-layout: brightness (30-130%) and
contrast (70-130%) sliders with live preview, plus Night Mode
(invert) — also a quick row in the ⋯ tools menu. One --fx-filter
CSS var drives everything: ::part(filter) on foliate-view iframes
(forwarded via the new exportparts attribute) and the webtoon
page images alike. Persisted as fx_brightness/fx_contrast/fx_invert
(types + defaults both sides); Restore Defaults resets them.
Help menu (the reader had a growing shortcut/gesture vocabulary with
no discoverability): a ? topbar button, the '?' key, and F1 open a
glass modal listing navigation, zoom/pan, highlight, and touch
gesture reference — format-aware (fixed-layout/PDF rows appear only
where they apply), Esc closes it first in the dismiss chain.
Desktop edge zones: clickable page-turn strips on the left/right
viewport edges (8% width, 44-72px), desktop only (hover+fine-pointer
media query — touch devices use tap zones, avoiding double paging).
Hovering reveals a chevron arrow and a subtle edge gradient. Zones
disable (pointer-events pass-through) while a fixed-layout page is
zoomed so edge clicks belong to content: panning, selection,
highlight editing. fxZoomed tracks zoom state via the renderer zoom
event, reset/fit actions, and init.
The 🏷️ bookmark button (and the 'b' shortcut) saved silently — an
accidental click gave no reaction at all. addBookmark() now shows a
short success toast ('Bookmark added — <progress>') using the
existing toast system, which the reader bundle hadn't been importing.
Importing it also activates the shared fetch interceptor, so failed
reader API calls (incl. bookmark saves) surface error toasts instead
of being swallowed.
Diagnosed in a real browser (playwright/chromium against the running
app + Head First SQL): the engine's book.toc held all 18 entries with
correct labels/hrefs and the tab counter even showed 380, yet zero
links rendered while the console flooded with 'Alpine Warning:
Duplicate key on x-for'.
Root cause: the drawer keyed TOC rows by item.href. PDF outline
entries frequently share the same destination (e.g. the printed TOC
page is targeted by several bookmark entries), so flattened items
carried duplicate keys — and Alpine's x-for renders NOTHING for a
duplicated key, not even the unique ones. EPUB TOCs never collided
because their hrefs are unique file paths, which is why this only
surfaced on PDFs.
Key is now href + row index (the list is static once loaded, so
positional keys are safe). Verified end-to-end in the browser: 18
entries render and the drawer populates.
Investigation: the contents drawer read book.toc, which makePDF
builds from pdf.getOutline() — verified against the real library PDF
(Head First SQL) through the exact vendored pdf.js build AND the exact
range transport the browser uses: 18 chapter entries come back. So
the source is right; manga-scan PDFs and CBZs simply have no embedded
outline, which made Contents look broken exactly where users expect
page-based navigation.
- TOC now populates eagerly right after the book opens (toggle-time
lazy population removed), so an existing outline can never silently
miss due to timing; the drawer keeps the honest empty-state text
for books without outlines.
- New 'Pages' tab in the contents drawer for fixed-layout books:
a Kavita-style thumbnail grid (3-up, current page highlighted and
scrolled into view, click to jump — recorded on the back-to-
location stack). Thumbnails render client-side: PDFs via the
in-memory pdf.js document (small viewport render, Alpine.raw
unwrap); comics via the page's image blob drawn down to a 110px
canvas, then unloading the full-size blob so thumbnailling doesn't
hoard page images. Lazy via IntersectionObserver scoped to the
drawer's scroll container (200px margin), canvases cached at module
level so revisits are instant; failures warn in console and allow
retry. The backend /readers/thumbnails endpoint turned out to be an
empty stub, so nothing server-side was worth wiring.
The display:none for the full toolbar lived in @layer components while
the div also carried Tailwind's flex utility (@layer utilities). Layer
order beats specificity, so the utilities layer always won and the
full bar never hid below the breakpoint (the compact row only worked
because it had no display utility of its own).
Switch to Tailwind's own responsive utilities in the markup — full
toolbar 'hidden md:flex', compact row 'flex md:hidden' — and delete
the custom rules; responsive display now resolves inside a single
layer where source order (responsive variants after base) guarantees
the right winner.
Wrapping alone isn't how polished mobile readers work. Adopt the
standard pattern (Kindle/Apple Books/Mihon) responsively:
- >= 768px: the full fixed-layout toolbar stays (wrap still absorbs
mid-size widths) — power users keep one-click zoom/fit/spread.
- < 768px: single-line compact row — page back, back-to-location pin,
slider, page forward, progress, and a ⋯ overflow button. No
wrapping, no horizontal scroll.
- ⋯ opens a glass menu anchored above the bar with LABELED rows
(Zoom −/%/+, Fit, Page position/Recenter, Magnifier, Pointer
Smart/Pan/Text, Double page, Contents) — labels beat mystery icons
on touch. Pointer row hides for comics; menu scrolls if tall.
- Dismissal: Esc, outside click (⋯ button exempt so it re-toggles
cleanly), opening any drawer or TOC closes it; hides with the
chrome. Compact slider registered in progressSliders() so all
three stay in sync with relocate events.
- Back-to-location moves from the topbar (where it sat between Back
and the title, too subtle and disconnected from navigation) into
both bottom-bar rows, beside the page-back arrow — the natural
'go back' cluster. New icon: a location pin, clearly distinct from
the back arrow and page controls. Appears only when the stack has
a return target; Alt+← unchanged.
- New recenter button in the fixed-layout row (crosshair icon, next
to zoom): resets pan offsets while keeping the current zoom —
backed by foliate's new recenter() (1c812e8), which zeroes the
wrapper translate and re-syncs the spread side.
- Both bottom-bar rows wrap gracefully on narrow windows instead of
overflowing/h-scrolling: controls are grouped (paging+back | slider |
fit+zoom+magnifier+recenter | pointer mode | spread | progress+TOC)
so groups flow to a second line at small widths; the slider shrinks
first (grow + min-width), everything else stays whole. Fixed-layout
row drops its overflow-x-auto.
Highlights landed on the right line but shifted right and oversized
on any display with devicePixelRatio != 1. Cause: selection fractions
divided the textLayer span rects by documentElement's screen rect,
but pdf.js scales the iframe's <html> by 1/dpr — that rect is dpr×
smaller than the visible page, inflating every x/w fraction by dpr
(on a 2× display a highlight started twice as far right and was twice
as wide). dpr=1 displays were coincidentally correct, which is why
the geometry looked sound when written.
The denominator is now the rendered canvas (#canvas canvas), whose
post-transform rect IS the visible page and shares the textLayer's
transform space — the dpr scaling cancels exactly. Comics keep the
img denominator; a viewport fallback covers any page without either.
The popover-placement scale factors (frame/denominator) become 1 for
PDFs as a side effect, fixing popover drift too. The fork's click
hit-test (86e234d) gets the same canvas-aware denominator so clicking
highlights opens the editor at the right spot.
Highlights saved before this fix stored dpr-inflated fractions and
will still render misplaced — delete and re-create them.
PDF search diagnosis: extraction and matching were proven correct
against the real 609-page library PDF (pdfjs 5.5.207, incl. the exact
range-transport setup makePDF uses — 841 hits for 'SELECT'), and the
served bundle had every piece. The failure was Alpine's reactivity:
this.book is a plain object, so reading .pdf through component state
returns a reactive Proxy around the PDFDocumentProxy — and pdf.js
v5 uses #private fields, so getPage() through the proxy throws
'cannot read private member', which the empty catch rendered as a
silent empty result set. runPdfSearch now unwraps via Alpine.raw
(falls back to the raw read), and search failures surface in the
drawer ('Search failed — see console') plus console.warn instead of
masquerading as 'No matches'.
Back-to-location stack (research/footnote workflow): the current
position is recorded before every programmatic jump — search-result
clicks, TOC entries, bookmark and highlight jumps — and on every
internal link click (footnotes, cross-references) via foliate's
'link' event. A ↩ button appears in the topbar once a return target
exists; Alt+← works everywhere. Ordinary paging never pollutes the
stack (max depth 50, consecutive duplicates collapse).
PDFs have fully searchable text (pdf.js text layer) — the previous
reflowable-only gate existed only because foliate's generic search
needs DOM documents that PDF sections don't provide. This adds a PDF
pipeline alongside it:
- Fork d065495 exposes the pdf.js document proxy as book.pdf so the
host can drive text extraction directly.
- New web/src/reader/pdf-search.ts: extractPdfPages() pulls each
page's textContent with item geometry (progress-reported, cached
after first search). PDF text items often omit inter-word spaces
(gaps are positional), so pages are joined gap-aware — baseline
changes, hasEOL, or horizontal gaps past a font-size threshold
become spaces — recording a char→item map. searchPdfPages() does
case-insensitive matching over the joined text and maps each hit
back to the page-fraction rects of the items it spans, with
ellipsized pre/match/post excerpts. Pure functions, unit-sanity
checked (cross-item 'brave new' → two rects).
- runSearch branches: EPUB keeps foliate's DOM search; PDFs search
the extracted pages, group hits per page ('Page 12'), and render
on-page hit rectangles through the existing fraction-rect overlay
(addRectAnnotation) — which re-render automatically when pages
revisit, same as highlights. Clearing the query removes them.
- Results navigate by page index; the 🔍 button and '/' shortcut now
appear for PDFs too (comics remain without searchable text).
Wires foliate's search engine into the new drawer system:
- 🔍 topbar button (reflowable-only; PDF/comic sections have no
searchable text documents) and the '/' keyboard shortcut open a
Search drawer: query input (Enter to run), live progress while
scanning (per-section percent), match count, and results grouped
by section with TOC labels.
- Each result shows pre/match/post excerpt rendered as three text
nodes (no x-html — book content never enters the DOM as markup);
the match is styled with a translucent <mark>. Clicking jumps to
the hit's CFI and closes the drawer.
- Hits are drawn on the page through foliate's overlayer (outline
style) and persist across page turns — the engine re-applies
search results when a section's overlay is created. Clearing the
query removes the outlines.
- A generation counter discards results and progress from superseded
searches (rapid re-query), and starting a new search clears the
previous one server-side via view.clearSearch().
- Search integrates with the drawer system: scrim, Esc-to-close,
one-drawer-at-a-time, / focuses the input via .
Two bugs broke the Phase 3b PDF highlight flow end to end:
1. Selection capture never attached: reader.ts read renderer.isPDF
before view.init() rendered the first spread, but the renderer
only sets that flag once frames exist (PDF frames carry pdf.js
onZoom). The stale undefined copy gated the pointerup selection
listener off, so selecting PDF text did nothing. The listener now
gates structurally on the loaded document having a .textLayer
(true for every PDF page, false for comics), and isPDF is re-read
after init — which also finally makes the Smart|Pan|Text control
and the saved pointer mode apply on PDFs.
2. Highlights rendered invisibly: the overlay SVG lived inside the
page iframe, whose <html> pdf.js scales by 1/devicePixelRatio —
shrinking the overlay into the top-left corner on any dpr != 1
display. The fork (1c0ebf3) now renders annotation rects
host-side, inside the frame wrapper element, positioned in
percentages of the visible page box — immune to the html
transform, zoom re-renders, comic iframe scaling, and pan/zoom.
Phase 3b of the reader redesign — highlighting for fixed-layout PDFs:
- Select text on a PDF page → same glass popover as EPUBs (colors,
note, copy). The selection's client rects are normalized to
page-fraction quads using a transform-inclusive denominator so
pdf.js's devicePixelRatio scaling on <html> cancels out, then
stored as a JSON anchor {page, rects} in epubcfi_start.
- Rendering goes through the fork's new rect-annotation pipeline
(foliate-js aba68d8): a full-bleed viewBox-0-100 SVG inside the
page iframe, so highlights stay aligned through pan/zoom, iframe
CSS-scaling, and PDF hi-res re-renders with zero re-anchoring.
Frames carry their page index and re-render annotations when
recreated on spread changes.
- Clicking an existing highlight hit-tests in fraction space and
opens the edit popover (recolor, note, copy, delete) at the
host-space click position; drag-selecting text never triggers it.
- Annotations drawer: PDF highlights jump by page index; notes and
recolors round-trip through the same LWW/dedup sync path as EPUBs
(same dedup key derivation on the JSON anchor).
- Comics keep bookmark-only highlighting (no text layer) by design.
Phase 3 (EPUB half) of the reader redesign:
- Select text in a reflowable book → floating glass popover at the
selection (5 colors, note, copy). Clicking a color creates the
highlight via POST /api/media-items/:id/highlights, anchored by the
foliate range CFI (epubcfi_start) with percentage position.
- Highlights render through foliate's overlayer pipeline: draw-
annotation draws Overlayer.highlight with the stored color,
create-overlay re-adds persisted highlights as sections load,
show-annotation opens the edit popover when a highlight is clicked
(recolor, edit note, copy, delete).
- Backend: highlight create/update accept epubcfi_start/end,
note_text, and percentage fields; position validation relaxed
(CFIs exceed the old 100-char cap); PUT routes through
AnnotationService.SaveHighlight so edits get dedup/LWW treatment
and actually persist note_text (the plain query can't).
- Bookmarks drawer becomes the Annotations drawer with tabs:
Highlights (color-bar list, note previews, jump/edit/delete),
Notes (add note at current position, list, delete — backed by the
existing notes API), and Bookmarks (unchanged behavior).
- Popover dismissed on outside click, collapsed selection, page
navigation, or Esc (new top-priority Esc branch).
emptyOutDir is false because web/static also holds tracked assets,
so *-<hash>.js chunks from every previous build accumulated
indefinitely and leaked into Docker images via the build context
(the reader serves whichever chunk the import chain names, so the
orphans are pure confusion + bloat). A closeBundle plugin now
deletes any hashed chunk this build did not produce.
Phase 2 of the reader redesign:
- Fixed-layout touch engine (foliate-js e9e61d8): pinch-zoom around
the midpoint, two-finger pan, single-finger pan while zoomed,
horizontal swipe page-turn at fit (RTL-aware via next()/prev()),
and double-tap to zoom 2.5x / reset. Touch events forwarded from
page iframes with converted coordinates; preventDefault only when
the engine consumes the gesture, so PDF text selection and native
taps stay intact. touch-action: none on the host and in comic/pdf
page documents keeps the browser from fighting the engine.
- Tap zones (Kindle-style) for touch devices: tap the outer margins
to page, center to toggle chrome. Size configurable (10-50%) via
the revived tap_zone_size setting; toggle via new tap_zones_enabled
(Behavior section of the settings drawer). Pointer-based + passive
so drags/swipes/selection never trigger; attached both to the
viewport and inside every page document (iframe events don't
bubble); debounced 280ms so double-tap zoom doesn't also page; no
zone actions while a fixed-layout page is zoomed.
- Drawers become full-width sheets on screens <= 640px.
Deleting a bookmark/highlight/note and then re-adding the same content
at the same position (same dedup key — e.g. the reader's auto-titled
'Bookmark at X%') was silently swallowed: the save hit the tombstone
branch, returned 201 with the deleted row, and the list (which filters
deleted) stayed empty. Bookmarks were further blocked by the
UNIQUE(media_item_id, user_id, title) slot the tombstoned row holds,
and notes had no TTL escape at all.
Tombstones now only block saves that predate them (stale replays from
a device that still has the annotation). A save whose modification
time is newer than max(deleted_at, last_modified_at) — a deliberate
re-create from the web or a device — resurrects the row via the LWW
update queries, which now clear deleted/deleted_at.
Modernize the reader chrome bars without touching the drawer system:
- Bars become theme-tinted glass: 70% bg-primary translucency over
the edge-to-edge page, 18px backdrop blur + saturation, hairline
translucent borders, soft directional shadows (single .reader-glass
class owns the effect; replaces solid opaque backgrounds and the
tailwind backdrop-blur that would override it).
- Chrome hide/show now slides the bars off-screen (translateY) in
addition to the opacity fade, via .chrome-hidden on #reader-chrome.
- Theme-aware hover pills (translucent currentColor tint) replace
hard-coded gray-700 hovers; focus-visible rings added.
- Progress slider gets a custom thin rounded track with a floating
white thumb (webkit + gecko), replacing native range styling.
- Separators and the fit-mode select match the glass language
(.reader-sep, .reader-select).
Phase 1 of the reader redesign:
- Reading surface is edge-to-edge; top/bottom bars overlay
translucently (backdrop-blur) instead of reserving insets, killing
the inset-coordination bug class entirely. Chrome auto-hides after
2.5s of pointer inactivity (chrome_behavior setting finally wired:
auto-hide / always-visible; legacy values map to auto-hide). Pointer
activity inside page iframes keeps it awake; Esc toggles.
- TOC / Settings / Bookmarks become slide-over drawers with a scrim
(z-50, full-height, safe-area aware), replacing the dockable-panel
system and its window-shade headers. Only one drawer opens at a
time; Esc or scrim click closes.
- Bottom bar is contextual: reflowable keeps nav/slider/progress/TOC;
fixed-layout row adds Fit Page/Width select, zoom cluster,
magnifier (now shows active state), Double Page Spread toggle, and
a Smart | Pan | Text segmented control replacing the cryptic
two-state icon. Smart = text-aware drag; Text = selection-only
(manual smart-detect off); Pan = force pan. Choice persists via
pdf_interaction_mode (new setting + foliate 29bc958 'text' mode).
- Settings drawer: Behavior (chrome, progress mode), Appearance with
18 Kindle-style theme swatches (single source of truth from
THEME_COLORS), Typography, Layout — each scoped by format.
- Keyboard: t/s/b open TOC/settings/bookmark, Esc closes drawers
before toggling chrome, shortcuts skip form inputs; both slider
rows tracked correctly (no duplicate-ID lookups).
- Topbar: Back, title, add-bookmark, bookmarks drawer, Aa settings;
chrome follows user theme.
Phase 0 of the reader redesign:
- Panels no longer render under the top/bottom bars: sidebars get
measured insets (same resize/safe-area mechanism as the viewport);
panel max-height now derives from the bounded sidebar instead of a
100vh guess; right-side border targets the actual sidebar.
- Bookmarks work end-to-end for the first time: REST CRUD under
/api/media-items/:id/bookmarks (create/delete route through
AnnotationService for dedup/LWW/tombstones), fix UpdateMediaBookmark
referencing nonexistent updated_at column, frontend posts to the
real API with per-format position (CFI vs page), live list with
jump + delete instead of SSR-only snapshot.
- Fix chapter matching in progress saves: boundaries were compared by
a nonexistent tocItem property, so chapter was never persisted.
- Remove dead UI: Navigator panel stub, empty dictionary popup shell,
unwired Chrome Behavior select; purge 160 stale build artifacts.
- Reader chrome now follows the user's app theme instead of hardcoded
theme-tokyo-night.
Complete the SHA-256 lifecycle for preexisting databases: items
imported before hashing existed get hashed automatically, and any
content duplicates discovered in the process land on the new admin
Hash Conflicts page for an explicit keep/merge decision.
HashBackfillService (runs once 30s after startup, independent of
auto-scan):
- hashes every media_items row where file_sha256 IS NULL, resolving
each path through LibraryService; per-item failures are logged and
skipped so one unreadable file cannot block the pass
- no-op once everything is hashed (logged and skipped)
- finishes with a conflict sweep flagging every content-duplicate
group via FindHashConflictGroups + CreateHashConflict; the sweep
runs after the per-item pass because a preexisting pair only
becomes detectable once both sides have their hash
API (admin-only):
- GET /api/admin/hash-conflicts - pending groups with member items
and usage counts
- POST /api/admin/hash-conflicts/:id/resolve - action=keep_all, or
action=keep with keep_uuid: validates the uuid belongs to the
group, re-parents every other copy's child rows onto the kept item
(reparent_media_item_children), deletes the losers, and records
the resolution + resolving admin; accepts form or JSON bodies and
returns the htmx resolved fragment
Page route /admin/hash-conflicts (admin-only) renders the template
with hydrated conflict data; HashConflictsHandler wired into the
router Config and constructed in main.
Verified end-to-end against the live database: duplicate detection,
pending listing, keep_all resolution, merge path (re-parent +
delete), and - critically - a resolved group is not re-flagged by a
later sweep (upsert no-op). Database restored afterward.
New /admin/hash-conflicts page (admin-only) listing pending
content-duplicate groups. Each group card shows the library, a
shortened SHA-256, and one row per copy with title, author, path,
size, and per-copy reading-data counts (progress, highlights,
bookmarks, notes, collections) - copies that own user data are
highlighted so the keep choice is informed.
Per copy: 'Keep this copy' merges the other copies' child rows into
it and deletes them. Per group: 'Keep both' for intentional
duplicates. Both confirm first, resolve via htmx POST, and swap the
card for a resolved confirmation inline. The confirmation fragment is
built inline in the handler rather than the templates package
(templates imports handlers; a back-import would be a cycle).
Empty state shown when no conflicts are pending. Adds a 'Hash
Conflicts' entry to the admin sidebar section between Libraries and
Users.
Force rescan was metadata-only: updateMediaItem never touched the
hash identifiers, so a force scan could not backfill file_sha256 for
items imported before hashing existed (or where extraction originally
failed). Those items were invisible to content dedup and SHA-256
device matching with no way to fix short of delete + re-import.
processMediaFile now refreshes hash identifiers in three cases:
- force rescan (the admin Scan button becomes the backfill tool)
- file size change (stored hash is stale - the bytes changed)
- unchanged file with no stored hash (ordinary scans self-heal the
legacy backlog incrementally, no admin action required)
Each recompute runs recordHashConflictIfAny: when the freshly stored
hash is now shared by more than one item in the library, the group is
upserted into hash_conflicts for the admin Hash Conflicts page. The
upsert is a no-op for already-tracked groups, so resolved 'keep both'
decisions stick.
Also extract a package-level computeFileSHA256 (the scanner method
now delegates to it) so the startup backfill service can hash files
without a scanner instance.
Content duplicates (same library + file_sha256 at different paths,
e.g. the same book imported twice under two names on a preexisting
database) cannot be auto-collapsed the way path duplicates were:
keeping both copies may be intentional. Surface them for an explicit
admin decision instead.
Schema:
- new hash_conflicts table keyed (library_id, file_sha256) with a
status/resolution lifecycle: 'pending' until an admin resolves via
'keep_all' or 'kept:<uuid>' (which copy was kept after merging)
- resolution is VARCHAR(50) - 'kept:<uuid>' is 41 chars; include a
widening ALTER for databases created with the initial 30-char width
- resolution/resolved_by/resolved_at record who decided what and when
Queries:
- ListMediaItemsMissingHash: items imported before hashing existed
(file_sha256 IS NULL), ordered oldest-first for the backfill pass
- FindHashConflictGroups: the content-duplicate group detection
(GROUP BY library_id, file_sha256 HAVING COUNT(*) > 1)
- ListMediaItemsBySHA256AndLibrary: full membership of one group
- CreateHashConflict: upsert with DO NOTHING so already-tracked groups
are untouched - critical behavior: a group an admin resolved as
'keep both' is never re-flagged by later sweeps
- ListPendingHashConflicts: admin listing with library name and live
item counts (items may have been deleted since flagging)
- GetHashConflict / ResolveHashConflict: lifecycle
- GetMediaItemUsageCounts: per-item progress/highlight/bookmark/note/
collection counts so the admin can make an informed keep choice
- ReparentMediaItemChildren: sqlc binding for the existing
reparent_media_item_children() migration function, used to merge a
losing copy's child rows into the kept copy
Bring the koreader protocol doc in line with the hash-sharing work:
- Request table: uuid is no longer required (it is absent on the first
sync of a newly downloaded book); document sha256 and file_path and
the resolution priority uuid -> sha256 -> file_path alias ->
title/author
- Note that SHA-256 matching is format-aware (media_items hash first,
media_item_formats fallback) so converted KEPUB/PDF downloads match
- Document the sha256 field returned by the metadata and library
endpoints
- Add a 'Book identification' section pointing current and future
clients (koreader, kobo, OPDS, device-link UI, mobile apps) at the
shared BookResolver as the single resolution path
john-okeefe