Web colors now flow to KOReader instead of every synced highlight
rendering in the device default. The server maps the web hex swatch
to KOReader's named palette (#ce93d8→purple, #90caf9→blue,
#a5d6a7→green, #ffd54f→yellow, pink→purple as the palette's
closest); applyServerAnnotations uses the served name when it is a
plain color name, falling back to the device default
(view.highlight.saved_color) when absent or malformed (hex rejected
as belt-and-braces). v1-model entries accept named colors too.
Round-trip safety is unchanged: un-edited applied entries still
suppress their color on echo (a pink→purple palette mismatch must not
rewrite the stored web hex), and a device edit still pushes its color
which the server lets win. Verified in the harness: purple applied
from a served #ce93d8; unedited echo colorless; edited echo sends
its color and updates the web row.
Two rendering observations from live use:
- Synced highlights rendered grey while device-native ones are yellow:
KOReader natively stamps view.highlight.saved_color ("yellow" on
color screens) onto every new highlight; a nil color takes the
darkenRect path (grey). Applied entries now carry the device's own
saved_color, so they look exactly like native ones.
- They only appeared after a restart: ReaderView caches each rendered
page's highlight boxes and only invalidates the cache on
AnnotationsModified — a plain setDirty repaints from the stale
cache. applyServerAnnotations now dispatches AnnotationsModified
with a non-table payload (clears the whole cache) before setDirty.
Echo safety for the new default color: an applied entry's yellow is
the device default, not user intent — pushing it back would clobber
the web color. collectAnnotations suppresses the color for entries we
applied (bookhoard_dedup_key set) that the user has not modified
since (datetime_updated, which KOReader sets on any edit); edited
entries push their color and the server lets the edit win. Verified
in the harness: applied entry gets color=yellow; un-edited echo
carries no color; an edited (datetime_updated + green) echo sends
green and the web row updates; dedup keys round-trip throughout.
- Identity: applied entries carry bookhoard_dedup_key (served by the
server) and echo it back in pushes, so a pull→push cycle updates the
original server row instead of minting a duplicate (device locators
never matched web locators under the computed key — every round-trip
duplicated every annotation). Matching in applyServerAnnotations is
key-first; a keyed entry that is absent is genuinely new, so pos0
fallback applies to legacy keyless entries only — a different
annotation that merely shares a position can no longer be updated
or, for tombstones, cross-deleted (demonstrated: a deleted
duplicate's tombstone used to pos0-match a live highlight sharing
the spot and remove it). Tombstone matching is likewise key-only
for keyed entries; legacy pos0 fallback is restricted to the same
annotation kind.
- Classification (v2): the drawer field is the only reliable
highlight discriminator — KOReader auto-fills text="in Chapter X"
on page bookmarks, so text-presence turned every echoed bookmark
into a junk highlight on the web. Bookmarks now take their label
from note.
- Colors: no color is imposed on applied entries; devices render
their own default. The web color only changes when the user edits
the highlight on the device (the edit sets a device color name,
which the server maps and stores).
Two rendering gaps once annotations actually synced:
- KOReader draws highlight colors from a fixed name set
(Blitbuffer.HIGHLIGHT_COLORS); a web hex value (#ffd54f) resolves
to nothing useful. The server now serves mapped names, but keep a
belt-and-braces guard: only accept plain-name colors, let anything
else (hex or unknown) fall back to the device default.
- applyServerAnnotations mutated the annotation model but never
requested a repaint — ReaderView paints annotations from the model
on each redraw, so applied highlights stayed invisible until an
unrelated refresh. Mark the reader dialog dirty when anything
changed (same call the native highlight flows make).
KOReader replaced the bookmark store: since 2024.07, highlights/notes/
bookmarks live in ui.annotation.annotations (sidecar key 'annotations')
with new field semantics — text = highlighted text, note = user note,
drawer/color styling, pos0/pos1 as xpointers (CRE) or {page=N} tables
(pdf). ui.bookmark.bookmarks no longer exists, so collectAnnotations
returned empty and applyServerAnnotations bailed on its first guard:
annotations NEVER crossed in either direction on current KOReader
(reproduced: device log showed highlights:[] with a highlight present,
and the book's sidecar showed annotations={} with no bookmarks key).
Both functions are now dual-model via getAnnotationStore():
- collect: v2 entries normalized to the wire format (text/note from
the new fields, pdf table positions stringified to their page,
classification text+note=note / text=highlight / else=bookmark with
the label traveling in the entry text, as before); v1 mapping
corrected to KOReader's own migration semantics (v1 notes held the
highlighted text, text held the note).
- apply: v2 entries are built with drawer='lighten' for highlights
(its absence marks a page bookmark), pdf positions as {page=N},
and inserted via ui.annotation:addItem() — the module's supported
API (sorted insertion, datetime/pageno defaults). v1 path preserved
for older devices. Tombstone removal works on either array; the
v1-only onSortBookmarks call is now guarded.
Verified both models in the stub harness against the live server:
v2 push collects the device highlight; v2 pull lands server
highlights (with notes), and bookmarks with correct field shapes;
v1 roundtrip unchanged.
Pulling progress crashed KOReader in syncToProgress when the target
wasn't a page number or xpointer (e.g. an epubcfi string from the
server on a rolling document, falling to the percentage branch):
'attempt to call method gotoPercent (a nil value)'. Documents have no
such method — navigation goes through events, exactly as the other
branches in this function already do.
The percentage fallback now dispatches Event GotoPercent (0-100),
handled by ReaderRolling:onGotoPercent and ReaderPaging:onGotoPercent
(the built-in Go-to-% handlers, which also refresh the view). The
redundant UpdatePos follow-up is gone; handlers do it.
Verified all four paths in the stub harness: rolling+percentage →
Event:GotoPercent, rolling+xpointer → Event:GotoXPointer, paging+
page → Event:GotoPage, paging+unresolvable → no-op. No crashes.
A device log showed pushes carrying highlights:[] while the document
had highlights — progress synced fine, annotations never moved.
Root cause: G_reader_settings:readSetting(key, default) returns the
STORED table verbatim and does not merge defaults into it. A settings
file saved by an older plugin version has sync_progress=true but no
sync_bookmarks/sync_highlights/sync_notes keys, so all three read as
nil; the 'or' chain in _doUpdateProgress then fell to the else branch
that pushes empty annotation arrays — silently, since progress in the
same request succeeds.
init() now reads into a fresh table, merges any missing
default_settings keys, and persists once (so the settings file and
the menu checkboxes self-heal on first run). Verified in the KOReader
stub harness against the live server with the exact scenario: stale
settings shaped like the device's, one highlight, one local bookmark
list — after init sync_highlights=true, and the pushed request body
contains the highlight (previously empty).
isValidPos0 compared pos0:sub(1,7) against the 6-char string '/body/',
so it tested '/body/D' == '/body/' — false for every CRE xpointer, and
applyServerAnnotations silently dropped ALL pulled highlights/notes/
bookmarks on rolling documents. Caught by running the real plugin
code against the live server in a stubbed KOReader harness: push
stored rows on the server, pull applied nothing — matching the
reported 'neither direction syncs' symptom.
sub(1,6) now matches. Both directions re-verified end-to-end through
the actual plugin: a KOReader-style highlight pushes and lands in the
server DB (with server-derived percentage), and a metadata pull
populates a fresh device's bookmark list with web- and device-sourced
highlights (xpointer pos0s) plus web bookmarks (numeric page pos0s on
paging documents).
- collectAnnotations no longer calls getPageFromXPointer per
annotation (a CRE-engine lookup each — the most expensive thing the
sync loop did on weak hardware). Paging documents compute
percentage arithmetically; CRE documents omit it entirely and the
server derives it from the locator against the actual book.
- chapter/page are sent raw; the server's tolerant types accept
strings, empty strings, and CRE xpointers in the page field.
Previously a single annotation with chapter:'' failed the ENTIRE
progress push with a 400, silently killing progress sync too.
- applyServerAnnotations validates pos0 before applying: rolling
documents require '/body/...' xpointers, paging documents numeric
pages. Unplaceable locators are skipped instead of becoming junk
local bookmarks that would re-push as duplicates.
On the first open of a freshly downloaded book there is no cached
bookhoard UUID yet, since it is only learned from a successful push
response. Pulling at that point simply failed with "Push progress
first", which also broke auto sync on open.
- onReaderReady auto sync: if no UUID is cached, push once to
bootstrap identity (the server resolves the book by format-aware
SHA-256 and returns the UUID, which we then cache); otherwise pull
as before
- getProgress: when no UUID is cached, instead of showing the dead-end
"Push progress first" message, show "Linking this book to Bookhoard
first…", push once via _bootstrapUUIDThenPull, and retry the pull
once the UUID is cached
- _bootstrapUUIDThenPull: new helper that pushes, waits 2s for the
UUID to be cached, then performs the originally-requested pull,
falling back to an error message on failure
After this bootstrap both push and pull work without ordering.
Bring the plugin to feature parity with the server's annotation sync
system. Previously annotations were push-only (and even that was broken
because the server ignored inline annotations in progress pushes).
ANNOTATION PULL (server → device):
- Add applyServerAnnotations() method called from _doGetProgress
after progress handling
- Parses annotations.highlights, annotations.notes, and
annotations.bookmarks from GetMetadata response
- Matches incoming annotations against local bookmarks by pos0
(start position) to detect existing entries
- New entries are inserted with correct KOReader bookmark format:
page, pos0, pos1, datetime, text, notes, color, chapter
- Existing entries with changed content are updated (LWW: server
version is authoritative since the server already resolved conflicts)
- Respects sync_highlights, sync_notes, sync_bookmarks toggles for
inbound sync, not just outbound
- Triggers onSortBookmarks + saveSettings on change for persistence
DELETE PROPAGATION:
- Parses annotations.deleted_highlights and annotations.deleted_bookmarks
from GetMetadata response
- Each entry contains device_sync_data (pos0, datetime, page) from the
original device push
- Matches local bookmarks by pos0 and removes them
- Deleted highlights only remove entries that have text (to avoid
removing plain bookmarks at the same position)
CLEANUP:
- Fix BookhoardAPI.lua: rename pcall return _ to err for clarity
(was functional but misleading variable name)
- Remove dead code: unused getLibrary() and syncBookmarks() API methods
- Remove sync_endpoints field (stored from registration response but
never read by any code)
The pull path (_doGetProgress + syncToProgress) still used the
reflowable-oriented nav_target selection that prefers xpointer/cfi
over page. For has_pages documents, if the server returned an epubcfi
(e.g., a fake CFI from the web reader's comic renderer), nav_target
became a CFI string, tonumber() returned nil, and GotoPage(nil)
crashed koreader when the user confirmed the sync prompt. After the
crash the old position was retained because GotoPage never completed.
This makes the pull path symmetric with the push path (ef129fd, which
omits epubcfi for paging documents):
- _doGetProgress: for has_pages, select nav_target from progress.page
and show the actual page number in the sync prompt instead of
computing it from server_percentage × total_pages (which could
display the wrong page due to rounding differences between foliate
and koreader page math).
- syncToProgress: add a nil guard so a missing page number skips
gracefully instead of crashing.
collectBookData put getLastProgress() directly into the epubcfi field
for every document. For paging documents (has_pages == true: PDF, CBZ,
CBR, CB7, CBT, DjVu) that value is the current page as a *number*, so
the payload carried "epubcfi": 5 (a JSON number). The server types
KOReaderBookProgress.Epubcfi as *string, and the strict JSON binder
rejected it with a 400 ("cannot unmarshal number into ... epububcfi of
type string"), which the plugin surfaced as the misleading
"Failed to push progress. Check your network connection." Reflowable
EPUBs were unaffected because CREngine returns an xpointer string.
Only set epubcfi for rolling (reflowable) documents. Paging documents
carry their position in the numeric page/total_pages fields, which the
server already treats as the canonical locator for fixed-layout/comic
formats. The pull path already falls back to progress.page, so restore
keeps working.
Lua's json.encode produces {} for empty tables instead of [].
Use json.util.InitArray to mark tables as arrays so they encode
correctly as [] for authors, bookmarks, highlights, notes, and books.
- Append ?token= auth param to OPDS catalog URL
- Replace any existing stale Bookhoard entry (with or without token)
- Hot-update OPDS plugin's in-memory servers table so catalog
appears immediately without restarting KOReader
Writes directly to KOReader's opds.lua settings so the user
doesn't have to type the long URL. Also updates the manual
menu item to auto-configure instead of showing the URL.
Instead of showing a long auth URL that's impractical to type on
e-ink, tell users to open their Bookhoard web UI and approve from
Device Management. Uses persistent InfoMessage (no timeout) so
the message stays visible while they switch devices.
john-okeefe