H1's doctrine applied to the annotation service, which had been left on
setters: NewMediaHandler/NewKoboHandler/NewKOReaderHandler/
NewSyncQueueProcessor(WithConfig) now take it as a parameter; the four
SetAnnotationService setters are deleted; wiring updated (main.go,
router/sync.go, test harness).
With the service unskippable, the 18 nil-guards and their plain-query
fallbacks die:
- media.go: the create fallbacks for notes/highlights/bookmarks (the
bookmark one was the H3 census's Bug #1 — it dropped
percentage/chapter_reference the sync path carries), the highlight
PUT fallback, and the three delete-path plain deletes (tombstone is
now the only delete semantics).
- kobo.go: both markup-ingest annotation loops (the bookmark fallback
wrote the text into media_notes — a category error the fallback
never should have had).
- koreader.go: all three push-stack guards.
- queue.go: the three "annotation service not available" errors in
syncNote/syncHighlight/syncBookmark.
Behavior note: the deleted highlight create/PUT fallbacks were the only
paths passing note_id to the plain query — the sync service has no
NoteID field, so the web's highlight→note link was already being
dropped on the live path; this commit makes that pre-existing
limitation explicit rather than papering it over (surfacing note_id on
SaveHighlightRequest is left for a behavior commit that needs it).
Pins (now 20 incl. the 3 annotation CRUD suites) + H1 tests + both
unit suites green.
Pin-first batch for H3: freeze web annotation CRUD response shapes
(create 201 / update 200 / delete 204; full sqlc row models) before
the stack consolidation touches anything.
The new pins immediately caught a live server bug the 28 unit tests
could never see (annotations_test.go is pure-unit: keys and compare
logic, no DB): applyNoteLWW built UpdateMediaNoteForSyncParams without
UserID, so the query's WHERE id=$1 AND user_id=$13 compared against
zero — every dedup-hit note re-save (same content+position) returned
500 "update note: no rows in result set" and the update never landed.
The bookmark twin passes UserID and works; the note stack is the clone
that dropped the field.
Fix: pass UserID in the params (one line). Pins go green and become
the regression net; changelog records the new surfaces.
Found while verifying H1's "app offline-play -> flush -> position lands"
with the emulator path: the flush has been dead code in two independent
ways, invisible because the first break made the second unreachable.
1. ListPendingSyncQueueItems matched `device_id = $1` with a NULL $1
(the processor passes DeviceID{Valid:false} to enumerate all devices;
uuid = NULL matches no rows). The ticker therefore processed nothing,
ever. The WHERE now treats NULL as "every device" (the same
narg::uuid IS NULL idiom the visibility queries use; EXPLAIN'd — seq
scan on a bounded table, idx_sync_queue_status_priority_created
fits). The processor also dropped its fake-pagination loop, which
re-fetched the same first batch forever once the backlog hit the
batch size (no OFFSET in the query).
2. dbItemToSyncQueueItem dropped SyncData — even with a working
enumerator, every flushed item arrived with an empty payload and
failed JSON unmarshal. Dead code harboring silent breakage, exactly
the plan's thesis.
Callers updated for the renamed RowLimit param (queue handler, offline
detector). TestH1_SyncQueueFlush_LandsAndPreservesAudio drives the real
flow: web PUT seeds audio 900s, a pending checkpoint-style queue row
applies percentage 0.8 through SaveProgress, audio survives, queue row
completes.
Also fixes dashboard_test.go's stale BuildSections signature (the
package's test binary has been unbuildable) — user-approved: tests
designed against old shapes are not worth keeping.
Batch H1 step 3b — THE data fix. Conflict resolution was the one live
erasure writer: applyProgressResolution and its verbatim twin
applyResolution both wrote the winner via a raw UpdateUniversalProgress
whose param list omits audio_offset_seconds/context_text/viewports, and
the SQL's DO UPDATE SET ... = EXCLUDED... NULLed exactly those. The web
UI resolves through POST /api/conflicts/:id/resolve and
/api/conflicts/bulk-resolve (main.js resolveConflict/bulkResolve), and
the reader's mark-as-read fires a resolution first when the reader SSR
stamps data-conflict-id/-winner — so every resolution today erased the
audiobook position and reader context.
Now: ProgressService is a required ConflictHandler dependency; BOTH
call sites (single resolve + bulk resolve) share one
applyProgressResolution that funnels through SaveProgress (the verbatim
twin is deleted). Deliberate choices, per the plan:
- SkipConflictDetection on the resolution write: both flows apply
BEFORE the conflict row is marked resolved, so the funnel's LWW
detector would otherwise file a duplicate pending conflict
(HasRecentConflictResolution is false at apply time).
- Source "conflict_resolution" — the row's last-write attribution; the
winner stays durably recorded in resolution_data.winner.
- Broadcast: true — live devices jump to the winning position.
- The winner snapshot maps percentage/epubcfi/context_text/chapter/
character/page/total_pages; a winner without a CFI clears the locator
pair (ClearLocator) instead of re-pairing with the loser's stale
anchor. Everything the snapshot doesn't carry SURVIVES via the LWW
seed: audio_offset_seconds above all. chapter_progress keeps its
stored value (was overwritten with the book percentage), an absent
percentage survives (was written as 0%).
- Accepted: the resolution captures one reading_history session
(attributed to conflict_resolution; the 2h idle ceiling applies).
Verification: TestH1_ConflictResolutionPreservesAudio drives the real
flow — web push with audio 300s, diverging push from a second source
detects a conflict, resolve with the second source as winner, GET shows
percentage 0.3 AND audio_offset_seconds 300 (previously NULLed), and no
duplicate pending conflict remains. Full contract pin suite green.
The host-run TestConflict*/TestProgress* failures are environmental
(container-only /app/uploads path + db hostname; fail identically at
baseline b4ae423) — the same suites pass in the test-runner container.
Batch H1 step 3a. The ceiling census (exhaustive UpdateUniversalProgress
grep) found 9 nil-guarded `if progressSvc != nil { funnel } else { raw
erase-write }` fallbacks — the survey's "four" under-counted kobo.go,
which has FOUR sites (Markup ReadingSync, Markup last-read-place,
AnalyticsGettests, SyncFromServer), plus one each in media.go, queue.go
and koreader.go. All unreachable: the service is created unconditionally
at main.go:127 and wired into every handler and the queue processor; the
test harness wires them too.
The erasure pattern is now unrepresentable:
- NewMediaHandler / NewKOReaderHandler / NewKoboHandler / both
NewSyncQueueProcessor flavors take the ProgressService as a REQUIRED
constructor arg; the SetProgressService setters are gone.
- Every fallback body (raw UpdateUniversalProgress with explicit-NULL
pgtypes for viewport/scroll/panel/reading_mode/zoom and no
audio_offset_seconds/context_text) is deleted; each write funnels
through SaveProgress with the LWW seed, so fields the writer doesn't
know survive.
- Removed the now-dead koreader pg-type helpers (textPtrToPgText,
intPtrToPgInt4, int64PtrToPgInt8) and their self-tests.
Behavior deltas (stored-row content, no wire shape changes —
KOReader/Kobo response pins stayed green):
- KOReader fallback used to write explicit NULL viewports and drop the
context_text the koplugin sends; the funneled mainline already maps
book.ContextText.
- Queue flushes no longer overwrite chapter_progress with the book
percentage (bogus data; preserved via the seed).
- A web PUT without percentage no longer writes 0% via the deleted
fallback; percentage seeds from the stored row.
Contract suite: KOReader×4 + Kobo×3 pins green.
Batch H1 steps 1-2 of the hardening plan. Wire contracts frozen; the
pinned shapes are unchanged modulo the whitelisted additive GET keys
(docs/contract_pins.md).
1. One canonical request struct. The web PUT binds SaveProgressRequest
directly (identity/attribution fields are json:"-" so they stay
server-owned) and copies its server fields once — the hand-rolled DTO
and the 20-line field-copy block are gone. Parse set for existing
clients is IDENTICAL to the old DTO (verified in web/src: the ebook
reader sends percentage/current_page/total_pages/reading_mode/epubcfi/
context_text/chapter/zoom_level — reader.ts:3113-3143; the audiobook
reader sends percentage/chapter/audio_offset_seconds/elapsed_seconds/
reading_mode — audio.ts:518-520).
2. GET /api/media-items/:id/progress now serializes through one
row-struct-driven builder instead of a hand-map, so a row column
cannot be silently forgotten again (the audio_offset_seconds
omission). Existing keys are verbatim; the row already carried
audio_offset_seconds/reading_mode/viewport_x-y/zoom_level/
scroll_position_x-y/panel_number and they surface as WHITELISTED
additive keys. The web audiobook reader already reads
audio_offset_seconds (audio.ts:219) — with the key absent,
durationless books resumed at chapter start (offset always 0); they
now resume correctly. The conditional context_text bolt-on and the
server-derived css_selector/anchor_href block are unchanged —
restore precedence stays percentage-first, never re-derived.
Phase 1 (foundation): 'audiobooks' library type seeded (schema + DB);
audio_metadata JSONB on media_items (the kobo_metadata precedent —
only audio paths read it); reading_progress.audio_offset_seconds for
the honest {chapter, offset} locator; extensions/mimes registered for
ALL breadth formats (m4b/m4a/mp3/flac — deliberate per the DOC/LIT
doctrine, m4b scans richly first); classifyFormatGroup 'audiobook'
case (gotcha 39 recomputes on rescan); Go mime table extended at boot
so media elements accept the streams; ffmpeg/ffprobe in the image
(8.1.2 — also the xHE-AAC identification tripwire).
Phase 2 (ingest): sidecar-first (Libation/ABS metadata.json —
narrators/ASIN/curated chapters), ffprobe fallback/filler (duration,
bitrate, codec/profile, embedded chapters). Fixed three real
parser brittlenesses the Libation sidecar exposed: series accepts
string OR array-of-objects, publishedYear accepts number OR string,
ffprobe chapter timings accept number OR string (flexFloat). Folder-
structure fallback now runs on the UPDATE path too (rescans were
clearing sidecar-absent series/author); audio facts re-derive on
every rescan (gotcha 39).
Phase 3 (playback + sync): GET /api/media-items/:id/chapters (the
format-abstraction list); streaming rides the existing cookie-authed
Range /uploads path; PUT progress carries audio_offset_seconds +
elapsed_seconds (honest wall-clock playing time) — the reading-
history capture prefers player-reported elapsed over the inter-push
delta, same 2h ceiling; reading_mode='audio' marks the row.
Phase 4 (web player): dedicated audio reader page (foliate not
involved) — cover, chapter title, whole-book scrubber, skip ± (chips
1/5/10/15/30s, default 10s), speed (0.75×–2×), chapter list with
jump-to, Media Session lockscreen/notification transport, periodic
playing-only progress pushes, position restore on open.
Verified live: ingest of the Libation sample (Gardens of the Moon —
title/author/series/cover/date from sidecar+folders, 34 chapters,
26h duration, aac/LC); chapters endpoint; the reader route serving
the player; playback with position restore (0:57 round-trip);
progress rows in audio mode; wall-clock session rows in
reading_history.
The design parked in the handoff (multi-tier stats off reading_history;
bookorbit-style in data, not looks) is in. The real prerequisite work
was data: reading_history existed but NOTHING wrote to it — the whole
stats feature was sitting on an empty table.
- CAPTURE (no client changes, no repo-boundary crossings): every
successful progress write through SaveProgress (KOReader + app + web
+ Kobo + offline queue — the one chokepoint) now records one
reading_history session row: span = the interval since the PREVIOUS
write instant, pages = positive page delta when both sides know the
page index, marker rows for first contact. Spans over 2h are kept as
zero-second markers (overnight suspend→resume, offline flushes must
not inflate totals) — the honest 'sync cadence, not a stopwatch'
caveat stays. device_id only when a REAL device row matches (the
web/app progress path historically stuffs the USER id into DeviceID
— probe instead of trusting it; FK violation would silently drop
every capture). Source preserved in device_metadata.
- API: /api/stats/summary?range=7|30|90|year (totals, zero-filled
daily arrays, streaks, most-active day/weekday, top books, library
breakdown — one call), /api/stats/this-week (widget: rolling 7d vs
previous + streak), /api/stats/books/:id (lifetime + recent
sessions). All bucketed in the user's timezone; finished ≡ any push
≥0.995; visible-library lens enforced at source. Legacy
/api/analytics/* kept untouched. Range windows anchor end-of-day to
the START of today (blasting the clock time into tomorrow resolved
'today' a calendar day late — found by the streak test).
- /stats page (sidebar Stats between Progress and Devices, previously
/analytics was an orphan): range selector, five stat cards, two
Chart.js v4 charts (daily minutes bar, pages/day line — chunked via
dynamic import so main.js stays untouched), per-library breakdown,
most-read list, empty-state explainer. /analytics now 301s to /stats.
The orphaned analytics.templ + analytics.ts removed (their containers
and the page JSON never matched; the page was astub).
- Dashboard 'This Week' widget (server-rendered, suppressed on service
failure) and the book detail 'Your Reading Stats' card (lifetime
totals of the viewer's own sessions) complete the tiers: per-user
default, per-library breakdown inline, per-book on the page,
per-system/admin aggregate parked as before.
Tests: statistics_test.go (capture semantics at the chokepoint,
tz/day buckets, zero-fill, streak math, finished detection, range
validation, auth gate, widget, per-book lifetime). Full suite PASS.
The family-lens hardening: library_visibility was only enforced where
file bytes flowed (download/reader/OPDS) — metadata surfaces leaked
hidden libraries to any authenticated account, and registration was
open by default with no admin-side user creation.
- Series are grouped PER LIBRARY now (same name in two libraries =
two entries: manga adaptations / LN splits no longer merge);
/api/series aggregates only the requester's VISIBLE libraries and
/api/series/books REQUIRES a visible library_id (breaking contract;
web + app clients updated in the same change).
- /api/media-items, item detail, and the reader content endpoints
(chapters/panels/outline/THUMBNAILS — page images leaked unguarded
next to a gated GetPage) all enforce the visible set; resolveLibrary
no longer honors a hand-typed library_id outside the visible set.
- Annotation/note/bookmark mutations (update/delete/tombstone, sync
LWW paths incl. conflict resolution) carry ownership predicates:
another account's bare row id is a 404, not a silent delete.
- Admin user creation: POST /api/auth/users (AdminMiddleware) + a
Create User card in the admin Users UI — accounts stay creatable
with self-service registration OFF, which is now the schema default
(fresh installs; the first-admin bootstrap exception is unchanged).
- Tests: series contract re-pinned (per-library, visibility 403s),
new ownership + admin-create integration suites, OPDS all=1 test
now passes an explicit page size (shared-DB growth made the fresh
books fall off page 1), device-cap helper switched to the admin
endpoint.
TestManualCleanRepro replays a real device highlight export through the
same conversion path the sync handler uses, diffs the output against
the CFIs stored at ingest time, and computes a ground-truth oracle by
locating each selection in the flattened document itself. Classifies
every row as healthy / diff-base / backwards for both the converter
and the stored data. Skips automatically when fixture files are not
present on the machine (manual test, like cfi_verify_manual_test.go).
REPRO_VERBOSE=1 prints per-row detail; REPRO_EMIT_SQL=<path> emits
idempotent UPDATE statements that heal database rows written by the
old converter from the preserved raw device positions - used to
backfill dev and validated against production data.
KOReader highlights converted to wrong or unresolvable web CFIs
whenever a selection crossed text-node boundaries. Three compounding
defects: ParseCREXPointer discarded the text()[N] node index and
applied .offset as a global block offset; the end anchor converted
with an empty context (blind structural landing, zero verification);
and the block-containment check accepted wrong-node landings because
a selection's text always occurs somewhere in its own block. pos0 and
pos1 were converted independently, yielding mismatched start/end base
paths and backwards ranges - highlights stored but never painted, or
painted at the wrong location, in the web reader.
Add ConvertHighlightRange: both endpoints are anchored by locating the
selection text in the flattened document text (whitespace-normalized,
crossing block boundaries), with the device locators demoted to
disambiguation hints. A single document match yields a same-spine,
ordered start/end pair by construction. Falls back to the legacy
per-endpoint path when the text cannot be located, so behavior is
never worse than before.
Also fix the shared flatten stream: zero-width format characters
(ZWSP/ZWNJ/BOM) and whitespace-only inline spans (<span> </span>)
now act as word separators instead of fusing adjacent words, which
prevented text matching in real books.
Verified against 28 real device highlights from production: converter
output now matches the text-search oracle on all of them (previously
8/28 correct, 18 with mismatched base paths, 2 backwards).
The verify/heal pipeline verified only INBOUND context-bearing writes;
stored-row coherence was never enforced. SaveProgress seeds missing
fields from the existing row, so every percentage-only write (KOReader
low-confidence conversion, KOReader no-pointer payloads, Kobo markup
ReadingSync / analytics / sync-from-server, locator-less queue entries)
silently re-paired its NEW percentage with the PREVIOUS writer's stale
CFI+context — the incoherent row shape that fed the client-side
far-anchor restore floor (and, via the read-side anchor handles, could
even hand a first-party client the OLD position's scroll address).
KOReader's converter already declared the right intent ("storing
percentage only — the row's percentage restores approximately"); the
seeding betrayed it. SaveProgressRequest gains ClearLocator, set by the
percentage-only call sites, and the seeding rule honors it: stale
epubcfi/context/character_offset clear (the upsert assigns columns
directly, so NULL sticks) unless the request carries its own. A
request-carried context survives so the verify/heal pass can still mint
a fresh anchor from it; the HTTP merge path (field-absent PUTs
preserve) and all anchor-bearing writes are untouched. The fixed-layout
branch already cleared locators unconditionally — unchanged.
Consumer audit: percentage — the only thing every sync round-trip
exchanges — is untouched on all paths; KOReader pulls on locator-less
rows already support the nil-CFI shape (builder guards
progress.Epubcfi.Valid); first-party restores go percentage-first
instead of floor-fighting a stale anchor.
Tests: TestProgressWeb_LocatorCoherence pins clear-on-declared,
request-carried-context survival, and merge-path preservation. Full
suite with the change: 218 groups, zero failures.
.IMAGE — extensions removed from AllowedExtensions (scan gate),
bookExtensions, MimeTypes and sync/format.go maps: .doc/.lit files
are no longer scanned or indexed. classifyFormatGroup gains the
RTF and PDB reflowable arms that were missing when those rungs
shipped (rows only reclassify on creation). Scanner docs updated.
DOC + LIT have no credible JS tooling (mammoth is docx-only; LIT
needs LZX and its DRM variants are dead) and text-extraction-only
support would misrepresent what the reader can do. Verified: probe
.doc/.lit files watched but never scanned; probe .txt control
scanned; TestClassifyFormatGroup extended and passing.
Docs change granted in-session by the user (extensions + format
docs).
countTextCharsBefore started its sibling walk at the target node itself
and recursed after counting the current node's subtree, so every
ancestor level re-counted everything accumulated so far — a node at
depth 4 in the single-document 1984 epub reported 2.38M chars before it
in a 589k-char document, producing healed percentages of 4.04 (>1) and
a 500 on the reading_progress percentage check constraint. Every
healed progress save from a fresh client failed; only exact-context
matches (no heal) stored.
The walk now starts at target.PrevSibling: strictly the characters
before the node, per the contract all three callers already assume
(healed book offsets, CRE convert percentage, kepub offsets).
Found by the real-phone validation pass: the phone's first healed
submission 500ed where the emulator's had matched context exactly and
never taken the heal path.
Offset currency policy, now explicit: EPUB CFI terminals, CRE text()
offsets and the served char_offset handle are UTF-16 code units (the
EPUB CFI spec, and what foliate/readium/KOReader/Kobo clients actually
observe), while internal arithmetic — the book-wide character_offset
column and percentage fractions — stays rune-based, consistent with
TotalCharacters. For all-BMP books the currencies are identical, so no
stored value changes; astral-plane text (emoji, rare CJK) no longer
drifts.
Boundaries converted: resolveCFIToNode interprets incoming CFI terminal
offsets as UTF-16; textNodeAtUTF16Offset (née textNodeAtRuneOffset)
interprets CRE text() offsets as UTF-16; buildCFI and buildCREXPointer
emit UTF-16 terminals; blockCharOffset (the served char_offset) is
UTF-16.
Also fixes two character_offset column defects: heals wrote a BLOCK-
relative offset into the book-wide column, and verified-but-unhealed
saves (e.g. KOReader pushes) never refreshed it, leaving it stale
behind the anchor. VerifyProgressAnchor now returns the verified book-
wide rune offset and SaveProgress refreshes the column on every
verified save.
Tests: astral currency round trip (offset after an emoji must shift by
one unit between currencies, in both heal and exact-verify directions)
and book-offset ordering. The cmd/server/tests integration harness
failures under docker (library folder 400 during setup) reproduce on
the pre-change tree and are unrelated.
Progress submissions now carry (percentage, context_text, epubcfi) and
the server becomes the position authority:
- VerifyProgressAnchor resolves the submitted standard CFI against the
book's own XHTML, extracts the text at the anchor, and cross-checks it
with the submitted context_text. A mismatch heals the anchor by text
search (percentage disambiguates repeats) instead of storing a bad
position.
- The anchor's block element is derived as a cssSelector plus a block-
relative character offset, and served on progress GET alongside the
anchor document's href — readium-native handles that let clients
re-open a book without parsing CFIs themselves.
- context_text-only submissions (no CFI — the dumb-client tier) are
anchored structurally from the context text.
Motivation: cross-client progress sync (web foliate CFIs, KOReader CRE
xpointers, readium-native apps) previously trusted each client's own
locator math; the app's EPUB restore drifted ±pages because readium's
paginator does not lay out far-from-viewport columns and the foliate-
ported CFI walk ran against readium's mutated WebView DOM. Server-side
verification heals both classes at ingest.
The LWW skip path compared only annotation content (text, color, note,
percentages) — locator columns were not part of 'changed'. A device echo
with identical content therefore resolved to skip, and the freshly
re-derived canonical CFIs were discarded in the same request that
computed them: a highlight whose stored start anchor had been corrupted
by the old percentage-exact bug could never heal, because every
subsequent echo carried the same text and was skipped before the
locator columns were written. Observed live: a push converted the
cross-block highlight's anchors correctly (structural, heading to
paragraph) yet the row kept its garbage start CFI and the highlight
stayed unpaintable on the web.
Echo saves now treat a non-empty incoming locator that differs from the
stored one as a change (locatorRefresher): empty locators still coalesce
(no drift), and once healed the echo produces identical CFIs, so the
steady state remains skip — no write churn. Highlights check
epubcfi_start/end; bookmarks check cfi_position/position.
applyBookmarkLWW also gains the empty-locator coalescing the highlight
path already had: web bookmark edits carry no device locators, and a
title/note edit must not wipe the stored device-native position.
Tonight's failures all traced to one blind spot: the converter could
only reason about text within a single block. A position at a chapter
heading sends walk-up context (heading + the paragraphs below, joined by
the plugin's block capture); a selection can span several paragraphs.
Neither shape could be verified (containment compared one block against
a multi-block quote, so the CORRECT structural landing at the heading
was rejected) nor matched by text search (it never crossed block
boundaries). The ladder then fell to the percentage rung — which labeled
its char-count guess Precision "exact" — and that confidently-wrong CFI
was stored: reading positions reopened paragraphs away from the true
spot, and a highlight echo overwrote the row's good web CFIs with a
garbage start anchor that made the highlight unpaintable ("disappeared").
Four changes, all in the forward converter and its consumers:
- Quote verification: after the structural walk lands, read the
whitespace-normalized document text forward from the landing point
(crossing block boundaries; inline spans join directly so drop-cap
splits still read as one word). A usable context must be a prefix of
that stream — which is exactly what device captures are: the text from
the position onward, or the selection between two anchors. The old
single-block containment checks remain as secondary acceptance.
- Cross-block text search: the search rung matches against the whole
document flattened in reading order, with every rune mapped back to
its source node and offset. A context spanning blocks now matches, and
the matched extent yields a true range end (EndEPUBCFI) that
highlights use as their end anchor, threaded through the facade as
CanonicalLocator.EndCFI.
- Honest labels: the percentage rung returns Precision "percentage" —
a char-count estimate must never masquerade as an exact anchor.
- Confident-only storage: progress adopts a converted locator solely at
structural/exact precision (section hrefs keep their legacy handling;
anything lower stores percentage only), and highlight conversion
returns CFIs only at structural/exact precision — a low-confidence
echo yields empty, which applyLWW coalescing turns into preservation
of the row's existing web CFIs instead of clobbering them.
Tests: walk-up context at a heading verifies structurally and lands in
the heading; a block-spanning context is found by search with a range
end landing in the following paragraph; the percentage rung is honestly
labeled; all drop-cap guards stay green.
PUT /highlights/:id parsed the row id from the URL and then dropped it:
the sync-aware path routed through SaveHighlight's content-derived dedup
key, on the assumption that the same text + CFI always resolves to the
same key. That assumption breaks in practice — the stored epubcfi_start
drifts from foliate's range shape to the converter's point shape after a
device echo rewrites the row (bucketPosition cuts at the last colon, so
'…/6,/1:367,…' and '…/6/1:367' bucket differently), and the user can
edit the selection text. The recomputed key then misses the row being
edited and createHighlight mints a second one: the edited row (with
note, no device pos0) beside the original — served to KOReader as two
highlights, one noted and one not. Editing a selection's text would hit
the same trap.
SaveHighlightRequest gains an optional HighlightID. When set, the save
resolves the row by id (ownership-checked), LWWs against it under its
stored dedup key, and never re-derives identity from content. The PUT
handler passes the already-parsed id. Device pushes, Kobo, and the sync
queue send no id and keep the identity-based flow untouched.
applyLWW also stops wiping stored locators on web edits: the web reader
sends empty start/end positions (it never had a CRE xpointer), so a
note/color edit now keeps the device-native positions and CFIs instead
of blanking them — round-trip serve-back for device-created highlights
survives web-side edits.
Progress, highlights, notes, and bookmarks entered position conversion
through three different doors: progress converted inline with an
uncached converter, annotations through the facade, bookmarks not at
all (the raw xpointer was stored verbatim, cfi_position stayed empty,
and the web drawer's goToBookmark silently no-ops on cfi-less entries).
Unify on the facade (ConvertToCanonical/ConvertFromCanonical):
- annotationEpub context resolved once per push: media item + EPUB path
shared by every annotation instead of re-fetched per entry
- progress forward: the inline block becomes one facade call;
non-reflowable formats pass through unchanged, and the cached
converter stops re-parsing the book on every sync
- progress reverse: convertCFIToXPointer delegates to reverseConvertCFI,
keeping the stored percentage in play for the fallback ladder
- bookmarks (bulk progress and /sync-bookmarks): pos0 resolves
structural-only — bookmark text is a display label, never book text,
so no context is supplied; webUsableCFI stores the result only for
structural/exact epubcfi landings, discarding href/percentage results
rather than storing dead drawer links. Also records percentage_location
and origin_source on the legacy endpoint.
- percentages thread through: highlights/notes/bookmarks pass the device
percentage or the derived section percentage instead of a hardcoded 0,
so the last-resort fallback lands near the true position instead of
the document start
- extendCFIByLength end-derivation now also fires on structural starts
(it had silently stopped matching when the structural rung began
landing starts with precision 'structural' rather than 'exact')
Tests: the drop-cap xpointer through the facade with empty context (the
bookmark scenario) must land structurally, not doc-start; webUsableCFI
table covers the store/discard gate.
Per-annotation percentage derivation (deriveAnnotationPercentage) built a
fresh CFIConverter for every highlight/note/bookmark, re-reading and
re-parsing the whole EPUB each time. Export SectionPercentageCached so
handlers reach the same bounded cache ConvertToCanonical already uses
(8 books, insertion-order eviction): one parse per book per push instead
of one per annotation.
- Drop UNIQUE(media_item_id,user_id,title): titles are display labels
shared verbatim across clients; same-title bookmarks on different pages
now coexist instead of 500ing (deleting over a tombstone no longer
blocks future creates with that title)
- Add origin_source column recording the creating client (android/web/
koreader), set once at insert, exposed in API responses
- Web reader auto-title mirrors KOReader's 'in <chapter>' convention,
falling back to 'Bookmark'; adds bookmark rename in the drawer
Drop-cap markup like <p><span>C</span>onvergence of Heaven and Earth</p>
made getTextFromXPointer return just 'C'. The text search then matched
the first 'C' in the chapter and stored doc-start (/4/2/1:0) with
'precision: exact', so the web reader reopened at the chapter start
while the percentage looked mid-chapter.
- Add convertByStructuralPath: walk the parsed CRE ElementPath against
the raw XHTML (same-tag 1-based indexing, mirroring buildCREXPointer),
map CharOffset into the target element's text, and build the CFI.
Usable device text verifies the landing; disagreement falls through
instead of storing a confident-but-wrong CFI.
- Add usableContextText guard (>=8 runes, >=2 words): single chars can
never claim an exact text-search hit in either direction
(ConvertCREToStandard and reverseByTextSearch).
- Add drop-cap regression fixtures plus usable-context unit tests.
- Verified against the real book: DocFragment[26]/p[12]/span header now
converts to epubcfi(/6/52!/4/28/2/1:0) structural both with 'C' and the
full header, and round-trips back to DocFragment[26].
RestoreAnnotationByID and PurgeAnnotationByID dispatch on annotation kind
(highlight/note/bookmark) to the new queries, broadcasting an annotation
update on restore so connected web sessions refresh. Both report whether
a row actually changed.
TombstoneBookmarkByDedupKey mirrors the existing TombstoneHighlight for
bookmarks: devices report deletions by dedup key (they have no row IDs),
and until now only highlights had a key-based tombstone path — device
bookmark deletions had nowhere to land.
ValidAnnotationKind centralizes the kind check the HTTP handlers share.
Six converter tests pointed at absolute paths for 1984 and Crime and
Punishment under uploads/ — books that don't exist on most checkouts
(CI included), so the suite shipped with 5 permanently failing tests
(and a sixth passing only by accident: the percentage-fallback path
triggered by the missing file is the outcome it asserts).
A writeTestEPUB helper now builds a minimal deterministic EPUB in
t.TempDir() (zip → container.xml → OPF → 6-doc spine), so the tests
exercise the real zip/OPF/spine/document pipeline with no external
dependencies. The xpointer→CFI conversion, fragment-ID conversion,
both round-trips (bare and context-text-anchored), and the text-search
and percentage fallbacks all keep their original assertions, now
against known document content. internal/sync is green for the first
time on this machine.
Echo duplication: devices push their full annotation list on every
sync, and an echo of a web-created annotation computed a different
dedup key than the original (device locators differ from web locators)
— every pull→push cycle minted a duplicate row, and cleaning those up
on the web tombstoned them back to the device, deleting the
just-applied copies. That was the "web highlights never appear on
KOReader" experience. GetMetadata now serves each annotation's
dedup_key; the device stores it on the applied entry and echoes it in
pushes; SaveHighlight/SaveBookmark/SaveNote accept a DedupKey
override so echoes converge onto the original row (verified: pull →
echo push creates no rows, LWW skips identical content).
Color semantics (per user preference): devices render their own
default and cannot round-trip web colors, so GetMetadata no longer
serves colors at all — every highlight syncs regardless of its web
color and the device draws its default. An echo carries no color;
ingest then PRESERVES the stored web color (existingHighlightColor
lookup by dedup key) so round-trips never change it. A non-empty
device color means the user edited the highlight there: it maps
name→hex (green→#a5d6a7, default yellow) and wins. Verified: echo
kept #ffd54f; a simulated device edit with "green" updated the web
row to #a5d6a7.
Classification: KOReader auto-fills text="in Chapter X" on page
bookmarks (ReaderAnnotation:updateItemByXPointer), so the plugin's
text-presence classification turned every echoed bookmark into a junk
highlight on the web. v2 classification now keys off the drawer field
(present = highlight/note, absent = bookmark with its label in note).
ConvertToCanonical/ConvertFromCanonical built a fresh CFIConverter
per call, and each annotation converts twice (pos0+pos1) — a book
with 200 highlights re-opened and re-parsed the EPUB 400+ times per
sync, and again per metadata pull. A bounded 8-entry cache keyed by
path now shares converters (the parsing work belongs on the server;
clients stay thin). CFIConverter gained a mutex around its lazily
built spine/doc caches since instances are now shared between
concurrent requests.
Adds CFIConverter.SectionPercentage: book-wide percentage for a CRE
xpointer from the spine char distribution (midpoint of its document)
— the server-side counterpart to dropping per-annotation
getPageFromXPointer lookups from the plugin.
Deleting a bookmark/highlight/note and then re-adding the same content
at the same position (same dedup key — e.g. the reader's auto-titled
'Bookmark at X%') was silently swallowed: the save hit the tombstone
branch, returned 201 with the deleted row, and the list (which filters
deleted) stayed empty. Bookmarks were further blocked by the
UNIQUE(media_item_id, user_id, title) slot the tombstoned row holds,
and notes had no TTL escape at all.
Tombstones now only block saves that predate them (stale replays from
a device that still has the annotation). A save whose modification
time is newer than max(deleted_at, last_modified_at) — a deliberate
re-create from the web or a device — resurrects the row via the LWW
update queries, which now clear deleted/deleted_at.
CleanupExpiredRefreshTokens and CleanupExpiredOpdsTokens were generated
by sqlc but never invoked anywhere in the codebase, so expired/revoked
tokens accumulated in the database indefinitely. The refresh-token query
was parameterized in the settings-registry work specifically so its
retention window could follow the configurable session duration, but the
periodic caller was never wired up.
annotations.go:
- Rename StartTombstonePurger to StartDailyMaintenance, which now runs
all periodic cleanup tasks from a single 24h-tick goroutine.
- Add runDailyMaintenance helper: tombstones, then OPDS tokens, then
refresh tokens, each logging independently so one failure never skips
the others.
- Refresh-token retention is read from the registry (SessionDuration)
on every tick so live admin edits are honored; guarded on the registry
being wired so unwired test paths simply skip cleanup.
- All three queries only delete rows that are already expired or
revoked, so active sessions are never logged out.
main.go:
- Update the call site: tombstonePurgerCancel becomes maintenanceCancel
and calls StartDailyMaintenance.
Net footprint: still one goroutine and one ticker; the cleanup adds one
DELETE per table per day.
Split each constructor into a default-args wrapper and a config-accepting
variant so the sync queue interval/batch size and the worker pool size/
queue cap can be sourced from the settings registry at startup. These
values are constructed once at boot, so they are tagged requires_restart
in the admin UI.
queue.go:
- NewSyncQueueProcessorWithConfig(db, interval, batchSize) takes the
flush interval and batch size as parameters; NewSyncQueueProcessor
becomes a thin wrapper with the historical 5s / 50 defaults.
worker.go:
- NewWorkerWithConfig(numWorkers, queueCap, connManager) takes the
queue capacity as a parameter; NewWorker becomes a thin wrapper with
the historical cap of 100.
No behavior change for existing callers; main.go will switch to the
config-accepting variants in a follow-up wiring commit.
The 30-day retention window for soft-deleted annotations was a package
const; move it behind the registry so it can be tuned live.
annotations.go:
- AnnotationService gains an optional *database.SettingsRegistry and a
tombstoneTTL() helper. The skip-resurrect checks and the purge cutoff
now call it instead of reading the TombstoneTTL const directly.
- Add ActiveTombstoneTTL() so callers outside the sync package can
compute cutoffs consistently with the service.
- The package-level TombstoneTTL const is retained as the fallback for
tests / unwired code paths.
kobo.go, koreader.go:
- The per-book tombstone sweep cutoff now uses
h.annotationSvc.ActiveTombstoneTTL() instead of the wsync.TombstoneTTL
const, so both the service and the handlers honor the configured TTL.
Wire AnnotationService into SyncQueueProcessor and implement the three
previously-stubbed execute methods:
- syncHighlight: unmarshals syncData JSON into SaveHighlightRequest,
applies CRE→CFI conversion via AnnotationService
- syncNote: unmarshals into SaveNoteRequest
- syncBookmark: unmarshals into SaveBookmarkRequest
- Add SyncTypeBookmark to executeSync switch (was hitting default error)
Add enqueue methods for future offline/batch use:
- EnqueueHighlight / EnqueueNote / EnqueueBookmark
- Shared enqueueAnnotation helper creates queue items with
PriorityCriticalNote and 3 max attempts
- Update types (HighlightUpdate, NoteUpdate, BookmarkUpdate) mirror the
existing ProgressUpdate pattern
Existing handler behavior is unchanged — annotations still sync
synchronously via AnnotationService. The queue path is available for
retry-on-failure and offline batch processing scenarios.
AnnotationService is the central service for cross-device annotation sync.
It provides SaveHighlight, SaveNote, and SaveBookmark methods that handle
the full sync lifecycle:
Identity (3-layer):
1. Server UUID (primary key)
2. Per-device native ID stored in device_sync_data JSONB
3. Content dedup_key: sha1(normalize(selection_text) + bucket_position)
- CFI character offsets are stripped for bucketing so the same
highlight at slightly different offsets still deduplicates
- Raw positions are preserved in the DB for precise restoration
Resolution policy (LWW):
- When the incoming annotation has an explicit ModifiedAt timestamp,
last_modified_at wins
- When the device sends zero ModifiedAt (creation time only), field-diff
mode compares content fields (text/color/note/percentage) — if all
match, the save is skipped; if any differ, the save is applied with
server-receive-time as the new last_modified_at
Conflict detection:
- When incoming and existing annotations have different sources (e.g.
koreader vs kobo) and content differs, an auto_resolved sync_conflict
is recorded with both sides' data for audit trail
- Broadcasts a WebSocket conflict notification for real-time UI updates
Tombstone management:
- Delete-wins: tombstoned annotations block recreation from stale pushes
- 30-day TTL before physical purge
- PurgeExpiredTombstones method + StartTombstonePurger goroutine (24h ticker)
Add locators.go with unified bidirectional CFI conversion:
ConvertToCanonical / ConvertFromCanonical
- CRE XPointer <-> standard EPUB CFI (for KOReader)
- KEPUB CFI passthrough (for Kobo)
- Skips non-reflowable formats (PDF, CBZ, fixed-layout EPUBs)
Add 25 unit tests covering:
- Dedup key determinism, text normalization, position sensitivity
- Offset insensitivity (CFI char-offset bucketing)
- Device sync data merge (preserves existing, overwrites same source)
- Cross-source detection
- LWW comparison (newer wins, older skipped, fallback to updated_at)
- Field-diff mode (identical content skipped, changes applied)
- Tombstone TTL constant
- CRE XPointer parsing and classification
- Standard EPUB CFI classification
Remove TestConvertHessBook and TestDebugHess (both referenced a
non-existent Hess EPUB at an absolute local path) and
TestConvertCPByTextSearch (referenced a Crime and Punishment EPUB
in the local uploads directory). These were development-time debug
tests that only worked on the author's machine.
All CFI/KEPUB conversion behavior is already covered by the proper
fixture-based tests (TestKEPUBRoundTrip, TestKEPUBConvertKEPUBToStandard,
TestKEPUBConvertWithEmElements, etc.) which use createTestEPUB and
createTestKEPUB helpers.
When a Kobo device pushes a last-read-place bookmark, the server now
converts the KEPUB CFI (with koboSpan wrappers) to a standard EPUB CFI
and extracts surrounding text as context_text for use by other devices
(KOReader, web reader) during their pull-side CFI conversions.
Previously the raw KEPUB CFI was stored verbatim as epubcfi, which
meant foliate and CREngine couldn't resolve it (wrong child indices
due to koboSpan wrappers), and no context_text was available for the
text-search fallback in ConvertStandardToCRE.
Changes:
- kepub_cfi_converter.go: Add ExtractedContext field to
KEPUBConversionResult, populated from the already-computed
searchText in both ConvertKEPUBCFIToStandard and
ConvertStandardCFIToKEPUB (exact-match and percentage-fallback
paths).
- kobo.go: Add libraryService field and SetLibraryService setter
(mirrors KOReaderHandler pattern). Add convertKoboCFIToStandard
helper that resolves EPUB+KEPUB paths, instantiates the converter,
and returns the converted CFI + extracted context. The last-read-place
branch in Markup now calls this helper for reflowable formats,
skipping fixed-layout/comic archives (page-index only).
- router.go: Add LibraryService to router Config.
- sync.go: Wire LibraryService to KoboHandler via SetLibraryService.
- main.go: Pass libraryService through router config.
The conversion is purely additive — if no KEPUB file exists on disk
(e.g. side-loaded EPUB without kepubify conversion), the handler
gracefully skips conversion and stores the raw CFI as before.
The push path (koreader→server) was already updated to omit epubcfi
for paging documents, and SaveProgress derives percentage from
page/total_pages for fixed formats. However, the pull path
(server→koreader) still returned any stored epubcfi and attempted
CFI→XPointer conversion, and SaveProgress preserved stale CFI values
written by the web reader (which generates fake CFIs via
CFI.fake.fromIndex for every comic/PDF page).
These fake CFI strings lingered in the database and koreader's pull
path picked them up over progress.page, causing tonumber("epubcfi(...)")
→ nil → GotoPage(nil) → crash when the user confirmed the sync prompt.
Changes:
- GetMetadata (koreader.go): for fixed_layout/comic_archive formats,
skip returning epubcfi and skip the CFI→CRE XPointer conversion.
Reflowable documents are byte-for-byte unchanged.
- SaveProgress (progress.go): for fixed formats, explicitly clear
epubcfi and character_offset on every save so stale values from
prior web-reader sessions are cleaned up over time.
Fixed-layout EPUBs, PDFs, DjVu, and comic archives (cbz/cbr/cb7/cbt)
are page-based: each page is a fixed image, so a page index is an exact,
universal locator regardless of screen size or device. Sync previously
treated these like reflowable content (CFI-first restore, percentage
fallback, character-offset math), which was both wrong and lossy. This
makes the page index the canonical position for fixed-layout and comic
formats while leaving the reflowable path byte-for-byte unchanged.
Backend:
- progress.go SaveProgress: branch on mediaItem.FormatGroup. For
fixed_layout/comic_archive derive percentage from current_page/
total_pages and skip the CFI/character-offset back-fills (meaningless
for image content). The reflowable derivation block is preserved
verbatim under an else.
- kobo.go: Kobo only sends a percentage, so for fixed-layout/comic
formats derive CurrentPage via PercentageToPage(percentage, pageCount)
using the media item's known page count, so Kobo->web lands on the
exact page.
Reader:
- reader.templ readerInitExpr: pass savedPage/savedTotalPages to the
reader config for fixed_layout/comic_archive formats.
- reader.ts: when isFixedLayout and savedPage is present, restore via
view.init({ lastLocation: savedPage - 1 }) (a bare number navigates
foliate directly to the section index). Reflowable falls through to
the existing CFI->percentage path, unchanged.
cfi_converter_test.go:
- TestFindTextInNode_SingleTextNode: baseline single-node match
- TestFindTextInNode_CrossEmElement: 'Vokalia and Consonantia' across
two <em> elements — verifies match returns the 'Vokalia' text node
- TestFindTextInNode_CrossStrongElement: text crossing <strong> boundary
- TestFindTextInNode_DoesNotCrossParagraphs: verifies block boundary
enforcement — text split across <p> elements is NOT matched
- TestFindTextInNode_NestedFormatting: <em><strong> nesting
- TestFindBlockParent: verifies findBlockParent walks up through inline
elements to find block-level <p>
- TestCollectInlineText: verifies text segments are collected in order
with correct content
kepub_cfi_converter_test.go:
- TestKEPUBConvertWithEmElements: previously skipped, now expects exact
precision and verifies round-trip conversion works for text spanning
<em> element boundaries
Instead of reading only from the single resolved text node (which
produces a tiny window when the position is inside <em> or other inline
elements), extractSurroundingText now:
1. Finds the block-level parent (e.g. <p>) of the resolved text node
2. Collects all text within that block, transparently crossing inline
formatting elements via collectInlineText
3. Computes the global offset of the original text node within the
concatenated block text
4. Extracts the [offset-window : offset+window] slice
This gives a full context window regardless of inline element
boundaries, enabling accurate text bridging between EPUB and KEPUB
documents even when reading positions fall inside <em>, <strong>,
<span class="koboSpan">, etc.
Falls back to single-node extraction when no block parent is found
(e.g. orphan text nodes in tests).
When finding or extracting text in EPUB/KEPUB DOM trees, inline
formatting elements like <em>, <strong>, <i>, <b>, <span>, etc. should
not break text continuity. A reader sees 'Vokalia and Consonantia' as
one phrase regardless of the <em> wrappers around each word.
Add inline formatting element set and helper functions:
- isInlineFormatting: checks if an element is an inline phrasing element
- collectInlineText: flattens text across formatting elements within
a block-level parent, returning segments that map back to original
text nodes
- findBlockParent: walks up from a text node to find the nearest
block-level ancestor (used to scope text collection)
- findTextAcrossInlineElements: fallback for findTextInNode that
concatenates text within each block element (transparently crossing
formatting elements) and maps match positions back to actual nodes
- collectBlockElements: gathers all block-level elements containing text
The key invariant: text collection NEVER crosses block-level element
boundaries (<p>, <div>, <h1>-<h6>, <li>, etc.) to avoid concatenating
text from different paragraphs.
The findTextInNode function now tries single-text-node matching first
(fast path, unchanged), then falls back to cross-element matching only
when needed. This preserves performance for the common case.
Test coverage for KEPUBCFIConverter with programmatically generated
EPUB and KEPUB zip fixtures:
- KEPUB->Standard text search conversion (exact precision)
- Standard->KEPUB text search conversion with round-trip verification
- Multi-position round-trip (3 phrases across different paragraphs)
- Cross-chapter conversion (spine index 1)
- No-context-text conversion (extracts surrounding text from resolved node)
- Invalid CFI handling (falls back to percentage)
- Percentage fallback for unresolvable CFIs
- 5-paragraph round-trip covering different document positions
- CFI structural difference verification (koboSpan adds DOM steps)
- Spine index consistency between EPUB and KEPUB
- Real book conversion test (skips if file not present)
- extractSurroundingText unit tests
Add KEPUBCFIConverter in internal/sync that converts between KEPUB CFIs
(which include extra koboSpan DOM steps) and standard EPUB CFIs at sync
time, so only standard epubcfi values are stored in the database.
The converter works by:
1. Resolving the source CFI in the source document (EPUB or KEPUB)
2. Extracting surrounding text at the resolved position
3. Searching for that same text in the target document
4. Building a new CFI pointing to the matched text in the target
This text-content bridging handles the DOM structural differences
between EPUB (text nodes at depth 2) and KEPUB (text nodes wrapped in
<span class="koboSpan"> at depth 3).
Falls back to percentage-based positioning when text search fails,
matching the pattern used by the existing CRE converter.
No changes to existing cfi_converter.go or KOReader conversion code.
Same-package access to unexported functions (resolveCFIToNode, buildCFI,
findTextInNode, etc.) via internal/sync package placement.
- SaveProgressRequest: add ContextText field
- SaveProgress: carry existing ContextText from DB, overwrite when provided
- ProgressUpdate: add ContextText field for checkpoint sync
- SyncQueueProcessor: serialize/deserialize context_text in sync data
- buildProgressSnapshot: include context_text in snapshot data
Implements ConvertCREToStandard which converts CREngine XPointers
(e.g. /body/DocFragment[6]/body/div/p[47]/text().2399) to standard
epubcfi format (e.g. epubcfi(/6/12!/4/2[id]/4/1:7)).
Key components:
- indexChildNodes: faithful port of foliate-js's epubcfi.js algorithm
for computing CFI-compatible child node indices including virtual
positions, null positions between adjacent elements, and text chunks
- preprocessXHTML: converts XHTML self-closing tags (e.g. <a id="x"/>)
to open/close pairs so Go's HTML parser produces the same DOM as the
browser's XHTML parser
- buildCFI: walks up from a text node to body, computing CFI indices
at each level using indexChildNodes
- findTextInNode: regex-based whitespace-flexible text search for
context_text fallback positioning
- convertByPercentageOffset: estimates position via book-wide character
counts when no context_text is available
- ConvertCREToStandard: orchestrates text search → percentage fallback
Supports CREngine XPointer format, CREngine fragment ID format
(#_doc_fragment_N_anchor), and includes round-trip test coverage for
1984 and Crime and Punishment EPUBs.
All four progress write paths now delegate to ProgressService.SaveProgress:
- MediaHandler: UpdateMediaReadingProgress uses ProgressService for web
saves with richer request body (reading_mode, zoom_level, scroll). GET
now uses GetUniversalProgress query that JOINs media_items for
format_group, total_characters, chapter_count.
- KOReaderHandler: updateProgressForBook delegates to ProgressService.
Fixed device ID bug (was using userID, now uses deviceID). Removed
duplicate UpdateDeviceLastSync with zero UUID. Added pgtype helper
functions (textPtrToPgText, intPtrToPgInt4, int64PtrToPgInt8).
- KoboHandler: all four progress write points (Markup ReadingSync, Markup
last-read-place, AnalyticsGettests, SyncFromServer) delegate to
ProgressService. Fixed empty epubcfi string now correctly set to
Valid: false. SyncFromServer preserves last_sync_source=bookhoard
and Broadcast: false.
- QueueProcessor: syncProgress delegates to ProgressService.
- main.go: creates ProgressService after ConnectionManager, injects via
SetProgressService() on all handlers and queue processor.
Handler tests cover pgtype conversion helpers (textPtrToPgText, etc.)
and device icon mapping.
Introduces a centralized ProgressService that handles all reading progress
writes across web, KOReader, and Kobo clients. The service implements:
- Merge strategy: reads existing progress first, then only overwrites
non-nil fields from the incoming request. This fixes the data loss bug
where partial updates (e.g., Kobo last-read-place sending only epubcfi
and chapter) would NULL out percentage, character_offset, etc.
- Enrichment: computes missing fields from available data:
- character_offset from percentage + total_characters
- current_page from percentage + total_pages
- percentage from current_page + total_pages (reverse)
- percentage from character_offset + total_characters (reverse)
- Conflict detection: when a different source writes progress within 5
minutes with >1% difference, records a sync_conflicts row and broadcasts
a WebSocket notification for real-time UI alerts.
- Broadcast control: SaveProgressRequest.Broadcast flag lets Kobo
last-read-place and SyncFromServer skip WebSocket broadcasts.
- Pointer fields on SaveProgressRequest: nil means preserve existing,
non-nil means overwrite. Eliminates ambiguity between zero values
and not-provided fields.
Also adds unit tests for buildProgressSnapshot helper function.
Reflowable ebooks (EPUBs) don't have inherent page numbers since layout
depends on device settings. Add an EstimatedPages() function that converts
total character count to an estimated print page count using the industry
standard of 1800 characters per page.
This provides a consistent, device-independent page count for progress
display (e.g., "Page 89 of 196" for a reflowable EPUB), matching how
KOReader and similar readers handle the same problem.
Go's new() builtin takes a type and allocates a zero value — it cannot
wrap an expression. All instances of new(someExpression) were compile
errors. Replace each with a local variable assignment and address-of
operator.
Affected files:
- handlers/koreader.go: progress field pointers (Chapter, Page, etc.)
- handlers/kobo.go: pagesRemaining pointer
- handlers/queue.go: uuidPtrToString and timestamptzPtrToString helpers
- router/reader.go: bookmark pageNumber and chapterNumber pointers
- services/media_scanner.go: validation error message pointers
- services/worker.go: StartedAt and CompletedAt timestamps
- sync/offline.go: GetDeviceStatus return pointer
- tests/device_test.go: SyncEnabled and SyncFrequencyMinutes pointers
Remove redundant type conversions that Go 1.26 makes unnecessary or that
were already no-ops:
- uuid.UUID(x.Bytes) → x.Bytes (uuid.UUID is [16]byte, same as pgtype UUID Bytes)
- pgtype.UUID{Bytes: [16]byte(u), Valid: true} → pgtype.UUID{Bytes: u, Valid: true}
- (*time.Time)(&x.Time) → &x.Time
- json.RawMessage(x) → x where x is already []byte
- []byte(stringVal) → stringVal where []byte is expected
- int()/int64()/byte() casts on values already of the target type
- Decompressor(fn) → fn (type is identical)
Handle previously ignored error returns:
- collections.go: check json.Unmarshal error in GetCollection
- conversion_service.go: check fileSize.Scan() error
- app_test.go: check app.Shutdown() error in benchmark
John O'Keefe
john-okeefe