Tonight's failures all traced to one blind spot: the converter could
only reason about text within a single block. A position at a chapter
heading sends walk-up context (heading + the paragraphs below, joined by
the plugin's block capture); a selection can span several paragraphs.
Neither shape could be verified (containment compared one block against
a multi-block quote, so the CORRECT structural landing at the heading
was rejected) nor matched by text search (it never crossed block
boundaries). The ladder then fell to the percentage rung — which labeled
its char-count guess Precision "exact" — and that confidently-wrong CFI
was stored: reading positions reopened paragraphs away from the true
spot, and a highlight echo overwrote the row's good web CFIs with a
garbage start anchor that made the highlight unpaintable ("disappeared").
Four changes, all in the forward converter and its consumers:
- Quote verification: after the structural walk lands, read the
whitespace-normalized document text forward from the landing point
(crossing block boundaries; inline spans join directly so drop-cap
splits still read as one word). A usable context must be a prefix of
that stream — which is exactly what device captures are: the text from
the position onward, or the selection between two anchors. The old
single-block containment checks remain as secondary acceptance.
- Cross-block text search: the search rung matches against the whole
document flattened in reading order, with every rune mapped back to
its source node and offset. A context spanning blocks now matches, and
the matched extent yields a true range end (EndEPUBCFI) that
highlights use as their end anchor, threaded through the facade as
CanonicalLocator.EndCFI.
- Honest labels: the percentage rung returns Precision "percentage" —
a char-count estimate must never masquerade as an exact anchor.
- Confident-only storage: progress adopts a converted locator solely at
structural/exact precision (section hrefs keep their legacy handling;
anything lower stores percentage only), and highlight conversion
returns CFIs only at structural/exact precision — a low-confidence
echo yields empty, which applyLWW coalescing turns into preservation
of the row's existing web CFIs instead of clobbering them.
Tests: walk-up context at a heading verifies structurally and lands in
the heading; a block-spanning context is found by search with a range
end landing in the following paragraph; the percentage rung is honestly
labeled; all drop-cap guards stay green.
Progress, highlights, notes, and bookmarks entered position conversion
through three different doors: progress converted inline with an
uncached converter, annotations through the facade, bookmarks not at
all (the raw xpointer was stored verbatim, cfi_position stayed empty,
and the web drawer's goToBookmark silently no-ops on cfi-less entries).
Unify on the facade (ConvertToCanonical/ConvertFromCanonical):
- annotationEpub context resolved once per push: media item + EPUB path
shared by every annotation instead of re-fetched per entry
- progress forward: the inline block becomes one facade call;
non-reflowable formats pass through unchanged, and the cached
converter stops re-parsing the book on every sync
- progress reverse: convertCFIToXPointer delegates to reverseConvertCFI,
keeping the stored percentage in play for the fallback ladder
- bookmarks (bulk progress and /sync-bookmarks): pos0 resolves
structural-only — bookmark text is a display label, never book text,
so no context is supplied; webUsableCFI stores the result only for
structural/exact epubcfi landings, discarding href/percentage results
rather than storing dead drawer links. Also records percentage_location
and origin_source on the legacy endpoint.
- percentages thread through: highlights/notes/bookmarks pass the device
percentage or the derived section percentage instead of a hardcoded 0,
so the last-resort fallback lands near the true position instead of
the document start
- extendCFIByLength end-derivation now also fires on structural starts
(it had silently stopped matching when the structural rung began
landing starts with precision 'structural' rather than 'exact')
Tests: the drop-cap xpointer through the facade with empty context (the
bookmark scenario) must land structurally, not doc-start; webUsableCFI
table covers the store/discard gate.
Drop-cap markup like <p><span>C</span>onvergence of Heaven and Earth</p>
made getTextFromXPointer return just 'C'. The text search then matched
the first 'C' in the chapter and stored doc-start (/4/2/1:0) with
'precision: exact', so the web reader reopened at the chapter start
while the percentage looked mid-chapter.
- Add convertByStructuralPath: walk the parsed CRE ElementPath against
the raw XHTML (same-tag 1-based indexing, mirroring buildCREXPointer),
map CharOffset into the target element's text, and build the CFI.
Usable device text verifies the landing; disagreement falls through
instead of storing a confident-but-wrong CFI.
- Add usableContextText guard (>=8 runes, >=2 words): single chars can
never claim an exact text-search hit in either direction
(ConvertCREToStandard and reverseByTextSearch).
- Add drop-cap regression fixtures plus usable-context unit tests.
- Verified against the real book: DocFragment[26]/p[12]/span header now
converts to epubcfi(/6/52!/4/28/2/1:0) structural both with 'C' and the
full header, and round-trips back to DocFragment[26].
Six converter tests pointed at absolute paths for 1984 and Crime and
Punishment under uploads/ — books that don't exist on most checkouts
(CI included), so the suite shipped with 5 permanently failing tests
(and a sixth passing only by accident: the percentage-fallback path
triggered by the missing file is the outcome it asserts).
A writeTestEPUB helper now builds a minimal deterministic EPUB in
t.TempDir() (zip → container.xml → OPF → 6-doc spine), so the tests
exercise the real zip/OPF/spine/document pipeline with no external
dependencies. The xpointer→CFI conversion, fragment-ID conversion,
both round-trips (bare and context-text-anchored), and the text-search
and percentage fallbacks all keep their original assertions, now
against known document content. internal/sync is green for the first
time on this machine.
Remove TestConvertHessBook and TestDebugHess (both referenced a
non-existent Hess EPUB at an absolute local path) and
TestConvertCPByTextSearch (referenced a Crime and Punishment EPUB
in the local uploads directory). These were development-time debug
tests that only worked on the author's machine.
All CFI/KEPUB conversion behavior is already covered by the proper
fixture-based tests (TestKEPUBRoundTrip, TestKEPUBConvertKEPUBToStandard,
TestKEPUBConvertWithEmElements, etc.) which use createTestEPUB and
createTestKEPUB helpers.
cfi_converter_test.go:
- TestFindTextInNode_SingleTextNode: baseline single-node match
- TestFindTextInNode_CrossEmElement: 'Vokalia and Consonantia' across
two <em> elements — verifies match returns the 'Vokalia' text node
- TestFindTextInNode_CrossStrongElement: text crossing <strong> boundary
- TestFindTextInNode_DoesNotCrossParagraphs: verifies block boundary
enforcement — text split across <p> elements is NOT matched
- TestFindTextInNode_NestedFormatting: <em><strong> nesting
- TestFindBlockParent: verifies findBlockParent walks up through inline
elements to find block-level <p>
- TestCollectInlineText: verifies text segments are collected in order
with correct content
kepub_cfi_converter_test.go:
- TestKEPUBConvertWithEmElements: previously skipped, now expects exact
precision and verifies round-trip conversion works for text spanning
<em> element boundaries
Implements ConvertCREToStandard which converts CREngine XPointers
(e.g. /body/DocFragment[6]/body/div/p[47]/text().2399) to standard
epubcfi format (e.g. epubcfi(/6/12!/4/2[id]/4/1:7)).
Key components:
- indexChildNodes: faithful port of foliate-js's epubcfi.js algorithm
for computing CFI-compatible child node indices including virtual
positions, null positions between adjacent elements, and text chunks
- preprocessXHTML: converts XHTML self-closing tags (e.g. <a id="x"/>)
to open/close pairs so Go's HTML parser produces the same DOM as the
browser's XHTML parser
- buildCFI: walks up from a text node to body, computing CFI indices
at each level using indexChildNodes
- findTextInNode: regex-based whitespace-flexible text search for
context_text fallback positioning
- convertByPercentageOffset: estimates position via book-wide character
counts when no context_text is available
- ConvertCREToStandard: orchestrates text search → percentage fallback
Supports CREngine XPointer format, CREngine fragment ID format
(#_doc_fragment_N_anchor), and includes round-trip test coverage for
1984 and Crime and Punishment EPUBs.
john-okeefe