The UI had no surface showing how many media items have been imported.
Surface the total in the library switcher shown on the Dashboard, Series,
and Collections pages (via the LibrarySwitcher component) and in the
Bookshelf's inline library filter.
- Add a MediaCount field to LibraryData and a TotalMediaCount helper to
sum counts for the "All Libraries" / "All Books" option.
- resolveLibrary() now fetches per-library counts (one query) and maps
them onto each LibraryData entry, so the switcher reflects the active
scope without changing the component's signature.
- Each library option renders "(N)" and the "All" option renders the
grand total across the user's visible libraries.
The "All" total is the sum of the user's visible libraries, correctly
respecting per-user library visibility rather than a raw global count.
Regenerated templ files for library_switcher and bookshelf.
Add GetVisibleLibraryMediaCounts, which returns the media item count for
each library visible to a given user in a single GROUP BY query over
media_items. It mirrors the visibility logic in GetUserVisibleLibraries
(libraries default to visible unless an explicit false row exists) so
counts can be resolved in one round-trip instead of N per-library
lookups.
Regenerated sqlc bindings (querier.go, queries.sql.go).
Gitea's auto GITHUB_TOKEN lacks the package scope needed to push to the container registry, causing the login step to fail. Switch the login password to a PAT stored as the REGISTRY_TOKEN repo Actions secret (scopes: write:package, read:package).
Adds .gitea/workflows/release.yml. On a v* git tag push (or manual dispatch), builds the Dockerfile and publishes to git.linuxhg.com/bookhoard/bookhoard under two tags: the version (${{ gitea.ref_name }}) and 'latest'. Auth uses the auto-provided GITHUB_TOKEN; no secret to manage. Pushes to main do nothing, so work-in-progress commits never ship.
Expose the recently-added env-driven compose settings as commented examples so self-hosters and deployers can discover them. All remain optional with defaults.
Replace hardcoded port literals with env-driven variables so a single change
in .env reconfigures the full stack consistently. Defaults are unchanged
(DB 5432, app 8765), so existing setups need no .env changes.
- DB_PORT (default 5432): drives the db host<->container port mapping,
Postgres PGPORT (so it listens on the chosen port), and the app's
DATABASE_PORT connection setting. Lets deployers avoid a host port conflict
(e.g. another local Postgres) by setting DB_PORT once.
- SERVER_PORT (default 8765): drives the app host<->container mapping, the
SERVER_PORT the app listens on, and the healthcheck target URL.
- Applied to both the base (docker-compose.yml) and the dev override
(docker-compose.dev.yml, tests service) so dev and prod stay in sync.
Address compose issues surfaced on first production deploy:
- Remove obsolete `version: "3.8"` (ignored by Compose v2; caused a warning).
- Fix BASE_URL: it used compose-time interpolation of ${SERVER_PORT}, which is
only defined as a runtime container env var (invisible to interpolation) and
absent from .env. This resolved to an empty string, producing a broken
`http://localhost:` (no port) and a startup warning. Now
${BASE_URL:-http://localhost:8765}, overridable per-deployment via .env.
- Move COOKIE_SECURE from the db service to the app service and make it
configurable (${COOKIE_SECURE:-false}). It controls the session cookie Secure
flag, an app concern; on the db service it was a no-op, so the app never
received it and cookies were always non-secure. Set COOKIE_SECURE=true behind
a TLS-terminating reverse proxy (Caddy/nginx/traefik), where the app speaks
plain HTTP internally.
- Image reference unchanged: ${IMAGE_TAG:-latest} (no hardcoded version).
Restructure the container setup to support registry-based deployment:
the default docker-compose.yml now pulls a prebuilt app image from the
Gitea container registry instead of building locally, while a new
docker-compose.dev.yml override preserves the local build + integration
test workflow for development.
Why:
- Production and self-hosting should consume a published image, not
rebuild from source on the host. The default `docker compose up` now
pulls the app image (git.linuxhg.com/bookhoard/bookhoard) alongside the
public postgres image, with no build step required.
- Development still needs to build from source and run integration
tests, so those concerns move to an override file the Makefile applies.
Shared config (env, volumes, ports, healthchecks) lives in one place to
avoid drift between environments.
Changes:
- docker-compose.yml (prod base): the app service now references
`image: git.linuxhg.com/bookhoard/bookhoard:${IMAGE_TAG:-latest}` instead
of a build context. The tests service is removed (moved to the
override). IMAGE_TAG lets deployers pin or roll back a specific version.
- docker-compose.dev.yml (new override): adds the local `build:` context
for the app and defines the integration `tests` service (profile-gated).
Everything else is inherited from the base file via compose merging.
- Makefile: introduce a COMPOSE variable that merges the base and
override (`-f docker-compose.yml -f docker-compose.dev.yml`); all dev
targets now use it. Plain `docker compose` against the base file only
remains the production path.
- README: quickstart updated to pull and start prebuilt images; clone URL
points at the Gitea instance.
The development workflow (`make rebuild-app`, `make test-integration`,
etc.) is functionally unchanged.
Complete the annotation sync pipeline across all ingest and serve paths.
Previously, annotations sent inline with KOReader progress pushes were
silently discarded, and no annotations were ever served back to devices.
INGEST (device → server):
KOReader (koreader.go):
- Add processBookAnnotations helper that processes inline highlights,
notes, and bookmarks from every progress push (immediate + checkpoint)
- Highlights get CRE→CFI position conversion before SaveHighlight
- KOReader 'notes' (text + notes) stored as highlights with NoteText
to ensure correct round-trip classification
- Bookmarks routed through SaveBookmark with device sync data
- Called from both updateProgressForBook and handleCheckpointSync
Kobo (kobo.go):
- Markup handler: annotations and bookmarks route through
AnnotationService (SaveHighlight/SaveBookmark)
- Bookmark handler: same routing with device sync data
- SyncFromServer handler: same routing
- All handlers fall back to direct DB calls when annotationSvc == nil
Web reader (media.go):
- CreateMediaHighlight → SaveHighlight (Source="web", ModifiedAt=now)
- CreateMediaNote → SaveNote (Source="web")
- DeleteMediaHighlight → TombstoneHighlightByID
- DeleteMediaNote → TombstoneNoteByID (was hard delete, now tombstone)
- All fall back to old behavior when annotationSvc == nil
SERVE (server → device):
KOReader GetMetadata (koreader.go):
- Query and serve bookmarks from media_bookmarks table (was missing)
- Serve deleted_highlights and deleted_bookmarks arrays containing
device_sync_data + dedup_key for client-side deletion
- Highlights/notes already served with reverse CFI conversion
Kobo Markup handler (kobo.go):
- Track processed books during sync
- Query tombstones per book, extract bookmark_id from device_sync_data
- Return DeletedAnnotations array in KoboSyncStatus response
Conflict resolution (conflicts.go):
- Enable annotation conflict types in ResolveConflict handler
- Add applyAnnotationResolution dispatching to:
applyHighlightResolution / applyBookmarkResolution / applyNoteResolution
- Each looks up by dedup_key and applies winner's fields
- Allow manual override of auto_resolved conflicts
(changed check from != "unresolved" to == "user_resolved")
Infrastructure:
- AnnotationService field + SetAnnotationService in router Config
- Inject AnnotationService into KOReader, Kobo, Media handlers
- Start tombstone purger goroutine in main.go (24h interval)
- Test helpers: construct AnnotationService in test setup
Wire AnnotationService into SyncQueueProcessor and implement the three
previously-stubbed execute methods:
- syncHighlight: unmarshals syncData JSON into SaveHighlightRequest,
applies CRE→CFI conversion via AnnotationService
- syncNote: unmarshals into SaveNoteRequest
- syncBookmark: unmarshals into SaveBookmarkRequest
- Add SyncTypeBookmark to executeSync switch (was hitting default error)
Add enqueue methods for future offline/batch use:
- EnqueueHighlight / EnqueueNote / EnqueueBookmark
- Shared enqueueAnnotation helper creates queue items with
PriorityCriticalNote and 3 max attempts
- Update types (HighlightUpdate, NoteUpdate, BookmarkUpdate) mirror the
existing ProgressUpdate pattern
Existing handler behavior is unchanged — annotations still sync
synchronously via AnnotationService. The queue path is available for
retry-on-failure and offline batch processing scenarios.
AnnotationService is the central service for cross-device annotation sync.
It provides SaveHighlight, SaveNote, and SaveBookmark methods that handle
the full sync lifecycle:
Identity (3-layer):
1. Server UUID (primary key)
2. Per-device native ID stored in device_sync_data JSONB
3. Content dedup_key: sha1(normalize(selection_text) + bucket_position)
- CFI character offsets are stripped for bucketing so the same
highlight at slightly different offsets still deduplicates
- Raw positions are preserved in the DB for precise restoration
Resolution policy (LWW):
- When the incoming annotation has an explicit ModifiedAt timestamp,
last_modified_at wins
- When the device sends zero ModifiedAt (creation time only), field-diff
mode compares content fields (text/color/note/percentage) — if all
match, the save is skipped; if any differ, the save is applied with
server-receive-time as the new last_modified_at
Conflict detection:
- When incoming and existing annotations have different sources (e.g.
koreader vs kobo) and content differs, an auto_resolved sync_conflict
is recorded with both sides' data for audit trail
- Broadcasts a WebSocket conflict notification for real-time UI updates
Tombstone management:
- Delete-wins: tombstoned annotations block recreation from stale pushes
- 30-day TTL before physical purge
- PurgeExpiredTombstones method + StartTombstonePurger goroutine (24h ticker)
Add locators.go with unified bidirectional CFI conversion:
ConvertToCanonical / ConvertFromCanonical
- CRE XPointer <-> standard EPUB CFI (for KOReader)
- KEPUB CFI passthrough (for Kobo)
- Skips non-reflowable formats (PDF, CBZ, fixed-layout EPUBs)
Add 25 unit tests covering:
- Dedup key determinism, text normalization, position sensitivity
- Offset insensitivity (CFI char-offset bucketing)
- Device sync data merge (preserves existing, overwrites same source)
- Cross-source detection
- LWW comparison (newer wins, older skipped, fallback to updated_at)
- Field-diff mode (identical content skipped, changes applied)
- Tombstone TTL constant
- CRE XPointer parsing and classification
- Standard EPUB CFI classification
Add migration columns to media_highlights, media_notes, and media_bookmarks
for cross-device annotation sync:
- dedup_key: SHA-1 of normalized selection text + bucketed position, used
as the stable cross-device identity for annotations
- last_modified_at / last_modified_source: edit clock for LWW resolution
and cross-source conflict detection
- deleted / deleted_at: sticky tombstone columns for delete-wins semantics
with a 30-day TTL before rows are physically purged
- note_text on highlights: stores attached notes from KOReader entries that
have both selected text and a user note
- Location columns on bookmarks (cfi_position, percentage_location,
epubcfi_location, chapter_reference, paragraph_reference)
- device_sync_data JSONB on all three tables: stores per-device native
identifiers (e.g. KOReader pos0/datetime, Kobo bookmark_id) so each
device can locate and manipulate its own copy of an annotation
Add partial unique indexes on (user_id, media_item_id, dedup_key) where
deleted = FALSE to enforce one active annotation per dedup key.
Add tombstone purge indexes on (deleted, deleted_at) for efficient GC.
New queries:
- GetByDedupKey for all three tables (returns active or most-recent tombstone)
- CreateFull / UpdateForSync for all three tables (populate sync columns)
- TombstoneByDedupKey / TombstoneByID for all three tables
- PurgeExpired* for all three tables (GC past TTL)
- GetActiveAnnotationsForBook (filtered union of highlights + notes)
- GetTombstonedAnnotationsForBook (union of all 3 deleted within TTL)
- GetMediaBookmarks with deleted filter
- GetMediaBookmark (singular) with deleted filter
- Added deleted=FALSE filter to GetMediaHighlights, GetAnnotationsForBook
- CreateAutoResolvedSyncConflict (INSERT with resolution_status='auto_resolved')
Quote DATABASE_PORT and SERVER_PORT ("5432", "8765") in docker-compose.yml so they are treated as strings rather than YAML integers, avoiding type-coercion warnings from compose runtimes.
Setup completion was previously tracked by a manually-flipped setup_complete row in system_settings, written via a JWT-protected PUT /api/setup/complete endpoint. This meant any admin user created outside the setup wizard (future CLI, seed scripts, direct DB inserts) would not flip the switch, leaving the app stuck redirecting to /setup.
The trigger is now derived from real data: setup is complete iff at least one admin user exists. This is self-correcting regardless of how users are created, and re-engages setup automatically if all admins are ever removed.
Changes:
- Add internal/setupstatus package with IsSetupComplete() (queries CountAdmins, 10s in-memory cache, fails open on DB error) and Invalidate() to clear the cache. Uses an AdminCounter interface to avoid importing the database package.
- Add CountAdmins sqlc query (SELECT COUNT(*) FROM users WHERE role = 'admin') and regenerate.
- Rewire router/setup.go isSetupComplete() to delegate to setupstatus; drop the old setup_complete setting read, cache vars, and the PUT /api/setup/complete route.
- Call setupstatus.Invalidate() in the auth handler after CreateUser, UpdateUserRole, and DeleteUser so the cache reflects admin-count changes immediately.
- Align first-user promotion in Register to key off !adminExists instead of len(users) == 0, so the two checks cannot diverge.
- Remove the now-dead SetSetupComplete/GetSetupStatus handlers.
- Drop the setup_complete seed row from schema.sql.
- Remove the apiPut('/setup/complete') call from the setup wizard finishSetup(); the admin account created in submitAdmin already marks setup complete server-side.
The btn-secondary class was used 39 times across 15 templates but had
no definition in any global CSS file. The only definition existed in
error.templ's inline styles (intentionally self-contained).
Added .btn-secondary and .btn-secondary:hover to input.css using
theme-aware CSS variables (--accent) consistent with the existing
.btn-primary pattern. Rebuilt style.css via Tailwind.
The Manga Type and Reading Direction select options used invalid inline
if syntax: 'if cond { selected }' which is not valid templ. Replaced
with the correct templ attribute syntax: selected={ boolExpression }
which properly renders the selected attribute when true and omits it
when false.
Also add for/id attributes to link labels to their select elements
for accessibility (manga_type and reading_direction).
Affected lines: book_detail_modals.templ:478-496
Remove TestConvertHessBook and TestDebugHess (both referenced a
non-existent Hess EPUB at an absolute local path) and
TestConvertCPByTextSearch (referenced a Crime and Punishment EPUB
in the local uploads directory). These were development-time debug
tests that only worked on the author's machine.
All CFI/KEPUB conversion behavior is already covered by the proper
fixture-based tests (TestKEPUBRoundTrip, TestKEPUBConvertKEPUBToStandard,
TestKEPUBConvertWithEmElements, etc.) which use createTestEPUB and
createTestKEPUB helpers.
When a Kobo device pushes a last-read-place bookmark, the server now
converts the KEPUB CFI (with koboSpan wrappers) to a standard EPUB CFI
and extracts surrounding text as context_text for use by other devices
(KOReader, web reader) during their pull-side CFI conversions.
Previously the raw KEPUB CFI was stored verbatim as epubcfi, which
meant foliate and CREngine couldn't resolve it (wrong child indices
due to koboSpan wrappers), and no context_text was available for the
text-search fallback in ConvertStandardToCRE.
Changes:
- kepub_cfi_converter.go: Add ExtractedContext field to
KEPUBConversionResult, populated from the already-computed
searchText in both ConvertKEPUBCFIToStandard and
ConvertStandardCFIToKEPUB (exact-match and percentage-fallback
paths).
- kobo.go: Add libraryService field and SetLibraryService setter
(mirrors KOReaderHandler pattern). Add convertKoboCFIToStandard
helper that resolves EPUB+KEPUB paths, instantiates the converter,
and returns the converted CFI + extracted context. The last-read-place
branch in Markup now calls this helper for reflowable formats,
skipping fixed-layout/comic archives (page-index only).
- router.go: Add LibraryService to router Config.
- sync.go: Wire LibraryService to KoboHandler via SetLibraryService.
- main.go: Pass libraryService through router config.
The conversion is purely additive — if no KEPUB file exists on disk
(e.g. side-loaded EPUB without kepubify conversion), the handler
gracefully skips conversion and stores the raw CFI as before.
The web reader now captures ~100 chars of visible text from the
foliate relocate event's range and includes it as context_text in
the progress PUT body for reflowable formats.
This enables the server's reverseByTextSearch fallback in
ConvertStandardToCRE, which is critical for single-file EPUBs
(e.g. 1984) where foliate emits coarse or fake-section CFIs that
CREngine cannot directly resolve. Previously only the KOReader
plugin sent context_text; the web reader's omission left the
fallback unusable, causing percentage-based position estimation
that was off by ~1 page.
Changes:
- Add contextText state property (line 387)
- Extract visible text from e.detail.range in relocate handler,
normalize whitespace, and slice to 100 chars (lines 508-512)
- Include context_text in saveProgress PUT body for reflowable
EPUBs only, alongside epubcfi (line 575)
The server-side chain was already wired: media.go accepts it,
progress.go stores it, and koreader.go passes it to the CFI
converter. No Go changes needed.
The web reader's saveProgress always sent epubcfi: cfi || "", even
for fixed-layout comics/PDFs. Foliate's comic renderer generates a
fake CFI (epubcfi(/6/{n})) for every page via CFI.fake.fromIndex,
which the server stored as a valid locator. These fake CFIs are
meaningless — the page index is the canonical locator for image-based
content — and they caused koreader to crash on pull (see prior commit).
Only set epubcfi in the request body for reflowable documents.
The push path (koreader→server) was already updated to omit epubcfi
for paging documents, and SaveProgress derives percentage from
page/total_pages for fixed formats. However, the pull path
(server→koreader) still returned any stored epubcfi and attempted
CFI→XPointer conversion, and SaveProgress preserved stale CFI values
written by the web reader (which generates fake CFIs via
CFI.fake.fromIndex for every comic/PDF page).
These fake CFI strings lingered in the database and koreader's pull
path picked them up over progress.page, causing tonumber("epubcfi(...)")
→ nil → GotoPage(nil) → crash when the user confirmed the sync prompt.
Changes:
- GetMetadata (koreader.go): for fixed_layout/comic_archive formats,
skip returning epubcfi and skip the CFI→CRE XPointer conversion.
Reflowable documents are byte-for-byte unchanged.
- SaveProgress (progress.go): for fixed formats, explicitly clear
epubcfi and character_offset on every save so stale values from
prior web-reader sessions are cleaned up over time.
The CRE->CFI converter works by text search / character-offset mapping
across the EPUB spine. Image-based fixed content (fixed-layout comic
EPUBs, PDFs, comic archives) has no extractable text, so the conversion
can never succeed and only wastes time parsing content docs while
logging a failed percentage-precision result.
Guard the conversion in the KOReader progress handler: when the matched
media item's FormatGroup is fixed_layout or comic_archive, skip
ConvertCREToStandard entirely. The incoming xpointer is left as-is so
KOReader<->KOReader restore via GotoXPointer still works; the web reader
restores by page index (the canonical locator for these formats).
This covers fixed-layout comic EPUBs in particular: KOReader routes all
EPUBs through CREngine (has_pages == false), so they send a real
xpointer that passes IsCREXPointer and would otherwise trigger the
doomed text extraction. Reflowable EPUBs (format_group == reflowable)
still run the conversion exactly as before.
foliate's goLeft/goRight (and spread ordering) swap on book.dir ===
"rtl", but makeComicBook never sets dir, so manga/comic archives
always paged left-to-right even when the metadata says right-to-left.
For fixed-layout content, set this.book.dir = "rtl" from the
readingDirection metadata after the book opens. Reflowable EPUBs are
unaffected: they keep whatever direction foliate read from the OPF.
This is scoped to isFixedLayout (FXL EPUB, PDF, comics) so it cannot
reach the reflowable path.
Fixed-layout EPUBs, PDFs, DjVu, and comic archives (cbz/cbr/cb7/cbt)
are page-based: each page is a fixed image, so a page index is an exact,
universal locator regardless of screen size or device. Sync previously
treated these like reflowable content (CFI-first restore, percentage
fallback, character-offset math), which was both wrong and lossy. This
makes the page index the canonical position for fixed-layout and comic
formats while leaving the reflowable path byte-for-byte unchanged.
Backend:
- progress.go SaveProgress: branch on mediaItem.FormatGroup. For
fixed_layout/comic_archive derive percentage from current_page/
total_pages and skip the CFI/character-offset back-fills (meaningless
for image content). The reflowable derivation block is preserved
verbatim under an else.
- kobo.go: Kobo only sends a percentage, so for fixed-layout/comic
formats derive CurrentPage via PercentageToPage(percentage, pageCount)
using the media item's known page count, so Kobo->web lands on the
exact page.
Reader:
- reader.templ readerInitExpr: pass savedPage/savedTotalPages to the
reader config for fixed_layout/comic_archive formats.
- reader.ts: when isFixedLayout and savedPage is present, restore via
view.init({ lastLocation: savedPage - 1 }) (a bare number navigates
foliate directly to the section index). Reflowable falls through to
the existing CFI->percentage path, unchanged.
The OPDS device catalog previously hard-coded every acquisition link as
application/epub+zip and always offered kepub/pdf alternate links, which
is wrong for comic archives (cbz/cbr/cb7/cbt) and other non-epub media.
- Resolve the acquisition mime type from the media item's stored
mime_type (falling back to format_mimetype, then epub) instead of
assuming epub
- Only offer reflowable conversions (kepub for kobo, pdf) for ebooks;
comic archives are served as-is in their native format
- Derive the native format label (epub/pdf/cbz/cbr/...) from the file
path in ListFormats rather than always reporting epub
- Add resolveMimeType, isComicArchive, and formatLabelFromPath helpers
Add a Server URL field to the admin-account step of the setup wizard so
the public base URL (used for device sync, OPDS feed, and API endpoints)
is configured up front instead of requiring a later visit to admin
settings.
- setup.templ: base URL input on the admin step plus a summary entry
- setup.ts: default baseUrl to window.location.origin and persist it via
PUT /system/config ({ base_url }) right after the admin account is
created, with a non-fatal warning if the save fails
Add a single-page multi-step setup wizard that guides new users
through initial configuration:
Step 1 - Admin Registration: Creates the first user (auto-admin)
using the existing POST /api/auth/register endpoint, with
real-time password validation and confirmation matching.
Step 2 - Library Creation: Create one or more libraries (Ebooks,
Audiobooks, Comics, Manga) using POST /api/libraries. Libraries
list updates inline as they're added.
Step 3 - Folder Configuration: Add filesystem folders to each
library using the existing GET /api/libraries/browse endpoint for
a visual directory browser. Folders are attached via POST
/api/libraries/:id/folders.
Step 4 - Initial Scan: Triggers a manual scan of all libraries via
POST /api/libraries/scan with real-time progress polling using the
existing scan status endpoint.
On completion, the wizard calls PUT /api/setup/complete and sets the
selectedLibrary cookie to the first library's UUID, ensuring the
dashboard loads with populated content instead of an empty 'All
Libraries' view. Handles the edge case where a stale JWT from a
previous database instance triggers an 'already exists' error by
auto-advancing to step 2.
The wizard reuses all existing API calls, Alpine.js utilities, and
form validation functions — no backend logic was duplicated.
Add setupRedirectMiddleware that checks the setup_complete system
setting on every request. If setup is incomplete, all non-setup
requests are redirected to /setup so the wizard is the first thing
new users see. The check uses an in-memory cache (10s TTL) to avoid
hitting the database on every request, with cache invalidation on
setup completion.
The middleware skips /setup, /api/*, /static/*, /health, and
/favicon.ico so the wizard page, API calls, and static assets load
normally during setup.
Register two new routes:
- GET /setup: renders the setup wizard SSR template
- PUT /api/setup/complete: marks setup as complete (JWT-protected,
requires an authenticated admin user created in step 1)
Add 'setup_complete' boolean to the system_settings seed data (defaults
to false) so fresh databases start in the unconfigured state.
Add two new handlers to SystemSettingsHandler:
- SetSetupComplete: marks setup_complete=true in the database
- GetSetupStatus: reads the current setup_complete value, returns
{setup_complete: bool} JSON response, defaults to false if the
setting row is missing or unparseable
cfi_converter_test.go:
- TestFindTextInNode_SingleTextNode: baseline single-node match
- TestFindTextInNode_CrossEmElement: 'Vokalia and Consonantia' across
two <em> elements — verifies match returns the 'Vokalia' text node
- TestFindTextInNode_CrossStrongElement: text crossing <strong> boundary
- TestFindTextInNode_DoesNotCrossParagraphs: verifies block boundary
enforcement — text split across <p> elements is NOT matched
- TestFindTextInNode_NestedFormatting: <em><strong> nesting
- TestFindBlockParent: verifies findBlockParent walks up through inline
elements to find block-level <p>
- TestCollectInlineText: verifies text segments are collected in order
with correct content
kepub_cfi_converter_test.go:
- TestKEPUBConvertWithEmElements: previously skipped, now expects exact
precision and verifies round-trip conversion works for text spanning
<em> element boundaries
Instead of reading only from the single resolved text node (which
produces a tiny window when the position is inside <em> or other inline
elements), extractSurroundingText now:
1. Finds the block-level parent (e.g. <p>) of the resolved text node
2. Collects all text within that block, transparently crossing inline
formatting elements via collectInlineText
3. Computes the global offset of the original text node within the
concatenated block text
4. Extracts the [offset-window : offset+window] slice
This gives a full context window regardless of inline element
boundaries, enabling accurate text bridging between EPUB and KEPUB
documents even when reading positions fall inside <em>, <strong>,
<span class="koboSpan">, etc.
Falls back to single-node extraction when no block parent is found
(e.g. orphan text nodes in tests).
When finding or extracting text in EPUB/KEPUB DOM trees, inline
formatting elements like <em>, <strong>, <i>, <b>, <span>, etc. should
not break text continuity. A reader sees 'Vokalia and Consonantia' as
one phrase regardless of the <em> wrappers around each word.
Add inline formatting element set and helper functions:
- isInlineFormatting: checks if an element is an inline phrasing element
- collectInlineText: flattens text across formatting elements within
a block-level parent, returning segments that map back to original
text nodes
- findBlockParent: walks up from a text node to find the nearest
block-level ancestor (used to scope text collection)
- findTextAcrossInlineElements: fallback for findTextInNode that
concatenates text within each block element (transparently crossing
formatting elements) and maps match positions back to actual nodes
- collectBlockElements: gathers all block-level elements containing text
The key invariant: text collection NEVER crosses block-level element
boundaries (<p>, <div>, <h1>-<h6>, <li>, etc.) to avoid concatenating
text from different paragraphs.
The findTextInNode function now tries single-text-node matching first
(fast path, unchanged), then falls back to cross-element matching only
when needed. This preserves performance for the common case.
Test coverage for KEPUBCFIConverter with programmatically generated
EPUB and KEPUB zip fixtures:
- KEPUB->Standard text search conversion (exact precision)
- Standard->KEPUB text search conversion with round-trip verification
- Multi-position round-trip (3 phrases across different paragraphs)
- Cross-chapter conversion (spine index 1)
- No-context-text conversion (extracts surrounding text from resolved node)
- Invalid CFI handling (falls back to percentage)
- Percentage fallback for unresolvable CFIs
- 5-paragraph round-trip covering different document positions
- CFI structural difference verification (koboSpan adds DOM steps)
- Spine index consistency between EPUB and KEPUB
- Real book conversion test (skips if file not present)
- extractSurroundingText unit tests
Add KEPUBCFIConverter in internal/sync that converts between KEPUB CFIs
(which include extra koboSpan DOM steps) and standard EPUB CFIs at sync
time, so only standard epubcfi values are stored in the database.
The converter works by:
1. Resolving the source CFI in the source document (EPUB or KEPUB)
2. Extracting surrounding text at the resolved position
3. Searching for that same text in the target document
4. Building a new CFI pointing to the matched text in the target
This text-content bridging handles the DOM structural differences
between EPUB (text nodes at depth 2) and KEPUB (text nodes wrapped in
<span class="koboSpan"> at depth 3).
Falls back to percentage-based positioning when text search fails,
matching the pattern used by the existing CRE converter.
No changes to existing cfi_converter.go or KOReader conversion code.
Same-package access to unexported functions (resolveCFIToNode, buildCFI,
findTextInNode, etc.) via internal/sync package placement.
Templ requires Go expression interpolation for dynamic attributes,
not string embedding. Change from @click="func('{ id }')" to
@click={ "func('" + id + "')" } for device ID and registration ID
buttons.
Replace the 'Go to Conflicts Page' link with inline conflict resolution.
Each conflict source now has a 'Keep This' button that resolves the
conflict directly from the book detail page.
- Conflict data now keyed by source name (koreader, web) instead of
new/existing, with Source and Timestamp fields
- Display percentage scaled correctly (* 100)
- Fix page field name from current_page to page
- Add conflict resolution JavaScript in book-detail.ts
- Add 10-minute cooldown after resolution to prevent re-detection
- reader.templ: divide DB percentage (0-1) by 100 was wrong; actually
the router multiplies by 100, so template now divides by 100 to get
back to 0-1 range for foliate-js savedPercentage
- reader.ts: add 5-second initTime guard to prevent overwriting
existing progress with initial position on page load
- UpdateSystemConfiguration: when base_url changes, automatically
update opds_base_url and api_base_url derived configs
- convertPending: format time.Time as RFC3339 string instead of
relying on string type assertion which would panic
- Add <title> and <author><name> elements to Atom feed for compatibility
- Remove doubled /opds/opds/ path prefix in feed URLs
- Include ?token= auth param on all OPDS URLs
- Serve kepub links only for Kobo devices
- Fix URL construction for entries and acquisitions
Previously device creation happened in CheckRegistrationStatus (polling
endpoint), which was racy. Now the admin's ApproveDevice handler
creates the device record and stores auth token + device ID on the
registration entry. CheckRegistrationStatus just returns the pre-created
credentials.
Also adds approved/authToken/deviceID/syncEndpoints fields to
PendingRegistration struct.
When resolving a conflict, the last_sync_source is now set to the
winner's actual source name (koreader, web, etc.) rather than always
'manual'. This prevents subsequent saves from re-triggering conflicts.
Also removes the strict oneof validation on the winner field since
the source name is dynamic.
Web reader can now send surrounding text at current reading position.
Stored in reading_progress.context_text for use as CFI resolution
fallback when converting epubcfi to CREngine XPointer.
Forward (push): When KOReader pushes a CREngine XPointer, convert it
to standard epubcfi before storing. Uses CFIConverter.ConvertCREToStandard
with context_text for text search fallback.
Reverse (pull): When KOReader pulls progress, convert stored standard
epubcfi back to CREngine XPointer via CFIConverter.ConvertStandardToCRE.
Returns as koreader_xpointer field in GetMetadata response.
Other changes:
- updateProgressForBook: pass context_text to SaveProgress
- enqueueProgressForBook: pass context_text to queue
- KOReaderProgressData: add KoreaderXPointer field
- New convertCFIToXPointer helper method
- Wire libraryService in main.go for EPUB path resolution
- SaveProgressRequest: add ContextText field
- SaveProgress: carry existing ContextText from DB, overwrite when provided
- ProgressUpdate: add ContextText field for checkpoint sync
- SyncQueueProcessor: serialize/deserialize context_text in sync data
- buildProgressSnapshot: include context_text in snapshot data
Stores surrounding text (~100 chars) at the reader's current position.
Used as fallback for CFI resolution when converting between epubcfi
and CREngine XPointer formats.
Updates:
- schema.sql: add context_text TEXT column, update stored procedure
- queries.sql: add context_text to GetUniversalProgress and
UpdateUniversalProgress queries
- Regenerate sqlc Go code (models.go, querier.go, queries.sql.go)
Implements ConvertCREToStandard which converts CREngine XPointers
(e.g. /body/DocFragment[6]/body/div/p[47]/text().2399) to standard
epubcfi format (e.g. epubcfi(/6/12!/4/2[id]/4/1:7)).
Key components:
- indexChildNodes: faithful port of foliate-js's epubcfi.js algorithm
for computing CFI-compatible child node indices including virtual
positions, null positions between adjacent elements, and text chunks
- preprocessXHTML: converts XHTML self-closing tags (e.g. <a id="x"/>)
to open/close pairs so Go's HTML parser produces the same DOM as the
browser's XHTML parser
- buildCFI: walks up from a text node to body, computing CFI indices
at each level using indexChildNodes
- findTextInNode: regex-based whitespace-flexible text search for
context_text fallback positioning
- convertByPercentageOffset: estimates position via book-wide character
counts when no context_text is available
- ConvertCREToStandard: orchestrates text search → percentage fallback
Supports CREngine XPointer format, CREngine fragment ID format
(#_doc_fragment_N_anchor), and includes round-trip test coverage for
1984 and Crime and Punishment EPUBs.
Previously, when HTMX form submissions (profile update, password change)
returned a successful JSON response like {"message": "profile updated
successfully"}, the raw JSON was swapped into the target div as plain text.
The htmx:afterSwap listener in toast.ts only handled error responses.
Extend it to also intercept successful 2xx JSON responses that contain a
"message" field, showing a green success toast and clearing the raw JSON
from the target element. Only JSON responses are intercepted (checked via
Content-Type header), so legitimate HTML swaps are unaffected.
The timezone feature has been fully implemented across the codebase
(database queries, API handlers, profile form, reader settings).
This planning document is no longer needed.
john-okeefe