Complete the annotation sync pipeline across all ingest and serve paths.
Previously, annotations sent inline with KOReader progress pushes were
silently discarded, and no annotations were ever served back to devices.
INGEST (device → server):
KOReader (koreader.go):
- Add processBookAnnotations helper that processes inline highlights,
notes, and bookmarks from every progress push (immediate + checkpoint)
- Highlights get CRE→CFI position conversion before SaveHighlight
- KOReader 'notes' (text + notes) stored as highlights with NoteText
to ensure correct round-trip classification
- Bookmarks routed through SaveBookmark with device sync data
- Called from both updateProgressForBook and handleCheckpointSync
Kobo (kobo.go):
- Markup handler: annotations and bookmarks route through
AnnotationService (SaveHighlight/SaveBookmark)
- Bookmark handler: same routing with device sync data
- SyncFromServer handler: same routing
- All handlers fall back to direct DB calls when annotationSvc == nil
Web reader (media.go):
- CreateMediaHighlight → SaveHighlight (Source="web", ModifiedAt=now)
- CreateMediaNote → SaveNote (Source="web")
- DeleteMediaHighlight → TombstoneHighlightByID
- DeleteMediaNote → TombstoneNoteByID (was hard delete, now tombstone)
- All fall back to old behavior when annotationSvc == nil
SERVE (server → device):
KOReader GetMetadata (koreader.go):
- Query and serve bookmarks from media_bookmarks table (was missing)
- Serve deleted_highlights and deleted_bookmarks arrays containing
device_sync_data + dedup_key for client-side deletion
- Highlights/notes already served with reverse CFI conversion
Kobo Markup handler (kobo.go):
- Track processed books during sync
- Query tombstones per book, extract bookmark_id from device_sync_data
- Return DeletedAnnotations array in KoboSyncStatus response
Conflict resolution (conflicts.go):
- Enable annotation conflict types in ResolveConflict handler
- Add applyAnnotationResolution dispatching to:
applyHighlightResolution / applyBookmarkResolution / applyNoteResolution
- Each looks up by dedup_key and applies winner's fields
- Allow manual override of auto_resolved conflicts
(changed check from != "unresolved" to == "user_resolved")
Infrastructure:
- AnnotationService field + SetAnnotationService in router Config
- Inject AnnotationService into KOReader, Kobo, Media handlers
- Start tombstone purger goroutine in main.go (24h interval)
- Test helpers: construct AnnotationService in test setup
When a Kobo device pushes a last-read-place bookmark, the server now
converts the KEPUB CFI (with koboSpan wrappers) to a standard EPUB CFI
and extracts surrounding text as context_text for use by other devices
(KOReader, web reader) during their pull-side CFI conversions.
Previously the raw KEPUB CFI was stored verbatim as epubcfi, which
meant foliate and CREngine couldn't resolve it (wrong child indices
due to koboSpan wrappers), and no context_text was available for the
text-search fallback in ConvertStandardToCRE.
Changes:
- kepub_cfi_converter.go: Add ExtractedContext field to
KEPUBConversionResult, populated from the already-computed
searchText in both ConvertKEPUBCFIToStandard and
ConvertStandardCFIToKEPUB (exact-match and percentage-fallback
paths).
- kobo.go: Add libraryService field and SetLibraryService setter
(mirrors KOReaderHandler pattern). Add convertKoboCFIToStandard
helper that resolves EPUB+KEPUB paths, instantiates the converter,
and returns the converted CFI + extracted context. The last-read-place
branch in Markup now calls this helper for reflowable formats,
skipping fixed-layout/comic archives (page-index only).
- router.go: Add LibraryService to router Config.
- sync.go: Wire LibraryService to KoboHandler via SetLibraryService.
- main.go: Pass libraryService through router config.
The conversion is purely additive — if no KEPUB file exists on disk
(e.g. side-loaded EPUB without kepubify conversion), the handler
gracefully skips conversion and stores the raw CFI as before.
Fixed-layout EPUBs, PDFs, DjVu, and comic archives (cbz/cbr/cb7/cbt)
are page-based: each page is a fixed image, so a page index is an exact,
universal locator regardless of screen size or device. Sync previously
treated these like reflowable content (CFI-first restore, percentage
fallback, character-offset math), which was both wrong and lossy. This
makes the page index the canonical position for fixed-layout and comic
formats while leaving the reflowable path byte-for-byte unchanged.
Backend:
- progress.go SaveProgress: branch on mediaItem.FormatGroup. For
fixed_layout/comic_archive derive percentage from current_page/
total_pages and skip the CFI/character-offset back-fills (meaningless
for image content). The reflowable derivation block is preserved
verbatim under an else.
- kobo.go: Kobo only sends a percentage, so for fixed-layout/comic
formats derive CurrentPage via PercentageToPage(percentage, pageCount)
using the media item's known page count, so Kobo->web lands on the
exact page.
Reader:
- reader.templ readerInitExpr: pass savedPage/savedTotalPages to the
reader config for fixed_layout/comic_archive formats.
- reader.ts: when isFixedLayout and savedPage is present, restore via
view.init({ lastLocation: savedPage - 1 }) (a bare number navigates
foliate directly to the section index). Reflowable falls through to
the existing CFI->percentage path, unchanged.
All four progress write paths now delegate to ProgressService.SaveProgress:
- MediaHandler: UpdateMediaReadingProgress uses ProgressService for web
saves with richer request body (reading_mode, zoom_level, scroll). GET
now uses GetUniversalProgress query that JOINs media_items for
format_group, total_characters, chapter_count.
- KOReaderHandler: updateProgressForBook delegates to ProgressService.
Fixed device ID bug (was using userID, now uses deviceID). Removed
duplicate UpdateDeviceLastSync with zero UUID. Added pgtype helper
functions (textPtrToPgText, intPtrToPgInt4, int64PtrToPgInt8).
- KoboHandler: all four progress write points (Markup ReadingSync, Markup
last-read-place, AnalyticsGettests, SyncFromServer) delegate to
ProgressService. Fixed empty epubcfi string now correctly set to
Valid: false. SyncFromServer preserves last_sync_source=bookhoard
and Broadcast: false.
- QueueProcessor: syncProgress delegates to ProgressService.
- main.go: creates ProgressService after ConnectionManager, injects via
SetProgressService() on all handlers and queue processor.
Handler tests cover pgtype conversion helpers (textPtrToPgText, etc.)
and device icon mapping.
Go's new() builtin takes a type and allocates a zero value — it cannot
wrap an expression. All instances of new(someExpression) were compile
errors. Replace each with a local variable assignment and address-of
operator.
Affected files:
- handlers/koreader.go: progress field pointers (Chapter, Page, etc.)
- handlers/kobo.go: pagesRemaining pointer
- handlers/queue.go: uuidPtrToString and timestamptzPtrToString helpers
- router/reader.go: bookmark pageNumber and chapterNumber pointers
- services/media_scanner.go: validation error message pointers
- services/worker.go: StartedAt and CompletedAt timestamps
- sync/offline.go: GetDeviceStatus return pointer
- tests/device_test.go: SyncEnabled and SyncFrequencyMinutes pointers
Remove redundant type conversions that Go 1.26 makes unnecessary or that
were already no-ops:
- uuid.UUID(x.Bytes) → x.Bytes (uuid.UUID is [16]byte, same as pgtype UUID Bytes)
- pgtype.UUID{Bytes: [16]byte(u), Valid: true} → pgtype.UUID{Bytes: u, Valid: true}
- (*time.Time)(&x.Time) → &x.Time
- json.RawMessage(x) → x where x is already []byte
- []byte(stringVal) → stringVal where []byte is expected
- int()/int64()/byte() casts on values already of the target type
- Decompressor(fn) → fn (type is identical)
Handle previously ignored error returns:
- collections.go: check json.Unmarshal error in GetCollection
- conversion_service.go: check fileSize.Scan() error
- app_test.go: check app.Shutdown() error in benchmark
Simplify pointer creation across kobo, koreader, and queue handlers by
replacing the two-step pattern (assign to local, then take address) with
inline new() calls. This reduces verbosity without changing behavior:
Before:
remaining := int(a - b)
pagesRemaining = &remaining
After:
pagesRemaining = new(int(a - b))
Covers page calculations, chapter/progress fields, UUID formatting,
and timestamp string conversions.
- Add nil UUID checks after mapContentIdToBookhoardUUID in all handlers
- Add ContentType detection for Kobo EPUB/PDF sync (EPUB=6, PDF=5)
- Add "last-read-place" bookmark type support with EPUB CFI position tracking
- Restore broken mapContentIdToBookhoardUUID function with UUID parsing
- Restore mapBookhoardUUIDToKoboContentId helper function
- Restore getCollectionMetadataForBook helper function
This fixes the catastrophic file corruption from commit 2200720 which
deleted 414 lines and inserted code in the wrong location.
Phase 1-3 of KOBO_IMPLEMENTATION_PLAN.md completed:
- Step 4: Nil UUID checks in Markup, Bookmark, AnalyticsGettests, SyncFromServer
- Step 5: ContentType field added to KoboReadingSync struct
- Step 6: last-read-place case added to Markup handler switch statement
Testing: Code compiles successfully, all handlers properly structured
Complete the rename by updating:
- DeviceCatalogs struct field: BookmannUuid → BookhoardUuid (models.go)
- Generated queries: Update all references (queries.sql.go)
- Local variables: bookmannUUID → bookhoardUUID (kobo.go)
- Struct field access: catalog.BookmannUuid → catalog.BookhoardUuid
All "bookmann" and "BOOKMANN" references are now eliminated from the codebase.
Part of project rename to Bookhoard.
Changes:
- Update comments: "Bookmann UUID" → "Bookhoard UUID"
- Rename sidecar struct field: Bookmann → Bookhoard
- Update type names: SidecarBookmannConfig → SidecarBookhoardConfig
- Fix test database name in queue_test.go
- Fix uppercase env var examples in KOBO_SETUP.md
Internal Go variable names (BookmannUuid, bookmannUUID) left unchanged
as they're implementation details that don't affect functionality.
Part of project rename to Bookhoard.
- Add Kobo sync handler with markup, bookmark, analytics, and initialization endpoints
- Add Kobo integration tests and Bruno API test collection
- Move device approve/reject routes from public to protected routes
- Enhance test infrastructure with DATABASE_URL support and helper functions
- Fix device GetDevice handler nil pointer handling
- Clean up test reports and session files
john-okeefe