A read-then-write race in processMediaFile allowed the same file to be
imported twice: two concurrent scan jobs (startup scan, fsnotify dirty-
directory scan, periodic backup poll, or a manual scan each run on
separate worker goroutines with separate MediaScanner instances) could
both SELECT 'not found' and both INSERT. There was no transaction, no
row lock, no unique constraint on (library_id, file_path), and no
ON CONFLICT clause, so nothing stopped the double insert. Observed in
production as two identical 'Head First SQL' rows created in the same
second (same sha256, size, path, library).
Database enforcement:
- schema.sql: add UNIQUE(library_id, file_path) constraint, guarded so
re-runs don't error
- schema.sql: add self-healing migration that runs on every startup -
dedup_media_items_by_path() collapses existing path-duplicates and
reparent_media_item_children() moves all child rows (progress,
highlights, bookmarks, notes, collections, formats, aliases, kobo
entitlements, etc.) onto a survivor before deleting losers, so the
constraint applies cleanly on already-duplicated servers without
losing reading history. Survivor picks the row with the most user
data, ties broken by lowest id
- CreateMediaItem: upsert via ON CONFLICT (library_id, file_path) DO
UPDATE so concurrent inserts collapse to one row and return it
- CreateMediaItemFormat: upsert via ON CONFLICT (media_item_id,
format_type), closing the same race on format rows
Application-level guards:
- media_scanner processMediaFile: after computing the file hash, check
GetMediaItemBySHA256AndLibrary (new query) and treat the file as
existing when identical content is already in the library under a
different path (content dedup, library-scoped so multi-library
setups still work)
Ops tooling:
- scripts/dedup_media_items.sql: standalone idempotent maintenance
script with a dry-run report (path + content duplicate groups, child
row counts) and transactional cleanup, for servers that prefer to
dedup manually before upgrading
Verified against the live database: the duplicate pair was collapsed
(reading_progress preserved on the survivor), schema.sql re-runs are a
no-op, and the constraint is in place with 62 unique books remaining.
The double_page_spread checkbox in the reader settings panel was inert:
it had no Alpine binding, no apply logic, and no persistence. Default
was also inconsistent (false in settings-manager, absent from server
defaults).
- Add doublePageSpread state to the reader Alpine component, loaded
from saved settings (default true)
- Add applyDoublePageSpread() which sets the renderer's 'spread'
attribute to auto/none and persists the setting via saveSettings
- Apply the spread attribute during fixed-layout renderer init
- Bind the settings checkbox with x-model and @change
- Add double_page_spread: true to ReaderService server defaults so
new users get the same starting value the client expects
- Also improve the PDF pan/select toolbar button: distinct smart-
select vs pan icons, highlighted state while pan mode is active,
and dynamic tooltips/aria-labels explaining each mode
CleanupExpiredRefreshTokens and CleanupExpiredOpdsTokens were generated
by sqlc but never invoked anywhere in the codebase, so expired/revoked
tokens accumulated in the database indefinitely. The refresh-token query
was parameterized in the settings-registry work specifically so its
retention window could follow the configurable session duration, but the
periodic caller was never wired up.
annotations.go:
- Rename StartTombstonePurger to StartDailyMaintenance, which now runs
all periodic cleanup tasks from a single 24h-tick goroutine.
- Add runDailyMaintenance helper: tombstones, then OPDS tokens, then
refresh tokens, each logging independently so one failure never skips
the others.
- Refresh-token retention is read from the registry (SessionDuration)
on every tick so live admin edits are honored; guarded on the registry
being wired so unwired test paths simply skip cleanup.
- All three queries only delete rows that are already expired or
revoked, so active sessions are never logged out.
main.go:
- Update the call site: tombstonePurgerCancel becomes maintenanceCancel
and calls StartDailyMaintenance.
Net footprint: still one goroutine and one ticker; the cleanup adds one
DELETE per table per day.
Replace the read-only "System Information" card (which listed hardcoded
values) with editable HTMX forms, organized so the live vs restart
distinction and related settings are visually clear.
admin_settings.templ:
- AdminSettings signature now takes liveGroups and restartGroups
([]SettingGroup) instead of a flat entry list.
- Remove the static System Information list. Render two cards: "Live"
(green, applies immediately) and "Restart Required" (warning header,
saved but only takes effect after restart).
- Within each card, TunableSettingsSection clusters entries into
labeled sub-sections by Group (e.g. "Password Quality", "Device Rate
Limits", "Login Lockout", "Worker Pool") with uppercase tracked
sub-headers.
- TunableSettingRow renders an inline HTMX form per setting: a Yes/No
select for bools, a number input with min/max for ints, text
otherwise, posting to /admin/settings/tunable. Rows show "modified
from default" when the value differs from the compiled default.
types.go:
- Add SettingEntry (template-local mirror of database.SettingEntry,
keeps templates from importing database) and SettingGroup.
utils.go:
- Add GroupTunableSettings: splits a flat, group-sorted entry list into
live and restart []SettingGroup buckets preserving source order.
utils_test.go covers the multi-group + empty cases.
frontend.go:
- The /admin/settings page handler now loads entries from the registry,
drops the three keys that have dedicated UI cards (default_timezone
dropdown, scan_poll_interval_seconds, auto_scan_enabled) so they are
not listed twice, groups the rest, and passes liveGroups/restartGroups
into the template.
Construct the SettingsRegistry at boot, load it, and thread it through
every consumer so the configurable values take effect and stay cached.
cmd/server/main.go:
- Build the registry from the Queries handle and Load() it right after
schema init; a load failure logs and continues (getters fall back to
compiled defaults, so startup is never blocked).
- Wire the registry into the package-level password validator
(SetDefaultPasswordSettings) and call SetSettings on every handler/
service that reads tunables: AuthHandler, DeviceAuthMiddleware,
OPDSHandler, SidecarHandler, SystemSettingsHandler,
AnnotationService, ConversionService.
- Source the restart-time values from the registry: login lockout
(max attempts + duration) feeds NewLoginAttemptTracker, and the new
NewSyncQueueProcessorWithConfig / NewWorkerWithConfig take the sync
queue and worker pool configs.
router.go:
- Config gains a Settings *database.SettingsRegistry field.
- The global auth rate limiter now reads RequestsPerMinute from
registry.AuthRateLimit() (env stays as the enabled/disabled switch
and as the fallback if the registry is unset).
admin_library.go:
- The HTMX scan-settings save endpoint reloads the registry after
writing so the change is visible without a page reload.
- Add PUT /admin/settings/tunable: a small HTMX endpoint that calls
SystemSettingsHandler.ApplySetting and returns a colored status
snippet ("Saved" or "Saved — restart required") for the admin UI's
per-row forms.
Add a single pair of admin-only endpoints that supersede the scattered
scan-settings JSON routes as the canonical way to read and write
tunable system settings. Existing legacy routes are kept working for
backward compatibility and now refresh the registry cache on write.
system_settings.go:
- GET /api/system/settings returns every known setting with full
metadata (value, type, min, max, requires_restart, category, group,
description, is_default) via SettingsRegistry.All().
- PUT /api/system/settings accepts {key, value}; ApplySetting() looks
up the compiled Default for the key, runs type-aware validation
(int range, bool parse, non-empty string, timezone via
time.LoadLocation), upserts via UpsertSystemSetting, reloads the
registry, and reports whether a restart is needed for the change to
take full effect. Shared by the JSON endpoint and the HTMX endpoint.
- Legacy UpdateScanSettings / GetScanSettings / UpdateTimezoneSettings
now reload the registry after writing and prefer the registry when
reading, so the cache stays consistent regardless of entry point.
sidecar.go:
- SidecarHandler gains an optional registry; the timezone branch of
UpdateSystemConfiguration (PUT /api/system/config) calls
settings.Reload() after the write so the new value is visible
immediately. base_url handling is unchanged.
system.go:
- Register GET/PUT /api/system/settings under the existing admin
/api/system group.
Split each constructor into a default-args wrapper and a config-accepting
variant so the sync queue interval/batch size and the worker pool size/
queue cap can be sourced from the settings registry at startup. These
values are constructed once at boot, so they are tagged requires_restart
in the admin UI.
queue.go:
- NewSyncQueueProcessorWithConfig(db, interval, batchSize) takes the
flush interval and batch size as parameters; NewSyncQueueProcessor
becomes a thin wrapper with the historical 5s / 50 defaults.
worker.go:
- NewWorkerWithConfig(numWorkers, queueCap, connManager) takes the
queue capacity as a parameter; NewWorker becomes a thin wrapper with
the historical cap of 100.
No behavior change for existing callers; main.go will switch to the
config-accepting variants in a follow-up wiring commit.
The 30-day retention window for soft-deleted annotations was a package
const; move it behind the registry so it can be tuned live.
annotations.go:
- AnnotationService gains an optional *database.SettingsRegistry and a
tombstoneTTL() helper. The skip-resurrect checks and the purge cutoff
now call it instead of reading the TombstoneTTL const directly.
- Add ActiveTombstoneTTL() so callers outside the sync package can
compute cutoffs consistently with the service.
- The package-level TombstoneTTL const is retained as the fallback for
tests / unwired code paths.
kobo.go, koreader.go:
- The per-book tombstone sweep cutoff now uses
h.annotationSvc.ActiveTombstoneTTL() instead of the wsync.TombstoneTTL
const, so both the service and the handlers honor the configured TTL.
Move three more hardcoded values behind the settings registry. All
apply immediately on the next request (no restart needed).
device_auth.go:
- DeviceAuthMiddleware reads per-route device rate limits (sync /
progress / metadata per minute) from the registry on each
authenticated request via a rateLimitConfig() helper, falling back to
the Default* constants when no registry is wired.
- The X-RateLimit-Limit response header previously hardcoded "60" for
every request type; it now reflects the actual configured limit for
the request type via rateLimitForRequestType().
opds.go:
- Default (50) and maximum (200) OPDS page sizes come from the
registry's OpdsDefaultPageSize()/OpdsMaxPageSize() instead of inline
literals, so catalog pagination can be tuned without a redeploy.
conversion_service.go:
- The 24h kepub cache lifetime is read from the registry via a
cacheTTL() helper (was a bare 24 * time.Hour literal in the
constructor). The field default is retained for tests that construct
the service directly.
- conversion_service_test.go updated to assert both the field default
and the cacheTTL() accessor return 24h.
Replace the hardcoded 7-day session lifetime and fixed password
complexity rules with registry-backed accessors so they can be tuned
from the admin UI without a code change.
auth.go:
- Drop the SessionDuration const; keep DefaultSessionDuration (7 days)
as the fallback used when no registry is wired (e.g. in tests).
- AuthHandler gains an optional *database.SettingsRegistry and a
sessionDuration() helper that reads the registry, falling back to
DefaultSessionDuration.
- Cookie MaxAge, JWT exp claim, and ExpiresIn responses now derive from
sessionDuration() instead of the package-level SessionDurationSec, so
a settings change takes effect on the next login.
refresh_token.go:
- Refresh-token lifetime follows sessionDuration() via a new
refreshTokenTTL() helper (was a separate refreshTokenExpiration const
that silently had to be kept in sync with the session duration).
password_validator.go:
- PasswordValidator now reads min length and the upper/lower/number/
special toggles from the registry at validation time, so rule
changes apply immediately. The special-character regex is compiled
once and reused (sync.Once).
- GetPasswordRequirements() and ValidatePassword() reflect the active
configured rules instead of a static list.
- Add SetDefaultPasswordSettings() so the package-level default
validator (used by echo's struct-tag validator) follows live config.
All paths degrade gracefully to the historical defaults when no
registry is wired.
Add a typed, cached registry over the system_settings table so that
values which used to be hardcoded Go literals can be changed at runtime.
Schema (database/schema/schema.sql):
- Extend system_settings with setting_type, min_value, max_value,
requires_restart, and category columns (all ADD COLUMN IF NOT EXISTS,
nullable for backward compat with the original three rows).
- Seed rows for every tunable: session duration, password rules,
login lockout, auth/device rate limits, OPDS page size, tombstone TTL,
conversion cache TTL, sync queue interval/batch, and worker pool
size/cap. Seed values equal the previous hardcoded literals, so
behavior is unchanged on upgrade. ON CONFLICT DO NOTHING preserves
any admin-modified values.
Queries (queries.sql):
- Add UpsertSystemSetting (RETURNING *) so new keys without a seed row
can still be written through the API.
- Add GetSystemSettingFull + GetAllSystemSettingsFull returning the
full typed row.
- Refactor CleanupExpiredRefreshTokens to take the retention window as
a parameter (make_interval(secs => $1)) instead of the INTERVAL '7
days' literal, so it can follow a configurable session duration.
Registry (internal/database/settings_registry.go):
- SettingsRegistry holds an in-memory cache of all known settings,
populated by Load at startup and refreshed by Reload on writes.
- Typed domain getters (SessionDuration, PasswordRules, DeviceRateLimits,
TombstoneTTL, OpdsPageSize, ConversionCacheTTL, SyncQueueConfig,
WorkerPoolConfig, LoginLockout, AuthRateLimit, ...) with compiled-in
fallback defaults and min/max clamping, so a corrupt or missing row
can never break the app.
- SettingDefaults is the single source of truth for keys, types, bounds,
and human descriptions; All() exposes metadata + current values for
the admin UI/API.
The registry lives in the database package (rather than its own
internal/settings package) because a quirk in this custom go1.26.5
toolchain prevented the large handlers package from importing any
newly-created package; every consumer already imports database.
Tests: settings_registry_test.go covers default validity per type,
int clamping at both bounds, garbage-value fallback, and unknown-key
lookup.
GetAllProgressData (SSR handler) was missing LastUpdated, DeviceIcon,
DeviceName, DeviceType, and EpubCFI fields that the template expects.
All showed blank. Now matches the API handler's field population.
Three bugs fixed:
1. Schema seeded base_url with fake placeholder 'bookhoard.example.com'.
Removed seed; startup now seeds from BASE_URL env var only if DB row
is empty (admin changes persist across restarts). One-time UPDATE
clears the placeholder in existing installs.
2. config.GetBaseURL() had a broken type assertion (local SystemConfigRow
vs database.SystemConfig) that always failed, returning . Admin panel
showed env var fallback instead of actual DB value. Fixed with a
function-type getter that properly wraps the DB query.
3. OPDS handler read base_url only from DB with no fallback. When DB had
the placeholder, all feed links pointed to an unreachable domain,
breaking KOReader search/download. Added deriveBaseURL() helper that
falls back to the request Host/scheme when DB value is empty.
Setup gate improvements:
- isSetupComplete now requires both admin user AND non-empty base_url
- Setup middleware no longer exempts all /api/ routes; only allows
/api/auth/register, /api/auth/login, /api/system/config before setup
is complete. All other API routes get 503.
- Cache invalidated when base_url is saved via admin settings
Dev workflow:
- New bruno/NewDevDBSetup/SetBaseUrl.yml for dev DB setup
- NewDB.sh runs SetBaseUrl between RegisterUser and CreateEbookLibrary
System collections (Not Started, Continue Reading, etc.) compute
their contents dynamically from reading_progress, so manual
add/remove has no effect. Hide the search bar, Remove Selected
button, Add Books button, per-card checkboxes, and Remove buttons
when the collection is system, making the page visually read-only.
- Add IsSystem bool to CollectionData, populated from the database
is_system_collection flag.
- Add data-is-system to #collection-data so the JS renderer can
also conditionally omit controls on library switch.
- Wrap toolbar controls, book picker modal, card checkboxes, and
remove buttons in if !collection.IsSystem in the template.
The /collections/:id page had several broken features because three
referenced functions (removeBook, toggleBookForRemoval,
filterCollectionBooks) were never defined, and every book card was
wrapped in <a href="/media/..."> so clicking the checkbox or remove
button navigated to the book detail page instead.
Card restructure:
- Remove the <a> wrapper; title and cover are now individual links.
- Checkbox sits in a <label> with expanded click area (p-2 -m-2).
- Checkbox uses Alpine :checked/@change bound to a reactive
selectedBooks array on the collections component.
Remove (single + bulk):
- Add removeBook(id) and bulkRemove() methods with confirm() dialogs.
- Wire the "Remove Selected" button with :disabled binding and @click.
- Selected-count badge is now Alpine-reactive (x-show/x-text).
Search within collection:
- Add filterCollectionBooks() that filters cards client-side by
title/author via data-* attributes and @input.
Book picker ("Add Books"):
- Point the HTMX search inputs at the existing /api/media-items/search
endpoint instead of the non-existent /api/media-items/filtered.
- Add hx-trigger="loadBooks" + hx-get to the grid so loadBooks()
actually fires an initial request when the picker opens.
- Merge the hidden limit/offset inputs into the #book-picker-filters
div so hx-include picks them up (was a separate <form id=filter-form>
that nobody referenced).
- Add show_checkbox mode to handleSearchHTML: when present, render a
new BookPickerGrid template with clickable, selectable cards instead
of the reader BookCard.
- Fix bookPicker submit() to location.reload() instead of a non-existent
reloadCollection HTMX event, and clearFilters() to target text inputs.
The dashboard re-renders its sections client-side (library switch,
refresh, saving settings) via renderBookCard in dashboard.ts, which was
still the old markup with no .book-card-action overlay. So the read
button appeared on the server-rendered cards but vanished as soon as the
dashboard re-rendered, while the bookshelf (always templ-rendered) kept
working.
- Rewrite renderBookCard to match the templ BookCard: detail link plus
the play/read action overlay, routing to the reader or the detail page
when the book has an active conflict.
- Add has_conflict to the BookInfo TS type and stamp it in the dashboard
sections API (GetSections) so client-rendered cards can route correctly.
- Add pointer-events-none / group-hover:pointer-events-auto to the
client-rendered carousel nav buttons so they no longer swallow hover
over edge cards, matching the templ fix.
The play button on book cards now opens the reader directly, instead of
always going to the detail page. Cards with an active progress sync
conflict route the play button to the detail page (which hosts the
conflict dialogue and resolves before writing progress), so the user is
never silently dropped into the reader with an unresolved conflict.
Backend:
- Add HasConflict to BookInfo and stamp it via ListSyncConflictsByUser
(MarkActiveConflicts / MarkActiveConflictsSections) on the dashboard,
bookshelf, series, tag, and search result card builders.
- Each page issues a single conflict query regardless of card count.
BookCard:
- Restructure into a detail link (cover + meta) with the play action as a
sibling overlay using a pointer-events split: the container passes
clicks through to detail while only the circular button routes to the
reader. No nested anchors.
- On touch devices (hover: none) the play button stays visible.
Fix: carousel nav buttons had opacity-0 without pointer-events-none, so
they swallowed hover/clicks over book cards on the dashboard. They are
now click-through until the carousel is hovered.
The bookhoard container crashed with 'panic: Failed to create file
watcher: too many open files' (media_scanner.go) after running for a few
hours, preceded by floods of 'no space left on device' from watcher.Add.
Root cause: every scan job called NewMediaScanner(), which eagerly
created an fsnotify watcher. SetFolders() then walked the entire library
tree and registered one inotify watch per directory (~3,000+ across the
libraries), and ScanFolders() registered them again during its walk. The
worker never called scanner.Close() on these ephemeral per-job scanners,
and the worker loop had no recover(), so:
1. Leaked watchers accumulated until the kernel inotify watch cap was
hit (ENOSPC -> 'no space left on device'), then
2. the process fd limit (ulimit -n 1024) was exhausted, causing
fsnotify.NewWatcher() to fail with EMFILE, and
3. NewMediaScanner panicked on that error, taking down the whole
process (exit code 2). With no restart policy the container stayed
down.
The scan jobs run frequently (scan_poll_interval), so the leak built up
within hours. Note this was NOT a disk-space issue; df showed plenty free.
Fix:
- media_scanner.go: NewMediaScanner no longer creates a watcher eagerly
(s.watcher starts nil), which removes the panic site entirely -- there
is nothing to fail at construction. The watcher is created lazily only
when needed.
- media_scanner.go: SetFolders gains a [?1049h[22;0;0t[1;24r(B[m[4l[?7h[?25l[H[2JEvery 2.0s: bool[1;37Hgaruda-ser8: Fri 31 Jul 2026 10:45:41 AM EDT[2;66Hin 0.002s (127)[2;80H
[3dsh: line 1: bool: command not found
[4d[24;1H[?12l[?25h[?1049l[23;0;0t
[?1l> parameter. It creates
and populates a watcher (returning an error instead of panicking) only
when watch=true; otherwise it skips all watcher.Add calls. ScanFolders
guards its watcher.Add with a nil check, and the WatchChanges event
loop exits cleanly when there is no watcher (polling still runs).
- worker.go: the worker() loop now wraps each job in defer/recover() so a
panicking job is recorded as failed and can never kill the process.
- worker.go: the three ephemeral scan handlers (processScanJob,
processSetFoldersJob, processDirectoryScanJob) now defer scanner.Close()
and call SetFolders(..., false), so scan jobs allocate zero watchers and
zero inotify watches. Any pre-existing leak is also bounded by Close().
- handlers/scanner.go: the long-lived watch-mode scanners (StartScanner
and StartWatchModeForLibrary) pass watch=true since they actually read
watcher.Events for live change detection.
- calibre_integration_test.go: updated to the new SetFolders signature
(watch=false, matching one-off scan usage).
Auto-add is fully preserved: new files are still detected by the periodic
poller (startBackupScan), which is independent of fsnotify and unaffected
by these changes. The watch-mode event loop remains as bonus responsiveness
when inotify is available; through Docker bind mounts where inotify is
unreliable, polling is what catches new books.
Search results navigated to /bookshelf with no filters instead of the
selected book's page. Results now link to /media/:id and display cover
thumbnails, with cover URLs resolved server-side via ResolveMediaURL.
Removes the dead selectedBook localStorage plumbing.
The device catalog feed was unusable on paged OPDS clients such as
KOReader: it sliced results into pages but never advertised how to reach
the next page, so clients could only ever fetch the first page (~50 books)
and could not search the catalog.
GetDeviceCatalog:
- Emit the full set of OPDS pagination link relations (self, start, first,
previous, next, last) pointing at catalog?page=N&per_page=M, with the
device auth token appended for path-based auth.
- Emit OpenSearch totalResults/itemsPerPage/startIndex metadata.
- Point rel=search at the OpenSearch description (correct MIME type).
SearchDeviceCatalog now branches on the q parameter:
- No q: return an OpenSearch description document whose Url template
contains the {searchTerms} placeholder, so clients can formulate a query.
- With q: return the existing acquisition results feed, now including
totalResults.
A pure addCatalogPaginationLinks helper holds the page/URL logic so it can
be unit tested without a database. New handler tests cover middle/first/
last/single/empty pages (correct presence of next/previous) and token
appending.
Ordering is intentionally left unchanged (created_at DESC, grouped by
library).
Extend the OPDS feed model so clients can page through large catalogs and
discover how to search them.
Feed changes:
- Add the OpenSearch namespace (xmlns:opensearch) to all feeds.
- Add optional TotalResults/ItemsPerPage/StartIndex fields, serialized as
<opensearch:totalResults>, <opensearch:itemsPerPage> and
<opensearch:startIndex>, plus a SetPagination helper.
- Add OpenSearchDescription/OpenSearchUrl types and a NewSearchDescription
constructor with GenerateXML/GenerateXMLString. This produces the
OpenSearch description document (application/opensearchdescription+xml)
that OPDS clients like KOReader fetch to learn the {searchTerms} search
URL template.
These are building blocks; the handlers are wired up in a follow-up commit.
Tests cover SetPagination, omission when unset, XML emission of the
paging metadata, and OpenSearch description generation/serialization.
The OPDS feed test suite did not compile or pass:
- TestNewEntry asserted on entry.Creator, but the Entry struct stores the
creator under Author.Name (the Atom <author><name> element). Assert on
entry.Author.Name instead.
- TestFeedGenerateXML expected <dc:title>/<dc:creator> elements, but the
Entry struct emits standard Atom <title> and <author><name>. Update the
expected substrings to match the actual (correct) output.
These are pre-existing assertion errors unrelated to any field being
removed; the code under test was already correct.
The UI had no surface showing how many media items have been imported.
Surface the total in the library switcher shown on the Dashboard, Series,
and Collections pages (via the LibrarySwitcher component) and in the
Bookshelf's inline library filter.
- Add a MediaCount field to LibraryData and a TotalMediaCount helper to
sum counts for the "All Libraries" / "All Books" option.
- resolveLibrary() now fetches per-library counts (one query) and maps
them onto each LibraryData entry, so the switcher reflects the active
scope without changing the component's signature.
- Each library option renders "(N)" and the "All" option renders the
grand total across the user's visible libraries.
The "All" total is the sum of the user's visible libraries, correctly
respecting per-user library visibility rather than a raw global count.
Regenerated templ files for library_switcher and bookshelf.
Add GetVisibleLibraryMediaCounts, which returns the media item count for
each library visible to a given user in a single GROUP BY query over
media_items. It mirrors the visibility logic in GetUserVisibleLibraries
(libraries default to visible unless an explicit false row exists) so
counts can be resolved in one round-trip instead of N per-library
lookups.
Regenerated sqlc bindings (querier.go, queries.sql.go).
Complete the annotation sync pipeline across all ingest and serve paths.
Previously, annotations sent inline with KOReader progress pushes were
silently discarded, and no annotations were ever served back to devices.
INGEST (device → server):
KOReader (koreader.go):
- Add processBookAnnotations helper that processes inline highlights,
notes, and bookmarks from every progress push (immediate + checkpoint)
- Highlights get CRE→CFI position conversion before SaveHighlight
- KOReader 'notes' (text + notes) stored as highlights with NoteText
to ensure correct round-trip classification
- Bookmarks routed through SaveBookmark with device sync data
- Called from both updateProgressForBook and handleCheckpointSync
Kobo (kobo.go):
- Markup handler: annotations and bookmarks route through
AnnotationService (SaveHighlight/SaveBookmark)
- Bookmark handler: same routing with device sync data
- SyncFromServer handler: same routing
- All handlers fall back to direct DB calls when annotationSvc == nil
Web reader (media.go):
- CreateMediaHighlight → SaveHighlight (Source="web", ModifiedAt=now)
- CreateMediaNote → SaveNote (Source="web")
- DeleteMediaHighlight → TombstoneHighlightByID
- DeleteMediaNote → TombstoneNoteByID (was hard delete, now tombstone)
- All fall back to old behavior when annotationSvc == nil
SERVE (server → device):
KOReader GetMetadata (koreader.go):
- Query and serve bookmarks from media_bookmarks table (was missing)
- Serve deleted_highlights and deleted_bookmarks arrays containing
device_sync_data + dedup_key for client-side deletion
- Highlights/notes already served with reverse CFI conversion
Kobo Markup handler (kobo.go):
- Track processed books during sync
- Query tombstones per book, extract bookmark_id from device_sync_data
- Return DeletedAnnotations array in KoboSyncStatus response
Conflict resolution (conflicts.go):
- Enable annotation conflict types in ResolveConflict handler
- Add applyAnnotationResolution dispatching to:
applyHighlightResolution / applyBookmarkResolution / applyNoteResolution
- Each looks up by dedup_key and applies winner's fields
- Allow manual override of auto_resolved conflicts
(changed check from != "unresolved" to == "user_resolved")
Infrastructure:
- AnnotationService field + SetAnnotationService in router Config
- Inject AnnotationService into KOReader, Kobo, Media handlers
- Start tombstone purger goroutine in main.go (24h interval)
- Test helpers: construct AnnotationService in test setup
Wire AnnotationService into SyncQueueProcessor and implement the three
previously-stubbed execute methods:
- syncHighlight: unmarshals syncData JSON into SaveHighlightRequest,
applies CRE→CFI conversion via AnnotationService
- syncNote: unmarshals into SaveNoteRequest
- syncBookmark: unmarshals into SaveBookmarkRequest
- Add SyncTypeBookmark to executeSync switch (was hitting default error)
Add enqueue methods for future offline/batch use:
- EnqueueHighlight / EnqueueNote / EnqueueBookmark
- Shared enqueueAnnotation helper creates queue items with
PriorityCriticalNote and 3 max attempts
- Update types (HighlightUpdate, NoteUpdate, BookmarkUpdate) mirror the
existing ProgressUpdate pattern
Existing handler behavior is unchanged — annotations still sync
synchronously via AnnotationService. The queue path is available for
retry-on-failure and offline batch processing scenarios.
AnnotationService is the central service for cross-device annotation sync.
It provides SaveHighlight, SaveNote, and SaveBookmark methods that handle
the full sync lifecycle:
Identity (3-layer):
1. Server UUID (primary key)
2. Per-device native ID stored in device_sync_data JSONB
3. Content dedup_key: sha1(normalize(selection_text) + bucket_position)
- CFI character offsets are stripped for bucketing so the same
highlight at slightly different offsets still deduplicates
- Raw positions are preserved in the DB for precise restoration
Resolution policy (LWW):
- When the incoming annotation has an explicit ModifiedAt timestamp,
last_modified_at wins
- When the device sends zero ModifiedAt (creation time only), field-diff
mode compares content fields (text/color/note/percentage) — if all
match, the save is skipped; if any differ, the save is applied with
server-receive-time as the new last_modified_at
Conflict detection:
- When incoming and existing annotations have different sources (e.g.
koreader vs kobo) and content differs, an auto_resolved sync_conflict
is recorded with both sides' data for audit trail
- Broadcasts a WebSocket conflict notification for real-time UI updates
Tombstone management:
- Delete-wins: tombstoned annotations block recreation from stale pushes
- 30-day TTL before physical purge
- PurgeExpiredTombstones method + StartTombstonePurger goroutine (24h ticker)
Add locators.go with unified bidirectional CFI conversion:
ConvertToCanonical / ConvertFromCanonical
- CRE XPointer <-> standard EPUB CFI (for KOReader)
- KEPUB CFI passthrough (for Kobo)
- Skips non-reflowable formats (PDF, CBZ, fixed-layout EPUBs)
Add 25 unit tests covering:
- Dedup key determinism, text normalization, position sensitivity
- Offset insensitivity (CFI char-offset bucketing)
- Device sync data merge (preserves existing, overwrites same source)
- Cross-source detection
- LWW comparison (newer wins, older skipped, fallback to updated_at)
- Field-diff mode (identical content skipped, changes applied)
- Tombstone TTL constant
- CRE XPointer parsing and classification
- Standard EPUB CFI classification
Add migration columns to media_highlights, media_notes, and media_bookmarks
for cross-device annotation sync:
- dedup_key: SHA-1 of normalized selection text + bucketed position, used
as the stable cross-device identity for annotations
- last_modified_at / last_modified_source: edit clock for LWW resolution
and cross-source conflict detection
- deleted / deleted_at: sticky tombstone columns for delete-wins semantics
with a 30-day TTL before rows are physically purged
- note_text on highlights: stores attached notes from KOReader entries that
have both selected text and a user note
- Location columns on bookmarks (cfi_position, percentage_location,
epubcfi_location, chapter_reference, paragraph_reference)
- device_sync_data JSONB on all three tables: stores per-device native
identifiers (e.g. KOReader pos0/datetime, Kobo bookmark_id) so each
device can locate and manipulate its own copy of an annotation
Add partial unique indexes on (user_id, media_item_id, dedup_key) where
deleted = FALSE to enforce one active annotation per dedup key.
Add tombstone purge indexes on (deleted, deleted_at) for efficient GC.
New queries:
- GetByDedupKey for all three tables (returns active or most-recent tombstone)
- CreateFull / UpdateForSync for all three tables (populate sync columns)
- TombstoneByDedupKey / TombstoneByID for all three tables
- PurgeExpired* for all three tables (GC past TTL)
- GetActiveAnnotationsForBook (filtered union of highlights + notes)
- GetTombstonedAnnotationsForBook (union of all 3 deleted within TTL)
- GetMediaBookmarks with deleted filter
- GetMediaBookmark (singular) with deleted filter
- Added deleted=FALSE filter to GetMediaHighlights, GetAnnotationsForBook
- CreateAutoResolvedSyncConflict (INSERT with resolution_status='auto_resolved')
Setup completion was previously tracked by a manually-flipped setup_complete row in system_settings, written via a JWT-protected PUT /api/setup/complete endpoint. This meant any admin user created outside the setup wizard (future CLI, seed scripts, direct DB inserts) would not flip the switch, leaving the app stuck redirecting to /setup.
The trigger is now derived from real data: setup is complete iff at least one admin user exists. This is self-correcting regardless of how users are created, and re-engages setup automatically if all admins are ever removed.
Changes:
- Add internal/setupstatus package with IsSetupComplete() (queries CountAdmins, 10s in-memory cache, fails open on DB error) and Invalidate() to clear the cache. Uses an AdminCounter interface to avoid importing the database package.
- Add CountAdmins sqlc query (SELECT COUNT(*) FROM users WHERE role = 'admin') and regenerate.
- Rewire router/setup.go isSetupComplete() to delegate to setupstatus; drop the old setup_complete setting read, cache vars, and the PUT /api/setup/complete route.
- Call setupstatus.Invalidate() in the auth handler after CreateUser, UpdateUserRole, and DeleteUser so the cache reflects admin-count changes immediately.
- Align first-user promotion in Register to key off !adminExists instead of len(users) == 0, so the two checks cannot diverge.
- Remove the now-dead SetSetupComplete/GetSetupStatus handlers.
- Drop the setup_complete seed row from schema.sql.
- Remove the apiPut('/setup/complete') call from the setup wizard finishSetup(); the admin account created in submitAdmin already marks setup complete server-side.
Remove TestConvertHessBook and TestDebugHess (both referenced a
non-existent Hess EPUB at an absolute local path) and
TestConvertCPByTextSearch (referenced a Crime and Punishment EPUB
in the local uploads directory). These were development-time debug
tests that only worked on the author's machine.
All CFI/KEPUB conversion behavior is already covered by the proper
fixture-based tests (TestKEPUBRoundTrip, TestKEPUBConvertKEPUBToStandard,
TestKEPUBConvertWithEmElements, etc.) which use createTestEPUB and
createTestKEPUB helpers.
When a Kobo device pushes a last-read-place bookmark, the server now
converts the KEPUB CFI (with koboSpan wrappers) to a standard EPUB CFI
and extracts surrounding text as context_text for use by other devices
(KOReader, web reader) during their pull-side CFI conversions.
Previously the raw KEPUB CFI was stored verbatim as epubcfi, which
meant foliate and CREngine couldn't resolve it (wrong child indices
due to koboSpan wrappers), and no context_text was available for the
text-search fallback in ConvertStandardToCRE.
Changes:
- kepub_cfi_converter.go: Add ExtractedContext field to
KEPUBConversionResult, populated from the already-computed
searchText in both ConvertKEPUBCFIToStandard and
ConvertStandardCFIToKEPUB (exact-match and percentage-fallback
paths).
- kobo.go: Add libraryService field and SetLibraryService setter
(mirrors KOReaderHandler pattern). Add convertKoboCFIToStandard
helper that resolves EPUB+KEPUB paths, instantiates the converter,
and returns the converted CFI + extracted context. The last-read-place
branch in Markup now calls this helper for reflowable formats,
skipping fixed-layout/comic archives (page-index only).
- router.go: Add LibraryService to router Config.
- sync.go: Wire LibraryService to KoboHandler via SetLibraryService.
- main.go: Pass libraryService through router config.
The conversion is purely additive — if no KEPUB file exists on disk
(e.g. side-loaded EPUB without kepubify conversion), the handler
gracefully skips conversion and stores the raw CFI as before.
The push path (koreader→server) was already updated to omit epubcfi
for paging documents, and SaveProgress derives percentage from
page/total_pages for fixed formats. However, the pull path
(server→koreader) still returned any stored epubcfi and attempted
CFI→XPointer conversion, and SaveProgress preserved stale CFI values
written by the web reader (which generates fake CFIs via
CFI.fake.fromIndex for every comic/PDF page).
These fake CFI strings lingered in the database and koreader's pull
path picked them up over progress.page, causing tonumber("epubcfi(...)")
→ nil → GotoPage(nil) → crash when the user confirmed the sync prompt.
Changes:
- GetMetadata (koreader.go): for fixed_layout/comic_archive formats,
skip returning epubcfi and skip the CFI→CRE XPointer conversion.
Reflowable documents are byte-for-byte unchanged.
- SaveProgress (progress.go): for fixed formats, explicitly clear
epubcfi and character_offset on every save so stale values from
prior web-reader sessions are cleaned up over time.
The CRE->CFI converter works by text search / character-offset mapping
across the EPUB spine. Image-based fixed content (fixed-layout comic
EPUBs, PDFs, comic archives) has no extractable text, so the conversion
can never succeed and only wastes time parsing content docs while
logging a failed percentage-precision result.
Guard the conversion in the KOReader progress handler: when the matched
media item's FormatGroup is fixed_layout or comic_archive, skip
ConvertCREToStandard entirely. The incoming xpointer is left as-is so
KOReader<->KOReader restore via GotoXPointer still works; the web reader
restores by page index (the canonical locator for these formats).
This covers fixed-layout comic EPUBs in particular: KOReader routes all
EPUBs through CREngine (has_pages == false), so they send a real
xpointer that passes IsCREXPointer and would otherwise trigger the
doomed text extraction. Reflowable EPUBs (format_group == reflowable)
still run the conversion exactly as before.
Fixed-layout EPUBs, PDFs, DjVu, and comic archives (cbz/cbr/cb7/cbt)
are page-based: each page is a fixed image, so a page index is an exact,
universal locator regardless of screen size or device. Sync previously
treated these like reflowable content (CFI-first restore, percentage
fallback, character-offset math), which was both wrong and lossy. This
makes the page index the canonical position for fixed-layout and comic
formats while leaving the reflowable path byte-for-byte unchanged.
Backend:
- progress.go SaveProgress: branch on mediaItem.FormatGroup. For
fixed_layout/comic_archive derive percentage from current_page/
total_pages and skip the CFI/character-offset back-fills (meaningless
for image content). The reflowable derivation block is preserved
verbatim under an else.
- kobo.go: Kobo only sends a percentage, so for fixed-layout/comic
formats derive CurrentPage via PercentageToPage(percentage, pageCount)
using the media item's known page count, so Kobo->web lands on the
exact page.
Reader:
- reader.templ readerInitExpr: pass savedPage/savedTotalPages to the
reader config for fixed_layout/comic_archive formats.
- reader.ts: when isFixedLayout and savedPage is present, restore via
view.init({ lastLocation: savedPage - 1 }) (a bare number navigates
foliate directly to the section index). Reflowable falls through to
the existing CFI->percentage path, unchanged.
The OPDS device catalog previously hard-coded every acquisition link as
application/epub+zip and always offered kepub/pdf alternate links, which
is wrong for comic archives (cbz/cbr/cb7/cbt) and other non-epub media.
- Resolve the acquisition mime type from the media item's stored
mime_type (falling back to format_mimetype, then epub) instead of
assuming epub
- Only offer reflowable conversions (kepub for kobo, pdf) for ebooks;
comic archives are served as-is in their native format
- Derive the native format label (epub/pdf/cbz/cbr/...) from the file
path in ListFormats rather than always reporting epub
- Add resolveMimeType, isComicArchive, and formatLabelFromPath helpers
Add setupRedirectMiddleware that checks the setup_complete system
setting on every request. If setup is incomplete, all non-setup
requests are redirected to /setup so the wizard is the first thing
new users see. The check uses an in-memory cache (10s TTL) to avoid
hitting the database on every request, with cache invalidation on
setup completion.
The middleware skips /setup, /api/*, /static/*, /health, and
/favicon.ico so the wizard page, API calls, and static assets load
normally during setup.
Register two new routes:
- GET /setup: renders the setup wizard SSR template
- PUT /api/setup/complete: marks setup as complete (JWT-protected,
requires an authenticated admin user created in step 1)
Add 'setup_complete' boolean to the system_settings seed data (defaults
to false) so fresh databases start in the unconfigured state.
Add two new handlers to SystemSettingsHandler:
- SetSetupComplete: marks setup_complete=true in the database
- GetSetupStatus: reads the current setup_complete value, returns
{setup_complete: bool} JSON response, defaults to false if the
setting row is missing or unparseable
cfi_converter_test.go:
- TestFindTextInNode_SingleTextNode: baseline single-node match
- TestFindTextInNode_CrossEmElement: 'Vokalia and Consonantia' across
two <em> elements — verifies match returns the 'Vokalia' text node
- TestFindTextInNode_CrossStrongElement: text crossing <strong> boundary
- TestFindTextInNode_DoesNotCrossParagraphs: verifies block boundary
enforcement — text split across <p> elements is NOT matched
- TestFindTextInNode_NestedFormatting: <em><strong> nesting
- TestFindBlockParent: verifies findBlockParent walks up through inline
elements to find block-level <p>
- TestCollectInlineText: verifies text segments are collected in order
with correct content
kepub_cfi_converter_test.go:
- TestKEPUBConvertWithEmElements: previously skipped, now expects exact
precision and verifies round-trip conversion works for text spanning
<em> element boundaries
Instead of reading only from the single resolved text node (which
produces a tiny window when the position is inside <em> or other inline
elements), extractSurroundingText now:
1. Finds the block-level parent (e.g. <p>) of the resolved text node
2. Collects all text within that block, transparently crossing inline
formatting elements via collectInlineText
3. Computes the global offset of the original text node within the
concatenated block text
4. Extracts the [offset-window : offset+window] slice
This gives a full context window regardless of inline element
boundaries, enabling accurate text bridging between EPUB and KEPUB
documents even when reading positions fall inside <em>, <strong>,
<span class="koboSpan">, etc.
Falls back to single-node extraction when no block parent is found
(e.g. orphan text nodes in tests).
When finding or extracting text in EPUB/KEPUB DOM trees, inline
formatting elements like <em>, <strong>, <i>, <b>, <span>, etc. should
not break text continuity. A reader sees 'Vokalia and Consonantia' as
one phrase regardless of the <em> wrappers around each word.
Add inline formatting element set and helper functions:
- isInlineFormatting: checks if an element is an inline phrasing element
- collectInlineText: flattens text across formatting elements within
a block-level parent, returning segments that map back to original
text nodes
- findBlockParent: walks up from a text node to find the nearest
block-level ancestor (used to scope text collection)
- findTextAcrossInlineElements: fallback for findTextInNode that
concatenates text within each block element (transparently crossing
formatting elements) and maps match positions back to actual nodes
- collectBlockElements: gathers all block-level elements containing text
The key invariant: text collection NEVER crosses block-level element
boundaries (<p>, <div>, <h1>-<h6>, <li>, etc.) to avoid concatenating
text from different paragraphs.
The findTextInNode function now tries single-text-node matching first
(fast path, unchanged), then falls back to cross-element matching only
when needed. This preserves performance for the common case.
Test coverage for KEPUBCFIConverter with programmatically generated
EPUB and KEPUB zip fixtures:
- KEPUB->Standard text search conversion (exact precision)
- Standard->KEPUB text search conversion with round-trip verification
- Multi-position round-trip (3 phrases across different paragraphs)
- Cross-chapter conversion (spine index 1)
- No-context-text conversion (extracts surrounding text from resolved node)
- Invalid CFI handling (falls back to percentage)
- Percentage fallback for unresolvable CFIs
- 5-paragraph round-trip covering different document positions
- CFI structural difference verification (koboSpan adds DOM steps)
- Spine index consistency between EPUB and KEPUB
- Real book conversion test (skips if file not present)
- extractSurroundingText unit tests
Add KEPUBCFIConverter in internal/sync that converts between KEPUB CFIs
(which include extra koboSpan DOM steps) and standard EPUB CFIs at sync
time, so only standard epubcfi values are stored in the database.
The converter works by:
1. Resolving the source CFI in the source document (EPUB or KEPUB)
2. Extracting surrounding text at the resolved position
3. Searching for that same text in the target document
4. Building a new CFI pointing to the matched text in the target
This text-content bridging handles the DOM structural differences
between EPUB (text nodes at depth 2) and KEPUB (text nodes wrapped in
<span class="koboSpan"> at depth 3).
Falls back to percentage-based positioning when text search fails,
matching the pattern used by the existing CRE converter.
No changes to existing cfi_converter.go or KOReader conversion code.
Same-package access to unexported functions (resolveCFIToNode, buildCFI,
findTextInNode, etc.) via internal/sync package placement.
- UpdateSystemConfiguration: when base_url changes, automatically
update opds_base_url and api_base_url derived configs
- convertPending: format time.Time as RFC3339 string instead of
relying on string type assertion which would panic
- Add <title> and <author><name> elements to Atom feed for compatibility
- Remove doubled /opds/opds/ path prefix in feed URLs
- Include ?token= auth param on all OPDS URLs
- Serve kepub links only for Kobo devices
- Fix URL construction for entries and acquisitions
Previously device creation happened in CheckRegistrationStatus (polling
endpoint), which was racy. Now the admin's ApproveDevice handler
creates the device record and stores auth token + device ID on the
registration entry. CheckRegistrationStatus just returns the pre-created
credentials.
Also adds approved/authToken/deviceID/syncEndpoints fields to
PendingRegistration struct.
When resolving a conflict, the last_sync_source is now set to the
winner's actual source name (koreader, web, etc.) rather than always
'manual'. This prevents subsequent saves from re-triggering conflicts.
Also removes the strict oneof validation on the winner field since
the source name is dynamic.
Web reader can now send surrounding text at current reading position.
Stored in reading_progress.context_text for use as CFI resolution
fallback when converting epubcfi to CREngine XPointer.
john-okeefe