bbedbfb13fd90f89ff1df94263d2161e687b7d9d
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
6216bd9bcf |
feat(audiobooks): multi-file books (folder-of-tracks + zip) + exhaustive metadata ingestion
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
|
||
|
John O'Keefe
|
45b8542e25 |
fix(api): honor the request's origin as the writer label — app rows stop wearing "web"
Release / build-and-push (push) Successful in 3m36s
Every web-API write endpoint hardcoded Source:"web", so the Android app's own pushes landed with last_sync_source/last_modified_source "web" — the app was indistinguishable from the web SPA in progress conflicts, device_sync_data, and annotation bookkeeping. The app CAN say who it is (bookmark create already carried origin → origin_source); the server just ignored it for labeling. All six write sites (progress PUT, bookmark/note/highlight create+PUT) now parse an ADDITIVE optional "origin" key, normalized through one gate: "android"/"ios" pass through, anything absent/unknown stays "web" (the label keys conflict claims and device_sync_data — free-form client strings would pollute them). Precedence on progress: a paired X-Device-Token still outranks origin. Bookmark origin_source now rides the same gate. The web SPA sends no origin — its rows are unchanged; KOReader/Kobo handlers keep their own labels. Wire discipline: additive request keys only (rule 2), response shapes untouched, pins changelog widened in-commit, locked by TestOriginWriterAttribution (android lands, default stays web, unknown normalizes). Full battery: only the two known pre-existing failures (TestOPDSLibraryFolders rot, TestWebSocketPingPong — both reproduce with the change stashed). |
||
|
John O'Keefe
|
8fc74c908a |
fix(conflicts): resolution writes funnel through SaveProgress — audio position survives
Batch H1 step 3b — THE data fix. Conflict resolution was the one live
erasure writer: applyProgressResolution and its verbatim twin
applyResolution both wrote the winner via a raw UpdateUniversalProgress
whose param list omits audio_offset_seconds/context_text/viewports, and
the SQL's DO UPDATE SET ... = EXCLUDED... NULLed exactly those. The web
UI resolves through POST /api/conflicts/:id/resolve and
/api/conflicts/bulk-resolve (main.js resolveConflict/bulkResolve), and
the reader's mark-as-read fires a resolution first when the reader SSR
stamps data-conflict-id/-winner — so every resolution today erased the
audiobook position and reader context.
Now: ProgressService is a required ConflictHandler dependency; BOTH
call sites (single resolve + bulk resolve) share one
applyProgressResolution that funnels through SaveProgress (the verbatim
twin is deleted). Deliberate choices, per the plan:
- SkipConflictDetection on the resolution write: both flows apply
BEFORE the conflict row is marked resolved, so the funnel's LWW
detector would otherwise file a duplicate pending conflict
(HasRecentConflictResolution is false at apply time).
- Source "conflict_resolution" — the row's last-write attribution; the
winner stays durably recorded in resolution_data.winner.
- Broadcast: true — live devices jump to the winning position.
- The winner snapshot maps percentage/epubcfi/context_text/chapter/
character/page/total_pages; a winner without a CFI clears the locator
pair (ClearLocator) instead of re-pairing with the loser's stale
anchor. Everything the snapshot doesn't carry SURVIVES via the LWW
seed: audio_offset_seconds above all. chapter_progress keeps its
stored value (was overwritten with the book percentage), an absent
percentage survives (was written as 0%).
- Accepted: the resolution captures one reading_history session
(attributed to conflict_resolution; the 2h idle ceiling applies).
Verification: TestH1_ConflictResolutionPreservesAudio drives the real
flow — web push with audio 300s, diverging push from a second source
detects a conflict, resolve with the second source as winner, GET shows
percentage 0.3 AND audio_offset_seconds 300 (previously NULLed), and no
duplicate pending conflict remains. Full contract pin suite green.
The host-run TestConflict*/TestProgress* failures are environmental
(container-only /app/uploads path + db hostname; fail identically at
baseline
|
||
|
John O'Keefe
|
28bd323505 |
refactor(progress): SaveProgressRequest becomes the canonical PUT payload; GET serializes from the row
Batch H1 steps 1-2 of the hardening plan. Wire contracts frozen; the pinned shapes are unchanged modulo the whitelisted additive GET keys (docs/contract_pins.md). 1. One canonical request struct. The web PUT binds SaveProgressRequest directly (identity/attribution fields are json:"-" so they stay server-owned) and copies its server fields once — the hand-rolled DTO and the 20-line field-copy block are gone. Parse set for existing clients is IDENTICAL to the old DTO (verified in web/src: the ebook reader sends percentage/current_page/total_pages/reading_mode/epubcfi/ context_text/chapter/zoom_level — reader.ts:3113-3143; the audiobook reader sends percentage/chapter/audio_offset_seconds/elapsed_seconds/ reading_mode — audio.ts:518-520). 2. GET /api/media-items/:id/progress now serializes through one row-struct-driven builder instead of a hand-map, so a row column cannot be silently forgotten again (the audio_offset_seconds omission). Existing keys are verbatim; the row already carried audio_offset_seconds/reading_mode/viewport_x-y/zoom_level/ scroll_position_x-y/panel_number and they surface as WHITELISTED additive keys. The web audiobook reader already reads audio_offset_seconds (audio.ts:219) — with the key absent, durationless books resumed at chapter start (offset always 0); they now resume correctly. The conditional context_text bolt-on and the server-derived css_selector/anchor_href block are unchanged — restore precedence stays percentage-first, never re-derived. |
||
|
John O'Keefe
|
902cbe5915 |
feat(audiobooks): Batch A — foundation, m4b ingest, playback sync, web player
Phase 1 (foundation): 'audiobooks' library type seeded (schema + DB);
audio_metadata JSONB on media_items (the kobo_metadata precedent —
only audio paths read it); reading_progress.audio_offset_seconds for
the honest {chapter, offset} locator; extensions/mimes registered for
ALL breadth formats (m4b/m4a/mp3/flac — deliberate per the DOC/LIT
doctrine, m4b scans richly first); classifyFormatGroup 'audiobook'
case (gotcha 39 recomputes on rescan); Go mime table extended at boot
so media elements accept the streams; ffmpeg/ffprobe in the image
(8.1.2 — also the xHE-AAC identification tripwire).
Phase 2 (ingest): sidecar-first (Libation/ABS metadata.json —
narrators/ASIN/curated chapters), ffprobe fallback/filler (duration,
bitrate, codec/profile, embedded chapters). Fixed three real
parser brittlenesses the Libation sidecar exposed: series accepts
string OR array-of-objects, publishedYear accepts number OR string,
ffprobe chapter timings accept number OR string (flexFloat). Folder-
structure fallback now runs on the UPDATE path too (rescans were
clearing sidecar-absent series/author); audio facts re-derive on
every rescan (gotcha 39).
Phase 3 (playback + sync): GET /api/media-items/:id/chapters (the
format-abstraction list); streaming rides the existing cookie-authed
Range /uploads path; PUT progress carries audio_offset_seconds +
elapsed_seconds (honest wall-clock playing time) — the reading-
history capture prefers player-reported elapsed over the inter-push
delta, same 2h ceiling; reading_mode='audio' marks the row.
Phase 4 (web player): dedicated audio reader page (foliate not
involved) — cover, chapter title, whole-book scrubber, skip ± (chips
1/5/10/15/30s, default 10s), speed (0.75×–2×), chapter list with
jump-to, Media Session lockscreen/notification transport, periodic
playing-only progress pushes, position restore on open.
Verified live: ingest of the Libation sample (Gardens of the Moon —
title/author/series/cover/date from sidecar+folders, 34 chapters,
26h duration, aac/LC); chapters endpoint; the reader route serving
the player; playback with position restore (0:57 round-trip);
progress rows in audio mode; wall-clock session rows in
reading_history.
|
||
|
John O'Keefe
|
6457fbaf72 |
feat(stats): reading statistics — session capture wired + /stats page + widget + per-book card
The design parked in the handoff (multi-tier stats off reading_history; bookorbit-style in data, not looks) is in. The real prerequisite work was data: reading_history existed but NOTHING wrote to it — the whole stats feature was sitting on an empty table. - CAPTURE (no client changes, no repo-boundary crossings): every successful progress write through SaveProgress (KOReader + app + web + Kobo + offline queue — the one chokepoint) now records one reading_history session row: span = the interval since the PREVIOUS write instant, pages = positive page delta when both sides know the page index, marker rows for first contact. Spans over 2h are kept as zero-second markers (overnight suspend→resume, offline flushes must not inflate totals) — the honest 'sync cadence, not a stopwatch' caveat stays. device_id only when a REAL device row matches (the web/app progress path historically stuffs the USER id into DeviceID — probe instead of trusting it; FK violation would silently drop every capture). Source preserved in device_metadata. - API: /api/stats/summary?range=7|30|90|year (totals, zero-filled daily arrays, streaks, most-active day/weekday, top books, library breakdown — one call), /api/stats/this-week (widget: rolling 7d vs previous + streak), /api/stats/books/:id (lifetime + recent sessions). All bucketed in the user's timezone; finished ≡ any push ≥0.995; visible-library lens enforced at source. Legacy /api/analytics/* kept untouched. Range windows anchor end-of-day to the START of today (blasting the clock time into tomorrow resolved 'today' a calendar day late — found by the streak test). - /stats page (sidebar Stats between Progress and Devices, previously /analytics was an orphan): range selector, five stat cards, two Chart.js v4 charts (daily minutes bar, pages/day line — chunked via dynamic import so main.js stays untouched), per-library breakdown, most-read list, empty-state explainer. /analytics now 301s to /stats. The orphaned analytics.templ + analytics.ts removed (their containers and the page JSON never matched; the page was astub). - Dashboard 'This Week' widget (server-rendered, suppressed on service failure) and the book detail 'Your Reading Stats' card (lifetime totals of the viewer's own sessions) complete the tiers: per-user default, per-library breakdown inline, per-book on the page, per-system/admin aggregate parked as before. Tests: statistics_test.go (capture semantics at the chokepoint, tz/day buckets, zero-fill, streak math, finished detection, range validation, auth gate, widget, per-book lifetime). Full suite PASS. |
||
|
John O'Keefe
|
69516ae165 |
feat(server): progress locator coherence — percentage-only writes clear stale anchors
The verify/heal pipeline verified only INBOUND context-bearing writes;
stored-row coherence was never enforced. SaveProgress seeds missing
fields from the existing row, so every percentage-only write (KOReader
low-confidence conversion, KOReader no-pointer payloads, Kobo markup
ReadingSync / analytics / sync-from-server, locator-less queue entries)
silently re-paired its NEW percentage with the PREVIOUS writer's stale
CFI+context — the incoherent row shape that fed the client-side
far-anchor restore floor (and, via the read-side anchor handles, could
even hand a first-party client the OLD position's scroll address).
KOReader's converter already declared the right intent ("storing
percentage only — the row's percentage restores approximately"); the
seeding betrayed it. SaveProgressRequest gains ClearLocator, set by the
percentage-only call sites, and the seeding rule honors it: stale
epubcfi/context/character_offset clear (the upsert assigns columns
directly, so NULL sticks) unless the request carries its own. A
request-carried context survives so the verify/heal pass can still mint
a fresh anchor from it; the HTTP merge path (field-absent PUTs
preserve) and all anchor-bearing writes are untouched. The fixed-layout
branch already cleared locators unconditionally — unchanged.
Consumer audit: percentage — the only thing every sync round-trip
exchanges — is untouched on all paths; KOReader pulls on locator-less
rows already support the nil-CFI shape (builder guards
progress.Epubcfi.Valid); first-party restores go percentage-first
instead of floor-fighting a stale anchor.
Tests: TestProgressWeb_LocatorCoherence pins clear-on-declared,
request-carried-context survival, and merge-path preservation. Full
suite with the change: 218 groups, zero failures.
|
||
|
John O'Keefe
|
8c3273a0fc |
fix(sync): normalize character offsets to UTF-16 at the wire; refresh book offset on every verified save
Offset currency policy, now explicit: EPUB CFI terminals, CRE text() offsets and the served char_offset handle are UTF-16 code units (the EPUB CFI spec, and what foliate/readium/KOReader/Kobo clients actually observe), while internal arithmetic — the book-wide character_offset column and percentage fractions — stays rune-based, consistent with TotalCharacters. For all-BMP books the currencies are identical, so no stored value changes; astral-plane text (emoji, rare CJK) no longer drifts. Boundaries converted: resolveCFIToNode interprets incoming CFI terminal offsets as UTF-16; textNodeAtUTF16Offset (née textNodeAtRuneOffset) interprets CRE text() offsets as UTF-16; buildCFI and buildCREXPointer emit UTF-16 terminals; blockCharOffset (the served char_offset) is UTF-16. Also fixes two character_offset column defects: heals wrote a BLOCK- relative offset into the book-wide column, and verified-but-unhealed saves (e.g. KOReader pushes) never refreshed it, leaving it stale behind the anchor. VerifyProgressAnchor now returns the verified book- wide rune offset and SaveProgress refreshes the column on every verified save. Tests: astral currency round trip (offset after an emoji must shift by one unit between currencies, in both heal and exact-verify directions) and book-offset ordering. The cmd/server/tests integration harness failures under docker (library folder 400 during setup) reproduce on the pre-change tree and are unrelated. |
||
|
John O'Keefe
|
aec226af1a |
feat(sync): server-side position authority — verify/heal progress anchors
Progress submissions now carry (percentage, context_text, epubcfi) and the server becomes the position authority: - VerifyProgressAnchor resolves the submitted standard CFI against the book's own XHTML, extracts the text at the anchor, and cross-checks it with the submitted context_text. A mismatch heals the anchor by text search (percentage disambiguates repeats) instead of storing a bad position. - The anchor's block element is derived as a cssSelector plus a block- relative character offset, and served on progress GET alongside the anchor document's href — readium-native handles that let clients re-open a book without parsing CFIs themselves. - context_text-only submissions (no CFI — the dumb-client tier) are anchored structurally from the context text. Motivation: cross-client progress sync (web foliate CFIs, KOReader CRE xpointers, readium-native apps) previously trusted each client's own locator math; the app's EPUB restore drifted ±pages because readium's paginator does not lay out far-from-viewport columns and the foliate- ported CFI walk ran against readium's mutated WebView DOM. Server-side verification heals both classes at ingest. |
||
 john-okeefe
|
762181c123 |
fix(sync): treat page index as sole locator for fixed-layout pull path
The push path (koreader→server) was already updated to omit epubcfi
for paging documents, and SaveProgress derives percentage from
page/total_pages for fixed formats. However, the pull path
(server→koreader) still returned any stored epubcfi and attempted
CFI→XPointer conversion, and SaveProgress preserved stale CFI values
written by the web reader (which generates fake CFIs via
CFI.fake.fromIndex for every comic/PDF page).
These fake CFI strings lingered in the database and koreader's pull
path picked them up over progress.page, causing tonumber("epubcfi(...)")
→ nil → GotoPage(nil) → crash when the user confirmed the sync prompt.
Changes:
- GetMetadata (koreader.go): for fixed_layout/comic_archive formats,
skip returning epubcfi and skip the CFI→CRE XPointer conversion.
Reflowable documents are byte-for-byte unchanged.
- SaveProgress (progress.go): for fixed formats, explicitly clear
epubcfi and character_offset on every save so stale values from
prior web-reader sessions are cleaned up over time.
|
||
|
john-okeefe
|
423a1c0bf7 |
feat(sync): use page index as canonical locator for fixed-layout & comic formats
Fixed-layout EPUBs, PDFs, DjVu, and comic archives (cbz/cbr/cb7/cbt)
are page-based: each page is a fixed image, so a page index is an exact,
universal locator regardless of screen size or device. Sync previously
treated these like reflowable content (CFI-first restore, percentage
fallback, character-offset math), which was both wrong and lossy. This
makes the page index the canonical position for fixed-layout and comic
formats while leaving the reflowable path byte-for-byte unchanged.
Backend:
- progress.go SaveProgress: branch on mediaItem.FormatGroup. For
fixed_layout/comic_archive derive percentage from current_page/
total_pages and skip the CFI/character-offset back-fills (meaningless
for image content). The reflowable derivation block is preserved
verbatim under an else.
- kobo.go: Kobo only sends a percentage, so for fixed-layout/comic
formats derive CurrentPage via PercentageToPage(percentage, pageCount)
using the media item's known page count, so Kobo->web lands on the
exact page.
Reader:
- reader.templ readerInitExpr: pass savedPage/savedTotalPages to the
reader config for fixed_layout/comic_archive formats.
- reader.ts: when isFixedLayout and savedPage is present, restore via
view.init({ lastLocation: savedPage - 1 }) (a bare number navigates
foliate directly to the section index). Reflowable falls through to
the existing CFI->percentage path, unchanged.
|
||
|
john-okeefe
|
b2b1804aa3 |
feat(sync): plumb context_text through SaveProgress and sync queue
- SaveProgressRequest: add ContextText field - SaveProgress: carry existing ContextText from DB, overwrite when provided - ProgressUpdate: add ContextText field for checkpoint sync - SyncQueueProcessor: serialize/deserialize context_text in sync data - buildProgressSnapshot: include context_text in snapshot data |
||
|
john-okeefe
|
f699899408 |
feat(sync): add ProgressService with merge, enrichment, and conflict detection
Introduces a centralized ProgressService that handles all reading progress writes across web, KOReader, and Kobo clients. The service implements: - Merge strategy: reads existing progress first, then only overwrites non-nil fields from the incoming request. This fixes the data loss bug where partial updates (e.g., Kobo last-read-place sending only epubcfi and chapter) would NULL out percentage, character_offset, etc. - Enrichment: computes missing fields from available data: - character_offset from percentage + total_characters - current_page from percentage + total_pages - percentage from current_page + total_pages (reverse) - percentage from character_offset + total_characters (reverse) - Conflict detection: when a different source writes progress within 5 minutes with >1% difference, records a sync_conflicts row and broadcasts a WebSocket notification for real-time UI alerts. - Broadcast control: SaveProgressRequest.Broadcast flag lets Kobo last-read-place and SyncFromServer skip WebSocket broadcasts. - Pointer fields on SaveProgressRequest: nil means preserve existing, non-nil means overwrite. Eliminates ambiguity between zero values and not-provided fields. Also adds unit tests for buildProgressSnapshot helper function. |
||
|
john-okeefe
|
44ec2f496a |
feat(sync): add estimated pages calculation for reflowable ebooks
Reflowable ebooks (EPUBs) don't have inherent page numbers since layout depends on device settings. Add an EstimatedPages() function that converts total character count to an estimated print page count using the industry standard of 1800 characters per page. This provides a consistent, device-independent page count for progress display (e.g., "Page 89 of 196" for a reflowable EPUB), matching how KOReader and similar readers handle the same problem. |
||
|
john-okeefe
|
89b0b93ffc |
fix: correct JSON struct tags in ProgressData
Fix incorrect struct tags for Page, TotalPages, and PageY fields. Previously used 'int' tag instead of proper JSON field names, which would cause serialization issues. |
||
|
john-okeefe
|
bb3c32c59f |
Phase 1 Week 2: Format detection and progress conversion engine
- Add internal/sync package with format detection - FormatGroup types: reflowable, fixed_layout, comic_archive - DetectFormatGroup() function based on mimetype and file extension - MimeType mappings for common ebook formats - Progress conversion engine with: - ConvertProgress() between format groups - Extract percentage from various progress formats - PageToPercentage / PercentageToPage helpers - CharacterToPercentage / PercentageToCharacter helpers - MergeProgress() with 'max progress wins' strategy - FormatProgressForDisplay() for UI rendering - Add sqlc queries for format detection and progress updates - BulkUpdateFormatGroups query for auto-format detection - GetUniversalProgress query with all location references - UpdateUniversalProgress query with device sync metadata - ReadingHistory queries for session tracking |