John O'Keefe
6216bd9bcf
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
122 lines
8.9 KiB
Markdown
122 lines
8.9 KiB
Markdown
# Wire-contract pins (hardening batches)
|
|
|
|
These are the response SHAPES (JSON key sets, not values) frozen by the
|
|
`TestContractPin_*` suite in `cmd/server/tests/contract_pins_test.go`
|
|
(batch H0). The plan they serve is `~/.opencode/plan/hardening-canonical-types.md`.
|
|
|
|
## Rules
|
|
|
|
1. **Frozen**: no existing response key may be RENAMED or REMOVED on any
|
|
endpoint below (JSON or KOReader/Kobo adapters). Requests are frozen too —
|
|
no parsing changes.
|
|
2. **Additive keys are allowed and expected** — but a batch must widen the
|
|
`allowed` slice in the test AND record the addition in the changelog
|
|
below, in the same commit.
|
|
3. KOReader / Kobo / progress surfaces are the hard freeze: the KOReader
|
|
plugin is an external client whose handlers keep response shapes
|
|
byte-identical (only their internals may route through shared funnels).
|
|
4. A behavior fix that would require narrowing or REMOVING a pinned key
|
|
needs the user's explicit sign-off before the pin is touched — a pin
|
|
freezing buggy behavior is a known-bugs ledger, not a license.
|
|
|
|
## Pinned surfaces
|
|
|
|
| Test | Surface | Shape |
|
|
|---|---|---|
|
|
| `ProgressWeb` | `GET /api/media-items/:id/progress` (percentage-only write) | flat map: `id, media_item_id, user_id, current_page, total_pages, last_read_at, percentage, character_offset, epubcfi, chapter, chapter_progress, format_group, total_characters, chapter_count, last_sync_device, last_sync_source, last_sync_timestamp` |
|
|
| | … conditional bolt-ons | `context_text` (when stored valid); `css_selector`, `anchor_href`, `char_offset`, rewritten `epubcfi` (when the anchor verify path fires) |
|
|
| | `PUT /api/media-items/:id/progress` response | `database.ReadingProgress` JSON (25 keys incl. `audio_offset_seconds`, viewport/zoom/scroll/panel/reading_mode) |
|
|
| | `GET` with no rows | exactly `{current_page, total_pages}` |
|
|
| `KOReaderProgressPush` | `POST /api/sync/koreader/progress` (immediate, 202) | `sync_status, books_synced, book_results[]?{sha256, book_uuid, synced}, conflicts[]?{book_uuid, conflict_type, device_progress, server_progress, resolution}, timestamp, device_updated` |
|
|
| `KOReaderProgressPull` | `GET /api/sync/koreader/metadata/:uuid` | `uuid, sha256?, title, authors, progress{percentage + optional character/epubcfi/koreader_xpointer/chapter/chapter_progress/page/total_pages}, annotations{highlights?, notes?, bookmarks?, deleted_highlights?, deleted_bookmarks?}, last_sync` |
|
|
| `KOReaderLibrary` | `GET /api/sync/koreader/library` | `library_sync[]{uuid, sha256?, title, author, content_type, percent_read, pages_remaining?, bookmark_count, last_modified}, total_books, last_sync` |
|
|
| `KOReaderBookmarksPush` | `POST /api/sync/koreader/bookmarks` | `sync_status, bookmarks_synced, notes_synced, highlights_synced, total_synced, timestamp` |
|
|
| `KoboInitialization` | `GET /api/sync/kobo/:token/v1/initialization` | `library_sync[]{ContentId, ContentType, Title, Author, PercentRead, PagesRemaining?, BookmarkCount, LastModified, EntitlementId, Revision, MimeType, FileSize, Categories?, BookhoardUUID?}, total_books, last_sync` |
|
|
| `KoboMarkupSync` | `POST /api/sync/kobo/:token/markup` | `Status, MarkupsSynced, BookmarksSynced, DeletedAnnotations?` |
|
|
| `KoboSyncFromServer` | `POST /api/sync/kobo/:token/sync-from-server` | `BooksSynced, BookmarksSent, HighlightsSent` |
|
|
| `CollectionsJSON` | `GET /api/collections/:id` (all 4 branches) | envelope `id, user_id, name, description, color, icon, auto_assign_rules, view_settings, books[], book_count, is_smart, created_at`; books = `BookInfo` |
|
|
| | `POST /api/collections/preview` | bare `{items: BookInfo[]}` |
|
|
| `SeriesJSON` | `GET /api/series/books?name=&library_id=` | `{name, library_id, books: BookInfo[], total}` |
|
|
| `SearchJSON` | `GET /api/media-items/search?q=&library_id=` | top-level array of `SearchMediaItemsUnifiedRow` (47 keys) |
|
|
| `DashboardSections` | `GET /api/dashboard/sections` | `{sections[]: {id, collection_id, is_system, title, description, icon, items: BookInfo[], view_all_url, priority}}` |
|
|
| `MediaListBothBranches` | `GET /api/media-items` (unscoped) | `{data: [38-key resolved map]}` (intentionally divergent from the scoped branch until the row-shape collapse batch) |
|
|
| | `GET /api/media-items?library_id=&sort=` | `{data: [raw ListMediaItemsSortedRow, 70 sqlc keys]}` |
|
|
| `SSRPages` | `/bookshelf`, `/collections`, `/collections/:id`, `/series`, `/series/detail`, `/tags/detail` | 200 + stable title/substring markers |
|
|
|
|
`BookInfo` (every card surface): `media_item_id, title, author,
|
|
cover_image_path, has_conflict, format_group, badge`.
|
|
|
|
## Changelog (additive keys per batch)
|
|
|
|
- **H0 (initial pin)** — the shapes above, captured on the unmodified tree
|
|
at commit `b4ae423`.
|
|
- **H1 (progress canonicalization)** — `GET /api/media-items/:id/progress`
|
|
Whitelisted ADDITIVE keys (fields the `GetUniversalProgressRow` already
|
|
carries, now surfaced by the shared row serializer):
|
|
`audio_offset_seconds, reading_mode, viewport_x, viewport_y, zoom_level,
|
|
scroll_position_x, scroll_position_y, panel_number`. No key removed or
|
|
renamed; the no-rows and PUT-response shapes are untouched. The web PUT
|
|
binds `SaveProgressRequest` directly — its parse set for existing
|
|
clients is IDENTICAL to the old DTO (verified against web/src
|
|
reader.ts:3113-3143 and audio.ts:518-520); keys the web never sent
|
|
(viewport_x/y, panel_number) merely become honor-able. All other pins
|
|
unchanged.
|
|
- **H3-prep (web annotation CRUD freeze)** — NEW pinned surfaces, captured
|
|
pre-refactor per the pin-first doctrine: `POST/PUT/DELETE
|
|
/api/media-items/:id/{notes,highlights,bookmarks}` — create 201 / update
|
|
200 / delete 204, response bodies are the full sqlc row models
|
|
(MediaNotes 19 keys, MediaBookmarks 20 keys, MediaHighlights 27 keys —
|
|
listed in contract_pins_test.go). No keys removed; nothing was pinned
|
|
here before. Capturing these immediately caught a LIVE server bug:
|
|
`applyNoteLWW` omitted `UserID` from its
|
|
`UpdateMediaNoteForSync` params, so every dedup-hit note re-save
|
|
(same content+position ⇒ same key) returned 500
|
|
("update note: no rows in result set") — the LWW update could never
|
|
land. Fixed in the same commit (server-side fix, freely allowed);
|
|
the pins are its regression net.
|
|
- **H3 (PUT fixes routed through the sync service)** — `PUT
|
|
/api/media-items/:id/bookmarks/:bookmarkId` now parses ADDITIONAL
|
|
optional request keys mirroring its create sibling:
|
|
`cfi_position, page_number, chapter_number, percentage,
|
|
chapter_reference` (all zero-value-preserving: omitted keys coalesce
|
|
with the stored row inside the service, so legacy clients that send
|
|
only title/notes/position keep every stored locator — verified by
|
|
TestH3_BookmarkPUTPreservesLocators). The PUT lands as an id-targeted
|
|
LWW edit under the row's stored dedup key (BookmarkID/NoteID mirror
|
|
SaveHighlightRequest.HighlightID), so `last_modified_source` /
|
|
`last_modified_at` are now maintained. Note PUT unchanged on the wire
|
|
(content+position) but routed through the service the same way.
|
|
Response shapes unchanged (same row models — pins green).
|
|
- **H4 (unified row completion)** — `/api/media-items/search` rows[]
|
|
whitelisted ADDITIVE keys `abridged, audio_metadata`: the unified
|
|
search row gained the two columns it lagged (EXPLAIN plan-identical —
|
|
both ride the already-scanned media_items side), and the endpoint
|
|
serializes the row directly, so they emit. No key removed or renamed.
|
|
- **Origin writer attribution (post-hardening parity)** — every web-API
|
|
write endpoint now parses an ADDITIVE optional request key `origin`
|
|
("android" | "ios"; anything else stays "web") and records it as the
|
|
writer: progress PUT → `last_sync_source`, bookmark/note/highlight
|
|
create+PUT → `last_modified_source` (and `origin_source` on bookmark
|
|
creates, now normalized through the same gate). Precedence on progress:
|
|
a paired X-Device-Token still outranks origin. The web SPA sends no
|
|
origin — its rows are unchanged. Request-parse widening documented per
|
|
rule 2; response shapes untouched; locked by
|
|
TestOriginWriterAttribution.
|
|
|
|
## Changelog entry — multi-file audiobooks (2026-10-18)
|
|
|
|
- `GET /api/media-items/:id/chapters` gained ADDITIVE response keys
|
|
`layout` ("" for single-file books, `"folder"` or `"zip"` for
|
|
multi-file books) and `tracks[]` (`index, url, title, path, start,
|
|
end, duration_seconds, size_bytes`; empty for single-file books).
|
|
Existing keys (`media_item_id, chapters, duration_seconds,
|
|
codec_profile`) unchanged — no pin existed on this endpoint, so no
|
|
pin widened; recorded here per rule 2's spirit.
|
|
- New sibling endpoint `GET /api/media-items/:id/audio/:trackIndex`
|
|
streams one track of a multi-file book (Range-capable). New surface,
|
|
not a modification; gated by library visibility like `ServeFile`
|
|
(403 on hidden libraries).
|
|
- `media_items.audio_metadata` payload gained additive JSON keys
|
|
`layout` and `tracks` (see schema.sql comment). Consumers parse
|
|
additively; stored single-file payloads are unchanged.
|