John O'Keefe
1dbb0ce406
Phase 2b + 3 of the metadata overhaul (plan §2.4, §3.1-3.7). Auto-playlist collections (§2.4): the scanner turns obviously-playlist sidecar series (explicit '(Reading Order)'/chronological/publication/ in-order/timeline markers only — precision over recall) into curated static collections: upsert by view_settings.imported_from_series then name (rename-safe), position set ONLY on insert (clean integers; 1.5/1a stay unpositioned rather than guessing), membership add-if-missing, never removed or re-ordered by rescans, provenance in description + view_settings, show_on_dashboard true (dashboard-eligible by design), 'scanner.auto_playlist_collections' setting (default on, Settings Registry + schema seed). Ordering (§3.1-3.2): collection listings order position ASC NULLS LAST then added_at ASC (was added_at DESC — walked playlists backwards); PUT /api/collections/:id/items/order applies full-order rewrites (owner-only, service-gated). hide_finished + Up Next (§3.3-3.4): GetCollectionItemsForDashboard gains viewer_user_id + hide_finished (LEFT JOIN reading_progress, COALESCE percentage < 0.999) and viewer_percentage rides the row for card graying; GetCollectionUpNext serves the collection's reading-queue head with the family-lens visible-library array enforced. Dashboard (§3.5): user collections are dashboard-ELIGIBLE BY DEFAULT — the show_on_dashboard gate (never writable for user rows since the column shipped) is gone from GetUserCollectionsForDashboard; hiding stays dashboard-prefs (hidden_collections) territory; hide_finished filtered-everything now emits a Complete section instead of silently vanishing; sections JSON carries is_playlist/complete. Series page ordering hardening (§3.6): GetSeriesBooks tiebreaks by title ASC — NULL/tied numbers previously returned heap (scan) order, which is what the user saw. Frontend custom-section-builder note: the builder has sent show_on_dashboard: true since its creation; the API never bound it. User-row visibility is now intentionally prefs-governed, so the field stays unbound by design (documented in code). Pins widened (docs/contract_pins.md): collection detail + up_next; dashboard sections + is_playlist/complete; new reorder endpoint recorded. Verified: TestPlaylistCollectionsEndToEnd (new, full e2e: sidecar series split, collection auto-creation with position 7, rescan idempotency, owner-only reorder, finished-filter + completion marker + Up Next drain) plus the full affected integration sweep green.
155 lines
11 KiB
Markdown
155 lines
11 KiB
Markdown
# Wire-contract pins (hardening batches)
|
|
|
|
These are the response SHAPES (JSON key sets, not values) frozen by the
|
|
`TestContractPin_*` suite in `cmd/server/tests/contract_pins_test.go`
|
|
(batch H0). The plan they serve is `~/.opencode/plan/hardening-canonical-types.md`.
|
|
|
|
## Rules
|
|
|
|
1. **Frozen**: no existing response key may be RENAMED or REMOVED on any
|
|
endpoint below (JSON or KOReader/Kobo adapters). Requests are frozen too —
|
|
no parsing changes.
|
|
2. **Additive keys are allowed and expected** — but a batch must widen the
|
|
`allowed` slice in the test AND record the addition in the changelog
|
|
below, in the same commit.
|
|
3. KOReader / Kobo / progress surfaces are the hard freeze: the KOReader
|
|
plugin is an external client whose handlers keep response shapes
|
|
byte-identical (only their internals may route through shared funnels).
|
|
4. A behavior fix that would require narrowing or REMOVING a pinned key
|
|
needs the user's explicit sign-off before the pin is touched — a pin
|
|
freezing buggy behavior is a known-bugs ledger, not a license.
|
|
|
|
## Pinned surfaces
|
|
|
|
| Test | Surface | Shape |
|
|
|---|---|---|
|
|
| `ProgressWeb` | `GET /api/media-items/:id/progress` (percentage-only write) | flat map: `id, media_item_id, user_id, current_page, total_pages, last_read_at, percentage, character_offset, epubcfi, chapter, chapter_progress, format_group, total_characters, chapter_count, last_sync_device, last_sync_source, last_sync_timestamp` |
|
|
| | … conditional bolt-ons | `context_text` (when stored valid); `css_selector`, `anchor_href`, `char_offset`, rewritten `epubcfi` (when the anchor verify path fires) |
|
|
| | `PUT /api/media-items/:id/progress` response | `database.ReadingProgress` JSON (25 keys incl. `audio_offset_seconds`, viewport/zoom/scroll/panel/reading_mode) |
|
|
| | `GET` with no rows | exactly `{current_page, total_pages}` |
|
|
| `KOReaderProgressPush` | `POST /api/sync/koreader/progress` (immediate, 202) | `sync_status, books_synced, book_results[]?{sha256, book_uuid, synced}, conflicts[]?{book_uuid, conflict_type, device_progress, server_progress, resolution}, timestamp, device_updated` |
|
|
| `KOReaderProgressPull` | `GET /api/sync/koreader/metadata/:uuid` | `uuid, sha256?, title, authors, progress{percentage + optional character/epubcfi/koreader_xpointer/chapter/chapter_progress/page/total_pages}, annotations{highlights?, notes?, bookmarks?, deleted_highlights?, deleted_bookmarks?}, last_sync` |
|
|
| `KOReaderLibrary` | `GET /api/sync/koreader/library` | `library_sync[]{uuid, sha256?, title, author, content_type, percent_read, pages_remaining?, bookmark_count, last_modified}, total_books, last_sync` |
|
|
| `KOReaderBookmarksPush` | `POST /api/sync/koreader/bookmarks` | `sync_status, bookmarks_synced, notes_synced, highlights_synced, total_synced, timestamp` |
|
|
| `KoboInitialization` | `GET /api/sync/kobo/:token/v1/initialization` | `library_sync[]{ContentId, ContentType, Title, Author, PercentRead, PagesRemaining?, BookmarkCount, LastModified, EntitlementId, Revision, MimeType, FileSize, Categories?, BookhoardUUID?}, total_books, last_sync` |
|
|
| `KoboMarkupSync` | `POST /api/sync/kobo/:token/markup` | `Status, MarkupsSynced, BookmarksSynced, DeletedAnnotations?` |
|
|
| `KoboSyncFromServer` | `POST /api/sync/kobo/:token/sync-from-server` | `BooksSynced, BookmarksSent, HighlightsSent` |
|
|
| `CollectionsJSON` | `GET /api/collections/:id` (all 4 branches) | envelope `id, user_id, name, description, color, icon, auto_assign_rules, view_settings, books[], book_count, is_smart, created_at`; books = `BookInfo` |
|
|
| | `POST /api/collections/preview` | bare `{items: BookInfo[]}` |
|
|
| `SeriesJSON` | `GET /api/series/books?name=&library_id=` | `{name, library_id, books: BookInfo[], total}` |
|
|
| `SearchJSON` | `GET /api/media-items/search?q=&library_id=` | top-level array of `SearchMediaItemsUnifiedRow` (47 keys) |
|
|
| `DashboardSections` | `GET /api/dashboard/sections` | `{sections[]: {id, collection_id, is_system, title, description, icon, items: BookInfo[], view_all_url, priority}}` |
|
|
| `MediaListBothBranches` | `GET /api/media-items` (unscoped) | `{data: [38-key resolved map]}` (intentionally divergent from the scoped branch until the row-shape collapse batch) |
|
|
| | `GET /api/media-items?library_id=&sort=` | `{data: [raw ListMediaItemsSortedRow, 70 sqlc keys]}` |
|
|
| `SSRPages` | `/bookshelf`, `/collections`, `/collections/:id`, `/series`, `/series/detail`, `/tags/detail` | 200 + stable title/substring markers |
|
|
|
|
`BookInfo` (every card surface): `media_item_id, title, author,
|
|
cover_image_path, has_conflict, format_group, badge`.
|
|
|
|
## Changelog (additive keys per batch)
|
|
|
|
- **H0 (initial pin)** — the shapes above, captured on the unmodified tree
|
|
at commit `b4ae423`.
|
|
- **H1 (progress canonicalization)** — `GET /api/media-items/:id/progress`
|
|
Whitelisted ADDITIVE keys (fields the `GetUniversalProgressRow` already
|
|
carries, now surfaced by the shared row serializer):
|
|
`audio_offset_seconds, reading_mode, viewport_x, viewport_y, zoom_level,
|
|
scroll_position_x, scroll_position_y, panel_number`. No key removed or
|
|
renamed; the no-rows and PUT-response shapes are untouched. The web PUT
|
|
binds `SaveProgressRequest` directly — its parse set for existing
|
|
clients is IDENTICAL to the old DTO (verified against web/src
|
|
reader.ts:3113-3143 and audio.ts:518-520); keys the web never sent
|
|
(viewport_x/y, panel_number) merely become honor-able. All other pins
|
|
unchanged.
|
|
- **H3-prep (web annotation CRUD freeze)** — NEW pinned surfaces, captured
|
|
pre-refactor per the pin-first doctrine: `POST/PUT/DELETE
|
|
/api/media-items/:id/{notes,highlights,bookmarks}` — create 201 / update
|
|
200 / delete 204, response bodies are the full sqlc row models
|
|
(MediaNotes 19 keys, MediaBookmarks 20 keys, MediaHighlights 27 keys —
|
|
listed in contract_pins_test.go). No keys removed; nothing was pinned
|
|
here before. Capturing these immediately caught a LIVE server bug:
|
|
`applyNoteLWW` omitted `UserID` from its
|
|
`UpdateMediaNoteForSync` params, so every dedup-hit note re-save
|
|
(same content+position ⇒ same key) returned 500
|
|
("update note: no rows in result set") — the LWW update could never
|
|
land. Fixed in the same commit (server-side fix, freely allowed);
|
|
the pins are its regression net.
|
|
- **H3 (PUT fixes routed through the sync service)** — `PUT
|
|
/api/media-items/:id/bookmarks/:bookmarkId` now parses ADDITIONAL
|
|
optional request keys mirroring its create sibling:
|
|
`cfi_position, page_number, chapter_number, percentage,
|
|
chapter_reference` (all zero-value-preserving: omitted keys coalesce
|
|
with the stored row inside the service, so legacy clients that send
|
|
only title/notes/position keep every stored locator — verified by
|
|
TestH3_BookmarkPUTPreservesLocators). The PUT lands as an id-targeted
|
|
LWW edit under the row's stored dedup key (BookmarkID/NoteID mirror
|
|
SaveHighlightRequest.HighlightID), so `last_modified_source` /
|
|
`last_modified_at` are now maintained. Note PUT unchanged on the wire
|
|
(content+position) but routed through the service the same way.
|
|
Response shapes unchanged (same row models — pins green).
|
|
- **H4 (unified row completion)** — `/api/media-items/search` rows[]
|
|
whitelisted ADDITIVE keys `abridged, audio_metadata`: the unified
|
|
search row gained the two columns it lagged (EXPLAIN plan-identical —
|
|
both ride the already-scanned media_items side), and the endpoint
|
|
serializes the row directly, so they emit. No key removed or renamed.
|
|
- **Origin writer attribution (post-hardening parity)** — every web-API
|
|
write endpoint now parses an ADDITIVE optional request key `origin`
|
|
("android" | "ios"; anything else stays "web") and records it as the
|
|
writer: progress PUT → `last_sync_source`, bookmark/note/highlight
|
|
create+PUT → `last_modified_source` (and `origin_source` on bookmark
|
|
creates, now normalized through the same gate). Precedence on progress:
|
|
a paired X-Device-Token still outranks origin. The web SPA sends no
|
|
origin — its rows are unchanged. Request-parse widening documented per
|
|
rule 2; response shapes untouched; locked by
|
|
TestOriginWriterAttribution.
|
|
|
|
## Changelog entry — multi-file audiobooks (2026-10-18)
|
|
|
|
- `GET /api/media-items/:id/chapters` gained ADDITIVE response keys
|
|
`layout` ("" for single-file books, `"folder"` or `"zip"` for
|
|
multi-file books) and `tracks[]` (`index, url, title, path, start,
|
|
end, duration_seconds, size_bytes`; empty for single-file books).
|
|
Existing keys (`media_item_id, chapters, duration_seconds,
|
|
codec_profile`) unchanged — no pin existed on this endpoint, so no
|
|
pin widened; recorded here per rule 2's spirit.
|
|
- New sibling endpoint `GET /api/media-items/:id/audio/:trackIndex`
|
|
streams one track of a multi-file book (Range-capable). New surface,
|
|
not a modification; gated by library visibility like `ServeFile`
|
|
(403 on hidden libraries).
|
|
- `media_items.audio_metadata` payload gained additive JSON keys
|
|
`layout` and `tracks` (see schema.sql comment). Consumers parse
|
|
additively; stored single-file payloads are unchanged.
|
|
|
|
## 2026-10-19 — A2b: metadata array/model overhaul (phase 1)
|
|
|
|
- `media_items.series_number` widened INTEGER → DOUBLE PRECISION across all
|
|
surfaces. JSON serializers emit the same key; values are JSON numbers, so
|
|
integer-valued ordinals are byte-identical ("3") and decimal ordinals
|
|
(0.5/1.5) now serialize as decimals instead of truncating. Clients reading
|
|
JSON numbers are unaffected; the Android client parses either form.
|
|
- `GET /api/media-items*` / detail responses gain ADDITIVE keys:
|
|
`subtitle` (string), `explicit` (bool), `authors` ([]string, write truth —
|
|
legacy `author` display string still emitted, now trigger-regenerated),
|
|
`publishers` ([]string; legacy `publisher` likewise).
|
|
- `POST/PUT /api/media-items*` accept ADDITIVE request keys `subtitle`,
|
|
`explicit`, `authors`, `publishers`; legacy `author`/`publisher` singles
|
|
still accepted (arrays win when both present).
|
|
- Override keys unchanged: `author`/`publisher` override names are preserved
|
|
for existing rows and now protect the arrays.
|
|
|
|
## 2026-10-19 — A2c: collections/playlists phase
|
|
|
|
- `GET /api/collections/:id` gained ADDITIVE `up_next` (array of
|
|
`{id, title, author, cover, format, progress}` — the static collection's
|
|
reading queue head in position order, viewer-progress-aware; empty array
|
|
for smart collections).
|
|
- `GET /api/dashboard/sections` rows gained ADDITIVE `is_playlist` (bool —
|
|
collection was imported from a sidecar ordering construct) and
|
|
`complete` (bool — hide_finished filtered every item out of a non-empty
|
|
collection; render "playlist complete").
|
|
- `PUT /api/collections/:id/items/order` is a NEW endpoint (owner-only,
|
|
full ordered id list → positions 1..n).
|
|
- User collections are dashboard-eligible by default (the
|
|
show_on_dashboard gate is gone from GetUserCollectionsForDashboard);
|
|
hiding stays dashboard-prefs (`hidden_collections`) territory.
|