1dbb0ce406bc5d765cebe80930e2500301f0f7f1
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 John O'Keefe
|
1dbb0ce406 |
feat(collections): playlist auto-creation, position ordering, hide_finished, Up Next, dashboard eligibility
Phase 2b + 3 of the metadata overhaul (plan §2.4, §3.1-3.7). Auto-playlist collections (§2.4): the scanner turns obviously-playlist sidecar series (explicit '(Reading Order)'/chronological/publication/ in-order/timeline markers only — precision over recall) into curated static collections: upsert by view_settings.imported_from_series then name (rename-safe), position set ONLY on insert (clean integers; 1.5/1a stay unpositioned rather than guessing), membership add-if-missing, never removed or re-ordered by rescans, provenance in description + view_settings, show_on_dashboard true (dashboard-eligible by design), 'scanner.auto_playlist_collections' setting (default on, Settings Registry + schema seed). Ordering (§3.1-3.2): collection listings order position ASC NULLS LAST then added_at ASC (was added_at DESC — walked playlists backwards); PUT /api/collections/:id/items/order applies full-order rewrites (owner-only, service-gated). hide_finished + Up Next (§3.3-3.4): GetCollectionItemsForDashboard gains viewer_user_id + hide_finished (LEFT JOIN reading_progress, COALESCE percentage < 0.999) and viewer_percentage rides the row for card graying; GetCollectionUpNext serves the collection's reading-queue head with the family-lens visible-library array enforced. Dashboard (§3.5): user collections are dashboard-ELIGIBLE BY DEFAULT — the show_on_dashboard gate (never writable for user rows since the column shipped) is gone from GetUserCollectionsForDashboard; hiding stays dashboard-prefs (hidden_collections) territory; hide_finished filtered-everything now emits a Complete section instead of silently vanishing; sections JSON carries is_playlist/complete. Series page ordering hardening (§3.6): GetSeriesBooks tiebreaks by title ASC — NULL/tied numbers previously returned heap (scan) order, which is what the user saw. Frontend custom-section-builder note: the builder has sent show_on_dashboard: true since its creation; the API never bound it. User-row visibility is now intentionally prefs-governed, so the field stays unbound by design (documented in code). Pins widened (docs/contract_pins.md): collection detail + up_next; dashboard sections + is_playlist/complete; new reorder endpoint recorded. Verified: TestPlaylistCollectionsEndToEnd (new, full e2e: sidecar series split, collection auto-creation with position 7, rescan idempotency, owner-only reorder, finished-filter + completion marker + Up Next drain) plus the full affected integration sweep green. |
||
|
John O'Keefe
|
646477afad |
feat(schema): subtitle/explicit columns, authors+publishers arrays, float series numbers
Phase 1 of the series/collections/metadata overhaul
(~/.opencode/plan/series-collections-metadata-overhaul.md).
Schema (in place, idempotent):
- media_items.series_number INTEGER -> DOUBLE PRECISION: real-world
ordinals are decimals (0.5, 1.5 novellas); INTEGER silently truncated
them (calibre 2.5 pinned as 2 in tests).
- + subtitle TEXT, explicit BOOLEAN (sidecar ingest; ABS explicit /
Audible isAdult).
- + authors/authors_search/publishers/publishers_search TEXT[] NOT NULL
(narrators A2a precedent). Arrays are write truth; author/publisher
varchars become trigger-regenerated display strings
(sync_author_publisher_display) so every varchar consumer (trgm
search, facets, API, OPDS/Kobo) keeps working unchanged.
- Row-level NULL normalization for the new arrays in
set_library_type_name (A2 lesson: Go nil -> explicit NULL -> 23502).
- Backfills ARRAY[author]/ARRAY[publisher] — deliberately no
comma-splitting ('Tolkien, J.R.R.' would corrupt; scanner kept only
authors[0] historically).
- collection_items.position INTEGER + (collection_id, position) index
for playlist ordering.
Go: sqlc regen; MediaMetadata/handler params gain Subtitle/Explicit/
Authors/Publishers (legacy singles still accepted, arrays win);
SeriesNumber int32 -> float64 end to end; metadata_overrides adapted
with legacy override keys ('author'/'publisher') preserved so existing
user overrides survive; bulk author edit writes the array.
Pins widened (docs/contract_pins.md): raw sqlc row key set + the six
new columns; changelog entry added.
Verified: unit suite green; integration sweep (bulk/smart collections/
scanner/stats/sync/queue/calibre/pins/audiobooks) green against the
shared dev DB after in-place migration. Pre-existing unrelated red:
TestOwnershipAndAdminTestSuite/TestAdminCreateUser timezone inheritance
fails identically at the pre-change commit (live shared state).
|
||
|
John O'Keefe
|
6216bd9bcf |
feat(audiobooks): multi-file books (folder-of-tracks + zip) + exhaustive metadata ingestion
Multi-file book ingest: a directory of ≥2 audio files (recursively —
disc-part folders like CD1/Disc 2 merge upward, siblings never merge)
ingests as ONE book whose file_path is the book-root directory; a zip
of audio tracks is its own book (file_path = the archive). Both write
the documented chapters abstraction {index,title,start,end} into
audio_metadata and add additive keys: layout ("folder"|"zip") and a
tracks[] manifest (per-track path, global-seconds start/end, duration,
size). Pre-grouping one-track strays under a book root are absorbed on
rescan (annotations cascade). Rescans take a stat-only fast path
(track count + total size + known durations unchanged → no re-probe —
a 1189-track book must not re-ffprobe every auto-scan); a zero-duration
book counts as changed so a ffprobe-equipped rescan fills durations in.
Folder-book roots count as present in the archive lifecycle pass, and
per-item rescan (RescanMediaItem) routes multi-file books through a
book-level refresh — the generic updater would read the root/zip as if
it were the book file and clobber format_group with "unknown".
Serving: GET /api/media-items/:id/chapters gains additive layout +
tracks keys (urls → the new endpoint). New GET
/api/media-items/:id/audio/:trackIndex streams one track — folder
tracks served from the real file (full Range via ServeContent), zip
tracks via a reopen-and-skip ReadSeeker over the entry (deflate has no
random access; acceptable at audio bitrates; stored entries stay
section readers). Both endpoints gate on library visibility (403 on
hidden libraries — family-lens rule). ServeFile and OPDS DownloadBook
refuse directory paths (folder books have no single-file download).
Web player (next commit) consumes the manifest; progress/bookmarks/
deep-links keep their global-seconds coordinates, so the server-side
progress model is untouched. sync learns FormatGroupAudiobook
(percentage passthrough in ConvertProgress); detectFormatType stops
reporting "unknown" for audio.
Metadata ingestion (Batch B) — read everything available, degrade
gracefully, documented precedence sidecars → folder structure →
embedded tags → filename: ffprobe tags are now a case-insensitive
lookup map (freeform keys pass through verbatim); composer fills
narrators (Audible's own convention, kept as ONE string — no reliable
multi-narrator delimiter); grouping + Audible freeform atoms
(SERIES/SERIES-PART/SUBTITLE/ASIN/ABRIDGED) fill series/index/
subtitle-join/ASIN/abridged (explicit sidecar abridged-false is
guarded); date/genre fall back to tags; album/album_artist/artist
fill title/author. Genre persists at create for all formats.
Embedded cover art (mp3 APIC, m4b covr, flac picture) extracts via one
optional ffmpeg call as a "<book>.cover.jpg" sibling — sidecar covers
win. CUE sheets (FILE-matched, INDEX 01 at 75fps) are the lowest-
priority chapter source under sidecars and embedded m4b chapters. A
lone m3u/m3u8 in a book root imposes playlist ordering + #EXTINF
names; (disc, track) tags reorder tracks when coverage is complete.
extractMetadata's default branch no longer pre-fills the filename as
title (tags must outrank it) — final filename fallbacks now exist on
BOTH the create and rescan paths (TestRescanKeepsFilenameTitle pins
that rescans cannot blank a title); tagless zip books take the zip's
own name, not their parent folder's.
Accepted containers widen to .m4b .m4a .mp3 .flac .zip .opus .ogg .oga
.aac .wav with real media types registered (audio/ogg per RFC 7845 for
Opus-in-Ogg) — all browser-playable; .wma deliberately excluded.
Tests: unit tests for CUE parsing, playlist hints, tag fallbacks
(synthetic ffprobe reports — ffmpeg cannot write Audible freeform
atoms), ordinal parsing, tag ordering, natural sort, grouping rules;
integration suite (11 tests) over the live dev DB: folder/zip ingest +
idempotency, stray absorption, chapters manifest, per-track streaming
with Range, hidden-library 403s, embedded covers + sidecar-wins, CUE
chapters, embedded m4b chapters, playlist ordering, tag fallbacks, all
five new containers with content-type assertions, rescan title pin.
Fixture media is real committed-format binary content living untracked
under internal/services/testdata/ (gitignored by policy; recipes in
docs/contributing/development.md → Media fixtures). Test image now
ships ffmpeg to match the runtime's audio toolchain.
|
||
|
John O'Keefe
|
45b8542e25 |
fix(api): honor the request's origin as the writer label — app rows stop wearing "web"
Release / build-and-push (push) Successful in 3m36s
Every web-API write endpoint hardcoded Source:"web", so the Android app's own pushes landed with last_sync_source/last_modified_source "web" — the app was indistinguishable from the web SPA in progress conflicts, device_sync_data, and annotation bookkeeping. The app CAN say who it is (bookmark create already carried origin → origin_source); the server just ignored it for labeling. All six write sites (progress PUT, bookmark/note/highlight create+PUT) now parse an ADDITIVE optional "origin" key, normalized through one gate: "android"/"ios" pass through, anything absent/unknown stays "web" (the label keys conflict claims and device_sync_data — free-form client strings would pollute them). Precedence on progress: a paired X-Device-Token still outranks origin. Bookmark origin_source now rides the same gate. The web SPA sends no origin — its rows are unchanged; KOReader/Kobo handlers keep their own labels. Wire discipline: additive request keys only (rule 2), response shapes untouched, pins changelog widened in-commit, locked by TestOriginWriterAttribution (android lands, default stays web, unknown normalizes). Full battery: only the two known pre-existing failures (TestOPDSLibraryFolders rot, TestWebSocketPingPong — both reproduce with the change stashed). |
||
|
John O'Keefe
|
010c0cb287 |
test(contracts): whitelist the unified row's two new keys on search rows
The H4 row-completion commit made abridged/audio_metadata emit on the search wire (the endpoint serializes the row directly) — exactly what the pin exists to surface. Whitelisted per the additive rule with the changelog entry. |
||
|
John O'Keefe
|
7d698ea0d8 |
fix(annotations): bookmark and note PUTs land through the sync service (LWW by id)
H3 census bugs #2 and #3: both web PUTs used plain queries that bypassed the sync-aware layer entirely — no LWW, no source bookkeeping, and locator fields the client couldn't even send. The highlight PUT was the only one already routed (via HighlightID); the other two now mirror it: - SaveBookmarkRequest gains BookmarkID, SaveNoteRequest gains NoteID: id-targeted resolution reads the exact row, verifies ownership, LWWs under its stored dedup key, and resurrects tombstones when the edit is newer (same doctrine and comment structure as the highlight path). - Fields the PUT omits are coalesced from the stored row before the full-column sync update — the legacy web PUT sends only title/notes/position, and zeroing cfi/percentage/page on every edit would have been data loss. Verified both directions by TestH3_BookmarkPUTPreservesLocators (omitted survives, explicit lands). - The bookmark PUT's wire DTO gains the create endpoint's locator keys as optional fields (documented additive request keys in contract_pins.md); the note PUT keeps its content+position wire. UpdateMediaBookmark/UpdateMediaNote plain queries are now uncalled (the generated queries stay until the H4 query sweep). |
||
|
John O'Keefe
|
ff029f1ad5 |
fix(annotations): note LWW re-save 500'd — UserID missing from the sync update params
Pin-first batch for H3: freeze web annotation CRUD response shapes (create 201 / update 200 / delete 204; full sqlc row models) before the stack consolidation touches anything. The new pins immediately caught a live server bug the 28 unit tests could never see (annotations_test.go is pure-unit: keys and compare logic, no DB): applyNoteLWW built UpdateMediaNoteForSyncParams without UserID, so the query's WHERE id=$1 AND user_id=$13 compared against zero — every dedup-hit note re-save (same content+position) returned 500 "update note: no rows in result set" and the update never landed. The bookmark twin passes UserID and works; the note stack is the clone that dropped the field. Fix: pass UserID in the params (one line). Pins go green and become the regression net; changelog records the new surfaces. |
||
|
John O'Keefe
|
de9e66636a |
docs(agents): test-failure triage discipline — suspect the test first, but test edits stay asymmetric
2026-10-17 lessons from the hardening session: most test/bruno failures were harness/schema/environment drift (drifted body keys, truncated payloads, container-only paths), found by triaging the test before the server. Codified as: triage order (test schema -> environment -> cheap probe), with the guard against "writing tests to work" — test/contract edits are asymmetric with code fixes: only harness plumbing fixes, additive pin widenings with a changelog entry, or deletions of tests of removed behavior. Pins were green at a known commit; git history arbitrates stale-vs-changed. A right-looking test that fails is prima facie a server bug: stop and flag. contract_pins.md rule 4: narrowing/removing a pinned key requires the user's explicit sign-off — a pin freezing buggy behavior is a ledger, not a license. |
||
|
John O'Keefe
|
28bd323505 |
refactor(progress): SaveProgressRequest becomes the canonical PUT payload; GET serializes from the row
Batch H1 steps 1-2 of the hardening plan. Wire contracts frozen; the pinned shapes are unchanged modulo the whitelisted additive GET keys (docs/contract_pins.md). 1. One canonical request struct. The web PUT binds SaveProgressRequest directly (identity/attribution fields are json:"-" so they stay server-owned) and copies its server fields once — the hand-rolled DTO and the 20-line field-copy block are gone. Parse set for existing clients is IDENTICAL to the old DTO (verified in web/src: the ebook reader sends percentage/current_page/total_pages/reading_mode/epubcfi/ context_text/chapter/zoom_level — reader.ts:3113-3143; the audiobook reader sends percentage/chapter/audio_offset_seconds/elapsed_seconds/ reading_mode — audio.ts:518-520). 2. GET /api/media-items/:id/progress now serializes through one row-struct-driven builder instead of a hand-map, so a row column cannot be silently forgotten again (the audio_offset_seconds omission). Existing keys are verbatim; the row already carried audio_offset_seconds/reading_mode/viewport_x-y/zoom_level/ scroll_position_x-y/panel_number and they surface as WHITELISTED additive keys. The web audiobook reader already reads audio_offset_seconds (audio.ts:219) — with the key absent, durationless books resumed at chapter start (offset always 0); they now resume correctly. The conditional context_text bolt-on and the server-derived css_selector/anchor_href block are unchanged — restore precedence stays percentage-first, never re-derived. |
||
|
John O'Keefe
|
a2ea34232c |
test(contracts): pin wire shapes before the canonical-types migrations
Batch H0 of the hardening plan: freeze the JSON key SETS of every
load-bearing surface so H1-H7 can only change them additively.
- cmd/server/tests/contract_pins_test.go: 14 tests / ~30 subtests pinning
progress GET+PUT, the KOReader plugin surfaces (progress push/pull,
library, bookmarks), the Kobo adapter shapes (initialization, markup,
sync-from-server), collections detail all four branches + preview,
series JSON, search rows, dashboard sections, media list BOTH branches
(raw sqlc rows vs resolved 38-key map), and SSR smoke for
bookshelf/collections/series/tags.
- docs/contract_pins.md: the pinned-shape inventory + the rule that a
batch widening a shape must widen the pin and this doc in one commit.
Suite is green on the unmodified tree at
|