feat(podcasts): concurrent download pool — fair share, tickets, admin tunables
Replaces the sequential worker with a download-manager-style pool (§0 item 4). Model: the user's connection is the limit; semantics are fair-share ACROSS feeds, not goroutine-per-book (a global FIFO would make Book B wait behind Book A's 100 files). - Atomic claim: GetNextQueuedEpisode + SetPodcastEpisodeDownloading was a read-then-write race under concurrency; ClaimQueuedPodcastEpisode converts queued→downloading in ONE guarded statement (two claimants, one winner — the loser gets ErrNoRows). SetPodcastEpisodeDownloading retired. - Queue tickets: download_queue_pos BIGINT fed by a Postgres SEQUENCE stamped at enqueue — the enqueue loop's ordering IS the claim order (the foundation §0 item 5 builds on; previously all rows in a batch shared one updated_at, so within-feed order was arbitrary). - Pool: up to podcast_download_concurrency (default 4, live range 1–16) claim-loop goroutines per drain round; the setting re-reads every round so admin changes take effect without restart. - Fair share: claim prefers the feed least-recently-served (pickFeedToServe; never-served first), earliest ticket within a feed — books proceed in parallel while chapters land in order (start listening while the rest download). - Bandwidth: shared token bucket across all workers (podcast_download_speed_mbps, default 0 = unlimited); take() reserves bandwidth and callers sleep outside the lock. - Settings: new float setting type end to end (registry defaults + clamping, save validation, admin UI number-with-step input) + the two schema seeds; worker wires the registry via SetSettings (tests run on registered defaults). - Tests: unit coverage for the bucket reservation math + fair-share pick; integration TestPodcastDownloadPool pins ticket order = enqueue order, claim exclusivity, and a full 3-episode pool drain. Dev-DB note: the column/sequence/settings were applied live (idempotent DDL, same statements the schema runs at next boot) since the running container predates them.
This commit is contained in:
1 parent
e3675536aa
commit
0f3c4a8cec
14 files changed
+893
-264
No files matched your search
@@ -4862,10 +4862,13 @@ SELECT * FROM media_items
|
||||
WHERE id = $1 AND source = 'podcast_episode';
|
||||
|
||||
-- name: SetPodcastEpisodeDownloadQueued :execrows
|
||||
-- Worker claim: pending|failed -> queued. Attempts incremented at
|
||||
-- claim time so a crashed worker still counts the try.
|
||||
-- Enqueue: pending|failed -> queued, stamped with the NEXT QUEUE TICKET
|
||||
-- (§0 items 4/5) so claim order = enqueue order (the effective
|
||||
-- listening order the enqueue loop applied). Attempts increment at
|
||||
-- enqueue so a crashed worker still counts the try.
|
||||
UPDATE media_items
|
||||
SET download_state = 'queued',
|
||||
download_queue_pos = nextval('podcast_queue_seq'),
|
||||
download_error = NULL,
|
||||
download_attempts = download_attempts + 1,
|
||||
updated_at = NOW()
|
||||
@@ -4873,10 +4876,9 @@ WHERE id = $1
|
||||
AND source = 'podcast_episode'
|
||||
AND download_state IN ('pending', 'failed');
|
||||
|
||||
-- name: SetPodcastEpisodeDownloading :exec
|
||||
UPDATE media_items
|
||||
SET download_state = 'downloading', updated_at = NOW()
|
||||
WHERE id = $1 AND source = 'podcast_episode';
|
||||
-- SetPodcastEpisodeDownloading RETIRED (§0 item 4): the pair
|
||||
-- GetNextQueuedEpisode + SetPodcastEpisodeDownloading raced under
|
||||
-- concurrency; ClaimQueuedPodcastEpisode does both atomically.
|
||||
|
||||
-- name: MarkPodcastEpisodeDownloaded :one
|
||||
-- SUCCESS-only write (invariant 7): the file is on disk and probed.
|
||||
@@ -4964,12 +4966,26 @@ WHERE s.user_id = sqlc.arg('user_id')
|
||||
OR COALESCE(f.publisher, '') ILIKE '%' || sqlc.arg('search_query')::text || '%')
|
||||
ORDER BY COALESCE(s.display_name, f.title) ASC;
|
||||
|
||||
-- name: GetNextQueuedEpisode :one
|
||||
-- Sequential download worker's claim source (single worker, day one).
|
||||
SELECT * FROM media_items
|
||||
-- name: ListQueuedPodcastEpisodes :many
|
||||
-- Fair-share candidate scan (§0 item 4): the download pool claims in
|
||||
-- TICKET order (download_queue_pos = SEQUENCE nextval at enqueue — the
|
||||
-- enqueue loop's ordering IS the download order, item 5). The worker
|
||||
-- groups these per feed and prefers the least-recently-served feed.
|
||||
SELECT id, source_feed_id FROM media_items
|
||||
WHERE source = 'podcast_episode' AND download_state = 'queued'
|
||||
ORDER BY updated_at ASC
|
||||
LIMIT 1;
|
||||
ORDER BY download_queue_pos ASC NULLS LAST;
|
||||
|
||||
-- name: ClaimQueuedPodcastEpisode :one
|
||||
-- Atomic claim (§0 item 4): converts queued -> downloading in ONE
|
||||
-- statement. Two claimants racing on the same row: exactly one UPDATE
|
||||
-- matches download_state='queued'; the loser gets no rows (ErrNoRows).
|
||||
-- This is the SKIP-LOCKED guarantee without a second round-trip.
|
||||
UPDATE media_items
|
||||
SET download_state = 'downloading', updated_at = NOW()
|
||||
WHERE id = $1
|
||||
AND source = 'podcast_episode'
|
||||
AND download_state = 'queued'
|
||||
RETURNING media_items.*;
|
||||
|
||||
-- name: CountQueuedEpisodes :one
|
||||
SELECT COUNT(*) FROM media_items
|
||||
|
||||
Reference in new issue
Block a user