perf(scanner): bulk scan-state load — one query per library scan
Part 2 of the faceted plan, Phase 3 (steps 1-2). The periodic scan
issued one GetMediaItemByFilePath per scannable file (~3,094 queries
per scan in prod, nearly all answering 'unchanged, skip') plus one
GetLibraryByFolder per file for folder resolution.
- NEW media_scan_state table: the scanner's per-item fingerprint
cache. Deliberately NOT a media_items column — the library-scoped
list emits raw sqlc rows as JSON (7537063c), so scanner bookkeeping
must never leak into API payloads nor churn the model or pins.
Manifest JSONB shape v1 documented in the schema (n/s/m/h; ev
reserved for the Phase 4+ extractor-version mechanism).
- GetScanStateByLibrary: lean bulk projection (id, file_path,
file_size, archived_at, missing_scan_count, file_sha256, manifest).
archived/hash columns ride along so the skip path's archive-restore
and hash self-heal checks stay DB-free too (refines the plan's
4-column sketch; same goal, narrower invariant).
- ScanFolders: per-run reset, per-folder library cache, one bulk load
per distinct library. processMediaFile: map hit answers existence +
size + flags with zero DB reads; force-rescan and size-change
branches re-read the full row (the change path may be expensive);
write-through on create/size-change keeps later passes honest.
- Multifile folder/zip books keep their full-row lookup: it is one
query per BOOK ROOT (not per track) and needs audio_metadata for
the duration-probe check. They join the map in Phase 4, when the
manifest compare replaces their fingerprint check.
- nil map (watcher's scanDirectory path, bulk-load failure) falls
back to per-file lookups — behavior-preserving; the failure case
logged once per scan.
Test: TestScanStateBulkLoad (unchanged rescan leaves updated_at
stable; API-created rows are seen by the bulk load; size changes
still update in place). Container runner.
This commit is contained in:
1 parent
bbedbfb13f
commit
693053a9f5
7 files changed
+512
-54
No files matched your search
@@ -400,6 +400,17 @@ type Querier interface {
|
||||
GetRefreshToken(ctx context.Context, token pgtype.UUID) (GetRefreshTokenRow, error)
|
||||
GetSavedFilterByID(ctx context.Context, arg GetSavedFilterByIDParams) (SavedFilters, error)
|
||||
GetSavedFilters(ctx context.Context, arg GetSavedFiltersParams) ([]SavedFilters, error)
|
||||
// GetScanStateByLibrary: the scanner's ONE bulk query per library scan
|
||||
// (faceted plan Part 2 §2) — every item's existence + scanner-owned
|
||||
// facts + sidecar manifest in a single pass. Replaces the per-file
|
||||
// GetMediaItemByFilePath flood (~3,094 queries per scan in prod).
|
||||
// Lean projection: metadata blobs (audio_metadata, chapter_metadata,
|
||||
// authors, description...) are NEVER loaded — the skip path never
|
||||
// needs them, and the change path re-reads the full row.
|
||||
// archived_at + missing_scan_count + file_sha256 ride along so the
|
||||
// skip path's archive-restore and hash self-heal checks stay
|
||||
// DB-free too.
|
||||
GetScanStateByLibrary(ctx context.Context, libraryID pgtype.UUID) ([]GetScanStateByLibraryRow, error)
|
||||
// Scoped to ONE library: same-name series in different libraries are
|
||||
// different stories (manga adaptations, LN splits). Callers must pass a
|
||||
// library_id and verify the requester can see that library.
|
||||
|
||||
Reference in new issue
Block a user