feat(scanner): extractor-version (ev) mechanism — parser fixes reach old rows without a force rescan (faceted plan Phase 10)
Parser fixes (7e4c97f6, genre model v2) never reached rows ingested
under an older parser: the sidecar never changed, so every fingerprint
said 'unchanged'. The manifest JSONB now carries the reserved 'ev'
stamp — the parser version that produced the item's row.
Semantics (all pinned by tests):
- ABSENT ev decodes as 1 ('v1-or-unknown'); stamping is truthful-only:
create paths + post-extraction stores stamp the setting's current
value; deploy-day backfill and the extract-failure guard stay UNSET
(a row never claims a version its data can't prove — a mid-backfill
bump self-heals next poll); repoint reconciliation preserves the
stored ev (stat-only refresh re-extracted nothing).
- NEW setting scan_extractor_version (seeded + registry + getter,
int default 1, category scanner): a manifest whose ev != the setting
re-extracts through the SAME change path (full extraction →
updateMediaItemWithMetadata → fresh stamped manifest). Sidecar-less
books get an entries-less manifest as their ev home (create: stamped;
first-sighting: unset).
- Catch-up bounded: evReextractBudgetPerScan=100, atomic (watcher pass
can overlap a poll scan), reset per ScanFolders — amortized +
resumable across polls; no force rescan, no flood.
- Multifile books included: the ev gate rides multifileSidecarChanged's
shared ladder; ev-driven manifestChanged drives the editorial apply
+ stamped re-store.
Settings API: the unified GET/PUT /api/system/settings now lists the
key (registry-driven, auto-surfaces in the admin UI). Test harness:
the registry is now Loaded + wired into the SystemSettingsHandler
(main.go parity) — settings writes through the API were 503s before.
Verified: unit (ev normalization, budget CAS, GetExtractorVersion
cache paths, registry clamping); integration state machine —
create-stamp ev=1 → equality skips (corruption probe persists) →
wiped scan-state backfills UNSET without re-extract → bump re-extracts
ONCE (sidecar value restored, ev=2) → stable (second corruption
persists); multifile twin green; adjacent suites (Audiobook, Calibre,
CrossLibraryMove, SchemaIdempotency) green; live app restart re-scans
1198 files with 0 changed (deploy-day rule), setting listed at
/api/system/settings.
This commit is contained in:
1 parent
43987262a6
commit
91b0c45dd4
10 files changed
+486
-10
No files matched your search
@@ -467,6 +467,17 @@ func setupTestServer(t *testing.T) *TestServerSetup {
|
||||
|
||||
queries := database.New(dbPool)
|
||||
|
||||
// Wire the tunable settings registry exactly like main.go: Load
|
||||
// populates the cache (missing rows fall back to compiled defaults)
|
||||
// and the SystemSettingsHandler must hold it or every
|
||||
// /api/system/settings call 503s ("settings registry not
|
||||
// initialized") — tests that write settings through the API (the
|
||||
// only sanctioned path) need this wired.
|
||||
settingsRegistry := database.NewSettingsRegistry(queries)
|
||||
if err := settingsRegistry.Load(context.Background()); err != nil {
|
||||
t.Logf("settings registry Load failed (defaults apply): %v", err)
|
||||
}
|
||||
|
||||
// Create login attempt tracker
|
||||
loginAttemptTracker := ratelimit.NewLoginAttemptTracker(5, 15*time.Minute, 5*time.Minute)
|
||||
|
||||
@@ -493,6 +504,7 @@ func setupTestServer(t *testing.T) *TestServerSetup {
|
||||
analyticsHandler := handlers.NewAnalyticsHandler(queries)
|
||||
queueHandler := handlers.NewQueueHandler(queries, queueProcessor)
|
||||
systemSettingsHandler := handlers.NewSystemSettingsHandler(queries)
|
||||
systemSettingsHandler.SetSettings(settingsRegistry)
|
||||
processingIssuesHandler := handlers.NewProcessingIssuesHandler(queries)
|
||||
|
||||
// Create refactored handlers (matching main.go)
|
||||
@@ -571,7 +583,7 @@ func setupTestServer(t *testing.T) *TestServerSetup {
|
||||
// RegisterRoutes dereferences Settings for the auth rate limit;
|
||||
// the main binary wires database.NewSettingsRegistry(queries) —
|
||||
// the test harness must too or route registration nil-panics.
|
||||
Settings: database.NewSettingsRegistry(queries),
|
||||
Settings: settingsRegistry,
|
||||
// main.go:246 passes the shared LibraryService here; omitting it
|
||||
// hands the kobo/kepub CFI converter a typed-nil (kobo markup
|
||||
// panic-recovered as 500s in TestProgressWeb_KoboIntegration).
|
||||
|
||||
Reference in new issue
Block a user