feat(scanner): add debounced file watching with polling fallback

- Implement event queue with 3-second debouncing for file system events - Add configurable polling fallback (default 3 min) via SCAN_POLL_INTERVAL_MINUTES - Add SyncFilesystemWithDatabase to detect orphaned DB entries and new files - Integrate utils.ResolveMediaURL for consistent media file path resolution - Add COOKIE_SECURE env var with SameSite=LaxMode for session cookies - Update media handler to properly decode URL paths for file serving - Refactor scanner initialization to accept poll interval configuration
2026-02-28 01:16:27 -05:00
parent 594c630b99
commit 037e7c1189
14 changed files with 485 additions and 113 deletions
@@ -0,0 +1,207 @@
+# Autoscanner Improvements - Implementation Plan
+
+## Problem Statement
+
+The current autoscanner has two issues:
+
+1. **Bulk file detection unreliable**: When files are copied in bulk (e.g., 6 files at once), fsnotify only detects 1 or 0 files
+2. **Delete detection broken**: When files are deleted from the filesystem, they remain in the database
+
+### Root Causes
+
+1. **fsnotify limitations**: The file system watcher can miss events during bulk file operations
+2. **Relative path mismatch**: Delete detection uses absolute paths for database lookups, but database stores relative paths (partially fixed)
+
+## Solution Architecture
+
+```
+[fsnotify Events] → [Event Queue] → [Debounce Timer (3s)] → [Process Queue]
+        ↓
+[Polling Fallback (3 min)] → [Full Sync Check] → [Add missing / Remove deleted]
+```
+
+Two complementary systems working together:
+- **fsnotify + debounce**: Handles most file changes in real-time
+- **Polling fallback**: Safety net that catches anything fsnotify misses
+
+## Implementation Plan
+
+### Phase 1: Add Debounce to fsnotify Handler
+
+**File**: `internal/services/media_scanner.go`
+
+**Changes**:
+1. Create an event queue to accumulate fsnotify events
+2. Add debounce timer (3 seconds) that resets on each new event
+3. When timer fires, process all queued events in batch
+4. Process each event: new files → scan, deleted files → remove from DB
+
+**Implementation Details**:
+- Use a buffered channel as the event queue
+- Use `time.After()` or `time.Timer` for debounce
+- Process events in order, skip duplicates for same file
+
+```go
+// Pseudo-code structure
+type FileEvent struct {
+    path    string
+    isDelete bool
+}
+
+eventQueue := make(chan FileEvent, 100)
+var debounceTimer *time.Timer
+
+func handleFsEvent(event fsnotify.Event) {
+    select {
+    case eventQueue <- FileEvent{path: event.Name, isDelete: event.Has(fsnotify.Remove)}:
+    default:
+        // Queue full, log warning
+    }
+    
+    // Reset debounce timer
+    if debounceTimer != nil {
+        debounceTimer.Stop()
+    }
+    debounceTimer = time.AfterFunc(3*time.Second, processEventQueue)
+}
+
+func processEventQueue() {
+    // Drain queue and process unique paths
+}
+```
+
+### Phase 2: Add Polling Fallback
+
+**File**: `internal/services/media_scanner.go` (or new file)
+
+**Changes**:
+1. Add polling interval configuration (default: 3 minutes)
+2. Create sync function that:
+   - Walks all library folders
+   - Compares filesystem against database
+   - Adds missing files (triggers scan for new files)
+   - Removes orphaned database entries (files no longer exist)
+3. Start polling goroutine alongside existing fsnotify watcher
+
+**Configuration**:
+- Environment variable: `SCAN_POLL_INTERVAL_MINUTES` (default: 3)
+- Use existing config system or add to `internal/config/config.go`
+
+**Implementation Details**:
+```go
+type ScannerSyncOptions struct {
+    PollInterval time.Duration // default: 3 minutes
+}
+
+func (s *MediaScanner) StartPolling(ctx context.Context, opts ScannerSyncOptions) {
+    ticker := time.NewTicker(opts.PollInterval)
+    defer ticker.Stop()
+    
+    for {
+        select {
+        case <-ctx.Done():
+            return
+        case <-ticker.C:
+            s.SyncFilesystemWithDatabase(ctx)
+        }
+    }
+}
+
+func (s *MediaScanner) SyncFilesystemWithDatabase(ctx context.Context) error {
+    // 1. Get all media items from database
+    // 2. For each library folder:
+    //    - Walk filesystem, build map of existing files (relative paths)
+    //    - Compare against database
+    //    - Add: file in filesystem but not in DB → scan
+    //    - Remove: file in DB but not on filesystem → delete
+}
+```
+
+### Phase 3: Reuse Existing Cleanup Logic
+
+**File**: `internal/services/media_scanner.go`
+
+The cleanup logic already exists (lines 252-299 in `ScanFolders`) - it removes orphaned items after each manual scan. We can refactor this into a reusable function called by both:
+- Manual scan (existing behavior)
+- Polling fallback (new behavior)
+
+```go
+func (s *MediaScanner) CleanupOrphanedItems(ctx context.Context) error {
+    // Existing cleanup code from ScanFolders
+    // Extract to reusable function
+}
+```
+
+### Phase 4: Configuration
+
+**File**: `internal/config/config.go`
+
+Add new configuration option:
+
+```go
+type Config struct {
+    // ... existing fields ...
+    ScanPollIntervalMinutes int `env:"SCAN_POLL_INTERVAL_MINUTES"`
+}
+
+func LoadConfig() *Config {
+    return &Config{
+        // ... existing fields ...
+        ScanPollIntervalMinutes: getEnvInt("SCAN_POLL_INTERVAL_MINUTES", 3), // 3 minutes default
+    }
+}
+```
+
+### Phase 5: Integration with Scheduler
+
+**File**: `internal/services/scheduler.go`
+
+The scheduler already handles periodic tasks. Consider:
+1. Adding polling fallback to scheduler, OR
+2. Starting polling directly in MediaScanner initialization
+
+## Files to Modify
+
+| File | Changes |
+|------|---------|
+| `internal/services/media_scanner.go` | Add debounce queue, polling fallback, reuse cleanup |
+| `internal/config/config.go` | Add `SCAN_POLL_INTERVAL_MINUTES` config |
+| `docker-compose.yml` | Add environment variable (optional) |
+
+## Testing Plan
+
+1. **Bulk file addition**: Copy 10+ files at once, verify all detected within 3 seconds (fsnotify) or 3 minutes (polling)
+2. **Bulk file deletion**: Delete 5+ files, verify all removed from database within 3 minutes
+3. **Mixed operations**: Add some, delete some, verify correct state
+4. **Large library**: Test with 100+ files to ensure performance is acceptable
+
+## Backward Compatibility
+
+- Default polling interval: 3 minutes (user can configure)
+- Existing manual scan functionality unchanged
+- fsnotify continues to work as before (with debounce improvement)
+
+## Security Considerations
+
+- Polling runs on same goroutine as scanner (no new attack surface)
+- File operations are read-only until changes detected
+- Database operations use existing service layer (already authorized)
+
+## Timeline Estimate
+
+| Phase | Complexity | Estimate |
+|-------|-------------|----------|
+| Phase 1: Debounce | Medium | 1-2 hours |
+| Phase 2: Polling | Medium | 1-2 hours |
+| Phase 3: Reuse cleanup | Low | 30 min |
+| Phase 4: Config | Low | 15 min |
+| Phase 5: Integration | Low | 15 min |
+| Testing | Medium | 1 hour |
+| **Total** | - | **4-6 hours** |
+
+## Future Improvements (Out of Scope)
+
+1. **Configurable debounce duration**
+2. **Per-library polling intervals**
+3. **Event history/logging for debugging**
+4. **Manual trigger for full sync**