bookhoard/AUTOSCANNER_IMPROVEMENTS.md

Autoscanner Improvements - Implementation Plan

Problem Statement

The current autoscanner has two issues:

1. Bulk file detection unreliable: When files are copied in bulk (e.g., 6 files at once), fsnotify only detects 1 or 0 files
2. Delete detection broken: When files are deleted from the filesystem, they remain in the database

Root Causes

1. fsnotify limitations: The file system watcher can miss events during bulk file operations
2. Relative path mismatch: Delete detection uses absolute paths for database lookups, but database stores relative paths (partially fixed)

Solution Architecture

[fsnotify Events] → [Event Queue] → [Debounce Timer (3s)] → [Process Queue]
        ↓
[Polling Fallback (3 min)] → [Full Sync Check] → [Add missing / Remove deleted]

Two complementary systems working together:

• fsnotify + debounce: Handles most file changes in real-time
• Polling fallback: Safety net that catches anything fsnotify misses

Implementation Plan

Phase 1: Add Debounce to fsnotify Handler

File: internal/services/media_scanner.go

Changes:

1. Create an event queue to accumulate fsnotify events
2. Add debounce timer (3 seconds) that resets on each new event
3. When timer fires, process all queued events in batch
4. Process each event: new files → scan, deleted files → remove from DB

Implementation Details:

• Use a buffered channel as the event queue
• Use time.After() or time.Timer for debounce
• Process events in order, skip duplicates for same file

// Pseudo-code structure
type FileEvent struct {
    path    string
    isDelete bool
}

eventQueue := make(chan FileEvent, 100)
var debounceTimer *time.Timer

func handleFsEvent(event fsnotify.Event) {
    select {
    case eventQueue <- FileEvent{path: event.Name, isDelete: event.Has(fsnotify.Remove)}:
    default:
        // Queue full, log warning
    }
    
    // Reset debounce timer
    if debounceTimer != nil {
        debounceTimer.Stop()
    }
    debounceTimer = time.AfterFunc(3*time.Second, processEventQueue)
}

func processEventQueue() {
    // Drain queue and process unique paths
}

Phase 2: Add Polling Fallback

File: internal/services/media_scanner.go (or new file)

Changes:

1. Add polling interval configuration (default: 3 minutes)
2. Create sync function that:
   • Walks all library folders
   • Compares filesystem against database
   • Adds missing files (triggers scan for new files)
   • Removes orphaned database entries (files no longer exist)
3. Start polling goroutine alongside existing fsnotify watcher

Configuration:

• Environment variable: SCAN_POLL_INTERVAL_MINUTES (default: 3)
• Use existing config system or add to internal/config/config.go

Implementation Details:

type ScannerSyncOptions struct {
    PollInterval time.Duration // default: 3 minutes
}

func (s *MediaScanner) StartPolling(ctx context.Context, opts ScannerSyncOptions) {
    ticker := time.NewTicker(opts.PollInterval)
    defer ticker.Stop()
    
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            s.SyncFilesystemWithDatabase(ctx)
        }
    }
}

func (s *MediaScanner) SyncFilesystemWithDatabase(ctx context.Context) error {
    // 1. Get all media items from database
    // 2. For each library folder:
    //    - Walk filesystem, build map of existing files (relative paths)
    //    - Compare against database
    //    - Add: file in filesystem but not in DB → scan
    //    - Remove: file in DB but not on filesystem → delete
}

Phase 3: Reuse Existing Cleanup Logic

File: internal/services/media_scanner.go

The cleanup logic already exists (lines 252-299 in ScanFolders) - it removes orphaned items after each manual scan. We can refactor this into a reusable function called by both:

• Manual scan (existing behavior)
• Polling fallback (new behavior)

func (s *MediaScanner) CleanupOrphanedItems(ctx context.Context) error {
    // Existing cleanup code from ScanFolders
    // Extract to reusable function
}

Phase 4: Configuration

File: internal/config/config.go

Add new configuration option:

type Config struct {
    // ... existing fields ...
    ScanPollIntervalMinutes int `env:"SCAN_POLL_INTERVAL_MINUTES"`
}

func LoadConfig() *Config {
    return &Config{
        // ... existing fields ...
        ScanPollIntervalMinutes: getEnvInt("SCAN_POLL_INTERVAL_MINUTES", 3), // 3 minutes default
    }
}

Phase 5: Integration with Scheduler

File: internal/services/scheduler.go

The scheduler already handles periodic tasks. Consider:

1. Adding polling fallback to scheduler, OR
2. Starting polling directly in MediaScanner initialization

Files to Modify

File	Changes
internal/services/media_scanner.go	Add debounce queue, polling fallback, reuse cleanup
internal/config/config.go	Add SCAN_POLL_INTERVAL_MINUTES config
docker-compose.yml	Add environment variable (optional)

Testing Plan

1. Bulk file addition: Copy 10+ files at once, verify all detected within 3 seconds (fsnotify) or 3 minutes (polling)
2. Bulk file deletion: Delete 5+ files, verify all removed from database within 3 minutes
3. Mixed operations: Add some, delete some, verify correct state
4. Large library: Test with 100+ files to ensure performance is acceptable

Backward Compatibility

• Default polling interval: 3 minutes (user can configure)
• Existing manual scan functionality unchanged
• fsnotify continues to work as before (with debounce improvement)

Security Considerations

• Polling runs on same goroutine as scanner (no new attack surface)
• File operations are read-only until changes detected
• Database operations use existing service layer (already authorized)

Timeline Estimate

Phase	Complexity	Estimate
Phase 1: Debounce	Medium	1-2 hours
Phase 2: Polling	Medium	1-2 hours
Phase 3: Reuse cleanup	Low	30 min
Phase 4: Config	Low	15 min
Phase 5: Integration	Low	15 min
Testing	Medium	1 hour
Total	-	4-6 hours

Future Improvements (Out of Scope)

1. Configurable debounce duration
2. Per-library polling intervals
3. Event history/logging for debugging
4. Manual trigger for full sync