bookhoard/AUTOSCANNER_IMPROVEMENTS.md

# Autoscanner Improvements - Implementation Plan

## Problem Statement

The current autoscanner has two issues:

1. **Bulk file detection unreliable**: When files are copied in bulk (e.g., 6 files at once), fsnotify only detects 1 or 0 files
2. **Delete detection broken**: When files are deleted from the filesystem, they remain in the database

### Root Causes

1. **fsnotify limitations**: The file system watcher can miss events during bulk file operations
2. **Relative path mismatch**: Delete detection uses absolute paths for database lookups, but database stores relative paths (partially fixed)

## Solution Architecture

```
[fsnotify Events] → [Event Queue] → [Debounce Timer (3s)] → [Process Queue]
        ↓
[Polling Fallback (3 min)] → [Full Sync Check] → [Add missing / Remove deleted]
```

Two complementary systems working together:
- **fsnotify + debounce**: Handles most file changes in real-time
- **Polling fallback**: Safety net that catches anything fsnotify misses

## Implementation Plan

### Phase 1: Add Debounce to fsnotify Handler

**File**: `internal/services/media_scanner.go`

**Changes**:
1. Create an event queue to accumulate fsnotify events
2. Add debounce timer (3 seconds) that resets on each new event
3. When timer fires, process all queued events in batch
4. Process each event: new files → scan, deleted files → remove from DB

**Implementation Details**:
- Use a buffered channel as the event queue
- Use `time.After()` or `time.Timer` for debounce
- Process events in order, skip duplicates for same file

```go
// Pseudo-code structure
type FileEvent struct {
    path    string
    isDelete bool
}

eventQueue := make(chan FileEvent, 100)
var debounceTimer *time.Timer

func handleFsEvent(event fsnotify.Event) {
    select {
    case eventQueue <- FileEvent{path: event.Name, isDelete: event.Has(fsnotify.Remove)}:
    default:
        // Queue full, log warning
    }
    
    // Reset debounce timer
    if debounceTimer != nil {
        debounceTimer.Stop()
    }
    debounceTimer = time.AfterFunc(3*time.Second, processEventQueue)
}

func processEventQueue() {
    // Drain queue and process unique paths
}
```

### Phase 2: Add Polling Fallback

**File**: `internal/services/media_scanner.go` (or new file)

**Changes**:
1. Add polling interval configuration (default: 3 minutes)
2. Create sync function that:
   - Walks all library folders
   - Compares filesystem against database
   - Adds missing files (triggers scan for new files)
   - Removes orphaned database entries (files no longer exist)
3. Start polling goroutine alongside existing fsnotify watcher

**Configuration**:
- Environment variable: `SCAN_POLL_INTERVAL_MINUTES` (default: 3)
- Use existing config system or add to `internal/config/config.go`

**Implementation Details**:
```go
type ScannerSyncOptions struct {
    PollInterval time.Duration // default: 3 minutes
}

func (s *MediaScanner) StartPolling(ctx context.Context, opts ScannerSyncOptions) {
    ticker := time.NewTicker(opts.PollInterval)
    defer ticker.Stop()
    
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            s.SyncFilesystemWithDatabase(ctx)
        }
    }
}

func (s *MediaScanner) SyncFilesystemWithDatabase(ctx context.Context) error {
    // 1. Get all media items from database
    // 2. For each library folder:
    //    - Walk filesystem, build map of existing files (relative paths)
    //    - Compare against database
    //    - Add: file in filesystem but not in DB → scan
    //    - Remove: file in DB but not on filesystem → delete
}
```

### Phase 3: Reuse Existing Cleanup Logic

**File**: `internal/services/media_scanner.go`

The cleanup logic already exists (lines 252-299 in `ScanFolders`) - it removes orphaned items after each manual scan. We can refactor this into a reusable function called by both:
- Manual scan (existing behavior)
- Polling fallback (new behavior)

```go
func (s *MediaScanner) CleanupOrphanedItems(ctx context.Context) error {
    // Existing cleanup code from ScanFolders
    // Extract to reusable function
}
```

### Phase 4: Configuration

**File**: `internal/config/config.go`

Add new configuration option:

```go
type Config struct {
    // ... existing fields ...
    ScanPollIntervalMinutes int `env:"SCAN_POLL_INTERVAL_MINUTES"`
}

func LoadConfig() *Config {
    return &Config{
        // ... existing fields ...
        ScanPollIntervalMinutes: getEnvInt("SCAN_POLL_INTERVAL_MINUTES", 3), // 3 minutes default
    }
}
```

### Phase 5: Integration with Scheduler

**File**: `internal/services/scheduler.go`

The scheduler already handles periodic tasks. Consider:
1. Adding polling fallback to scheduler, OR
2. Starting polling directly in MediaScanner initialization

## Files to Modify

| File | Changes |
|------|---------|
| `internal/services/media_scanner.go` | Add debounce queue, polling fallback, reuse cleanup |
| `internal/config/config.go` | Add `SCAN_POLL_INTERVAL_MINUTES` config |
| `docker-compose.yml` | Add environment variable (optional) |

## Testing Plan

1. **Bulk file addition**: Copy 10+ files at once, verify all detected within 3 seconds (fsnotify) or 3 minutes (polling)
2. **Bulk file deletion**: Delete 5+ files, verify all removed from database within 3 minutes
3. **Mixed operations**: Add some, delete some, verify correct state
4. **Large library**: Test with 100+ files to ensure performance is acceptable

## Backward Compatibility

- Default polling interval: 3 minutes (user can configure)
- Existing manual scan functionality unchanged
- fsnotify continues to work as before (with debounce improvement)

## Security Considerations

- Polling runs on same goroutine as scanner (no new attack surface)
- File operations are read-only until changes detected
- Database operations use existing service layer (already authorized)

## Timeline Estimate

| Phase | Complexity | Estimate |
|-------|-------------|----------|
| Phase 1: Debounce | Medium | 1-2 hours |
| Phase 2: Polling | Medium | 1-2 hours |
| Phase 3: Reuse cleanup | Low | 30 min |
| Phase 4: Config | Low | 15 min |
| Phase 5: Integration | Low | 15 min |
| Testing | Medium | 1 hour |
| **Total** | - | **4-6 hours** |

## Future Improvements (Out of Scope)

1. **Configurable debounce duration**
2. **Per-library polling intervals**
3. **Event history/logging for debugging**
4. **Manual trigger for full sync**