docs: update scan settings documentation for new polling system
- Update validation from minutes (15-1440) to seconds (1-3600) - Clarify behavior: real-time file watching with polling fallback - Remove scheduler references from development docs - Update migration notes for the new implementation
This commit is contained in:
@@ -1,207 +0,0 @@
|
||||
# Autoscanner Improvements - Implementation Plan
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The current autoscanner has two issues:
|
||||
|
||||
1. **Bulk file detection unreliable**: When files are copied in bulk (e.g., 6 files at once), fsnotify only detects 1 or 0 files
|
||||
2. **Delete detection broken**: When files are deleted from the filesystem, they remain in the database
|
||||
|
||||
### Root Causes
|
||||
|
||||
1. **fsnotify limitations**: The file system watcher can miss events during bulk file operations
|
||||
2. **Relative path mismatch**: Delete detection uses absolute paths for database lookups, but database stores relative paths (partially fixed)
|
||||
|
||||
## Solution Architecture
|
||||
|
||||
```
|
||||
[fsnotify Events] → [Event Queue] → [Debounce Timer (3s)] → [Process Queue]
|
||||
↓
|
||||
[Polling Fallback (3 min)] → [Full Sync Check] → [Add missing / Remove deleted]
|
||||
```
|
||||
|
||||
Two complementary systems working together:
|
||||
- **fsnotify + debounce**: Handles most file changes in real-time
|
||||
- **Polling fallback**: Safety net that catches anything fsnotify misses
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Add Debounce to fsnotify Handler
|
||||
|
||||
**File**: `internal/services/media_scanner.go`
|
||||
|
||||
**Changes**:
|
||||
1. Create an event queue to accumulate fsnotify events
|
||||
2. Add debounce timer (3 seconds) that resets on each new event
|
||||
3. When timer fires, process all queued events in batch
|
||||
4. Process each event: new files → scan, deleted files → remove from DB
|
||||
|
||||
**Implementation Details**:
|
||||
- Use a buffered channel as the event queue
|
||||
- Use `time.After()` or `time.Timer` for debounce
|
||||
- Process events in order, skip duplicates for same file
|
||||
|
||||
```go
|
||||
// Pseudo-code structure
|
||||
type FileEvent struct {
|
||||
path string
|
||||
isDelete bool
|
||||
}
|
||||
|
||||
eventQueue := make(chan FileEvent, 100)
|
||||
var debounceTimer *time.Timer
|
||||
|
||||
func handleFsEvent(event fsnotify.Event) {
|
||||
select {
|
||||
case eventQueue <- FileEvent{path: event.Name, isDelete: event.Has(fsnotify.Remove)}:
|
||||
default:
|
||||
// Queue full, log warning
|
||||
}
|
||||
|
||||
// Reset debounce timer
|
||||
if debounceTimer != nil {
|
||||
debounceTimer.Stop()
|
||||
}
|
||||
debounceTimer = time.AfterFunc(3*time.Second, processEventQueue)
|
||||
}
|
||||
|
||||
func processEventQueue() {
|
||||
// Drain queue and process unique paths
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: Add Polling Fallback
|
||||
|
||||
**File**: `internal/services/media_scanner.go` (or new file)
|
||||
|
||||
**Changes**:
|
||||
1. Add polling interval configuration (default: 3 minutes)
|
||||
2. Create sync function that:
|
||||
- Walks all library folders
|
||||
- Compares filesystem against database
|
||||
- Adds missing files (triggers scan for new files)
|
||||
- Removes orphaned database entries (files no longer exist)
|
||||
3. Start polling goroutine alongside existing fsnotify watcher
|
||||
|
||||
**Configuration**:
|
||||
- Environment variable: `SCAN_POLL_INTERVAL_MINUTES` (default: 3)
|
||||
- Use existing config system or add to `internal/config/config.go`
|
||||
|
||||
**Implementation Details**:
|
||||
```go
|
||||
type ScannerSyncOptions struct {
|
||||
PollInterval time.Duration // default: 3 minutes
|
||||
}
|
||||
|
||||
func (s *MediaScanner) StartPolling(ctx context.Context, opts ScannerSyncOptions) {
|
||||
ticker := time.NewTicker(opts.PollInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
s.SyncFilesystemWithDatabase(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *MediaScanner) SyncFilesystemWithDatabase(ctx context.Context) error {
|
||||
// 1. Get all media items from database
|
||||
// 2. For each library folder:
|
||||
// - Walk filesystem, build map of existing files (relative paths)
|
||||
// - Compare against database
|
||||
// - Add: file in filesystem but not in DB → scan
|
||||
// - Remove: file in DB but not on filesystem → delete
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 3: Reuse Existing Cleanup Logic
|
||||
|
||||
**File**: `internal/services/media_scanner.go`
|
||||
|
||||
The cleanup logic already exists (lines 252-299 in `ScanFolders`) - it removes orphaned items after each manual scan. We can refactor this into a reusable function called by both:
|
||||
- Manual scan (existing behavior)
|
||||
- Polling fallback (new behavior)
|
||||
|
||||
```go
|
||||
func (s *MediaScanner) CleanupOrphanedItems(ctx context.Context) error {
|
||||
// Existing cleanup code from ScanFolders
|
||||
// Extract to reusable function
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 4: Configuration
|
||||
|
||||
**File**: `internal/config/config.go`
|
||||
|
||||
Add new configuration option:
|
||||
|
||||
```go
|
||||
type Config struct {
|
||||
// ... existing fields ...
|
||||
ScanPollIntervalMinutes int `env:"SCAN_POLL_INTERVAL_MINUTES"`
|
||||
}
|
||||
|
||||
func LoadConfig() *Config {
|
||||
return &Config{
|
||||
// ... existing fields ...
|
||||
ScanPollIntervalMinutes: getEnvInt("SCAN_POLL_INTERVAL_MINUTES", 3), // 3 minutes default
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 5: Integration with Scheduler
|
||||
|
||||
**File**: `internal/services/scheduler.go`
|
||||
|
||||
The scheduler already handles periodic tasks. Consider:
|
||||
1. Adding polling fallback to scheduler, OR
|
||||
2. Starting polling directly in MediaScanner initialization
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `internal/services/media_scanner.go` | Add debounce queue, polling fallback, reuse cleanup |
|
||||
| `internal/config/config.go` | Add `SCAN_POLL_INTERVAL_MINUTES` config |
|
||||
| `docker-compose.yml` | Add environment variable (optional) |
|
||||
|
||||
## Testing Plan
|
||||
|
||||
1. **Bulk file addition**: Copy 10+ files at once, verify all detected within 3 seconds (fsnotify) or 3 minutes (polling)
|
||||
2. **Bulk file deletion**: Delete 5+ files, verify all removed from database within 3 minutes
|
||||
3. **Mixed operations**: Add some, delete some, verify correct state
|
||||
4. **Large library**: Test with 100+ files to ensure performance is acceptable
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
- Default polling interval: 3 minutes (user can configure)
|
||||
- Existing manual scan functionality unchanged
|
||||
- fsnotify continues to work as before (with debounce improvement)
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Polling runs on same goroutine as scanner (no new attack surface)
|
||||
- File operations are read-only until changes detected
|
||||
- Database operations use existing service layer (already authorized)
|
||||
|
||||
## Timeline Estimate
|
||||
|
||||
| Phase | Complexity | Estimate |
|
||||
|-------|-------------|----------|
|
||||
| Phase 1: Debounce | Medium | 1-2 hours |
|
||||
| Phase 2: Polling | Medium | 1-2 hours |
|
||||
| Phase 3: Reuse cleanup | Low | 30 min |
|
||||
| Phase 4: Config | Low | 15 min |
|
||||
| Phase 5: Integration | Low | 15 min |
|
||||
| Testing | Medium | 1 hour |
|
||||
| **Total** | - | **4-6 hours** |
|
||||
|
||||
## Future Improvements (Out of Scope)
|
||||
|
||||
1. **Configurable debounce duration**
|
||||
2. **Per-library polling intervals**
|
||||
3. **Event history/logging for debugging**
|
||||
4. **Manual trigger for full sync**
|
||||
File diff suppressed because it is too large
Load Diff
@@ -66,9 +66,8 @@ bookhoard/
|
||||
**Services** (`internal/services/`):
|
||||
|
||||
- `library_service.go` - Library operations
|
||||
- `media_scanner.go` - File scanning & metadata extraction
|
||||
- `media_scanner.go` - File scanning, metadata extraction, and real-time file watching
|
||||
- `worker.go` - Job queue worker pool
|
||||
- `scheduler.go` - Scheduled task manager
|
||||
- `collection_service.go` - Collection rules processing
|
||||
- `conversion_service.go` - EPUB→KEPUB conversion
|
||||
- `book_matching.go` - Book matching algorithms
|
||||
|
||||
@@ -38,7 +38,7 @@ Retrieve the current system-wide scan settings.
|
||||
|
||||
**Fields**:
|
||||
|
||||
- `scan_poll_interval_seconds` (integer): How often to scan all libraries in minutes (15-1440)
|
||||
- `scan_poll_interval_seconds` (integer): How often to poll for file changes in seconds (1-3600)
|
||||
- `auto_scan_enabled` (boolean): Whether auto-scanning is enabled system-wide
|
||||
|
||||
**Example**:
|
||||
@@ -69,9 +69,9 @@ Update the system-wide scan settings.
|
||||
|
||||
**Fields**:
|
||||
|
||||
- `scan_poll_interval_seconds` (integer, required): How often to scan all libraries in minutes
|
||||
- Minimum: 15 (15 minutes)
|
||||
- Maximum: 1440 (24 hours)
|
||||
- `scan_poll_interval_seconds` (integer, required): How often to poll for file changes in seconds
|
||||
- Minimum: 1 (1 second)
|
||||
- Maximum: 3600 (1 hour)
|
||||
- Default: 60
|
||||
- `auto_scan_enabled` (boolean, required): Whether auto-scanning is enabled system-wide
|
||||
- Default: true
|
||||
@@ -104,7 +104,7 @@ Update the system-wide scan settings.
|
||||
|
||||
**Validation Rules**:
|
||||
|
||||
- `scan_poll_interval_seconds` must be between 15 and 1440 minutes
|
||||
- `scan_poll_interval_seconds` must be between 1 and 3600 seconds (1 second to 1 hour)
|
||||
- Both fields are required
|
||||
|
||||
**Example**:
|
||||
@@ -123,26 +123,31 @@ curl -X PUT https://bookhoard.example.com/api/libraries/scan-settings \
|
||||
|
||||
## Behavior
|
||||
|
||||
### Scan Frequency
|
||||
### Poll Interval
|
||||
|
||||
The `scan_poll_interval_seconds` setting determines how often the system will automatically scan all libraries for new media files. The scheduler will trigger scans for all libraries at the configured interval.
|
||||
The `scan_poll_interval_seconds` setting determines how often the system will poll library folders for file changes as a fallback to real-time file watching.
|
||||
|
||||
**Constraints**:
|
||||
|
||||
- Minimum: 15 minutes (to prevent excessive scanning)
|
||||
- Maximum: 1440 minutes (24 hours)
|
||||
- Default: 60 minutes (1 hour)
|
||||
- Minimum: 1 second
|
||||
- Maximum: 3600 seconds (1 hour)
|
||||
- Default: 60 seconds
|
||||
|
||||
### Auto-Scan Toggle
|
||||
|
||||
The `auto_scan_enabled` setting acts as a master switch for automatic scanning:
|
||||
|
||||
- When `true`: All libraries will be scanned automatically at the configured interval
|
||||
- When `false`: No automatic scans will occur (manual scans still available)
|
||||
- When `true`: File watching and polling fallback are active for all libraries
|
||||
- When `false`: No automatic file monitoring occurs (manual scans still available)
|
||||
|
||||
### System-Wide Scope
|
||||
### File Watching System
|
||||
|
||||
These settings apply to **all libraries** in the system. Individual users can no longer configure per-user scan settings. This ensures consistent scanning behavior across the entire Bookhoard instance.
|
||||
The scan settings control the file watching system which consists of:
|
||||
|
||||
1. **Real-time file watching**: Uses fsnotify to detect file changes immediately
|
||||
2. **Polling fallback**: If file watching fails or is unavailable, polls folders at the configured interval
|
||||
|
||||
The system applies these settings to all configured libraries automatically on startup.
|
||||
|
||||
---
|
||||
|
||||
@@ -167,16 +172,18 @@ These settings apply to **all libraries** in the system. Individual users can no
|
||||
|
||||
## Migration Notes
|
||||
|
||||
This API replaces the previous per-user scan settings system. The following changes were made:
|
||||
This API has been updated to use a new polling-based scanning system. The following changes were made:
|
||||
|
||||
- **Removed**: Per-user scan settings (previously in users table)
|
||||
- **Added**: System-wide scan settings (now in system_settings table)
|
||||
- **Changed**: Access control from user-specific to admin-only
|
||||
- **Preserved**: Endpoint paths remain the same for backward compatibility
|
||||
- **Changed**: `scan_frequency_minutes` renamed to `scan_poll_interval_seconds`
|
||||
- **Changed**: Unit changed from minutes to seconds (15-1440 minutes → 1-3600 seconds)
|
||||
- **Removed**: Old scheduler-based scanning system
|
||||
- **Added**: Real-time file watching with polling fallback
|
||||
- **Preserved**: API endpoint paths remain the same
|
||||
|
||||
The migration ensures that:
|
||||
The new system ensures that:
|
||||
|
||||
1. All libraries scan at the same frequency
|
||||
2. Only administrators can modify scan settings
|
||||
3. The API endpoints remain unchanged for existing clients
|
||||
4. The scheduler uses system-wide settings instead of user-specific settings
|
||||
1. File changes are detected in real-time when possible (via fsnotify)
|
||||
2. Polling fallback catches missed events at the configured interval
|
||||
3. Settings apply to all libraries system-wide
|
||||
4. Only administrators can modify scan settings
|
||||
5. The `auto_scan_enabled` setting controls both file watching and polling
|
||||
|
||||
Reference in New Issue
Block a user