docs: update scan settings documentation for new polling system
- Update validation from minutes (15-1440) to seconds (1-3600) - Clarify behavior: real-time file watching with polling fallback - Remove scheduler references from development docs - Update migration notes for the new implementation
This commit is contained in:
@@ -1,207 +0,0 @@
|
|||||||
# Autoscanner Improvements - Implementation Plan
|
|
||||||
|
|
||||||
## Problem Statement
|
|
||||||
|
|
||||||
The current autoscanner has two issues:
|
|
||||||
|
|
||||||
1. **Bulk file detection unreliable**: When files are copied in bulk (e.g., 6 files at once), fsnotify only detects 1 or 0 files
|
|
||||||
2. **Delete detection broken**: When files are deleted from the filesystem, they remain in the database
|
|
||||||
|
|
||||||
### Root Causes
|
|
||||||
|
|
||||||
1. **fsnotify limitations**: The file system watcher can miss events during bulk file operations
|
|
||||||
2. **Relative path mismatch**: Delete detection uses absolute paths for database lookups, but database stores relative paths (partially fixed)
|
|
||||||
|
|
||||||
## Solution Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
[fsnotify Events] → [Event Queue] → [Debounce Timer (3s)] → [Process Queue]
|
|
||||||
↓
|
|
||||||
[Polling Fallback (3 min)] → [Full Sync Check] → [Add missing / Remove deleted]
|
|
||||||
```
|
|
||||||
|
|
||||||
Two complementary systems working together:
|
|
||||||
- **fsnotify + debounce**: Handles most file changes in real-time
|
|
||||||
- **Polling fallback**: Safety net that catches anything fsnotify misses
|
|
||||||
|
|
||||||
## Implementation Plan
|
|
||||||
|
|
||||||
### Phase 1: Add Debounce to fsnotify Handler
|
|
||||||
|
|
||||||
**File**: `internal/services/media_scanner.go`
|
|
||||||
|
|
||||||
**Changes**:
|
|
||||||
1. Create an event queue to accumulate fsnotify events
|
|
||||||
2. Add debounce timer (3 seconds) that resets on each new event
|
|
||||||
3. When timer fires, process all queued events in batch
|
|
||||||
4. Process each event: new files → scan, deleted files → remove from DB
|
|
||||||
|
|
||||||
**Implementation Details**:
|
|
||||||
- Use a buffered channel as the event queue
|
|
||||||
- Use `time.After()` or `time.Timer` for debounce
|
|
||||||
- Process events in order, skip duplicates for same file
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Pseudo-code structure
|
|
||||||
type FileEvent struct {
|
|
||||||
path string
|
|
||||||
isDelete bool
|
|
||||||
}
|
|
||||||
|
|
||||||
eventQueue := make(chan FileEvent, 100)
|
|
||||||
var debounceTimer *time.Timer
|
|
||||||
|
|
||||||
func handleFsEvent(event fsnotify.Event) {
|
|
||||||
select {
|
|
||||||
case eventQueue <- FileEvent{path: event.Name, isDelete: event.Has(fsnotify.Remove)}:
|
|
||||||
default:
|
|
||||||
// Queue full, log warning
|
|
||||||
}
|
|
||||||
|
|
||||||
// Reset debounce timer
|
|
||||||
if debounceTimer != nil {
|
|
||||||
debounceTimer.Stop()
|
|
||||||
}
|
|
||||||
debounceTimer = time.AfterFunc(3*time.Second, processEventQueue)
|
|
||||||
}
|
|
||||||
|
|
||||||
func processEventQueue() {
|
|
||||||
// Drain queue and process unique paths
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Phase 2: Add Polling Fallback
|
|
||||||
|
|
||||||
**File**: `internal/services/media_scanner.go` (or new file)
|
|
||||||
|
|
||||||
**Changes**:
|
|
||||||
1. Add polling interval configuration (default: 3 minutes)
|
|
||||||
2. Create sync function that:
|
|
||||||
- Walks all library folders
|
|
||||||
- Compares filesystem against database
|
|
||||||
- Adds missing files (triggers scan for new files)
|
|
||||||
- Removes orphaned database entries (files no longer exist)
|
|
||||||
3. Start polling goroutine alongside existing fsnotify watcher
|
|
||||||
|
|
||||||
**Configuration**:
|
|
||||||
- Environment variable: `SCAN_POLL_INTERVAL_MINUTES` (default: 3)
|
|
||||||
- Use existing config system or add to `internal/config/config.go`
|
|
||||||
|
|
||||||
**Implementation Details**:
|
|
||||||
```go
|
|
||||||
type ScannerSyncOptions struct {
|
|
||||||
PollInterval time.Duration // default: 3 minutes
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MediaScanner) StartPolling(ctx context.Context, opts ScannerSyncOptions) {
|
|
||||||
ticker := time.NewTicker(opts.PollInterval)
|
|
||||||
defer ticker.Stop()
|
|
||||||
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return
|
|
||||||
case <-ticker.C:
|
|
||||||
s.SyncFilesystemWithDatabase(ctx)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MediaScanner) SyncFilesystemWithDatabase(ctx context.Context) error {
|
|
||||||
// 1. Get all media items from database
|
|
||||||
// 2. For each library folder:
|
|
||||||
// - Walk filesystem, build map of existing files (relative paths)
|
|
||||||
// - Compare against database
|
|
||||||
// - Add: file in filesystem but not in DB → scan
|
|
||||||
// - Remove: file in DB but not on filesystem → delete
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Phase 3: Reuse Existing Cleanup Logic
|
|
||||||
|
|
||||||
**File**: `internal/services/media_scanner.go`
|
|
||||||
|
|
||||||
The cleanup logic already exists (lines 252-299 in `ScanFolders`) - it removes orphaned items after each manual scan. We can refactor this into a reusable function called by both:
|
|
||||||
- Manual scan (existing behavior)
|
|
||||||
- Polling fallback (new behavior)
|
|
||||||
|
|
||||||
```go
|
|
||||||
func (s *MediaScanner) CleanupOrphanedItems(ctx context.Context) error {
|
|
||||||
// Existing cleanup code from ScanFolders
|
|
||||||
// Extract to reusable function
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Phase 4: Configuration
|
|
||||||
|
|
||||||
**File**: `internal/config/config.go`
|
|
||||||
|
|
||||||
Add new configuration option:
|
|
||||||
|
|
||||||
```go
|
|
||||||
type Config struct {
|
|
||||||
// ... existing fields ...
|
|
||||||
ScanPollIntervalMinutes int `env:"SCAN_POLL_INTERVAL_MINUTES"`
|
|
||||||
}
|
|
||||||
|
|
||||||
func LoadConfig() *Config {
|
|
||||||
return &Config{
|
|
||||||
// ... existing fields ...
|
|
||||||
ScanPollIntervalMinutes: getEnvInt("SCAN_POLL_INTERVAL_MINUTES", 3), // 3 minutes default
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Phase 5: Integration with Scheduler
|
|
||||||
|
|
||||||
**File**: `internal/services/scheduler.go`
|
|
||||||
|
|
||||||
The scheduler already handles periodic tasks. Consider:
|
|
||||||
1. Adding polling fallback to scheduler, OR
|
|
||||||
2. Starting polling directly in MediaScanner initialization
|
|
||||||
|
|
||||||
## Files to Modify
|
|
||||||
|
|
||||||
| File | Changes |
|
|
||||||
|------|---------|
|
|
||||||
| `internal/services/media_scanner.go` | Add debounce queue, polling fallback, reuse cleanup |
|
|
||||||
| `internal/config/config.go` | Add `SCAN_POLL_INTERVAL_MINUTES` config |
|
|
||||||
| `docker-compose.yml` | Add environment variable (optional) |
|
|
||||||
|
|
||||||
## Testing Plan
|
|
||||||
|
|
||||||
1. **Bulk file addition**: Copy 10+ files at once, verify all detected within 3 seconds (fsnotify) or 3 minutes (polling)
|
|
||||||
2. **Bulk file deletion**: Delete 5+ files, verify all removed from database within 3 minutes
|
|
||||||
3. **Mixed operations**: Add some, delete some, verify correct state
|
|
||||||
4. **Large library**: Test with 100+ files to ensure performance is acceptable
|
|
||||||
|
|
||||||
## Backward Compatibility
|
|
||||||
|
|
||||||
- Default polling interval: 3 minutes (user can configure)
|
|
||||||
- Existing manual scan functionality unchanged
|
|
||||||
- fsnotify continues to work as before (with debounce improvement)
|
|
||||||
|
|
||||||
## Security Considerations
|
|
||||||
|
|
||||||
- Polling runs on same goroutine as scanner (no new attack surface)
|
|
||||||
- File operations are read-only until changes detected
|
|
||||||
- Database operations use existing service layer (already authorized)
|
|
||||||
|
|
||||||
## Timeline Estimate
|
|
||||||
|
|
||||||
| Phase | Complexity | Estimate |
|
|
||||||
|-------|-------------|----------|
|
|
||||||
| Phase 1: Debounce | Medium | 1-2 hours |
|
|
||||||
| Phase 2: Polling | Medium | 1-2 hours |
|
|
||||||
| Phase 3: Reuse cleanup | Low | 30 min |
|
|
||||||
| Phase 4: Config | Low | 15 min |
|
|
||||||
| Phase 5: Integration | Low | 15 min |
|
|
||||||
| Testing | Medium | 1 hour |
|
|
||||||
| **Total** | - | **4-6 hours** |
|
|
||||||
|
|
||||||
## Future Improvements (Out of Scope)
|
|
||||||
|
|
||||||
1. **Configurable debounce duration**
|
|
||||||
2. **Per-library polling intervals**
|
|
||||||
3. **Event history/logging for debugging**
|
|
||||||
4. **Manual trigger for full sync**
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -66,9 +66,8 @@ bookhoard/
|
|||||||
**Services** (`internal/services/`):
|
**Services** (`internal/services/`):
|
||||||
|
|
||||||
- `library_service.go` - Library operations
|
- `library_service.go` - Library operations
|
||||||
- `media_scanner.go` - File scanning & metadata extraction
|
- `media_scanner.go` - File scanning, metadata extraction, and real-time file watching
|
||||||
- `worker.go` - Job queue worker pool
|
- `worker.go` - Job queue worker pool
|
||||||
- `scheduler.go` - Scheduled task manager
|
|
||||||
- `collection_service.go` - Collection rules processing
|
- `collection_service.go` - Collection rules processing
|
||||||
- `conversion_service.go` - EPUB→KEPUB conversion
|
- `conversion_service.go` - EPUB→KEPUB conversion
|
||||||
- `book_matching.go` - Book matching algorithms
|
- `book_matching.go` - Book matching algorithms
|
||||||
|
|||||||
@@ -38,7 +38,7 @@ Retrieve the current system-wide scan settings.
|
|||||||
|
|
||||||
**Fields**:
|
**Fields**:
|
||||||
|
|
||||||
- `scan_poll_interval_seconds` (integer): How often to scan all libraries in minutes (15-1440)
|
- `scan_poll_interval_seconds` (integer): How often to poll for file changes in seconds (1-3600)
|
||||||
- `auto_scan_enabled` (boolean): Whether auto-scanning is enabled system-wide
|
- `auto_scan_enabled` (boolean): Whether auto-scanning is enabled system-wide
|
||||||
|
|
||||||
**Example**:
|
**Example**:
|
||||||
@@ -69,9 +69,9 @@ Update the system-wide scan settings.
|
|||||||
|
|
||||||
**Fields**:
|
**Fields**:
|
||||||
|
|
||||||
- `scan_poll_interval_seconds` (integer, required): How often to scan all libraries in minutes
|
- `scan_poll_interval_seconds` (integer, required): How often to poll for file changes in seconds
|
||||||
- Minimum: 15 (15 minutes)
|
- Minimum: 1 (1 second)
|
||||||
- Maximum: 1440 (24 hours)
|
- Maximum: 3600 (1 hour)
|
||||||
- Default: 60
|
- Default: 60
|
||||||
- `auto_scan_enabled` (boolean, required): Whether auto-scanning is enabled system-wide
|
- `auto_scan_enabled` (boolean, required): Whether auto-scanning is enabled system-wide
|
||||||
- Default: true
|
- Default: true
|
||||||
@@ -104,7 +104,7 @@ Update the system-wide scan settings.
|
|||||||
|
|
||||||
**Validation Rules**:
|
**Validation Rules**:
|
||||||
|
|
||||||
- `scan_poll_interval_seconds` must be between 15 and 1440 minutes
|
- `scan_poll_interval_seconds` must be between 1 and 3600 seconds (1 second to 1 hour)
|
||||||
- Both fields are required
|
- Both fields are required
|
||||||
|
|
||||||
**Example**:
|
**Example**:
|
||||||
@@ -123,26 +123,31 @@ curl -X PUT https://bookhoard.example.com/api/libraries/scan-settings \
|
|||||||
|
|
||||||
## Behavior
|
## Behavior
|
||||||
|
|
||||||
### Scan Frequency
|
### Poll Interval
|
||||||
|
|
||||||
The `scan_poll_interval_seconds` setting determines how often the system will automatically scan all libraries for new media files. The scheduler will trigger scans for all libraries at the configured interval.
|
The `scan_poll_interval_seconds` setting determines how often the system will poll library folders for file changes as a fallback to real-time file watching.
|
||||||
|
|
||||||
**Constraints**:
|
**Constraints**:
|
||||||
|
|
||||||
- Minimum: 15 minutes (to prevent excessive scanning)
|
- Minimum: 1 second
|
||||||
- Maximum: 1440 minutes (24 hours)
|
- Maximum: 3600 seconds (1 hour)
|
||||||
- Default: 60 minutes (1 hour)
|
- Default: 60 seconds
|
||||||
|
|
||||||
### Auto-Scan Toggle
|
### Auto-Scan Toggle
|
||||||
|
|
||||||
The `auto_scan_enabled` setting acts as a master switch for automatic scanning:
|
The `auto_scan_enabled` setting acts as a master switch for automatic scanning:
|
||||||
|
|
||||||
- When `true`: All libraries will be scanned automatically at the configured interval
|
- When `true`: File watching and polling fallback are active for all libraries
|
||||||
- When `false`: No automatic scans will occur (manual scans still available)
|
- When `false`: No automatic file monitoring occurs (manual scans still available)
|
||||||
|
|
||||||
### System-Wide Scope
|
### File Watching System
|
||||||
|
|
||||||
These settings apply to **all libraries** in the system. Individual users can no longer configure per-user scan settings. This ensures consistent scanning behavior across the entire Bookhoard instance.
|
The scan settings control the file watching system which consists of:
|
||||||
|
|
||||||
|
1. **Real-time file watching**: Uses fsnotify to detect file changes immediately
|
||||||
|
2. **Polling fallback**: If file watching fails or is unavailable, polls folders at the configured interval
|
||||||
|
|
||||||
|
The system applies these settings to all configured libraries automatically on startup.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -167,16 +172,18 @@ These settings apply to **all libraries** in the system. Individual users can no
|
|||||||
|
|
||||||
## Migration Notes
|
## Migration Notes
|
||||||
|
|
||||||
This API replaces the previous per-user scan settings system. The following changes were made:
|
This API has been updated to use a new polling-based scanning system. The following changes were made:
|
||||||
|
|
||||||
- **Removed**: Per-user scan settings (previously in users table)
|
- **Changed**: `scan_frequency_minutes` renamed to `scan_poll_interval_seconds`
|
||||||
- **Added**: System-wide scan settings (now in system_settings table)
|
- **Changed**: Unit changed from minutes to seconds (15-1440 minutes → 1-3600 seconds)
|
||||||
- **Changed**: Access control from user-specific to admin-only
|
- **Removed**: Old scheduler-based scanning system
|
||||||
- **Preserved**: Endpoint paths remain the same for backward compatibility
|
- **Added**: Real-time file watching with polling fallback
|
||||||
|
- **Preserved**: API endpoint paths remain the same
|
||||||
|
|
||||||
The migration ensures that:
|
The new system ensures that:
|
||||||
|
|
||||||
1. All libraries scan at the same frequency
|
1. File changes are detected in real-time when possible (via fsnotify)
|
||||||
2. Only administrators can modify scan settings
|
2. Polling fallback catches missed events at the configured interval
|
||||||
3. The API endpoints remain unchanged for existing clients
|
3. Settings apply to all libraries system-wide
|
||||||
4. The scheduler uses system-wide settings instead of user-specific settings
|
4. Only administrators can modify scan settings
|
||||||
|
5. The `auto_scan_enabled` setting controls both file watching and polling
|
||||||
|
|||||||
Reference in New Issue
Block a user