Commit Graph
909 Commits
Author SHA1 Message Date
john-okeefe ba2f29983c test: add integration and unit tests for file watching
Add comprehensive test coverage for media scanning functionality:

- fsnotify_integration_test.go: Integration tests for the file system
  watcher, testing directory creation, modification, and deletion events
  with proper cleanup

- media_scanner_test.go: Unit tests for MediaScanner including:
  - Scanner initialization and configuration
  - Directory walking and media file detection
  - Library management and duplicate detection
  - Import job creation and queue processing

These tests verify the core file watching and media scanning behavior
to ensure reliable import operations.
2026-03-05 20:26:49 -05:00
john-okeefe d740442ca4 feat: refactor health check endpoint with real-time worker status
Extract health check logic into GetHealth method on Config struct and
integrate with Worker service for accurate scan status reporting.

Changes:
- Move health check handler from inline function to Config.GetHealth()
- Add Worker field to Config struct for dependency injection
- Wire Worker into main server dependencies
- Report actual scan_in_progress status using Worker.HasActiveScans()
- Report actual active_jobs count using Worker.GetActiveJobCount()

This provides more accurate health monitoring by checking the real state
of background jobs rather than returning static placeholder values.
2026-03-05 20:26:42 -05:00
john-okeefe e8efc2ee3e fix: remove unsupported sync job type from job handler
Remove "sync" from the list of valid job types to align with
the removal of JobTypeSync from the Worker service.
2026-03-05 20:26:35 -05:00
john-okeefe 71c415e958 feat: enhance Worker service with job tracking capabilities
- Add Priority field to Job struct for future job prioritization
- Add HasActiveScans() method to check if any scans are currently running
- Add GetActiveJobCount() method to count running and pending jobs
- Remove unused JobTypeSync constant

These changes enable more accurate health check reporting and prepare
for future job priority queue implementation.
2026-03-05 20:26:33 -05:00
john-okeefe d9356f0f85 feat: enhance health check endpoint with detailed error info and scan status
- Return actual database error message instead of generic "unavailable"
- Add scan status information to healthy response (scan_in_progress, active_jobs)
- Maintain backward compatibility while providing more actionable diagnostics
- Use map[string]interface{} to support nested scan status structure

These changes improve observability by providing administrators with
specific error messages and scan status information, making it easier
to diagnose issues and monitor system state.
2026-03-05 19:35:04 -05:00
john-okeefe b3263b2611 feat: add settings cache to reduce database queries in MediaScanner
- Add SettingsCache with TTL-based invalidation (30 seconds)
- Cache scan_poll_interval_seconds and auto_scan_enabled settings
- Reduce database queries from every poll/check to once per TTL period
- Improve error handling with proper fallback values
- Simplify boolean parsing with strings.ToLower for consistency

This optimization reduces database load when checking scan settings,
which occurs frequently during media scanning operations.
2026-03-05 19:35:02 -05:00
john-okeefe ab11eade68 refactor: inject ConnectionManager into Worker
Pass ConnectionManager to Worker constructor to enable WebSocket
broadcasting capabilities. Updated:
- main.go: server initialization
- test_helpers.go: test setup
- commonhandlers.go: handler initialization

This change enables Worker to broadcast job updates to connected clients.
2026-03-05 17:13:26 -05:00
john-okeefe 39a87ddabc feat: integrate WebSocket for real-time scan progress in admin panel
- Add WebSocket connection for scan progress updates
- Display live progress bar and file count during scans
- Handle scan_complete and scan_error messages
- Store polling interval in module variable for cleanup
- Expose stopScanStatusPolling function for manual control

Replaces or supplements HTTP polling with push-based updates for
better UX and reduced server load.
2026-03-05 17:13:23 -05:00
john-okeefe 40f303b004 feat: add user-scoped WebSocket broadcasting for scan progress
- Add UserID field to Job struct for tracking job ownership
- Broadcast scan progress updates to user's WebSocket connections
- Send real-time updates during scanning (progress, files scanned, new items, errors)

This allows the frontend to display live scan progress without HTTP polling.
Scanner now associates scan jobs with requesting user for targeted updates.
2026-03-05 17:13:21 -05:00
john-okeefe ff480129a3 feat: add WebSocket message types for scan progress
Add new message type constants for real-time scan progress updates:
- MessageTypeScanProgress: broadcast progress during scanning
- MessageTypeScanComplete: notify when scan completes
- MessageTypeScanError: report scan errors

These enable frontend to receive live scan updates instead of polling.
2026-03-05 17:13:17 -05:00
john-okeefe 89b0b93ffc fix: correct JSON struct tags in ProgressData
Fix incorrect struct tags for Page, TotalPages, and PageY fields.
Previously used 'int' tag instead of proper JSON field names,
which would cause serialization issues.
2026-03-05 17:13:15 -05:00
john-okeefe 51077887a1 Remove obsolete worker_test.go
The old test file is replaced by the new test structure in cmd/server/tests/
2026-03-05 16:28:51 -05:00
john-okeefe fe7eb5e308 Add tests for Jobs API and Worker job processing
- Add jobs_test.go with tests for job creation and status retrieval
- Add worker_test.go with tests for job processing
2026-03-05 16:28:45 -05:00
john-okeefe 54bfd778db Refactor MediaScanner for improved file watching and job queue integration
- Replace event queue with dirty directories tracking (Jellyfin approach)
- Add file stability checking to wait for file writes to complete
- Add initial scan on startup to detect existing files
- Integrate with Worker job queue for directory scanning
- Change WatchChanges to return error and use atomic.Bool for state
- Add scan_mutex to prevent concurrent scans
- Add Close method with proper cleanup of resources
- Enhance polling with configurable interval
2026-03-05 16:28:40 -05:00
john-okeefe a5ac1137e5 Enhance Worker with new job types and singleton pattern
- Add WorkerInstance global singleton for global access
- Add new job types: import, convert, thumbnails, backup, analytics, sync
- Add Enqueue method for non-blocking job submission
- Add job processors for each new job type:
  - processImportJob: OPDS and Calibre import support
  - processConvertJob: EPUB to KEPUB conversion
  - processThumbnailsJob: Cover thumbnail generation
  - processBackupJob: Database backup functionality
  - processAnalyticsJob: Library and system statistics
  - processDirectoryScanJob: Directory scanning for media scanner
- Add helper getTopN function for analytics
2026-03-05 16:28:32 -05:00
john-okeefe 5e97f14008 Add Jobs API for background task management
- Add JobsHandler with CreateJob and GetJobStatus endpoints
- Add jobs router with POST /api/jobs and GET /api/jobs/:jobId routes
- Integrate JobsHandler into main server and router config
2026-03-05 16:28:24 -05:00
john-okeefe 605aff104b docs: Fix Phase 1 job type duplication with Phase 0.5
Fixed issue where Phase 1 tried to add job types that were already added in Phase 0.5.

Changes:
1. Step 1.1 - Updated title from 'Add All Job Type Constants' to 'Add NEW Job Type Constants'
   - Now shows current state after Phase 0.5 (JobTypeScan, JobTypeSetFolders, JobTypeDirectoryScan)
   - Only adds NEW Phase 1 job types: Import, Convert, Thumbnails, Reindex, Backup, Analytics, Sync
   - Clarifies that JobTypeScan, JobTypeSetFolders, JobTypeDirectoryScan were added in Phase 0.5

2. Step 1.2 - Updated title from 'Add Job Handlers to Switch Statement' to 'Add NEW Job Handlers to Switch Statement'
   - Now shows current state after Phase 0.5 (handlers for Scan, SetFolders, DirectoryScan)
   - Only adds NEW Phase 1 handlers for the new job types
   - Clarifies that existing handlers were added in Phase 0.5

Impact: Developers now have clear guidance on which job types/handlers to add in each phase, avoiding confusion and potential merge conflicts.
2026-03-05 13:57:48 -05:00
john-okeefe f2e5114813 docs: Fix critical inconsistencies in Phase 0.5 plan
Fixed issues identified during review:

1. Commit message accuracy (lines 1207, 1209):
   - Changed 'worker *Worker (job queue reference)' to 'WorkerInstance *Worker global (no circular dependency)'
   - Changed 'fileStability map[string]atomic.Bool' to 'fileStability map[string]*atomic.Bool (pointer)'
   - Removed claim that worker field was ADDED (it was REMOVED in clean rewrite)

2. Polling interval consistency (60s chosen):
   - Constructor: 60s (correct, no change)
   - Test: Changed from expecting 300s to 60s
   - Commit message: Changed all references from 300s to 60s
   - Benefits: 'Delete detection via 60s polling (fast safety net)'
   - Rationale: Real-time fsnotify + 60s polling = best UX

3. Added Step 0.5.3.8: Initialize WorkerInstance in main():
   - Previously buried as inline comment in Step 0.5.3.7
   - Now dedicated step with file location (cmd/server/main.go)
   - Critical for system initialization

4. Removed duplicate benefits lines:
   - Lines 1252-1254 were duplicates of 1249-1251

5. Updated 'Code to ADD' section:
   - Clarified '*atomic.Bool (pointer to atomic.Bool, not value type)'
   - Clarified 'WorkerInstance *Worker global (no circular dependency)'
   - Added 'JobTypeDirectoryScan' to constants list

6. Updated Files modified section:
   - Added cmd/server/main.go (initialize WorkerInstance)
   - Clarified worker.go changes (JobTypeDirectoryScan, processDirectoryScanJob, WorkerInstance, Enqueue)
   - Changed scan_settings_integration_test.go description to 'test expects 60s polling'

7. Enhanced Concurrency Control section:
   - Added 'No circular dependency (WorkerInstance global)'

Result: Plan now accurately reflects clean architecture approach with 60s polling.
2026-03-05 13:04:05 -05:00
john-okeefe 1a552e03f1 docs: Rewrite Phase 0.5 with clean architecture (Phase 0.5 + Phase 1 robustness)
Critical rewrite to fix broken hybrid approach that tried to merge two incompatible systems.

PROBLEM WITH PREVIOUS APPROACH:
- Tried to use job queue AND direct scanning simultaneously
- Created job parameters that didn't match handler expectations
- Referenced non-existent activeScans map
- Never-initialized worker field in MediaScanner
- performInitialScan() bypassed job queue
- Like building a car with parts from two different manufacturers

CLEAN ARCHITECTURE:
- Job queue handles concurrency control ONLY
- Scanner handles all scanning logic
- Global WorkerInstance provides access (no circular dependency)
- Simple scan_mutex for double-protection
- Clear separation of concerns

KEY CHANGES:
1. MediaScanner struct:
   - Removed: worker *Worker field (circular dependency)
   - Removed: activeScans map (too complex)
   - Fixed: fileStability map[string]*atomic.Bool (was value, now pointer)
   - Added: scan_mutex sync.Mutex (simple, effective)

2. Job queue integration:
   - processDirtyDirectories() submits jobs to WorkerInstance
   - Job parameters: {directory: dirPath, db: s.db}
   - Added JobTypeDirectoryScan constant
   - Added processDirectoryScanJob() handler in Worker
   - performInitialScan() submits jobs (not direct calls)

3. Worker changes:
   - Added WorkerInstance *Worker global variable
   - Added Enqueue() method (non-blocking with fallback)
   - processDirectoryScanJob() creates scanner, calls scanDirectory()

PRESERVED FROM PHASE 0.5:
- Directory watching with dirty dirs tracking
- File stability checks (Audiobookshelf approach)
- Smart event merging (Jellyfin approach)
- 10-second batch processing
- 60-second polling fallback

ADDED FROM PHASE 1 ROBUSTNESS:
- Job queue for concurrency control
- Test isolation
- Fixed default values (30s → 60s)

RESULT:
- No parameter mismatches
- No non-existent fields
- No memory leaks
- Clean separation of concerns
- Best of both worlds without the complexity
2026-03-05 12:55:48 -05:00
john-okeefe eecfb52996 docs: Incorporate old Phase 1 features into Phase 0.5
Incorporates all old Phase 1 job queue concurrency control features
into Phase 0.5 to fix fsnotify reliability issues comprehensively.

Features Added from Old Phase 1:
- Job queue for all directory scans (serialized by worker pool)
- JobTypeSetFolders: Async folder configuration via job queue
- processSetFoldersJob() handler
- Test isolation: Snapshot/restore system_settings
- Fix GetPollInterval() default value: 30s → 60s (matches handler/schema)
- Fix test expectation: 60 → 300 (5 min polling interval)

Critical Fixes:
- File stability race condition: Uses atomic.Bool to prevent concurrent checks
- Double-unlock bug: Removed defer unlock, use explicit cleanup only
- Unbounded goroutine spawn: Job queue serializes scans (no semaphore needed)
- activeScans inconsistency: Removed (job queue handles concurrency)
- Worker constructor conflicts: Use Phase 4's signature (db parameter)

Concurrency Control (Job Queue Approach):
- All directory scans submitted as jobs to worker pool
- Worker pool serializes scans naturally (no concurrent access)
- No unbounded goroutine spawn (worker limits concurrency)
- Atomic file stability checks prevent duplicate entries
- No race conditions in fileStability map
- No memory leaks from orphaned map entries

Phase 0.5 Time Estimate: 3-4 hours (was 2-3 hours)
- Added JobTypeSetFolders integration
- Added test isolation implementation
- Fixed all critical issues (double-unlock, race conditions, etc.)

This plan now incorporates the best of both approaches:
- Directory-based watching (Jellyfin)
- File stability checks (Audiobookshelf)
- Job queue concurrency control (old Phase 1)
2026-03-05 12:38:31 -05:00
john-okeefe 7ff565a068 docs: Fix 12 critical issues in Phase 0.5 fsnotify implementation
CRITICAL FIXES (would cause test failures):
1. Integration test timing: 5s → 12s
   - Test waited 5s but implementation uses 10s batch delay
   - Would fail intermittently detecting all 20 files

2. Race condition in fileStability map access
   - Lock released between check and insert (lines 342-352)
   - Concurrent calls could create duplicate map entries
   - Fixed by holding lock during entire function

3. Missing concurrency protection in scanDirectory()
   - Multiple scans of same directory could run simultaneously
   - Could cause race conditions in fileStability map
   - Fixed with activeScans map to prevent duplicate scans

MAJOR FIXES (production issues under load):
4. Unbounded goroutine spawn
   - Spawns unlimited goroutines for directory scans
   - 100 changed directories = 100 concurrent scans = 1000s of goroutines
   - Fixed with semaphore limiting concurrent scans to 10
   - You correctly identified this as the same problem Phase 1 job queue solved

5. Memory leak in fileStability map
   - Entries never cleaned up if waitForFileStability() called concurrently
   - Fixed by proper lock pattern and cleanup on all code paths

6. No initial scan of root folders
   - Only watches for NEW changes, misses existing files
   - Fixed by adding performInitialScan() function

MEDIUM FIXES (edge cases / code quality):
7. Removed unused batchTimeout variable
   - Was declared but never actually used

8. Completed smart event merging
   - Added sibling directory consolidation logic
   - Prevents redundant scans of sibling folders

9. Clarified subdirectory handling
   - Updated comment to explain subdirs trigger own events
   - scanDirectory() doesn't walk into them (by design)

10. Added cleanup on shutdown
   - New Close() method cleans up all maps
   - Waits for active scans with 5-second timeout

11. Test timing: 11s → 15s
   - Prevents flaky tests under load

12. Added database error handling
   - Checks libraryID.Valid before scanning
   - Handles orphaned folders gracefully

NEW CODE ADDED:
- scanSemaphore chan struct{} - limits concurrent scans to 10
- activeScans map[string]bool - prevents duplicate scans
- activeScansMu sync.Mutex - protects activeScans
- performInitialScan() - scans root folders on startup
- Close() method - cleanup and graceful shutdown

ARCHITECTURAL IMPROVEMENT:
- Semaphore pattern (from Phase 1 job queue) applied at directory level
- Higher concurrency limit (10 directory scans vs 3 library scans)
- Prevents resource exhaustion while maintaining parallelism
- All map entries properly cleaned up (no memory leaks)
- Graceful shutdown with timeout

Document size: 3,130 lines (increased from 2,974 lines)
Total changes: 191 insertions, 35 deletions
2026-03-05 12:31:32 -05:00
john-okeefe a1b6820dba docs: Restructure infrastructure plan - remove Phase 1 dependencies
Problem:
- Phase 1 (Core Fixes Using Job Queue) conflicted with Phase 0.5
- Phase 1 added JobTypeSetFolders and processSetFoldersJob
- Phase 0.5 makes folder configuration automatic via database
- Phase 1's SetFolders() job queue approach is obsolete

Changes Made:
1. Removed Phase 1 entirely (636 lines deleted)
   - Removed JobTypeSetFolders job type
   - Removed processSetFoldersJob handler
   - Removed test isolation fixes (to be added elsewhere if needed)
   - Removed default value fixes (to be added elsewhere if needed)

2. Renumbered all subsequent phases:
   - Phase 2 (Job Queue Expansion) → Phase 1
   - Phase 3 (WebSocket Scan Progress) → Phase 2
   - Phase 4 (Caching and Monitoring) → Phase 3
   - Phase 5 (Job Queue Enhancements) → Phase 4

3. Updated all step numbers:
   - All steps renumbered to match new phase numbers
   - Step 2.x → Step 1.x, Step 3.x → Step 2.x, etc.

4. Updated job type counts:
   - Changed "8 async job types" to "7 async job types"
   - Removed setfolders from commit messages

5. Updated Summary section:
   - Removed Phase 1 time estimate
   - Added Phase 0.5 time estimate
   - Removed folder config from key design decisions
   - Updated Files Modified section

6. Updated references throughout:
   - All phase references updated to new numbers
   - All step references updated to match phase numbers

Rationale:
Phase 0.5's directory-based watching approach reads folder paths
directly from the library_folders table via GetLibraryFolders(),
making manual SetFolders() configuration unnecessary. The watcher
automatically discovers subdirectories, so job queue-based folder
configuration is no longer needed.

Phase 0.5 structure:
- Phase 0.5: Fix fsnotify Reliability (2-3 hours)
  - Directory-based watching replaces file-based event queue
  - File stability checks prevent processing incomplete files
  - Smart event merging consolidates parent/child/sibling events
  - 10-second batch processing for efficient bulk operations

- Phase 1: Job Queue Expansion (6-8 hours)
  - 7 async job types (import, convert, thumbnails, reindex, backup, analytics, sync)
  - Job management API for CRUD operations
  - Real-time job status tracking

- Phase 2: WebSocket Scan Progress (2-3 hours)
  - Real-time progress updates via WebSocket
  - Eliminates polling for job status

- Phase 3: Caching and Monitoring (2 hours)
  - Settings cache reduces database load
  - Enhanced /health endpoint

- Phase 4: Job Queue Enhancements (4-6 hours)
  - Job persistence across restarts
  - Job history and audit trail
  - Priority queue support

Total document size: 2,974 lines (reduced from 3,620 lines)

All dependencies on removed Phase 1 functionality have been eliminated.
Job queue for other tasks (import, convert, etc.) and WebSocket
integration remain unchanged and fully compatible.
2026-03-05 12:21:54 -05:00
john-okeefe 9e5c6d4566 docs: Update Phase 0.5 with Jellyfin and Audiobookshelf research findings
Research Summary:
Analyzed how two mature media servers handle filesystem watching to
identify best practices for fixing Bookhoard's fsnotify reliability issues.

Jellyfin (C#/.NET) Approach:
- Uses directory-based watching with 64KB internal buffer (16x default)
- Smart event merging: consolidates parent/sibling/subpath events
- 45-second self-ignore delay for internal changes
- Per-library enable/disable via configuration
- Weakness: No file stability check, processes immediately

Audiobookshelf (Node.js) Approach:
- Custom watcher wrapper for cross-platform support
- File stability check: polls mtime every 3s until stable (up to 10min timeout!)
- 10-second batch delay for processing multiple changes together
- renameDetection for move operations
- Weakness: Complex custom implementation

Phase 0.5 Plan Updates:
1. Added file stability check (Audiobookshelf approach)
   - New waitForFileStability() function
   - Polls file mtime every 3 seconds until stable
   - 60-second timeout prevents infinite waiting
   - Prevents processing files still being copied/downloaded

2. Added smart event merging (Jellyfin approach)
   - Updated markDirectoryDirty() with consolidation logic
   - Replaces child events with parent directory events
   - Handles sibling consolidation (merges to common parent)
   - Reduces redundant scans during bulk operations

3. Changed to 10-second batch delay (Audiobookshelf approach)
   - Changed from 2-second debounce to 10-second batch
   - Processes all ready directories together
   - Better balance between responsiveness and efficiency

4. Updated MediaScanner struct
   - Added fileStability map[string]time.Time field
   - Added fileStabilityMu sync.RWMutex field

5. Added comprehensive unit tests
   - TestMarkDirectoryDirty_SmartEventMerging
   - TestWaitForFileStability_StableFile
   - TestWaitForFileStability_UnstableFile
   - TestProcessDirtyDirectories_BatchesScans

6. Added comparison table showing research insights

Benefits of Combined Approach:
- No event queue overflow (directory-based watching)
- Reliable bulk import with file stability checks
- Smart event consolidation reduces redundant scans
- 10-second batch provides good responsiveness/efficiency balance
- Delete detection via 60-second polling safety net
- Works on Docker and network mounts

Files Changed:
- COMPLETE_INFRASTRUCTURE_ENHANCEMENT_PLAN.md (23 lines added)

Research Sources:
- https://github.com/jellyfin/jellyfin
- https://github.com/advplyr/audiobookshelf
2026-03-05 11:59:39 -05:00
john-okeefe b34f0fe156 docs: add comprehensive infrastructure enhancement plan
Add detailed implementation plan for leveraging underutilized job queue
and WebSocket infrastructure. Key focus areas:

**Core Design Principle:**
- Job queue as concurrency control mechanism (not mutex blocking)
- Non-blocking API responses for long-running operations
- Expand job queue from 10% to 90% utilization

**Phase 1 (2-3 hours): Core Concurrency Fixes**
- Add watching atomic flag to MediaScanner (prevents duplicate WatchChanges)
- Use job queue for folder configuration instead of blocking calls
- Fix test isolation with system_settings snapshot/restore
- Add job queue serialization tests

**Phase 2 (6-8 hours): Job Queue Expansion**
- Add 7 new job types: import, convert, thumbnails, reindex, backup, analytics, sync
- Create JobsHandler with REST API endpoints
- All operations support progress tracking via callbacks

**Phase 3 (2-3 hours): WebSocket Scan Progress**
- Real-time scan progress broadcasts to user's devices
- Pass ConnectionManager to Worker for WebSocket integration
- Add user ID to Job for targeted messaging

**Phase 4 (2 hours): Caching & Monitoring**
- Redis caching for frequently accessed data
- Prometheus metrics for job queue performance

**Phase 5 (4-6 hours): Job Queue Enhancements**
- Priority queues for different job types
- Job cancellation and retry logic
- Rate limiting and backpressure handling

Total estimated time: 16-22 hours for full implementation
2026-03-05 00:43:03 -05:00
john-okeefe cb46cd310f feat(collections): add WebSocket broadcast on RemoveBook operation
Add real-time synchronization for collection book removal:
- Extract user ID from context for targeted broadcasts
- Broadcast 'collection_updated' message to user's other devices
- Includes collection_id, action, and book_id in message payload

This ensures that when a user removes a book from a collection,
all their connected devices (browser tabs, mobile apps, etc.)
receive real-time updates via WebSocket.

Consistent with existing AddBooks and BulkRemoveBooks operations
which already use BroadcastToUser for synchronization.
2026-03-05 00:42:52 -05:00
john-okeefe 75260d1b10 chore: remove obsolete collection planning documents
Remove completed implementation plans that have been superseded:
- COLLECTION_LIBRARY_FILTERING_PLAN.md (library filtering feature completed)
- IMPLEMENTATION_COLLECTION_FIX.md (collection detail fix implemented)

These plans were for features that have already been implemented
in recent commits. Keeping only current/future planning docs.
2026-03-05 00:42:41 -05:00
john-okeefe 9b3d8cc949 feat: implement collection library filter with WebSocket improvements and test coverage
This commit adds comprehensive functionality for filtering collections by library,
improves WebSocket real-time updates with user activity detection, and adds
extensive test coverage.

## Core Features

### Collection Library Filter
- Added library_id parameter to media-items search API
- Collections can now be filtered by specific library
- Toggle UI component for enabling/disabling library filter
- Default state is "checked" when library_id is present
- Consistent behavior across partial and fuzzy search modes

### WebSocket Auto-Reload Mitigation
- Added user activity detection to prevent disruptive page reloads
- Checks if user is actively typing in INPUT/TEXTAREA/SELECT elements
- Skips auto-reload when user is interacting with form elements
- Toast notifications still show for awareness
- Prevents data loss during editing operations

## Implementation Changes

### Backend
- internal/database/queries.sql.go: Added library filter support to search queries
- internal/handlers/media.go: Enhanced search with library_id parameter validation
- internal/handlers/collections.go: Updated collection handlers with library filtering
- internal/sync/websocket.go: Improved broadcast mechanism with user-scoped updates
- internal/router/frontend.go: Pass libraryID to collection templates

### Frontend
- templates/collections.templ: Added library filter toggle UI component
- web/src/collections.ts: TypeScript implementation with WebSocket integration
- templates/collections_templ.go: Generated template code

### Testing
- cmd/server/tests/search_test.go: Added TestCollectionSearchLibraryFilter
- cmd/server/tests/websocket_test.go: Added TestWebSocketUserScopedBroadcast
- New helper functions for creating libraries and media items via API
- Comprehensive test coverage for library filtering and user-scoped broadcasts

## API Documentation Updates

### Bruno Tests (Comprehensive Documentation)
- bruno/collections/*: Added detailed API documentation for all collection endpoints
- bruno/devices/*: Added device management and sync API documentation
- bruno/devices/kobo/api.yml: Kobo-specific sync protocol docs
- bruno/devices/koreader/api.yml: KOReader-specific sync protocol docs
- bruno/opds/*: Added OPDS feed and download endpoint documentation
- bruno/library/browse-folders.yml: Library folder browsing API docs

### New Bruno Tests
- bruno/media-items/Search All Libraries.yml: Test search without library filter
- bruno/media-items/Search Specific Library.yml: Test search with library filter
- bruno/media-items/Search Invalid Library ID.yml: Test error handling

## Documentation

- docs/developer/api/media-items/search_media_items.md: Updated with library_id parameter
- IMPLEMENTATION_COLLECTION_FIX.md: Comprehensive implementation guide with test scenarios

## Testing

### Integration Tests
- Library filter tests verify correct filtering across multiple libraries
- Invalid library_id tests ensure proper error handling
- WebSocket tests verify user-scoped broadcast behavior
- User A no longer receives User B's collection updates

### Manual Testing Scenarios
- Open collection in multiple tabs - updates propagate correctly
- Type in search box while another tab adds books - no disruptive reload
- Add/remove books from collection - toast notifications appear
- Toggle library filter - results update dynamically

## Technical Details

- WebSocket broadcasts are now user-scoped for privacy
- Active element detection uses tagName and contenteditable attributes
- Library ID validation uses UUID format checking
- Progressive enhancement maintained - page works without JavaScript
- All changes follow PROJECT_GUIDELINES.md conventions
- TypeScript only for frontend logic
- TailwindCSS only for styling
- Procedural programming style throughout

## Breaking Changes

None - all changes are additive and backward compatible.
2026-03-04 22:37:47 -05:00
john-okeefe 72f053d179 docs: comprehensive implementation plan for collection detail fix
This implementation plan addresses multiple architectural improvements:

**Security Fix:**
- Add user-scoped WebSocket broadcasts to prevent cross-user data leaks
- Current broadcast sends ALL collection updates to ALL users
- New BroadcastToUser() method ensures privacy between users

**Features:**
- Add optional library_id filter to search API (partial + fuzzy)
- Add library filter toggle UI in Add Books modal
- Remove 265 lines of inline JavaScript from template
- Convert to proper TypeScript with type safety

**Architecture:**
- Full-stack task: backend, database, frontend, documentation
- User-scoped broadcasts follow JWT + device auth patterns
- Progressive enhancement maintained (SSR + JS enhancement)
- WebSocket real-time sync preserved for multi-device support

**Testing:**
- Integration tests using setupTestServer() helper
- Tests for library filtering (no filter, lib1, lib2, invalid)
- Tests for user-scoped WebSocket broadcasts
- Bruno API tests for new library_id parameter

**Documentation:**
- API docs at docs/developer/api/search.md
- Git strategy: 6 logical commits outlined
- Testing checklist for manual + automated verification

**Files Modified:**
- internal/sync/websocket.go: Add BroadcastToUser()
- internal/handlers/collections.go: Use user-scoped broadcasts
- internal/database/queries.sql: Add library_id filter
- internal/handlers/media.go: Accept library_id parameter
- templates/collections.templ: Remove inline JS, add toggle UI
- web/src/collections.ts: TypeScript with WebSocket support
- internal/router/frontend.go: Pass libraryID to template
- Tests, docs, Bruno tests

This plan follows all PROJECT_GUIDELINES.md requirements including
TypeScript conversion, TailwindCSS only, procedural style, proper
commit organization, and comprehensive testing.
2026-03-02 21:02:47 -05:00
john-okeefe 240b3247aa docs: Add implementation plan for collection detail page fix and library filtering
This document outlines the plan to fix the broken /collections/:id page
which has an inline JavaScript bug, and add library_id support to the
search API.

Key changes planned:
- Remove 265+ lines of inline JavaScript from collections.templ template
- Add minimal TypeScript module (~180 lines) in web/src/collections.ts
- Add optional library_id parameter to SearchMediaItems API endpoint
- Add library filter toggle UI to the Add Books modal
- Update template to accept libraryID parameter

The implementation uses a hybrid approach: minimal TypeScript for
client-only features while maintaining HTMX-like patterns for CRUD
operations. This reduces maintenance burden and improves code
organization.

Steps detailed:
1. Update Search API to accept optional library_id parameter
2. Add library_id filter to SQL query if not present
3. Remove inline JS from template, add data attributes
4. Add toggle UI for filtering books by library
5. Add TypeScript functions for modal, search, and book management
6. Update handler to pass libraryID to template
7. Update template function signature

Testing checklist included to verify:
- Page loads without JS errors
- Library filter toggle visibility
- Search results with/without library filtering
- Add/remove books functionality
- Client-side search filtering
2026-03-02 15:45:52 -05:00
john-okeefe 6454ade2f7 fix(dashboard): Return default preferences instead of 404
The GetPreferences API was returning 404 when no preferences existed
for a library, breaking the dashboard settings modal. Now returns
default preferences (empty hidden_collections, empty collection_order,
20 items_per_section) when no preferences are found, matching the
behavior of the frontend dashboard page.
2026-03-02 13:48:29 -05:00
john-okeefe 38be055149 chore: Remove obsolete collections HTMX planning document
This file was a planning document that has been superseded by
the implementation and is no longer needed.
2026-03-02 13:11:48 -05:00
john-okeefe 0426391835 docs(collections): Add user documentation for library filtering
- Document collection viewing from collections page vs dashboard
- Explain library filtering behavior with query parameters
- Clarify backward compatible behavior (no filter = all books)
2026-03-02 13:11:43 -05:00
john-okeefe fb6a57884d test(dashboard): Update tests for library filtering feature
- Update TestGetViewAllURL_SystemCollections to use collectionID and libraryID parameters
- Test both with and without library_id in URL
- Update TestBuildSections_ConvertsServiceTypesToHandlerTypes expected values
- All collections now link to /collections/{id} (system and user treated equally)
2026-03-02 13:11:39 -05:00
john-okeefe be4230266e feat(collections): Add library-aware filtering to collection detail pages
- Add library_id parameter to BuildSections and getViewAllURL functions
- Update dashboard handler to pass libraryID when building sections
- Add library_id query param support to collection detail page handler
- When library_id is provided, filter collection items by that library
- When no library_id, show all books (backward compatible)
- Reuses GetCollectionItemsForDashboard query for filtered results
- Preserves context when navigating from dashboard to collection detail
2026-03-02 13:11:35 -05:00
john-okeefe 8f83403342 docs: add collection library filtering implementation plan
Add comprehensive step-by-step plan for implementing library-aware
filtering on collection detail pages.

Purpose:
- Preserve dashboard context when navigating to collection details
- Support both filtered (single library) and unfiltered (all libraries) views
- Maintain backward compatibility with existing URLs

Plan includes:
- Detailed code changes for dashboard.go, frontend.go, dashboard_test.go
- Line-by-line modifications with before/after code snippets
- Implementation order with 10 steps
- Testing checklist for verification
- Documentation requirements

Follows PROJECT_GUIDELINES.md:
- No cascading fix-up edits
- Sequential implementation order
- Post-edit verification steps
- Test-driven approach with additions to dashboard_test.go
- Documentation updates for user-facing feature

This is a planning document only - no implementation changes yet.
2026-03-01 21:37:31 -05:00
john-okeefe a14b9c82ef fix(dashboard): normalize nil slices to empty arrays in preferences API
Ensure consistent JSON responses by converting nil slices to empty arrays
in the GetPreferences handler. This prevents null values from being
returned to the client for hidden_collections and collection_order fields,
making the API response more predictable and easier to consume.
2026-03-01 21:35:31 -05:00
john-okeefe 4f37a13519 feat(dashboard): add HTMX form data binding and redirect to RestoreSystemCollection
Update RestoreSystemCollection handler to support form-encoded requests from HTMX:

- Add 'form' struct tags to CollectionName and ResetType fields to enable binding
  from both JSON payloads and form submissions (required for HTMX compatibility)
- Add conditional HTMX redirect handling that sets HX-Redirect header when
  the request originates from HTMX, directing users to /collections after
  successful restoration

This change enables the system collection restore functionality to work seamlessly
with HTMX-based modal forms, improving the user experience by providing proper
navigation after the restore operation completes without requiring JavaScript
redirect logic.
2026-03-01 21:10:05 -05:00
john-okeefe 07d1143b7b chore(gitignore): ignore JavaScript sourcemap files
Add *.map pattern to .gitignore to exclude JavaScript sourcemap files
from version control. These files are generated during the build process
and are not needed in the repository, matching the existing pattern for
TypeScript declaration maps (*.d.ts.map).

This prevents accidentally committing generated sourcemap files like
collections.js.map that provide debugging information but are not
necessary for deployment or source control.
2026-03-01 21:09:59 -05:00
john-okeefe 1352d05ca3 docs: add Collections HTMX implementation documentation
Add comprehensive documentation tracking the HTMX Server-Side Rendering
implementation for the Collections page.

Document contents:
- Summary of completed implementation (March 2025)
- Detailed list of all files created and modified
- Step-by-step workflow for each CRUD operation
  (Create, Edit, Delete, Restore System Collection)
- Verification instructions
- Key discoveries and lessons learned:
  * Templ syntax limitations in conditionals
  * Route registration order requirements
  * HTMX fragment theming inheritance
  * Color handling best practices
  * Browser caching considerations

Purpose:
- Historical record of implementation approach
- Reference for future developers
- Documentation of project patterns and conventions
- Guide for troubleshooting similar features
2026-03-01 21:00:43 -05:00
john-okeefe 08b7f13079 feat(collections): add HTMX auth, icon picker, and navigation helpers
Add comprehensive TypeScript utilities for collections page functionality.

1. HTMX Authentication (setupHTMXAuth):
   - Adds Authorization header to all HTMX requests automatically
   - Listens for htmx:configRequest event on document.body
   - Injects Bearer token from localStorage
   - Eliminates need for hx-headers attributes on individual elements

2. Smart Card Navigation (navigateToCollection):
   - Implements event delegation to distinguish button clicks from card clicks
   - Checks event.target to determine what user clicked
   - Returns early if button clicked (lets HTMX handle button actions)
   - Navigates to collection detail page only when card body clicked
   - Uses data-href attribute for navigation target

3. Color Selection Helpers:
   - selectColor(): Updates hidden input and visual selection state
   - closeCollectionModal(): Removes modal from DOM after HTMX swap
   - initColorSelection(): Applies border color classes to collection cards
     using borderClasses mapping (blue→border-blue-500, etc.)

4. Icon Picker with Search:
   - Hardcoded iconData object: 30 emojis with searchable keywords
     (e.g., "📚": ["book", "books", "library", "read", "reading"])
   - populateIconGrid(): Dynamically generates icon buttons from iconData
   - selectIcon(): Updates hidden input with selected emoji
   - filterIcons(): Real-time search filtering by emoji OR keywords
   - showAllIcons(): Clears search filter
   - initIconSelection(): Auto-initializes after HTMX modal swap
     (listens for htmx:afterSwap event on #modal-container)

5. HTMX Modal Initialization:
   - setupHTMXModalInit(): Listens for modal loads via HTMX
   - Auto-initializes icon picker when modal content swapped into
     #modal-container

All functions exported to window object for onclick attribute access.
Auto-initializes on DOMContentLoaded or immediately if DOM ready.

Pattern consistency:
- Follows same pattern as toast.js (global exports, auto-init)
- Uses TypeScript type annotations
- No OOP (functional style per project guidelines)
- Server-side rendering with HTMX (no AJAX data fetching)
2026-03-01 21:00:28 -05:00
john-okeefe bdc3dcff96 refactor(templates): migrate collections page to HTMX modals
Refactor collections.templ to use HTMX-powered modals instead of
client-side JavaScript modals. This aligns with project guidelines
for server-side rendering and progressive enhancement.

Key changes:

1. Remove inline modal HTML and JavaScript:
   - Delete hardcoded create-modal div with inline form
   - Remove all inline JavaScript (showCreateModal, hideCreateModal,
     selectColor, handleCreate, viewCollection, editCollection,
     deleteCollection, logout)

2. Add HTMX modal infrastructure:
   - Add modal container div: <div id="modal-container"></div>
   - Load modals dynamically via hx-get attributes
   - Remove JavaScript modal toggling functions

3. Refactor collection cards for event delegation:
   - Change from <a> wrapper to <div> with onclick="navigateToCollection()"
   - Add data-href attribute for navigation target
   - Wrap edit/delete buttons in separate container to prevent
     unwanted card navigation when clicking buttons

4. Update buttons to use HTMX:
   - Create button: hx-get="/collections/create-modal"
   - Edit button: hx-get="/collections/{id}/edit-modal"
   - Delete button: hx-delete="/api/collections/{id}" with hx-confirm
   - Restore System button: hx-get="/collections/restore-modal"

5. Remove redundant forms:
   - Delete empty-state "Create Your First Collection" button's
     inline onclick (now uses HTMX like the main create button)

6. Add external JavaScript:
   - Load /static/collections.js for helper functions
     (navigateToCollection, setupHTMXAuth, etc.)

Benefits:
- Smaller initial page load (modal HTML loaded on-demand)
- Server-side rendering follows project guidelines
- Progressive enhancement (page works without JavaScript)
- Consistent with auth page modal pattern
- Easier to maintain (modal logic separated into dedicated templates)
2026-03-01 21:00:20 -05:00
john-okeefe d70a770504 chore(templates): add generated Go code for modal templates
Add auto-generated Go code for new modal templates:
- collection_modal_templ.go (from collection_modal.templ)
- restore_system_collection_modal_templ.go (from restore_system_collection_modal.templ)

These files are generated by templ compiler and contain the Render()
implementations. Do not edit manually.

Regenerate with: templ generate
2026-03-01 21:00:14 -05:00
john-okeefe 5d5012c0f7 feat(templates): add collection modals for create/edit/restore
Add two new template components:

1. CollectionModal(collection CollectionData)
   - Reusable modal for both creating and editing collections
   - When collection.ID is empty: shows "Create Collection" form
   - When collection.ID is set: shows "Edit Collection" form with pre-filled data
   - Features:
     * Name and description fields
     * Icon picker with search input and emoji grid
       (grid populated dynamically by JavaScript)
       (supports typing emoji directly or searching by keywords)
     * Color selection buttons (blue/red/yellow/green/purple)
     * HTMX form submission (hx-post for create, hx-put for update)
     - HX-Redirect to /collections after successful submission

2. RestoreSystemCollectionModal()
   - Modal for restoring deleted system collections
   - Dropdown with options: Continue Reading, Recently Added,
     Recently Read, Not Started
   - HTMX form submission to /api/dashboard/restore-system-collection
   - HX-Redirect to /collections after restoration

Both modals:
- Use fixed inset-0 positioning with black/70 backdrop
- Inherit theme from parent page (no html/head/body tags)
- Include close button (✕) that calls closeCollectionModal()
- Follow existing card styling conventions
- Use CSS custom properties for theming (--bg-secondary, --text-primary, etc.)
2026-03-01 21:00:07 -05:00
john-okeefe 87f53b56e8 feat(router): add collection modal routes for HTMX
Add three new frontend routes to support HTMX-powered modal dialogs:

1. GET /collections/create-modal
   - Renders empty collection creation modal
   - Uses CollectionModal template with empty CollectionData

2. GET /collections/:id/edit-modal
   - Fetches collection by ID from database
   - Pre-populates modal with existing collection data
   - Returns 400 for invalid UUID, 404 if collection not found

3. GET /collections/restore-modal
   - Renders system collection restoration modal
   - Allows users to restore deleted system collections

Route registration order:
- /collections/:id/edit-modal must be registered before /collections/:id
  to avoid path conflicts in Echo's router

These routes enable the collections page to load modals dynamically via
HTMX (hx-get) instead of embedding modal HTML in the base page.
2026-03-01 21:00:00 -05:00
john-okeefe 511ae66688 fix(collections): add form binding and HTMX redirect support
Add form:"" tags to CreateCollectionRequest and UpdateCollectionRequest
structs to enable proper form data binding with Echo's c.Bind().

This change aligns with the pattern used in auth handlers where both
form:"" and json:"" tags are present, allowing the same request structs
to work with both JSON payloads (API) and form data (HTMX).

Changes:
- Add form:"name", form:"description", form:"color", form:"icon",
  form:"auto_assign_rules", and form:"view_settings" tags to both
  CreateCollectionRequest and UpdateCollectionRequest

Additionally, add HTMX redirect support to CreateCollection and
UpdateCollection handlers:
- Add HX-Redirect header for HTMX requests after successful create/update
- Add HTML redirect response to DeleteCollection for HTMX requests
  (follows pattern from auth.go: inline script with window.location.href)

This ensures HTMX form submissions properly redirect to /collections
after successful operations, while maintaining API compatibility for
JSON requests.
2026-03-01 20:59:56 -05:00
john-okeefe 42a20e3be3 feat: Improve wood paneling border colors and background blend
- Update wood-light border from harsh black (#2a2a2a) to lighter warm brown (#8b5a2b) for better harmony with light background
- Update wood-dark border from #5c3317 to #7a5228 (slightly lighter medium brown) for improved visibility on dark backgrounds
- Update wood-mahogany border from #5c3317 to #8b3a3a (medium red-brown) to enhance mahogany's characteristic reddish tones
- Reduce background blend opacity from 60% to 40% to create more subtle text area background that complements new border colors

These changes improve visual consistency between border colors and their respective wood paneling backgrounds while maintaining good text contrast across all wood themes.
2026-03-01 12:22:36 -05:00
john-okeefe 0a0b7f4d2e fix: Update test files to match refactored method signatures
Update test files to work with recent backend refactoring changes.

Test changes in internal/services/dashboard_service_test.go:
- Fix method name casing for FilterHiddenCollections
  - Change from filterHiddenCollections (lowercase 'f')
  - Change to FilterHiddenCollections (uppercase 'F')
  - Matches exported method signature in DashboardService
  - Line 57: Update test call to use correct exported method

Test changes in internal/handlers/dashboard_test.go:
- Update getViewAllURL test to match simplified function signature
  - Remove queryType parameter from test call
  - Function now only takes collectionName parameter
  - Aligns with refactoring to use /collections/{id} routing
  - Line 178: Update test call to use new signature

These fixes ensure tests compile and run correctly after the
collection detail page refactoring where:
1. getViewAllURL() was simplified to return /collections/{id}
2. System collections now use the same routing as user collections
2026-03-01 00:33:20 -05:00
john-okeefe c6fa217092 feat: Add library ID support to media scanner and worker
Add default library ID functionality to improve library targeting
during media scans.

Service changes in internal/services/media_scanner.go:
- Add defaultLibraryID field to MediaScanner struct
- Add SetLibraryID() method to set default library
- Modify processMediaFile() to use defaultLibraryID when set
  - Prioritizes defaultLibraryID over folder-based library detection
  - Provides explicit library targeting for scans

Service changes in internal/services/worker.go:
- Add libraryUUID conversion from string to pgtype.UUID
- Call scanner.SetLibraryID() before ScanFolders()
  - Ensures scanner respects the job's library ID

These changes enable more precise library targeting during media scans,
allowing scans to be directed to specific libraries rather than relying
solely on folder-based detection.
2026-03-01 00:29:39 -05:00
john-okeefe eb2da1e05b fix: Change library ordering to oldest-first
Change library ordering in dropdown from DESC to ASC to display
libraries in creation order (oldest first).

Database changes in internal/database/queries/queries.sql:
- Modify GetUserLibraries query ORDER BY clause
  - Change from ORDER BY l.created_at DESC to ASC
  - Displays oldest libraries first in dropdown

This provides a more intuitive ordering where users see their
first-created libraries at the top of the list.
2026-03-01 00:29:27 -05:00
john-okeefe 486c16172b fix: Wood paneling overscroll and alignment issues
Fix multiple issues with wood paneling background image display
affecting overscroll area and page-specific rendering.

CSS changes in web/static/input.css:
- Add background-attachment: fixed to all wood paneling classes
  - Prevents wood paneling from moving during page scroll
  - Ensures wood paneling extends into overscroll area
  - Applied to bg-wood-dark, bg-wood-light, bg-wood-mahogany

- Fix body and container selectors for wood paneling
  - Ensure proper selector targeting for wood paneling application
  - Use background-position: center for better alignment
  - Use background-size: cover for full coverage

TypeScript changes in web/src/woodPanelingInit.ts:
- Add page detection to prevent wood paneling on collections page
  - Check if #collections-container exists in DOM
  - Only apply wood paneling on dashboard, not collections page
  - Prevents ID collision between dashboard and collections containers

Template changes in templates/header.templ:
- No functional changes, only reformatting

These fixes ensure that:
1. Wood paneling displays consistently across the entire viewport
2. Wood paneling extends into the overscroll area when scrolling past content
3. Wood paneling is properly aligned and centered
4. Wood paneling doesn't interfere with collections page rendering
5. Both dashboard and collections pages can coexist without visual conflicts
2026-03-01 00:29:11 -05:00