- Add UserID field to Job struct for tracking job ownership
- Broadcast scan progress updates to user's WebSocket connections
- Send real-time updates during scanning (progress, files scanned, new items, errors)
This allows the frontend to display live scan progress without HTTP polling.
Scanner now associates scan jobs with requesting user for targeted updates.
- Add JobsHandler with CreateJob and GetJobStatus endpoints
- Add jobs router with POST /api/jobs and GET /api/jobs/:jobId routes
- Integrate JobsHandler into main server and router config
Add real-time synchronization for collection book removal:
- Extract user ID from context for targeted broadcasts
- Broadcast 'collection_updated' message to user's other devices
- Includes collection_id, action, and book_id in message payload
This ensures that when a user removes a book from a collection,
all their connected devices (browser tabs, mobile apps, etc.)
receive real-time updates via WebSocket.
Consistent with existing AddBooks and BulkRemoveBooks operations
which already use BroadcastToUser for synchronization.
This commit adds comprehensive functionality for filtering collections by library,
improves WebSocket real-time updates with user activity detection, and adds
extensive test coverage.
## Core Features
### Collection Library Filter
- Added library_id parameter to media-items search API
- Collections can now be filtered by specific library
- Toggle UI component for enabling/disabling library filter
- Default state is "checked" when library_id is present
- Consistent behavior across partial and fuzzy search modes
### WebSocket Auto-Reload Mitigation
- Added user activity detection to prevent disruptive page reloads
- Checks if user is actively typing in INPUT/TEXTAREA/SELECT elements
- Skips auto-reload when user is interacting with form elements
- Toast notifications still show for awareness
- Prevents data loss during editing operations
## Implementation Changes
### Backend
- internal/database/queries.sql.go: Added library filter support to search queries
- internal/handlers/media.go: Enhanced search with library_id parameter validation
- internal/handlers/collections.go: Updated collection handlers with library filtering
- internal/sync/websocket.go: Improved broadcast mechanism with user-scoped updates
- internal/router/frontend.go: Pass libraryID to collection templates
### Frontend
- templates/collections.templ: Added library filter toggle UI component
- web/src/collections.ts: TypeScript implementation with WebSocket integration
- templates/collections_templ.go: Generated template code
### Testing
- cmd/server/tests/search_test.go: Added TestCollectionSearchLibraryFilter
- cmd/server/tests/websocket_test.go: Added TestWebSocketUserScopedBroadcast
- New helper functions for creating libraries and media items via API
- Comprehensive test coverage for library filtering and user-scoped broadcasts
## API Documentation Updates
### Bruno Tests (Comprehensive Documentation)
- bruno/collections/*: Added detailed API documentation for all collection endpoints
- bruno/devices/*: Added device management and sync API documentation
- bruno/devices/kobo/api.yml: Kobo-specific sync protocol docs
- bruno/devices/koreader/api.yml: KOReader-specific sync protocol docs
- bruno/opds/*: Added OPDS feed and download endpoint documentation
- bruno/library/browse-folders.yml: Library folder browsing API docs
### New Bruno Tests
- bruno/media-items/Search All Libraries.yml: Test search without library filter
- bruno/media-items/Search Specific Library.yml: Test search with library filter
- bruno/media-items/Search Invalid Library ID.yml: Test error handling
## Documentation
- docs/developer/api/media-items/search_media_items.md: Updated with library_id parameter
- IMPLEMENTATION_COLLECTION_FIX.md: Comprehensive implementation guide with test scenarios
## Testing
### Integration Tests
- Library filter tests verify correct filtering across multiple libraries
- Invalid library_id tests ensure proper error handling
- WebSocket tests verify user-scoped broadcast behavior
- User A no longer receives User B's collection updates
### Manual Testing Scenarios
- Open collection in multiple tabs - updates propagate correctly
- Type in search box while another tab adds books - no disruptive reload
- Add/remove books from collection - toast notifications appear
- Toggle library filter - results update dynamically
## Technical Details
- WebSocket broadcasts are now user-scoped for privacy
- Active element detection uses tagName and contenteditable attributes
- Library ID validation uses UUID format checking
- Progressive enhancement maintained - page works without JavaScript
- All changes follow PROJECT_GUIDELINES.md conventions
- TypeScript only for frontend logic
- TailwindCSS only for styling
- Procedural programming style throughout
## Breaking Changes
None - all changes are additive and backward compatible.
The GetPreferences API was returning 404 when no preferences existed
for a library, breaking the dashboard settings modal. Now returns
default preferences (empty hidden_collections, empty collection_order,
20 items_per_section) when no preferences are found, matching the
behavior of the frontend dashboard page.
- Update TestGetViewAllURL_SystemCollections to use collectionID and libraryID parameters
- Test both with and without library_id in URL
- Update TestBuildSections_ConvertsServiceTypesToHandlerTypes expected values
- All collections now link to /collections/{id} (system and user treated equally)
- Add library_id parameter to BuildSections and getViewAllURL functions
- Update dashboard handler to pass libraryID when building sections
- Add library_id query param support to collection detail page handler
- When library_id is provided, filter collection items by that library
- When no library_id, show all books (backward compatible)
- Reuses GetCollectionItemsForDashboard query for filtered results
- Preserves context when navigating from dashboard to collection detail
Ensure consistent JSON responses by converting nil slices to empty arrays
in the GetPreferences handler. This prevents null values from being
returned to the client for hidden_collections and collection_order fields,
making the API response more predictable and easier to consume.
Update RestoreSystemCollection handler to support form-encoded requests from HTMX:
- Add 'form' struct tags to CollectionName and ResetType fields to enable binding
from both JSON payloads and form submissions (required for HTMX compatibility)
- Add conditional HTMX redirect handling that sets HX-Redirect header when
the request originates from HTMX, directing users to /collections after
successful restoration
This change enables the system collection restore functionality to work seamlessly
with HTMX-based modal forms, improving the user experience by providing proper
navigation after the restore operation completes without requiring JavaScript
redirect logic.
Add form:"" tags to CreateCollectionRequest and UpdateCollectionRequest
structs to enable proper form data binding with Echo's c.Bind().
This change aligns with the pattern used in auth handlers where both
form:"" and json:"" tags are present, allowing the same request structs
to work with both JSON payloads (API) and form data (HTMX).
Changes:
- Add form:"name", form:"description", form:"color", form:"icon",
form:"auto_assign_rules", and form:"view_settings" tags to both
CreateCollectionRequest and UpdateCollectionRequest
Additionally, add HTMX redirect support to CreateCollection and
UpdateCollection handlers:
- Add HX-Redirect header for HTMX requests after successful create/update
- Add HTML redirect response to DeleteCollection for HTMX requests
(follows pattern from auth.go: inline script with window.location.href)
This ensures HTMX form submissions properly redirect to /collections
after successful operations, while maintaining API compatibility for
JSON requests.
Update test files to work with recent backend refactoring changes.
Test changes in internal/services/dashboard_service_test.go:
- Fix method name casing for FilterHiddenCollections
- Change from filterHiddenCollections (lowercase 'f')
- Change to FilterHiddenCollections (uppercase 'F')
- Matches exported method signature in DashboardService
- Line 57: Update test call to use correct exported method
Test changes in internal/handlers/dashboard_test.go:
- Update getViewAllURL test to match simplified function signature
- Remove queryType parameter from test call
- Function now only takes collectionName parameter
- Aligns with refactoring to use /collections/{id} routing
- Line 178: Update test call to use new signature
These fixes ensure tests compile and run correctly after the
collection detail page refactoring where:
1. getViewAllURL() was simplified to return /collections/{id}
2. System collections now use the same routing as user collections
Add comprehensive collection detail page that works for both system collections
(continue-reading, recently-added, not-started) and user collections.
Backend changes:
- Add new /collections/:id route in internal/router/frontend.go
- Fetches collection using GetCollection with UUID parameter
- Determines collection type from QueryType field
- Resolves library_id for system collections
- Converts database.MediaItems to handlers.BookInfo for display
- Renders CollectionDetail template with collection and books data
- Update SectionData struct in internal/handlers/collections.go
- Add CollectionID string field for view all links
- Update BuildSections() in internal/handlers/dashboard.go
- Pass CollectionID to SectionData for proper link generation
- Simplify getViewAllURL() in internal/handlers/dashboard.go
- Return /collections/{collectionID} instead of /section/{type}
- Works uniformly for both system and user collections
Frontend changes:
- Fix CollectionDetail template in templates/collections.templ
- Fix broken div nesting causing compilation error
- Add null check for CoverImagePath to prevent broken images
- Update aspect ratio to modern aspect-[3/4] syntax
- Use responsive widths (w-16 sm:w-20) for mobile/desktop
- Improve card layout with horizontal flex structure
- Add placeholder image fallback for books without covers
- Remove erroneous renderBooks() function call
This change aligns with the backend update where system collections are
now pre-made user collections in the database with query_type fields.
All collections can now use the same CollectionDetail template for a
consistent viewing experience.
- Update validation to use scan_poll_interval_seconds field (1-3600 seconds)
- Update all test cases and assertions to use new field name
- Update integration test to reflect new field name
- Add check for auto_scan_enabled setting in router before starting watch mode
- Update StartScanner handler to verify auto-scan is enabled
- Switch scanner to use watchModeCtx/watchModeCancel instead of ctx/cancel
- Update StartWatchModeForLibrary to use new MediaScanner signature
- Add GetPollInterval() method to MediaScanner to read from database
- Add GetAutoScanEnabled() method to check if auto-scan is enabled
- Remove ScanPollIntervalSeconds from config (now DB-driven)
- Update NewMediaScanner signature to not require interval parameter
- Remove SCAN_POLL_INTERVAL_SECONDS from docker-compose env var
- Rename SCAN_POLL_INTERVAL_MINUTES to SCAN_POLL_INTERVAL_SECONDS in config
- Update MediaScanner to accept interval in seconds instead of minutes
- Adjust default polling interval from 3 minutes to 30 seconds for faster response
- Add debug logging for fsnotify events to aid troubleshooting file watching
This change improves media file detection responsiveness by reducing the
polling interval from minutes to seconds, while maintaining the file
watcher as the primary detection mechanism.
- Implement event queue with 3-second debouncing for file system events
- Add configurable polling fallback (default 3 min) via SCAN_POLL_INTERVAL_MINUTES
- Add SyncFilesystemWithDatabase to detect orphaned DB entries and new files
- Integrate utils.ResolveMediaURL for consistent media file path resolution
- Add COOKIE_SECURE env var with SameSite=LaxMode for session cookies
- Update media handler to properly decode URL paths for file serving
- Refactor scanner initialization to accept poll interval configuration
Update the DownloadBook function to properly resolve media file paths using
the LibraryService.ResolveMediaPath method instead of directly accessing the
FilePath field. This ensures correct file resolution after the migration to
relative path storage.
The change affects three code paths in the download handler:
- KEPUB conversion path
- Direct file serve path (non-EPUB with conversion service)
- Default EPUB path
Error handling added to return 404 when path resolution fails, preventing
potential errors when accessing non-existent files.
This fixes potential file access issues after the relative path storage
implementation.
Remove the FilePath field from book metadata responses in the GetShelf endpoint.
This change improves security by not exposing internal file paths to API clients,
as the application now uses relative path storage with URL resolution via the
library service.
Changes:
- Remove FilePath field from BookPreview struct in GetShelf response
- Remove FilePath field from shelf items response
Related to previous commit implementing relative path storage.
- Add libraryService dependency to CollectionHandler and OPDSHandler for centralized path resolution
- Create internal/utils/mediaurl.go with ResolveMediaURL() function as single source of truth
- Update GetMediaItem and ListMediaItems handlers to return resolved URLs in API responses
- Update collection handlers (GetCollection, TestRules, PreviewCollection) to use resolved cover URLs
- Update progress handler (GetAllProgress) to use resolved cover URLs
- Add library_id to GetCollectionItems SQL query to enable URL resolution
- Refactor media scanner to store relative paths instead of absolute filesystem paths
- Add ResolveMediaPath() to LibraryService for resolving relative paths to absolute paths
- Add ServeFile endpoint at /uploads/library-:id/* for authenticated file serving
- Add MimeTypes map to library_service.go for consistent MIME type handling
- Update DownloadBook handler to use resolved filesystem paths
- Add getRelativePath() helper to MediaScanner for converting absolute to relative paths
- Use strings.EqualFold for case-insensitive path comparisons in zip extraction
This change enables the application to work with relative paths stored in the
database, making it portable across different server environments while
maintaining backward compatibility with existing absolute paths.
Extract Handler struct, constructor, and shared utilities from scanner.go
into a new commonhandlers.go file for better code organization.
Changes:
- Move Handler struct definition to commonhandlers.go
- Move NewHandler constructor to commonhandlers.go
- Move SetupRoutes function to commonhandlers.go
- Move parseDate utility function to commonhandlers.go
- Remove unused imports from scanner.go
- Create dedicated commonhandlers.go for shared HTTP handler code
This refactoring improves code maintainability by separating
concerns between scanner-specific logic and common handler utilities,
making it easier to understand and extend the handlers package.
Fix 1 - File modification time for created_at:
- Get file.ModTime() in processMediaFile and pass to CreateMediaItem
- Modified SQL INSERT to include created_at column
Fix 2 - Force rescan UPDATE instead of DELETE+INSERT:
- Changed force rescan logic to call updateMediaItem instead of delete + create
- Preserves created_at timestamp on force rescan
Fix 3 - GetMediaItemByFilePath filters by library_id:
- Added library_id to WHERE clause in SQL query
- Created GetMediaItemByFilePathAnyLibrary for cross-library lookups (KOReader)
- Added SetLibraryID method to MediaScanner
- Updated handler to call SetLibraryID for watch mode
Fix 4 - File deletion handling with persistent logging:
- Added fsnotify.Remove handler in WatchChanges
- Added orphan cleanup in ScanFolders after scan completes
- Created scanner_logger.go with daily log rotation (7 days)
- Logs to /app/logs/scanner-deletes-YYYY-MM-DD.log and scanner-errors-YYYY-MM-DD.log
- Individual deletes with enhanced safety logging
Note: Integration tests can now safely scan /app/uploads because
GetMediaItemByFilePath now filters by library_id, preventing
cross-library interference.
- Add Force bool field to ScanLibraryRequest in handlers
- Pass force param through job params to worker
- Add forceRescan field and SetForce method to MediaScanner
- Modify processMediaFile to delete and re-create existing items when force=true
- Default behavior unchanged (force=false maintains skip-if-exists)
Implements comprehensive progress tracking for scan jobs to provide real-time
statistics to the frontend (files_scanned, new_items, errors).
Changes:
1. Extended JobResult struct with new fields:
- FilesScanned: total files processed
- NewItems: books added to database
- Errors: scan errors encountered
2. Added progress callback mechanism:
- Job.ProgressCallback function field for real-time updates
- Job.UpdateProgress() method to trigger callbacks
- Worker stores callback and updates JobResult during scan
3. MediaScanner now tracks statistics:
- totalFiles, newItems, errors counters
- GetStats() method to retrieve statistics
- First pass counts total files for progress calculation
- Batches progress updates every 10 files (reduces mutex contention)
- Final update ensures 100% progress is reported
4. Updated processMediaFile signature:
- Returns (bool, error) instead of (error)
- true = new item created, false = existing/updated/error
- Increments newItems counter when creating database entries
- Updated WatchChanges to handle new return value
5. Worker job completion extracts stats:
- Parses result map for files_scanned, new_items, errors
- Stores in final JobResult for API response
6. GetScanStatus API response includes new fields:
- files_scanned, new_items, errors now in JSON response
- Frontend can display real-time progress
Design decisions:
- Batching every 10 files balances performance vs. granularity
- Thread-safe via worker mutex (w.mu.Lock/Unlock)
- Callback pattern decouples scanner from job management
- processMediaFile return type allows tracking new vs. updated items
- Maintains backward compatibility (uses || 0 fallbacks in frontend)
Testing:
- All code compiles successfully
- Follows service layer pattern (no business logic in handlers)
- No database schema changes
- Integration tests to be added in Step 8 (separate commit)
Files modified:
- internal/services/worker.go (JobResult, Job struct, processScanJob, processJob)
- internal/services/media_scanner.go (struct fields, GetStats, ScanFolders, processMediaFile)
- internal/handlers/scanner.go (GetScanStatus response)
Related: TASKS-backend-progress-tracking.md Steps 1-7
- Add BrowseDirectories() to library service with path traversal protection
- Add BrowseDirectories handler with proper error handling
- Register GET /api/libraries/browse endpoint (admin-only)
- Returns current path, parent path, and list of subdirectories
- Security: blocks "..", validates path exists, checks is directory
Fixes: Issue 2 (backend)
Add ListLibrariesData() method to LibraryHandler to support
server-side rendering of all libraries on the /admin/library page.
This follows the existing pattern of GetUserVisibleLibrariesData() and
GetLibraryTypeData() methods, which return data structures instead of
JSON for template rendering.
Changes:
- Add ListLibrariesData() method (3 lines)
- Returns []database.ListLibrariesRow for template consumption
- Called by frontend route handler for SSR
Add explicit check to verify target user exists in database before
attempting deletion. Previously, the handler would return 200 OK when
trying to delete non-existent users.
Changes:
- Add userFound flag to track if target user was found in user list
- Explicitly check pgtype.UUID.Bytes against all users' IDs
- Return 404 Not Found if user doesn't exist (before last admin check)
- Supports both JSON and HTML (HTMX) response formats
This fixes the failing test:
- TestDeleteUserConsolidated/DELETE_/api/auth/profile/:id_-_Delete_non-existent_user
The check uses the existing ListUsers result, so no additional database
query is required. The pgtype.UUID.Bytes comparison ensures exact
16-byte UUID matching.
Implement DeleteUser, ResetUserPassword, and UpdateUserAdmin handlers.
Update collections handler to check soft-deleted users. Update dashboard
service to exclude deleted users from statistics.
Removed unit tests that couldn't work without a database (nil db would
panic). Added comprehensive integration tests for the PreviewCollection
endpoint covering:
- Authentication (no auth, valid auth)
- Input validation (missing/invalid library ID, invalid JSON)
- Manual book selection
- Rule-based filtering
- Limit parameter handling
- Duplicate and invalid book ID handling
The PreviewCollection endpoint was missing authentication verification,
allowing unauthenticated access to the preview functionality. Added
check for user in context, returning 401 Unauthorized if missing.
Phase 11 - Unit and Integration Tests
Service Layer Tests (dashboard_service_test.go):
- Test filterHiddenCollections with multiple scenarios
- Test reorderCollections with custom orders
- Test sortByPriority sorting logic
- All 6 tests passing
Handler Tests (dashboard_test.go):
- Test BuildSections type conversion
- Test textToString helper function
- Test getViewAllURL mapping
- All 7 tests passing
Preview Tests (collections_preview_test.go):
- Test preview endpoint validation
- Test limit validation
- Test rule validation
- 6 test scenarios
Integration Tests (dashboard_integration_test.go):
- Test GET /api/dashboard/sections end-to-end
- Test PUT /api/dashboard/preferences
- Test POST /api/dashboard/restore-system-collection
- Test authentication and validation
- 9 test scenarios total
Part of Carousel Dashboard Plan completion
Add preview endpoint for custom section builder and rule evaluation:
Handler Implementation (internal/handlers/collections.go):
- PreviewCollection method: Evaluates filter rules and returns matching items without saving
* Accepts library_id, rules array, manual_book_ids array, and limit
* Evaluates rules against all library items using collectionService.EvaluateRules
* Adds manually selected books to results
* Deduplicates manual books (avoids adding same book twice)
* Applies limit (default: 20, max: 100)
* Returns array of BookInfo with matching items
- Helper function: mediaItemsToListMediaItemsRow
* Converts database.MediaItems to database.ListMediaItemsRow
* Required for EvaluateRules which expects ListMediaItemsRow type
Route Registration (internal/router/collections.go):
- POST /api/collections/preview
- Protected by JWT middleware
- Part of collections API group
Why This Endpoint is Necessary:
- Allows users to see what books match their filter rules BEFORE saving
- Avoids creating incorrect collections
- Enables testing different rule combinations quickly
- Reuses existing service logic (collectionService.EvaluateRules)
- Client-side preview would require downloading entire library (10,000+ books)
- Would duplicate 500+ lines of rule evaluation logic in TypeScript
- Would create maintenance nightmare keeping Go and TypeScript in sync
Bruno Test (bruno/collections/preview-collection.bru):
- Tests POST /api/collections/preview endpoint
- Validates status 200 response
- Validates items array in response
- Example request with genre filter rule
This endpoint is required for both the web UI Custom Section Builder and future mobile apps.
Add dashboard API endpoints with handler layer:
Step 1: Add SectionData to collections.go
- SectionData struct represents dashboard section (carousel of books)
- Used by: Dashboard handler, Templates (SSR), API JSON responses
- Shared type from collections.go (no duplicate definitions)
- Fields: ID, IsSystem, Title, Description, Icon, Items, ViewAllURL, Priority
Step 2: Create dashboard.go handler
- DashboardHandler struct with injected database and dashboard service
- GetSections: Returns dashboard sections as JSON (mobile apps, web UI TypeScript, plugins)
* Validates library_id parameter
* Fetches user dashboard preferences
* Configurable limit (default 20, max 100)
* Calls service layer for business logic
* Converts service types to handler types for JSON serialization
- UpdatePreferences: Saves dashboard preferences
* Validates library_id
* Upserts user dashboard preferences
- RestoreSystemCollection: Resets system collection to defaults
* Validates collection_name against allowed system collections
* Deletes user's copy (system collection reappears automatically)
- BuildSections: Converts service DashboardSection to handler SectionData
* Converts database.MediaItems to handlers.BookInfo
* Uses shared types from collections.go
- getViewAllURL: Maps system collections to their view-all URLs
- Reuses existing textToString helper from collections.go
Architecture Compliance:
- Generic API handler for reuse by SSR, mobile, plugins
- Uses shared types from collections.go (SectionData, BookInfo)
- IsSystem bool matches database field (no string conversion)
- Single service method returns structured data (simpler, less bugs)
- Handler just converts types (no matching logic needed)
- Reusable by mobile apps, web UI, plugins
- Add SessionDuration constant (7 days) and SessionDurationSec computed value
- Update JWT token expiration to use SessionDuration instead of 1 hour
- Update register/login cookie MaxAge to use SessionDurationSec (604800)
- Update register/login API response ExpiresIn to use SessionDurationSec
- Update refresh token endpoint ExpiresIn to use SessionDurationSec
- Remove redundant client-side document.cookie lines from login/register
- Add TODO comment for HTTPS cookie Secure flag
This provides Google-like persistent sessions with a single source of truth
for session duration, eliminating hardcoded values throughout the codebase.
- Set HTTP-only cookie in login handler for SSR authentication
- Set HTTP-only cookie in registration handler
- Change default redirect from /bookshelf to /dashboard
- Cookie enables browser page navigation without JavaScript
- Add LibraryTestData struct to TestDeviceSetup
- Implement CreateLibrary() for proper library creation in tests
- Implement CreateCollection() for test collection support
- Improve test isolation with dedicated library creation
This provides a more robust foundation for integration tests that need
proper library management support.
- Add handler to regenerate device auth tokens
- Add PUT /api/devices/:id/regenerate-token route
- Returns new token and sync URLs for device configuration
- Allow media items to be created/updated with invalid ISBN by storing empty string
- Fix test to use valid ISBN-13 format (9780306406157)
- Add small delay to prevent race condition in pagination test
Enhance NormalizeISBN to validate and convert ISBNs:
- Validate length (10 or 13 digits), return error if invalid
- Convert ISBN-10 to ISBN-13 by prefixing '978' and recalculating checksum
- Add NormalizeISBNSafe for backward compatibility in scanners
This ensures all ISBNs stored in database are valid ISBN-13 format.
Add strict validation to return 400 Bad Request when any media_item_id
is empty in the bulk-update request, rather than treating it as a
partial failure with 200 OK.
This aligns the handler behavior with test expectations for the
BulkUpdateBooks_EmptyBookIDs test case.
john-okeefe