Verifying the per-user chokepoint end to end surfaced a UI regression
it would have caused: web/src/header.ts runs the sitewide scan
indicator on scan_progress/scan_complete events in every browser, and
routing those to the triggering identity alone froze the indicator for
everyone else at its page-load state (active-summary stays the
load-time fallback, but live movement died). The source's own comment
declares the indicator sitewide by design, and scan payloads carry no
user data — job ids, file counts, progress percentages only.
Resolution honors both truths: BroadcastToAll is re-added as the ONE
sanctioned instance-wide fan-out, its doc comment making the decision
explicit (no user-derived data; anything carrying books, notes, or
annotations belongs in BroadcastToUser), and the two scan emitters use
it. TestBroadcastToAllReachesEveryone pins the exception alongside
TestBroadcastIsPerUser — audiences are now named decisions the
compiler and the tests both enforce.
Owner-flagged after the dev-docs audit: BroadcastProgressUpdate,
BroadcastAnnotationUpdate, and BroadcastConflictNotification fanned out
through the untargeted Broadcast, delivering one user's book progress,
conflict notices, and — worst — annotation payloads (which can carry
note/highlight text) to EVERY connected client on the instance. A
family-lens violation of exactly the kind the RLS move eliminated for
database reads: user B's client received events for user A's books,
including books in libraries hidden from B.
Fix is the RLS philosophy applied to the fan-out layer: enforcement at
one choke point. The untargeted Broadcast method and its channel/loop
machinery are deleted — BroadcastToUser is now the only door, and the
compiler forces every future emitter to name its audience instead of
relying on review. The three helpers take the acting user's ID; the
four sync call sites (progress.go x2, annotations.go x2) already had
the acting user in scope. Scan events were already per-triggering-
identity and collections already per-user — unchanged. Logging is
untouched: the only log line in the diff is the deleted method's own,
so docker logs remains the complete administrator audit trail.
TestBroadcastIsPerUser pins the invariant (alice's two devices receive
her three events; bob's client receives nothing — any message there is
a cross-user leak). Both websocket docs updated in the same commit:
the wire doc's fan-out section now states per-user delivery and the
chokepoint, and the gated endpoint doc's intro matches. Full make test
green.
CancelJob existed but was dead code: it only flipped a status in the
in-memory results map - it never cancelled anything, and processJob's
unconditional completion write overwrote the marker anyway.
- derive a per-job context in processJob (parent is the handler ctx
for manual scans, Background for auto-scans) and register its
cancel func under the job id; previously every manual scan shared
the handler's process-lifetime context, so no targeted cancel was
possible without killing everything
- skip execution at dequeue for jobs cancelled while still queued
- record a pending JobResult on BOTH enqueue paths (Enqueue used by
auto-scans, EnqueueJob used by the API): queued jobs were invisible
to status polling (404 until a worker picked them up) and could not
be cancelled before execution started
- cancelled jobs keep their last real progress instead of a fake 100%
and drop the meaningless "context canceled" error text
- broadcast the new scan_cancelled websocket event for cancelled scan
jobs instead of scan_complete, so clients reset their scanning
indicators rather than reading the abort as a finished scan
- sentinel errors (ErrJobNotFound, ErrJobNotCancellable) so HTTP
handlers map outcomes to 404/409 without string matching
Unit tests (worker_cancel_test.go) pin: queued-cancel never executes,
running-cancel invokes the context, error mapping, pending recording
on both enqueue paths.
Add new message type constants for real-time scan progress updates:
- MessageTypeScanProgress: broadcast progress during scanning
- MessageTypeScanComplete: notify when scan completes
- MessageTypeScanError: report scan errors
These enable frontend to receive live scan updates instead of polling.
This commit adds comprehensive functionality for filtering collections by library,
improves WebSocket real-time updates with user activity detection, and adds
extensive test coverage.
## Core Features
### Collection Library Filter
- Added library_id parameter to media-items search API
- Collections can now be filtered by specific library
- Toggle UI component for enabling/disabling library filter
- Default state is "checked" when library_id is present
- Consistent behavior across partial and fuzzy search modes
### WebSocket Auto-Reload Mitigation
- Added user activity detection to prevent disruptive page reloads
- Checks if user is actively typing in INPUT/TEXTAREA/SELECT elements
- Skips auto-reload when user is interacting with form elements
- Toast notifications still show for awareness
- Prevents data loss during editing operations
## Implementation Changes
### Backend
- internal/database/queries.sql.go: Added library filter support to search queries
- internal/handlers/media.go: Enhanced search with library_id parameter validation
- internal/handlers/collections.go: Updated collection handlers with library filtering
- internal/sync/websocket.go: Improved broadcast mechanism with user-scoped updates
- internal/router/frontend.go: Pass libraryID to collection templates
### Frontend
- templates/collections.templ: Added library filter toggle UI component
- web/src/collections.ts: TypeScript implementation with WebSocket integration
- templates/collections_templ.go: Generated template code
### Testing
- cmd/server/tests/search_test.go: Added TestCollectionSearchLibraryFilter
- cmd/server/tests/websocket_test.go: Added TestWebSocketUserScopedBroadcast
- New helper functions for creating libraries and media items via API
- Comprehensive test coverage for library filtering and user-scoped broadcasts
## API Documentation Updates
### Bruno Tests (Comprehensive Documentation)
- bruno/collections/*: Added detailed API documentation for all collection endpoints
- bruno/devices/*: Added device management and sync API documentation
- bruno/devices/kobo/api.yml: Kobo-specific sync protocol docs
- bruno/devices/koreader/api.yml: KOReader-specific sync protocol docs
- bruno/opds/*: Added OPDS feed and download endpoint documentation
- bruno/library/browse-folders.yml: Library folder browsing API docs
### New Bruno Tests
- bruno/media-items/Search All Libraries.yml: Test search without library filter
- bruno/media-items/Search Specific Library.yml: Test search with library filter
- bruno/media-items/Search Invalid Library ID.yml: Test error handling
## Documentation
- docs/developer/api/media-items/search_media_items.md: Updated with library_id parameter
- IMPLEMENTATION_COLLECTION_FIX.md: Comprehensive implementation guide with test scenarios
## Testing
### Integration Tests
- Library filter tests verify correct filtering across multiple libraries
- Invalid library_id tests ensure proper error handling
- WebSocket tests verify user-scoped broadcast behavior
- User A no longer receives User B's collection updates
### Manual Testing Scenarios
- Open collection in multiple tabs - updates propagate correctly
- Type in search box while another tab adds books - no disruptive reload
- Add/remove books from collection - toast notifications appear
- Toggle library filter - results update dynamically
## Technical Details
- WebSocket broadcasts are now user-scoped for privacy
- Active element detection uses tagName and contenteditable attributes
- Library ID validation uses UUID format checking
- Progressive enhancement maintained - page works without JavaScript
- All changes follow PROJECT_GUIDELINES.md conventions
- TypeScript only for frontend logic
- TailwindCSS only for styling
- Procedural programming style throughout
## Breaking Changes
None - all changes are additive and backward compatible.
Critical fixes to prevent goroutine leaks during application shutdown:
1. Sync Queue Processor:
- Changed StartCleanupTask() to return context.CancelFunc
- Modified to accept and watch cancellable context
- Added queue context/cancel to Handler struct
- Created StartBackgroundTasks() method for main handler instance
- Cancel queue processor during shutdown in StopScheduler()
2. Connection Manager:
- Modified StartCleanupTask() to use cancellable context
- Returns cancel function that can be called during shutdown
- Goroutine now properly exits when context is cancelled
3. Handler Lifecycle:
- Added StartBackgroundTasks() to Handler
- Only main handler instance starts background goroutines
- Temporary handler instances (library/sync routes) don't start tasks
- StopScheduler() now properly shuts down all background goroutines
4. Router Integration:
- Updated SetupRoutes to accept queueProcessor parameter
- Main scanner handler starts background tasks after creation
- Library and sync route handlers don't start duplicate tasks
Impact:
- Fixes 2 major goroutine leaks (queue processor + connection cleanup)
- Application now properly shuts down all goroutines on exit
- No more resource leaks from long-running goroutines
- Test added to detect future goroutine regressions
Test: TestGoroutineCleanup verifies background services can be stopped.
Implement conflict detection for concurrent reading progress updates from different devices. Adds conflict management endpoints for listing, viewing, and resolving conflicts.
- Add ConflictHandler with CRUD endpoints for conflict management
- Implement automatic conflict detection in KOReader progress updates
- Add WebSocket broadcast for real-time conflict notifications
- Add database query for listing user conflicts by status
- Add integration tests and Bruno API test collection
Add ConnectionManager for real-time WebSocket communication:
- Message types for progress updates, annotations, conflicts
- Broadcast message structure with source device tracking
- Device connection tracking with user and device metadata
- Automatic broadcast loop with concurrent message delivery
- Connection management (add, remove, get by ID/user)
- Stale connection cleanup (2-minute timeout)
- Connection statistics by device type
- Background cleanup task runs every minute
This implements the core WebSocket infrastructure needed for
Week 9 of the Universal Sync Implementation Guide.
John O'Keefe
john-okeefe