Follow-up to the sweep route audit, correcting its diagnosis: the CLI
DOES read .yml bodies — via OpenCollection's `body.data` string form
(the NewDevDBSetup script always worked because it used it). The
failing scenario POSTs used `body.json` (structured) / `body.jsonBody`
(string) — keys the CLI silently ignores, sending Content-Length: 0
with form-urlencoded content-type, so every server bind failed.
- 81 bodies converted to data-form (json maps serialized; jsonBody
strings kept verbatim, {{vars}} intact).
- 6 pre-existing TRUNCATED payloads repaired: the original generator
cut bodies at the first {{var}} (every SHA-256-identified scenario
plus Sync Progress/Sync Bookmarks shipped mid-JSON and 400'd from
day one). Rewritten whole, faithful to each file's documented
purpose; book_sha256/book_uuid added to the environment.
- environments: opds_base_url correction from the sweep retained.
Verified live against the rebuilt dev container: the full koreader
scenario tree runs green through `bru run` — 14/14 (202s across
immediate/checkpoint/SHA-256 progress, 200s bookmarks, highlights,
notes; SHA-256 book identification exercised against a real
file_sha256).
Audited every request (247 .yml files) against the routes the server
actually registers (extracted from internal/router, group prefixes
resolved). Results:
- 44 stale requests: deleted where the endpoint no longer exists
(universal-progress GET/PUT /api/progress/:id — routes never existed,
the handlers were deleted in H1; queue stats/clear-failed/process;
admin profile page; auth/account) and FIXED where the endpoint exists
under a different path (kobo token-in-path scheme: every
/api/sync/kobo/<seg> and /api/v1/kobo/<seg> URL now carries
{{kobo_device_token}}; /sync/* missing their /api prefix; koreader
/api/v1/koreader/sync/* -> /api/sync/koreader/*; link-book and
unlinked-books moved to /api/devices/:id/sync/*).
- koreader Sync Highlights/Sync Notes pointed at endpoints that never
existed with a body shape to match — rewritten against POST
/api/sync/koreader/bookmarks (the annotations funnel carries
highlights and notes arrays).
- 9 files the GUI refused to load: 4 had bruno-invalid script type
"tests" (converted to after-response), 4 were info/type:collection
doc files with bogus http blocks (converted to folder.yml folder
docs), 1 had a dead pre-YAML schema and no route (deleted).
- devices/koreader requests now carry request-level bearer auth
({{koreader_device_token}}) — they inherited the collection's user-JWT
bearer, which device endpoints rightly reject — and their bodies
reference {{media_item_id}} instead of dead placeholder uuids.
- environments: opds_base_url was empty (every OPDS request missed the
/opds prefix); set to http://localhost:8765/opds and un-doubled the
prefix in opds/scenarios.
Known limitation, for the handoff: bru CLI cannot execute .yml request
bodies (the collection is GUI OpenCollection format; the CLI runner
reads bodies from .bru files only). A .bru migration is the remaining
bruno work if CLI runs are wanted.
This commit adds comprehensive functionality for filtering collections by library,
improves WebSocket real-time updates with user activity detection, and adds
extensive test coverage.
## Core Features
### Collection Library Filter
- Added library_id parameter to media-items search API
- Collections can now be filtered by specific library
- Toggle UI component for enabling/disabling library filter
- Default state is "checked" when library_id is present
- Consistent behavior across partial and fuzzy search modes
### WebSocket Auto-Reload Mitigation
- Added user activity detection to prevent disruptive page reloads
- Checks if user is actively typing in INPUT/TEXTAREA/SELECT elements
- Skips auto-reload when user is interacting with form elements
- Toast notifications still show for awareness
- Prevents data loss during editing operations
## Implementation Changes
### Backend
- internal/database/queries.sql.go: Added library filter support to search queries
- internal/handlers/media.go: Enhanced search with library_id parameter validation
- internal/handlers/collections.go: Updated collection handlers with library filtering
- internal/sync/websocket.go: Improved broadcast mechanism with user-scoped updates
- internal/router/frontend.go: Pass libraryID to collection templates
### Frontend
- templates/collections.templ: Added library filter toggle UI component
- web/src/collections.ts: TypeScript implementation with WebSocket integration
- templates/collections_templ.go: Generated template code
### Testing
- cmd/server/tests/search_test.go: Added TestCollectionSearchLibraryFilter
- cmd/server/tests/websocket_test.go: Added TestWebSocketUserScopedBroadcast
- New helper functions for creating libraries and media items via API
- Comprehensive test coverage for library filtering and user-scoped broadcasts
## API Documentation Updates
### Bruno Tests (Comprehensive Documentation)
- bruno/collections/*: Added detailed API documentation for all collection endpoints
- bruno/devices/*: Added device management and sync API documentation
- bruno/devices/kobo/api.yml: Kobo-specific sync protocol docs
- bruno/devices/koreader/api.yml: KOReader-specific sync protocol docs
- bruno/opds/*: Added OPDS feed and download endpoint documentation
- bruno/library/browse-folders.yml: Library folder browsing API docs
### New Bruno Tests
- bruno/media-items/Search All Libraries.yml: Test search without library filter
- bruno/media-items/Search Specific Library.yml: Test search with library filter
- bruno/media-items/Search Invalid Library ID.yml: Test error handling
## Documentation
- docs/developer/api/media-items/search_media_items.md: Updated with library_id parameter
- IMPLEMENTATION_COLLECTION_FIX.md: Comprehensive implementation guide with test scenarios
## Testing
### Integration Tests
- Library filter tests verify correct filtering across multiple libraries
- Invalid library_id tests ensure proper error handling
- WebSocket tests verify user-scoped broadcast behavior
- User A no longer receives User B's collection updates
### Manual Testing Scenarios
- Open collection in multiple tabs - updates propagate correctly
- Type in search box while another tab adds books - no disruptive reload
- Add/remove books from collection - toast notifications appear
- Toggle library filter - results update dynamically
## Technical Details
- WebSocket broadcasts are now user-scoped for privacy
- Active element detection uses tagName and contenteditable attributes
- Library ID validation uses UUID format checking
- Progressive enhancement maintained - page works without JavaScript
- All changes follow PROJECT_GUIDELINES.md conventions
- TypeScript only for frontend logic
- TailwindCSS only for styling
- Procedural programming style throughout
## Breaking Changes
None - all changes are additive and backward compatible.
Fix 1 - File modification time for created_at:
- Get file.ModTime() in processMediaFile and pass to CreateMediaItem
- Modified SQL INSERT to include created_at column
Fix 2 - Force rescan UPDATE instead of DELETE+INSERT:
- Changed force rescan logic to call updateMediaItem instead of delete + create
- Preserves created_at timestamp on force rescan
Fix 3 - GetMediaItemByFilePath filters by library_id:
- Added library_id to WHERE clause in SQL query
- Created GetMediaItemByFilePathAnyLibrary for cross-library lookups (KOReader)
- Added SetLibraryID method to MediaScanner
- Updated handler to call SetLibraryID for watch mode
Fix 4 - File deletion handling with persistent logging:
- Added fsnotify.Remove handler in WatchChanges
- Added orphan cleanup in ScanFolders after scan completes
- Created scanner_logger.go with daily log rotation (7 days)
- Logs to /app/logs/scanner-deletes-YYYY-MM-DD.log and scanner-errors-YYYY-MM-DD.log
- Individual deletes with enhanced safety logging
Note: Integration tests can now safely scan /app/uploads because
GetMediaItemByFilePath now filters by library_id, preventing
cross-library interference.
Phase 12 - Bruno API Tests Standardization
Convert all custom section builder tests to .yml format:
- preview-collection.bru → preview-collection.yml
- create-custom-section-manual.bru → create-custom-section-manual.yml
- create-custom-section-missing-fields.bru → create-custom-section-missing-fields.yml
- create-custom-section-rules.bru → create-custom-section-rules.yml
All Bruno tests now consistently use .yml extension across the project.
Tests cover:
- Preview collection with filter rules
- Create collection with manual book selection
- Validation of required fields
- Multiple filter rules with operators
Part of Carousel Dashboard Plan completion
Phase 10.5.1: Add /custom-section frontend route
- Added route handler in internal/router/frontend.go
- Fetches user libraries and renders custom section builder template
Phase 10.5.2: Create custom section builder template
- Created templates/custom_section.templ with full UI
- Includes section details form, filter rules builder, manual book selection
- Live preview functionality with preview container
- Form actions for save/cancel
Phase 10.5.3: Create custom-section-builder TypeScript
- Created web/src/custom-section-builder.ts with 13+ filter fields
- Filter fields: title, author, genre, series, progress, rating, date_added, last_read, publisher, language, format, tags, narrators
- Procedural/imperative style (no OOP) as per guidelines
- Rule builder with AND/OR logic support
- Book search and multi-select functionality
- Live preview via /api/collections/preview endpoint
- Form validation and submission to /api/collections
Phase 10.5.4: Build TypeScript modules
- Compiled custom-section-builder.ts to web/static/custom-section-builder.js
- Verified successful compilation with no errors
- All existing TypeScript modules continue to compile
Phase 10.5.5: Add Bruno tests for custom section creation
- create-custom-section-rules.bru: Test creating section with filter rules
- create-custom-section-manual.bru: Test creating section with manual book selection
- create-custom-section-missing-fields.bru: Test error handling for missing required fields
Phase 10.6: Build Verification
- ✅ TypeScript modules compile successfully
- ✅ Templates generate successfully
- ✅ Go build succeeds with no compilation errors
- ✅ All build artifacts verified (dashboard.js, custom-section-builder.js, dashboard_templ.go, custom_section_templ.go)
This completes the Custom Section Builder feature, allowing users to create
personalized dashboard sections with flexible filter rules or manual book selection.
Add preview endpoint for custom section builder and rule evaluation:
Handler Implementation (internal/handlers/collections.go):
- PreviewCollection method: Evaluates filter rules and returns matching items without saving
* Accepts library_id, rules array, manual_book_ids array, and limit
* Evaluates rules against all library items using collectionService.EvaluateRules
* Adds manually selected books to results
* Deduplicates manual books (avoids adding same book twice)
* Applies limit (default: 20, max: 100)
* Returns array of BookInfo with matching items
- Helper function: mediaItemsToListMediaItemsRow
* Converts database.MediaItems to database.ListMediaItemsRow
* Required for EvaluateRules which expects ListMediaItemsRow type
Route Registration (internal/router/collections.go):
- POST /api/collections/preview
- Protected by JWT middleware
- Part of collections API group
Why This Endpoint is Necessary:
- Allows users to see what books match their filter rules BEFORE saving
- Avoids creating incorrect collections
- Enables testing different rule combinations quickly
- Reuses existing service logic (collectionService.EvaluateRules)
- Client-side preview would require downloading entire library (10,000+ books)
- Would duplicate 500+ lines of rule evaluation logic in TypeScript
- Would create maintenance nightmare keeping Go and TypeScript in sync
Bruno Test (bruno/collections/preview-collection.bru):
- Tests POST /api/collections/preview endpoint
- Validates status 200 response
- Validates items array in response
- Example request with genre filter rule
This endpoint is required for both the web UI Custom Section Builder and future mobile apps.
- Move all files from bruno-yaml/* to bruno/*
- Maintains existing directory structure within categories
- Updates bruno/user/auth files with OAuth2 refresh token flow
- Updates bruno/user/profile files for user profile management
- Adds bruno/dashboard/ directory with dashboard API tests
- Preserves all existing test scenarios and OpenCollection YAML format
- No functional changes - file reorganization only
- Convert all existing .bru files from JSON to Bruno DSL format
- Remove obsolete files (conflicts/api.bru, kobo/Kobo Initialization.bru)
- Update auth configuration to use 'inherit' instead of explicit bearer tokens
- Add comprehensive documentation to all test files
- Improve test scripts with proper assertions and error handling
Add comprehensive Bruno tests for Phase 9 limitations features:
Test Collection Rules.bru:
- Test rule: genre equals "Science Fiction"
- Test rule: author contains "Asimov"
- Test rule: copyright_year greater than 2000
- Test rule: non-existent genre (expects 0 matches)
- Test validation: empty rules array (expects 400)
Bulk Remove Books.bru:
- Setup: Create test collection
- Add multiple books to collection
- Test 1: Bulk remove all books
- Test 2: Bulk remove with some invalid IDs
- Test 3: Empty list validation
- Test 4: Single book removal (bulk should work for 1 too)
- Cleanup: Delete test collection
Test Coverage:
- Rule evaluation API endpoint
- Bulk remove API endpoint
- Request validation
- Response structure verification
- Edge cases and error handling
Bruno Test Format:
- JSON request bodies
- Status code assertions
- Response structure validation
- Setup/teardown for integration tests
These tests ensure the new Phase 9 API endpoints work correctly
and maintain backward compatibility.
John O'Keefe
john-okeefe