Phase 3 (EPUB half) of the reader redesign:
- Select text in a reflowable book → floating glass popover at the
selection (5 colors, note, copy). Clicking a color creates the
highlight via POST /api/media-items/:id/highlights, anchored by the
foliate range CFI (epubcfi_start) with percentage position.
- Highlights render through foliate's overlayer pipeline: draw-
annotation draws Overlayer.highlight with the stored color,
create-overlay re-adds persisted highlights as sections load,
show-annotation opens the edit popover when a highlight is clicked
(recolor, edit note, copy, delete).
- Backend: highlight create/update accept epubcfi_start/end,
note_text, and percentage fields; position validation relaxed
(CFIs exceed the old 100-char cap); PUT routes through
AnnotationService.SaveHighlight so edits get dedup/LWW treatment
and actually persist note_text (the plain query can't).
- Bookmarks drawer becomes the Annotations drawer with tabs:
Highlights (color-bar list, note previews, jump/edit/delete),
Notes (add note at current position, list, delete — backed by the
existing notes API), and Bookmarks (unchanged behavior).
- Popover dismissed on outside click, collapsed selection, page
navigation, or Esc (new top-priority Esc branch).
Phase 0 of the reader redesign:
- Panels no longer render under the top/bottom bars: sidebars get
measured insets (same resize/safe-area mechanism as the viewport);
panel max-height now derives from the bounded sidebar instead of a
100vh guess; right-side border targets the actual sidebar.
- Bookmarks work end-to-end for the first time: REST CRUD under
/api/media-items/:id/bookmarks (create/delete route through
AnnotationService for dedup/LWW/tombstones), fix UpdateMediaBookmark
referencing nonexistent updated_at column, frontend posts to the
real API with per-format position (CFI vs page), live list with
jump + delete instead of SSR-only snapshot.
- Fix chapter matching in progress saves: boundaries were compared by
a nonexistent tocItem property, so chapter was never persisted.
- Remove dead UI: Navigator panel stub, empty dictionary popup shell,
unwired Chrome Behavior select; purge 160 stale build artifacts.
- Reader chrome now follows the user's app theme instead of hardcoded
theme-tokyo-night.
Search results navigated to /bookshelf with no filters instead of the
selected book's page. Results now link to /media/:id and display cover
thumbnails, with cover URLs resolved server-side via ResolveMediaURL.
Removes the dead selectedBook localStorage plumbing.
Complete the annotation sync pipeline across all ingest and serve paths.
Previously, annotations sent inline with KOReader progress pushes were
silently discarded, and no annotations were ever served back to devices.
INGEST (device → server):
KOReader (koreader.go):
- Add processBookAnnotations helper that processes inline highlights,
notes, and bookmarks from every progress push (immediate + checkpoint)
- Highlights get CRE→CFI position conversion before SaveHighlight
- KOReader 'notes' (text + notes) stored as highlights with NoteText
to ensure correct round-trip classification
- Bookmarks routed through SaveBookmark with device sync data
- Called from both updateProgressForBook and handleCheckpointSync
Kobo (kobo.go):
- Markup handler: annotations and bookmarks route through
AnnotationService (SaveHighlight/SaveBookmark)
- Bookmark handler: same routing with device sync data
- SyncFromServer handler: same routing
- All handlers fall back to direct DB calls when annotationSvc == nil
Web reader (media.go):
- CreateMediaHighlight → SaveHighlight (Source="web", ModifiedAt=now)
- CreateMediaNote → SaveNote (Source="web")
- DeleteMediaHighlight → TombstoneHighlightByID
- DeleteMediaNote → TombstoneNoteByID (was hard delete, now tombstone)
- All fall back to old behavior when annotationSvc == nil
SERVE (server → device):
KOReader GetMetadata (koreader.go):
- Query and serve bookmarks from media_bookmarks table (was missing)
- Serve deleted_highlights and deleted_bookmarks arrays containing
device_sync_data + dedup_key for client-side deletion
- Highlights/notes already served with reverse CFI conversion
Kobo Markup handler (kobo.go):
- Track processed books during sync
- Query tombstones per book, extract bookmark_id from device_sync_data
- Return DeletedAnnotations array in KoboSyncStatus response
Conflict resolution (conflicts.go):
- Enable annotation conflict types in ResolveConflict handler
- Add applyAnnotationResolution dispatching to:
applyHighlightResolution / applyBookmarkResolution / applyNoteResolution
- Each looks up by dedup_key and applies winner's fields
- Allow manual override of auto_resolved conflicts
(changed check from != "unresolved" to == "user_resolved")
Infrastructure:
- AnnotationService field + SetAnnotationService in router Config
- Inject AnnotationService into KOReader, Kobo, Media handlers
- Start tombstone purger goroutine in main.go (24h interval)
- Test helpers: construct AnnotationService in test setup
Web reader can now send surrounding text at current reading position.
Stored in reading_progress.context_text for use as CFI resolution
fallback when converting epubcfi to CREngine XPointer.
All four progress write paths now delegate to ProgressService.SaveProgress:
- MediaHandler: UpdateMediaReadingProgress uses ProgressService for web
saves with richer request body (reading_mode, zoom_level, scroll). GET
now uses GetUniversalProgress query that JOINs media_items for
format_group, total_characters, chapter_count.
- KOReaderHandler: updateProgressForBook delegates to ProgressService.
Fixed device ID bug (was using userID, now uses deviceID). Removed
duplicate UpdateDeviceLastSync with zero UUID. Added pgtype helper
functions (textPtrToPgText, intPtrToPgInt4, int64PtrToPgInt8).
- KoboHandler: all four progress write points (Markup ReadingSync, Markup
last-read-place, AnalyticsGettests, SyncFromServer) delegate to
ProgressService. Fixed empty epubcfi string now correctly set to
Valid: false. SyncFromServer preserves last_sync_source=bookhoard
and Broadcast: false.
- QueueProcessor: syncProgress delegates to ProgressService.
- main.go: creates ProgressService after ConnectionManager, injects via
SetProgressService() on all handlers and queue processor.
Handler tests cover pgtype conversion helpers (textPtrToPgText, etc.)
and device icon mapping.
Replace direct error equality checks (err == pgx.ErrNoRows, err != http.ErrServerClosed)
with the idiomatic errors.Is() function throughout handlers, services, middleware, and app
startup. This correctly handles wrapped error chains.
Also replace a raw type assertion (*HTTPError) with errors.AsType[*HTTPError]() in the
error handler middleware for consistency.
Additionally, rename shadowed variables for clarity:
- sidecar.go: config -> sidecarConfig, systemConfig (shadowed package-level vars)
- media_scanner.go: uuid -> uuidString (shadowed the uuid package import)
Refactor UpdateMediaReadingProgress handler to use UpdateUniversalProgress
database function, providing comprehensive progress tracking capabilities
including device sync information, viewport data, reading mode, and scroll
position for enhanced cross-platform reading synchronization.
Enhance reading progress tracking to support EPUB-specific location data:
- Add epubcfi field to store EPUB Canonical Fragment Identifier
- Add percentage field for normalized position across formats
- Update UpdateReadingProgress API handler to accept new fields
- Modify database queries to persist additional progress metadata
This enables precise position tracking in reflowable EPUB content
where page numbers are insufficient for accurate bookmarking.
- Add library_type_name to GetMediaItem handler response in media.go
- Remove empty LibraryName and LibraryTypeName fields from:
- ListMediaItemsRow in collections.go handler
- ListMediaItemsRow in dashboard_service.go
- These fields are now populated at the database level via trigger
The library_type_name is now automatically populated in the database
when a media item is created, so we remove the manual empty string
assignments and expose the actual value in the API response.
- Fix pgtype.UUID usage in test files by properly converting string UUIDs to pgtype.UUID
- Update numericToFloat to use pgtype.Float8 instead of pgtype.Numeric for DOUBLE PRECISION support
- Fix field name from WebURL to WebUrl to match current schema
These changes align with the recent community_rating type change to DOUBLE PRECISION
and ensure consistent type handling across the codebase.
Updated the backend services and handlers to properly detect and pass
the has_cover parameter's validity state to the database layer.
Changes:
- services/search.go: Changed HasCover type from bool to pgtype.Bool
to support 3-state logic (NULL, TRUE, FALSE)
- handlers/media.go: Fixed 3-state detection by checking if has_cover
exists in query params before setting Valid flag
- router/search.go: Fixed 3-state detection to match media.go logic
- router/frontend.go: Use pgtype.Bool{Valid: false} for SSR initial
load to ensure no filtering occurs on first page load
The key fix is detecting whether the has_cover parameter was actually
sent in the request:
- Parameter not sent → pgtype.Bool{Bool: false, Valid: false}
- Parameter sent as "true" → pgtype.Bool{Bool: true, Valid: true}
- Parameter sent as "false" → pgtype.Bool{Bool: false, Valid: true}
Previously, media.go was hardcoding Valid: true, which meant it was
always filtering by has_cover=false (only books without covers) when
the parameter wasn't sent, causing searches to incorrectly return
0 results for queries like "1984".
This ensures consistency between the JSON API endpoint (media.go) and
the HTML endpoint (search.go), and fixes the critical bug where SSR
was returning 0 books on initial page load.
Add public ExecuteSearch method to MediaHandler that delegates to SearchService. Update SearchMediaItems to use the new shared service method instead of calling SearchMediaItemsUnified directly.
- Add ExecuteSearch wrapper method (line 160-162)
- Update SearchMediaItems to use searchService.ExecuteSearch
- Maintains existing JSON API behavior while enabling shared logic
- Extract tags_filter query parameter in handler
- Add tags autocomplete route handler
- Add tags case to field values search endpoint
- Keep genre_filter for backward compatibility
Provides HTTP endpoints for filtering by tags and getting autocomplete
suggestions for tag values.
Relates to IMPLEMENTATION_TAGS_FILTER.md Phase 3
- Update SearchMediaItems handler to use SearchService
- Add autocomplete detection for field value queries (author=value, genre=value, etc.)
- Add handleFieldValuesSearch method for dropdown population
- Add sort parameter extraction with default "title ASC"
- Remove deprecated ListMediaItemsFiltered handler
- Remove deprecated /api/media-items/filtered route registration
- Update frontend.go to use SearchMediaItemsUnified instead of ListMediaItemsFiltered
- Fix parameter passing (empty filters use Valid:true with empty values, not Valid:false)
- Add SearchQuery, IsExactSearch, SearchPattern parameters for query parsing
Handler is now a thin wrapper that extracts params and delegates to service layer.
This commit enhances the SearchMediaItems handler to support dual-mode
operation: unified search with filters AND autocomplete queries for
dropdown suggestions.
**Autocomplete Detection:**
- Detects autocomplete queries: author=value, genre=value, series=value, language=value
- Routes to new handleFieldValuesSearch method for dropdown population
- Returns JSON format: {"results": [{"value": "...", "count": 47, "score": 0.8}], "total": 1}
**Unified Search Integration:**
- Replaced direct DB calls (SearchMediaItems, SearchMediaItemsFuzzy) with SearchService
- Added support for all fuzzy filters: author_filter, genre_filter, series_filter, language_filter
- Added exact filters: year_min, year_max, has_cover
- Combined search query + filters in single SearchMediaItemsUnified call
- Removed fallback logic (partial → fuzzy), now single query with smart ordering
**New Method: handleFieldValuesSearch**
- Handles autocomplete queries for all field types (author, genre, series, language)
- Validates library_id requirement
- Applies default limit=50 if not specified
- Calls SearchService.SearchFieldValues() with FieldSearchParams
- Returns consistent JSON format with results array and total count
**QueryParam Handling:**
- Fixed to not use default values (Echo QueryParam only accepts single argument)
- Properly handles empty limit parameter with default fallback
- Extracts all filter parameters for unified search
**Behavior Changes:**
- SearchMediaItems no longer requires 'q' parameter (filters-only queries now valid)
- Autocomplete queries detected before filter processing (correct priority)
- Better error messages and logging
**Service Layer Pattern:**
- Follows established pattern (FiltersService, CollectionService)
- Handler is thin - extracts params and calls service
- Business logic in SearchService (created in commit 9ab2796)
**Backward Compatibility:**
- All existing query parameters still supported
- Response format unchanged for media items search
- New response format for autocomplete queries (distinct field values)
Update MediaHandler to use new SearchService:
Changes:
- Add searchService field to MediaHandler struct
- Instantiate SearchService in NewMediaHandler constructor
- Follows established pattern (FiltersService, CollectionService)
- Keeps handler dependencies self-contained, no main.go changes needed
Design rationale:
- Handler owns its service dependencies
- Simpler initialization than passing from main.go
- More testable with direct service instantiation
- Consistent with existing codebase patterns
Next steps: Handler methods will delegate to searchService for
search operations (implementation in follow-up commits).
Replace echo's c.File() with standard library http.ServeFile() in the
ServeFile handler. This provides more reliable static file serving and
better handles edge cases in file delivery.
- Fix SearchMediaItems to retrieve user object from context instead of string
- Remove redundant UUID parsing, use user.ID directly
- Add error logging for search failures with query details
- Fix JWT middleware to use echo.NewHTTPError for consistent error format
- Improves debugging and error response consistency across API
This commit adds comprehensive functionality for filtering collections by library,
improves WebSocket real-time updates with user activity detection, and adds
extensive test coverage.
## Core Features
### Collection Library Filter
- Added library_id parameter to media-items search API
- Collections can now be filtered by specific library
- Toggle UI component for enabling/disabling library filter
- Default state is "checked" when library_id is present
- Consistent behavior across partial and fuzzy search modes
### WebSocket Auto-Reload Mitigation
- Added user activity detection to prevent disruptive page reloads
- Checks if user is actively typing in INPUT/TEXTAREA/SELECT elements
- Skips auto-reload when user is interacting with form elements
- Toast notifications still show for awareness
- Prevents data loss during editing operations
## Implementation Changes
### Backend
- internal/database/queries.sql.go: Added library filter support to search queries
- internal/handlers/media.go: Enhanced search with library_id parameter validation
- internal/handlers/collections.go: Updated collection handlers with library filtering
- internal/sync/websocket.go: Improved broadcast mechanism with user-scoped updates
- internal/router/frontend.go: Pass libraryID to collection templates
### Frontend
- templates/collections.templ: Added library filter toggle UI component
- web/src/collections.ts: TypeScript implementation with WebSocket integration
- templates/collections_templ.go: Generated template code
### Testing
- cmd/server/tests/search_test.go: Added TestCollectionSearchLibraryFilter
- cmd/server/tests/websocket_test.go: Added TestWebSocketUserScopedBroadcast
- New helper functions for creating libraries and media items via API
- Comprehensive test coverage for library filtering and user-scoped broadcasts
## API Documentation Updates
### Bruno Tests (Comprehensive Documentation)
- bruno/collections/*: Added detailed API documentation for all collection endpoints
- bruno/devices/*: Added device management and sync API documentation
- bruno/devices/kobo/api.yml: Kobo-specific sync protocol docs
- bruno/devices/koreader/api.yml: KOReader-specific sync protocol docs
- bruno/opds/*: Added OPDS feed and download endpoint documentation
- bruno/library/browse-folders.yml: Library folder browsing API docs
### New Bruno Tests
- bruno/media-items/Search All Libraries.yml: Test search without library filter
- bruno/media-items/Search Specific Library.yml: Test search with library filter
- bruno/media-items/Search Invalid Library ID.yml: Test error handling
## Documentation
- docs/developer/api/media-items/search_media_items.md: Updated with library_id parameter
- IMPLEMENTATION_COLLECTION_FIX.md: Comprehensive implementation guide with test scenarios
## Testing
### Integration Tests
- Library filter tests verify correct filtering across multiple libraries
- Invalid library_id tests ensure proper error handling
- WebSocket tests verify user-scoped broadcast behavior
- User A no longer receives User B's collection updates
### Manual Testing Scenarios
- Open collection in multiple tabs - updates propagate correctly
- Type in search box while another tab adds books - no disruptive reload
- Add/remove books from collection - toast notifications appear
- Toggle library filter - results update dynamically
## Technical Details
- WebSocket broadcasts are now user-scoped for privacy
- Active element detection uses tagName and contenteditable attributes
- Library ID validation uses UUID format checking
- Progressive enhancement maintained - page works without JavaScript
- All changes follow PROJECT_GUIDELINES.md conventions
- TypeScript only for frontend logic
- TailwindCSS only for styling
- Procedural programming style throughout
## Breaking Changes
None - all changes are additive and backward compatible.
- Implement event queue with 3-second debouncing for file system events
- Add configurable polling fallback (default 3 min) via SCAN_POLL_INTERVAL_MINUTES
- Add SyncFilesystemWithDatabase to detect orphaned DB entries and new files
- Integrate utils.ResolveMediaURL for consistent media file path resolution
- Add COOKIE_SECURE env var with SameSite=LaxMode for session cookies
- Update media handler to properly decode URL paths for file serving
- Refactor scanner initialization to accept poll interval configuration
Remove the FilePath field from book metadata responses in the GetShelf endpoint.
This change improves security by not exposing internal file paths to API clients,
as the application now uses relative path storage with URL resolution via the
library service.
Changes:
- Remove FilePath field from BookPreview struct in GetShelf response
- Remove FilePath field from shelf items response
Related to previous commit implementing relative path storage.
- Add libraryService dependency to CollectionHandler and OPDSHandler for centralized path resolution
- Create internal/utils/mediaurl.go with ResolveMediaURL() function as single source of truth
- Update GetMediaItem and ListMediaItems handlers to return resolved URLs in API responses
- Update collection handlers (GetCollection, TestRules, PreviewCollection) to use resolved cover URLs
- Update progress handler (GetAllProgress) to use resolved cover URLs
- Add library_id to GetCollectionItems SQL query to enable URL resolution
- Refactor media scanner to store relative paths instead of absolute filesystem paths
- Add ResolveMediaPath() to LibraryService for resolving relative paths to absolute paths
- Add ServeFile endpoint at /uploads/library-:id/* for authenticated file serving
- Add MimeTypes map to library_service.go for consistent MIME type handling
- Update DownloadBook handler to use resolved filesystem paths
- Add getRelativePath() helper to MediaScanner for converting absolute to relative paths
- Use strings.EqualFold for case-insensitive path comparisons in zip extraction
This change enables the application to work with relative paths stored in the
database, making it portable across different server environments while
maintaining backward compatibility with existing absolute paths.
- Allow media items to be created/updated with invalid ISBN by storing empty string
- Fix test to use valid ISBN-13 format (9780306406157)
- Add small delay to prevent race condition in pagination test
Enhance NormalizeISBN to validate and convert ISBNs:
- Validate length (10 or 13 digits), return error if invalid
- Convert ISBN-10 to ISBN-13 by prefixing '978' and recalculating checksum
- Add NormalizeISBNSafe for backward compatibility in scanners
This ensures all ISBNs stored in database are valid ISBN-13 format.
Add strict validation to return 400 Bad Request when any media_item_id
is empty in the bulk-update request, rather than treating it as a
partial failure with 200 OK.
This aligns the handler behavior with test expectations for the
BulkUpdateBooks_EmptyBookIDs test case.
- Check library has folders before creating media items
- Return HTTP 400 with clear error message if no folders
- Proper error code (400) instead of generic 500
- Improved user feedback for invalid operations
- Inject LibraryService into MediaHandler
Fixes: TestCollectionsBulkOperations HTTP 500 errors
Related: Service layer validation commit
Update CreateMediaItem handler:
- Normalize tags for display using NormalizeTags()
- Normalize contributors for display using NormalizeContributors()
- Generate tags_search using NormalizeTagsSearch()
- Generate contributors_search using NormalizeContributorsSearch()
- Pass search fields to database
Update UpdateMediaItem handler:
- Same normalization logic as CreateMediaItem
- Regenerate search fields on updates
Update HandleBulkUpdate handler:
- Add tag normalization with punctuation preference
- Regenerate search fields when tags/contributors updated
All handlers now populate both display and search fields, ensuring
consistent normalization throughout the application.
Relates to Tags & Contributors Migration Phase 6
Changed HandleBulkUpdate to check tags array length before assignment
instead of checking for nil, improving consistency with array handling
and preparing for dual-field normalization implementation.
Convert tags and contributors columns from comma-separated strings to PostgreSQL
TEXT[] arrays for better data normalization and query performance.
Database Changes:
- schema.sql: Change tags/contributors from TEXT to TEXT[]
- schema.sql: Add GIN indexes for fast array searches
- queries.sql: Update search queries to use ANY() operator
- queries.sql: Update fuzzy search with unnest() for arrays
Generated Code (sqlc):
- models.go: Auto-generated with []string types for tags/contributors
- queries.sql.go: Auto-generated with proper array handling
Handler Changes:
- media.go: Update request structs to use []string for tags/contributors
- media.go: Remove pgtype.Text wrapping, use direct array assignment
- media.go: Add tag normalization in CreateMediaItemHandler
- collections.go: Update tags evaluation to join arrays for comparison
- collections.go: Add strings import for Join() function
Service Changes:
- ebook_scanner.go: Update EbookMetadata struct to use []string
- ebook_scanner.go: Remove string Join(), assign arrays directly
- collection_service.go: Update tags rule evaluation to join arrays
- collection_service.go: Add strings import
New Utilities:
- internal/utils/tags.go: Create NormalizeTags(), JoinTags(), SplitTags()
- Normalizes tags by trimming, lowercasing, removing duplicates/empties
API Documentation:
- bruno/media-items/Create Media Item.bru: Update examples to use arrays
- bruno/media-items/Update Media Item.bru: Update examples to use arrays
- Update docs: tags/contributors now array of string
Breaking Change:
- JSON format changes from "tags": "tag1,tag2" to "tags": ["tag1", "tag2"]
- Tests already use array format (no changes needed)
Benefits:
- GIN indexes enable faster array searches
- Normalization prevents data quality issues (case, duplicates)
- Array operations use PostgreSQL native operators (ANY, &&, unnest)
- Better separation of concerns (no string parsing in application)
- Remove 24 duplicate media CRUD methods from ebook.go (884 lines removed)
- Keep 12 scanner/watch/scheduler methods on Handler
- Move request type declarations to media.go:
* CreateMediaItemRequest
* UpdateMediaItemRequest
* CreateMediaNoteRequest
* UpdateMediaNoteRequest
* CreateMediaHighlightRequest
* UpdateMediaHighlightRequest
- Remove unused imports from ebook.go (strconv, pgx)
- Fix library.go to use MediaHandler.ListMediaItems instead of Handler
ebook.go reduced from 1266 lines to 382 lines (70% reduction)
Handler now has focused responsibility: scanner and scheduler operations only
This completes Phase 7 of the ebook.go refactoring plan.
Result: Clean separation of concerns with no duplicate code
john-okeefe