- Remove CreateEbookNote, UpdateEbookNote, DeleteEbookNote queries
- These were marked as backward compatibility but never used
- API uses CreateMediaNote, UpdateMediaNote, DeleteMediaNote instead
- Remove misleading backward compatibility comments
- Regenerate sqlc code
- Add detailed line-by-line plan for renaming EbookScanner to MediaScanner
- Include database cleanup (remove unused backward compatibility functions)
- Cover all test files, handlers, routers, and documentation
- 9 phases with specific file/line references for safe implementation
- Includes verification steps and rollback plan
- Remove EBOOK_REFACTOR_PLAN.md (superseded by comprehensive plan)
- Remove TAGS_CONTRIBUTORS_IMPLEMENTATION_PLAN.md (completed/combined into other work)
- Update api-reference.md with current endpoint list
- Update bulk_update_books.md with improved documentation
- Update scanner/overview.md to reference scan_library.md
- Add create_media_item.md for media item creation API
- Add delete_rating.md for rating deletion endpoint
- Add update_rating.md for rating update endpoint
- Rename ScanEbooks method to ScanLibrary to reflect generic media scanning
- Rename ScanEbooksRequest to ScanLibraryRequest
- Update route handlers in scanner.go and library.go
- Rename scan_ebooks.md to scan_library.md
- No breaking changes: API endpoint remains POST /api/scanner/scan
- sync_progress.md - POST /api/sync/koreader/progress
- get_metadata.md - GET /api/sync/koreader/metadata/:uuid
- get_library.md - GET /api/sync/koreader/library
- sync_bookmarks.md - POST /api/sync/koreader/bookmarks
KOReader device sync endpoints with device authentication
for progress, metadata, library, and bookmarks
- get_universal_progress.md - GET /api/progress/:id
- update_universal_progress.md - POST /api/progress/:id
- get_progress_history.md - GET /api/progress/:id/history
Documents device-agnostic (universal) reading progress tracking
that works across all devices (Kobo, KOReader, etc.)
- Add bulk_delete_books.md for POST /api/books/bulk-delete
- Add bulk_update_books.md for POST /api/books/bulk-update
- Includes tag/contributor normalization details
- Documents dual-field normalization behavior
- Add download_book.md for GET /api/books/:uuid/download
- Public endpoint with auth for non-public libraries
- Documents Content-Type headers for different formats
Completes books operations API section
Update Update Media Item.bru to document normalization behavior:
- Note that tags and contributors are auto-normalized (same as Create)
- Document response includes updated search fields
Relates to Tags & Contributors Migration documentation updates
Update Bulk Update Books.bru to demonstrate tag and contributor updates:
- Add tags update example with normalized output
- Add contributors update example
- Shows punctuation preference behavior
Demonstrates that tags and contributors are automatically normalized
when updated via bulk operations, ensuring consistency across the database.
Relates to Tags & Contributors Migration documentation updates
Create comprehensive documentation explaining:
- Dual-field architecture (display vs search fields)
- Normalization rules for tags and contributors
- API request/response examples
- Frontend implementation guidelines
- Search query behavior with examples
- Checkbox filter integration
- Common mistakes to avoid
- Schema reference with indexes
- Complete example flows
This guide helps frontend developers understand:
- How to display normalized tags/contributors
- How to implement search functionality
- Why there are two sets of fields
- Best practices for filter UIs
Relates to Tags & Contributors Migration Phase 9
Update extractEPUBMetadata:
- Normalize tags for display using NormalizeTags()
- Normalize contributors for display using NormalizeContributors()
- Preserves extracted metadata formatting while ensuring consistency
Update processEbookFile:
- Add normalization before database insert
- Generate tags_search using NormalizeTagsSearch()
- Generate contributors_search using NormalizeContributorsSearch()
- Pass both display and search fields to CreateMediaItem
Scanner now produces normalized metadata matching user input normalization,
ensuring consistency between scanned and manually entered media items.
Relates to Tags & Contributors Migration Phase 7
Update CreateMediaItem handler:
- Normalize tags for display using NormalizeTags()
- Normalize contributors for display using NormalizeContributors()
- Generate tags_search using NormalizeTagsSearch()
- Generate contributors_search using NormalizeContributorsSearch()
- Pass search fields to database
Update UpdateMediaItem handler:
- Same normalization logic as CreateMediaItem
- Regenerate search fields on updates
Update HandleBulkUpdate handler:
- Add tag normalization with punctuation preference
- Regenerate search fields when tags/contributors updated
All handlers now populate both display and search fields, ensuring
consistent normalization throughout the application.
Relates to Tags & Contributors Migration Phase 6
Schema changes:
- Add tags_search TEXT[] column for case-insensitive, punctuation-free search
- Add contributors_search TEXT[] column for case-insensitive, punctuation-free search
- Create GIN indexes for fast array searches on both search fields
Query updates:
- CreateMediaItem: Include tags_search and contributors_search parameters
- UpdateMediaItem: Include tags_search and contributors_search parameters
- SearchMediaItems: Search against tags_search instead of tags
- SearchMediaItems: Search against contributors_search instead of contributors
- SearchMediaItemsFuzzy: Use tags_search and contributors_search for fuzzy matching
- Update ranking and priority logic to use search fields
Benefits:
- Case-insensitive search: "acme corp" finds "ACME CORP."
- Punctuation-agnostic search: "oreilly" finds "O'Reilly Media"
- Better UX: Users don't need to match exact casing or punctuation
- Improved performance with dedicated GIN indexes
Relates to Tags & Contributors Migration Phases 5 & 8
Add 100+ test cases covering all normalization scenarios:
- Empty/nil inputs, whitespace trimming
- Titlecasing with hyphens, apostrophes, multi-word tags
- Punctuation preference (hyphens, periods, apostrophes)
- Case-insensitive deduplication with and without punctuation
- Contributor case preservation (CAPSLOCK, Title Case, lowercase)
- Edge cases: only punctuation, multiple spaces, mixed content
New test scenarios for punctuation preference:
- Prefer "Science-Fiction" over "science fiction"
- Prefer "O'Reilly Media" over "OReilly Media"
- Prefer "ACME CORP." over "acme corp"
- Test deduplication when punctuated version appears later in array
All tests passing ✓
Relates to Tags & Contributors Migration Phase 4
Add direct dependency on golang.org/x/text to support proper
titlecasing with punctuation preservation (hyphens, apostrophes).
Required for enhanced tag and contributor normalization.
Changed HandleBulkUpdate to check tags array length before assignment
instead of checking for nil, improving consistency with array handling
and preparing for dual-field normalization implementation.
Added golang.org/x/text v0.33.0 for proper titlecasing support in tag
normalization. Required for dual-field normalization to display tags in
title case (e.g., "Science Fiction", "Non-Fiction") while maintaining
search fields in lowercase without punctuation.
Convert tags and contributors columns from comma-separated strings to PostgreSQL
TEXT[] arrays for better data normalization and query performance.
Database Changes:
- schema.sql: Change tags/contributors from TEXT to TEXT[]
- schema.sql: Add GIN indexes for fast array searches
- queries.sql: Update search queries to use ANY() operator
- queries.sql: Update fuzzy search with unnest() for arrays
Generated Code (sqlc):
- models.go: Auto-generated with []string types for tags/contributors
- queries.sql.go: Auto-generated with proper array handling
Handler Changes:
- media.go: Update request structs to use []string for tags/contributors
- media.go: Remove pgtype.Text wrapping, use direct array assignment
- media.go: Add tag normalization in CreateMediaItemHandler
- collections.go: Update tags evaluation to join arrays for comparison
- collections.go: Add strings import for Join() function
Service Changes:
- ebook_scanner.go: Update EbookMetadata struct to use []string
- ebook_scanner.go: Remove string Join(), assign arrays directly
- collection_service.go: Update tags rule evaluation to join arrays
- collection_service.go: Add strings import
New Utilities:
- internal/utils/tags.go: Create NormalizeTags(), JoinTags(), SplitTags()
- Normalizes tags by trimming, lowercasing, removing duplicates/empties
API Documentation:
- bruno/media-items/Create Media Item.bru: Update examples to use arrays
- bruno/media-items/Update Media Item.bru: Update examples to use arrays
- Update docs: tags/contributors now array of string
Breaking Change:
- JSON format changes from "tags": "tag1,tag2" to "tags": ["tag1", "tag2"]
- Tests already use array format (no changes needed)
Benefits:
- GIN indexes enable faster array searches
- Normalization prevents data quality issues (case, duplicates)
- Array operations use PostgreSQL native operators (ANY, &&, unnest)
- Better separation of concerns (no string parsing in application)
Remove the getJSONInt helper function and update all test assertions to
expect float64 instead of int for JSON numeric fields, as Go's JSON
decoder unmarshals all numbers to float64 by default.
This simplifies the codebase by removing an unnecessary conversion
helper and makes tests more accurate to the actual JSON format.
Changes:
- Remove getJSONInt function from book_matching_test.go
- Update 5 assertions in book_matching_test.go to use float64
- Update 2 assertions in collections_bulk_test.go to use float64
- Update 2 assertions in media_bulk_test.go to use float64
- Add nil checks for optional numeric fields to prevent panics
Affected tests:
- TestBookMatchingBulkLink
- TestBookMatchingAutoLink
- TestCollectionsBulkOperations
- TestMediaBulkOperations
Note: Some test failures remain (API returning 400 instead of 200) but
these are legitimate test issues unrelated to type assertions.
Fix nil pointer panics in integration tests by initializing the
four refactored handlers (MediaHandler, SearchHandler, MatchingHandler,
CollectionHandler) that were added during Phase 6 refactoring but
never added to the test setup.
These handlers were properly instantiated in cmd/server/main.go
(commit 9fd8a39) but were missing from cmd/server/tests/test_helpers.go,
causing panics when tests tried to use /api/collections and /api/media-items
endpoints.
Changes:
- Create worker with 3 concurrent workers
- Initialize CollectionHandler with queries and connManager
- Initialize MediaHandler with queries and worker
- Initialize SearchHandler with queries
- Initialize MatchingHandler with queries and connManager
- Add all four handlers to router.Config struct
Fixes panic errors:
- internal/handlers/collections.go:78 (CreateCollection nil pointer)
- internal/handlers/media.go:891 (CreateMediaItem nil pointer)
Tests now pass:
- TestCollectionsBulkOperations: PASS
- TestAnalytics*: PASS (all analytics tests)
Note: Bruno API tests and frontend were NOT affected as they use the
real running application (which has complete handler setup).
- Update Login User.bru to use testuser@example.com
- Update Register User.bru to use testuser@example.com
- Update Register Admin User.bru to use maxdevices@example.com
- Standardize password to Test@Pass123! across all tests
- Add TEST_DATA.md documenting shared test credentials
- Update bruno.json with documentation reference
- Add comments linking to TEST_DATA.md for cross-reference
This alignment makes it easier to verify test failures between
Bruno API tests and Go integration tests using identical credentials.
- Replace hardcoded /app/uploads with getUploadPath()
- Update assertions to use dynamic paths
- Improve test log message to show actual path used
- Ensures tests work in both container and host environments
- Add isRunningInContainer() to detect test runtime environment
- Add getUploadPath() to resolve upload paths (container vs host)
- Add getCachePath() to resolve cache paths appropriately
- Update setupTestServer() to use dynamic path helpers
- Support environment variable overrides for flexibility
- Add os import for file system checks
- Add .env file inclusion for single source of truth
- Update test-integration to build and start all containers
- Add health check waiting for database and application
- Run tests from host against containerized database
- Add test-stop target for manual container cleanup
- Improve help text for better clarity
- Remove 24 duplicate media CRUD methods from ebook.go (884 lines removed)
- Keep 12 scanner/watch/scheduler methods on Handler
- Move request type declarations to media.go:
* CreateMediaItemRequest
* UpdateMediaItemRequest
* CreateMediaNoteRequest
* UpdateMediaNoteRequest
* CreateMediaHighlightRequest
* UpdateMediaHighlightRequest
- Remove unused imports from ebook.go (strconv, pgx)
- Fix library.go to use MediaHandler.ListMediaItems instead of Handler
ebook.go reduced from 1266 lines to 382 lines (70% reduction)
Handler now has focused responsibility: scanner and scheduler operations only
This completes Phase 7 of the ebook.go refactoring plan.
Result: Clean separation of concerns with no duplicate code
- Remove ROUTER_REFACTOR_PLAN.md (superseded by EBOOK_REFACTOR_PLAN.md)
- Remove SCANNER_RESTORATION_PLAN.md (no longer needed)
EBOOK_REFACTOR_PLAN.md remains as the active refactoring plan.
- Create worker for background tasks (3 concurrent workers)
- Create CollectionHandler for collection endpoints
- Create MediaHandler with worker for media CRUD operations
- Create SearchHandler for query operations
- Create MatchingHandler for book matching/linking operations
- Update routerConfig to include new handlers instead of EbookHandler
All handlers properly initialized and passed to router package.
System is fully operational with new handler architecture.
This is Phase 6 of the ebook.go refactoring plan.
- Update Config struct to replace EbookHandler with MediaHandler, SearchHandler, MatchingHandler
- Add CollectionHandler to Config struct
- Update RegisterRoutes to call new registration functions
- Create registerCollectionsRoutes for 15 collection endpoints
- Create registerSearchRoutes for search endpoints (uses MediaHandler.SearchMediaItems, MatchingHandler.QueryBooks)
- Create registerMatchingRoutes for 8 matching/linking endpoints
- Update registerMediaRoutes to use cfg.MediaHandler (adds 24 media endpoints)
- Create registerProgressRoutes for 3 universal progress endpoints
All 57 routes preserved and properly registered with correct handlers.
No functionality lost, all endpoints work identically.
This is Phase 5 of the ebook.go refactoring plan.
- Create SearchHandler struct with db field
- Add NewSearchHandler constructor
- Note: SearchMediaItems already moved to MediaHandler in Phase 2
- SearchHandler reserved for future search-specific operations
This is Phase 3 of the ebook.go refactoring plan.
- Remove all route registration from SetupRoutes
- Make SetupRoutes a pure factory function that only returns Handler
- Routes will be registered via router package in Phase 5
- Maintains backward compatibility with existing function signature
This is Phase 1 of the ebook.go refactoring plan to split the monolithic
Handler into focused handlers (MediaHandler, SearchHandler, MatchingHandler).
Remove the CollectionHandler field from router.Config struct literal
in cmd/server/tests/test_helpers.go. This field was removed from
the Config struct in a previous commit.
The collection routes are registered directly in handlers.SetupRoutes()
and don't need to be passed through the router config.
Remove the CollectionHandler field from router.Config struct and its
initialization in main.go. This field was never used - collections are
registered directly in handlers.SetupRoutes() where a CollectionHandler
is created locally.
Changes:
- Remove CollectionHandler field from internal/router/router.go Config
- Remove CollectionHandler: nil line from cmd/server/main.go
This cleans up dead code from the router refactoring. Collections
continue to work correctly as they are registered in SetupRoutes().
Related: Router refactoring completion
Phase 5: Application Lifecycle Management
Creates internal/app package for proper lifecycle management, signal
handling, and graceful shutdown of all services.
Changes:
- Create internal/app/app.go with App lifecycle manager
- Handles SIGINT, SIGTERM, SIGQUIT signals
- Graceful shutdown with 30-second timeout
- Manages HTTP server shutdown
- Manages scheduler start/stop
- Update cmd/server/main.go to use app lifecycle manager
- Replace defer-based cleanup with proper signal handling
- Server starts in background goroutine
- Blocks on app.Start() until shutdown signal
- Clean shutdown of all services
Benefits:
- Proper signal handling (Ctrl+C, kill, docker stop)
- Graceful shutdown prevents data corruption
- No more os.Exit(1) bypassing defer cleanup
- All services stopped in correct order
- Server stops accepting new connections first
- Then scheduler and background services stopped
Technical details:
- Uses sync.Mutex for shutdown safety
- Context with timeout for shutdown operations
- Channel-based coordination for shutdown completion
- Logs all lifecycle events for debugging
Fixes issue where e.Logger.Fatal() would call os.Exit(1)
immediately, skipping defer cleanup and causing unclean shutdown.
Change 'server' to '/server' to make pattern more explicit.
This prevents editors from confusing the ignored server binary
with the tracked cmd/server/ source code directory.
Pattern now only matches:
- /server (binary at root, ignored)
- NOT cmd/server/ (source directory, tracked)
Phase 4 of code organization plan
Changes:
- Create internal/router/scanner.go with registerScannerRoutes()
- Move scanner route registration from handlers to router package
- Update internal/router/router.go to call registerScannerRoutes
- Remove inline scanner routes from internal/handlers/ebook.go
Scanner routes now centralized in router/scanner.go:
- POST /scanner/scan - Scan ebooks
- POST /scanner/start - Start scanner
- POST /scanner/stop - Stop scanner
- GET /scanner/status/:jobId - Get scan status
- POST /scanner/watch/start - Start watch mode
- POST /scanner/watch/stop - Stop watch mode
- GET /scanner/watch/status - Get watch mode status
This improves code organization by separating route registration
from handler logic, making the codebase easier to maintain and
follows the established pattern of organizing routes by feature.
Phase 3 of scanner enhancement plan
Supported archive formats:
- .cbz (ZIP archives)
- .cbr (RAR archives)
- .cb7 (7-Zip archives)
- .cbt (TAR archives)
- .tar.gz, .tar.bz2 (Compressed TAR)
Features:
- Extract ComicInfo.xml metadata from all supported formats
- Extract cover images from archives
- Fall back to filename-based metadata if ComicInfo.xml not found
- Integrate into scanner workflow for automatic metadata extraction
Dependencies added:
- github.com/nwaples/rardecode v1.1.3 (MIT license, pure Go RAR)
- github.com/bodgit/sevenzip v1.6.1 (MIT license, pure Go 7-Zip)
Uses pure Go libraries only - no CGO required, ensuring maximum
compatibility and cross-platform builds.
All formats use a unified archiveFile interface for clean,
maintainable code.
Phase 2 of scanner enhancement plan
Changes:
- Add libraryTypes map[string][]string field to EbookScanner
- Initialize libraryTypes cache in NewEbookScanner
- Build library types cache in SetFolders by querying database
- Replace isEbookFile with isScannableFile for library-aware filtering
- Update ScanFolders and WatchChanges to use isScannableFile
This prevents cross-contamination between library types:
- Epub libraries only scan .epub files
- Comic libraries only scan .cbz/.cbr files
- Manga libraries only scan appropriate formats
- Each library type has configurable allowed extensions
Files are now filtered based on their library's allowed extensions,
ensuring only supported formats are scanned for each library type.
Phase 1 of scanner restoration plan
Changes:
- cmd/server/main.go: Capture ebookHandler from router.RegisterRoutes
- cmd/server/main.go: Start scheduler in background goroutine
- cmd/server/main.go: Defer StopScheduler() for graceful shutdown
- cmd/server/main.go: Start watch mode for all libraries after 2-second delay
- internal/router/router.go: Return ebookHandler from RegisterRoutes
This restores critical functionality that was removed during router refactor:
- Auto-scanning now works again
- Watch mode starts automatically for all libraries
- Graceful shutdown properly stops scheduler
Fixes issue where scheduler and watch mode were not starting on server boot.
- Rename phase1_integration_test.go to universal_progress_integration_test.go
(tests universal reading progress feature)
- Rename ebook_scanner_phase2_test.go to ebook_scanner_hash_test.go
(tests hash calculation and file identification utilities)
These renames make the test suite more maintainable and self-documenting.
john-okeefe