- Add /devices route for device management interface
- Add /conflicts route for sync conflict resolution
- Add /queue route for sync queue management
- Add comprehensive tests for device cap management
- Add test suite for queue management
- Add comprehensive implementation plan for universal sync system
- Update README with API reference and device setup guides
- Add KOBO_SETUP.md device configuration guide
This plan implements:
- Universal book identification (SHA-256, UUID, ISBN, ASIN, OPF identifiers)
- Enhanced collection management with auto-assign rules
- OPDS-based wireless book delivery for Kobo, KOReader, Web, and Mobile
- Bidirectional progress sync with ContentId mapping
- Device-specific shelf mappings and configuration
- Complete database schema with 7 new tables
- 50+ documented API endpoints
- 10-week phased implementation plan
Features include:
- Cross-device book matching regardless of file paths
- Collections as device-neutral metadata with per-device shelf mapping
- Format conversion (EPUB → KEPUB) with hash integrity preservation
- Three-tier authentication (JWT, device tokens, OPDS)
- Two-layer architecture: OPDS for acquisition + internal APIs for state management
Key design principles:
1. Canonical UUID (Bookmann UUID) always wins for progress tracking
2. Collections ≠ device inventory - organizational metadata only
3. OPDS primary for all devices, internal APIs for web/mobile
4. Dual hash storage prevents format conversion issues
See IMPLEMENTATION_PLAN.md for complete technical details.
- Add SECURITY_AUDIT.md with A- security rating
- Add SECURITY_ENHANCEMENTS.md for improvements
- Add DEVICE_CAP_IMPLEMENTATION.md complete guide
- Add KOREADER_SETUP.md device setup guide
- Add SYNC_USER_GUIDE.md user documentation
- Document all API endpoints and features
- Include security considerations and best practices
- Update User Max Devices - Complete API documentation
- Update User Max Devices - Success test case
- Update User Max Devices - Invalid Zero test case
- Update User Max Devices - Exceeds Maximum test case
- Update User Max Devices - Missing ID test case
- Include validation rules and example payloads
- Document all status codes and error responses
- Test successful updates (5, 10, 50, 100 devices)
- Test validation failures (0, -1, 101, 1000 devices)
- Test authentication requirements (no token, non-admin)
- Test non-existent user ID
- Test missing user ID in URL
- Test max_devices field in user list response
- Add 20+ test cases across 7 test functions
- Helper functions for admin user creation and login
- Add max_devices column to users table (default: 10)
- Add UpdateUserMaxDevices database query
- Add CountUserDevices database query
- Add UpdateUserMaxDevices handler with validation (1-100 devices)
- Add PUT /api/auth/users/:id/max-devices endpoint (admin only)
- Update UserList struct to include max_devices field
- Validate user ID format and max_devices range
- Return appropriate errors for invalid requests
- Add 12 composite database indexes for sync operations
- Composite indexes for sync_queue (device/status/priority)
- Composite indexes for reading_progress (user/media timestamps)
- Composite indexes for devices (user/sync_enabled)
- Composite indexes for annotations (user/media)
- Comment out ALTER SYSTEM commands for sqlc compatibility
- PostgreSQL tuning recommendations included for manual application
Implement conflict detection for concurrent reading progress updates from different devices. Adds conflict management endpoints for listing, viewing, and resolving conflicts.
- Add ConflictHandler with CRUD endpoints for conflict management
- Implement automatic conflict detection in KOReader progress updates
- Add WebSocket broadcast for real-time conflict notifications
- Add database query for listing user conflicts by status
- Add integration tests and Bruno API test collection
- Test login returns both access_token and refresh_token
- Test refresh endpoint accepts UUID token and returns new access_token
- Verifies end-to-end refresh token flow works correctly
- Add parseTokenUUID() helper to convert string to pgtype.UUID
- Update RefreshAccessToken to parse token string to UUID before validation
- Update Logout to parse token string to UUID before revoking
- Update CreateRefreshToken to pass UUID directly to database
- Update auth.go: fix return value order from CreateRefreshToken
- Remove unnecessary comments for cleaner code
- Change token column type from VARCHAR(255) to UUID
- Add gen_random_uuid() as default value for token
- Improves type safety and performance for token storage
- Remove integration_test.sh (redundant with Go integration tests)
- Remove BRUNO_PHASE1_TEST_REPORT.md (temporary test report)
- Remove integration_test_results.txt (temporary test output)
Go-based tests in cmd/server/tests/ provide comprehensive coverage and
are better integrated with the project testing infrastructure.
- Add Kobo sync handler with markup, bookmark, analytics, and initialization endpoints
- Add Kobo integration tests and Bruno API test collection
- Move device approve/reject routes from public to protected routes
- Enhance test infrastructure with DATABASE_URL support and helper functions
- Fix device GetDevice handler nil pointer handling
- Clean up test reports and session files
Add Bruno v3.0 request for testing WebSocket endpoint:
- WebSocket connection type
- JWT token authentication via query parameter
- Ping message body for heartbeat testing
- Assertions for successful WebSocket upgrade (101 status)
Provides API documentation and testing capability for
WebSocket connection functionality per project standards.
Add WebSocket infrastructure to main server:
- Import sync package for ConnectionManager
- Create and start ConnectionManager with cleanup task
- Initialize WSHandler with auth dependencies
- Add /ws/sync WebSocket endpoint
- Update handler initialization to pass ConnectionManager
The WebSocket endpoint at /ws/sync enables real-time progress
updates across all connected clients (web, mobile, devices).
Add WebSocket broadcast to KOReader progress sync:
- Integrate ConnectionManager into KOReaderHandler
- Broadcast progress updates on successful sync
- Include source device information (model, type)
- Real-time updates to all connected clients
When KOReader devices sync reading progress, all connected
WebSocket clients (web browsers, mobile apps, other devices)
receive instant updates.
Add WebSocket ConnectionManager to Handler:
- Add connManager field to Handler struct
- Update NewHandler to accept ConnectionManager parameter
- Update SetupRoutes to pass ConnectionManager through
- Import sync package with alias to avoid conflicts
This enables progress handlers to broadcast updates via WebSocket.
Add device token validation method for WebSocket authentication:
- Validates device auth tokens against database
- Returns device information for valid tokens
- Used by WebSocket handler for device authentication
This enables devices to authenticate WebSocket connections
using their bearer tokens.
Add ConnectionManager for real-time WebSocket communication:
- Message types for progress updates, annotations, conflicts
- Broadcast message structure with source device tracking
- Device connection tracking with user and device metadata
- Automatic broadcast loop with concurrent message delivery
- Connection management (add, remove, get by ID/user)
- Stale connection cleanup (2-minute timeout)
- Connection statistics by device type
- Background cleanup task runs every minute
This implements the core WebSocket infrastructure needed for
Week 9 of the Universal Sync Implementation Guide.
- Add bulk_update_progress_from_koreader() function for batch processing
- Handles progress, annotations, and conflict detection
- Returns success/failure status for each book
- Supports device matching by UUID, file path, or title/author
- Implements automatic conflict detection for concurrent syncs
- Part of Phase 3 KOReader Integration implementation
- Add KOReader sync endpoints to main application router
- Create Bruno API collection for testing KOReader endpoints
- Add integration tests for KOReader functionality
- Include comprehensive README with setup instructions
- Test coverage for progress, metadata, library, and bookmarks sync
- Part of Phase 3 KOReader Integration implementation
- Update device auth middleware to set actual device object
- Change from DeviceContext to database.Devices
- Fix RequirePermission to use database.Devices
- Ensures handlers can access full device information
- Required for KOReader sync handlers to function properly
- Create SyncProgress for bidirectional progress synchronization
- Create GetMetadata for book progress and annotation retrieval
- Create GetLibrary for user library sync
- Create SyncBookmarks for annotation management
- Support device matching by UUID, file path, or title/author
- Implement immediate and checkpoint sync modes
- Handle bookmarks, highlights, and notes synchronization
- Part of Phase 3 KOReader Integration implementation
- Add GetMediaItemByFilePathForSync for file path matching
- Add GetUserProgressForBooks for bulk progress retrieval
- Add GetAnnotationsForBook for annotation sync
- Add UpdateDeviceSyncTimestamp for device tracking
- Add GetUserMediaItemsForSync for library sync
- Add CheckForProgressConflicts for conflict detection
- Regenerate sqlc code for all new queries
- Part of Phase 3 KOReader Integration implementation
- Add bulk_update_progress_from_koreader() function for batch processing
- Handles progress, annotations, and conflict detection
- Returns success/failure status for each book
- Supports device matching by UUID, file path, or title/author
- Implements automatic conflict detection for concurrent syncs
- Part of Phase 3 KOReader Integration implementation
- Add POST /api/libraries/:id/scan endpoint for admin library scanning
- Add GET /api/libraries/:id/media-items endpoint for library media items
- Move /api/libraries/types to public endpoint (no auth required)
- Update ScanEbooks handler to support library_id parameter
- Remove manual device ID generation, use database-generated IDs
- Add comprehensive test helpers (setupTestServer, loginTestUser, getTestUserID)
- Add cleanup step for existing test users in integration tests
- Fix UUID parsing from database responses
Test Files Added:
- integration_test.sh: Automated integration test script
* Tests full user flow: register, login, library creation, scanning
* Tests device registration and management
* Color-coded output with pass/fail tracking
* Generates detailed test results report
- cmd/server/tests/device_test.go: Unit tests for device endpoints
* TestDeviceRegistrationFlow: Full registration flow test
* TestListDevices: Device listing functionality
* TestUpdateDevice: Device settings updates
* TestDeleteDevice: Device removal
* TestDeviceAuthentication: Device auth middleware test
- cmd/server/tests/phase1_integration_test.go: Phase 1 integration tests
* Tests universal progress tracking
* Tests format group detection
* Tests progress conversion
Test Coverage:
- Device registration with web-based approval flow
- Device management (list, update, delete)
- Device authentication and token validation
- User authentication and authorization
- Library creation and management
- Scanner integration
- Media items listing
Notes:
- Tests designed to run against live server on localhost:8765
- Integration test script uses bash/curl for endpoint testing
- Device tests require helper functions to be implemented
- Add internal/sync package with format detection
- FormatGroup types: reflowable, fixed_layout, comic_archive
- DetectFormatGroup() function based on mimetype and file extension
- MimeType mappings for common ebook formats
- Progress conversion engine with:
- ConvertProgress() between format groups
- Extract percentage from various progress formats
- PageToPercentage / PercentageToPage helpers
- CharacterToPercentage / PercentageToCharacter helpers
- MergeProgress() with 'max progress wins' strategy
- FormatProgressForDisplay() for UI rendering
- Add sqlc queries for format detection and progress updates
- BulkUpdateFormatGroups query for auto-format detection
- GetUniversalProgress query with all location references
- UpdateUniversalProgress query with device sync metadata
- ReadingHistory queries for session tracking
Add missing Bruno requests for all API endpoints:
Library Management:
- Get Library - retrieve single library details
- Update Library - modify existing library
- Delete Library - remove library and media items
- Delete Library Folder - remove folder from library
- Get Library Stats - retrieve library statistics
Media Items Management:
- Create Media Item - add new media with full metadata
- Update Media Item - modify existing media metadata
- Delete Media Item - remove media from library
- Get Media Rating - retrieve single media rating
- Delete Media Rating - remove user's media rating
Coverage now complete: 47/47 API endpoints have Bruno requests
Organized requests in proper folder structure for maintainability
- Remove MIGRATION_FINAL_STATUS.md - migration is complete
- Remove REMOVE_EBOOKS_SYSTEM.md - system cleanup finished
- Repository now focused on current working code
- Remove ebook-specific test files (ebook_test.go, integration_test.go, notes_highlights_test.go)
- Update search_test.go for media-items API paths
- Regenerate templates (bookshelf_templ.go, header_templ.go)
- Add ISBN normalization utility function
- Clean up test suite to focus on media-items functionality
Aligns tests and templates with unified media-items architecture
- Remove all ebook-specific API requests (15 files deleted)
- Rename Scan Ebooks.bru to Scan Media Items.bru
- Update API paths from /api/ebooks to /api/media-items
- Update base URL and environment configuration
- Maintain all existing media-items, library, auth, and progress tests
Aligns Bruno collection with unified media-items API architecture
- Remove API_TESTING_SUMMARY.md, IMPLEMENTATION_SUMMARY.md
- Remove MIGRATION_PROGRESS.md, SECURITY_*.md files
- Clean up temporary documentation files from previous sessions
- Repository now focused on current working code
- ebook_scanner.go:293 now compiles successfully
- GetLibraryByFolder method available after database code regeneration
- Scanner service can properly find libraries by folder path
Fixes primary compilation error blocking build
- Fix CreateMediaItem ISBN field to use pgtype.Text wrapper
- Fix UpdateMediaItem to use correct Isbn field name
- Resolve type mismatch between request and database params
Resolves compilation errors in media item handlers
Major changes:
- Rename testEbooks() function to testMediaItems()
- Remove all old ebook test cases
- Update all /api/ebooks paths to /api/media-items
- Update TestContext: remove EbookID, add MediaItemID field
- Add admin media-items tests (Create, Update, Delete)
- Fix compilation errors and missing imports
Tests updated to use new API structure while maintaining test coverage.
Breaking change: /api/ebooks endpoints removed (use /api/media-items instead)
- README: Document new sorting options (12 fields)
- README: Document new filtering capabilities (6 filter types)
- README: Document enhanced metadata fields (9 new fields)
- README: Update prerequisites to mention Podman
- IMPLEMENTATION_SUMMARY: Mark all phases as complete
- Add API usage examples for sorting and filtering
- Add integration tests for sorting (sorting_test.go)
- Test sort by title, author, page_count, copyright_year, genre
- Test pagination with sorting
- Test invalid sort parameter defaults
- Cover no user, user, and admin contexts
- Add integration tests for filtering (filtering_test.go)
- Test filter by genre, language, year range, has_cover
- Test combining multiple filters
- Test filtering with pagination and sorting
- Cover no user, user, and admin contexts
- Add Bruno API test for sorting
- Add Bruno API test for filtering
Comprehensive documentation of:
- Create Library 500 error bug and fix
- Root cause analysis (type mismatch in context extraction)
- Testing issues discovered (poor error reporting, mock vs real tests)
- Test improvements implemented
- Tomorrow's 5-phase action plan for API reliability
- Complete endpoint checklist for testing
- Correct vs incorrect code patterns
- Success criteria for "rock solid" API
Reference document for tomorrow's comprehensive API review session.
- Add explicit status code check (require.Equal 201)
- Remove conditional success/failure branching
- Provide clear error message with actual vs expected status
Now if CreateLibrary returns 500, test will clearly show:
"Failed to create library: expected 201, got 500"
Instead of vague "Library ID is empty" message that hid the 500 error.
- Use c.Get("user").(database.Users) instead of c.Get("user_id").(string)
- Extract userUUID from user.ID.Bytes ([16]byte)
- Properly convert to pgtype.UUID for service layer
- Remove unnecessary uuid.Parse call
This fixes 500 Internal Server Error when creating libraries via API.
The JWT middleware sets user as database.Users struct, not string.
Related to Bruno Create Library request testing.
- Document search capabilities in Media Management section
- Add API endpoint documentation for /api/media-items/search
- Include search behavior, ranking, and examples
- Document fuzzy search fallback for typos
- Add comprehensive search integration tests (search_test.go)
- Test no user, user, and admin contexts
- Test partial matching, fuzzy fallback, special characters
- Add Bruno API test for search endpoint
- Fix missing closing parenthesis in test structure
- Create search.js with debounced input (300ms)
- Display results in dropdown modal with highlighted matches
- Support keyboard navigation (arrows, Enter, Escape)
- Show result count and 'no results' state
- Highlight matching terms in results
- Add autocomplete attribute to search input
- Minimum 2 characters to trigger search
- Fix scoping issue with err variable in os.Stat check
- Properly check for non-existent vs inaccessible folders
- Use reassignment (=) instead of declaration (:=) since err already declared
- Add GET /api/media-items/search endpoint
- Try partial matching first (ILIKE with wildcards)
- Fallback to fuzzy search if no results found
- Return 404 with 'no results found' when no matches
- Limit results to 50 items by default
- Supports search across title, author, series, tags, contributors
- Respects library visibility settings per user
- Add SearchMediaItems query with ILIKE partial matching
- Add SearchMediaItemsFuzzy query with word_similarity()
- Use sqlc.narg() for named parameters (search_pattern, search_query)
- Rank results by relevance: title > author > series > tags
- Fuzzy threshold set to 0.3 for word_similarity
- Generated Go models with proper parameter types
- Changed Header component calls from text to proper templ syntax (@Header)
- Header now properly renders navigation, search, theme switcher, and user menu
- Fixed both bookshelf.templ and dashboard.templ templates
- Added direct /bookshelf route that works with both Authorization header and cookie token
- Imported missing strings package
- Users can now access /bookshelf directly instead of /api/bookshelf
- Regenerate all template files after adding header component
- Clean up generated session files
- Templates now use new header component consistently
All templates have been regenerated to include the new
header functionality and updated routing.
- Replace navigation bar with Header() component
- Remove duplicate logout function (now in header.js)
- Maintain consistent header across pages
- Update generated template files
This provides consistent navigation and theme switching
functionality across all pages using the reusable header component.
- Add JavaScript to check for valid JWT token on homepage load
- Auto-redirect to /bookshelf if user is already logged in
- Shows login/register form if not authenticated
- Improves UX by taking logged-in users directly to bookshelf
Implementation:
- Fetch /api/auth/profile with stored token
- On success, redirect to /bookshelf
- On failure, silently stay on homepage
- Runs on DOMContentLoaded for fast execution
- Add /bookshelf route as default page for logged-in users
- Update login and register handlers to redirect to /bookshelf
- Update homepage to auto-redirect to /bookshelf when logged in
- Preserve /dashboard route for backward compatibility
- Update test redirects to use /bookshelf
Changes:
- main.go: Add /bookshelf protected route
- auth.go: Change login/register redirects from /api/dashboard to /bookshelf (2 locations)
- edge_cases_test.go: Update test redirect to /bookshelf
- Maintains backward compatibility with existing /dashboard route
This makes the beautiful bookshelf the default landing page
for all authenticated users while keeping the old dashboard accessible.
- Create bookshelf.templ with beautiful visual bookshelf interface
- Implement wooden shelf appearance with CSS gradients
- Add responsive grid layout (2/3/6 columns based on screen size)
- Books display with 3D spine effect and hover animations
- Auto-select first library and load books on page load
- Empty state and loading state handling
Visual Features:
- Wooden shelves with gradient shadows (12px bottom border)
- Books hover with lift (translateY) and rotation effects
- Book covers with aspect ratio 2/3 and inset spine highlight
- Error handling falls back to placeholder-book.svg
- 6 books per shelf for optimal display
JavaScript Features:
- Fetch visible libraries from API
- Populate library selector dropdown
- Load and display media items on shelves
- Handle empty states gracefully
- Book detail placeholder (to be implemented)
- Add header.templ component with app title, search, theme switcher, user menu
- Implement dropdown menus for theme selection and user actions
- Add wood theme options (Wood Light, Wood Dark, Wood Mahogany)
- Support all existing themes with visual color swatches
- Auto-close dropdowns when clicking outside
- TypeScript header functionality with proper type safety
Features:
- Left: App title "📚 Bookmann" linking to /bookshelf
- Center: Search box (ready for future search functionality)
- Right: Theme switcher button with color dropdown → User icon menu
- User menu includes Settings, Admin Panel (if admin), and Logout
- Theme persistence to localStorage and server via API
- Add fallback to login when user registration returns 409 Conflict
- Prevents empty user token error when test user already exists
- Allows integration tests to run reliably across multiple executions
- Test now attempts to log in with existing credentials if registration fails
This fixes the issue where the test would fail if the user
'integrationuser@test.com' already existed from a previous test run.
- Fix scheduler.go log.Printf calls to convert pgtype.UUID to string before formatting
- Fix ebook.go fmt.Printf calls to convert pgtype.UUID to string before formatting
- Add missing Enabled field to rate limiter config in security test
- Prevents format string errors when logging library IDs
This resolves compilation errors where pgtype.UUID was being formatted
with %s which expects a string, not a UUID struct.
john-okeefe