Add support for Carousel-style dashboard with unified collections architecture:
Database Schema Changes:
- Add user_dashboard_preferences table:
- hidden_collections: TEXT[] for managing section visibility
- collection_order: TEXT[] for custom ordering
- items_per_section: INT for limiting items per section
- Update collections table:
- user_id: Make nullable to support system-owned collections (NULL = system)
- show_on_dashboard: BOOLEAN for controlling visibility
- query_type: TEXT for different query types (continue-reading, recently-added, etc.)
- priority: INT for display order (lower = higher priority)
- is_system_collection: BOOLEAN for flagging system defaults
- Update collection_items table:
- Add excluded BOOLEAN for user overrides of auto-assigned items
Indexes:
- idx_collections_dashboard: (user_id, show_on_dashboard, priority) WHERE show_on_dashboard = true
- idx_dashboard_prefs_user_library: (user_id, library_id)
- idx_collection_items_excluded: (collection_id, excluded) WHERE excluded = true
System Collections (pre-seeded defaults):
- continue-reading: Books with 0 < progress < 1
- recently-added: Newly added items to library
- recently-read: Books with progress >= 1
- not-started: Books with progress = 0 or no record
This implements Phase 1 of the Carousel-style dashboard redesign plan.
- Try multiple locations for schema.sql file
- Support both local dev and containerized deployment paths
- Add informative logging when schema is loaded
- Prevent runtime.Caller issues in containers
Locations checked:
- database/schema/schema.sql (working directory)
- /app/database/schema/schema.sql (container)
- ../database/schema/schema.sql (relative)
- ../../database/schema/schema.sql (relative)
This fixes the 'no such file or directory' error in production containers.
- Create internal/database/schema.go with full initialization logic
- Parse table names from schema.sql using regex (handles both formats)
- Execute schema in atomic transaction
- Verify all expected tables exist
- Verify all critical functions exist (6 functions)
- PostgreSQL advisory locking with 30-second timeout
- Self-healing from partial/corrupted state
- Load schema.sql from filesystem at runtime
Features:
- Defensive regex handles IF NOT EXISTS and legacy CREATE TABLE
- Lock timeout prevents indefinite hangs
- Function verification ensures sync operations work
- Clear error messages with debug hints
Auto-generated changes from running 'sqlc generate' after query updates:
- models.go: updated with SystemSettings struct, removed scan fields from Users
- querier.go: updated interface with new system settings methods
- queries.sql.go: regenerated with new query methods
Generated via: cd internal/database && sqlc generate
System Settings Migration:
- Add GetSystemSetting query for single setting retrieval
- Add UpdateSystemSetting query for updating settings
- Add GetAllSystemSettings query for all settings
- Remove UpdateScanSettings and GetScanSettings (per-user queries)
User Query Enhancement:
- Add max_devices field to GetUser query
- Add device_count computed field to GetUser query
- Add max_devices field to ListUsers query
- Add device_count computed field to ListUsers query
These changes support:
1. System-wide scan settings instead of per-user settings
2. Users can now see their device count and limits
3. Admins can monitor device usage across all users
- Change query from :exec to :one with RETURNING *
- Allows handler to detect when user doesn't exist
- Follows pattern established by UpdateMediaItem
- Required for 404 response on non-existent user
Related: Fix for TestUpdateUserMaxDevicesNonExistentUser
- Remove CreateEbookNote, UpdateEbookNote, DeleteEbookNote queries
- These were marked as backward compatibility but never used
- API uses CreateMediaNote, UpdateMediaNote, DeleteMediaNote instead
- Remove misleading backward compatibility comments
- Regenerate sqlc code
Schema changes:
- Add tags_search TEXT[] column for case-insensitive, punctuation-free search
- Add contributors_search TEXT[] column for case-insensitive, punctuation-free search
- Create GIN indexes for fast array searches on both search fields
Query updates:
- CreateMediaItem: Include tags_search and contributors_search parameters
- UpdateMediaItem: Include tags_search and contributors_search parameters
- SearchMediaItems: Search against tags_search instead of tags
- SearchMediaItems: Search against contributors_search instead of contributors
- SearchMediaItemsFuzzy: Use tags_search and contributors_search for fuzzy matching
- Update ranking and priority logic to use search fields
Benefits:
- Case-insensitive search: "acme corp" finds "ACME CORP."
- Punctuation-agnostic search: "oreilly" finds "O'Reilly Media"
- Better UX: Users don't need to match exact casing or punctuation
- Improved performance with dedicated GIN indexes
Relates to Tags & Contributors Migration Phases 5 & 8
Convert tags and contributors columns from comma-separated strings to PostgreSQL
TEXT[] arrays for better data normalization and query performance.
Database Changes:
- schema.sql: Change tags/contributors from TEXT to TEXT[]
- schema.sql: Add GIN indexes for fast array searches
- queries.sql: Update search queries to use ANY() operator
- queries.sql: Update fuzzy search with unnest() for arrays
Generated Code (sqlc):
- models.go: Auto-generated with []string types for tags/contributors
- queries.sql.go: Auto-generated with proper array handling
Handler Changes:
- media.go: Update request structs to use []string for tags/contributors
- media.go: Remove pgtype.Text wrapping, use direct array assignment
- media.go: Add tag normalization in CreateMediaItemHandler
- collections.go: Update tags evaluation to join arrays for comparison
- collections.go: Add strings import for Join() function
Service Changes:
- ebook_scanner.go: Update EbookMetadata struct to use []string
- ebook_scanner.go: Remove string Join(), assign arrays directly
- collection_service.go: Update tags rule evaluation to join arrays
- collection_service.go: Add strings import
New Utilities:
- internal/utils/tags.go: Create NormalizeTags(), JoinTags(), SplitTags()
- Normalizes tags by trimming, lowercasing, removing duplicates/empties
API Documentation:
- bruno/media-items/Create Media Item.bru: Update examples to use arrays
- bruno/media-items/Update Media Item.bru: Update examples to use arrays
- Update docs: tags/contributors now array of string
Breaking Change:
- JSON format changes from "tags": "tag1,tag2" to "tags": ["tag1", "tag2"]
- Tests already use array format (no changes needed)
Benefits:
- GIN indexes enable faster array searches
- Normalization prevents data quality issues (case, duplicates)
- Array operations use PostgreSQL native operators (ANY, &&, unnest)
- Better separation of concerns (no string parsing in application)
Complete the rename by updating:
- DeviceCatalogs struct field: BookmannUuid → BookhoardUuid (models.go)
- Generated queries: Update all references (queries.sql.go)
- Local variables: bookmannUUID → bookhoardUUID (kobo.go)
- Struct field access: catalog.BookmannUuid → catalog.BookhoardUuid
All "bookmann" and "BOOKMANN" references are now eliminated from the codebase.
Part of project rename to Bookhoard.
Changes:
- Update comments: "Bookmann UUID" → "Bookhoard UUID"
- Rename sidecar struct field: Bookmann → Bookhoard
- Update type names: SidecarBookmannConfig → SidecarBookhoardConfig
- Fix test database name in queue_test.go
- Fix uppercase env var examples in KOBO_SETUP.md
Internal Go variable names (BookmannUuid, bookmannUUID) left unchanged
as they're implementation details that don't affect functionality.
Part of project rename to Bookhoard.
Database changes:
- schema.sql: Update column name bookmann_uuid → bookhoard_uuid
- schema.sql: Update index names and example URLs
- queries.sql: Update all SQL queries to use bookhoard_uuid
- Update example configuration values
Part of project rename to Bookhoard.
Add analytics queries:
- GetUserReadingHistory: detailed reading history with device info
- GetUserDeviceUsage: device usage statistics (sync count, time spent)
- GetPopularBooks: most read books with completion rates
Add book matching queries:
- GetUnlinkedBookByID: fetch single unlinked book
- DeleteUnlinkedBook: remove resolved unlinked book
- ListUnresolvedUnlinkedBooks: paginated list of unresolved books
Implement conflict detection for concurrent reading progress updates from different devices. Adds conflict management endpoints for listing, viewing, and resolving conflicts.
- Add ConflictHandler with CRUD endpoints for conflict management
- Implement automatic conflict detection in KOReader progress updates
- Add WebSocket broadcast for real-time conflict notifications
- Add database query for listing user conflicts by status
- Add integration tests and Bruno API test collection
- Add bulk_update_progress_from_koreader() function for batch processing
- Handles progress, annotations, and conflict detection
- Returns success/failure status for each book
- Supports device matching by UUID, file path, or title/author
- Implements automatic conflict detection for concurrent syncs
- Part of Phase 3 KOReader Integration implementation
- Add internal/sync package with format detection
- FormatGroup types: reflowable, fixed_layout, comic_archive
- DetectFormatGroup() function based on mimetype and file extension
- MimeType mappings for common ebook formats
- Progress conversion engine with:
- ConvertProgress() between format groups
- Extract percentage from various progress formats
- PageToPercentage / PercentageToPage helpers
- CharacterToPercentage / PercentageToCharacter helpers
- MergeProgress() with 'max progress wins' strategy
- FormatProgressForDisplay() for UI rendering
- Add sqlc queries for format detection and progress updates
- BulkUpdateFormatGroups query for auto-format detection
- GetUniversalProgress query with all location references
- UpdateUniversalProgress query with device sync metadata
- ReadingHistory queries for session tracking
- Add SearchMediaItems query with ILIKE partial matching
- Add SearchMediaItemsFuzzy query with word_similarity()
- Use sqlc.narg() for named parameters (search_pattern, search_query)
- Rank results by relevance: title > author > series > tags
- Fuzzy threshold set to 0.3 for word_similarity
- Generated Go models with proper parameter types
- Increase ISBN column from VARCHAR(13) to VARCHAR(17) to support ISBN-13 with hyphens
- Add normalize_isbn() database function to automatically remove hyphens and spaces
- Create trigger to auto-normalize ISBNs on INSERT/UPDATE operations
- Update all Ebook and MediaItem queries to use ISBN normalization
- Add GetEbookLibraryID query to check for existing ebook libraries
- Add graceful error handling when no ebook library exists
- Return helpful error message: 'no ebook library found. Please create an ebook library first'
- Create comprehensive tests for ISBN normalization and library selection
- Add Bruno test files for various ISBN formats and error scenarios
- Update documentation with ISBN normalization details
- Regenerate queries.sql.go with refresh token queries
- Update models.go with RefreshTokens type
- Update querier.go with new query methods
- Update db.go with generated code
- Implement strict password requirements:
- Minimum 8 characters
- At least one uppercase letter
- At least one lowercase letter
- At least one number
- At least one special character
- Add custom validator for Echo integration
- Add GetPasswordRequirements helper function
- Add ValidatePassword function for manual validation
This major update implements a complete user annotation system:
## 🎯 New Features
- User notes with position tracking for media items
- Text highlighting with customizable colors
- Highlight-note associations for detailed annotations
- Full CRUD API for both notes and highlights
- Backward compatibility with existing ebook endpoints
## 📊 Database Changes
- Add media_notes table (id, media_item_id, user_id, content, position, timestamps)
- Add media_highlights table (id, media_item_id, user_id, selection_text, start/end_position, color, optional note_id)
- Add foreign key relationships with CASCADE deletes
- Add proper indexes for performance
- Add database schema views for ebook backward compatibility
## 🔧 API Implementation
- Complete REST API endpoints for notes and highlights
- JWT authentication with proper middleware bypass
- Request validation with meaningful error responses
- UUID validation and type safety
- Support for hex color codes in highlights
## 🧪 Testing & Documentation
- Comprehensive test suite covering authentication scenarios
- Bruno API collection for manual testing
- Detailed testing guide with troubleshooting
- Updated documentation in README and TESTING.md
## 📁 Backward Compatibility
- Existing ebook endpoints continue working
- Database views maintain API contracts
- No breaking changes for existing integrations
The annotation system is now fully functional and ready for production use.
- Add MediaNotes and MediaHighlights model structs with pgx v5 types
- Add EbookNotes and EbookHighlights for backward compatibility
- Add complete CRUD SQL queries for notes and highlights
- Add database connection pool function using pgx v5
- Generate sqlc code for new annotation functionality
- Add library_types table with ebooks, comics, manga types
- Add libraries table for multiple library support
- Add library_folders table for multi-folder libraries
- Add library_visibility table for user access control
- Add media_items table replacing ebooks for broader media support
- Create backward compatibility views for existing API
- Implement library service with type validation and file extension handling
- Support modular extension for future media types
Manga type includes cbz/cbr archives as requested
- Update CreateUser SQL query to accept role parameter
- Regenerate SQLC code to include Role field in CreateUserParams
- Support for explicit role assignment during user registration
- Add role column to users table with admin/user constraint
- Add added_by_admin_id column to ebooks table for tracking
- Add constraint to ensure only admins can manage folders
- Update all SQL queries to include role field
- Regenerate database models with new schema
- Change schema path to use relative path from internal/database
- Update queries path to use current directory
- Modify output directory to current directory for better organization
- Move migrations/ to database/schema/ for clarity on database schema definitions
- Move sqlc.yaml to internal/database/ to group with database code
- Move static/ to cmd/server/static/ to co-locate with server
- Update all configuration files and documentation
- Follow Go project conventions for better organization
- Update CreateUser query to explicitly select only existing columns
- Update all user SELECT queries to explicitly select columns to avoid scan_frequency_minutes column issues
- Add GetUserForLogin query that includes password_hash for authentication
- Update login handler to use GetUserForLogin instead of GetUserByEmailOrUsername
- This prevents errors when migration hasn't been applied yet, allowing user registration to work
- Add admin dashboard at /admin route consolidating all user settings
- Move user preferences (username, email, password, theme) from separate page
- Add ebook library preferences section with folder management
- Add scan settings (frequency and auto-scan toggle) with database persistence
- Create database migration for scan_frequency_minutes and auto_scan_enabled columns
- Add API endpoints for scan settings management:
- PUT /api/library/scan-settings - Update scan preferences
- GET /api/library/scan-settings - Get current scan settings
- Update dashboard navigation to link to admin dashboard
- Remove old preferences.html template (functionality moved to admin)
- Create Bruno API testing files for library endpoints
- Add real-time folder loading and management in admin interface
- Implement scan settings persistence and retrieval from database
- Add /preferences route and preferences.html template for user settings
- Implement username, email, password, and theme update functionality
- Add account deletion feature with confirmation
- Add navigation link to preferences from dashboard
- Create API endpoints:
- PUT /api/user/username - Update username
- PUT /api/user/email - Update email address
- PUT /api/user/password - Change password with verification
- DELETE /api/user/account - Delete user account
- Add database queries for user updates and account deletion
- Create Bruno API testing files for all user preference endpoints
- Add proper validation, error handling, and security checks
- Rename project from 'bookmann' to 'shelf'
- Move all backend/ contents to root level (flatten structure)
- Update Go module name from 'bookmann' to 'shelf'
- Update all import paths to use new 'shelf' module
- Update Dockerfile to work without backend/ subdirectory
- Update docker-compose.yml to use new structure and rename containers
- Update .gitignore for new file paths
- Update README.md with new project name and structure
- Regenerate database code with new module imports
john-okeefe