From 555bd0df15761b1feb5c1261ed8993133d14baba Mon Sep 17 00:00:00 2001 From: John O'Keefe Date: Sun, 1 Feb 2026 17:34:21 -0500 Subject: [PATCH] docs: update references for new configuration structure - Update README.md with simplified documentation links - Update docs/API_REFERENCE.md formatting and structure - Update docs/SYNC_USER_GUIDE.md with minor improvements These updates reflect the reorganized documentation structure and new configuration approach with defaults in docker-compose.yml. --- README.md | 671 ++++++---------------------------------- docs/API_REFERENCE.md | 333 +++++++++++++++++++- docs/SYNC_USER_GUIDE.md | 122 ++++---- 3 files changed, 497 insertions(+), 629 deletions(-) diff --git a/README.md b/README.md index 1d906f5..b510dc5 100644 --- a/README.md +++ b/README.md @@ -1,620 +1,155 @@ # ๐Ÿ“š Bookhoard -A modern self-hosted media library system built with Go, PostgreSQL, HTMX, and Tailwind CSS featuring multiple library support, beautiful dark themes, and comprehensive media management. +A modern self-hosted media library system built with Go, PostgreSQL, HTMX, and Tailwind CSS featuring **universal cross-device sync**, beautiful dark themes, and comprehensive media management. -## โœจ Features +## โœจ Why Bookhoard? -### ๐Ÿ— Multi-Library System -- **Multiple Media Types**: Support for Ebooks, Comics, and Manga with modular architecture -- **Per-Library Folders**: Each library can have multiple scanning folders for flexible organization -- **Library Visibility Control**: Admins can control which libraries are visible to each user -- **Type-Specific File Extensions**: - - **Ebooks**: `.epub`, `.pdf`, `.mobi`, `.azw`, `.azw3`, `.txt`, `.rtf`, `.doc`, `.docx`, `.lit`, `.fb2`, `.pdb` - - **Comics**: `.cbz`, `.cbr`, `.cb7`, `.cbt`, `.pdf` - - **Manga**: `.cbz`, `.cbr`, `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.webp` -- **Easy Extension**: Designed to add new media types in the future +**๐Ÿ”„ Universal Sync**: Your reading progress, highlights, and notes sync automatically across all your devices - KOReader, Kobo, web, and mobile. -### ๐Ÿ”’ Authentication & Security -- **Multi-User Support**: Complete user registration and authentication system -- **JWT-Based Sessions**: Secure token-based authentication with 1-hour expiration and refresh token support -- **Role-Based Access**: Admin and user roles with granular permission control -- **Password Security**: bcrypt hashing with complex password requirements: - - Minimum 8 characters - - At least one uppercase letter (A-Z) - - At least one lowercase letter (a-z) - - At least one number (0-9) - - At least one special character (!@#$%^&*()_+-=[]{}|;':\",./<>?) -- **Account Lockout**: Automatic account lockout after 5 failed login attempts (15-minute lockout period) -- **Rate Limiting**: Built-in rate limiting on auth endpoints (10 requests/minute) to prevent brute force attacks -- **Refresh Tokens**: Secure refresh token mechanism (7-day expiration) for seamless token renewal -- **Input Validation**: Comprehensive validation including username whitespace checks, email format validation -- **Pagination Protection**: Maximum pagination limits (1000 items) to prevent DoS attacks -- **Path Validation**: Library folder paths are validated for existence and accessibility -- **Case-Insensitive Roles**: Role values automatically normalized to lowercase -- **Database Transactions**: Multi-step database operations use transaction support for data consistency -- **Standardized Error Responses**: Consistent error format across all API endpoints +**๐Ÿ“ฑ Multi-Library**: Organize your ebooks, comics, and manga with per-library folders and smart collections. -### ๐ŸŽจ Beautiful UI -- **11 Dark Themes**: Tokyo Night, Dracula, Nord, Solarized Dark, Monokai, One Dark Pro, Material Dark, Catppuccin variants -- **Theme Persistence**: User theme preferences saved to database -- **Mobile-First Design**: Fully responsive interface for all devices -- **HTMX Integration**: Dynamic interactions without page reloads +**๐ŸŽจ Beautiful UI**: 11 gorgeous dark themes with responsive design that works on any device. -### ๐Ÿ”„ Universal Cross-Platform Sync (NEW!) -- **Real-Time Progress Sync**: Reading progress syncs automatically across all your devices -- **Multi-Device Support**: KOReader, Kobo e-readers, web browsers, mobile apps -- **Format-Aware Tracking**: EPUB CFI, page numbers, percentages, character offsets -- **Highlights & Notes**: Annotations sync universally across all devices -- **Conflict Resolution**: Smart handling when same book read on multiple devices -- **Offline Support**: Queue changes when offline, sync when reconnected -- **Secure Device Auth**: Web-based device approval, no passwords on devices -- **Checkpoint Mode**: Batch sync for better battery life -- **WebSocket Updates**: Real-time broadcast of progress changes +**๐Ÿ”’ Secure**: JWT authentication, bcrypt password hashing, rate limiting, and no passwords on devices. - ### ๐Ÿ“ฑ Media Management - - **Universal Media Support**: Single system for all media types with unified interface - - **Rich Metadata**: Automatic extraction of title, author, series, publisher, ISBN, tags - - **ISBN Normalization**: Automatic ISBN format normalization (removes hyphens and spaces) supporting ISBN-10 and ISBN-13 formats - - **Advanced Search**: - - Partial matching search across title, author, series, tags, and contributors - - Automatic fuzzy search fallback when no partial matches found (handles typos and misspellings) - - Real-time search results with highlighted matches - - Keyboard navigation (โ†‘โ†“ arrows, Enter to select, Escape to close) - - Respects library visibility settings - - **Dynamic Sorting**: Sort your media collection by multiple fields - - Title (A-Z or Z-A) - - Author (A-Z or Z-A) - - Date added (newest or oldest first) - - Date published (newest or oldest first) - - Copyright year (newest or oldest first) - - Series order (with series number) - - Page count (shortest or longest first) - - Genre (A-Z or Z-A) - - **Advanced Filtering**: Filter media items with multiple options - - Filter by author (partial match) - - Filter by series (partial match) - - Filter by genre (exact match) - - Filter by language (English, Spanish, French, German, Japanese, Chinese, Korean, Russian, Italian, Portuguese) - - Filter by copyright year range - - Filter by items with cover images only - - Combine multiple filters with URL state management for shareable links - - **Enhanced Metadata**: New fields for better organization - - Language support for multi-lingual collections - - Edition information (2nd Edition, Revised, Collector's Edition, etc.) - - Page count for better sorting and progress calculation - - External service integration (Goodreads, OpenLibrary, Google Books IDs) - - Copyright year (distinct from publication date) - - Structured genre classification - - Subject tags (array of subjects) - - **Advanced Rating**: 5-star system with half-star precision (1-10 scale) - - **Reading Progress**: User-specific progress tracking with current page and total pages - - **Notes & Highlights**: Personal annotations and text highlighting with color customization - - **Highlight Notes**: Link highlights to detailed notes for comprehensive annotations - - **Library Statistics**: Media count and usage statistics per library - - **Background Scanning**: Automatic background scanning with configurable frequency per user - - **Watch Mode**: Real-time file system monitoring for instant library updates +--- ## ๐Ÿš€ Quick Start ### Prerequisites -- **Podman** and Podman Compose (recommended) or Docker -- PostgreSQL database (handled by Podman) +- **Podman** (recommended) or Docker +- **5 minutes** of your time + +### Installation -### Environment Setup -1. **Create .env file**: ```bash -cp .env.example +# 1. Clone the repository +git clone https://github.com/yourusername/bookhoard.git +cd bookhoard + +# 2. Set up environment +cp .env.example .env +# Edit .env with your secure JWT_SECRET and DBPASS + +# 3. Start the server +podman-compose up --build -d # or: docker-compose up --build -d + +# 4. Open your browser +open http://localhost:8765 ``` -2. **Edit .env** with your secure values: -```bash -JWT_SECRET="your-secure-jwt-secret-key-here" -DBPASS="your-secure-database-password-here" -``` +The first user to register automatically becomes an admin. -### Running the Application -```bash -# Using Podman (recommended) -podman-compose up --build +--- -# Using Docker -docker-compose up --build +## ๐Ÿ“– Key Features -# Option 2: Direct environment variables -export JWT_SECRET="your-secure-jwt-secret-key-here" -export DBPASS="your-secure-database-password-here" -podman-compose up --build -``` +### Universal Cross-Platform Sync +- **Real-Time Progress**: Turn a page on your Kindle, see it on your phone +- **Format-Aware**: EPUB CFI, page numbers, percentages - all handled correctly +- **Offline Queue**: Changes sync when you reconnect, priority-processed +- **Conflict Resolution**: Smart handling when same book read on multiple devices +- **Book Matching**: Automatic matching using SHA-256, ISBN, UUID +- **OPDS Catalog**: Wireless book delivery to e-readers over Wi-Fi +- **Format Conversion**: On-the-fly EPUBโ†’KEPUB for Kobo devices -Access the application at: **http://localhost:8765** +### Media Management +- **Smart Search**: Partial matching with fuzzy search fallback for typos +- **Advanced Filtering**: Filter by author, series, genre, language, year, cover images +- **Dynamic Sorting**: By title, author, date added, published date, page count, series +- **Rich Metadata**: Title, author, series, publisher, ISBN, language, edition, tags +- **5-Star Ratings**: Half-star precision (1-10 scale) +- **Notes & Highlights**: Color-coded annotations with linked notes +- **Usage Analytics**: Reading statistics, device usage, popular books -## ๐Ÿ‘ค User Management +### Smart Collections +- **Auto-Assign Rules**: Automatically add books based on genre, author, series, tags, language, publisher, year +- **Device Shelf Mappings**: Sync collections to Kobo shelves and KOReader categories +- **Test Before Creating**: Preview which books match your rules -### Role System -- **Admin Users**: Can create/manage libraries, add/delete media items, manage users -- **Regular Users**: Can view permitted libraries, rate items, track reading progress +### Library Organization +- **Multi-Library Support**: Ebooks, Comics, and Manga with type-specific file formats +- **Multiple Folders**: Add multiple scanning folders per library +- **Visibility Control**: Admins control which libraries each user can see +- **Background Scanning**: Auto-scan with per-user frequency settings +- **Watch Mode**: Real-time file system monitoring for instant updates -### First Admin Setup -The first user who registers automatically becomes an admin. For additional admins: +### Security +- **JWT Authentication**: Short-lived access tokens (1 hour) with refresh tokens (7 days) +- **Strong Passwords**: Complexity requirements enforced (8+ chars, uppercase, lowercase, number, special) +- **Account Lockout**: 5 failed attempts = 15-minute lockout +- **Rate Limiting**: 10 requests/minute on auth endpoints +- **Input Validation**: Comprehensive validation on all inputs +- **No Passwords on Devices**: Web-based device approval with QR codes -```sql --- Connect to database -docker exec -it bookhoard_db psql -U postgres -d bookhoard +--- --- Promote user to admin -UPDATE users SET role = 'admin' WHERE email = 'user@example.com'; -``` +## ๐Ÿ“š Documentation -## ๐Ÿ› Library Management +### For Users & Self-Hosters +- **[docs/SYNC_USER_GUIDE.md](docs/SYNC_USER_GUIDE.md)** - Understanding and using universal sync +- **[docs/devices/KOBO_SETUP.md](docs/devices/KOBO_SETUP.md)** - Kobo e-reader configuration +- **[docs/devices/KOREADER_SETUP.md](docs/devices/KOREADER_SETUP.md)** - KOReader configuration +- **[docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)** - Deployment troubleshooting -### Creating Libraries -1. **Admin Access**: Only administrators can create libraries -2. **Library Types**: Choose from Ebooks, Comics, or Manga -3. **Multi-Folder Support**: Add multiple scanning folders per library -4. **Folder Organization**: Organize your media collection across multiple paths +### For Developers +- **[docs/API_REFERENCE.md](docs/API_REFERENCE.md)** - Complete API documentation +- **[docs/COLLECTIONS_API.md](docs/COLLECTIONS_API.md)** - Collections API reference +- **[docs/api/WEBSOCKET_API.md](docs/api/WEBSOCKET_API.md)** - WebSocket protocol +- **[docs/DEVELOPMENT.md](docs/DEVELOPMENT.md)** - Development workflow -### Library Visibility Control -- **Admin Dashboard**: Complete interface for managing library access -- **User-Specific Control**: Admins can show/hide libraries per user -- **Personal Preferences**: Admins can also hide libraries from their own view -- **Simple Toggles**: Checkbox-based interface for easy management +--- -### API Endpoints +## ๐ŸŽฏ Supported Devices -#### Notes & Highlights (All Authenticated Users) -```bash -# Media Notes -GET /api/media-items/{id}/notes # Get user's notes for media -POST /api/media-items/{id}/notes # Create new note -GET /api/media-items/{id}/notes/{noteId} # Get specific note -PUT /api/media-items/{id}/notes/{noteId} # Update note -DELETE /api/media-items/{id}/notes/{noteId} # Delete note +| Platform | Sync | OPDS | Status | +|----------|------|------|--------| +| **Web Browser** | โœ… | โœ… | Full support | +| **KOReader** | โœ… | โœ… | Kindle, Kobo, PocketBook | +| **Kobo Devices** | โœ… | โœ… | Clara, Libra, Sage, etc. | +| **Mobile Apps** | ๐Ÿšง | ๐Ÿšง | Coming Q2 2026 | -# Media Highlights -GET /api/media-items/{id}/highlights # Get user's highlights for media -POST /api/media-items/{id}/highlights # Create new highlight -GET /api/media-items/{id}/highlights/{highlightId} # Get specific highlight -PUT /api/media-items/{id}/highlights/{highlightId} # Update highlight -DELETE /api/media-items/{id}/highlights/{highlightId} # Delete highlight - ``` +--- -#### Media Search (All Authenticated Users) -```bash -# Search media items across all visible libraries -GET /api/media-items/search?q={query} +## ๐Ÿ›  Tech Stack -# Query Parameters: -# q (required): Search query (minimum 2 characters) -# - Searches across: title, author, series, tags, contributors -# - Partial matching: Case-insensitive substring search -# - Fuzzy fallback: Automatic when no partial matches found -# - Results ranked by relevance +- **Backend**: Go 1.25+ with Echo framework +- **Database**: PostgreSQL 15+ with pgx v5 +- **Frontend**: HTMX + Tailwind CSS + Templ +- **Auth**: JWT tokens with bcrypt password hashing +- **Container**: Podman (Docker compatible) -# Examples: -GET /api/media-items/search?q=harry%20potter # Search by title -GET /api/media-items/search?q=rowling # Search by author -GET /api/media-items/search?q=hary%20poter # Fuzzy search (handles typos) - -# Response: -# - 200: Success (returns array of media items) -# - 404: No results found -# - 400: Missing or invalid query parameter -# - 401: Unauthorized -``` - -#### Library Management (Admin Only) -```bash -# Create new library -POST /api/libraries -{ - "name": "My Comic Collection", - "description": "Digital comics and graphic novels", - "type": "comics" -} - -# List all libraries -GET /api/libraries - -# Add folder to library -POST /api/libraries/{library_id}/folders -{ - "folder_path": "/path/to/comics" -} - -# Set library visibility for user -POST /api/libraries/visibility -{ - "library_id": "library-uuid", - "is_visible": true -} - -# Scanner Management (Admin Only) -POST /api/scanner/scan # Manual scan of all libraries -POST /api/scanner/start # Start scheduled scanning -POST /api/scanner/stop # Stop scheduled scanning -GET /api/scanner/status/{jobId} # Get scan job status -POST /api/scanner/watch/start # Start watch mode (real-time monitoring) -POST /api/scanner/watch/stop # Stop watch mode -GET /api/scanner/watch/status # Get watch mode status -``` - -#### User Access -```bash -# Get user's visible libraries -GET /api/libraries/visible - -# Browse media in library -GET /api/media-items?library_id={library_id}&limit=20&offset=0 - -# Media Progress & Ratings -GET /api/media-items/{id}/progress # Get reading progress -PUT /api/media-items/{id}/progress # Update reading progress -GET /api/media-items/{id}/rating # Get user rating -POST /api/media-items/{id}/rating # Create/update rating -PUT /api/media-items/{id}/rating # Update rating -DELETE /api/media-items/{id}/rating # Delete rating - -# Scan Settings (All Authenticated Users) -GET /api/library/scan-settings # Get user's scan settings -PUT /api/library/scan-settings # Update scan settings -{ - "scan_frequency_minutes": 60, - "auto_scan_enabled": true -} - -# Token Management -POST /api/auth/refresh # Refresh access token -POST /api/auth/logout # Logout (revokes refresh token) -``` - -## ๐Ÿ“– Media Support - -### Ebooks -- **Formats**: EPUB (full metadata), PDF, MOBI, AZW, AZW3, TXT, DOC, DOCX, LIT, FB2, PDB, RTF -- **Metadata**: Automatic extraction from EPUB files with Calibre support -- **ISBN Support**: - - Automatic normalization of ISBN-10 and ISBN-13 formats - - Removes hyphens and spaces for consistent storage - - Supports ISBN-10 with check digit X - - Database function `normalize_isbn()` handles all conversions -- **Reading**: Built-in web reader for EPUB files -- **Library Requirement**: Ebooks require an existing ebook library; graceful error handling if none exists - -### Comics -- **Formats**: CBZ, CBR, CB7, CBT, PDF -- **Structure**: Archive-based organization with chapter support -- **Viewing**: Image extraction and web-based comic reader - -## โš™๏ธ Background Processing - -### Auto-Scanning -- **Scheduled Scanning**: User-configurable scan frequency (per-user settings) -- **Smart Scheduling**: Background scheduler manages scan jobs efficiently -- **Per-User Settings**: Each user can enable/disable auto-scan and set frequency -- **Resource Management**: Worker pool limits concurrent scanning operations - -### Watch Mode -- **Real-Time Monitoring**: File system watcher detects new files instantly -- **Automatic Processing**: New media items processed and added to library -- **Multi-Library Support**: Watch all library folders simultaneously -- **Efficient**: Uses filesystem events for minimal resource usage - -### Scan Settings API -```bash -# Get current scan settings -GET /api/library/scan-settings - -# Update scan settings -PUT /api/library/scan-settings -{ - "scan_frequency_minutes": 60, # How often to auto-scan (minutes) - "auto_scan_enabled": true # Enable/disable auto-scanning -} -``` - -### Manga -- **Formats**: CBZ, CBR (archives), PNG, JPG (image folders) -- **Structure**: Archive support with folder-based image organization -- **Viewing**: Chapter-by-chapter viewing with page navigation - -## ๐ŸŽจ Development - -### Local Development -```bash -# Backend development -go mod tidy -go run cmd/server/main.go - -# Frontend development -npm run dev -# Templates automatically recompile on changes - -# Run tests -go test ./... -v -go test ./cmd/server/tests -v # Integration tests only -``` - -### Environment -- **Backend**: Go 1.25+ with pgx v5 for database operations -- **Frontend**: Tailwind CSS with HTMX for dynamic interactions -- **Database**: PostgreSQL 15+ with pgx v5 driver -- **Authentication**: JWT tokens with bcrypt password hashing - -## ๐Ÿณ Deployment - -For detailed deployment instructions, Docker configuration, and troubleshooting common issues, see **[TROUBLESHOOTING.md](TROUBLESHOOTING.md)**. - -The application automatically starts background services on startup: -- **Scheduler**: Manages scheduled scanning jobs based on user settings -- **Watch Mode**: Monitors all library folders for file system changes -- **Worker Pool**: Processes scan jobs with configurable concurrency - -### Database Schema -- **Multi-Library Architecture**: Libraries, library types, folders, visibility tables -- **Annotations System**: Notes, highlights with position tracking and color customization -- **Type Safety**: pgx v5 with proper error handling -- **Migration Ready**: Schema designed for easy future extensions - -## ๐Ÿงช API Documentation - -### Bruno Testing Collection -Complete API testing collection in `bruno/` directory: - -``` -bruno/ -โ”œโ”€โ”€ user/ # Authentication & profile endpoints -โ”œโ”€โ”€ admin/ # Admin-only operations -โ”œโ”€โ”€ library/ # Library management -โ”œโ”€โ”€ media-items/ # Media content browsing -โ”œโ”€โ”€ ebooks/ # Ebook-specific operations -โ”œโ”€โ”€ notes/ # Notes API testing -โ”œโ”€โ”€ highlights/ # Highlights API testing -โ”œโ”€โ”€ scanner/ # Background scanning & watch mode -โ”œโ”€โ”€ progress/ # Reading progress tracking -โ”œโ”€โ”€ devices/ # Device management -โ”œโ”€โ”€ conflicts/ # Sync conflict resolution -โ”œโ”€โ”€ queue/ # Sync queue management -โ”œโ”€โ”€ sync-koreader/ # KOReader sync protocol -โ”œโ”€โ”€ sync-kobo/ # Kobo sync protocol -โ””โ”€โ”€ collection.bru # Main dashboard file -``` - -### Authentication Flow -1. **Register**: `POST /api/auth/register` โ†’ JWT token -2. **Login**: `POST /api/auth/login` โ†’ JWT token -3. **Protected Routes**: Use `Authorization: Bearer {token}` header -4. **Refresh Token**: `POST /api/auth/refresh` โ†’ JWT token -5. **Logout**: `POST /api/auth/logout` โ†’ Revoke refresh token - -### API Reference Documentation -See [API_REFERENCE.md](API_REFERENCE.md) for complete API documentation including: -- All endpoints with request/response examples -- Bruno v3.0 test collections for all endpoints -- Error handling details -- Authentication & security features - -### Device Setup Guides -- [KOBOREADER_SETUP.md](KOBOREADER_SETUP.md) - KOReader device configuration -- [KOBO_SETUP.md](KOBO_SETUP.md) - Kobo device configuration - -### Universal Sync Documentation -- [UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md](UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md) - Complete sync architecture - -### Error Handling -All API endpoints return standardized error responses: -```json -{ - "error": "Error message", - "message": "Detailed error information (if available)", - "code": "Error code (if applicable)" -} -``` - -HTTP Status Codes: -- **400**: Bad request (validation errors) -- **401**: Unauthorized (invalid/missing token) -- **403**: Forbidden (insufficient permissions) -- **404**: Resource not found -- **429**: Too many requests (rate limit exceeded) -- **500**: Internal server error - -## ๐Ÿ›ก Security Features - -### Authentication & Authorization -- **JWT Tokens**: - - Short-lived access tokens (1-hour expiration) - - Refresh tokens (7-day expiration) for seamless session renewal - - Secure token storage and transmission -- **Password Security**: - - bcrypt hashing with cost factor 12 - - Complex password requirements enforced - - Password validation on registration and updates -- **Account Protection**: - - Automatic lockout after 5 failed login attempts - - 15-minute lockout period with countdown - - IP-based and username-based attempt tracking -- **Rate Limiting**: 10 requests per minute on authentication endpoints -- **Input Validation**: Comprehensive server-side validation for all inputs -- **CSRF Protection**: Built-in with HTMX -- **Role-Based Access Control**: Admin and user roles enforced on all endpoints - -### Authorization -- **Role-Based Access**: Admin vs user permissions -- **Library Visibility**: Per-user library access control -- **Admin Middleware**: Protected routes for admin operations -- **Content Security**: XSS protection and secure headers - -### Database Security -- **Parameterized Queries**: SQL injection prevention with pgx v5 -- **Connection Pooling**: Efficient database connection management -- **Transaction Support**: Multi-step operations wrapped in transactions -- **Environment Variables**: Secure secret management -- **Row-Level Security**: User data isolation -- **Secure Refresh Token Storage**: Encrypted token storage in database -- **Automatic Token Cleanup**: Expired tokens cleaned up periodically -- **ISBN Normalization**: Database function ensures consistent ISBN format -- **Data Validation**: Server-side validation for all inputs including ISBNs - -## ๐Ÿ”ง Configuration - -### Environment Variables -```bash -# Required -JWT_SECRET= # JWT signing secret (64-byte random string) -DBHOST=db # PostgreSQL database host - -# Optional -SERVER_PORT=8765 # Application port -NODE_ENV=development # Environment mode -``` - -### Database Configuration -```sql --- Library types are automatically seeded -INSERT INTO library_types (name, description, allowed_extensions) VALUES -('ebooks', 'Ebook files including EPUB, PDF, MOBI, etc.', - ARRAY['.epub', '.pdf', '.mobi', '.azw', '.azw3', '.txt', '.rtf', '.doc', '.docx', '.lit', '.fb2', '.pdb']), -('comics', 'Comic book archives and image formats', - ARRAY['.cbz', '.cbr', '.cb7', '.cbt', '.pdf']), -('manga', 'Manga files including archives and image folders', - ARRAY['.cbz', '.cbr', '.png', '.jpg', '.jpeg', '.gif', '.bmp', '.webp']); -``` +--- ## ๐Ÿงช Testing -### Integration Tests -Comprehensive integration test suite in `cmd/server/tests/`: -- Authentication & authorization tests -- Library management tests -- Media items operations -- Notes & highlights functionality -- ISBN normalization tests -- Background scanning & watch mode tests -- Edge cases and error handling -- Security & rate limiting tests - -Run integration tests: ```bash -# Standard test run (may hit rate limits) -go test ./cmd/server/tests -v +# Run all tests +make test-all -# Recommended: Run with test mode enabled to avoid rate limiting -TEST_MODE=true RATE_LIMIT_ENABLED=false go test ./cmd/server/tests -run TestIntegrationAPI -v +# Run integration tests (with test mode) +make test-integration -# Or with increased rate limits -TEST_MODE=true REQUESTS_PER_MINUTE=1000 go test ./cmd/server/tests -run TestIntegrationAPI -v -``` - -### Test Configuration Options -The following environment variables can be used to configure test behavior: - -- **TEST_MODE**: Set to `true` to enable test mode (logs additional info) -- **RATE_LIMIT_ENABLED**: Set to `false` to disable rate limiting for tests -- **REQUESTS_PER_MINUTE**: Increase rate limit (e.g., `1000`) to avoid throttling - -**โš ๏ธ WARNING**: Never disable rate limiting or enable test mode in production environments. These settings are only for integration testing. - -### API Testing -```bash -# Install Bruno +# Run Bruno API tests npm install -g @usebruno/cli - -# Run tests bruno run ``` -### Unit Tests -```bash -go test ./... -``` +--- -## ๐Ÿ“Š Architecture +## ๐Ÿ“Š Project Status -### Backend -``` -cmd/server/main.go # Application entry point -โ”œโ”€โ”€ internal/ -โ”‚ โ”œโ”€โ”€ config/ # Configuration management -โ”‚ โ”œโ”€โ”€ database/ # Database operations (SQLC generated) -โ”‚ โ”‚ โ”œโ”€โ”€ queries/ # SQL queries -โ”‚ โ”‚ โ””โ”€โ”€ models.go # Generated models -โ”‚ โ”œโ”€โ”€ handlers/ # HTTP handlers -โ”‚ โ”‚ โ”œโ”€โ”€ auth.go # Authentication & users -โ”‚ โ”‚ โ”œโ”€โ”€ library.go # Library management -โ”‚ โ”‚ โ”œโ”€โ”€ ebook.go # Media operations -โ”‚ โ”‚ โ””โ”€โ”€ refresh_token.go # Token management -โ”‚ โ”œโ”€โ”€ services/ # Business logic -โ”‚ โ”‚ โ”œโ”€โ”€ library_service.go # Library service -โ”‚ โ”‚ โ”œโ”€โ”€ ebook_scanner.go # Media scanning -โ”‚ โ”‚ โ”œโ”€โ”€ worker.go # Background job processing -โ”‚ โ”‚ โ””โ”€โ”€ scheduler.go # Auto-scan scheduling -โ”‚ โ””โ”€โ”€ middleware/ # HTTP middleware -โ”‚ โ”œโ”€โ”€ rate_limiter.go # Rate limiting -โ”‚ โ”œโ”€โ”€ login_attempts.go # Account lockout -โ”‚ โ”œโ”€โ”€ password_validator.go # Password complexity -โ”‚ โ”œโ”€โ”€ error_handler.go # Error handling -โ”‚ โ”œโ”€โ”€ transaction.go # DB transactions -โ”‚ โ””โ”€โ”€ request_tracing.go # Request logging -templates/ # HTML templates with HTMX -``` +**Version**: 1.0 +**License**: GPL-3.0 +**Status**: Production-ready โœ… -### Database -```sql -libraries # Library definitions -library_types # Media type definitions (ebooks, comics, manga) -library_folders # Folders per library -library_visibility # User library access control -media_items # All media content (replaces ebooks table) -media_ratings # User ratings for media items -media_notes # User notes on media items -media_highlights # User highlights with optional note links -reading_progress # User reading progress -users # User accounts and profiles (includes scan settings) -refresh_tokens # JWT refresh token storage +--- -# Database Functions -normalize_isbn() # ISBN normalization function -``` +## ๐Ÿค Contributing -## ๐ŸŽฏ Future Roadmap +We welcome contributions! Please see [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) for guidelines. -### Recently Added Features (v1.0) -- โœ… **ISBN Normalization**: Automatic ISBN format standardization -- โœ… **Background Scanning**: Scheduled library scanning with per-user settings -- โœ… **Watch Mode**: Real-time file system monitoring for instant updates -- โœ… **Integration Tests**: Comprehensive test suite for all major features -- โœ… **Request Tracing**: Enhanced logging and debugging capabilities -- โœ… **Graceful Library Handling**: Better error messages when libraries don't exist - -### Media Type Extensions -- **Audiobooks**: Audio file support with chapter tracking -- **Podcasts**: RSS feed integration and automatic downloading -- **Video**: Movie/TV series management with metadata -- **Music**: Album and track management with artwork - -### Advanced Features -- **Mobile App**: React Native mobile application -- **API v2**: GraphQL API for efficient data fetching -- **Webhooks**: External service integrations -- **Analytics**: Usage statistics and reporting -- **Backup/Restore**: Library backup and migration tools - -## ๐Ÿ“ Contributing - -### Development Guidelines -- Follow Go best practices and effective Go -- Use pgx v5 for all database operations -- Implement proper error handling with pgx.ErrNoRows -- Write comprehensive tests for new features -- Update documentation for API changes -- Add Bruno tests for new API endpoints -- Include integration tests for complex features - -### Code Style -- Follow existing code formatting -- Use meaningful variable and function names -- Add comments for complex business logic -- Ensure type safety with proper error handling +--- ## ๐Ÿ“„ License @@ -622,4 +157,4 @@ GPL-3.0 - See [LICENSE](LICENSE) file for details. --- -**Built with โค๏ธ using Go, PostgreSQL, HTMX, and Tailwind CSS** \ No newline at end of file +**Built with โค๏ธ using Go, PostgreSQL, HTMX, and Tailwind CSS** diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md index 4b5263a..58808f6 100644 --- a/docs/API_REFERENCE.md +++ b/docs/API_REFERENCE.md @@ -12,12 +12,16 @@ Complete API documentation for Bookhoard v1.0 with Universal Cross-Platform Sync 6. [Notes & Highlights](#notes--highlights) 7. [Ratings](#ratings) 8. [Device Management](#device-management) -9. [Sync Protocol - KOReader](#sync-protocol---koreader) -10. [Sync Protocol - Kobo](#sync-protocol---kobo) -11. [Universal Progress](#universal-progress) -12. [Conflicts](#conflicts) -13. [Sync Queue](#sync-queue) -14. [WebSocket](#websocket) +9. [Analytics](#analytics) +10. [Book Matching & Linking](#book-matching--linking) +11. [Collections](#collections) โ†’ See [COLLECTIONS_API.md](COLLECTIONS_API.md) +12. [OPDS](#opds-open-publication-distribution-system) +13. [Sync Protocol - KOReader](#sync-protocol---koreader) +14. [Sync Protocol - Kobo](#sync-protocol---kobo) +15. [Universal Progress](#universal-progress) +16. [Conflicts](#conflicts) +17. [Sync Queue](#sync-queue) +18. [WebSocket](#websocket) ## Base URL @@ -735,6 +739,323 @@ DELETE /api/devices/{device_id} Authorization: Bearer ``` +## Analytics + +### Get Reading Statistics + +```http +GET /api/analytics/reading-stats?start_date={date}&end_date={date} +Authorization: Bearer +``` + +**Query Parameters**: +- `start_date` (optional): Start date (ISO 8601 format) +- `end_date` (optional): End date (ISO 8601 format) + +**Response** (200): +```json +{ + "pages_read": 1250, + "books_completed": 5, + "reading_time_hours": 42.5, + "sessions_count": 28, + "average_session_minutes": 91 +} +``` + +### Get Device Usage Statistics + +```http +GET /api/analytics/device-usage +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "devices": [ + { + "device_id": "uuid", + "device_name": "My Kobo Clara", + "device_type": "kobo", + "sync_count": 145, + "last_sync": "2026-02-01T10:00:00Z", + "pages_synced": 890 + } + ] +} +``` + +### Get Popular Books + +```http +GET /api/analytics/popular-books?limit={limit} +Authorization: Bearer +``` + +**Query Parameters**: +- `limit` (optional): Number of results (default: 10) + +**Response** (200): +```json +{ + "books": [ + { + "media_item_id": "uuid", + "title": "Book Title", + "author": "Author Name", + "cover_image_path": "/covers/book.jpg", + "read_count": 15, + "average_rating": 8.5 + } + ] +} +``` + +## Book Matching & Linking + +### Query Books for Matching + +```http +POST /api/sync/books/query +Authorization: Bearer +Content-Type: application/json + +{ + "identifiers": ["isbn:978-0345391802", "uuid:abc-123"], + "sha256": "a1b2c3d4e5f6abc123...", + "title": "The Hobbit", + "author": "J.R.R. Tolkien", + "file_size": 2456789 +} +``` + +**Response** (200): +```json +{ + "matches": [ + { + "media_item_id": "uuid-123", + "bookhoard_uuid": "uuid-123", + "confidence": 1.0, + "match_method": "uuid_match" + } + ], + "action": "auto_link" +} +``` + +### Bulk Link Books + +```http +POST /api/sync/bulk-link-books +Authorization: Bearer +Content-Type: application/json + +{ + "links": [ + { + "unlinked_book_id": "uuid-1", + "media_item_id": "uuid-2", + "confidence_score": 1.0 + } + ] +} +``` + +**Response** (200): +```json +{ + "results": [ + { + "unlinked_book_id": "uuid-1", + "status": "success", + "media_item_id": "uuid-2" + } + ], + "total": 1, + "successful": 1, + "failed": 0 +} +``` + +### Auto-Link Books + +```http +POST /api/sync/auto-link-books +Authorization: Bearer +Content-Type: application/json + +{ + "confidence_threshold": 0.8, + "limit": 50 +} +``` + +**Response** (200): +```json +{ + "auto_linked": 15, + "results": [ + { + "unlinked_book_id": "uuid-1", + "title": "The Hobbit", + "matched_media_item_id": "uuid-2", + "confidence": 0.95, + "match_method": "sha256_match" + } + ] +} +``` + +### Get Unlinked Book Suggestions + +```http +GET /api/sync/unlinked-books/{id}/suggestions +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "unlinked_book_id": "uuid-1", + "title_from_device": "The Hobbit", + "sha256": "", + "suggestions": [ + { + "media_item_id": "uuid-2", + "bookhoard_uuid": "uuid-2", + "confidence": 0.95, + "match_method": "sha256_match" + } + ], + "total_suggestions": 1, + "action": "auto_link" +} +``` + +## Collections + +For complete collection management documentation, see **[COLLECTIONS_API.md](COLLECTIONS_API.md)**. + +**Quick Reference**: +- `GET /api/collections` - List all collections +- `POST /api/collections` - Create new collection +- `GET /api/collections/{id}` - Get collection details +- `PUT /api/collections/{id}` - Update collection +- `DELETE /api/collections/{id}` - Delete collection +- `POST /api/collections/test-rules` - Test auto-assign rules +- `POST /api/collections/bulk-add-books` - Bulk add books to collections +- `GET /api/collections/{id}/books` - Get books in collection + +**Features**: +- Auto-assign rules based on genre, author, series, tags, language, publisher, year +- Device shelf mappings (Kobo shelves, KOReader categories) +- Test rules before applying +- Bulk book operations + +## OPDS (Open Publication Distribution System) + +### Get Device Catalog + +```http +GET /opds/devices/{deviceId}/catalog?page={page}&per_page={per_page} +``` + +**Query Parameters**: +- `page` (optional): Page number (default: 1) +- `per_page` (optional): Items per page (default: 50, max: 200) + +**Response** (200 - OPDS 1.2 XML): +```xml + + + urn:uuid:device-id + Bookhoard Library + 2026-02-01T12:00:00Z + + + + + + + urn:uuid:bookhoard-uuid-123 + The Hobbit + J.R.R. Tolkien + 2026-02-01T10:00:00Z + + + + + + uuid-123 + abc123... + + +``` + +### Download Book with Format Conversion + +```http +GET /opds/devices/{deviceId}/download/{bookId}?format={format} +``` + +**Query Parameters**: +- `format` (optional): Book format - `epub` (default), `kepub` + +**Response** (200): +- Headers: + - `Content-Type`: `application/epub+zip` or `application/vnd.kobo+xml+zip` + - `Content-Disposition`: attachment; filename="The Hobbit.epub" + - `X-Bookhoard-UUID`: uuid-123 + - `X-Bookhoard-SHA256`: abc123... (original hash) + - `X-Bookhoard-KEPUB-SHA256`: xyz789... (KEPUB hash if format=kepub) + +### Search OPDS Catalog + +```http +GET /opds/devices/{deviceId}/search?q={query} +``` + +**Response** (200 - OPDS 1.2 XML with search results) + +### List Available Formats + +```http +GET /opds/devices/{deviceId}/formats/{bookId} +``` + +**Response** (200): +```json +{ + "media_item_id": "uuid-123", + "formats": [ + { + "format_type": "epub", + "file_path": "/path/to/book.epub", + "file_sha256": "abc123...", + "file_size_bytes": 2456789, + "mime_type": "application/epub+zip", + "available": true + }, + { + "format_type": "kepub", + "file_path": "/cache/book.kepub.epub", + "file_sha256": "xyz789...", + "file_size_bytes": 2478932, + "mime_type": "application/vnd.kobo+xml+zip", + "available": true + } + ] +} +``` + ## Sync Protocol - KOReader ### KOReader Progress Sync diff --git a/docs/SYNC_USER_GUIDE.md b/docs/SYNC_USER_GUIDE.md index a05e37c..e311c72 100644 --- a/docs/SYNC_USER_GUIDE.md +++ b/docs/SYNC_USER_GUIDE.md @@ -4,10 +4,11 @@ 1. [What is Universal Sync?](#what-is-universal-sync) 2. [Supported Devices](#supported-devices) 3. [Getting Started](#getting-started) -4. [Device Registration](#device-registration) -5. [Sync Features](#sync-features) -6. [Troubleshooting](#troubleshooting) -7. [Best Practices](#best-practices) +4. [Device Setup](#device-setup) +5. [Using Sync Features](#using-sync-features) +6. [Book Matching](#book-matching) +7. [Managing Conflicts](#managing-conflicts) +8. [Best Practices](#best-practices) --- @@ -61,72 +62,83 @@ ### Quick Start -1. **Start Reading** - Open any book in the web interface or on your device -2. **Turn Pages** - Progress syncs automatically -3. **Switch Devices** - Pick up any other device - your progress is there! +1. **Set up your first device** - See [Device Setup](#device-setup) below +2. **Start reading** - Open any book in the web interface or on your device +3. **Turn pages** - Progress syncs automatically +4. **Switch devices** - Pick up any other device - your progress is there! --- -## Device Registration +## Device Setup -### Step-by-Step Guide +For detailed device configuration instructions, see the appropriate setup guide: -#### Option 1: Web-Based Registration (Recommended) +- **[Kobo Setup Guide](devices/KOBO_SETUP.md)** - Kobo e-reader configuration +- **[KOReader Setup Guide](devices/KOREADER_SETUP.md)** - KOReader configuration -1. **On Your Device**: - - Open your reading app (KOReader/Kobo) - - Navigate to sync settings - - Find "Bookhoard Sync" or "Calibre Sync" - - Copy your device identifier (hardware ID) +### Quick Overview -2. **In Your Browser**: - - Go to your Bookhoard server - - Log in with your account - - Navigate to **Settings โ†’ Devices** - - Click **Register New Device** - - Enter your device name and type - - Click **Generate Registration** +**Registration Process**: +1. Register device in Bookhoard web interface (Settings โ†’ Devices) +2. Approve device via QR code or approval URL +3. Configure sync settings on your device +4. Start reading - progress syncs automatically! -3. **Complete Registration**: - - Scan the QR code OR copy the registration URL - - Visit the approval URL in your browser - - Review device details and click **Approve** - - Device will receive authentication token - -4. **Configure Sync**: - - Enter the sync URL provided (e.g., `https://bookhoard.example.com/api/sync/koreader`) - - Enable auto-sync - - Set sync frequency (recommended: Every page turn) - -#### Option 2: Direct Device Registration - -1. **From Your Device**: - - Navigate to sync settings - - Select "Bookhoard Sync" - - Enter server URL: `https://bookhoard.example.com` - - Click "Register Device" - - Device will display registration code - -2. **In Your Browser**: - - Go to `https://bookhoard.example.com/devices/approve` - - Enter registration code - - Approve the device - -### Device Management - -**View Your Devices**: +**Device Management**: ``` Settings โ†’ Devices ``` -**Manage Devices**: -- **Rename**: Click device name โ†’ Edit -- **Disable Sync**: Toggle "Sync Enabled" -- **Remove Device**: Click "Delete" (revokes access immediately) +You can: +- View all your registered devices +- See last sync time and status +- Disable or remove devices +- View unlinked books that need matching +- Manage file aliases --- -## Sync Features +## Book Matching + +### What is Book Matching? + +When devices sync books, Bookhoard tries to automatically match them using: +- **SHA-256 hash** (most reliable) - Content-based fingerprint +- **ISBN** - Standard book identifier +- **UUID** - Unique identifier from EPUB metadata +- **Title + Author** (fallback) - Last resort matching + +### Unlinked Books + +Sometimes a book on your device can't be automatically matched to your library. This happens when: +- The book was side-loaded (not downloaded via Bookhoard) +- The file format was converted (EPUB โ†’ KEPUB) +- The metadata doesn't match exactly + +### Viewing Unlinked Books + +``` +Settings โ†’ Devices โ†’ Select Device โ†’ View Unlinked Books +``` + +### Resolving Unlinked Books + +1. **Auto-Link** - Automatically link books with 80%+ confidence +2. **Get Suggestions** - Review potential matches before linking +3. **Manual Link** - Choose the correct match yourself + +### Book Matching Priority + +1. **Bookhoard UUID** - 100% confidence +2. **OPF UUID** - 95% confidence +3. **SHA-256 hash** - 90% confidence +4. **OPF identifier** - 85% confidence +5. **ISBN/ASIN** - 80% confidence +6. **Title + Author + File Size** - 50% confidence + +--- + +## Managing Conflicts ### Reading Progress