From 5f355266e4d0da6206e3702f18f066d07460bf75 Mon Sep 17 00:00:00 2001 From: John O'Keefe Date: Thu, 29 Jan 2026 12:12:13 -0500 Subject: [PATCH] docs(readme): update README with current project features - Add ISBN normalization documentation - Document background scanning and watch mode features - Add scan settings API endpoints - Include integration testing section - Update architecture section with new services - Document auto-starting services - Add recently added features section - Update testing documentation with integration tests - Enhance security section with ISBN validation --- README.md | 126 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 119 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 6373922..c68fd65 100644 --- a/README.md +++ b/README.md @@ -43,11 +43,14 @@ A modern self-hosted media library system built with Go, PostgreSQL, HTMX, and T ### ๐Ÿ“ฑ Media Management - **Universal Media Support**: Single system for all media types with unified interface - **Rich Metadata**: Automatic extraction of title, author, series, publisher, ISBN, tags +- **ISBN Normalization**: Automatic ISBN format normalization (removes hyphens and spaces) supporting ISBN-10 and ISBN-13 formats - **Advanced Rating**: 5-star system with half-star precision (1-10 scale) - **Reading Progress**: User-specific progress tracking with current page and total pages - **Notes & Highlights**: Personal annotations and text highlighting with color customization - **Highlight Notes**: Link highlights to detailed notes for comprehensive annotations - **Library Statistics**: Media count and usage statistics per library +- **Background Scanning**: Automatic background scanning with configurable frequency per user +- **Watch Mode**: Real-time file system monitoring for instant library updates ## ๐Ÿš€ Quick Start @@ -166,6 +169,15 @@ POST /api/libraries/visibility "library_id": "library-uuid", "is_visible": true } + +# Scanner Management (Admin Only) +POST /api/scanner/scan # Manual scan of all libraries +POST /api/scanner/start # Start scheduled scanning +POST /api/scanner/stop # Stop scheduled scanning +GET /api/scanner/status/{jobId} # Get scan job status +POST /api/scanner/watch/start # Start watch mode (real-time monitoring) +POST /api/scanner/watch/stop # Stop watch mode +GET /api/scanner/watch/status # Get watch mode status ``` #### User Access @@ -184,6 +196,14 @@ POST /api/media-items/{id}/rating # Create/update rating PUT /api/media-items/{id}/rating # Update rating DELETE /api/media-items/{id}/rating # Delete rating +# Scan Settings (All Authenticated Users) +GET /api/library/scan-settings # Get user's scan settings +PUT /api/library/scan-settings # Update scan settings +{ + "scan_frequency_minutes": 60, + "auto_scan_enabled": true +} + # Token Management POST /api/auth/refresh # Refresh access token POST /api/auth/logout # Logout (revokes refresh token) @@ -192,15 +212,48 @@ POST /api/auth/logout # Logout (revokes refresh token) ## ๐Ÿ“– Media Support ### Ebooks -- **Formats**: EPUB (full metadata), PDF, MOBI, AZW, TXT, DOC, FB2 +- **Formats**: EPUB (full metadata), PDF, MOBI, AZW, AZW3, TXT, DOC, DOCX, LIT, FB2, PDB, RTF - **Metadata**: Automatic extraction from EPUB files with Calibre support +- **ISBN Support**: + - Automatic normalization of ISBN-10 and ISBN-13 formats + - Removes hyphens and spaces for consistent storage + - Supports ISBN-10 with check digit X + - Database function `normalize_isbn()` handles all conversions - **Reading**: Built-in web reader for EPUB files +- **Library Requirement**: Ebooks require an existing ebook library; graceful error handling if none exists ### Comics - **Formats**: CBZ, CBR, CB7, CBT, PDF - **Structure**: Archive-based organization with chapter support - **Viewing**: Image extraction and web-based comic reader +## โš™๏ธ Background Processing + +### Auto-Scanning +- **Scheduled Scanning**: User-configurable scan frequency (per-user settings) +- **Smart Scheduling**: Background scheduler manages scan jobs efficiently +- **Per-User Settings**: Each user can enable/disable auto-scan and set frequency +- **Resource Management**: Worker pool limits concurrent scanning operations + +### Watch Mode +- **Real-Time Monitoring**: File system watcher detects new files instantly +- **Automatic Processing**: New media items processed and added to library +- **Multi-Library Support**: Watch all library folders simultaneously +- **Efficient**: Uses filesystem events for minimal resource usage + +### Scan Settings API +```bash +# Get current scan settings +GET /api/library/scan-settings + +# Update scan settings +PUT /api/library/scan-settings +{ + "scan_frequency_minutes": 60, # How often to auto-scan (minutes) + "auto_scan_enabled": true # Enable/disable auto-scanning +} +``` + ### Manga - **Formats**: CBZ, CBR (archives), PNG, JPG (image folders) - **Structure**: Archive support with folder-based image organization @@ -217,6 +270,10 @@ go run cmd/server/main.go # Frontend development npm run dev # Templates automatically recompile on changes + +# Run tests +go test ./... -v +go test ./cmd/server/tests -v # Integration tests only ``` ### Environment @@ -240,8 +297,17 @@ services: - DBHOST=db depends_on: - db + volumes: + - /path/to/media:/media # Mount media directories + - /path/to/uploads:/uploads # Upload directory ``` +### Auto-Starting Services +The application automatically starts background services on startup: +- **Scheduler**: Manages scheduled scanning jobs based on user settings +- **Watch Mode**: Monitors all library folders for file system changes +- **Worker Pool**: Processes scan jobs with configurable concurrency + ### Database Schema - **Multi-Library Architecture**: Libraries, library types, folders, visibility tables - **Annotations System**: Notes, highlights with position tracking and color customization @@ -258,10 +324,13 @@ Complete API testing collection in `bruno/` directory: bruno/ โ”œโ”€โ”€ user/ # Authentication & profile endpoints โ”œโ”€โ”€ admin/ # Admin-only operations -โ”œโ”€โ”€ library/ # Library management +โ”œโ”€โ”€ library/ # Library management โ”œโ”€โ”€ media-items/ # Media content browsing +โ”œโ”€โ”€ ebooks/ # Ebook-specific operations โ”œโ”€โ”€ notes/ # Notes API testing โ”œโ”€โ”€ highlights/ # Highlights API testing +โ”œโ”€โ”€ scanner/ # Background scanning & watch mode +โ”œโ”€โ”€ progress/ # Reading progress tracking โ””โ”€โ”€ collection.bru # Main dashboard ``` @@ -322,6 +391,8 @@ HTTP Status Codes: - **Row-Level Security**: User data isolation - **Secure Refresh Token Storage**: Encrypted token storage in database - **Automatic Token Cleanup**: Expired tokens cleaned up periodically +- **ISBN Normalization**: Database function ensures consistent ISBN format +- **Data Validation**: Server-side validation for all inputs including ISBNs ## ๐Ÿ”ง Configuration @@ -350,6 +421,22 @@ INSERT INTO library_types (name, description, allowed_extensions) VALUES ## ๐Ÿงช Testing +### Integration Tests +Comprehensive integration test suite in `cmd/server/tests/`: +- Authentication & authorization tests +- Library management tests +- Media items operations +- Notes & highlights functionality +- ISBN normalization tests +- Background scanning & watch mode tests +- Edge cases and error handling +- Security & rate limiting tests + +Run integration tests: +```bash +go test ./cmd/server/tests -v +``` + ### API Testing ```bash # Install Bruno @@ -377,9 +464,20 @@ cmd/server/main.go # Application entry point โ”‚ โ”œโ”€โ”€ handlers/ # HTTP handlers โ”‚ โ”‚ โ”œโ”€โ”€ auth.go # Authentication & users โ”‚ โ”‚ โ”œโ”€โ”€ library.go # Library management -โ”‚ โ”‚ โ””โ”€โ”€ ebook.go # Media operations -โ”‚ โ””โ”€โ”€ services/ # Business logic -โ”‚ โ””โ”€โ”€ library_service.go # Library service +โ”‚ โ”‚ โ”œโ”€โ”€ ebook.go # Media operations +โ”‚ โ”‚ โ””โ”€โ”€ refresh_token.go # Token management +โ”‚ โ”œโ”€โ”€ services/ # Business logic +โ”‚ โ”‚ โ”œโ”€โ”€ library_service.go # Library service +โ”‚ โ”‚ โ”œโ”€โ”€ ebook_scanner.go # Media scanning +โ”‚ โ”‚ โ”œโ”€โ”€ worker.go # Background job processing +โ”‚ โ”‚ โ””โ”€โ”€ scheduler.go # Auto-scan scheduling +โ”‚ โ””โ”€โ”€ middleware/ # HTTP middleware +โ”‚ โ”œโ”€โ”€ rate_limiter.go # Rate limiting +โ”‚ โ”œโ”€โ”€ login_attempts.go # Account lockout +โ”‚ โ”œโ”€โ”€ password_validator.go # Password complexity +โ”‚ โ”œโ”€โ”€ error_handler.go # Error handling +โ”‚ โ”œโ”€โ”€ transaction.go # DB transactions +โ”‚ โ””โ”€โ”€ request_tracing.go # Request logging templates/ # HTML templates with HTMX ``` @@ -393,8 +491,12 @@ media_items # All media content (replaces ebooks table) media_ratings # User ratings for media items media_notes # User notes on media items media_highlights # User highlights with optional note links -reading_progress # User reading progress -users # User accounts and profiles +reading_progress # User reading progress +users # User accounts and profiles (includes scan settings) +refresh_tokens # JWT refresh token storage + +# Database Functions +normalize_isbn() # ISBN normalization function # Backward Compatibility Views ebook_notes # Notes view for ebook API compatibility @@ -403,6 +505,14 @@ ebook_highlights # Highlights view for ebook API compatibility ## ๐ŸŽฏ Future Roadmap +### Recently Added Features (v1.0) +- โœ… **ISBN Normalization**: Automatic ISBN format standardization +- โœ… **Background Scanning**: Scheduled library scanning with per-user settings +- โœ… **Watch Mode**: Real-time file system monitoring for instant updates +- โœ… **Integration Tests**: Comprehensive test suite for all major features +- โœ… **Request Tracing**: Enhanced logging and debugging capabilities +- โœ… **Graceful Library Handling**: Better error messages when libraries don't exist + ### Media Type Extensions - **Audiobooks**: Audio file support with chapter tracking - **Podcasts**: RSS feed integration and automatic downloading @@ -424,6 +534,8 @@ ebook_highlights # Highlights view for ebook API compatibility - Implement proper error handling with pgx.ErrNoRows - Write comprehensive tests for new features - Update documentation for API changes +- Add Bruno tests for new API endpoints +- Include integration tests for complex features ### Code Style - Follow existing code formatting