diff --git a/AUTOSCANNER_IMPROVEMENTS.md b/AUTOSCANNER_IMPROVEMENTS.md deleted file mode 100644 index 1939bad..0000000 --- a/AUTOSCANNER_IMPROVEMENTS.md +++ /dev/null @@ -1,207 +0,0 @@ -# Autoscanner Improvements - Implementation Plan - -## Problem Statement - -The current autoscanner has two issues: - -1. **Bulk file detection unreliable**: When files are copied in bulk (e.g., 6 files at once), fsnotify only detects 1 or 0 files -2. **Delete detection broken**: When files are deleted from the filesystem, they remain in the database - -### Root Causes - -1. **fsnotify limitations**: The file system watcher can miss events during bulk file operations -2. **Relative path mismatch**: Delete detection uses absolute paths for database lookups, but database stores relative paths (partially fixed) - -## Solution Architecture - -``` -[fsnotify Events] → [Event Queue] → [Debounce Timer (3s)] → [Process Queue] - ↓ -[Polling Fallback (3 min)] → [Full Sync Check] → [Add missing / Remove deleted] -``` - -Two complementary systems working together: -- **fsnotify + debounce**: Handles most file changes in real-time -- **Polling fallback**: Safety net that catches anything fsnotify misses - -## Implementation Plan - -### Phase 1: Add Debounce to fsnotify Handler - -**File**: `internal/services/media_scanner.go` - -**Changes**: -1. Create an event queue to accumulate fsnotify events -2. Add debounce timer (3 seconds) that resets on each new event -3. When timer fires, process all queued events in batch -4. Process each event: new files → scan, deleted files → remove from DB - -**Implementation Details**: -- Use a buffered channel as the event queue -- Use `time.After()` or `time.Timer` for debounce -- Process events in order, skip duplicates for same file - -```go -// Pseudo-code structure -type FileEvent struct { - path string - isDelete bool -} - -eventQueue := make(chan FileEvent, 100) -var debounceTimer *time.Timer - -func handleFsEvent(event fsnotify.Event) { - select { - case eventQueue <- FileEvent{path: event.Name, isDelete: event.Has(fsnotify.Remove)}: - default: - // Queue full, log warning - } - - // Reset debounce timer - if debounceTimer != nil { - debounceTimer.Stop() - } - debounceTimer = time.AfterFunc(3*time.Second, processEventQueue) -} - -func processEventQueue() { - // Drain queue and process unique paths -} -``` - -### Phase 2: Add Polling Fallback - -**File**: `internal/services/media_scanner.go` (or new file) - -**Changes**: -1. Add polling interval configuration (default: 3 minutes) -2. Create sync function that: - - Walks all library folders - - Compares filesystem against database - - Adds missing files (triggers scan for new files) - - Removes orphaned database entries (files no longer exist) -3. Start polling goroutine alongside existing fsnotify watcher - -**Configuration**: -- Environment variable: `SCAN_POLL_INTERVAL_MINUTES` (default: 3) -- Use existing config system or add to `internal/config/config.go` - -**Implementation Details**: -```go -type ScannerSyncOptions struct { - PollInterval time.Duration // default: 3 minutes -} - -func (s *MediaScanner) StartPolling(ctx context.Context, opts ScannerSyncOptions) { - ticker := time.NewTicker(opts.PollInterval) - defer ticker.Stop() - - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - s.SyncFilesystemWithDatabase(ctx) - } - } -} - -func (s *MediaScanner) SyncFilesystemWithDatabase(ctx context.Context) error { - // 1. Get all media items from database - // 2. For each library folder: - // - Walk filesystem, build map of existing files (relative paths) - // - Compare against database - // - Add: file in filesystem but not in DB → scan - // - Remove: file in DB but not on filesystem → delete -} -``` - -### Phase 3: Reuse Existing Cleanup Logic - -**File**: `internal/services/media_scanner.go` - -The cleanup logic already exists (lines 252-299 in `ScanFolders`) - it removes orphaned items after each manual scan. We can refactor this into a reusable function called by both: -- Manual scan (existing behavior) -- Polling fallback (new behavior) - -```go -func (s *MediaScanner) CleanupOrphanedItems(ctx context.Context) error { - // Existing cleanup code from ScanFolders - // Extract to reusable function -} -``` - -### Phase 4: Configuration - -**File**: `internal/config/config.go` - -Add new configuration option: - -```go -type Config struct { - // ... existing fields ... - ScanPollIntervalMinutes int `env:"SCAN_POLL_INTERVAL_MINUTES"` -} - -func LoadConfig() *Config { - return &Config{ - // ... existing fields ... - ScanPollIntervalMinutes: getEnvInt("SCAN_POLL_INTERVAL_MINUTES", 3), // 3 minutes default - } -} -``` - -### Phase 5: Integration with Scheduler - -**File**: `internal/services/scheduler.go` - -The scheduler already handles periodic tasks. Consider: -1. Adding polling fallback to scheduler, OR -2. Starting polling directly in MediaScanner initialization - -## Files to Modify - -| File | Changes | -|------|---------| -| `internal/services/media_scanner.go` | Add debounce queue, polling fallback, reuse cleanup | -| `internal/config/config.go` | Add `SCAN_POLL_INTERVAL_MINUTES` config | -| `docker-compose.yml` | Add environment variable (optional) | - -## Testing Plan - -1. **Bulk file addition**: Copy 10+ files at once, verify all detected within 3 seconds (fsnotify) or 3 minutes (polling) -2. **Bulk file deletion**: Delete 5+ files, verify all removed from database within 3 minutes -3. **Mixed operations**: Add some, delete some, verify correct state -4. **Large library**: Test with 100+ files to ensure performance is acceptable - -## Backward Compatibility - -- Default polling interval: 3 minutes (user can configure) -- Existing manual scan functionality unchanged -- fsnotify continues to work as before (with debounce improvement) - -## Security Considerations - -- Polling runs on same goroutine as scanner (no new attack surface) -- File operations are read-only until changes detected -- Database operations use existing service layer (already authorized) - -## Timeline Estimate - -| Phase | Complexity | Estimate | -|-------|-------------|----------| -| Phase 1: Debounce | Medium | 1-2 hours | -| Phase 2: Polling | Medium | 1-2 hours | -| Phase 3: Reuse cleanup | Low | 30 min | -| Phase 4: Config | Low | 15 min | -| Phase 5: Integration | Low | 15 min | -| Testing | Medium | 1 hour | -| **Total** | - | **4-6 hours** | - -## Future Improvements (Out of Scope) - -1. **Configurable debounce duration** -2. **Per-library polling intervals** -3. **Event history/logging for debugging** -4. **Manual trigger for full sync** diff --git a/cover_image_serving_plan.md b/cover_image_serving_plan.md deleted file mode 100644 index 90e9b9a..0000000 --- a/cover_image_serving_plan.md +++ /dev/null @@ -1,1212 +0,0 @@ -# Cover & File Serving - Implementation Plan - -## Overview - -Fix file and cover image serving to support: - -1. Multiple library folders in docker compose (flexible mount points) -2. Keep files with books (no hardcoded paths) -3. Store relative paths in database (for both files AND covers) -4. Serve everything via authenticated API endpoints -5. Mobile app compatibility (same auth for all requests) - -## Architecture - -### Current Behavior - -- File path stored as absolute: `/app/uploads/Jane Austen/Pride and Prejudice/book.epub` -- Cover path stored as absolute: `/app/uploads/Jane Austen/Pride and Prejudice/cover.jpg` -- Frontend uses path directly - doesn't work (browser can't access container paths) -- No route serves `/app/uploads/*` - -### Target Behavior - -- File path stored as relative: `Jane Austen/Pride and Prejudice/book.epub` -- Cover path stored as relative: `Jane Austen/Pride and Prejudice/cover.jpg` -- Handler resolves relative path using library folder base path -- Authenticated static-style handler serves files: `/uploads/library-{id}/path/to/file` -- Backend resolves full URLs in API/SSR responses (one source of truth) -- Works with mobile apps, Kobo, KOReader devices via same endpoints - -### URL Format - -To handle same relative paths in different libraries, use: - -``` -/uploads/library-{library_id}/relative/path -``` - -- Requires JWT authentication (like API endpoints) -- Works for both covers and book files -- Single handler handles all file serving - -### Universal Path Resolution - -All handlers use the same `LibraryService.ResolveMediaPath()` function: - -- MediaHandler (downloads) -- OPDSHandler (device cover images) -- Future handlers - -This ensures one source of truth for path resolution. - ---- - -## Phase 1: Update Scanner to Store Relative Paths (Files AND Covers) - -### File: `internal/services/media_scanner.go` - -#### Change 1: Store relative file path - -**Location**: In `internal/services/media_scanner.go` - wherever `FilePath` is set in the database insert - -**Current code** (line 579): - -```go -FilePath: path, // path is absolute like /app/uploads/Author/Book/file.epub -``` - -**New code**: - -```go -FilePath: s.getRelativePath(path), -``` - -**Also update** line 617 for format file paths: - -```go -FilePath: pgtype.Text{String: s.getRelativePath(format.FilePath), Valid: true}, -``` - ---- - -#### Change 2: Store relative cover path - -**Location**: In `internal/services/media_scanner.go` - wherever `metadata.CoverPath` is set - -**Current code** (example at line 517): - -```go -if len(coverImage) > 0 && metadata.CoverPath == "" { - coverPath := path + ".cover.jpg" - if err := os.WriteFile(coverPath, coverImage, 0644); err == nil { - metadata.CoverPath = coverPath - } -} -``` - -**New code**: - -```go -if len(coverImage) > 0 && metadata.CoverPath == "" { - coverPath := path + ".cover.jpg" - if err := os.WriteFile(coverPath, coverImage, 0644); err == nil { - // Store relative path - derive from library folder base - metadata.CoverPath = s.getRelativePath(coverPath) - } -} -``` - -**All locations where metadata.CoverPath is set**: - -- Line 517 (main cover) -- Line 645 (sidecar cover) -- Line 651 (sidecar cover alternative) -- Line 1060 (main cover) -- Line 1067 (sidecar cover) - -#### Change 3: Add helper function - -**Add new function** in `internal/services/media_scanner.go`: - -```go -// getRelativePath converts absolute filesystem path to relative path -// using the library folder base path -func (s *MediaScanner) getRelativePath(absolutePath string) string { - // Get the base folder paths from scanner - for _, baseFolder := range s.folders { - // Check if path is within this base folder - if strings.HasPrefix(absolutePath, baseFolder) { - // Return relative path (without leading slash) - relPath := strings.TrimPrefix(absolutePath, baseFolder) - // Remove leading slash if present - relPath = strings.TrimPrefix(relPath, "/") - return relPath - } - } - // Fallback: if no match, return as-is (shouldn't happen) - return absolutePath -} -``` - -**Note**: This uses `s.folders` which is already populated in the scanner. - -#### Change 4: Update force rescan path handling - -**Location**: Around line 1472 (in the force rescan/update flow) - -Apply same `getRelativePath()` conversion when updating existing items. - ---- - -## Phase 2: Create Path Resolution Helper (Service Layer) - -### File: `internal/services/library_service.go` (or new file) - -Create a reusable function that resolves relative paths to absolute filesystem paths: - -```go -// ResolveMediaPath resolves a relative path to absolute filesystem path -// using the library's configured folder(s) -func (s *LibraryService) ResolveMediaPath(ctx context.Context, libraryID pgtype.UUID, relativePath string) (string, error) { - // Get library folders for this library - folders, err := s.db.GetLibraryFolders(ctx, libraryID) - if err != nil || len(folders) == 0 { - return "", fmt.Errorf("no library folders found for library") - } - - // Try each folder - find one where the relative path makes sense - for _, folder := range folders { - fullPath := filepath.Join(folder.FolderPath, relativePath) - if _, err := os.Stat(fullPath); err == nil { - return fullPath, nil - } - } - - // Fallback: use first folder (file might not exist yet during scan) - if len(folders) > 0 { - return filepath.Join(folders[0].FolderPath, relativePath), nil - } - - return "", fmt.Errorf("could not resolve path") -} -``` - ---- - -## Phase 3: Add URL Resolution Helper to MediaHandler - -### Strategy - -Use `LibraryService.ResolveMediaPath()` to resolve paths. Add a simple wrapper in the handler for convenience. - -### File: `internal/handlers/media.go` - -Add helper method that uses the service: - -```go -// getFullFilePath returns the absolute filesystem path for a media item -// Uses LibraryService for resolution (one source of truth) -func (mh *MediaHandler) getFullFilePath(ctx context.Context, libraryID pgtype.UUID, relativePath string) (string, error) { - if relativePath == "" { - return "", fmt.Errorf("no file path") - } - - // Check if already absolute (backward compatibility) - if filepath.IsAbs(relativePath) { - return relativePath, nil - } - - // Use service for resolution (one source of truth) - return mh.libraryService.ResolveMediaPath(ctx, libraryID, relativePath) -} -``` - -Note: The handler already has `libraryService` injected, so this just calls through to it. - ---- - -## Phase 4: Update Download Handler to Use Relative Paths - -### File: `internal/handlers/media.go` - -#### Modify DownloadBook function - -**Current code** (line 103-144): - -```go -func (h *MediaHandler) DownloadBook(c echo.Context) error { - // ... - mediaItem, err := h.db.GetMediaItem(c.Request().Context(), pgBookUUID) - if err != nil { - return c.JSON(http.StatusNotFound, map[string]string{"error": "book not found"}) - } - - if _, err := os.Stat(mediaItem.FilePath); os.IsNotExist(err) { - return c.JSON(http.StatusNotFound, map[string]string{"error": "book file not found on disk"}) - } - - file, err := os.Open(mediaItem.FilePath) - // ... -} -``` - -**New code**: - -```go -func (h *MediaHandler) DownloadBook(c echo.Context) error { - // ... - mediaItem, err := h.db.GetMediaItem(c.Request().Context(), pgBookUUID) - if err != nil { - return c.JSON(http.StatusNotFound, map[string]string{"error": "book not found"}) - } - - // Resolve relative path to absolute filesystem path - fullPath, err := h.getFullFilePath(c.Request().Context(), mediaItem.LibraryID, mediaItem.FilePath) - if err != nil { - return c.JSON(http.StatusNotFound, map[string]string{"error": "book file not found on disk"}) - } - - if _, err := os.Stat(fullPath); os.IsNotExist(err) { - return c.JSON(http.StatusNotFound, map[string]string{"error": "book file not found on disk"}) - } - - file, err := os.Open(fullPath) - // ... -} -``` - ---- - -## Phase 5: Create Authenticated File Serving Handler - -### File: `internal/handlers/media.go` - -Create a single handler that serves both covers and book files: - -```go -// ServeFile serves files (covers or books) via /uploads/library-{id}/path -// Requires JWT authentication -func (mh *MediaHandler) ServeFile(c echo.Context) error { - // URL format: /uploads/library-{libraryID}/{relativePath} - path := c.Param("*") // Gets everything after /uploads/library-{id}/ - - // Extract library ID from path - parts := strings.SplitN(path, "/", 2) - if len(parts) < 2 { - return c.JSON(http.StatusBadRequest, map[string]string{"error": "invalid path"}) - } - - libraryIDStr := strings.TrimPrefix(parts[0], "library-") - libraryUUID, err := uuid.Parse(libraryIDStr) - if err != nil { - return c.JSON(http.StatusBadRequest, map[string]string{"error": "invalid library ID"}) - } - - relativePath := parts[1] - - // Resolve using service - fullPath, err := mh.getFullFilePath(c.Request().Context(), pgtype.UUID{Bytes: libraryUUID, Valid: true}, relativePath) - if err != nil { - return c.JSON(http.StatusNotFound, map[string]string{"error": "file not found"}) - } - - // Check if file exists - if _, err := os.Stat(fullPath); os.IsNotExist(err) { - return c.JSON(http.StatusNotFound, map[string]string{"error": "file not found"}) - } - - // Determine content type - ext := strings.ToLower(filepath.Ext(fullPath)) - contentType := "application/octet-stream" - if ext == ".jpg" || ext == ".jpeg" { - contentType = "image/jpeg" - } else if ext == ".png" { - contentType = "image/png" - } else if ext == ".webp" { - contentType = "image/webp" - } else if ext == ".epub" { - contentType = "application/epub+zip" - } else if ext == ".pdf" { - contentType = "application/pdf" - } - - c.Response().Header().Set("Content-Type", contentType) - c.Response().Header().Set("Cache-Control", "public, max-age=86400") - return c.File(fullPath) -} -``` - -### File: `internal/router/media.go` - -**Location**: After existing media routes (NOT in the protected group) - -```go -// File serving - authenticated (registered on Echo to avoid /api prefix) -// Note: Must be registered LAST as it's a wildcard route -e.GET("/uploads/library-:id/*", createJWTMiddleware(cfg), cfg.MediaHandler.ServeFile) -``` - -**Important**: This route must be registered LAST because `/*` is a wildcard that matches everything. - ---- - -## Phase 6: Update OPDS Handler for Device Support - -### File: `internal/handlers/opds.go` - -#### Modify GetCoverImage function - -**Current code** (around line 477-549): - -```go -func (h *OPDSHandler) GetCoverImage(c echo.Context) error { - // ... - coverPath := mediaItem.CoverImagePath.String - - // Check if file exists - if _, err := os.Stat(coverPath); os.IsNotExist(err) { - return c.NoContent(http.StatusNoContent) - } - - // Open file - file, err := os.Open(coverPath) - // ... -} -``` - -**New code**: - -```go -func (h *OPDSHandler) GetCoverImage(c echo.Context) error { - // ... - coverPath := mediaItem.CoverImagePath.String - - // Resolve relative path using library service - fullPath, err := h.libraryService.ResolveMediaPath(c.Request().Context(), mediaItem.LibraryID, coverPath) - if err != nil { - return c.NoContent(http.StatusNoContent) - } - - // Check if file exists - if _, err := os.Stat(fullPath); os.IsNotExist(err) { - return c.NoContent(http.StatusNoContent) - } - - // Open file - file, err := os.Open(fullPath) - // ... -} -``` - -**Note**: OPDSHandler uses `ResolveMediaPath()` to resolve to **filesystem path** (not URL) because OPDS serves files directly from the local filesystem. This is different from API handlers which resolve to `/uploads/library-{id}/...` URLs. - -**If OPDSHandler doesn't have libraryService**, add it: - -```go -type OPDSHandler struct { - db *database.Queries - libraryService *services.LibraryService - conversionService interface {...} -} - -func NewOPDSHandler(db *database.Queries, libraryService *services.LibraryService, conversionService ...) *OPDSHandler { - return &OPDSHandler{ - db: db, - libraryService: libraryService, - conversionService: conversionService, - } -} -``` - ---- - -## Phase 7: Update ALL Handlers to Resolve URLs in API Responses - -Every handler that returns `cover_image_path` or `file_path` in API responses must resolve the relative path to a full URL before sending to the client. - -### URL Resolution Flow - -1. **Database stores**: Relative path (e.g., `Author/Book/cover.jpg`) -2. **Handler fetches**: Gets relative path from DB (as pgtype.Text) -3. **Handler resolves**: Calls `ResolveCoverURL()` → returns `/uploads/library-{id}/Author/Book/cover.jpg` -4. **Handler replaces**: Sets resolved URL string in response (not pgtype.Text) -5. **Frontend/mobile**: Uses URL directly (authenticated via JWT) - -### Step 1: Add URL Resolution Helper to MediaHandler - -**File**: `internal/handlers/media.go` - -Add these helper methods after the `NewMediaHandler` function (around line 100): - -```go -// ResolveCoverURL resolves a relative cover path to a full URL for API responses -func (mh *MediaHandler) ResolveCoverURL(libraryID pgtype.UUID, coverPath pgtype.Text) string { - if !coverPath.Valid || coverPath.String == "" { - return "" - } - - return mh.resolveMediaURL(libraryID, coverPath.String) -} - -// ResolveFileURL resolves a relative file path to a full URL for API responses -func (mh *MediaHandler) ResolveFileURL(libraryID pgtype.UUID, filePath pgtype.Text) string { - if !filePath.Valid || filePath.String == "" { - return "" - } - - return mh.resolveMediaURL(libraryID, filePath.String) -} - -// resolveMediaURL is the internal helper that does the actual resolution -func (mh *MediaHandler) resolveMediaURL(libraryID pgtype.UUID, relativePath string) string { - // Already a full URL? Return as-is - if strings.HasPrefix(relativePath, "/uploads/") { - return relativePath - } - - // Already absolute path? Convert to URL format (backward compatibility) - // Note: This loses library ID info, but existing data won't have it - if filepath.IsAbs(relativePath) { - return relativePath - } - - // Resolve relative path to URL format - libraryIDStr := libraryID.Bytes.String() - return fmt.Sprintf("/uploads/library-%s/%s", libraryIDStr, relativePath) -} -``` - -**Note**: Add `"strings"` and `"fmt"` to the imports if not already present. - ---- - -### Step 1b: Add libraryService to CollectionHandler - -**File**: `internal/handlers/collections.go` - -Add `libraryService` field to CollectionHandler struct (near line 10): - -```go -type CollectionHandler struct { - db *database.Queries - collectionService *services.CollectionService - libraryService *services.LibraryService // ADD THIS - connManager *wsync.ConnectionManager -} -``` - -Update constructor to accept and set libraryService: - -```go -func NewCollectionHandler(db *database.Queries, libraryService *services.LibraryService, connManager *wsync.ConnectionManager) *CollectionHandler { - return &CollectionHandler{ - db: db, - collectionService: services.NewCollectionService(db), - libraryService: libraryService, // ADD THIS - connManager: connManager, - } -} -``` - -**Update router** where CollectionHandler is instantiated (likely in router/collections.go or similar): - -```go -cfg.CollectionHandler, err = handlers.NewCollectionHandler(cfg.Queries, cfg.LibraryService, cfg.ConnManager) -``` - ---- - -### Step 2: Update collections.go - GetCollection handler (line ~193-201) - -**File**: `internal/handlers/collections.go` - -**Current code** (lines 193-201 in GetCollection function): - -```go -bookList := make([]BookInfo, 0, len(books)) -for _, book := range books { - bookList = append(bookList, BookInfo{ - MediaItemID: uuid.UUID(book.MediaItemID.Bytes).String(), - Title: book.Title, - Author: textToString(book.Author), - CoverImagePath: textToString(book.CoverImagePath), - }) -} -``` - -**New code**: - -```go -bookList := make([]BookInfo, 0, len(books)) -for _, book := range books { - bookList = append(bookList, BookInfo{ - MediaItemID: uuid.UUID(book.MediaItemID.Bytes).String(), - Title: book.Title, - Author: textToString(book.Author), - CoverImagePath: h.resolveCoverURL(book.LibraryID, book.CoverImagePath), - }) -} -``` - -**Add helper method** to CollectionHandler struct (near line 10): - -```go -// resolveCoverURL resolves a relative cover path to a full URL -func (h *CollectionHandler) resolveCoverURL(libraryID pgtype.UUID, coverPath pgtype.Text) string { - if !coverPath.Valid || coverPath.String == "" { - return "" - } - - // Already a full URL? Return as-is - if strings.HasPrefix(coverPath.String, "/uploads/") { - return coverPath.String - } - - // Already absolute path? Return as-is (backward compatibility) - if filepath.IsAbs(coverPath.String) { - return coverPath.String - } - - // Resolve relative path to URL format - libraryIDStr := libraryID.Bytes.String() - return fmt.Sprintf("/uploads/library-%s/%s", libraryIDStr, coverPath.String) -} -``` - -**Add imports** if not present: `"fmt"`, `"path/filepath"`, `"strings"` - ---- - -### Step 3: Update collections.go - TestRules/BookMatch (lines 620-641) - -**File**: `internal/handlers/collections.go` - -**Current code** (lines 620-641 in TestRules function): - -```go -var matches []BookMatch -for _, item := range mediaItems { - matchReason := h.checkRulesAgainstBook(item, req.Rules) - if matchReason != "" { - coverPath := "" - if item.CoverImagePath.Valid { - coverPath = item.CoverImagePath.String - } - author := "" - if item.Author.Valid { - author = item.Author.String - } - - matches = append(matches, BookMatch{ - MediaItemID: uuid.UUID(item.ID.Bytes).String(), - Title: item.Title, - Author: author, - CoverImagePath: coverPath, - MatchReason: matchReason, - }) - } -} -``` - -**New code**: - -```go -var matches []BookMatch -for _, item := range mediaItems { - matchReason := h.checkRulesAgainstBook(item, req.Rules) - if matchReason != "" { - author := "" - if item.Author.Valid { - author = item.Author.String - } - - matches = append(matches, BookMatch{ - MediaItemID: uuid.UUID(item.ID.Bytes).String(), - Title: item.Title, - Author: author, - CoverImagePath: h.resolveCoverURL(item.LibraryID, item.CoverImagePath), - MatchReason: matchReason, - }) - } -} -``` - ---- - -### Step 4: Update collections.go - PreviewCollection (lines 910-919) and mediaItemsToListMediaItemsRow helper (line 935) - -**File**: `internal/handlers/collections.go` - -**Location 1 - PreviewCollection function** (lines 910-919): - -**Current code**: - -```go -bookCards := make([]BookInfo, len(matchedItems)) -for i, item := range matchedItems { - itemUUID, _ := uuid.FromBytes(item.ID.Bytes[0:16]) - bookCards[i] = BookInfo{ - MediaItemID: itemUUID.String(), - Title: item.Title, - Author: textToString(item.Author), - CoverImagePath: textToString(item.CoverImagePath), - } -} -``` - -**New code**: - -```go -bookCards := make([]BookInfo, len(matchedItems)) -for i, item := range matchedItems { - itemUUID, _ := uuid.FromBytes(item.ID.Bytes[0:16]) - bookCards[i] = BookInfo{ - MediaItemID: itemUUID.String(), - Title: item.Title, - Author: textToString(item.Author), - CoverImagePath: h.resolveCoverURL(item.LibraryID, item.CoverImagePath), - } -} -``` - -**Location 2 - mediaItemsToListMediaItemsRow helper** (line 935): - -**Current code**: - -```go -func mediaItemsToListMediaItemsRow(item database.MediaItems) database.ListMediaItemsRow { - return database.ListMediaItemsRow{ - // ... - CoverImagePath: item.CoverImagePath, - // ... - } -} -``` - -**New code**: - -```go -// NOTE: This helper function doesn't have access to libraryID -// Consider refactoring to pass libraryID or handle URL resolution at call site -``` - -**Add helper method** for file URL resolution: - -```go -// resolveFileURL resolves a relative file path to a full URL -func (h *CollectionHandler) resolveFileURL(libraryID pgtype.UUID, filePath pgtype.Text) string { - if !filePath.Valid || filePath.String == "" { - return "" - } - - if strings.HasPrefix(filePath.String, "/uploads/") { - return filePath.String - } - - if filepath.IsAbs(filePath.String) { - return filePath.String - } - - libraryIDStr := libraryID.Bytes.String() - return fmt.Sprintf("/uploads/library-%s/%s", libraryIDStr, filePath.String) -} -``` - ---- - -### Step 5: Update progress.go - two locations (lines 286-289 and 357-360) - -**File**: `internal/handlers/progress.go` - -First, add helper methods to Handler struct (defined in `commonhandlers.go`, used by progress.go): - -```go -// resolveCoverURL resolves a relative cover path to a full URL -func (h *Handler) resolveCoverURL(libraryID pgtype.UUID, coverPath pgtype.Text) string { - if !coverPath.Valid || coverPath.String == "" { - return "" - } - - if strings.HasPrefix(coverPath.String, "/uploads/") { - return coverPath.String - } - - if filepath.IsAbs(coverPath.String) { - return coverPath.String - } - - libraryIDStr := libraryID.Bytes.String() - return fmt.Sprintf("/uploads/library-%s/%s", libraryIDStr, coverPath.String) -} -``` - -**Location 1 - GetAllProgress function** (lines 286-289): - -**Current code**: - -```go -coverPath := "" -if mediaItem.CoverImagePath.Valid { - coverPath = mediaItem.CoverImagePath.String -} -``` - -**New code** (remove the manual resolution, use helper): - -```go -coverPath := h.resolveCoverURL(mediaItem.LibraryID, mediaItem.CoverImagePath) -``` - -**Location 2 - GetAllProgressData function** (lines 357-360): - -**Current code**: - -```go -coverPath := "" -if mediaItem.CoverImagePath.Valid { - coverPath = mediaItem.CoverImagePath.String -} -``` - -**New code**: - -```go -coverPath := h.resolveCoverURL(mediaItem.LibraryID, mediaItem.CoverImagePath) -``` - ---- - -### Step 6: Update media.go - GetMediaItem and ListMediaItems - -**File**: `internal/handlers/media.go` - -Add to imports: - -```go -"bookhoard/internal/utils" -``` - -**GetMediaItem** - Find where it returns the response (around line 770): - -**Current code**: - -```go -return c.JSON(http.StatusOK, item) -``` - -**New code**: - -```go -return c.JSON(http.StatusOK, map[string]interface{}{ - "id": uuid.UUID(item.ID.Bytes).String(), - "library_id": uuid.UUID(item.LibraryID.Bytes).String(), - "title": item.Title, - "author": textToString(item.Author), - "cover_image_path": utils.ResolveMediaURL(item.LibraryID, item.CoverImagePath), - "file_path": utils.ResolveMediaURL(item.LibraryID, item.FilePath), - "file_size": item.FileSize, - "mime_type": textToString(item.MimeType), - // ... add other fields as needed -}) -``` - -**ListMediaItems** - Find where it returns items (around line 609): - -Wrap each item in the response with resolved URLs. The exact implementation depends on how ListMediaItems currently returns data - you may need to build a custom response map similar to GetMediaItem. - -**Note**: Unlike collections.go and progress.go where we added helper methods to the handler, here we use the utils package function directly since we've consolidated URL resolution into utils. - ---- - -### Step 7: Fix Frontend /covers/ Prefix - -**File**: `web/src/bookshelf.ts` - -**Current code** (line 49-50): - -```typescript -${book.cover_image_path ? - `${book.title}` : -``` - -**New code**: - -```typescript -${book.cover_image_path ? - `${book.title}` : -``` - -The backend now returns full URLs like `/uploads/library-{id}/path/to/cover.jpg`, so no prefix is needed. - ---- - -### Summary of Changes for Phase 7 - -| File | Changes | -| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -| `internal/utils/mediaurl.go` | Create with `ResolveMediaURL()` function for URL resolution (one source of truth) | -| `internal/handlers/media.go` | Update GetMediaItem and ListMediaItems to use `utils.ResolveMediaURL()` for resolved URLs in responses | -| `internal/handlers/collections.go` | Use `utils.ResolveMediaURL()` in GetCollection, TestRules, PreviewCollection; update lines 193-201, 620-641, 910-919 | -| `internal/handlers/progress.go` | Use `utils.ResolveMediaURL()` in GetAllProgress; update lines 286-289, 357-360 | -| `web/src/bookshelf.ts` | Remove `/covers/` prefix from cover image URL | - ---- - -### Additional Plan Updates Needed - -| Item | Status | -| --------------------------------------------------- | ------------------------------------------------------------------- | -| Add `mi.library_id` to GetCollectionItems SQL query | Needs to be done before implementing Step 2 in collections.go | -| Create `internal/utils/mediaurl.go` | Needs to be created before implementing URL resolution | -| Update callers to use utils package | Replace h.resolveCoverURL/resolveFileURL with utils.ResolveMediaURL | - -## Phase 8: Backward Compatibility - -Handle existing absolute paths in database: - -### Option A: Migration (One-time) - -Create a script to convert existing absolute paths to relative paths using known library folder paths. - -### Option B: Runtime Resolution (No migration) - -Add backward compatibility in handlers: - -```go -func (mh *MediaHandler) getFullFilePath(ctx context.Context, libraryID pgtype.UUID, relativePath string) (string, error) { - // Already absolute? Use as-is (backward compatibility) - if filepath.IsAbs(relativePath) { - return relativePath, nil - } - - // Otherwise resolve as relative path - return mh.libraryService.ResolveMediaPath(ctx, libraryID, relativePath) -} -``` - -**Recommended**: Option B - no database migration needed, handles both old and new data. - ---- - -## Phase 9: Tests - -### Unit Tests - -#### File: `internal/handlers/media_test.go` - -```go -// TestGetCoverImage_ValidItem tests successful cover image retrieval -func TestGetCoverImage_ValidItem(t *testing.T) { - // Setup test server with mock database - // Create a test cover image file - // Call GetCoverImage - // Verify response has correct Content-Type and status code -} - -// TestGetCoverImage_NotFound tests 404 for non-existent media item -func TestGetCoverImage_NotFound(t *testing.T) { - // Call with invalid UUID - // Verify 404 response -} - -// TestGetCoverImage_NoCover tests 404 when media item has no cover -func TestGetCoverImage_NoCover(t *testing.T) { - // Create media item with empty cover_image_path - // Verify 404 response -} - -// TestGetFullFilePath_RelativePath tests relative path resolution -func TestGetFullFilePath_RelativePath(t *testing.T) { - // Setup: Create library with folder /app/uploads - // Media item with file_path: "Author/Book/book.epub" - // Call getFullFilePath - // Verify returns: "/app/uploads/Author/Book/book.epub" -} - -// TestGetFullFilePath_AbsolutePath tests backward compatibility -func TestGetFullFilePath_AbsolutePath(t *testing.T) { - // Media item with absolute file_path - // Verify returns same path -} -``` - -### Scanner Tests - -#### File: `internal/services/media_scanner_test.go` - -```go -// TestGetRelativePath tests path conversion -func TestGetRelativePath(t *testing.T) { - scanner := &MediaScanner{ - folders: []string{"/app/uploads", "/var/books"}, - } - - tests := []struct { - absolute string - expected string - }{ - {"/app/uploads/Author/Book/epub", "Author/Book/epub"}, - {"/var/books/manga/Naruto/vol1", "manga/Naruto/vol1"}, - {"/other/path/file.pdf", "/other/path/file.pdf"}, // fallback - } - - for _, tt := range tests { - result := scanner.getRelativePath(tt.absolute) - assert.Equal(t, tt.expected, result) - } -} -``` - -### Integration Tests - -#### File: `cmd/server/tests/cover_file_serving_test.go` - -```go -package main - -import ( - "bytes" - "encoding/json" - "fmt" - "io" - "net/http" - "os" - "path/filepath" - "testing" - "time" - - "github.com/stretchr/testify/require" - "github.com/stretchr/testify/suite" -) -``` - -Note: The integration tests use `setupTestServer(s.T())` from `cmd/server/tests/test_helpers.go` as per PROJECT_GUIDELINES.md requirements. - -### Bruno API Tests - -Create new Bruno test files for the new endpoints: - -#### File: `bruno/media-items/Get Cover Image.yml` - -```yaml -info: - name: Get Cover Image - type: http - seq: 1 -http: - method: GET - url: "{{base_url}}/uploads/library-{{library_id}}/path/to/cover.jpg" - auth: none - -docs: |- - ## Get Cover Image - - Retrieve the cover image for a media item via authenticated static-style URL. - Uses JWT authentication (same as API endpoints). - - **Method:** GET - - **Endpoint:** /uploads/library-{id}/{path} - - **Authentication:** Bearer token required - - **Response:** Binary image data (JPEG, PNG, etc.) - - **Status Codes:** - - 200: Success - returns image - - 401: Unauthorized (missing/invalid JWT) - - 404: File not found - - **Note:** The actual path would come from the API response which provides - the resolved URL. This test is a template showing the URL format. - -vars: - library_id: "" -``` - -#### File: `bruno/media-items/EPUB Download.yml` (Update existing) - -Update the existing file to note that downloads now work through the same `/uploads/library-{id}/` endpoint: - -```yaml -info: - name: Download Media Item - type: http - seq: 1 -http: - method: GET - url: "{{base_url}}/uploads/library-{{library_id}}/path/to/book.epub" - auth: none - -docs: |- - ## Download Media Item - - Download a media item file (EPUB, PDF, CBZ, etc.) via authenticated static-style URL. - Uses JWT authentication (same as API endpoints). - - **Method:** GET - - **Endpoint:** /uploads/library-{id}/{path} - - **Authentication:** Bearer token required - - **Path Resolution:** The handler resolves the relative file path stored in the - database against the library's configured folder(s) to locate the actual file. - - **Backward Compatibility:** Supports both relative paths (new) and absolute - paths (legacy data). - - **Response:** Binary file data with appropriate Content-Type header - - **Status Codes:** - - 200: Success - returns file - - 401: Unauthorized (missing/invalid JWT) - - 404: File not found - - **Note:** The actual path would come from the API response which provides - the resolved URL. This test shows the URL format. - -vars: - library_id: "" -``` - ---- - -## Phase 9: Documentation - -### File: `docs/developer/api/media-items/get_cover_image.md` - -````markdown ---- -title: Get Cover Image -description: Retrieve the cover image for a media item ---- - -# Get Cover Image - -Retrieve the cover image for a media item. - -## Endpoint - -`GET /api/covers/:id` - -## Path Parameters - -| Parameter | Type | Description | -| --------- | ------ | ------------------------ | -| id | string | The media item ID (UUID) | - -## Headers - -| Header | Required | Description | -| ------------- | -------- | ------------ | -| Authorization | Yes | Bearer token | - -## Response - -- **200 OK**: Cover image returned - - Content-Type: `image/jpeg`, `image/png`, etc. - - Cache-Control: `public, max-age=86400` - -- **400 Bad Request**: Invalid media item ID - -- **404 Not Found**: - - Media item not found - - No cover image configured - - Cover image file not found on disk - -## Example - -```bash -curl -H "Authorization: Bearer YOUR_TOKEN" \ - http://localhost:8765/api/covers/550e8400-e29b-41d4-a716-446655440000 \ - --output cover.jpg -``` -```` - -## Notes - -- Cover images are stored relative to their library folder -- The API resolves the full path using the library's configured folder(s) -- Supports backward compatibility with existing absolute paths -- Images are cached for 24 hours by clients -- All endpoints require authentication (JWT) - -### File: `docs/developer/api/media-items/download_book.md` - -Update existing documentation to note: - -- File paths are stored relative to library folders -- Handler resolves path at request time -- Backward compatible with existing absolute paths - ---- - -## Summary of Changes - -| Phase | File | Change | -| -------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -| Refactor | `internal/handlers/commonhandlers.go` | Move Handler struct from scanner.go (kitchen sink handler for progress/book matching) | -| Refactor | `internal/routers/*.go` | Update NewHandler instantiation if needed | -| 1 | `internal/services/media_scanner.go` | Add `getRelativePath()` function; use for both file_path and cover_path | -| 2 | `internal/services/library_service.go` | Add `ResolveMediaPath()` function (one source of truth) | -| 3 | `internal/handlers/media.go` | Add `getFullFilePath()` helper that calls service | -| 4 | `internal/handlers/media.go` | Modify `DownloadBook` to use `getFullFilePath()` | -| 5 | `internal/handlers/media.go` | Add `ServeFile()` handler for authenticated static-style routes | -| 5 | `internal/router/media.go` | Add route `GET /uploads/library-:id/*` on Echo (not protected group) | -| 6 | `internal/handlers/opds.go` | Add `libraryService` to struct; update `GetCoverImage` to use service | -| 7 | `internal/handlers/collections.go` | Add `libraryService` to struct/constructor; resolve cover paths to URLs in API responses | -| 7 | `internal/router/*.go` | Update CollectionHandler instantiation to pass LibraryService | -| 7 | `internal/handlers/commonhandlers.go` (Handler struct, used by progress.go) | Resolve cover paths to URLs in API responses | -| 7 | `internal/handlers/media.go` | Resolve file paths to URLs in API responses | -| 7 | `web/src/bookshelf.ts` | Remove `/covers/` prefix (use resolved URL directly) | -| 8 | Runtime resolution | Handles both absolute (old) and relative (new) paths | -| 9 | `internal/handlers/media_test.go` | Add unit tests for path resolution | -| 9 | `internal/services/media_scanner_test.go` | Add unit tests for `getRelativePath()` | -| 9 | `cmd/server/tests/cover_file_serving_test.go` | Add integration tests using test_helpers | -| 10 | `bruno/media-items/Get Cover Image.yml` | Add Bruno API test | -| 10 | `bruno/media-items/EPUB Download.yml` | Update to document relative path handling | -| 10 | `docs/developer/api/media-items/` | Update API documentation | - ---- - -## Verification Steps - -After implementation: - -1. **Test new scan**: Add a new book with cover, verify: - - Database `file_path` is relative (e.g., `Author/Book/book.epub`) - - Database `cover_image_path` is relative (e.g., `Author/Book/cover.jpg`) - - GET `/uploads/library-{id}/Author/Book/cover.jpg` returns the image - - GET `/api/media-items/:id/download` returns the file - -2. **Test existing data**: For items with absolute paths: - - Downloads still work (backward compatibility) - - Cover images still work (backward compatibility) - -3. **Test multiple mount points**: - - Library A with folder `/app/epubs` - - Library B with folder `/var/manga` - - Books in each resolve correctly via their library ID - -4. **Test frontend**: - - Dashboard shows cover images (SSR - initial load) - - Library switch works (dynamic - uses resolved URLs) - - Bookshelf shows cover images - - Downloads work - -5. **Test mobile app** (future): - - Same JWT auth works for files and covers - - `/uploads/library-{id}/...` URLs work - -6. **Test device integration**: - - Kobo devices can fetch cover images via OPDS - - KOReader sync continues to work - ---- - -## Flexibility for Users - -Users can configure any mount point in docker-compose: - -```yaml -services: - bookhoard: - volumes: - - ./epubs:/app/epubs # ebooks - - ./manga:/var/manga # manga - - ./comics:/media/comics # comics -``` - -The system stores relative paths, so it works with any configuration. diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 06b269f..37b02d4 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -66,9 +66,8 @@ bookhoard/ **Services** (`internal/services/`): - `library_service.go` - Library operations -- `media_scanner.go` - File scanning & metadata extraction +- `media_scanner.go` - File scanning, metadata extraction, and real-time file watching - `worker.go` - Job queue worker pool -- `scheduler.go` - Scheduled task manager - `collection_service.go` - Collection rules processing - `conversion_service.go` - EPUB→KEPUB conversion - `book_matching.go` - Book matching algorithms diff --git a/docs/developer/api/system/settings.md b/docs/developer/api/system/settings.md index 07b2e8c..13e24a7 100644 --- a/docs/developer/api/system/settings.md +++ b/docs/developer/api/system/settings.md @@ -38,7 +38,7 @@ Retrieve the current system-wide scan settings. **Fields**: -- `scan_poll_interval_seconds` (integer): How often to scan all libraries in minutes (15-1440) +- `scan_poll_interval_seconds` (integer): How often to poll for file changes in seconds (1-3600) - `auto_scan_enabled` (boolean): Whether auto-scanning is enabled system-wide **Example**: @@ -69,9 +69,9 @@ Update the system-wide scan settings. **Fields**: -- `scan_poll_interval_seconds` (integer, required): How often to scan all libraries in minutes - - Minimum: 15 (15 minutes) - - Maximum: 1440 (24 hours) +- `scan_poll_interval_seconds` (integer, required): How often to poll for file changes in seconds + - Minimum: 1 (1 second) + - Maximum: 3600 (1 hour) - Default: 60 - `auto_scan_enabled` (boolean, required): Whether auto-scanning is enabled system-wide - Default: true @@ -104,7 +104,7 @@ Update the system-wide scan settings. **Validation Rules**: -- `scan_poll_interval_seconds` must be between 15 and 1440 minutes +- `scan_poll_interval_seconds` must be between 1 and 3600 seconds (1 second to 1 hour) - Both fields are required **Example**: @@ -123,26 +123,31 @@ curl -X PUT https://bookhoard.example.com/api/libraries/scan-settings \ ## Behavior -### Scan Frequency +### Poll Interval -The `scan_poll_interval_seconds` setting determines how often the system will automatically scan all libraries for new media files. The scheduler will trigger scans for all libraries at the configured interval. +The `scan_poll_interval_seconds` setting determines how often the system will poll library folders for file changes as a fallback to real-time file watching. **Constraints**: -- Minimum: 15 minutes (to prevent excessive scanning) -- Maximum: 1440 minutes (24 hours) -- Default: 60 minutes (1 hour) +- Minimum: 1 second +- Maximum: 3600 seconds (1 hour) +- Default: 60 seconds ### Auto-Scan Toggle The `auto_scan_enabled` setting acts as a master switch for automatic scanning: -- When `true`: All libraries will be scanned automatically at the configured interval -- When `false`: No automatic scans will occur (manual scans still available) +- When `true`: File watching and polling fallback are active for all libraries +- When `false`: No automatic file monitoring occurs (manual scans still available) -### System-Wide Scope +### File Watching System -These settings apply to **all libraries** in the system. Individual users can no longer configure per-user scan settings. This ensures consistent scanning behavior across the entire Bookhoard instance. +The scan settings control the file watching system which consists of: + +1. **Real-time file watching**: Uses fsnotify to detect file changes immediately +2. **Polling fallback**: If file watching fails or is unavailable, polls folders at the configured interval + +The system applies these settings to all configured libraries automatically on startup. --- @@ -167,16 +172,18 @@ These settings apply to **all libraries** in the system. Individual users can no ## Migration Notes -This API replaces the previous per-user scan settings system. The following changes were made: +This API has been updated to use a new polling-based scanning system. The following changes were made: -- **Removed**: Per-user scan settings (previously in users table) -- **Added**: System-wide scan settings (now in system_settings table) -- **Changed**: Access control from user-specific to admin-only -- **Preserved**: Endpoint paths remain the same for backward compatibility +- **Changed**: `scan_frequency_minutes` renamed to `scan_poll_interval_seconds` +- **Changed**: Unit changed from minutes to seconds (15-1440 minutes → 1-3600 seconds) +- **Removed**: Old scheduler-based scanning system +- **Added**: Real-time file watching with polling fallback +- **Preserved**: API endpoint paths remain the same -The migration ensures that: +The new system ensures that: -1. All libraries scan at the same frequency -2. Only administrators can modify scan settings -3. The API endpoints remain unchanged for existing clients -4. The scheduler uses system-wide settings instead of user-specific settings +1. File changes are detected in real-time when possible (via fsnotify) +2. Polling fallback catches missed events at the configured interval +3. Settings apply to all libraries system-wide +4. Only administrators can modify scan settings +5. The `auto_scan_enabled` setting controls both file watching and polling