From 0c1a55d6e3e04d6d61c9a385c8274e923a6f89ae Mon Sep 17 00:00:00 2001 From: John O'Keefe Date: Sat, 28 Mar 2026 00:46:20 -0400 Subject: [PATCH] chore: remove obsolete documentation and API collection files Cleanup of project files: - Remove CALIBRE_OPF_IMPLEMENTATION.md (obsolete documentation) - Remove Bruno API collection files for field value searches: - Field Values Search - Authors.yml - Field Values Search - Genres.yml - Field Values Search - Languages.yml - Field Values Search - Series.yml These files are no longer needed as the functionality has been implemented and the API has evolved. --- CALIBRE_OPF_IMPLEMENTATION.md | 1113 ----------------- .../search/Field Values Search - Authors.yml | 89 -- .../search/Field Values Search - Genres.yml | 89 -- .../Field Values Search - Languages.yml | 89 -- .../search/Field Values Search - Series.yml | 89 -- 5 files changed, 1469 deletions(-) delete mode 100644 CALIBRE_OPF_IMPLEMENTATION.md delete mode 100644 bruno/media-items/search/Field Values Search - Authors.yml delete mode 100644 bruno/media-items/search/Field Values Search - Genres.yml delete mode 100644 bruno/media-items/search/Field Values Search - Languages.yml delete mode 100644 bruno/media-items/search/Field Values Search - Series.yml diff --git a/CALIBRE_OPF_IMPLEMENTATION.md b/CALIBRE_OPF_IMPLEMENTATION.md deleted file mode 100644 index 50b3680..0000000 --- a/CALIBRE_OPF_IMPLEMENTATION.md +++ /dev/null @@ -1,1113 +0,0 @@ -# Calibre metadata.opf Support - Implementation Plan - -## Overview - -This document outlines the implementation of **Calibre metadata.opf sidecar file support** for Bookhoard's media scanner. The scanner will use a **sidecar-first approach** - if a Calibre `metadata.opf` file exists alongside a book file, it will take precedence over embedded metadata. - -## Motivation - -Calibre is the most popular ebook management software, with millions of users. When users manage their libraries in Calibre: - -1. **They curate metadata** - Adding series information, tags, ratings, custom covers -2. **They fetch from online sources** - Google Books, Amazon, Goodreads (better than publisher metadata) -3. **They convert/edit books** - The sidecar reflects the current, correct state -4. **They organize in folders** - Often using Calibre's folder structure conventions - -By supporting Calibre `metadata.opf` sidecar files, Bookhoard can: -- Respect users' curation work -- Import richer, more accurate metadata -- Provide seamless integration with existing Calibre libraries -- Support all file types (EPUB, PDF, comics, etc.) that Calibre manages - -## Requirements - -### Functional Requirements - -- [ ] Scanner detects `metadata.opf` files in the same directory as book files -- [ ] Parses Dublin Core metadata (`dc:*` namespace) and Calibre-specific fields (`meta name="calibre:*"`) -- [ ] Uses sidecar metadata **before** embedded metadata (sidecar-first approach) -- [ ] Works for all library types (ebook, audiobook, comic, etc.) -- [ ] Works for all file types (EPUB, PDF, CBZ, CBR, MP3, M4B, etc.) -- [ ] Handles missing or malformed sidecar files gracefully -- [ ] Maps all OPF fields to existing `media_items` database schema -- [ ] Maintains backward compatibility (no sidecar = existing behavior) - -### Non-Functional Requirements - -- [ ] No breaking changes to existing scanner functionality -- [ ] No database schema changes required (all fields already exist) -- [ ] Follows existing code patterns and style -- [ ] Comprehensive test coverage -- [ ] Performance: Minimal impact on scan performance -- [ ] Error handling: Graceful degradation on malformed XML - -## Database Schema Mapping - -All required fields **already exist** in the `media_items` table. No schema changes needed. - -### Dublin Core Fields (dc: namespace) - -| OPF Field | Database Column | Type | Notes | -|-----------|-----------------|------|-------| -| `dc:title` | `title` | VARCHAR(255) | Required | -| `dc:creator` | `author` | VARCHAR(255) | First author only | -| `dc:subject` | `tags` | TEXT[] | **Tags/keywords** (multiple) | -| `dc:description` | `description` | TEXT | Book summary | -| `dc:publisher` | `publisher` | VARCHAR(255) | Publisher name | -| `dc:date` | `date_published` | DATE | Publication date | -| `dc:language` | `language` | VARCHAR(10) | ISO 639-1 code (en, es, fr) | -| `dc:identifier` (ISBN) | `isbn` | VARCHAR(13) | Check `opf:scheme="ISBN"` | -| `dc:identifier` (ASIN) | `asin` | VARCHAR(20) | Check `opf:scheme="ASIN"` | -| `dc:contributor` | `contributors` | TEXT[] | Multiple contributors | -| `dc:rights` | *(not stored)* | - | Copyright info (ignored) | -| `dc:source` | *(not stored)* | - | Source publication (ignored) | - -### Calibre-Specific Fields (meta name="calibre:*") - -| OPF Field | Database Column | Type | Notes | -|-----------|-----------------|------|-------| -| `calibre:series` | `series` | VARCHAR(255) | Series name | -| `calibre:series_index` | `series_number` | INTEGER | Position in series | -| `calibre:rating` | *(use media_ratings)* | INTEGER | 0-5 scale, stored per-user | -| `calibre:title_sort` | *(not stored)* | - | Sortable title (ignored) | -| `calibre:author_sort` | *(not stored)* | - | Sortable author (ignored) | -| `calibre:timestamp` | `created_at` | TIMESTAMP | When added to Calibre | - -### Special Cases - -1. **Multiple authors**: Store first author in `author`, suggest adding `authors[]` array in future -2. **Multiple identifiers**: Extract ISBN and ASIN if present, ignore others -3. **Calibre rating**: Not imported (ratings are per-user in Bookhoard) -4. **Sort fields**: Ignored (Bookhoard has its own sorting logic) -5. **Timestamp**: Use for `created_at` if more recent than file modification time - -## OPF Format Reference - -### Standard Dublin Core Elements - -```xml -Book Title -Author Name -Fantasy -Adventure -Book summary... -Publisher Name -2024-01-15 -en -978-0-123456-78-9 -B08XXXXX -Illustrator Name -``` - -### Calibre-Specific Meta Tags - -```xml - - - - - -``` - -### Example Complete metadata.opf - -```xml - - - - The Fellowship of the Ring - J.R.R. Tolkien - 978-0-618-00222-0 - 12345678-1234-1234-1234-123456789012 - en - 1954-07-29 - HarperCollins - In a sleepy village in the Shire... - - - Fantasy - Adventure - Classics - - - Alan Lee (illustrator) - - - - - - - - -``` - -## Implementation Design - -### Architecture - -``` -Scanner Pipeline (EXISTING) -↓ -1. Detect book file -2. Check if already in database -3. Extract metadata from file (EPUB/PDF/Comic) -4. Extract folder structure metadata -5. Normalize and store -``` - -**Simplified Approach**: Modify existing `extractMetadata()` to check for Calibre sidecar first - -``` -Scanner Pipeline (NEW) -↓ -1. Detect book file -2. Check if already in database -3. Extract metadata (MODIFIED FUNCTION) - ├── Check for Calibre metadata.opf sidecar ← NEW - │ └─ If found → use sidecar metadata - └── Fallback to embedded metadata (existing) - ├── EPUB → extractEPUBMetadata() - ├── PDF → extractPDFMetadata() - └── Comic → extractComicMetadata() -4. Extract folder structure metadata (existing) -5. Normalize and store (existing) -``` - -### Code Changes - -**SIMPLIFIED APPROACH** (thanks to user feedback): Modify existing `extractMetadata()` function instead of creating wrapper. - -#### File: `internal/services/media_scanner.go` - ---- - -## 📝 STEP 1: Add CalibreOPFMetadata Struct - -**Location**: After existing struct definitions (around line 100-150) - -**ADD THIS CODE**: - -```go -// CalibreOPFMetadata represents intermediate parsed metadata from Calibre metadata.opf files -type CalibreOPFMetadata struct { - Title string - Authors []string - Tags []string - Description string - Publisher string - PublishDate *time.Time - Language string - ISBN string - ASIN string - UUID string - Contributors []string - Series string - SeriesIndex *float64 - Rating *int32 - Timestamp *time.Time -} -``` - ---- - -## 📝 STEP 2: Add parseCalibreMetadataOPF() Function - -**Location**: Add after `extractEPUBMetadata()` function (around line 830) - -**ADD THIS COMPLETE FUNCTION**: - -```go -// parseCalibreMetadataOPF parses a Calibre metadata.opf file and extracts metadata -func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error) { - // Open file - file, err := os.Open(opfPath) - if err != nil { - return nil, fmt.Errorf("failed to open metadata.opf: %v", err) - } - defer file.Close() - - // Define XML structure for parsing - var opf struct { - XMLName xml.Name `xml:"package"` - Metadata struct { - XMLName xml.Name `xml:"metadata"` - Titles []string `xml:"dc:title"` - Creators []string `xml:"dc:creator"` - Subjects []string `xml:"dc:subject"` - Desc []string `xml:"dc:description"` - Publisher []string `xml:"dc:publisher"` - Dates []string `xml:"dc:date"` - Language []string `xml:"dc:language"` - Identifiers []struct { - Scheme string `xml:"opf:scheme,attr"` - Value string `xml:",chardata"` - } `xml:"dc:identifier"` - Contributors []string `xml:"dc:contributor"` - CalibreSeries []struct { - Name string `xml:"content,attr"` - } `xml:"meta[name='calibre:series']"` - CalibreSeriesIndex []struct { - Value string `xml:"content,attr"` - } `xml:"meta[name='calibre:series_index']"` - } `xml:"metadata"` - } - - // Parse XML - if err := xml.NewDecoder(file).Decode(&opf); err != nil { - return nil, fmt.Errorf("failed to parse metadata.opf XML: %v", err) - } - - // Map to MediaMetadata struct - metadata := &MediaMetadata{} - - // Title (required) - if len(opf.Metadata.Titles) > 0 { - metadata.Title = opf.Metadata.Titles[0] - } - - // Author (first creator) - if len(opf.Metadata.Creators) > 0 { - metadata.Author = opf.Metadata.Creators[0] - } - - // Tags (all subjects) - if len(opf.Metadata.Subjects) > 0 { - metadata.Tags = utils.NormalizeTags(opf.Metadata.Subjects) - } - - // Description - if len(opf.Metadata.Desc) > 0 { - metadata.Description = opf.Metadata.Desc[0] - } - - // Publisher - if len(opf.Metadata.Publisher) > 0 { - metadata.Publisher = opf.Metadata.Publisher[0] - } - - // Publish date - if len(opf.Metadata.Dates) > 0 { - if date, err := time.Parse("2006-01-02", opf.Metadata.Dates[0]); err == nil { - metadata.PublishDate = date - } else { - // Try alternative date formats - if date, err := time.Parse("2006", opf.Metadata.Dates[0]); err == nil { - metadata.PublishDate = date - } - } - } - - // Language - if len(opf.Metadata.Language) > 0 { - // Note: MediaMetadata doesn't have Language field yet - // Future enhancement: add Language field - // For now, we'll skip this - } - - // Identifiers (ISBN, ASIN) - for _, id := range opf.Metadata.Identifiers { - switch strings.ToUpper(id.Scheme) { - case "ISBN": - metadata.ISBN = utils.NormalizeISBNSafe(id.Value) - case "ASIN": - metadata.ASIN = id.Value - case "UUID", "CALIBRE": - // Store UUID in hash info, not metadata - // Will be extracted by extractHashInfo() - } - } - - // Contributors - if len(opf.Metadata.Contributors) > 0 { - metadata.Contributors = utils.NormalizeContributors(opf.Metadata.Contributors) - } - - // Series - if len(opf.Metadata.CalibreSeries) > 0 { - metadata.Series = opf.Metadata.CalibreSeries[0].Name - } - - // Series index - if len(opf.Metadata.CalibreSeriesIndex) > 0 { - if index, err := strconv.ParseFloat(opf.Metadata.CalibreSeriesIndex[0].Value, 32); err == nil { - metadata.SeriesNumber = int32(index) - } - } - - return metadata, nil -} -``` - ---- - -## 📝 STEP 3: Add extractCalibreSidecar() Function - -**Location**: Add before `extractMetadata()` function (around line 710) - -**ADD THIS COMPLETE FUNCTION**: - -```go -// extractCalibreSidecar checks for and parses a Calibre metadata.opf sidecar file -func (s *MediaScanner) extractCalibreSidecar(path string) *MediaMetadata { - // Get directory of media file - dir := filepath.Dir(path) - opfPath := filepath.Join(dir, "metadata.opf") - - // Check if sidecar exists - if _, err := os.Stat(opfPath); os.IsNotExist(err) { - return nil // No sidecar, not an error - } - - // Parse sidecar - metadata, err := s.parseCalibreMetadataOPF(opfPath) - if err != nil { - fmt.Printf("Warning: failed to parse Calibre metadata.opf: %v\n", err) - return nil // Parsing failed, fall back to embedded - } - - return metadata -} -``` - ---- - -## 📝 STEP 4: Modify extractMetadata() Function - -**Location**: `internal/services/media_scanner.go` line 712 - -**FIND THIS EXISTING CODE**: - -```go -func (s *MediaScanner) extractMetadata(path string) (*MediaMetadata, error) { - ext := strings.ToLower(filepath.Ext(path)) - - switch ext { - case ".epub": - metadata, err := s.extractEPUBMetadata(path) - // ... rest of function -``` - -**REPLACE WITH THIS CODE** (add sidecar check at the beginning): - -```go -func (s *MediaScanner) extractMetadata(path string) (*MediaMetadata, error) { - // NEW: Try Calibre sidecar first - if metadata := s.extractCalibreSidecar(path); metadata != nil { - fmt.Printf("Using Calibre metadata.opf for %s\n", path) - - // Try to find cover image for sidecar metadata - coverPath := findSidecarCover(path) - if coverPath != "" { - metadata.CoverPath = s.getRelativePath(coverPath) - } - - return metadata, nil - } - - // EXISTING: Fallback to embedded metadata - ext := strings.ToLower(filepath.Ext(path)) - - switch ext { - case ".epub": - metadata, err := s.extractEPUBMetadata(path) - if err != nil { - return metadata, err - } - // Try to extract embedded cover - coverPath, err := s.extractEPUBCover(path) - if err != nil { - fmt.Printf("Warning: failed to extract EPUB cover from %s: %v\n", path, err) - } else if coverPath != "" { - metadata.CoverPath = s.getRelativePath(coverPath) - } - // If no embedded cover, try sidecar - if metadata.CoverPath == "" { - sidecarCover := findSidecarCover(path) - if sidecarCover != "" { - metadata.CoverPath = s.getRelativePath(sidecarCover) - } - } - return metadata, nil - case ".pdf": - return s.extractPDFMetadata(path) - default: - // For other formats, return basic metadata - return &MediaMetadata{ - Title: strings.TrimSuffix(filepath.Base(path), ext), - }, nil - } -} -``` - -**NOTE**: The rest of the function remains EXACTLY the same. Only the sidecar check at the beginning is new. - ---- - -#### New Struct: `CalibreOPFMetadata` - -```go -// Intermediate struct for parsed OPF data -type CalibreOPFMetadata struct { - Title string - Authors []string - Tags []string - Description string - Publisher string - PublishDate *time.Time - Language string - ISBN string - ASIN string - UUID string - Contributors []string - Series string - SeriesIndex *float64 - Rating *int32 - Timestamp *time.Time -} -``` - -### Metadata Priority (Sidecar-First) - -``` -Priority Order: -1. Calibre metadata.opf sidecar ← NEW (highest priority) -2. Embedded file metadata (EPUB OPF, PDF info, ComicInfo.xml) -3. Folder structure parsing -4. Filename parsing (last resort) -``` - -**Rationale**: If a user has a `metadata.opf` file, they're actively managing their library in Calibre. The sidecar represents their curated "source of truth." - -### Error Handling - -1. **Sidecar file not found** → Return nil, continue to embedded metadata -2. **Malformed XML** → Log warning, continue to embedded metadata -3. **Missing required fields** → Use available fields, continue -4. **Invalid date format** → Log warning, skip field -5. **File permission errors** → Log error, continue to embedded metadata - -**Principle**: Graceful degradation - never fail a scan due to sidecar issues - -### Testing Strategy - -#### Unit Tests - -File: `internal/services/media_scanner_calibre_test.go` - -- Test parsing complete metadata.opf with all fields -- Test parsing minimal metadata.opf (title only) -- Test parsing malformed XML (graceful failure) -- Test missing metadata.opf (returns nil, not error) -- Test multiple `` tags (tags array) -- Test multiple `` tags (extract ISBN/ASIN) -- Test Calibre-specific fields (series, series_index) -- Test invalid date formats -- Test empty/whitespace values - -#### Integration Tests - -File: `cmd/server/tests/calibre_integration_test.go` - -- Test scanning directory with Calibre library structure -- Test scanning directory without sidecar files (existing behavior) -- Test mixed: some books with sidecar, some without -- Test force rescan updates from sidecar changes -- Test all file types: EPUB, PDF, CBZ, MP3, M4B -- Test all library types: ebook, audiobook, comic - -#### Test Data - -Create `testdata/calibre/` with sample files: -- `Complete Book/metadata.opf` - All fields populated -- `Minimal Book/metadata.opf` - Only required fields -- `Malformed Book/metadata.opf` - Invalid XML -- No sidecar files (rely on existing test data) - -## Implementation Steps - -### ✅ IMPLEMENTATION CHECKLIST - -Follow these steps in order. Each step includes ready-to-copy code. - -#### Phase 1: Core Functionality - -**Step 1.1**: Add `CalibreOPFMetadata` struct -- [ ] Open `internal/services/media_scanner.go` -- [ ] Find location after existing struct definitions (around line 100-150) -- [ ] Copy and paste the struct from **📝 STEP 1** above -- [ ] Save file - -**Step 1.2**: Add `parseCalibreMetadataOPF()` function -- [ ] Open `internal/services/media_scanner.go` -- [ ] Find location after `extractEPUBMetadata()` function (around line 830) -- [ ] Copy and paste the complete function from **📝 STEP 2** above -- [ ] Save file -- [ ] Run `go build ./internal/services` to verify no syntax errors - -**Step 1.3**: Add `extractCalibreSidecar()` function -- [ ] Open `internal/services/media_scanner.go` -- [ ] Find location before `extractMetadata()` function (around line 710) -- [ ] Copy and paste the complete function from **📝 STEP 3** above -- [ ] Save file -- [ ] Run `go build ./internal/services` to verify no syntax errors - -**Step 1.4**: Modify `extractMetadata()` function -- [ ] Open `internal/services/media_scanner.go` -- [ ] Go to line 712 (function `extractMetadata`) -- [ ] Find the line: `func (s *MediaScanner) extractMetadata(path string) (*MediaMetadata, error) {` -- [ ] Find the next line: `ext := strings.ToLower(filepath.Ext(path))` -- [ ] **REPLACE** those two lines with the code from **📝 STEP 4** above -- [ ] Save file -- [ ] Run `go build ./internal/services` to verify compilation -- [ ] Verify: `go test ./internal/services -v` passes existing tests - -**Step 1.5**: Quick manual test -- [ ] Create test directory with a Calibre book + metadata.opf -- [ ] Run scanner on that directory -- [ ] Check logs for "Using Calibre metadata.opf" message -- [ ] Verify metadata was imported correctly - -**Total new code: ~150 lines** (simplified from original ~250 lines) - ---- - -### Phase 2: Testing (REQUIRED) - -#### Unit Tests - -**File**: Create `internal/services/media_scanner_calibre_test.go` - -**ADD THIS COMPLETE TEST FILE**: - -```go -package services - -import ( - "os" - "path/filepath" - "testing" -) - -func TestParseCalibreMetadataOPF(t *testing.T) { - tests := []struct { - name string - opfContent string - wantTitle string - wantAuthor string - wantSeries string - wantTags int - wantErr bool - }{ - { - name: "Complete metadata.opf", - opfContent: ` - - - Test Book - Test Author - Fantasy - Adventure - Test description - Test Publisher - 2024-01-15 - en - 978-0-123456-78-9 - Contributor Name - - - -`, - wantTitle: "Test Book", - wantAuthor: "Test Author", - wantSeries: "Test Series", - wantTags: 2, - wantErr: false, - }, - { - name: "Minimal metadata.opf", - opfContent: ` - - - Minimal Book - -`, - wantTitle: "Minimal Book", - wantAuthor: "", - wantSeries: "", - wantTags: 0, - wantErr: false, - }, - { - name: "Malformed XML", - opfContent: ` - - - Test`, - wantTitle: "", - wantAuthor: "", - wantSeries: "", - wantTags: 0, - wantErr: true, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - // Create temporary OPF file - tmpDir := t.TempDir() - opfPath := filepath.Join(tmpDir, "metadata.opf") - if err := os.WriteFile(opfPath, []byte(tt.opfContent), 0644); err != nil { - t.Fatalf("Failed to create test OPF: %v", err) - } - - // Parse - scanner := &MediaScanner{} - got, err := scanner.parseCalibreMetadataOPF(opfPath) - - if (err != nil) != tt.wantErr { - t.Errorf("parseCalibreMetadataOPF() error = %v, wantErr %v", err, tt.wantErr) - return - } - - if !tt.wantErr && got.Title != tt.wantTitle { - t.Errorf("Title = %v, want %v", got.Title, tt.wantTitle) - } - if !tt.wantErr && got.Author != tt.wantAuthor { - t.Errorf("Author = %v, want %v", got.Author, tt.wantAuthor) - } - if !tt.wantErr && got.Series != tt.wantSeries { - t.Errorf("Series = %v, want %v", got.Series, tt.wantSeries) - } - if !tt.wantErr && len(got.Tags) != tt.wantTags { - t.Errorf("Tags length = %v, want %v", len(got.Tags), tt.wantTags) - } - }) - } -} - -func TestExtractCalibreSidecar(t *testing.T) { - t.Run("Sidecar exists", func(t *testing.T) { - tmpDir := t.TempDir() - opfPath := filepath.Join(tmpDir, "metadata.opf") - bookPath := filepath.Join(tmpDir, "book.epub") - - // Create OPF file - opfContent := ` - - - Sidecar Test - Test Author - -` - if err := os.WriteFile(opfPath, []byte(opfContent), 0644); err != nil { - t.Fatal(err) - } - - // Test extraction - scanner := &MediaScanner{} - metadata := scanner.extractCalibreSidecar(bookPath) - - if metadata == nil { - t.Error("Expected metadata, got nil") - return - } - - if metadata.Title != "Sidecar Test" { - t.Errorf("Title = %v, want 'Sidecar Test'", metadata.Title) - } - }) - - t.Run("No sidecar", func(t *testing.T) { - tmpDir := t.TempDir() - bookPath := filepath.Join(tmpDir, "book.epub") - - scanner := &MediaScanner{} - metadata := scanner.extractCalibreSidecar(bookPath) - - if metadata != nil { - t.Error("Expected nil, got metadata") - } - }) -} -``` - -**Step 2.1**: Run unit tests -- [ ] Save test file as `internal/services/media_scanner_calibre_test.go` -- [ ] Run: `go test ./internal/services -v -run TestParseCalibreMetadataOPF` -- [ ] Run: `go test ./internal/services -v -run TestExtractCalibreSidecar` -- [ ] All tests should pass - ---- - -### Phase 3: Integration Testing (OPTIONAL but RECOMMENDED) - -**File**: Create `cmd/server/tests/calibre_integration_test.go` - -**ADD THIS INTEGRATION TEST** (optional, for comprehensive testing): - -```go -package main - -import ( - "context" - "os" - "path/filepath" - "testing" - - "github.com/yourusername/bookhoard/internal/database" - "github.com/yourusername/bookhoard/internal/services" -) - -func TestCalibreLibraryScan(t *testing.T) { - // Setup test server - setup := setupTestServer(t) - defer setup.Teardown() - - // Create test Calibre library structure - tmpDir := t.TempDir() - - // Create author directory - authorDir := filepath.Join(tmpDir, "Test Author") - if err := os.Mkdir(authorDir, 0755); err != nil { - t.Fatal(err) - } - - // Create book directory - bookDir := filepath.Join(authorDir, "Test Book") - if err := os.Mkdir(bookDir, 0755); err != nil { - t.Fatal(err) - } - - // Create metadata.opf - opfPath := filepath.Join(bookDir, "metadata.opf") - opfContent := ` - - - Test Book - Test Author - Fantasy - Test description - Test Publisher - - - -` - if err := os.WriteFile(opfPath, []byte(opfContent), 0644); err != nil { - t.Fatal(err) - } - - // Create dummy EPUB file - epubPath := filepath.Join(bookDir, "Test Book.epub") - if err := os.WriteFile(epubPath, []byte("dummy epub content"), 0644); err != nil { - t.Fatal(err) - } - - // Add library - libraryID := createTestLibrary(t, setup.DB, tmpDir) - - // Scan library - scanner := services.NewMediaScanner(setup.DB, setup.AdminID, []string{tmpDir}) - err := scanner.ScanFolders(context.Background()) - if err != nil { - t.Fatalf("ScanFolders() error = %v", err) - } - - // Verify imported book - books, err := setup.DB.ListMediaItems(context.Background(), libraryID) - if err != nil { - t.Fatalf("ListMediaItems() error = %v", err) - } - - if len(books) != 1 { - t.Fatalf("Got %d books, want 1", len(books)) - } - - book := books[0] - - // Verify metadata from sidecar - if book.Title != "Test Book" { - t.Errorf("Title = %v, want 'Test Book'", book.Title) - } - if book.Author.String != "Test Author" { - t.Errorf("Author = %v, want 'Test Author'", book.Author.String) - } - if book.Series.String != "Test Series" { - t.Errorf("Series = %v, want 'Test Series'", book.Series.String) - } - if book.SeriesNumber.Int32 != 1 { - t.Errorf("SeriesNumber = %v, want 1", book.SeriesNumber.Int32) - } -} -``` - -**Step 3.1**: Run integration test -- [ ] Save test file -- [ ] Run: `go test ./cmd/server/tests -v -run TestCalibreLibraryScan` -- [ ] Should pass - ---- - -### Phase 4: Verification & Testing - -**Step 4.1**: Build verification -```bash -# Build entire project -go build ./... - -# Run all tests -go test ./... -v - -# Run specific scanner tests -go test ./internal/services -v -go test ./cmd/server/tests -v -``` - -**Step 4.2**: Manual verification with real Calibre library -- [ ] Have a real Calibre library available -- [ ] Add library in Bookhoard admin panel -- [ ] Run manual scan -- [ ] Check that Calibre metadata is imported: - - [ ] Titles correct - - [ ] Authors correct - - [ ] Series info present - - [ ] Tags imported - - [ ] Covers display -- [ ] Check logs for "Using Calibre metadata.opf" messages - -**Step 4.3**: Test force rescan -- [ ] Edit metadata in Calibre (change title, add series) -- [ ] In Bookhoard, trigger force rescan -- [ ] Verify metadata updates from sidecar - -**Step 4.4**: Test backward compatibility -- [ ] Scan library WITHOUT metadata.opf files -- [ ] Verify existing behavior (embedded metadata) still works -- [ ] Check logs show embedded extraction, not sidecar - ---- - -## 📦 REQUIRED IMPORTS - -Before implementing, verify these imports are in `internal/services/media_scanner.go`: - -```go -import ( - "encoding/xml" // NEW: Needed for XML parsing - "fmt" - "os" - "path/filepath" - "strconv" - "strings" - "time" - - "github.com/yourusername/bookhoard/internal/utils" - // ... other existing imports -) -``` - -**If `encoding/xml` is missing**, add it to the imports section. - ---- - -## 🎯 IMPLEMENTATION SUMMARY - -### Files to Modify: 1 -- `internal/services/media_scanner.go` (~150 lines added) - -### Files to Create: 1 -- `internal/services/media_scanner_calibre_test.go` (~200 lines) - -### Optional Files to Create: 1 -- `cmd/server/tests/calibre_integration_test.go` (~100 lines) - -### Total Work -- **~150 lines** of new production code -- **~200 lines** of test code -- **1 function** to modify (add 10 lines at beginning) -- **2 functions** to add (complete functions provided above) -- **1 struct** to add (provided above) - -### Expected Time -- **Phase 1 (Core)**: 30-45 minutes -- **Phase 2 (Unit Tests)**: 30 minutes -- **Phase 3 (Integration)**: 30 minutes (optional) -- **Phase 4 (Verification)**: 30 minutes -- **Total**: ~2-2.5 hours - ---- - -## ✅ SUCCESS CRITERIA - -Implementation is complete when: - -- [ ] All code compiles without errors (`go build ./...`) -- [ ] All existing tests pass (`go test ./...`) -- [ ] New unit tests pass (`go test ./internal/services -v -run TestParseCalibreMetadataOPF`) -- [ ] Real Calibre library scans successfully -- [ ] Logs show "Using Calibre metadata.opf" for sidecar files -- [ ] Non-Calibre libraries still work (backward compatibility) -- [ ] Metadata fields correctly imported (title, author, series, tags, etc.) -- [ ] No performance regression (scan time increase <5%) - ---- - -## 🐛 TROUBLESHOOTING - -### Compilation Errors - -**Error**: `undefined: parseCalibreMetadataOPF` -- **Fix**: Make sure you added the function in STEP 2 before STEP 3 - -**Error**: `undefined: extractCalibreSidecar` -- **Fix**: Make sure you added the function in STEP 3 before STEP 4 - -**Error**: `xml.Name undefined` -- **Fix**: Add import `"encoding/xml"` to file imports - -### Test Failures - -**Error**: Test fails with "no such file or directory" -- **Fix**: Make sure test creates temporary directory with `t.TempDir()` - -**Error**: "Expected nil, got metadata" -- **Fix**: Verify `extractCalibreSidecar()` returns `nil` when no sidecar exists - -### Runtime Issues - -**Problem**: Scanner doesn't find metadata.opf -- **Check**: Is metadata.opf in the SAME directory as the book file? -- **Check**: File permissions (scanner needs read access) -- **Check**: Logs for error messages - -**Problem**: XML parsing fails -- **Check**: Is metadata.opf valid XML? -- **Check**: Does it have proper namespaces? -- **Check**: Logs for "failed to parse" warnings - ---- - -## 📚 REFERENCE IMPLEMENTATION - -All code in this document is production-ready and can be copied/pasted directly. The implementation: - -- ✅ Follows Go best practices -- ✅ Uses existing Bookhoard patterns -- ✅ Handles errors gracefully -- ✅ Includes comprehensive tests -- ✅ Maintains backward compatibility -- ✅ No breaking changes - -**Ready to implement!** 🚀 - -### Phase 4: Future Enhancements (OPTIONAL) - -- [ ] Support Calibre `author_link` metadata -- [ ] Support Calibre custom columns -- [ ] Import Calibre ratings as user's initial rating -- [ ] Detect Calibre library structure patterns -- [ ] Configuration option for merge strategy (sidecar-first vs. smart-merge) - -## Backward Compatibility - -**No breaking changes.** Existing behavior is preserved: - -- Libraries without `metadata.opf` files → Existing behavior (embedded metadata) -- Scanning logic unchanged → Single function modification, no call site changes -- Database unchanged → All fields already exist -- API unchanged → No new endpoints or response fields -- **No wrapper function needed** → Simpler, cleaner implementation - -## Success Criteria - -- [ ] All existing tests pass -- [ ] New unit tests pass (90%+ coverage) -- [ ] New integration tests pass -- [ ] Scans real Calibre library successfully -- [ ] Scans non-Calibre library without errors -- [ ] No performance regression (scan time +5% max) -- [ ] Documentation complete and accurate -- [ ] Code review approved - -## Open Questions - -1. **Q**: Should we import Calibre ratings? - **A**: No - ratings are per-user in Bookhoard. Calibre rating is the owner's opinion, not necessarily the current user's. - -2. **Q**: Should we support Calibre custom columns? - **A**: Not in initial implementation. Custom columns are user-defined and would require dynamic schema or JSON storage. Future enhancement. - -3. **Q**: How to handle multiple authors? - **A**: Store first author in `author` field (existing behavior). Future enhancement could add `authors[]` array. - -4. **Q**: Should we follow Calibre folder structure? - **A**: Already implemented! `extractFolderStructureMetadata()` handles Calibre patterns. - -## References - -- [EPUB Publications 3.0 Specification](https://www.idpf.org/epub/30/spec/epub30-publications.html) -- [Dublin Core Metadata Element Set](https://www.dublincore.org/specifications/dces/1998-08-13/) -- [Calibre Manual - Editing Metadata](https://manual.calibre-ebook.com/metadata.html) -- [Calibre Source Code](https://github.com/kovidgoyal/calibre) - -## Appendix: Decision Log - -### Decision 1: Sidecar-First Approach -**Date**: 2026-03-26 -**Decision**: Use sidecar-first (not smart-merge) -**Rationale**: -- Users with Calibre sidecars actively curate their metadata -- Sidecar metadata is typically richer and more accurate -- Simple implementation, predictable behavior -- Easy to test and debug -- Future enhancement can add configurable merge strategy - -### Decision 2: No Database Schema Changes -**Date**: 2026-03-26 -**Decision**: Use existing schema only -**Rationale**: -- All required fields already exist in `media_items` table -- Pre-production application (no legacy data to migrate) -- Simpler implementation -- No backward compatibility concerns - -### Decision 3: All Library Types Supported -**Date**: 2026-03-26 -**Decision**: Support OPF for all libraries, not just ebooks -**Rationale**: -- Calibre supports all file types -- Users may organize audiobooks/comics in Calibre -- Consistent behavior across library types -- No additional complexity - -### Decision 4: Simplified Implementation (REVISED) -**Date**: 2026-03-26 -**Decision**: Modify `extractMetadata()` directly instead of creating wrapper function -**Rationale**: -- **User feedback**: Suggested simpler approach with if statement check -- **Less code**: ~150 lines vs. ~250 lines -- **No call site changes**: All existing code continues to work -- **Clearer flow**: Single entry point for metadata extraction -- **Better testability**: Test `extractMetadata()` with/without sidecar -- **Easier to maintain**: All metadata logic in one place - -**Original plan**: -```go -// Wrapper function approach -extractMetadataWithSidecar() { - if sidecar exists → return sidecar - return extractMetadata() -} -``` - -**Revised plan (SIMPLER)**: -```go -// Direct modification -extractMetadata() { - if sidecar exists → return sidecar // ← NEW: Just add this at top - // existing switch statement continues... -} -``` - -This simplification reduces code complexity and makes the implementation cleaner and easier to understand. diff --git a/bruno/media-items/search/Field Values Search - Authors.yml b/bruno/media-items/search/Field Values Search - Authors.yml deleted file mode 100644 index 75a8a3d..0000000 --- a/bruno/media-items/search/Field Values Search - Authors.yml +++ /dev/null @@ -1,89 +0,0 @@ -info: - name: Field Values Search - Authors - type: http - seq: 6 - -http: - method: GET - url: "{{base_url}}/api/media-items/search?library_id={{library_id}}&author=orwell&limit=50" - params: - - name: library_id - value: "{{library_id}}" - type: query - - name: author - value: orwell - type: query - - name: limit - value: "50" - type: query - auth: inherit - -settings: - encodeUrl: true - timeout: 0 - followRedirects: true - maxRedirects: 5 - -docs: |- - ## Field Values Search - Authors - - Fetches distinct author values for autocomplete dropdowns with counts and similarity scores. - - **Method:** GET - - **Endpoint:** /api/media-items/search - - **Authentication:** Required (Bearer token) - - **Query Parameters:** - - `library_id` (string, required): Library UUID to search within - - `author` (string, required): Author name to search for (fuzzy match) - - `limit` (integer, optional): Maximum results to return (default: 50) - - **How It Works:** - - Returns distinct author names from the library - - Fuzzy matches the search query against author names - - Includes count of books per author - - Includes similarity score (0-1, higher = better match) - - Sorted by similarity score, then by count - - **Use Case:** - - Populate autocomplete dropdown when user types in author filter field - - Show user available authors with book counts - - Help users discover similar author names - - **Response:** HTTP 200 (OK) - ```json - { - "results": [ - { - "value": "Asimov, Isaac", - "count": 47, - "score": 0.8 - }, - { - "value": "Asimov, Isaac & Robert Silverberg", - "count": 2, - "score": 0.75 - } - ], - "total": 2 - } - ``` - - **Examples:** - - `author=asimov` → Returns "Asimov, Isaac" (47 books), "Asimov, Isaac & Robert Silverberg" (2 books) - - `author=rowling` → Returns "Rowling, J.K." (10 books) - - `author=tolkien` → Returns "Tolkien, J.R.R." (15 books) - - **Frontend Integration:** - - Call this endpoint when user types ≥2 characters in author field - - Display results in `` or custom dropdown - - Show: "Author Name (count)" format - - Allow user to select from suggestions - - **Success Criteria:** - - Status: 200 - - Returns array of author values with counts - - Results sorted by relevance (similarity score) - - Only returns authors matching the fuzzy search diff --git a/bruno/media-items/search/Field Values Search - Genres.yml b/bruno/media-items/search/Field Values Search - Genres.yml deleted file mode 100644 index 9bdfced..0000000 --- a/bruno/media-items/search/Field Values Search - Genres.yml +++ /dev/null @@ -1,89 +0,0 @@ -info: - name: Field Values Search - Genres - type: http - seq: 7 - -http: - method: GET - url: "{{base_url}}/api/media-items/search?library_id={{library_id}}&tags=sci&limit=50" - params: - - name: library_id - value: "{{library_id}}" - type: query - - name: tags - value: sci - type: query - - name: limit - value: "50" - type: query - auth: inherit - -settings: - encodeUrl: true - timeout: 0 - followRedirects: true - maxRedirects: 5 - -docs: |- - ## Field Values Search - Genres - - Fetches distinct tags values for autocomplete dropdowns with counts and similarity scores. - - **Method:** GET - - **Endpoint:** /api/media-items/search - - **Authentication:** Required (Bearer token) - - **Query Parameters:** - - `library_id` (string, required): Library UUID to search within - - `tags` (string, required): Genre name to search for (fuzzy match) - - `limit` (integer, optional): Maximum results to return (default: 50) - - **How It Works:** - - Returns distinct tags names from the library - - Fuzzy matches the search query against tags names - - Includes count of books per tags - - Includes similarity score (0-1, higher = better match) - - Sorted by similarity score, then by count - - **Use Case:** - - Populate autocomplete dropdown when user types in tags filter field - - Show user available tagss with book counts - - Help users discover similar tags names - - **Response:** HTTP 200 (OK) - ```json - { - "results": [ - { - "value": "Sci-Fi", - "count": 234, - "score": 0.85 - }, - { - "value": "Science Fiction", - "count": 156, - "score": 0.82 - } - ], - "total": 2 - } - ``` - - **Examples:** - - `tags=sci` → Returns "Sci-Fi" (234 books), "Science Fiction" (156 books) - - `tags=fant` → Returns "Fantasy" (345 books), "High Fantasy" (89 books) - - `tags=mystery` → Returns "Mystery" (123 books), "Mystery/Thriller" (45 books) - - **Frontend Integration:** - - Call this endpoint when user types ≥2 characters in tags field - - Display results in `` or custom dropdown - - Show: "Genre Name (count)" format - - Allow user to select from suggestions - - **Success Criteria:** - - Status: 200 - - Returns array of tags values with counts - - Results sorted by relevance (similarity score) - - Only returns tagss matching the fuzzy search diff --git a/bruno/media-items/search/Field Values Search - Languages.yml b/bruno/media-items/search/Field Values Search - Languages.yml deleted file mode 100644 index 777144c..0000000 --- a/bruno/media-items/search/Field Values Search - Languages.yml +++ /dev/null @@ -1,89 +0,0 @@ -info: - name: Field Values Search - Languages - type: http - seq: 8 - -http: - method: GET - url: "{{base_url}}/api/media-items/search?library_id={{library_id}}&language=eng&limit=50" - params: - - name: library_id - value: "{{library_id}}" - type: query - - name: language - value: eng - type: query - - name: limit - value: "50" - type: query - auth: inherit - -settings: - encodeUrl: true - timeout: 0 - followRedirects: true - maxRedirects: 5 - -docs: |- - ## Field Values Search - Languages - - Fetches distinct language values for autocomplete dropdowns with counts and similarity scores. - - **Method:** GET - - **Endpoint:** /api/media-items/search - - **Authentication:** Required (Bearer token) - - **Query Parameters:** - - `library_id` (string, required): Library UUID to search within - - `language` (string, required): Language name/code to search for (fuzzy match) - - `limit` (integer, optional): Maximum results to return (default: 50) - - **How It Works:** - - Returns distinct language names from the library - - Fuzzy matches the search query against language names - - Includes count of books per language - - Includes similarity score (0-1, higher = better match) - - Sorted by similarity score, then by count - - **Use Case:** - - Populate autocomplete dropdown when user types in language filter field - - Show user available languages with book counts - - Help users discover similar language codes/names - - **Response:** HTTP 200 (OK) - ```json - { - "results": [ - { - "value": "English", - "count": 1245, - "score": 0.9 - }, - { - "value": "eng", - "count": 856, - "score": 0.85 - } - ], - "total": 2 - } - ``` - - **Examples:** - - `language=eng` → Returns "English" (1245 books), "eng" (856 books) - - `language=fre` → Returns "French" (234 books), "fre" (123 books) - - `language=german` → Returns "German" (189 books), "deutsch" (45 books) - - **Frontend Integration:** - - Call this endpoint when user types ≥2 characters in language field - - Display results in `` or custom dropdown - - Show: "Language Name (count)" format - - Allow user to select from suggestions - - **Success Criteria:** - - Status: 200 - - Returns array of language values with counts - - Results sorted by relevance (similarity score) - - Only returns languages matching the fuzzy search diff --git a/bruno/media-items/search/Field Values Search - Series.yml b/bruno/media-items/search/Field Values Search - Series.yml deleted file mode 100644 index b4eda9d..0000000 --- a/bruno/media-items/search/Field Values Search - Series.yml +++ /dev/null @@ -1,89 +0,0 @@ -info: - name: Field Values Search - Series - type: http - seq: 9 - -http: - method: GET - url: "{{base_url}}/api/media-items/search?library_id={{library_id}}&series=haley&limit=50" - params: - - name: library_id - value: "{{library_id}}" - type: query - - name: series - value: haley - type: query - - name: limit - value: "50" - type: query - auth: inherit - -settings: - encodeUrl: true - timeout: 0 - followRedirects: true - maxRedirects: 5 - -docs: |- - ## Field Values Search - Series - - Fetches distinct series values for autocomplete dropdowns with counts and similarity scores. - - **Method:** GET - - **Endpoint:** /api/media-items/search - - **Authentication:** Required (Bearer token) - - **Query Parameters:** - - `library_id` (string, required): Library UUID to search within - - `series` (string, required): Series name to search for (fuzzy match) - - `limit` (integer, optional): Maximum results to return (default: 50) - - **How It Works:** - - Returns distinct series names from the library - - Fuzzy matches the search query against series names - - Includes count of books per series - - Includes similarity score (0-1, higher = better match) - - Sorted by similarity score, then by count - - **Use Case:** - - Populate autocomplete dropdown when user types in series filter field - - Show user available series with book counts - - Help users discover similar series names - - **Response:** HTTP 200 (OK) - ```json - { - "results": [ - { - "value": "Foundation", - "count": 7, - "score": 0.9 - }, - { - "value": "Foundation Series", - "count": 7, - "score": 0.85 - } - ], - "total": 2 - } - ``` - - **Examples:** - - `series=foundation` → Returns "Foundation" (7 books), "Foundation Series" (7 books) - - `series=harry` → Returns "Harry Potter" (10 books) - - `series=lord` → Returns "Lord of the Rings" (3 books) - - **Frontend Integration:** - - Call this endpoint when user types ≥2 characters in series field - - Display results in `` or custom dropdown - - Show: "Series Name (count)" format - - Allow user to select from suggestions - - **Success Criteria:** - - Status: 200 - - Returns array of series values with counts - - Results sorted by relevance (similarity score) - - Only returns series matching the fuzzy search