bookhoard/docs/development/calibre-opf-implementation.md

# Calibre metadata.opf Implementation

This document provides technical details for implementing Calibre `metadata.opf` sidecar file support in Bookhoard's media scanner.

## Overview

The media scanner detects and imports metadata from Calibre's `metadata.opf` sidecar files using a **sidecar-first approach**. If a `metadata.opf` file exists alongside a book file, its metadata takes precedence over embedded file metadata.

## Architecture

### Scanner Pipeline

```go
func (s *MediaScanner) processMediaFile(ctx context.Context, path string) (bool, error) {
    // 1. Get file info
    // 2. Find library
    // 3. Check if media item exists
    // 4. EXTRACT METADATA (NEW: sidecar-first)
    metadata := s.extractMetadataWithSidecar(path)
    // 5. Extract hash info
    // 6. Extract comic metadata (if applicable)
    // 7. Extract folder structure metadata
    // 8. Normalize metadata
    // 9. Create or update in database
}
```

### Metadata Extraction Flow

```
extractMetadataWithSidecar()
    ↓
1. Try Calibre metadata.opf sidecar
   ├─ Found → parseCalibreMetadataOPF()
   └─ Not found → continue
    ↓
2. Extract embedded metadata
   ├─ EPUB → extractEPUBMetadata()
   ├─ PDF → extractPDFMetadata()
   └─ Comic → extractComicMetadata()
    ↓
3. Extract folder structure metadata
   └─ extractFolderStructureMetadata()
    ↓
4. Return merged MediaMetadata
```

## Data Structures

### CalibreOPFMetadata

Intermediate struct for parsed OPF data:

```go
// CalibreOPFMetadata represents parsed metadata from Calibre metadata.opf
type CalibreOPFMetadata struct {
    Title          string
    Authors        []string
    Tags           []string
    Description    string
    Publisher      string
    PublishDate    *time.Time
    Language       string
    ISBN           string
    ASIN           string
    UUID           string
    Contributors   []string
    Series         string
    SeriesIndex    *float64
    Rating         *int32
    Timestamp      *time.Time
}
```

### MediaMetadata

Existing struct (already defined in `media_scanner.go`):

```go
// MediaMetadata represents extracted book metadata
type MediaMetadata struct {
    Title          string
    Author         string
    ISBN           string
    Description    string
    Publisher      string
    Series         string
    SeriesNumber   int32
    Tags           []string
    Contributors   []string
    PublishDate    time.Time
    ASIN           string
    // ... other fields
}
```

## Implementation Functions

### 1. extractCalibreSidecar()

**Location**: `internal/services/media_scanner.go`

**Signature**:
```go
func (s *MediaScanner) extractCalibreSidecar(path string) *MediaMetadata
```

**Purpose**: Checks for `metadata.opf` in the same directory as the book file.

**Logic**:
```go
func (s *MediaScanner) extractCalibreSidecar(path string) *MediaMetadata {
    // Get directory of media file
    dir := filepath.Dir(path)
    opfPath := filepath.Join(dir, "metadata.opf")

    // Check if sidecar exists
    if _, err := os.Stat(opfPath); os.IsNotExist(err) {
        return nil  // No sidecar, not an error
    }

    // Parse sidecar
    metadata, err := s.parseCalibreMetadataOPF(opfPath)
    if err != nil {
        fmt.Printf("Warning: failed to parse Calibre metadata.opf: %v\n", err)
        return nil  // Parsing failed, fall back to embedded
    }

    return metadata
}
```

**Error Handling**:
- File not found → Return `nil` (not an error)
- Permission denied → Log warning, return `nil`
- Parse errors → Log warning, return `nil` (graceful fallback)

**Note**: Simplified from original plan - no wrapper function needed

### 2. parseCalibreMetadataOPF()

**Location**: `internal/services/media_scanner.go`

**Signature**:
```go
func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error)
```

**Purpose**: Parses XML file and extracts Dublin Core + Calibre-specific fields.

**Logic**:
```go
func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error) {
    // Open file
    file, err := os.Open(opfPath)
    if err != nil {
        return nil, fmt.Errorf("failed to open metadata.opf: %v", err)
    }
    defer file.Close()

    // Parse XML
    var opf struct {
        XMLName  xml.Name `xml:"package"`
        Metadata struct {
            XMLName   xml.Name `xml:"metadata"`
            Titles    []string `xml:"dc:title"`
            Creators []string `xml:"dc:creator"`
            Subjects  []string `xml:"dc:subject"`
            Desc      []string `xml:"dc:description"`
            Publisher []string `xml:"dc:publisher"`
            Dates     []string `xml:"dc:date"`
            Language  []string `xml:"dc:language"`
            Identifiers []struct {
                Scheme string `xml:"opf:scheme,attr"`
                Value  string `xml:",chardata"`
            } `xml:"dc:identifier"`
            Contributors []string `xml:"dc:contributor"`
            CalibreSeries []struct {
                Name  string `xml:"content,attr"`
            } `xml:"meta[name='calibre:series']"`
            CalibreSeriesIndex []struct {
                Value  string `xml:"content,attr"`
            } `xml:"meta[name='calibre:series_index']"`
        } `xml:"metadata"`
    }

    if err := xml.NewDecoder(file).Decode(&opf); err != nil {
        return nil, fmt.Errorf("failed to parse metadata.opf XML: %v", err)
    }

    // Map to MediaMetadata
    metadata := &MediaMetadata{}

    // Title (required)
    if len(opf.Metadata.Titles) > 0 {
        metadata.Title = opf.Metadata.Titles[0]
    }

    // Author (first creator)
    if len(opf.Metadata.Creators) > 0 {
        metadata.Author = opf.Metadata.Creators[0]
    }

    // Tags (all subjects)
    if len(opf.Metadata.Subjects) > 0 {
        metadata.Tags = utils.NormalizeTags(opf.Metadata.Subjects)
    }

    // Description
    if len(opf.Metadata.Desc) > 0 {
        metadata.Description = opf.Metadata.Desc[0]
    }

    // Publisher
    if len(opf.Metadata.Publisher) > 0 {
        metadata.Publisher = opf.Metadata.Publisher[0]
    }

    // Publish date
    if len(opf.Metadata.Dates) > 0 {
        if date, err := time.Parse("2006-01-02", opf.Metadata.Dates[0]); err == nil {
            metadata.PublishDate = date
        }
    }

    // Language
    if len(opf.Metadata.Language) > 0 {
        // Store language (already in schema)
        // Note: MediaMetadata doesn't have Language field yet
        // Future enhancement: add Language field
    }

    // Identifiers (ISBN, ASIN)
    for _, id := range opf.Metadata.Identifiers {
        switch strings.ToUpper(id.Scheme) {
        case "ISBN":
            metadata.ISBN = utils.NormalizeISBNSafe(id.Value)
        case "ASIN":
            metadata.ASIN = id.Value
        case "UUID", "CALIBRE":
            // Store UUID in hash info, not metadata
            // Will be extracted by extractHashInfo()
        }
    }

    // Contributors
    if len(opf.Metadata.Contributors) > 0 {
        metadata.Contributors = utils.NormalizeContributors(opf.Metadata.Contributors)
    }

    // Series
    if len(opf.Metadata.CalibreSeries) > 0 {
        metadata.Series = opf.Metadata.CalibreSeries[0].Name
    }

    // Series index
    if len(opf.Metadata.CalibreSeriesIndex) > 0 {
        if index, err := strconv.ParseFloat(opf.Metadata.CalibreSeriesIndex[0].Value, 32); err == nil {
            metadata.SeriesNumber = int32(index)
        }
    }

    return metadata, nil
}
```

**XML Namespaces**: Handle Dublin Core (`dc:`) and Calibre (`calibre:`) namespaces properly.

**Error Handling**:
- Malformed XML → Log warning, return `nil`
- Missing required fields → Use available fields
- Invalid date format → Log warning, skip field

### 3. Modified: extractMetadata()

**Location**: `internal/services/media_scanner.go` (line 712)

**Signature**: (Unchanged - existing function)
```go
func (s *MediaScanner) extractMetadata(path string) (*MediaMetadata, error)
```

**Purpose**: Extracts metadata from book files with sidecar-first approach.

**Modified Logic**:
```go
func (s *MediaScanner) extractMetadata(path string) (*MediaMetadata, error) {
    // NEW: Try Calibre sidecar first
    if metadata := s.extractCalibreSidecar(path); metadata != nil {
        fmt.Printf("Using Calibre metadata.opf for %s\n", path)

        // Try to find cover image for sidecar metadata
        coverPath := findSidecarCover(path)
        if coverPath != "" {
            metadata.CoverPath = s.getRelativePath(coverPath)
        }

        return metadata, nil
    }

    // EXISTING: Fallback to embedded metadata
    ext := strings.ToLower(filepath.Ext(path))

    switch ext {
    case ".epub":
        metadata, err := s.extractEPUBMetadata(path)
        // ... existing EPUB logic
    case ".pdf":
        return s.extractPDFMetadata(path)
    default:
        return &MediaMetadata{
            Title: strings.TrimSuffix(filepath.Base(path), ext),
        }, nil
    }
}
```

**Changes**:
- Add sidecar check at very beginning (before switch statement)
- If sidecar found → return immediately with sidecar metadata
- Otherwise → continue to existing embedded extraction logic
- **No call site changes needed** - all existing code continues to work

**Priority**: Sidecar → Embedded → Folder → Filename

**Note**: This is a SIMPLIFIED approach based on user feedback. Original plan called for a wrapper function, but directly modifying `extractMetadata()` is cleaner and requires less code.

## Database Schema Mapping

All OPF fields map to existing `media_items` columns:

| OPF Field | Database Column | Type | Conversion |
|-----------|-----------------|------|------------|
| `dc:title` | `title` | VARCHAR(255) | Direct |
| `dc:creator` | `author` | VARCHAR(255) | First author only |
| `dc:subject` | `tags` | TEXT[] | Array conversion |
| `dc:description` | `description` | TEXT | Direct |
| `dc:publisher` | `publisher` | VARCHAR(255) | Direct |
| `dc:date` | `date_published` | DATE | Parse ISO 8601 |
| `dc:language` | `language` | VARCHAR(10) | Store ISO code |
| `dc:identifier[ISBN]` | `isbn` | VARCHAR(13) | Normalize with utils |
| `dc:identifier[ASIN]` | `asin` | VARCHAR(20) | Direct |
| `dc:contributor` | `contributors` | TEXT[] | Array conversion |
| `calibre:series` | `series` | VARCHAR(255) | Direct |
| `calibre:series_index` | `series_number` | INTEGER | Parse float → int |

**No schema changes required** - all fields already exist.

## Normalization

Use existing normalization utilities:

```go
// Tags
metadata.Tags = utils.NormalizeTags(tags)

// Contributors
metadata.Contributors = utils.NormalizeContributors(contributors)

// ISBN
metadata.ISBN = utils.NormalizeISBNSafe(isbn)

// Search versions (auto-generated by database triggers)
// tags_search, contributors_search
```

## Testing

### Unit Tests

**File**: `internal/services/media_scanner_calibre_test.go`

```go
func TestParseCalibreMetadataOPF(t *testing.T) {
    tests := []struct {
        name       string
        opfContent string
        want       *MediaMetadata
        wantErr    bool
    }{
        {
            name: "Complete metadata.opf",
            opfContent: `<?xml version='1.0' encoding='utf-8'?>
<package xmlns="http://www.idpf.org/2007/opf" version="2.0">
  <metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
    <dc:title>Test Book</dc:title>
    <dc:creator>Test Author</dc:creator>
    <dc:subject>Fantasy</dc:subject>
    <dc:subject>Adventure</dc:subject>
    <dc:description>Test description</dc:description>
    <dc:publisher>Test Publisher</dc:publisher>
    <dc:date>2024-01-15</dc:date>
    <dc:language>en</dc:language>
    <dc:identifier opf:scheme="ISBN">978-0-123456-78-9</dc:identifier>
    <meta name="calibre:series" content="Test Series"/>
    <meta name="calibre:series_index" content="1"/>
  </metadata>
</package>`,
            want: &MediaMetadata{
                Title:       "Test Book",
                Author:      "Test Author",
                Tags:        []string{"Fantasy", "Adventure"},
                Description: "Test description",
                Publisher:   "Test Publisher",
                Series:      "Test Series",
                SeriesNumber: 1,
            },
            wantErr: false,
        },
        // ... more test cases
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            // Create temporary OPF file
            tmpDir := t.TempDir()
            opfPath := filepath.Join(tmpDir, "metadata.opf")
            if err := os.WriteFile(opfPath, []byte(tt.opfContent), 0644); err != nil {
                t.Fatal(err)
            }

            // Parse
            scanner := &MediaScanner{}
            got, err := scanner.parseCalibreMetadataOPF(opfPath)

            if (err != nil) != tt.wantErr {
                t.Errorf("parseCalibreMetadataOPF() error = %v, wantErr %v", err, tt.wantErr)
                return
            }

            // Verify fields
            if got.Title != tt.want.Title {
                t.Errorf("Title = %v, want %v", got.Title, tt.want.Title)
            }
            // ... more assertions
        })
    }
}
```

**Test Cases**:
1. Complete metadata.opf (all fields)
2. Minimal metadata.opf (title only)
3. Malformed XML
4. Missing metadata.opf (returns nil)
5. Multiple `<dc:subject>` tags
6. Multiple `<dc:identifier>` tags
7. Invalid date formats
8. Empty/whitespace values

### Integration Tests

**File**: `cmd/server/tests/calibre_integration_test.go`

```go
func TestCalibreLibraryScan(t *testing.T) {
    // Setup test server
    setup := setupTestServer(t)
    defer setup.Teardown()

    // Create test Calibre library structure
    tmpDir := t.TempDir()
    createTestCalibreLibrary(t, tmpDir)

    // Add library
    libraryID := createTestLibrary(t, setup.DB, tmpDir)

    // Scan library
    scanner := services.NewMediaScanner(setup.DB, setup.AdminID, []string{tmpDir})
    err := scanner.ScanFolders(context.Background())
    if err != nil {
        t.Fatalf("ScanFolders() error = %v", err)
    }

    // Verify imported books
    books, err := setup.DB.ListMediaItems(context.Background(), libraryID)
    if err != nil {
        t.Fatalf("ListMediaItems() error = %v", err)
    }

    // Assertions
    if len(books) != 3 {
        t.Errorf("Got %d books, want 3", len(books))
    }

    // Verify metadata from sidecar
    for _, book := range books {
        if book.Title == "" {
            t.Errorf("Book %s: Title should not be empty", book.ID)
        }
        if book.Series.Valid && book.SeriesNumber.Int32 == 0 {
            t.Errorf("Book %s: Series number should be set", book.ID)
        }
    }
}
```

**Test Scenarios**:
1. Scan Calibre library (with sidecars)
2. Scan non-Calibre library (no sidecars)
3. Mixed library (some with sidecars, some without)
4. Force rescan updates from sidecar changes
5. All file types (EPUB, PDF, CBZ, MP3)

## Performance Considerations

### Impact on Scan Performance

**Expected overhead**: +5-10% scan time increase

**Optimizations**:
1. **File system cache**: OS caches `metadata.opf` reads
2. **Early exit**: Return immediately if sidecar not found
3. **Parallel processing**: Sidecar extraction doesn't block other files
4. **Lazy parsing**: Only parse XML when sidecar exists

**Benchmarking**:
```go
func BenchmarkSidecarExtraction(b *testing.B) {
    scanner := &MediaScanner{}
    tmpDir := b.TempDir()
    createTestCalibreLibrary(b, tmpDir)

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _, err := scanner.extractMetadataFromCalibreSidecar(filepath.Join(tmpDir, "book.epub"))
        if err != nil {
            b.Fatal(err)
        }
    }
}
```

### Memory Usage

**XML parsing**: Small memory footprint (OPF files are typically <10KB)

**No leaks**: Ensure file handles are closed with `defer file.Close()`

## Error Handling

### Graceful Degradation

The scanner must **never fail** due to sidecar issues:

```go
// Pattern: Log error, continue to next source
if metadata, err := s.extractMetadataFromCalibreSidecar(path); err != nil {
    // Log warning
    fmt.Printf("Warning: failed to read Calibre sidecar: %v\n", err)
    // Continue to embedded extraction
    metadata = s.extractMetadata(path)
}
```

### Error Types

| Error | Action | Log Level |
|-------|--------|-----------|
| File not found | Continue to embedded | DEBUG |
| Permission denied | Continue to embedded | WARN |
| Malformed XML | Continue to embedded | WARN |
| Invalid date format | Skip field | WARN |
| Empty required field | Use available fields | INFO |
| UTF-8 decode error | Continue to embedded | ERROR |

### Logging

Use structured logging:

```go
fmt.Printf("Using Calibre metadata.opf for %s\n", path)
fmt.Printf("Warning: failed to parse metadata.opf: %v\n", err)
fmt.Printf("Info: No sidecar found, using embedded metadata for %s\n", path)
```

## Future Enhancements

### Phase 4 (Optional)

1. **Custom column support**: Parse Calibre custom columns
2. **Smart merge**: Configurable merge strategy
3. **Rating import**: Import Calibre rating as user's initial rating
4. **Library detection**: Auto-detect Calibre library structure
5. **Incremental sync**: Only re-scan changed sidecar files

### Configuration Option

```go
type ScannerConfig struct {
    PreferCalibreSidecar bool   // Default: true
    CalibreMergeStrategy string // "sidecar-first" | "smart-merge" | "embedded-first"
}
```

## References

- [EPUB 3.0 Spec](https://www.idpf.org/epub/30/spec/epub30-publications.html)
- [Dublin Core](https://www.dublincore.org/specifications/dces/1998-08-13/)
- [Calibre Manual](https://manual.calibre-ebook.com/)
- [Go XML Encoding](https://pkg.go.dev/encoding/xml)

## Implementation Note: Simplified Approach

**Original Plan**: Create a new wrapper function `extractMetadataWithSidecar()` that calls the existing `extractMetadata()`.

**Revised Plan** (based on user feedback): Modify `extractMetadata()` directly to check for sidecar files at the beginning of the function.

**Benefits of Simplified Approach**:
- **Less code**: ~150 lines vs. ~250 lines
- **No call site changes**: All existing code continues to work without modification
- **Clearer flow**: Single entry point for metadata extraction
- **Better testability**: Test `extractMetadata()` with/without sidecar in one place
- **Easier to maintain**: All metadata extraction logic in one function

**What Changed**:
- No new wrapper function needed
- Only 2 new functions: `extractCalibreSidecar()` and `parseCalibreMetadataOPF()`
- Single modification to existing `extractMetadata()` function
- Simpler architecture, easier to understand

This is a great example of how user feedback can improve implementation design!

## See Also

- [User Guide: Calibre Integration](../user/calibre-integration.md)
- The original implementation plan file was removed from the repo when this page absorbed its content.
- The scanner implementation lives in `internal/services/` (not hyperlinked — the docs site serves only docs).