bookhoard/docs/development/calibre-opf-implementation.md

Calibre metadata.opf Implementation

This document provides technical details for implementing Calibre metadata.opf sidecar file support in Bookhoard's media scanner.

Overview

The media scanner detects and imports metadata from Calibre's metadata.opf sidecar files using a sidecar-first approach. If a metadata.opf file exists alongside a book file, its metadata takes precedence over embedded file metadata.

Architecture

Scanner Pipeline

func (s *MediaScanner) processMediaFile(ctx context.Context, path string) (bool, error) {
    // 1. Get file info
    // 2. Find library
    // 3. Check if media item exists
    // 4. EXTRACT METADATA (NEW: sidecar-first)
    metadata := s.extractMetadataWithSidecar(path)
    // 5. Extract hash info
    // 6. Extract comic metadata (if applicable)
    // 7. Extract folder structure metadata
    // 8. Normalize metadata
    // 9. Create or update in database
}

Metadata Extraction Flow

extractMetadataWithSidecar()
    ↓
1. Try Calibre metadata.opf sidecar
   ├─ Found → parseCalibreMetadataOPF()
   └─ Not found → continue
    ↓
2. Extract embedded metadata
   ├─ EPUB → extractEPUBMetadata()
   ├─ PDF → extractPDFMetadata()
   └─ Comic → extractComicMetadata()
    ↓
3. Extract folder structure metadata
   └─ extractFolderStructureMetadata()
    ↓
4. Return merged MediaMetadata

Data Structures

CalibreOPFMetadata

Intermediate struct for parsed OPF data:

// CalibreOPFMetadata represents parsed metadata from Calibre metadata.opf
type CalibreOPFMetadata struct {
    Title          string
    Authors        []string
    Tags           []string
    Description    string
    Publisher      string
    PublishDate    *time.Time
    Language       string
    ISBN           string
    ASIN           string
    UUID           string
    Contributors   []string
    Series         string
    SeriesIndex    *float64
    Rating         *int32
    Timestamp      *time.Time
}

MediaMetadata

Existing struct (already defined in media_scanner.go):

// MediaMetadata represents extracted book metadata
type MediaMetadata struct {
    Title          string
    Author         string
    ISBN           string
    Description    string
    Publisher      string
    Series         string
    SeriesNumber   int32
    Tags           []string
    Contributors   []string
    PublishDate    time.Time
    ASIN           string
    // ... other fields
}

Implementation Functions

1. extractCalibreSidecar()

Location: internal/services/media_scanner.go

Signature:

func (s *MediaScanner) extractCalibreSidecar(path string) *MediaMetadata

Purpose: Checks for metadata.opf in the same directory as the book file.

Logic:

func (s *MediaScanner) extractCalibreSidecar(path string) *MediaMetadata {
    // Get directory of media file
    dir := filepath.Dir(path)
    opfPath := filepath.Join(dir, "metadata.opf")

    // Check if sidecar exists
    if _, err := os.Stat(opfPath); os.IsNotExist(err) {
        return nil  // No sidecar, not an error
    }

    // Parse sidecar
    metadata, err := s.parseCalibreMetadataOPF(opfPath)
    if err != nil {
        fmt.Printf("Warning: failed to parse Calibre metadata.opf: %v\n", err)
        return nil  // Parsing failed, fall back to embedded
    }

    return metadata
}

Error Handling:

• File not found → Return nil (not an error)
• Permission denied → Log warning, return nil
• Parse errors → Log warning, return nil (graceful fallback)

Note: Simplified from original plan - no wrapper function needed

2. parseCalibreMetadataOPF()

Location: internal/services/media_scanner.go

Signature:

func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error)

Purpose: Parses XML file and extracts Dublin Core + Calibre-specific fields.

Logic:

func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error) {
    // Open file
    file, err := os.Open(opfPath)
    if err != nil {
        return nil, fmt.Errorf("failed to open metadata.opf: %v", err)
    }
    defer file.Close()

    // Parse XML
    var opf struct {
        XMLName  xml.Name `xml:"package"`
        Metadata struct {
            XMLName   xml.Name `xml:"metadata"`
            Titles    []string `xml:"dc:title"`
            Creators []string `xml:"dc:creator"`
            Subjects  []string `xml:"dc:subject"`
            Desc      []string `xml:"dc:description"`
            Publisher []string `xml:"dc:publisher"`
            Dates     []string `xml:"dc:date"`
            Language  []string `xml:"dc:language"`
            Identifiers []struct {
                Scheme string `xml:"opf:scheme,attr"`
                Value  string `xml:",chardata"`
            } `xml:"dc:identifier"`
            Contributors []string `xml:"dc:contributor"`
            CalibreSeries []struct {
                Name  string `xml:"content,attr"`
            } `xml:"meta[name='calibre:series']"`
            CalibreSeriesIndex []struct {
                Value  string `xml:"content,attr"`
            } `xml:"meta[name='calibre:series_index']"`
        } `xml:"metadata"`
    }

    if err := xml.NewDecoder(file).Decode(&opf); err != nil {
        return nil, fmt.Errorf("failed to parse metadata.opf XML: %v", err)
    }

    // Map to MediaMetadata
    metadata := &MediaMetadata{}

    // Title (required)
    if len(opf.Metadata.Titles) > 0 {
        metadata.Title = opf.Metadata.Titles[0]
    }

    // Author (first creator)
    if len(opf.Metadata.Creators) > 0 {
        metadata.Author = opf.Metadata.Creators[0]
    }

    // Tags (all subjects)
    if len(opf.Metadata.Subjects) > 0 {
        metadata.Tags = utils.NormalizeTags(opf.Metadata.Subjects)
    }

    // Description
    if len(opf.Metadata.Desc) > 0 {
        metadata.Description = opf.Metadata.Desc[0]
    }

    // Publisher
    if len(opf.Metadata.Publisher) > 0 {
        metadata.Publisher = opf.Metadata.Publisher[0]
    }

    // Publish date
    if len(opf.Metadata.Dates) > 0 {
        if date, err := time.Parse("2006-01-02", opf.Metadata.Dates[0]); err == nil {
            metadata.PublishDate = date
        }
    }

    // Language
    if len(opf.Metadata.Language) > 0 {
        // Store language (already in schema)
        // Note: MediaMetadata doesn't have Language field yet
        // Future enhancement: add Language field
    }

    // Identifiers (ISBN, ASIN)
    for _, id := range opf.Metadata.Identifiers {
        switch strings.ToUpper(id.Scheme) {
        case "ISBN":
            metadata.ISBN = utils.NormalizeISBNSafe(id.Value)
        case "ASIN":
            metadata.ASIN = id.Value
        case "UUID", "CALIBRE":
            // Store UUID in hash info, not metadata
            // Will be extracted by extractHashInfo()
        }
    }

    // Contributors
    if len(opf.Metadata.Contributors) > 0 {
        metadata.Contributors = utils.NormalizeContributors(opf.Metadata.Contributors)
    }

    // Series
    if len(opf.Metadata.CalibreSeries) > 0 {
        metadata.Series = opf.Metadata.CalibreSeries[0].Name
    }

    // Series index
    if len(opf.Metadata.CalibreSeriesIndex) > 0 {
        if index, err := strconv.ParseFloat(opf.Metadata.CalibreSeriesIndex[0].Value, 32); err == nil {
            metadata.SeriesNumber = int32(index)
        }
    }

    return metadata, nil
}

XML Namespaces: Handle Dublin Core (dc:) and Calibre (calibre:) namespaces properly.

Error Handling:

• Malformed XML → Log warning, return nil
• Missing required fields → Use available fields
• Invalid date format → Log warning, skip field

3. Modified: extractMetadata()

Location: internal/services/media_scanner.go (line 712)

Signature: (Unchanged - existing function)

func (s *MediaScanner) extractMetadata(path string) (*MediaMetadata, error)

Purpose: Extracts metadata from book files with sidecar-first approach.

Modified Logic:

func (s *MediaScanner) extractMetadata(path string) (*MediaMetadata, error) {
    // NEW: Try Calibre sidecar first
    if metadata := s.extractCalibreSidecar(path); metadata != nil {
        fmt.Printf("Using Calibre metadata.opf for %s\n", path)

        // Try to find cover image for sidecar metadata
        coverPath := findSidecarCover(path)
        if coverPath != "" {
            metadata.CoverPath = s.getRelativePath(coverPath)
        }

        return metadata, nil
    }

    // EXISTING: Fallback to embedded metadata
    ext := strings.ToLower(filepath.Ext(path))

    switch ext {
    case ".epub":
        metadata, err := s.extractEPUBMetadata(path)
        // ... existing EPUB logic
    case ".pdf":
        return s.extractPDFMetadata(path)
    default:
        return &MediaMetadata{
            Title: strings.TrimSuffix(filepath.Base(path), ext),
        }, nil
    }
}

Changes:

• Add sidecar check at very beginning (before switch statement)
• If sidecar found → return immediately with sidecar metadata
• Otherwise → continue to existing embedded extraction logic
• No call site changes needed - all existing code continues to work

Priority: Sidecar → Embedded → Folder → Filename

Note: This is a SIMPLIFIED approach based on user feedback. Original plan called for a wrapper function, but directly modifying extractMetadata() is cleaner and requires less code.

Database Schema Mapping

All OPF fields map to existing media_items columns:

OPF Field	Database Column	Type	Conversion
dc:title	title	VARCHAR(255)	Direct
dc:creator	author	VARCHAR(255)	First author only
dc:subject	tags	TEXT[]	Array conversion
dc:description	description	TEXT	Direct
dc:publisher	publisher	VARCHAR(255)	Direct
dc:date	date_published	DATE	Parse ISO 8601
dc:language	language	VARCHAR(10)	Store ISO code
dc:identifier[ISBN]	isbn	VARCHAR(13)	Normalize with utils
dc:identifier[ASIN]	asin	VARCHAR(20)	Direct
dc:contributor	contributors	TEXT[]	Array conversion
calibre:series	series	VARCHAR(255)	Direct
calibre:series_index	series_number	INTEGER	Parse float → int

No schema changes required - all fields already exist.

Normalization

Use existing normalization utilities:

// Tags
metadata.Tags = utils.NormalizeTags(tags)

// Contributors
metadata.Contributors = utils.NormalizeContributors(contributors)

// ISBN
metadata.ISBN = utils.NormalizeISBNSafe(isbn)

// Search versions (auto-generated by database triggers)
// tags_search, contributors_search

Testing

Unit Tests

File: internal/services/media_scanner_calibre_test.go

func TestParseCalibreMetadataOPF(t *testing.T) {
    tests := []struct {
        name       string
        opfContent string
        want       *MediaMetadata
        wantErr    bool
    }{
        {
            name: "Complete metadata.opf",
            opfContent: `<?xml version='1.0' encoding='utf-8'?>
<package xmlns="http://www.idpf.org/2007/opf" version="2.0">
  <metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
    <dc:title>Test Book</dc:title>
    <dc:creator>Test Author</dc:creator>
    <dc:subject>Fantasy</dc:subject>
    <dc:subject>Adventure</dc:subject>
    <dc:description>Test description</dc:description>
    <dc:publisher>Test Publisher</dc:publisher>
    <dc:date>2024-01-15</dc:date>
    <dc:language>en</dc:language>
    <dc:identifier opf:scheme="ISBN">978-0-123456-78-9</dc:identifier>
    <meta name="calibre:series" content="Test Series"/>
    <meta name="calibre:series_index" content="1"/>
  </metadata>
</package>`,
            want: &MediaMetadata{
                Title:       "Test Book",
                Author:      "Test Author",
                Tags:        []string{"Fantasy", "Adventure"},
                Description: "Test description",
                Publisher:   "Test Publisher",
                Series:      "Test Series",
                SeriesNumber: 1,
            },
            wantErr: false,
        },
        // ... more test cases
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            // Create temporary OPF file
            tmpDir := t.TempDir()
            opfPath := filepath.Join(tmpDir, "metadata.opf")
            if err := os.WriteFile(opfPath, []byte(tt.opfContent), 0644); err != nil {
                t.Fatal(err)
            }

            // Parse
            scanner := &MediaScanner{}
            got, err := scanner.parseCalibreMetadataOPF(opfPath)

            if (err != nil) != tt.wantErr {
                t.Errorf("parseCalibreMetadataOPF() error = %v, wantErr %v", err, tt.wantErr)
                return
            }

            // Verify fields
            if got.Title != tt.want.Title {
                t.Errorf("Title = %v, want %v", got.Title, tt.want.Title)
            }
            // ... more assertions
        })
    }
}

Test Cases:

1. Complete metadata.opf (all fields)
2. Minimal metadata.opf (title only)
3. Malformed XML
4. Missing metadata.opf (returns nil)
5. Multiple <dc:subject> tags
6. Multiple <dc:identifier> tags
7. Invalid date formats
8. Empty/whitespace values

Integration Tests

File: cmd/server/tests/calibre_integration_test.go

func TestCalibreLibraryScan(t *testing.T) {
    // Setup test server
    setup := setupTestServer(t)
    defer setup.Teardown()

    // Create test Calibre library structure
    tmpDir := t.TempDir()
    createTestCalibreLibrary(t, tmpDir)

    // Add library
    libraryID := createTestLibrary(t, setup.DB, tmpDir)

    // Scan library
    scanner := services.NewMediaScanner(setup.DB, setup.AdminID, []string{tmpDir})
    err := scanner.ScanFolders(context.Background())
    if err != nil {
        t.Fatalf("ScanFolders() error = %v", err)
    }

    // Verify imported books
    books, err := setup.DB.ListMediaItems(context.Background(), libraryID)
    if err != nil {
        t.Fatalf("ListMediaItems() error = %v", err)
    }

    // Assertions
    if len(books) != 3 {
        t.Errorf("Got %d books, want 3", len(books))
    }

    // Verify metadata from sidecar
    for _, book := range books {
        if book.Title == "" {
            t.Errorf("Book %s: Title should not be empty", book.ID)
        }
        if book.Series.Valid && book.SeriesNumber.Int32 == 0 {
            t.Errorf("Book %s: Series number should be set", book.ID)
        }
    }
}

Test Scenarios:

1. Scan Calibre library (with sidecars)
2. Scan non-Calibre library (no sidecars)
3. Mixed library (some with sidecars, some without)
4. Force rescan updates from sidecar changes
5. All file types (EPUB, PDF, CBZ, MP3)

Performance Considerations

Impact on Scan Performance

Expected overhead: +5-10% scan time increase

Optimizations:

1. File system cache: OS caches metadata.opf reads
2. Early exit: Return immediately if sidecar not found
3. Parallel processing: Sidecar extraction doesn't block other files
4. Lazy parsing: Only parse XML when sidecar exists

Benchmarking:

func BenchmarkSidecarExtraction(b *testing.B) {
    scanner := &MediaScanner{}
    tmpDir := b.TempDir()
    createTestCalibreLibrary(b, tmpDir)

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _, err := scanner.extractMetadataFromCalibreSidecar(filepath.Join(tmpDir, "book.epub"))
        if err != nil {
            b.Fatal(err)
        }
    }
}

Memory Usage

XML parsing: Small memory footprint (OPF files are typically <10KB)

No leaks: Ensure file handles are closed with defer file.Close()

Error Handling

Graceful Degradation

The scanner must never fail due to sidecar issues:

// Pattern: Log error, continue to next source
if metadata, err := s.extractMetadataFromCalibreSidecar(path); err != nil {
    // Log warning
    fmt.Printf("Warning: failed to read Calibre sidecar: %v\n", err)
    // Continue to embedded extraction
    metadata = s.extractMetadata(path)
}

Error Types

Error	Action	Log Level
File not found	Continue to embedded	DEBUG
Permission denied	Continue to embedded	WARN
Malformed XML	Continue to embedded	WARN
Invalid date format	Skip field	WARN
Empty required field	Use available fields	INFO
UTF-8 decode error	Continue to embedded	ERROR

Logging

Use structured logging:

fmt.Printf("Using Calibre metadata.opf for %s\n", path)
fmt.Printf("Warning: failed to parse metadata.opf: %v\n", err)
fmt.Printf("Info: No sidecar found, using embedded metadata for %s\n", path)

Future Enhancements

Phase 4 (Optional)

1. Custom column support: Parse Calibre custom columns
2. Smart merge: Configurable merge strategy
3. Rating import: Import Calibre rating as user's initial rating
4. Library detection: Auto-detect Calibre library structure
5. Incremental sync: Only re-scan changed sidecar files

Configuration Option

type ScannerConfig struct {
    PreferCalibreSidecar bool   // Default: true
    CalibreMergeStrategy string // "sidecar-first" | "smart-merge" | "embedded-first"
}

References

• EPUB 3.0 Spec
• Dublin Core
• Calibre Manual
• Go XML Encoding

Implementation Note: Simplified Approach

Original Plan: Create a new wrapper function extractMetadataWithSidecar() that calls the existing extractMetadata().

Revised Plan (based on user feedback): Modify extractMetadata() directly to check for sidecar files at the beginning of the function.

Benefits of Simplified Approach:

• Less code: ~150 lines vs. ~250 lines
• No call site changes: All existing code continues to work without modification
• Clearer flow: Single entry point for metadata extraction
• Better testability: Test extractMetadata() with/without sidecar in one place
• Easier to maintain: All metadata extraction logic in one function

What Changed:

• No new wrapper function needed
• Only 2 new functions: extractCalibreSidecar() and parseCalibreMetadataOPF()
• Single modification to existing extractMetadata() function
• Simpler architecture, easier to understand

This is a great example of how user feedback can improve implementation design!

See Also

• User Guide: Calibre Integration
• Implementation Plan
• Scanner Source Code