bookhoard/CALIBRE_OPF_IMPLEMENTATION.md

Calibre metadata.opf Support - Implementation Plan

Overview

This document outlines the implementation of Calibre metadata.opf sidecar file support for Bookhoard's media scanner. The scanner will use a sidecar-first approach - if a Calibre metadata.opf file exists alongside a book file, it will take precedence over embedded metadata.

Motivation

Calibre is the most popular ebook management software, with millions of users. When users manage their libraries in Calibre:

1. They curate metadata - Adding series information, tags, ratings, custom covers
2. They fetch from online sources - Google Books, Amazon, Goodreads (better than publisher metadata)
3. They convert/edit books - The sidecar reflects the current, correct state
4. They organize in folders - Often using Calibre's folder structure conventions

By supporting Calibre metadata.opf sidecar files, Bookhoard can:

• Respect users' curation work
• Import richer, more accurate metadata
• Provide seamless integration with existing Calibre libraries
• Support all file types (EPUB, PDF, comics, etc.) that Calibre manages

Requirements

Functional Requirements

• Scanner detects metadata.opf files in the same directory as book files
• Parses Dublin Core metadata (dc:* namespace) and Calibre-specific fields (meta name="calibre:*")
• Uses sidecar metadata before embedded metadata (sidecar-first approach)
• Works for all library types (ebook, audiobook, comic, etc.)
• Works for all file types (EPUB, PDF, CBZ, CBR, MP3, M4B, etc.)
• Handles missing or malformed sidecar files gracefully
• Maps all OPF fields to existing media_items database schema
• Maintains backward compatibility (no sidecar = existing behavior)

Non-Functional Requirements

• No breaking changes to existing scanner functionality
• No database schema changes required (all fields already exist)
• Follows existing code patterns and style
• Comprehensive test coverage
• Performance: Minimal impact on scan performance
• Error handling: Graceful degradation on malformed XML

Database Schema Mapping

All required fields already exist in the media_items table. No schema changes needed.

Dublin Core Fields (dc: namespace)

OPF Field	Database Column	Type	Notes
dc:title	title	VARCHAR(255)	Required
dc:creator	author	VARCHAR(255)	First author only
dc:subject	tags	TEXT[]	Tags/keywords (multiple)
dc:description	description	TEXT	Book summary
dc:publisher	publisher	VARCHAR(255)	Publisher name
dc:date	date_published	DATE	Publication date
dc:language	language	VARCHAR(10)	ISO 639-1 code (en, es, fr)
dc:identifier (ISBN)	isbn	VARCHAR(13)	Check opf:scheme="ISBN"
dc:identifier (ASIN)	asin	VARCHAR(20)	Check opf:scheme="ASIN"
dc:contributor	contributors	TEXT[]	Multiple contributors
dc:rights	(not stored)	-	Copyright info (ignored)
dc:source	(not stored)	-	Source publication (ignored)

Calibre-Specific Fields (meta name="calibre:*")

OPF Field	Database Column	Type	Notes
calibre:series	series	VARCHAR(255)	Series name
calibre:series_index	series_number	INTEGER	Position in series
calibre:rating	(use media_ratings)	INTEGER	0-5 scale, stored per-user
calibre:title_sort	(not stored)	-	Sortable title (ignored)
calibre:author_sort	(not stored)	-	Sortable author (ignored)
calibre:timestamp	created_at	TIMESTAMP	When added to Calibre

Special Cases

1. Multiple authors: Store first author in author, suggest adding authors[] array in future
2. Multiple identifiers: Extract ISBN and ASIN if present, ignore others
3. Calibre rating: Not imported (ratings are per-user in Bookhoard)
4. Sort fields: Ignored (Bookhoard has its own sorting logic)
5. Timestamp: Use for created_at if more recent than file modification time

OPF Format Reference

Standard Dublin Core Elements

<dc:title>Book Title</dc:title>
<dc:creator>Author Name</dc:creator>
<dc:subject>Fantasy</dc:subject>          <!-- Tag -->
<dc:subject>Adventure</dc:subject>         <!-- Another tag -->
<dc:description>Book summary...</dc:description>
<dc:publisher>Publisher Name</dc:publisher>
<dc:date>2024-01-15</dc:date>
<dc:language>en</dc:language>
<dc:identifier opf:scheme="ISBN">978-0-123456-78-9</dc:identifier>
<dc:identifier opf:scheme="ASIN">B08XXXXX</dc:identifier>
<dc:contributor>Illustrator Name</dc:contributor>

Calibre-Specific Meta Tags

<meta name="calibre:series" content="Series Name"/>
<meta name="calibre:series_index" content="3"/>
<meta name="calibre:rating" content="4"/>
<meta name="calibre:title_sort" content="Book Title"/>
<meta name="calibre:author_sort" content="Author, Name"/>

Example Complete metadata.opf

<?xml version='1.0' encoding='utf-8'?>
<package xmlns="http://www.idpf.org/2007/opf" version="2.0" unique-identifier="uuid_id">
  <metadata xmlns:opf="http://www.idpf.org/2007/opf"
            xmlns:calibre="http://calibre.kovidgoyal.net/2009/metadata"
            xmlns:dc="http://purl.org/dc/elements/1.1/">
    <dc:title>The Fellowship of the Ring</dc:title>
    <dc:creator opf:role="aut">J.R.R. Tolkien</dc:creator>
    <dc:identifier opf:scheme="ISBN">978-0-618-00222-0</dc:identifier>
    <dc:identifier opf:scheme="calibre">12345678-1234-1234-1234-123456789012</dc:identifier>
    <dc:language>en</dc:language>
    <dc:date>1954-07-29</dc:date>
    <dc:publisher>HarperCollins</dc:publisher>
    <dc:description>In a sleepy village in the Shire...</dc:description>

    <!-- Tags -->
    <dc:subject>Fantasy</dc:subject>
    <dc:subject>Adventure</dc:subject>
    <dc:subject>Classics</dc:subject>

    <!-- Contributors -->
    <dc:contributor>Alan Lee (illustrator)</dc:contributor>

    <!-- Calibre-specific fields -->
    <meta name="calibre:series" content="The Lord of the Rings"/>
    <meta name="calibre:series_index" content="1"/>
    <meta name="calibre:rating" content="5"/>
    <meta name="calibre:title_sort" content="Fellowship of the Ring"/>
  </metadata>
</package>

Implementation Design

Architecture

Scanner Pipeline (EXISTING)
↓
1. Detect book file
2. Check if already in database
3. Extract metadata from file (EPUB/PDF/Comic)
4. Extract folder structure metadata
5. Normalize and store

Simplified Approach: Modify existing extractMetadata() to check for Calibre sidecar first

Scanner Pipeline (NEW)
↓
1. Detect book file
2. Check if already in database
3. Extract metadata (MODIFIED FUNCTION)
   ├── Check for Calibre metadata.opf sidecar ← NEW
   │   └─ If found → use sidecar metadata
   └── Fallback to embedded metadata (existing)
       ├── EPUB → extractEPUBMetadata()
       ├── PDF → extractPDFMetadata()
       └── Comic → extractComicMetadata()
4. Extract folder structure metadata (existing)
5. Normalize and store (existing)

Code Changes

SIMPLIFIED APPROACH (thanks to user feedback): Modify existing extractMetadata() function instead of creating wrapper.

File: internal/services/media_scanner.go

New Functions (to add):

1. extractCalibreSidecar(path string) *MediaMetadata

   • Checks for metadata.opf in same directory as book file
   • Calls parseCalibreMetadataOPF() if found
   • Returns nil if no sidecar exists (not an error)
   • Returns nil on parse errors (graceful fallback)

2. parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error)

   • Opens and parses XML file
   • Extracts Dublin Core fields (dc:*)
   • Extracts Calibre-specific fields (meta name="calibre:*")
   • Maps to MediaMetadata struct
   • Handles errors gracefully (malformed XML, missing fields)

Modified Function:

1. extractMetadata(path string) (*MediaMetadata, error) at line 712
   • Add sidecar check at very beginning (before switch statement)
   • If sidecar found → return sidecar metadata immediately
   • Otherwise → continue to existing embedded extraction logic
   • No call site changes needed - all existing code continues to work

New Struct: CalibreOPFMetadata

// Intermediate struct for parsed OPF data
type CalibreOPFMetadata struct {
    Title          string
    Authors        []string
    Tags           []string
    Description    string
    Publisher      string
    PublishDate    *time.Time
    Language       string
    ISBN           string
    ASIN           string
    UUID           string
    Contributors   []string
    Series         string
    SeriesIndex    *float64
    Rating         *int32
    Timestamp      *time.Time
}

Metadata Priority (Sidecar-First)

Priority Order:
1. Calibre metadata.opf sidecar ← NEW (highest priority)
2. Embedded file metadata (EPUB OPF, PDF info, ComicInfo.xml)
3. Folder structure parsing
4. Filename parsing (last resort)

Rationale: If a user has a metadata.opf file, they're actively managing their library in Calibre. The sidecar represents their curated "source of truth."

Error Handling

1. Sidecar file not found → Return nil, continue to embedded metadata
2. Malformed XML → Log warning, continue to embedded metadata
3. Missing required fields → Use available fields, continue
4. Invalid date format → Log warning, skip field
5. File permission errors → Log error, continue to embedded metadata

Principle: Graceful degradation - never fail a scan due to sidecar issues

Testing Strategy

Unit Tests

File: internal/services/media_scanner_calibre_test.go

• Test parsing complete metadata.opf with all fields
• Test parsing minimal metadata.opf (title only)
• Test parsing malformed XML (graceful failure)
• Test missing metadata.opf (returns nil, not error)
• Test multiple <dc:subject> tags (tags array)
• Test multiple <dc:identifier> tags (extract ISBN/ASIN)
• Test Calibre-specific fields (series, series_index)
• Test invalid date formats
• Test empty/whitespace values

Integration Tests

File: cmd/server/tests/calibre_integration_test.go

• Test scanning directory with Calibre library structure
• Test scanning directory without sidecar files (existing behavior)
• Test mixed: some books with sidecar, some without
• Test force rescan updates from sidecar changes
• Test all file types: EPUB, PDF, CBZ, MP3, M4B
• Test all library types: ebook, audiobook, comic

Test Data

Create testdata/calibre/ with sample files:

• Complete Book/metadata.opf - All fields populated
• Minimal Book/metadata.opf - Only required fields
• Malformed Book/metadata.opf - Invalid XML
• No sidecar files (rely on existing test data)

Implementation Steps

Phase 1: Core Functionality (REQUIRED)

• Step 1.1: Add CalibreOPFMetadata struct (intermediate parsing struct)
• Step 1.2: Implement parseCalibreMetadataOPF() function (~120 lines)
• Step 1.3: Implement extractCalibreSidecar() function (~20 lines)
• Step 1.4: Modify extractMetadata() function (~10 lines)
  • Add sidecar check at beginning of function (line ~712)
  • Return sidecar metadata if found
  • Otherwise continue to existing switch statement
• Step 1.5: Add unit tests for OPF parsing
• Step 1.6: Test with real Calibre library

Total new code: ~150 lines (simplified from original ~250 lines thanks to direct modification of extractMetadata())

Phase 2: Documentation (REQUIRED)

• Step 2.1: Create user documentation (docs/user/calibre-support.md)
• Step 2.2: Update developer documentation (docs/development/calibre-opf-implementation.md)
• Step 2.3: Update scanner API docs if needed
• Step 2.4: Add examples to documentation

Phase 3: Testing & Polish (REQUIRED)

• Step 3.1: Add integration tests
• Step 3.2: Test with various Calibre library configurations
• Step 3.3: Performance testing (ensure no regression)
• Step 3.4: Error handling review
• Step 3.5: Code review and refinement

Phase 4: Future Enhancements (OPTIONAL)

• Support Calibre author_link metadata
• Support Calibre custom columns
• Import Calibre ratings as user's initial rating
• Detect Calibre library structure patterns
• Configuration option for merge strategy (sidecar-first vs. smart-merge)

Backward Compatibility

No breaking changes. Existing behavior is preserved:

• Libraries without metadata.opf files → Existing behavior (embedded metadata)
• Scanning logic unchanged → Single function modification, no call site changes
• Database unchanged → All fields already exist
• API unchanged → No new endpoints or response fields
• No wrapper function needed → Simpler, cleaner implementation

Success Criteria

• All existing tests pass
• New unit tests pass (90%+ coverage)
• New integration tests pass
• Scans real Calibre library successfully
• Scans non-Calibre library without errors
• No performance regression (scan time +5% max)
• Documentation complete and accurate
• Code review approved

Open Questions

1. Q: Should we import Calibre ratings? A: No - ratings are per-user in Bookhoard. Calibre rating is the owner's opinion, not necessarily the current user's.

2. Q: Should we support Calibre custom columns? A: Not in initial implementation. Custom columns are user-defined and would require dynamic schema or JSON storage. Future enhancement.

3. Q: How to handle multiple authors? A: Store first author in author field (existing behavior). Future enhancement could add authors[] array.

4. Q: Should we follow Calibre folder structure? A: Already implemented! extractFolderStructureMetadata() handles Calibre patterns.

References

• EPUB Publications 3.0 Specification
• Dublin Core Metadata Element Set
• Calibre Manual - Editing Metadata
• Calibre Source Code

Appendix: Decision Log

Decision 1: Sidecar-First Approach

Date: 2026-03-26 Decision: Use sidecar-first (not smart-merge) Rationale:

• Users with Calibre sidecars actively curate their metadata
• Sidecar metadata is typically richer and more accurate
• Simple implementation, predictable behavior
• Easy to test and debug
• Future enhancement can add configurable merge strategy

Decision 2: No Database Schema Changes

Date: 2026-03-26 Decision: Use existing schema only Rationale:

• All required fields already exist in media_items table
• Pre-production application (no legacy data to migrate)
• Simpler implementation
• No backward compatibility concerns

Decision 3: All Library Types Supported

Date: 2026-03-26 Decision: Support OPF for all libraries, not just ebooks Rationale:

• Calibre supports all file types
• Users may organize audiobooks/comics in Calibre
• Consistent behavior across library types
• No additional complexity

Decision 4: Simplified Implementation (REVISED)

Date: 2026-03-26 Decision: Modify extractMetadata() directly instead of creating wrapper function Rationale:

• User feedback: Suggested simpler approach with if statement check
• Less code: ~150 lines vs. ~250 lines
• No call site changes: All existing code continues to work
• Clearer flow: Single entry point for metadata extraction
• Better testability: Test extractMetadata() with/without sidecar
• Easier to maintain: All metadata logic in one place

Original plan:

// Wrapper function approach
extractMetadataWithSidecar() {
    if sidecar exists → return sidecar
    return extractMetadata()
}

Revised plan (SIMPLER):

// Direct modification
extractMetadata() {
    if sidecar exists → return sidecar  // ← NEW: Just add this at top
    // existing switch statement continues...
}

This simplification reduces code complexity and makes the implementation cleaner and easier to understand.