bookhoard/CALIBRE_OPF_IMPLEMENTATION.md

# Calibre metadata.opf Support - Implementation Plan

## Overview

This document outlines the implementation of **Calibre metadata.opf sidecar file support** for Bookhoard's media scanner. The scanner will use a **sidecar-first approach** - if a Calibre `metadata.opf` file exists alongside a book file, it will take precedence over embedded metadata.

## Motivation

Calibre is the most popular ebook management software, with millions of users. When users manage their libraries in Calibre:

1. **They curate metadata** - Adding series information, tags, ratings, custom covers
2. **They fetch from online sources** - Google Books, Amazon, Goodreads (better than publisher metadata)
3. **They convert/edit books** - The sidecar reflects the current, correct state
4. **They organize in folders** - Often using Calibre's folder structure conventions

By supporting Calibre `metadata.opf` sidecar files, Bookhoard can:
- Respect users' curation work
- Import richer, more accurate metadata
- Provide seamless integration with existing Calibre libraries
- Support all file types (EPUB, PDF, comics, etc.) that Calibre manages

## Requirements

### Functional Requirements

- [ ] Scanner detects `metadata.opf` files in the same directory as book files
- [ ] Parses Dublin Core metadata (`dc:*` namespace) and Calibre-specific fields (`meta name="calibre:*"`)
- [ ] Uses sidecar metadata **before** embedded metadata (sidecar-first approach)
- [ ] Works for all library types (ebook, audiobook, comic, etc.)
- [ ] Works for all file types (EPUB, PDF, CBZ, CBR, MP3, M4B, etc.)
- [ ] Handles missing or malformed sidecar files gracefully
- [ ] Maps all OPF fields to existing `media_items` database schema
- [ ] Maintains backward compatibility (no sidecar = existing behavior)

### Non-Functional Requirements

- [ ] No breaking changes to existing scanner functionality
- [ ] No database schema changes required (all fields already exist)
- [ ] Follows existing code patterns and style
- [ ] Comprehensive test coverage
- [ ] Performance: Minimal impact on scan performance
- [ ] Error handling: Graceful degradation on malformed XML

## Database Schema Mapping

All required fields **already exist** in the `media_items` table. No schema changes needed.

### Dublin Core Fields (dc: namespace)

| OPF Field | Database Column | Type | Notes |
|-----------|-----------------|------|-------|
| `dc:title` | `title` | VARCHAR(255) | Required |
| `dc:creator` | `author` | VARCHAR(255) | First author only |
| `dc:subject` | `tags` | TEXT[] | **Tags/keywords** (multiple) |
| `dc:description` | `description` | TEXT | Book summary |
| `dc:publisher` | `publisher` | VARCHAR(255) | Publisher name |
| `dc:date` | `date_published` | DATE | Publication date |
| `dc:language` | `language` | VARCHAR(10) | ISO 639-1 code (en, es, fr) |
| `dc:identifier` (ISBN) | `isbn` | VARCHAR(13) | Check `opf:scheme="ISBN"` |
| `dc:identifier` (ASIN) | `asin` | VARCHAR(20) | Check `opf:scheme="ASIN"` |
| `dc:contributor` | `contributors` | TEXT[] | Multiple contributors |
| `dc:rights` | *(not stored)* | - | Copyright info (ignored) |
| `dc:source` | *(not stored)* | - | Source publication (ignored) |

### Calibre-Specific Fields (meta name="calibre:*")

| OPF Field | Database Column | Type | Notes |
|-----------|-----------------|------|-------|
| `calibre:series` | `series` | VARCHAR(255) | Series name |
| `calibre:series_index` | `series_number` | INTEGER | Position in series |
| `calibre:rating` | *(use media_ratings)* | INTEGER | 0-5 scale, stored per-user |
| `calibre:title_sort` | *(not stored)* | - | Sortable title (ignored) |
| `calibre:author_sort` | *(not stored)* | - | Sortable author (ignored) |
| `calibre:timestamp` | `created_at` | TIMESTAMP | When added to Calibre |

### Special Cases

1. **Multiple authors**: Store first author in `author`, suggest adding `authors[]` array in future
2. **Multiple identifiers**: Extract ISBN and ASIN if present, ignore others
3. **Calibre rating**: Not imported (ratings are per-user in Bookhoard)
4. **Sort fields**: Ignored (Bookhoard has its own sorting logic)
5. **Timestamp**: Use for `created_at` if more recent than file modification time

## OPF Format Reference

### Standard Dublin Core Elements

```xml
<dc:title>Book Title</dc:title>
<dc:creator>Author Name</dc:creator>
<dc:subject>Fantasy</dc:subject>          <!-- Tag -->
<dc:subject>Adventure</dc:subject>         <!-- Another tag -->
<dc:description>Book summary...</dc:description>
<dc:publisher>Publisher Name</dc:publisher>
<dc:date>2024-01-15</dc:date>
<dc:language>en</dc:language>
<dc:identifier opf:scheme="ISBN">978-0-123456-78-9</dc:identifier>
<dc:identifier opf:scheme="ASIN">B08XXXXX</dc:identifier>
<dc:contributor>Illustrator Name</dc:contributor>
```

### Calibre-Specific Meta Tags

```xml
<meta name="calibre:series" content="Series Name"/>
<meta name="calibre:series_index" content="3"/>
<meta name="calibre:rating" content="4"/>
<meta name="calibre:title_sort" content="Book Title"/>
<meta name="calibre:author_sort" content="Author, Name"/>
```

### Example Complete metadata.opf

```xml
<?xml version='1.0' encoding='utf-8'?>
<package xmlns="http://www.idpf.org/2007/opf" version="2.0" unique-identifier="uuid_id">
  <metadata xmlns:opf="http://www.idpf.org/2007/opf"
            xmlns:calibre="http://calibre.kovidgoyal.net/2009/metadata"
            xmlns:dc="http://purl.org/dc/elements/1.1/">
    <dc:title>The Fellowship of the Ring</dc:title>
    <dc:creator opf:role="aut">J.R.R. Tolkien</dc:creator>
    <dc:identifier opf:scheme="ISBN">978-0-618-00222-0</dc:identifier>
    <dc:identifier opf:scheme="calibre">12345678-1234-1234-1234-123456789012</dc:identifier>
    <dc:language>en</dc:language>
    <dc:date>1954-07-29</dc:date>
    <dc:publisher>HarperCollins</dc:publisher>
    <dc:description>In a sleepy village in the Shire...</dc:description>

    <!-- Tags -->
    <dc:subject>Fantasy</dc:subject>
    <dc:subject>Adventure</dc:subject>
    <dc:subject>Classics</dc:subject>

    <!-- Contributors -->
    <dc:contributor>Alan Lee (illustrator)</dc:contributor>

    <!-- Calibre-specific fields -->
    <meta name="calibre:series" content="The Lord of the Rings"/>
    <meta name="calibre:series_index" content="1"/>
    <meta name="calibre:rating" content="5"/>
    <meta name="calibre:title_sort" content="Fellowship of the Ring"/>
  </metadata>
</package>
```

## Implementation Design

### Architecture

```
Scanner Pipeline (EXISTING)
↓
1. Detect book file
2. Check if already in database
3. Extract metadata from file (EPUB/PDF/Comic)
4. Extract folder structure metadata
5. Normalize and store
```

**Simplified Approach**: Modify existing `extractMetadata()` to check for Calibre sidecar first

```
Scanner Pipeline (NEW)
↓
1. Detect book file
2. Check if already in database
3. Extract metadata (MODIFIED FUNCTION)
   ├── Check for Calibre metadata.opf sidecar ← NEW
   │   └─ If found → use sidecar metadata
   └── Fallback to embedded metadata (existing)
       ├── EPUB → extractEPUBMetadata()
       ├── PDF → extractPDFMetadata()
       └── Comic → extractComicMetadata()
4. Extract folder structure metadata (existing)
5. Normalize and store (existing)
```

### Code Changes

**SIMPLIFIED APPROACH** (thanks to user feedback): Modify existing `extractMetadata()` function instead of creating wrapper.

#### File: `internal/services/media_scanner.go`

**New Functions (to add):**

1. `extractCalibreSidecar(path string) *MediaMetadata`
   - Checks for `metadata.opf` in same directory as book file
   - Calls `parseCalibreMetadataOPF()` if found
   - Returns nil if no sidecar exists (not an error)
   - Returns nil on parse errors (graceful fallback)

2. `parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error)`
   - Opens and parses XML file
   - Extracts Dublin Core fields (`dc:*`)
   - Extracts Calibre-specific fields (`meta name="calibre:*"`)
   - Maps to `MediaMetadata` struct
   - Handles errors gracefully (malformed XML, missing fields)

**Modified Function:**

1. `extractMetadata(path string) (*MediaMetadata, error)` at line 712
   - Add sidecar check at very beginning (before switch statement)
   - If sidecar found → return sidecar metadata immediately
   - Otherwise → continue to existing embedded extraction logic
   - **No call site changes needed** - all existing code continues to work

#### New Struct: `CalibreOPFMetadata`

```go
// Intermediate struct for parsed OPF data
type CalibreOPFMetadata struct {
    Title          string
    Authors        []string
    Tags           []string
    Description    string
    Publisher      string
    PublishDate    *time.Time
    Language       string
    ISBN           string
    ASIN           string
    UUID           string
    Contributors   []string
    Series         string
    SeriesIndex    *float64
    Rating         *int32
    Timestamp      *time.Time
}
```

### Metadata Priority (Sidecar-First)

```
Priority Order:
1. Calibre metadata.opf sidecar ← NEW (highest priority)
2. Embedded file metadata (EPUB OPF, PDF info, ComicInfo.xml)
3. Folder structure parsing
4. Filename parsing (last resort)
```

**Rationale**: If a user has a `metadata.opf` file, they're actively managing their library in Calibre. The sidecar represents their curated "source of truth."

### Error Handling

1. **Sidecar file not found** → Return nil, continue to embedded metadata
2. **Malformed XML** → Log warning, continue to embedded metadata
3. **Missing required fields** → Use available fields, continue
4. **Invalid date format** → Log warning, skip field
5. **File permission errors** → Log error, continue to embedded metadata

**Principle**: Graceful degradation - never fail a scan due to sidecar issues

### Testing Strategy

#### Unit Tests

File: `internal/services/media_scanner_calibre_test.go`

- Test parsing complete metadata.opf with all fields
- Test parsing minimal metadata.opf (title only)
- Test parsing malformed XML (graceful failure)
- Test missing metadata.opf (returns nil, not error)
- Test multiple `<dc:subject>` tags (tags array)
- Test multiple `<dc:identifier>` tags (extract ISBN/ASIN)
- Test Calibre-specific fields (series, series_index)
- Test invalid date formats
- Test empty/whitespace values

#### Integration Tests

File: `cmd/server/tests/calibre_integration_test.go`

- Test scanning directory with Calibre library structure
- Test scanning directory without sidecar files (existing behavior)
- Test mixed: some books with sidecar, some without
- Test force rescan updates from sidecar changes
- Test all file types: EPUB, PDF, CBZ, MP3, M4B
- Test all library types: ebook, audiobook, comic

#### Test Data

Create `testdata/calibre/` with sample files:
- `Complete Book/metadata.opf` - All fields populated
- `Minimal Book/metadata.opf` - Only required fields
- `Malformed Book/metadata.opf` - Invalid XML
- No sidecar files (rely on existing test data)

## Implementation Steps

### Phase 1: Core Functionality (REQUIRED)

- [ ] **Step 1.1**: Add `CalibreOPFMetadata` struct (intermediate parsing struct)
- [ ] **Step 1.2**: Implement `parseCalibreMetadataOPF()` function (~120 lines)
- [ ] **Step 1.3**: Implement `extractCalibreSidecar()` function (~20 lines)
- [ ] **Step 1.4**: Modify `extractMetadata()` function (~10 lines)
  - Add sidecar check at beginning of function (line ~712)
  - Return sidecar metadata if found
  - Otherwise continue to existing switch statement
- [ ] **Step 1.5**: Add unit tests for OPF parsing
- [ ] **Step 1.6**: Test with real Calibre library

**Total new code: ~150 lines** (simplified from original ~250 lines thanks to direct modification of `extractMetadata()`)

### Phase 2: Documentation (REQUIRED)

- [ ] **Step 2.1**: Create user documentation (`docs/user/calibre-support.md`)
- [ ] **Step 2.2**: Update developer documentation (`docs/development/calibre-opf-implementation.md`)
- [ ] **Step 2.3**: Update scanner API docs if needed
- [ ] **Step 2.4**: Add examples to documentation

### Phase 3: Testing & Polish (REQUIRED)

- [ ] **Step 3.1**: Add integration tests
- [ ] **Step 3.2**: Test with various Calibre library configurations
- [ ] **Step 3.3**: Performance testing (ensure no regression)
- [ ] **Step 3.4**: Error handling review
- [ ] **Step 3.5**: Code review and refinement

### Phase 4: Future Enhancements (OPTIONAL)

- [ ] Support Calibre `author_link` metadata
- [ ] Support Calibre custom columns
- [ ] Import Calibre ratings as user's initial rating
- [ ] Detect Calibre library structure patterns
- [ ] Configuration option for merge strategy (sidecar-first vs. smart-merge)

## Backward Compatibility

**No breaking changes.** Existing behavior is preserved:

- Libraries without `metadata.opf` files → Existing behavior (embedded metadata)
- Scanning logic unchanged → Single function modification, no call site changes
- Database unchanged → All fields already exist
- API unchanged → No new endpoints or response fields
- **No wrapper function needed** → Simpler, cleaner implementation

## Success Criteria

- [ ] All existing tests pass
- [ ] New unit tests pass (90%+ coverage)
- [ ] New integration tests pass
- [ ] Scans real Calibre library successfully
- [ ] Scans non-Calibre library without errors
- [ ] No performance regression (scan time +5% max)
- [ ] Documentation complete and accurate
- [ ] Code review approved

## Open Questions

1. **Q**: Should we import Calibre ratings?
   **A**: No - ratings are per-user in Bookhoard. Calibre rating is the owner's opinion, not necessarily the current user's.

2. **Q**: Should we support Calibre custom columns?
   **A**: Not in initial implementation. Custom columns are user-defined and would require dynamic schema or JSON storage. Future enhancement.

3. **Q**: How to handle multiple authors?
   **A**: Store first author in `author` field (existing behavior). Future enhancement could add `authors[]` array.

4. **Q**: Should we follow Calibre folder structure?
   **A**: Already implemented! `extractFolderStructureMetadata()` handles Calibre patterns.

## References

- [EPUB Publications 3.0 Specification](https://www.idpf.org/epub/30/spec/epub30-publications.html)
- [Dublin Core Metadata Element Set](https://www.dublincore.org/specifications/dces/1998-08-13/)
- [Calibre Manual - Editing Metadata](https://manual.calibre-ebook.com/metadata.html)
- [Calibre Source Code](https://github.com/kovidgoyal/calibre)

## Appendix: Decision Log

### Decision 1: Sidecar-First Approach
**Date**: 2026-03-26
**Decision**: Use sidecar-first (not smart-merge)
**Rationale**:
- Users with Calibre sidecars actively curate their metadata
- Sidecar metadata is typically richer and more accurate
- Simple implementation, predictable behavior
- Easy to test and debug
- Future enhancement can add configurable merge strategy

### Decision 2: No Database Schema Changes
**Date**: 2026-03-26
**Decision**: Use existing schema only
**Rationale**:
- All required fields already exist in `media_items` table
- Pre-production application (no legacy data to migrate)
- Simpler implementation
- No backward compatibility concerns

### Decision 3: All Library Types Supported
**Date**: 2026-03-26
**Decision**: Support OPF for all libraries, not just ebooks
**Rationale**:
- Calibre supports all file types
- Users may organize audiobooks/comics in Calibre
- Consistent behavior across library types
- No additional complexity

### Decision 4: Simplified Implementation (REVISED)
**Date**: 2026-03-26
**Decision**: Modify `extractMetadata()` directly instead of creating wrapper function
**Rationale**:
- **User feedback**: Suggested simpler approach with if statement check
- **Less code**: ~150 lines vs. ~250 lines
- **No call site changes**: All existing code continues to work
- **Clearer flow**: Single entry point for metadata extraction
- **Better testability**: Test `extractMetadata()` with/without sidecar
- **Easier to maintain**: All metadata logic in one place

**Original plan**:
```go
// Wrapper function approach
extractMetadataWithSidecar() {
    if sidecar exists → return sidecar
    return extractMetadata()
}
```

**Revised plan (SIMPLER)**:
```go
// Direct modification
extractMetadata() {
    if sidecar exists → return sidecar  // ← NEW: Just add this at top
    // existing switch statement continues...
}
```

This simplification reduces code complexity and makes the implementation cleaner and easier to understand.