docs: add comprehensive Calibre integration documentation

This commit adds complete documentation for the planned Calibre metadata.opf sidecar file support feature. ## New Documentation ### Implementation Planning - CALIBRE_OPF_IMPLEMENTATION.md: Detailed implementation plan with requirements, architecture, database mapping, and step-by-step implementation guide for adding Calibre metadata.opf support ### Technical Documentation - docs/development/calibre-opf-implementation.md: Technical implementation details including: - Scanner pipeline architecture with sidecar-first approach - Data structures (CalibreOPFMetadata, MediaMetadata) - Function signatures and logic for parseCalibreMetadataOPF() - Database schema mapping (no changes required) - Testing strategy (unit and integration tests) - Error handling and performance considerations - Code examples and benchmarking approach ### User Documentation - docs/user/calibre-integration.md: Comprehensive user guide covering: - What is Calibre and how Bookhoard integrates with it - Automatic metadata import from metadata.opf sidecar files - Supported metadata fields (Dublin Core + Calibre-specific) - Setup instructions for Calibre libraries - Workflow examples (fresh library, mixed library, updating metadata) - Troubleshooting common issues - Best practices for Calibre + Bookhoard workflow - FAQ and resources ## Updated Documentation - README.md: Added Calibre integration feature to media management section - docs/user/user-guide.md: Added link to Calibre integration guide - docs/developer/development.md: Added link to Calibre implementation guide ## Feature Summary The Calibre integration feature will allow Bookhoard to automatically import curated metadata from Calibre's metadata.opf sidecar files, including titles, authors, series, tags, descriptions, publishers, identifiers (ISBN/ASIN), and contributors. Uses sidecar-first approach: metadata.opf → embedded metadata → folder structure → filename. All database fields already exist; no schema changes required.
2026-03-26 10:34:15 -04:00
parent 333f4ae026
commit 22fd28c7db
6 changed files with 1295 additions and 0 deletions
@@ -0,0 +1,381 @@
+# Calibre metadata.opf Support - Implementation Plan
+
+## Overview
+
+This document outlines the implementation of **Calibre metadata.opf sidecar file support** for Bookhoard's media scanner. The scanner will use a **sidecar-first approach** - if a Calibre `metadata.opf` file exists alongside a book file, it will take precedence over embedded metadata.
+
+## Motivation
+
+Calibre is the most popular ebook management software, with millions of users. When users manage their libraries in Calibre:
+
+1. **They curate metadata** - Adding series information, tags, ratings, custom covers
+2. **They fetch from online sources** - Google Books, Amazon, Goodreads (better than publisher metadata)
+3. **They convert/edit books** - The sidecar reflects the current, correct state
+4. **They organize in folders** - Often using Calibre's folder structure conventions
+
+By supporting Calibre `metadata.opf` sidecar files, Bookhoard can:
+- Respect users' curation work
+- Import richer, more accurate metadata
+- Provide seamless integration with existing Calibre libraries
+- Support all file types (EPUB, PDF, comics, etc.) that Calibre manages
+
+## Requirements
+
+### Functional Requirements
+
+- [ ] Scanner detects `metadata.opf` files in the same directory as book files
+- [ ] Parses Dublin Core metadata (`dc:*` namespace) and Calibre-specific fields (`meta name="calibre:*"`)
+- [ ] Uses sidecar metadata **before** embedded metadata (sidecar-first approach)
+- [ ] Works for all library types (ebook, audiobook, comic, etc.)
+- [ ] Works for all file types (EPUB, PDF, CBZ, CBR, MP3, M4B, etc.)
+- [ ] Handles missing or malformed sidecar files gracefully
+- [ ] Maps all OPF fields to existing `media_items` database schema
+- [ ] Maintains backward compatibility (no sidecar = existing behavior)
+
+### Non-Functional Requirements
+
+- [ ] No breaking changes to existing scanner functionality
+- [ ] No database schema changes required (all fields already exist)
+- [ ] Follows existing code patterns and style
+- [ ] Comprehensive test coverage
+- [ ] Performance: Minimal impact on scan performance
+- [ ] Error handling: Graceful degradation on malformed XML
+
+## Database Schema Mapping
+
+All required fields **already exist** in the `media_items` table. No schema changes needed.
+
+### Dublin Core Fields (dc: namespace)
+
+| OPF Field | Database Column | Type | Notes |
+|-----------|-----------------|------|-------|
+| `dc:title` | `title` | VARCHAR(255) | Required |
+| `dc:creator` | `author` | VARCHAR(255) | First author only |
+| `dc:subject` | `tags` | TEXT[] | **Tags/keywords** (multiple) |
+| `dc:description` | `description` | TEXT | Book summary |
+| `dc:publisher` | `publisher` | VARCHAR(255) | Publisher name |
+| `dc:date` | `date_published` | DATE | Publication date |
+| `dc:language` | `language` | VARCHAR(10) | ISO 639-1 code (en, es, fr) |
+| `dc:identifier` (ISBN) | `isbn` | VARCHAR(13) | Check `opf:scheme="ISBN"` |
+| `dc:identifier` (ASIN) | `asin` | VARCHAR(20) | Check `opf:scheme="ASIN"` |
+| `dc:contributor` | `contributors` | TEXT[] | Multiple contributors |
+| `dc:rights` | *(not stored)* | - | Copyright info (ignored) |
+| `dc:source` | *(not stored)* | - | Source publication (ignored) |
+
+### Calibre-Specific Fields (meta name="calibre:*")
+
+| OPF Field | Database Column | Type | Notes |
+|-----------|-----------------|------|-------|
+| `calibre:series` | `series` | VARCHAR(255) | Series name |
+| `calibre:series_index` | `series_number` | INTEGER | Position in series |
+| `calibre:rating` | *(use media_ratings)* | INTEGER | 0-5 scale, stored per-user |
+| `calibre:title_sort` | *(not stored)* | - | Sortable title (ignored) |
+| `calibre:author_sort` | *(not stored)* | - | Sortable author (ignored) |
+| `calibre:timestamp` | `created_at` | TIMESTAMP | When added to Calibre |
+
+### Special Cases
+
+1. **Multiple authors**: Store first author in `author`, suggest adding `authors[]` array in future
+2. **Multiple identifiers**: Extract ISBN and ASIN if present, ignore others
+3. **Calibre rating**: Not imported (ratings are per-user in Bookhoard)
+4. **Sort fields**: Ignored (Bookhoard has its own sorting logic)
+5. **Timestamp**: Use for `created_at` if more recent than file modification time
+
+## OPF Format Reference
+
+### Standard Dublin Core Elements
+
+```xml
+<dc:title>Book Title</dc:title>
+<dc:creator>Author Name</dc:creator>
+<dc:subject>Fantasy</dc:subject>          <!-- Tag -->
+<dc:subject>Adventure</dc:subject>         <!-- Another tag -->
+<dc:description>Book summary...</dc:description>
+<dc:publisher>Publisher Name</dc:publisher>
+<dc:date>2024-01-15</dc:date>
+<dc:language>en</dc:language>
+<dc:identifier opf:scheme="ISBN">978-0-123456-78-9</dc:identifier>
+<dc:identifier opf:scheme="ASIN">B08XXXXX</dc:identifier>
+<dc:contributor>Illustrator Name</dc:contributor>
+```
+
+### Calibre-Specific Meta Tags
+
+```xml
+<meta name="calibre:series" content="Series Name"/>
+<meta name="calibre:series_index" content="3"/>
+<meta name="calibre:rating" content="4"/>
+<meta name="calibre:title_sort" content="Book Title"/>
+<meta name="calibre:author_sort" content="Author, Name"/>
+```
+
+### Example Complete metadata.opf
+
+```xml
+<?xml version='1.0' encoding='utf-8'?>
+<package xmlns="http://www.idpf.org/2007/opf" version="2.0" unique-identifier="uuid_id">
+  <metadata xmlns:opf="http://www.idpf.org/2007/opf"
+            xmlns:calibre="http://calibre.kovidgoyal.net/2009/metadata"
+            xmlns:dc="http://purl.org/dc/elements/1.1/">
+    <dc:title>The Fellowship of the Ring</dc:title>
+    <dc:creator opf:role="aut">J.R.R. Tolkien</dc:creator>
+    <dc:identifier opf:scheme="ISBN">978-0-618-00222-0</dc:identifier>
+    <dc:identifier opf:scheme="calibre">12345678-1234-1234-1234-123456789012</dc:identifier>
+    <dc:language>en</dc:language>
+    <dc:date>1954-07-29</dc:date>
+    <dc:publisher>HarperCollins</dc:publisher>
+    <dc:description>In a sleepy village in the Shire...</dc:description>
+
+    <!-- Tags -->
+    <dc:subject>Fantasy</dc:subject>
+    <dc:subject>Adventure</dc:subject>
+    <dc:subject>Classics</dc:subject>
+
+    <!-- Contributors -->
+    <dc:contributor>Alan Lee (illustrator)</dc:contributor>
+
+    <!-- Calibre-specific fields -->
+    <meta name="calibre:series" content="The Lord of the Rings"/>
+    <meta name="calibre:series_index" content="1"/>
+    <meta name="calibre:rating" content="5"/>
+    <meta name="calibre:title_sort" content="Fellowship of the Ring"/>
+  </metadata>
+</package>
+```
+
+## Implementation Design
+
+### Architecture
+
+```
+Scanner Pipeline (EXISTING)
+↓
+1. Detect book file
+2. Check if already in database
+3. Extract metadata from file (EPUB/PDF/Comic)
+4. Extract folder structure metadata
+5. Normalize and store
+```
+
+**New Step 3a**: Check for Calibre sidecar **before** embedded extraction
+
+```
+Scanner Pipeline (NEW)
+↓
+1. Detect book file
+2. Check if already in database
+3. **NEW** Extract metadata from Calibre metadata.opf (if exists)
+   - If sidecar found → use sidecar metadata
+   - If no sidecar → use embedded metadata (existing)
+4. Extract folder structure metadata (existing)
+5. Normalize and store (existing)
+```
+
+### Code Changes
+
+#### File: `internal/services/media_scanner.go`
+
+**New Functions:**
+
+1. `extractMetadataFromCalibreSidecar(path string) (*MediaMetadata, error)`
+   - Checks for `metadata.opf` in same directory as book file
+   - Calls `parseCalibreMetadataOPF()` if found
+   - Returns nil if no sidecar exists (not an error)
+
+2. `parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error)`
+   - Opens and parses XML file
+   - Extracts Dublin Core fields (`dc:*`)
+   - Extracts Calibre-specific fields (`meta name="calibre:*"`)
+   - Maps to `MediaMetadata` struct
+   - Handles errors gracefully (malformed XML, missing fields)
+
+**Modified Function:**
+
+1. `processMediaFile(ctx context.Context, path string) (bool, error)`
+   - Add sidecar extraction check at line ~544 (before embedded extraction)
+   - Flow: sidecar → embedded → folder → filename
+
+#### New Struct: `CalibreOPFMetadata`
+
+```go
+// Intermediate struct for parsed OPF data
+type CalibreOPFMetadata struct {
+    Title          string
+    Authors        []string
+    Tags           []string
+    Description    string
+    Publisher      string
+    PublishDate    *time.Time
+    Language       string
+    ISBN           string
+    ASIN           string
+    UUID           string
+    Contributors   []string
+    Series         string
+    SeriesIndex    *float64
+    Rating         *int32
+    Timestamp      *time.Time
+}
+```
+
+### Metadata Priority (Sidecar-First)
+
+```
+Priority Order:
+1. Calibre metadata.opf sidecar ← NEW (highest priority)
+2. Embedded file metadata (EPUB OPF, PDF info, ComicInfo.xml)
+3. Folder structure parsing
+4. Filename parsing (last resort)
+```
+
+**Rationale**: If a user has a `metadata.opf` file, they're actively managing their library in Calibre. The sidecar represents their curated "source of truth."
+
+### Error Handling
+
+1. **Sidecar file not found** → Return nil, continue to embedded metadata
+2. **Malformed XML** → Log warning, continue to embedded metadata
+3. **Missing required fields** → Use available fields, continue
+4. **Invalid date format** → Log warning, skip field
+5. **File permission errors** → Log error, continue to embedded metadata
+
+**Principle**: Graceful degradation - never fail a scan due to sidecar issues
+
+### Testing Strategy
+
+#### Unit Tests
+
+File: `internal/services/media_scanner_calibre_test.go`
+
+- Test parsing complete metadata.opf with all fields
+- Test parsing minimal metadata.opf (title only)
+- Test parsing malformed XML (graceful failure)
+- Test missing metadata.opf (returns nil, not error)
+- Test multiple `<dc:subject>` tags (tags array)
+- Test multiple `<dc:identifier>` tags (extract ISBN/ASIN)
+- Test Calibre-specific fields (series, series_index)
+- Test invalid date formats
+- Test empty/whitespace values
+
+#### Integration Tests
+
+File: `cmd/server/tests/calibre_integration_test.go`
+
+- Test scanning directory with Calibre library structure
+- Test scanning directory without sidecar files (existing behavior)
+- Test mixed: some books with sidecar, some without
+- Test force rescan updates from sidecar changes
+- Test all file types: EPUB, PDF, CBZ, MP3, M4B
+- Test all library types: ebook, audiobook, comic
+
+#### Test Data
+
+Create `testdata/calibre/` with sample files:
+- `Complete Book/metadata.opf` - All fields populated
+- `Minimal Book/metadata.opf` - Only required fields
+- `Malformed Book/metadata.opf` - Invalid XML
+- No sidecar files (rely on existing test data)
+
+## Implementation Steps
+
+### Phase 1: Core Functionality (REQUIRED)
+
+- [ ] **Step 1.1**: Add `CalibreOPFMetadata` struct
+- [ ] **Step 1.2**: Implement `parseCalibreMetadataOPF()` function
+- [ ] **Step 1.3**: Implement `extractMetadataFromCalibreSidecar()` function
+- [ ] **Step 1.4**: Modify `processMediaFile()` to call sidecar extraction
+- [ ] **Step 1.5**: Add unit tests for OPF parsing
+- [ ] **Step 1.6**: Test with real Calibre library
+
+### Phase 2: Documentation (REQUIRED)
+
+- [ ] **Step 2.1**: Create user documentation (`docs/user/calibre-support.md`)
+- [ ] **Step 2.2**: Update developer documentation (`docs/development/calibre-opf-implementation.md`)
+- [ ] **Step 2.3**: Update scanner API docs if needed
+- [ ] **Step 2.4**: Add examples to documentation
+
+### Phase 3: Testing & Polish (REQUIRED)
+
+- [ ] **Step 3.1**: Add integration tests
+- [ ] **Step 3.2**: Test with various Calibre library configurations
+- [ ] **Step 3.3**: Performance testing (ensure no regression)
+- [ ] **Step 3.4**: Error handling review
+- [ ] **Step 3.5**: Code review and refinement
+
+### Phase 4: Future Enhancements (OPTIONAL)
+
+- [ ] Support Calibre `author_link` metadata
+- [ ] Support Calibre custom columns
+- [ ] Import Calibre ratings as user's initial rating
+- [ ] Detect Calibre library structure patterns
+- [ ] Configuration option for merge strategy (sidecar-first vs. smart-merge)
+
+## Backward Compatibility
+
+**No breaking changes.** Existing behavior is preserved:
+
+- Libraries without `metadata.opf` files → Existing behavior (embedded metadata)
+- Scanning logic unchanged → Only adds sidecar check before embedded extraction
+- Database unchanged → All fields already exist
+- API unchanged → No new endpoints or response fields
+
+## Success Criteria
+
+- [ ] All existing tests pass
+- [ ] New unit tests pass (90%+ coverage)
+- [ ] New integration tests pass
+- [ ] Scans real Calibre library successfully
+- [ ] Scans non-Calibre library without errors
+- [ ] No performance regression (scan time +5% max)
+- [ ] Documentation complete and accurate
+- [ ] Code review approved
+
+## Open Questions
+
+1. **Q**: Should we import Calibre ratings?
+   **A**: No - ratings are per-user in Bookhoard. Calibre rating is the owner's opinion, not necessarily the current user's.
+
+2. **Q**: Should we support Calibre custom columns?
+   **A**: Not in initial implementation. Custom columns are user-defined and would require dynamic schema or JSON storage. Future enhancement.
+
+3. **Q**: How to handle multiple authors?
+   **A**: Store first author in `author` field (existing behavior). Future enhancement could add `authors[]` array.
+
+4. **Q**: Should we follow Calibre folder structure?
+   **A**: Already implemented! `extractFolderStructureMetadata()` handles Calibre patterns.
+
+## References
+
+- [EPUB Publications 3.0 Specification](https://www.idpf.org/epub/30/spec/epub30-publications.html)
+- [Dublin Core Metadata Element Set](https://www.dublincore.org/specifications/dces/1998-08-13/)
+- [Calibre Manual - Editing Metadata](https://manual.calibre-ebook.com/metadata.html)
+- [Calibre Source Code](https://github.com/kovidgoyal/calibre)
+
+## Appendix: Decision Log
+
+### Decision 1: Sidecar-First Approach
+**Date**: 2026-03-26
+**Decision**: Use sidecar-first (not smart-merge)
+**Rationale**:
+- Users with Calibre sidecars actively curate their metadata
+- Sidecar metadata is typically richer and more accurate
+- Simple implementation, predictable behavior
+- Easy to test and debug
+- Future enhancement can add configurable merge strategy
+
+### Decision 2: No Database Schema Changes
+**Date**: 2026-03-26
+**Decision**: Use existing schema only
+**Rationale**:
+- All required fields already exist in `media_items` table
+- Pre-production application (no legacy data to migrate)
+- Simpler implementation
+- No backward compatibility concerns
+
+### Decision 3: All Library Types Supported
+**Date**: 2026-03-26
+**Decision**: Support OPF for all libraries, not just ebooks
+**Rationale**:
+- Calibre supports all file types
+- Users may organize audiobooks/comics in Calibre
+- Consistent behavior across library types
+- No additional complexity