Bookhoard/bookhoard
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 12 Wiki Activity

docs: add comprehensive Calibre integration documentation

Browse Source 
This commit adds complete documentation for the planned Calibre
metadata.opf sidecar file support feature.

## New Documentation

### Implementation Planning
- CALIBRE_OPF_IMPLEMENTATION.md: Detailed implementation plan with
  requirements, architecture, database mapping, and step-by-step
  implementation guide for adding Calibre metadata.opf support

### Technical Documentation
- docs/development/calibre-opf-implementation.md: Technical implementation
  details including:
  - Scanner pipeline architecture with sidecar-first approach
  - Data structures (CalibreOPFMetadata, MediaMetadata)
  - Function signatures and logic for parseCalibreMetadataOPF()
  - Database schema mapping (no changes required)
  - Testing strategy (unit and integration tests)
  - Error handling and performance considerations
  - Code examples and benchmarking approach

### User Documentation
- docs/user/calibre-integration.md: Comprehensive user guide covering:
  - What is Calibre and how Bookhoard integrates with it
  - Automatic metadata import from metadata.opf sidecar files
  - Supported metadata fields (Dublin Core + Calibre-specific)
  - Setup instructions for Calibre libraries
  - Workflow examples (fresh library, mixed library, updating metadata)
  - Troubleshooting common issues
  - Best practices for Calibre + Bookhoard workflow
  - FAQ and resources

## Updated Documentation

- README.md: Added Calibre integration feature to media management section
- docs/user/user-guide.md: Added link to Calibre integration guide
- docs/developer/development.md: Added link to Calibre implementation guide

## Feature Summary

The Calibre integration feature will allow Bookhoard to automatically
import curated metadata from Calibre's metadata.opf sidecar files,
including titles, authors, series, tags, descriptions, publishers,
identifiers (ISBN/ASIN), and contributors. Uses sidecar-first approach:
metadata.opf → embedded metadata → folder structure → filename.

All database fields already exist; no schema changes required.
This commit is contained in:
john-okeefe
2026-03-26 10:34:15 -04:00
parent 333f4ae026
commit 22fd28c7db
6 changed files with 1295 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/development/calibre-opf-implementation.md
+576
View File
@@ -0,0 +1,576 @@
# Calibre metadata.opf Implementation
This document provides technical details for implementing Calibre `metadata.opf` sidecar file support in Bookhoard's media scanner.
## Overview
The media scanner detects and imports metadata from Calibre's `metadata.opf` sidecar files using a **sidecar-first approach**. If a `metadata.opf` file exists alongside a book file, its metadata takes precedence over embedded file metadata.
## Architecture
### Scanner Pipeline
```go
func (s *MediaScanner) processMediaFile(ctx context.Context, path string) (bool, error) {
// 1. Get file info
// 2. Find library
// 3. Check if media item exists
// 4. EXTRACT METADATA (NEW: sidecar-first)
metadata := s.extractMetadataWithSidecar(path)
// 5. Extract hash info
// 6. Extract comic metadata (if applicable)
// 7. Extract folder structure metadata
// 8. Normalize metadata
// 9. Create or update in database
}
```
### Metadata Extraction Flow
```
extractMetadataWithSidecar()
↓
1. Try Calibre metadata.opf sidecar
├─ Found → parseCalibreMetadataOPF()
└─ Not found → continue
↓
2. Extract embedded metadata
├─ EPUB → extractEPUBMetadata()
├─ PDF → extractPDFMetadata()
└─ Comic → extractComicMetadata()
↓
3. Extract folder structure metadata
└─ extractFolderStructureMetadata()
↓
4. Return merged MediaMetadata
```
## Data Structures
### CalibreOPFMetadata
Intermediate struct for parsed OPF data:
```go
// CalibreOPFMetadata represents parsed metadata from Calibre metadata.opf
type CalibreOPFMetadata struct {
Title string
Authors []string
Tags []string
Description string
Publisher string
PublishDate *time.Time
Language string
ISBN string
ASIN string
UUID string
Contributors []string
Series string
SeriesIndex *float64
Rating *int32
Timestamp *time.Time
}
```
### MediaMetadata
Existing struct (already defined in `media_scanner.go`):
```go
// MediaMetadata represents extracted book metadata
type MediaMetadata struct {
Title string
Author string
ISBN string
Description string
Publisher string
Series string
SeriesNumber int32
Tags []string
Contributors []string
PublishDate time.Time
ASIN string
// ... other fields
}
```
## Implementation Functions
### 1. extractMetadataFromCalibreSidecar()
**Location**: `internal/services/media_scanner.go`
**Signature**:
```go
func (s *MediaScanner) extractMetadataFromCalibreSidecar(mediaFilePath string) (*MediaMetadata, error)
```
**Purpose**: Checks for `metadata.opf` in the same directory as the book file.
**Logic**:
```go
func (s *MediaScanner) extractMetadataFromCalibreSidecar(mediaFilePath string) (*MediaMetadata, error) {
// Get directory of media file
dir := filepath.Dir(mediaFilePath)
// Check for metadata.opf
opfPath := filepath.Join(dir, "metadata.opf")
if _, err := os.Stat(opfPath); os.IsNotExist(err) {
// No sidecar file - not an error, just return nil
return nil, nil
}
// Parse metadata.opf
return s.parseCalibreMetadataOPF(opfPath)
}
```
**Error Handling**:
- File not found → Return `nil, nil` (not an error)
- Permission denied → Log warning, return `nil, nil`
- Other errors → Log error, return `nil, err`
### 2. parseCalibreMetadataOPF()
**Location**: `internal/services/media_scanner.go`
**Signature**:
```go
func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error)
```
**Purpose**: Parses XML file and extracts Dublin Core + Calibre-specific fields.
**Logic**:
```go
func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error) {
// Open file
file, err := os.Open(opfPath)
if err != nil {
return nil, fmt.Errorf("failed to open metadata.opf: %v", err)
}
defer file.Close()
// Parse XML
var opf struct {
XMLName xml.Name `xml:"package"`
Metadata struct {
XMLName xml.Name `xml:"metadata"`
Titles []string `xml:"dc:title"`
Creators []string `xml:"dc:creator"`
Subjects []string `xml:"dc:subject"`
Desc []string `xml:"dc:description"`
Publisher []string `xml:"dc:publisher"`
Dates []string `xml:"dc:date"`
Language []string `xml:"dc:language"`
Identifiers []struct {
Scheme string `xml:"opf:scheme,attr"`
Value string `xml:",chardata"`
} `xml:"dc:identifier"`
Contributors []string `xml:"dc:contributor"`
CalibreSeries []struct {
Name string `xml:"content,attr"`
} `xml:"meta[name='calibre:series']"`
CalibreSeriesIndex []struct {
Value string `xml:"content,attr"`
} `xml:"meta[name='calibre:series_index']"`
} `xml:"metadata"`
}
if err := xml.NewDecoder(file).Decode(&opf); err != nil {
return nil, fmt.Errorf("failed to parse metadata.opf XML: %v", err)
}
// Map to MediaMetadata
metadata := &MediaMetadata{}
// Title (required)
if len(opf.Metadata.Titles) > 0 {
metadata.Title = opf.Metadata.Titles[0]
}
// Author (first creator)
if len(opf.Metadata.Creators) > 0 {
metadata.Author = opf.Metadata.Creators[0]
}
// Tags (all subjects)
if len(opf.Metadata.Subjects) > 0 {
metadata.Tags = utils.NormalizeTags(opf.Metadata.Subjects)
}
// Description
if len(opf.Metadata.Desc) > 0 {
metadata.Description = opf.Metadata.Desc[0]
}
// Publisher
if len(opf.Metadata.Publisher) > 0 {
metadata.Publisher = opf.Metadata.Publisher[0]
}
// Publish date
if len(opf.Metadata.Dates) > 0 {
if date, err := time.Parse("2006-01-02", opf.Metadata.Dates[0]); err == nil {
metadata.PublishDate = date
}
}
// Language
if len(opf.Metadata.Language) > 0 {
// Store language (already in schema)
// Note: MediaMetadata doesn't have Language field yet
// Future enhancement: add Language field
}
// Identifiers (ISBN, ASIN)
for _, id := range opf.Metadata.Identifiers {
switch strings.ToUpper(id.Scheme) {
case "ISBN":
metadata.ISBN = utils.NormalizeISBNSafe(id.Value)
case "ASIN":
metadata.ASIN = id.Value
case "UUID", "CALIBRE":
// Store UUID in hash info, not metadata
// Will be extracted by extractHashInfo()
}
}
// Contributors
if len(opf.Metadata.Contributors) > 0 {
metadata.Contributors = utils.NormalizeContributors(opf.Metadata.Contributors)
}
// Series
if len(opf.Metadata.CalibreSeries) > 0 {
metadata.Series = opf.Metadata.CalibreSeries[0].Name
}
// Series index
if len(opf.Metadata.CalibreSeriesIndex) > 0 {
if index, err := strconv.ParseFloat(opf.Metadata.CalibreSeriesIndex[0].Value, 32); err == nil {
metadata.SeriesNumber = int32(index)
}
}
return metadata, nil
}
```
**XML Namespaces**: Handle Dublin Core (`dc:`) and Calibre (`calibre:`) namespaces properly.
**Error Handling**:
- Malformed XML → Log warning, return `nil, nil`
- Missing required fields → Use available fields
- Invalid date format → Log warning, skip field
### 3. extractMetadataWithSidecar()
**Location**: `internal/services/media_scanner.go`
**Signature**:
```go
func (s *MediaScanner) extractMetadataWithSidecar(path string) (*MediaMetadata, error)
```
**Purpose**: Implements sidecar-first metadata extraction.
**Logic**:
```go
func (s *MediaScanner) extractMetadataWithSidecar(path string) (*MediaMetadata, error) {
// Try Calibre sidecar first
if metadata, err := s.extractMetadataFromCalibreSidecar(path); metadata != nil {
fmt.Printf("Using Calibre metadata.opf for %s\n", path)
return metadata, nil
} else if err != nil {
// Log error but continue to embedded extraction
fmt.Printf("Warning: failed to read Calibre sidecar: %v\n", err)
}
// Fallback to embedded metadata
return s.extractMetadata(path)
}
```
**Priority**: Sidecar → Embedded → Folder → Filename
## Database Schema Mapping
All OPF fields map to existing `media_items` columns:
| OPF Field | Database Column | Type | Conversion |
|-----------|-----------------|------|------------|
| `dc:title` | `title` | VARCHAR(255) | Direct |
| `dc:creator` | `author` | VARCHAR(255) | First author only |
| `dc:subject` | `tags` | TEXT[] | Array conversion |
| `dc:description` | `description` | TEXT | Direct |
| `dc:publisher` | `publisher` | VARCHAR(255) | Direct |
| `dc:date` | `date_published` | DATE | Parse ISO 8601 |
| `dc:language` | `language` | VARCHAR(10) | Store ISO code |
| `dc:identifier[ISBN]` | `isbn` | VARCHAR(13) | Normalize with utils |
| `dc:identifier[ASIN]` | `asin` | VARCHAR(20) | Direct |
| `dc:contributor` | `contributors` | TEXT[] | Array conversion |
| `calibre:series` | `series` | VARCHAR(255) | Direct |
| `calibre:series_index` | `series_number` | INTEGER | Parse float → int |
**No schema changes required** - all fields already exist.
## Normalization
Use existing normalization utilities:
```go
// Tags
metadata.Tags = utils.NormalizeTags(tags)
// Contributors
metadata.Contributors = utils.NormalizeContributors(contributors)
// ISBN
metadata.ISBN = utils.NormalizeISBNSafe(isbn)
// Search versions (auto-generated by database triggers)
// tags_search, contributors_search
```
## Testing
### Unit Tests
**File**: `internal/services/media_scanner_calibre_test.go`
```go
func TestParseCalibreMetadataOPF(t *testing.T) {
tests := []struct {
name string
opfContent string
want *MediaMetadata
wantErr bool
}{
{
name: "Complete metadata.opf",
opfContent: `<?xml version='1.0' encoding='utf-8'?>
<package xmlns="http://www.idpf.org/2007/opf" version="2.0">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:title>Test Book</dc:title>
<dc:creator>Test Author</dc:creator>
<dc:subject>Fantasy</dc:subject>
<dc:subject>Adventure</dc:subject>
<dc:description>Test description</dc:description>
<dc:publisher>Test Publisher</dc:publisher>
<dc:date>2024-01-15</dc:date>
<dc:language>en</dc:language>
<dc:identifier opf:scheme="ISBN">978-0-123456-78-9</dc:identifier>
<meta name="calibre:series" content="Test Series"/>
<meta name="calibre:series_index" content="1"/>
</metadata>
</package>`,
want: &MediaMetadata{
Title: "Test Book",
Author: "Test Author",
Tags: []string{"Fantasy", "Adventure"},
Description: "Test description",
Publisher: "Test Publisher",
Series: "Test Series",
SeriesNumber: 1,
},
wantErr: false,
},
// ... more test cases
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Create temporary OPF file
tmpDir := t.TempDir()
opfPath := filepath.Join(tmpDir, "metadata.opf")
if err := os.WriteFile(opfPath, []byte(tt.opfContent), 0644); err != nil {
t.Fatal(err)
}
// Parse
scanner := &MediaScanner{}
got, err := scanner.parseCalibreMetadataOPF(opfPath)
if (err != nil) != tt.wantErr {
t.Errorf("parseCalibreMetadataOPF() error = %v, wantErr %v", err, tt.wantErr)
return
}
// Verify fields
if got.Title != tt.want.Title {
t.Errorf("Title = %v, want %v", got.Title, tt.want.Title)
}
// ... more assertions
})
}
}
```
**Test Cases**:
1. Complete metadata.opf (all fields)
2. Minimal metadata.opf (title only)
3. Malformed XML
4. Missing metadata.opf (returns nil)
5. Multiple `<dc:subject>` tags
6. Multiple `<dc:identifier>` tags
7. Invalid date formats
8. Empty/whitespace values
### Integration Tests
**File**: `cmd/server/tests/calibre_integration_test.go`
```go
func TestCalibreLibraryScan(t *testing.T) {
// Setup test server
setup := setupTestServer(t)
defer setup.Teardown()
// Create test Calibre library structure
tmpDir := t.TempDir()
createTestCalibreLibrary(t, tmpDir)
// Add library
libraryID := createTestLibrary(t, setup.DB, tmpDir)
// Scan library
scanner := services.NewMediaScanner(setup.DB, setup.AdminID, []string{tmpDir})
err := scanner.ScanFolders(context.Background())
if err != nil {
t.Fatalf("ScanFolders() error = %v", err)
}
// Verify imported books
books, err := setup.DB.ListMediaItems(context.Background(), libraryID)
if err != nil {
t.Fatalf("ListMediaItems() error = %v", err)
}
// Assertions
if len(books) != 3 {
t.Errorf("Got %d books, want 3", len(books))
}
// Verify metadata from sidecar
for _, book := range books {
if book.Title == "" {
t.Errorf("Book %s: Title should not be empty", book.ID)
}
if book.Series.Valid && book.SeriesNumber.Int32 == 0 {
t.Errorf("Book %s: Series number should be set", book.ID)
}
}
}
```
**Test Scenarios**:
1. Scan Calibre library (with sidecars)
2. Scan non-Calibre library (no sidecars)
3. Mixed library (some with sidecars, some without)
4. Force rescan updates from sidecar changes
5. All file types (EPUB, PDF, CBZ, MP3)
## Performance Considerations
### Impact on Scan Performance
**Expected overhead**: +5-10% scan time increase
**Optimizations**:
1. **File system cache**: OS caches `metadata.opf` reads
2. **Early exit**: Return immediately if sidecar not found
3. **Parallel processing**: Sidecar extraction doesn't block other files
4. **Lazy parsing**: Only parse XML when sidecar exists
**Benchmarking**:
```go
func BenchmarkSidecarExtraction(b *testing.B) {
scanner := &MediaScanner{}
tmpDir := b.TempDir()
createTestCalibreLibrary(b, tmpDir)
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := scanner.extractMetadataFromCalibreSidecar(filepath.Join(tmpDir, "book.epub"))
if err != nil {
b.Fatal(err)
}
}
}
```
### Memory Usage
**XML parsing**: Small memory footprint (OPF files are typically <10KB)
**No leaks**: Ensure file handles are closed with `defer file.Close()`
## Error Handling
### Graceful Degradation
The scanner must **never fail** due to sidecar issues:
```go
// Pattern: Log error, continue to next source
if metadata, err := s.extractMetadataFromCalibreSidecar(path); err != nil {
// Log warning
fmt.Printf("Warning: failed to read Calibre sidecar: %v\n", err)
// Continue to embedded extraction
metadata = s.extractMetadata(path)
}
```
### Error Types
| Error | Action | Log Level |
|-------|--------|-----------|
| File not found | Continue to embedded | DEBUG |
| Permission denied | Continue to embedded | WARN |
| Malformed XML | Continue to embedded | WARN |
| Invalid date format | Skip field | WARN |
| Empty required field | Use available fields | INFO |
| UTF-8 decode error | Continue to embedded | ERROR |
### Logging
Use structured logging:
```go
fmt.Printf("Using Calibre metadata.opf for %s\n", path)
fmt.Printf("Warning: failed to parse metadata.opf: %v\n", err)
fmt.Printf("Info: No sidecar found, using embedded metadata for %s\n", path)
```
## Future Enhancements
### Phase 4 (Optional)
1. **Custom column support**: Parse Calibre custom columns
2. **Smart merge**: Configurable merge strategy
3. **Rating import**: Import Calibre rating as user's initial rating
4. **Library detection**: Auto-detect Calibre library structure
5. **Incremental sync**: Only re-scan changed sidecar files
### Configuration Option
```go
type ScannerConfig struct {
PreferCalibreSidecar bool // Default: true
CalibreMergeStrategy string // "sidecar-first" | "smart-merge" | "embedded-first"
}
```
## References
- [EPUB 3.0 Spec](https://www.idpf.org/epub/30/spec/epub30-publications.html)
- [Dublin Core](https://www.dublincore.org/specifications/dces/1998-08-13/)
- [Calibre Manual](https://manual.calibre-ebook.com/)
- [Go XML Encoding](https://pkg.go.dev/encoding/xml)
## See Also
- [User Guide: Calibre Integration](../../user/calibre-integration.md)
- [Implementation Plan](../../CALIBRE_OPF_IMPLEMENTATION.md)
- [Scanner Source Code](../../internal/services/media_scanner.go)