docs: add comprehensive Calibre integration documentation
This commit adds complete documentation for the planned Calibre metadata.opf sidecar file support feature. ## New Documentation ### Implementation Planning - CALIBRE_OPF_IMPLEMENTATION.md: Detailed implementation plan with requirements, architecture, database mapping, and step-by-step implementation guide for adding Calibre metadata.opf support ### Technical Documentation - docs/development/calibre-opf-implementation.md: Technical implementation details including: - Scanner pipeline architecture with sidecar-first approach - Data structures (CalibreOPFMetadata, MediaMetadata) - Function signatures and logic for parseCalibreMetadataOPF() - Database schema mapping (no changes required) - Testing strategy (unit and integration tests) - Error handling and performance considerations - Code examples and benchmarking approach ### User Documentation - docs/user/calibre-integration.md: Comprehensive user guide covering: - What is Calibre and how Bookhoard integrates with it - Automatic metadata import from metadata.opf sidecar files - Supported metadata fields (Dublin Core + Calibre-specific) - Setup instructions for Calibre libraries - Workflow examples (fresh library, mixed library, updating metadata) - Troubleshooting common issues - Best practices for Calibre + Bookhoard workflow - FAQ and resources ## Updated Documentation - README.md: Added Calibre integration feature to media management section - docs/user/user-guide.md: Added link to Calibre integration guide - docs/developer/development.md: Added link to Calibre implementation guide ## Feature Summary The Calibre integration feature will allow Bookhoard to automatically import curated metadata from Calibre's metadata.opf sidecar files, including titles, authors, series, tags, descriptions, publishers, identifiers (ISBN/ASIN), and contributors. Uses sidecar-first approach: metadata.opf → embedded metadata → folder structure → filename. All database fields already exist; no schema changes required.
This commit is contained in:
@@ -0,0 +1,381 @@
|
|||||||
|
# Calibre metadata.opf Support - Implementation Plan
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This document outlines the implementation of **Calibre metadata.opf sidecar file support** for Bookhoard's media scanner. The scanner will use a **sidecar-first approach** - if a Calibre `metadata.opf` file exists alongside a book file, it will take precedence over embedded metadata.
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
Calibre is the most popular ebook management software, with millions of users. When users manage their libraries in Calibre:
|
||||||
|
|
||||||
|
1. **They curate metadata** - Adding series information, tags, ratings, custom covers
|
||||||
|
2. **They fetch from online sources** - Google Books, Amazon, Goodreads (better than publisher metadata)
|
||||||
|
3. **They convert/edit books** - The sidecar reflects the current, correct state
|
||||||
|
4. **They organize in folders** - Often using Calibre's folder structure conventions
|
||||||
|
|
||||||
|
By supporting Calibre `metadata.opf` sidecar files, Bookhoard can:
|
||||||
|
- Respect users' curation work
|
||||||
|
- Import richer, more accurate metadata
|
||||||
|
- Provide seamless integration with existing Calibre libraries
|
||||||
|
- Support all file types (EPUB, PDF, comics, etc.) that Calibre manages
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
### Functional Requirements
|
||||||
|
|
||||||
|
- [ ] Scanner detects `metadata.opf` files in the same directory as book files
|
||||||
|
- [ ] Parses Dublin Core metadata (`dc:*` namespace) and Calibre-specific fields (`meta name="calibre:*"`)
|
||||||
|
- [ ] Uses sidecar metadata **before** embedded metadata (sidecar-first approach)
|
||||||
|
- [ ] Works for all library types (ebook, audiobook, comic, etc.)
|
||||||
|
- [ ] Works for all file types (EPUB, PDF, CBZ, CBR, MP3, M4B, etc.)
|
||||||
|
- [ ] Handles missing or malformed sidecar files gracefully
|
||||||
|
- [ ] Maps all OPF fields to existing `media_items` database schema
|
||||||
|
- [ ] Maintains backward compatibility (no sidecar = existing behavior)
|
||||||
|
|
||||||
|
### Non-Functional Requirements
|
||||||
|
|
||||||
|
- [ ] No breaking changes to existing scanner functionality
|
||||||
|
- [ ] No database schema changes required (all fields already exist)
|
||||||
|
- [ ] Follows existing code patterns and style
|
||||||
|
- [ ] Comprehensive test coverage
|
||||||
|
- [ ] Performance: Minimal impact on scan performance
|
||||||
|
- [ ] Error handling: Graceful degradation on malformed XML
|
||||||
|
|
||||||
|
## Database Schema Mapping
|
||||||
|
|
||||||
|
All required fields **already exist** in the `media_items` table. No schema changes needed.
|
||||||
|
|
||||||
|
### Dublin Core Fields (dc: namespace)
|
||||||
|
|
||||||
|
| OPF Field | Database Column | Type | Notes |
|
||||||
|
|-----------|-----------------|------|-------|
|
||||||
|
| `dc:title` | `title` | VARCHAR(255) | Required |
|
||||||
|
| `dc:creator` | `author` | VARCHAR(255) | First author only |
|
||||||
|
| `dc:subject` | `tags` | TEXT[] | **Tags/keywords** (multiple) |
|
||||||
|
| `dc:description` | `description` | TEXT | Book summary |
|
||||||
|
| `dc:publisher` | `publisher` | VARCHAR(255) | Publisher name |
|
||||||
|
| `dc:date` | `date_published` | DATE | Publication date |
|
||||||
|
| `dc:language` | `language` | VARCHAR(10) | ISO 639-1 code (en, es, fr) |
|
||||||
|
| `dc:identifier` (ISBN) | `isbn` | VARCHAR(13) | Check `opf:scheme="ISBN"` |
|
||||||
|
| `dc:identifier` (ASIN) | `asin` | VARCHAR(20) | Check `opf:scheme="ASIN"` |
|
||||||
|
| `dc:contributor` | `contributors` | TEXT[] | Multiple contributors |
|
||||||
|
| `dc:rights` | *(not stored)* | - | Copyright info (ignored) |
|
||||||
|
| `dc:source` | *(not stored)* | - | Source publication (ignored) |
|
||||||
|
|
||||||
|
### Calibre-Specific Fields (meta name="calibre:*")
|
||||||
|
|
||||||
|
| OPF Field | Database Column | Type | Notes |
|
||||||
|
|-----------|-----------------|------|-------|
|
||||||
|
| `calibre:series` | `series` | VARCHAR(255) | Series name |
|
||||||
|
| `calibre:series_index` | `series_number` | INTEGER | Position in series |
|
||||||
|
| `calibre:rating` | *(use media_ratings)* | INTEGER | 0-5 scale, stored per-user |
|
||||||
|
| `calibre:title_sort` | *(not stored)* | - | Sortable title (ignored) |
|
||||||
|
| `calibre:author_sort` | *(not stored)* | - | Sortable author (ignored) |
|
||||||
|
| `calibre:timestamp` | `created_at` | TIMESTAMP | When added to Calibre |
|
||||||
|
|
||||||
|
### Special Cases
|
||||||
|
|
||||||
|
1. **Multiple authors**: Store first author in `author`, suggest adding `authors[]` array in future
|
||||||
|
2. **Multiple identifiers**: Extract ISBN and ASIN if present, ignore others
|
||||||
|
3. **Calibre rating**: Not imported (ratings are per-user in Bookhoard)
|
||||||
|
4. **Sort fields**: Ignored (Bookhoard has its own sorting logic)
|
||||||
|
5. **Timestamp**: Use for `created_at` if more recent than file modification time
|
||||||
|
|
||||||
|
## OPF Format Reference
|
||||||
|
|
||||||
|
### Standard Dublin Core Elements
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<dc:title>Book Title</dc:title>
|
||||||
|
<dc:creator>Author Name</dc:creator>
|
||||||
|
<dc:subject>Fantasy</dc:subject> <!-- Tag -->
|
||||||
|
<dc:subject>Adventure</dc:subject> <!-- Another tag -->
|
||||||
|
<dc:description>Book summary...</dc:description>
|
||||||
|
<dc:publisher>Publisher Name</dc:publisher>
|
||||||
|
<dc:date>2024-01-15</dc:date>
|
||||||
|
<dc:language>en</dc:language>
|
||||||
|
<dc:identifier opf:scheme="ISBN">978-0-123456-78-9</dc:identifier>
|
||||||
|
<dc:identifier opf:scheme="ASIN">B08XXXXX</dc:identifier>
|
||||||
|
<dc:contributor>Illustrator Name</dc:contributor>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Calibre-Specific Meta Tags
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<meta name="calibre:series" content="Series Name"/>
|
||||||
|
<meta name="calibre:series_index" content="3"/>
|
||||||
|
<meta name="calibre:rating" content="4"/>
|
||||||
|
<meta name="calibre:title_sort" content="Book Title"/>
|
||||||
|
<meta name="calibre:author_sort" content="Author, Name"/>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example Complete metadata.opf
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<?xml version='1.0' encoding='utf-8'?>
|
||||||
|
<package xmlns="http://www.idpf.org/2007/opf" version="2.0" unique-identifier="uuid_id">
|
||||||
|
<metadata xmlns:opf="http://www.idpf.org/2007/opf"
|
||||||
|
xmlns:calibre="http://calibre.kovidgoyal.net/2009/metadata"
|
||||||
|
xmlns:dc="http://purl.org/dc/elements/1.1/">
|
||||||
|
<dc:title>The Fellowship of the Ring</dc:title>
|
||||||
|
<dc:creator opf:role="aut">J.R.R. Tolkien</dc:creator>
|
||||||
|
<dc:identifier opf:scheme="ISBN">978-0-618-00222-0</dc:identifier>
|
||||||
|
<dc:identifier opf:scheme="calibre">12345678-1234-1234-1234-123456789012</dc:identifier>
|
||||||
|
<dc:language>en</dc:language>
|
||||||
|
<dc:date>1954-07-29</dc:date>
|
||||||
|
<dc:publisher>HarperCollins</dc:publisher>
|
||||||
|
<dc:description>In a sleepy village in the Shire...</dc:description>
|
||||||
|
|
||||||
|
<!-- Tags -->
|
||||||
|
<dc:subject>Fantasy</dc:subject>
|
||||||
|
<dc:subject>Adventure</dc:subject>
|
||||||
|
<dc:subject>Classics</dc:subject>
|
||||||
|
|
||||||
|
<!-- Contributors -->
|
||||||
|
<dc:contributor>Alan Lee (illustrator)</dc:contributor>
|
||||||
|
|
||||||
|
<!-- Calibre-specific fields -->
|
||||||
|
<meta name="calibre:series" content="The Lord of the Rings"/>
|
||||||
|
<meta name="calibre:series_index" content="1"/>
|
||||||
|
<meta name="calibre:rating" content="5"/>
|
||||||
|
<meta name="calibre:title_sort" content="Fellowship of the Ring"/>
|
||||||
|
</metadata>
|
||||||
|
</package>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Implementation Design
|
||||||
|
|
||||||
|
### Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Scanner Pipeline (EXISTING)
|
||||||
|
↓
|
||||||
|
1. Detect book file
|
||||||
|
2. Check if already in database
|
||||||
|
3. Extract metadata from file (EPUB/PDF/Comic)
|
||||||
|
4. Extract folder structure metadata
|
||||||
|
5. Normalize and store
|
||||||
|
```
|
||||||
|
|
||||||
|
**New Step 3a**: Check for Calibre sidecar **before** embedded extraction
|
||||||
|
|
||||||
|
```
|
||||||
|
Scanner Pipeline (NEW)
|
||||||
|
↓
|
||||||
|
1. Detect book file
|
||||||
|
2. Check if already in database
|
||||||
|
3. **NEW** Extract metadata from Calibre metadata.opf (if exists)
|
||||||
|
- If sidecar found → use sidecar metadata
|
||||||
|
- If no sidecar → use embedded metadata (existing)
|
||||||
|
4. Extract folder structure metadata (existing)
|
||||||
|
5. Normalize and store (existing)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Code Changes
|
||||||
|
|
||||||
|
#### File: `internal/services/media_scanner.go`
|
||||||
|
|
||||||
|
**New Functions:**
|
||||||
|
|
||||||
|
1. `extractMetadataFromCalibreSidecar(path string) (*MediaMetadata, error)`
|
||||||
|
- Checks for `metadata.opf` in same directory as book file
|
||||||
|
- Calls `parseCalibreMetadataOPF()` if found
|
||||||
|
- Returns nil if no sidecar exists (not an error)
|
||||||
|
|
||||||
|
2. `parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error)`
|
||||||
|
- Opens and parses XML file
|
||||||
|
- Extracts Dublin Core fields (`dc:*`)
|
||||||
|
- Extracts Calibre-specific fields (`meta name="calibre:*"`)
|
||||||
|
- Maps to `MediaMetadata` struct
|
||||||
|
- Handles errors gracefully (malformed XML, missing fields)
|
||||||
|
|
||||||
|
**Modified Function:**
|
||||||
|
|
||||||
|
1. `processMediaFile(ctx context.Context, path string) (bool, error)`
|
||||||
|
- Add sidecar extraction check at line ~544 (before embedded extraction)
|
||||||
|
- Flow: sidecar → embedded → folder → filename
|
||||||
|
|
||||||
|
#### New Struct: `CalibreOPFMetadata`
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Intermediate struct for parsed OPF data
|
||||||
|
type CalibreOPFMetadata struct {
|
||||||
|
Title string
|
||||||
|
Authors []string
|
||||||
|
Tags []string
|
||||||
|
Description string
|
||||||
|
Publisher string
|
||||||
|
PublishDate *time.Time
|
||||||
|
Language string
|
||||||
|
ISBN string
|
||||||
|
ASIN string
|
||||||
|
UUID string
|
||||||
|
Contributors []string
|
||||||
|
Series string
|
||||||
|
SeriesIndex *float64
|
||||||
|
Rating *int32
|
||||||
|
Timestamp *time.Time
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Metadata Priority (Sidecar-First)
|
||||||
|
|
||||||
|
```
|
||||||
|
Priority Order:
|
||||||
|
1. Calibre metadata.opf sidecar ← NEW (highest priority)
|
||||||
|
2. Embedded file metadata (EPUB OPF, PDF info, ComicInfo.xml)
|
||||||
|
3. Folder structure parsing
|
||||||
|
4. Filename parsing (last resort)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rationale**: If a user has a `metadata.opf` file, they're actively managing their library in Calibre. The sidecar represents their curated "source of truth."
|
||||||
|
|
||||||
|
### Error Handling
|
||||||
|
|
||||||
|
1. **Sidecar file not found** → Return nil, continue to embedded metadata
|
||||||
|
2. **Malformed XML** → Log warning, continue to embedded metadata
|
||||||
|
3. **Missing required fields** → Use available fields, continue
|
||||||
|
4. **Invalid date format** → Log warning, skip field
|
||||||
|
5. **File permission errors** → Log error, continue to embedded metadata
|
||||||
|
|
||||||
|
**Principle**: Graceful degradation - never fail a scan due to sidecar issues
|
||||||
|
|
||||||
|
### Testing Strategy
|
||||||
|
|
||||||
|
#### Unit Tests
|
||||||
|
|
||||||
|
File: `internal/services/media_scanner_calibre_test.go`
|
||||||
|
|
||||||
|
- Test parsing complete metadata.opf with all fields
|
||||||
|
- Test parsing minimal metadata.opf (title only)
|
||||||
|
- Test parsing malformed XML (graceful failure)
|
||||||
|
- Test missing metadata.opf (returns nil, not error)
|
||||||
|
- Test multiple `<dc:subject>` tags (tags array)
|
||||||
|
- Test multiple `<dc:identifier>` tags (extract ISBN/ASIN)
|
||||||
|
- Test Calibre-specific fields (series, series_index)
|
||||||
|
- Test invalid date formats
|
||||||
|
- Test empty/whitespace values
|
||||||
|
|
||||||
|
#### Integration Tests
|
||||||
|
|
||||||
|
File: `cmd/server/tests/calibre_integration_test.go`
|
||||||
|
|
||||||
|
- Test scanning directory with Calibre library structure
|
||||||
|
- Test scanning directory without sidecar files (existing behavior)
|
||||||
|
- Test mixed: some books with sidecar, some without
|
||||||
|
- Test force rescan updates from sidecar changes
|
||||||
|
- Test all file types: EPUB, PDF, CBZ, MP3, M4B
|
||||||
|
- Test all library types: ebook, audiobook, comic
|
||||||
|
|
||||||
|
#### Test Data
|
||||||
|
|
||||||
|
Create `testdata/calibre/` with sample files:
|
||||||
|
- `Complete Book/metadata.opf` - All fields populated
|
||||||
|
- `Minimal Book/metadata.opf` - Only required fields
|
||||||
|
- `Malformed Book/metadata.opf` - Invalid XML
|
||||||
|
- No sidecar files (rely on existing test data)
|
||||||
|
|
||||||
|
## Implementation Steps
|
||||||
|
|
||||||
|
### Phase 1: Core Functionality (REQUIRED)
|
||||||
|
|
||||||
|
- [ ] **Step 1.1**: Add `CalibreOPFMetadata` struct
|
||||||
|
- [ ] **Step 1.2**: Implement `parseCalibreMetadataOPF()` function
|
||||||
|
- [ ] **Step 1.3**: Implement `extractMetadataFromCalibreSidecar()` function
|
||||||
|
- [ ] **Step 1.4**: Modify `processMediaFile()` to call sidecar extraction
|
||||||
|
- [ ] **Step 1.5**: Add unit tests for OPF parsing
|
||||||
|
- [ ] **Step 1.6**: Test with real Calibre library
|
||||||
|
|
||||||
|
### Phase 2: Documentation (REQUIRED)
|
||||||
|
|
||||||
|
- [ ] **Step 2.1**: Create user documentation (`docs/user/calibre-support.md`)
|
||||||
|
- [ ] **Step 2.2**: Update developer documentation (`docs/development/calibre-opf-implementation.md`)
|
||||||
|
- [ ] **Step 2.3**: Update scanner API docs if needed
|
||||||
|
- [ ] **Step 2.4**: Add examples to documentation
|
||||||
|
|
||||||
|
### Phase 3: Testing & Polish (REQUIRED)
|
||||||
|
|
||||||
|
- [ ] **Step 3.1**: Add integration tests
|
||||||
|
- [ ] **Step 3.2**: Test with various Calibre library configurations
|
||||||
|
- [ ] **Step 3.3**: Performance testing (ensure no regression)
|
||||||
|
- [ ] **Step 3.4**: Error handling review
|
||||||
|
- [ ] **Step 3.5**: Code review and refinement
|
||||||
|
|
||||||
|
### Phase 4: Future Enhancements (OPTIONAL)
|
||||||
|
|
||||||
|
- [ ] Support Calibre `author_link` metadata
|
||||||
|
- [ ] Support Calibre custom columns
|
||||||
|
- [ ] Import Calibre ratings as user's initial rating
|
||||||
|
- [ ] Detect Calibre library structure patterns
|
||||||
|
- [ ] Configuration option for merge strategy (sidecar-first vs. smart-merge)
|
||||||
|
|
||||||
|
## Backward Compatibility
|
||||||
|
|
||||||
|
**No breaking changes.** Existing behavior is preserved:
|
||||||
|
|
||||||
|
- Libraries without `metadata.opf` files → Existing behavior (embedded metadata)
|
||||||
|
- Scanning logic unchanged → Only adds sidecar check before embedded extraction
|
||||||
|
- Database unchanged → All fields already exist
|
||||||
|
- API unchanged → No new endpoints or response fields
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
- [ ] All existing tests pass
|
||||||
|
- [ ] New unit tests pass (90%+ coverage)
|
||||||
|
- [ ] New integration tests pass
|
||||||
|
- [ ] Scans real Calibre library successfully
|
||||||
|
- [ ] Scans non-Calibre library without errors
|
||||||
|
- [ ] No performance regression (scan time +5% max)
|
||||||
|
- [ ] Documentation complete and accurate
|
||||||
|
- [ ] Code review approved
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. **Q**: Should we import Calibre ratings?
|
||||||
|
**A**: No - ratings are per-user in Bookhoard. Calibre rating is the owner's opinion, not necessarily the current user's.
|
||||||
|
|
||||||
|
2. **Q**: Should we support Calibre custom columns?
|
||||||
|
**A**: Not in initial implementation. Custom columns are user-defined and would require dynamic schema or JSON storage. Future enhancement.
|
||||||
|
|
||||||
|
3. **Q**: How to handle multiple authors?
|
||||||
|
**A**: Store first author in `author` field (existing behavior). Future enhancement could add `authors[]` array.
|
||||||
|
|
||||||
|
4. **Q**: Should we follow Calibre folder structure?
|
||||||
|
**A**: Already implemented! `extractFolderStructureMetadata()` handles Calibre patterns.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [EPUB Publications 3.0 Specification](https://www.idpf.org/epub/30/spec/epub30-publications.html)
|
||||||
|
- [Dublin Core Metadata Element Set](https://www.dublincore.org/specifications/dces/1998-08-13/)
|
||||||
|
- [Calibre Manual - Editing Metadata](https://manual.calibre-ebook.com/metadata.html)
|
||||||
|
- [Calibre Source Code](https://github.com/kovidgoyal/calibre)
|
||||||
|
|
||||||
|
## Appendix: Decision Log
|
||||||
|
|
||||||
|
### Decision 1: Sidecar-First Approach
|
||||||
|
**Date**: 2026-03-26
|
||||||
|
**Decision**: Use sidecar-first (not smart-merge)
|
||||||
|
**Rationale**:
|
||||||
|
- Users with Calibre sidecars actively curate their metadata
|
||||||
|
- Sidecar metadata is typically richer and more accurate
|
||||||
|
- Simple implementation, predictable behavior
|
||||||
|
- Easy to test and debug
|
||||||
|
- Future enhancement can add configurable merge strategy
|
||||||
|
|
||||||
|
### Decision 2: No Database Schema Changes
|
||||||
|
**Date**: 2026-03-26
|
||||||
|
**Decision**: Use existing schema only
|
||||||
|
**Rationale**:
|
||||||
|
- All required fields already exist in `media_items` table
|
||||||
|
- Pre-production application (no legacy data to migrate)
|
||||||
|
- Simpler implementation
|
||||||
|
- No backward compatibility concerns
|
||||||
|
|
||||||
|
### Decision 3: All Library Types Supported
|
||||||
|
**Date**: 2026-03-26
|
||||||
|
**Decision**: Support OPF for all libraries, not just ebooks
|
||||||
|
**Rationale**:
|
||||||
|
- Calibre supports all file types
|
||||||
|
- Users may organize audiobooks/comics in Calibre
|
||||||
|
- Consistent behavior across library types
|
||||||
|
- No additional complexity
|
||||||
@@ -60,6 +60,7 @@ The first user to register automatically becomes an admin.
|
|||||||
|
|
||||||
### Media Management
|
### Media Management
|
||||||
|
|
||||||
|
- **Calibre Integration**: Automatic metadata import from Calibre `metadata.opf` sidecar files
|
||||||
- **Smart Search**: Partial matching with fuzzy search fallback for typos
|
- **Smart Search**: Partial matching with fuzzy search fallback for typos
|
||||||
- **Advanced Filtering**: Filter by author, series, genre, language, year, cover images
|
- **Advanced Filtering**: Filter by author, series, genre, language, year, cover images
|
||||||
- **Dynamic Sorting**: By title, author, date added, published date, page count, series
|
- **Dynamic Sorting**: By title, author, date added, published date, page count, series
|
||||||
@@ -97,6 +98,7 @@ The first user to register automatically becomes an admin.
|
|||||||
|
|
||||||
### For Users & Self-Hosters
|
### For Users & Self-Hosters
|
||||||
|
|
||||||
|
- **[docs/user/calibre-integration.md](docs/user/calibre-integration.md)** - Calibre library integration
|
||||||
- **[docs/user/sync-guide.md](docs/user/sync-guide.md)** - Understanding and using universal sync
|
- **[docs/user/sync-guide.md](docs/user/sync-guide.md)** - Understanding and using universal sync
|
||||||
- **[docs/user/devices/kobo-setup.md](docs/user/devices/kobo-setup.md)** - Kobo e-reader configuration
|
- **[docs/user/devices/kobo-setup.md](docs/user/devices/kobo-setup.md)** - Kobo e-reader configuration
|
||||||
- **[docs/user/devices/koreader-setup.md](docs/user/devices/koreader-setup.md)** - KOReader configuration
|
- **[docs/user/devices/koreader-setup.md](docs/user/devices/koreader-setup.md)** - KOReader configuration
|
||||||
|
|||||||
@@ -71,6 +71,18 @@ Browse individual endpoint documentation with interactive API Explorer:
|
|||||||
- Event types
|
- Event types
|
||||||
- Authentication
|
- Authentication
|
||||||
|
|
||||||
|
## 🛠 Implementation Guides
|
||||||
|
|
||||||
|
### Calibre Integration
|
||||||
|
|
||||||
|
- **[Calibre OPF Implementation](../development/calibre-opf-implementation.md)** - Technical implementation details
|
||||||
|
- Architecture and design
|
||||||
|
- Data structures and functions
|
||||||
|
- Database schema mapping
|
||||||
|
- Testing strategy
|
||||||
|
- Error handling
|
||||||
|
- Performance considerations
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
**Looking for user documentation?** See the [User Portal](../user/user-guide.md)
|
**Looking for user documentation?** See the [User Portal](../user/user-guide.md)
|
||||||
|
|||||||
@@ -0,0 +1,576 @@
|
|||||||
|
# Calibre metadata.opf Implementation
|
||||||
|
|
||||||
|
This document provides technical details for implementing Calibre `metadata.opf` sidecar file support in Bookhoard's media scanner.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The media scanner detects and imports metadata from Calibre's `metadata.opf` sidecar files using a **sidecar-first approach**. If a `metadata.opf` file exists alongside a book file, its metadata takes precedence over embedded file metadata.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Scanner Pipeline
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (s *MediaScanner) processMediaFile(ctx context.Context, path string) (bool, error) {
|
||||||
|
// 1. Get file info
|
||||||
|
// 2. Find library
|
||||||
|
// 3. Check if media item exists
|
||||||
|
// 4. EXTRACT METADATA (NEW: sidecar-first)
|
||||||
|
metadata := s.extractMetadataWithSidecar(path)
|
||||||
|
// 5. Extract hash info
|
||||||
|
// 6. Extract comic metadata (if applicable)
|
||||||
|
// 7. Extract folder structure metadata
|
||||||
|
// 8. Normalize metadata
|
||||||
|
// 9. Create or update in database
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Metadata Extraction Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
extractMetadataWithSidecar()
|
||||||
|
↓
|
||||||
|
1. Try Calibre metadata.opf sidecar
|
||||||
|
├─ Found → parseCalibreMetadataOPF()
|
||||||
|
└─ Not found → continue
|
||||||
|
↓
|
||||||
|
2. Extract embedded metadata
|
||||||
|
├─ EPUB → extractEPUBMetadata()
|
||||||
|
├─ PDF → extractPDFMetadata()
|
||||||
|
└─ Comic → extractComicMetadata()
|
||||||
|
↓
|
||||||
|
3. Extract folder structure metadata
|
||||||
|
└─ extractFolderStructureMetadata()
|
||||||
|
↓
|
||||||
|
4. Return merged MediaMetadata
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data Structures
|
||||||
|
|
||||||
|
### CalibreOPFMetadata
|
||||||
|
|
||||||
|
Intermediate struct for parsed OPF data:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// CalibreOPFMetadata represents parsed metadata from Calibre metadata.opf
|
||||||
|
type CalibreOPFMetadata struct {
|
||||||
|
Title string
|
||||||
|
Authors []string
|
||||||
|
Tags []string
|
||||||
|
Description string
|
||||||
|
Publisher string
|
||||||
|
PublishDate *time.Time
|
||||||
|
Language string
|
||||||
|
ISBN string
|
||||||
|
ASIN string
|
||||||
|
UUID string
|
||||||
|
Contributors []string
|
||||||
|
Series string
|
||||||
|
SeriesIndex *float64
|
||||||
|
Rating *int32
|
||||||
|
Timestamp *time.Time
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### MediaMetadata
|
||||||
|
|
||||||
|
Existing struct (already defined in `media_scanner.go`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
// MediaMetadata represents extracted book metadata
|
||||||
|
type MediaMetadata struct {
|
||||||
|
Title string
|
||||||
|
Author string
|
||||||
|
ISBN string
|
||||||
|
Description string
|
||||||
|
Publisher string
|
||||||
|
Series string
|
||||||
|
SeriesNumber int32
|
||||||
|
Tags []string
|
||||||
|
Contributors []string
|
||||||
|
PublishDate time.Time
|
||||||
|
ASIN string
|
||||||
|
// ... other fields
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Implementation Functions
|
||||||
|
|
||||||
|
### 1. extractMetadataFromCalibreSidecar()
|
||||||
|
|
||||||
|
**Location**: `internal/services/media_scanner.go`
|
||||||
|
|
||||||
|
**Signature**:
|
||||||
|
```go
|
||||||
|
func (s *MediaScanner) extractMetadataFromCalibreSidecar(mediaFilePath string) (*MediaMetadata, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Purpose**: Checks for `metadata.opf` in the same directory as the book file.
|
||||||
|
|
||||||
|
**Logic**:
|
||||||
|
```go
|
||||||
|
func (s *MediaScanner) extractMetadataFromCalibreSidecar(mediaFilePath string) (*MediaMetadata, error) {
|
||||||
|
// Get directory of media file
|
||||||
|
dir := filepath.Dir(mediaFilePath)
|
||||||
|
|
||||||
|
// Check for metadata.opf
|
||||||
|
opfPath := filepath.Join(dir, "metadata.opf")
|
||||||
|
if _, err := os.Stat(opfPath); os.IsNotExist(err) {
|
||||||
|
// No sidecar file - not an error, just return nil
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse metadata.opf
|
||||||
|
return s.parseCalibreMetadataOPF(opfPath)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Error Handling**:
|
||||||
|
- File not found → Return `nil, nil` (not an error)
|
||||||
|
- Permission denied → Log warning, return `nil, nil`
|
||||||
|
- Other errors → Log error, return `nil, err`
|
||||||
|
|
||||||
|
### 2. parseCalibreMetadataOPF()
|
||||||
|
|
||||||
|
**Location**: `internal/services/media_scanner.go`
|
||||||
|
|
||||||
|
**Signature**:
|
||||||
|
```go
|
||||||
|
func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Purpose**: Parses XML file and extracts Dublin Core + Calibre-specific fields.
|
||||||
|
|
||||||
|
**Logic**:
|
||||||
|
```go
|
||||||
|
func (s *MediaScanner) parseCalibreMetadataOPF(opfPath string) (*MediaMetadata, error) {
|
||||||
|
// Open file
|
||||||
|
file, err := os.Open(opfPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to open metadata.opf: %v", err)
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
|
||||||
|
// Parse XML
|
||||||
|
var opf struct {
|
||||||
|
XMLName xml.Name `xml:"package"`
|
||||||
|
Metadata struct {
|
||||||
|
XMLName xml.Name `xml:"metadata"`
|
||||||
|
Titles []string `xml:"dc:title"`
|
||||||
|
Creators []string `xml:"dc:creator"`
|
||||||
|
Subjects []string `xml:"dc:subject"`
|
||||||
|
Desc []string `xml:"dc:description"`
|
||||||
|
Publisher []string `xml:"dc:publisher"`
|
||||||
|
Dates []string `xml:"dc:date"`
|
||||||
|
Language []string `xml:"dc:language"`
|
||||||
|
Identifiers []struct {
|
||||||
|
Scheme string `xml:"opf:scheme,attr"`
|
||||||
|
Value string `xml:",chardata"`
|
||||||
|
} `xml:"dc:identifier"`
|
||||||
|
Contributors []string `xml:"dc:contributor"`
|
||||||
|
CalibreSeries []struct {
|
||||||
|
Name string `xml:"content,attr"`
|
||||||
|
} `xml:"meta[name='calibre:series']"`
|
||||||
|
CalibreSeriesIndex []struct {
|
||||||
|
Value string `xml:"content,attr"`
|
||||||
|
} `xml:"meta[name='calibre:series_index']"`
|
||||||
|
} `xml:"metadata"`
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := xml.NewDecoder(file).Decode(&opf); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to parse metadata.opf XML: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Map to MediaMetadata
|
||||||
|
metadata := &MediaMetadata{}
|
||||||
|
|
||||||
|
// Title (required)
|
||||||
|
if len(opf.Metadata.Titles) > 0 {
|
||||||
|
metadata.Title = opf.Metadata.Titles[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
// Author (first creator)
|
||||||
|
if len(opf.Metadata.Creators) > 0 {
|
||||||
|
metadata.Author = opf.Metadata.Creators[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tags (all subjects)
|
||||||
|
if len(opf.Metadata.Subjects) > 0 {
|
||||||
|
metadata.Tags = utils.NormalizeTags(opf.Metadata.Subjects)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Description
|
||||||
|
if len(opf.Metadata.Desc) > 0 {
|
||||||
|
metadata.Description = opf.Metadata.Desc[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
// Publisher
|
||||||
|
if len(opf.Metadata.Publisher) > 0 {
|
||||||
|
metadata.Publisher = opf.Metadata.Publisher[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
// Publish date
|
||||||
|
if len(opf.Metadata.Dates) > 0 {
|
||||||
|
if date, err := time.Parse("2006-01-02", opf.Metadata.Dates[0]); err == nil {
|
||||||
|
metadata.PublishDate = date
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Language
|
||||||
|
if len(opf.Metadata.Language) > 0 {
|
||||||
|
// Store language (already in schema)
|
||||||
|
// Note: MediaMetadata doesn't have Language field yet
|
||||||
|
// Future enhancement: add Language field
|
||||||
|
}
|
||||||
|
|
||||||
|
// Identifiers (ISBN, ASIN)
|
||||||
|
for _, id := range opf.Metadata.Identifiers {
|
||||||
|
switch strings.ToUpper(id.Scheme) {
|
||||||
|
case "ISBN":
|
||||||
|
metadata.ISBN = utils.NormalizeISBNSafe(id.Value)
|
||||||
|
case "ASIN":
|
||||||
|
metadata.ASIN = id.Value
|
||||||
|
case "UUID", "CALIBRE":
|
||||||
|
// Store UUID in hash info, not metadata
|
||||||
|
// Will be extracted by extractHashInfo()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Contributors
|
||||||
|
if len(opf.Metadata.Contributors) > 0 {
|
||||||
|
metadata.Contributors = utils.NormalizeContributors(opf.Metadata.Contributors)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Series
|
||||||
|
if len(opf.Metadata.CalibreSeries) > 0 {
|
||||||
|
metadata.Series = opf.Metadata.CalibreSeries[0].Name
|
||||||
|
}
|
||||||
|
|
||||||
|
// Series index
|
||||||
|
if len(opf.Metadata.CalibreSeriesIndex) > 0 {
|
||||||
|
if index, err := strconv.ParseFloat(opf.Metadata.CalibreSeriesIndex[0].Value, 32); err == nil {
|
||||||
|
metadata.SeriesNumber = int32(index)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return metadata, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**XML Namespaces**: Handle Dublin Core (`dc:`) and Calibre (`calibre:`) namespaces properly.
|
||||||
|
|
||||||
|
**Error Handling**:
|
||||||
|
- Malformed XML → Log warning, return `nil, nil`
|
||||||
|
- Missing required fields → Use available fields
|
||||||
|
- Invalid date format → Log warning, skip field
|
||||||
|
|
||||||
|
### 3. extractMetadataWithSidecar()
|
||||||
|
|
||||||
|
**Location**: `internal/services/media_scanner.go`
|
||||||
|
|
||||||
|
**Signature**:
|
||||||
|
```go
|
||||||
|
func (s *MediaScanner) extractMetadataWithSidecar(path string) (*MediaMetadata, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Purpose**: Implements sidecar-first metadata extraction.
|
||||||
|
|
||||||
|
**Logic**:
|
||||||
|
```go
|
||||||
|
func (s *MediaScanner) extractMetadataWithSidecar(path string) (*MediaMetadata, error) {
|
||||||
|
// Try Calibre sidecar first
|
||||||
|
if metadata, err := s.extractMetadataFromCalibreSidecar(path); metadata != nil {
|
||||||
|
fmt.Printf("Using Calibre metadata.opf for %s\n", path)
|
||||||
|
return metadata, nil
|
||||||
|
} else if err != nil {
|
||||||
|
// Log error but continue to embedded extraction
|
||||||
|
fmt.Printf("Warning: failed to read Calibre sidecar: %v\n", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fallback to embedded metadata
|
||||||
|
return s.extractMetadata(path)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Priority**: Sidecar → Embedded → Folder → Filename
|
||||||
|
|
||||||
|
## Database Schema Mapping
|
||||||
|
|
||||||
|
All OPF fields map to existing `media_items` columns:
|
||||||
|
|
||||||
|
| OPF Field | Database Column | Type | Conversion |
|
||||||
|
|-----------|-----------------|------|------------|
|
||||||
|
| `dc:title` | `title` | VARCHAR(255) | Direct |
|
||||||
|
| `dc:creator` | `author` | VARCHAR(255) | First author only |
|
||||||
|
| `dc:subject` | `tags` | TEXT[] | Array conversion |
|
||||||
|
| `dc:description` | `description` | TEXT | Direct |
|
||||||
|
| `dc:publisher` | `publisher` | VARCHAR(255) | Direct |
|
||||||
|
| `dc:date` | `date_published` | DATE | Parse ISO 8601 |
|
||||||
|
| `dc:language` | `language` | VARCHAR(10) | Store ISO code |
|
||||||
|
| `dc:identifier[ISBN]` | `isbn` | VARCHAR(13) | Normalize with utils |
|
||||||
|
| `dc:identifier[ASIN]` | `asin` | VARCHAR(20) | Direct |
|
||||||
|
| `dc:contributor` | `contributors` | TEXT[] | Array conversion |
|
||||||
|
| `calibre:series` | `series` | VARCHAR(255) | Direct |
|
||||||
|
| `calibre:series_index` | `series_number` | INTEGER | Parse float → int |
|
||||||
|
|
||||||
|
**No schema changes required** - all fields already exist.
|
||||||
|
|
||||||
|
## Normalization
|
||||||
|
|
||||||
|
Use existing normalization utilities:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Tags
|
||||||
|
metadata.Tags = utils.NormalizeTags(tags)
|
||||||
|
|
||||||
|
// Contributors
|
||||||
|
metadata.Contributors = utils.NormalizeContributors(contributors)
|
||||||
|
|
||||||
|
// ISBN
|
||||||
|
metadata.ISBN = utils.NormalizeISBNSafe(isbn)
|
||||||
|
|
||||||
|
// Search versions (auto-generated by database triggers)
|
||||||
|
// tags_search, contributors_search
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Unit Tests
|
||||||
|
|
||||||
|
**File**: `internal/services/media_scanner_calibre_test.go`
|
||||||
|
|
||||||
|
```go
|
||||||
|
func TestParseCalibreMetadataOPF(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
opfContent string
|
||||||
|
want *MediaMetadata
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "Complete metadata.opf",
|
||||||
|
opfContent: `<?xml version='1.0' encoding='utf-8'?>
|
||||||
|
<package xmlns="http://www.idpf.org/2007/opf" version="2.0">
|
||||||
|
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
|
||||||
|
<dc:title>Test Book</dc:title>
|
||||||
|
<dc:creator>Test Author</dc:creator>
|
||||||
|
<dc:subject>Fantasy</dc:subject>
|
||||||
|
<dc:subject>Adventure</dc:subject>
|
||||||
|
<dc:description>Test description</dc:description>
|
||||||
|
<dc:publisher>Test Publisher</dc:publisher>
|
||||||
|
<dc:date>2024-01-15</dc:date>
|
||||||
|
<dc:language>en</dc:language>
|
||||||
|
<dc:identifier opf:scheme="ISBN">978-0-123456-78-9</dc:identifier>
|
||||||
|
<meta name="calibre:series" content="Test Series"/>
|
||||||
|
<meta name="calibre:series_index" content="1"/>
|
||||||
|
</metadata>
|
||||||
|
</package>`,
|
||||||
|
want: &MediaMetadata{
|
||||||
|
Title: "Test Book",
|
||||||
|
Author: "Test Author",
|
||||||
|
Tags: []string{"Fantasy", "Adventure"},
|
||||||
|
Description: "Test description",
|
||||||
|
Publisher: "Test Publisher",
|
||||||
|
Series: "Test Series",
|
||||||
|
SeriesNumber: 1,
|
||||||
|
},
|
||||||
|
wantErr: false,
|
||||||
|
},
|
||||||
|
// ... more test cases
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
// Create temporary OPF file
|
||||||
|
tmpDir := t.TempDir()
|
||||||
|
opfPath := filepath.Join(tmpDir, "metadata.opf")
|
||||||
|
if err := os.WriteFile(opfPath, []byte(tt.opfContent), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse
|
||||||
|
scanner := &MediaScanner{}
|
||||||
|
got, err := scanner.parseCalibreMetadataOPF(opfPath)
|
||||||
|
|
||||||
|
if (err != nil) != tt.wantErr {
|
||||||
|
t.Errorf("parseCalibreMetadataOPF() error = %v, wantErr %v", err, tt.wantErr)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify fields
|
||||||
|
if got.Title != tt.want.Title {
|
||||||
|
t.Errorf("Title = %v, want %v", got.Title, tt.want.Title)
|
||||||
|
}
|
||||||
|
// ... more assertions
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test Cases**:
|
||||||
|
1. Complete metadata.opf (all fields)
|
||||||
|
2. Minimal metadata.opf (title only)
|
||||||
|
3. Malformed XML
|
||||||
|
4. Missing metadata.opf (returns nil)
|
||||||
|
5. Multiple `<dc:subject>` tags
|
||||||
|
6. Multiple `<dc:identifier>` tags
|
||||||
|
7. Invalid date formats
|
||||||
|
8. Empty/whitespace values
|
||||||
|
|
||||||
|
### Integration Tests
|
||||||
|
|
||||||
|
**File**: `cmd/server/tests/calibre_integration_test.go`
|
||||||
|
|
||||||
|
```go
|
||||||
|
func TestCalibreLibraryScan(t *testing.T) {
|
||||||
|
// Setup test server
|
||||||
|
setup := setupTestServer(t)
|
||||||
|
defer setup.Teardown()
|
||||||
|
|
||||||
|
// Create test Calibre library structure
|
||||||
|
tmpDir := t.TempDir()
|
||||||
|
createTestCalibreLibrary(t, tmpDir)
|
||||||
|
|
||||||
|
// Add library
|
||||||
|
libraryID := createTestLibrary(t, setup.DB, tmpDir)
|
||||||
|
|
||||||
|
// Scan library
|
||||||
|
scanner := services.NewMediaScanner(setup.DB, setup.AdminID, []string{tmpDir})
|
||||||
|
err := scanner.ScanFolders(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ScanFolders() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify imported books
|
||||||
|
books, err := setup.DB.ListMediaItems(context.Background(), libraryID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ListMediaItems() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Assertions
|
||||||
|
if len(books) != 3 {
|
||||||
|
t.Errorf("Got %d books, want 3", len(books))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify metadata from sidecar
|
||||||
|
for _, book := range books {
|
||||||
|
if book.Title == "" {
|
||||||
|
t.Errorf("Book %s: Title should not be empty", book.ID)
|
||||||
|
}
|
||||||
|
if book.Series.Valid && book.SeriesNumber.Int32 == 0 {
|
||||||
|
t.Errorf("Book %s: Series number should be set", book.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test Scenarios**:
|
||||||
|
1. Scan Calibre library (with sidecars)
|
||||||
|
2. Scan non-Calibre library (no sidecars)
|
||||||
|
3. Mixed library (some with sidecars, some without)
|
||||||
|
4. Force rescan updates from sidecar changes
|
||||||
|
5. All file types (EPUB, PDF, CBZ, MP3)
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
### Impact on Scan Performance
|
||||||
|
|
||||||
|
**Expected overhead**: +5-10% scan time increase
|
||||||
|
|
||||||
|
**Optimizations**:
|
||||||
|
1. **File system cache**: OS caches `metadata.opf` reads
|
||||||
|
2. **Early exit**: Return immediately if sidecar not found
|
||||||
|
3. **Parallel processing**: Sidecar extraction doesn't block other files
|
||||||
|
4. **Lazy parsing**: Only parse XML when sidecar exists
|
||||||
|
|
||||||
|
**Benchmarking**:
|
||||||
|
```go
|
||||||
|
func BenchmarkSidecarExtraction(b *testing.B) {
|
||||||
|
scanner := &MediaScanner{}
|
||||||
|
tmpDir := b.TempDir()
|
||||||
|
createTestCalibreLibrary(b, tmpDir)
|
||||||
|
|
||||||
|
b.ResetTimer()
|
||||||
|
for i := 0; i < b.N; i++ {
|
||||||
|
_, err := scanner.extractMetadataFromCalibreSidecar(filepath.Join(tmpDir, "book.epub"))
|
||||||
|
if err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Memory Usage
|
||||||
|
|
||||||
|
**XML parsing**: Small memory footprint (OPF files are typically <10KB)
|
||||||
|
|
||||||
|
**No leaks**: Ensure file handles are closed with `defer file.Close()`
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Graceful Degradation
|
||||||
|
|
||||||
|
The scanner must **never fail** due to sidecar issues:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Pattern: Log error, continue to next source
|
||||||
|
if metadata, err := s.extractMetadataFromCalibreSidecar(path); err != nil {
|
||||||
|
// Log warning
|
||||||
|
fmt.Printf("Warning: failed to read Calibre sidecar: %v\n", err)
|
||||||
|
// Continue to embedded extraction
|
||||||
|
metadata = s.extractMetadata(path)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Error Types
|
||||||
|
|
||||||
|
| Error | Action | Log Level |
|
||||||
|
|-------|--------|-----------|
|
||||||
|
| File not found | Continue to embedded | DEBUG |
|
||||||
|
| Permission denied | Continue to embedded | WARN |
|
||||||
|
| Malformed XML | Continue to embedded | WARN |
|
||||||
|
| Invalid date format | Skip field | WARN |
|
||||||
|
| Empty required field | Use available fields | INFO |
|
||||||
|
| UTF-8 decode error | Continue to embedded | ERROR |
|
||||||
|
|
||||||
|
### Logging
|
||||||
|
|
||||||
|
Use structured logging:
|
||||||
|
|
||||||
|
```go
|
||||||
|
fmt.Printf("Using Calibre metadata.opf for %s\n", path)
|
||||||
|
fmt.Printf("Warning: failed to parse metadata.opf: %v\n", err)
|
||||||
|
fmt.Printf("Info: No sidecar found, using embedded metadata for %s\n", path)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Future Enhancements
|
||||||
|
|
||||||
|
### Phase 4 (Optional)
|
||||||
|
|
||||||
|
1. **Custom column support**: Parse Calibre custom columns
|
||||||
|
2. **Smart merge**: Configurable merge strategy
|
||||||
|
3. **Rating import**: Import Calibre rating as user's initial rating
|
||||||
|
4. **Library detection**: Auto-detect Calibre library structure
|
||||||
|
5. **Incremental sync**: Only re-scan changed sidecar files
|
||||||
|
|
||||||
|
### Configuration Option
|
||||||
|
|
||||||
|
```go
|
||||||
|
type ScannerConfig struct {
|
||||||
|
PreferCalibreSidecar bool // Default: true
|
||||||
|
CalibreMergeStrategy string // "sidecar-first" | "smart-merge" | "embedded-first"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [EPUB 3.0 Spec](https://www.idpf.org/epub/30/spec/epub30-publications.html)
|
||||||
|
- [Dublin Core](https://www.dublincore.org/specifications/dces/1998-08-13/)
|
||||||
|
- [Calibre Manual](https://manual.calibre-ebook.com/)
|
||||||
|
- [Go XML Encoding](https://pkg.go.dev/encoding/xml)
|
||||||
|
|
||||||
|
## See Also
|
||||||
|
|
||||||
|
- [User Guide: Calibre Integration](../../user/calibre-integration.md)
|
||||||
|
- [Implementation Plan](../../CALIBRE_OPF_IMPLEMENTATION.md)
|
||||||
|
- [Scanner Source Code](../../internal/services/media_scanner.go)
|
||||||
@@ -0,0 +1,313 @@
|
|||||||
|
# Calibre Integration Guide
|
||||||
|
|
||||||
|
Bookhoard provides seamless integration with [Calibre](https://calibre-ebook.com/) libraries. If you manage your ebooks with Calibre, Bookhoard can automatically import your curated metadata, including series information, tags, and custom covers.
|
||||||
|
|
||||||
|
## What is Calibre?
|
||||||
|
|
||||||
|
Calibre is a free and open-source ebook management software. It allows you to:
|
||||||
|
|
||||||
|
- **Organize your library** - Create collections, add tags, manage series
|
||||||
|
- **Edit metadata** - Update titles, authors, descriptions, covers
|
||||||
|
- **Fetch metadata online** - Download from Google Books, Amazon, Goodreads
|
||||||
|
- **Convert formats** - Convert between EPUB, MOBI, PDF, and more
|
||||||
|
- **Sync to devices** - Send books to Kindle, Kobo, and other e-readers
|
||||||
|
|
||||||
|
## How Bookhoard Integrates with Calibre
|
||||||
|
|
||||||
|
### Automatic Metadata Import
|
||||||
|
|
||||||
|
When you scan a Calibre library in Bookhoard, it automatically detects and imports metadata from Calibre's `metadata.opf` files. This includes:
|
||||||
|
|
||||||
|
- ✅ **Titles and authors** - Your curated edits
|
||||||
|
- ✅ **Series information** - Series name and position
|
||||||
|
- ✅ **Tags and genres** - Your custom organization
|
||||||
|
- ✅ **Descriptions** - Book summaries
|
||||||
|
- ✅ **Publishers and dates** - Publication information
|
||||||
|
- ✅ **Identifiers** - ISBN, ASIN, and other IDs
|
||||||
|
- ✅ **Contributors** - Illustrators, editors, translators
|
||||||
|
- ✅ **Custom covers** - Your chosen cover images
|
||||||
|
|
||||||
|
### Sidecar-First Approach
|
||||||
|
|
||||||
|
Bookhoard uses a **sidecar-first** approach for Calibre libraries:
|
||||||
|
|
||||||
|
1. **If `metadata.opf` exists** → Use Calibre's curated metadata
|
||||||
|
2. **If no sidecar** → Use embedded metadata from the book file
|
||||||
|
3. **Fallback** → Use folder structure and filename
|
||||||
|
|
||||||
|
This ensures your Calibre curation work is respected and imported accurately.
|
||||||
|
|
||||||
|
## Setting Up Your Calibre Library in Bookhoard
|
||||||
|
|
||||||
|
### Step 1: Organize Your Calibre Library
|
||||||
|
|
||||||
|
Ensure your Calibre library has `metadata.opf` files in each book's folder. Calibre creates these automatically when you add books to your library.
|
||||||
|
|
||||||
|
Typical Calibre folder structure:
|
||||||
|
```
|
||||||
|
Calibre Library/
|
||||||
|
├── Author Name/
|
||||||
|
│ ├── Book Title (Series #1)/
|
||||||
|
│ │ ├── Book Title.epub
|
||||||
|
│ │ ├── metadata.opf ← Bookhoard reads this
|
||||||
|
│ │ └── cover.jpg
|
||||||
|
│ └── Book Title 2/
|
||||||
|
│ ├── Book Title 2.epub
|
||||||
|
│ └── metadata.opf ← Bookhoard reads this
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Add Library in Bookhoard
|
||||||
|
|
||||||
|
1. Navigate to **Admin** → **Libraries**
|
||||||
|
2. Click **Add Library**
|
||||||
|
3. Configure:
|
||||||
|
- **Name**: "My Calibre Library"
|
||||||
|
- **Type**: Ebook (or Audiobook/Comic)
|
||||||
|
- **Folder**: Path to your Calibre library
|
||||||
|
- **Scan on save**: ✅ Checked
|
||||||
|
4. Click **Save**
|
||||||
|
|
||||||
|
Bookhoard will automatically scan the library and import all books with their Calibre metadata.
|
||||||
|
|
||||||
|
### Step 3: Verify Import
|
||||||
|
|
||||||
|
1. Navigate to **Library** view
|
||||||
|
2. Browse your imported books
|
||||||
|
3. Check that:
|
||||||
|
- Titles and authors are correct
|
||||||
|
- Series information appears (if applicable)
|
||||||
|
- Tags are imported
|
||||||
|
- Covers display correctly
|
||||||
|
- Descriptions are present
|
||||||
|
|
||||||
|
## Supported Metadata
|
||||||
|
|
||||||
|
### Dublin Core Fields
|
||||||
|
|
||||||
|
Bookhoard imports standard Dublin Core metadata from Calibre:
|
||||||
|
|
||||||
|
| Field | Source | Notes |
|
||||||
|
|-------|--------|-------|
|
||||||
|
| Title | `dc:title` | Book title |
|
||||||
|
| Author | `dc:creator` | Primary author |
|
||||||
|
| Tags | `dc:subject` | **Keywords/tags** (multiple) |
|
||||||
|
| Description | `dc:description` | Book summary |
|
||||||
|
| Publisher | `dc:publisher` | Publisher name |
|
||||||
|
| Date | `dc:date` | Publication date |
|
||||||
|
| Language | `dc:language` | ISO language code |
|
||||||
|
| ISBN | `dc:identifier` | ISBN (if present) |
|
||||||
|
| ASIN | `dc:identifier` | Amazon ID (if present) |
|
||||||
|
| Contributors | `dc:contributor` | Additional contributors |
|
||||||
|
|
||||||
|
### Calibre-Specific Fields
|
||||||
|
|
||||||
|
Bookhoard also imports Calibre's custom metadata:
|
||||||
|
|
||||||
|
| Field | Source | Notes |
|
||||||
|
|-------|--------|-------|
|
||||||
|
| Series | `calibre:series` | Series name |
|
||||||
|
| Series Number | `calibre:series_index` | Position in series |
|
||||||
|
| Timestamp | `dc:date` | When added to Calibre |
|
||||||
|
|
||||||
|
### Not Imported
|
||||||
|
|
||||||
|
The following Calibre metadata is **not** imported (by design):
|
||||||
|
|
||||||
|
- ❌ **Ratings** - Calibre ratings are personal; Bookhoard has per-user ratings
|
||||||
|
- ❌ **Sort fields** - Bookhoard has its own sorting logic
|
||||||
|
- ❌ **Custom columns** - User-defined columns (future enhancement)
|
||||||
|
- ❌ **Last read** - Reading progress is tracked per-user in Bookhoard
|
||||||
|
|
||||||
|
## Supported File Types
|
||||||
|
|
||||||
|
Bookhoard imports Calibre metadata for **all file types**:
|
||||||
|
|
||||||
|
- 📚 **Ebooks**: EPUB, MOBI, AZW3, PDF, TXT, etc.
|
||||||
|
- 🎧 **Audiobooks**: MP3, M4B, M4A, etc.
|
||||||
|
- 📰 **Comics**: CBZ, CBR, CB7, CBT, etc.
|
||||||
|
- 📄 **Documents**: PDF, DOCX, etc.
|
||||||
|
|
||||||
|
As long as a `metadata.opf` file exists in the folder, Bookhoard will import the metadata.
|
||||||
|
|
||||||
|
## Workflow Examples
|
||||||
|
|
||||||
|
### Example 1: Fresh Calibre Library
|
||||||
|
|
||||||
|
**Scenario**: You have a Calibre library with 500 ebooks, all organized with series, tags, and custom covers.
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
1. Add the Calibre library folder in Bookhoard
|
||||||
|
2. Enable "Scan on save"
|
||||||
|
3. Bookhoard imports all 500 books with:
|
||||||
|
- Correct titles and authors
|
||||||
|
- Series information (e.g., "Harry Potter #2")
|
||||||
|
- Your custom tags (e.g., "Fantasy", "Favorites", "To Read")
|
||||||
|
- Your chosen cover images
|
||||||
|
- Descriptions and publisher info
|
||||||
|
|
||||||
|
**Result**: Your entire Calibre library is instantly available in Bookhoard with all your curation work preserved.
|
||||||
|
|
||||||
|
### Example 2: Mixed Library (Calibre + Non-Calibre)
|
||||||
|
|
||||||
|
**Scenario**: Your library has some books from Calibre (with `metadata.opf`) and some downloaded from elsewhere (no sidecar).
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
1. Add the library folder in Bookhoard
|
||||||
|
2. Bookhoard scans all files:
|
||||||
|
- **Calibre books**: Import from `metadata.opf`
|
||||||
|
- **Non-Calibre books**: Use embedded metadata or filename
|
||||||
|
|
||||||
|
**Result**: Seamless import of both Calibre and non-Calibre books, with appropriate metadata sources for each.
|
||||||
|
|
||||||
|
### Example 3: Updating Calibre Metadata
|
||||||
|
|
||||||
|
**Scenario**: You edit metadata in Calibre (fix author name, add series, change cover).
|
||||||
|
|
||||||
|
**Steps**:
|
||||||
|
1. Edit metadata in Calibre (it updates `metadata.opf`)
|
||||||
|
2. In Bookhoard, trigger a rescan:
|
||||||
|
- Navigate to **Admin** → **Libraries**
|
||||||
|
- Click **Rescan** on your library
|
||||||
|
- Or use the **Scanner API** to force rescan
|
||||||
|
3. Bookhoard detects updated `metadata.opf` and refreshes metadata
|
||||||
|
|
||||||
|
**Result**: Bookhoard reflects your Calibre changes automatically.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Metadata Not Importing
|
||||||
|
|
||||||
|
**Problem**: Bookhoard doesn't import Calibre metadata.
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. **Check file structure**: Ensure `metadata.opf` is in the same folder as the book file
|
||||||
|
2. **Verify library type**: Ensure library type matches content (ebook vs. audiobook)
|
||||||
|
3. **Force rescan**: Use the "Force Rescan" option to re-import all metadata
|
||||||
|
4. **Check logs**: Review Bookhoard logs for parsing errors
|
||||||
|
|
||||||
|
### Incorrect Metadata
|
||||||
|
|
||||||
|
**Problem**: Imported metadata is wrong or incomplete.
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. **Fix in Calibre**: Edit metadata in Calibre, then rescan in Bookhoard
|
||||||
|
2. **Verify sidecar content**: Open `metadata.opf` in a text editor to check content
|
||||||
|
3. **Check date formats**: Ensure dates are ISO format (YYYY-MM-DD)
|
||||||
|
4. **Re-scan**: Force rescan to re-import from updated sidecar
|
||||||
|
|
||||||
|
### Missing Series Information
|
||||||
|
|
||||||
|
**Problem**: Books in a series don't show series info.
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. **Check Calibre**: Verify series is set in Calibre
|
||||||
|
2. **Check sidecar**: Ensure `calibre:series` and `calibre:series_index` are in `metadata.opf`
|
||||||
|
3. **Re-scan**: Force rescan to update series info
|
||||||
|
|
||||||
|
### Performance Issues
|
||||||
|
|
||||||
|
**Problem**: Scanning takes a long time with Calibre library.
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. **Large libraries**: Initial scan may take time; subsequent scans are faster
|
||||||
|
2. **Network storage**: Scanning over network is slower; use local storage if possible
|
||||||
|
3. **Watch mode**: Enable watch mode for real-time updates instead of full rescans
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
### 1. Keep Calibre as Primary Source
|
||||||
|
|
||||||
|
**Do**:
|
||||||
|
- ✅ Edit metadata in Calibre
|
||||||
|
- ✅ Rescan in Bookhoard to sync changes
|
||||||
|
- ✅ Use Calibre for library management
|
||||||
|
|
||||||
|
**Don't**:
|
||||||
|
- ❌ Edit `metadata.opf` files manually
|
||||||
|
- ❌ Modify metadata in Bookhoard if you plan to resync from Calibre
|
||||||
|
- ❌ Keep conflicting metadata sources
|
||||||
|
|
||||||
|
### 2. Organize with Tags and Collections
|
||||||
|
|
||||||
|
- Use **tags in Calibre** for genres, moods, status (To Read, Read, etc.)
|
||||||
|
- Create **collections in Bookhoard** for automatic organization
|
||||||
|
- Leverage both systems for different purposes
|
||||||
|
|
||||||
|
### 3. Regular Syncing
|
||||||
|
|
||||||
|
- Set up **watch mode** in Bookhoard for automatic updates
|
||||||
|
- Run **full rescan** after bulk edits in Calibre
|
||||||
|
- Keep metadata consistent between Calibre and Bookhoard
|
||||||
|
|
||||||
|
### 4. Backup Your Library
|
||||||
|
|
||||||
|
- Back up your Calibre library regularly
|
||||||
|
- Include both book files and `metadata.opf` files
|
||||||
|
- Test restore process periodically
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Custom Calibre Columns (Future)
|
||||||
|
|
||||||
|
Bookhoard may support Calibre custom columns in a future release. This would allow you to import:
|
||||||
|
- Read status
|
||||||
|
- Favorite flags
|
||||||
|
- Custom metadata fields
|
||||||
|
- User-defined categories
|
||||||
|
|
||||||
|
Stay tuned for updates!
|
||||||
|
|
||||||
|
### OPDS Integration
|
||||||
|
|
||||||
|
You can access your Bookhoard library (including Calibre-imported books) via OPDS from Calibre-aware devices:
|
||||||
|
- Kobo e-readers
|
||||||
|
- KOReader
|
||||||
|
- Phone/tablet apps (KYBook, Chunky, etc.)
|
||||||
|
|
||||||
|
See the [Kobo Setup Guide](devices/kobo-setup.md) or [KOReader Setup Guide](devices/koreader-setup.md) for details.
|
||||||
|
|
||||||
|
## FAQ
|
||||||
|
|
||||||
|
**Q: Will Bookhoard modify my Calibre library?**
|
||||||
|
|
||||||
|
A: No. Bookhoard only **reads** Calibre metadata. It never modifies your Calibre library files or `metadata.opf` files.
|
||||||
|
|
||||||
|
**Q: Can I use both Calibre and Bookhoard?**
|
||||||
|
|
||||||
|
A: Yes! They're complementary:
|
||||||
|
- Use **Calibre** for library management, conversion, and device syncing
|
||||||
|
- Use **Bookhoard** for web access, sync across devices, and sharing
|
||||||
|
|
||||||
|
**Q: What if I don't use Calibre?**
|
||||||
|
|
||||||
|
A: Bookhoard works perfectly without Calibre. It will use embedded metadata from your book files, folder structure, and filenames.
|
||||||
|
|
||||||
|
**Q: Does Bookhoard import Calibre ratings?**
|
||||||
|
|
||||||
|
A: No. Ratings are **per-user** in Bookhoard. Calibre ratings reflect the owner's opinion, which may not match other users' opinions.
|
||||||
|
|
||||||
|
**Q: Can I edit Calibre metadata in Bookhoard?**
|
||||||
|
|
||||||
|
A: You can edit metadata in Bookhoard, but it won't sync back to Calibre. For permanent changes, edit in Calibre and rescan in Bookhoard.
|
||||||
|
|
||||||
|
**Q: Does this work with Calibre Web?**
|
||||||
|
|
||||||
|
A: Calibre Web uses the same `metadata.opf` files, so yes - Bookhoard can scan a Calibre Web library folder.
|
||||||
|
|
||||||
|
**Q: What about Calibre's "author sort" and "title sort"?**
|
||||||
|
|
||||||
|
A: Bookhoard has its own sorting logic and doesn't import Calibre sort fields. This allows for consistent sorting across all books.
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
|
||||||
|
- [Calibre Website](https://calibre-ebook.com/)
|
||||||
|
- [Calibre User Manual](https://manual.calibre-ebook.com/)
|
||||||
|
- [Calibre Forum](https://www.mobileread.com/forums/forumdisplay.php?f=166)
|
||||||
|
- [Bookhoard GitHub](https://github.com/yourusername/bookhoard)
|
||||||
|
|
||||||
|
## Need Help?
|
||||||
|
|
||||||
|
- **Documentation**: See [User Guide](user-guide.md) for general Bookhoard usage
|
||||||
|
- **Troubleshooting**: See [Operations Guide](../operations/operations.md) for common issues
|
||||||
|
- **Developer Docs**: See [Developer Portal](../developer/development.md) for technical details
|
||||||
|
- **Issues**: Report bugs or request features on [GitHub Issues](https://github.com/yourusername/bookhoard/issues)
|
||||||
@@ -27,6 +27,17 @@ Learn how to configure your e-reader devices to sync with Bookhoard:
|
|||||||
- Conflict resolution
|
- Conflict resolution
|
||||||
- Best practices
|
- Best practices
|
||||||
|
|
||||||
|
## 📚 Calibre Integration
|
||||||
|
|
||||||
|
- **[Calibre Integration Guide](calibre-integration.md)** - Integrating with Calibre libraries
|
||||||
|
- What is Calibre?
|
||||||
|
- Automatic metadata import
|
||||||
|
- Setting up your Calibre library
|
||||||
|
- Supported metadata and file types
|
||||||
|
- Workflow examples
|
||||||
|
- Troubleshooting
|
||||||
|
- Best practices
|
||||||
|
|
||||||
## 🎨 Frontend Guide
|
## 🎨 Frontend Guide
|
||||||
|
|
||||||
**[Frontend Guide](frontend-guide.md)** - Learn how to use the Bookhoard web interface
|
**[Frontend Guide](frontend-guide.md)** - Learn how to use the Bookhoard web interface
|
||||||
|
|||||||
Reference in New Issue
Block a user