bookhoard/docs/CONVERSION_SERVICE.md

EPUB to KEPUB Conversion Service

Overview

The Conversion Service provides on-the-fly EPUB to KEPUB conversion with dual hash storage to ensure cross-device book matching continues to work after format conversion.

Key Features

1. On-Demand Conversion: Converts EPUB to KEPUB when requested via OPDS with ?format=kepub
2. Dual Hash Storage: Stores both original EPUB hash AND converted KEPUB hash in media_item_formats table
3. Conversion Caching: Caches converted files for 24 hours (configurable) to avoid re-conversion
4. Hash Preservation: After conversion, both hashes remain queryable for book matching
5. Format Integrity: Ensures converted KEPUB maintains all reading progress markers

Architecture

User requests book via OPDS with ?format=kepub
    ↓
Check media_item_formats table for existing KEPUB
    ↓
If KEPUB exists and is recent (< 24 hours):
    → Serve pre-converted file
    → Set X-Bookhoard-KEPUB-SHA256 header
    ↓
If KEPUB doesn't exist or is stale:
    → Convert EPUB→KEPUB on-the-fly
    → Calculate SHA-256 of converted KEPUB
    → Store in media_item_formats (with converted_from_format_id)
    → Serve converted file
    → Set X-Bookhoard-KEPUB-SHA256 header
    ↓
Device downloads book with hash in response header
    ↓
Device syncs progress using hash for matching

Configuration

Environment Variables

Add these to your .env file or system_config table:

# Conversion service configuration
BOOKHOARD_CONVERSION_CACHE_DIR=/var/bookhoard/cache/kepub
BOOKHOARD_CONVERSION_TOOL=/usr/bin/kepubify  # or /usr/bin/ebook-convert
BOOKHOARD_CONVERSION_CACHE_TTL=24h

Dockerfile Updates

If using kepubify (recommended for Kobo):

# Install kepubify for EPUB→KEPUB conversion
RUN wget -O /usr/bin/kepubify https://github.com/pgaskin/kepubify/releases/latest/download/kepubify-linux-64bit \
    && chmod +x /usr/bin/kepubify

Or install Calibre for ebook-convert:

# Install Calibre for ebook-convert
RUN apt-get update && apt-get install -y calibre

API Usage

Download KEPUB via OPDS

GET /opds/devices/{deviceId}/download/{bookId}?format=kepub

Response Headers:

• Content-Type: application/vnd.kobo+xml+zip
• Content-Disposition: attachment; filename="book.kepub.epub"
• X-Bookhoard-UUID: uuid-123
• X-Bookhoard-KEPUB-SHA256: abc123... (KEPUB-specific hash)

Database Schema

media_item_formats Table

CREATE TABLE media_item_formats (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    format_type VARCHAR(10) NOT NULL,           -- 'epub', 'kepub', 'pdf', 'cbz'
    file_path VARCHAR(500),
    file_sha256 CHAR(64),                       -- Hash for THIS format version
    file_size_bytes BIGINT,
    mime_type VARCHAR(100),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    converted_from_format_id UUID REFERENCES media_item_formats(id),  -- Track conversion chain
    UNIQUE(media_item_id, format_type)
);

Example Data:

Row 1: media_item_id=uuid-123, format_type='epub', file_sha256='abc123...'
Row 2: media_item_id=uuid-123, format_type='kepub', file_sha256='xyz789...', converted_from_format_id=Row1.id

Implementation Details

Service Methods

ConvertEPUBToKEPUB

Converts an EPUB file to KEPUB format with hash storage.

Parameters:

• ctx context.Context: Request context
• mediaItemID pgtype.UUID: ID of the media item
• epubPath string: Path to the source EPUB file

Returns:

• *ConvertedKEPUB: Contains path, SHA256 hash, and cached status
• error: Conversion error if any

Behavior:

1. Checks cache for existing KEPUB (recent conversions are reused)
2. Performs EPUB→KEPUB conversion using kepubify or ebook-convert
3. Calculates SHA-256 hash of converted file
4. Stores format record in database with dual hash
5. Returns converted file path and hash

Conversion Tools

The service tries conversion tools in this order:

1. kepubify (recommended): Purpose-built KEPUB converter

   • Faster and more reliable for Kobo devices
   • Download: https://github.com/pgaskin/kepubify/releases

2. ebook-convert (fallback): Part of Calibre suite

   • More versatile but slower
   • Requires full Calibre installation

Cache Invalidation

Converted KEPUB files are cached for 24 hours by default. This TTL is configurable via:

• Environment variable: BOOKHOARD_CONVERSION_CACHE_TTL
• Code: conversionCacheTTL field in ConversionService

Testing

Unit Tests

go test ./internal/services/... -v

Manual Testing with Bruno

Use the provided Bruno test:

• bruno/opds/Download Book KEPUB (On-the-fly Conversion).bru

This test verifies:

• KEPUB hash header is present
• Hash is 64 characters (SHA-256 format)
• Bookhoard UUID header is present
• Content-Type is correct for KEPUB

Troubleshooting

Conversion Failures

Problem: KEPUB conversion fails Solution:

1. Check if kepubify or ebook-convert is installed
2. Verify EPUB file is valid and accessible
3. Check cache directory permissions: /var/bookhoard/cache/kepub
4. Review conversion logs for specific error messages

Cache Issues

Problem: Converted files not being cached Solution:

1. Verify cache directory exists and is writable
2. Check BOOKHOARD_CONVERSION_CACHE_DIR environment variable
3. Ensure database can create media_item_formats records

Hash Mismatches

Problem: Progress sync fails after conversion Solution:

1. Verify dual hash storage: both EPUB and KEPUB hashes should exist in media_item_formats
2. Check X-Bookhoard-KEPUB-SHA256 header in response
3. Ensure converted_from_format_id links KEPUB to source EPUB

Performance Considerations

First Conversion

• Time: 2-5 seconds per book (depends on file size)
• CPU: Medium (single-threaded conversion)
• I/O: Read EPUB, write KEPUB to cache

Cached Conversions

• Time: < 100ms (database lookup + file serve)
• CPU: Minimal
• I/O: Read cached KEPUB file

Storage Requirements

• Cache Size: ~1.1x original EPUB size (KEPUB is slightly larger)
• Database: ~200 bytes per converted format record
• Recommendation: 10 GB cache per 1000 books

Security

File Access

• Conversion service only processes files from library folders
• Converted files are stored in secure cache directory
• Original files are never modified

Input Validation

• All file paths are validated before conversion
• Media item IDs are verified against database
• User access permissions are checked via OPDS handler

Future Enhancements

Potential improvements:

1. Async Conversion: Queue conversions for background processing
2. Batch Conversion: Pre-convert entire libraries during off-hours
3. Format Variants: Support PDF→EPUB, CBZ→EPUB, etc.
4. Quality Settings: Configurable conversion quality/size tradeoffs
5. Distributed Caching: Share cache across multiple server instances