bookhoard/docs/CONVERSION_SERVICE.md

# EPUB to KEPUB Conversion Service

## Overview

The Conversion Service provides on-the-fly EPUB to KEPUB conversion with dual hash storage to ensure cross-device book matching continues to work after format conversion.

## Key Features

1. **On-Demand Conversion**: Converts EPUB to KEPUB when requested via OPDS with `?format=kepub`
2. **Dual Hash Storage**: Stores both original EPUB hash AND converted KEPUB hash in `media_item_formats` table
3. **Conversion Caching**: Caches converted files for 24 hours (configurable) to avoid re-conversion
4. **Hash Preservation**: After conversion, both hashes remain queryable for book matching
5. **Format Integrity**: Ensures converted KEPUB maintains all reading progress markers

## Architecture

```
User requests book via OPDS with ?format=kepub
    ↓
Check media_item_formats table for existing KEPUB
    ↓
If KEPUB exists and is recent (< 24 hours):
    → Serve pre-converted file
    → Set X-Bookmann-KEPUB-SHA256 header
    ↓
If KEPUB doesn't exist or is stale:
    → Convert EPUB→KEPUB on-the-fly
    → Calculate SHA-256 of converted KEPUB
    → Store in media_item_formats (with converted_from_format_id)
    → Serve converted file
    → Set X-Bookmann-KEPUB-SHA256 header
    ↓
Device downloads book with hash in response header
    ↓
Device syncs progress using hash for matching
```

## Configuration

### Environment Variables

Add these to your `.env` file or `system_config` table:

```bash
# Conversion service configuration
BOOKMANN_CONVERSION_CACHE_DIR=/var/bookmann/cache/kepub
BOOKMANN_CONVERSION_TOOL=/usr/bin/kepubify  # or /usr/bin/ebook-convert
BOOKMANN_CONVERSION_CACHE_TTL=24h
```

### Dockerfile Updates

If using kepubify (recommended for Kobo):

```dockerfile
# Install kepubify for EPUB→KEPUB conversion
RUN wget -O /usr/bin/kepubify https://github.com/pgaskin/kepubify/releases/latest/download/kepubify-linux-64bit \
    && chmod +x /usr/bin/kepubify
```

Or install Calibre for ebook-convert:

```dockerfile
# Install Calibre for ebook-convert
RUN apt-get update && apt-get install -y calibre
```

## API Usage

### Download KEPUB via OPDS

```http
GET /opds/devices/{deviceId}/download/{bookId}?format=kepub
```

**Response Headers:**
- `Content-Type`: application/vnd.kobo+xml+zip
- `Content-Disposition`: attachment; filename="book.kepub.epub"
- `X-Bookmann-UUID`: uuid-123
- `X-Bookmann-KEPUB-SHA256`: abc123... (KEPUB-specific hash)

## Database Schema

### media_item_formats Table

```sql
CREATE TABLE media_item_formats (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    format_type VARCHAR(10) NOT NULL,           -- 'epub', 'kepub', 'pdf', 'cbz'
    file_path VARCHAR(500),
    file_sha256 CHAR(64),                       -- Hash for THIS format version
    file_size_bytes BIGINT,
    mime_type VARCHAR(100),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    converted_from_format_id UUID REFERENCES media_item_formats(id),  -- Track conversion chain
    UNIQUE(media_item_id, format_type)
);
```

**Example Data:**
```
Row 1: media_item_id=uuid-123, format_type='epub', file_sha256='abc123...'
Row 2: media_item_id=uuid-123, format_type='kepub', file_sha256='xyz789...', converted_from_format_id=Row1.id
```

## Implementation Details

### Service Methods

#### ConvertEPUBToKEPUB

Converts an EPUB file to KEPUB format with hash storage.

**Parameters:**
- `ctx context.Context`: Request context
- `mediaItemID pgtype.UUID`: ID of the media item
- `epubPath string`: Path to the source EPUB file

**Returns:**
- `*ConvertedKEPUB`: Contains path, SHA256 hash, and cached status
- `error`: Conversion error if any

**Behavior:**
1. Checks cache for existing KEPUB (recent conversions are reused)
2. Performs EPUB→KEPUB conversion using kepubify or ebook-convert
3. Calculates SHA-256 hash of converted file
4. Stores format record in database with dual hash
5. Returns converted file path and hash

### Conversion Tools

The service tries conversion tools in this order:

1. **kepubify** (recommended): Purpose-built KEPUB converter
   - Faster and more reliable for Kobo devices
   - Download: https://github.com/pgaskin/kepubify/releases

2. **ebook-convert** (fallback): Part of Calibre suite
   - More versatile but slower
   - Requires full Calibre installation

### Cache Invalidation

Converted KEPUB files are cached for 24 hours by default. This TTL is configurable via:
- Environment variable: `BOOKMANN_CONVERSION_CACHE_TTL`
- Code: `conversionCacheTTL` field in `ConversionService`

## Testing

### Unit Tests

```bash
go test ./internal/services/... -v
```

### Manual Testing with Bruno

Use the provided Bruno test:
- `bruno/opds/Download Book KEPUB (On-the-fly Conversion).bru`

This test verifies:
- KEPUB hash header is present
- Hash is 64 characters (SHA-256 format)
- Bookmann UUID header is present
- Content-Type is correct for KEPUB

## Troubleshooting

### Conversion Failures

**Problem**: KEPUB conversion fails
**Solution**:
1. Check if kepubify or ebook-convert is installed
2. Verify EPUB file is valid and accessible
3. Check cache directory permissions: `/var/bookmann/cache/kepub`
4. Review conversion logs for specific error messages

### Cache Issues

**Problem**: Converted files not being cached
**Solution**:
1. Verify cache directory exists and is writable
2. Check `BOOKMANN_CONVERSION_CACHE_DIR` environment variable
3. Ensure database can create media_item_formats records

### Hash Mismatches

**Problem**: Progress sync fails after conversion
**Solution**:
1. Verify dual hash storage: both EPUB and KEPUB hashes should exist in `media_item_formats`
2. Check `X-Bookmann-KEPUB-SHA256` header in response
3. Ensure `converted_from_format_id` links KEPUB to source EPUB

## Performance Considerations

### First Conversion
- **Time**: 2-5 seconds per book (depends on file size)
- **CPU**: Medium (single-threaded conversion)
- **I/O**: Read EPUB, write KEPUB to cache

### Cached Conversions
- **Time**: < 100ms (database lookup + file serve)
- **CPU**: Minimal
- **I/O**: Read cached KEPUB file

### Storage Requirements
- **Cache Size**: ~1.1x original EPUB size (KEPUB is slightly larger)
- **Database**: ~200 bytes per converted format record
- **Recommendation**: 10 GB cache per 1000 books

## Security

### File Access
- Conversion service only processes files from library folders
- Converted files are stored in secure cache directory
- Original files are never modified

### Input Validation
- All file paths are validated before conversion
- Media item IDs are verified against database
- User access permissions are checked via OPDS handler

## Future Enhancements

Potential improvements:
1. **Async Conversion**: Queue conversions for background processing
2. **Batch Conversion**: Pre-convert entire libraries during off-hours
3. **Format Variants**: Support PDF→EPUB, CBZ→EPUB, etc.
4. **Quality Settings**: Configurable conversion quality/size tradeoffs
5. **Distributed Caching**: Share cache across multiple server instances