docs: add phase summaries and conversion service documentation
- Phase 6: WebSocket verification and bulk operations summary - Phase 1: Device management completion summary - Phase 2: Quick completion summary and detailed notes - Conversion service: Architecture and implementation details - Document caching strategy, TTL configuration, and performance considerations
This commit is contained in:
@@ -0,0 +1,231 @@
|
||||
# EPUB to KEPUB Conversion Service
|
||||
|
||||
## Overview
|
||||
|
||||
The Conversion Service provides on-the-fly EPUB to KEPUB conversion with dual hash storage to ensure cross-device book matching continues to work after format conversion.
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **On-Demand Conversion**: Converts EPUB to KEPUB when requested via OPDS with `?format=kepub`
|
||||
2. **Dual Hash Storage**: Stores both original EPUB hash AND converted KEPUB hash in `media_item_formats` table
|
||||
3. **Conversion Caching**: Caches converted files for 24 hours (configurable) to avoid re-conversion
|
||||
4. **Hash Preservation**: After conversion, both hashes remain queryable for book matching
|
||||
5. **Format Integrity**: Ensures converted KEPUB maintains all reading progress markers
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
User requests book via OPDS with ?format=kepub
|
||||
↓
|
||||
Check media_item_formats table for existing KEPUB
|
||||
↓
|
||||
If KEPUB exists and is recent (< 24 hours):
|
||||
→ Serve pre-converted file
|
||||
→ Set X-Bookmann-KEPUB-SHA256 header
|
||||
↓
|
||||
If KEPUB doesn't exist or is stale:
|
||||
→ Convert EPUB→KEPUB on-the-fly
|
||||
→ Calculate SHA-256 of converted KEPUB
|
||||
→ Store in media_item_formats (with converted_from_format_id)
|
||||
→ Serve converted file
|
||||
→ Set X-Bookmann-KEPUB-SHA256 header
|
||||
↓
|
||||
Device downloads book with hash in response header
|
||||
↓
|
||||
Device syncs progress using hash for matching
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Add these to your `.env` file or `system_config` table:
|
||||
|
||||
```bash
|
||||
# Conversion service configuration
|
||||
BOOKMANN_CONVERSION_CACHE_DIR=/var/bookmann/cache/kepub
|
||||
BOOKMANN_CONVERSION_TOOL=/usr/bin/kepubify # or /usr/bin/ebook-convert
|
||||
BOOKMANN_CONVERSION_CACHE_TTL=24h
|
||||
```
|
||||
|
||||
### Dockerfile Updates
|
||||
|
||||
If using kepubify (recommended for Kobo):
|
||||
|
||||
```dockerfile
|
||||
# Install kepubify for EPUB→KEPUB conversion
|
||||
RUN wget -O /usr/bin/kepubify https://github.com/pgaskin/kepubify/releases/latest/download/kepubify-linux-64bit \
|
||||
&& chmod +x /usr/bin/kepubify
|
||||
```
|
||||
|
||||
Or install Calibre for ebook-convert:
|
||||
|
||||
```dockerfile
|
||||
# Install Calibre for ebook-convert
|
||||
RUN apt-get update && apt-get install -y calibre
|
||||
```
|
||||
|
||||
## API Usage
|
||||
|
||||
### Download KEPUB via OPDS
|
||||
|
||||
```http
|
||||
GET /opds/devices/{deviceId}/download/{bookId}?format=kepub
|
||||
```
|
||||
|
||||
**Response Headers:**
|
||||
- `Content-Type`: application/vnd.kobo+xml+zip
|
||||
- `Content-Disposition`: attachment; filename="book.kepub.epub"
|
||||
- `X-Bookmann-UUID`: uuid-123
|
||||
- `X-Bookmann-KEPUB-SHA256`: abc123... (KEPUB-specific hash)
|
||||
|
||||
## Database Schema
|
||||
|
||||
### media_item_formats Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE media_item_formats (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
|
||||
format_type VARCHAR(10) NOT NULL, -- 'epub', 'kepub', 'pdf', 'cbz'
|
||||
file_path VARCHAR(500),
|
||||
file_sha256 CHAR(64), -- Hash for THIS format version
|
||||
file_size_bytes BIGINT,
|
||||
mime_type VARCHAR(100),
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
converted_from_format_id UUID REFERENCES media_item_formats(id), -- Track conversion chain
|
||||
UNIQUE(media_item_id, format_type)
|
||||
);
|
||||
```
|
||||
|
||||
**Example Data:**
|
||||
```
|
||||
Row 1: media_item_id=uuid-123, format_type='epub', file_sha256='abc123...'
|
||||
Row 2: media_item_id=uuid-123, format_type='kepub', file_sha256='xyz789...', converted_from_format_id=Row1.id
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Service Methods
|
||||
|
||||
#### ConvertEPUBToKEPUB
|
||||
|
||||
Converts an EPUB file to KEPUB format with hash storage.
|
||||
|
||||
**Parameters:**
|
||||
- `ctx context.Context`: Request context
|
||||
- `mediaItemID pgtype.UUID`: ID of the media item
|
||||
- `epubPath string`: Path to the source EPUB file
|
||||
|
||||
**Returns:**
|
||||
- `*ConvertedKEPUB`: Contains path, SHA256 hash, and cached status
|
||||
- `error`: Conversion error if any
|
||||
|
||||
**Behavior:**
|
||||
1. Checks cache for existing KEPUB (recent conversions are reused)
|
||||
2. Performs EPUB→KEPUB conversion using kepubify or ebook-convert
|
||||
3. Calculates SHA-256 hash of converted file
|
||||
4. Stores format record in database with dual hash
|
||||
5. Returns converted file path and hash
|
||||
|
||||
### Conversion Tools
|
||||
|
||||
The service tries conversion tools in this order:
|
||||
|
||||
1. **kepubify** (recommended): Purpose-built KEPUB converter
|
||||
- Faster and more reliable for Kobo devices
|
||||
- Download: https://github.com/pgaskin/kepubify/releases
|
||||
|
||||
2. **ebook-convert** (fallback): Part of Calibre suite
|
||||
- More versatile but slower
|
||||
- Requires full Calibre installation
|
||||
|
||||
### Cache Invalidation
|
||||
|
||||
Converted KEPUB files are cached for 24 hours by default. This TTL is configurable via:
|
||||
- Environment variable: `BOOKMANN_CONVERSION_CACHE_TTL`
|
||||
- Code: `conversionCacheTTL` field in `ConversionService`
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
go test ./internal/services/... -v
|
||||
```
|
||||
|
||||
### Manual Testing with Bruno
|
||||
|
||||
Use the provided Bruno test:
|
||||
- `bruno/opds/Download Book KEPUB (On-the-fly Conversion).bru`
|
||||
|
||||
This test verifies:
|
||||
- KEPUB hash header is present
|
||||
- Hash is 64 characters (SHA-256 format)
|
||||
- Bookmann UUID header is present
|
||||
- Content-Type is correct for KEPUB
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Conversion Failures
|
||||
|
||||
**Problem**: KEPUB conversion fails
|
||||
**Solution**:
|
||||
1. Check if kepubify or ebook-convert is installed
|
||||
2. Verify EPUB file is valid and accessible
|
||||
3. Check cache directory permissions: `/var/bookmann/cache/kepub`
|
||||
4. Review conversion logs for specific error messages
|
||||
|
||||
### Cache Issues
|
||||
|
||||
**Problem**: Converted files not being cached
|
||||
**Solution**:
|
||||
1. Verify cache directory exists and is writable
|
||||
2. Check `BOOKMANN_CONVERSION_CACHE_DIR` environment variable
|
||||
3. Ensure database can create media_item_formats records
|
||||
|
||||
### Hash Mismatches
|
||||
|
||||
**Problem**: Progress sync fails after conversion
|
||||
**Solution**:
|
||||
1. Verify dual hash storage: both EPUB and KEPUB hashes should exist in `media_item_formats`
|
||||
2. Check `X-Bookmann-KEPUB-SHA256` header in response
|
||||
3. Ensure `converted_from_format_id` links KEPUB to source EPUB
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### First Conversion
|
||||
- **Time**: 2-5 seconds per book (depends on file size)
|
||||
- **CPU**: Medium (single-threaded conversion)
|
||||
- **I/O**: Read EPUB, write KEPUB to cache
|
||||
|
||||
### Cached Conversions
|
||||
- **Time**: < 100ms (database lookup + file serve)
|
||||
- **CPU**: Minimal
|
||||
- **I/O**: Read cached KEPUB file
|
||||
|
||||
### Storage Requirements
|
||||
- **Cache Size**: ~1.1x original EPUB size (KEPUB is slightly larger)
|
||||
- **Database**: ~200 bytes per converted format record
|
||||
- **Recommendation**: 10 GB cache per 1000 books
|
||||
|
||||
## Security
|
||||
|
||||
### File Access
|
||||
- Conversion service only processes files from library folders
|
||||
- Converted files are stored in secure cache directory
|
||||
- Original files are never modified
|
||||
|
||||
### Input Validation
|
||||
- All file paths are validated before conversion
|
||||
- Media item IDs are verified against database
|
||||
- User access permissions are checked via OPDS handler
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements:
|
||||
1. **Async Conversion**: Queue conversions for background processing
|
||||
2. **Batch Conversion**: Pre-convert entire libraries during off-hours
|
||||
3. **Format Variants**: Support PDF→EPUB, CBZ→EPUB, etc.
|
||||
4. **Quality Settings**: Configurable conversion quality/size tradeoffs
|
||||
5. **Distributed Caching**: Share cache across multiple server instances
|
||||
Reference in New Issue
Block a user