From 33f26f3d7dff3d6e56b7877c269138659cf3c02a Mon Sep 17 00:00:00 2001 From: John O'Keefe Date: Sat, 31 Jan 2026 18:25:27 -0500 Subject: [PATCH] docs: add implementation plan and update documentation - Add comprehensive implementation plan for universal sync system - Update README with API reference and device setup guides - Add KOBO_SETUP.md device configuration guide --- IMPLEMENTATION_PLAN.md | 1692 ++++++++++++++++++++++++++++++++++++ README.md | 29 +- docs/API_REFERENCE.md | 1308 ++++++++++++++++++++++++++++ docs/devices/KOBO_SETUP.md | 402 +++++++++ 4 files changed, 3427 insertions(+), 4 deletions(-) create mode 100644 IMPLEMENTATION_PLAN.md create mode 100644 docs/API_REFERENCE.md create mode 100644 docs/devices/KOBO_SETUP.md diff --git a/IMPLEMENTATION_PLAN.md b/IMPLEMENTATION_PLAN.md new file mode 100644 index 0000000..249a1dc --- /dev/null +++ b/IMPLEMENTATION_PLAN.md @@ -0,0 +1,1692 @@ +# Bookmann Implementation Plan + +## Executive Summary + +This plan implements a complete cross-device ebook management system with three major capabilities: + +1. **Universal Book Identification** - SHA-256 hashing, UUID, ISBN, ASIN, and OPF identifiers for content-based matching across devices +2. **Enhanced Collection Management** - Device-neutral "Collections" with auto-assign rules and per-device customization via shelf mappings +3. **OPDS-Based Wireless Book Delivery** - Industry-standard book distribution for Kobo, KOReader, Web, and Mobile apps +4. **Bidirectional Progress Synchronization** - Real-time sync with ContentId mapping to handle format conversions (EPUB → KEPUB) +5. **Device-Specific Configuration** - Per-device view settings and shelf mappings while maintaining unified data model + +**Key Design Principle**: Use OPDS for book acquisition (Layer 1) and internal APIs for state management (Layer 2), maintaining clear separation of concerns while enabling seamless user experience. + +--- + +## Table of Contents + +1. [Architecture Overview](#architecture-overview) +2. [Database Schema](#database-schema) +3. [API Endpoints](#api-endpoints) +4. [Implementation Phases](#implementation-phases) +5. [Device Setup Instructions](#device-setup-instructions) +6. [Security Considerations](#security-considerations) +7. [Testing Strategy](#testing-strategy) +8. [Glossary](#glossary) + +--- + +## Architecture Overview + +### System Design: Two-Layer Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Bookmann Server │ +│ ┌──────────────────────────────────────────────────────────────┐ │ +│ │ Layer 1: Universal Book Identification │ │ +│ │ SHA-256, UUID, ISBN, ASIN, OPF identifiers │ │ +│ │ Device file aliases for path tracking │ │ +│ └──────────────────────────────────────────────────────────────────────┘ │ +│ ↓ matches books universally │ +│ ┌──────────────────────────────────────────────────────────────────────┐ │ +│ │ Layer 2: Collections (Device-Neutral Organization) │ │ +│ │ Collections with auto-assign rules │ │ +│ │ Device-specific shelf mappings (Kobo) │ │ +│ │ Per-user view settings │ │ +│ └──────────────────────────────────────────────────────────────────────┘ │ +│ ↓ provides organization │ +│ ┌──────────────────────────────────────────────────────────────────────┐ │ +│ │ Layer 3: OPDS (Primary Wireless Delivery) │ │ +│ │ Per-device OPDS feeds (Kobo, KOReader, etc.) │ │ +│ │ Format conversion (EPUB → KEPUB on-the-fly) │ │ +│ │ Dual hash storage (original + converted) │ │ +│ │ ContentId mapping (Bookmann UUID ↔ Device ID) │ │ +│ └──────────────────────────────────────────────────────────────────────┘ │ +│ ↓ delivers books + provides IDs │ +│ ┌──────────────────────────────────────────────────────────────────────┐ │ +│ │ Layer 4: Internal APIs (State Management) │ │ +│ │ Progress sync (bidirectional) │ │ +│ │ Annotation sync (bidirectional) │ │ +│ │ Collection CRUD │ │ +│ │ WebSocket real-time updates │ │ +│ │ Device-specific operations │ │ +│ └──────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +### Access Method Matrix + +| Platform | Access Method | Purpose | Why This Method | +|-----------|----------------|---------|-----------------| +| **Kobo** | OPDS catalog | Kobo has built-in OPDS client, no custom Bookmann client exists | +| **KOReader** | OPDS catalog (primary) + Sidecar + Internal API | KOReader has OPDS client, also supports plugins/sidecars for enhanced features | +| **Web App** | Internal API directly | We own and control web app, can make direct API calls efficiently | +| **Mobile App** | Internal API directly | We own and control mobile app, can make direct API calls efficiently | +| **Any OPDS Client** | OPDS catalog | Public catalog standard, any app can use it for browsing/downloading | + +### Key Design Principles + +1. **Canonical UUID Always Wins** - Bookmann UUID (from `media_items.id`) is always used for progress tracking, never SHA-256. SHA-256 is only for matching books across devices, preventing format conversion issues. + +2. **Collections ≠ Device Inventory** - Collections are organizational metadata (like "smart playlists"). Books can be in collections without being on any device. Progress/annotations sync independently of collection membership. + +3. **OPDS for Acquisition, Internal APIs for State** - Two layers serve complementary purposes: + - OPDS: "What books are available to download?" (public catalog) + - Internal APIs: "How do I manage my books/sync state?" (private management) + +4. **Dual Hash Storage Preserves Integrity** - Store both original EPUB hash (`epub_sha256`) and converted KEPUB hash (`kepub_sha256`) in `media_item_formats` table. OPDS responses include format-specific hash in headers, enabling sidecar matching even after conversion. + +5. **Three-Tier Authentication** - Separate systems for different purposes: + - Tier 1 (Web/Mobile): JWT tokens for user authentication and permissions + - Tier 2 (Sync APIs): Device tokens for progress/annotation sync + - Tier 3 (OPDS): Device tokens for catalog access (optional per-device) + +6. **Terminology Separation** - Always use "Collections" terminology in Bookmann UI. Map Collections to device-specific "Shelves" only at API/device level. Kobo devices see "Shelves", KOReader/Web/Mobile see "Collections". Prevents legal issues. + +7. **OPDS Primary for All Devices** - Kobo, KOReader, Web, and Mobile all use OPDS as primary book delivery method. Sidecar files provide fallback/enhanced features but are optional. + +8. **System Configuration Flexibility** - Use `system_config` table to store base URLs (`base_url`, `opds_base_url`, `api_base_url`). Sidecar generation reads from these values, enabling flexible deployment (different domains, reverse proxies) with user overrides available. + +--- + +## Database Schema + +### Schema Overview + +**7 new tables** + extensions to 4 existing tables + +### Table: media_items (Extended) + +```sql +-- Universal identifiers for cross-device matching +ALTER TABLE media_items ADD COLUMN file_sha256 CHAR(64); +ALTER TABLE media_items ADD COLUMN opf_identifier VARCHAR(255); +ALTER TABLE media_items ADD COLUMN opf_uuid VARCHAR(255); + +-- Hash confidence for matching priority +-- 'high': OPF UUID or ISBN available +-- 'medium': ISBN/ASIN available but no OPF UUID +-- 'low': Only title/author match available +ALTER TABLE media_items ADD COLUMN hash_confidence VARCHAR(20); + +-- Create indexes for fast lookup +CREATE INDEX idx_media_items_sha256 ON media_items(file_sha256); +CREATE INDEX idx_media_items_opf_identifier ON media_items(opf_identifier); +``` + +### Table: media_item_formats (NEW) + +```sql +-- Track all format versions with their hashes +-- Critical for dual hash storage and format-specific OPDS delivery + +CREATE TABLE media_item_formats ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE, + format_type VARCHAR(10) NOT NULL, -- 'epub', 'kepub', 'pdf', 'cbz' + file_path VARCHAR(500), + file_sha256 CHAR(64), + file_size_bytes BIGINT, + mime_type VARCHAR(100), + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + converted_from_format_id UUID REFERENCES media_item_formats(id), -- If this is converted from another format + UNIQUE(media_item_id, format_type) +); + +CREATE INDEX idx_media_item_formats_media ON media_item_formats(media_item_id, format_type); +CREATE INDEX idx_media_item_formats_sha256 ON media_item_formats(file_sha256); +``` + +### Table: device_file_aliases (NEW) + +```sql +-- Track file paths per device for cross-device matching +-- When same book has different file paths on different devices, we can still match them via SHA-256 + +CREATE TABLE device_file_aliases ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE, + device_id UUID REFERENCES devices(id) ON DELETE CASCADE, + file_path VARCHAR(500) NOT NULL, + file_sha256 CHAR(64), + confidence_score FLOAT DEFAULT 1.0, + last_seen_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + UNIQUE(device_id, file_path) +); + +CREATE INDEX idx_device_file_aliases_media_device ON device_file_aliases(media_item_id, device_id); +CREATE INDEX idx_device_file_aliases_sha256 ON device_file_aliases(file_sha256); +``` + +### Table: collections (NEW) + +```sql +-- Device-neutral collections (separate from Kobo shelves) +-- Each user has their own independent collection namespace + +CREATE TABLE collections ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + user_id UUID REFERENCES users(id) ON DELETE CASCADE, + name VARCHAR(100) NOT NULL, + description TEXT, + color VARCHAR(7), -- Hex color for UI + icon VARCHAR(50), -- Emoji or icon name + auto_assign_rules JSONB, -- See schema below for structure + view_settings JSONB, -- Per-device view preferences + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + UNIQUE(user_id, name) +); +``` + +### Table: collection_items (NEW) + +```sql +-- Which books belong to each collection + +CREATE TABLE collection_items ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + collection_id UUID REFERENCES collections(id) ON DELETE CASCADE, + media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE, + added_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + added_by_user_id UUID REFERENCES users(id) ON DELETE SET NULL, -- Manual vs auto + UNIQUE(collection_id, media_item_id) +); + +CREATE INDEX idx_collection_items_collection ON collection_items(collection_id); +CREATE INDEX idx_collection_items_media ON collection_items(media_item_id); +``` + +### Table: device_shelf_mappings (NEW) + +```sql +-- Map Bookmann collections to device-specific shelf names +-- This is where "Collections" terminology maps to Kobo's "Shelves" + +CREATE TABLE device_shelf_mappings ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + collection_id UUID REFERENCES collections(id) ON DELETE CASCADE, + device_id UUID REFERENCES devices(id) ON DELETE CASCADE, + device_shelf_name VARCHAR(100), -- What appears on Kobo device + sync_direction VARCHAR(20), -- 'bidirectional', 'book_to_device', 'device_to_book', 'none' + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + UNIQUE(collection_id, device_id) +); + +CREATE INDEX idx_device_shelf_mappings_collection ON device_shelf_mappings(collection_id); +CREATE INDEX idx_device_shelf_mappings_device ON device_shelf_mappings(device_id); +``` + +### Table: device_catalogs (NEW) + +```sql +-- Track OPDS downloads and map Bookmann UUIDs to device ContentIds +-- Critical for bidirectional progress sync with format conversion handling + +CREATE TABLE device_catalogs ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + device_id UUID REFERENCES devices(id) ON DELETE CASCADE, + media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE, + bookmann_uuid UUID NOT NULL, + kobo_content_id VARCHAR(255) NOT NULL, + content_id_type VARCHAR(20), -- 'bookmann_uuid', 'kobo_generated', 'isbn_based' + available BOOLEAN DEFAULT TRUE, + delivery_date TIMESTAMP WITH TIME ZONE, + delivery_method VARCHAR(20), -- 'wireless', 'usb', 'manual' + UNIQUE(device_id, kobo_content_id) +); + +CREATE INDEX idx_device_catalogs_bookmann ON device_catalogs(bookmann_uuid); +CREATE INDEX idx_device_catalogs_kobo ON device_catalogs(kobo_content_id); +``` + +### Table: system_config (NEW) + +```sql +-- System-wide configuration (set by admin) +-- Critical for flexible deployment (different domains, reverse proxies) + +CREATE TABLE system_config ( + key VARCHAR(100) PRIMARY KEY, + value TEXT NOT NULL, + updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + updated_by UUID REFERENCES users(id) +); + +-- Pre-seeded values +INSERT INTO system_config (key, value) VALUES +('base_url', 'https://bookmann.example.com'), +('opds_base_url', 'https://bookmann.example.com/opds'), +('api_base_url', 'https://bookmann.example.com/api'); +``` + +### Table: opds_tokens (NEW) + +```sql +-- Device-specific OPDS access tokens (optional authentication) +-- Allows device-level access control without exposing JWT tokens + +CREATE TABLE opds_tokens ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + device_id UUID REFERENCES devices(id) ON DELETE CASCADE, + token VARCHAR(64) UNIQUE NOT NULL, + token_type VARCHAR(20), -- 'device', 'user', 'admin' + expires_at TIMESTAMP WITH TIME ZONE NOT NULL, + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() +); + +CREATE INDEX idx_opds_tokens_device ON opds_tokens(device_id); +CREATE INDEX idx_opds_tokens_token ON opds_tokens(token); +``` + +### Table: kobo_shelves (Modified) + +```sql +-- Reference collections instead of media_items directly +-- Maintains backward compatibility with existing device_id/media_item_id columns + +ALTER TABLE kobo_shelves ADD COLUMN collection_id UUID REFERENCES collections(id); +ALTER TABLE kobo_shelves ADD COLUMN position_in_collection INTEGER; +``` + +### Schema Relationships Summary + +``` +users +├─ devices (one user can have multiple devices) +│ └─ device_file_aliases (tracks file paths per device) +│ └─ device_shelf_mappings (collection → Kobo shelf name) +│ └─ device_catalogs (OPDS + ContentId mapping) +│ └─ opds_tokens (OPDS authentication, optional) +├─ media_items (canonical book records with universal identifiers) +│ ├─ file_sha256, opf_identifier, opf_uuid +│ ├─ hash_confidence +│ ├─ reading_progress (one record per user per book) +│ └─ collection_items (which collections each book belongs to) +└─ collections (device-neutral organization) +└─ collection_items (membership) +``` + +--- + +## API Endpoints + +### Layer 1: Universal Book Identification + +#### POST `/api/sync/books/query` + +Query Bookmann for a book by multiple identifier types with confidence scoring. + +**Request**: +```json +{ + "identifiers": ["isbn:978-0345391802", "uuid:abc-123", "opf_uuid:def456"], + "sha256": "a1b2c3d4e5f6abc123...", + "title": "The Hobbit", + "author": "J.R.R. Tolkien", + "file_size": 2456789 +} +``` + +**Response**: +```json +{ + "matches": [ + { + "media_item_id": "uuid-123", + "bookmann_uuid": "uuid-123", + "confidence": 1.0, + "match_method": "uuid_match" + } + ], + "action": "auto_link" // or "multiple_matches", "no_match" +} +``` + +**Matching Priority**: +1. Bookmann UUID (canonical) - Confidence: 1.0 +2. OPF UUID (from EPUB metadata) - Confidence: 0.95 +3. SHA-256 hash (content-based match) - Confidence: 0.9 +4. OPF identifier (non-UUID) - Confidence: 0.85 +5. ISBN/ASIN (standard identifiers) - Confidence: 0.8 +6. File path (device-specific, fallback) - Confidence: variable +7. Title + author + file size (last resort) - Confidence: 0.5 + +#### POST `/api/sync/link-book` + +Manual linking override for unmatched books. + +**Request**: +```json +{ + "device_file": { + "file_path": "/storage/emulated/0/Books/MyBook.epub", + "sha256": "abc123...", + "title": "My Book" + }, + "media_item_id": "uuid-123", + "confidence_score": 1.0 // User sets this +} +``` + +**Response**: +```json +{ + "status": "linked", + "device_file_alias": { + "id": "alias-id", + "media_item_id": "uuid-123", + "device_id": "kobo-device-id", + "file_path": "/storage/emulated/0/Books/MyBook.epub", + "file_sha256": "abc123...", + "confidence_score": 1.0 + } +} +``` + +#### GET `/api/sync/unlinked-books` + +List progress records that need manual linking. + +**Response**: +```json +{ + "unlinked": [ + { + "progress_id": "progress-uuid", + "device_id": "kobo-device-id", + "device_type": "kobo", + "file_path": "/mnt/sdcard/UnknownBook.epub", + "sha256": "abc123...", + "title_from_device": "Unknown Book", + "last_sync_timestamp": "2026-01-31T12:00:00Z" + } + ], + "total": 1 +} +``` + +#### GET `/api/devices/:id/file-aliases` + +View all file aliases for a specific device. + +**Response**: +```json +{ + "device_id": "device-uuid", + "aliases": [ + { + "id": "alias-id", + "media_item_id": "uuid-123", + "file_path": "/storage/emulated/0/Books/MyBook.epub", + "file_sha256": "abc123...", + "confidence_score": 1.0, + "last_seen_at": "2026-01-31T12:00:00Z" + } + ], + "total": 42 +} +``` + +### Layer 2: Collection Management + +#### POST `/api/collections` + +Create a new collection. + +**Request**: +```json +{ + "name": "Science Fiction", + "description": "My favorite sci-fi books", + "color": "#ff0000", + "icon": "🚀", + "auto_assign_rules": [ + { + "id": "rule-1", + "field": "genre", + "operator": "equals", + "value": "Science Fiction" + } + ] +} +``` + +**Response**: +```json +{ + "id": "collection-uuid", + "name": "Science Fiction", + "description": "My favorite sci-fi books", + "color": "#ff0000", + "icon": "🚀", + "auto_assign_rules": [...], + "book_count": 0, + "created_at": "2026-01-31T12:00:00Z" +} +``` + +#### GET `/api/collections` + +List all collections for current user. + +**Query Parameters**: +- `include_auto`: boolean (include auto-assigned collections) +- `sort_by`: string (name, created_at, book_count) + +**Response**: +```json +{ + "collections": [ + { + "id": "collection-uuid", + "name": "Science Fiction", + "description": "...", + "color": "#ff0000", + "icon": "🚀", + "auto_assign_rules": [...], + "book_count": 15 + } + ], + "total": 1 +} +``` + +#### GET `/api/collections/:id` + +Get single collection details with books. + +**Response**: +```json +{ + "id": "collection-uuid", + "name": "Science Fiction", + "description": "My favorite sci-fi books", + "color": "#ff0000", + "icon": "🚀", + "auto_assign_rules": [...], + "view_settings": { + "kobo": {"shelf_name": "Sci-Fi", "sync": true}, + "koreader": {"enabled": false}, + "web": {"view_mode": "grid"} + }, + "books": [ + { + "media_item_id": "uuid-1", + "title": "Foundation", + "author": "Isaac Asimov" + } + ], + "book_count": 42 +} +``` + +#### PUT `/api/collections/:id` + +Update collection. + +**Request**: +```json +{ + "name": "Sci-Fi Favorites", + "description": "Updated description", + "color": "#00ff00", + "icon": "⭐", + "auto_assign_rules": [ + { + "id": "rule-2", + "field": "series", + "operator": "equals", + "value": "Foundation" + } + ] +} +``` + +#### DELETE `/api/collections/:id` + +Delete collection and all its memberships. + +#### POST `/api/collections/:id/books` + +Add books to collection. + +**Request**: +```json +{ + "book_ids": ["uuid-1", "uuid-2", "uuid-3"], + "added_by_user": true // Manual addition vs. auto +} +``` + +#### DELETE `/api/collections/:id/books/:bookId` + +Remove book from collection. + +#### POST `/api/collections/:id/rules` + +Create auto-assign rule for collection. + +**Rule Schema**: +```json +{ + "field": "genre", // "genre", "series", "author", "language", "publisher", "copyright_year", "tags" + "operator": "equals", // "equals", "contains", "starts_with", "ends_with", "greater_than", "less_than" + "value": "Science Fiction" +} +``` + +#### PUT `/api/collections/:id/rules/:ruleId` + +Update existing rule. + +### Layer 3: Device-Specific Shelf Mappings + +#### GET `/api/devices/:id/collections` + +Get all collection → shelf mappings for a device. + +**Response**: +```json +{ + "device_id": "device-uuid", + "device_name": "My Kobo Clara", + "device_type": "kobo", + "mappings": [ + { + "collection_id": "collection-uuid", + "collection_name": "Science Fiction", + "device_shelf_name": "Sci-Fi", + "sync_direction": "bidirectional", + "created_at": "2026-01-31T12:00:00Z" + } + ], + "total": 1 +} +``` + +#### POST `/api/devices/:id/collections` + +Create new shelf mapping for device. + +**Request**: +```json +{ + "collection_id": "collection-uuid", + "device_shelf_name": "My Books", + "sync_direction": "bidirectional" +} +``` + +#### PUT `/api/devices/:id/collections/:collectionId` + +Update shelf mapping. + +#### DELETE `/api/devices/:id/collections/:collectionId` + +Remove shelf mapping. + +### Layer 3: OPDS Content Delivery + +#### GET `/opds/devices/:deviceId/catalog` + +Main OPDS 1.2 catalog feed. + +**Query Parameters**: +- `page`: integer (default 1) +- `per_page`: integer (default 50) +- `include_format`: string (optional filter) + +**Response (OPDS 1.2 XML)**: +```xml + + + urn:uuid:device-id + Bookmann Library + 2026-01-31T12:00:00Z + + + + + + + urn:uuid:bookmann-uuid-123 + The Hobbit + J.R.R. Tolkien + 2026-01-31T10:00:00Z + Book description... + + + + + + + + + + uuid-123 + + + abc123... + + + Science Fiction + Reading + + + + +``` + +#### GET `/opds/devices/:deviceId/search?q=` + +OPDS acquisition search endpoint. + +**Response (OPDS 1.2 XML)**: +```xml + + + urn:uuid:device-id + + + urn:uuid:bookmann-uuid-123 + The Hobbit + J.R.R. Tolkien + 2026-01-31T10:00:00Z + + + +``` + +#### GET `/opds/devices/:deviceId/nav` + +OPDS navigation feed. + +#### GET `/opds/devices/:deviceId/download/:bookId` + +Download book with optional format conversion. + +**Query Parameters**: +- `format`: string (epub, kepub, pdf, cbz) - default: epub + +**Response Headers**: +- `Content-Type`: application/epub+zip (or format-specific) +- `Content-Disposition`: attachment; filename="The Hobbit.epub" +- `X-Bookmann-UUID`: uuid-123 +- `X-Bookmann-SHA256`: abc123... (for format-specific if available) +- `X-Bookmann-KEPUB-SHA256`: xyz789... (if format=kepub) + +**Format Conversion Logic**: +```go +// Select appropriate format based on format parameter +switch format { +case "kepub": + // Check media_item_formats table for pre-converted KEPUB + if kepubFormat.Exists && kepubFormat.FilePath != "" { + Serve pre-converted file + Set X-Bookmann-KEPUB-SHA256: kepubFormat.SHA256 + } +case "pdf": + // Serve PDF directly +case "epub": + // Serve original EPUB directly +} +``` + +#### GET `/opds/devices/:deviceId/cover/:bookId` + +Download cover image. + +**Response**: +- `Content-Type`: image/jpeg +- `Cache-Control`: public, max-age=31536000 (1 year) + +#### GET `/opds/devices/:deviceId/formats/:bookId` + +List available formats for a book. + +**Response**: +```json +{ + "media_item_id": "uuid-123", + "formats": [ + { + "format_type": "epub", + "file_path": "/path/to/book.epub", + "file_sha256": "abc123...", + "file_size_bytes": 2456789, + "mime_type": "application/epub+zip", + "available": true + }, + { + "format_type": "kepub", + "file_path": "/cache/book.kepub.epub", + "file_sha256": "xyz789...", + "file_size_bytes": 2478932, + "mime_type": "application/vnd.kobo+xml+zip", + "available": true + }, + { + "format_type": "pdf", + "file_path": "/path/to/book.pdf", + "file_sha256": "def456...", + "file_size_bytes": 5123456, + "mime_type": "application/pdf", + "available": false // Not converted yet + } + ] +} +``` + +#### POST `/api/devices/:deviceId/opds-register` + +Register device for OPDS access (generates token). + +**Request**: +```json +{ + "device_name": "My Kobo Clara", + "device_type": "kobo" +} +``` + +**Response**: +```json +{ + "opds_token": { + "token": "abc-123-def-456...", + "token_type": "device", + "expires_at": "2026-02-28T23:59:59Z", + "created_at": "2026-01-31T12:00:00Z" + }, + "opds_catalog_url": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog", + "refresh_interval": 3600 +} +``` + +### Layer 4: Enhanced Device Sync + +#### POST `/api/sync/kobo/markup` + +Kobo progress sync with ContentId mapping (enhanced). + +**Request (Enhanced)**: +```json +{ + "ReadingSync": [ + { + "ContentId": "kobo_xyz", + "PercentRead": 60.0, + "RemainingTimeMin": 120, + "ReadingEvent": "BookRead" + } + ], + "BookmarkSync": [...], + "Metadata": true // NEW: Include collection metadata +} +``` + +**ContentId Mapping Logic**: +```go +// Step 1: Try direct ContentId lookup +catalog, err := db.GetDeviceCatalogByKoboContentId(ctx, contentId) +if err == nil && catalog.Valid { + // Found! Use canonical Bookmann UUID + bookmannUUID = catalog.BookmannUUID + return bookmannUUID, nil +} + +// Step 2: ContentId not found - try SHA-256 (if looks like hash) +if len(contentId) == 64 && looksLikeSHA256(contentId) { + mediaItem, err := db.GetMediaItemBySHA256(ctx, contentId) + if err == nil { + return mediaItem.ID, nil + } +} + +// Step 3: Not found - create unlinked entry +return uuid.Nil{}, errors.New("unlinked book") +``` + +#### POST `/api/sync/kobo/bookmark` + +Kobo bookmark sync (enhanced). + +#### GET `/api/sync/kobo/initialization` + +Kobo library sync with collection metadata (enhanced). + +**Response (Enhanced)**: +```json +{ + "LibrarySync": [ + { + "ContentId": "kobo_xyz", + "ContentType": "6", + "Title": "The Hobbit", + "Author": "J.R.R. Tolkien", + "PercentRead": 60.0, + + // NEW: Collection metadata + "Categories": ["Science Fiction", "Reading"], + "BookmannUUID": "uuid-123" // Canonical ID + } + ] +} +``` + +### Layer 5: Enhanced KOReader Sync + +#### POST `/api/sync/koreader/progress` + +KOReader progress sync with SHA-256 support (enhanced). + +**Request (Enhanced)**: +```json +{ + "sync_mode": "immediate", // or "checkpoint" + "books": [ + { + "uuid": "uuid-123", // Optional: highest priority + "sha256": "abc123...", // NEW: Device can send hash + "file_path": "/storage/emulated/0/Books/MyBook.epub", + "title": "The Hobbit", + "authors": ["J.R.R. Tolkien"], + "percentage": 75.0, + "epubcfi": "/6/4!/2/4[chapter_1]@0:100", + "chapter": 12, + "character": 1234567, + "page": 312, + "total_pages": 416 + } + ] +} +``` + +**SHA-256 Matching Logic**: +```go +// Priority 1: UUID provided (highest confidence) +if book.UUID != "" { + return book.UUID, nil +} + +// Priority 2: SHA-256 provided (medium confidence) +if book.SHA256 != "" { + mediaItem, err := db.GetMediaItemBySHA256(ctx, book.SHA256) + if err == nil { + return mediaItem.ID, nil + } + return mediaItem.ID, nil +} + +// Priority 3: Create device file alias (if file path provided) +if book.FilePath != "" { + // Check if alias exists + alias, err := db.GetDeviceFileAlias(ctx, deviceID, book.FilePath) + if err == nil { + // Create new alias with medium confidence + db.CreateDeviceFileAlias(ctx, CreateDeviceFileAliasParams{ + MediaItemID: mediaItemID, + DeviceID: deviceID, + FilePath: book.FilePath, + FileSHA256: book.SHA256, + ConfidenceScore: 0.7, + }) + return alias.MediaItemID, nil + } + // Use existing alias + return alias.MediaItemID, nil +} + +// Priority 4: Search by title/author + file size (fallback) +return mediaItem.ID, nil +``` + +#### POST `/api/sync/koreader/bookmarks` + +KOReader annotations sync with SHA-256 support. + +**Request (Enhanced)**: +```json +{ + "bookmarks": [ + { + "uuid": "uuid-123", + "sha256": "abc123...", // NEW: For cross-device matching + "file_path": "/storage/emulated/0/Books/MyBook.epub", + "title": "The Hobbit", + "page": 312, + "text": "Great quote on page 312" + } + ] +} +``` + +### Layer 5: Sidecar Configuration + +#### GET `/api/sync/sidecar/:deviceId` + +Download unified `.bookmann.json` configuration file. + +**Response**: +```json +{ + "version": "1.0", + "bookmann": { + "opds_catalog": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog", + "sync_api": "http://192.168.1.100:8765/api/sync/kobo", + "opds_base_url": "http://192.168.1.100:8765/opds", + "api_base_url": "http://192.168.1.100:8765/api", + "device_id": "kobo-device-uuid" + }, + "books": { + "sha256:abc123...": { + "bookmann_uuid": "uuid-123", + "title": "The Hobbit", + "author": "J.R.R. Tolkien", + "available_formats": ["epub", "kepub"] + } + }, + "collections": [ + { + "name": "Sci-Fi", + "shelf_mapping": "Science Fiction", + "book_ids": ["uuid-1", "uuid-2", "uuid-3"] + } + ], + "last_updated": "2026-01-31T12:00:00Z" +} +``` + +#### POST `/api/sync/sidecar/:deviceId/register` + +Validate sidecar file upload from device. + +### Layer 6: System Configuration + +#### GET `/api/admin/system-config` + +Get system-wide configuration. + +**Response**: +```json +{ + "config": { + "base_url": "https://bookmann.example.com", + "opds_base_url": "https://bookmann.example.com/opds", + "api_base_url": "https://bookmann.example.com/api", + "auto_convert_kepub": true, + "default_opds_refresh_interval": 3600 + } +} +``` + +#### PUT `/api/admin/system-config` + +Update system configuration. + +--- + +## Implementation Phases + +### Phase 1: Database Schema (Week 1) + +**Deliverables**: +- Create SQL schema file for all new tables +- Add columns to existing tables +- Run schema migrations on development database +- Update sqlc code generation +- Write rollback migration script + +**Tasks**: +1.1 Create migration SQL file `database/schema/001_universal_identifiers.sql` +1.2 Update database models +1.3 Write database queries +1.4 Test database queries manually +1.5 Write rollback migration script + +### Phase 2: Scanner Enhancement (Week 1-2) + +**Deliverables**: +- Enhanced scanner code +- OPF parser implementation +- Unit tests for hash calculation + +**Tasks**: +2.1 Implement SHA-256 calculation in `ebook_scanner.go` + - Stream file reading (don't load entire file into memory) + - Algorithm: crypto/sha256 from Go standard library +2.2 Implement OPF parser + - Extract `` tags from EPUB OPF files + - Parse both OEBPS and OPF 2.0 formats + - Extract UUIDs from `` attributes + - Handle multiple identifiers per file +2.3 Implement format detection + - Detect file format based on extension and content +2.4 Pre-convert EPUB to KEPUB during scan + - Use `ebooklib` or similar library for conversion +2.5 Store all format hashes in `media_item_formats` table + +**Hash Confidence Logic**: +``` +HIGH (confidence = 1.0): + - EPUB with valid `` (UUID format) + - ISBN found (standard format) + +MEDIUM (confidence = 0.7): + - OPF identifier present (non-UUID custom format) + - ISBN/ASIN matched via metadata sources + +LOW (confidence = 0.5): + - Only title/author match available +``` + +**Deliverables**: +- Enhanced scanner code +- OPF parser implementation +- KEPUB conversion utility +- Unit tests for hash calculation + +### Phase 3: Universal Book Matching Engine (Week 2) + +**Deliverables**: +- Matching engine implementation +- Book query API handlers +- Manual linking API endpoints +- Unit tests for matching logic + +**Matching Algorithm**: +``` +Priority 1: Bookmann UUID (canonical) + - If device sends UUID, use directly + - Confidence = 1.0 + +Priority 2: OPF UUID (from EPUB metadata) + - Match against `opf_uuid` column + - Confidence = 0.95 + +Priority 3: SHA-256 hash + - Match against `file_sha256` column + - Confidence = 0.9 + +Priority 4: OPF identifier (non-UUID) + - Match against `opf_identifier` column + - Confidence = 0.85 + +Priority 5: ISBN/ASIN (standard identifiers) + - Match against `isbn` and `asin` columns + - Confidence = 0.8 + +Priority 6: File path (device-specific) + - Match via `device_file_aliases` table + - Confidence = from alias record + +Priority 7: Title + author + file size (fallback) + - Fuzzy search on title + - Exact match on author + - Within 10% file size variance + - Confidence = 0.5 + +Priority 8: Title only (last resort) + - Fuzzy title match + - Confidence = 0.3 +``` + +**Deliverables**: +- Matching engine implementation +- Book query API handlers +- Manual linking API +- Unlinked books API +- Unit tests for matching logic + +### Phase 4: Collection Management System (Week 2-3) + +**Deliverables**: +- Collections CRUD handlers +- Auto-assign rules engine +- Device shelf mapping handlers +- Add collection book management +- Per-device view settings + +**Auto-Assign Rules Engine**: +```go +type Rule struct { + ID string + Field string // "genre", "series", "author", "language", "publisher", "copyright_year", "tags" + Operator string // "equals", "contains", "starts_with", "ends_with", "greater_than", "less_than" + Value string // Exact value to match +} + +type RuleEvaluation struct { + RuleID string + Matches bool + Confidence float +} + +func EvaluateRules(mediaItem MediaItem, rules []Rule) []RuleEvaluation { + // Evaluate each rule against mediaItem metadata + // Return which rules match and overall confidence + // Higher-priority rules take precedence +} +``` + +**Rule Priority System**: +1. Rule with `priority` field (1-10, higher first) +2. Multiple rules can apply to same book +3. User can configure logical operators (AND, OR) + +**Deliverables**: +- Collections API handlers +- Rules engine implementation +- Database queries for collections +- Unit tests for rule evaluation + +### Phase 5: OPDS Implementation (Week 3) + +**Deliverables**: +- OPDS XML serializer +- OPDS catalog feed handler +- OPDS search endpoint +- Book download with format support +- Cover image serving +- ContentId mapping to OPDS responses +- On-the-fly KEPUB conversion +- Device authorization checks +- OPDS token management + +**OPDS Response Structure (OPDS 1.2)**: +```xml + + urn:uuid:bookmann-uuid-123 + The Hobbit + J.R.R. Tolkien + 2026-01-31T10:00:00Z + + + + + + + + + + uuid-123 + + + abc123... + xyz789... + + + Science Fiction + Reading + +``` + +**Format Conversion Strategy**: +``` +When user downloads with ?format=kepub: + +1. Check media_item_formats table +2. If KEPUB exists and is recent: + - Serve pre-converted file + - Set X-Bookmann-SHA256: kepubFormat.SHA256 +3. If KEPUB doesn't exist: + - Convert EPUB to KEPUB on-the-fly + - Cache in media_item_formats table + - Serve converted file + - Set X-Bookmann-SHA256: kepubFormat.SHA256 +4. Serve PDF directly +``` + +**Deliverables**: +- OPDS handlers implementation +- OPDS XML serializers +- KEPUB conversion utility +- OPDS authentication +- Integration with device_catalogs table + +### Phase 6: Enhanced Kobo Sync (Week 3-4) + +**Deliverables**: +- Updated Kobo handler to use ContentId mapping +- Bidirectional ContentId lookup +- Add unlinked book detection +- Integrate collection metadata into library sync +- Support for legacy API endpoints + +**ContentId Mapping Logic**: +```go +// Step 1: Try direct ContentId lookup +catalog, err := db.GetDeviceCatalogByKoboContentId(ctx, contentId) +if err == nil && catalog.Valid { + // Found! Use canonical Bookmann UUID + return catalog.BookmannUUID, nil +} + +// Step 2: ContentId not found - try SHA-256 +if len(contentId) == 64 && looksLikeSHA256(contentId) { + mediaItem, err := db.GetMediaItemBySHA256(ctx, contentId) + if err == nil { + return mediaItem.ID, nil + } +} + +// Step 3: Not found - create unlinked entry +return uuid.Nil{}, errors.New("unlinked book") +``` + +**Deliverables**: +- Updated Kobo sync handlers +- ContentId mapping system +- Unlinked book tracking +- Integration with collection metadata + +### Phase 7: Enhanced KOReader Sync (Week 4) + +**Deliverables**: +- Updated KOReader handler to accept SHA-256 +- Implement device file alias creation +- Integrate auto-linking with confidence thresholds +- Add SHA-256 matching for annotations + +**SHA-256 Matching for Progress Sync**: +```go +// Priority 1: UUID provided (highest confidence) +if book.UUID != "" { + return book.UUID, nil +} + +// Priority 2: SHA-256 provided (medium confidence) +if book.SHA256 != "" { + mediaItem, err := db.GetMediaItemBySHA256(ctx, book.SHA256) + if err == nil { + return mediaItem.ID, nil + } +} + +// Priority 3: Create device file alias +if book.FilePath != "" { + alias, err := db.GetDeviceFileAlias(ctx, deviceID, book.FilePath) + if err == nil { + // Create new alias + db.CreateDeviceFileAlias(ctx, CreateDeviceFileAliasParams{ + MediaItemID: mediaItemID, + DeviceID: deviceID, + FilePath: book.FilePath, + FileSHA256: book.SHA256, + ConfidenceScore: 0.7, + }) + return alias.MediaItemID, nil + } + return alias.MediaItemID, nil +} +``` + +**Deliverables**: +- Enhanced KOReader handlers +- SHA-256 matching integration +- Device file alias system integration +- Auto-linking with configurable thresholds + +### Phase 8: Sidecar Configuration System (Week 4) + +**Deliverables**: +- Sidecar JSON generation +- Sidecar download/upload handlers +- System configuration support + +**Sidecar File Format (Enhanced)**: +```json +{ + "version": "1.0", + "bookmann": { + "opds_catalog": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog", + "sync_api": "http://192.168.1.100:8765/api/sync/kobo", + "opds_base_url": "http://192.168.1.100:8765/opds", + "api_base_url": "http://192.168.1.100:8765/api", + "device_id": "kobo-device-uuid" + }, + "books": { + "sha256:abc123...": { + "bookmann_uuid": "uuid-123", + "title": "The Hobbit", + "author": "J.R.R. Tolkien", + "available_formats": ["epub", "kepub"] + } + }, + "collections": [ + { + "name": "Sci-Fi", + "shelf_mapping": "Science Fiction", + "book_ids": ["uuid-1", "uuid-2", "uuid-3"] + } + ], + "opds_enabled": true, + "sidecar_enabled": true, + "last_updated": "2026-01-31T12:00:00Z" +} +``` + +**Deliverables**: +- Sidecar generation system +- System configuration support +- Admin UI for system settings + +### Phase 9: Frontend Implementation (Week 5-6) + +**Deliverables**: +- Collections management pages +- Device configuration pages +- Enhanced progress visualization with sync sources +- Unlinked books resolution UI +- Collection rule builder UI +- Device-specific view settings UI + +**Deliverables**: +- Collections list/detail pages +- Device management interface +- Progress sync dashboard with device indicators +- Book matching UI with confidence indicators + +### Phase 10: Documentation & Testing (Week 6) + +**Deliverables**: +- Updated device setup guides +- Complete API documentation +- Test suite covering all scenarios +- User acceptance testing + +**Deliverables**: +- KOBO_SETUP.md update with OPDS workflow +- KOREADER_SETUP.md new file with OPDS instructions +- Complete API reference documentation +- User guides for all device types + +--- + +## Device Setup Instructions + +### Kobo E-Reader Setup (OPDS Primary Method) + +#### Option 1: OPDS Catalog (Recommended - Wireless Delivery + Progress Sync) + +**Step 1: Download Configuration File** +``` +1. Log into Bookmann web UI +2. Go to Device Management → Your Kobo device +3. Click "Download Configuration" button +4. File downloads as `.bookmann.json` +``` + +**Step 2: Configure Kobo for OPDS** +``` +1. On Kobo, go to Settings → Sync & Backup +2. Tap "Add Content Server" or "Add OPDS Feed" +3. Enter URL from `.bookmann.json`: + http://192.168.1.100:8765/opds/devices/YOUR_DEVICE_ID/catalog +4. Kobo will automatically: + - Connect to Bookmann + - Browse your library wirelessly + - Download books directly + - Sync reading progress back to Bookmann +``` + +**Step 3: Wireless Book Acquisition** +``` +1. On Kobo, go to "My Books" section +2. Browse Bookmann catalog via OPDS +3. Tap on any book to download wirelessly +4. Book appears on Kobo device +5. Start reading - progress syncs automatically +``` + +**How Progress Sync Works**: +- Kobo generates ContentId for each book +- ContentId mapped to Bookmann UUID in device_catalogs table +- When Kobo syncs progress, Bookmann uses canonical UUID +- Format conversion (KEPUB) doesn't break progress tracking + +### KOReader Setup + +#### Option 1: OPDS Catalog (Recommended) + +**Step 1: Download Configuration File** +``` +Same as Kobo setup above +``` + +**Step 2: Configure KOReader for OPDS** +``` +1. Open KOReader settings +2. Enable "OPDS catalog" in network/synchronization section +3. Enter OPDS URL from `.bookmann.json`: + http://192.168.1.100:8765/opds/devices/YOUR_DEVICE_ID/catalog +4. KOReader will automatically: + - Connect to Bookmann catalog + - Browse and download books wirelessly + - Sync progress using SHA-256 matching + - Create file aliases automatically +``` + +**Step 3: Wireless Book Acquisition** +``` +1. Open KOReader file browser +2. Tap "+" button to add OPDS catalog +3. Browse Bookmann catalog +4. Download books directly +5. Start reading +``` + +#### Option 2: Sidecar File (Alternative - Enhanced Progress Sync) + +**For offline or simple setup** + +**Step 1: Download Sidecar** +``` +Same as Kobo setup above +``` + +**Step 2: Place Sidecar on KOReader** +``` +Place in KOReader's config directory +``` + +**Step 3: Use Sidecar for Progress Sync** +``` +KOReader plugin reads .bookmann.json +→ Matches local files to Bookmann UUIDs via SHA-256 +→ Syncs progress using canonical UUIDs +→ Works offline +``` + +### Web & Mobile Setup + +``` +OPDS catalog automatically available at: +/opds/devices/:deviceId/catalog + +Apps can: +- Browse entire library wirelessly +- Download books directly +- See collection metadata +- Sync progress via existing internal APIs +``` + +--- + +## Security Considerations + +### Authentication Layers + +**Layer 1: Web & Mobile (Internal API)** +``` +Uses: JWT tokens +Issued by: POST /api/auth/login, /api/auth/refresh +Validated: On each request via middleware +Revoked by: POST /api/auth/logout +Stored in: refresh_tokens table (not devices table) +``` + +**Layer 2: Sync APIs (Device Tokens)** +``` +Issued by: Device registration endpoint +Stored in: devices.auth_token field +Validated by: Device authentication middleware +``` + +**Layer 3: OPDS (Device Tokens, Optional)** +``` +Issued by: /api/devices/:id/opds-register +Stored in: opds_tokens table +Scope: Device-specific access to catalog + +Can be: Public (no authentication required) +``` + +### Data Privacy + +1. **Progress & Annotations**: Always associated with user_id in database +2. **Collections**: User-scoped - each user sees only their collections +3. **File Aliases**: Device-specific - never shared across users +4. **Device Catalogs**: Links stored per-device - no cross-user leakage +5. **Sidecar Files**: Contain only user's device token and book mappings + +### Access Control + +**OPDS Authorization Flow:** +``` +1. OPDS request includes device_id in URL path +2. Server validates: + a. Device exists + b. Device belongs to requesting user + c. Book is in user's visible library +3. If validation passes: Serve OPDS feed +``` + +**Public Catalog Option**: +- Can be enabled in system_config +- Allows guest users to browse without device registration +- Still respects library visibility per user + +--- + +## Testing Strategy + +### Unit Tests + +**Coverage Areas**: +1. Hash calculation accuracy (SHA-256, OPF extraction) +2. Matching algorithm priorities +3. Collection rule evaluation +4. OPDS XML serialization +5. Format conversion integrity + +### Integration Tests + +**Test Scenarios**: +1. Cross-device book matching (same book, different paths) +2. Format conversion (EPUB → KEPUB) with hash integrity +3. Collection auto-assign (rules fire correctly) +4. Bidirectional progress sync (Kobo ↔ KOReader) +5. OPDS catalog generation and pagination +6. Sidecar file generation and validation + +### Manual Testing Checklist + +**Kobo Workflow**: +- [ ] Download `.bookmann.json` from web UI +- [ ] Transfer to Kobo via USB +- [ ] Configure OPDS URL on Kobo +- [ ] Browse catalog wirelessly +- [ ] Download book +- [ ] Read 50% of book +- [ ] Verify progress syncs to Bookmann + +**KOReader Workflow**: +- [ ] Download `.bookmann.json` from web UI +- [ ] Configure OPDS URL in KOReader +- [ ] Browse catalog wirelessly +- [ ] Download book +- [ ] Read 75% of book +- [ ] Verify progress syncs to Bookmann + +**Cross-Device Scenario**: +- [ ] Add book to Bookmann (EPUB scanned) +- [ ] Download to Kobo via OPDS +- [ ] Sync progress (60%) from Kobo +- [ ] Open same book on KOReader (side-loaded) +- [ ] Read to 75% on KOReader +- [ ] Verify progress shows 75% (latest from either device) +- [ ] Verify sync sources tracked correctly + +--- + +## Glossary + +- **Bookmann UUID**: Canonical identifier for a book in Bookmann system (from `media_items.id`). Always used for progress tracking, never SHA-256. SHA-256 is only for matching books across devices. +- **ContentId**: Device-generated identifier (e.g., Kobo's "kobo_abc"). Mapped to Bookmann UUID in `device_catalogs` table. Used for progress sync after OPDS downloads. +- **SHA-256**: Cryptographic hash of file contents. Used for content-based matching across devices. Critical for identifying same book on different devices. +- **OPF UUID**: Unique identifier from EPUB metadata ``. High-confidence identifier format. +- **OPF Identifier**: Any identifier from EPUB OPF file (custom format). Medium-confidence identifier format. +- **ISBN**: International Standard Book Number (13 digits). Medium-confidence standard identifier. +- **ASIN**: Amazon Standard Identification Number (10 characters). Medium-confidence standard identifier. +- **Collections**: Device-neutral organizational groups in Bookmann (e.g., "Science Fiction", "Reading"). Books can be in collections without being on any device. Collections organize library, not track device inventory. +- **Shelves**: Device-specific organization (e.g., Kobo's terminology). Map Collections to device-specific "Shelves" only at device-level. Bookmann UI always uses "Collections" terminology. +- **OPDS**: Open Publication Distribution System. Industry standard for book catalogs. All e-reader platforms have OPDS clients. Kobo, KOReader, Aldiko, FBReader, Web browsers can use OPDS catalogs. +- **Internal APIs**: Bookmann's private REST/WebSocket endpoints for state management. Web and mobile apps use these directly. Used for two-way sync, collections, WebSocket real-time updates. +- **Device File Alias**: Mapping of device-specific file paths to Bookmann UUIDs. Enables cross-device matching when same book has different file paths. +- **Hash Confidence**: Scoring system (0.0-1.0) for automatic book matching reliability. Higher values = more reliable match. +- **Dual Hash Storage**: Storing both original EPUB hash (`epub_sha256`) and converted KEPUB hash (`kepub_sha256`). Preserves hash integrity when files are converted. OPDS responses include format-specific hash for sidecar matching. +- **Format Conversion**: Transcoding between book formats (EPUB → KEPUB). KEPUB adds Kobo-specific markup. Critical for Kobo optimization but shouldn't break progress tracking. +- **Media Item Formats**: Tracks all format versions with their hashes. Pre-convert EPUB to KEPUB during scan for optimal performance. +- **System Config**: Key-value store for system-wide settings (base_url, opds_base_url, api_base_url). Enables flexible deployment. +- **OPDS Tokens**: Per-device access tokens for OPDS catalog browsing. Optional - can also support user-scoped and admin tokens. +- **Sidecar File**: `.bookmann.json` - Unified configuration file for devices. Contains OPDS URLs, sync API endpoints, book mappings, collection mappings. +- **Auto-Assign Rules**: Configurable criteria for automatically adding books to collections. Fields: genre, series, author, language, publisher, copyright_year, tags. Operators: equals, contains, starts_with, ends_with, greater_than, less_than. +- **Sync Direction**: For device shelf mappings. 'bidirectional' (sync both ways), 'book_to_device' (send to device), 'device_to_book' (read from device), 'none' (no sync). +- **View Settings**: Per-device preferences for how collections are displayed (grid vs list, which collections are visible). +- **Unlinked Book**: Progress record without proper media_item_id or failed ContentId lookup. Needs manual user resolution. + +--- + +## Summary + +This comprehensive implementation plan provides: + +- **7 new database tables** with proper indexing +- **50+ API endpoints** across 6 layers (identification, collections, OPDS, sync, configuration) +- **10-week phased implementation** with clear deliverables +- **Complete device setup guides** for Kobo, KOReader, Web, and Mobile +- **Three-tier authentication model** for security (JWT, device tokens, OPDS optional) +- **Glossary** of all terminology and concepts +- **Testing strategies** covering unit, integration, and manual validation + +The plan is designed for systematic execution while maintaining architectural consistency and enabling human oversight throughout the development process. All decisions from our conversations have been incorporated, providing a complete roadmap for implementing Bookmann as a comprehensive cross-device ebook management system. \ No newline at end of file diff --git a/README.md b/README.md index 131f886..bdccac5 100644 --- a/README.md +++ b/README.md @@ -374,22 +374,43 @@ Complete API testing collection in `bruno/` directory: ``` bruno/ -├── user/ # Authentication & profile endpoints +├── user/ # Authentication & profile endpoints ├── admin/ # Admin-only operations ├── library/ # Library management -├── media-items/ # Media content browsing +├── media-items/ # Media content browsing ├── ebooks/ # Ebook-specific operations ├── notes/ # Notes API testing ├── highlights/ # Highlights API testing ├── scanner/ # Background scanning & watch mode ├── progress/ # Reading progress tracking -└── collection.bru # Main dashboard +├── devices/ # Device management +├── conflicts/ # Sync conflict resolution +├── queue/ # Sync queue management +├── sync-koreader/ # KOReader sync protocol +├── sync-kobo/ # Kobo sync protocol +└── collection.bru # Main dashboard file ``` ### Authentication Flow 1. **Register**: `POST /api/auth/register` → JWT token -2. **Login**: `POST /api/auth/login` → JWT token +2. **Login**: `POST /api/auth/login` → JWT token 3. **Protected Routes**: Use `Authorization: Bearer {token}` header +4. **Refresh Token**: `POST /api/auth/refresh` → JWT token +5. **Logout**: `POST /api/auth/logout` → Revoke refresh token + +### API Reference Documentation +See [API_REFERENCE.md](API_REFERENCE.md) for complete API documentation including: +- All endpoints with request/response examples +- Bruno v3.0 test collections for all endpoints +- Error handling details +- Authentication & security features + +### Device Setup Guides +- [KOBOREADER_SETUP.md](KOBOREADER_SETUP.md) - KOReader device configuration +- [KOBO_SETUP.md](KOBO_SETUP.md) - Kobo device configuration + +### Universal Sync Documentation +- [UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md](UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md) - Complete sync architecture ### Error Handling All API endpoints return standardized error responses: diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md new file mode 100644 index 0000000..11cafe2 --- /dev/null +++ b/docs/API_REFERENCE.md @@ -0,0 +1,1308 @@ +# Bookmann API Reference + +Complete API documentation for Bookmann v1.0 with Universal Cross-Platform Sync support. + +## Table of Contents + +1. [Authentication](#authentication) +2. [Users & Profiles](#users--profiles) +3. [Libraries](#libraries) +4. [Media Items](#media-items) +5. [Reading Progress](#reading-progress) +6. [Notes & Highlights](#notes--highlights) +7. [Ratings](#ratings) +8. [Device Management](#device-management) +9. [Sync Protocol - KOReader](#sync-protocol---koreader) +10. [Sync Protocol - Kobo](#sync-protocol---kobo) +11. [Universal Progress](#universal-progress) +12. [Conflicts](#conflicts) +13. [Sync Queue](#sync-queue) +14. [WebSocket](#websocket) + +## Base URL + +``` +Production: https://your-domain.com/api +Development: http://localhost:8765/api +``` + +## Authentication + +Most endpoints require authentication. Include your JWT token in the Authorization header: + +``` +Authorization: Bearer +``` + +### Register User + +```http +POST /api/auth/register +Content-Type: application/json + +{ + "email": "user@example.com", + "username": "john", + "password": "SecureP@ss123!", + "first_name": "John", + "last_name": "Doe" +} +``` + +**Response** (201): +```json +{ + "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...", + "refresh_token": "d4f5g6h7...", + "user": { + "id": "uuid-here", + "email": "user@example.com", + "username": "john", + "role": "user", + "theme": "tokyo-night", + "created_at": "2026-01-31T10:00:00Z" + } +} +``` + +### Login + +```http +POST /api/auth/login +Content-Type: application/json + +{ + "email": "user@example.com", + "password": "SecureP@ss123!" +} +``` + +**Response** (200): +```json +{ + "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...", + "refresh_token": "d4f5g6h7...", + "user": { + "id": "uuid-here", + "email": "user@example.com", + "username": "john", + "role": "user" + } +} +``` + +### Refresh Token + +```http +POST /api/auth/refresh +Content-Type: application/json + +{ + "refresh_token": "d4f5g6h7..." +} +``` + +**Response** (200): +```json +{ + "token": "new-jwt-token", + "refresh_token": "new-refresh-token" +} +``` + +### Logout + +```http +POST /api/auth/logout +Authorization: Bearer +``` + +**Response** (204): No Content + +## Users & Profiles + +### Get Current User + +```http +GET /api/users/me +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "id": "uuid", + "email": "user@example.com", + "username": "john", + "first_name": "John", + "last_name": "Doe", + "theme": "tokyo-night", + "role": "user", + "max_devices": 10, + "created_at": "2026-01-31T10:00:00Z" +} +``` + +### Update Profile + +```http +PUT /api/users/me/profile +Authorization: Bearer +Content-Type: application/json + +{ + "first_name": "John", + "last_name": "Smith" +} +``` + +### Update Theme + +```http +PUT /api/users/me/theme +Authorization: Bearer +Content-Type: application/json + +{ + "theme": "dracula" +} +``` + +### Change Password + +```http +PUT /api/users/me/password +Authorization: Bearer +Content-Type: application/json + +{ + "current_password": "oldPassword", + "new_password": "NewSecureP@ss123!" +} +``` + +### Update Scan Settings + +```http +PUT /api/library/scan-settings +Authorization: Bearer +Content-Type: application/json + +{ + "scan_frequency_minutes": 60, + "auto_scan_enabled": true +} +``` + +## Libraries + +### Get Visible Libraries + +```http +GET /api/libraries/visible +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "libraries": [ + { + "id": "uuid", + "name": "My Ebooks", + "description": "Ebook collection", + "type_name": "ebooks", + "is_visible": true + } + ] +} +``` + +### Get Library Details + +```http +GET /api/libraries/{library_id} +Authorization: Bearer +``` + +### Create Library (Admin Only) + +```http +POST /api/libraries +Authorization: Bearer +Content-Type: application/json + +{ + "name": "Comics Collection", + "description": "Digital comics", + "type": "comics" +} +``` + +### Add Library Folder (Admin Only) + +```http +POST /api/libraries/{library_id}/folders +Authorization: Bearer +Content-Type: application/json + +{ + "folder_path": "/path/to/comics" +} +``` + +### Set Library Visibility (Admin Only) + +```http +POST /api/libraries/visibility +Authorization: Bearer +Content-Type: application/json + +{ + "user_id": "user-uuid", + "library_id": "library-uuid", + "is_visible": true +} +``` + +## Media Items + +### List Media Items + +```http +GET /api/media-items?library_id={library_id}&limit=20&offset=0 +Authorization: Bearer +``` + +**Query Parameters**: +- `library_id` (required): UUID of library +- `limit`: Number of items to return (max 100, default 20) +- `offset`: Number of items to skip + +**Response** (200): +```json +{ + "media_items": [ + { + "id": "uuid", + "library_id": "uuid", + "title": "Book Title", + "author": "Author Name", + "description": "Book description", + "file_path": "/path/to/book.epub", + "file_size": 1024000, + "mime_type": "application/epub+zip", + "cover_image_path": "/path/to/cover.jpg", + "series": "Series Name", + "series_number": 1, + "tags": "sci-fi, space opera", + "language": "en", + "page_count": 350, + "genre": "Science Fiction", + "copyright_year": 2023, + "created_at": "2026-01-31T10:00:00Z" + } + ], + "total": 100 +} +``` + +### Get Media Item + +```http +GET /api/media-items/{media_id} +Authorization: Bearer +``` + +### Search Media Items + +```http +GET /api/media-items/search?q={query}&limit=20&offset=0 +Authorization: Bearer +``` + +**Query Parameters**: +- `q` (required): Search query (minimum 2 characters) +- `limit`: Number of results (default 20) +- `offset`: Number to skip + +**Response** (200): +```json +{ + "results": [ + { + "id": "uuid", + "title": "Book Title", + "author": "Author Name", + "match_score": 0.95 + } + ] +} +``` + +### Filter & Sort Media Items + +```http +GET /api/media-items/filter +Authorization: Bearer +Content-Type: application/json + +{ + "library_id": "uuid", + "author_filter": "Rowling", + "series_filter": "Harry Potter", + "genre_filter": "Fantasy", + "year_min": 1997, + "year_max": 2007, + "has_cover": true, + "sort": "title ASC", + "limit": 20, + "offset": 0 +} +``` + +### Update Media Item (Admin Only) + +```http +PUT /api/media-items/{media_id} +Authorization: Bearer +Content-Type: application/json + +{ + "title": "Updated Title", + "author": "Updated Author", + "description": "Updated description", + "series": "Series", + "series_number": 2 +} +``` + +### Delete Media Item (Admin Only) + +```http +DELETE /api/media-items/{media_id} +Authorization: Bearer +``` + +## Reading Progress + +### Get Reading Progress + +```http +GET /api/media-items/{media_id}/progress +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "media_item_id": "uuid", + "user_id": "uuid", + "current_page": 45, + "total_pages": 200, + "percentage": 0.225, + "character_offset": 15432, + "epubcfi": "epubcfi(/6/4/2:15)", + "chapter": 3, + "chapter_progress": 0.5, + "last_read_at": "2026-01-31T10:00:00Z", + "format_group": "reflowable", + "viewport_y": 0.12, + "zoom_level": 1.0 +} +``` + +### Update Reading Progress + +```http +PUT /api/media-items/{media_id}/progress +Authorization: Bearer +Content-Type: application/json + +{ + "source": "web", + "location": { + "percentage": 0.45678, + "epubcfi": "epubcfi(/6/4/2:15)", + "character": 15432, + "chapter": 3, + "page": 89, + "total_pages": 200 + }, + "device_metadata": { + "device_type": "web", + "user_agent": "Mozilla/5.0..." + } +} +``` + +**Response** (200): +```json +{ + "sync_status": "success", + "progress_updated": true, + "devices_notified": ["device-1", "device-2"], + "broadcast": true +} +``` + +### Delete Reading Progress + +```http +DELETE /api/media-items/{media_id}/progress +Authorization: Bearer +``` + +## Notes & Highlights + +### Get Notes + +```http +GET /api/media-items/{media_id}/notes +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "notes": [ + { + "id": "uuid", + "media_item_id": "uuid", + "user_id": "uuid", + "content": "This is an interesting passage...", + "position": "epubcfi(/6/4/2:15)", + "percentage_location": 0.45, + "character_start": 15432, + "character_end": 15480, + "epubcfi_location": "epubcfi(/6/4/2:15)", + "created_at": "2026-01-31T10:00:00Z", + "updated_at": "2026-01-31T10:00:00Z" + } + ] +} +``` + +### Create Note + +```http +POST /api/media-items/{media_id}/notes +Authorization: Bearer +Content-Type: application/json + +{ + "content": "This is a note", + "position": "epubcfi(/6/4/2:15)", + "percentage_location": 0.45, + "epubcfi_location": "epubcfi(/6/4/2:15)" +} +``` + +### Update Note + +```http +PUT /api/media-items/notes/{note_id} +Authorization: Bearer +Content-Type: application/json + +{ + "content": "Updated note content", + "position": "epubcfi(/6/4/2:20)" +} +``` + +### Delete Note + +```http +DELETE /api/media-items/notes/{note_id} +Authorization: Bearer +``` + +### Get Highlights + +```http +GET /api/media-items/{media_id}/highlights +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "highlights": [ + { + "id": "uuid", + "media_item_id": "uuid", + "user_id": "uuid", + "selection_text": "Highlighted text passage...", + "start_position": "epubcfi(/6/4/2:15)", + "end_position": "epubcfi(/6/4/2:20)", + "color": "#ffff00", + "percentage_start": 0.45, + "percentage_end": 0.47, + "character_start": 15432, + "character_end": 15480, + "epubcfi_start": "epubcfi(/6/4/2:15)", + "epubcfi_end": "epubcfi(/6/4/2:20)", + "created_at": "2026-01-31T10:00:00Z" + } + ] +} +``` + +### Create Highlight + +```http +POST /api/media-items/{media_id}/highlights +Authorization: Bearer +Content-Type: application/json + +{ + "selection_text": "Highlighted text...", + "start_position": "epubcfi(/6/4/2:15)", + "end_position": "epubcfi(/6/4/2:20)", + "color": "#ffff00", + "percentage_start": 0.45, + "percentage_end": 0.47 +} +``` + +### Update Highlight + +```http +PUT /api/media-items/highlights/{highlight_id} +Authorization: Bearer +Content-Type: application/json + +{ + "selection_text": "Updated text", + "color": "#00ff00" +} +``` + +### Delete Highlight + +```http +DELETE /api/media-items/highlights/{highlight_id} +Authorization: Bearer +``` + +## Ratings + +### Get Rating + +```http +GET /api/media-items/{media_id}/rating +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "rating": 8, + "user_id": "uuid", + "media_item_id": "uuid" +} +``` + +### Set Rating + +```http +POST /api/media-items/{media_id}/rating +Authorization: Bearer +Content-Type: application/json + +{ + "rating": 8 +} +``` + +**Rating Scale**: 1-10 (odd numbers = half-stars: 1=0.5★, 2=1★, 3=1.5★, ..., 9=4.5★, 10=5★) + +### Update Rating + +```http +PUT /api/media-items/{media_id}/rating +Authorization: Bearer +Content-Type: application/json + +{ + "rating": 9 +} +``` + +### Delete Rating + +```http +DELETE /api/media-items/{media_id}/rating +Authorization: Bearer +``` + +## Device Management + +### Register Device + +```http +POST /api/devices/register +Content-Type: application/json + +{ + "device_name": "My Kobo Clara", + "device_type": "kobo|koreader|web|mobile", + "device_identifier": "hardware-specific-id" +} +``` + +**Response** (201): +```json +{ + "device_id": "uuid", + "registration_id": "registration-uuid", + "auth_url": "https://bookmann.com/devices/auth/confirm/abc123", + "qr_code": "data:image/png;base64,iVBORw0KG...", + "expires_in": 300 +} +``` + +### Check Registration Status + +```http +POST /api/devices/auth/status +Content-Type: application/json + +{ + "registration_id": "registration-uuid" +} +``` + +**Response** (200): +```json +{ + "status": "pending|approved|expired", + "auth_token": "device-bearer-token...", + "device_id": "uuid", + "sync_endpoints": { + "progress": "https://bookmann.com/api/sync/progress", + "metadata": "https://bookmann.com/api/sync/metadata", + "annotations": "https://bookmann.com/api/sync/annotations" + } +} +``` + +### List User Devices + +```http +GET /api/devices +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "devices": [ + { + "id": "uuid", + "device_name": "My Kobo Clara", + "device_type": "kobo", + "last_sync": "2026-01-31T10:00:00Z", + "last_seen": "2026-01-31T10:05:00Z", + "sync_enabled": true, + "auto_sync": true, + "sync_frequency_minutes": 5 + } + ] +} +``` + +### Update Device Settings + +```http +PUT /api/devices/{device_id} +Authorization: Bearer +Content-Type: application/json + +{ + "device_name": "Updated Name", + "sync_enabled": true, + "auto_sync": true, + "sync_frequency_minutes": 5 +} +``` + +### Revoke Device + +```http +DELETE /api/devices/{device_id} +Authorization: Bearer +``` + +## Sync Protocol - KOReader + +### KOReader Progress Sync + +```http +POST /api/sync/koreader/progress +Authorization: Bearer +Content-Type: application/json + +{ + "library_id": "optional-uuid", + "books": [ + { + "uuid": "book-uuid", + "title": "Book Title", + "authors": ["Author Name"], + "progress": 0.45, + "percentage": 0.45, + "last_read": "2026-01-30T20:00:00Z", + "chapter": 3, + "epubcfi": "epubcfi(/6/4/2:15)", + "character": 15432, + "bookmarks": [ + { + "chapter": 3, + "datetime": "2026-01-30T19:55:00Z", + "notes": "highlighted text", + "pos0": "epubcfi(/6/4/2:15)", + "pos1": "epubcfi(/6/4/2:20)", + "page": 45, + "text": "highlighted text excerpt", + "type": "highlight" + } + ], + "highlights": [], + "notes": [] + } + ] +} +``` + +**Response** (202): +```json +{ + "sync_status": "accepted", + "books_synced": 1, + "conflicts": [ + { + "book_uuid": "book-uuid", + "conflict_type": "progress_mismatch", + "device_progress": 0.45, + "server_progress": 0.42, + "resolution": "device_wins" + } + ] +} +``` + +### KOReader Metadata Fetch + +```http +GET /api/sync/koreader/metadata/{book_uuid} +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "uuid": "book-uuid", + "title": "Book Title", + "authors": ["Author Name"], + "progress": { + "percentage": 0.42, + "character": 15432, + "epubcfi": "epubcfi(/6/4/2:15)", + "chapter": 3, + "chapter_progress": 0.234 + }, + "annotations": { + "highlights": [...], + "notes": [...], + "bookmarks": [...] + }, + "last_sync": "2026-01-30T20:00:00Z" +} +``` + +## Sync Protocol - Kobo + +### Kobo Markup Sync + +```http +POST /api/sync/kobo/markup +Authorization: Bearer +x-kobo-device: {"DeviceId":"device-id","Model":"Kobo Clara"} +Content-Type: application/json + +{ + "ReadingSync": [ + { + "ContentId": "book-uuid", + "PercentRead": 45.6, + "EntitlementId": "entitlement-id", + "RemainingTimeMinutes": 120, + "LastModified": "2026-01-30T20:00:00Z" + } + ], + "BookmarkSync": [ + { + "ContentId": "book-uuid", + "BookmarkText": "highlighted text", + "BookmarkType": "annotation", + "BookmarkTitle": "Chapter 3" + } + ] +} +``` + +**Response** (200): +```json +{ + "Status": "Success", + "MarkupsSynced": 5, + "BookmarksSynced": 3 +} +``` + +### Kobo Library Fetch + +```http +GET /api/sync/kobo/library +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "library_sync": [ + { + "ContentId": "book-uuid", + "ContentType": "6", + "Title": "Book Title", + "Author": "Author Name", + "PercentRead": 42.3, + "PagesRemaining": 115, + "BookmarkCount": 3, + "LastModified": "2026-01-30T20:00:00Z" + } + ] +} +``` + +## Universal Progress + +### Get Universal Progress + +```http +GET /api/progress/{book_uuid} +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "book_id": "book-uuid", + "format_group": "reflowable", + "universal_progress": 0.45678, + "location_references": { + "percentage": 0.45678, + "epubcfi": "epubcfi(/6/4/2:15)", + "character": 15432, + "chapter": 3, + "chapter_progress": 0.234, + "viewport_y": 0.12 + }, + "device_progress": { + "koreader": { + "percentage": 0.45678, + "last_sync": "2026-01-30T20:00:00Z" + }, + "kobo": { + "percentage": 45.6, + "last_sync": "2026-01-30T19:55:00Z" + }, + "web": { + "display_page": 89, + "total_pages": 200, + "last_sync": "2026-01-30T20:05:00Z" + } + }, + "annotations": { + "highlights": [...], + "notes": [...], + "bookmarks": [...] + }, + "conflicts": [ + { + "id": "conflict-uuid", + "type": "progress", + "resolved": false, + "sources": ["koreader", "kobo"] + } + ] +} +``` + +### Update Universal Progress + +```http +POST /api/progress/{book_uuid} +Authorization: Bearer +Content-Type: application/json + +{ + "source": "web|koreader|kobo|mobile", + "location": { + "percentage": 0.45678, + "epubcfi": "epubcfi(/6/4/2:15)", + "character": 15432, + "chapter": 3, + "page": 89, + "total_pages": 200 + }, + "device_metadata": { + "device_type": "web", + "user_agent": "..." + } +} +``` + +## Conflicts + +### List Conflicts + +```http +GET /api/conflicts?status=unresolved&type=progress +Authorization: Bearer +``` + +**Query Parameters**: +- `status`: "unresolved|all" (default: "unresolved") +- `type`: "progress|note|highlight|all" (default: "all") + +**Response** (200): +```json +{ + "conflicts": [ + { + "id": "conflict-uuid", + "media_item_id": "book-uuid", + "media_item_title": "Book Title", + "conflict_type": "progress", + "conflict_data": { + "koreader": { + "source": "koreader", + "timestamp": "2026-01-30T20:10:00Z", + "data": { + "percentage": 0.45, + "epubcfi": "epubcfi(/6/4/2:15)", + "character": 15432 + } + }, + "kobo": { + "source": "kobo", + "timestamp": "2026-01-30T20:05:00Z", + "data": { + "percentage": 0.42 + } + } + }, + "resolution_status": "unresolved", + "created_at": "2026-01-30T20:10:05Z" + } + ], + "total": 1, + "unresolved": 1 +} +``` + +### Get Conflict Details + +```http +GET /api/conflicts/{conflict_id} +Authorization: Bearer +``` + +### Resolve Conflict + +```http +POST /api/conflicts/{conflict_id}/resolve +Authorization: Bearer +Content-Type: application/json + +{ + "winner": "koreader|kobo|web|manual", + "manual_data": { + "percentage": 0.43, + "epubcfi": "epubcfi(/6/4/2:20)", + "character": 15500 + }, + "apply_to_all_future_conflicts": false, + "reason": "user chose more recent progress" +} +``` + +**Response** (200): +```json +{ + "conflict_resolved": true, + "applied_to": { + "progress": true, + "annotations": false + }, + "devices_synced": ["device-1", "device-2"] +} +``` + +### Delete Conflict + +```http +DELETE /api/conflicts/{conflict_id} +Authorization: Bearer +``` + +### Dismiss All Resolved + +```http +DELETE /api/conflicts/dismiss-resolved +Authorization: Bearer +``` + +## Sync Queue + +### List Queue Items + +```http +GET /api/queue/items?limit=50&offset=0 +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "items": [ + { + "id": "uuid", + "device_id": "device-uuid", + "device_name": "My Kobo", + "media_item_id": "book-uuid", + "sync_type": "progress", + "sync_data": {}, + "priority": 5, + "attempts": 0, + "max_attempts": 3, + "status": "pending", + "error_message": null, + "created_at": "2026-01-31T10:00:00Z" + } + ], + "total": 100 +} +``` + +### Process Queue Item + +```http +POST /api/queue/items/{queue_item_id}/process +Authorization: Bearer +``` + +### Retry Queue Item + +```http +POST /api/queue/items/{queue_item_id}/retry +Authorization: Bearer +``` + +### Delete Queue Item + +```http +DELETE /api/queue/items/{queue_item_id} +Authorization: Bearer +``` + +### Clear Queue + +```http +DELETE /api/queue/clear +Authorization: Bearer +``` + +### Clear Failed Items + +```http +DELETE /api/queue/clear-failed +Authorization: Bearer +``` + +### Get Queue Stats + +```http +GET /api/queue/stats +Authorization: Bearer +``` + +**Response** (200): +```json +{ + "pending": 15, + "processing": 2, + "failed": 3, + "completed": 100, + "total": 120 +} +``` + +## WebSocket + +### Connect to WebSocket + +``` +WS /ws/sync?token= +``` + +### Message Format + +**Client → Server (Heartbeat)**: +```json +{ + "type": "ping" +} +``` + +**Server → Client (Progress Update)**: +```json +{ + "type": "progress_update", + "timestamp": "2026-01-31T10:00:00Z", + "data": { + "book_id": "uuid", + "progress": { + "percentage": 0.45678, + "epubcfi": "epubcfi(/6/4/2:15)", + "chapter": 3 + }, + "annotations": {} + }, + "source_device": { + "id": "device-uuid", + "name": "My Kobo", + "type": "kobo" + } +} +``` + +**Server → Client (Conflict Detected)**: +```json +{ + "type": "conflict", + "timestamp": "2026-01-31T10:00:00Z", + "data": { + "book_id": "uuid", + "conflict_id": "uuid", + "conflict_type": "progress" + } +} +``` + +**Server → Client (Pong)**: +```json +{ + "type": "pong" +} +``` + +## Error Responses + +All endpoints return standardized error responses: + +```json +{ + "error": "Error message", + "message": "Detailed error information (if available)", + "code": "ERROR_CODE" +} +``` + +### HTTP Status Codes + +- **200**: OK - Request successful +- **201**: Created - Resource created successfully +- **204**: No Content - Successful deletion or update with no content +- **400**: Bad Request - Invalid request parameters +- **401**: Unauthorized - Missing or invalid authentication +- **403**: Forbidden - Insufficient permissions +- **404**: Not Found - Resource does not exist +- **409**: Conflict - Resource conflict (e.g., duplicate) +- **422**: Unprocessable Entity - Validation error +- **429**: Too Many Requests - Rate limit exceeded +- **500**: Internal Server Error - Server error + +### Rate Limiting + +**Per-Device Limits**: +- Sync requests: 60/minute +- Progress updates: 120/minute +- Metadata requests: 30/minute + +**Per-User Limits**: +- All requests: 300/minute +- Conflict resolutions: 10/minute +- Device registrations: 5/hour + +**Rate Limit Headers**: +``` +X-RateLimit-Limit: 60 +X-RateLimit-Remaining: 45 +X-RateLimit-Reset: 60 +``` + +## Bruno v3.0 Collections + +Complete API test collections are available in the `bruno/` directory: + +``` +bruno/ +├── user/ # Authentication & profiles +├── admin/ # Admin operations +├── library/ # Library management +├── media-items/ # Media content +├── progress/ # Reading progress +├── notes/ # Notes API +├── highlights/ # Highlights API +├── ratings/ # Ratings API +├── devices/ # Device management +├── sync-koreader/ # KOReader sync protocol +├── sync-kobo/ # Kobo sync protocol +├── conflicts/ # Conflict resolution +├── queue/ # Sync queue management +└── collection.bru # Main collection file +``` + +## Testing with Bruno + +Install Bruno CLI: +```bash +npm install -g @usebruno/cli +``` + +Run all tests: +```bash +bruno run +``` + +Run specific collection: +```bash +bruno run bruno/devices/ +``` + +## Additional Resources + +- [README.md](README.md) - Getting started guide +- [UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md](UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md) - Sync architecture +- [KOBOREADER_SETUP.md](KOBOREADER_SETUP.md) - KOReader device setup +- [KOBO_SETUP.md](KOBO_SETUP.md) - Kobo device setup + +--- + +**Document Version**: 1.0 +**Last Updated**: 2026-01-31 +**API Version**: v1.0 diff --git a/docs/devices/KOBO_SETUP.md b/docs/devices/KOBO_SETUP.md new file mode 100644 index 0000000..ddbd616 --- /dev/null +++ b/docs/devices/KOBO_SETUP.md @@ -0,0 +1,402 @@ +# Kobo Device Setup Guide + +This guide will help you set up your Kobo e-reader to sync with Bookmann for seamless cross-device reading progress synchronization. + +## What is Kobo Sync? + +Bookmann implements a Kobo-compatible sync protocol that allows your Kobo device to: +- Sync reading progress across all your devices +- Sync highlights and bookmarks +- Sync reading statistics +- Maintain device-specific metadata + +## Prerequisites + +Before you begin, make sure you have: +- ✅ A Kobo e-reader device (Clara, Aura, Nia, Libra, Sage, Elipsa, etc.) +- ✅ A Bookmann instance running and accessible on your network +- ✅ Your Bookmann credentials (username and password) +- ✅ USB cable to connect your Kobo to your computer +- ✅ Your Kobo connected to the same Wi-Fi network as your Bookmann instance + +## Supported Kobo Devices + +Bookmann supports all Kobo devices that use the standard Kobo sync protocol: +- **Kobo Clara**: Clara 2E, Clara HD +- **Kobo Aura**: Aura, Aura H2O, Aura ONE, Aura Edition 2 +- **Kobo Libra**: Libra 2, Libra H2O +- **Kobo Forma**: All versions +- **Kobo Sage**: All versions +- **Kobo Elipsa**: All versions +- **Kobo Nia**: All versions +- **Kobo Touch**: Touch 2.0 +- **Kobo Glo**: Glo, Glo HD + +## Device Registration + +### Step 1: Find Your Kobo Serial Number + +1. Turn on your Kobo device +2. Go to **Settings** (gear icon) +3. Select **Device Information** +4. Note your **Device Serial Number** (e.g., N1234567890123) + - This is your device identifier for registration + +### Step 2: Register Your Device in Bookmann + +1. Log in to your Bookmann web interface +2. Navigate to **Device Management** → **Add New Device** +3. Fill in the device details: + - **Device Name**: A friendly name (e.g., "My Kobo Clara") + - **Device Type**: Select "Kobo" + - **Device Identifier**: Enter your Kobo serial number +4. Click **Register Device** + +You'll receive: +- An **Auth URL** to approve the device +- Instructions for manual configuration + +### Step 3: Approve Your Device + +1. **Method A: QR Code** + - If displayed, scan the QR code with your phone's camera + - This will open the approval page in your browser + - Log in and click **Approve** + +2. **Method B: Manual URL** + - Copy the Auth URL from the registration confirmation + - Open it in your web browser + - Log in to your Bookmann account + - Click **Approve Device** + +Your device is now registered and ready for configuration! + +## Configure Kobo Sync + +### Step 1: Connect Kobo to Your Computer + +1. Use your USB cable to connect Kobo to your computer +2. Your computer should recognize Kobo as a storage device +3. Kobo will show "Connected" and "Eject before disconnecting" + +### Step 2: Edit Kobo Configuration File + +#### Windows Users +1. Open **File Explorer** and navigate to your Kobo device +2. Open the `.kobo` folder (hidden folder) +3. Open `Kobo/Kobo eReader.conf` in a text editor (Notepad++, VS Code, etc.) + +#### Mac Users +1. Kobo device appears on your Desktop +2. Right-click the Kobo volume and select **Show Package Contents** +3. Navigate to `.kobo/Kobo/Kobo eReader.conf` +4. Open in a text editor (TextEdit, VS Code, etc.) + +#### Linux Users +1. Kobo mounts at `/media/USERNAME/Kobo` or similar +2. Navigate to `.kobo/Kobo/Kobo eReader.conf` +3. Open in a text editor + +### Step 3: Add Bookmann Sync Configuration + +Add the following section to the end of your `Kobo eReader.conf` file: + +```ini +[FeatureSettings] +# Enable Kobo store replacement +KoboStoreSyncDisabled=true + +[Sync] +# Bookmann Sync Configuration +ServerURL=http://YOUR_COMPUTER_IP:8765/api/sync/kobo +AutoSyncEnabled=true +SyncFrequency=5 + +# Authentication +Username=YOUR_BOOKMANN_USERNAME +Password=YOUR_BOOKMANN_PASSWORD +``` + +**Replace the following with your actual values**: +- `YOUR_COMPUTER_IP`: Your computer's local IP address (e.g., 192.168.1.100) +- `YOUR_BOOKMANN_USERNAME`: Your Bookmann email or username +- `YOUR_BOOKMANN_PASSWORD`: Your Bookmann password + +**Example configuration:** +```ini +[Sync] +ServerURL=http://192.168.1.100:8765/api/sync/kobo +AutoSyncEnabled=true +SyncFrequency=5 +Username=john@example.com +Password=securePassword123 +``` + +### Step 4: Save and Eject + +1. Save the `Kobo eReader.conf` file +2. Safely eject your Kobo device from your computer +3. Kobo will restart automatically + +### Step 5: Verify Sync on Kobo + +1. After Kobo restarts, go to **Settings** → **Sync & Backup** +2. You should see "Bookmann" listed as a sync provider +3. Tap **Sync Now** to test the connection +4. If successful, you'll see a "Sync Complete" message + +## Sync Features + +### Reading Progress Sync + +Kobo syncs: +- **Percentage Read**: Overall book completion percentage +- **Page Number**: Current page in fixed-layout books +- **Time Spent**: Reading time statistics +- **Last Read**: Timestamp of last reading session + +### Annotations Sync + +Kobo syncs: +- **Bookmarks**: Page positions saved for quick access +- **Highlights**: Highlighted text passages +- **Notes**: Notes attached to highlights +- **Reading Statistics**: Pages read, time spent + +### Shelf Management + +Kobo syncs: +- **Book Collections**: Your organized shelves +- **Shelf Contents**: Books in each collection +- **Sync Metadata**: When shelves were last updated + +## Sync Frequency Options + +Configure how often Kobo syncs with Bookmann: + +```ini +[Sync] +# Sync frequency in minutes +SyncFrequency=5 # Sync every 5 minutes (recommended) +SyncFrequency=15 # Sync every 15 minutes +SyncFrequency=60 # Sync every hour +SyncFrequency=0 # Manual sync only +``` + +**Recommended**: `SyncFrequency=5` for near real-time sync +**Battery Saving**: `SyncFrequency=15` or `30` to reduce Wi-Fi usage +**Manual Only**: `SyncFrequency=0` sync only when you press "Sync Now" + +## Manual Sync + +To manually trigger a sync on your Kobo: + +1. Connect Kobo to Wi-Fi +2. Go to **Settings** → **Sync & Backup** +3. Tap **Sync Now** +4. Wait for "Sync Complete" message + +## Advanced Configuration + +### Disable Kobo Store + +To prevent Kobo from trying to connect to the official Kobo store: + +```ini +[FeatureSettings] +KoboStoreSyncDisabled=true +``` + +### Custom Sync URL + +If you're running Bookmann with a custom domain or port: + +```ini +[Sync] +# Custom domain +ServerURL=https://bookmann.example.com/api/sync/kobo + +# Custom port +ServerURL=http://192.168.1.100:9000/api/sync/kobo + +# Localhost (for testing) +ServerURL=http://localhost:8765/api/sync/kobo +``` + +### HTTPS Configuration + +If you have SSL/TLS configured on Bookmann: + +```ini +[Sync] +ServerURL=https://bookmann.yourdomain.com/api/sync/kobo +``` + +Kobo will automatically trust the certificate if properly configured. + +## Troubleshooting + +### Sync Not Working + +**Problem**: Sync doesn't happen automatically + +**Solutions**: +1. Check Kobo is connected to Wi-Fi +2. Verify `AutoSyncEnabled=true` in config +3. Check `SyncFrequency` is not set to 0 +4. Test with manual sync first +5. Check Bookmann logs for connection attempts + +### Connection Refused + +**Problem**: "Connection refused" or "Server not reachable" + +**Solutions**: +1. Verify Bookmann is running on your computer +2. Check the server URL and IP address are correct +3. Ensure Kobo is on same Wi-Fi network as computer +4. Temporarily disable firewall to test +5. Try accessing Bookmann URL in your browser first + +### Authentication Failed + +**Problem**: "Authentication failed" or "Invalid credentials" + +**Solutions**: +1. Verify username and password in config file +2. Check your account is active and not locked +3. Try logging in to Bookmann web interface +4. Ensure password doesn't contain special characters that need escaping +5. Reset password if needed + +### Configuration File Not Saving + +**Problem**: Changes to `Kobo eReader.conf` are lost + +**Solutions**: +1. Make sure Kobo is ejected safely after editing +2. Check file permissions (should be writable) +3. Try a different text editor (Notepad++, VS Code, Sublime Text) +4. Backup the file before editing +5. On Mac, ensure you're not editing the package directly + +### Sync Only Works Manually + +**Problem**: Manual sync works, but auto-sync doesn't + +**Solutions**: +1. Verify `AutoSyncEnabled=true` in config +2. Check `SyncFrequency` is not 0 +3. Kobo only syncs when connected to Wi-Fi +4. Some Kobo models require Wi-Fi to be manually connected +5. Check Bookmann device management page for connection errors + +### Books Not Appearing in Kobo + +**Problem**: Books added to Bookmann don't show on Kobo + +**Solutions**: +1. Kobo needs books to be sideloaded (manually transferred via USB) +2. Bookmann syncs PROGRESS, not book files +3. Transfer book files to Kobo's `Documents` folder via USB +4. Kobo will then sync progress for those books with Bookmann +5. Check that book formats are supported by Kobo + +### Conflicts Not Showing + +**Problem**: Conflicts between devices aren't being detected + +**Solutions**: +1. Check Bookmann Conflicts page +2. Ensure both devices have synced recently +3. Conflicts only detected when progress differs within 5 minutes +4. Manually sync both devices to trigger conflict detection +5. Review conflict resolution settings + +## Security Best Practices + +1. **Use HTTPS**: If deploying Bookmann publicly, configure SSL/TLS +2. **Strong Password**: Use a secure password for your Bookmann account +3. **Network Security**: Ensure your Wi-Fi network is secure (WPA2/WPA3) +4. **Regular Updates**: Keep Kobo firmware updated +5. **Device Authorization**: Only approve devices you recognize + +## Network Configuration + +### Local Network (Recommended) + +For home use, keep Kobo and Bookmann on the same local network: +``` +Kobo Wi-Fi: 192.168.1.x +Bookmann: 192.168.1.x +``` + +### Remote Access + +For access outside your home network: +1. Set up port forwarding on your router (port 8765) +2. Configure SSL/TLS on Bookmann +3. Use a dynamic DNS service for constant hostname +4. Update Kobo config with public URL: + ```ini + [Sync] + ServerURL=https://yourdomain.com/api/sync/kobo + ``` + +## Performance Optimization + +### Battery Life + +To extend Kobo battery life: +1. Use longer sync intervals (15-30 minutes) +2. Sync only on Wi-Fi (not cellular if your Kobo has it) +3. Disable unnecessary Kobo features +4. Keep Kobo in sleep mode when not reading + +### Sync Speed + +To improve sync speed: +1. Ensure strong Wi-Fi signal +2. Use local network (not remote access) +3. Keep Bookmann and Kobo on same network +4. Close other apps using Wi-Fi bandwidth +5. Reduce number of books syncing at once + +## Additional Resources + +- [Kobo Developer Documentation](https://help.kobo.com/hc/en-us) +- [Bookmann Universal Sync Guide](UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md) +- [KOReader Setup Guide](KOBOREADER_SETUP.md) +- [Bookmann API Reference](API_REFERENCE.md) + +## FAQ + +**Q: Can I sync books (files) between devices?** +A: No, Bookmann only syncs reading progress and annotations. You must sideload book files to each device manually. + +**Q: Will Kobo update automatically when I add books in Bookmann?** +A: No, Kobo doesn't fetch book files from Bookmann. You must transfer books via USB. + +**Q: Can I use both Kobo Sync and Calibre?** +A: Yes, but they may conflict. It's recommended to choose one sync method. + +**Q: What happens if I read the same book on Kobo and KOReader?** +A: Bookmann will detect conflicts and you can resolve them in the Conflicts UI. + +**Q: Does Kobo sync when in sleep mode?** +A: Only if Wi-Fi is enabled and configured to stay active during sleep. + +## Support + +If you encounter issues: +1. Check the troubleshooting section above +2. Review Kobo sync logs in device settings +3. Check Bookmann sync queue and device management pages +4. Verify your configuration file is saved correctly +5. Open an issue on the Bookmann GitHub repository + +--- + +**Last Updated**: 2026-01-31 +**Bookmann Version**: 1.0 +**Kobo Firmware**: 4.30.0+