bookhoard/docs/Code/bookmann/IMPLEMENTATION_PLAN.md

Bookmann Implementation Plan

Executive Summary

This plan implements a complete cross-device ebook management system with three major capabilities:

1. Universal Book Identification - SHA-256 hashing, UUID, ISBN, ASIN, and OPF identifiers for content-based matching across devices
2. Enhanced Collection Management - Device-neutral "Collections" with auto-assign rules and per-device customization via shelf mappings
3. OPDS-Based Wireless Book Delivery - Industry-standard book distribution for Kobo, KOReader, Web, and Mobile apps
4. Bidirectional Progress Synchronization - Real-time sync with ContentId mapping to handle format conversions (EPUB → KEPUB)
5. Device-Specific Configuration - Per-device view settings and shelf mappings while maintaining unified data model

Key Design Principle: Use OPDS for book acquisition (Layer 1) and internal APIs for state management (Layer 2), maintaining clear separation of concerns while enabling seamless user experience.

---

Table of Contents

1. Architecture Overview
2. Database Schema
3. API Endpoints
4. Implementation Phases
5. Device Setup Instructions
6. Security Considerations
7. Testing Strategy
8. Glossary

---

Architecture Overview

System Design: Two-Layer Architecture

┌─────────────────────────────────────────────────────────────────┐
│                  Bookmann Server                    │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │ Layer 1: Universal Book Identification                          │  │
│  │ SHA-256, UUID, ISBN, ASIN, OPF identifiers              │  │
│  │ Device file aliases for path tracking                             │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                          ↓ matches books universally                         │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │ Layer 2: Collections (Device-Neutral Organization)              │  │
│  │ Collections with auto-assign rules                                │  │
│  │ Device-specific shelf mappings (Kobo)                             │  │
│  │ Per-user view settings                                          │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                          ↓ provides organization                       │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │ Layer 3: OPDS (Primary Wireless Delivery)                     │  │
│  │ Per-device OPDS feeds (Kobo, KOReader, etc.)                │  │
│  │ Format conversion (EPUB → KEPUB on-the-fly)                │  │
│  │ Dual hash storage (original + converted)                         │  │
│  │ ContentId mapping (Bookmann UUID ↔ Device ID)               │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                          ↓ delivers books + provides IDs            │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │ Layer 4: Internal APIs (State Management)                 │  │
│  │ Progress sync (bidirectional)                                    │  │
│  │ Annotation sync (bidirectional)                                   │  │
│  │ Collection CRUD                                                 │  │
│  │ WebSocket real-time updates                                       │  │
│  │ Device-specific operations                                       │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────────┘

Access Method Matrix

Platform	Access Method	Purpose	Why This Method
Kobo	OPDS catalog	Kobo has built-in OPDS client, no custom Bookmann client exists
KOReader	OPDS catalog (primary) + Sidecar + Internal API	KOReader has OPDS client, also supports plugins/sidecars for enhanced features
Web App	Internal API directly	We own and control web app, can make direct API calls efficiently
Mobile App	Internal API directly	We own and control mobile app, can make direct API calls efficiently
Any OPDS Client	OPDS catalog	Public catalog standard, any app can use it for browsing/downloading

Key Design Principles

1. Canonical UUID Always Wins - Bookmann UUID (from media_items.id) is always used for progress tracking, never SHA-256. SHA-256 is only for matching books across devices, preventing format conversion issues.

2. Collections ≠ Device Inventory - Collections are organizational metadata (like "smart playlists"). Books can be in collections without being on any device. Progress/annotations sync independently of collection membership.

3. OPDS for Acquisition, Internal APIs for State - Two layers serve complementary purposes:

   • OPDS: "What books are available to download?" (public catalog)
   • Internal APIs: "How do I manage my books/sync state?" (private management)

4. Dual Hash Storage Preserves Integrity - Store both original EPUB hash (epub_sha256) and converted KEPUB hash (kepub_sha256) in media_item_formats table. OPDS responses include format-specific hash in headers, enabling sidecar matching even after conversion.

5. Three-Tier Authentication - Separate systems for different purposes:

   • Tier 1 (Web/Mobile): JWT tokens for user authentication and permissions
   • Tier 2 (Sync APIs): Device tokens for progress/annotation sync
   • Tier 3 (OPDS): Device tokens for catalog access (optional per-device)

6. Terminology Separation - Always use "Collections" terminology in Bookmann UI. Map Collections to device-specific "Shelves" only at API/device level. Kobo devices see "Shelves", KOReader/Web/Mobile see "Collections". Prevents legal issues.

7. OPDS Primary for All Devices - Kobo, KOReader, Web, and Mobile all use OPDS as primary book delivery method. Sidecar files provide fallback/enhanced features but are optional.

8. System Configuration Flexibility - Use system_config table to store base URLs (base_url, opds_base_url, api_base_url). Sidecar generation reads from these values, enabling flexible deployment (different domains, reverse proxies) with user overrides available.

---

Database Schema

Schema Overview

7 new tables + extensions to 4 existing tables

Table: media_items (Extended)

-- Universal identifiers for cross-device matching
ALTER TABLE media_items ADD COLUMN file_sha256 CHAR(64);
ALTER TABLE media_items ADD COLUMN opf_identifier VARCHAR(255);
ALTER TABLE media_items ADD COLUMN opf_uuid VARCHAR(255);

-- Hash confidence for matching priority
-- 'high': OPF UUID or ISBN available
-- 'medium': ISBN/ASIN available but no OPF UUID
-- 'low': Only title/author match available
ALTER TABLE media_items ADD COLUMN hash_confidence VARCHAR(20);

-- Create indexes for fast lookup
CREATE INDEX idx_media_items_sha256 ON media_items(file_sha256);
CREATE INDEX idx_media_items_opf_identifier ON media_items(opf_identifier);

Table: media_item_formats (NEW)

-- Track all format versions with their hashes
-- Critical for dual hash storage and format-specific OPDS delivery

CREATE TABLE media_item_formats (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    format_type VARCHAR(10) NOT NULL,           -- 'epub', 'kepub', 'pdf', 'cbz'
    file_path VARCHAR(500),
    file_sha256 CHAR(64),
    file_size_bytes BIGINT,
    mime_type VARCHAR(100),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    converted_from_format_id UUID REFERENCES media_item_formats(id),  -- If this is converted from another format
    UNIQUE(media_item_id, format_type)
);

CREATE INDEX idx_media_item_formats_media ON media_item_formats(media_item_id, format_type);
CREATE INDEX idx_media_item_formats_sha256 ON media_item_formats(file_sha256);

Table: device_file_aliases (NEW)

-- Track file paths per device for cross-device matching
-- When same book has different file paths on different devices, we can still match them via SHA-256

CREATE TABLE device_file_aliases (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    device_id UUID REFERENCES devices(id) ON DELETE CASCADE,
    file_path VARCHAR(500) NOT NULL,
    file_sha256 CHAR(64),
    confidence_score FLOAT DEFAULT 1.0,
    last_seen_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    UNIQUE(device_id, file_path)
);

CREATE INDEX idx_device_file_aliases_media_device ON device_file_aliases(media_item_id, device_id);
CREATE INDEX idx_device_file_aliases_sha256 ON device_file_aliases(file_sha256);

Table: collections (NEW)

-- Device-neutral collections (separate from Kobo shelves)
-- Each user has their own independent collection namespace

CREATE TABLE collections (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    name VARCHAR(100) NOT NULL,
    description TEXT,
    color VARCHAR(7),                    -- Hex color for UI
    icon VARCHAR(50),                    -- Emoji or icon name
    auto_assign_rules JSONB,          -- See schema below for structure
    view_settings JSONB,                -- Per-device view preferences
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    UNIQUE(user_id, name)
);

Table: collection_items (NEW)

-- Which books belong to each collection

CREATE TABLE collection_items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    collection_id UUID REFERENCES collections(id) ON DELETE CASCADE,
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    added_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    added_by_user_id UUID REFERENCES users(id) ON DELETE SET NULL,  -- Manual vs auto
    UNIQUE(collection_id, media_item_id)
);

CREATE INDEX idx_collection_items_collection ON collection_items(collection_id);
CREATE INDEX idx_collection_items_media ON collection_items(media_item_id);

Table: device_shelf_mappings (NEW)

-- Map Bookmann collections to device-specific shelf names
-- This is where "Collections" terminology maps to Kobo's "Shelves"

CREATE TABLE device_shelf_mappings (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    collection_id UUID REFERENCES collections(id) ON DELETE CASCADE,
    device_id UUID REFERENCES devices(id) ON DELETE CASCADE,
    device_shelf_name VARCHAR(100),        -- What appears on Kobo device
    sync_direction VARCHAR(20),           -- 'bidirectional', 'book_to_device', 'device_to_book', 'none'
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    UNIQUE(collection_id, device_id)
);

CREATE INDEX idx_device_shelf_mappings_collection ON device_shelf_mappings(collection_id);
CREATE INDEX idx_device_shelf_mappings_device ON device_shelf_mappings(device_id);

Table: device_catalogs (NEW)

-- Track OPDS downloads and map Bookmann UUIDs to device ContentIds
-- Critical for bidirectional progress sync with format conversion handling

CREATE TABLE device_catalogs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    device_id UUID REFERENCES devices(id) ON DELETE CASCADE,
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    bookmann_uuid UUID NOT NULL,
    kobo_content_id VARCHAR(255) NOT NULL,
    content_id_type VARCHAR(20),            -- 'bookmann_uuid', 'kobo_generated', 'isbn_based'
    available BOOLEAN DEFAULT TRUE,
    delivery_date TIMESTAMP WITH TIME ZONE,
    delivery_method VARCHAR(20),              -- 'wireless', 'usb', 'manual'
    UNIQUE(device_id, kobo_content_id)
);

CREATE INDEX idx_device_catalogs_bookmann ON device_catalogs(bookmann_uuid);
CREATE INDEX idx_device_catalogs_kobo ON device_catalogs(kobo_content_id);

Table: system_config (NEW)

-- System-wide configuration (set by admin)
-- Critical for flexible deployment (different domains, reverse proxies)

CREATE TABLE system_config (
    key VARCHAR(100) PRIMARY KEY,
    value TEXT NOT NULL,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_by UUID REFERENCES users(id)
);

-- Pre-seeded values
INSERT INTO system_config (key, value) VALUES
('base_url', 'https://bookmann.example.com'),
('opds_base_url', 'https://bookmann.example.com/opds'),
('api_base_url', 'https://bookmann.example.com/api');

Table: opds_tokens (NEW)

-- Device-specific OPDS access tokens (optional authentication)
-- Allows device-level access control without exposing JWT tokens

CREATE TABLE opds_tokens (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    device_id UUID REFERENCES devices(id) ON DELETE CASCADE,
    token VARCHAR(64) UNIQUE NOT NULL,
    token_type VARCHAR(20),                    -- 'device', 'user', 'admin'
    expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE INDEX idx_opds_tokens_device ON opds_tokens(device_id);
CREATE INDEX idx_opds_tokens_token ON opds_tokens(token);

Table: kobo_shelves (Modified)

-- Reference collections instead of media_items directly
-- Maintains backward compatibility with existing device_id/media_item_id columns

ALTER TABLE kobo_shelves ADD COLUMN collection_id UUID REFERENCES collections(id);
ALTER TABLE kobo_shelves ADD COLUMN position_in_collection INTEGER;

Schema Relationships Summary

users
├─ devices (one user can have multiple devices)
│   └─ device_file_aliases (tracks file paths per device)
│   └─ device_shelf_mappings (collection → Kobo shelf name)
│   └─ device_catalogs (OPDS + ContentId mapping)
│   └─ opds_tokens (OPDS authentication, optional)
├─ media_items (canonical book records with universal identifiers)
│   ├─ file_sha256, opf_identifier, opf_uuid
│   ├─ hash_confidence
│   ├─ reading_progress (one record per user per book)
│   └─ collection_items (which collections each book belongs to)
└─ collections (device-neutral organization)
└─ collection_items (membership)

---

API Endpoints

Layer 1: Universal Book Identification

POST /api/sync/books/query

Query Bookmann for a book by multiple identifier types with confidence scoring.

Request:

{
  "identifiers": ["isbn:978-0345391802", "uuid:abc-123", "opf_uuid:def456"],
  "sha256": "a1b2c3d4e5f6abc123...",
  "title": "The Hobbit",
  "author": "J.R.R. Tolkien",
  "file_size": 2456789
}

Response:

{
  "matches": [
    {
      "media_item_id": "uuid-123",
      "bookmann_uuid": "uuid-123",
      "confidence": 1.0,
      "match_method": "uuid_match"
    }
  ],
  "action": "auto_link"  // or "multiple_matches", "no_match"
}

Matching Priority:

1. Bookmann UUID (canonical) - Confidence: 1.0
2. OPF UUID (from EPUB metadata) - Confidence: 0.95
3. SHA-256 hash (content-based match) - Confidence: 0.9
4. OPF identifier (non-UUID) - Confidence: 0.85
5. ISBN/ASIN (standard identifiers) - Confidence: 0.8
6. File path (device-specific, fallback) - Confidence: variable
7. Title + author + file size (last resort) - Confidence: 0.5

POST /api/sync/link-book

Manual linking override for unmatched books.

Request:

{
  "device_file": {
    "file_path": "/storage/emulated/0/Books/MyBook.epub",
    "sha256": "abc123...",
    "title": "My Book"
  },
  "media_item_id": "uuid-123",
  "confidence_score": 1.0  // User sets this
}

Response:

{
  "status": "linked",
  "device_file_alias": {
      "id": "alias-id",
      "media_item_id": "uuid-123",
      "device_id": "kobo-device-id",
      "file_path": "/storage/emulated/0/Books/MyBook.epub",
      "file_sha256": "abc123...",
      "confidence_score": 1.0
  }
}

GET /api/sync/unlinked-books

List progress records that need manual linking.

Response:

{
  "unlinked": [
    {
      "progress_id": "progress-uuid",
      "device_id": "kobo-device-id",
      "device_type": "kobo",
      "file_path": "/mnt/sdcard/UnknownBook.epub",
      "sha256": "abc123...",
      "title_from_device": "Unknown Book",
      "last_sync_timestamp": "2026-01-31T12:00:00Z"
    }
  ],
  "total": 1
}

GET /api/devices/:id/file-aliases

View all file aliases for a specific device.

Response:

{
  "device_id": "device-uuid",
  "aliases": [
    {
      "id": "alias-id",
      "media_item_id": "uuid-123",
      "file_path": "/storage/emulated/0/Books/MyBook.epub",
      "file_sha256": "abc123...",
      "confidence_score": 1.0,
      "last_seen_at": "2026-01-31T12:00:00Z"
    }
  ],
  "total": 42
}

Layer 2: Collection Management

POST /api/collections

Create a new collection.

Request:

{
  "name": "Science Fiction",
  "description": "My favorite sci-fi books",
  "color": "#ff0000",
  "icon": "🚀",
  "auto_assign_rules": [
    {
      "id": "rule-1",
      "field": "genre",
      "operator": "equals",
      "value": "Science Fiction"
    }
  ]
}

Response:

{
  "id": "collection-uuid",
  "name": "Science Fiction",
  "description": "My favorite sci-fi books",
  "color": "#ff0000",
  "icon": "🚀",
  "auto_assign_rules": [...],
  "book_count": 0,
  "created_at": "2026-01-31T12:00:00Z"
}

GET /api/collections

List all collections for current user.

Query Parameters:

• include_auto: boolean (include auto-assigned collections)
• sort_by: string (name, created_at, book_count)

Response:

{
  "collections": [
    {
      "id": "collection-uuid",
      "name": "Science Fiction",
      "description": "...",
      "color": "#ff0000",
      "icon": "🚀",
      "auto_assign_rules": [...],
      "book_count": 15
    }
  ],
  "total": 1
}

GET /api/collections/:id

Get single collection details with books.

Response:

{
  "id": "collection-uuid",
  "name": "Science Fiction",
  "description": "My favorite sci-fi books",
  "color": "#ff0000",
  "icon": "🚀",
  "auto_assign_rules": [...],
  "view_settings": {
    "kobo": {"shelf_name": "Sci-Fi", "sync": true},
    "koreader": {"enabled": false},
    "web": {"view_mode": "grid"}
  },
  "books": [
    {
      "media_item_id": "uuid-1",
      "title": "Foundation",
      "author": "Isaac Asimov"
    }
  ],
  "book_count": 42
}

PUT /api/collections/:id

Update collection.

Request:

{
  "name": "Sci-Fi Favorites",
  "description": "Updated description",
  "color": "#00ff00",
  "icon": "⭐",
  "auto_assign_rules": [
    {
      "id": "rule-2",
      "field": "series",
      "operator": "equals",
      "value": "Foundation"
    }
  ]
}

DELETE /api/collections/:id

Delete collection and all its memberships.

POST /api/collections/:id/books

Add books to collection.

Request:

{
  "book_ids": ["uuid-1", "uuid-2", "uuid-3"],
  "added_by_user": true  // Manual addition vs. auto
}

DELETE /api/collections/:id/books/:bookId

Remove book from collection.

POST /api/collections/:id/rules

Create auto-assign rule for collection.

Rule Schema:

{
  "field": "genre",        // "genre", "series", "author", "language", "publisher", "copyright_year", "tags"
  "operator": "equals",       // "equals", "contains", "starts_with", "ends_with", "greater_than", "less_than"
  "value": "Science Fiction"
}

PUT /api/collections/:id/rules/:ruleId

Update existing rule.

Layer 3: Device-Specific Shelf Mappings

GET /api/devices/:id/collections

Get all collection → shelf mappings for a device.

Response:

{
  "device_id": "device-uuid",
  "device_name": "My Kobo Clara",
  "device_type": "kobo",
  "mappings": [
    {
      "collection_id": "collection-uuid",
      "collection_name": "Science Fiction",
      "device_shelf_name": "Sci-Fi",
      "sync_direction": "bidirectional",
      "created_at": "2026-01-31T12:00:00Z"
    }
  ],
  "total": 1
}

POST /api/devices/:id/collections

Create new shelf mapping for device.

Request:

{
  "collection_id": "collection-uuid",
  "device_shelf_name": "My Books",
  "sync_direction": "bidirectional"
}

PUT /api/devices/:id/collections/:collectionId

Update shelf mapping.

DELETE /api/devices/:id/collections/:collectionId

Remove shelf mapping.

Layer 3: OPDS Content Delivery

GET /opds/devices/:deviceId/catalog

Main OPDS 1.2 catalog feed.

Query Parameters:

• page: integer (default 1)
• per_page: integer (default 50)
• include_format: string (optional filter)

Response (OPDS 1.2 XML):

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" 
       xmlns:opds="http://opds-spec.org/2010/"
       xmlns:dc="http://purl.org/dc/elements/1.1/">
  <id>urn:uuid:device-id</id>
  <title>Bookmann Library</title>
  <updated>2026-01-31T12:00:00Z</updated>
  
  <link rel="self" href="http://192.168.1.100:8765/opds/devices/kobo-id/catalog"/>
  <link rel="search" href="http://192.168.1.100:8765/opds/devices/kobo-id/search"/>
  <link rel="start" href="http://192.168.1.100:8765/opds/devices/kobo-id/nav"/>
  
  <entry>
    <id>urn:uuid:bookmann-uuid-123</id>
    <dc:title>The Hobbit</dc:title>
    <dc:creator>J.R.R. Tolkien</dc:creator>
    <updated>2026-01-31T10:00:00Z</updated>
    <summary>Book description...</summary>
    
    <!-- Acquisition links (OPDS standard) -->
    <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123" 
          type="application/epub+zip" 
          rel="http://opds-spec.org/acquisition/open-access"/>
    
    <!-- Format variants (all devices get same) -->
    <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123?format=kepub" 
          type="application/vnd.kobo+xml+zip" 
          rel="alternate"/>
    <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123?format=pdf" 
          type="application/pdf" 
          rel="alternate"/>
    
    <!-- Canonical ID for progress matching -->
    <dc:identifier id="bookmann">uuid-123</dc:identifier>
    
    <!-- Hash for sidecar matching -->
    <meta property="bookmann:sha256">abc123...</meta>
    
    <!-- Collections as categories -->
    <category scheme="http://bookmann.example.com/collections">Science Fiction</category>
    <category scheme="http://bookmann.example.com/collections">Reading</category>
  </entry>
  
  <!-- More entries... -->
</feed>

GET /opds/devices/:deviceId/search?q=

OPDS acquisition search endpoint.

Response (OPDS 1.2 XML):

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" 
       xmlns:opds="http://opds-spec.org/2010/">
  <id>urn:uuid:device-id</id>
  
  <entry>
    <id>urn:uuid:bookmann-uuid-123</id>
    <dc:title>The Hobbit</dc:title>
    <dc:creator>J.R.R. Tolkien</dc:creator>
    <updated>2026-01-31T10:00:00Z</updated>
    <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123" 
          type="application/epub+zip" 
          rel="http://opds-spec.org/acquisition/open-access"/>
  </entry>
</feed>

GET /opds/devices/:deviceId/nav

OPDS navigation feed.

GET /opds/devices/:deviceId/download/:bookId

Download book with optional format conversion.

Query Parameters:

• format: string (epub, kepub, pdf, cbz) - default: epub

Response Headers:

• Content-Type: application/epub+zip (or format-specific)
• Content-Disposition: attachment; filename="The Hobbit.epub"
• X-Bookmann-UUID: uuid-123
• X-Bookmann-SHA256: abc123... (for format-specific if available)
• X-Bookmann-KEPUB-SHA256: xyz789... (if format=kepub)

Format Conversion Logic:

// Select appropriate format based on format parameter
switch format {
case "kepub":
    // Check media_item_formats table for pre-converted KEPUB
    if kepubFormat.Exists && kepubFormat.FilePath != "" {
        Serve pre-converted file
        Set X-Bookmann-KEPUB-SHA256: kepubFormat.SHA256
    }
case "pdf":
    // Serve PDF directly
case "epub":
    // Serve original EPUB directly
}

GET /opds/devices/:deviceId/cover/:bookId

Download cover image.

Response:

• Content-Type: image/jpeg
• Cache-Control: public, max-age=31536000 (1 year)

GET /opds/devices/:deviceId/formats/:bookId

List available formats for a book.

Response:

{
  "media_item_id": "uuid-123",
  "formats": [
    {
      "format_type": "epub",
      "file_path": "/path/to/book.epub",
      "file_sha256": "abc123...",
      "file_size_bytes": 2456789,
      "mime_type": "application/epub+zip",
      "available": true
    },
    {
      "format_type": "kepub",
      "file_path": "/cache/book.kepub.epub",
      "file_sha256": "xyz789...",
      "file_size_bytes": 2478932,
      "mime_type": "application/vnd.kobo+xml+zip",
      "available": true
    },
    {
      "format_type": "pdf",
      "file_path": "/path/to/book.pdf",
      "file_sha256": "def456...",
      "file_size_bytes": 5123456,
      "mime_type": "application/pdf",
      "available": false  // Not converted yet
    }
  ]
}

POST /api/devices/:deviceId/opds-register

Register device for OPDS access (generates token).

Request:

{
  "device_name": "My Kobo Clara",
  "device_type": "kobo"
}

Response:

{
  "opds_token": {
    "token": "abc-123-def-456...",
    "token_type": "device",
    "expires_at": "2026-02-28T23:59:59Z",
    "created_at": "2026-01-31T12:00:00Z"
  },
  "opds_catalog_url": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog",
  "refresh_interval": 3600
}

Layer 4: Enhanced Device Sync

POST /api/sync/kobo/markup

Kobo progress sync with ContentId mapping (enhanced).

Request (Enhanced):

{
  "ReadingSync": [
    {
      "ContentId": "kobo_xyz",
      "PercentRead": 60.0,
      "RemainingTimeMin": 120,
      "ReadingEvent": "BookRead"
    }
  ],
  "BookmarkSync": [...],
  "Metadata": true  // NEW: Include collection metadata
}

ContentId Mapping Logic:

// Step 1: Try direct ContentId lookup
catalog, err := db.GetDeviceCatalogByKoboContentId(ctx, contentId)
if err == nil && catalog.Valid {
    // Found! Use canonical Bookmann UUID
    bookmannUUID = catalog.BookmannUUID
    return bookmannUUID, nil
}

// Step 2: ContentId not found - try SHA-256 (if looks like hash)
if len(contentId) == 64 && looksLikeSHA256(contentId) {
    mediaItem, err := db.GetMediaItemBySHA256(ctx, contentId)
    if err == nil {
        return mediaItem.ID, nil
    }
}

// Step 3: Not found - create unlinked entry
return uuid.Nil{}, errors.New("unlinked book")

POST /api/sync/kobo/bookmark

Kobo bookmark sync (enhanced).

GET /api/sync/kobo/initialization

Kobo library sync with collection metadata (enhanced).

Response (Enhanced):

{
  "LibrarySync": [
    {
      "ContentId": "kobo_xyz",
      "ContentType": "6",
      "Title": "The Hobbit",
      "Author": "J.R.R. Tolkien",
      "PercentRead": 60.0,
      
      // NEW: Collection metadata
      "Categories": ["Science Fiction", "Reading"],
      "BookmannUUID": "uuid-123"  // Canonical ID
    }
  ]
}

Layer 5: Enhanced KOReader Sync

POST /api/sync/koreader/progress

KOReader progress sync with SHA-256 support (enhanced).

Request (Enhanced):

{
  "sync_mode": "immediate",  // or "checkpoint"
  "books": [
    {
      "uuid": "uuid-123",  // Optional: highest priority
      "sha256": "abc123...",  // NEW: Device can send hash
      "file_path": "/storage/emulated/0/Books/MyBook.epub",
      "title": "The Hobbit",
      "authors": ["J.R.R. Tolkien"],
      "percentage": 75.0,
      "epubcfi": "/6/4!/2/4[chapter_1]@0:100",
      "chapter": 12,
      "character": 1234567,
      "page": 312,
      "total_pages": 416
    }
  ]
}

SHA-256 Matching Logic:

// Priority 1: UUID provided (highest confidence)
if book.UUID != "" {
    return book.UUID, nil
}

// Priority 2: SHA-256 provided (medium confidence)
if book.SHA256 != "" {
    mediaItem, err := db.GetMediaItemBySHA256(ctx, book.SHA256)
    if err == nil {
        return mediaItem.ID, nil
    }
    return mediaItem.ID, nil
}

// Priority 3: Create device file alias (if file path provided)
if book.FilePath != "" {
    // Check if alias exists
    alias, err := db.GetDeviceFileAlias(ctx, deviceID, book.FilePath)
    if err == nil {
        // Create new alias with medium confidence
        db.CreateDeviceFileAlias(ctx, CreateDeviceFileAliasParams{
            MediaItemID: mediaItemID,
            DeviceID: deviceID,
            FilePath: book.FilePath,
            FileSHA256: book.SHA256,
            ConfidenceScore: 0.7,
        })
        return alias.MediaItemID, nil
    }
    // Use existing alias
    return alias.MediaItemID, nil
}

// Priority 4: Search by title/author + file size (fallback)
return mediaItem.ID, nil

POST /api/sync/koreader/bookmarks

KOReader annotations sync with SHA-256 support.

Request (Enhanced):

{
  "bookmarks": [
    {
      "uuid": "uuid-123",
      "sha256": "abc123...",  // NEW: For cross-device matching
      "file_path": "/storage/emulated/0/Books/MyBook.epub",
      "title": "The Hobbit",
      "page": 312,
      "text": "Great quote on page 312"
    }
  ]
}

Layer 5: Sidecar Configuration

GET /api/sync/sidecar/:deviceId

Download unified .bookmann.json configuration file.

Response:

{
  "version": "1.0",
  "bookmann": {
    "opds_catalog": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog",
    "sync_api": "http://192.168.1.100:8765/api/sync/kobo",
    "opds_base_url": "http://192.168.1.100:8765/opds",
    "api_base_url": "http://192.168.1.100:8765/api",
    "device_id": "kobo-device-uuid"
  },
  "books": {
    "sha256:abc123...": {
      "bookmann_uuid": "uuid-123",
      "title": "The Hobbit",
      "author": "J.R.R. Tolkien",
      "available_formats": ["epub", "kepub"]
    }
  },
  "collections": [
    {
      "name": "Sci-Fi",
      "shelf_mapping": "Science Fiction",
      "book_ids": ["uuid-1", "uuid-2", "uuid-3"]
    }
  ],
  "last_updated": "2026-01-31T12:00:00Z"
}

POST /api/sync/sidecar/:deviceId/register

Validate sidecar file upload from device.

Layer 6: System Configuration

GET /api/admin/system-config

Get system-wide configuration.

Response:

{
  "config": {
    "base_url": "https://bookmann.example.com",
    "opds_base_url": "https://bookmann.example.com/opds",
    "api_base_url": "https://bookmann.example.com/api",
    "auto_convert_kepub": true,
    "default_opds_refresh_interval": 3600
  }
}

PUT /api/admin/system-config

Update system configuration.

---

Implementation Phases

Phase 1: Database Schema (Week 1)

Deliverables:

• Create SQL schema file for all new tables
• Add columns to existing tables
• Run schema migrations on development database
• Update sqlc code generation
• Write rollback migration script

Tasks: 1.1 Create migration SQL file database/schema/001_universal_identifiers.sql 1.2 Update database models 1.3 Write database queries 1.4 Test database queries manually 1.5 Write rollback migration script

Phase 2: Scanner Enhancement (Week 1-2)

Deliverables:

• Enhanced scanner code
• OPF parser implementation
• Unit tests for hash calculation

Tasks: 2.1 Implement SHA-256 calculation in ebook_scanner.go

• Stream file reading (don't load entire file into memory)
• Algorithm: crypto/sha256 from Go standard library 2.2 Implement OPF parser
• Extract <dc:identifier> tags from EPUB OPF files
• Parse both OEBPS and OPF 2.0 formats
• Extract UUIDs from <dc:identifier id="..."> attributes
• Handle multiple identifiers per file 2.3 Implement format detection
• Detect file format based on extension and content 2.4 Pre-convert EPUB to KEPUB during scan
• Use ebooklib or similar library for conversion 2.5 Store all format hashes in media_item_formats table

Hash Confidence Logic:

HIGH (confidence = 1.0):
  - EPUB with valid `<dc:identifier>` (UUID format)
  - ISBN found (standard format)

MEDIUM (confidence = 0.7):
  - OPF identifier present (non-UUID custom format)
  - ISBN/ASIN matched via metadata sources

LOW (confidence = 0.5):
  - Only title/author match available

Deliverables:

• Enhanced scanner code
• OPF parser implementation
• KEPUB conversion utility
• Unit tests for hash calculation

Phase 3: Universal Book Matching Engine (Week 2)

Deliverables:

• Matching engine implementation
• Book query API handlers
• Manual linking API endpoints
• Unit tests for matching logic

Matching Algorithm:

Priority 1: Bookmann UUID (canonical)
  - If device sends UUID, use directly
  - Confidence = 1.0

Priority 2: OPF UUID (from EPUB metadata)
  - Match against `opf_uuid` column
  - Confidence = 0.95

Priority 3: SHA-256 hash
  - Match against `file_sha256` column
  - Confidence = 0.9

Priority 4: OPF identifier (non-UUID)
  - Match against `opf_identifier` column
  - Confidence = 0.85

Priority 5: ISBN/ASIN (standard identifiers)
  - Match against `isbn` and `asin` columns
  - Confidence = 0.8

Priority 6: File path (device-specific)
  - Match via `device_file_aliases` table
  - Confidence = from alias record

Priority 7: Title + author + file size (fallback)
  - Fuzzy search on title
  - Exact match on author
  - Within 10% file size variance
  - Confidence = 0.5

Priority 8: Title only (last resort)
  - Fuzzy title match
  - Confidence = 0.3

Deliverables:

• Matching engine implementation
• Book query API handlers
• Manual linking API
• Unlinked books API
• Unit tests for matching logic

Phase 4: Collection Management System (Week 2-3)

Deliverables:

• Collections CRUD handlers
• Auto-assign rules engine
• Device shelf mapping handlers
• Add collection book management
• Per-device view settings

Auto-Assign Rules Engine:

type Rule struct {
    ID       string
    Field    string  // "genre", "series", "author", "language", "publisher", "copyright_year", "tags"
    Operator string  // "equals", "contains", "starts_with", "ends_with", "greater_than", "less_than"
    Value     string  // Exact value to match
}

type RuleEvaluation struct {
    RuleID    string
    Matches    bool
    Confidence float
}

func EvaluateRules(mediaItem MediaItem, rules []Rule) []RuleEvaluation {
    // Evaluate each rule against mediaItem metadata
    // Return which rules match and overall confidence
    // Higher-priority rules take precedence
}

Rule Priority System:

1. Rule with priority field (1-10, higher first)
2. Multiple rules can apply to same book
3. User can configure logical operators (AND, OR)

Deliverables:

• Collections API handlers
• Rules engine implementation
• Database queries for collections
• Unit tests for rule evaluation

Phase 5: OPDS Implementation (Week 3)

Deliverables:

• OPDS XML serializer
• OPDS catalog feed handler
• OPDS search endpoint
• Book download with format support
• Cover image serving
• ContentId mapping to OPDS responses
• On-the-fly KEPUB conversion
• Device authorization checks
• OPDS token management

OPDS Response Structure (OPDS 1.2):

<entry>
  <id>urn:uuid:bookmann-uuid-123</id>
  <dc:title>The Hobbit</dc:title>
  <dc:creator>J.R.R. Tolkien</dc:creator>
  <updated>2026-01-31T10:00:00Z</updated>
  
  <!-- Acquisition links (OPDS standard) -->
  <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123" 
          type="application/epub+zip" 
          rel="http://opds-spec.org/acquisition/open-access"/>
  
  <!-- Format variants (all devices get same) -->
  <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123?format=kepub" 
          type="application/vnd.kobo+xml+zip" 
          rel="alternate"/>
  <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123?format=pdf" 
          type="application/pdf" 
          rel="alternate"/>
  
  <!-- Canonical ID for progress matching -->
  <dc:identifier id="bookmann">uuid-123</dc:identifier>
  
  <!-- Hash for sidecar matching (format-specific if available) -->
  <meta property="bookmann:sha256">abc123...</meta>
  <meta property="bookmann:kepub_sha256">xyz789...</meta>
  
  <!-- Collections as categories -->
  <category scheme="http://bookmann.example.com/collections">Science Fiction</category>
  <category scheme="http://bookmann.example.com/collections">Reading</category>
  </entry>

Format Conversion Strategy:

When user downloads with ?format=kepub:

1. Check media_item_formats table
2. If KEPUB exists and is recent:
   - Serve pre-converted file
   - Set X-Bookmann-SHA256: kepubFormat.SHA256
3. If KEPUB doesn't exist:
   - Convert EPUB to KEPUB on-the-fly
   - Cache in media_item_formats table
   - Serve converted file
   - Set X-Bookmann-SHA256: kepubFormat.SHA256
4. Serve PDF directly

Deliverables:

• OPDS handlers implementation
• OPDS XML serializers
• KEPUB conversion utility
• OPDS authentication
• Integration with device_catalogs table

Phase 6: Enhanced Kobo Sync (Week 3-4)

Deliverables:

• Updated Kobo handler to use ContentId mapping
• Bidirectional ContentId lookup
• Add unlinked book detection
• Integrate collection metadata into library sync
• Support for legacy API endpoints

ContentId Mapping Logic:

// Step 1: Try direct ContentId lookup
catalog, err := db.GetDeviceCatalogByKoboContentId(ctx, contentId)
if err == nil && catalog.Valid {
    // Found! Use canonical Bookmann UUID
    return catalog.BookmannUUID, nil
}

// Step 2: ContentId not found - try SHA-256
if len(contentId) == 64 && looksLikeSHA256(contentId) {
    mediaItem, err := db.GetMediaItemBySHA256(ctx, contentId)
    if err == nil {
        return mediaItem.ID, nil
    }
}

// Step 3: Not found - create unlinked entry
return uuid.Nil{}, errors.New("unlinked book")

Deliverables:

• Updated Kobo sync handlers
• ContentId mapping system
• Unlinked book tracking
• Integration with collection metadata

Phase 7: Enhanced KOReader Sync (Week 4)

Deliverables:

• Updated KOReader handler to accept SHA-256
• Implement device file alias creation
• Integrate auto-linking with confidence thresholds
• Add SHA-256 matching for annotations

SHA-256 Matching for Progress Sync:

// Priority 1: UUID provided (highest confidence)
if book.UUID != "" {
    return book.UUID, nil
}

// Priority 2: SHA-256 provided (medium confidence)
if book.SHA256 != "" {
    mediaItem, err := db.GetMediaItemBySHA256(ctx, book.SHA256)
    if err == nil {
        return mediaItem.ID, nil
    }
}

// Priority 3: Create device file alias
if book.FilePath != "" {
    alias, err := db.GetDeviceFileAlias(ctx, deviceID, book.FilePath)
    if err == nil {
        // Create new alias
        db.CreateDeviceFileAlias(ctx, CreateDeviceFileAliasParams{
            MediaItemID: mediaItemID,
            DeviceID: deviceID,
            FilePath: book.FilePath,
            FileSHA256: book.SHA256,
            ConfidenceScore: 0.7,
        })
        return alias.MediaItemID, nil
    }
    return alias.MediaItemID, nil
}

Deliverables:

• Enhanced KOReader handlers
• SHA-256 matching integration
• Device file alias system integration
• Auto-linking with configurable thresholds

Phase 8: Sidecar Configuration System (Week 4)

Deliverables:

• Sidecar JSON generation
• Sidecar download/upload handlers
• System configuration support

Sidecar File Format (Enhanced):

{
  "version": "1.0",
  "bookmann": {
    "opds_catalog": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog",
    "sync_api": "http://192.168.1.100:8765/api/sync/kobo",
    "opds_base_url": "http://192.168.1.100:8765/opds",
    "api_base_url": "http://192.168.1.100:8765/api",
    "device_id": "kobo-device-uuid"
  },
  "books": {
    "sha256:abc123...": {
      "bookmann_uuid": "uuid-123",
      "title": "The Hobbit",
      "author": "J.R.R. Tolkien",
      "available_formats": ["epub", "kepub"]
    }
  },
  "collections": [
    {
      "name": "Sci-Fi",
      "shelf_mapping": "Science Fiction",
      "book_ids": ["uuid-1", "uuid-2", "uuid-3"]
    }
  ],
  "opds_enabled": true,
  "sidecar_enabled": true,
  "last_updated": "2026-01-31T12:00:00Z"
}

Deliverables:

• Sidecar generation system
• System configuration support
• Admin UI for system settings

Phase 9: Frontend Implementation (Week 5-6)

Deliverables:

• Collections management pages
• Device configuration pages
• Enhanced progress visualization with sync sources
• Unlinked books resolution UI
• Collection rule builder UI
• Device-specific view settings UI

Deliverables:

• Collections list/detail pages
• Device management interface
• Progress sync dashboard with device indicators
• Book matching UI with confidence indicators

Phase 10: Documentation & Testing (Week 6)

Deliverables:

• Updated device setup guides
• Complete API documentation
• Test suite covering all scenarios
• User acceptance testing

Deliverables:

• KOBO_SETUP.md update with OPDS workflow
• KOREADER_SETUP.md new file with OPDS instructions
• Complete API reference documentation
• User guides for all device types

---

Device Setup Instructions

Kobo E-Reader Setup (OPDS Primary Method)

Option 1: OPDS Catalog (Recommended - Wireless Delivery + Progress Sync)

Step 1: Download Configuration File

1. Log into Bookmann web UI
2. Go to Device Management → Your Kobo device
3. Click "Download Configuration" button
4. File downloads as `.bookmann.json`

Step 2: Configure Kobo for OPDS

1. On Kobo, go to Settings → Sync & Backup
2. Tap "Add Content Server" or "Add OPDS Feed"
3. Enter URL from `.bookmann.json`:
   http://192.168.1.100:8765/opds/devices/YOUR_DEVICE_ID/catalog
4. Kobo will automatically:
   - Connect to Bookmann
   - Browse your library wirelessly
   - Download books directly
   - Sync reading progress back to Bookmann

Step 3: Wireless Book Acquisition

1. On Kobo, go to "My Books" section
2. Browse Bookmann catalog via OPDS
3. Tap on any book to download wirelessly
4. Book appears on Kobo device
5. Start reading - progress syncs automatically

How Progress Sync Works:

• Kobo generates ContentId for each book
• ContentId mapped to Bookmann UUID in device_catalogs table
• When Kobo syncs progress, Bookmann uses canonical UUID
• Format conversion (KEPUB) doesn't break progress tracking

KOReader Setup

Option 1: OPDS Catalog (Recommended)

Step 1: Download Configuration File

Same as Kobo setup above

Step 2: Configure KOReader for OPDS

1. Open KOReader settings
2. Enable "OPDS catalog" in network/synchronization section
3. Enter OPDS URL from `.bookmann.json`:
   http://192.168.1.100:8765/opds/devices/YOUR_DEVICE_ID/catalog
4. KOReader will automatically:
   - Connect to Bookmann catalog
   - Browse and download books wirelessly
   - Sync progress using SHA-256 matching
   - Create file aliases automatically

Step 3: Wireless Book Acquisition

1. Open KOReader file browser
2. Tap "+" button to add OPDS catalog
3. Browse Bookmann catalog
4. Download books directly
5. Start reading

Option 2: Sidecar File (Alternative - Enhanced Progress Sync)

For offline or simple setup

Step 1: Download Sidecar

Same as Kobo setup above

Step 2: Place Sidecar on KOReader

Place in KOReader's config directory

Step 3: Use Sidecar for Progress Sync

KOReader plugin reads .bookmann.json
→ Matches local files to Bookmann UUIDs via SHA-256
→ Syncs progress using canonical UUIDs
→ Works offline

Web & Mobile Setup

OPDS catalog automatically available at:
/opds/devices/:deviceId/catalog

Apps can:
- Browse entire library wirelessly
- Download books directly
- See collection metadata
- Sync progress via existing internal APIs

---

Security Considerations

Authentication Layers

Layer 1: Web & Mobile (Internal API)

Uses: JWT tokens
Issued by: POST /api/auth/login, /api/auth/refresh
Validated: On each request via middleware
Revoked by: POST /api/auth/logout
Stored in: refresh_tokens table (not devices table)

Layer 2: Sync APIs (Device Tokens)

Issued by: Device registration endpoint
Stored in: devices.auth_token field
Validated by: Device authentication middleware

Layer 3: OPDS (Device Tokens, Optional)

Issued by: /api/devices/:id/opds-register
Stored in: opds_tokens table
Scope: Device-specific access to catalog

Can be: Public (no authentication required)

Data Privacy

1. Progress & Annotations: Always associated with user_id in database
2. Collections: User-scoped - each user sees only their collections
3. File Aliases: Device-specific - never shared across users
4. Device Catalogs: Links stored per-device - no cross-user leakage
5. Sidecar Files: Contain only user's device token and book mappings

Access Control

OPDS Authorization Flow:

1. OPDS request includes device_id in URL path
2. Server validates:
   a. Device exists
   b. Device belongs to requesting user
   c. Book is in user's visible library
3. If validation passes: Serve OPDS feed

Public Catalog Option:

• Can be enabled in system_config
• Allows guest users to browse without device registration
• Still respects library visibility per user

---

Testing Strategy

Unit Tests

Coverage Areas:

1. Hash calculation accuracy (SHA-256, OPF extraction)
2. Matching algorithm priorities
3. Collection rule evaluation
4. OPDS XML serialization
5. Format conversion integrity

Integration Tests

Test Scenarios:

1. Cross-device book matching (same book, different paths)
2. Format conversion (EPUB → KEPUB) with hash integrity
3. Collection auto-assign (rules fire correctly)
4. Bidirectional progress sync (Kobo ↔ KOReader)
5. OPDS catalog generation and pagination
6. Sidecar file generation and validation

Manual Testing Checklist

Kobo Workflow:

• Download .bookmann.json from web UI
• Transfer to Kobo via USB
• Configure OPDS URL on Kobo
• Browse catalog wirelessly
• Download book
• Read 50% of book
• Verify progress syncs to Bookmann

KOReader Workflow:

• Download .bookmann.json from web UI
• Configure OPDS URL in KOReader
• Browse catalog wirelessly
• Download book
• Read 75% of book
• Verify progress syncs to Bookmann

Cross-Device Scenario:

• Add book to Bookmann (EPUB scanned)
• Download to Kobo via OPDS
• Sync progress (60%) from Kobo
• Open same book on KOReader (side-loaded)
• Read to 75% on KOReader
• Verify progress shows 75% (latest from either device)
• Verify sync sources tracked correctly

---

Glossary

• Bookmann UUID: Canonical identifier for a book in Bookmann system (from media_items.id). Always used for progress tracking, never SHA-256. SHA-256 is only for matching books across devices.
• ContentId: Device-generated identifier (e.g., Kobo's "kobo_abc"). Mapped to Bookmann UUID in device_catalogs table. Used for progress sync after OPDS downloads.
• SHA-256: Cryptographic hash of file contents. Used for content-based matching across devices. Critical for identifying same book on different devices.
• OPF UUID: Unique identifier from EPUB metadata <dc:identifier id="...">. High-confidence identifier format.
• OPF Identifier: Any identifier from EPUB OPF file (custom format). Medium-confidence identifier format.
• ISBN: International Standard Book Number (13 digits). Medium-confidence standard identifier.
• ASIN: Amazon Standard Identification Number (10 characters). Medium-confidence standard identifier.
• Collections: Device-neutral organizational groups in Bookmann (e.g., "Science Fiction", "Reading"). Books can be in collections without being on any device. Collections organize library, not track device inventory.
• Shelves: Device-specific organization (e.g., Kobo's terminology). Map Collections to device-specific "Shelves" only at device-level. Bookmann UI always uses "Collections" terminology.
• OPDS: Open Publication Distribution System. Industry standard for book catalogs. All e-reader platforms have OPDS clients. Kobo, KOReader, Aldiko, FBReader, Web browsers can use OPDS catalogs.
• Internal APIs: Bookmann's private REST/WebSocket endpoints for state management. Web and mobile apps use these directly. Used for two-way sync, collections, WebSocket real-time updates.
• Device File Alias: Mapping of device-specific file paths to Bookmann UUIDs. Enables cross-device matching when same book has different file paths.
• Hash Confidence: Scoring system (0.0-1.0) for automatic book matching reliability. Higher values = more reliable match.
• Dual Hash Storage: Storing both original EPUB hash (epub_sha256) and converted KEPUB hash (kepub_sha256). Preserves hash integrity when files are converted. OPDS responses include format-specific hash for sidecar matching.
• Format Conversion: Transcoding between book formats (EPUB → KEPUB). KEPUB adds Kobo-specific markup. Critical for Kobo optimization but shouldn't break progress tracking.
• Media Item Formats: Tracks all format versions with their hashes. Pre-convert EPUB to KEPUB during scan for optimal performance.
• System Config: Key-value store for system-wide settings (base_url, opds_base_url, api_base_url). Enables flexible deployment.
• OPDS Tokens: Per-device access tokens for OPDS catalog browsing. Optional - can also support user-scoped and admin tokens.
• Sidecar File: .bookmann.json - Unified configuration file for devices. Contains OPDS URLs, sync API endpoints, book mappings, collection mappings.
• Auto-Assign Rules: Configurable criteria for automatically adding books to collections. Fields: genre, series, author, language, publisher, copyright_year, tags. Operators: equals, contains, starts_with, ends_with, greater_than, less_than.
• Sync Direction: For device shelf mappings. 'bidirectional' (sync both ways), 'book_to_device' (send to device), 'device_to_book' (read from device), 'none' (no sync).
• View Settings: Per-device preferences for how collections are displayed (grid vs list, which collections are visible).
• Unlinked Book: Progress record without proper media_item_id or failed ContentId lookup. Needs manual user resolution.

---

Summary

This comprehensive implementation plan provides:

• 7 new database tables with proper indexing
• 50+ API endpoints across 6 layers (identification, collections, OPDS, sync, configuration)
• 10-week phased implementation with clear deliverables
• Complete device setup guides for Kobo, KOReader, Web, and Mobile
• Three-tier authentication model for security (JWT, device tokens, OPDS optional)
• Glossary of all terminology and concepts
• Testing strategies covering unit, integration, and manual validation

The plan is designed for systematic execution while maintaining architectural consistency and enabling human oversight throughout the development process. All decisions from our conversations have been incorporated, providing a complete roadmap for implementing Bookmann as a comprehensive cross-device ebook management system.