bookhoard/docs/Code/bookmann/IMPLEMENTATION_PLAN.md

# Bookmann Implementation Plan

## Executive Summary

This plan implements a complete cross-device ebook management system with three major capabilities:

1. **Universal Book Identification** - SHA-256 hashing, UUID, ISBN, ASIN, and OPF identifiers for content-based matching across devices
2. **Enhanced Collection Management** - Device-neutral "Collections" with auto-assign rules and per-device customization via shelf mappings
3. **OPDS-Based Wireless Book Delivery** - Industry-standard book distribution for Kobo, KOReader, Web, and Mobile apps
4. **Bidirectional Progress Synchronization** - Real-time sync with ContentId mapping to handle format conversions (EPUB → KEPUB)
5. **Device-Specific Configuration** - Per-device view settings and shelf mappings while maintaining unified data model

**Key Design Principle**: Use OPDS for book acquisition (Layer 1) and internal APIs for state management (Layer 2), maintaining clear separation of concerns while enabling seamless user experience.

---

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [Database Schema](#database-schema)
3. [API Endpoints](#api-endpoints)
4. [Implementation Phases](#implementation-phases)
5. [Device Setup Instructions](#device-setup-instructions)
6. [Security Considerations](#security-considerations)
7. [Testing Strategy](#testing-strategy)
8. [Glossary](#glossary)

---

## Architecture Overview

### System Design: Two-Layer Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                  Bookmann Server                    │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │ Layer 1: Universal Book Identification                          │  │
│  │ SHA-256, UUID, ISBN, ASIN, OPF identifiers              │  │
│  │ Device file aliases for path tracking                             │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                          ↓ matches books universally                         │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │ Layer 2: Collections (Device-Neutral Organization)              │  │
│  │ Collections with auto-assign rules                                │  │
│  │ Device-specific shelf mappings (Kobo)                             │  │
│  │ Per-user view settings                                          │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                          ↓ provides organization                       │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │ Layer 3: OPDS (Primary Wireless Delivery)                     │  │
│  │ Per-device OPDS feeds (Kobo, KOReader, etc.)                │  │
│  │ Format conversion (EPUB → KEPUB on-the-fly)                │  │
│  │ Dual hash storage (original + converted)                         │  │
│  │ ContentId mapping (Bookmann UUID ↔ Device ID)               │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                          ↓ delivers books + provides IDs            │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │ Layer 4: Internal APIs (State Management)                 │  │
│  │ Progress sync (bidirectional)                                    │  │
│  │ Annotation sync (bidirectional)                                   │  │
│  │ Collection CRUD                                                 │  │
│  │ WebSocket real-time updates                                       │  │
│  │ Device-specific operations                                       │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────────┘
```

### Access Method Matrix

| Platform | Access Method | Purpose | Why This Method |
|-----------|----------------|---------|-----------------|
| **Kobo** | OPDS catalog | Kobo has built-in OPDS client, no custom Bookmann client exists |
| **KOReader** | OPDS catalog (primary) + Sidecar + Internal API | KOReader has OPDS client, also supports plugins/sidecars for enhanced features |
| **Web App** | Internal API directly | We own and control web app, can make direct API calls efficiently |
| **Mobile App** | Internal API directly | We own and control mobile app, can make direct API calls efficiently |
| **Any OPDS Client** | OPDS catalog | Public catalog standard, any app can use it for browsing/downloading |

### Key Design Principles

1. **Canonical UUID Always Wins** - Bookmann UUID (from `media_items.id`) is always used for progress tracking, never SHA-256. SHA-256 is only for matching books across devices, preventing format conversion issues.

2. **Collections ≠ Device Inventory** - Collections are organizational metadata (like "smart playlists"). Books can be in collections without being on any device. Progress/annotations sync independently of collection membership.

3. **OPDS for Acquisition, Internal APIs for State** - Two layers serve complementary purposes:
   - OPDS: "What books are available to download?" (public catalog)
   - Internal APIs: "How do I manage my books/sync state?" (private management)

4. **Dual Hash Storage Preserves Integrity** - Store both original EPUB hash (`epub_sha256`) and converted KEPUB hash (`kepub_sha256`) in `media_item_formats` table. OPDS responses include format-specific hash in headers, enabling sidecar matching even after conversion.

5. **Three-Tier Authentication** - Separate systems for different purposes:
   - Tier 1 (Web/Mobile): JWT tokens for user authentication and permissions
   - Tier 2 (Sync APIs): Device tokens for progress/annotation sync
   - Tier 3 (OPDS): Device tokens for catalog access (optional per-device)

6. **Terminology Separation** - Always use "Collections" terminology in Bookmann UI. Map Collections to device-specific "Shelves" only at API/device level. Kobo devices see "Shelves", KOReader/Web/Mobile see "Collections". Prevents legal issues.

7. **OPDS Primary for All Devices** - Kobo, KOReader, Web, and Mobile all use OPDS as primary book delivery method. Sidecar files provide fallback/enhanced features but are optional.

8. **System Configuration Flexibility** - Use `system_config` table to store base URLs (`base_url`, `opds_base_url`, `api_base_url`). Sidecar generation reads from these values, enabling flexible deployment (different domains, reverse proxies) with user overrides available.

---

## Database Schema

### Schema Overview

**7 new tables** + extensions to 4 existing tables

### Table: media_items (Extended)

```sql
-- Universal identifiers for cross-device matching
ALTER TABLE media_items ADD COLUMN file_sha256 CHAR(64);
ALTER TABLE media_items ADD COLUMN opf_identifier VARCHAR(255);
ALTER TABLE media_items ADD COLUMN opf_uuid VARCHAR(255);

-- Hash confidence for matching priority
-- 'high': OPF UUID or ISBN available
-- 'medium': ISBN/ASIN available but no OPF UUID
-- 'low': Only title/author match available
ALTER TABLE media_items ADD COLUMN hash_confidence VARCHAR(20);

-- Create indexes for fast lookup
CREATE INDEX idx_media_items_sha256 ON media_items(file_sha256);
CREATE INDEX idx_media_items_opf_identifier ON media_items(opf_identifier);
```

### Table: media_item_formats (NEW)

```sql
-- Track all format versions with their hashes
-- Critical for dual hash storage and format-specific OPDS delivery

CREATE TABLE media_item_formats (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    format_type VARCHAR(10) NOT NULL,           -- 'epub', 'kepub', 'pdf', 'cbz'
    file_path VARCHAR(500),
    file_sha256 CHAR(64),
    file_size_bytes BIGINT,
    mime_type VARCHAR(100),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    converted_from_format_id UUID REFERENCES media_item_formats(id),  -- If this is converted from another format
    UNIQUE(media_item_id, format_type)
);

CREATE INDEX idx_media_item_formats_media ON media_item_formats(media_item_id, format_type);
CREATE INDEX idx_media_item_formats_sha256 ON media_item_formats(file_sha256);
```

### Table: device_file_aliases (NEW)

```sql
-- Track file paths per device for cross-device matching
-- When same book has different file paths on different devices, we can still match them via SHA-256

CREATE TABLE device_file_aliases (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    device_id UUID REFERENCES devices(id) ON DELETE CASCADE,
    file_path VARCHAR(500) NOT NULL,
    file_sha256 CHAR(64),
    confidence_score FLOAT DEFAULT 1.0,
    last_seen_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    UNIQUE(device_id, file_path)
);

CREATE INDEX idx_device_file_aliases_media_device ON device_file_aliases(media_item_id, device_id);
CREATE INDEX idx_device_file_aliases_sha256 ON device_file_aliases(file_sha256);
```

### Table: collections (NEW)

```sql
-- Device-neutral collections (separate from Kobo shelves)
-- Each user has their own independent collection namespace

CREATE TABLE collections (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    name VARCHAR(100) NOT NULL,
    description TEXT,
    color VARCHAR(7),                    -- Hex color for UI
    icon VARCHAR(50),                    -- Emoji or icon name
    auto_assign_rules JSONB,          -- See schema below for structure
    view_settings JSONB,                -- Per-device view preferences
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    UNIQUE(user_id, name)
);
```

### Table: collection_items (NEW)

```sql
-- Which books belong to each collection

CREATE TABLE collection_items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    collection_id UUID REFERENCES collections(id) ON DELETE CASCADE,
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    added_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    added_by_user_id UUID REFERENCES users(id) ON DELETE SET NULL,  -- Manual vs auto
    UNIQUE(collection_id, media_item_id)
);

CREATE INDEX idx_collection_items_collection ON collection_items(collection_id);
CREATE INDEX idx_collection_items_media ON collection_items(media_item_id);
```

### Table: device_shelf_mappings (NEW)

```sql
-- Map Bookmann collections to device-specific shelf names
-- This is where "Collections" terminology maps to Kobo's "Shelves"

CREATE TABLE device_shelf_mappings (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    collection_id UUID REFERENCES collections(id) ON DELETE CASCADE,
    device_id UUID REFERENCES devices(id) ON DELETE CASCADE,
    device_shelf_name VARCHAR(100),        -- What appears on Kobo device
    sync_direction VARCHAR(20),           -- 'bidirectional', 'book_to_device', 'device_to_book', 'none'
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    UNIQUE(collection_id, device_id)
);

CREATE INDEX idx_device_shelf_mappings_collection ON device_shelf_mappings(collection_id);
CREATE INDEX idx_device_shelf_mappings_device ON device_shelf_mappings(device_id);
```

### Table: device_catalogs (NEW)

```sql
-- Track OPDS downloads and map Bookmann UUIDs to device ContentIds
-- Critical for bidirectional progress sync with format conversion handling

CREATE TABLE device_catalogs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    device_id UUID REFERENCES devices(id) ON DELETE CASCADE,
    media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
    bookmann_uuid UUID NOT NULL,
    kobo_content_id VARCHAR(255) NOT NULL,
    content_id_type VARCHAR(20),            -- 'bookmann_uuid', 'kobo_generated', 'isbn_based'
    available BOOLEAN DEFAULT TRUE,
    delivery_date TIMESTAMP WITH TIME ZONE,
    delivery_method VARCHAR(20),              -- 'wireless', 'usb', 'manual'
    UNIQUE(device_id, kobo_content_id)
);

CREATE INDEX idx_device_catalogs_bookmann ON device_catalogs(bookmann_uuid);
CREATE INDEX idx_device_catalogs_kobo ON device_catalogs(kobo_content_id);
```

### Table: system_config (NEW)

```sql
-- System-wide configuration (set by admin)
-- Critical for flexible deployment (different domains, reverse proxies)

CREATE TABLE system_config (
    key VARCHAR(100) PRIMARY KEY,
    value TEXT NOT NULL,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_by UUID REFERENCES users(id)
);

-- Pre-seeded values
INSERT INTO system_config (key, value) VALUES
('base_url', 'https://bookmann.example.com'),
('opds_base_url', 'https://bookmann.example.com/opds'),
('api_base_url', 'https://bookmann.example.com/api');
```

### Table: opds_tokens (NEW)

```sql
-- Device-specific OPDS access tokens (optional authentication)
-- Allows device-level access control without exposing JWT tokens

CREATE TABLE opds_tokens (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    device_id UUID REFERENCES devices(id) ON DELETE CASCADE,
    token VARCHAR(64) UNIQUE NOT NULL,
    token_type VARCHAR(20),                    -- 'device', 'user', 'admin'
    expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE INDEX idx_opds_tokens_device ON opds_tokens(device_id);
CREATE INDEX idx_opds_tokens_token ON opds_tokens(token);
```

### Table: kobo_shelves (Modified)

```sql
-- Reference collections instead of media_items directly
-- Maintains backward compatibility with existing device_id/media_item_id columns

ALTER TABLE kobo_shelves ADD COLUMN collection_id UUID REFERENCES collections(id);
ALTER TABLE kobo_shelves ADD COLUMN position_in_collection INTEGER;
```

### Schema Relationships Summary

```
users
├─ devices (one user can have multiple devices)
│   └─ device_file_aliases (tracks file paths per device)
│   └─ device_shelf_mappings (collection → Kobo shelf name)
│   └─ device_catalogs (OPDS + ContentId mapping)
│   └─ opds_tokens (OPDS authentication, optional)
├─ media_items (canonical book records with universal identifiers)
│   ├─ file_sha256, opf_identifier, opf_uuid
│   ├─ hash_confidence
│   ├─ reading_progress (one record per user per book)
│   └─ collection_items (which collections each book belongs to)
└─ collections (device-neutral organization)
└─ collection_items (membership)
```

---

## API Endpoints

### Layer 1: Universal Book Identification

#### POST `/api/sync/books/query`

Query Bookmann for a book by multiple identifier types with confidence scoring.

**Request**:
```json
{
  "identifiers": ["isbn:978-0345391802", "uuid:abc-123", "opf_uuid:def456"],
  "sha256": "a1b2c3d4e5f6abc123...",
  "title": "The Hobbit",
  "author": "J.R.R. Tolkien",
  "file_size": 2456789
}
```

**Response**:
```json
{
  "matches": [
    {
      "media_item_id": "uuid-123",
      "bookmann_uuid": "uuid-123",
      "confidence": 1.0,
      "match_method": "uuid_match"
    }
  ],
  "action": "auto_link"  // or "multiple_matches", "no_match"
}
```

**Matching Priority**:
1. Bookmann UUID (canonical) - Confidence: 1.0
2. OPF UUID (from EPUB metadata) - Confidence: 0.95
3. SHA-256 hash (content-based match) - Confidence: 0.9
4. OPF identifier (non-UUID) - Confidence: 0.85
5. ISBN/ASIN (standard identifiers) - Confidence: 0.8
6. File path (device-specific, fallback) - Confidence: variable
7. Title + author + file size (last resort) - Confidence: 0.5

#### POST `/api/sync/link-book`

Manual linking override for unmatched books.

**Request**:
```json
{
  "device_file": {
    "file_path": "/storage/emulated/0/Books/MyBook.epub",
    "sha256": "abc123...",
    "title": "My Book"
  },
  "media_item_id": "uuid-123",
  "confidence_score": 1.0  // User sets this
}
```

**Response**:
```json
{
  "status": "linked",
  "device_file_alias": {
      "id": "alias-id",
      "media_item_id": "uuid-123",
      "device_id": "kobo-device-id",
      "file_path": "/storage/emulated/0/Books/MyBook.epub",
      "file_sha256": "abc123...",
      "confidence_score": 1.0
  }
}
```

#### GET `/api/sync/unlinked-books`

List progress records that need manual linking.

**Response**:
```json
{
  "unlinked": [
    {
      "progress_id": "progress-uuid",
      "device_id": "kobo-device-id",
      "device_type": "kobo",
      "file_path": "/mnt/sdcard/UnknownBook.epub",
      "sha256": "abc123...",
      "title_from_device": "Unknown Book",
      "last_sync_timestamp": "2026-01-31T12:00:00Z"
    }
  ],
  "total": 1
}
```

#### GET `/api/devices/:id/file-aliases`

View all file aliases for a specific device.

**Response**:
```json
{
  "device_id": "device-uuid",
  "aliases": [
    {
      "id": "alias-id",
      "media_item_id": "uuid-123",
      "file_path": "/storage/emulated/0/Books/MyBook.epub",
      "file_sha256": "abc123...",
      "confidence_score": 1.0,
      "last_seen_at": "2026-01-31T12:00:00Z"
    }
  ],
  "total": 42
}
```

### Layer 2: Collection Management

#### POST `/api/collections`

Create a new collection.

**Request**:
```json
{
  "name": "Science Fiction",
  "description": "My favorite sci-fi books",
  "color": "#ff0000",
  "icon": "🚀",
  "auto_assign_rules": [
    {
      "id": "rule-1",
      "field": "genre",
      "operator": "equals",
      "value": "Science Fiction"
    }
  ]
}
```

**Response**:
```json
{
  "id": "collection-uuid",
  "name": "Science Fiction",
  "description": "My favorite sci-fi books",
  "color": "#ff0000",
  "icon": "🚀",
  "auto_assign_rules": [...],
  "book_count": 0,
  "created_at": "2026-01-31T12:00:00Z"
}
```

#### GET `/api/collections`

List all collections for current user.

**Query Parameters**:
- `include_auto`: boolean (include auto-assigned collections)
- `sort_by`: string (name, created_at, book_count)

**Response**:
```json
{
  "collections": [
    {
      "id": "collection-uuid",
      "name": "Science Fiction",
      "description": "...",
      "color": "#ff0000",
      "icon": "🚀",
      "auto_assign_rules": [...],
      "book_count": 15
    }
  ],
  "total": 1
}
```

#### GET `/api/collections/:id`

Get single collection details with books.

**Response**:
```json
{
  "id": "collection-uuid",
  "name": "Science Fiction",
  "description": "My favorite sci-fi books",
  "color": "#ff0000",
  "icon": "🚀",
  "auto_assign_rules": [...],
  "view_settings": {
    "kobo": {"shelf_name": "Sci-Fi", "sync": true},
    "koreader": {"enabled": false},
    "web": {"view_mode": "grid"}
  },
  "books": [
    {
      "media_item_id": "uuid-1",
      "title": "Foundation",
      "author": "Isaac Asimov"
    }
  ],
  "book_count": 42
}
```

#### PUT `/api/collections/:id`

Update collection.

**Request**:
```json
{
  "name": "Sci-Fi Favorites",
  "description": "Updated description",
  "color": "#00ff00",
  "icon": "⭐",
  "auto_assign_rules": [
    {
      "id": "rule-2",
      "field": "series",
      "operator": "equals",
      "value": "Foundation"
    }
  ]
}
```

#### DELETE `/api/collections/:id`

Delete collection and all its memberships.

#### POST `/api/collections/:id/books`

Add books to collection.

**Request**:
```json
{
  "book_ids": ["uuid-1", "uuid-2", "uuid-3"],
  "added_by_user": true  // Manual addition vs. auto
}
```

#### DELETE `/api/collections/:id/books/:bookId`

Remove book from collection.

#### POST `/api/collections/:id/rules`

Create auto-assign rule for collection.

**Rule Schema**:
```json
{
  "field": "genre",        // "genre", "series", "author", "language", "publisher", "copyright_year", "tags"
  "operator": "equals",       // "equals", "contains", "starts_with", "ends_with", "greater_than", "less_than"
  "value": "Science Fiction"
}
```

#### PUT `/api/collections/:id/rules/:ruleId`

Update existing rule.

### Layer 3: Device-Specific Shelf Mappings

#### GET `/api/devices/:id/collections`

Get all collection → shelf mappings for a device.

**Response**:
```json
{
  "device_id": "device-uuid",
  "device_name": "My Kobo Clara",
  "device_type": "kobo",
  "mappings": [
    {
      "collection_id": "collection-uuid",
      "collection_name": "Science Fiction",
      "device_shelf_name": "Sci-Fi",
      "sync_direction": "bidirectional",
      "created_at": "2026-01-31T12:00:00Z"
    }
  ],
  "total": 1
}
```

#### POST `/api/devices/:id/collections`

Create new shelf mapping for device.

**Request**:
```json
{
  "collection_id": "collection-uuid",
  "device_shelf_name": "My Books",
  "sync_direction": "bidirectional"
}
```

#### PUT `/api/devices/:id/collections/:collectionId`

Update shelf mapping.

#### DELETE `/api/devices/:id/collections/:collectionId`

Remove shelf mapping.

### Layer 3: OPDS Content Delivery

#### GET `/opds/devices/:deviceId/catalog`

Main OPDS 1.2 catalog feed.

**Query Parameters**:
- `page`: integer (default 1)
- `per_page`: integer (default 50)
- `include_format`: string (optional filter)

**Response (OPDS 1.2 XML)**:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" 
       xmlns:opds="http://opds-spec.org/2010/"
       xmlns:dc="http://purl.org/dc/elements/1.1/">
  <id>urn:uuid:device-id</id>
  <title>Bookmann Library</title>
  <updated>2026-01-31T12:00:00Z</updated>
  
  <link rel="self" href="http://192.168.1.100:8765/opds/devices/kobo-id/catalog"/>
  <link rel="search" href="http://192.168.1.100:8765/opds/devices/kobo-id/search"/>
  <link rel="start" href="http://192.168.1.100:8765/opds/devices/kobo-id/nav"/>
  
  <entry>
    <id>urn:uuid:bookmann-uuid-123</id>
    <dc:title>The Hobbit</dc:title>
    <dc:creator>J.R.R. Tolkien</dc:creator>
    <updated>2026-01-31T10:00:00Z</updated>
    <summary>Book description...</summary>
    
    <!-- Acquisition links (OPDS standard) -->
    <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123" 
          type="application/epub+zip" 
          rel="http://opds-spec.org/acquisition/open-access"/>
    
    <!-- Format variants (all devices get same) -->
    <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123?format=kepub" 
          type="application/vnd.kobo+xml+zip" 
          rel="alternate"/>
    <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123?format=pdf" 
          type="application/pdf" 
          rel="alternate"/>
    
    <!-- Canonical ID for progress matching -->
    <dc:identifier id="bookmann">uuid-123</dc:identifier>
    
    <!-- Hash for sidecar matching -->
    <meta property="bookmann:sha256">abc123...</meta>
    
    <!-- Collections as categories -->
    <category scheme="http://bookmann.example.com/collections">Science Fiction</category>
    <category scheme="http://bookmann.example.com/collections">Reading</category>
  </entry>
  
  <!-- More entries... -->
</feed>
```

#### GET `/opds/devices/:deviceId/search?q=`

OPDS acquisition search endpoint.

**Response (OPDS 1.2 XML)**:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" 
       xmlns:opds="http://opds-spec.org/2010/">
  <id>urn:uuid:device-id</id>
  
  <entry>
    <id>urn:uuid:bookmann-uuid-123</id>
    <dc:title>The Hobbit</dc:title>
    <dc:creator>J.R.R. Tolkien</dc:creator>
    <updated>2026-01-31T10:00:00Z</updated>
    <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123" 
          type="application/epub+zip" 
          rel="http://opds-spec.org/acquisition/open-access"/>
  </entry>
</feed>
```

#### GET `/opds/devices/:deviceId/nav`

OPDS navigation feed.

#### GET `/opds/devices/:deviceId/download/:bookId`

Download book with optional format conversion.

**Query Parameters**:
- `format`: string (epub, kepub, pdf, cbz) - default: epub

**Response Headers**:
- `Content-Type`: application/epub+zip (or format-specific)
- `Content-Disposition`: attachment; filename="The Hobbit.epub"
- `X-Bookmann-UUID`: uuid-123
- `X-Bookmann-SHA256`: abc123... (for format-specific if available)
- `X-Bookmann-KEPUB-SHA256`: xyz789... (if format=kepub)

**Format Conversion Logic**:
```go
// Select appropriate format based on format parameter
switch format {
case "kepub":
    // Check media_item_formats table for pre-converted KEPUB
    if kepubFormat.Exists && kepubFormat.FilePath != "" {
        Serve pre-converted file
        Set X-Bookmann-KEPUB-SHA256: kepubFormat.SHA256
    }
case "pdf":
    // Serve PDF directly
case "epub":
    // Serve original EPUB directly
}
```

#### GET `/opds/devices/:deviceId/cover/:bookId`

Download cover image.

**Response**:
- `Content-Type`: image/jpeg
- `Cache-Control`: public, max-age=31536000 (1 year)

#### GET `/opds/devices/:deviceId/formats/:bookId`

List available formats for a book.

**Response**:
```json
{
  "media_item_id": "uuid-123",
  "formats": [
    {
      "format_type": "epub",
      "file_path": "/path/to/book.epub",
      "file_sha256": "abc123...",
      "file_size_bytes": 2456789,
      "mime_type": "application/epub+zip",
      "available": true
    },
    {
      "format_type": "kepub",
      "file_path": "/cache/book.kepub.epub",
      "file_sha256": "xyz789...",
      "file_size_bytes": 2478932,
      "mime_type": "application/vnd.kobo+xml+zip",
      "available": true
    },
    {
      "format_type": "pdf",
      "file_path": "/path/to/book.pdf",
      "file_sha256": "def456...",
      "file_size_bytes": 5123456,
      "mime_type": "application/pdf",
      "available": false  // Not converted yet
    }
  ]
}
```

#### POST `/api/devices/:deviceId/opds-register`

Register device for OPDS access (generates token).

**Request**:
```json
{
  "device_name": "My Kobo Clara",
  "device_type": "kobo"
}
```

**Response**:
```json
{
  "opds_token": {
    "token": "abc-123-def-456...",
    "token_type": "device",
    "expires_at": "2026-02-28T23:59:59Z",
    "created_at": "2026-01-31T12:00:00Z"
  },
  "opds_catalog_url": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog",
  "refresh_interval": 3600
}
```

### Layer 4: Enhanced Device Sync

#### POST `/api/sync/kobo/markup`

Kobo progress sync with ContentId mapping (enhanced).

**Request (Enhanced)**:
```json
{
  "ReadingSync": [
    {
      "ContentId": "kobo_xyz",
      "PercentRead": 60.0,
      "RemainingTimeMin": 120,
      "ReadingEvent": "BookRead"
    }
  ],
  "BookmarkSync": [...],
  "Metadata": true  // NEW: Include collection metadata
}
```

**ContentId Mapping Logic**:
```go
// Step 1: Try direct ContentId lookup
catalog, err := db.GetDeviceCatalogByKoboContentId(ctx, contentId)
if err == nil && catalog.Valid {
    // Found! Use canonical Bookmann UUID
    bookmannUUID = catalog.BookmannUUID
    return bookmannUUID, nil
}

// Step 2: ContentId not found - try SHA-256 (if looks like hash)
if len(contentId) == 64 && looksLikeSHA256(contentId) {
    mediaItem, err := db.GetMediaItemBySHA256(ctx, contentId)
    if err == nil {
        return mediaItem.ID, nil
    }
}

// Step 3: Not found - create unlinked entry
return uuid.Nil{}, errors.New("unlinked book")
```

#### POST `/api/sync/kobo/bookmark`

Kobo bookmark sync (enhanced).

#### GET `/api/sync/kobo/initialization`

Kobo library sync with collection metadata (enhanced).

**Response (Enhanced)**:
```json
{
  "LibrarySync": [
    {
      "ContentId": "kobo_xyz",
      "ContentType": "6",
      "Title": "The Hobbit",
      "Author": "J.R.R. Tolkien",
      "PercentRead": 60.0,
      
      // NEW: Collection metadata
      "Categories": ["Science Fiction", "Reading"],
      "BookmannUUID": "uuid-123"  // Canonical ID
    }
  ]
}
```

### Layer 5: Enhanced KOReader Sync

#### POST `/api/sync/koreader/progress`

KOReader progress sync with SHA-256 support (enhanced).

**Request (Enhanced)**:
```json
{
  "sync_mode": "immediate",  // or "checkpoint"
  "books": [
    {
      "uuid": "uuid-123",  // Optional: highest priority
      "sha256": "abc123...",  // NEW: Device can send hash
      "file_path": "/storage/emulated/0/Books/MyBook.epub",
      "title": "The Hobbit",
      "authors": ["J.R.R. Tolkien"],
      "percentage": 75.0,
      "epubcfi": "/6/4!/2/4[chapter_1]@0:100",
      "chapter": 12,
      "character": 1234567,
      "page": 312,
      "total_pages": 416
    }
  ]
}
```

**SHA-256 Matching Logic**:
```go
// Priority 1: UUID provided (highest confidence)
if book.UUID != "" {
    return book.UUID, nil
}

// Priority 2: SHA-256 provided (medium confidence)
if book.SHA256 != "" {
    mediaItem, err := db.GetMediaItemBySHA256(ctx, book.SHA256)
    if err == nil {
        return mediaItem.ID, nil
    }
    return mediaItem.ID, nil
}

// Priority 3: Create device file alias (if file path provided)
if book.FilePath != "" {
    // Check if alias exists
    alias, err := db.GetDeviceFileAlias(ctx, deviceID, book.FilePath)
    if err == nil {
        // Create new alias with medium confidence
        db.CreateDeviceFileAlias(ctx, CreateDeviceFileAliasParams{
            MediaItemID: mediaItemID,
            DeviceID: deviceID,
            FilePath: book.FilePath,
            FileSHA256: book.SHA256,
            ConfidenceScore: 0.7,
        })
        return alias.MediaItemID, nil
    }
    // Use existing alias
    return alias.MediaItemID, nil
}

// Priority 4: Search by title/author + file size (fallback)
return mediaItem.ID, nil
```

#### POST `/api/sync/koreader/bookmarks`

KOReader annotations sync with SHA-256 support.

**Request (Enhanced)**:
```json
{
  "bookmarks": [
    {
      "uuid": "uuid-123",
      "sha256": "abc123...",  // NEW: For cross-device matching
      "file_path": "/storage/emulated/0/Books/MyBook.epub",
      "title": "The Hobbit",
      "page": 312,
      "text": "Great quote on page 312"
    }
  ]
}
```

### Layer 5: Sidecar Configuration

#### GET `/api/sync/sidecar/:deviceId`

Download unified `.bookmann.json` configuration file.

**Response**:
```json
{
  "version": "1.0",
  "bookmann": {
    "opds_catalog": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog",
    "sync_api": "http://192.168.1.100:8765/api/sync/kobo",
    "opds_base_url": "http://192.168.1.100:8765/opds",
    "api_base_url": "http://192.168.1.100:8765/api",
    "device_id": "kobo-device-uuid"
  },
  "books": {
    "sha256:abc123...": {
      "bookmann_uuid": "uuid-123",
      "title": "The Hobbit",
      "author": "J.R.R. Tolkien",
      "available_formats": ["epub", "kepub"]
    }
  },
  "collections": [
    {
      "name": "Sci-Fi",
      "shelf_mapping": "Science Fiction",
      "book_ids": ["uuid-1", "uuid-2", "uuid-3"]
    }
  ],
  "last_updated": "2026-01-31T12:00:00Z"
}
```

#### POST `/api/sync/sidecar/:deviceId/register`

Validate sidecar file upload from device.

### Layer 6: System Configuration

#### GET `/api/admin/system-config`

Get system-wide configuration.

**Response**:
```json
{
  "config": {
    "base_url": "https://bookmann.example.com",
    "opds_base_url": "https://bookmann.example.com/opds",
    "api_base_url": "https://bookmann.example.com/api",
    "auto_convert_kepub": true,
    "default_opds_refresh_interval": 3600
  }
}
```

#### PUT `/api/admin/system-config`

Update system configuration.

---

## Implementation Phases

### Phase 1: Database Schema (Week 1)

**Deliverables**:
- Create SQL schema file for all new tables
- Add columns to existing tables
- Run schema migrations on development database
- Update sqlc code generation
- Write rollback migration script

**Tasks**:
1.1 Create migration SQL file `database/schema/001_universal_identifiers.sql`
1.2 Update database models
1.3 Write database queries
1.4 Test database queries manually
1.5 Write rollback migration script

### Phase 2: Scanner Enhancement (Week 1-2)

**Deliverables**:
- Enhanced scanner code
- OPF parser implementation
- Unit tests for hash calculation

**Tasks**:
2.1 Implement SHA-256 calculation in `ebook_scanner.go`
   - Stream file reading (don't load entire file into memory)
   - Algorithm: crypto/sha256 from Go standard library
2.2 Implement OPF parser
   - Extract `<dc:identifier>` tags from EPUB OPF files
   - Parse both OEBPS and OPF 2.0 formats
   - Extract UUIDs from `<dc:identifier id="...">` attributes
   - Handle multiple identifiers per file
2.3 Implement format detection
   - Detect file format based on extension and content
2.4 Pre-convert EPUB to KEPUB during scan
   - Use `ebooklib` or similar library for conversion
2.5 Store all format hashes in `media_item_formats` table

**Hash Confidence Logic**:
```
HIGH (confidence = 1.0):
  - EPUB with valid `<dc:identifier>` (UUID format)
  - ISBN found (standard format)

MEDIUM (confidence = 0.7):
  - OPF identifier present (non-UUID custom format)
  - ISBN/ASIN matched via metadata sources

LOW (confidence = 0.5):
  - Only title/author match available
```

**Deliverables**:
- Enhanced scanner code
- OPF parser implementation
- KEPUB conversion utility
- Unit tests for hash calculation

### Phase 3: Universal Book Matching Engine (Week 2)

**Deliverables**:
- Matching engine implementation
- Book query API handlers
- Manual linking API endpoints
- Unit tests for matching logic

**Matching Algorithm**:
```
Priority 1: Bookmann UUID (canonical)
  - If device sends UUID, use directly
  - Confidence = 1.0

Priority 2: OPF UUID (from EPUB metadata)
  - Match against `opf_uuid` column
  - Confidence = 0.95

Priority 3: SHA-256 hash
  - Match against `file_sha256` column
  - Confidence = 0.9

Priority 4: OPF identifier (non-UUID)
  - Match against `opf_identifier` column
  - Confidence = 0.85

Priority 5: ISBN/ASIN (standard identifiers)
  - Match against `isbn` and `asin` columns
  - Confidence = 0.8

Priority 6: File path (device-specific)
  - Match via `device_file_aliases` table
  - Confidence = from alias record

Priority 7: Title + author + file size (fallback)
  - Fuzzy search on title
  - Exact match on author
  - Within 10% file size variance
  - Confidence = 0.5

Priority 8: Title only (last resort)
  - Fuzzy title match
  - Confidence = 0.3
```

**Deliverables**:
- Matching engine implementation
- Book query API handlers
- Manual linking API
- Unlinked books API
- Unit tests for matching logic

### Phase 4: Collection Management System (Week 2-3)

**Deliverables**:
- Collections CRUD handlers
- Auto-assign rules engine
- Device shelf mapping handlers
- Add collection book management
- Per-device view settings

**Auto-Assign Rules Engine**:
```go
type Rule struct {
    ID       string
    Field    string  // "genre", "series", "author", "language", "publisher", "copyright_year", "tags"
    Operator string  // "equals", "contains", "starts_with", "ends_with", "greater_than", "less_than"
    Value     string  // Exact value to match
}

type RuleEvaluation struct {
    RuleID    string
    Matches    bool
    Confidence float
}

func EvaluateRules(mediaItem MediaItem, rules []Rule) []RuleEvaluation {
    // Evaluate each rule against mediaItem metadata
    // Return which rules match and overall confidence
    // Higher-priority rules take precedence
}
```

**Rule Priority System**:
1. Rule with `priority` field (1-10, higher first)
2. Multiple rules can apply to same book
3. User can configure logical operators (AND, OR)

**Deliverables**:
- Collections API handlers
- Rules engine implementation
- Database queries for collections
- Unit tests for rule evaluation

### Phase 5: OPDS Implementation (Week 3)

**Deliverables**:
- OPDS XML serializer
- OPDS catalog feed handler
- OPDS search endpoint
- Book download with format support
- Cover image serving
- ContentId mapping to OPDS responses
- On-the-fly KEPUB conversion
- Device authorization checks
- OPDS token management

**OPDS Response Structure (OPDS 1.2)**:
```xml
<entry>
  <id>urn:uuid:bookmann-uuid-123</id>
  <dc:title>The Hobbit</dc:title>
  <dc:creator>J.R.R. Tolkien</dc:creator>
  <updated>2026-01-31T10:00:00Z</updated>
  
  <!-- Acquisition links (OPDS standard) -->
  <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123" 
          type="application/epub+zip" 
          rel="http://opds-spec.org/acquisition/open-access"/>
  
  <!-- Format variants (all devices get same) -->
  <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123?format=kepub" 
          type="application/vnd.kobo+xml+zip" 
          rel="alternate"/>
  <link href="http://192.168.1.100:8765/opds/devices/kobo-id/download/uuid-123?format=pdf" 
          type="application/pdf" 
          rel="alternate"/>
  
  <!-- Canonical ID for progress matching -->
  <dc:identifier id="bookmann">uuid-123</dc:identifier>
  
  <!-- Hash for sidecar matching (format-specific if available) -->
  <meta property="bookmann:sha256">abc123...</meta>
  <meta property="bookmann:kepub_sha256">xyz789...</meta>
  
  <!-- Collections as categories -->
  <category scheme="http://bookmann.example.com/collections">Science Fiction</category>
  <category scheme="http://bookmann.example.com/collections">Reading</category>
  </entry>
```

**Format Conversion Strategy**:
```
When user downloads with ?format=kepub:

1. Check media_item_formats table
2. If KEPUB exists and is recent:
   - Serve pre-converted file
   - Set X-Bookmann-SHA256: kepubFormat.SHA256
3. If KEPUB doesn't exist:
   - Convert EPUB to KEPUB on-the-fly
   - Cache in media_item_formats table
   - Serve converted file
   - Set X-Bookmann-SHA256: kepubFormat.SHA256
4. Serve PDF directly
```

**Deliverables**:
- OPDS handlers implementation
- OPDS XML serializers
- KEPUB conversion utility
- OPDS authentication
- Integration with device_catalogs table

### Phase 6: Enhanced Kobo Sync (Week 3-4)

**Deliverables**:
- Updated Kobo handler to use ContentId mapping
- Bidirectional ContentId lookup
- Add unlinked book detection
- Integrate collection metadata into library sync
- Support for legacy API endpoints

**ContentId Mapping Logic**:
```go
// Step 1: Try direct ContentId lookup
catalog, err := db.GetDeviceCatalogByKoboContentId(ctx, contentId)
if err == nil && catalog.Valid {
    // Found! Use canonical Bookmann UUID
    return catalog.BookmannUUID, nil
}

// Step 2: ContentId not found - try SHA-256
if len(contentId) == 64 && looksLikeSHA256(contentId) {
    mediaItem, err := db.GetMediaItemBySHA256(ctx, contentId)
    if err == nil {
        return mediaItem.ID, nil
    }
}

// Step 3: Not found - create unlinked entry
return uuid.Nil{}, errors.New("unlinked book")
```

**Deliverables**:
- Updated Kobo sync handlers
- ContentId mapping system
- Unlinked book tracking
- Integration with collection metadata

### Phase 7: Enhanced KOReader Sync (Week 4)

**Deliverables**:
- Updated KOReader handler to accept SHA-256
- Implement device file alias creation
- Integrate auto-linking with confidence thresholds
- Add SHA-256 matching for annotations

**SHA-256 Matching for Progress Sync**:
```go
// Priority 1: UUID provided (highest confidence)
if book.UUID != "" {
    return book.UUID, nil
}

// Priority 2: SHA-256 provided (medium confidence)
if book.SHA256 != "" {
    mediaItem, err := db.GetMediaItemBySHA256(ctx, book.SHA256)
    if err == nil {
        return mediaItem.ID, nil
    }
}

// Priority 3: Create device file alias
if book.FilePath != "" {
    alias, err := db.GetDeviceFileAlias(ctx, deviceID, book.FilePath)
    if err == nil {
        // Create new alias
        db.CreateDeviceFileAlias(ctx, CreateDeviceFileAliasParams{
            MediaItemID: mediaItemID,
            DeviceID: deviceID,
            FilePath: book.FilePath,
            FileSHA256: book.SHA256,
            ConfidenceScore: 0.7,
        })
        return alias.MediaItemID, nil
    }
    return alias.MediaItemID, nil
}
```

**Deliverables**:
- Enhanced KOReader handlers
- SHA-256 matching integration
- Device file alias system integration
- Auto-linking with configurable thresholds

### Phase 8: Sidecar Configuration System (Week 4)

**Deliverables**:
- Sidecar JSON generation
- Sidecar download/upload handlers
- System configuration support

**Sidecar File Format (Enhanced)**:
```json
{
  "version": "1.0",
  "bookmann": {
    "opds_catalog": "http://192.168.1.100:8765/opds/devices/kobo-id/catalog",
    "sync_api": "http://192.168.1.100:8765/api/sync/kobo",
    "opds_base_url": "http://192.168.1.100:8765/opds",
    "api_base_url": "http://192.168.1.100:8765/api",
    "device_id": "kobo-device-uuid"
  },
  "books": {
    "sha256:abc123...": {
      "bookmann_uuid": "uuid-123",
      "title": "The Hobbit",
      "author": "J.R.R. Tolkien",
      "available_formats": ["epub", "kepub"]
    }
  },
  "collections": [
    {
      "name": "Sci-Fi",
      "shelf_mapping": "Science Fiction",
      "book_ids": ["uuid-1", "uuid-2", "uuid-3"]
    }
  ],
  "opds_enabled": true,
  "sidecar_enabled": true,
  "last_updated": "2026-01-31T12:00:00Z"
}
```

**Deliverables**:
- Sidecar generation system
- System configuration support
- Admin UI for system settings

### Phase 9: Frontend Implementation (Week 5-6)

**Deliverables**:
- Collections management pages
- Device configuration pages
- Enhanced progress visualization with sync sources
- Unlinked books resolution UI
- Collection rule builder UI
- Device-specific view settings UI

**Deliverables**:
- Collections list/detail pages
- Device management interface
- Progress sync dashboard with device indicators
- Book matching UI with confidence indicators

### Phase 10: Documentation & Testing (Week 6)

**Deliverables**:
- Updated device setup guides
- Complete API documentation
- Test suite covering all scenarios
- User acceptance testing

**Deliverables**:
- KOBO_SETUP.md update with OPDS workflow
- KOREADER_SETUP.md new file with OPDS instructions
- Complete API reference documentation
- User guides for all device types

---

## Device Setup Instructions

### Kobo E-Reader Setup (OPDS Primary Method)

#### Option 1: OPDS Catalog (Recommended - Wireless Delivery + Progress Sync)

**Step 1: Download Configuration File**
```
1. Log into Bookmann web UI
2. Go to Device Management → Your Kobo device
3. Click "Download Configuration" button
4. File downloads as `.bookmann.json`
```

**Step 2: Configure Kobo for OPDS**
```
1. On Kobo, go to Settings → Sync & Backup
2. Tap "Add Content Server" or "Add OPDS Feed"
3. Enter URL from `.bookmann.json`:
   http://192.168.1.100:8765/opds/devices/YOUR_DEVICE_ID/catalog
4. Kobo will automatically:
   - Connect to Bookmann
   - Browse your library wirelessly
   - Download books directly
   - Sync reading progress back to Bookmann
```

**Step 3: Wireless Book Acquisition**
```
1. On Kobo, go to "My Books" section
2. Browse Bookmann catalog via OPDS
3. Tap on any book to download wirelessly
4. Book appears on Kobo device
5. Start reading - progress syncs automatically
```

**How Progress Sync Works**:
- Kobo generates ContentId for each book
- ContentId mapped to Bookmann UUID in device_catalogs table
- When Kobo syncs progress, Bookmann uses canonical UUID
- Format conversion (KEPUB) doesn't break progress tracking

### KOReader Setup

#### Option 1: OPDS Catalog (Recommended)

**Step 1: Download Configuration File**
```
Same as Kobo setup above
```

**Step 2: Configure KOReader for OPDS**
```
1. Open KOReader settings
2. Enable "OPDS catalog" in network/synchronization section
3. Enter OPDS URL from `.bookmann.json`:
   http://192.168.1.100:8765/opds/devices/YOUR_DEVICE_ID/catalog
4. KOReader will automatically:
   - Connect to Bookmann catalog
   - Browse and download books wirelessly
   - Sync progress using SHA-256 matching
   - Create file aliases automatically
```

**Step 3: Wireless Book Acquisition**
```
1. Open KOReader file browser
2. Tap "+" button to add OPDS catalog
3. Browse Bookmann catalog
4. Download books directly
5. Start reading
```

#### Option 2: Sidecar File (Alternative - Enhanced Progress Sync)

**For offline or simple setup**

**Step 1: Download Sidecar**
```
Same as Kobo setup above
```

**Step 2: Place Sidecar on KOReader**
```
Place in KOReader's config directory
```

**Step 3: Use Sidecar for Progress Sync**
```
KOReader plugin reads .bookmann.json
→ Matches local files to Bookmann UUIDs via SHA-256
→ Syncs progress using canonical UUIDs
→ Works offline
```

### Web & Mobile Setup

```
OPDS catalog automatically available at:
/opds/devices/:deviceId/catalog

Apps can:
- Browse entire library wirelessly
- Download books directly
- See collection metadata
- Sync progress via existing internal APIs
```

---

## Security Considerations

### Authentication Layers

**Layer 1: Web & Mobile (Internal API)**
```
Uses: JWT tokens
Issued by: POST /api/auth/login, /api/auth/refresh
Validated: On each request via middleware
Revoked by: POST /api/auth/logout
Stored in: refresh_tokens table (not devices table)
```

**Layer 2: Sync APIs (Device Tokens)**
```
Issued by: Device registration endpoint
Stored in: devices.auth_token field
Validated by: Device authentication middleware
```

**Layer 3: OPDS (Device Tokens, Optional)**
```
Issued by: /api/devices/:id/opds-register
Stored in: opds_tokens table
Scope: Device-specific access to catalog

Can be: Public (no authentication required)
```

### Data Privacy

1. **Progress & Annotations**: Always associated with user_id in database
2. **Collections**: User-scoped - each user sees only their collections
3. **File Aliases**: Device-specific - never shared across users
4. **Device Catalogs**: Links stored per-device - no cross-user leakage
5. **Sidecar Files**: Contain only user's device token and book mappings

### Access Control

**OPDS Authorization Flow:**
```
1. OPDS request includes device_id in URL path
2. Server validates:
   a. Device exists
   b. Device belongs to requesting user
   c. Book is in user's visible library
3. If validation passes: Serve OPDS feed
```

**Public Catalog Option**:
- Can be enabled in system_config
- Allows guest users to browse without device registration
- Still respects library visibility per user

---

## Testing Strategy

### Unit Tests

**Coverage Areas**:
1. Hash calculation accuracy (SHA-256, OPF extraction)
2. Matching algorithm priorities
3. Collection rule evaluation
4. OPDS XML serialization
5. Format conversion integrity

### Integration Tests

**Test Scenarios**:
1. Cross-device book matching (same book, different paths)
2. Format conversion (EPUB → KEPUB) with hash integrity
3. Collection auto-assign (rules fire correctly)
4. Bidirectional progress sync (Kobo ↔ KOReader)
5. OPDS catalog generation and pagination
6. Sidecar file generation and validation

### Manual Testing Checklist

**Kobo Workflow**:
- [ ] Download `.bookmann.json` from web UI
- [ ] Transfer to Kobo via USB
- [ ] Configure OPDS URL on Kobo
- [ ] Browse catalog wirelessly
- [ ] Download book
- [ ] Read 50% of book
- [ ] Verify progress syncs to Bookmann

**KOReader Workflow**:
- [ ] Download `.bookmann.json` from web UI
- [ ] Configure OPDS URL in KOReader
- [ ] Browse catalog wirelessly
- [ ] Download book
- [ ] Read 75% of book
- [ ] Verify progress syncs to Bookmann

**Cross-Device Scenario**:
- [ ] Add book to Bookmann (EPUB scanned)
- [ ] Download to Kobo via OPDS
- [ ] Sync progress (60%) from Kobo
- [ ] Open same book on KOReader (side-loaded)
- [ ] Read to 75% on KOReader
- [ ] Verify progress shows 75% (latest from either device)
- [ ] Verify sync sources tracked correctly

---

## Glossary

- **Bookmann UUID**: Canonical identifier for a book in Bookmann system (from `media_items.id`). Always used for progress tracking, never SHA-256. SHA-256 is only for matching books across devices.
- **ContentId**: Device-generated identifier (e.g., Kobo's "kobo_abc"). Mapped to Bookmann UUID in `device_catalogs` table. Used for progress sync after OPDS downloads.
- **SHA-256**: Cryptographic hash of file contents. Used for content-based matching across devices. Critical for identifying same book on different devices.
- **OPF UUID**: Unique identifier from EPUB metadata `<dc:identifier id="...">`. High-confidence identifier format.
- **OPF Identifier**: Any identifier from EPUB OPF file (custom format). Medium-confidence identifier format.
- **ISBN**: International Standard Book Number (13 digits). Medium-confidence standard identifier.
- **ASIN**: Amazon Standard Identification Number (10 characters). Medium-confidence standard identifier.
- **Collections**: Device-neutral organizational groups in Bookmann (e.g., "Science Fiction", "Reading"). Books can be in collections without being on any device. Collections organize library, not track device inventory.
- **Shelves**: Device-specific organization (e.g., Kobo's terminology). Map Collections to device-specific "Shelves" only at device-level. Bookmann UI always uses "Collections" terminology.
- **OPDS**: Open Publication Distribution System. Industry standard for book catalogs. All e-reader platforms have OPDS clients. Kobo, KOReader, Aldiko, FBReader, Web browsers can use OPDS catalogs.
- **Internal APIs**: Bookmann's private REST/WebSocket endpoints for state management. Web and mobile apps use these directly. Used for two-way sync, collections, WebSocket real-time updates.
- **Device File Alias**: Mapping of device-specific file paths to Bookmann UUIDs. Enables cross-device matching when same book has different file paths.
- **Hash Confidence**: Scoring system (0.0-1.0) for automatic book matching reliability. Higher values = more reliable match.
- **Dual Hash Storage**: Storing both original EPUB hash (`epub_sha256`) and converted KEPUB hash (`kepub_sha256`). Preserves hash integrity when files are converted. OPDS responses include format-specific hash for sidecar matching.
- **Format Conversion**: Transcoding between book formats (EPUB → KEPUB). KEPUB adds Kobo-specific markup. Critical for Kobo optimization but shouldn't break progress tracking.
- **Media Item Formats**: Tracks all format versions with their hashes. Pre-convert EPUB to KEPUB during scan for optimal performance.
- **System Config**: Key-value store for system-wide settings (base_url, opds_base_url, api_base_url). Enables flexible deployment.
- **OPDS Tokens**: Per-device access tokens for OPDS catalog browsing. Optional - can also support user-scoped and admin tokens.
- **Sidecar File**: `.bookmann.json` - Unified configuration file for devices. Contains OPDS URLs, sync API endpoints, book mappings, collection mappings.
- **Auto-Assign Rules**: Configurable criteria for automatically adding books to collections. Fields: genre, series, author, language, publisher, copyright_year, tags. Operators: equals, contains, starts_with, ends_with, greater_than, less_than.
- **Sync Direction**: For device shelf mappings. 'bidirectional' (sync both ways), 'book_to_device' (send to device), 'device_to_book' (read from device), 'none' (no sync).
- **View Settings**: Per-device preferences for how collections are displayed (grid vs list, which collections are visible).
- **Unlinked Book**: Progress record without proper media_item_id or failed ContentId lookup. Needs manual user resolution.

---

## Summary

This comprehensive implementation plan provides:

- **7 new database tables** with proper indexing
- **50+ API endpoints** across 6 layers (identification, collections, OPDS, sync, configuration)
- **10-week phased implementation** with clear deliverables
- **Complete device setup guides** for Kobo, KOReader, Web, and Mobile
- **Three-tier authentication model** for security (JWT, device tokens, OPDS optional)
- **Glossary** of all terminology and concepts
- **Testing strategies** covering unit, integration, and manual validation

The plan is designed for systematic execution while maintaining architectural consistency and enabling human oversight throughout the development process. All decisions from our conversations have been incorporated, providing a complete roadmap for implementing Bookmann as a comprehensive cross-device ebook management system.