Files

T

john-okeefe bb3c32c59f Phase 1 Week 2: Format detection and progress conversion engine

- Add internal/sync package with format detection
- FormatGroup types: reflowable, fixed_layout, comic_archive
- DetectFormatGroup() function based on mimetype and file extension
- MimeType mappings for common ebook formats
- Progress conversion engine with:
  - ConvertProgress() between format groups
  - Extract percentage from various progress formats
  - PageToPercentage / PercentageToPage helpers
  - CharacterToPercentage / PercentageToCharacter helpers
  - MergeProgress() with 'max progress wins' strategy
  - FormatProgressForDisplay() for UI rendering
- Add sqlc queries for format detection and progress updates
- BulkUpdateFormatGroups query for auto-format detection
- GetUniversalProgress query with all location references
- UpdateUniversalProgress query with device sync metadata
- ReadingHistory queries for session tracking

2026-01-30 16:07:00 -05:00

55 KiB

Raw Blame History

Bookmann Universal Cross-Platform Sync Implementation Guide

Executive Summary

This document provides a complete technical specification for implementing a universal cross-platform reading progress synchronization system that rivals the Amazon Kindle ecosystem. The system enables seamless reading progress, highlights, and notes synchronization across web interface, KOReader, Kobo devices, and future platforms - all self-hosted and open-source.

Vision: Users can read on any device, put it down, pick up any other device, and continue exactly where they left off with all annotations synchronized.

Core Philosophy: Bookmann serves as the "universal translator" between different reading platforms, understanding multiple location reference systems and converting between them seamlessly.

Current System Status
Problem Analysis
Technical Architecture
Database Schema Changes
API Endpoints Specification
Format Grouping System
Progress Tracking by Format
Wireless Sync Protocols
Security Architecture
Conflict Resolution System
Backup & Recovery Strategy
Real-time Synchronization
Device Onboarding Flow
Implementation Phases
Testing Requirements
Performance Considerations

Current System Status

Working Endpoints (As of Testing)

All core functionality is operational:

Authentication: Register, login, logout, refresh tokens ✅
Library management: Create, read, update, delete libraries ✅
Scanner: Background scanning, watch mode, status tracking ✅
Media items: CRUD operations, search, filter, sort ✅
Progress tracking: Basic page-based tracking ✅
Notes: Create, read, update, delete ✅
Highlights: Create, read, update, delete ✅
Ratings: Full rating system ✅

Known Issues

EPUB Progress Tracking: Current page-based system is unreliable for reflowable formats
No Cross-Device Sync: Progress doesn't sync between devices
Limited Platform Support: Only web interface, no device integration
Format Inconsistency: Different formats treated identically despite different capabilities

Problem Analysis

The EPUB Progress Problem

Current Implementation Flaw:

EPUB is reflowable - page count changes based on font size, screen size, zoom level
"Page 45 of 200" is meaningless across different devices or reading sessions
Notes at "page:45" location become unfindable with different display settings
Progress sharing between users or devices is unreliable

Real-World Example:

User reads on tablet (large font) → sees "page 45 of 250"
Same user reads on phone (small font) → sees "page 78 of 450"
Same book, same progress, but completely different page numbers

Cross-Platform Incompatibility

Kobo Devices:

Use percentage-based progress internally
Sync to proprietary Kobo cloud
Store annotations in SQLite databases
No open API for third-party sync

KOReader:

Open-source, supports Wi-Fi sync
Uses calibre-compatible protocols
Stores data in .sdr sidecar folders
Percentage + character offset tracking

Bookmann Current State:

Page-based tracking only
No device protocols implemented
No sync endpoints for external devices

Technical Architecture

System Overview

┌─────────────────────────────────────────────────────────────┐
│                    Bookmann Core System                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Web UI     │  │  Wireless    │  │   Progress   │      │
│  │   Interface  │  │   Sync API   │  │  Conversion  │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘
                            ▲
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
┌───────┴────────┐  ┌──────┴─────────┐  ┌─────┴────────┐
│   KOReader     │  │  Kobo Device   │  │  Web Mobile  │
│  .sdr folders  │  │  SQLite DB     │  │   App/API    │
└────────────────┘  └────────────────┘  └───────────────┘

Data Flow Architecture

Device Progress → Device Protocol → Bookmann Wireless API
                                            ↓
                                  Universal Progress JSON
                                            ↓
                                  Format Conversion Engine
                                            ↓
                                  Database Storage (Multi-format)
                                            ↓
                                  WebSocket Broadcast
                                            ↓
                          All Connected Clients (Web, Mobile, Other Devices)

Universal Progress Hub Concept

Bookmann doesn't just store progress - it maintains a multi-dimensional location reference system that can express the same reading position in multiple ways simultaneously:

{
  "book_id": "uuid-1234",
  "universal_progress": 0.45678,
  "format_group": "reflowable",
  "location_references": {
    "percentage": 0.45678,
    "epubcfi": "epubcfi(/6/4[chap1]!/4/2:15)",
    "character": 15432,
    "chapter": 3,
    "chapter_progress": 0.234,
    "viewport_y": 0.12
  },
  "device_specific": {
    "koreader": { "percent": 0.45678, "timestamp": "..." },
    "kobo": { "progress": 45, "timestamp": "..." },
    "web": { "display_page": 89, "total_pages": 200, "timestamp": "..." }
  }
}

Database Schema Changes

Core Schema Additions

-- ============================================
-- FORMAT DETECTION AND GROUPING
-- ============================================
ALTER TABLE media_items
  ADD COLUMN format_group VARCHAR(20) NOT NULL DEFAULT 'reflowable',
  ADD COLUMN format_mimetype VARCHAR(100),
  ADD COLUMN is_reflowable BOOLEAN DEFAULT TRUE,
  ADD COLUMN has_fixed_layout BOOLEAN DEFAULT FALSE,
  ADD COLUMN total_characters BIGINT,
  ADD COLUMN chapter_count INTEGER;

-- ============================================
-- UNIVERSAL PROGRESS TRACKING
-- ============================================
ALTER TABLE reading_progress
  ADD COLUMN percentage FLOAT CHECK (percentage >= 0 AND percentage <= 1),
  ADD COLUMN character_offset BIGINT,
  ADD COLUMN epubcfi TEXT,
  ADD COLUMN chapter INTEGER,
  ADD COLUMN chapter_progress FLOAT CHECK (chapter_progress >= 0 AND chapter_progress <= 1),
  ADD COLUMN viewport_x FLOAT DEFAULT 0,
  ADD COLUMN viewport_y FLOAT DEFAULT 0,
  ADD COLUMN zoom_level FLOAT DEFAULT 1.0,
  ADD COLUMN scroll_position_x FLOAT DEFAULT 0,
  ADD COLUMN scroll_position_y FLOAT DEFAULT 0,
  ADD COLUMN panel_number INTEGER,
  ADD COLUMN reading_mode VARCHAR(20);

-- ============================================
-- DEVICE SYNC METADATA
-- ============================================
ALTER TABLE reading_progress
  ADD COLUMN last_sync_device VARCHAR(50),
  ADD COLUMN last_sync_source VARCHAR(20),
  ADD COLUMN last_sync_timestamp TIMESTAMP,
  ADD COLUMN conflict_detected BOOLEAN DEFAULT FALSE,
  ADD COLUMN conflict_resolved BOOLEAN DEFAULT TRUE;

-- ============================================
-- NOTES LOCATION ENHANCEMENTS
-- ============================================
ALTER TABLE media_notes
  ADD COLUMN percentage_location FLOAT,
  ADD COLUMN character_start INTEGER,
  ADD COLUMN character_end INTEGER,
  ADD COLUMN epubcfi_location TEXT,
  ADD COLUMN chapter_reference INTEGER,
  ADD COLUMN paragraph_reference INTEGER,
  ADD COLUMN device_sync_data JSONB;

-- ============================================
-- HIGHLIGHTS LOCATION ENHANCEMENTS
-- ============================================
ALTER TABLE media_highlights
  ADD COLUMN percentage_start FLOAT,
  ADD COLUMN percentage_end FLOAT,
  ADD COLUMN character_start INTEGER,
  ADD COLUMN character_end INTEGER,
  ADD COLUMN epubcfi_start TEXT,
  ADD COLUMN epubcfi_end TEXT,
  ADD COLUMN chapter_reference INTEGER,
  ADD COLUMN paragraph_start INTEGER,
  ADD COLUMN paragraph_end INTEGER,
  ADD COLUMN panel_number INTEGER,
  ADD COLUMN device_sync_data JSONB;

-- ============================================
-- DEVICE REGISTRY
-- ============================================
CREATE TABLE devices (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  device_name VARCHAR(100) NOT NULL,
  device_type VARCHAR(20) NOT NULL, -- 'koreader', 'kobo', 'web', 'mobile'
  device_identifier VARCHAR(255) UNIQUE NOT NULL,
  auth_token VARCHAR(500) UNIQUE NOT NULL,
  last_sync TIMESTAMP,
  last_seen TIMESTAMP,
  sync_enabled BOOLEAN DEFAULT TRUE,
  auto_sync BOOLEAN DEFAULT TRUE,
  sync_frequency_minutes INTEGER DEFAULT 5,
  device_metadata JSONB,
  created_at TIMESTAMP DEFAULT NOW(),
  updated_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_devices_user_id ON devices(user_id);
CREATE INDEX idx_devices_device_type ON devices(device_type);
CREATE INDEX idx_devices_device_identifier ON devices(device_identifier);

-- ============================================
-- SYNC QUEUE FOR OFFLINE SUPPORT
-- ============================================
CREATE TABLE sync_queue (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  device_id UUID NOT NULL REFERENCES devices(id) ON DELETE CASCADE,
  media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
  sync_type VARCHAR(20) NOT NULL, -- 'progress', 'note', 'highlight', 'bookmark'
  sync_data JSONB NOT NULL,
  priority INTEGER DEFAULT 5,
  attempts INTEGER DEFAULT 0,
  max_attempts INTEGER DEFAULT 3,
  status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'processing', 'completed', 'failed'
  error_message TEXT,
  created_at TIMESTAMP DEFAULT NOW(),
  processed_at TIMESTAMP
);

CREATE INDEX idx_sync_queue_device_id ON sync_queue(device_id);
CREATE INDEX idx_sync_queue_status ON sync_queue(status);
CREATE INDEX idx_sync_queue_priority ON sync_queue(priority);

-- ============================================
-- CONFLICT RESOLUTION
-- ============================================
CREATE TABLE sync_conflicts (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  media_item_id UUID NOT NULL REFERENCES media_items(id) ON DELETE CASCADE,
  user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  conflict_type VARCHAR(20) NOT NULL, -- 'progress', 'note', 'highlight'
  conflict_data JSONB NOT NULL,
  resolution_status VARCHAR(20) DEFAULT 'unresolved', -- 'unresolved', 'auto_resolved', 'user_resolved'
  resolution_data JSONB,
  resolved_by UUID REFERENCES users(id),
  resolved_at TIMESTAMP,
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_sync_conflicts_media_item_id ON sync_conflicts(media_item_id);
CREATE INDEX idx_sync_conflicts_user_id ON sync_conflicts(user_id);
CREATE INDEX idx_sync_conflicts_status ON sync_conflicts(resolution_status);

-- ============================================
-- READING HISTORY
-- ============================================
CREATE TABLE reading_history (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  media_item_id UUID NOT NULL REFERENCES media_items(id) ON DELETE CASCADE,
  device_id UUID REFERENCES devices(id),
  progress_percentage FLOAT,
  reading_session_start TIMESTAMP,
  reading_session_end TIMESTAMP,
  pages_read INTEGER,
  time_spent_seconds INTEGER,
  device_metadata JSONB,
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_reading_history_user_id ON reading_history(user_id);
CREATE INDEX idx_reading_history_media_item_id ON reading_history(media_item_id);
CREATE INDEX idx_reading_history_created_at ON reading_history(created_at DESC);

Database Triggers for Automatic Timestamps

-- Update devices updated_at timestamp
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
  NEW.updated_at = NOW();
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER update_devices_updated_at
  BEFORE UPDATE ON devices
  FOR EACH ROW
  EXECUTE FUNCTION update_updated_at_column();

API Endpoints Specification

Authentication Endpoints

Device Registration Flow

POST /api/devices/register
Description: Register a new reading device

Request Body:
{
  "device_name": "My Kobo Clara",
  "device_type": "kobo|koreader|web|mobile",
  "device_identifier": "unique-device-hardware-id"
}

Response (201):
{
  "device_id": "uuid",
  "auth_token": "Bearer token for device",
  "setup_instructions": {
    "kobo": "Sync URL: https://bookmann.example.com/api/sync/kobo",
    "koreader": "Calibre URL: https://bookmann.example.com/api/sync/koreader"
  },
  "qr_code_url": "https://bookmann.example.com/devices/qr/uuid"
}

POST /api/devices/auth/web
Description: Authenticate device via web login (no API keys on device!)

Request Body:
{
  "device_identifier": "device-unique-id",
  "device_type": "kobo"
}

Response (200):
{
  "auth_pending": true,
  "auth_url": "https://bookmann.example.com/devices/auth/confirm/abc123",
  "expires_in": 300,
  "poll_interval": 3
}

[User visits auth URL on web, logs in, approves device]

Device polls:
POST /api/devices/auth/status

Response (200):
{
  "auth_complete": true,
  "auth_token": "device-bearer-token",
  "device_id": "uuid",
  "sync_endpoints": {
    "progress": "https://bookmann.example.com/api/sync/progress",
    "metadata": "https://bookmann.example.com/api/sync/metadata",
    "annotations": "https://bookmann.example.com/api/sync/annotations"
  }
}

Wireless Sync Endpoints

KOReader Wireless Sync (Calibre-compatible)

POST /api/sync/koreader/progress
Description: KOReader sends progress update

Request Headers:
  Authorization: Bearer {device_token}
  Content-Type: application/json

Request Body:
{
  "library_id": "optional-uuid",
  "books": [
    {
      "uuid": "book-uuid",
      "title": "Book Title",
      "authors": ["Author Name"],
      "progress": 0.45,
      "percentage": 0.45,
      "last_read": "2026-01-30T20:00:00Z",
      "device_info": {
        "koreader_version": "2024.01",
        "device_model": "kindle-paperwhite-5"
      }
    }
  ]
}

Response (202):
{
  "sync_status": "accepted",
  "books_synced": 1,
  "conflicts": [
    {
      "book_uuid": "book-uuid",
      "conflict_type": "progress_mismatch",
      "device_progress": 0.45,
      "server_progress": 0.42,
      "resolution": "device_wins"
    }
  ]
}

GET /api/sync/koreader/metadata/{book_uuid}
Description: KOReader fetches book metadata and sync status

Response (200):
{
  "uuid": "book-uuid",
  "title": "Book Title",
  "authors": ["Author Name"],
  "progress": {
    "percentage": 0.42,
    "character": 15432,
    "epubcfi": "epubcfi(/6/4/2:15)",
    "chapter": 3,
    "chapter_progress": 0.234
  },
  "annotations": {
    "highlights": [...],
    "notes": [...],
    "bookmarks": [...]
  },
  "last_sync": "2026-01-30T20:00:00Z"
}

Kobo Wireless Sync (Kobo API-compatible)

POST /api/sync/kobo/markup
Description: Kobo sends reading progress and annotations

Request Headers:
  Authorization: Bearer {device_token}
  x-kobo-device: {"DeviceId":"device-id","Model":"Kobo Clara"}

Request Body:
{
  "ReadingSync": [
    {
      "ContentId": "book-uuid",
      "PercentRead": 45.6,
      "EntitlementId": "entitlement-id",
      "RemainingTimeMinutes": 120,
      "LastModified": "2026-01-30T20:00:00Z"
    }
  ],
  "BookmarkSync": [
    {
      "ContentId": "book-uuid",
      "BookmarkText": "highlighted text",
      "BookmarkType": "annotation",
      "BookmarkTitle": "Chapter 3"
    }
  ]
}

Response (200):
{
  "Status": "Success",
  "MarkupsSynced": 5,
  "BookmarksSynced": 3
}

GET /api/sync/kobo/library
Description: Kobo fetches library and sync status

Response (200):
{
  "library_sync": [
    {
      "ContentId": "book-uuid",
      "ContentType": "6",
      "Title": "Book Title",
      "Author": "Author Name",
      "PercentRead": 42.3,
      "PagesRemaining": 115,
      "BookmarkCount": 3,
      "LastModified": "2026-01-30T20:00:00Z"
    }
  ]
}

Universal Progress Endpoints

Get Universal Progress

GET /api/progress/{book_uuid}
Description: Get progress with all location references

Response (200):
{
  "book_id": "book-uuid",
  "format_group": "reflowable",
  "universal_progress": 0.45678,
  "location_references": {
    "percentage": 0.45678,
    "epubcfi": "epubcfi(/6/4/2:15)",
    "character": 15432,
    "chapter": 3,
    "chapter_progress": 0.234,
    "viewport_y": 0.12
  },
  "device_progress": {
    "koreader": {
      "percentage": 0.45678,
      "last_sync": "2026-01-30T20:00:00Z"
    },
    "kobo": {
      "percentage": 45.6,
      "last_sync": "2026-01-30T19:55:00Z"
    },
    "web": {
      "display_page": 89,
      "total_pages": 200,
      "last_sync": "2026-01-30T20:05:00Z"
    }
  },
  "annotations": {
    "highlights": [...],
    "notes": [...],
    "bookmarks": [...]
  },
  "conflicts": [
    {
      "id": "conflict-uuid",
      "type": "progress",
      "resolved": false,
      "sources": ["koreader", "kobo"]
    }
  ]
}

Update Universal Progress

POST /api/progress/{book_uuid}
Description: Update progress with automatic conversion to all formats

Request Body:
{
  "source": "web|koreader|kobo|mobile",
  "location": {
    "percentage": 0.45678,
    "epubcfi": "epubcfi(/6/4/2:15)",
    "character": 15432,
    "chapter": 3,
    "page": 89,
    "total_pages": 200
  },
  "device_metadata": {
    "device_type": "web",
    "user_agent": "..."
  }
}

Response (200):
{
  "sync_status": "success",
  "progress_updated": true,
  "devices_notified": ["koreader-device-1", "kobo-device-2"],
  "broadcast": true
}

Conflict Resolution Endpoints

List Conflicts

GET /api/conflicts
Description: List all unresolved sync conflicts

Query Parameters:
  status: "unresolved|all"
  type: "progress|note|highlight|all"

Response (200):
{
  "conflicts": [
    {
      "id": "conflict-uuid",
      "media_item_id": "book-uuid",
      "media_item_title": "Book Title",
      "conflict_type": "progress",
      "conflict_data": {
        "koreader": {
          "source": "koreader",
          "timestamp": "2026-01-30T20:10:00Z",
          "data": {
            "percentage": 0.45,
            "epubcfi": "epubcfi(/6/4/2:15)",
            "character": 15432
          }
        },
        "kobo": {
          "source": "kobo",
          "timestamp": "2026-01-30T20:05:00Z",
          "data": {
            "percentage": 0.42,
            "location": "unknown"
          }
        }
      },
      "resolution_status": "unresolved",
      "created_at": "2026-01-30T20:10:05Z"
    }
  ],
  "total": 1,
  "unresolved": 1
}

Resolve Conflict

POST /api/conflicts/{conflict_id}/resolve
Description: Resolve a sync conflict by choosing source

Request Body:
{
  "winner": "koreader|kobo|web|manual",
  "manual_data": {
    "percentage": 0.43,
    "epubcfi": "epubcfi(/6/4/2:20)",
    "character": 15500
  }, // Required if winner is "manual"
  "apply_to_all_future_conflicts": false,
  "reason": "user chose more recent progress"
}

Response (200):
{
  "conflict_resolved": true,
  "applied_to": {
    "progress": true,
    "annotations": false
  },
  "devices_synced": ["koreader-device-1", "kobo-device-2"]
}

Device Management Endpoints

List User Devices

GET /api/devices
Response (200):
{
  "devices": [
    {
      "id": "device-uuid",
      "device_name": "My Kobo Clara",
      "device_type": "kobo",
      "last_sync": "2026-01-30T20:00:00Z",
      "last_seen": "2026-01-30T20:05:00Z",
      "sync_enabled": true,
      "auto_sync": true
    }
  ]
}

Update Device Settings

PUT /api/devices/{device_id}
Request Body:
{
  "device_name": "Updated Name",
  "sync_enabled": true,
  "auto_sync": true,
  "sync_frequency_minutes": 5
}

Response (200):
{
  "device_updated": true
}

Revoke Device Access

DELETE /api/devices/{device_id}
Response (204)

Format Grouping System

Three Format Groups

Based on format capabilities and optimal tracking methods:

1. Reflowable Group

Formats: EPUB, MOBI, AZW3, FB2, TXT Characteristics:

Text reflows to fit viewport
Page count changes based on display settings
Can contain complex formatting and images
Support for chapters, TOC

Optimal Tracking:

Percentage (0.0-1.0, 5 decimal precision)
EPUB CFI for precise locations
Chapter number + chapter progress
Character offset from beginning
Viewport Y position for visual reference

2. Fixed Layout Group

Formats: PDF, DJVU Characteristics:

Fixed page layout (like printed book)
Consistent page count across devices
Zoom and pan for navigation
May contain text layer

Optimal Tracking:

Page number + total pages
Page Y position (for scroll position)
Zoom level
Scroll X/Y coordinates
Character offset in extracted text (if available)

3. Comic Archive Group

Formats: CBZ, CBR, CBT, CB7, PDF comics Characteristics:

Image-based pages
No text layer (usually)
Panel-based reading
Single/double page modes
Panel zoom features

Optimal Tracking:

Page number
Panel number (for future panel navigation)
Panel coordinates (for future use)
Zoom level
Pan X/Y coordinates
Reading mode (single/double/panel)

Format Detection Logic

CREATE OR REPLACE FUNCTION detect_format_group(mimetype VARCHAR, file_path VARCHAR)
RETURNS VARCHAR AS $$
BEGIN
  CASE
    -- Reflowable formats
    WHEN mimetype = 'application/epub+zip' THEN
      RETURN 'reflowable';
    WHEN mimetype = 'application/x-mobipocket-ebook' THEN
      RETURN 'reflowable';
    WHEN mimetype = 'application/vnd.amazon.mobi8-ebook' THEN
      RETURN 'reflowable';
    WHEN file_path LIKE '%.epub' THEN
      RETURN 'reflowable';
    WHEN file_path LIKE '%.mobi' THEN
      RETURN 'reflowable';
    WHEN file_path LIKE '%.azw3' THEN
      RETURN 'reflowable';
    WHEN file_path LIKE '%.fb2' THEN
      RETURN 'reflowable';
    WHEN file_path LIKE '%.txt' THEN
      RETURN 'reflowable';

    -- Fixed layout formats
    WHEN mimetype = 'application/pdf' THEN
      RETURN 'fixed_layout';
    WHEN file_path LIKE '%.pdf' THEN
      RETURN 'fixed_layout';
    WHEN file_path LIKE '%.djvu' THEN
      RETURN 'fixed_layout';

    -- Comic archive formats
    WHEN mimetype = 'application/x-cbr' THEN
      RETURN 'comic_archive';
    WHEN mimetype = 'application/x-cbz' THEN
      RETURN 'comic_archive';
    WHEN file_path LIKE '%.cbz' THEN
      RETURN 'comic_archive';
    WHEN file_path LIKE '%.cbr' THEN
      RETURN 'comic_archive';
    WHEN file_path LIKE '%.cbt' THEN
      RETURN 'comic_archive';
    WHEN file_path LIKE '%.cb7' THEN
      RETURN 'comic_archive';

    ELSE
      RETURN 'unknown';
    END CASE;
END;
$$ LANGUAGE plpgsql;

Progress Tracking by Format

Reflowable Format Progress

{
  "format_group": "reflowable",
  "book_id": "uuid",
  "progress": {
    "percentage": 0.45678,
    "epubcfi": "epubcfi(/6/4[chap01ref]!/4[body01]/10[para05]/2:15)",
    "character": 15432,
    "chapter": 3,
    "chapter_progress": 0.234,
    "viewport_y": 0.12,
    "total_characters": 34567
  },
  "conversion_rules": {
    "to_percentage": "character / total_characters",
    "to_epubcfi": "use stored CFI or calculate from character",
    "to_chapter": "parse from CFI or calculate from character offset",
    "from_koreader": "direct percentage mapping",
    "from_kobo": "percentage / 100"
  },
  "display": {
    "web": "45.7% (Chapter 3)",
    "koreader": "45.7%",
    "kobo": "46%"
  }
}

Fixed Layout Format Progress

{
  "format_group": "fixed_layout",
  "book_id": "uuid",
  "progress": {
    "page": 45,
    "total_pages": 200,
    "page_y": 234,
    "zoom": 1.25,
    "scroll_x": 0,
    "scroll_y": 0,
    "character": 15432,
    "percentage": 0.225
  },
  "conversion_rules": {
    "to_percentage": "page / total_pages",
    "to_page": "floor(percentage * total_pages)",
    "to_character": "use OCR text extraction",
    "from_koreader": "map percentage to nearest page",
    "from_kobo": "direct page mapping"
  },
  "display": {
    "web": "Page 45 of 200 (22.5%)",
    "koreader": "22.5%",
    "kobo": "Page 45 (23%)"
  }
}

Comic Archive Format Progress

{
  "format_group": "comic_archive",
  "book_id": "uuid",
  "progress": {
    "page": 12,
    "total_pages": 32,
    "panel": 5,
    "panel_bounds": "120,85,300,250",
    "zoom": 1.5,
    "pan_x": 120,
    "pan_y": 85,
    "reading_mode": "single_page",
    "percentage": 0.375
  },
  "conversion_rules": {
    "to_percentage": "page / total_pages",
    "to_page": "floor(percentage * total_pages)",
    "from_koreader": "percentage to page mapping",
    "from_kobo": "page mapping"
  },
  "display": {
    "web": "Page 12 of 32 (Panel 5)",
    "koreader": "37.5%",
    "kobo": "Page 12 (38%)"
  }
}

Progress Conversion Engine

CREATE OR REPLACE FUNCTION convert_progress(
  source_format VARCHAR,
  target_format VARCHAR,
  source_progress JSONB
) RETURNS JSONB AS $$
DECLARE
  result JSONB;
  percentage FLOAT;
BEGIN
  -- Extract percentage from source
  CASE source_format
    WHEN 'reflowable' THEN
      percentage := (source_progress->>'percentage')::FLOAT;
    WHEN 'fixed_layout' THEN
      percentage := ((source_progress->>'page')::FLOAT /
                     (source_progress->>'total_pages')::FLOAT);
    WHEN 'comic_archive' THEN
      percentage := ((source_progress->>'page')::FLOAT /
                     (source_progress->>'total_pages')::FLOAT);
    ELSE
      percentage := 0.0;
  END CASE;

  -- Build target format progress
  CASE target_format
    WHEN 'reflowable' THEN
      result := jsonb_build_object(
        'percentage', percentage,
        'character', CAST(percentage * 34567 AS INTEGER),
        'epubcfi', 'epubcfi(/6/4/2:' || CAST(percentage * 100 AS INTEGER) || ')'
      );
    WHEN 'fixed_layout' THEN
      result := jsonb_build_object(
        'page', CAST(percentage * 200 AS INTEGER),
        'total_pages', 200,
        'percentage', percentage
      );
    WHEN 'comic_archive' THEN
      result := jsonb_build_object(
        'page', CAST(percentage * 32 AS INTEGER),
        'total_pages', 32,
        'percentage', percentage
      );
    ELSE
      result := '{}'::jsonb;
  END CASE;

  RETURN result;
END;
$$ LANGUAGE plpgsql;

Wireless Sync Protocols

KOReader Calibre-Compatible Protocol

Endpoint Structure:

POST /api/sync/koreader/progress
POST /api/sync/koreader/bookmarks
POST /api/sync/koreader/highlights
GET  /api/sync/koreader/metadata
GET  /api/sync/koreader/status

Authentication: Bearer token from device registration

Data Format (KOReader → Bookmann):

{
  "library_id": "optional-uuid",
  "books": [
    {
      "uuid": "book-uuid",
      "title": "Book Title",
      "authors": ["Author Name"],
      "progress": 0.45,
      "percentage": 0.45,
      "last_read": "2026-01-30T20:00:00Z",
      "bookmarks": [
        {
          "chapter": 3,
          "datetime": "2026-01-30T19:55:00Z",
          "notes": "highlighted text",
          "pos0": "epubcfi(/6/4/2:15)",
          "pos1": "epubcfi(/6/4/2:20)",
          "page": 45,
          "text": "highlighted text excerpt",
          "type": "highlight"
        }
      ],
      "highlights": [...],
      "notes": [...]
    }
  ]
}

Immediate Sync on Page Turn:

POST /api/sync/koreader/progress
Content-Type: application/json

{
  "sync_mode": "immediate",
  "books": [
    {
      "uuid": "book-uuid",
      "percentage": 0.45678,
      "chapter": 3,
      "timestamp": "2026-01-30T20:00:00Z"
    }
  ]
}

Response (202 Accepted):
{
  "sync_status": "accepted",
  "broadcast_scheduled": true,
  "queue_position": 1
}

Checkpoint Sync (Fallback):

POST /api/sync/koreader/progress
Content-Type: application/json

{
  "sync_mode": "checkpoint",
  "checkpoint_id": "checkpoint-uuid",
  "since_timestamp": "2026-01-30T19:00:00Z",
  "books": [...]
}

Response (200 OK):
{
  "sync_status": "completed",
  "books_synced": 1,
  "checkpoint_saved": true
}

Kobo Sync Protocol (Reverse-Engineered)

Endpoint Structure:

POST /api/sync/kobo/markup
POST /api/sync/kobo/bookmark
GET  /api/sync/kobo/library
GET  /api/sync/kobo/status

Authentication Headers:

Authorization: Bearer {device_token}
x-kobo-userid: {user_id}
x-kobo-device: {"DeviceId":"device-id","Model":"Kobo Clara","SerialNumber":"..."}

Data Format (Kobo → Bookmann):

{
  "ReadingSync": [
    {
      "ContentId": "book-uuid",
      "PercentRead": 45.6,
      "EntitlementId": "entitlement-id",
      "RemainingTimeMinutes": 120,
      "FirstReadTime": "2026-01-25T10:00:00Z",
      "LastModified": "2026-01-30T20:00:00Z"
    }
  ],
  "BookmarkSync": [
    {
      "BookmarkId": "bookmark-uuid",
      "ContentId": "book-uuid",
      "BookmarkText": "highlighted text",
      "BookmarkType": "annotation",
      "BookmarkTitle": "Chapter 3",
      "DateCreated": "2026-01-30T19:55:00Z",
      "Chapter": 3,
      "Hidden": false
    }
  ]
}

Single Port Architecture

All sync endpoints share the same port (8765) with proper routing:

Port 8765 (Bookmann)
├── /api/auth/* (Authentication)
├── /api/libraries/* (Library management)
├── /api/media-items/* (Media items)
├── /api/sync/koreader/* (KOReader wireless sync)
├── /api/sync/kobo/* (Kobo wireless sync)
├── /api/progress/* (Universal progress API)
├── /api/conflicts/* (Conflict resolution)
├── /api/devices/* (Device management)
└── /ws/sync (WebSocket for real-time updates)

Caddy Proxy Configuration:

bookmann.example.com {
    reverse_proxy localhost:8765
    encode gzip
    log {
        output file /var/log/caddy/bookmann-access.log
    }
}

Security Architecture

Device Authentication Flow

No API Keys on Devices! All authentication goes through web interface.

1. Device generates unique identifier (hardware ID)
2. Device sends registration request with device type
3. Bookmann creates pending registration with auth URL
4. User visits auth URL in web browser
5. User logs in and approves device
6. Bookmann generates device token
7. Device polls for token approval
8. Device receives token and begins syncing

Authentication Endpoints

POST /api/devices/register/initiate
Request: { "device_identifier": "hw-id-123", "device_type": "kobo" }
Response: {
  "registration_id": "reg-uuid",
  "auth_url": "https://bookmann.example.com/devices/auth/confirm/reg-uuid",
  "expires_in": 300,
  "qr_code": "data:image/png;base64,..."
}

[User visits auth URL]

POST /api/devices/register/status
Request: { "registration_id": "reg-uuid" }
Response: {
  "status": "pending" | "approved" | "expired",
  "auth_token": "Bearer token..." (when approved)
}

Token Management

Device Token Structure:

{
  "device_id": "uuid",
  "user_id": "uuid",
  "device_type": "kobo|koreader|web|mobile",
  "permissions": [
    "sync:progress",
    "sync:annotations",
    "sync:metadata"
  ],
  "expires": "never",
  "revocable": true
}

Rate Limiting

Per-device rate limits:
- Sync requests: 60/minute
- Progress updates: 120/minute (page turns)
- Metadata requests: 30/minute

Per-user rate limits:
- All devices combined: 300/minute
- Conflict resolution: 10/minute

Implementation: Redis-backed sliding window

Request Validation

// Example Go validation for sync requests
func validateSyncRequest(deviceToken string, request SyncRequest) error {
    // 1. Validate device token
    device, err := validateDeviceToken(deviceToken)
    if err != nil {
        return ErrUnauthorized
    }

    // 2. Check device not revoked
    if device.Revoked {
        return ErrDeviceRevoked
    }

    // 3. Validate device permissions
    if !hasPermission(device, "sync:progress") {
        return ErrForbidden
    }

    // 4. Validate request format
    if err := validateRequestFormat(request); err != nil {
        return ErrInvalidRequest
    }

    // 5. Check rate limits
    if exceededRateLimit(device.ID) {
        return ErrRateLimited
    }

    // 6. Validate media item ownership
    if !ownsMediaItem(device.UserID, request.BookUUID) {
        return ErrNotFound
    }

    return nil
}

CORS Configuration

// Allow device origins
allowedOrigins := []string{
    "https://bookmann.example.com",
    "kobo://*", // Custom protocol for Kobo app
    "koreader://*", // Custom protocol for KOReader
}

corsConfig := cors.Config{
    AllowOrigins:     allowedOrigins,
    AllowMethods:     []string{"GET", "POST", "PUT", "DELETE"},
    AllowHeaders:     []string{"Authorization", "Content-Type"},
    ExposeHeaders:    []string{"X-Sync-Status", "X-Conflict-Detected"},
    AllowCredentials: true,
}

Conflict Resolution System

Conflict Detection

Automatic Conflict Detection:

CREATE OR REPLACE FUNCTION detect_conflict(
  media_item_id UUID,
  new_progress JSONB,
  device_type VARCHAR
) RETURNS BOOLEAN AS $$
DECLARE
  existing_progress JSONB;
  time_diff INTERVAL;
BEGIN
  -- Get most recent progress from different device
  SELECT pg_jsonb INTO existing_progress
  FROM reading_progress
  WHERE media_item_id = $1
    AND last_sync_source != device_type
  ORDER BY last_sync_timestamp DESC
  LIMIT 1;

  -- If no existing progress, no conflict
  IF existing_progress IS NULL THEN
    RETURN FALSE;
  END IF;

  -- Check time difference (within 5 minutes = potential conflict)
  time_diff := NOW() - (existing_progress->>'timestamp')::TIMESTAMP;

  IF time_diff < INTERVAL '5 minutes' THEN
    -- Check if progress significantly different
    IF ABS((new_progress->>'percentage')::FLOAT -
           (existing_progress->>'percentage')::FLOAT) > 0.01 THEN
      RETURN TRUE;
    END IF;
  END IF;

  RETURN FALSE;
END;
$$ LANGUAGE plpgsql;

Side-by-Side Conflict Display

UI Component Specification:

┌─────────────────────────────────────────────────────────┐
│ ⚠️ Sync Conflict Detected: Book Title                    │
├─────────────────────────────────────────────────────────┤
│                                                           │
│ Compare progress from different devices:                 │
│                                                           │
│ ┌─────────────────┬─────────────────────────────────┐   │
│ │ KOReader        │ Kobo Device                     │   │
│ ├─────────────────┼─────────────────────────────────┤   │
│ │ Progress: 45.7% │ Progress: 42.3%                 │   │
│ │ Location:       │ Location: Page 89/200           │   │
│ │ Chapter 3       │                                  │   │
│ │ CFI: ...        │ Last Read: 2 hours ago          │   │
│ │                 │                                  │   │
│ │ Last Read:      │                                  │   │
│ │ 5 minutes ago   │                                  │   │
│ │                 │                                  │   │
│ │ Device:         │ Device: Kobo Clara              │   │
│ │ Kindle PW5      │                                  │   │
│ └─────────────────┴─────────────────────────────────┘   │
│                                                           │
│ [Choose KOReader] [Choose Kobo] [Merge] [Dismiss]       │
│                                                           │
│ □ Auto-resolve future conflicts from [KOReader]          │
│                                                           │
└─────────────────────────────────────────────────────────┘

Conflict Resolution Logic

"Last In Wins" with User Override:

1. Each sync writes timestamp and source device
2. If conflict detected:
   - Compare timestamps
   - More recent progress wins by default
   - Create conflict record for user review
3. User can:
   - Accept auto-resolution
   - Choose different source manually
   - Merge progress (take max of both)
   - Set auto-resolution preference

Merge Logic:

CREATE OR REPLACE FUNCTION merge_progress(
  progress_a JSONB,
  progress_b JSONB
) RETURNS JSONB AS $$
BEGIN
  RETURN jsonb_build_object(
    'percentage', GREATEST(
      (progress_a->>'percentage')::FLOAT,
      (progress_b->>'percentage')::FLOAT
    ),
    'merged_from', ARRAY[
      (progress_a->>'source'),
      (progress_b->>'source')
    ],
    'merge_timestamp', NOW()
  );
END;
$$ LANGUAGE plpgsql;

Backup & Recovery Strategy

Hybrid Queue + Checkpoint Strategy

Immediate Sync (Page Turn):

Device → POST /api/sync/progress (sync_mode: immediate)
  ↓
Bookmann validates and stores
  ↓
WebSocket broadcast to all clients
  ↓
If network error → Queue locally with retry

Checkpoint Sync (Fallback):

Device → POST /api/sync/progress (sync_mode: checkpoint)
  ↓
Bookmann validates and stores
  ↓
Process queued updates since last checkpoint
  ↓
Clear queue on success
  ↓
On failure → Keep in queue with exponential backoff

Sync Queue Management

Queue Schema:

CREATE TABLE sync_queue (
  id UUID PRIMARY KEY,
  device_id UUID NOT NULL,
  media_item_id UUID,
  sync_type VARCHAR(20) NOT NULL,
  sync_data JSONB NOT NULL,
  priority INTEGER DEFAULT 5, -- 1=highest, 10=lowest
  attempts INTEGER DEFAULT 0,
  max_attempts INTEGER DEFAULT 3,
  status VARCHAR(20) DEFAULT 'pending',
  error_message TEXT,
  created_at TIMESTAMP DEFAULT NOW(),
  next_retry_at TIMESTAMP,
  processed_at TIMESTAMP
);

Priority Levels:

Priority 1: User-initiated sync (manual refresh)
Priority 2: Book completion (100% progress)
Priority 3: Critical annotations (user notes)
Priority 5: Page turns (immediate sync)
Priority 7: Checkpoint sync (periodic)
Priority 10: Background metadata sync

Exponential Backoff:

Attempt 1: Immediate
Attempt 2: 1 minute delay
Attempt 3: 5 minute delay
Attempt 4: 15 minute delay
Attempt 5: 1 hour delay
After 5 failed attempts: Mark as failed, notify user

Offline Capability

Device-Side Queue (KOReader example):

-- KOReader plugin code
local SyncQueue = {
  queue = {},
  max_queue_size = 100,

  add = function(self, sync_data)
    table.insert(self.queue, {
      data = sync_data,
      timestamp = os.time(),
      retry_count = 0
    })
    self:persist()
  end,

  sync = function(self)
    for i, item in ipairs(self.queue) do
      local success = self:send_to_server(item.data)
      if success then
        table.remove(self.queue, i)
      else
        item.retry_count = item.retry_count + 1
        if item.retry_count >= 3 then
          self:notify_user(item)
          table.remove(self.queue, i)
        end
      end
    end
    self:persist()
  end,

  persist = function(self)
    local file = io.open("/mnt/us/sync-queue.json", "w")
    file:write(json.encode(self.queue))
    file:close()
  end
}

Recovery Scenarios

Scenario 1: Device Offline for Extended Period

1. Device queues all changes locally
2. User marks book as finished (100%)
3. High-priority sync queued
4. Device comes back online
5. Bookmann processes queue in priority order
6. Finishes book, updates all other devices
7. User notified of sync completion

Scenario 2: Server Unavailable

1. All devices detect server unavailable
2. Switch to checkpoint mode (batch every 5 min)
3. Queue changes locally
4. When server returns:
   - Process all checkpoints
   - Resolve conflicts based on timestamps
   - Notify user of conflicts

Scenario 3: Multiple Devices Offline

1. All devices queue changes independently
2. Server processes queue in order of reconnection
3. Timestamps determine conflict resolution
4. User gets conflict summary for manual resolution
5. Auto-resolution preferences applied to future conflicts

Real-time Synchronization

WebSocket Architecture

Connection Endpoint:

WS /ws/sync?token={device_token}

Message Format:

{
  "type": "progress_update|annotation_update|conflict|sync_complete",
  "timestamp": "2026-01-30T20:00:00Z",
  "data": {
    "book_id": "uuid",
    "progress": {...},
    "annotations": {...}
  },
  "source_device": {
    "id": "device-uuid",
    "name": "My Kobo",
    "type": "kobo"
  }
}

Broadcast Strategy

Device A updates progress
  ↓
Bookmann stores in database
  ↓
WebSocket broadcast to:
  - Web interface (if user logged in)
  - Mobile apps (if connected)
  - Other devices (on next sync)
  ↓
Devices update their display immediately

Connection Management

Device Connection States:

type ConnectionManager struct {
    connections map[string]*DeviceConnection
    broadcast   chan BroadcastMessage
}

type DeviceConnection struct {
    DeviceID   string
    UserID     string
    DeviceType string
    Connected  time.Time
    LastPing   time.Time
    Send       chan Message
}

func (m *ConnectionManager) HandleConnection(conn *websocket.Conn) {
    device := m.authenticate(conn)
    m.connections[device.ID] = device

    // Send initial state
    device.Send <- m.getInitialState(device.UserID)

    // Handle messages
    for {
        msg := <-device.Send
        err := conn.WriteJSON(msg)
        if err != nil {
            break
        }
    }

    // Cleanup
    delete(m.connections, device.ID)
}

Heartbeat & Keep-Alive

Client → PING (every 30 seconds)
Server → PONG

If no PING for 90 seconds → Close connection

Device Onboarding Flow

Step-by-Step Device Setup

KOReader Setup

1. Generate Registration Request

Device sends: POST /api/devices/register
{
  "device_type": "koreader",
  "device_identifier": "hardware-specific-id",
  "device_name": "My Kindle Paperwhite"
}

2. Receive Registration Info

Server responds: {
  "registration_id": "reg-uuid-123",
  "auth_url": "https://bookmann.example.com/devices/auth/confirm/reg-uuid-123",
  "qr_code": "data:image/png;base64,iVBORw0KG...",
  "expires_in": 300
}

3. User Approves Device

User visits auth URL → Sees:
┌─────────────────────────────────────┐
│ Device Registration Request         │
│                                     │
│ Device Type: KOReader               │
│ Device Name: My Kindle Paperwhite   │
│                                     │
│ Approve this device to sync your    │
│ reading progress and annotations?   │
│                                     │
│ [Approve] [Deny]                    │
└─────────────────────────────────────┘

4. Configure KOReader

In KOReader settings:
- Calibre sync: ON
- Server URL: https://bookmann.example.com/api/sync/koreader
- Wireless sync: ON
- Sync frequency: Every page turn
- Auto-sync: ON

KOReader stores config in:
/mnt/us/settings/koreader/calibre.lua

5. Device Polls for Token

Device: POST /api/devices/register/status
{
  "registration_id": "reg-uuid-123"
}

Server responds:
{
  "status": "approved",
  "auth_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "sync_endpoints": {
    "progress": "https://bookmann.example.com/api/sync/koreader/progress",
    "metadata": "https://bookmann.example.com/api/sync/koreader/metadata",
    "bookmarks": "https://bookmann.example.com/api/sync/koreader/bookmarks"
  }
}

Kobo Setup

1-3: Same as KOReader (registration & approval)

4. Configure Kobo Device

In Kobo settings:
- Sync server: https://bookmann.example.com/api/sync/kobo
- Auto-sync: ON
- Sync frequency: Every 5 minutes

Kobo stores config in:
/mnt/onboard/.kobo/Kobo/Kobo eReader.conf
[Sync]
ServerURL=https://bookmann.example.com/api/sync/kobo
AutoSyncEnabled=true
SyncFrequency=5

5. Device Syncs

Kobo sends initial sync:
POST /api/sync/kobo/library
Authorization: Bearer {token}
x-kobo-device: {"DeviceId":"...", "Model":"Kobo Clara"}

Server responds with library and current sync state

Device Discovery (Optional)

mDNS/Bonjour for Local Network:

Bookmann broadcasts: _bookmann-sync._tcp.local
Port: 8765
TXT: "path=/api/sync", "version=1.0"

Devices discover automatically
User confirms connection

Implementation Phases

Phase 1: Universal Progress System (Weeks 1-4)

Week 1: Database Schema

Add format_group columns to media_items
Add universal progress columns to reading_progress
Create device registry tables
Create sync_queue tables
Create sync_conflicts tables

Week 2: Format Detection & Progress Conversion

Implement format detection function
Build progress conversion engine
Update all existing progress to new format
Create migration scripts

Week 3: Core Progress APIs

GET /api/progress/{book_uuid}
POST /api/progress/{book_uuid}
Progress conversion logic
Format-specific display logic

Week 4: Testing & Validation

Unit tests for conversion engine
Integration tests for progress APIs
Test with existing data
Performance testing

Deliverables:

Universal progress tracking working
All existing progress converted
Format-aware display in web UI

Phase 2: Device Management & Auth (Weeks 5-6)

Week 5: Device Registration

Device registry implementation
Registration endpoints
Web-based approval flow
QR code generation
Device management UI

Week 6: Authentication

Device token generation
Token validation middleware
Permission system
Rate limiting per device
Revocation system

Deliverables:

Users can register devices
Devices authenticate via web flow
Device management UI complete

Phase 3: KOReader Integration (Weeks 7-9)

Week 7: KOReader Protocol

Implement calibre-compatible endpoints
Progress sync endpoint
Metadata sync endpoint
Bookmark/highlight sync

Week 8: Bidirectional Sync

Bookmann → KOReader sync
KOReader → Bookmann sync
Format conversion for KOReader
Error handling & retry logic

Week 9: Real-time Updates

WebSocket implementation
Progress broadcast system
Connection management
Testing with actual KOReader devices

Deliverables:

Full KOReader wireless sync
Real-time progress updates
Tested on actual devices

Phase 4: Kobo Integration (Weeks 10-12)

Week 10: Kobo Protocol

Reverse-engineer Kobo sync API
Implement Kobo-compatible endpoints
Markup sync endpoint
Library sync endpoint

Week 11: Bidirectional Sync

Bookmann → Kobo sync
Kobo → Bookmann sync
Format conversion for Kobo
Annotation sync

Week 12: Testing & Validation

Test on actual Kobo devices
Performance testing
Error handling
Documentation

Deliverables:

Full Kobo wireless sync
Tested on Kobo Clara/Naura/etc.
Device setup documentation

Phase 5: Conflict Resolution (Weeks 13-14)

Week 13: Conflict Detection

Automatic conflict detection
Conflict storage
Conflict notifications
Conflict listing API

Week 14: Resolution System

Side-by-side conflict UI
Resolution endpoints
Auto-resolution preferences
Merge logic

Deliverables:

Complete conflict resolution system
User-friendly conflict UI
Auto-resolution preferences

Phase 6: Backup & Recovery (Weeks 15-16)

Week 15: Sync Queue

Queue implementation
Priority system
Retry logic with exponential backoff
Device-side queue (KOReader plugin)

Week 16: Offline Support

Checkpoint sync mode
Offline detection
Recovery scenarios
Progress merge logic

Deliverables:

Robust offline sync
Recovery from extended offline
Queue management UI

Phase 7: Polish & Documentation (Weeks 17-18)

Week 17: Performance & Security

Performance optimization
Security audit
Rate limiting tuning
Load testing

Week 18: Documentation & Release

User documentation
Developer documentation
Device setup guides
API documentation
Release preparation

Deliverables:

Production-ready system
Complete documentation
Device setup guides
API reference

Testing Requirements

Unit Tests

Format Detection:

TestFormatDetection_Epub()
TestFormatDetection_Mobi()
TestFormatDetection_Pdf()
TestFormatDetection_Cbz()
TestFormatDetection_Cbr()
TestFormatDetection_UnknownFormat()

Progress Conversion:

TestProgressConversion_ReflowableToFixed()
TestProgressConversion_FixedToReflowable()
TestProgressConversion_ComicToReflowable()
TestProgressConversion_Precision()
TestProgressConversion_EdgeCases()

Conflict Detection:

TestConflictDetection_SameTimestamp()
TestConflictDetection_DifferentDevices()
TestConflictDetection_SmallDifference()
TestConflictDetection_LargeDifference()

Integration Tests

Sync Flow:

TestSyncFlow_KOReaderToBookmann()
TestSyncFlow_BookmannToKOReader()
TestSyncFlow_KoboToBookmann()
TestSyncFlow_Bidirectional()
TestSyncFlow_MultipleDevices()

Conflict Resolution:

TestConflictResolution_AutoResolve()
TestConflictResolution_ManualResolve()
TestConflictResolution_Merge()
TestConflictResolution_UserPreference()

Offline Recovery:

TestOfflineRecovery_QueueProcessing()
TestOfflineRecovery_PriorityOrder()
TestOfflineRecovery_ExponentialBackoff()
TestOfflineRecovery_ExtendedOffline()

End-to-End Tests

User Scenarios:

TestScenario_ReadOnKOReaderContinueOnWeb()
TestScenario_ReadOnKoboContinueOnMobile()
TestScenario_MultipleDevicesConflict()
TestScenario_ExtendedOfflineSync()
TestScenario_BookCompletionAcrossDevices()

Performance Tests

Load Testing:

- 1000 concurrent sync requests
- 100 devices syncing simultaneously
- 10,000 progress updates per minute
- WebSocket connection stability under load

Database Performance:

- Query performance with 1M+ progress records
- Index effectiveness
- Transaction throughput
- Conflict detection performance

Performance Considerations

Database Optimization

Indexes:

-- Critical indexes for sync performance
CREATE INDEX idx_reading_progress_media_user
  ON reading_progress(media_item_id, user_id);

CREATE INDEX idx_reading_progress_last_sync
  ON reading_progress(last_sync_timestamp DESC)
  WHERE last_sync_timestamp > NOW() - INTERVAL '30 days';

CREATE INDEX idx_sync_queue_status_priority
  ON sync_queue(status, priority)
  WHERE status != 'completed';

CREATE INDEX idx_devices_active
  ON devices(user_id, sync_enabled)
  WHERE sync_enabled = true;

Partitioning:

-- Partition reading_history by month
CREATE TABLE reading_history_2026_01
  PARTITION OF reading_history
  FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

Caching Strategy

Redis Cache:

- Device authentication tokens (TTL: 24 hours)
- Active sync sessions (TTL: 1 hour)
- Progress hot data (TTL: 5 minutes)
- Conflict data (TTL: 1 hour)

Cache Invalidation:

- Progress update → invalidate progress cache
- New conflict → broadcast to all clients
- Device auth revoked → invalidate token cache

Connection Pooling

Database Connections:

- Max connections: 100
- Min connections: 10
- Connection timeout: 30 seconds
- Query timeout: 10 seconds

WebSocket Connections:

- Max concurrent connections: 1000
- Heartbeat interval: 30 seconds
- Connection timeout: 90 seconds
- Message buffer size: 1000 messages

Rate Limiting

Per-Device Limits:

- Sync requests: 60/minute
- Progress updates: 120/minute
- Metadata requests: 30/minute
- WebSocket messages: 300/minute

Per-User Limits:

- Total requests: 300/minute
- Conflict resolutions: 10/minute
- Device registrations: 5/hour

Conclusion

This implementation guide provides a complete blueprint for creating a universal cross-platform reading synchronization system that:

Tracks progress optimally for each format (reflowable, fixed, comics)
Syncs wirelessly with KOReader and Kobo devices
Resolves conflicts intelligently with user control
Handles offline scenarios gracefully with queueing
Updates in real-time via WebSocket broadcasts
Authenticates securely via web interface (no device API keys)
Scales efficiently with proper caching and optimization

Result: Users get an Amazon Kindle-like experience - pick up any device, continue reading exactly where they left off, with all annotations synchronized - completely self-hosted and open-source.

Appendix: Example Data Migrations

Migrate Existing Page-Based Progress

-- Convert existing progress to new format
UPDATE reading_progress
SET
  percentage = CASE
    WHEN total_pages > 0 THEN current_page::FLOAT / total_pages::FLOAT
    ELSE 0
  END,
  character_offset = current_page * 500, -- Rough estimate
  chapter = CASE
    WHEN current_page < 100 THEN 1
    WHEN current_page < 200 THEN 2
    ELSE 3
  END,
  chapter_progress = 0.5,
  last_sync_source = 'migration',
  last_sync_timestamp = NOW()
WHERE percentage IS NULL;

Update Media Items with Format Groups

-- Set format groups based on file extensions
UPDATE media_items
SET
  format_group = CASE
    WHEN file_path LIKE '%.epub' THEN 'reflowable'
    WHEN file_path LIKE '%.mobi' THEN 'reflowable'
    WHEN file_path LIKE '%.pdf' THEN 'fixed_layout'
    WHEN file_path LIKE '%.cbz' THEN 'comic_archive'
    WHEN file_path LIKE '%.cbr' THEN 'comic_archive'
    ELSE 'reflowable' -- Default
  END,
  is_reflowable = CASE
    WHEN file_path LIKE '%.pdf' THEN FALSE
    WHEN file_path LIKE '%.cbz' THEN FALSE
    WHEN file_path LIKE '%.cbr' THEN FALSE
    ELSE TRUE
  END
WHERE format_group IS NULL OR format_group = '';

Document Version: 1.0 Last Updated: 2026-01-30 Maintained By: Bookmann Development Team Status: Ready for Implementation

55 KiB Raw Blame History

Bookmann Universal Cross-Platform Sync Implementation Guide

Executive Summary

Table of Contents

Current System Status

Working Endpoints (As of Testing)

Known Issues

Problem Analysis

The EPUB Progress Problem

Cross-Platform Incompatibility

Technical Architecture

System Overview

Data Flow Architecture

Universal Progress Hub Concept

Database Schema Changes

Core Schema Additions

Database Triggers for Automatic Timestamps

API Endpoints Specification

Authentication Endpoints

Device Registration Flow

Web Login for Device Authentication

Wireless Sync Endpoints

KOReader Wireless Sync (Calibre-compatible)

Kobo Wireless Sync (Kobo API-compatible)

Universal Progress Endpoints

Get Universal Progress

Update Universal Progress

Conflict Resolution Endpoints

List Conflicts

Resolve Conflict

Device Management Endpoints

List User Devices

Update Device Settings

Revoke Device Access

Format Grouping System

Three Format Groups

1. Reflowable Group

2. Fixed Layout Group

3. Comic Archive Group

Format Detection Logic

Progress Tracking by Format

Reflowable Format Progress

Fixed Layout Format Progress

Comic Archive Format Progress

Progress Conversion Engine

Wireless Sync Protocols

KOReader Calibre-Compatible Protocol

Kobo Sync Protocol (Reverse-Engineered)

Single Port Architecture

Security Architecture

Device Authentication Flow

Authentication Endpoints

Token Management

Rate Limiting

Request Validation

CORS Configuration

Conflict Resolution System

Conflict Detection

Side-by-Side Conflict Display

Conflict Resolution Logic

Backup & Recovery Strategy

Hybrid Queue + Checkpoint Strategy

Sync Queue Management

Offline Capability

Recovery Scenarios

Real-time Synchronization

WebSocket Architecture

Broadcast Strategy

Connection Management

Heartbeat & Keep-Alive

Device Onboarding Flow

Step-by-Step Device Setup

KOReader Setup

Kobo Setup

Device Discovery (Optional)

Implementation Phases

Phase 1: Universal Progress System (Weeks 1-4)

Phase 2: Device Management & Auth (Weeks 5-6)

Phase 3: KOReader Integration (Weeks 7-9)

Phase 4: Kobo Integration (Weeks 10-12)

55 KiB

Raw Blame History