Bookhoard/bookhoard
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 12 Wiki Activity
Files
030d30a225db902b77ff2a17c48e501897ef62cb
bookhoard/UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md
T
john-okeefe bb3c32c59f Phase 1 Week 2: Format detection and progress conversion engine 
- Add internal/sync package with format detection
- FormatGroup types: reflowable, fixed_layout, comic_archive
- DetectFormatGroup() function based on mimetype and file extension
- MimeType mappings for common ebook formats
- Progress conversion engine with:
  - ConvertProgress() between format groups
  - Extract percentage from various progress formats
  - PageToPercentage / PercentageToPage helpers
  - CharacterToPercentage / PercentageToCharacter helpers
  - MergeProgress() with 'max progress wins' strategy
  - FormatProgressForDisplay() for UI rendering
- Add sqlc queries for format detection and progress updates
- BulkUpdateFormatGroups query for auto-format detection
- GetUniversalProgress query with all location references
- UpdateUniversalProgress query with device sync metadata
- ReadingHistory queries for session tracking
2026-01-30 16:07:00 -05:00

2167 lines
55 KiB
Markdown
Raw Blame History

# Bookmann Universal Cross-Platform Sync Implementation Guide
## Executive Summary
This document provides a complete technical specification for implementing a universal cross-platform reading progress synchronization system that rivals the Amazon Kindle ecosystem. The system enables seamless reading progress, highlights, and notes synchronization across web interface, KOReader, Kobo devices, and future platforms - all self-hosted and open-source.
**Vision**: Users can read on any device, put it down, pick up any other device, and continue exactly where they left off with all annotations synchronized.
**Core Philosophy**: Bookmann serves as the "universal translator" between different reading platforms, understanding multiple location reference systems and converting between them seamlessly.
---
## Table of Contents
1. [Current System Status](#current-system-status)
2. [Problem Analysis](#problem-analysis)
3. [Technical Architecture](#technical-architecture)
4. [Database Schema Changes](#database-schema-changes)
5. [API Endpoints Specification](#api-endpoints-specification)
6. [Format Grouping System](#format-grouping-system)
7. [Progress Tracking by Format](#progress-tracking-by-format)
8. [Wireless Sync Protocols](#wireless-sync-protocols)
9. [Security Architecture](#security-architecture)
10. [Conflict Resolution System](#conflict-resolution-system)
11. [Backup & Recovery Strategy](#backup--recovery-strategy)
12. [Real-time Synchronization](#real-time-synchronization)
13. [Device Onboarding Flow](#device-onboarding-flow)
14. [Implementation Phases](#implementation-phases)
15. [Testing Requirements](#testing-requirements)
16. [Performance Considerations](#performance-considerations)
---
## Current System Status
### Working Endpoints (As of Testing)
All core functionality is operational:
- Authentication: Register, login, logout, refresh tokens ✅
- Library management: Create, read, update, delete libraries ✅
- Scanner: Background scanning, watch mode, status tracking ✅
- Media items: CRUD operations, search, filter, sort ✅
- Progress tracking: Basic page-based tracking ✅
- Notes: Create, read, update, delete ✅
- Highlights: Create, read, update, delete ✅
- Ratings: Full rating system ✅
### Known Issues
1. **EPUB Progress Tracking**: Current page-based system is unreliable for reflowable formats
2. **No Cross-Device Sync**: Progress doesn't sync between devices
3. **Limited Platform Support**: Only web interface, no device integration
4. **Format Inconsistency**: Different formats treated identically despite different capabilities
---
## Problem Analysis
### The EPUB Progress Problem
**Current Implementation Flaw:**
- EPUB is reflowable - page count changes based on font size, screen size, zoom level
- "Page 45 of 200" is meaningless across different devices or reading sessions
- Notes at "page:45" location become unfindable with different display settings
- Progress sharing between users or devices is unreliable
**Real-World Example:**
- User reads on tablet (large font) → sees "page 45 of 250"
- Same user reads on phone (small font) → sees "page 78 of 450"
- Same book, same progress, but completely different page numbers
### Cross-Platform Incompatibility
**Kobo Devices:**
- Use percentage-based progress internally
- Sync to proprietary Kobo cloud
- Store annotations in SQLite databases
- No open API for third-party sync
**KOReader:**
- Open-source, supports Wi-Fi sync
- Uses calibre-compatible protocols
- Stores data in `.sdr` sidecar folders
- Percentage + character offset tracking
**Bookmann Current State:**
- Page-based tracking only
- No device protocols implemented
- No sync endpoints for external devices
---
## Technical Architecture
### System Overview
```
┌─────────────────────────────────────────────────────────────┐
│ Bookmann Core System │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Web UI │ │ Wireless │ │ Progress │ │
│ │ Interface │ │ Sync API │ │ Conversion │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘
▲
│
┌───────────────────┼───────────────────┐
│ │ │
┌───────┴────────┐ ┌──────┴─────────┐ ┌─────┴────────┐
│ KOReader │ │ Kobo Device │ │ Web Mobile │
│ .sdr folders │ │ SQLite DB │ │ App/API │
└────────────────┘ └────────────────┘ └───────────────┘
```
### Data Flow Architecture
```
Device Progress → Device Protocol → Bookmann Wireless API
↓
Universal Progress JSON
↓
Format Conversion Engine
↓
Database Storage (Multi-format)
↓
WebSocket Broadcast
↓
All Connected Clients (Web, Mobile, Other Devices)
```
### Universal Progress Hub Concept
Bookmann doesn't just store progress - it maintains a **multi-dimensional location reference system** that can express the same reading position in multiple ways simultaneously:
```json
{
"book_id": "uuid-1234",
"universal_progress": 0.45678,
"format_group": "reflowable",
"location_references": {
"percentage": 0.45678,
"epubcfi": "epubcfi(/6/4[chap1]!/4/2:15)",
"character": 15432,
"chapter": 3,
"chapter_progress": 0.234,
"viewport_y": 0.12
},
"device_specific": {
"koreader": { "percent": 0.45678, "timestamp": "..." },
"kobo": { "progress": 45, "timestamp": "..." },
"web": { "display_page": 89, "total_pages": 200, "timestamp": "..." }
}
}
```
---
## Database Schema Changes
### Core Schema Additions
```sql
-- ============================================
-- FORMAT DETECTION AND GROUPING
-- ============================================
ALTER TABLE media_items
ADD COLUMN format_group VARCHAR(20) NOT NULL DEFAULT 'reflowable',
ADD COLUMN format_mimetype VARCHAR(100),
ADD COLUMN is_reflowable BOOLEAN DEFAULT TRUE,
ADD COLUMN has_fixed_layout BOOLEAN DEFAULT FALSE,
ADD COLUMN total_characters BIGINT,
ADD COLUMN chapter_count INTEGER;
-- ============================================
-- UNIVERSAL PROGRESS TRACKING
-- ============================================
ALTER TABLE reading_progress
ADD COLUMN percentage FLOAT CHECK (percentage >= 0 AND percentage <= 1),
ADD COLUMN character_offset BIGINT,
ADD COLUMN epubcfi TEXT,
ADD COLUMN chapter INTEGER,
ADD COLUMN chapter_progress FLOAT CHECK (chapter_progress >= 0 AND chapter_progress <= 1),
ADD COLUMN viewport_x FLOAT DEFAULT 0,
ADD COLUMN viewport_y FLOAT DEFAULT 0,
ADD COLUMN zoom_level FLOAT DEFAULT 1.0,
ADD COLUMN scroll_position_x FLOAT DEFAULT 0,
ADD COLUMN scroll_position_y FLOAT DEFAULT 0,
ADD COLUMN panel_number INTEGER,
ADD COLUMN reading_mode VARCHAR(20);
-- ============================================
-- DEVICE SYNC METADATA
-- ============================================
ALTER TABLE reading_progress
ADD COLUMN last_sync_device VARCHAR(50),
ADD COLUMN last_sync_source VARCHAR(20),
ADD COLUMN last_sync_timestamp TIMESTAMP,
ADD COLUMN conflict_detected BOOLEAN DEFAULT FALSE,
ADD COLUMN conflict_resolved BOOLEAN DEFAULT TRUE;
-- ============================================
-- NOTES LOCATION ENHANCEMENTS
-- ============================================
ALTER TABLE media_notes
ADD COLUMN percentage_location FLOAT,
ADD COLUMN character_start INTEGER,
ADD COLUMN character_end INTEGER,
ADD COLUMN epubcfi_location TEXT,
ADD COLUMN chapter_reference INTEGER,
ADD COLUMN paragraph_reference INTEGER,
ADD COLUMN device_sync_data JSONB;
-- ============================================
-- HIGHLIGHTS LOCATION ENHANCEMENTS
-- ============================================
ALTER TABLE media_highlights
ADD COLUMN percentage_start FLOAT,
ADD COLUMN percentage_end FLOAT,
ADD COLUMN character_start INTEGER,
ADD COLUMN character_end INTEGER,
ADD COLUMN epubcfi_start TEXT,
ADD COLUMN epubcfi_end TEXT,
ADD COLUMN chapter_reference INTEGER,
ADD COLUMN paragraph_start INTEGER,
ADD COLUMN paragraph_end INTEGER,
ADD COLUMN panel_number INTEGER,
ADD COLUMN device_sync_data JSONB;
-- ============================================
-- DEVICE REGISTRY
-- ============================================
CREATE TABLE devices (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
device_name VARCHAR(100) NOT NULL,
device_type VARCHAR(20) NOT NULL, -- 'koreader', 'kobo', 'web', 'mobile'
device_identifier VARCHAR(255) UNIQUE NOT NULL,
auth_token VARCHAR(500) UNIQUE NOT NULL,
last_sync TIMESTAMP,
last_seen TIMESTAMP,
sync_enabled BOOLEAN DEFAULT TRUE,
auto_sync BOOLEAN DEFAULT TRUE,
sync_frequency_minutes INTEGER DEFAULT 5,
device_metadata JSONB,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_devices_user_id ON devices(user_id);
CREATE INDEX idx_devices_device_type ON devices(device_type);
CREATE INDEX idx_devices_device_identifier ON devices(device_identifier);
-- ============================================
-- SYNC QUEUE FOR OFFLINE SUPPORT
-- ============================================
CREATE TABLE sync_queue (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
device_id UUID NOT NULL REFERENCES devices(id) ON DELETE CASCADE,
media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
sync_type VARCHAR(20) NOT NULL, -- 'progress', 'note', 'highlight', 'bookmark'
sync_data JSONB NOT NULL,
priority INTEGER DEFAULT 5,
attempts INTEGER DEFAULT 0,
max_attempts INTEGER DEFAULT 3,
status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'processing', 'completed', 'failed'
error_message TEXT,
created_at TIMESTAMP DEFAULT NOW(),
processed_at TIMESTAMP
);
CREATE INDEX idx_sync_queue_device_id ON sync_queue(device_id);
CREATE INDEX idx_sync_queue_status ON sync_queue(status);
CREATE INDEX idx_sync_queue_priority ON sync_queue(priority);
-- ============================================
-- CONFLICT RESOLUTION
-- ============================================
CREATE TABLE sync_conflicts (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
media_item_id UUID NOT NULL REFERENCES media_items(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
conflict_type VARCHAR(20) NOT NULL, -- 'progress', 'note', 'highlight'
conflict_data JSONB NOT NULL,
resolution_status VARCHAR(20) DEFAULT 'unresolved', -- 'unresolved', 'auto_resolved', 'user_resolved'
resolution_data JSONB,
resolved_by UUID REFERENCES users(id),
resolved_at TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_sync_conflicts_media_item_id ON sync_conflicts(media_item_id);
CREATE INDEX idx_sync_conflicts_user_id ON sync_conflicts(user_id);
CREATE INDEX idx_sync_conflicts_status ON sync_conflicts(resolution_status);
-- ============================================
-- READING HISTORY
-- ============================================
CREATE TABLE reading_history (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
media_item_id UUID NOT NULL REFERENCES media_items(id) ON DELETE CASCADE,
device_id UUID REFERENCES devices(id),
progress_percentage FLOAT,
reading_session_start TIMESTAMP,
reading_session_end TIMESTAMP,
pages_read INTEGER,
time_spent_seconds INTEGER,
device_metadata JSONB,
created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_reading_history_user_id ON reading_history(user_id);
CREATE INDEX idx_reading_history_media_item_id ON reading_history(media_item_id);
CREATE INDEX idx_reading_history_created_at ON reading_history(created_at DESC);
```
### Database Triggers for Automatic Timestamps
```sql
-- Update devices updated_at timestamp
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER update_devices_updated_at
BEFORE UPDATE ON devices
FOR EACH ROW
EXECUTE FUNCTION update_updated_at_column();
```
---
## API Endpoints Specification
### Authentication Endpoints
#### Device Registration Flow
```
POST /api/devices/register
Description: Register a new reading device
Request Body:
{
"device_name": "My Kobo Clara",
"device_type": "kobo|koreader|web|mobile",
"device_identifier": "unique-device-hardware-id"
}
Response (201):
{
"device_id": "uuid",
"auth_token": "Bearer token for device",
"setup_instructions": {
"kobo": "Sync URL: https://bookmann.example.com/api/sync/kobo",
"koreader": "Calibre URL: https://bookmann.example.com/api/sync/koreader"
},
"qr_code_url": "https://bookmann.example.com/devices/qr/uuid"
}
```
#### Web Login for Device Authentication
```
POST /api/devices/auth/web
Description: Authenticate device via web login (no API keys on device!)
Request Body:
{
"device_identifier": "device-unique-id",
"device_type": "kobo"
}
Response (200):
{
"auth_pending": true,
"auth_url": "https://bookmann.example.com/devices/auth/confirm/abc123",
"expires_in": 300,
"poll_interval": 3
}
[User visits auth URL on web, logs in, approves device]
Device polls:
POST /api/devices/auth/status
Response (200):
{
"auth_complete": true,
"auth_token": "device-bearer-token",
"device_id": "uuid",
"sync_endpoints": {
"progress": "https://bookmann.example.com/api/sync/progress",
"metadata": "https://bookmann.example.com/api/sync/metadata",
"annotations": "https://bookmann.example.com/api/sync/annotations"
}
}
```
### Wireless Sync Endpoints
#### KOReader Wireless Sync (Calibre-compatible)
```
POST /api/sync/koreader/progress
Description: KOReader sends progress update
Request Headers:
Authorization: Bearer {device_token}
Content-Type: application/json
Request Body:
{
"library_id": "optional-uuid",
"books": [
{
"uuid": "book-uuid",
"title": "Book Title",
"authors": ["Author Name"],
"progress": 0.45,
"percentage": 0.45,
"last_read": "2026-01-30T20:00:00Z",
"device_info": {
"koreader_version": "2024.01",
"device_model": "kindle-paperwhite-5"
}
}
]
}
Response (202):
{
"sync_status": "accepted",
"books_synced": 1,
"conflicts": [
{
"book_uuid": "book-uuid",
"conflict_type": "progress_mismatch",
"device_progress": 0.45,
"server_progress": 0.42,
"resolution": "device_wins"
}
]
}
GET /api/sync/koreader/metadata/{book_uuid}
Description: KOReader fetches book metadata and sync status
Response (200):
{
"uuid": "book-uuid",
"title": "Book Title",
"authors": ["Author Name"],
"progress": {
"percentage": 0.42,
"character": 15432,
"epubcfi": "epubcfi(/6/4/2:15)",
"chapter": 3,
"chapter_progress": 0.234
},
"annotations": {
"highlights": [...],
"notes": [...],
"bookmarks": [...]
},
"last_sync": "2026-01-30T20:00:00Z"
}
```
#### Kobo Wireless Sync (Kobo API-compatible)
```
POST /api/sync/kobo/markup
Description: Kobo sends reading progress and annotations
Request Headers:
Authorization: Bearer {device_token}
x-kobo-device: {"DeviceId":"device-id","Model":"Kobo Clara"}
Request Body:
{
"ReadingSync": [
{
"ContentId": "book-uuid",
"PercentRead": 45.6,
"EntitlementId": "entitlement-id",
"RemainingTimeMinutes": 120,
"LastModified": "2026-01-30T20:00:00Z"
}
],
"BookmarkSync": [
{
"ContentId": "book-uuid",
"BookmarkText": "highlighted text",
"BookmarkType": "annotation",
"BookmarkTitle": "Chapter 3"
}
]
}
Response (200):
{
"Status": "Success",
"MarkupsSynced": 5,
"BookmarksSynced": 3
}
GET /api/sync/kobo/library
Description: Kobo fetches library and sync status
Response (200):
{
"library_sync": [
{
"ContentId": "book-uuid",
"ContentType": "6",
"Title": "Book Title",
"Author": "Author Name",
"PercentRead": 42.3,
"PagesRemaining": 115,
"BookmarkCount": 3,
"LastModified": "2026-01-30T20:00:00Z"
}
]
}
```
### Universal Progress Endpoints
#### Get Universal Progress
```
GET /api/progress/{book_uuid}
Description: Get progress with all location references
Response (200):
{
"book_id": "book-uuid",
"format_group": "reflowable",
"universal_progress": 0.45678,
"location_references": {
"percentage": 0.45678,
"epubcfi": "epubcfi(/6/4/2:15)",
"character": 15432,
"chapter": 3,
"chapter_progress": 0.234,
"viewport_y": 0.12
},
"device_progress": {
"koreader": {
"percentage": 0.45678,
"last_sync": "2026-01-30T20:00:00Z"
},
"kobo": {
"percentage": 45.6,
"last_sync": "2026-01-30T19:55:00Z"
},
"web": {
"display_page": 89,
"total_pages": 200,
"last_sync": "2026-01-30T20:05:00Z"
}
},
"annotations": {
"highlights": [...],
"notes": [...],
"bookmarks": [...]
},
"conflicts": [
{
"id": "conflict-uuid",
"type": "progress",
"resolved": false,
"sources": ["koreader", "kobo"]
}
]
}
```
#### Update Universal Progress
```
POST /api/progress/{book_uuid}
Description: Update progress with automatic conversion to all formats
Request Body:
{
"source": "web|koreader|kobo|mobile",
"location": {
"percentage": 0.45678,
"epubcfi": "epubcfi(/6/4/2:15)",
"character": 15432,
"chapter": 3,
"page": 89,
"total_pages": 200
},
"device_metadata": {
"device_type": "web",
"user_agent": "..."
}
}
Response (200):
{
"sync_status": "success",
"progress_updated": true,
"devices_notified": ["koreader-device-1", "kobo-device-2"],
"broadcast": true
}
```
### Conflict Resolution Endpoints
#### List Conflicts
```
GET /api/conflicts
Description: List all unresolved sync conflicts
Query Parameters:
status: "unresolved|all"
type: "progress|note|highlight|all"
Response (200):
{
"conflicts": [
{
"id": "conflict-uuid",
"media_item_id": "book-uuid",
"media_item_title": "Book Title",
"conflict_type": "progress",
"conflict_data": {
"koreader": {
"source": "koreader",
"timestamp": "2026-01-30T20:10:00Z",
"data": {
"percentage": 0.45,
"epubcfi": "epubcfi(/6/4/2:15)",
"character": 15432
}
},
"kobo": {
"source": "kobo",
"timestamp": "2026-01-30T20:05:00Z",
"data": {
"percentage": 0.42,
"location": "unknown"
}
}
},
"resolution_status": "unresolved",
"created_at": "2026-01-30T20:10:05Z"
}
],
"total": 1,
"unresolved": 1
}
```
#### Resolve Conflict
```
POST /api/conflicts/{conflict_id}/resolve
Description: Resolve a sync conflict by choosing source
Request Body:
{
"winner": "koreader|kobo|web|manual",
"manual_data": {
"percentage": 0.43,
"epubcfi": "epubcfi(/6/4/2:20)",
"character": 15500
}, // Required if winner is "manual"
"apply_to_all_future_conflicts": false,
"reason": "user chose more recent progress"
}
Response (200):
{
"conflict_resolved": true,
"applied_to": {
"progress": true,
"annotations": false
},
"devices_synced": ["koreader-device-1", "kobo-device-2"]
}
```
### Device Management Endpoints
#### List User Devices
```
GET /api/devices
Response (200):
{
"devices": [
{
"id": "device-uuid",
"device_name": "My Kobo Clara",
"device_type": "kobo",
"last_sync": "2026-01-30T20:00:00Z",
"last_seen": "2026-01-30T20:05:00Z",
"sync_enabled": true,
"auto_sync": true
}
]
}
```
#### Update Device Settings
```
PUT /api/devices/{device_id}
Request Body:
{
"device_name": "Updated Name",
"sync_enabled": true,
"auto_sync": true,
"sync_frequency_minutes": 5
}
Response (200):
{
"device_updated": true
}
```
#### Revoke Device Access
```
DELETE /api/devices/{device_id}
Response (204)
```
---
## Format Grouping System
### Three Format Groups
Based on format capabilities and optimal tracking methods:
#### 1. Reflowable Group
**Formats**: EPUB, MOBI, AZW3, FB2, TXT
**Characteristics**:
- Text reflows to fit viewport
- Page count changes based on display settings
- Can contain complex formatting and images
- Support for chapters, TOC
**Optimal Tracking**:
- Percentage (0.0-1.0, 5 decimal precision)
- EPUB CFI for precise locations
- Chapter number + chapter progress
- Character offset from beginning
- Viewport Y position for visual reference
#### 2. Fixed Layout Group
**Formats**: PDF, DJVU
**Characteristics**:
- Fixed page layout (like printed book)
- Consistent page count across devices
- Zoom and pan for navigation
- May contain text layer
**Optimal Tracking**:
- Page number + total pages
- Page Y position (for scroll position)
- Zoom level
- Scroll X/Y coordinates
- Character offset in extracted text (if available)
#### 3. Comic Archive Group
**Formats**: CBZ, CBR, CBT, CB7, PDF comics
**Characteristics**:
- Image-based pages
- No text layer (usually)
- Panel-based reading
- Single/double page modes
- Panel zoom features
**Optimal Tracking**:
- Page number
- Panel number (for future panel navigation)
- Panel coordinates (for future use)
- Zoom level
- Pan X/Y coordinates
- Reading mode (single/double/panel)
### Format Detection Logic
```sql
CREATE OR REPLACE FUNCTION detect_format_group(mimetype VARCHAR, file_path VARCHAR)
RETURNS VARCHAR AS $$
BEGIN
CASE
-- Reflowable formats
WHEN mimetype = 'application/epub+zip' THEN
RETURN 'reflowable';
WHEN mimetype = 'application/x-mobipocket-ebook' THEN
RETURN 'reflowable';
WHEN mimetype = 'application/vnd.amazon.mobi8-ebook' THEN
RETURN 'reflowable';
WHEN file_path LIKE '%.epub' THEN
RETURN 'reflowable';
WHEN file_path LIKE '%.mobi' THEN
RETURN 'reflowable';
WHEN file_path LIKE '%.azw3' THEN
RETURN 'reflowable';
WHEN file_path LIKE '%.fb2' THEN
RETURN 'reflowable';
WHEN file_path LIKE '%.txt' THEN
RETURN 'reflowable';
-- Fixed layout formats
WHEN mimetype = 'application/pdf' THEN
RETURN 'fixed_layout';
WHEN file_path LIKE '%.pdf' THEN
RETURN 'fixed_layout';
WHEN file_path LIKE '%.djvu' THEN
RETURN 'fixed_layout';
-- Comic archive formats
WHEN mimetype = 'application/x-cbr' THEN
RETURN 'comic_archive';
WHEN mimetype = 'application/x-cbz' THEN
RETURN 'comic_archive';
WHEN file_path LIKE '%.cbz' THEN
RETURN 'comic_archive';
WHEN file_path LIKE '%.cbr' THEN
RETURN 'comic_archive';
WHEN file_path LIKE '%.cbt' THEN
RETURN 'comic_archive';
WHEN file_path LIKE '%.cb7' THEN
RETURN 'comic_archive';
ELSE
RETURN 'unknown';
END CASE;
END;
$$ LANGUAGE plpgsql;
```
---
## Progress Tracking by Format
### Reflowable Format Progress
```json
{
"format_group": "reflowable",
"book_id": "uuid",
"progress": {
"percentage": 0.45678,
"epubcfi": "epubcfi(/6/4[chap01ref]!/4[body01]/10[para05]/2:15)",
"character": 15432,
"chapter": 3,
"chapter_progress": 0.234,
"viewport_y": 0.12,
"total_characters": 34567
},
"conversion_rules": {
"to_percentage": "character / total_characters",
"to_epubcfi": "use stored CFI or calculate from character",
"to_chapter": "parse from CFI or calculate from character offset",
"from_koreader": "direct percentage mapping",
"from_kobo": "percentage / 100"
},
"display": {
"web": "45.7% (Chapter 3)",
"koreader": "45.7%",
"kobo": "46%"
}
}
```
### Fixed Layout Format Progress
```json
{
"format_group": "fixed_layout",
"book_id": "uuid",
"progress": {
"page": 45,
"total_pages": 200,
"page_y": 234,
"zoom": 1.25,
"scroll_x": 0,
"scroll_y": 0,
"character": 15432,
"percentage": 0.225
},
"conversion_rules": {
"to_percentage": "page / total_pages",
"to_page": "floor(percentage * total_pages)",
"to_character": "use OCR text extraction",
"from_koreader": "map percentage to nearest page",
"from_kobo": "direct page mapping"
},
"display": {
"web": "Page 45 of 200 (22.5%)",
"koreader": "22.5%",
"kobo": "Page 45 (23%)"
}
}
```
### Comic Archive Format Progress
```json
{
"format_group": "comic_archive",
"book_id": "uuid",
"progress": {
"page": 12,
"total_pages": 32,
"panel": 5,
"panel_bounds": "120,85,300,250",
"zoom": 1.5,
"pan_x": 120,
"pan_y": 85,
"reading_mode": "single_page",
"percentage": 0.375
},
"conversion_rules": {
"to_percentage": "page / total_pages",
"to_page": "floor(percentage * total_pages)",
"from_koreader": "percentage to page mapping",
"from_kobo": "page mapping"
},
"display": {
"web": "Page 12 of 32 (Panel 5)",
"koreader": "37.5%",
"kobo": "Page 12 (38%)"
}
}
```
### Progress Conversion Engine
```sql
CREATE OR REPLACE FUNCTION convert_progress(
source_format VARCHAR,
target_format VARCHAR,
source_progress JSONB
) RETURNS JSONB AS $$
DECLARE
result JSONB;
percentage FLOAT;
BEGIN
-- Extract percentage from source
CASE source_format
WHEN 'reflowable' THEN
percentage := (source_progress->>'percentage')::FLOAT;
WHEN 'fixed_layout' THEN
percentage := ((source_progress->>'page')::FLOAT /
(source_progress->>'total_pages')::FLOAT);
WHEN 'comic_archive' THEN
percentage := ((source_progress->>'page')::FLOAT /
(source_progress->>'total_pages')::FLOAT);
ELSE
percentage := 0.0;
END CASE;
-- Build target format progress
CASE target_format
WHEN 'reflowable' THEN
result := jsonb_build_object(
'percentage', percentage,
'character', CAST(percentage * 34567 AS INTEGER),
'epubcfi', 'epubcfi(/6/4/2:' || CAST(percentage * 100 AS INTEGER) || ')'
);
WHEN 'fixed_layout' THEN
result := jsonb_build_object(
'page', CAST(percentage * 200 AS INTEGER),
'total_pages', 200,
'percentage', percentage
);
WHEN 'comic_archive' THEN
result := jsonb_build_object(
'page', CAST(percentage * 32 AS INTEGER),
'total_pages', 32,
'percentage', percentage
);
ELSE
result := '{}'::jsonb;
END CASE;
RETURN result;
END;
$$ LANGUAGE plpgsql;
```
---
## Wireless Sync Protocols
### KOReader Calibre-Compatible Protocol
**Endpoint Structure**:
```
POST /api/sync/koreader/progress
POST /api/sync/koreader/bookmarks
POST /api/sync/koreader/highlights
GET /api/sync/koreader/metadata
GET /api/sync/koreader/status
```
**Authentication**: Bearer token from device registration
**Data Format** (KOReader → Bookmann):
```json
{
"library_id": "optional-uuid",
"books": [
{
"uuid": "book-uuid",
"title": "Book Title",
"authors": ["Author Name"],
"progress": 0.45,
"percentage": 0.45,
"last_read": "2026-01-30T20:00:00Z",
"bookmarks": [
{
"chapter": 3,
"datetime": "2026-01-30T19:55:00Z",
"notes": "highlighted text",
"pos0": "epubcfi(/6/4/2:15)",
"pos1": "epubcfi(/6/4/2:20)",
"page": 45,
"text": "highlighted text excerpt",
"type": "highlight"
}
],
"highlights": [...],
"notes": [...]
}
]
}
```
**Immediate Sync on Page Turn**:
```
POST /api/sync/koreader/progress
Content-Type: application/json
{
"sync_mode": "immediate",
"books": [
{
"uuid": "book-uuid",
"percentage": 0.45678,
"chapter": 3,
"timestamp": "2026-01-30T20:00:00Z"
}
]
}
Response (202 Accepted):
{
"sync_status": "accepted",
"broadcast_scheduled": true,
"queue_position": 1
}
```
**Checkpoint Sync (Fallback)**:
```
POST /api/sync/koreader/progress
Content-Type: application/json
{
"sync_mode": "checkpoint",
"checkpoint_id": "checkpoint-uuid",
"since_timestamp": "2026-01-30T19:00:00Z",
"books": [...]
}
Response (200 OK):
{
"sync_status": "completed",
"books_synced": 1,
"checkpoint_saved": true
}
```
### Kobo Sync Protocol (Reverse-Engineered)
**Endpoint Structure**:
```
POST /api/sync/kobo/markup
POST /api/sync/kobo/bookmark
GET /api/sync/kobo/library
GET /api/sync/kobo/status
```
**Authentication Headers**:
```
Authorization: Bearer {device_token}
x-kobo-userid: {user_id}
x-kobo-device: {"DeviceId":"device-id","Model":"Kobo Clara","SerialNumber":"..."}
```
**Data Format** (Kobo → Bookmann):
```json
{
"ReadingSync": [
{
"ContentId": "book-uuid",
"PercentRead": 45.6,
"EntitlementId": "entitlement-id",
"RemainingTimeMinutes": 120,
"FirstReadTime": "2026-01-25T10:00:00Z",
"LastModified": "2026-01-30T20:00:00Z"
}
],
"BookmarkSync": [
{
"BookmarkId": "bookmark-uuid",
"ContentId": "book-uuid",
"BookmarkText": "highlighted text",
"BookmarkType": "annotation",
"BookmarkTitle": "Chapter 3",
"DateCreated": "2026-01-30T19:55:00Z",
"Chapter": 3,
"Hidden": false
}
]
}
```
### Single Port Architecture
All sync endpoints share the same port (8765) with proper routing:
```
Port 8765 (Bookmann)
├── /api/auth/* (Authentication)
├── /api/libraries/* (Library management)
├── /api/media-items/* (Media items)
├── /api/sync/koreader/* (KOReader wireless sync)
├── /api/sync/kobo/* (Kobo wireless sync)
├── /api/progress/* (Universal progress API)
├── /api/conflicts/* (Conflict resolution)
├── /api/devices/* (Device management)
└── /ws/sync (WebSocket for real-time updates)
```
**Caddy Proxy Configuration**:
```
bookmann.example.com {
reverse_proxy localhost:8765
encode gzip
log {
output file /var/log/caddy/bookmann-access.log
}
}
```
---
## Security Architecture
### Device Authentication Flow
**No API Keys on Devices!** All authentication goes through web interface.
```
1. Device generates unique identifier (hardware ID)
2. Device sends registration request with device type
3. Bookmann creates pending registration with auth URL
4. User visits auth URL in web browser
5. User logs in and approves device
6. Bookmann generates device token
7. Device polls for token approval
8. Device receives token and begins syncing
```
### Authentication Endpoints
```
POST /api/devices/register/initiate
Request: { "device_identifier": "hw-id-123", "device_type": "kobo" }
Response: {
"registration_id": "reg-uuid",
"auth_url": "https://bookmann.example.com/devices/auth/confirm/reg-uuid",
"expires_in": 300,
"qr_code": "data:image/png;base64,..."
}
[User visits auth URL]
POST /api/devices/register/status
Request: { "registration_id": "reg-uuid" }
Response: {
"status": "pending" | "approved" | "expired",
"auth_token": "Bearer token..." (when approved)
}
```
### Token Management
**Device Token Structure**:
```json
{
"device_id": "uuid",
"user_id": "uuid",
"device_type": "kobo|koreader|web|mobile",
"permissions": [
"sync:progress",
"sync:annotations",
"sync:metadata"
],
"expires": "never",
"revocable": true
}
```
### Rate Limiting
```
Per-device rate limits:
- Sync requests: 60/minute
- Progress updates: 120/minute (page turns)
- Metadata requests: 30/minute
Per-user rate limits:
- All devices combined: 300/minute
- Conflict resolution: 10/minute
Implementation: Redis-backed sliding window
```
### Request Validation
```go
// Example Go validation for sync requests
func validateSyncRequest(deviceToken string, request SyncRequest) error {
// 1. Validate device token
device, err := validateDeviceToken(deviceToken)
if err != nil {
return ErrUnauthorized
}
// 2. Check device not revoked
if device.Revoked {
return ErrDeviceRevoked
}
// 3. Validate device permissions
if !hasPermission(device, "sync:progress") {
return ErrForbidden
}
// 4. Validate request format
if err := validateRequestFormat(request); err != nil {
return ErrInvalidRequest
}
// 5. Check rate limits
if exceededRateLimit(device.ID) {
return ErrRateLimited
}
// 6. Validate media item ownership
if !ownsMediaItem(device.UserID, request.BookUUID) {
return ErrNotFound
}
return nil
}
```
### CORS Configuration
```go
// Allow device origins
allowedOrigins := []string{
"https://bookmann.example.com",
"kobo://*", // Custom protocol for Kobo app
"koreader://*", // Custom protocol for KOReader
}
corsConfig := cors.Config{
AllowOrigins: allowedOrigins,
AllowMethods: []string{"GET", "POST", "PUT", "DELETE"},
AllowHeaders: []string{"Authorization", "Content-Type"},
ExposeHeaders: []string{"X-Sync-Status", "X-Conflict-Detected"},
AllowCredentials: true,
}
```
---
## Conflict Resolution System
### Conflict Detection
**Automatic Conflict Detection**:
```sql
CREATE OR REPLACE FUNCTION detect_conflict(
media_item_id UUID,
new_progress JSONB,
device_type VARCHAR
) RETURNS BOOLEAN AS $$
DECLARE
existing_progress JSONB;
time_diff INTERVAL;
BEGIN
-- Get most recent progress from different device
SELECT pg_jsonb INTO existing_progress
FROM reading_progress
WHERE media_item_id = $1
AND last_sync_source != device_type
ORDER BY last_sync_timestamp DESC
LIMIT 1;
-- If no existing progress, no conflict
IF existing_progress IS NULL THEN
RETURN FALSE;
END IF;
-- Check time difference (within 5 minutes = potential conflict)
time_diff := NOW() - (existing_progress->>'timestamp')::TIMESTAMP;
IF time_diff < INTERVAL '5 minutes' THEN
-- Check if progress significantly different
IF ABS((new_progress->>'percentage')::FLOAT -
(existing_progress->>'percentage')::FLOAT) > 0.01 THEN
RETURN TRUE;
END IF;
END IF;
RETURN FALSE;
END;
$$ LANGUAGE plpgsql;
```
### Side-by-Side Conflict Display
**UI Component Specification**:
```
┌─────────────────────────────────────────────────────────┐
│ ⚠️ Sync Conflict Detected: Book Title │
├─────────────────────────────────────────────────────────┤
│ │
│ Compare progress from different devices: │
│ │
│ ┌─────────────────┬─────────────────────────────────┐ │
│ │ KOReader │ Kobo Device │ │
│ ├─────────────────┼─────────────────────────────────┤ │
│ │ Progress: 45.7% │ Progress: 42.3% │ │
│ │ Location: │ Location: Page 89/200 │ │
│ │ Chapter 3 │ │ │
│ │ CFI: ... │ Last Read: 2 hours ago │ │
│ │ │ │ │
│ │ Last Read: │ │ │
│ │ 5 minutes ago │ │ │
│ │ │ │ │
│ │ Device: │ Device: Kobo Clara │ │
│ │ Kindle PW5 │ │ │
│ └─────────────────┴─────────────────────────────────┘ │
│ │
│ [Choose KOReader] [Choose Kobo] [Merge] [Dismiss] │
│ │
│ □ Auto-resolve future conflicts from [KOReader] │
│ │
└─────────────────────────────────────────────────────────┘
```
### Conflict Resolution Logic
**"Last In Wins" with User Override**:
```
1. Each sync writes timestamp and source device
2. If conflict detected:
- Compare timestamps
- More recent progress wins by default
- Create conflict record for user review
3. User can:
- Accept auto-resolution
- Choose different source manually
- Merge progress (take max of both)
- Set auto-resolution preference
```
**Merge Logic**:
```sql
CREATE OR REPLACE FUNCTION merge_progress(
progress_a JSONB,
progress_b JSONB
) RETURNS JSONB AS $$
BEGIN
RETURN jsonb_build_object(
'percentage', GREATEST(
(progress_a->>'percentage')::FLOAT,
(progress_b->>'percentage')::FLOAT
),
'merged_from', ARRAY[
(progress_a->>'source'),
(progress_b->>'source')
],
'merge_timestamp', NOW()
);
END;
$$ LANGUAGE plpgsql;
```
---
## Backup & Recovery Strategy
### Hybrid Queue + Checkpoint Strategy
**Immediate Sync (Page Turn)**:
```
Device → POST /api/sync/progress (sync_mode: immediate)
↓
Bookmann validates and stores
↓
WebSocket broadcast to all clients
↓
If network error → Queue locally with retry
```
**Checkpoint Sync (Fallback)**:
```
Device → POST /api/sync/progress (sync_mode: checkpoint)
↓
Bookmann validates and stores
↓
Process queued updates since last checkpoint
↓
Clear queue on success
↓
On failure → Keep in queue with exponential backoff
```
### Sync Queue Management
**Queue Schema**:
```sql
CREATE TABLE sync_queue (
id UUID PRIMARY KEY,
device_id UUID NOT NULL,
media_item_id UUID,
sync_type VARCHAR(20) NOT NULL,
sync_data JSONB NOT NULL,
priority INTEGER DEFAULT 5, -- 1=highest, 10=lowest
attempts INTEGER DEFAULT 0,
max_attempts INTEGER DEFAULT 3,
status VARCHAR(20) DEFAULT 'pending',
error_message TEXT,
created_at TIMESTAMP DEFAULT NOW(),
next_retry_at TIMESTAMP,
processed_at TIMESTAMP
);
```
**Priority Levels**:
```
Priority 1: User-initiated sync (manual refresh)
Priority 2: Book completion (100% progress)
Priority 3: Critical annotations (user notes)
Priority 5: Page turns (immediate sync)
Priority 7: Checkpoint sync (periodic)
Priority 10: Background metadata sync
```
**Exponential Backoff**:
```
Attempt 1: Immediate
Attempt 2: 1 minute delay
Attempt 3: 5 minute delay
Attempt 4: 15 minute delay
Attempt 5: 1 hour delay
After 5 failed attempts: Mark as failed, notify user
```
### Offline Capability
**Device-Side Queue** (KOReader example):
```lua
-- KOReader plugin code
local SyncQueue = {
queue = {},
max_queue_size = 100,
add = function(self, sync_data)
table.insert(self.queue, {
data = sync_data,
timestamp = os.time(),
retry_count = 0
})
self:persist()
end,
sync = function(self)
for i, item in ipairs(self.queue) do
local success = self:send_to_server(item.data)
if success then
table.remove(self.queue, i)
else
item.retry_count = item.retry_count + 1
if item.retry_count >= 3 then
self:notify_user(item)
table.remove(self.queue, i)
end
end
end
self:persist()
end,
persist = function(self)
local file = io.open("/mnt/us/sync-queue.json", "w")
file:write(json.encode(self.queue))
file:close()
end
}
```
### Recovery Scenarios
**Scenario 1: Device Offline for Extended Period**
```
1. Device queues all changes locally
2. User marks book as finished (100%)
3. High-priority sync queued
4. Device comes back online
5. Bookmann processes queue in priority order
6. Finishes book, updates all other devices
7. User notified of sync completion
```
**Scenario 2: Server Unavailable**
```
1. All devices detect server unavailable
2. Switch to checkpoint mode (batch every 5 min)
3. Queue changes locally
4. When server returns:
- Process all checkpoints
- Resolve conflicts based on timestamps
- Notify user of conflicts
```
**Scenario 3: Multiple Devices Offline**
```
1. All devices queue changes independently
2. Server processes queue in order of reconnection
3. Timestamps determine conflict resolution
4. User gets conflict summary for manual resolution
5. Auto-resolution preferences applied to future conflicts
```
---
## Real-time Synchronization
### WebSocket Architecture
**Connection Endpoint**:
```
WS /ws/sync?token={device_token}
```
**Message Format**:
```json
{
"type": "progress_update|annotation_update|conflict|sync_complete",
"timestamp": "2026-01-30T20:00:00Z",
"data": {
"book_id": "uuid",
"progress": {...},
"annotations": {...}
},
"source_device": {
"id": "device-uuid",
"name": "My Kobo",
"type": "kobo"
}
}
```
### Broadcast Strategy
```
Device A updates progress
↓
Bookmann stores in database
↓
WebSocket broadcast to:
- Web interface (if user logged in)
- Mobile apps (if connected)
- Other devices (on next sync)
↓
Devices update their display immediately
```
### Connection Management
**Device Connection States**:
```go
type ConnectionManager struct {
connections map[string]*DeviceConnection
broadcast chan BroadcastMessage
}
type DeviceConnection struct {
DeviceID string
UserID string
DeviceType string
Connected time.Time
LastPing time.Time
Send chan Message
}
func (m *ConnectionManager) HandleConnection(conn *websocket.Conn) {
device := m.authenticate(conn)
m.connections[device.ID] = device
// Send initial state
device.Send <- m.getInitialState(device.UserID)
// Handle messages
for {
msg := <-device.Send
err := conn.WriteJSON(msg)
if err != nil {
break
}
}
// Cleanup
delete(m.connections, device.ID)
}
```
### Heartbeat & Keep-Alive
```
Client → PING (every 30 seconds)
Server → PONG
If no PING for 90 seconds → Close connection
```
---
## Device Onboarding Flow
### Step-by-Step Device Setup
#### KOReader Setup
**1. Generate Registration Request**
```
Device sends: POST /api/devices/register
{
"device_type": "koreader",
"device_identifier": "hardware-specific-id",
"device_name": "My Kindle Paperwhite"
}
```
**2. Receive Registration Info**
```
Server responds: {
"registration_id": "reg-uuid-123",
"auth_url": "https://bookmann.example.com/devices/auth/confirm/reg-uuid-123",
"qr_code": "data:image/png;base64,iVBORw0KG...",
"expires_in": 300
}
```
**3. User Approves Device**
```
User visits auth URL → Sees:
┌─────────────────────────────────────┐
│ Device Registration Request │
│ │
│ Device Type: KOReader │
│ Device Name: My Kindle Paperwhite │
│ │
│ Approve this device to sync your │
│ reading progress and annotations? │
│ │
│ [Approve] [Deny] │
└─────────────────────────────────────┘
```
**4. Configure KOReader**
```
In KOReader settings:
- Calibre sync: ON
- Server URL: https://bookmann.example.com/api/sync/koreader
- Wireless sync: ON
- Sync frequency: Every page turn
- Auto-sync: ON
KOReader stores config in:
/mnt/us/settings/koreader/calibre.lua
```
**5. Device Polls for Token**
```
Device: POST /api/devices/register/status
{
"registration_id": "reg-uuid-123"
}
Server responds:
{
"status": "approved",
"auth_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
"sync_endpoints": {
"progress": "https://bookmann.example.com/api/sync/koreader/progress",
"metadata": "https://bookmann.example.com/api/sync/koreader/metadata",
"bookmarks": "https://bookmann.example.com/api/sync/koreader/bookmarks"
}
}
```
#### Kobo Setup
**1-3**: Same as KOReader (registration & approval)
**4. Configure Kobo Device**
```
In Kobo settings:
- Sync server: https://bookmann.example.com/api/sync/kobo
- Auto-sync: ON
- Sync frequency: Every 5 minutes
Kobo stores config in:
/mnt/onboard/.kobo/Kobo/Kobo eReader.conf
[Sync]
ServerURL=https://bookmann.example.com/api/sync/kobo
AutoSyncEnabled=true
SyncFrequency=5
```
**5. Device Syncs**
```
Kobo sends initial sync:
POST /api/sync/kobo/library
Authorization: Bearer {token}
x-kobo-device: {"DeviceId":"...", "Model":"Kobo Clara"}
Server responds with library and current sync state
```
### Device Discovery (Optional)
**mDNS/Bonjour for Local Network**:
```
Bookmann broadcasts: _bookmann-sync._tcp.local
Port: 8765
TXT: "path=/api/sync", "version=1.0"
Devices discover automatically
User confirms connection
```
---
## Implementation Phases
### Phase 1: Universal Progress System (Weeks 1-4)
**Week 1: Database Schema**
- Add format_group columns to media_items
- Add universal progress columns to reading_progress
- Create device registry tables
- Create sync_queue tables
- Create sync_conflicts tables
**Week 2: Format Detection & Progress Conversion**
- Implement format detection function
- Build progress conversion engine
- Update all existing progress to new format
- Create migration scripts
**Week 3: Core Progress APIs**
- GET /api/progress/{book_uuid}
- POST /api/progress/{book_uuid}
- Progress conversion logic
- Format-specific display logic
**Week 4: Testing & Validation**
- Unit tests for conversion engine
- Integration tests for progress APIs
- Test with existing data
- Performance testing
**Deliverables**:
- Universal progress tracking working
- All existing progress converted
- Format-aware display in web UI
### Phase 2: Device Management & Auth (Weeks 5-6)
**Week 5: Device Registration**
- Device registry implementation
- Registration endpoints
- Web-based approval flow
- QR code generation
- Device management UI
**Week 6: Authentication**
- Device token generation
- Token validation middleware
- Permission system
- Rate limiting per device
- Revocation system
**Deliverables**:
- Users can register devices
- Devices authenticate via web flow
- Device management UI complete
### Phase 3: KOReader Integration (Weeks 7-9)
**Week 7: KOReader Protocol**
- Implement calibre-compatible endpoints
- Progress sync endpoint
- Metadata sync endpoint
- Bookmark/highlight sync
**Week 8: Bidirectional Sync**
- Bookmann → KOReader sync
- KOReader → Bookmann sync
- Format conversion for KOReader
- Error handling & retry logic
**Week 9: Real-time Updates**
- WebSocket implementation
- Progress broadcast system
- Connection management
- Testing with actual KOReader devices
**Deliverables**:
- Full KOReader wireless sync
- Real-time progress updates
- Tested on actual devices
### Phase 4: Kobo Integration (Weeks 10-12)
**Week 10: Kobo Protocol**
- Reverse-engineer Kobo sync API
- Implement Kobo-compatible endpoints
- Markup sync endpoint
- Library sync endpoint
**Week 11: Bidirectional Sync**
- Bookmann → Kobo sync
- Kobo → Bookmann sync
- Format conversion for Kobo
- Annotation sync
**Week 12: Testing & Validation**
- Test on actual Kobo devices
- Performance testing
- Error handling
- Documentation
**Deliverables**:
- Full Kobo wireless sync
- Tested on Kobo Clara/Naura/etc.
- Device setup documentation
### Phase 5: Conflict Resolution (Weeks 13-14)
**Week 13: Conflict Detection**
- Automatic conflict detection
- Conflict storage
- Conflict notifications
- Conflict listing API
**Week 14: Resolution System**
- Side-by-side conflict UI
- Resolution endpoints
- Auto-resolution preferences
- Merge logic
**Deliverables**:
- Complete conflict resolution system
- User-friendly conflict UI
- Auto-resolution preferences
### Phase 6: Backup & Recovery (Weeks 15-16)
**Week 15: Sync Queue**
- Queue implementation
- Priority system
- Retry logic with exponential backoff
- Device-side queue (KOReader plugin)
**Week 16: Offline Support**
- Checkpoint sync mode
- Offline detection
- Recovery scenarios
- Progress merge logic
**Deliverables**:
- Robust offline sync
- Recovery from extended offline
- Queue management UI
### Phase 7: Polish & Documentation (Weeks 17-18)
**Week 17: Performance & Security**
- Performance optimization
- Security audit
- Rate limiting tuning
- Load testing
**Week 18: Documentation & Release**
- User documentation
- Developer documentation
- Device setup guides
- API documentation
- Release preparation
**Deliverables**:
- Production-ready system
- Complete documentation
- Device setup guides
- API reference
---
## Testing Requirements
### Unit Tests
**Format Detection**:
```go
TestFormatDetection_Epub()
TestFormatDetection_Mobi()
TestFormatDetection_Pdf()
TestFormatDetection_Cbz()
TestFormatDetection_Cbr()
TestFormatDetection_UnknownFormat()
```
**Progress Conversion**:
```go
TestProgressConversion_ReflowableToFixed()
TestProgressConversion_FixedToReflowable()
TestProgressConversion_ComicToReflowable()
TestProgressConversion_Precision()
TestProgressConversion_EdgeCases()
```
**Conflict Detection**:
```go
TestConflictDetection_SameTimestamp()
TestConflictDetection_DifferentDevices()
TestConflictDetection_SmallDifference()
TestConflictDetection_LargeDifference()
```
### Integration Tests
**Sync Flow**:
```go
TestSyncFlow_KOReaderToBookmann()
TestSyncFlow_BookmannToKOReader()
TestSyncFlow_KoboToBookmann()
TestSyncFlow_Bidirectional()
TestSyncFlow_MultipleDevices()
```
**Conflict Resolution**:
```go
TestConflictResolution_AutoResolve()
TestConflictResolution_ManualResolve()
TestConflictResolution_Merge()
TestConflictResolution_UserPreference()
```
**Offline Recovery**:
```go
TestOfflineRecovery_QueueProcessing()
TestOfflineRecovery_PriorityOrder()
TestOfflineRecovery_ExponentialBackoff()
TestOfflineRecovery_ExtendedOffline()
```
### End-to-End Tests
**User Scenarios**:
```go
TestScenario_ReadOnKOReaderContinueOnWeb()
TestScenario_ReadOnKoboContinueOnMobile()
TestScenario_MultipleDevicesConflict()
TestScenario_ExtendedOfflineSync()
TestScenario_BookCompletionAcrossDevices()
```
### Performance Tests
**Load Testing**:
```
- 1000 concurrent sync requests
- 100 devices syncing simultaneously
- 10,000 progress updates per minute
- WebSocket connection stability under load
```
**Database Performance**:
```
- Query performance with 1M+ progress records
- Index effectiveness
- Transaction throughput
- Conflict detection performance
```
---
## Performance Considerations
### Database Optimization
**Indexes**:
```sql
-- Critical indexes for sync performance
CREATE INDEX idx_reading_progress_media_user
ON reading_progress(media_item_id, user_id);
CREATE INDEX idx_reading_progress_last_sync
ON reading_progress(last_sync_timestamp DESC)
WHERE last_sync_timestamp > NOW() - INTERVAL '30 days';
CREATE INDEX idx_sync_queue_status_priority
ON sync_queue(status, priority)
WHERE status != 'completed';
CREATE INDEX idx_devices_active
ON devices(user_id, sync_enabled)
WHERE sync_enabled = true;
```
**Partitioning**:
```sql
-- Partition reading_history by month
CREATE TABLE reading_history_2026_01
PARTITION OF reading_history
FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');
```
### Caching Strategy
**Redis Cache**:
```
- Device authentication tokens (TTL: 24 hours)
- Active sync sessions (TTL: 1 hour)
- Progress hot data (TTL: 5 minutes)
- Conflict data (TTL: 1 hour)
```
**Cache Invalidation**:
```
- Progress update → invalidate progress cache
- New conflict → broadcast to all clients
- Device auth revoked → invalidate token cache
```
### Connection Pooling
**Database Connections**:
```
- Max connections: 100
- Min connections: 10
- Connection timeout: 30 seconds
- Query timeout: 10 seconds
```
**WebSocket Connections**:
```
- Max concurrent connections: 1000
- Heartbeat interval: 30 seconds
- Connection timeout: 90 seconds
- Message buffer size: 1000 messages
```
### Rate Limiting
**Per-Device Limits**:
```
- Sync requests: 60/minute
- Progress updates: 120/minute
- Metadata requests: 30/minute
- WebSocket messages: 300/minute
```
**Per-User Limits**:
```
- Total requests: 300/minute
- Conflict resolutions: 10/minute
- Device registrations: 5/hour
```
---
## Conclusion
This implementation guide provides a complete blueprint for creating a universal cross-platform reading synchronization system that:
1. **Tracks progress optimally** for each format (reflowable, fixed, comics)
2. **Syncs wirelessly** with KOReader and Kobo devices
3. **Resolves conflicts** intelligently with user control
4. **Handles offline scenarios** gracefully with queueing
5. **Updates in real-time** via WebSocket broadcasts
6. **Authenticates securely** via web interface (no device API keys)
7. **Scales efficiently** with proper caching and optimization
**Result**: Users get an Amazon Kindle-like experience - pick up any device, continue reading exactly where they left off, with all annotations synchronized - completely self-hosted and open-source.
---
## Appendix: Example Data Migrations
### Migrate Existing Page-Based Progress
```sql
-- Convert existing progress to new format
UPDATE reading_progress
SET
percentage = CASE
WHEN total_pages > 0 THEN current_page::FLOAT / total_pages::FLOAT
ELSE 0
END,
character_offset = current_page * 500, -- Rough estimate
chapter = CASE
WHEN current_page < 100 THEN 1
WHEN current_page < 200 THEN 2
ELSE 3
END,
chapter_progress = 0.5,
last_sync_source = 'migration',
last_sync_timestamp = NOW()
WHERE percentage IS NULL;
```
### Update Media Items with Format Groups
```sql
-- Set format groups based on file extensions
UPDATE media_items
SET
format_group = CASE
WHEN file_path LIKE '%.epub' THEN 'reflowable'
WHEN file_path LIKE '%.mobi' THEN 'reflowable'
WHEN file_path LIKE '%.pdf' THEN 'fixed_layout'
WHEN file_path LIKE '%.cbz' THEN 'comic_archive'
WHEN file_path LIKE '%.cbr' THEN 'comic_archive'
ELSE 'reflowable' -- Default
END,
is_reflowable = CASE
WHEN file_path LIKE '%.pdf' THEN FALSE
WHEN file_path LIKE '%.cbz' THEN FALSE
WHEN file_path LIKE '%.cbr' THEN FALSE
ELSE TRUE
END
WHERE format_group IS NULL OR format_group = '';
```
---
**Document Version**: 1.0
**Last Updated**: 2026-01-30
**Maintained By**: Bookmann Development Team
**Status**: Ready for Implementation
Reference in New Issue View Git Blame Copy Permalink