From e40be80e78ca7eddd9ab6a91056ef77335247499 Mon Sep 17 00:00:00 2001 From: John O'Keefe Date: Mon, 2 Nov 2026 14:55:16 -0500 Subject: [PATCH] docs(dev): websocket wire doc rewritten from source; endpoint doc ping claim fixed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Feb-era 700-line page documented client->server JSON messages the server never acted on (handleMessage logs only), a wrong ping/pong exchange, and missed every message type added since. Rewritten tight (~100 lines), every claim pinned: dual auth (JWT ?token= or device Bearer header), initial_state on connect under the connecting identity's RLS scope, the full 12-type message catalog (including collection_updated, found per-user-targeted in the collections handler and defined as a raw string outside the sync constants), WS-frame keepalive (30s server ping, 90s read deadline — browsers answer automatically), and the backpressure contract clients must honor: bounded per-connection queues skip messages when full, so the socket is a hint and REST refetch + fresh initial_state is the gap-healer. Fan-out documented precisely: sync/scan events instance-wide, collection events per-user. Also corrected the gated endpoint doc (api/websocket/protocol.md): its 'client sends {type:ping}, server responds pong' section contradicted the handler; replaced with the true keepalive story. Route pin untouched — the drift gates stay green. --- docs/developer/api/websocket/protocol.md | 15 +- docs/developer/websocket-api.md | 759 +++-------------------- 2 files changed, 82 insertions(+), 692 deletions(-) diff --git a/docs/developer/api/websocket/protocol.md b/docs/developer/api/websocket/protocol.md index d6b93f3c..624559a2 100644 --- a/docs/developer/api/websocket/protocol.md +++ b/docs/developer/api/websocket/protocol.md @@ -24,15 +24,16 @@ All messages are JSON objects with a `type` field. ### Client → Server Messages -#### Ping (Heartbeat) +None are interpreted. The socket is a one-directional push feed — the +server logs (but ignores) anything a client sends. All reads and writes +go through the REST API. -```json -{ - "type": "ping" -} -``` +#### Keepalive -Keep connection alive. Server responds with `pong`. +The server sends a WebSocket **ping frame every 30 seconds**; a +connection that misses pongs for 90 seconds is removed. Standard +WebSocket clients (browsers included) answer pongs automatically — +there is nothing to implement. ### Server → Client Messages diff --git a/docs/developer/websocket-api.md b/docs/developer/websocket-api.md index cf07a644..5e7b7faa 100644 --- a/docs/developer/websocket-api.md +++ b/docs/developer/websocket-api.md @@ -1,700 +1,89 @@ -# WebSocket API Documentation + -**Base URL**: `ws://localhost:8765/ws/sync` -**Protocol**: WebSocket (RFC 6455) -**Authentication**: Required (JWT token or device token) +# WebSocket API (real-time events) ---- +One channel, one direction: `/ws/sync` is a **server → client push feed** +for sync and scan events. It is not a command channel — clients do all +reading and writing through the REST API, and the server logs (but does +not act on) anything a client sends. The one client obligation is the +standard WebSocket pong, below. -## Table of Contents +## Connecting -1. [Authentication](#authentication) -2. [Connection](#connection) -3. [Message Format](#message-format) -4. [Message Types](#message-types) -5. [Client Implementation Guide](#client-implementation-guide) -6. [Examples](#examples) -7. [Troubleshooting](#troubleshooting) +`WS /ws/sync` — two accepted identities: ---- +| Method | How | +| --- | --- | +| User (JWT) | `?token=` query parameter | +| Device (KOReader, Kobo) | `Authorization: Bearer ` header | -## Authentication +No token → `401`. On connect the server immediately sends one +**`initial_state`** message: the connecting identity's media items for +sync, read under that identity's own data visibility. -### Option 1: JWT Token (Web/Mobile Clients) +## Message envelope -Pass JWT token as query parameter: - -``` -ws://localhost:8765/ws/sync?token= -``` - -**How to get JWT token**: - -```bash -curl -X POST http://localhost:8765/api/auth/login \ - -H "Content-Type: application/json" \ - -d '{"email":"user@example.com","password":"password"}' -``` - -Response: - -```json -{ - "token": "eyJhbGciOiJIUzI1NiIs...", - "user": { - "id": "user-uuid", - "email": "user@example.com" - } -} -``` - -### Option 2: Device Token (Kobo, KOReader, etc.) - -Pass device token in Authorization header: - -``` -ws://localhost:8765/ws/sync?token=any-value -Authorization: Bearer -``` - -**How to get device token**: -Device tokens are generated when devices are registered via the API. - ---- - -## Connection - -### Step 1: Connect to WebSocket - -**JavaScript Example**: - -```javascript -const token = "your-jwt-token"; -const ws = new WebSocket(`ws://localhost:8765/ws/sync?token=${token}`); - -ws.onopen = () => { - console.log("Connected to Bookhoard WebSocket"); -}; - -ws.onerror = (error) => { - console.error("WebSocket error:", error); -}; - -ws.onclose = (event) => { - console.log("WebSocket closed:", event.code, event.reason); -}; -``` - -**Go Example**: - -```go -import ( - "github.com/gorilla/websocket" -) - -url := "ws://localhost:8765/ws/sync?token=" + token -ws, _, err := websocket.DefaultDialer.Dial(url, nil) -if err != nil { - log.Fatal("Dial error:", err) -} -defer ws.Close() -``` - -**Python Example**: - -```python -import websocket - -token = "your-jwt-token" -url = f"ws://localhost:8765/ws/sync?token={token}" - -def on_message(ws, message): - print(f"Received: {message}") - -def on_error(ws, error): - print(f"Error: {error}") - -def on_close(ws, close_status_code, close_msg): - print("Closed") - -def on_open(ws): - print("Connected") - -ws = websocket.WebSocketApp( - url, - on_open=on_open, - on_message=on_message, - on_error=on_error, - on_close=on_close -) - -ws.run_forever() -``` - -### Step 2: Handle Initial State - -Immediately after connecting, you'll receive the initial state: - -```javascript -ws.onmessage = (event) => { - const message = JSON.parse(event.data); - - if (message.type === "initial_state") { - console.log("Initial progress:", message.data.progress); - console.log("Connected devices:", message.data.devices); - - // Store initial state - initialState = message.data; - } -}; -``` - -### Step 3: Handle Real-time Updates - -Listen for subsequent updates: - -```javascript -ws.onmessage = (event) => { - const message = JSON.parse(event.data); - - switch (message.type) { - case "progress_update": - handleProgressUpdate(message); - break; - case "annotation_update": - handleAnnotationUpdate(message); - break; - case "conflict": - handleConflict(message); - break; - case "heartbeat": - // Server ping, ignore - break; - default: - console.log("Unknown message type:", message.type); - } -}; - -function handleProgressUpdate(message) { - const { book_id, percentage } = message.data; - const source = message.source_device; - - console.log(`Progress update for ${book_id}: ${percentage * 100}%`); - console.log(`Source: ${source.name} (${source.type})`); - - // Update UI - updateProgressBar(book_id, percentage); -} - -function handleAnnotationUpdate(message) { - const { book_id, annotation_type, data } = message.data; - const source = message.source_device; - - console.log(`Annotation update for ${book_id}: ${annotation_type}`); - console.log(`Source: ${source.name}`); - - // Update UI - showAnnotation(book_id, data); -} - -function handleConflict(message) { - const { book_id, notification_type, conflict_id } = message.data; - - console.log(`Conflict detected for ${book_id}: ${notification_type}`); - - // Show conflict resolution UI - showConflictDialog(book_id, conflict_id); -} -``` - ---- - -## Message Format - -All messages follow this structure: - -```json -{ - "type": "message_type", - "timestamp": "2026-02-01T12:00:00Z", - "data": { - // Message-specific data - }, - "source_device": { - "id": "device-uuid", - "name": "Device Name", - "type": "kobo|koreader|web|mobile" - } -} -``` - -**Fields**: - -- `type` (string, required): Message type identifier -- `timestamp` (string, required): ISO 8601 timestamp -- `data` (object, required): Message payload -- `source_device` (object, optional): Device that sent the update (not present for initial_state) - ---- - -## Message Types - -### 1. initial_state - -**When**: Immediately after connection - -**Purpose**: Send current progress for all user's books - -**Data Structure**: - -```json -{ - "type": "initial_state", - "timestamp": "2026-02-01T12:00:00Z", - "data": { - "progress": { - "": { - "percentage": 0.5, - "current_page": 150, - "total_pages": 300, - "last_read": "2026-02-01T11:30:00Z" - } - }, - "devices": { - "kobo": 2, - "koreader": 1, - "web": 3 - } - } -} -``` - -**Fields**: - -- `progress` (object): Map of book UUID → progress data - - `percentage` (number): 0.0 to 1.0 - - `current_page` (number): Current page number - - `total_pages` (number): Total pages in book - - `last_read` (string): ISO 8601 timestamp of last read -- `devices` (object): Connection statistics by device type - -### 2. progress_update - -**When**: Any device syncs reading progress - -**Purpose**: Notify all connected clients of progress change - -**Data Structure**: +Every server message is a JSON object: ```json { "type": "progress_update", - "timestamp": "2026-02-01T12:00:00Z", - "data": { - "book_id": "book-uuid-1", - "percentage": 0.75 - }, - "source_device": { - "id": "device-uuid-1", - "name": "My Kobo Clara", - "type": "kobo" - } + "timestamp": "2026-11-02T12:00:00Z", + "data": { "...": "..." }, + "source_device": { "id": "…", "name": "Living-room Kobo", "type": "kobo" } } ``` -**Fields**: - -- `book_id` (string): UUID of book with updated progress -- `percentage` (number): New progress value (0.0 to 1.0) -- `source_device` (object): Device that sent the update - -**Use Cases**: - -- Update progress bar in real-time -- Sync reading position across devices -- Update "currently reading" lists - -### 3. annotation_update - -**When**: Any device syncs annotations, bookmarks, or highlights - -**Purpose**: Share annotations across devices - -**Data Structure**: - -```json -{ - "type": "annotation_update", - "timestamp": "2026-02-01T12:00:00Z", - "data": { - "book_id": "book-uuid-1", - "annotation_type": "bookmark", - "data": { - "page": 150, - "text": "Great quote on page 150", - "chapter": 5, - "created_at": "2026-02-01T12:00:00Z" - } - }, - "source_device": { - "id": "device-uuid-1", - "name": "My Kobo Clara", - "type": "kobo" - } -} -``` - -**Fields**: - -- `book_id` (string): UUID of book -- `annotation_type` (string): Type of annotation (bookmark, highlight, note) -- `data` (object): Annotation-specific data - - `page` (number): Page number - - `text` (string): Annotated text or note - - `chapter` (number): Chapter number (optional) - - `created_at` (string): ISO 8601 timestamp -- `source_device` (object): Device that created the annotation - -**Use Cases**: - -- Show bookmarks on all devices -- Share highlights between devices -- Display reading notes - -### 4. conflict - -**When**: Sync conflict is detected - -**Purpose**: Notify user of conflicting updates - -**Data Structure**: - -```json -{ - "type": "conflict", - "timestamp": "2026-02-01T12:00:00Z", - "data": { - "book_id": "book-uuid-1", - "notification_type": "progress_conflict", - "conflict_id": "conflict-uuid-1" - } -} -``` - -**Fields**: - -- `book_id` (string): UUID of book with conflict -- `notification_type` (string): Type of conflict (progress_conflict, annotation_conflict) -- `conflict_id` (string): UUID of conflict record - -**Use Cases**: - -- Prompt user to resolve conflict -- Show conflict resolution UI -- Log conflict for manual review - -### 5. heartbeat - -**When**: Server sends ping every 30 seconds - -**Purpose**: Keep connection alive - -**Data Structure**: - -```json -{ - "type": "heartbeat", - "timestamp": "2026-02-01T12:00:00Z" -} -``` - -**Handling**: Clients should respond automatically (handled by WebSocket protocol) - ---- - -## Client Implementation Guide - -### Reconnection Strategy - -Implement exponential backoff for reconnection: - -```javascript -let reconnectAttempts = 0; -const maxReconnectAttempts = 10; -const baseReconnectDelay = 1000; // 1 second - -function connect() { - const token = getAuthToken(); - const ws = new WebSocket(`ws://localhost:8765/ws/sync?token=${token}`); - - ws.onopen = () => { - console.log("Connected"); - reconnectAttempts = 0; // Reset on successful connection - }; - - ws.onclose = (event) => { - if (reconnectAttempts < maxReconnectAttempts) { - const delay = baseReconnectDelay * Math.pow(2, reconnectAttempts); - console.log(`Reconnecting in ${delay}ms...`); - - setTimeout(() => { - reconnectAttempts++; - connect(); - }, delay); - } else { - console.error("Max reconnection attempts reached"); - } - }; - - ws.onmessage = handleIncomingMessage; - - return ws; -} -``` - -### Error Handling - -Handle common errors: - -```javascript -ws.onerror = (error) => { - console.error("WebSocket error:", error); - - // Check specific error types - if (error.code === 1006) { - console.error("Abnormal closure - server may be down"); - } else if (error.code === 1008) { - console.error("Policy violation - check authentication"); - } -}; -``` - -### Message Queueing - -Queue messages while disconnected: - -```javascript -let messageQueue = []; -let isConnected = false; - -function queueMessage(message) { - if (isConnected) { - ws.send(JSON.stringify(message)); - } else { - messageQueue.push(message); - } -} - -ws.onopen = () => { - isConnected = true; - - // Send queued messages - while (messageQueue.length > 0) { - const message = messageQueue.shift(); - ws.send(JSON.stringify(message)); - } -}; - -ws.onclose = () => { - isConnected = false; -}; -``` - ---- - -## Examples - -### Complete JavaScript Client - -```javascript -class BookhoardWebSocket { - constructor(token) { - this.token = token; - this.ws = null; - this.reconnectAttempts = 0; - this.maxReconnectAttempts = 10; - this.messageHandlers = {}; - } - - connect() { - const url = `ws://localhost:8765/ws/sync?token=${this.token}`; - this.ws = new WebSocket(url); - - this.ws.onopen = () => { - console.log("Connected to Bookhoard"); - this.reconnectAttempts = 0; - }; - - this.ws.onmessage = (event) => { - const message = JSON.parse(event.data); - this.handleMessage(message); - }; - - this.ws.onerror = (error) => { - console.error("WebSocket error:", error); - }; - - this.ws.onclose = () => { - this.reconnect(); - }; - } - - handleMessage(message) { - const handler = this.messageHandlers[message.type]; - if (handler) { - handler(message); - } - } - - on(messageType, callback) { - this.messageHandlers[messageType] = callback; - } - - reconnect() { - if (this.reconnectAttempts < this.maxReconnectAttempts) { - const delay = 1000 * Math.pow(2, this.reconnectAttempts); - console.log(`Reconnecting in ${delay}ms...`); - - setTimeout(() => { - this.reconnectAttempts++; - this.connect(); - }, delay); - } - } - - disconnect() { - if (this.ws) { - this.ws.close(); - } - } -} - -// Usage -const token = "your-jwt-token"; -const client = new BookhoardWebSocket(token); - -client.on("initial_state", (message) => { - console.log("Initial state received:", message.data); - updateUIWithInitialState(message.data); -}); - -client.on("progress_update", (message) => { - console.log("Progress updated:", message.data); - updateProgressBar(message.data.book_id, message.data.percentage); -}); - -client.on("annotation_update", (message) => { - console.log("Annotation updated:", message.data); - showNotification("New annotation from " + message.source_device.name); -}); - -client.on("conflict", (message) => { - console.log("Conflict detected:", message.data); - showConflictDialog(message.data); -}); - -client.connect(); -``` - ---- - -## Troubleshooting - -### Connection Fails - -**Problem**: Can't connect to WebSocket - -**Solutions**: - -1. Check JWT token is valid -2. Verify server is running: `curl http://localhost:8765/health` -3. Check firewall settings -4. Verify WebSocket URL format - -### Connection Drops Frequently - -**Problem**: WebSocket disconnects unexpectedly - -**Solutions**: - -1. Check network stability -2. Verify server keepalive settings -3. Implement reconnection logic (see above) -4. Check server logs for errors - -### No Messages Received - -**Problem**: Connected but no messages - -**Solutions**: - -1. Check onmessage handler is registered -2. Verify initial_state message received -3. Test with manual progress update via API -4. Check browser console for errors - -### Authentication Errors - -**Problem**: 401 Unauthorized - -**Solutions**: - -1. Verify JWT token is not expired -2. Check token has correct claims (user_id) -3. For device tokens, verify Authorization header format -4. Regenerate token via `/api/auth/login` - -### High Memory Usage - -**Problem**: Client memory increases over time - -**Solutions**: - -1. Clean up old messages -2. Don't store entire message history -3. Use weak references for large data -4. Implement message pagination - ---- - -## Performance Tips - -1. **Debounce UI Updates**: Don't update DOM on every message - - ```javascript - const debouncedUpdate = debounce(updateUI, 100); - client.on("progress_update", (message) => { - debouncedUpdate(message); - }); - ``` - -2. **Use Web Workers** for heavy processing -3. **Limit Message History**: Keep only last N messages -4. **Virtual Scrolling**: For large lists of updates -5. **Connection Pooling**: Reuse connections across tabs - ---- - -## Security Best Practices - -1. **Use WSS in Production**: Always use secure WebSocket (wss://) in production -2. **Validate Tokens**: Always verify JWT tokens on server -3. **Rate Limiting**: Implement message rate limiting per connection -4. **Input Sanitization**: Sanitize all message data before displaying -5. **Origin Checks**: Validate WebSocket origin on server - ---- - -## Support - -For issues or questions: - -- GitHub Issues: [Bookhoard Repository] -- Documentation: [Bookhoard Docs] -- API Reference: [Bookhoard API Docs] - ---- - -**Last Updated**: 2026-02-01 -**Version**: 1.0.0 -**Protocol**: WebSocket (RFC 6455) +`source_device` is present on events that originate from a device +(progress and annotation updates), so clients can ignore their own echo. + +## Message catalog + +All ten types defined by `internal/sync/websocket.go`: + +| Type | Emitted | `data` carries | +| --- | --- | --- | +| `initial_state` | Once, on connect | The connecting user's media items for sync | +| `progress_update` | On any device's progress save | `book_id`, `percentage` | +| `annotation_update` | On annotation changes (highlights, notes, bookmarks) | `book_id`, `annotation_type`, payload | +| `conflict` | When a sync conflict is detected | `book_id`, `notification_type`, `conflict_id` | +| `scan_progress` | During scan jobs | Job progress for the admin scan panel | +| `scan_complete` | When a scan job finishes | Job summary | +| `scan_error` | When a scan job fails | Job error | +| `scan_cancelled` | When a scan job is cancelled | Job identity | +| `collection_updated` | When books are added to / removed from one of *your* collections | `collection_id`, `action` (`books_added` / `book_removed` / `books_removed`), affected book ids | +| `heartbeat` | *Defined, not currently emitted* | — | +| `sync_complete` | *Defined, not currently emitted* | — | + +(`collection_updated` is the one per-user-targeted event — it's emitted +with the connecting user's id directly. It's also currently the only +message type defined as a raw string in the collections handler rather +than in the sync package's constants block.) + +## Keepalive + +The server sends a WebSocket **ping frame every 30 seconds** and expects +the standard pong within a **90-second read deadline**. Browsers and +every mainstream WebSocket library answer pings automatically — there is +nothing to implement. A connection that stops answering is removed. + +## Fan-out and backpressure — read this before building a client + +- Sync and scan events fan out **instance-wide**: every connected client + receives progress, annotation, conflict, and scan events. The + exception is `collection_updated`, which is targeted to one user's + connections. (A per-user targeting helper exists and is used for + collections only — the sync/scan emitters do not use it.) +- Every connection has a bounded send queue. **If a client falls behind, + messages for that connection are skipped, not queued** (and the global + broadcast ring drops messages when saturated). Clients must therefore + treat the socket as a *hint* — on reconnect (or any gap), refetch real + state over REST. The fresh `initial_state` on reconnect is designed to + be that gap-healer.