bookhoard/IMPLEMENTATION_PLAN.md

Bookhoard Device Authentication & Sync Implementation Plan

Executive Summary

Current Problem:

• OPDS routes are publicly accessible (security vulnerability)
• Kobo sync uses Bearer tokens but Kobo firmware can't send them (broken)
• KOReader documentation is incorrect about Basic Auth
• No clear authentication strategy for different device types
• Sidecar implementation exists but isn't integrated

Vision: Transform Bookhoard into a comprehensive Kindle-replacement ecosystem with seamless device sync, supporting both Kobo (native) and KOReader (via plugin) with full functionality including progress, highlights, notes, and bookmarks.

---

1. Current State Analysis

1.1 Authentication Landscape

Existing Infrastructure:

// DeviceAuthMiddleware - Currently only supports Bearer tokens
// File: internal/middleware/device_auth.go:37-108
func (m *DeviceAuthMiddleware) Authenticate(next echo.HandlerFunc) echo.HandlerFunc {
    // Only checks Authorization: Bearer {token}
    // Validates against devices.auth_token
}

Routes Using Device Auth:

• /api/sync/kobo/* - Kobo sync endpoints
• /api/sync/koreader/* - KOReader sync endpoints
• /opds/devices/* - OPDS catalog (currently NOT protected, needs auth)

Database Schema:

-- devices table already has device_identifier field
-- Can store serial numbers for Kobo devices

1.2 Kobo Integration Status

What's Broken:

• Kobo sends Authorization: Bearer {kobo-store-token} (wrong token)
• Kobo sends x-kobo-device: {"SerialNumber":"N123..."} (what we need to use)
• Current middleware rejects Kobo requests because token doesn't match

What's Working:

• Kobo sync handler implementation exists
• Endpoints registered: /api/sync/kobo/markup, /bookmark, /v1/initialization, etc.
• Can sync progress, highlights, notes, bookmarks once auth works

1.3 KOReader Integration Status

What's Broken:

• Documentation says use "Basic Auth" - this is WRONG
• KOReader doesn't support custom headers natively for sync
• Bearer token can't be sent via standard KOReader settings

What's Working:

• KOReader sync handler supports full feature set
• Can sync: progress, bookmarks, highlights, notes, library metadata
• Plugin architecture exists but no Bookhoard plugin yet

1.4 OPDS Status

Current:

• Routes registered in internal/router/opds.go:11-12
• NOT using DeviceAuthMiddleware (security hole)
• Publicly accessible - anyone can enumerate device UUIDs

Required:

• Must require authentication
• Needs to work with both Kobo (serial) and KOReader (Bearer token)

---

2. User Priorities (Ranked)

Priority 1: Functionality First

User Quote: "I want a Kindle system replacement, otherwise I could just use Booklore or KOReader"

Requirements:

• ✅ Full sync: progress, highlights, notes, bookmarks
• ✅ Works on both Kobo and KOReader
• ✅ Seamless user experience
• ✅ OPDS for wireless book delivery
• ✅ Two-way sync (device ↔ server)

Priority 2: Security Second (with documentation)

User Quote: "This is a self-hosted app so as long as the user is aware of the security implications in the documentation it should be fine"

Acceptable Trade-offs:

• Serial-based auth for Kobo (less secure but easier)
• Security risks documented clearly
• Token-based for KOReader (more secure)
• Users responsible for network security (VPN, etc.)

Priority 3: Simple Setup

Goal: Minimal friction for users

Kobo Experience:

• Enter serial number during registration
• One line in Kobo eReader.conf
• No token management
• Works immediately

KOReader Experience:

• Copy plugin files
• Enter server URL and token
• One-time setup
• Background sync works automatically

---

3. Target State

3.1 Unified Authentication Architecture

┌─────────────────────────────────────────────────────────────┐
│                  DeviceAuthMiddleware                       │
│  Enhanced to support multiple auth methods:                 │
│                                                             │
│  1. Authorization: Bearer {token}                          │
│     → Lookup device by auth_token                          │
│     → Used by: KOReader, OPDS apps                         │
│                                                             │
│  2. X-Kobo-Device: {"SerialNumber":"N123..."}              │
│     → Extract serial from JSON                             │
│     → Lookup device by device_identifier                   │
│     → Used by: Kobo e-readers                              │
│                                                             │
│  Both paths result in: device context set in echo.Context   │
└─────────────────────────────────────────────────────────────┘

3.2 Feature Matrix

Feature	Kobo (Native)	KOReader (Plugin)
Progress Sync	✅ Automatic	✅ Automatic
Highlights	✅ Native support	✅ Via plugin API
Bookmarks	✅ Native support	✅ Via plugin API
Notes	✅ Native support	✅ Via plugin API
OPDS	✅ Via NickelMenu	✅ Native OPDS support
Setup Complexity	Low (1 config line)	Medium (plugin install)
Auth Method	Serial number	Bearer token
Security	Medium (documented)	High

3.3 User Flows

Kobo User Journey:

1. Install NickelMenu (one-time, jailbreak)
2. Open Bookhoard web UI
3. Register device → enter serial number (from Settings)
4. Approve device
5. Edit Kobo eReader.conf → add sync URL
6. Read books → automatic sync
7. Access OPDS via NickelMenu

KOReader User Journey:

1. Install KOReader on device
2. Open Bookhoard web UI
3. Register device → copy auth token
4. Approve device
5. Install Bookhoard plugin → paste token
6. Configure sync settings
7. Read books → automatic sync
8. Add OPDS catalog → browse library

---

4. Implementation Plan

Phase 1: Enhanced Authentication (Week 1)

Goal: Make auth work for both Kobo and KOReader

Tasks:

1. Update DeviceAuthMiddleware (internal/middleware/device_auth.go)

   • Try Bearer token lookup first (existing behavior)
   • If no Bearer token, check for x-kobo-device header
   • Parse JSON to extract SerialNumber field
   • Query database: GetDeviceByDeviceIdentifier(serial)
   • Set device context on successful auth
   • Return 401 if both methods fail

2. Add Database Query (internal/database/queries/devices.sql)

   • Add query: GetDeviceByDeviceIdentifier
   • Ensure index exists on device_identifier column

3. Update Device Registration UI (templates/devices.templ)

   • Add device type selector (Kobo/KOReader)
   • Show serial number field for Kobo (required)
   • Show serial number field for KOReader (optional, for fallback)
   • Generate/display auth token after approval (for KOReader)

4. Apply Middleware to OPDS (internal/router/opds.go)

   • Add DeviceAuthMiddleware to all /opds/devices/* routes
   • Update comments to reflect auth requirements

Testing:

• Kobo device sync with serial auth
• KOReader sync with Bearer token auth
• OPDS access with both auth methods

---

Phase 2: Kobo Integration (Week 1-2)

Goal: Seamless Kobo experience with serial-based auth

Tasks:

1. Kobo Setup Documentation (docs/user/devices/kobo-setup.md)

   • Revise to reflect serial-based auth (not username/password)
   • Step-by-step guide for finding serial number
   • NickelMenu configuration for sync
   • Security warning about serial-based auth
   • Network security recommendations (VPN, local network)

2. Kobo Sync Validation (cmd/server/tests/kobo_test.go)

   • Update tests to use serial auth
   • Test sync endpoints with x-kobo-device header
   • Verify all features work: progress, highlights, bookmarks, notes

3. Kobo OPDS Integration

   • Document OPDS URL format
   • Test OPDS browsing with serial auth
   • Test book downloads

Success Criteria:

• New Kobo device registered and syncing within 5 minutes
• All sync features working without token management
• Security implications clearly documented

---

Phase 3: KOReader Plugin Development (Week 2-4)

Goal: Full-featured KOReader plugin with Bearer token auth

Tasks:

1. Plugin Architecture

   • Create koreader-plugin/bookhoard.koplugin/ directory
   • Implement _meta.lua (plugin metadata)
   • Implement main.lua (entry point)

2. Core Plugin Features

   • Settings UI: server URL, auth token, sync options
   • HTTP client with Bearer token authentication
   • Auto-sync on events:
     • Page turn (configurable: every N pages)
     • Bookmark added/deleted
     • Highlight created/deleted
     • Note added/edited
   • Manual sync button in KOReader menu
   • Sync status indicator
   • Error handling and retry logic

3. Plugin API Integration

   • Call /api/sync/koreader/progress with reading position
   • Call /api/sync/koreader/bookmarks with annotations
   • Call /api/sync/koreader/metadata to get server-side progress
   • Handle conflicts (server vs local progress)

4. Plugin Documentation

   • Installation guide (copy files, restart KOReader)
   • Configuration guide (get token from Bookhoard UI)
   • Usage instructions
   • Troubleshooting guide

Success Criteria:

• Plugin installed and configured in under 10 minutes
• Background sync works transparently
• All annotation types sync properly
• No Lua errors or crashes

---

Phase 4: OPDS Security & Integration (Week 2-3)

Goal: Secure OPDS with unified device authentication

Tasks:

1. Secure OPDS Routes (internal/router/opds.go)

   • Apply DeviceAuthMiddleware to all OPDS endpoints
   • Ensure auth works with both serial and Bearer methods
   • Update router comments

2. Kobo OPDS Access

   • Document OPDS URL with device ID
   • Test with NickelMenu OPDS browser
   • Test book downloads

3. KOReader OPDS Access

   • Works natively via KOReader's OPDS support
   • Uses Bearer token in header
   • Document catalog URL format

4. Test Coverage (cmd/server/tests/opds_test.go)

   • Test unauthenticated access → expect 401
   • Test with valid Bearer token → expect 200
   • Test with valid serial header → expect 200
   • Test book download with auth

---

Phase 5: Testing & Documentation (Week 4)

Goal: Production-ready release with comprehensive docs

Tasks:

1. End-to-End Testing

   • Fresh Kobo setup: register → sync → highlights → OPDS
   • Fresh KOReader setup: install plugin → sync → highlights → OPDS
   • Cross-device sync: Kobo ↔ KOReader same book
   • Offline scenarios and conflict resolution
   • Error handling and recovery

2. Security Documentation

   • Security implications page
   • Network security recommendations
   • Comparison: Bearer vs Serial auth
   • Threat model for self-hosted users

3. User Documentation

   • Quick start guide (Kobo)
   • Quick start guide (KOReader)
   • Feature comparison table
   • FAQ and troubleshooting

4. Release Preparation

   • Version bump
   • Changelog
   • Migration guide (if needed)

---

5. Technical Decisions

5.1 Why Serial-Based Auth for Kobo?

Pros:

• No token management required
• Works with unmodified Kobo firmware
• Single configuration line
• User can't lose/forget credentials

Cons:

• Serial numbers are somewhat predictable
• No token rotation possible
• If leaked, can't be revoked (must delete device)

Mitigations:

• Document security risks
• Recommend VPN for remote access
• Use HTTPS (reverse proxy)
• Acceptable for self-hosted personal use

5.2 Why Bearer Token for KOReader?

Pros:

• Cryptographically secure
• Can be revoked/regenerated
• Standard authentication method
• Works with KOReader plugin architecture

Cons:

• Requires plugin (can't use native settings)
• Token must be copy-pasted during setup
• User must manage token

Trade-off Accepted: More secure but requires plugin installation. Justified because KOReader users are typically more technical and willing to install plugins.

5.3 Why Not Use Kobo's Token?

Kobo sends Authorization: Bearer {kobo-store-token} which is:

• Managed by Kobo (not under our control)
• May rotate/expire without warning
• Only valid for Kobo's servers
• Doesn't identify the device in our system

Decision: Parse the x-kobo-device header for serial number instead.

---

6. Files to Modify

Core Authentication

• internal/middleware/device_auth.go - Enhance with serial support
• internal/database/queries/devices.sql - Add serial lookup query
• internal/router/opds.go - Apply middleware

Device Management

• templates/devices.templ - Update registration form
• internal/handlers/devices.go - Handle serial input

Testing

• cmd/server/tests/kobo_test.go - Update for serial auth
• cmd/server/tests/opds_test.go - Add auth tests
• cmd/server/tests/test_helpers.go - Add serial auth helpers

Documentation

• docs/user/devices/kobo-setup.md - Rewrite for serial auth
• docs/user/devices/koreader-setup.md - Update for Bearer token
• docs/user/security.md - New security implications doc
• koreader-plugin/README.md - Plugin documentation

New Files

• koreader-plugin/bookhoard.koplugin/_meta.lua - Plugin metadata
• koreader-plugin/bookhoard.koplugin/main.lua - Plugin implementation
• docs/user/quickstart-kobo.md - Quick start guide
• docs/user/quickstart-koreader.md - Quick start guide

---

7. Success Metrics

Functionality:

• ✅ Kobo syncs progress, highlights, notes, bookmarks automatically
• ✅ KOReader syncs same features via plugin
• ✅ OPDS works on both platforms
• ✅ Cross-device sync works (read on Kobo, continue on KOReader)

Usability:

• ✅ Kobo setup time: < 5 minutes
• ✅ KOReader setup time: < 10 minutes
• ✅ No token management for Kobo users
• ✅ Clear documentation with security warnings

Security:

• ✅ OPDS no longer publicly accessible
• ✅ Auth required for all sync endpoints
• ✅ Security implications documented
• ✅ Network security recommendations provided

---

8. Future Enhancements (Out of Scope)

• Auto-discovery of devices on network
• QR code setup for mobile devices
• Background sync service (no plugin needed)
• Readwise/Joplin integration
• Calibre integration
• Mobile apps (iOS/Android)

---

9. Decisions Made (User Input)

Based on user feedback, the following decisions have been finalized:

9.1 KOReader Sync Trigger

Decision: Configurable with multiple options

• Auto-sync: Every N pages (configurable, default every 10 pages)
• Auto-sync: Every N minutes (configurable, default every 5 minutes)
• Manual sync: Always available via menu
• Event-triggered: On bookmark/highlight/note creation (immediate)

Rationale: User wants automatic sync for convenience but manual fallback for control.

9.2 Plugin Distribution

Decision: Separate repository under Bookhoard organization

• Repository: github.com/bookhoard/koreader-plugin
• Licensed under same terms as Bookhoard
• Versioned independently
• Referenced in main Bookhoard documentation

Rationale: Clean separation of concerns, easier plugin-specific issues/PRs.

9.3 Kobo Authentication

Decision: Serial number ONLY

• No Bearer token fallback for Kobo
• Serial entered during device registration
• Stored in devices.device_identifier

Rationale: Simplest user experience, one auth method per platform.

9.4 Conflict Resolution

Decision: Use existing codebase strategy

• Already implemented in internal/handlers/koreader.go
• Uses timestamp-based last-write-wins
• Device priority can be configured per-device

Rationale: Don't reinvent the wheel, existing implementation is sufficient.

9.5 Implementation Order

Decision: Parallel implementation with separate sections

• Phase 2a: Kobo Integration
• Phase 2b: KOReader Plugin (runs concurrently)
• Separate milestones and deliverables
• Can ship Kobo support before plugin is ready

Rationale: Faster time-to-market for Kobo users, plugin can follow.

---

10. Historical Context & Conversation Summary

Why This Plan Exists

This implementation plan emerged from a detailed analysis of the current authentication and sync architecture. Key discoveries from codebase review:

The Authentication Problem

Original Issue: The plan started with @fix-opds-device-authentication.md which identified:

• OPDS routes were publicly accessible (security vulnerability)
• Test GetDeviceCatalog_WithoutDeviceAuth expected 401 but got 404
• DeviceAuthMiddleware was already applied to OPDS routes, but tests were failing

Root Cause Discovery:

• Kobo devices send Authorization: Bearer {kobo-store-token} (Kobo's store token, not Bookhoard's)
• Kobo also sends x-kobo-device: {"SerialNumber":"N123..."} header
• The middleware was rejecting requests because the Bearer token didn't match devices.auth_token
• Kobo cannot send custom Bearer tokens through eReader.conf

The Sidecar File Red Herring

Initial Thought: Use .bookhoard.json sidecar file to pass device tokens to Kobo

Problem Discovered:

• Sidecar handler exists (internal/handlers/sidecar.go) but routes are NOT registered
• Kobo firmware is locked down - cannot read custom config files
• No plugin architecture on Kobo (unlike KOReader)
• Would require Kobo firmware modification (impossible for most users)

Decision: Abandon sidecar approach, use serial-based auth instead

KOReader Documentation Bug

Original Documentation: Claimed KOReader uses "Basic Auth" with username/password

Reality:

• KOReader sync sends Authorization: Bearer {token} to sync servers
• The official KOReader sync protocol uses MD5 hashed passwords
• Bookhoard's implementation expects Bearer tokens in DeviceAuthMiddleware
• KOReader cannot send custom headers through native settings

Solution: Create KOReader plugin that can send proper Bearer tokens

Booklore Analysis

Investigation: Analyzed how Booklore handles Kobo authentication

Booklore Approach:

• Token embedded in URL path: /api/kobo/{token}/...
• Works but token appears in logs
• One token per user (not per device)
• OPDS uses separate Basic Auth

Decision: Don't copy Booklore - less secure. Use device-specific auth with proper tokens for KOReader.

Plugin Feasibility Research

Concern: Is KOReader a "moving target" for plugin development?

Findings:

• UI plugins are unstable (Issue #13942) - frequent breaking changes
• Backend/sync plugins (like Wallabag) are stable
• Wallabag2 plugin has minimal updates over years
• HTTP client APIs don't change often

Decision: Proceed with plugin - sync plugins are low-risk

Technical Constraints Discovered

1. Kobo Firmware Limitations:

   • Cannot send custom HTTP headers (except x-kobo-device)
   • Cannot install plugins
   • Settings in eReader.conf are limited
   • NickelMenu required for OPDS access (jailbreak)

2. KOReader Capabilities:

   • Can install plugins (copy files to /koreader/plugins/)
   • Lua-based plugin architecture
   • Can send custom HTTP requests
   • Can access device serial for identification
   • Native OPDS client support

3. Security Trade-offs:

   • Serial numbers: Somewhat predictable, easy for users
   • Bearer tokens: Cryptographically secure, requires setup
   • User accepted security/usability trade-off for self-hosted use

Why Functionality Trumps Security

User Quote: "I want a Kindle system replacement, otherwise I could just use booklore or KOReader. My biggest concern is do any of these limit any of the syncing/downloading implementation I already have. Functionality and features are more important than anything else. Although security does come in at a close second."

Translation:

• Self-hosted = trusted network environment
• User understands and accepts risks
• Willing to document security implications
• Prioritizes working features over perfect security

Resulting Architecture:

• Kobo: Serial auth (medium security, zero friction)
• KOReader: Bearer token (high security, requires plugin setup)
• Both get full feature parity
• Security implications clearly documented

---

11. Updated Implementation Timeline

Parallel Track Structure

Track A: Kobo Integration

• Phase 1: Enhanced Auth (Week 1)
• Phase 2a: Kobo Implementation (Week 1-2)
• Phase 4: Kobo Testing (Week 3)

Track B: KOReader Plugin

• Phase 1: Enhanced Auth (Week 1) [shared]
• Phase 2b: Plugin Development (Week 2-4)
• Phase 4: Plugin Testing (Week 4)

Track C: OPDS Security

• Phase 3: OPDS Integration (Week 2-3)

Track D: Documentation

• Phase 5: Final Docs & Release (Week 4)

Milestones

Milestone 1 (End of Week 2): Kobo Fully Functional

• Serial auth working
• All sync features operational
• OPDS access secured
• Can ship to Kobo users

Milestone 2 (End of Week 4): KOReader Plugin Ready

• Plugin released in separate repo
• Full feature parity with Kobo
• Documentation complete
• Production release

---

12. Files & Components Reference

Critical Files for Implementation

Authentication Core:

• internal/middleware/device_auth.go:37-108 - Main auth middleware
• internal/database/queries/devices.sql - Device queries
• internal/database/models.go - Device model (has device_identifier)

Routing:

• internal/router/opds.go:11-12 - OPDS routes (need middleware)
• internal/router/sync.go:17-50 - Sync routes
• internal/router/device.go - Device management routes

Handlers:

• internal/handlers/kobo.go:26-679 - Kobo sync implementation
• internal/handlers/koreader.go:23-897 - KOReader sync implementation
• internal/handlers/opds.go - OPDS handler

Tests:

• cmd/server/tests/kobo_test.go - Kobo tests
• cmd/server/tests/opds_test.go - OPDS tests
• cmd/server/tests/test_helpers.go - Test utilities

Documentation (to update):

• docs/user/devices/kobo-setup.md - Currently has wrong info
• docs/user/devices/koreader-setup.md - Needs Bearer token info

Sidecar (reference only, not implementing):

• internal/handlers/sidecar.go - Exists but not wired up
• Routes not registered in router

---

13. Next Steps

Immediate Actions (When Ready to Proceed)

1. Review this plan - Ensure all decisions and context are captured
2. Create KOReader plugin repo - Set up github.com/bookhoard/koreader-plugin
3. Begin Phase 1 - Enhanced DeviceAuthMiddleware
4. Parallel development - Kobo and KOReader tracks

Success Criteria (Reiterated)

Functional:

• ✅ Kobo: Enter serial → register → sync works immediately
• ✅ KOReader: Install plugin → enter token → sync works
• ✅ Both: Full sync (progress, highlights, notes, bookmarks)
• ✅ OPDS: Authenticated access on both platforms
• ✅ Cross-device: Read on one, continue on another

Security:

• ✅ OPDS no longer public
• ✅ All sync endpoints require auth
• ✅ Security implications documented
• ✅ Network security recommendations provided

User Experience:

• ✅ Kobo setup: < 5 minutes
• ✅ KOReader setup: < 10 minutes
• ✅ Clear documentation
• ✅ Working examples

---

Plan Status: Ready for implementation Last Updated: Based on conversation ending with user decisions Note: User indicated it's late and not proceeding tonight

Ready to implement when you are. This plan captures all our discussion and decisions.

---

10. Risk Assessment

Low Risk:

• Serial auth implementation (well-understood pattern)
• OPDS security (straightforward middleware application)
• Kobo sync (already implemented, just fixing auth)

Medium Risk:

• KOReader plugin development (Lua learning curve, potential API changes)
• Cross-device sync conflicts (need clear resolution strategy)

High Risk:

• KOReader API instability (if they change plugin APIs frequently)
• User adoption (may resist plugin installation)

Mitigations:

• Start with Kobo to prove concept
• Simple plugin architecture (minimize breaking changes impact)
• Excellent documentation to reduce friction
• Community feedback loop

---

11. Timeline Summary

Phase	Duration	Key Deliverable
Phase 1	Week 1	Enhanced auth middleware working
Phase 2	Week 1-2	Kobo fully functional
Phase 3	Week 2-4	KOReader plugin complete
Phase 4	Week 2-3	OPDS secured
Phase 5	Week 4	Testing & docs complete

Total Duration: 4 weeks

---

Next Steps:

1. User reviews and approves plan
2. Answer open questions (Section 9)
3. Begin Phase 1 implementation
4. Weekly check-ins on progress

Ready to proceed?