bookhoard/docs/PROGRESS_ROUTES_ANALYSIS.md

Progress Routes Analysis & Thoughts

Overview

This document explores the current state of progress tracking in Bookmann, the migration from legacy media-item-specific routes to universal cross-device progress, and considerations for the future.

---

Current State

Legacy Routes (Marked as Deprecated)

Located in internal/handlers/ebook.go:114-117:

// Legacy progress routes (deprecated - use universal progress instead)
g.GET("/api/media-items/:id/progress", h.GetMediaReadingProgress)
g.PUT("/api/media-items/:id/progress", h.UpdateMediaReadingProgress)
g.DELETE("/api/media-items/:id/progress", h.DeleteMediaReadingProgress)

Purpose: These routes handle progress tracking for a specific media item from the media_items table.

Data Source: Likely queries the reading_progress table filtered by media_item_id.

Current Status: Explicitly marked as "legacy" and "deprecated" in code comments.

---

Universal Progress Routes (Phase 1 Implementation)

Located in internal/handlers/ebook.go:119-122:

// Universal Progress routes (Phase 1)
g.GET("/api/progress/:id", h.GetUniversalProgress)
g.POST("/api/progress/:id", h.UpdateUniversalProgress)
g.GET("/api/progress/:id/history", h.GetProgressHistory)

Purpose: These routes provide "universal" progress tracking that works across devices and media types.

Data Source: Uses enhanced reading_progress table with additional fields:

• percentage - Universal percentage (0-1)
• character_offset - Character-based positioning
• epubcfi - EPUB Canonical Fragment Identifier
• chapter + chapter_progress - Chapter-based tracking
• Viewport coordinates (viewport_x, viewport_y, zoom_level)
• Scroll positions (scroll_position_x, scroll_position_y)
• Panel number for comics/manga
• Reading mode indicator

Device Sync Metadata:

• last_sync_device - Which device last updated
• last_sync_source - Source type (koreader, kobo, web, etc.)
• last_sync_timestamp - When sync occurred
• conflict_detected - Boolean flag for conflicts
• conflict_resolved - Boolean flag for resolution status

---

Why the Migration Happened

1. Cross-Platform Kindle Ecosystem Vision

Bookmann aims to replace the Kindle ecosystem, which requires:

• Syncing progress across multiple devices (Kindle, Kobo, phone, web)
• Handling different progress formats (page numbers, percentages, CFI, character offsets)
• Maintaining reading state across different device types
• Supporting offline reading with sync queues

2. Format Diversity

Different e-readers and formats use different progress indicators:

Format/Device	Progress Type	Example
EPUB (KOReader)	EPUBCFI	epubcfi(/6/4[chap1ref]!/4/2/1:0)
EPUB (Kobo)	Page # + Total	page 234 of 456
PDF	Page #	page 45
Web Reader	Percentage	0.45 (45%)
TXT/Mobi	Character Offset	offset 12345
Comics/Manga	Panel #	panel 7
Kindle	Location #	location 1234

The legacy media-items/:id/progress routes couldn't handle this diversity.

3. Device Sync Architecture

Universal progress enables:

• Real-time sync via WebSocket (/ws/sync)
• Offline queue support (/api/queue/*)
• Conflict detection and resolution
• Checkpoint mode for battery optimization
• Progress history tracking

---

Current Database Schema

From database/schema/schema.sql:130-158:

CREATE TABLE reading_progress (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    media_item_id UUID NOT NULL REFERENCES media_items(id) ON DELETE CASCADE,
    user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,

    -- Legacy fields
    current_page INTEGER DEFAULT 0,
    total_pages INTEGER,
    last_read_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    -- Universal Progress Tracking (Phase 1)
    percentage FLOAT CHECK (percentage >= 0 AND percentage <= 1),
    character_offset BIGINT,
    epubcfi TEXT,
    chapter INTEGER,
    chapter_progress FLOAT CHECK (chapter_progress >= 0 AND chapter_progress <= 1),
    viewport_x FLOAT DEFAULT 0,
    viewport_y FLOAT DEFAULT 0,
    zoom_level FLOAT DEFAULT 1.0,
    scroll_position_x FLOAT DEFAULT 0,
    scroll_position_y FLOAT DEFAULT 0,
    panel_number INTEGER,
    reading_mode VARCHAR(20),

    -- Device Sync Metadata (Phase 1)
    last_sync_device VARCHAR(50),
    last_sync_source VARCHAR(20),
    last_sync_timestamp TIMESTAMP WITH TIME ZONE,
    conflict_detected BOOLEAN DEFAULT FALSE,
    conflict_resolved BOOLEAN DEFAULT TRUE,

    UNIQUE(media_item_id, user_id)
);

Backward Compatibility View (Line 160-168):

CREATE VIEW ebook_reading_progress AS
SELECT rp.*,
       mi.id as ebook_id -- Map media_item_id to ebook_id for compatibility
FROM reading_progress rp
JOIN media_items mi ON rp.media_item_id = mi.id
JOIN libraries l ON mi.library_id = l.id
JOIN library_types lt ON l.library_type_id = lt.id
WHERE lt.name = 'ebooks';

---

The Migration Challenge

Issue: Two Parallel Systems

Currently, both systems exist side-by-side:

1. Legacy routes (/api/media-items/:id/progress)

   • Likely use simple current_page / total_pages fields
   • Media-item scoped
   • No device sync metadata

2. Universal routes (/api/progress/:id)

   • Use rich progress tracking with multiple formats
   • Device-aware
   • Include sync metadata

Question: What Does :id Mean?

Legacy: :id = media_item_id (UUID of the book)

Universal: :id = ??? (Could be same media_item_id, or could be a different identifier)

Ambiguity: The routes use the same parameter name but might mean different things.

Potential Problems

1. Data Duplication: If both systems write to reading_progress table, they might overwrite each other
2. Client Confusion: Which endpoint should clients use?
3. Migration Path: How do existing clients using legacy endpoints transition?
4. API Consistency: Having two different endpoints for similar functionality is confusing

---

Observations & Concerns

1. Incomplete Migration

The legacy routes are marked as deprecated but still active. This suggests:

• Migration is ongoing, not complete
• Some clients might still depend on legacy routes
• Fear of breaking existing integrations

2. Backward Compatibility View

The ebook_reading_progress view exists to maintain compatibility with the old ebooks table. This adds:

• Query overhead (JOINs to filter by library type)
• Developer confusion (which table/view to query?)
• Technical debt (maintaining two ways to access data)

3. Route Naming Inconsistency

• Legacy: /api/media-items/:id/progress (RESTful, nested under media-item)
• Universal: /api/progress/:id (flat structure, not nested)

Question: Should universal progress be under /api/media-items/:id/universal-progress for consistency?

4. HTTP Method Mismatch

Legacy routes use:

• PUT /api/media-items/:id/progress (update progress)

Universal routes use:

• POST /api/progress/:id (update progress)

REST convention: PUT is idempotent, POST is not. For progress updates, PUT might be more appropriate since setting the same progress twice should have the same effect.

5. Missing Delete Operation

Universal routes don't have a DELETE /api/progress/:id endpoint. Legacy does:

• DELETE /api/media-items/:id/progress (clear progress)

Question: Should there be a way to reset progress via universal routes?

---

Potential Future Directions

Option 1: Full Migration (Clean Break)

Action: Remove all legacy routes and views.

Steps:

1. Deprecate legacy routes in API documentation (return Warning header)
2. Add a 6-month migration timeline
3. Remove /api/media-items/:id/progress routes
4. Drop ebook_reading_progress view
5. Update all clients to use universal routes

Pros:

• Cleaner API surface
• Single source of truth
• Less maintenance burden
• Clearer documentation

Cons:

• Breaking change for existing clients
• Mobile apps might need updates
• External integrations could break

Option 2: Compatibility Layer (Adapter Pattern)

Action: Keep legacy routes but make them thin wrappers around universal routes.

Implementation:

// Legacy route calls universal route internally
func (h *Handler) GetMediaReadingProgress(c echo.Context) error {
    mediaItemID := c.Param("id")
    // Extract user_id from JWT
    // Call h.GetUniversalProgress with same IDs
    // Transform response if needed
}

Pros:

• No breaking changes
• Gradual migration path
• Single implementation (universal routes)

Cons:

• Maintains API surface area
• Slight performance overhead (function call)
• Still confusing to have two endpoints

Option 3: Unified Endpoint (Best of Both)

Action: Create a single endpoint that handles both use cases.

Proposed:

GET /api/media-items/:id/progress?format=universal
PUT /api/media-items/:id/progress?format=universal
DELETE /api/media-items/:id/progress

The format query parameter determines:

• format=simple (default): Returns basic page/percentage (legacy behavior)
• format=universal: Returns full device-aware progress with metadata

Pros:

• Single endpoint
• Backward compatible
• Clear migration path via query parameter
• RESTful structure (nested under media-items)

Cons:

• More complex handler logic
• Need to maintain both formats in response

Option 4: Versioned API (Cleanest Long-Term)

Action: Use API versioning to separate old and new.

Proposed:

# v1 (Legacy)
GET /api/v1/media-items/:id/progress
PUT /api/v1/media-items/:id/progress
DELETE /api/v1/media-items/:id/progress

# v2 (Universal)
GET /api/v2/media-items/:id/progress
PUT /api/v2/media-items/:id/progress
GET /api/v2/media-items/:id/progress/history

Pros:

• Clean separation
• Can deprecate v1 independently
• Standard industry practice
• Clear migration documentation

Cons:

• Need to implement version routing
• More upfront work
• Maintenance of two versions temporarily

---

Unanswered Questions for Discussion

1. Are any clients currently using the legacy progress routes?

   • If yes, which ones? (mobile app, web app, third-party integrations?)
   • Can they be updated easily?

2. What does the :id parameter represent in universal progress routes?

   • Is it still media_item_id?
   • Or is it a reading_progress record ID?
   • Need to check implementation to confirm

3. Why was /api/progress/:id chosen instead of /api/media-items/:id/universal-progress?

   • Flat structure vs nested structure design decision
   • Might indicate plans for progress to exist independently of media items?

4. Is the legacy route implementation actually different, or just deprecated?

   • Need to read the handler implementations to compare
   • They might be calling the same underlying code

5. Should we maintain progress deletion functionality?

   • Universal routes don't have DELETE
   • Is deleting progress a necessary feature?

6. What's the timeline for removing legacy routes?

   • Already marked deprecated, but when can we delete them?
   • Need to coordinate with mobile app releases

7. How does the backward compatibility view affect performance?

   • The ebook_reading_progress view requires JOINs
   • Is it used anywhere, or can it be dropped?

---

Recommendations

Immediate Actions (Discussion Phase)

1. Audit Current Usage

   • Search codebase for references to legacy routes
   • Check if any external documentation mentions these endpoints
   • Identify all clients (web, mobile, third-party)

2. Compare Implementations

   • Read handler code for both legacy and universal routes
   • Document differences in behavior
   • Determine if they're truly different or just deprecated wrappers

3. Clarify API Contract

   • Define what :id means in universal routes
   • Document expected request/response formats
   • Add examples for different device types

4. Performance Analysis

   • Query database to see how many records use legacy fields vs universal
   • Check if backward compatibility view is actually used
   • Benchmark query performance with/without views

Future Considerations

1. Choose a Migration Strategy

   • Review Options 1-4 above
   • Consider breaking changes vs compatibility
   • Plan timeline based on client usage

2. API Versioning Decision

   • Decide if we want /api/v1/ and /api/v2/ structure
   • Or use different approach (headers, content negotiation)

3. Documentation Updates

   • Update API_REFERENCE.md with clear deprecation notices
   • Add migration guide for clients
   • Document best practices for progress tracking

4. Test Coverage

   • Ensure both legacy and universal routes have comprehensive tests
   • Add integration tests for cross-device sync scenarios
   • Test conflict resolution workflows

---

Next Steps for Discussion

1. Review handler implementations to understand actual differences
2. Check client usage (web app, mobile apps, Bruno tests)
3. Decide on migration timeline and breaking change tolerance
4. Choose unified strategy (Options 1-4 or hybrid)
5. Plan implementation with backward compatibility in mind

---

Document created for future discussion. No changes to be made without review.