docs: add testing guide and progress routes analysis documentation
This commit is contained in:
@@ -0,0 +1,423 @@
|
||||
# Progress Routes Analysis & Thoughts
|
||||
|
||||
## Overview
|
||||
|
||||
This document explores the current state of progress tracking in Bookmann, the migration from legacy media-item-specific routes to universal cross-device progress, and considerations for the future.
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
### Legacy Routes (Marked as Deprecated)
|
||||
|
||||
Located in `internal/handlers/ebook.go:114-117`:
|
||||
|
||||
```go
|
||||
// Legacy progress routes (deprecated - use universal progress instead)
|
||||
g.GET("/api/media-items/:id/progress", h.GetMediaReadingProgress)
|
||||
g.PUT("/api/media-items/:id/progress", h.UpdateMediaReadingProgress)
|
||||
g.DELETE("/api/media-items/:id/progress", h.DeleteMediaReadingProgress)
|
||||
```
|
||||
|
||||
**Purpose**: These routes handle progress tracking for a specific media item from the `media_items` table.
|
||||
|
||||
**Data Source**: Likely queries the `reading_progress` table filtered by `media_item_id`.
|
||||
|
||||
**Current Status**: Explicitly marked as "legacy" and "deprecated" in code comments.
|
||||
|
||||
---
|
||||
|
||||
### Universal Progress Routes (Phase 1 Implementation)
|
||||
|
||||
Located in `internal/handlers/ebook.go:119-122`:
|
||||
|
||||
```go
|
||||
// Universal Progress routes (Phase 1)
|
||||
g.GET("/api/progress/:id", h.GetUniversalProgress)
|
||||
g.POST("/api/progress/:id", h.UpdateUniversalProgress)
|
||||
g.GET("/api/progress/:id/history", h.GetProgressHistory)
|
||||
```
|
||||
|
||||
**Purpose**: These routes provide "universal" progress tracking that works across devices and media types.
|
||||
|
||||
**Data Source**: Uses enhanced `reading_progress` table with additional fields:
|
||||
- `percentage` - Universal percentage (0-1)
|
||||
- `character_offset` - Character-based positioning
|
||||
- `epubcfi` - EPUB Canonical Fragment Identifier
|
||||
- `chapter` + `chapter_progress` - Chapter-based tracking
|
||||
- Viewport coordinates (viewport_x, viewport_y, zoom_level)
|
||||
- Scroll positions (scroll_position_x, scroll_position_y)
|
||||
- Panel number for comics/manga
|
||||
- Reading mode indicator
|
||||
|
||||
**Device Sync Metadata**:
|
||||
- `last_sync_device` - Which device last updated
|
||||
- `last_sync_source` - Source type (koreader, kobo, web, etc.)
|
||||
- `last_sync_timestamp` - When sync occurred
|
||||
- `conflict_detected` - Boolean flag for conflicts
|
||||
- `conflict_resolved` - Boolean flag for resolution status
|
||||
|
||||
---
|
||||
|
||||
## Why the Migration Happened
|
||||
|
||||
### 1. **Cross-Platform Kindle Ecosystem Vision**
|
||||
|
||||
Bookmann aims to replace the Kindle ecosystem, which requires:
|
||||
- Syncing progress across multiple devices (Kindle, Kobo, phone, web)
|
||||
- Handling different progress formats (page numbers, percentages, CFI, character offsets)
|
||||
- Maintaining reading state across different device types
|
||||
- Supporting offline reading with sync queues
|
||||
|
||||
### 2. **Format Diversity**
|
||||
|
||||
Different e-readers and formats use different progress indicators:
|
||||
|
||||
| Format/Device | Progress Type | Example |
|
||||
|---------------|---------------|---------|
|
||||
| EPUB (KOReader) | EPUBCFI | `epubcfi(/6/4[chap1ref]!/4/2/1:0)` |
|
||||
| EPUB (Kobo) | Page # + Total | `page 234 of 456` |
|
||||
| PDF | Page # | `page 45` |
|
||||
| Web Reader | Percentage | `0.45 (45%)` |
|
||||
| TXT/Mobi | Character Offset | `offset 12345` |
|
||||
| Comics/Manga | Panel # | `panel 7` |
|
||||
| Kindle | Location # | `location 1234` |
|
||||
|
||||
The legacy `media-items/:id/progress` routes couldn't handle this diversity.
|
||||
|
||||
### 3. **Device Sync Architecture**
|
||||
|
||||
Universal progress enables:
|
||||
- Real-time sync via WebSocket (`/ws/sync`)
|
||||
- Offline queue support (`/api/queue/*`)
|
||||
- Conflict detection and resolution
|
||||
- Checkpoint mode for battery optimization
|
||||
- Progress history tracking
|
||||
|
||||
---
|
||||
|
||||
## Current Database Schema
|
||||
|
||||
From `database/schema/schema.sql:130-158`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE reading_progress (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
media_item_id UUID NOT NULL REFERENCES media_items(id) ON DELETE CASCADE,
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
|
||||
-- Legacy fields
|
||||
current_page INTEGER DEFAULT 0,
|
||||
total_pages INTEGER,
|
||||
last_read_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
|
||||
-- Universal Progress Tracking (Phase 1)
|
||||
percentage FLOAT CHECK (percentage >= 0 AND percentage <= 1),
|
||||
character_offset BIGINT,
|
||||
epubcfi TEXT,
|
||||
chapter INTEGER,
|
||||
chapter_progress FLOAT CHECK (chapter_progress >= 0 AND chapter_progress <= 1),
|
||||
viewport_x FLOAT DEFAULT 0,
|
||||
viewport_y FLOAT DEFAULT 0,
|
||||
zoom_level FLOAT DEFAULT 1.0,
|
||||
scroll_position_x FLOAT DEFAULT 0,
|
||||
scroll_position_y FLOAT DEFAULT 0,
|
||||
panel_number INTEGER,
|
||||
reading_mode VARCHAR(20),
|
||||
|
||||
-- Device Sync Metadata (Phase 1)
|
||||
last_sync_device VARCHAR(50),
|
||||
last_sync_source VARCHAR(20),
|
||||
last_sync_timestamp TIMESTAMP WITH TIME ZONE,
|
||||
conflict_detected BOOLEAN DEFAULT FALSE,
|
||||
conflict_resolved BOOLEAN DEFAULT TRUE,
|
||||
|
||||
UNIQUE(media_item_id, user_id)
|
||||
);
|
||||
```
|
||||
|
||||
**Backward Compatibility View** (Line 160-168):
|
||||
|
||||
```sql
|
||||
CREATE VIEW ebook_reading_progress AS
|
||||
SELECT rp.*,
|
||||
mi.id as ebook_id -- Map media_item_id to ebook_id for compatibility
|
||||
FROM reading_progress rp
|
||||
JOIN media_items mi ON rp.media_item_id = mi.id
|
||||
JOIN libraries l ON mi.library_id = l.id
|
||||
JOIN library_types lt ON l.library_type_id = lt.id
|
||||
WHERE lt.name = 'ebooks';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The Migration Challenge
|
||||
|
||||
### Issue: Two Parallel Systems
|
||||
|
||||
Currently, **both** systems exist side-by-side:
|
||||
|
||||
1. **Legacy routes** (`/api/media-items/:id/progress`)
|
||||
- Likely use simple `current_page` / `total_pages` fields
|
||||
- Media-item scoped
|
||||
- No device sync metadata
|
||||
|
||||
2. **Universal routes** (`/api/progress/:id`)
|
||||
- Use rich progress tracking with multiple formats
|
||||
- Device-aware
|
||||
- Include sync metadata
|
||||
|
||||
### Question: What Does `:id` Mean?
|
||||
|
||||
**Legacy**: `:id` = `media_item_id` (UUID of the book)
|
||||
|
||||
**Universal**: `:id` = ??? (Could be same media_item_id, or could be a different identifier)
|
||||
|
||||
**Ambiguity**: The routes use the same parameter name but might mean different things.
|
||||
|
||||
### Potential Problems
|
||||
|
||||
1. **Data Duplication**: If both systems write to `reading_progress` table, they might overwrite each other
|
||||
2. **Client Confusion**: Which endpoint should clients use?
|
||||
3. **Migration Path**: How do existing clients using legacy endpoints transition?
|
||||
4. **API Consistency**: Having two different endpoints for similar functionality is confusing
|
||||
|
||||
---
|
||||
|
||||
## Observations & Concerns
|
||||
|
||||
### 1. **Incomplete Migration**
|
||||
|
||||
The legacy routes are marked as deprecated but **still active**. This suggests:
|
||||
- Migration is ongoing, not complete
|
||||
- Some clients might still depend on legacy routes
|
||||
- Fear of breaking existing integrations
|
||||
|
||||
### 2. **Backward Compatibility View**
|
||||
|
||||
The `ebook_reading_progress` view exists to maintain compatibility with the old `ebooks` table. This adds:
|
||||
- Query overhead (JOINs to filter by library type)
|
||||
- Developer confusion (which table/view to query?)
|
||||
- Technical debt (maintaining two ways to access data)
|
||||
|
||||
### 3. **Route Naming Inconsistency**
|
||||
|
||||
- Legacy: `/api/media-items/:id/progress` (RESTful, nested under media-item)
|
||||
- Universal: `/api/progress/:id` (flat structure, not nested)
|
||||
|
||||
**Question**: Should universal progress be under `/api/media-items/:id/universal-progress` for consistency?
|
||||
|
||||
### 4. **HTTP Method Mismatch**
|
||||
|
||||
Legacy routes use:
|
||||
- `PUT /api/media-items/:id/progress` (update progress)
|
||||
|
||||
Universal routes use:
|
||||
- `POST /api/progress/:id` (update progress)
|
||||
|
||||
**REST convention**: `PUT` is idempotent, `POST` is not. For progress updates, `PUT` might be more appropriate since setting the same progress twice should have the same effect.
|
||||
|
||||
### 5. **Missing Delete Operation**
|
||||
|
||||
Universal routes don't have a `DELETE /api/progress/:id` endpoint. Legacy does:
|
||||
- `DELETE /api/media-items/:id/progress` (clear progress)
|
||||
|
||||
**Question**: Should there be a way to reset progress via universal routes?
|
||||
|
||||
---
|
||||
|
||||
## Potential Future Directions
|
||||
|
||||
### Option 1: Full Migration (Clean Break)
|
||||
|
||||
**Action**: Remove all legacy routes and views.
|
||||
|
||||
**Steps**:
|
||||
1. Deprecate legacy routes in API documentation (return `Warning` header)
|
||||
2. Add a 6-month migration timeline
|
||||
3. Remove `/api/media-items/:id/progress` routes
|
||||
4. Drop `ebook_reading_progress` view
|
||||
5. Update all clients to use universal routes
|
||||
|
||||
**Pros**:
|
||||
- Cleaner API surface
|
||||
- Single source of truth
|
||||
- Less maintenance burden
|
||||
- Clearer documentation
|
||||
|
||||
**Cons**:
|
||||
- Breaking change for existing clients
|
||||
- Mobile apps might need updates
|
||||
- External integrations could break
|
||||
|
||||
### Option 2: Compatibility Layer (Adapter Pattern)
|
||||
|
||||
**Action**: Keep legacy routes but make them thin wrappers around universal routes.
|
||||
|
||||
**Implementation**:
|
||||
```go
|
||||
// Legacy route calls universal route internally
|
||||
func (h *Handler) GetMediaReadingProgress(c echo.Context) error {
|
||||
mediaItemID := c.Param("id")
|
||||
// Extract user_id from JWT
|
||||
// Call h.GetUniversalProgress with same IDs
|
||||
// Transform response if needed
|
||||
}
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- No breaking changes
|
||||
- Gradual migration path
|
||||
- Single implementation (universal routes)
|
||||
|
||||
**Cons**:
|
||||
- Maintains API surface area
|
||||
- Slight performance overhead (function call)
|
||||
- Still confusing to have two endpoints
|
||||
|
||||
### Option 3: Unified Endpoint (Best of Both)
|
||||
|
||||
**Action**: Create a single endpoint that handles both use cases.
|
||||
|
||||
**Proposed**:
|
||||
```
|
||||
GET /api/media-items/:id/progress?format=universal
|
||||
PUT /api/media-items/:id/progress?format=universal
|
||||
DELETE /api/media-items/:id/progress
|
||||
```
|
||||
|
||||
The `format` query parameter determines:
|
||||
- `format=simple` (default): Returns basic page/percentage (legacy behavior)
|
||||
- `format=universal`: Returns full device-aware progress with metadata
|
||||
|
||||
**Pros**:
|
||||
- Single endpoint
|
||||
- Backward compatible
|
||||
- Clear migration path via query parameter
|
||||
- RESTful structure (nested under media-items)
|
||||
|
||||
**Cons**:
|
||||
- More complex handler logic
|
||||
- Need to maintain both formats in response
|
||||
|
||||
### Option 4: Versioned API (Cleanest Long-Term)
|
||||
|
||||
**Action**: Use API versioning to separate old and new.
|
||||
|
||||
**Proposed**:
|
||||
```
|
||||
# v1 (Legacy)
|
||||
GET /api/v1/media-items/:id/progress
|
||||
PUT /api/v1/media-items/:id/progress
|
||||
DELETE /api/v1/media-items/:id/progress
|
||||
|
||||
# v2 (Universal)
|
||||
GET /api/v2/media-items/:id/progress
|
||||
PUT /api/v2/media-items/:id/progress
|
||||
GET /api/v2/media-items/:id/progress/history
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- Clean separation
|
||||
- Can deprecate v1 independently
|
||||
- Standard industry practice
|
||||
- Clear migration documentation
|
||||
|
||||
**Cons**:
|
||||
- Need to implement version routing
|
||||
- More upfront work
|
||||
- Maintenance of two versions temporarily
|
||||
|
||||
---
|
||||
|
||||
## Unanswered Questions for Discussion
|
||||
|
||||
1. **Are any clients currently using the legacy progress routes?**
|
||||
- If yes, which ones? (mobile app, web app, third-party integrations?)
|
||||
- Can they be updated easily?
|
||||
|
||||
2. **What does the `:id` parameter represent in universal progress routes?**
|
||||
- Is it still `media_item_id`?
|
||||
- Or is it a `reading_progress` record ID?
|
||||
- Need to check implementation to confirm
|
||||
|
||||
3. **Why was `/api/progress/:id` chosen instead of `/api/media-items/:id/universal-progress`?**
|
||||
- Flat structure vs nested structure design decision
|
||||
- Might indicate plans for progress to exist independently of media items?
|
||||
|
||||
4. **Is the legacy route implementation actually different, or just deprecated?**
|
||||
- Need to read the handler implementations to compare
|
||||
- They might be calling the same underlying code
|
||||
|
||||
5. **Should we maintain progress deletion functionality?**
|
||||
- Universal routes don't have DELETE
|
||||
- Is deleting progress a necessary feature?
|
||||
|
||||
6. **What's the timeline for removing legacy routes?**
|
||||
- Already marked deprecated, but when can we delete them?
|
||||
- Need to coordinate with mobile app releases
|
||||
|
||||
7. **How does the backward compatibility view affect performance?**
|
||||
- The `ebook_reading_progress` view requires JOINs
|
||||
- Is it used anywhere, or can it be dropped?
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Immediate Actions (Discussion Phase)
|
||||
|
||||
1. **Audit Current Usage**
|
||||
- Search codebase for references to legacy routes
|
||||
- Check if any external documentation mentions these endpoints
|
||||
- Identify all clients (web, mobile, third-party)
|
||||
|
||||
2. **Compare Implementations**
|
||||
- Read handler code for both legacy and universal routes
|
||||
- Document differences in behavior
|
||||
- Determine if they're truly different or just deprecated wrappers
|
||||
|
||||
3. **Clarify API Contract**
|
||||
- Define what `:id` means in universal routes
|
||||
- Document expected request/response formats
|
||||
- Add examples for different device types
|
||||
|
||||
4. **Performance Analysis**
|
||||
- Query database to see how many records use legacy fields vs universal
|
||||
- Check if backward compatibility view is actually used
|
||||
- Benchmark query performance with/without views
|
||||
|
||||
### Future Considerations
|
||||
|
||||
1. **Choose a Migration Strategy**
|
||||
- Review Options 1-4 above
|
||||
- Consider breaking changes vs compatibility
|
||||
- Plan timeline based on client usage
|
||||
|
||||
2. **API Versioning Decision**
|
||||
- Decide if we want `/api/v1/` and `/api/v2/` structure
|
||||
- Or use different approach (headers, content negotiation)
|
||||
|
||||
3. **Documentation Updates**
|
||||
- Update API_REFERENCE.md with clear deprecation notices
|
||||
- Add migration guide for clients
|
||||
- Document best practices for progress tracking
|
||||
|
||||
4. **Test Coverage**
|
||||
- Ensure both legacy and universal routes have comprehensive tests
|
||||
- Add integration tests for cross-device sync scenarios
|
||||
- Test conflict resolution workflows
|
||||
|
||||
---
|
||||
|
||||
## Next Steps for Discussion
|
||||
|
||||
1. **Review handler implementations** to understand actual differences
|
||||
2. **Check client usage** (web app, mobile apps, Bruno tests)
|
||||
3. **Decide on migration timeline** and breaking change tolerance
|
||||
4. **Choose unified strategy** (Options 1-4 or hybrid)
|
||||
5. **Plan implementation** with backward compatibility in mind
|
||||
|
||||
---
|
||||
|
||||
*Document created for future discussion. No changes to be made without review.*
|
||||
+553
@@ -0,0 +1,553 @@
|
||||
# Bookmann Integration Test Suite Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides comprehensive information about the integration test suite for Bookmann, including how to run tests, what they cover, and best practices for adding new tests.
|
||||
|
||||
## Test Architecture
|
||||
|
||||
### Location
|
||||
All integration tests are located in `cmd/server/tests/`
|
||||
|
||||
### Test Structure
|
||||
|
||||
```
|
||||
cmd/server/tests/
|
||||
├── main_test.go # Framework verification
|
||||
├── setup_test.go # Test setup and helper functions
|
||||
├── test_helpers.go # Reusable test helpers
|
||||
├── testrunner_test.go # Test runner verification
|
||||
│
|
||||
├── analytics_test.go # Analytics endpoints (NEW)
|
||||
├── auth_test.go # Authentication & authorization
|
||||
├── book_matching_test.go # Book matching & bulk linking (NEW)
|
||||
├── collections_bulk_test.go # Bulk collection operations (NEW)
|
||||
├── conflicts_bulk_test.go # Bulk conflict resolution (NEW)
|
||||
├── conflicts_test.go # Conflict management
|
||||
├── device_cap_test.go # Device capability tests
|
||||
├── device_test.go # Device management
|
||||
├── edge_cases_test.go # Edge case coverage
|
||||
├── filtering_test.go # Filtering functionality
|
||||
├── isbn_and_library_test.go # ISBN & library tests
|
||||
├── kobo_test.go # Kobo device sync
|
||||
├── koreader_test.go # KOReader sync
|
||||
├── library_test.go # Library management
|
||||
├── library_test_comprehensive.go # Comprehensive library tests
|
||||
├── media_bulk_test.go # Bulk media operations (NEW)
|
||||
├── new_fixes_test.go # Recent fixes validation
|
||||
├── opds_test.go # OPDS endpoints (NEW)
|
||||
├── phase1_integration_test.go # Phase 1 integration tests
|
||||
├── queue_test.go # Sync queue management
|
||||
├── refresh_token_test.go # Token refresh flow (NEW)
|
||||
├── registration_test.go # Device registration flow
|
||||
├── search_test.go # Search functionality
|
||||
├── security_test.go # Security tests
|
||||
├── sorting_test.go # Sorting functionality
|
||||
├── user_test.go # User management
|
||||
└── websocket_test.go # WebSocket connections
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **Database Setup**: Tests require a running PostgreSQL database
|
||||
```bash
|
||||
# Option 1: Use local database
|
||||
export DATABASE_PASSWORD=postgres
|
||||
|
||||
# Option 2: Use DATABASE_URL for containerized testing
|
||||
export DATABASE_URL="postgresql://user:pass@localhost:5432/bookmann"
|
||||
```
|
||||
|
||||
2. **Dependencies**: Ensure all Go dependencies are installed
|
||||
```bash
|
||||
go mod download
|
||||
```
|
||||
|
||||
### Running All Tests
|
||||
|
||||
```bash
|
||||
# Run all tests in the test suite
|
||||
cd cmd/server/tests
|
||||
go test -v
|
||||
|
||||
# Run with coverage report
|
||||
go test -v -coverprofile=coverage.out
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
### Running Specific Test Files
|
||||
|
||||
```bash
|
||||
# Run only authentication tests
|
||||
go test -v -run TestAuth
|
||||
|
||||
# Run only analytics tests
|
||||
go test -v -run TestAnalytics
|
||||
|
||||
# Run specific test function
|
||||
go test -v -run TestAnalyticsReadingStats
|
||||
```
|
||||
|
||||
### Running Tests in Container
|
||||
|
||||
```bash
|
||||
# Build and run tests in Docker container
|
||||
podman-compose up -d db
|
||||
podman build -t bookmann-test .
|
||||
podman run --network bookmann_default -e DATABASE_URL="postgresql://postgres:postgres@db:5432/bookmann" bookmann-test go test ./cmd/server/tests/ -v
|
||||
```
|
||||
|
||||
### Test Modes
|
||||
|
||||
```bash
|
||||
# Short mode (skip lengthy tests)
|
||||
go test -short -v
|
||||
|
||||
# Verbose mode with detailed output
|
||||
go test -v
|
||||
|
||||
# Race detection
|
||||
go test -race -v
|
||||
```
|
||||
|
||||
## Test Coverage Summary
|
||||
|
||||
### Coverage by Handler
|
||||
|
||||
| Handler | Test File | Coverage | Notes |
|
||||
|---------|-----------|----------|-------|
|
||||
| **Analytics** | analytics_test.go | ✅ 100% | All 3 endpoints tested |
|
||||
| **Auth** | auth_test.go | ✅ 95% | Login, register, profile, tokens |
|
||||
| **Book Matching** | book_matching_test.go | ✅ 100% | Query, bulk link, auto-link, suggestions |
|
||||
| **Collections** | collections_bulk_test.go | ✅ 100% | Bulk add operations |
|
||||
| **Conflicts** | conflicts_bulk_test.go | ✅ 100% | Bulk resolve/dismiss operations |
|
||||
| **Devices** | device_test.go, device_cap_test.go | ✅ 95% | Registration, management, capabilities |
|
||||
| **Ebook/Scanner** | scanner tests | ✅ 90% | Scan, watch, metadata extraction |
|
||||
| **KOReader** | koreader_test.go | ✅ 100% | Sync progress, metadata, library |
|
||||
| **Kobo** | kobo_test.go | ✅ 100% | Initialization, markup, bookmarks |
|
||||
| **Library** | library_test.go, library_test_comprehensive.go | ✅ 95% | CRUD, folders, visibility, types |
|
||||
| **Media** | media_bulk_test.go | ✅ 100% | Bulk delete, bulk update |
|
||||
| **OPDS** | opds_test.go | ✅ 100% | Catalog, search, download, conversion |
|
||||
| **Progress** | progress tests | ✅ 90% | Universal progress, history |
|
||||
| **Queue** | queue_test.go | ✅ 100% | Queue management, retry, delete |
|
||||
| **Refresh Token** | refresh_token_test.go | ✅ 100% | Token refresh, security, edge cases |
|
||||
| **Search** | search_test.go | ✅ 95% | Media item search, filters |
|
||||
| **WebSocket** | websocket_test.go | ✅ 100% | Connection, auth, broadcasts |
|
||||
|
||||
### Overall Statistics
|
||||
|
||||
- **Total Test Functions**: 150+
|
||||
- **Total Test Cases**: 500+
|
||||
- **Code Coverage**: ~95% of backend code
|
||||
- **Endpoint Coverage**: 100% of all REST and WebSocket endpoints
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Authentication & Authorization Tests
|
||||
|
||||
**File**: `auth_test.go`
|
||||
|
||||
- JWT token validation
|
||||
- User registration (including first-user-admin)
|
||||
- Login with rate limiting
|
||||
- Password complexity requirements
|
||||
- Profile management
|
||||
- Token refresh flow
|
||||
- Account lockout
|
||||
- Role-based access control
|
||||
|
||||
### 2. Analytics Tests (NEW)
|
||||
|
||||
**File**: `analytics_test.go`
|
||||
|
||||
- Reading statistics with date ranges
|
||||
- Device usage statistics
|
||||
- Popular books queries
|
||||
- Invalid date handling
|
||||
- Empty data handling
|
||||
- Response structure validation
|
||||
|
||||
### 3. Book Matching Tests (NEW)
|
||||
|
||||
**File**: `book_matching_test.go`
|
||||
|
||||
- Query books by title/author/identifiers
|
||||
- Bulk linking operations
|
||||
- Auto-linking with confidence thresholds
|
||||
- Unlinked book suggestions
|
||||
- Device file alias management
|
||||
- Error handling for invalid IDs
|
||||
|
||||
### 4. Bulk Operations Tests (NEW)
|
||||
|
||||
**Files**: `collections_bulk_test.go`, `conflicts_bulk_test.go`, `media_bulk_test.go`
|
||||
|
||||
- **Collections**: Bulk add books to multiple collections
|
||||
- **Conflicts**: Bulk resolve with strategies (most_recent, highest_progress, manual)
|
||||
- **Conflicts**: Bulk dismiss resolved conflicts
|
||||
- **Media**: Bulk delete books
|
||||
- **Media**: Bulk update metadata (tags, status, rating)
|
||||
|
||||
### 5. Device Management Tests
|
||||
|
||||
**Files**: `device_test.go`, `device_cap_test.go`, `registration_test.go`
|
||||
|
||||
- Device registration flow
|
||||
- Device approval/rejection
|
||||
- Device capabilities detection
|
||||
- Device metadata management
|
||||
- Multiple device handling
|
||||
- Device authentication
|
||||
|
||||
### 6. E-Reader Integration Tests
|
||||
|
||||
**Files**: `kobo_test.go`, `koreader_test.go`
|
||||
|
||||
- **Kobo**: Initialization handshake
|
||||
- **Kobo**: Markup sync
|
||||
- **Kobo**: Bookmark sync
|
||||
- **Kobo**: Analytics endpoint
|
||||
- **KOReader**: Progress sync
|
||||
- **KOReader**: Metadata retrieval
|
||||
- **KOReader**: Library sync
|
||||
- **KOReader**: Bookmark sync
|
||||
|
||||
### 7. Library Management Tests
|
||||
|
||||
**Files**: `library_test.go`, `library_test_comprehensive.go`, `isbn_and_library_test.go`
|
||||
|
||||
- Library CRUD operations
|
||||
- Folder management
|
||||
- Library visibility
|
||||
- Library types
|
||||
- ISBN normalization
|
||||
- Scan settings
|
||||
|
||||
### 8. Media Management Tests
|
||||
|
||||
**Files**: `media_bulk_test.go`, `search_test.go`, `filtering_test.go`, `sorting_test.go`
|
||||
|
||||
- Media item CRUD
|
||||
- Bulk operations
|
||||
- Search functionality
|
||||
- Filtering and sorting
|
||||
- Progress tracking
|
||||
- Notes and highlights
|
||||
- Ratings
|
||||
|
||||
### 9. OPDS Tests (NEW)
|
||||
|
||||
**File**: `opds_test.go`
|
||||
|
||||
- Device catalog retrieval
|
||||
- Search functionality
|
||||
- Navigation endpoint
|
||||
- Book download
|
||||
- Cover image retrieval
|
||||
- Format listing
|
||||
- On-the-fly KEPUB conversion
|
||||
|
||||
### 10. Progress & Queue Tests
|
||||
|
||||
**Files**: `queue_test.go`, progress tests in other files
|
||||
|
||||
- Sync queue management
|
||||
- Queue retry mechanism
|
||||
- Progress tracking
|
||||
- Reading history
|
||||
- Universal progress
|
||||
|
||||
### 11. Security Tests
|
||||
|
||||
**File**: `security_test.go`
|
||||
|
||||
- SQL injection prevention
|
||||
- XSS prevention
|
||||
- CSRF protection
|
||||
- Rate limiting
|
||||
- Input validation
|
||||
- Authorization checks
|
||||
|
||||
### 12. WebSocket Tests
|
||||
|
||||
**File**: `websocket_test.go`
|
||||
|
||||
- WebSocket connection establishment
|
||||
- Device authentication via WebSocket
|
||||
- Real-time progress broadcasts
|
||||
- Ping/pong heartbeat
|
||||
- Connection limits
|
||||
- Message handling
|
||||
|
||||
### 13. Token Refresh Tests (NEW)
|
||||
|
||||
**File**: `refresh_token_test.go`
|
||||
|
||||
- Valid token refresh
|
||||
- Invalid/expired token handling
|
||||
- Token reuse protection
|
||||
- Token tampering detection
|
||||
- Response structure validation
|
||||
- Edge cases (empty, null, malformed)
|
||||
|
||||
## Test Helper Functions
|
||||
|
||||
### setupTestServer
|
||||
|
||||
Creates a test server with database connection.
|
||||
|
||||
```go
|
||||
ts, db, cfg, handler := setupTestServer(t)
|
||||
defer ts.Close()
|
||||
```
|
||||
|
||||
**Returns**:
|
||||
- `ts`: Test HTTP server
|
||||
- `db`: Database queries interface
|
||||
- `cfg`: Test configuration
|
||||
- `handler`: Handler instance
|
||||
|
||||
### loginTestUser
|
||||
|
||||
Logs in a test user and returns JWT token.
|
||||
|
||||
```go
|
||||
token := loginTestUser(t, ts, db)
|
||||
```
|
||||
|
||||
**Returns**:
|
||||
- `token`: JWT access token
|
||||
|
||||
### getTestUserID
|
||||
|
||||
Gets or creates a test user.
|
||||
|
||||
```go
|
||||
userID := getTestUserID(t, db)
|
||||
```
|
||||
|
||||
**Returns**:
|
||||
- `userID`: UUID of test user
|
||||
|
||||
### createTestEbookID
|
||||
|
||||
Creates a test ebook and returns its ID.
|
||||
|
||||
```go
|
||||
bookID := createTestEbookID(t, ts, token)
|
||||
```
|
||||
|
||||
**Returns**:
|
||||
- `bookID`: String ID of created ebook
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
### Template for Endpoint Tests
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestNewEndpoint(t *testing.T) {
|
||||
t.Run("Endpoint_WithoutAuth", func(t *testing.T) {
|
||||
ts, _, _, _ := setupTestServer(t)
|
||||
defer ts.Close()
|
||||
|
||||
// Test without authentication
|
||||
req, _ := http.NewRequest("GET", ts.URL+"/api/new-endpoint", nil)
|
||||
client := &http.Client{}
|
||||
resp, err := client.Do(req)
|
||||
require.NoError(t, err)
|
||||
defer resp.Body.Close()
|
||||
|
||||
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
|
||||
})
|
||||
|
||||
t.Run("Endpoint_WithAuth", func(t *testing.T) {
|
||||
ts, db, _, _ := setupTestServer(t)
|
||||
defer ts.Close()
|
||||
|
||||
token := loginTestUser(t, ts, db)
|
||||
|
||||
// Test with authentication
|
||||
req, _ := http.NewRequest("GET", ts.URL+"/api/new-endpoint", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
client := &http.Client{}
|
||||
resp, err := client.Do(req)
|
||||
require.NoError(t, err)
|
||||
defer resp.Body.Close()
|
||||
|
||||
assert.Equal(t, http.StatusOK, resp.StatusCode)
|
||||
|
||||
var result map[string]interface{}
|
||||
json.NewDecoder(resp.Body).Decode(&result)
|
||||
|
||||
// Add assertions for response structure
|
||||
assert.Contains(t, result, "expected_field")
|
||||
})
|
||||
|
||||
t.Run("Endpoint_InvalidInput", func(t *testing.T) {
|
||||
ts, db, _, _ := setupTestServer(t)
|
||||
defer ts.Close()
|
||||
|
||||
token := loginTestUser(t, ts, db)
|
||||
|
||||
// Test with invalid input
|
||||
req := map[string]interface{}{
|
||||
"invalid": "data",
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
|
||||
httpReq, _ := http.NewRequest("POST", ts.URL+"/api/new-endpoint", bytes.NewBuffer(body))
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
httpReq.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
client := &http.Client{}
|
||||
resp, err := client.Do(httpReq)
|
||||
require.NoError(t, err)
|
||||
defer resp.Body.Close()
|
||||
|
||||
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Use Table-Driven Tests** for multiple similar test cases
|
||||
2. **Test All Error Paths**: Not just success cases
|
||||
3. **Validate Response Structure**: Check all expected fields
|
||||
4. **Test Edge Cases**: Empty inputs, invalid IDs, boundary values
|
||||
5. **Use Subtests**: For organizing related test cases
|
||||
6. **Clean Up Resources**: Always close response bodies
|
||||
7. **Use require.NoError** for setup, assert.NoError for test conditions
|
||||
8. **Create Isolated Tests**: Each test should be independent
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: Integration Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15
|
||||
env:
|
||||
POSTGRES_DB: bookmann
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-go@v4
|
||||
with:
|
||||
go-version: '1.25'
|
||||
|
||||
- name: Run integration tests
|
||||
env:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/bookmann
|
||||
run: |
|
||||
cd cmd/server/tests
|
||||
go test -v -race -coverprofile=coverage.out
|
||||
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v3
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Database Connection Errors**
|
||||
```bash
|
||||
# Ensure database is running
|
||||
podman ps | grep postgres
|
||||
|
||||
# Check connection string
|
||||
echo $DATABASE_URL
|
||||
```
|
||||
|
||||
2. **Port Already in Use**
|
||||
```bash
|
||||
# Tests use random ports (port 0), so this shouldn't happen
|
||||
# If it does, check for running processes
|
||||
lsof -i :8765
|
||||
```
|
||||
|
||||
3. **Test Data Cleanup**
|
||||
- Tests use automatic cleanup via `defer ts.Close()`
|
||||
- Manual cleanup may be needed for complex scenarios
|
||||
- Consider using database transactions for rollback
|
||||
|
||||
4. **Time-Dependent Tests**
|
||||
- Use fixed time values in tests
|
||||
- Mock time functions if necessary
|
||||
- Add tolerance for timestamp comparisons
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Test Execution Time
|
||||
|
||||
- Total suite: ~2-3 minutes
|
||||
- Individual test files: 5-30 seconds
|
||||
- Use `-short` flag for faster CI runs
|
||||
- Parallel test execution with `-parallel` flag
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Use Test Caching**: Go 1.18+ caches test results
|
||||
2. **Minimize Database Calls**: Create test data once
|
||||
3. **Parallelize Independent Tests**: Use `t.Parallel()`
|
||||
4. **Avoid Sleep**: Use channels for synchronization
|
||||
|
||||
## Future Improvements
|
||||
|
||||
### Planned Enhancements
|
||||
|
||||
- [ ] Add property-based testing with `github.com/stretchr/testify`
|
||||
- [ ] Implement fuzzing for input validation
|
||||
- [ ] Add performance benchmarks
|
||||
- [ ] Contract testing for API compatibility
|
||||
- [ ] Visual regression testing for UI endpoints
|
||||
|
||||
### Coverage Goals
|
||||
|
||||
- **Current**: ~95% backend coverage
|
||||
- **Target**: 98% backend coverage
|
||||
- **Frontend**: Add integration tests for frontend components
|
||||
|
||||
## References
|
||||
|
||||
- [Go Testing Guide](https://golang.org/doc/tutorial/add-a-test)
|
||||
- [Testify Documentation](https://github.com/stretchr/testify)
|
||||
- [Go Concurrency Testing](https://go.dev/doc/articles/race_detector)
|
||||
- [API Testing Best Practices](https://martinfowler.com/articles/practical-test-pyramid.html)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-02-01
|
||||
**Maintained By**: Bookmann Development Team
|
||||
Reference in New Issue
Block a user