docs: reorganize documentation structure for users and self-hosters
- Remove internal development docs (phase tracking, implementation plans, security audits) - Move DEVELOPMENT.md to docs/contributing/ for contributor guidance - Move TROUBLESHOOTING.md from root to docs/ folder - Add docs/INDEX.md as navigation hub for all documentation - Clean up docs to focus on user/self-hoster facing content This reorganization separates user-facing documentation from internal contributor documentation, making the project more approachable for self-hosters.
This commit is contained in:
@@ -1,231 +0,0 @@
|
||||
# EPUB to KEPUB Conversion Service
|
||||
|
||||
## Overview
|
||||
|
||||
The Conversion Service provides on-the-fly EPUB to KEPUB conversion with dual hash storage to ensure cross-device book matching continues to work after format conversion.
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **On-Demand Conversion**: Converts EPUB to KEPUB when requested via OPDS with `?format=kepub`
|
||||
2. **Dual Hash Storage**: Stores both original EPUB hash AND converted KEPUB hash in `media_item_formats` table
|
||||
3. **Conversion Caching**: Caches converted files for 24 hours (configurable) to avoid re-conversion
|
||||
4. **Hash Preservation**: After conversion, both hashes remain queryable for book matching
|
||||
5. **Format Integrity**: Ensures converted KEPUB maintains all reading progress markers
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
User requests book via OPDS with ?format=kepub
|
||||
↓
|
||||
Check media_item_formats table for existing KEPUB
|
||||
↓
|
||||
If KEPUB exists and is recent (< 24 hours):
|
||||
→ Serve pre-converted file
|
||||
→ Set X-Bookhoard-KEPUB-SHA256 header
|
||||
↓
|
||||
If KEPUB doesn't exist or is stale:
|
||||
→ Convert EPUB→KEPUB on-the-fly
|
||||
→ Calculate SHA-256 of converted KEPUB
|
||||
→ Store in media_item_formats (with converted_from_format_id)
|
||||
→ Serve converted file
|
||||
→ Set X-Bookhoard-KEPUB-SHA256 header
|
||||
↓
|
||||
Device downloads book with hash in response header
|
||||
↓
|
||||
Device syncs progress using hash for matching
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Add these to your `.env` file or `system_config` table:
|
||||
|
||||
```bash
|
||||
# Conversion service configuration
|
||||
BOOKHOARD_CONVERSION_CACHE_DIR=/var/bookhoard/cache/kepub
|
||||
BOOKHOARD_CONVERSION_TOOL=/usr/bin/kepubify # or /usr/bin/ebook-convert
|
||||
BOOKHOARD_CONVERSION_CACHE_TTL=24h
|
||||
```
|
||||
|
||||
### Dockerfile Updates
|
||||
|
||||
If using kepubify (recommended for Kobo):
|
||||
|
||||
```dockerfile
|
||||
# Install kepubify for EPUB→KEPUB conversion
|
||||
RUN wget -O /usr/bin/kepubify https://github.com/pgaskin/kepubify/releases/latest/download/kepubify-linux-64bit \
|
||||
&& chmod +x /usr/bin/kepubify
|
||||
```
|
||||
|
||||
Or install Calibre for ebook-convert:
|
||||
|
||||
```dockerfile
|
||||
# Install Calibre for ebook-convert
|
||||
RUN apt-get update && apt-get install -y calibre
|
||||
```
|
||||
|
||||
## API Usage
|
||||
|
||||
### Download KEPUB via OPDS
|
||||
|
||||
```http
|
||||
GET /opds/devices/{deviceId}/download/{bookId}?format=kepub
|
||||
```
|
||||
|
||||
**Response Headers:**
|
||||
- `Content-Type`: application/vnd.kobo+xml+zip
|
||||
- `Content-Disposition`: attachment; filename="book.kepub.epub"
|
||||
- `X-Bookhoard-UUID`: uuid-123
|
||||
- `X-Bookhoard-KEPUB-SHA256`: abc123... (KEPUB-specific hash)
|
||||
|
||||
## Database Schema
|
||||
|
||||
### media_item_formats Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE media_item_formats (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
media_item_id UUID REFERENCES media_items(id) ON DELETE CASCADE,
|
||||
format_type VARCHAR(10) NOT NULL, -- 'epub', 'kepub', 'pdf', 'cbz'
|
||||
file_path VARCHAR(500),
|
||||
file_sha256 CHAR(64), -- Hash for THIS format version
|
||||
file_size_bytes BIGINT,
|
||||
mime_type VARCHAR(100),
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
converted_from_format_id UUID REFERENCES media_item_formats(id), -- Track conversion chain
|
||||
UNIQUE(media_item_id, format_type)
|
||||
);
|
||||
```
|
||||
|
||||
**Example Data:**
|
||||
```
|
||||
Row 1: media_item_id=uuid-123, format_type='epub', file_sha256='abc123...'
|
||||
Row 2: media_item_id=uuid-123, format_type='kepub', file_sha256='xyz789...', converted_from_format_id=Row1.id
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Service Methods
|
||||
|
||||
#### ConvertEPUBToKEPUB
|
||||
|
||||
Converts an EPUB file to KEPUB format with hash storage.
|
||||
|
||||
**Parameters:**
|
||||
- `ctx context.Context`: Request context
|
||||
- `mediaItemID pgtype.UUID`: ID of the media item
|
||||
- `epubPath string`: Path to the source EPUB file
|
||||
|
||||
**Returns:**
|
||||
- `*ConvertedKEPUB`: Contains path, SHA256 hash, and cached status
|
||||
- `error`: Conversion error if any
|
||||
|
||||
**Behavior:**
|
||||
1. Checks cache for existing KEPUB (recent conversions are reused)
|
||||
2. Performs EPUB→KEPUB conversion using kepubify or ebook-convert
|
||||
3. Calculates SHA-256 hash of converted file
|
||||
4. Stores format record in database with dual hash
|
||||
5. Returns converted file path and hash
|
||||
|
||||
### Conversion Tools
|
||||
|
||||
The service tries conversion tools in this order:
|
||||
|
||||
1. **kepubify** (recommended): Purpose-built KEPUB converter
|
||||
- Faster and more reliable for Kobo devices
|
||||
- Download: https://github.com/pgaskin/kepubify/releases
|
||||
|
||||
2. **ebook-convert** (fallback): Part of Calibre suite
|
||||
- More versatile but slower
|
||||
- Requires full Calibre installation
|
||||
|
||||
### Cache Invalidation
|
||||
|
||||
Converted KEPUB files are cached for 24 hours by default. This TTL is configurable via:
|
||||
- Environment variable: `BOOKHOARD_CONVERSION_CACHE_TTL`
|
||||
- Code: `conversionCacheTTL` field in `ConversionService`
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
go test ./internal/services/... -v
|
||||
```
|
||||
|
||||
### Manual Testing with Bruno
|
||||
|
||||
Use the provided Bruno test:
|
||||
- `bruno/opds/Download Book KEPUB (On-the-fly Conversion).bru`
|
||||
|
||||
This test verifies:
|
||||
- KEPUB hash header is present
|
||||
- Hash is 64 characters (SHA-256 format)
|
||||
- Bookhoard UUID header is present
|
||||
- Content-Type is correct for KEPUB
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Conversion Failures
|
||||
|
||||
**Problem**: KEPUB conversion fails
|
||||
**Solution**:
|
||||
1. Check if kepubify or ebook-convert is installed
|
||||
2. Verify EPUB file is valid and accessible
|
||||
3. Check cache directory permissions: `/var/bookhoard/cache/kepub`
|
||||
4. Review conversion logs for specific error messages
|
||||
|
||||
### Cache Issues
|
||||
|
||||
**Problem**: Converted files not being cached
|
||||
**Solution**:
|
||||
1. Verify cache directory exists and is writable
|
||||
2. Check `BOOKHOARD_CONVERSION_CACHE_DIR` environment variable
|
||||
3. Ensure database can create media_item_formats records
|
||||
|
||||
### Hash Mismatches
|
||||
|
||||
**Problem**: Progress sync fails after conversion
|
||||
**Solution**:
|
||||
1. Verify dual hash storage: both EPUB and KEPUB hashes should exist in `media_item_formats`
|
||||
2. Check `X-Bookhoard-KEPUB-SHA256` header in response
|
||||
3. Ensure `converted_from_format_id` links KEPUB to source EPUB
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### First Conversion
|
||||
- **Time**: 2-5 seconds per book (depends on file size)
|
||||
- **CPU**: Medium (single-threaded conversion)
|
||||
- **I/O**: Read EPUB, write KEPUB to cache
|
||||
|
||||
### Cached Conversions
|
||||
- **Time**: < 100ms (database lookup + file serve)
|
||||
- **CPU**: Minimal
|
||||
- **I/O**: Read cached KEPUB file
|
||||
|
||||
### Storage Requirements
|
||||
- **Cache Size**: ~1.1x original EPUB size (KEPUB is slightly larger)
|
||||
- **Database**: ~200 bytes per converted format record
|
||||
- **Recommendation**: 10 GB cache per 1000 books
|
||||
|
||||
## Security
|
||||
|
||||
### File Access
|
||||
- Conversion service only processes files from library folders
|
||||
- Converted files are stored in secure cache directory
|
||||
- Original files are never modified
|
||||
|
||||
### Input Validation
|
||||
- All file paths are validated before conversion
|
||||
- Media item IDs are verified against database
|
||||
- User access permissions are checked via OPDS handler
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements:
|
||||
1. **Async Conversion**: Queue conversions for background processing
|
||||
2. **Batch Conversion**: Pre-convert entire libraries during off-hours
|
||||
3. **Format Variants**: Support PDF→EPUB, CBZ→EPUB, etc.
|
||||
4. **Quality Settings**: Configurable conversion quality/size tradeoffs
|
||||
5. **Distributed Caching**: Share cache across multiple server instances
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,396 +0,0 @@
|
||||
# Device Cap Implementation - Task 2
|
||||
|
||||
**Date**: February 1, 2026
|
||||
**Status**: ✅ COMPLETE
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Implemented admin-configurable device cap per user as specified in the session requirements. This allows administrators to control the maximum number of devices each user can register.
|
||||
|
||||
---
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. Database Schema
|
||||
|
||||
**File**: `database/schema/schema.sql`
|
||||
|
||||
Added `max_devices` column to `users` table:
|
||||
```sql
|
||||
max_devices INTEGER DEFAULT 10
|
||||
```
|
||||
|
||||
- **Default Value**: 10 devices per user
|
||||
- **Constraints**: 1-100 devices (validated in handler)
|
||||
- **Purpose**: Prevent excessive device registrations per user
|
||||
|
||||
### 2. Database Queries
|
||||
|
||||
**File**: `internal/database/queries/queries.sql`
|
||||
|
||||
Added two new queries:
|
||||
|
||||
#### UpdateUserMaxDevices
|
||||
```sql
|
||||
-- name: UpdateUserMaxDevices :exec
|
||||
UPDATE users SET max_devices = $2, updated_at = NOW() WHERE id = $1;
|
||||
```
|
||||
- Updates max devices limit for a specific user
|
||||
- Parameters: user_id (UUID), max_devices (integer)
|
||||
|
||||
#### CountUserDevices
|
||||
```sql
|
||||
-- name: CountUserDevices :one
|
||||
SELECT COUNT(*) FROM devices WHERE user_id = $1;
|
||||
```
|
||||
- Counts current devices for a user
|
||||
- Useful for validation and display
|
||||
|
||||
### 3. Handler Implementation
|
||||
|
||||
**File**: `internal/handlers/auth.go`
|
||||
|
||||
Added new handler method:
|
||||
|
||||
#### UpdateUserMaxDevicesRequest
|
||||
```go
|
||||
type UpdateUserMaxDevicesRequest struct {
|
||||
MaxDevices int32 `json:"max_devices" validate:"required,min=1,max=100"`
|
||||
}
|
||||
```
|
||||
|
||||
#### UpdateUserMaxDevices Handler
|
||||
```go
|
||||
func (h *AuthHandler) UpdateUserMaxDevices(c echo.Context) error {
|
||||
userID := c.Param("id")
|
||||
if userID == "" {
|
||||
return c.JSON(http.StatusBadRequest, map[string]string{"error": "user id required"})
|
||||
}
|
||||
|
||||
var req UpdateUserMaxDevicesRequest
|
||||
if err := c.Bind(&req); err != nil {
|
||||
return c.JSON(http.StatusBadRequest, map[string]string{"error": "invalid request"})
|
||||
}
|
||||
|
||||
if err := c.Validate(&req); err != nil {
|
||||
return c.JSON(http.StatusBadRequest, map[string]string{"error": err.Error()})
|
||||
}
|
||||
|
||||
userUUID, err := uuid.Parse(userID)
|
||||
if err != nil {
|
||||
return c.JSON(http.StatusBadRequest, map[string]string{"error": "invalid user id"})
|
||||
}
|
||||
|
||||
err = h.db.UpdateUserMaxDevices(c.Request().Context(), database.UpdateUserMaxDevicesParams{
|
||||
ID: pgtype.UUID{Bytes: userUUID, Valid: true},
|
||||
MaxDevices: pgtype.Int4{Int32: req.MaxDevices, Valid: true},
|
||||
})
|
||||
if err != nil {
|
||||
return c.JSON(http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
}
|
||||
|
||||
return c.JSON(http.StatusOK, map[string]string{"message": "max devices updated"})
|
||||
}
|
||||
```
|
||||
|
||||
**Features**:
|
||||
- Validates user ID format (UUID)
|
||||
- Validates max_devices range (1-100)
|
||||
- Requires admin authentication
|
||||
- Updates user's max_devices in database
|
||||
- Returns success/error messages
|
||||
|
||||
### 4. UserList Update
|
||||
|
||||
**File**: `internal/handlers/auth.go`
|
||||
|
||||
Updated `UserList` struct to include max_devices:
|
||||
|
||||
```go
|
||||
type UserList struct {
|
||||
ID string `json:"id"`
|
||||
Email string `json:"email"`
|
||||
Username string `json:"username"`
|
||||
FirstName string `json:"first_name"`
|
||||
LastName string `json:"last_name"`
|
||||
Theme string `json:"theme"`
|
||||
Role string `json:"role"`
|
||||
MaxDevices int32 `json:"max_devices"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Route Registration
|
||||
|
||||
**File**: `cmd/server/main.go`
|
||||
|
||||
Added new admin route:
|
||||
|
||||
```go
|
||||
admin.PUT("/users/:id/max-devices", authHandler.UpdateUserMaxDevices)
|
||||
```
|
||||
|
||||
- **Path**: `/api/auth/users/:id/max-devices`
|
||||
- **Method**: PUT
|
||||
- **Auth**: Admin only (uses AdminMiddleware)
|
||||
- **Validation**: 1-100 devices
|
||||
|
||||
### 6. SQLC Code Generation
|
||||
|
||||
**File**: `internal/database/sqlc.yaml`
|
||||
|
||||
- Regenerated database code using `sqlc generate`
|
||||
- Created `UpdateUserMaxDevices` and `UpdateUserMaxDevicesParams` types
|
||||
- Created `CountUserDevices` function
|
||||
|
||||
### 7. Bruno API Collection
|
||||
|
||||
Created 4 Bruno files for API testing:
|
||||
|
||||
#### 1. Update User Max Devices (Documentation)
|
||||
- **Path**: `bruno/user/admin/Update User Max Devices.bru`
|
||||
- Contains complete API documentation
|
||||
- Includes all validation rules
|
||||
- Example payloads for common values
|
||||
|
||||
#### 2. Update User Max Devices - Success
|
||||
- **Path**: `bruno/user/admin/Update User Max Devices - Success.bru`
|
||||
- Tests successful update to 5 devices
|
||||
- Expected: 200 OK
|
||||
|
||||
#### 3. Update User Max Devices - Invalid Zero
|
||||
- **Path**: `bruno/user/admin/Update User Max Devices - Invalid Zero.bru`
|
||||
- Tests validation of zero devices (below minimum)
|
||||
- Expected: 400 Bad Request
|
||||
|
||||
#### 4. Update User Max Devices - Exceeds Maximum
|
||||
- **Path**: `bruno/user/admin/Update User Max Devices - Invalid Too High.bru`
|
||||
- Tests validation of 101 devices (above maximum)
|
||||
- Expected: 400 Bad Request
|
||||
|
||||
#### 5. Update User Max Devices - Missing ID
|
||||
- **Path**: `bruno/user/admin/Update User Max Devices - Missing ID.bru`
|
||||
- Tests missing user ID in URL
|
||||
- Expected: 400 Bad Request
|
||||
|
||||
### 8. Go Tests
|
||||
|
||||
**File**: `cmd/server/tests/device_cap_test.go`
|
||||
|
||||
Created comprehensive test suite with 7 test functions:
|
||||
|
||||
#### TestUpdateUserMaxDevices
|
||||
Tests successful updates:
|
||||
- Update to 5 devices
|
||||
- Update to 10 devices (default)
|
||||
- Update to 50 devices
|
||||
- Update to 100 devices (maximum)
|
||||
|
||||
#### TestUpdateUserMaxDevicesValidation
|
||||
Tests validation rules:
|
||||
- Zero devices (below minimum)
|
||||
- Negative devices
|
||||
- 101 devices (above maximum)
|
||||
- 1000 devices (far above maximum)
|
||||
|
||||
#### TestUpdateUserMaxDevicesAuth
|
||||
Tests authentication:
|
||||
- No authorization token
|
||||
- Non-admin user attempting to access endpoint
|
||||
- Expected: 401 Unauthorized or 403 Forbidden
|
||||
|
||||
#### TestUpdateUserMaxDevicesNonExistentUser
|
||||
Tests with non-existent user ID:
|
||||
- Expected: 500 Internal Server Error or 404 Not Found
|
||||
|
||||
#### TestUpdateUserMaxDevicesMissingUserID
|
||||
Tests with missing user ID in URL:
|
||||
- Expected: 400 Bad Request
|
||||
|
||||
#### TestListUsersIncludesMaxDevices
|
||||
Tests that max_devices field is included in user list response:
|
||||
- Ensures backward compatibility
|
||||
- Validates new field is present in API response
|
||||
|
||||
#### Helper Functions
|
||||
- `createAdminUser`: Creates admin user for testing
|
||||
- `createTestUserForMaxDevices`: Creates regular user for testing
|
||||
- `getAdminToken`: Retrieves admin JWT token
|
||||
- `loginTestUserByCredentials`: Logs in user with credentials
|
||||
|
||||
---
|
||||
|
||||
## API Specification
|
||||
|
||||
### PUT /api/auth/users/:id/max-devices
|
||||
|
||||
Updates the maximum number of devices a user can register.
|
||||
|
||||
**Authentication**: Required (Admin only)
|
||||
|
||||
**URL Parameters**:
|
||||
- `id` (string, required): User ID (UUID)
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"max_devices": 10
|
||||
}
|
||||
```
|
||||
|
||||
**Request Validation**:
|
||||
- `max_devices` (integer, required): Must be between 1 and 100
|
||||
|
||||
**Response** (Success):
|
||||
```json
|
||||
{
|
||||
"message": "max devices updated"
|
||||
}
|
||||
```
|
||||
|
||||
**Response** (Error):
|
||||
```json
|
||||
{
|
||||
"error": "validation error"
|
||||
}
|
||||
```
|
||||
|
||||
**Status Codes**:
|
||||
- `200`: Success
|
||||
- `400`: Bad Request (missing id, invalid UUID, validation error)
|
||||
- `401`: Unauthorized (missing or invalid token)
|
||||
- `403`: Forbidden (non-admin user)
|
||||
- `500`: Internal Server Error
|
||||
|
||||
### GET /api/auth/users
|
||||
|
||||
Updated to include `max_devices` field in response:
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "uuid",
|
||||
"email": "user@example.com",
|
||||
"username": "username",
|
||||
"first_name": "John",
|
||||
"last_name": "Doe",
|
||||
"role": "user",
|
||||
"theme": "tokyo-night",
|
||||
"max_devices": 10,
|
||||
"created_at": "2026-01-31T12:00:00Z",
|
||||
"updated_at": "2026-01-31T12:00:00Z"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
- ✅ Created comprehensive test suite
|
||||
- ✅ All tests compile successfully
|
||||
- ✅ Tests cover success cases
|
||||
- ✅ Tests cover validation
|
||||
- ✅ Tests cover authentication
|
||||
- ✅ Tests cover edge cases
|
||||
|
||||
### Bruno Tests
|
||||
- ✅ Created 4 test scenarios
|
||||
- ✅ Success case
|
||||
- ✅ Validation failure cases
|
||||
- ✅ Missing parameters
|
||||
|
||||
### Manual Testing Checklist
|
||||
- [ ] Admin can update max devices to valid values
|
||||
- [ ] Non-admin users cannot update max devices
|
||||
- [ ] Validation rejects values < 1
|
||||
- [ ] Validation rejects values > 100
|
||||
- [ ] Invalid user ID returns appropriate error
|
||||
- [ ] Missing user ID returns 400 error
|
||||
- [ ] User list includes max_devices field
|
||||
- [ ] Default value of 10 is enforced for new users
|
||||
|
||||
---
|
||||
|
||||
## Integration Notes
|
||||
|
||||
### Device Registration Enforcement
|
||||
|
||||
The `max_devices` setting should be enforced during device registration:
|
||||
|
||||
**In `InitiateRegistration` handler** (`internal/handlers/devices.go`):
|
||||
```go
|
||||
// Count user's current devices
|
||||
deviceCount, err := h.db.CountUserDevices(ctx, userID)
|
||||
|
||||
if deviceCount >= user.MaxDevices {
|
||||
return c.JSON(http.StatusForbidden, map[string]string{
|
||||
"error": "device limit reached",
|
||||
"max_devices": user.MaxDevices,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### Backward Compatibility
|
||||
|
||||
- ✅ Default value of 10 maintains existing behavior
|
||||
- ✅ Existing users without max_devices set use default
|
||||
- ✅ User list response enhanced with new field
|
||||
- ✅ No breaking changes to existing endpoints
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Admin-Only Access**: Endpoint protected by AdminMiddleware
|
||||
2. **Input Validation**: Strict validation of max_devices range (1-100)
|
||||
3. **UUID Validation**: User ID validated as proper UUID format
|
||||
4. **SQL Injection Protection**: Uses sqlc parameterized queries
|
||||
5. **Rate Limiting**: Inherits existing rate limiting from middleware
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
1. **Database Indexes**: Consider adding index on (user_id) for CountUserDevices
|
||||
2. **Caching**: User max_devices could be cached for frequent checks
|
||||
3. **Batch Operations**: Consider batch updates for multiple users
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. **Per-Device-Type Caps**: Allow different limits for different device types
|
||||
2. **Time-Based Limits**: Device limits that expire after time period
|
||||
3. **Plan-Based Limits**: Different device caps based on user subscription tier
|
||||
4. **Audit Logging**: Log when max_devices is changed (who changed, from, to, when)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
✅ **Complete**:
|
||||
- Database schema updated with max_devices column
|
||||
- Database queries added (UpdateUserMaxDevices, CountUserDevices)
|
||||
- Handler implemented with full validation
|
||||
- Route registered as admin-only
|
||||
- Bruno API collection created (4 files)
|
||||
- Go test suite created (7 test functions, 20+ test cases)
|
||||
- User list updated to include new field
|
||||
|
||||
**Production Ready**: Yes
|
||||
**Breaking Changes**: None
|
||||
**Backward Compatible**: Yes
|
||||
|
||||
---
|
||||
|
||||
**Next Steps**:
|
||||
1. Add device limit enforcement in device registration flow
|
||||
2. Update user management UI to display/edit max_devices
|
||||
3. Consider adding audit logging for admin actions
|
||||
4. Add user notifications when device limit is reached
|
||||
+229
@@ -0,0 +1,229 @@
|
||||
# Bookhoard Documentation Index
|
||||
|
||||
Complete guide to Bookhoard documentation. Find what you need quickly.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Links
|
||||
|
||||
### For New Users
|
||||
1. [README.md](../README.md) - **Start here!** Project overview and quick start
|
||||
2. [docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md) - Understanding and using sync
|
||||
3. [docs/devices/KOBO_SETUP.md](devices/KOBO_SETUP.md) - Kobo e-reader setup
|
||||
4. [docs/devices/KOREADER_SETUP.md](devices/KOREADER_SETUP.md) - KOReader setup
|
||||
|
||||
### For Self-Hosting
|
||||
1. [docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md) - Deployment and troubleshooting
|
||||
2. [.env.example](../.env.example) - Secrets configuration (JWT and DB password)
|
||||
|
||||
### For Contributors
|
||||
1. [docs/contributing/DEVELOPMENT.md](contributing/DEVELOPMENT.md) - Development workflow and architecture
|
||||
2. [docs/API_REFERENCE.md](API_REFERENCE.md) - Complete API documentation
|
||||
3. [docs/COLLECTIONS_API.md](COLLECTIONS_API.md) - Collections API
|
||||
4. [docs/api/WEBSOCKET_API.md](api/WEBSOCKET_API.md) - WebSocket protocol
|
||||
|
||||
---
|
||||
|
||||
## 📚 Documentation by Topic
|
||||
|
||||
### Getting Started
|
||||
- **[README.md](../README.md)** - Project overview, features, quick start guide
|
||||
- **[docs/contributing/DEVELOPMENT.md](contributing/DEVELOPMENT.md)** - Development environment setup
|
||||
|
||||
### Deployment & Operations
|
||||
- **[docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md)** - Common deployment issues and solutions
|
||||
- **[.env.example](../.env.example)** - Required secrets (JWT_SECRET, DBPASS)
|
||||
- **[docker-compose.yml](../docker-compose.yml)** - Operational configuration with defaults
|
||||
- **[Makefile](../Makefile)** - Build and test commands
|
||||
|
||||
### Using Sync Features
|
||||
- **[docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md)** - Universal sync user guide
|
||||
- Understanding sync
|
||||
- Book matching and auto-linking
|
||||
- Conflict resolution
|
||||
- Best practices
|
||||
|
||||
### Device Setup
|
||||
- **[docs/devices/KOBO_SETUP.md](devices/KOBO_SETUP.md)** - Kobo e-reader configuration
|
||||
- Device registration
|
||||
- Sync configuration
|
||||
- OPDS wireless book delivery
|
||||
- Troubleshooting
|
||||
|
||||
- **[docs/devices/KOREADER_SETUP.md](devices/KOREADER_SETUP.md)** - KOReader configuration
|
||||
- Installation on Kindle/Kobo/PocketBook
|
||||
- Sync setup
|
||||
- OPDS catalog access
|
||||
- Troubleshooting
|
||||
|
||||
### API Documentation
|
||||
- **[docs/API_REFERENCE.md](API_REFERENCE.md)** - Complete REST API reference
|
||||
- Authentication
|
||||
- User management
|
||||
- Libraries
|
||||
- Media items
|
||||
- Reading progress
|
||||
- Notes & highlights
|
||||
- Analytics
|
||||
- Book matching
|
||||
- OPDS
|
||||
- Sync protocols (KOReader, Kobo)
|
||||
- WebSocket
|
||||
|
||||
- **[docs/COLLECTIONS_API.md](COLLECTIONS_API.md)** - Collections API
|
||||
- Create and manage collections
|
||||
- Auto-assign rules
|
||||
- Test rules
|
||||
- Bulk operations
|
||||
- Device shelf mappings
|
||||
|
||||
- **[docs/api/WEBSOCKET_API.md](api/WEBSOCKET_API.md)** - WebSocket protocol
|
||||
- Connection flow
|
||||
- Message format
|
||||
- Real-time sync broadcasts
|
||||
- Authentication
|
||||
|
||||
### Contributing
|
||||
- **[docs/contributing/DEVELOPMENT.md](contributing/DEVELOPMENT.md)** - Development guide
|
||||
- Architecture overview
|
||||
- Directory structure
|
||||
- Local development setup
|
||||
- Testing guidelines
|
||||
- Code style
|
||||
- Deployment
|
||||
|
||||
- **[PROJECT_GUIDELINES.md](../PROJECT_GUIDELINES.md)** - Development rules and standards
|
||||
- Critical prohibitions
|
||||
- Mandatory requirements
|
||||
- Error recovery protocol
|
||||
|
||||
### Reference
|
||||
- **[go.mod](../go.mod)** - Go dependencies
|
||||
- **[database/schema/schema.sql](../database/schema/schema.sql)** - Database schema
|
||||
- **[bruno/](../bruno/)** - API test collections
|
||||
|
||||
---
|
||||
|
||||
## 📖 Reading Path by Role
|
||||
|
||||
### Self-Hoster / End User
|
||||
**Goal**: Set up and use Bookhoard for reading
|
||||
|
||||
1. Start with [README.md](../README.md) - Understand what Bookhoard is
|
||||
2. Follow quick start in README.md to get running
|
||||
3. Set up your device:
|
||||
- Kobo: [docs/devices/KOBO_SETUP.md](devices/KOBO_SETUP.md)
|
||||
- KOReader: [docs/devices/KOREADER_SETUP.md](devices/KOREADER_SETUP.md)
|
||||
4. Learn about sync: [docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md)
|
||||
5. If issues arise: [docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md)
|
||||
|
||||
|
||||
|
||||
### Developer
|
||||
**Goal**: Contribute to Bookhoard or integrate with it
|
||||
|
||||
1. Start with [README.md](../README.md) - Project overview
|
||||
2. Read [docs/contributing/DEVELOPMENT.md](contributing/DEVELOPMENT.md) - Architecture and setup
|
||||
3. Review [docs/API_REFERENCE.md](API_REFERENCE.md) - API endpoints
|
||||
4. Check [PROJECT_GUIDELINES.md](../PROJECT_GUIDELINES.md) - Development rules
|
||||
5. Explore codebase and contribute!
|
||||
|
||||
### API Integrator
|
||||
**Goal**: Build integration with Bookhoard
|
||||
|
||||
1. Review [README.md](../README.md) - Feature overview
|
||||
2. Study [docs/API_REFERENCE.md](API_REFERENCE.md) - All endpoints
|
||||
3. Check specialized docs:
|
||||
- Collections: [docs/COLLECTIONS_API.md](COLLECTIONS_API.md)
|
||||
- WebSocket: [docs/api/WEBSOCKET_API.md](api/WEBSOCKET_API.md)
|
||||
- Sync: [docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md)
|
||||
4. Test with [bruno/](../bruno/) collections
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Quick Find
|
||||
|
||||
### "How do I..."
|
||||
| ...do this? | See this document |
|
||||
|-------------|------------------|
|
||||
| ...install Bookhoard? | [README.md](../README.md) - Quick Start |
|
||||
| ...set up my Kobo? | [docs/devices/KOBO_SETUP.md](devices/KOBO_SETUP.md) |
|
||||
| ...set up KOReader? | [docs/devices/KOREADER_SETUP.md](devices/KOREADER_SETUP.md) |
|
||||
| ...understand sync? | [docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md) |
|
||||
| ...resolve conflicts? | [docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md) - Managing Conflicts |
|
||||
| ...match books? | [docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md) - Book Matching |
|
||||
| ...troubleshoot deployment? | [docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md) |
|
||||
| ...use the API? | [docs/API_REFERENCE.md](API_REFERENCE.md) |
|
||||
| ...set up development? | [docs/contributing/DEVELOPMENT.md](contributing/DEVELOPMENT.md) |
|
||||
| ...contribute code? | [docs/contributing/DEVELOPMENT.md](contributing/DEVELOPMENT.md) - Contributing |
|
||||
|
||||
### "Where is..."
|
||||
| ...this information? | See this document |
|
||||
|-------------------|------------------|
|
||||
| ...features list? | [README.md](../README.md) |
|
||||
| ...database schema? | [database/schema/schema.sql](../database/schema/schema.sql) |
|
||||
| ...API endpoints? | [docs/API_REFERENCE.md](API_REFERENCE.md) |
|
||||
| ...secrets config? | [.env.example](../.env.example) |
|
||||
| ...operational config? | [docker-compose.yml](../docker-compose.yml) |
|
||||
| ...deployment issues? | [docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md) |
|
||||
|
||||
---
|
||||
|
||||
## 📊 Documentation Statistics
|
||||
|
||||
| File | Lines | Purpose | Audience |
|
||||
|------|-------|---------|----------|
|
||||
| README.md | 150 | Overview & quick start | Everyone |
|
||||
| contributing/DEVELOPMENT.md | 450 | Development workflow | Contributors |
|
||||
| API_REFERENCE.md | 1,300+ | Complete REST API | Developers, integrators |
|
||||
| COLLECTIONS_API.md | 494 | Collections API | Developers, integrators |
|
||||
| SYNC_USER_GUIDE.md | 350+ | Sync usage guide | End users |
|
||||
| TROUBLESHOOTING.md | 300 | Deployment troubleshooting | Self-hosters |
|
||||
| KOBO_SETUP.md | 598 | Kobo setup | Kobo users |
|
||||
| KOREADER_SETUP.md | 504 | KOReader setup | KOReader users |
|
||||
| WEBSOCKET_API.md | 676 | WebSocket protocol | Developers |
|
||||
| PROJECT_GUIDELINES.md | 250 | Development rules | Developers |
|
||||
|
||||
**Total**: ~5,000 lines of comprehensive documentation
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Common Tasks
|
||||
|
||||
### Set up a new device
|
||||
1. Device setup guide: [docs/devices/KOBO_SETUP.md](devices/KOBO_SETUP.md) or [docs/devices/KOREADER_SETUP.md](devices/KOREADER_SETUP.md)
|
||||
2. Sync overview: [docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md)
|
||||
3. Troubleshooting: Device-specific setup guides
|
||||
|
||||
### Troubleshoot sync issues
|
||||
1. Check [docs/SYNC_USER_GUIDE.md](SYNC_USER_GUIDE.md) - "Managing Conflicts" and "Best Practices"
|
||||
2. Review device-specific guide for common issues
|
||||
3. Check [docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md) for general issues
|
||||
|
||||
### Integrate with Bookhoard API
|
||||
1. Start with [docs/API_REFERENCE.md](API_REFERENCE.md) - Complete API reference
|
||||
2. Check [docs/COLLECTIONS_API.md](COLLECTIONS_API.md) for collections
|
||||
3. Review [docs/api/WEBSOCKET_API.md](api/WEBSOCKET_API.md) for real-time updates
|
||||
4. Use [bruno/](../bruno/) test collections as examples
|
||||
|
||||
### Deploy to production
|
||||
1. Follow [README.md](../README.md) quick start
|
||||
2. Configure environment: [.env.example](../.env.example)
|
||||
3. Review [docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md) for common issues
|
||||
4. Check [docs/contributing/DEVELOPMENT.md](contributing/DEVELOPMENT.md) for performance tuning
|
||||
|
||||
---
|
||||
|
||||
## 📝 Contributing to Documentation
|
||||
|
||||
When adding new features:
|
||||
1. Update [README.md](../README.md) - Add to features list if user-facing
|
||||
2. Update [docs/API_REFERENCE.md](API_REFERENCE.md) - Document new endpoints
|
||||
3. Add/update tests in [bruno/](../bruno/)
|
||||
4. Update relevant guides (SYNC_USER_GUIDE.md, device guides, etc.)
|
||||
5. Keep [PROJECT_GUIDELINES.md](../PROJECT_GUIDELINES.md) in mind
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-02-01
|
||||
**Bookhoard Version**: 1.0
|
||||
@@ -1,192 +0,0 @@
|
||||
# Legacy Code Cleanup - Phases 1-3 Complete
|
||||
|
||||
## Summary
|
||||
|
||||
Successfully completed Phases 1-3 of the legacy migration code cleanup for Bookhoard.
|
||||
|
||||
---
|
||||
|
||||
## ✅ Phase 1: Documentation Cleanup (Complete)
|
||||
|
||||
### Changes Made:
|
||||
|
||||
1. **internal/handlers/ebook.go**
|
||||
- Removed misleading backward compatibility comments (lines 1294-1297)
|
||||
- Cleaned up references to non-existent `GetEbookNotes` and `GetEbookHighlights` handlers
|
||||
|
||||
2. **database/schema/schema.sql**
|
||||
- Removed historical migration comments (lines 370-371)
|
||||
- Deleted reference to `user_ebook_folders` table replacement
|
||||
|
||||
3. **README.md**
|
||||
- Removed "Ebook Compatibility (Backward Compatible)" section (lines 180-189)
|
||||
- Removed backward compatibility bullet point from Database Schema section (line 366)
|
||||
- Removed "Backward Compatibility Views" section from Database documentation (lines 590-592)
|
||||
|
||||
**Impact**: Cleaner documentation, no behavioral changes
|
||||
|
||||
---
|
||||
|
||||
## ✅ Phase 2: Dead Code Removal (Complete)
|
||||
|
||||
### Changes Made:
|
||||
|
||||
1. **internal/handlers/auth.go**
|
||||
- Deleted `AddEbookFolder` handler function (lines 566-569)
|
||||
- Deleted `GetEbookFolders` handler function (lines 571-574)
|
||||
- Deleted `DeleteEbookFolder` handler function (lines 576-579)
|
||||
- Deleted `DeleteEbookFolderRequest` struct (lines 562-564)
|
||||
|
||||
**Total**: ~20 lines of dead code removed
|
||||
|
||||
**Impact**: No behavioral changes (routes already unregistered, returning HTTP 410 Gone)
|
||||
|
||||
---
|
||||
|
||||
## ✅ Phase 3: Test Suite Cleanup (Complete)
|
||||
|
||||
### Files Deleted:
|
||||
|
||||
1. **cmd/server/tests/isbn_and_library_test.go** (507 lines)
|
||||
- All tests used deprecated `/api/ebooks` endpoint
|
||||
- No equivalent library-related tests to preserve
|
||||
- Tests covered:
|
||||
- ISBN normalization (8 test cases)
|
||||
- Library requirement validation
|
||||
- ISBN edge cases
|
||||
- Library auto-selection
|
||||
|
||||
2. **cmd/server/tests/edge_cases_test.go** (82 lines removed)
|
||||
- Removed `TestPaginationAndFiltering` function
|
||||
- Deleted 4 pagination test cases using `/api/ebooks` endpoint:
|
||||
- Negative limit
|
||||
- Negative offset
|
||||
- Very large limit
|
||||
- Valid pagination
|
||||
|
||||
**Total**: 589 lines of outdated tests removed
|
||||
|
||||
**Impact**: Cleaner test suite, no failing tests
|
||||
|
||||
---
|
||||
|
||||
## ✅ Phase 3: Equivalent Tests Created (Complete)
|
||||
|
||||
### New Test File: **cmd/server/tests/media_item_isbn_test.go** (467 lines)
|
||||
|
||||
Created comprehensive replacement tests using `/api/media-items` endpoint:
|
||||
|
||||
1. **TestMediaItemISBNNormalization**
|
||||
- 8 ISBN-10/ISBN-13 normalization test cases
|
||||
- Tests hyphens, spaces, mixed formats
|
||||
- Uses real API calls (not mocks)
|
||||
|
||||
2. **TestMediaItemISBNEdgeCases**
|
||||
- Empty ISBN handling
|
||||
- Multiple hyphens normalization
|
||||
- Trailing/leading hyphen removal
|
||||
|
||||
3. **TestMediaItemsPagination**
|
||||
- Valid pagination parameters
|
||||
- Pagination with offset
|
||||
- Negative limit validation
|
||||
- Negative offset validation
|
||||
- Maximum limit enforcement (1000 cap)
|
||||
|
||||
4. **TestMediaItemLibraryRequirement**
|
||||
- Media-item creation without library (should fail)
|
||||
- Media-item creation with existing library (should succeed)
|
||||
|
||||
5. **TestUpdateMediaItemISBN**
|
||||
- Update media-item with ISBN normalization
|
||||
|
||||
**Helper Function Added**:
|
||||
- `createTestLibrary(t, ts, token, name)` - Creates test library and returns ID
|
||||
|
||||
**Impact**: Modern, working tests that exercise actual API functionality
|
||||
|
||||
---
|
||||
|
||||
## 📊 Overall Statistics
|
||||
|
||||
| Category | Files Modified | Files Deleted | Files Created | Lines Removed | Lines Added |
|
||||
|----------|----------------|----------------|----------------|---------------|-------------|
|
||||
| Documentation | 3 | 0 | 0 | ~30 | 0 |
|
||||
| Dead Code | 1 | 0 | 0 | ~20 | 0 |
|
||||
| Old Tests | 1 | 1 | 0 | ~82 | 0 |
|
||||
| New Tests | 0 | 0 | 1 | 507 | 467 |
|
||||
| **TOTAL** | **5** | **1** | **1** | **~639** | **467** |
|
||||
|
||||
**Net Result**: -172 lines of code, significantly cleaner codebase
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Status
|
||||
|
||||
### Tests Deleted:
|
||||
- ✅ `isbn_and_library_test.go` - All using `/api/ebooks` (deprecated)
|
||||
- ✅ `edge_cases_test.go` - Pagination tests using `/api/ebooks` (deprecated)
|
||||
|
||||
### Tests Created:
|
||||
- ✅ `media_item_isbn_test.go` - Comprehensive replacement using `/api/media-items`
|
||||
|
||||
### Tests Preserved:
|
||||
- ✅ `library_test.go` - Contains equivalent pagination tests for `/api/media-items`
|
||||
- ✅ All other test files remain unchanged
|
||||
|
||||
---
|
||||
|
||||
## ⏭️ Next Steps: Phase 4 (Not Implemented Yet)
|
||||
|
||||
### Database Views Removal
|
||||
|
||||
**5 backward compatibility views to potentially drop**:
|
||||
1. `ebooks` view (lines 122-128)
|
||||
2. `ebook_reading_progress` view (lines 160-168)
|
||||
3. `ebook_ratings` view (lines 340-348)
|
||||
4. `ebook_notes` view (lines 350-358)
|
||||
5. `ebook_highlights` view (lines 360-368)
|
||||
|
||||
**Prerequisites**:
|
||||
1. ✅ User has requested verification of view usage first
|
||||
2. Search codebase for view references
|
||||
3. Run full test suite to ensure no dependencies
|
||||
4. Check Bruno API collections
|
||||
5. Verify no direct SQL queries use views
|
||||
|
||||
**Action Items** (When approved):
|
||||
1. Grep codebase for view names
|
||||
2. Check application logs
|
||||
3. Run integration tests
|
||||
4. If safe, drop views from schema.sql
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Success Criteria - All Met
|
||||
|
||||
- ✅ Documentation cleaned up (no backward compatibility mentions)
|
||||
- ✅ Dead code removed (unreachable handlers deleted)
|
||||
- ✅ Outdated tests removed (no `/api/ebooks` references remain)
|
||||
- ✅ Equivalent tests created (modern `/api/media-items` tests)
|
||||
- ✅ No behavioral changes (only cleanup, no functional modifications)
|
||||
- ✅ Code is cleaner and easier to maintain
|
||||
- ✅ Tests are more realistic (use actual API instead of mocks)
|
||||
|
||||
---
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- All changes are backward compatible (we only removed deprecated code)
|
||||
- No database schema changes required in Phases 1-3
|
||||
- Test file is syntactically correct (helper functions will be available in full test suite)
|
||||
- Ready to run full test suite to verify all changes
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Ready for Next Phase
|
||||
|
||||
Phases 1-3 are complete and tested. Ready to proceed with Phase 4 (Database Views Removal) when you approve the verification plan.
|
||||
|
||||
**Total legacy migration code removed**: ~639 lines
|
||||
**New modern tests added**: 467 lines
|
||||
**Net improvement**: Cleaner, more maintainable codebase with better test coverage
|
||||
@@ -1,255 +0,0 @@
|
||||
# Phase 1 Implementation Summary: File Conversion Pipeline
|
||||
|
||||
## Completed Tasks
|
||||
|
||||
### 1. Conversion Service Implementation ✅
|
||||
**File**: `internal/services/conversion_service.go`
|
||||
|
||||
Created a complete EPUB→KEPUB conversion service with:
|
||||
- On-demand conversion triggered by OPDS requests
|
||||
- Dual hash storage (EPUB and KEPUB hashes) in `media_item_formats` table
|
||||
- Conversion caching (24-hour TTL by default)
|
||||
- Support for kepubify (preferred) and ebook-convert (fallback)
|
||||
- SHA-256 hash calculation for converted files
|
||||
|
||||
**Key Methods**:
|
||||
- `ConvertEPUBToKEPUB(ctx, mediaItemID, epubPath)`: Main conversion method
|
||||
- `convertEPUB(epubPath, kepubPath)`: Executes conversion tool
|
||||
- `calculateSHA256(filePath)`: Computes file hash
|
||||
|
||||
### 2. OPDS Handler Updates ✅
|
||||
**File**: `internal/handlers/opds.go`
|
||||
|
||||
Updated the OPDS handler to integrate with conversion service:
|
||||
- Modified `NewOPDSHandler` to accept conversion service dependency
|
||||
- Enhanced `DownloadBook` method to support on-the-fly KEPUB conversion
|
||||
- Updated response headers to include `X-Bookhoard-KEPUB-SHA256` for KEPUB downloads
|
||||
- Properly handles format-specific hash headers
|
||||
|
||||
**Behavior**:
|
||||
- When `?format=kepub` is requested:
|
||||
1. Checks for cached KEPUB (serves if < 24 hours old)
|
||||
2. If not cached, converts EPUB→KEPUB on-the-fly
|
||||
3. Stores converted file with dual hash in database
|
||||
4. Serves converted file with KEPUB-specific hash header
|
||||
|
||||
### 3. Service Registration in Main ✅
|
||||
**File**: `cmd/server/main.go`
|
||||
|
||||
Integrated conversion service into server initialization:
|
||||
- Added `services` package import
|
||||
- Created `conversionService` instance with cache directory configuration
|
||||
- Updated `opdsHandler` initialization to include conversion service
|
||||
- Registered all OPDS routes (`/opds/devices/*`)
|
||||
|
||||
**New Routes**:
|
||||
- `GET /opds/devices/:deviceId/catalog` - OPDS catalog feed
|
||||
- `GET /opds/devices/:deviceId/search` - OPDS search endpoint
|
||||
- `GET /opds/devices/:deviceId/nav` - OPDS navigation feed
|
||||
- `GET /opds/devices/:deviceId/download/:bookId` - Book download with format conversion
|
||||
- `GET /opds/devices/:deviceId/cover/:bookId` - Cover image serving
|
||||
- `GET /opds/devices/:deviceId/formats/:bookId` - List available formats
|
||||
|
||||
### 4. Testing Infrastructure ✅
|
||||
**File**: `internal/services/conversion_service_test.go`
|
||||
|
||||
Created comprehensive unit tests:
|
||||
- `TestConvertEPUBToKEPUB`: Tests basic conversion and dual hash storage
|
||||
- `TestConvertCaching`: Verifies cache hit for recent conversions
|
||||
- `TestConversionChain`: Ensures conversion chain integrity
|
||||
|
||||
**File**: `bruno/opds/Download Book KEPUB (On-the-fly Conversion).bru`
|
||||
|
||||
Created Bruno API test that validates:
|
||||
- KEPUB hash header presence and format
|
||||
- Bookhoard UUID header
|
||||
- Correct Content-Type for KEPUB format
|
||||
|
||||
### 5. Documentation ✅
|
||||
**File**: `docs/CONVERSION_SERVICE.md`
|
||||
|
||||
Comprehensive documentation covering:
|
||||
- Architecture overview with flow diagrams
|
||||
- Configuration options (environment variables, Dockerfile)
|
||||
- API usage examples
|
||||
- Database schema details
|
||||
- Implementation details
|
||||
- Troubleshooting guide
|
||||
- Performance considerations
|
||||
- Security considerations
|
||||
|
||||
### 6. Configuration Updates ✅
|
||||
|
||||
**File**: `.env.example`
|
||||
Added conversion service configuration variables:
|
||||
- `BOOKHOARD_CONVERSION_CACHE_DIR` - Cache directory path
|
||||
- `BOOKHOARD_CONVERSION_TOOL` - Conversion tool to use
|
||||
- `BOOKHOARD_CONVERSION_CACHE_TTL` - Cache time-to-live
|
||||
|
||||
**File**: `Dockerfile`
|
||||
Added kepubify installation in final stage:
|
||||
```dockerfile
|
||||
RUN wget -O /usr/bin/kepubify https://github.com/pgaskin/kepubify/releases/latest/download/kepubify-linux-64bit \
|
||||
&& chmod +x /usr/bin/kepubify
|
||||
```
|
||||
|
||||
## Technical Implementation Details
|
||||
|
||||
### Dual Hash Storage Strategy
|
||||
|
||||
The conversion service maintains hash integrity for cross-device matching:
|
||||
|
||||
1. **Original EPUB Hash**: Stored in `media_item_formats` with `format_type='epub'`
|
||||
2. **Converted KEPUB Hash**: Stored in new row with `format_type='kepub'`
|
||||
3. **Conversion Chain**: KEPUB row references EPUB row via `converted_from_format_id`
|
||||
|
||||
**Example Database State**:
|
||||
```sql
|
||||
-- EPUB format (original)
|
||||
INSERT INTO media_item_formats (media_item_id, format_type, file_sha256, ...)
|
||||
VALUES (uuid-123, 'epub', 'abc123...', ...);
|
||||
|
||||
-- KEPUB format (converted)
|
||||
INSERT INTO media_item_formats (media_item_id, format_type, file_sha256, converted_from_format_id, ...)
|
||||
VALUES (uuid-123, 'kepub', 'xyz789...', <epub_format_id>, ...);
|
||||
```
|
||||
|
||||
### Conversion Process Flow
|
||||
|
||||
```
|
||||
OPDS Request: GET /opds/devices/{id}/download/{bookId}?format=kepub
|
||||
↓
|
||||
Check media_item_formats for existing KEPUB
|
||||
↓
|
||||
┌────┴────┐
|
||||
│ │
|
||||
Found Not Found
|
||||
│ │
|
||||
│ ├─ Is recent (< 24h)? ── No ──► Convert EPUB→KEPUB
|
||||
│ │ ↓
|
||||
│ │ Calculate SHA-256
|
||||
│ │ ↓
|
||||
│ │ Store in database
|
||||
│ │ ↓
|
||||
│ └───────────────────────── Serve converted file
|
||||
│
|
||||
└─ Serve cached file
|
||||
↓
|
||||
Set X-Bookhoard-KEPUB-SHA256 header
|
||||
↓
|
||||
Stream file to client
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
The conversion service handles multiple failure scenarios:
|
||||
|
||||
1. **EPUB Not Found**: Returns 404 error
|
||||
2. **Conversion Failure**: Returns 500 with error message
|
||||
3. **Hash Calculation Error**: Returns 500, prevents serving unhashed file
|
||||
4. **Database Storage Error**: Returns 500, preserves converted file for retry
|
||||
5. **Cache Directory Error**: Creates directory if missing, fails if permissions insufficient
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
- **First Conversion**: 2-5 seconds (file size dependent)
|
||||
- **Cached Conversion**: < 100ms (database lookup + file serve)
|
||||
- **Storage Overhead**: ~10% per converted file (KEPUB vs EPUB)
|
||||
- **Cache Hit Rate**: Expected > 95% after initial library conversion
|
||||
|
||||
## Verification Steps
|
||||
|
||||
### Build Verification
|
||||
```bash
|
||||
go build -o /tmp/bookhoard-test ./cmd/server
|
||||
# Success: Exit code 0
|
||||
```
|
||||
|
||||
### Manual Testing
|
||||
1. Start server with conversion service enabled
|
||||
2. Register a device and obtain device ID
|
||||
3. Add a book to library (EPUB format)
|
||||
4. Request KEPUB download via OPDS:
|
||||
```bash
|
||||
curl "http://localhost:8765/opds/devices/{deviceId}/download/{bookId}?format=kepub" \
|
||||
-I | grep -i "X-Bookhoard-KEPUB-SHA256"
|
||||
```
|
||||
5. Verify response headers:
|
||||
- `X-Bookhoard-KEPUB-SHA256` present (64-character hash)
|
||||
- `X-Bookhoard-UUID` present
|
||||
- `Content-Type: application/vnd.kobo+xml+zip`
|
||||
|
||||
### Automated Testing
|
||||
```bash
|
||||
# Run unit tests
|
||||
go test ./internal/services/... -v
|
||||
|
||||
# Run Bruno tests (via Bruno CLI or UI)
|
||||
bruno run "bruno/opds/Download Book KEPUB (On-the-fly Conversion).bru"
|
||||
```
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Existing Codebases
|
||||
- **OPDS Handler**: Enhanced with conversion service dependency
|
||||
- **Database Queries**: Uses existing `CreateMediaItemFormat` and `GetMediaItemFormatByType`
|
||||
- **Media Item Model**: Leverages existing `MediaItemFormats` struct
|
||||
- **Configuration System**: Integrates with existing `.env` pattern
|
||||
|
||||
### Future Enhancements
|
||||
The conversion service is designed to support:
|
||||
1. Additional format conversions (PDF→EPUB, CBZ→EPUB)
|
||||
2. Async/batch conversion queues
|
||||
3. Pre-conversion during library scan
|
||||
4. Distributed caching across multiple instances
|
||||
5. Custom conversion quality settings
|
||||
|
||||
## Compliance with Project Guidelines
|
||||
|
||||
✅ **Podman Only**: No Docker-specific code (kepubify works with any container runtime)
|
||||
✅ **No Local Builds**: Conversion happens via container, not local binary
|
||||
✅ **pgx v5 Standards**: Uses existing database queries with pgx types
|
||||
✅ **Atomic Changes**: Conversion doesn't modify original EPUB, creates new KEPUB
|
||||
✅ **Functional Programming**: Service uses pure functions for hash calculation
|
||||
✅ **TypeScript Only**: No new JavaScript (service is pure Go)
|
||||
✅ **Minimal Structure Changes**: Only adds new service file, updates existing handler
|
||||
✅ **Multiple Logical Commits**: Can be split into separate commits if desired
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Immediate (Phase 1 Complete)
|
||||
1. ✅ Conversion service implemented
|
||||
2. ✅ OPDS handler integrated
|
||||
3. ✅ Routes registered
|
||||
4. ✅ Tests created
|
||||
5. ✅ Documentation written
|
||||
6. ✅ Configuration updated
|
||||
|
||||
### Follow-up (Optional Enhancements)
|
||||
1. Add Prometheus metrics for conversion performance
|
||||
2. Implement async conversion queue for bulk operations
|
||||
3. Add conversion progress tracking via WebSocket
|
||||
4. Support for additional formats (PDF, CBZ)
|
||||
5. Pre-conversion during library scan
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
Before deploying to production:
|
||||
- [ ] Verify kepubify is installed in container
|
||||
- [ ] Set `BOOKHOARD_CONVERSION_CACHE_DIR` to persistent volume
|
||||
- [ ] Configure `BOOKHOARD_CONVERSION_CACHE_TTL` appropriately
|
||||
- [ ] Test conversion with actual EPUB files
|
||||
- [ ] Monitor cache directory size and set up cleanup
|
||||
- [ ] Verify database has `media_item_formats` table
|
||||
- [ ] Test dual hash storage with device sync
|
||||
- [ ] Document cache storage requirements (1.1x library size)
|
||||
- [ ] Set up monitoring for conversion failures
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Set `BOOKHOARD_CONVERSION_TOOL=""` to disable conversion
|
||||
2. Remove `conversionService` parameter from `NewOPDSHandler`
|
||||
3. OPDS handler will fall back to serving EPUB only
|
||||
4. No database schema changes required (schema already existed)
|
||||
5. No data migration needed (new rows are additive only)
|
||||
@@ -1,411 +0,0 @@
|
||||
# Phase 2: Advanced Unlinked Book Resolution - Implementation Summary
|
||||
|
||||
## Overview
|
||||
|
||||
Successfully implemented bulk resolution workflows and automated matching suggestions for unlinked books. This enhances the existing unlinked book tracking system with user-friendly bulk operations.
|
||||
|
||||
## Completed Tasks
|
||||
|
||||
### 1. Database Queries ✅
|
||||
|
||||
**File**: `internal/database/queries/queries.sql`
|
||||
|
||||
Added three new queries:
|
||||
- `GetUnlinkedBookByID` - Retrieve single unlinked book by ID
|
||||
- `DeleteUnlinkedBook` - Remove unlinked book entry
|
||||
- `ListUnresolvedUnlinkedBooks` - List unresolved books with pagination
|
||||
|
||||
### 2. Bulk Resolution API Endpoints ✅
|
||||
|
||||
**File**: `internal/handlers/book_matching.go`
|
||||
|
||||
#### POST `/api/sync/bulk-link-books`
|
||||
|
||||
Bulk link multiple unlinked books at once.
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"links": [
|
||||
{
|
||||
"unlinked_book_id": "uuid-1",
|
||||
"media_item_id": "uuid-2",
|
||||
"confidence_score": 1.0
|
||||
},
|
||||
{
|
||||
"unlinked_book_id": "uuid-3",
|
||||
"media_item_id": "uuid-4",
|
||||
"confidence_score": 0.9
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"unlinked_book_id": "uuid-1",
|
||||
"status": "success",
|
||||
"media_item_id": "uuid-2"
|
||||
}
|
||||
],
|
||||
"total": 2,
|
||||
"successful": 1,
|
||||
"failed": 1
|
||||
}
|
||||
```
|
||||
|
||||
**Status Values**:
|
||||
- `success` - Book linked successfully
|
||||
- `error` - Linking failed (book not found, alias creation failed)
|
||||
- `warning` - Linked but failed to mark as resolved
|
||||
|
||||
#### POST `/api/sync/auto-link-books`
|
||||
|
||||
Automatically attempt to link unlinked books using matching algorithm with confidence threshold.
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"confidence_threshold": 0.8,
|
||||
"limit": 50
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"auto_linked": 15,
|
||||
"results": [
|
||||
{
|
||||
"unlinked_book_id": "uuid-1",
|
||||
"title": "The Hobbit",
|
||||
"matched_media_item_id": "uuid-2",
|
||||
"confidence": 0.95,
|
||||
"match_method": "sha256_match"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior**:
|
||||
1. Fetches unresolved unlinked books (up to `limit`)
|
||||
2. Queries book matching service for each book
|
||||
3. Auto-links books with confidence ≥ threshold
|
||||
4. Creates device file aliases and marks as resolved
|
||||
5. Returns count and details of auto-linked books
|
||||
|
||||
#### GET `/api/sync/unlinked-books/:id/suggestions`
|
||||
|
||||
Get matching suggestions for a specific unlinked book.
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"unlinked_book_id": "uuid-1",
|
||||
"title_from_device": "The Hobbit",
|
||||
"sha256": "",
|
||||
"suggestions": [
|
||||
{
|
||||
"media_item_id": "uuid-2",
|
||||
"bookhoard_uuid": "uuid-2",
|
||||
"confidence": 0.95,
|
||||
"match_method": "sha256_match"
|
||||
}
|
||||
],
|
||||
"total_suggestions": 1,
|
||||
"action": "auto_link"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Frontend Template Enhancement ✅
|
||||
|
||||
**File**: `templates/unlinked_books.templ`
|
||||
|
||||
Added bulk operations UI:
|
||||
|
||||
**Bulk Actions Toolbar**:
|
||||
- Select All checkbox with count display
|
||||
- Auto-Link Selected button (high confidence, ≥80%)
|
||||
- Get Suggestions button (fetches matches for selected)
|
||||
- Bulk Manual Link button (initiates manual linking workflow)
|
||||
|
||||
**Per-Book Checkboxes**:
|
||||
- Each unlinked book card now has a checkbox
|
||||
- Checkboxes track `progress-id` and `title` for bulk operations
|
||||
- Real-time count of selected books
|
||||
|
||||
**JavaScript Functions**:
|
||||
- `toggleAllUnlinked()` - Select/deselect all books
|
||||
- `getSelectedUnlinked()` - Get selected books data
|
||||
- `updateSelectedCount()` - Update count display
|
||||
- `bulkAutoLink()` - Auto-link selected with confirmation
|
||||
- `bulkGetSuggestions()` - Fetch and display suggestions
|
||||
- `displaySuggestions()` - Render suggestions in UI
|
||||
- `showBulkManualLink()` - Initiate manual linking
|
||||
|
||||
### 4. Bruno API Tests ✅
|
||||
|
||||
Created three Bruno API test files:
|
||||
|
||||
1. **`bruno/sync-kobo/Bulk Link Books.bru`**
|
||||
- Tests bulk linking endpoint
|
||||
- Includes multiple books in single request
|
||||
- Verifies response structure
|
||||
|
||||
2. **`bruno/sync-kobo/Auto Link Books.bru`**
|
||||
- Tests auto-linking with confidence threshold
|
||||
- Configurable limit and threshold
|
||||
- Checks auto-linked count
|
||||
|
||||
3. **`bruno/sync-kobo/Get Unlinked Book Suggestions.bru`**
|
||||
- Tests suggestion retrieval
|
||||
- Uses unlinked book ID parameter
|
||||
- Validates suggestion structure
|
||||
|
||||
### 5. Route Registration ✅
|
||||
|
||||
**File**: `cmd/server/main.go`
|
||||
|
||||
Added protected routes:
|
||||
```go
|
||||
sync := protected.Group("/sync")
|
||||
sync.POST("/bulk-link-books", h.BulkLinkBooks)
|
||||
sync.POST("/auto-link-books", h.AutoLinkBooks)
|
||||
sync.GET("/unlinked-books/:id/suggestions", h.GetUnlinkedBookSuggestions)
|
||||
```
|
||||
|
||||
## Technical Implementation Details
|
||||
|
||||
### Database Schema Compatibility
|
||||
|
||||
The implementation works with the existing `unlinked_books` table:
|
||||
- Uses `id`, `device_id`, `content_id`, `file_path`, `title` fields
|
||||
- Links to `device_file_aliases` and `media_items` tables
|
||||
- Maintains `resolved` flag and `resolution_method`
|
||||
|
||||
**Note**: SHA-256 is not stored in `unlinked_books` table (not in original schema), so auto-linking relies on title matching primarily.
|
||||
|
||||
### Error Handling
|
||||
|
||||
Each bulk operation includes comprehensive error handling:
|
||||
|
||||
1. **Bulk Link**:
|
||||
- Validates each unlinked book exists
|
||||
- Creates device file alias for each link
|
||||
- Marks books as resolved
|
||||
- Returns individual status per book
|
||||
- Continues processing even if individual links fail
|
||||
|
||||
2. **Auto-Link**:
|
||||
- Fetches unlinked books with pagination
|
||||
- Queries matching service for each
|
||||
- Only auto-links if confidence ≥ threshold
|
||||
- Skips books on errors (continues processing)
|
||||
- Returns count of successful auto-links
|
||||
|
||||
3. **Suggestions**:
|
||||
- Validates unlinked book ID
|
||||
- Queries matching service
|
||||
- Returns all potential matches
|
||||
- Includes confidence scores and match methods
|
||||
|
||||
### Type Conversions
|
||||
|
||||
Helper function added to `book_matching.go`:
|
||||
```go
|
||||
func toFloat8(f float64) pgtype.Float8 {
|
||||
var result pgtype.Float8
|
||||
result.Scan(f)
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
Ensures proper type conversion for pgx v5 `Float8` type.
|
||||
|
||||
## API Usage Examples
|
||||
|
||||
### Example 1: Bulk Link Multiple Books
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8765/api/sync/bulk-link-books \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"links": [
|
||||
{
|
||||
"unlinked_book_id": "123e4567-e89b-12d3-a456-426614174000",
|
||||
"media_item_id": "987fcdeb-51a2-f43c-8877-123456789abc",
|
||||
"confidence_score": 1.0
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### Example 2: Auto-Link with High Confidence
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8765/api/sync/auto-link-books \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"confidence_threshold": 0.8,
|
||||
"limit": 50
|
||||
}'
|
||||
```
|
||||
|
||||
### Example 3: Get Suggestions
|
||||
|
||||
```bash
|
||||
curl -X GET http://localhost:8765/api/sync/unlinked-books/123e4567-e89b-12d3-a456-426614174000/suggestions \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
## Frontend Workflow
|
||||
|
||||
### User Experience Flow
|
||||
|
||||
1. **View Unlinked Books Page** (`/unlinked`)
|
||||
- Lists all unresolved unlinked books
|
||||
- Shows bulk actions toolbar at top
|
||||
|
||||
2. **Select Books**:
|
||||
- Click individual checkboxes OR
|
||||
- Click "Select All" to select all books
|
||||
- Selected count updates in real-time
|
||||
|
||||
3. **Choose Action**:
|
||||
- **Auto-Link**: One-click automatic linking (high confidence only)
|
||||
- **Get Suggestions**: Fetches potential matches for each book
|
||||
- **Bulk Manual Link**: Initiates manual selection workflow
|
||||
|
||||
4. **Review Results**:
|
||||
- Success/error status for each book
|
||||
- Toast notifications for overall status
|
||||
- Automatic page reload after successful bulk operations
|
||||
|
||||
### Matching Priority (Auto-Link)
|
||||
|
||||
The auto-link feature uses the existing book matching algorithm with priority:
|
||||
1. Bookhoard UUID (canonical) - 1.0 confidence
|
||||
2. OPF UUID - 0.95 confidence
|
||||
3. SHA-256 hash - 0.9 confidence
|
||||
4. OPF identifier - 0.85 confidence
|
||||
5. ISBN/ASIN - 0.8 confidence
|
||||
6. Title + author + file size - 0.5 confidence
|
||||
|
||||
With default threshold of 0.8, only matches with 80%+ confidence are auto-linked.
|
||||
|
||||
## Testing & Verification
|
||||
|
||||
### Unit Tests
|
||||
- Database query functions work correctly
|
||||
- Type conversions are proper
|
||||
- Error handling covers edge cases
|
||||
|
||||
### Integration Testing (Bruno)
|
||||
- Bulk link endpoint handles multiple books
|
||||
- Auto-link respects confidence threshold
|
||||
- Suggestions endpoint returns proper data
|
||||
|
||||
### Manual Testing
|
||||
1. Create unlinked book entries (via device sync or manual)
|
||||
2. Navigate to `/unlinked` page
|
||||
3. Select books using checkboxes
|
||||
4. Test each bulk action:
|
||||
- Auto-link with high confidence
|
||||
- Get suggestions and review matches
|
||||
- Manual link via suggestions
|
||||
|
||||
### Build Verification
|
||||
```bash
|
||||
cd /home/nymusicman/Code/bookhoard
|
||||
go build ./cmd/server # ✅ Successful
|
||||
cd internal/database && sqlc generate # ✅ Successful
|
||||
cd templates && templ generate # ✅ Successful
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Bulk Link
|
||||
- **Complexity**: O(n) where n = number of books
|
||||
- **Database**: N+1 queries (could be optimized in future)
|
||||
- **Time**: ~50ms per book (includes alias creation + resolution)
|
||||
- **Recommendation**: Limit to 50 books per request
|
||||
|
||||
### Auto-Link
|
||||
- **Complexity**: O(n*m) where n = books, m = matches checked
|
||||
- **Database**: 1 query + n matching queries
|
||||
- **Time**: ~100ms per book (includes matching service)
|
||||
- **Optimization**: Pagination prevents loading all books at once
|
||||
|
||||
### Get Suggestions
|
||||
- **Complexity**: O(1) for single book
|
||||
- **Database**: 1 query + 1 matching query
|
||||
- **Time**: ~50-100ms
|
||||
- **Caching**: Could be cached in future (TTL: 1 hour)
|
||||
|
||||
## Security & Permissions
|
||||
|
||||
All endpoints require:
|
||||
- JWT authentication (user must be logged in)
|
||||
- User can only link their own unlinked books
|
||||
- Device ownership verified via `device_id`
|
||||
- Media item access verified via library visibility
|
||||
|
||||
No cross-user data access possible.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements:
|
||||
1. **Optimized Bulk Link**: Batch database operations instead of N+1 queries
|
||||
2. **Background Processing**: Auto-link large datasets asynchronously
|
||||
3. **Confidence Learning**: Adjust thresholds based on user feedback
|
||||
4. **Suggestions Caching**: Cache suggestions to reduce load
|
||||
5. **Export/Import**: Export unlinked list for offline review
|
||||
6. **Bulk Delete**: Delete multiple unlinked entries at once
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Comment out route registrations in `main.go`
|
||||
2. Remove bulk actions toolbar from template
|
||||
3. Keep database queries (backward compatible)
|
||||
4. No data migration needed (no schema changes)
|
||||
|
||||
## Compliance with Project Guidelines
|
||||
|
||||
✅ **No Backend for Frontend Tasks**: Full-stack feature with API + UI
|
||||
✅ **pgx v5 Standards**: Uses generated queries with proper types
|
||||
✅ **Multiple Logical Commits**: Can be split into 3 commits
|
||||
✅ **Functional Programming**: Pure functions, no OOP patterns
|
||||
✅ **TypeScript Only**: Frontend uses vanilla JS (can convert later)
|
||||
✅ **KISS/DRY/YAGNI**: Minimal changes, reuses existing services
|
||||
✅ **Bruno Tests**: All endpoints tested with `.bru` files
|
||||
✅ **No Schema Changes**: Uses existing tables only
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
Before deploying to production:
|
||||
- [ ] Test bulk operations with sample unlinked books
|
||||
- [ ] Verify confidence thresholds work as expected
|
||||
- [ ] Check that suggestions return relevant matches
|
||||
- [ ] Test with 50+ unlinked books (performance)
|
||||
- [ ] Verify error messages are user-friendly
|
||||
- [ ] Test with multiple users (no cross-user data leakage)
|
||||
- [ ] Monitor database performance during bulk operations
|
||||
- [ ] Set up logging for bulk operations (audit trail)
|
||||
|
||||
## Summary
|
||||
|
||||
Phase 2 successfully adds bulk resolution capabilities to the unlinked books system:
|
||||
- ✅ 3 new API endpoints for bulk operations
|
||||
- ✅ Enhanced frontend with bulk actions UI
|
||||
- ✅ Comprehensive error handling and validation
|
||||
- ✅ Bruno API tests for all endpoints
|
||||
- ✅ Backward compatible with existing code
|
||||
- ✅ Ready for production use
|
||||
|
||||
The implementation makes it significantly easier for users to resolve large numbers of unlinked books efficiently.
|
||||
@@ -1,170 +0,0 @@
|
||||
# Phase 2: Advanced Unlinked Book Resolution - COMPLETE ✅
|
||||
|
||||
## Summary
|
||||
|
||||
Successfully implemented bulk resolution workflows for unlinked books with automated matching suggestions and user-friendly bulk operations.
|
||||
|
||||
## Files Created (6)
|
||||
|
||||
1. **`bruno/sync-kobo/Bulk Link Books.bru`** - Bruno test for bulk linking API
|
||||
2. **`bruno/sync-kobo/Auto Link Books.bru`** - Bruno test for auto-linking API
|
||||
3. **`bruno/sync-kobo/Get Unlinked Book Suggestions.bru`** - Bruno test for suggestions API
|
||||
4. **`docs/PHASE2_COMPLETION_SUMMARY.md`** - Comprehensive documentation
|
||||
|
||||
## Files Modified (7)
|
||||
|
||||
1. **`internal/database/queries/queries.sql`**
|
||||
- Added `GetUnlinkedBookByID` query
|
||||
- Added `DeleteUnlinkedBook` query
|
||||
- Added `ListUnresolvedUnlinkedBooks` query
|
||||
|
||||
2. **`internal/handlers/book_matching.go`**
|
||||
- Added `BulkLinkBooks()` handler
|
||||
- Added `AutoLinkBooks()` handler
|
||||
- Added `GetUnlinkedBookSuggestions()` handler
|
||||
- Added `toFloat8()` helper function
|
||||
- Added `BulkLinkBooksRequest` and `AutoLinkBooksRequest` types
|
||||
|
||||
3. **`cmd/server/main.go`**
|
||||
- Added bulk resolution routes under `/sync` group
|
||||
|
||||
4. **`templates/unlinked_books.templ`**
|
||||
- Added bulk actions toolbar with Select All
|
||||
- Added checkboxes to each book card
|
||||
- Added JavaScript functions for bulk operations
|
||||
- Enhanced UI with selected count display
|
||||
|
||||
5. **`internal/database/queries.sql.go`** (auto-generated)
|
||||
- Regenerated with new queries
|
||||
|
||||
6. **`internal/database/querier.go`** (auto-generated)
|
||||
- Updated interface with new methods
|
||||
|
||||
7. **`templates/unlinked_books_templ.go`** (auto-generated)
|
||||
- Regenerated template Go code
|
||||
|
||||
## New API Endpoints (3)
|
||||
|
||||
### 1. POST `/api/sync/bulk-link-books`
|
||||
Bulk link multiple unlinked books to media items.
|
||||
|
||||
**Features**:
|
||||
- Links multiple books in single request
|
||||
- Creates device file aliases
|
||||
- Marks books as resolved
|
||||
- Returns individual status per book
|
||||
- Continues on errors (partial success)
|
||||
|
||||
### 2. POST `/api/sync/auto-link-books`
|
||||
Automatically link unlinked books using matching algorithm.
|
||||
|
||||
**Features**:
|
||||
- Configurable confidence threshold (default 0.8)
|
||||
- Paginated processing (default 50 books)
|
||||
- Uses existing book matching service
|
||||
- Only links high-confidence matches
|
||||
- Returns count and details
|
||||
|
||||
### 3. GET `/api/sync/unlinked-books/:id/suggestions`
|
||||
Get matching suggestions for a specific unlinked book.
|
||||
|
||||
**Features**:
|
||||
- Returns all potential matches
|
||||
- Includes confidence scores
|
||||
- Shows match methods
|
||||
- Enables informed manual linking
|
||||
|
||||
## Frontend Enhancements
|
||||
|
||||
### Bulk Actions Toolbar
|
||||
- **Select All** checkbox with real-time count
|
||||
- **Auto-Link Selected** - One-click high-confidence linking
|
||||
- **Get Suggestions** - Fetch matches for selected books
|
||||
- **Bulk Manual Link** - Initiate manual workflow
|
||||
|
||||
### Per-Book Checkboxes
|
||||
- Individual selection control
|
||||
- Tracks progress ID and title
|
||||
- Updates selected count dynamically
|
||||
|
||||
### JavaScript Functions
|
||||
- `toggleAllUnlinked()` - Select/deselect all
|
||||
- `bulkAutoLink()` - Auto-link with confirmation
|
||||
- `bulkGetSuggestions()` - Fetch and display matches
|
||||
- `displaySuggestions()` - Render suggestions in UI
|
||||
- `updateSelectedCount()` - Update count display
|
||||
|
||||
## Database Queries Added
|
||||
|
||||
```sql
|
||||
-- Get unlinked book by ID
|
||||
GetUnlinkedBookByID(ctx, id) -> UnlinkedBooks
|
||||
|
||||
-- Delete unlinked book
|
||||
DeleteUnlinkedBook(ctx, id) -> exec
|
||||
|
||||
-- List unresolved unlinked books
|
||||
ListUnresolvedUnlinkedBooks(ctx, {limit, offset}) -> []UnlinkedBooksRow
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
✅ **Bulk Linking** - Link multiple books in one API call
|
||||
✅ **Auto-Linking** - Automatic high-confidence matching
|
||||
✅ **Suggestions API** - Get potential matches for manual review
|
||||
✅ **Error Resilience** - Continues processing on individual failures
|
||||
✅ **User-Friendly UI** - Checkboxes, select all, real-time count
|
||||
✅ **Comprehensive Testing** - Bruno tests for all endpoints
|
||||
✅ **Backward Compatible** - No schema changes, uses existing tables
|
||||
|
||||
## Testing & Verification
|
||||
|
||||
### Build Status
|
||||
```bash
|
||||
✅ go build ./cmd/server - Successful
|
||||
✅ sqlc generate - Successful
|
||||
✅ templ generate - Successful
|
||||
```
|
||||
|
||||
### Manual Testing Checklist
|
||||
- [ ] View unlinked books page
|
||||
- [ ] Select individual books
|
||||
- [ ] Use "Select All" checkbox
|
||||
- [ ] Test auto-link with high confidence
|
||||
- [ ] Get suggestions for selected books
|
||||
- [ ] Verify suggestions display correctly
|
||||
- [ ] Test bulk manual link workflow
|
||||
- [ ] Verify error handling for invalid IDs
|
||||
|
||||
### API Testing
|
||||
Use Bruno tests in `bruno/sync-kobo/`:
|
||||
- Bulk Link Books.bru
|
||||
- Auto Link Books.bru
|
||||
- Get Unlinked Book Suggestions.bru
|
||||
|
||||
## Performance
|
||||
|
||||
| Operation | Time | Complexity | Notes |
|
||||
|-----------|------|------------|-------|
|
||||
| Bulk Link (50 books) | ~2.5s | O(n) | ~50ms per book |
|
||||
| Auto-Link (50 books) | ~5s | O(n*m) | Includes matching |
|
||||
| Get Suggestions | ~100ms | O(1) | Single book |
|
||||
|
||||
## Security
|
||||
|
||||
- ✅ All endpoints require JWT authentication
|
||||
- ✅ User can only access their own unlinked books
|
||||
- ✅ Device ownership verified
|
||||
- ✅ No cross-user data access
|
||||
|
||||
## Next Steps
|
||||
|
||||
Phase 2 is complete and ready for:
|
||||
1. ✅ Manual testing with real unlinked books
|
||||
2. ✅ Integration testing with device sync
|
||||
3. ✅ Deployment to staging environment
|
||||
4. Ready for Phase 3: Conflict Resolution UI & API
|
||||
|
||||
## Summary
|
||||
|
||||
Phase 2 successfully adds **bulk resolution capabilities** to the unlinked books system, making it significantly easier for users to resolve large numbers of unlinked books efficiently. The implementation includes three new API endpoints, enhanced frontend with bulk operations UI, comprehensive error handling, and full test coverage.
|
||||
@@ -1,423 +0,0 @@
|
||||
# Progress Routes Analysis & Thoughts
|
||||
|
||||
## Overview
|
||||
|
||||
This document explores the current state of progress tracking in Bookhoard, the migration from legacy media-item-specific routes to universal cross-device progress, and considerations for the future.
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
### Legacy Routes (Marked as Deprecated)
|
||||
|
||||
Located in `internal/handlers/ebook.go:114-117`:
|
||||
|
||||
```go
|
||||
// Legacy progress routes (deprecated - use universal progress instead)
|
||||
g.GET("/api/media-items/:id/progress", h.GetMediaReadingProgress)
|
||||
g.PUT("/api/media-items/:id/progress", h.UpdateMediaReadingProgress)
|
||||
g.DELETE("/api/media-items/:id/progress", h.DeleteMediaReadingProgress)
|
||||
```
|
||||
|
||||
**Purpose**: These routes handle progress tracking for a specific media item from the `media_items` table.
|
||||
|
||||
**Data Source**: Likely queries the `reading_progress` table filtered by `media_item_id`.
|
||||
|
||||
**Current Status**: Explicitly marked as "legacy" and "deprecated" in code comments.
|
||||
|
||||
---
|
||||
|
||||
### Universal Progress Routes (Phase 1 Implementation)
|
||||
|
||||
Located in `internal/handlers/ebook.go:119-122`:
|
||||
|
||||
```go
|
||||
// Universal Progress routes (Phase 1)
|
||||
g.GET("/api/progress/:id", h.GetUniversalProgress)
|
||||
g.POST("/api/progress/:id", h.UpdateUniversalProgress)
|
||||
g.GET("/api/progress/:id/history", h.GetProgressHistory)
|
||||
```
|
||||
|
||||
**Purpose**: These routes provide "universal" progress tracking that works across devices and media types.
|
||||
|
||||
**Data Source**: Uses enhanced `reading_progress` table with additional fields:
|
||||
- `percentage` - Universal percentage (0-1)
|
||||
- `character_offset` - Character-based positioning
|
||||
- `epubcfi` - EPUB Canonical Fragment Identifier
|
||||
- `chapter` + `chapter_progress` - Chapter-based tracking
|
||||
- Viewport coordinates (viewport_x, viewport_y, zoom_level)
|
||||
- Scroll positions (scroll_position_x, scroll_position_y)
|
||||
- Panel number for comics/manga
|
||||
- Reading mode indicator
|
||||
|
||||
**Device Sync Metadata**:
|
||||
- `last_sync_device` - Which device last updated
|
||||
- `last_sync_source` - Source type (koreader, kobo, web, etc.)
|
||||
- `last_sync_timestamp` - When sync occurred
|
||||
- `conflict_detected` - Boolean flag for conflicts
|
||||
- `conflict_resolved` - Boolean flag for resolution status
|
||||
|
||||
---
|
||||
|
||||
## Why the Migration Happened
|
||||
|
||||
### 1. **Cross-Platform Kindle Ecosystem Vision**
|
||||
|
||||
Bookhoard aims to replace the Kindle ecosystem, which requires:
|
||||
- Syncing progress across multiple devices (Kindle, Kobo, phone, web)
|
||||
- Handling different progress formats (page numbers, percentages, CFI, character offsets)
|
||||
- Maintaining reading state across different device types
|
||||
- Supporting offline reading with sync queues
|
||||
|
||||
### 2. **Format Diversity**
|
||||
|
||||
Different e-readers and formats use different progress indicators:
|
||||
|
||||
| Format/Device | Progress Type | Example |
|
||||
|---------------|---------------|---------|
|
||||
| EPUB (KOReader) | EPUBCFI | `epubcfi(/6/4[chap1ref]!/4/2/1:0)` |
|
||||
| EPUB (Kobo) | Page # + Total | `page 234 of 456` |
|
||||
| PDF | Page # | `page 45` |
|
||||
| Web Reader | Percentage | `0.45 (45%)` |
|
||||
| TXT/Mobi | Character Offset | `offset 12345` |
|
||||
| Comics/Manga | Panel # | `panel 7` |
|
||||
| Kindle | Location # | `location 1234` |
|
||||
|
||||
The legacy `media-items/:id/progress` routes couldn't handle this diversity.
|
||||
|
||||
### 3. **Device Sync Architecture**
|
||||
|
||||
Universal progress enables:
|
||||
- Real-time sync via WebSocket (`/ws/sync`)
|
||||
- Offline queue support (`/api/queue/*`)
|
||||
- Conflict detection and resolution
|
||||
- Checkpoint mode for battery optimization
|
||||
- Progress history tracking
|
||||
|
||||
---
|
||||
|
||||
## Current Database Schema
|
||||
|
||||
From `database/schema/schema.sql:130-158`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE reading_progress (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
media_item_id UUID NOT NULL REFERENCES media_items(id) ON DELETE CASCADE,
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
|
||||
-- Legacy fields
|
||||
current_page INTEGER DEFAULT 0,
|
||||
total_pages INTEGER,
|
||||
last_read_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
|
||||
-- Universal Progress Tracking (Phase 1)
|
||||
percentage FLOAT CHECK (percentage >= 0 AND percentage <= 1),
|
||||
character_offset BIGINT,
|
||||
epubcfi TEXT,
|
||||
chapter INTEGER,
|
||||
chapter_progress FLOAT CHECK (chapter_progress >= 0 AND chapter_progress <= 1),
|
||||
viewport_x FLOAT DEFAULT 0,
|
||||
viewport_y FLOAT DEFAULT 0,
|
||||
zoom_level FLOAT DEFAULT 1.0,
|
||||
scroll_position_x FLOAT DEFAULT 0,
|
||||
scroll_position_y FLOAT DEFAULT 0,
|
||||
panel_number INTEGER,
|
||||
reading_mode VARCHAR(20),
|
||||
|
||||
-- Device Sync Metadata (Phase 1)
|
||||
last_sync_device VARCHAR(50),
|
||||
last_sync_source VARCHAR(20),
|
||||
last_sync_timestamp TIMESTAMP WITH TIME ZONE,
|
||||
conflict_detected BOOLEAN DEFAULT FALSE,
|
||||
conflict_resolved BOOLEAN DEFAULT TRUE,
|
||||
|
||||
UNIQUE(media_item_id, user_id)
|
||||
);
|
||||
```
|
||||
|
||||
**Backward Compatibility View** (Line 160-168):
|
||||
|
||||
```sql
|
||||
CREATE VIEW ebook_reading_progress AS
|
||||
SELECT rp.*,
|
||||
mi.id as ebook_id -- Map media_item_id to ebook_id for compatibility
|
||||
FROM reading_progress rp
|
||||
JOIN media_items mi ON rp.media_item_id = mi.id
|
||||
JOIN libraries l ON mi.library_id = l.id
|
||||
JOIN library_types lt ON l.library_type_id = lt.id
|
||||
WHERE lt.name = 'ebooks';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The Migration Challenge
|
||||
|
||||
### Issue: Two Parallel Systems
|
||||
|
||||
Currently, **both** systems exist side-by-side:
|
||||
|
||||
1. **Legacy routes** (`/api/media-items/:id/progress`)
|
||||
- Likely use simple `current_page` / `total_pages` fields
|
||||
- Media-item scoped
|
||||
- No device sync metadata
|
||||
|
||||
2. **Universal routes** (`/api/progress/:id`)
|
||||
- Use rich progress tracking with multiple formats
|
||||
- Device-aware
|
||||
- Include sync metadata
|
||||
|
||||
### Question: What Does `:id` Mean?
|
||||
|
||||
**Legacy**: `:id` = `media_item_id` (UUID of the book)
|
||||
|
||||
**Universal**: `:id` = ??? (Could be same media_item_id, or could be a different identifier)
|
||||
|
||||
**Ambiguity**: The routes use the same parameter name but might mean different things.
|
||||
|
||||
### Potential Problems
|
||||
|
||||
1. **Data Duplication**: If both systems write to `reading_progress` table, they might overwrite each other
|
||||
2. **Client Confusion**: Which endpoint should clients use?
|
||||
3. **Migration Path**: How do existing clients using legacy endpoints transition?
|
||||
4. **API Consistency**: Having two different endpoints for similar functionality is confusing
|
||||
|
||||
---
|
||||
|
||||
## Observations & Concerns
|
||||
|
||||
### 1. **Incomplete Migration**
|
||||
|
||||
The legacy routes are marked as deprecated but **still active**. This suggests:
|
||||
- Migration is ongoing, not complete
|
||||
- Some clients might still depend on legacy routes
|
||||
- Fear of breaking existing integrations
|
||||
|
||||
### 2. **Backward Compatibility View**
|
||||
|
||||
The `ebook_reading_progress` view exists to maintain compatibility with the old `ebooks` table. This adds:
|
||||
- Query overhead (JOINs to filter by library type)
|
||||
- Developer confusion (which table/view to query?)
|
||||
- Technical debt (maintaining two ways to access data)
|
||||
|
||||
### 3. **Route Naming Inconsistency**
|
||||
|
||||
- Legacy: `/api/media-items/:id/progress` (RESTful, nested under media-item)
|
||||
- Universal: `/api/progress/:id` (flat structure, not nested)
|
||||
|
||||
**Question**: Should universal progress be under `/api/media-items/:id/universal-progress` for consistency?
|
||||
|
||||
### 4. **HTTP Method Mismatch**
|
||||
|
||||
Legacy routes use:
|
||||
- `PUT /api/media-items/:id/progress` (update progress)
|
||||
|
||||
Universal routes use:
|
||||
- `POST /api/progress/:id` (update progress)
|
||||
|
||||
**REST convention**: `PUT` is idempotent, `POST` is not. For progress updates, `PUT` might be more appropriate since setting the same progress twice should have the same effect.
|
||||
|
||||
### 5. **Missing Delete Operation**
|
||||
|
||||
Universal routes don't have a `DELETE /api/progress/:id` endpoint. Legacy does:
|
||||
- `DELETE /api/media-items/:id/progress` (clear progress)
|
||||
|
||||
**Question**: Should there be a way to reset progress via universal routes?
|
||||
|
||||
---
|
||||
|
||||
## Potential Future Directions
|
||||
|
||||
### Option 1: Full Migration (Clean Break)
|
||||
|
||||
**Action**: Remove all legacy routes and views.
|
||||
|
||||
**Steps**:
|
||||
1. Deprecate legacy routes in API documentation (return `Warning` header)
|
||||
2. Add a 6-month migration timeline
|
||||
3. Remove `/api/media-items/:id/progress` routes
|
||||
4. Drop `ebook_reading_progress` view
|
||||
5. Update all clients to use universal routes
|
||||
|
||||
**Pros**:
|
||||
- Cleaner API surface
|
||||
- Single source of truth
|
||||
- Less maintenance burden
|
||||
- Clearer documentation
|
||||
|
||||
**Cons**:
|
||||
- Breaking change for existing clients
|
||||
- Mobile apps might need updates
|
||||
- External integrations could break
|
||||
|
||||
### Option 2: Compatibility Layer (Adapter Pattern)
|
||||
|
||||
**Action**: Keep legacy routes but make them thin wrappers around universal routes.
|
||||
|
||||
**Implementation**:
|
||||
```go
|
||||
// Legacy route calls universal route internally
|
||||
func (h *Handler) GetMediaReadingProgress(c echo.Context) error {
|
||||
mediaItemID := c.Param("id")
|
||||
// Extract user_id from JWT
|
||||
// Call h.GetUniversalProgress with same IDs
|
||||
// Transform response if needed
|
||||
}
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- No breaking changes
|
||||
- Gradual migration path
|
||||
- Single implementation (universal routes)
|
||||
|
||||
**Cons**:
|
||||
- Maintains API surface area
|
||||
- Slight performance overhead (function call)
|
||||
- Still confusing to have two endpoints
|
||||
|
||||
### Option 3: Unified Endpoint (Best of Both)
|
||||
|
||||
**Action**: Create a single endpoint that handles both use cases.
|
||||
|
||||
**Proposed**:
|
||||
```
|
||||
GET /api/media-items/:id/progress?format=universal
|
||||
PUT /api/media-items/:id/progress?format=universal
|
||||
DELETE /api/media-items/:id/progress
|
||||
```
|
||||
|
||||
The `format` query parameter determines:
|
||||
- `format=simple` (default): Returns basic page/percentage (legacy behavior)
|
||||
- `format=universal`: Returns full device-aware progress with metadata
|
||||
|
||||
**Pros**:
|
||||
- Single endpoint
|
||||
- Backward compatible
|
||||
- Clear migration path via query parameter
|
||||
- RESTful structure (nested under media-items)
|
||||
|
||||
**Cons**:
|
||||
- More complex handler logic
|
||||
- Need to maintain both formats in response
|
||||
|
||||
### Option 4: Versioned API (Cleanest Long-Term)
|
||||
|
||||
**Action**: Use API versioning to separate old and new.
|
||||
|
||||
**Proposed**:
|
||||
```
|
||||
# v1 (Legacy)
|
||||
GET /api/v1/media-items/:id/progress
|
||||
PUT /api/v1/media-items/:id/progress
|
||||
DELETE /api/v1/media-items/:id/progress
|
||||
|
||||
# v2 (Universal)
|
||||
GET /api/v2/media-items/:id/progress
|
||||
PUT /api/v2/media-items/:id/progress
|
||||
GET /api/v2/media-items/:id/progress/history
|
||||
```
|
||||
|
||||
**Pros**:
|
||||
- Clean separation
|
||||
- Can deprecate v1 independently
|
||||
- Standard industry practice
|
||||
- Clear migration documentation
|
||||
|
||||
**Cons**:
|
||||
- Need to implement version routing
|
||||
- More upfront work
|
||||
- Maintenance of two versions temporarily
|
||||
|
||||
---
|
||||
|
||||
## Unanswered Questions for Discussion
|
||||
|
||||
1. **Are any clients currently using the legacy progress routes?**
|
||||
- If yes, which ones? (mobile app, web app, third-party integrations?)
|
||||
- Can they be updated easily?
|
||||
|
||||
2. **What does the `:id` parameter represent in universal progress routes?**
|
||||
- Is it still `media_item_id`?
|
||||
- Or is it a `reading_progress` record ID?
|
||||
- Need to check implementation to confirm
|
||||
|
||||
3. **Why was `/api/progress/:id` chosen instead of `/api/media-items/:id/universal-progress`?**
|
||||
- Flat structure vs nested structure design decision
|
||||
- Might indicate plans for progress to exist independently of media items?
|
||||
|
||||
4. **Is the legacy route implementation actually different, or just deprecated?**
|
||||
- Need to read the handler implementations to compare
|
||||
- They might be calling the same underlying code
|
||||
|
||||
5. **Should we maintain progress deletion functionality?**
|
||||
- Universal routes don't have DELETE
|
||||
- Is deleting progress a necessary feature?
|
||||
|
||||
6. **What's the timeline for removing legacy routes?**
|
||||
- Already marked deprecated, but when can we delete them?
|
||||
- Need to coordinate with mobile app releases
|
||||
|
||||
7. **How does the backward compatibility view affect performance?**
|
||||
- The `ebook_reading_progress` view requires JOINs
|
||||
- Is it used anywhere, or can it be dropped?
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Immediate Actions (Discussion Phase)
|
||||
|
||||
1. **Audit Current Usage**
|
||||
- Search codebase for references to legacy routes
|
||||
- Check if any external documentation mentions these endpoints
|
||||
- Identify all clients (web, mobile, third-party)
|
||||
|
||||
2. **Compare Implementations**
|
||||
- Read handler code for both legacy and universal routes
|
||||
- Document differences in behavior
|
||||
- Determine if they're truly different or just deprecated wrappers
|
||||
|
||||
3. **Clarify API Contract**
|
||||
- Define what `:id` means in universal routes
|
||||
- Document expected request/response formats
|
||||
- Add examples for different device types
|
||||
|
||||
4. **Performance Analysis**
|
||||
- Query database to see how many records use legacy fields vs universal
|
||||
- Check if backward compatibility view is actually used
|
||||
- Benchmark query performance with/without views
|
||||
|
||||
### Future Considerations
|
||||
|
||||
1. **Choose a Migration Strategy**
|
||||
- Review Options 1-4 above
|
||||
- Consider breaking changes vs compatibility
|
||||
- Plan timeline based on client usage
|
||||
|
||||
2. **API Versioning Decision**
|
||||
- Decide if we want `/api/v1/` and `/api/v2/` structure
|
||||
- Or use different approach (headers, content negotiation)
|
||||
|
||||
3. **Documentation Updates**
|
||||
- Update API_REFERENCE.md with clear deprecation notices
|
||||
- Add migration guide for clients
|
||||
- Document best practices for progress tracking
|
||||
|
||||
4. **Test Coverage**
|
||||
- Ensure both legacy and universal routes have comprehensive tests
|
||||
- Add integration tests for cross-device sync scenarios
|
||||
- Test conflict resolution workflows
|
||||
|
||||
---
|
||||
|
||||
## Next Steps for Discussion
|
||||
|
||||
1. **Review handler implementations** to understand actual differences
|
||||
2. **Check client usage** (web app, mobile apps, Bruno tests)
|
||||
3. **Decide on migration timeline** and breaking change tolerance
|
||||
4. **Choose unified strategy** (Options 1-4 or hybrid)
|
||||
5. **Plan implementation** with backward compatibility in mind
|
||||
|
||||
---
|
||||
|
||||
*Document created for future discussion. No changes to be made without review.*
|
||||
@@ -1,518 +0,0 @@
|
||||
# Bookhoard Security Audit Report
|
||||
## Universal Sync Implementation (Phases 1-7)
|
||||
|
||||
**Date**: January 31, 2026
|
||||
**Version**: 1.0.0
|
||||
**Auditor**: Bookhoard Security Team
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This security audit covers the Universal Cross-Platform Sync implementation, including device authentication, wireless sync protocols, queue management, and offline recovery mechanisms.
|
||||
|
||||
### Overall Security Rating: **A- (Recommended for Production with Minor Enhancements)**
|
||||
|
||||
---
|
||||
|
||||
## 1. Authentication & Authorization
|
||||
|
||||
### 1.1 Device Registration Flow ✅ SECURE
|
||||
|
||||
**Implementation**: `internal/handlers/devices.go`
|
||||
|
||||
**Flow**:
|
||||
```
|
||||
1. Device generates unique identifier (hardware ID)
|
||||
2. Device POST /api/devices/register/initiate
|
||||
3. Server creates pending registration (5 min expiry)
|
||||
4. User visits auth URL in web browser
|
||||
5. User logs in and approves device
|
||||
6. Server generates device-specific JWT token
|
||||
7. Device polls for token approval
|
||||
8. Device receives token and begins syncing
|
||||
```
|
||||
|
||||
**Security Strengths**:
|
||||
- ✅ No API keys on devices (prevents credential exposure)
|
||||
- ✅ User approval required via web interface
|
||||
- ✅ Short-lived registration sessions (5 minutes)
|
||||
- ✅ Device-specific JWT tokens with limited permissions
|
||||
- ✅ Token revocation support
|
||||
|
||||
**Recommendations**:
|
||||
- ⚠️ Add rate limiting on registration endpoint (10 req/min per IP)
|
||||
- ⚠️ Implement device cap per user (max 10 devices)
|
||||
- ⚠️ Add notification when new device registered
|
||||
|
||||
### 1.2 Device Authentication Middleware ✅ SECURE
|
||||
|
||||
**Implementation**: `internal/middleware/device_auth.go`
|
||||
|
||||
**Security Features**:
|
||||
- ✅ Bearer token validation on every request
|
||||
- ✅ Device ownership verification
|
||||
- ✅ Token expiry checking
|
||||
- ✅ Permission validation per endpoint
|
||||
- ✅ Device revocation support
|
||||
|
||||
**Code Review**:
|
||||
```go
|
||||
// Validates device token and ownership
|
||||
func (m *DeviceAuthMiddleware) Authenticate(next echo.HandlerFunc) echo.HandlerFunc {
|
||||
return func(c echo.Context) error {
|
||||
device, err := m.validateToken(token)
|
||||
if err != nil || !device.SyncEnabled.Bool {
|
||||
return ErrUnauthorized
|
||||
}
|
||||
c.Set("device", device)
|
||||
return next(c)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 1.3 User JWT Authentication ✅ SECURE
|
||||
|
||||
**Implementation**: Existing user authentication system
|
||||
|
||||
**Security Features**:
|
||||
- ✅ bcrypt password hashing (cost 10)
|
||||
- ✅ JWT with short expiry (15 minutes)
|
||||
- ✅ Refresh token rotation
|
||||
- ✅ Secure password complexity requirements
|
||||
- ✅ Login attempt rate limiting (5 attempts / 15 min lockout)
|
||||
|
||||
---
|
||||
|
||||
## 2. Wireless Sync Protocols
|
||||
|
||||
### 2.1 KOReader Sync Protocol ✅ SECURE
|
||||
|
||||
**Implementation**: `internal/handlers/koreader.go`
|
||||
|
||||
**Endpoints**:
|
||||
```
|
||||
POST /api/sync/koreader/progress
|
||||
GET /api/sync/koreader/metadata/:uuid
|
||||
POST /api/sync/koreader/bookmarks
|
||||
```
|
||||
|
||||
**Security Analysis**:
|
||||
- ✅ Requires device authentication
|
||||
- ✅ Input validation on all fields
|
||||
- ✅ Media item ownership verification
|
||||
- ✅ SQL injection protection (parameterized queries)
|
||||
- ✅ No arbitrary file access
|
||||
|
||||
**Potential Issues**:
|
||||
- ⚠️ Large sync payloads could cause DoS (add size limits)
|
||||
- ⚠️ No request signing (add HMAC for integrity)
|
||||
|
||||
**Recommendations**:
|
||||
```go
|
||||
// Add payload size limit
|
||||
const MaxSyncPayloadSize = 10 * 1024 * 1024 // 10MB
|
||||
|
||||
func validatePayloadSize(r *http.Request) error {
|
||||
r.Body = http.MaxBytesReader(nil, r.Body, MaxSyncPayloadSize)
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Kobo Sync Protocol ✅ SECURE
|
||||
|
||||
**Implementation**: `internal/handlers/kobo.go`
|
||||
|
||||
**Security Features**:
|
||||
- ✅ Device authentication required
|
||||
- ✅ x-kobo-device header validation
|
||||
- ✅ Content-Type validation
|
||||
- ✅ Input sanitization
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Protection
|
||||
|
||||
### 3.1 Sensitive Data Storage ✅ SECURE
|
||||
|
||||
**Password Storage**:
|
||||
- ✅ bcrypt with cost factor 10
|
||||
- ✅ No plaintext storage
|
||||
- ✅ No password logging
|
||||
|
||||
**Device Tokens**:
|
||||
- ✅ Unique per device
|
||||
- ✅ Cryptographically random (UUID v4)
|
||||
- ✅ Revocable
|
||||
- ⚠️ Stored in plaintext (consider encryption at rest)
|
||||
|
||||
**Sync Data**:
|
||||
- ✅ JSONB stored in PostgreSQL
|
||||
- ✅ No SQL injection vectors
|
||||
- ✅ Media item ownership verification
|
||||
|
||||
### 3.2 Data Transmission ✅ SECURE
|
||||
|
||||
**HTTPS Enforcement**:
|
||||
```go
|
||||
// Recommended: Force HTTPS in production
|
||||
if !cfg.TestMode {
|
||||
e.Pre(echomiddleware.HTTPSRedirect())
|
||||
}
|
||||
```
|
||||
|
||||
**WebSocket Security**:
|
||||
- ✅ Token validation on connection
|
||||
- ✅ Origin checking
|
||||
- ✅ Automatic disconnection on token expiry
|
||||
|
||||
---
|
||||
|
||||
## 4. Rate Limiting & DoS Prevention
|
||||
|
||||
### 4.1 Current Implementation ⚠️ NEEDS ENHANCEMENT
|
||||
|
||||
**Existing**: `internal/middleware/rate_limiter.go`
|
||||
|
||||
**Per-Endpoint Limits**:
|
||||
```
|
||||
General: 100 req/min (configurable)
|
||||
Auth: 10 req/min
|
||||
```
|
||||
|
||||
**Sync-Specific Limits Needed**:
|
||||
```go
|
||||
const (
|
||||
SyncProgressRateLimit = 120 / time.Minute // Page turns
|
||||
SyncMetadataRateLimit = 30 / time.Minute // Metadata fetches
|
||||
SyncBookmarkRateLimit = 60 / time.Minute // Bookmarks/notes
|
||||
DeviceRegistrationLimit = 10 / time.Minute // Device registrations
|
||||
)
|
||||
```
|
||||
|
||||
### 4.2 Resource Limits
|
||||
|
||||
**Queue Processing**:
|
||||
- ✅ Batch size limit (50 items)
|
||||
- ✅ Concurrent worker limit (1 per instance)
|
||||
- ⚠️ Add per-device queue size limit (100 items max)
|
||||
|
||||
**Database Connections**:
|
||||
- ✅ Connection pooling (pgxpool)
|
||||
- ✅ Max connections: 200
|
||||
- ✅ Automatic connection reuse
|
||||
|
||||
---
|
||||
|
||||
## 5. Input Validation
|
||||
|
||||
### 5.1 Sync Data Validation ✅ SECURE
|
||||
|
||||
**Progress Updates**:
|
||||
```go
|
||||
type ProgressUpdate struct {
|
||||
Percentage float64 `validate:"gte=0,lte=1"`
|
||||
Page *int `validate:"gte=0"`
|
||||
TotalPages *int `validate:"gte=0,lte=10000"`
|
||||
}
|
||||
```
|
||||
|
||||
**Device Registration**:
|
||||
```go
|
||||
type DeviceRegistration struct {
|
||||
DeviceName string `validate:"required,min=1,max=100"`
|
||||
DeviceType string `validate:"required,oneof=koreader kobo web mobile"`
|
||||
}
|
||||
```
|
||||
|
||||
**Strengths**:
|
||||
- ✅ Struct validation using go-playground/validator
|
||||
- ✅ Type safety via pgx
|
||||
- ✅ Length constraints
|
||||
- ✅ Enum validation
|
||||
|
||||
---
|
||||
|
||||
## 6. SQL Injection Prevention
|
||||
|
||||
### 6.1 Parameterized Queries ✅ SECURE
|
||||
|
||||
**All queries use sqlc-generated code**:
|
||||
```go
|
||||
// Generated code uses parameterized queries
|
||||
func (q *Queries) CreateSyncQueueItem(ctx context.Context, arg CreateSyncQueueItemParams) (SyncQueue, error) {
|
||||
row := q.db.QueryRow(ctx, CreateSyncQueueItem,
|
||||
arg.DeviceID, // $1 - Parameterized
|
||||
arg.MediaItemID, // $2 - Parameterized
|
||||
arg.SyncType, // $3 - Parameterized
|
||||
// ... all parameters are safely bound
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
**No dynamic SQL construction** ✅
|
||||
|
||||
---
|
||||
|
||||
## 7. Cross-Site Request Forgery (CSRF)
|
||||
|
||||
### 7.1 State-Changing Operations
|
||||
|
||||
**JWT Authentication**: CSRF protected via JWT
|
||||
- ✅ All state-changing ops require valid JWT
|
||||
- ✅ Token stored in memory/secure storage
|
||||
- ✅ SameSite cookie attribute (when applicable)
|
||||
|
||||
**Device Authentication**: CSRF not applicable
|
||||
- ✅ Devices use Bearer tokens (no cookies)
|
||||
- ✅ Origin validation for WebSocket
|
||||
|
||||
**Recommendation**: Add CSRF double-submit tokens for web interface
|
||||
|
||||
---
|
||||
|
||||
## 8. Authorization Checks
|
||||
|
||||
### 8.1 Media Item Ownership ✅ SECURE
|
||||
|
||||
```go
|
||||
func (h *Handler) validateOwnership(userID, mediaItemID uuid.UUID) error {
|
||||
item, err := h.db.GetMediaItem(ctx, mediaItemID)
|
||||
if err != nil {
|
||||
return ErrNotFound
|
||||
}
|
||||
|
||||
library, err := h.db.GetLibrary(ctx, item.LibraryID)
|
||||
if err != nil {
|
||||
return ErrNotFound
|
||||
}
|
||||
|
||||
// Check user has access to library
|
||||
visible, err := h.db.GetLibraryVisibility(ctx, userID, library.ID)
|
||||
if !visible.IsVisible {
|
||||
return ErrForbidden
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
**All endpoints verify ownership** ✅
|
||||
|
||||
---
|
||||
|
||||
## 9. Error Handling & Information Disclosure
|
||||
|
||||
### 9.1 Error Messages ✅ SECURE
|
||||
|
||||
**Good Examples**:
|
||||
```
|
||||
"Media item not found" // Generic
|
||||
"Invalid request format" // No details
|
||||
"Authentication required" // Clear but generic
|
||||
```
|
||||
|
||||
**Avoid Information Leakage**:
|
||||
```
|
||||
❌ "User with ID 123 does not exist"
|
||||
❌ "Password incorrect for user@example.com"
|
||||
✅ "Invalid credentials"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Cryptographic Practices
|
||||
|
||||
### 10.1 Random Number Generation ✅ SECURE
|
||||
|
||||
```go
|
||||
// Using crypto/rand (via UUID v4)
|
||||
deviceID := uuid.New() // Uses crypto/rand
|
||||
authToken := "device-token-" + uuid.New().String()
|
||||
```
|
||||
|
||||
### 10.2 Token Generation ✅ SECURE
|
||||
|
||||
```go
|
||||
// JWT signing with HS256
|
||||
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
|
||||
tokenString, err := token.SignedString([]byte(secret))
|
||||
```
|
||||
|
||||
**Recommendation**: Consider RS256 for production (asymmetric keys)
|
||||
|
||||
---
|
||||
|
||||
## 11. Dependency Security
|
||||
|
||||
### 11.1 Key Dependencies
|
||||
|
||||
```
|
||||
github.com/jackc/pgx/v5 v5.5.0 ✅ Latest stable
|
||||
github.com/golang-jwt/jwt/v5 v5.2.0 ✅ Latest stable
|
||||
github.com/labstack/echo/v4 v4.12.0 ✅ Latest stable
|
||||
golang.org/x/crypto v0.18.0 ✅ Latest stable
|
||||
```
|
||||
|
||||
**All dependencies up-to-date** ✅
|
||||
|
||||
---
|
||||
|
||||
## 12. Recommended Security Enhancements
|
||||
|
||||
### Priority 1 (Implement Before Production)
|
||||
|
||||
1. **Add Request Signing** ⚠️ HIGH PRIORITY
|
||||
```go
|
||||
// Add HMAC signature to sync requests
|
||||
signature = HMAC-SHA256(deviceToken, requestBody + timestamp)
|
||||
```
|
||||
|
||||
2. **Increase Rate Limiting** ⚠️ HIGH PRIORITY
|
||||
```go
|
||||
// Per-device rate limits
|
||||
DeviceRateLimit = 60 req/min
|
||||
// Per-user rate limits
|
||||
UserSyncRateLimit = 300 req/min
|
||||
```
|
||||
|
||||
3. **Add Request Size Limits** ⚠️ HIGH PRIORITY
|
||||
```go
|
||||
MaxSyncPayload = 10MB
|
||||
MaxAnnotationSize = 100KB
|
||||
```
|
||||
|
||||
### Priority 2 (Implement Soon)
|
||||
|
||||
4. **HTTPS Enforcement** 📡 MEDIUM PRIORITY
|
||||
```go
|
||||
e.Pre(echomiddleware.HTTPSRedirect())
|
||||
e.Pre(middleware.SecureWithConfig(middleware.SecureConfig{
|
||||
XSSProtection: "1; mode=block",
|
||||
ContentTypeNosniff: "1",
|
||||
XFrameOptions: "DENY",
|
||||
}))
|
||||
```
|
||||
|
||||
5. **Device Cap** 📱 MEDIUM PRIORITY
|
||||
```go
|
||||
MaxDevicesPerUser = 10
|
||||
```
|
||||
|
||||
6. **Security Headers** 🔒 MEDIUM PRIORITY
|
||||
```go
|
||||
// Add to all responses
|
||||
X-Content-Type-Options: nosniff
|
||||
X-Frame-Options: DENY
|
||||
X-XSS-Protection: 1; mode=block
|
||||
Strict-Transport-Security: max-age=31536000
|
||||
```
|
||||
|
||||
### Priority 3 (Future Enhancements)
|
||||
|
||||
7. **Audit Logging** 📊 LOW PRIORITY
|
||||
```go
|
||||
type AuditLog struct {
|
||||
Timestamp time.Time
|
||||
UserID uuid.UUID
|
||||
DeviceID uuid.UUID
|
||||
Action string
|
||||
ResourceType string
|
||||
ResourceID uuid.UUID
|
||||
IPAddress string
|
||||
UserAgent string
|
||||
}
|
||||
```
|
||||
|
||||
8. **API Key Rotation** 🔑 LOW PRIORITY
|
||||
```go
|
||||
// Auto-rotate device tokens every 90 days
|
||||
TokenRotationPeriod = 90 * 24 * time.Hour
|
||||
```
|
||||
|
||||
9. **WebAuthn for Device Registration** 🔐 LOW PRIORITY
|
||||
```go
|
||||
// Use WebAuthn instead of password login for device approval
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13. Testing & Validation
|
||||
|
||||
### 13.1 Security Test Coverage
|
||||
|
||||
**Existing Tests**:
|
||||
- ✅ Device authentication flow
|
||||
- ✅ User authentication
|
||||
- ✅ Authorization checks
|
||||
- ✅ Input validation
|
||||
|
||||
**Recommended Security Tests**:
|
||||
```go
|
||||
func TestSQLInjectionPrevention(t *testing.T)
|
||||
func TestAuthenticationBypass(t *testing.T)
|
||||
func TestRateLimitEnforcement(t *testing.T)
|
||||
func TestCSRFProtection(t *testing.T)
|
||||
func TestPrivilegeEscalation(t *testing.T)
|
||||
func TestDoSProtection(t *testing.T)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Compliance Considerations
|
||||
|
||||
### 14.1 Data Privacy
|
||||
|
||||
**GDPR Compliance**:
|
||||
- ✅ User data export capability
|
||||
- ✅ Right to deletion (DELETE /api/users/:id)
|
||||
- ✅ Data minimization
|
||||
- ⚠️ Need privacy policy update for sync features
|
||||
|
||||
**Data Retention**:
|
||||
```
|
||||
Sync Queue: 30 days
|
||||
Reading History: 365 days
|
||||
Conflict Logs: 90 days
|
||||
Audit Logs: 180 days
|
||||
```
|
||||
|
||||
### 14.2 SOC 2 Considerations
|
||||
|
||||
- ✅ Access control (user + device authentication)
|
||||
- ✅ Change logging (reading_progress, sync_conflicts)
|
||||
- ⚠️ Need incident response plan
|
||||
- ⚠️ Need security monitoring/alerting
|
||||
|
||||
---
|
||||
|
||||
## 15. Conclusion
|
||||
|
||||
### Security Scorecard
|
||||
|
||||
| Category | Score | Status |
|
||||
|----------|-------|--------|
|
||||
| Authentication | 9/10 | ✅ Excellent |
|
||||
| Authorization | 10/10 | ✅ Excellent |
|
||||
| Input Validation | 9/10 | ✅ Excellent |
|
||||
| Data Protection | 8/10 | ✅ Good |
|
||||
| Rate Limiting | 6/10 | ⚠️ Needs Enhancement |
|
||||
| Error Handling | 9/10 | ✅ Excellent |
|
||||
| Cryptography | 8/10 | ✅ Good |
|
||||
| Dependency Security | 10/10 | ✅ Excellent |
|
||||
|
||||
**Overall: 8.6/10 (A-)**
|
||||
|
||||
### Production Readiness: ✅ APPROVED
|
||||
|
||||
**With Conditions**:
|
||||
1. Implement Priority 1 enhancements before production
|
||||
2. Add monitoring for security events
|
||||
3. Document incident response procedures
|
||||
4. Perform penetration testing before public release
|
||||
|
||||
---
|
||||
|
||||
**Audit Completed By**: Bookhoard Security Team
|
||||
**Next Audit**: Within 3 months of production deployment
|
||||
**Questions**: security@bookhoard.example.com
|
||||
@@ -1,597 +0,0 @@
|
||||
# Security Enhancements Implementation Report
|
||||
## Priority 1 Security Features - COMPLETED
|
||||
|
||||
**Date**: January 31, 2026
|
||||
**Version**: 1.0.1
|
||||
**Implemented By**: Bookhoard Security Team
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
All **Priority 1** security recommendations from the security audit have been successfully implemented, bringing Bookhoard's security rating from **A- (8.6/10)** to **A+ (9.2/10)**.
|
||||
|
||||
### Security Scorecard Update
|
||||
|
||||
| Category | Before | After | Improvement |
|
||||
|----------|--------|-------|-------------|
|
||||
| Authentication | 9/10 | 9.5/10 | +0.5 |
|
||||
| Authorization | 10/10 | 10/10 | ✓ Maintained |
|
||||
| Input Validation | 9/10 | 9.5/10 | +0.5 |
|
||||
| Data Protection | 8/10 | 9/10 | +1.0 |
|
||||
| Rate Limiting | 6/10 | 9/10 | +3.0 |
|
||||
| Error Handling | 9/10 | 9/10 | ✓ Maintained |
|
||||
| Cryptography | 8/10 | 9/10 | +1.0 |
|
||||
| Dependency Security | 10/10 | 10/10 | ✓ Maintained |
|
||||
|
||||
**Overall Score**: **9.2/10 (A+)** - **Production Ready with No Conditions**
|
||||
|
||||
---
|
||||
|
||||
## Implemented Enhancements
|
||||
|
||||
### 1. ✅ HMAC Request Signing
|
||||
|
||||
**File**: `internal/middleware/request_signing.go` (240 lines)
|
||||
|
||||
**What Was Implemented**:
|
||||
- HMAC-SHA256 signature validation for all sync requests
|
||||
- Timestamp-based replay attack prevention (5-minute window)
|
||||
- Clock skew detection (±1 minute tolerance)
|
||||
- Request ID tracing for audit trails
|
||||
- Device-specific secret keys
|
||||
|
||||
**Security Benefits**:
|
||||
- ✅ **Request Integrity**: Ensures requests aren't tampered with in transit
|
||||
- ✅ **Replay Prevention**: Timestamps prevent old requests from being replayed
|
||||
- �**Audit Trail**: Request IDs enable security monitoring
|
||||
- �**Tamper Detection**: Any modification invalidates signature
|
||||
|
||||
**How It Works**:
|
||||
```go
|
||||
// Client signs request
|
||||
signingString = requestID + "|" + timestamp + "|" + requestBody
|
||||
signature = HMAC-SHA256(signingString, deviceSecret)
|
||||
|
||||
// Server validates
|
||||
expectedSig = HMAC-SHA256(requestID + timestamp + body, deviceSecret)
|
||||
if !hmac.Equal(signature, expectedSig) {
|
||||
return "Invalid signature"
|
||||
}
|
||||
```
|
||||
|
||||
**Headers Required**:
|
||||
```
|
||||
X-Request-ID: unique-uuid-v4
|
||||
X-Timestamp: Unix timestamp (seconds)
|
||||
X-Signature: hex-encoded HMAC-SHA256
|
||||
```
|
||||
|
||||
**Configuration**:
|
||||
```go
|
||||
type RequestSigningConfig struct {
|
||||
Enabled: true
|
||||
TimestampHeader: "X-Timestamp"
|
||||
SignatureHeader: "X-Signature"
|
||||
TimestampTolerance: 5 minutes
|
||||
MaxClockSkew: 1 minute
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. ✅ Request Size Limits
|
||||
|
||||
**File**: `internal/middleware/request_size_limits.go` (150+ lines)
|
||||
|
||||
**What Was Implemented**:
|
||||
- Payload size validation for all endpoints
|
||||
- Per-endpoint size limits:
|
||||
- Sync payloads: 10MB max
|
||||
- Annotations: 100KB max
|
||||
- Metadata: 1MB max
|
||||
- Image uploads: 50MB max
|
||||
- Real-time size monitoring and logging
|
||||
|
||||
**Security Benefits**:
|
||||
- ✅ **DoS Prevention**: Prevents memory exhaustion attacks
|
||||
- ✅ **Resource Protection: Limits server memory usage
|
||||
- ✅**Abuse Prevention**: Blocks large payload attacks
|
||||
|
||||
**Implementation Details**:
|
||||
```go
|
||||
const (
|
||||
MaxSyncPayload = 10 * 1024 * 1024 // 10MB
|
||||
MaxAnnotationSize = 100 * 1024 // 100KB
|
||||
MaxMetadataSize = 1 * 1024 * 1024 // 1MB
|
||||
MaxImageUploadSize = 50 * 1024 * 1024 // 50MB
|
||||
)
|
||||
|
||||
// Applied automatically
|
||||
c.Request().Body = http.MaxBytesReader(nil, c.Request().Body, limit)
|
||||
```
|
||||
|
||||
**Smart Limiting**:
|
||||
```go
|
||||
sync endpoints → 10MB limit
|
||||
annotation endpoints → 100KB limit
|
||||
metadata endpoints → 1MB limit
|
||||
upload endpoints → 50MB limit
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. ✅ Enhanced Rate Limiting
|
||||
|
||||
**File**: `internal/middleware/sync_rate_limiter.go` (160+ lines)
|
||||
|
||||
**What Was Implemented**:
|
||||
- Per-device rate limiting (60 req/min for sync)
|
||||
- Per-user combined rate limiting (300 req/min total)
|
||||
- Global server rate limiting (600 req/min)
|
||||
- Automatic cleanup of stale limiters
|
||||
- Memory-efficient implementation
|
||||
|
||||
**Security Benefits**:
|
||||
- ✅ **DoS Prevention**: Blocks abusive request patterns
|
||||
- ✅ **Fair Resource Allocation**: Prevents one device from monopolizing resources
|
||||
- ✅ **Scalability**: Ensures server stability under load
|
||||
- ✅**Abuse Detection**: Identifies problematic devices
|
||||
|
||||
**Rate Limits Applied**:
|
||||
```go
|
||||
const (
|
||||
DeviceSyncRatePerSec = 2 // 120 req/min
|
||||
DeviceMetadataRatePerSec = 0.5 // 30 req/min
|
||||
UserSyncRatePerSec = 5 // 300 req/min
|
||||
GlobalRatePerSec = 10 // 600 req/min
|
||||
)
|
||||
```
|
||||
|
||||
**Automatic Cleanup**:
|
||||
- Removes unused limiters every 5 minutes
|
||||
- Prevents memory leaks from stale device limiters
|
||||
- Maintains peak performance
|
||||
|
||||
---
|
||||
|
||||
### 4. ✅ HTTPS Enforcement
|
||||
|
||||
**File**: `internal/middleware/security.go` (180+ lines)
|
||||
|
||||
**What Was Implemented**:
|
||||
- Automatic HTTP → HTTPS redirect
|
||||
- Security headers on all responses
|
||||
- SSL proxy support for load balancers
|
||||
- CORS with security best practices
|
||||
|
||||
**Security Headers Added**:
|
||||
```http
|
||||
X-Content-Type-Options: nosniff
|
||||
X-Frame-Options: DENY
|
||||
X-XSS-Protection: 1; mode=block
|
||||
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
|
||||
Content-Security-Policy: default-src 'self'
|
||||
Referrer-Policy: strict-origin-when-cross-origin
|
||||
Permissions-Policy: geolocation=(), microphone=(), camera=()
|
||||
```
|
||||
|
||||
**HTTPS Redirect**:
|
||||
```go
|
||||
// Automatic redirect in production
|
||||
if c.Scheme() == "http" {
|
||||
target.Scheme = "https"
|
||||
return c.Redirect(http.StatusMovedPermanently, target)
|
||||
}
|
||||
```
|
||||
|
||||
**SSL Proxy Support**:
|
||||
```go
|
||||
// Handles X-Forwarded-* headers from load balancers
|
||||
if proto := c.Request().Header.Get("X-Forwarded-Proto"); proto == "https" {
|
||||
c.Request().URL.Scheme = "https"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. ✅ Device Cap Per User
|
||||
|
||||
**File**: `internal/handlers/device_cap.go` (180+ lines)
|
||||
|
||||
**What Was Implemented**:
|
||||
- Maximum 10 devices per user (configurable)
|
||||
- Device usage statistics
|
||||
- Automatic enforcement on registration
|
||||
- Clear error messages with suggestions
|
||||
- Admin override capability
|
||||
|
||||
**Security Benefits**:
|
||||
- ✅ **Attack Surface Reduction**: Limits blast radius of compromised credentials
|
||||
- ✅ **Resource Protection**: Prevents account abuse
|
||||
- ✅ **Cost Control**: Manages server resources efficiently
|
||||
- ✅ **User Safety**: Helps users track their devices
|
||||
|
||||
**Implementation**:
|
||||
```go
|
||||
const MaxDevicesPerUser = 10
|
||||
|
||||
// Check before allowing device registration
|
||||
func ValidateUserDeviceCount(ctx, db, userID) error {
|
||||
devices := db.ListDevicesByUser(ctx, userID)
|
||||
if len(devices) >= MaxDevicesPerUser {
|
||||
return "Device limit reached"
|
||||
}
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
**Error Response**:
|
||||
```json
|
||||
{
|
||||
"error": "You have reached your device limit (10 devices)",
|
||||
"max_devices": 10,
|
||||
"current_count": 10,
|
||||
"device_list": [
|
||||
"My Kindle (koreader)",
|
||||
"My Kobo (kobo)",
|
||||
"Work iPad (web)"
|
||||
],
|
||||
"suggestions": [
|
||||
"Remove an unused device from Settings",
|
||||
"Contact support to increase your limit"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Middleware Chain (Recommended Order)
|
||||
|
||||
```go
|
||||
e.Pre(
|
||||
// Security first
|
||||
middleware.HTTPSRedirectMiddleware("8443"),
|
||||
middleware.SecurityHeadersMiddleware(),
|
||||
|
||||
// Rate limiting
|
||||
middleware.GlobalRateLimiter(config),
|
||||
middleware.SyncRateLimiterMiddleware(syncLimiter, "sync"),
|
||||
|
||||
// Request limits
|
||||
middleware.RequestSizeMiddleware(sizeConfig, logger),
|
||||
|
||||
// Device limits
|
||||
handlers.CheckDeviceCapMiddleware(capConfig, db),
|
||||
|
||||
// Authentication
|
||||
middleware.JWTMiddleware(jwtConfig),
|
||||
|
||||
// Device auth (if applicable)
|
||||
middleware.DeviceAuthMiddleware(db),
|
||||
|
||||
// Request signing (for sync endpoints)
|
||||
middleware.RequestSigningMiddleware(signingConfig, getSecret),
|
||||
|
||||
// CORS
|
||||
middleware.SecureCORSMiddleware(corsConfig),
|
||||
)
|
||||
```
|
||||
|
||||
### Example Usage in main.go
|
||||
|
||||
```go
|
||||
import (
|
||||
"bookhoard/internal/middleware"
|
||||
"bookhoard/internal/handlers"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// ... setup code ...
|
||||
|
||||
// Security middleware
|
||||
securityMiddleware := middleware.HTTPSProtectionMiddleware(
|
||||
true, // enable redirect
|
||||
"8443", // HTTPS port
|
||||
)
|
||||
|
||||
e.Pre(securityMiddleware...)
|
||||
|
||||
// Apply to sync routes
|
||||
syncGroup := e.Group("/api/sync")
|
||||
syncGroup.Use(
|
||||
middleware.RequestSigningMiddleware(signingConfig, getSecret),
|
||||
)
|
||||
|
||||
koreaderSync := syncGroup.Group("/koreader")
|
||||
koreaderSync.POST("/progress",
|
||||
middleware.SyncRateLimiterMiddleware(limiter, "sync"),
|
||||
koreaderHandler.SyncProgress,
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Security Enhancements
|
||||
|
||||
### Unit Tests Required
|
||||
|
||||
**HMAC Signing**:
|
||||
```go
|
||||
func TestRequestSigning_ValidRequest(t *testing.T)
|
||||
func TestRequestSigning_InvalidSignature(t *testing.T)
|
||||
func TestRequestSigning_ReplayAttack(t *testing.T)
|
||||
func TestRequestSigning_ClockSkew(t *testing.T)
|
||||
```
|
||||
|
||||
**Request Size Limits**:
|
||||
```go
|
||||
func TestRequestSizeLimit_SyncPayload(t *testing.T)
|
||||
func TestRequestSizeLimit_ExceedsLimit(t *testing.T)
|
||||
func TestRequestSizeLimit_DifferentEndpoints(t *testing.T)
|
||||
```
|
||||
|
||||
**Rate Limiting**:
|
||||
```go
|
||||
func TestRateLimiting_DeviceLimit(t *testing.T)
|
||||
func TestRateLimiting_UserLimit(t *testing.T)
|
||||
func TestRateLimiting_GlobalLimit(t *testing.T)
|
||||
func TestRateLimiting_Cleanup(t *testing.T)
|
||||
```
|
||||
|
||||
**Device Cap**:
|
||||
```go
|
||||
func TestDeviceCap_UnderLimit(t *testing.T)
|
||||
func TestDeviceCap_AtLimit(t *testing.T)
|
||||
func TestDeviceCap_ExceedsLimit(t *testing.T)
|
||||
func TestDeviceCap_AdminOverride(t *testing.T)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Impact
|
||||
|
||||
### Overhead Analysis
|
||||
|
||||
| Feature | CPU Overhead | Memory Overhead | Network Impact |
|
||||
|---------|-------------|----------------|---------------|
|
||||
| HMAC Signing | ~0.5ms per request | ~100 bytes/device | +40 bytes/req |
|
||||
| Size Limits | ~0.1ms per request | Minimal | None |
|
||||
| Enhanced Rate Limiting | ~0.2ms per request | ~1KB total | None |
|
||||
| Device Cap | ~1ms per registration | Minimal | None |
|
||||
| HTTPS Headers | <0.1ms per request | ~200 bytes | +500 bytes/req |
|
||||
|
||||
**Total Overhead**: ~1.9ms per request, ~1.3KB memory, +540 bytes/req
|
||||
|
||||
**Trade-offs**: Minimal overhead for significantly enhanced security
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Security settings
|
||||
ENABLE_REQUEST_SIGNING=true
|
||||
SIGNATURE_TIMESTAMP_TOLERANCE=300 # seconds
|
||||
SIGNATURE_MAX_CLOCK_SKEW=60 # seconds
|
||||
|
||||
# Rate limiting
|
||||
DEVICE_SYNC_RATE_LIMIT=120 # req/min
|
||||
DEVICE_METADATA_RATE_LIMIT=30 # req/min
|
||||
USER_SYNC_RATE_LIMIT=300 # req/min
|
||||
GLOBAL_RATE_LIMIT=600 # req/min
|
||||
|
||||
# Request size limits
|
||||
MAX_SYNC_PAYLOAD=10485760 # 10MB
|
||||
MAX_ANNOTATION_SIZE=102400 # 100KB
|
||||
MAX_METADATA_SIZE=1048576 # 1MB
|
||||
MAX_IMAGE_UPLOAD_SIZE=52428800 # 50MB
|
||||
|
||||
# Device limits
|
||||
MAX_DEVICES_PER_USER=10
|
||||
|
||||
# HTTPS
|
||||
HTTPS_PORT=8443
|
||||
HTTPS_REDIRECT_ENABLED=true
|
||||
```
|
||||
|
||||
### Runtime Configuration
|
||||
|
||||
```go
|
||||
// In main.go
|
||||
signingConfig := &middleware.RequestSigningConfig{
|
||||
Enabled: true,
|
||||
TimestampTolerance: 5 * time.Minute,
|
||||
MaxClockSkew: 1 * time.Minute,
|
||||
}
|
||||
|
||||
rateConfig := &middleware.SyncRateLimiterConfig{
|
||||
DeviceSyncRate: 120 / time.Minute,
|
||||
DeviceMetadataRate: 30 / time.Minute,
|
||||
UserSyncRate: 300 / time.Minute,
|
||||
GlobalRate: 600 / time.Minute,
|
||||
}
|
||||
|
||||
sizeConfig := &middleware.RequestSizeLimitConfig{
|
||||
MaxSyncPayloadSize: 10 * 1024 * 1024,
|
||||
MaxAnnotationSize: 100 * 1024,
|
||||
MaxMetadataSize: 1 * 1024 * 1024,
|
||||
MaxImageUploadSize: 50 * 1024 * 1024,
|
||||
}
|
||||
|
||||
capConfig := &handlers.DeviceCapConfig{
|
||||
MaxDevices: 10,
|
||||
AllowAdminOverride: true,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### For Existing Deployments
|
||||
|
||||
**Step 1: Update Dependencies**
|
||||
```bash
|
||||
# No new dependencies required
|
||||
# Uses existing crypto/hmac and uuid packages
|
||||
```
|
||||
|
||||
**Step 2: Update Environment Variables**
|
||||
```bash
|
||||
# Add to .env or docker-compose.yml
|
||||
ENABLE_REQUEST_SIGNING=true
|
||||
MAX_DEVICES_PER_USER=10
|
||||
```
|
||||
|
||||
**Step 3: Update Middleware Chain**
|
||||
```go
|
||||
// Add to main.go middleware chain
|
||||
import "bookhoard/internal/middleware"
|
||||
|
||||
// In main():
|
||||
securityMiddleware := middleware.HTTPSProtectionMiddleware(true, "8443")
|
||||
e.Pre(securityMiddleware...)
|
||||
```
|
||||
|
||||
**Step 4: Regenerate Device Secrets** (Optional)
|
||||
```sql
|
||||
-- For existing devices, generate signing secrets
|
||||
UPDATE devices
|
||||
SET auth_token =
|
||||
auth_token || gen_random_uuid() ||
|
||||
'device-secret-' || encode(gen_random_bytes(16), 'hex')
|
||||
WHERE auth_token IS NULL OR auth_token = '';
|
||||
```
|
||||
|
||||
**Step 5: Deploy**
|
||||
```bash
|
||||
# Build and restart server
|
||||
docker-compose down
|
||||
docker-compose up --build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Alerts
|
||||
|
||||
### Key Metrics to Monitor
|
||||
|
||||
1. **Security Events**:
|
||||
- Invalid signature attempts
|
||||
- Rate limit violations
|
||||
- Device cap rejections
|
||||
- Request size limit violations
|
||||
|
||||
2. **Performance Metrics**:
|
||||
- HMAC signing overhead
|
||||
- Rate limiter hit rates
|
||||
- Request size distribution
|
||||
- Device registration trends
|
||||
|
||||
3. **Alerts**:
|
||||
- > 100 failed signature attempts in 5 minutes
|
||||
- > 50 rate limit violations in 5 minutes
|
||||
- Device limit reached (alert admin)
|
||||
- Large request spike (potential DoS)
|
||||
|
||||
### Log Examples
|
||||
|
||||
**Security Event Log**:
|
||||
```json
|
||||
{
|
||||
"timestamp": "2026-01-31T12:00:00Z",
|
||||
"event": "invalid_signature",
|
||||
"device_id": "device-123",
|
||||
"request_id": "req-456",
|
||||
"ip_address": "192.168.1.100",
|
||||
"signature_provided": "abc123...",
|
||||
"signature_expected": "def456...",
|
||||
"user_agent": "KOReader/2024.01"
|
||||
}
|
||||
```
|
||||
|
||||
**Rate Limit Log**:
|
||||
```json
|
||||
{
|
||||
"timestamp": "2026-01-31T12:00:00Z",
|
||||
"event": "rate_limit_exceeded",
|
||||
"device_id": "device-123",
|
||||
"limit": 120,
|
||||
"window": "60s",
|
||||
"current": 150,
|
||||
"path": "/api/sync/koreader/progress"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Compliance
|
||||
|
||||
### GDPR Compliance
|
||||
|
||||
**Data Protection**:
|
||||
- ✅ Enhanced data integrity via HMAC signing
|
||||
- ✅ Secure data transmission (HTTPS enforced)
|
||||
- ✅ Access control (device limits, rate limiting)
|
||||
|
||||
**Privacy**:
|
||||
- ✅ Request ID tracing without PII
|
||||
- ✅ No sensitive data in logs
|
||||
- ✅ Device token protection
|
||||
|
||||
### OWASP Top 10 Coverage
|
||||
|
||||
| Risk | Coverage | Notes |
|
||||
|------|----------|-------|
|
||||
| A01 Broken Access Control | ✅ | Device auth + JWT + HMAC |
|
||||
| A02 Cryptographic Failures | ✅ | HMAC-SHA256 + TLS 1.3 |
|
||||
| A03 Injection | ✅ | Parameterized queries + validation |
|
||||
| A04 Insecure Design | ✅ | Rate limiting + size limits |
|
||||
| A05 Security Misconfiguration | ✅ | Security headers + HTTPS |
|
||||
| A06 Weak Auth | ✅ | bcrypt + JWT + device tokens |
|
||||
| A07 ID & Auth Failures | ✅ | Device cap + registration flow |
|
||||
| A08 Software/Data Integrity | ✅ | HMAC signing + validation |
|
||||
| A09 Logging & Monitoring | ✅ | Request tracing + audit logs |
|
||||
| A10 Server-Side Request Forgery | ✅ | CSRF headers + HMAC |
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
All **Priority 1** security enhancements from the audit have been successfully implemented. The system is now **production-ready** with significantly improved security posture.
|
||||
|
||||
### Key Achievements
|
||||
|
||||
✅ **Request Integrity**: HMAC signing prevents tampering
|
||||
✅ **DoS Protection**: Rate limiting + size limits
|
||||
✅ **HTTPS Enforcement**: Automatic redirects + security headers
|
||||
✅ **Access Control**: Device limits + enhanced authorization
|
||||
✅ **Audit Trail**: Request ID tracing for security monitoring
|
||||
|
||||
### Next Steps (Optional)
|
||||
|
||||
While the system is production-ready, you may consider:
|
||||
|
||||
1. **Performance Testing**: Load test with simulated sync traffic
|
||||
2. **Penetration Testing**: Professional security audit
|
||||
3. **Monitoring Setup**: Implement security event alerting
|
||||
4. **Documentation**: Update user docs with security info
|
||||
|
||||
---
|
||||
|
||||
**Implementation Status**: ✅ **COMPLETE**
|
||||
**Production Ready**: ✅ **YES**
|
||||
**Security Score**: **9.2/10 (A+)**
|
||||
**Recommendation**: **Deploy to Production**
|
||||
|
||||
---
|
||||
|
||||
**Implementation Completed**: January 31, 2026
|
||||
**Next Review**: Within 3 months
|
||||
**Questions**: security@bookhoard.example.com
|
||||
-553
@@ -1,553 +0,0 @@
|
||||
# Bookhoard Integration Test Suite Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides comprehensive information about the integration test suite for Bookhoard, including how to run tests, what they cover, and best practices for adding new tests.
|
||||
|
||||
## Test Architecture
|
||||
|
||||
### Location
|
||||
All integration tests are located in `cmd/server/tests/`
|
||||
|
||||
### Test Structure
|
||||
|
||||
```
|
||||
cmd/server/tests/
|
||||
├── main_test.go # Framework verification
|
||||
├── setup_test.go # Test setup and helper functions
|
||||
├── test_helpers.go # Reusable test helpers
|
||||
├── testrunner_test.go # Test runner verification
|
||||
│
|
||||
├── analytics_test.go # Analytics endpoints (NEW)
|
||||
├── auth_test.go # Authentication & authorization
|
||||
├── book_matching_test.go # Book matching & bulk linking (NEW)
|
||||
├── collections_bulk_test.go # Bulk collection operations (NEW)
|
||||
├── conflicts_bulk_test.go # Bulk conflict resolution (NEW)
|
||||
├── conflicts_test.go # Conflict management
|
||||
├── device_cap_test.go # Device capability tests
|
||||
├── device_test.go # Device management
|
||||
├── edge_cases_test.go # Edge case coverage
|
||||
├── filtering_test.go # Filtering functionality
|
||||
├── isbn_and_library_test.go # ISBN & library tests
|
||||
├── kobo_test.go # Kobo device sync
|
||||
├── koreader_test.go # KOReader sync
|
||||
├── library_test.go # Library management
|
||||
├── library_test_comprehensive.go # Comprehensive library tests
|
||||
├── media_bulk_test.go # Bulk media operations (NEW)
|
||||
├── new_fixes_test.go # Recent fixes validation
|
||||
├── opds_test.go # OPDS endpoints (NEW)
|
||||
├── phase1_integration_test.go # Phase 1 integration tests
|
||||
├── queue_test.go # Sync queue management
|
||||
├── refresh_token_test.go # Token refresh flow (NEW)
|
||||
├── registration_test.go # Device registration flow
|
||||
├── search_test.go # Search functionality
|
||||
├── security_test.go # Security tests
|
||||
├── sorting_test.go # Sorting functionality
|
||||
├── user_test.go # User management
|
||||
└── websocket_test.go # WebSocket connections
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **Database Setup**: Tests require a running PostgreSQL database
|
||||
```bash
|
||||
# Option 1: Use local database
|
||||
export DATABASE_PASSWORD=postgres
|
||||
|
||||
# Option 2: Use DATABASE_URL for containerized testing
|
||||
export DATABASE_URL="postgresql://user:pass@localhost:5432/bookhoard"
|
||||
```
|
||||
|
||||
2. **Dependencies**: Ensure all Go dependencies are installed
|
||||
```bash
|
||||
go mod download
|
||||
```
|
||||
|
||||
### Running All Tests
|
||||
|
||||
```bash
|
||||
# Run all tests in the test suite
|
||||
cd cmd/server/tests
|
||||
go test -v
|
||||
|
||||
# Run with coverage report
|
||||
go test -v -coverprofile=coverage.out
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
### Running Specific Test Files
|
||||
|
||||
```bash
|
||||
# Run only authentication tests
|
||||
go test -v -run TestAuth
|
||||
|
||||
# Run only analytics tests
|
||||
go test -v -run TestAnalytics
|
||||
|
||||
# Run specific test function
|
||||
go test -v -run TestAnalyticsReadingStats
|
||||
```
|
||||
|
||||
### Running Tests in Container
|
||||
|
||||
```bash
|
||||
# Build and run tests in Docker container
|
||||
podman-compose up -d db
|
||||
podman build -t bookhoard-test .
|
||||
podman run --network bookhoard_default -e DATABASE_URL="postgresql://postgres:postgres@db:5432/bookhoard" bookhoard-test go test ./cmd/server/tests/ -v
|
||||
```
|
||||
|
||||
### Test Modes
|
||||
|
||||
```bash
|
||||
# Short mode (skip lengthy tests)
|
||||
go test -short -v
|
||||
|
||||
# Verbose mode with detailed output
|
||||
go test -v
|
||||
|
||||
# Race detection
|
||||
go test -race -v
|
||||
```
|
||||
|
||||
## Test Coverage Summary
|
||||
|
||||
### Coverage by Handler
|
||||
|
||||
| Handler | Test File | Coverage | Notes |
|
||||
|---------|-----------|----------|-------|
|
||||
| **Analytics** | analytics_test.go | ✅ 100% | All 3 endpoints tested |
|
||||
| **Auth** | auth_test.go | ✅ 95% | Login, register, profile, tokens |
|
||||
| **Book Matching** | book_matching_test.go | ✅ 100% | Query, bulk link, auto-link, suggestions |
|
||||
| **Collections** | collections_bulk_test.go | ✅ 100% | Bulk add operations |
|
||||
| **Conflicts** | conflicts_bulk_test.go | ✅ 100% | Bulk resolve/dismiss operations |
|
||||
| **Devices** | device_test.go, device_cap_test.go | ✅ 95% | Registration, management, capabilities |
|
||||
| **Ebook/Scanner** | scanner tests | ✅ 90% | Scan, watch, metadata extraction |
|
||||
| **KOReader** | koreader_test.go | ✅ 100% | Sync progress, metadata, library |
|
||||
| **Kobo** | kobo_test.go | ✅ 100% | Initialization, markup, bookmarks |
|
||||
| **Library** | library_test.go, library_test_comprehensive.go | ✅ 95% | CRUD, folders, visibility, types |
|
||||
| **Media** | media_bulk_test.go | ✅ 100% | Bulk delete, bulk update |
|
||||
| **OPDS** | opds_test.go | ✅ 100% | Catalog, search, download, conversion |
|
||||
| **Progress** | progress tests | ✅ 90% | Universal progress, history |
|
||||
| **Queue** | queue_test.go | ✅ 100% | Queue management, retry, delete |
|
||||
| **Refresh Token** | refresh_token_test.go | ✅ 100% | Token refresh, security, edge cases |
|
||||
| **Search** | search_test.go | ✅ 95% | Media item search, filters |
|
||||
| **WebSocket** | websocket_test.go | ✅ 100% | Connection, auth, broadcasts |
|
||||
|
||||
### Overall Statistics
|
||||
|
||||
- **Total Test Functions**: 150+
|
||||
- **Total Test Cases**: 500+
|
||||
- **Code Coverage**: ~95% of backend code
|
||||
- **Endpoint Coverage**: 100% of all REST and WebSocket endpoints
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Authentication & Authorization Tests
|
||||
|
||||
**File**: `auth_test.go`
|
||||
|
||||
- JWT token validation
|
||||
- User registration (including first-user-admin)
|
||||
- Login with rate limiting
|
||||
- Password complexity requirements
|
||||
- Profile management
|
||||
- Token refresh flow
|
||||
- Account lockout
|
||||
- Role-based access control
|
||||
|
||||
### 2. Analytics Tests (NEW)
|
||||
|
||||
**File**: `analytics_test.go`
|
||||
|
||||
- Reading statistics with date ranges
|
||||
- Device usage statistics
|
||||
- Popular books queries
|
||||
- Invalid date handling
|
||||
- Empty data handling
|
||||
- Response structure validation
|
||||
|
||||
### 3. Book Matching Tests (NEW)
|
||||
|
||||
**File**: `book_matching_test.go`
|
||||
|
||||
- Query books by title/author/identifiers
|
||||
- Bulk linking operations
|
||||
- Auto-linking with confidence thresholds
|
||||
- Unlinked book suggestions
|
||||
- Device file alias management
|
||||
- Error handling for invalid IDs
|
||||
|
||||
### 4. Bulk Operations Tests (NEW)
|
||||
|
||||
**Files**: `collections_bulk_test.go`, `conflicts_bulk_test.go`, `media_bulk_test.go`
|
||||
|
||||
- **Collections**: Bulk add books to multiple collections
|
||||
- **Conflicts**: Bulk resolve with strategies (most_recent, highest_progress, manual)
|
||||
- **Conflicts**: Bulk dismiss resolved conflicts
|
||||
- **Media**: Bulk delete books
|
||||
- **Media**: Bulk update metadata (tags, status, rating)
|
||||
|
||||
### 5. Device Management Tests
|
||||
|
||||
**Files**: `device_test.go`, `device_cap_test.go`, `registration_test.go`
|
||||
|
||||
- Device registration flow
|
||||
- Device approval/rejection
|
||||
- Device capabilities detection
|
||||
- Device metadata management
|
||||
- Multiple device handling
|
||||
- Device authentication
|
||||
|
||||
### 6. E-Reader Integration Tests
|
||||
|
||||
**Files**: `kobo_test.go`, `koreader_test.go`
|
||||
|
||||
- **Kobo**: Initialization handshake
|
||||
- **Kobo**: Markup sync
|
||||
- **Kobo**: Bookmark sync
|
||||
- **Kobo**: Analytics endpoint
|
||||
- **KOReader**: Progress sync
|
||||
- **KOReader**: Metadata retrieval
|
||||
- **KOReader**: Library sync
|
||||
- **KOReader**: Bookmark sync
|
||||
|
||||
### 7. Library Management Tests
|
||||
|
||||
**Files**: `library_test.go`, `library_test_comprehensive.go`, `isbn_and_library_test.go`
|
||||
|
||||
- Library CRUD operations
|
||||
- Folder management
|
||||
- Library visibility
|
||||
- Library types
|
||||
- ISBN normalization
|
||||
- Scan settings
|
||||
|
||||
### 8. Media Management Tests
|
||||
|
||||
**Files**: `media_bulk_test.go`, `search_test.go`, `filtering_test.go`, `sorting_test.go`
|
||||
|
||||
- Media item CRUD
|
||||
- Bulk operations
|
||||
- Search functionality
|
||||
- Filtering and sorting
|
||||
- Progress tracking
|
||||
- Notes and highlights
|
||||
- Ratings
|
||||
|
||||
### 9. OPDS Tests (NEW)
|
||||
|
||||
**File**: `opds_test.go`
|
||||
|
||||
- Device catalog retrieval
|
||||
- Search functionality
|
||||
- Navigation endpoint
|
||||
- Book download
|
||||
- Cover image retrieval
|
||||
- Format listing
|
||||
- On-the-fly KEPUB conversion
|
||||
|
||||
### 10. Progress & Queue Tests
|
||||
|
||||
**Files**: `queue_test.go`, progress tests in other files
|
||||
|
||||
- Sync queue management
|
||||
- Queue retry mechanism
|
||||
- Progress tracking
|
||||
- Reading history
|
||||
- Universal progress
|
||||
|
||||
### 11. Security Tests
|
||||
|
||||
**File**: `security_test.go`
|
||||
|
||||
- SQL injection prevention
|
||||
- XSS prevention
|
||||
- CSRF protection
|
||||
- Rate limiting
|
||||
- Input validation
|
||||
- Authorization checks
|
||||
|
||||
### 12. WebSocket Tests
|
||||
|
||||
**File**: `websocket_test.go`
|
||||
|
||||
- WebSocket connection establishment
|
||||
- Device authentication via WebSocket
|
||||
- Real-time progress broadcasts
|
||||
- Ping/pong heartbeat
|
||||
- Connection limits
|
||||
- Message handling
|
||||
|
||||
### 13. Token Refresh Tests (NEW)
|
||||
|
||||
**File**: `refresh_token_test.go`
|
||||
|
||||
- Valid token refresh
|
||||
- Invalid/expired token handling
|
||||
- Token reuse protection
|
||||
- Token tampering detection
|
||||
- Response structure validation
|
||||
- Edge cases (empty, null, malformed)
|
||||
|
||||
## Test Helper Functions
|
||||
|
||||
### setupTestServer
|
||||
|
||||
Creates a test server with database connection.
|
||||
|
||||
```go
|
||||
ts, db, cfg, handler := setupTestServer(t)
|
||||
defer ts.Close()
|
||||
```
|
||||
|
||||
**Returns**:
|
||||
- `ts`: Test HTTP server
|
||||
- `db`: Database queries interface
|
||||
- `cfg`: Test configuration
|
||||
- `handler`: Handler instance
|
||||
|
||||
### loginTestUser
|
||||
|
||||
Logs in a test user and returns JWT token.
|
||||
|
||||
```go
|
||||
token := loginTestUser(t, ts, db)
|
||||
```
|
||||
|
||||
**Returns**:
|
||||
- `token`: JWT access token
|
||||
|
||||
### getTestUserID
|
||||
|
||||
Gets or creates a test user.
|
||||
|
||||
```go
|
||||
userID := getTestUserID(t, db)
|
||||
```
|
||||
|
||||
**Returns**:
|
||||
- `userID`: UUID of test user
|
||||
|
||||
### createTestEbookID
|
||||
|
||||
Creates a test ebook and returns its ID.
|
||||
|
||||
```go
|
||||
bookID := createTestEbookID(t, ts, token)
|
||||
```
|
||||
|
||||
**Returns**:
|
||||
- `bookID`: String ID of created ebook
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
### Template for Endpoint Tests
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestNewEndpoint(t *testing.T) {
|
||||
t.Run("Endpoint_WithoutAuth", func(t *testing.T) {
|
||||
ts, _, _, _ := setupTestServer(t)
|
||||
defer ts.Close()
|
||||
|
||||
// Test without authentication
|
||||
req, _ := http.NewRequest("GET", ts.URL+"/api/new-endpoint", nil)
|
||||
client := &http.Client{}
|
||||
resp, err := client.Do(req)
|
||||
require.NoError(t, err)
|
||||
defer resp.Body.Close()
|
||||
|
||||
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
|
||||
})
|
||||
|
||||
t.Run("Endpoint_WithAuth", func(t *testing.T) {
|
||||
ts, db, _, _ := setupTestServer(t)
|
||||
defer ts.Close()
|
||||
|
||||
token := loginTestUser(t, ts, db)
|
||||
|
||||
// Test with authentication
|
||||
req, _ := http.NewRequest("GET", ts.URL+"/api/new-endpoint", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
client := &http.Client{}
|
||||
resp, err := client.Do(req)
|
||||
require.NoError(t, err)
|
||||
defer resp.Body.Close()
|
||||
|
||||
assert.Equal(t, http.StatusOK, resp.StatusCode)
|
||||
|
||||
var result map[string]interface{}
|
||||
json.NewDecoder(resp.Body).Decode(&result)
|
||||
|
||||
// Add assertions for response structure
|
||||
assert.Contains(t, result, "expected_field")
|
||||
})
|
||||
|
||||
t.Run("Endpoint_InvalidInput", func(t *testing.T) {
|
||||
ts, db, _, _ := setupTestServer(t)
|
||||
defer ts.Close()
|
||||
|
||||
token := loginTestUser(t, ts, db)
|
||||
|
||||
// Test with invalid input
|
||||
req := map[string]interface{}{
|
||||
"invalid": "data",
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
|
||||
httpReq, _ := http.NewRequest("POST", ts.URL+"/api/new-endpoint", bytes.NewBuffer(body))
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
httpReq.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
client := &http.Client{}
|
||||
resp, err := client.Do(httpReq)
|
||||
require.NoError(t, err)
|
||||
defer resp.Body.Close()
|
||||
|
||||
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Use Table-Driven Tests** for multiple similar test cases
|
||||
2. **Test All Error Paths**: Not just success cases
|
||||
3. **Validate Response Structure**: Check all expected fields
|
||||
4. **Test Edge Cases**: Empty inputs, invalid IDs, boundary values
|
||||
5. **Use Subtests**: For organizing related test cases
|
||||
6. **Clean Up Resources**: Always close response bodies
|
||||
7. **Use require.NoError** for setup, assert.NoError for test conditions
|
||||
8. **Create Isolated Tests**: Each test should be independent
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: Integration Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15
|
||||
env:
|
||||
POSTGRES_DB: bookhoard
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-go@v4
|
||||
with:
|
||||
go-version: '1.25'
|
||||
|
||||
- name: Run integration tests
|
||||
env:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/bookhoard
|
||||
run: |
|
||||
cd cmd/server/tests
|
||||
go test -v -race -coverprofile=coverage.out
|
||||
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v3
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Database Connection Errors**
|
||||
```bash
|
||||
# Ensure database is running
|
||||
podman ps | grep postgres
|
||||
|
||||
# Check connection string
|
||||
echo $DATABASE_URL
|
||||
```
|
||||
|
||||
2. **Port Already in Use**
|
||||
```bash
|
||||
# Tests use random ports (port 0), so this shouldn't happen
|
||||
# If it does, check for running processes
|
||||
lsof -i :8765
|
||||
```
|
||||
|
||||
3. **Test Data Cleanup**
|
||||
- Tests use automatic cleanup via `defer ts.Close()`
|
||||
- Manual cleanup may be needed for complex scenarios
|
||||
- Consider using database transactions for rollback
|
||||
|
||||
4. **Time-Dependent Tests**
|
||||
- Use fixed time values in tests
|
||||
- Mock time functions if necessary
|
||||
- Add tolerance for timestamp comparisons
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Test Execution Time
|
||||
|
||||
- Total suite: ~2-3 minutes
|
||||
- Individual test files: 5-30 seconds
|
||||
- Use `-short` flag for faster CI runs
|
||||
- Parallel test execution with `-parallel` flag
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Use Test Caching**: Go 1.18+ caches test results
|
||||
2. **Minimize Database Calls**: Create test data once
|
||||
3. **Parallelize Independent Tests**: Use `t.Parallel()`
|
||||
4. **Avoid Sleep**: Use channels for synchronization
|
||||
|
||||
## Future Improvements
|
||||
|
||||
### Planned Enhancements
|
||||
|
||||
- [ ] Add property-based testing with `github.com/stretchr/testify`
|
||||
- [ ] Implement fuzzing for input validation
|
||||
- [ ] Add performance benchmarks
|
||||
- [ ] Contract testing for API compatibility
|
||||
- [ ] Visual regression testing for UI endpoints
|
||||
|
||||
### Coverage Goals
|
||||
|
||||
- **Current**: ~95% backend coverage
|
||||
- **Target**: 98% backend coverage
|
||||
- **Frontend**: Add integration tests for frontend components
|
||||
|
||||
## References
|
||||
|
||||
- [Go Testing Guide](https://golang.org/doc/tutorial/add-a-test)
|
||||
- [Testify Documentation](https://github.com/stretchr/testify)
|
||||
- [Go Concurrency Testing](https://go.dev/doc/articles/race_detector)
|
||||
- [API Testing Best Practices](https://martinfowler.com/articles/practical-test-pyramid.html)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-02-01
|
||||
**Maintained By**: Bookhoard Development Team
|
||||
@@ -15,13 +15,17 @@ cp .env.example .env
|
||||
|
||||
# 2. Edit with secure values
|
||||
nano .env
|
||||
# Required:
|
||||
JWT_SECRET="your-secure-jwt-secret-key-here" # 64+ char random string
|
||||
DBPASS="your-secure-database-password" # Strong password
|
||||
|
||||
# Optional:
|
||||
SERVER_PORT=8765
|
||||
DATABASE_HOST=localhost # For local development
|
||||
# Required variables:
|
||||
JWT_SECRET="your-secure-jwt-secret-key-here" # 64+ char random string
|
||||
DBPASS="your-secure-database-password" # Strong password
|
||||
|
||||
# Optional variables (with defaults in docker-compose.yml):
|
||||
# TEST_MODE=false # Disables rate limiting (NEVER in production)
|
||||
# RATE_LIMIT_ENABLED=true # Enable/disable rate limiting
|
||||
# REQUESTS_PER_MINUTE=10 # Rate limit per IP
|
||||
# BOOKHOARD_CONVERSION_TOOL=/usr/bin/ebook-convert
|
||||
# BOOKHOARD_CONVERSION_CACHE_TTL=48h
|
||||
```
|
||||
|
||||
### 2. **Port Conflicts**
|
||||
@@ -40,18 +44,24 @@ sudo kill -9 $(lsof -t -i:8765)
|
||||
SERVER_PORT=8766
|
||||
```
|
||||
|
||||
### 3. **Docker Engine Compatibility**
|
||||
### 3. **Container Runtime - Podman vs Docker**
|
||||
|
||||
**Issue:** Using Podman instead of Docker
|
||||
**Issue:** Container runtime compatibility
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Both work, but for full Docker compatibility:
|
||||
# Install Docker Desktop
|
||||
# or use Docker instead of podman command
|
||||
# Podman is recommended (podman-compose works with docker-compose.yml)
|
||||
# Install podman-compose:
|
||||
sudo apt install podman-compose # Debian/Ubuntu
|
||||
|
||||
# Podman users: ensure podman-compose is installed
|
||||
# Docker and Podman can both use the same compose file
|
||||
# Docker also works (use docker-compose with docker-compose.yml)
|
||||
# Both runtimes use the same docker-compose.yml file
|
||||
|
||||
# Podman users:
|
||||
podman-compose up -d
|
||||
|
||||
# Docker users:
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 4. **Database Permissions**
|
||||
@@ -61,12 +71,18 @@ SERVER_PORT=8766
|
||||
**Solution:**
|
||||
```bash
|
||||
# Clean database volume and restart:
|
||||
# Podman:
|
||||
podman-compose down -v
|
||||
podman volume rm bookhoard_postgres_data 2>/dev/null
|
||||
podman-compose up -d
|
||||
|
||||
# Docker:
|
||||
docker compose down -v
|
||||
docker volume rm bookhoard_postgres_data 2>/dev/null
|
||||
docker compose up -d
|
||||
|
||||
# Check database logs for errors:
|
||||
docker compose logs db
|
||||
podman-compose logs db # or: docker compose logs db
|
||||
```
|
||||
|
||||
### 5. **Build Dependencies**
|
||||
@@ -81,10 +97,35 @@ which sqlc # Check if sqlc is accessible
|
||||
which templ # Check if templ is accessible
|
||||
|
||||
# Rebuild if tools are missing:
|
||||
docker compose build --no-cache
|
||||
podman-compose build --no-cache # or: docker compose build --no-cache
|
||||
```
|
||||
|
||||
### 6. **Platform-Specific Issues**
|
||||
### 6. **Conversion Cache Issues**
|
||||
|
||||
**Issue:** KEPUB conversion fails or cache problems
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Check cache directory exists and is writable
|
||||
ls -la /var/bookhoard/cache/kepub
|
||||
|
||||
# Create cache directory if missing
|
||||
sudo mkdir -p /var/bookhoard/cache/kepub
|
||||
sudo chmod 755 /var/bookhoard/cache/kepub
|
||||
|
||||
# Clear conversion cache (safe - will reconvert on next download)
|
||||
sudo rm -rf /var/bookhoard/cache/kepub/*
|
||||
|
||||
# Verify kepubify is installed
|
||||
which kepubify
|
||||
# or: which ebook-convert
|
||||
|
||||
# Conversion service defaults are in docker-compose.yml
|
||||
# Check if you're overriding them in .env:
|
||||
grep BOOKHOARD_CONVERSION .env
|
||||
```
|
||||
|
||||
### 7. **Platform-Specific Issues**
|
||||
|
||||
**Issue:** Different OS architectures (ARM vs x86)
|
||||
|
||||
@@ -103,17 +144,17 @@ FROM golang:1.25-alpine AS builder
|
||||
# ... rest of Dockerfile remains same
|
||||
```
|
||||
|
||||
### 7. **Network Connectivity**
|
||||
### 8. **Network Connectivity**
|
||||
|
||||
**Issue:** Can't connect to localhost
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Check if containers are running:
|
||||
docker compose ps
|
||||
podman-compose ps # or: docker compose ps
|
||||
|
||||
# Test database connection:
|
||||
docker compose exec db psql -U postgres -d bookhoard -c "SELECT 1;"
|
||||
podman-compose exec db psql -U postgres -d bookhoard -c "SELECT 1;" # or: docker compose exec db ...
|
||||
|
||||
# Test API endpoint:
|
||||
curl -s http://localhost:8765/api/libraries/visible
|
||||
@@ -127,10 +168,10 @@ curl -s http://SERVER_IP:8765/api/libraries/visible
|
||||
### Basic Health Checks:
|
||||
```bash
|
||||
# Check container status
|
||||
docker compose ps
|
||||
podman-compose ps # or: docker compose ps
|
||||
|
||||
# Test database connection
|
||||
docker compose exec db psql -U postgres -d bookhoard -c "SELECT 1;"
|
||||
podman-compose exec db psql -U postgres -d bookhoard -c "SELECT 1;" # or: docker compose exec db ...
|
||||
|
||||
# Test API endpoint
|
||||
curl -s http://localhost:8765/api/libraries/visible
|
||||
@@ -0,0 +1,442 @@
|
||||
# Bookhoard Development Guide
|
||||
|
||||
This guide is for developers contributing to Bookhoard or setting up a development environment.
|
||||
|
||||
## 🏗 Architecture
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
bookhoard/
|
||||
├── cmd/server/ # Application entry point
|
||||
│ ├── main.go # Server initialization, route registration
|
||||
│ └── tests/ # Integration tests (30+ test files)
|
||||
├── internal/
|
||||
│ ├── config/ # Configuration management
|
||||
│ ├── database/ # Database layer (SQLC generated)
|
||||
│ ├── handlers/ # HTTP request handlers (18 files)
|
||||
│ ├── middleware/ # HTTP middleware (9 files)
|
||||
│ ├── services/ # Business logic (7 files)
|
||||
│ ├── sync/ # Sync framework (5 files)
|
||||
│ ├── opds/ # OPDS feed generation
|
||||
│ └── utils/ # Utility functions
|
||||
├── templates/ # UI templates (17 .templ files)
|
||||
├── web/src/ # Frontend TypeScript
|
||||
├── database/schema/ # Database schema
|
||||
├── docs/ # Documentation
|
||||
└── bruno/ # API test collections
|
||||
```
|
||||
|
||||
### Backend Components
|
||||
|
||||
**Handlers** (`internal/handlers/`):
|
||||
- `auth.go` - Authentication & user management
|
||||
- `library.go` - Library CRUD operations
|
||||
- `ebook.go` - Media item operations
|
||||
- `media.go` - Media downloads, shelves
|
||||
- `koreader.go` - KOReader sync protocol
|
||||
- `kobo.go` - Kobo sync protocol
|
||||
- `collections.go` - Collection management
|
||||
- `devices.go` - Device registration/management
|
||||
- `conflicts.go` - Sync conflict resolution
|
||||
- `queue.go` - Sync queue management
|
||||
- `progress.go` - Reading progress tracking
|
||||
- `analytics.go` - Usage analytics
|
||||
- `opds.go` - OPDS feed generation
|
||||
- `websocket.go` - WebSocket connections
|
||||
- `sync.go` - Sync orchestration
|
||||
- `book_matching.go` - Book linking/matching
|
||||
- `sidecar.go` - Sidecar file handling
|
||||
- `refresh_token.go` - Token refresh logic
|
||||
- `context.go` - Handler context utilities
|
||||
|
||||
**Middleware** (`internal/middleware/`):
|
||||
- `device_auth.go` - Device authentication
|
||||
- `device_rate_limiter.go` - Device-specific rate limiting
|
||||
- `error_handler.go` - Global error handling
|
||||
- `login_attempts.go` - Login attempt tracking
|
||||
- `password_validator.go` - Password complexity validation
|
||||
- `rate_limiter.go` - IP-based rate limiting
|
||||
- `request_tracing.go` - Request ID tracking
|
||||
- `security.go` - Security headers
|
||||
- `transaction.go` - Database transaction middleware
|
||||
|
||||
**Services** (`internal/services/`):
|
||||
- `library_service.go` - Library operations
|
||||
- `ebook_scanner.go` - File scanning & metadata extraction
|
||||
- `worker.go` - Job queue worker pool
|
||||
- `scheduler.go` - Scheduled task manager
|
||||
- `collection_service.go` - Collection rules processing
|
||||
- `conversion_service.go` - EPUB→KEPUB conversion
|
||||
- `book_matching.go` - Book matching algorithms
|
||||
|
||||
**Sync Framework** (`internal/sync/`):
|
||||
- `queue.go` - Sync queue processor
|
||||
- `progress.go` - Universal progress format
|
||||
- `websocket.go` - Real-time sync broadcast
|
||||
- `offline.go` - Offline sync support
|
||||
- `format.go` - Format group conversion
|
||||
|
||||
### Database Schema
|
||||
|
||||
**Core Tables**:
|
||||
- `users` - User accounts with authentication and settings
|
||||
- `libraries` - Library definitions
|
||||
- `library_types` - Media type definitions (ebooks, comics, manga)
|
||||
- `library_folders` - Multiple folders per library
|
||||
- `library_visibility` - User-specific library access control
|
||||
- `media_items` - Universal media storage (replaces ebooks table)
|
||||
- `media_ratings` - User ratings (1-10 scale for half-star precision)
|
||||
- `media_notes` - User annotations
|
||||
- `media_highlights` - User highlights with color customization
|
||||
- `reading_progress` - Universal progress tracking across devices
|
||||
- `devices` - Device registry for sync
|
||||
- `sync_queue` - Offline sync support
|
||||
- `sync_conflicts` - Conflict resolution tracking
|
||||
- `collections` - Device-neutral collections
|
||||
- `collection_items` - Books in collections
|
||||
- `device_shelf_mappings` - Map collections to device-specific shelves
|
||||
- `device_catalogs` - Track OPDS downloads and ContentId mappings
|
||||
- `kobo_shelves` - Kobo-specific shelf management
|
||||
- `reading_history` - Reading session tracking
|
||||
- `unlinked_books` - Track books that couldn't be auto-matched
|
||||
- `media_item_formats` - Track all format versions with hashes
|
||||
- `device_file_aliases` - Track file paths per device
|
||||
- `refresh_tokens` - JWT refresh token storage
|
||||
|
||||
**Database Functions**:
|
||||
- `normalize_isbn()` - ISBN format normalization
|
||||
- `detect_format_group()` - Detect format group (reflowable, fixed_layout, comic_archive)
|
||||
- `convert_progress()` - Convert progress between format groups
|
||||
- `detect_conflict()` - Detect sync conflicts
|
||||
- `merge_progress()` - Merge progress from multiple sources
|
||||
|
||||
### Technology Stack
|
||||
|
||||
**Backend**:
|
||||
- Go 1.25+
|
||||
- Echo v4 - HTTP framework
|
||||
- pgx v5 - PostgreSQL driver
|
||||
- SQLC - SQL code generation
|
||||
- jwt-go - JWT authentication
|
||||
- bcrypt - Password hashing
|
||||
|
||||
**Frontend**:
|
||||
- Templ - HTML templating with Go
|
||||
- HTMX - Dynamic interactions
|
||||
- Tailwind CSS - Styling
|
||||
- TypeScript - Frontend logic
|
||||
|
||||
**Database**:
|
||||
- PostgreSQL 15+
|
||||
- 30+ tables
|
||||
- 50+ indexes
|
||||
- JSONB for complex data
|
||||
|
||||
**Testing**:
|
||||
- Testify - Testing framework
|
||||
- Bruno - API testing
|
||||
- 30+ integration test files
|
||||
|
||||
## 🚀 Local Development
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Go 1.25+
|
||||
- Node.js 18+ (for frontend build)
|
||||
- Podman or Docker
|
||||
- PostgreSQL 15+ (or use Podman)
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# 1. Clone repository
|
||||
git clone https://github.com/yourusername/bookhoard.git
|
||||
cd bookhoard
|
||||
|
||||
# 2. Install Go dependencies
|
||||
go mod download
|
||||
|
||||
# 3. Install build tools
|
||||
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
|
||||
go install github.com/a-h/templ/cmd/templ@latest
|
||||
|
||||
# 4. Set up environment
|
||||
cp .env.example .env
|
||||
# Edit .env with your settings
|
||||
|
||||
# 5. Generate database code
|
||||
cd internal/database
|
||||
sqlc generate
|
||||
|
||||
# 6. Generate templates
|
||||
cd ../../templates
|
||||
templ generate
|
||||
|
||||
# 7. Build frontend
|
||||
cd ../web
|
||||
npm install
|
||||
npm run build
|
||||
|
||||
# 8. Run tests
|
||||
cd ..
|
||||
go test ./... -v
|
||||
```
|
||||
|
||||
### Running Locally
|
||||
|
||||
```bash
|
||||
# Option 1: Using containers (recommended)
|
||||
podman-compose up --build
|
||||
|
||||
# Option 2: Direct Go run (requires local PostgreSQL)
|
||||
export JWT_SECRET="your-dev-secret"
|
||||
export DBPASS="your-db-password"
|
||||
go run cmd/server/main.go
|
||||
```
|
||||
|
||||
### Development Workflow
|
||||
|
||||
**Backend Development**:
|
||||
```bash
|
||||
# Watch mode for Go (requires air or similar)
|
||||
air
|
||||
|
||||
# Or manual rebuild
|
||||
go build -o bookhoard cmd/server/main.go
|
||||
./bookhoard
|
||||
```
|
||||
|
||||
**Frontend Development**:
|
||||
```bash
|
||||
cd web
|
||||
npm run dev # Watch mode for TypeScript/CSS
|
||||
```
|
||||
|
||||
**Database Changes**:
|
||||
1. Edit `database/schema/schema.sql`
|
||||
2. Edit `internal/database/queries/queries.sql`
|
||||
3. Run: `cd internal/database && sqlc generate`
|
||||
4. Restart server
|
||||
|
||||
**Template Changes**:
|
||||
1. Edit `templates/*.templ`
|
||||
2. Run: `cd templates && templ generate`
|
||||
3. Restart server (templates auto-reload in dev mode)
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
# Run all unit tests
|
||||
go test ./... -v
|
||||
|
||||
# Run specific package tests
|
||||
go test ./internal/handlers/... -v
|
||||
|
||||
# Run with coverage
|
||||
go test ./... -coverprofile=coverage.out
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
```bash
|
||||
# Run integration tests (may hit rate limits)
|
||||
go test ./cmd/server/tests -v
|
||||
|
||||
# Run with test mode (recommended)
|
||||
TEST_MODE=true RATE_LIMIT_ENABLED=false go test ./cmd/server/tests -v
|
||||
|
||||
# Run specific test
|
||||
go test ./cmd/server/tests -run TestAuth -v
|
||||
```
|
||||
|
||||
### API Testing with Bruno
|
||||
|
||||
```bash
|
||||
# Install Bruno CLI
|
||||
npm install -g @usebruno/cli
|
||||
|
||||
# Run all tests
|
||||
bruno run
|
||||
|
||||
# Run specific collection
|
||||
bruno run bruno/user/
|
||||
bruno run bruno/sync-kobo/
|
||||
```
|
||||
|
||||
### Test Configuration
|
||||
|
||||
Environment variables for testing:
|
||||
- `TEST_MODE=true` - Enable test mode (disables rate limiting)
|
||||
- `RATE_LIMIT_ENABLED=false` - Disable rate limiting
|
||||
- `REQUESTS_PER_MINUTE=1000` - Increase rate limit
|
||||
|
||||
**⚠️ WARNING**: Never enable these in production!
|
||||
|
||||
## 📝 Code Style Guidelines
|
||||
|
||||
### Go Code
|
||||
|
||||
- Follow [Effective Go](https://go.dev/doc/effective_go) guidelines
|
||||
- Use `gofmt` for formatting
|
||||
- Use `golangci-lint` for linting
|
||||
- Use meaningful variable and function names
|
||||
- Add comments for complex business logic
|
||||
- Handle errors explicitly - don't ignore them
|
||||
|
||||
### Database Operations
|
||||
|
||||
- **Always use sqlc-generated code** - No raw SQL in handlers
|
||||
- Use transactions for multi-step operations
|
||||
- Handle `pgx.ErrNoRows` explicitly
|
||||
- Use `pgtype.UUID` for UUID parameters
|
||||
- Validate inputs before database operations
|
||||
|
||||
### Error Handling
|
||||
|
||||
```go
|
||||
// Good - Explicit error handling
|
||||
user, err := h.db.GetUser(c.Request().Context(), userID)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return c.JSON(http.StatusNotFound, map[string]string{"error": "user not found"})
|
||||
}
|
||||
return c.JSON(http.StatusInternalServerError, map[string]string{"error": "database error"})
|
||||
}
|
||||
|
||||
// Bad - Ignoring errors
|
||||
user, _ := h.db.GetUser(c.Request().Context(), userID)
|
||||
```
|
||||
|
||||
### Adding New Features
|
||||
|
||||
1. **Database First**: Add tables/columns to `schema.sql`
|
||||
2. **Generate Queries**: Add to `queries.sql` and run `sqlc generate`
|
||||
3. **Handler**: Implement in `internal/handlers/`
|
||||
4. **Routes**: Register in `cmd/server/main.go`
|
||||
5. **Tests**: Add integration test in `cmd/server/tests/`
|
||||
6. **Bruno**: Add API test in `bruno/`
|
||||
7. **Docs**: Update relevant documentation
|
||||
|
||||
### API Design Principles
|
||||
|
||||
- RESTful naming conventions
|
||||
- Consistent error responses
|
||||
- Proper HTTP status codes
|
||||
- JWT authentication on protected routes
|
||||
- Input validation with struct tags
|
||||
- Use echo.Context for request/response
|
||||
|
||||
## 🐳 Deployment
|
||||
|
||||
### Building for Production
|
||||
|
||||
```bash
|
||||
# Using Makefile
|
||||
make build-force
|
||||
|
||||
# Or manually
|
||||
podman-compose build --no-cache
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Required for production:
|
||||
- `JWT_SECRET` - 64-byte random string
|
||||
- `DBPASS` - Strong database password
|
||||
- `BASE_URL` - Public URL (e.g., https://bookhoard.example.com)
|
||||
|
||||
Optional:
|
||||
- `HTTPS_PROXY` - If behind reverse proxy
|
||||
|
||||
**Note**: Conversion service, rate limiting, and other operational settings have defaults in `docker-compose.yml` and can be overridden via `.env` if needed.
|
||||
|
||||
### Performance Tuning
|
||||
|
||||
**PostgreSQL Settings**:
|
||||
```sql
|
||||
-- In postgresql.conf
|
||||
shared_buffers = 256MB
|
||||
effective_cache_size = 1GB
|
||||
maintenance_work_mem = 64MB
|
||||
checkpoint_completion_target = 0.9
|
||||
wal_buffers = 16MB
|
||||
default_statistics_target = 100
|
||||
random_page_cost = 1.1
|
||||
effective_io_concurrency = 200
|
||||
work_mem = 2621kB
|
||||
min_wal_size = 1GB
|
||||
max_wal_size = 4GB
|
||||
```
|
||||
|
||||
**Go Settings**:
|
||||
- GOMAXPROCS = number of CPU cores
|
||||
- Worker pool concurrency: 3 (configurable in services/worker.go)
|
||||
|
||||
## 🔍 Debugging
|
||||
|
||||
### Enable Debug Logging
|
||||
|
||||
```bash
|
||||
# Set environment variable
|
||||
export DEBUG=true
|
||||
|
||||
# Or in .env
|
||||
DEBUG=true
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Database Connection Errors**:
|
||||
- Check PostgreSQL is running
|
||||
- Verify DATABASE_HOST and DATABASE_PORT
|
||||
- Check firewall settings
|
||||
|
||||
**Rate Limiting During Development**:
|
||||
- Enable test mode: `TEST_MODE=true RATE_LIMIT_ENABLED=false`
|
||||
- Or increase limit: `REQUESTS_PER_MINUTE=1000`
|
||||
|
||||
**Template Not Updating**:
|
||||
- Run `templ generate` in templates/ directory
|
||||
- Restart server
|
||||
|
||||
**Database Queries Not Working**:
|
||||
- Run `sqlc generate` in internal/database/
|
||||
- Check generated code in `queries.sql.go`
|
||||
- Verify SQL syntax in `queries.sql`
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [Project Guidelines](../../PROJECT_GUIDELINES.md) - Development rules and standards
|
||||
- [API Reference](../API_REFERENCE.md) - Complete API documentation
|
||||
- [Troubleshooting](../TROUBLESHOOTING.md) - Deployment issues
|
||||
- [Go Documentation](https://go.dev/doc/)
|
||||
- [Echo Framework](https://echo.labstack.com/docs)
|
||||
- [pgx Documentation](https://pgx.github.io/pgx/)
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
|
||||
3. Follow code style guidelines
|
||||
4. Add tests for new features
|
||||
5. Ensure all tests pass
|
||||
6. Commit with clear messages
|
||||
7. Push to branch (`git push origin feature/amazing-feature`)
|
||||
8. Open a Pull Request
|
||||
|
||||
### Pull Request Checklist
|
||||
|
||||
- [ ] Code follows style guidelines
|
||||
- [ ] Tests added/updated
|
||||
- [ ] Documentation updated
|
||||
- [ ] All tests passing
|
||||
- [ ] No new warnings
|
||||
- [ ] Commit messages are clear
|
||||
|
||||
---
|
||||
|
||||
**Happy Coding!** 🚀
|
||||
Reference in New Issue
Block a user