bookhoard/README.md

# 📚 Bookmann

A modern self-hosted media library system built with Go, PostgreSQL, HTMX, and Tailwind CSS featuring multiple library support, beautiful dark themes, and comprehensive media management.

## ✨ Features

### 🏗 Multi-Library System
- **Multiple Media Types**: Support for Ebooks, Comics, and Manga with modular architecture
- **Per-Library Folders**: Each library can have multiple scanning folders for flexible organization
- **Library Visibility Control**: Admins can control which libraries are visible to each user
- **Type-Specific File Extensions**: 
  - **Ebooks**: `.epub`, `.pdf`, `.mobi`, `.azw`, `.azw3`, `.txt`, `.rtf`, `.doc`, `.docx`, `.lit`, `.fb2`, `.pdb`
  - **Comics**: `.cbz`, `.cbr`, `.cb7`, `.cbt`, `.pdf`
  - **Manga**: `.cbz`, `.cbr`, `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.webp`
- **Easy Extension**: Designed to add new media types in the future

### 🔒 Authentication & Security
- **Multi-User Support**: Complete user registration and authentication system
- **JWT-Based Sessions**: Secure token-based authentication with 1-hour expiration and refresh token support
- **Role-Based Access**: Admin and user roles with granular permission control
- **Password Security**: bcrypt hashing with complex password requirements:
  - Minimum 8 characters
  - At least one uppercase letter (A-Z)
  - At least one lowercase letter (a-z)
  - At least one number (0-9)
  - At least one special character (!@#$%^&*()_+-=[]{}|;':\",./<>?)
- **Account Lockout**: Automatic account lockout after 5 failed login attempts (15-minute lockout period)
- **Rate Limiting**: Built-in rate limiting on auth endpoints (10 requests/minute) to prevent brute force attacks
- **Refresh Tokens**: Secure refresh token mechanism (7-day expiration) for seamless token renewal
- **Input Validation**: Comprehensive validation including username whitespace checks, email format validation
- **Pagination Protection**: Maximum pagination limits (1000 items) to prevent DoS attacks
- **Path Validation**: Library folder paths are validated for existence and accessibility
- **Case-Insensitive Roles**: Role values automatically normalized to lowercase
- **Database Transactions**: Multi-step database operations use transaction support for data consistency
- **Standardized Error Responses**: Consistent error format across all API endpoints

### 🎨 Beautiful UI
- **11 Dark Themes**: Tokyo Night, Dracula, Nord, Solarized Dark, Monokai, One Dark Pro, Material Dark, Catppuccin variants
- **Theme Persistence**: User theme preferences saved to database
- **Mobile-First Design**: Fully responsive interface for all devices
- **HTMX Integration**: Dynamic interactions without page reloads

### 🔄 Universal Cross-Platform Sync (NEW!)
- **Real-Time Progress Sync**: Reading progress syncs automatically across all your devices
- **Multi-Device Support**: KOReader, Kobo e-readers, web browsers, mobile apps
- **Format-Aware Tracking**: EPUB CFI, page numbers, percentages, character offsets
- **Highlights & Notes**: Annotations sync universally across all devices
- **Conflict Resolution**: Smart handling when same book read on multiple devices
- **Offline Support**: Queue changes when offline, sync when reconnected
- **Secure Device Auth**: Web-based device approval, no passwords on devices
- **Checkpoint Mode**: Batch sync for better battery life
- **WebSocket Updates**: Real-time broadcast of progress changes

 ### 📱 Media Management
 - **Universal Media Support**: Single system for all media types with unified interface
 - **Rich Metadata**: Automatic extraction of title, author, series, publisher, ISBN, tags
 - **ISBN Normalization**: Automatic ISBN format normalization (removes hyphens and spaces) supporting ISBN-10 and ISBN-13 formats
  - **Advanced Search**:
    - Partial matching search across title, author, series, tags, and contributors
    - Automatic fuzzy search fallback when no partial matches found (handles typos and misspellings)
    - Real-time search results with highlighted matches
    - Keyboard navigation (↑↓ arrows, Enter to select, Escape to close)
    - Respects library visibility settings
  - **Dynamic Sorting**: Sort your media collection by multiple fields
    - Title (A-Z or Z-A)
    - Author (A-Z or Z-A)
    - Date added (newest or oldest first)
    - Date published (newest or oldest first)
    - Copyright year (newest or oldest first)
    - Series order (with series number)
    - Page count (shortest or longest first)
    - Genre (A-Z or Z-A)
  - **Advanced Filtering**: Filter media items with multiple options
    - Filter by author (partial match)
    - Filter by series (partial match)
    - Filter by genre (exact match)
    - Filter by language (English, Spanish, French, German, Japanese, Chinese, Korean, Russian, Italian, Portuguese)
    - Filter by copyright year range
    - Filter by items with cover images only
    - Combine multiple filters with URL state management for shareable links
  - **Enhanced Metadata**: New fields for better organization
    - Language support for multi-lingual collections
    - Edition information (2nd Edition, Revised, Collector's Edition, etc.)
    - Page count for better sorting and progress calculation
    - External service integration (Goodreads, OpenLibrary, Google Books IDs)
    - Copyright year (distinct from publication date)
    - Structured genre classification
    - Subject tags (array of subjects)
  - **Advanced Rating**: 5-star system with half-star precision (1-10 scale)
 - **Reading Progress**: User-specific progress tracking with current page and total pages
 - **Notes & Highlights**: Personal annotations and text highlighting with color customization
 - **Highlight Notes**: Link highlights to detailed notes for comprehensive annotations
 - **Library Statistics**: Media count and usage statistics per library
 - **Background Scanning**: Automatic background scanning with configurable frequency per user
 - **Watch Mode**: Real-time file system monitoring for instant library updates

## 🚀 Quick Start

### Prerequisites
- **Podman** and Podman Compose (recommended) or Docker
- PostgreSQL database (handled by Podman)

### Environment Setup
1. **Create .env file**:
```bash
cp .env.example
```

2. **Edit .env** with your secure values:
```bash
JWT_SECRET="your-secure-jwt-secret-key-here"
DBPASS="your-secure-database-password-here"
```

### Running the Application
```bash
# Using Podman (recommended)
podman-compose up --build

# Using Docker
docker-compose up --build

# Option 2: Direct environment variables
export JWT_SECRET="your-secure-jwt-secret-key-here"
export DBPASS="your-secure-database-password-here"
podman-compose up --build
```

Access the application at: **http://localhost:8765**

## 👤 User Management

### Role System
- **Admin Users**: Can create/manage libraries, add/delete media items, manage users
- **Regular Users**: Can view permitted libraries, rate items, track reading progress

### First Admin Setup
The first user who registers automatically becomes an admin. For additional admins:

```sql
-- Connect to database
docker exec -it bookmann_db psql -U postgres -d bookmann

-- Promote user to admin
UPDATE users SET role = 'admin' WHERE email = 'user@example.com';
```

## 🏛 Library Management

### Creating Libraries
1. **Admin Access**: Only administrators can create libraries
2. **Library Types**: Choose from Ebooks, Comics, or Manga
3. **Multi-Folder Support**: Add multiple scanning folders per library
4. **Folder Organization**: Organize your media collection across multiple paths

### Library Visibility Control
- **Admin Dashboard**: Complete interface for managing library access
- **User-Specific Control**: Admins can show/hide libraries per user
- **Personal Preferences**: Admins can also hide libraries from their own view
- **Simple Toggles**: Checkbox-based interface for easy management

### API Endpoints

#### Notes & Highlights (All Authenticated Users)
```bash
# Media Notes
GET /api/media-items/{id}/notes              # Get user's notes for media
POST /api/media-items/{id}/notes             # Create new note
GET /api/media-items/{id}/notes/{noteId}   # Get specific note
PUT /api/media-items/{id}/notes/{noteId}    # Update note
DELETE /api/media-items/{id}/notes/{noteId} # Delete note

# Media Highlights  
GET /api/media-items/{id}/highlights                   # Get user's highlights for media
POST /api/media-items/{id}/highlights                  # Create new highlight
GET /api/media-items/{id}/highlights/{highlightId}      # Get specific highlight
PUT /api/media-items/{id}/highlights/{highlightId}       # Update highlight
DELETE /api/media-items/{id}/highlights/{highlightId}   # Delete highlight

# Ebook Compatibility (Backward Compatible)
GET /api/ebooks/{id}/notes              # Get user's notes for ebook
POST /api/ebooks/{id}/notes             # Create new ebook note
PUT /api/ebooks/{id}/notes/{noteId}    # Update ebook note
DELETE /api/ebooks/{id}/notes/{noteId} # Delete ebook note

GET /api/ebooks/{id}/highlights                   # Get user's highlights for ebook
POST /api/ebooks/{id}/highlights                  # Create new ebook highlight
PUT /api/ebooks/{id}/highlights/{highlightId}       # Update ebook highlight
DELETE /api/ebooks/{id}/highlights/{highlightId}   # Delete ebook highlight
 ```

#### Media Search (All Authenticated Users)
```bash
# Search media items across all visible libraries
GET /api/media-items/search?q={query}

# Query Parameters:
# q (required): Search query (minimum 2 characters)
# - Searches across: title, author, series, tags, contributors
# - Partial matching: Case-insensitive substring search
# - Fuzzy fallback: Automatic when no partial matches found
# - Results ranked by relevance

# Examples:
GET /api/media-items/search?q=harry%20potter    # Search by title
GET /api/media-items/search?q=rowling          # Search by author
GET /api/media-items/search?q=hary%20poter     # Fuzzy search (handles typos)

# Response:
# - 200: Success (returns array of media items)
# - 404: No results found
# - 400: Missing or invalid query parameter
# - 401: Unauthorized
```

#### Library Management (Admin Only)
```bash
# Create new library
POST /api/libraries
{
  "name": "My Comic Collection",
  "description": "Digital comics and graphic novels",
  "type": "comics"
}

# List all libraries  
GET /api/libraries

# Add folder to library
POST /api/libraries/{library_id}/folders
{
  "folder_path": "/path/to/comics"
}

# Set library visibility for user
POST /api/libraries/visibility
{
  "library_id": "library-uuid",
  "is_visible": true
}

# Scanner Management (Admin Only)
POST /api/scanner/scan                  # Manual scan of all libraries
POST /api/scanner/start                 # Start scheduled scanning
POST /api/scanner/stop                  # Stop scheduled scanning
GET /api/scanner/status/{jobId}        # Get scan job status
POST /api/scanner/watch/start          # Start watch mode (real-time monitoring)
POST /api/scanner/watch/stop           # Stop watch mode
GET /api/scanner/watch/status          # Get watch mode status
```

#### User Access
```bash
# Get user's visible libraries
GET /api/libraries/visible

# Browse media in library
GET /api/media-items?library_id={library_id}&limit=20&offset=0

# Media Progress & Ratings
GET /api/media-items/{id}/progress     # Get reading progress
PUT /api/media-items/{id}/progress     # Update reading progress
GET /api/media-items/{id}/rating       # Get user rating
POST /api/media-items/{id}/rating      # Create/update rating
PUT /api/media-items/{id}/rating       # Update rating
DELETE /api/media-items/{id}/rating   # Delete rating

# Scan Settings (All Authenticated Users)
GET /api/library/scan-settings         # Get user's scan settings
PUT /api/library/scan-settings         # Update scan settings
{
  "scan_frequency_minutes": 60,
  "auto_scan_enabled": true
}

# Token Management
POST /api/auth/refresh                 # Refresh access token
POST /api/auth/logout                  # Logout (revokes refresh token)
```

## 📖 Media Support

### Ebooks
- **Formats**: EPUB (full metadata), PDF, MOBI, AZW, AZW3, TXT, DOC, DOCX, LIT, FB2, PDB, RTF
- **Metadata**: Automatic extraction from EPUB files with Calibre support
- **ISBN Support**: 
  - Automatic normalization of ISBN-10 and ISBN-13 formats
  - Removes hyphens and spaces for consistent storage
  - Supports ISBN-10 with check digit X
  - Database function `normalize_isbn()` handles all conversions
- **Reading**: Built-in web reader for EPUB files
- **Library Requirement**: Ebooks require an existing ebook library; graceful error handling if none exists

### Comics  
- **Formats**: CBZ, CBR, CB7, CBT, PDF
- **Structure**: Archive-based organization with chapter support
- **Viewing**: Image extraction and web-based comic reader

## ⚙️ Background Processing

### Auto-Scanning
- **Scheduled Scanning**: User-configurable scan frequency (per-user settings)
- **Smart Scheduling**: Background scheduler manages scan jobs efficiently
- **Per-User Settings**: Each user can enable/disable auto-scan and set frequency
- **Resource Management**: Worker pool limits concurrent scanning operations

### Watch Mode
- **Real-Time Monitoring**: File system watcher detects new files instantly
- **Automatic Processing**: New media items processed and added to library
- **Multi-Library Support**: Watch all library folders simultaneously
- **Efficient**: Uses filesystem events for minimal resource usage

### Scan Settings API
```bash
# Get current scan settings
GET /api/library/scan-settings

# Update scan settings
PUT /api/library/scan-settings
{
  "scan_frequency_minutes": 60,    # How often to auto-scan (minutes)
  "auto_scan_enabled": true        # Enable/disable auto-scanning
}
```

### Manga
- **Formats**: CBZ, CBR (archives), PNG, JPG (image folders)
- **Structure**: Archive support with folder-based image organization
- **Viewing**: Chapter-by-chapter viewing with page navigation

## 🎨 Development

### Local Development
```bash
# Backend development
go mod tidy
go run cmd/server/main.go

# Frontend development
npm run dev
# Templates automatically recompile on changes

# Run tests
go test ./... -v
go test ./cmd/server/tests -v  # Integration tests only
```

### Environment
- **Backend**: Go 1.25+ with pgx v5 for database operations
- **Frontend**: Tailwind CSS with HTMX for dynamic interactions  
- **Database**: PostgreSQL 15+ with pgx v5 driver
- **Authentication**: JWT tokens with bcrypt password hashing

## 🐳 Deployment

For detailed deployment instructions, Docker configuration, and troubleshooting common issues, see **[TROUBLESHOOTING.md](TROUBLESHOOTING.md)**.

The application automatically starts background services on startup:
- **Scheduler**: Manages scheduled scanning jobs based on user settings
- **Watch Mode**: Monitors all library folders for file system changes
- **Worker Pool**: Processes scan jobs with configurable concurrency

### Database Schema
- **Multi-Library Architecture**: Libraries, library types, folders, visibility tables
- **Annotations System**: Notes, highlights with position tracking and color customization
- **Backward Compatibility**: Views maintain existing API contracts for ebooks
- **Type Safety**: pgx v5 with proper error handling
- **Migration Ready**: Schema designed for easy future extensions

## 🧪 API Documentation

### Bruno Testing Collection
Complete API testing collection in `bruno/` directory:

```
bruno/
├── user/              # Authentication & profile endpoints
├── admin/          # Admin-only operations  
├── library/        # Library management
├── media-items/     # Media content browsing
├── ebooks/         # Ebook-specific operations
├── notes/          # Notes API testing
├── highlights/     # Highlights API testing
├── scanner/        # Background scanning & watch mode
├── progress/       # Reading progress tracking
├── devices/        # Device management
├── conflicts/       # Sync conflict resolution
├── queue/          # Sync queue management
├── sync-koreader/ # KOReader sync protocol
├── sync-kobo/       # Kobo sync protocol
└── collection.bru   # Main dashboard file
```

### Authentication Flow
1. **Register**: `POST /api/auth/register` → JWT token
2. **Login**: `POST /api/auth/login` → JWT token
3. **Protected Routes**: Use `Authorization: Bearer {token}` header
4. **Refresh Token**: `POST /api/auth/refresh` → JWT token
5. **Logout**: `POST /api/auth/logout` → Revoke refresh token

### API Reference Documentation
See [API_REFERENCE.md](API_REFERENCE.md) for complete API documentation including:
- All endpoints with request/response examples
- Bruno v3.0 test collections for all endpoints
- Error handling details
- Authentication & security features

### Device Setup Guides
- [KOBOREADER_SETUP.md](KOBOREADER_SETUP.md) - KOReader device configuration
- [KOBO_SETUP.md](KOBO_SETUP.md) - Kobo device configuration

### Universal Sync Documentation
- [UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md](UNIVERSAL_SYNC_IMPLEMENTATION_GUIDE.md) - Complete sync architecture

### Error Handling
All API endpoints return standardized error responses:
```json
{
  "error": "Error message",
  "message": "Detailed error information (if available)",
  "code": "Error code (if applicable)"
}
```

HTTP Status Codes:
- **400**: Bad request (validation errors)
- **401**: Unauthorized (invalid/missing token)
- **403**: Forbidden (insufficient permissions)
- **404**: Resource not found
- **429**: Too many requests (rate limit exceeded)
- **500**: Internal server error

## 🛡 Security Features

### Authentication & Authorization
- **JWT Tokens**: 
  - Short-lived access tokens (1-hour expiration)
  - Refresh tokens (7-day expiration) for seamless session renewal
  - Secure token storage and transmission
- **Password Security**:
  - bcrypt hashing with cost factor 12
  - Complex password requirements enforced
  - Password validation on registration and updates
- **Account Protection**:
  - Automatic lockout after 5 failed login attempts
  - 15-minute lockout period with countdown
  - IP-based and username-based attempt tracking
- **Rate Limiting**: 10 requests per minute on authentication endpoints
- **Input Validation**: Comprehensive server-side validation for all inputs
- **CSRF Protection**: Built-in with HTMX
- **Role-Based Access Control**: Admin and user roles enforced on all endpoints

### Authorization
- **Role-Based Access**: Admin vs user permissions
- **Library Visibility**: Per-user library access control
- **Admin Middleware**: Protected routes for admin operations
- **Content Security**: XSS protection and secure headers

### Database Security
- **Parameterized Queries**: SQL injection prevention with pgx v5
- **Connection Pooling**: Efficient database connection management
- **Transaction Support**: Multi-step operations wrapped in transactions
- **Environment Variables**: Secure secret management
- **Row-Level Security**: User data isolation
- **Secure Refresh Token Storage**: Encrypted token storage in database
- **Automatic Token Cleanup**: Expired tokens cleaned up periodically
- **ISBN Normalization**: Database function ensures consistent ISBN format
- **Data Validation**: Server-side validation for all inputs including ISBNs

## 🔧 Configuration

### Environment Variables
```bash
# Required
JWT_SECRET=           # JWT signing secret (64-byte random string)
DBHOST=db            # PostgreSQL database host

# Optional  
SERVER_PORT=8765      # Application port
NODE_ENV=development   # Environment mode
```

### Database Configuration
```sql
-- Library types are automatically seeded
INSERT INTO library_types (name, description, allowed_extensions) VALUES
('ebooks', 'Ebook files including EPUB, PDF, MOBI, etc.', 
 ARRAY['.epub', '.pdf', '.mobi', '.azw', '.azw3', '.txt', '.rtf', '.doc', '.docx', '.lit', '.fb2', '.pdb']),
('comics', 'Comic book archives and image formats', 
 ARRAY['.cbz', '.cbr', '.cb7', '.cbt', '.pdf']),
('manga', 'Manga files including archives and image folders', 
 ARRAY['.cbz', '.cbr', '.png', '.jpg', '.jpeg', '.gif', '.bmp', '.webp']);
```

## 🧪 Testing

### Integration Tests
Comprehensive integration test suite in `cmd/server/tests/`:
- Authentication & authorization tests
- Library management tests
- Media items operations
- Notes & highlights functionality
- ISBN normalization tests
- Background scanning & watch mode tests
- Edge cases and error handling
- Security & rate limiting tests

Run integration tests:
```bash
# Standard test run (may hit rate limits)
go test ./cmd/server/tests -v

# Recommended: Run with test mode enabled to avoid rate limiting
TEST_MODE=true RATE_LIMIT_ENABLED=false go test ./cmd/server/tests -run TestIntegrationAPI -v

# Or with increased rate limits
TEST_MODE=true REQUESTS_PER_MINUTE=1000 go test ./cmd/server/tests -run TestIntegrationAPI -v
```

### Test Configuration Options
The following environment variables can be used to configure test behavior:

- **TEST_MODE**: Set to `true` to enable test mode (logs additional info)
- **RATE_LIMIT_ENABLED**: Set to `false` to disable rate limiting for tests
- **REQUESTS_PER_MINUTE**: Increase rate limit (e.g., `1000`) to avoid throttling

**⚠️ WARNING**: Never disable rate limiting or enable test mode in production environments. These settings are only for integration testing.

### API Testing
```bash
# Install Bruno
npm install -g @usebruno/cli

# Run tests
bruno run
```

### Unit Tests
```bash
go test ./...
```

## 📊 Architecture

### Backend
```
cmd/server/main.go              # Application entry point
├── internal/
│   ├── config/             # Configuration management
│   ├── database/           # Database operations (SQLC generated)
│   │   ├── queries/     # SQL queries
│   │   └── models.go    # Generated models
│   ├── handlers/           # HTTP handlers
│   │   ├── auth.go     # Authentication & users
│   │   ├── library.go  # Library management
│   │   ├── ebook.go    # Media operations
│   │   └── refresh_token.go # Token management
│   ├── services/          # Business logic
│   │   ├── library_service.go  # Library service
│   │   ├── ebook_scanner.go    # Media scanning
│   │   ├── worker.go           # Background job processing
│   │   └── scheduler.go        # Auto-scan scheduling
│   └── middleware/         # HTTP middleware
│       ├── rate_limiter.go     # Rate limiting
│       ├── login_attempts.go   # Account lockout
│       ├── password_validator.go # Password complexity
│       ├── error_handler.go    # Error handling
│       ├── transaction.go      # DB transactions
│       └── request_tracing.go  # Request logging
templates/                   # HTML templates with HTMX
```

### Database
```sql
libraries              # Library definitions
library_types          # Media type definitions (ebooks, comics, manga)
library_folders        # Folders per library
library_visibility     # User library access control
media_items           # All media content (replaces ebooks table)
media_ratings         # User ratings for media items
media_notes           # User notes on media items
media_highlights      # User highlights with optional note links
reading_progress      # User reading progress
users                 # User accounts and profiles (includes scan settings)
refresh_tokens        # JWT refresh token storage

# Database Functions
normalize_isbn()      # ISBN normalization function

# Backward Compatibility Views
ebook_notes           # Notes view for ebook API compatibility
ebook_highlights      # Highlights view for ebook API compatibility
```

## 🎯 Future Roadmap

### Recently Added Features (v1.0)
- ✅ **ISBN Normalization**: Automatic ISBN format standardization
- ✅ **Background Scanning**: Scheduled library scanning with per-user settings
- ✅ **Watch Mode**: Real-time file system monitoring for instant updates
- ✅ **Integration Tests**: Comprehensive test suite for all major features
- ✅ **Request Tracing**: Enhanced logging and debugging capabilities
- ✅ **Graceful Library Handling**: Better error messages when libraries don't exist

### Media Type Extensions
- **Audiobooks**: Audio file support with chapter tracking
- **Podcasts**: RSS feed integration and automatic downloading
- **Video**: Movie/TV series management with metadata
- **Music**: Album and track management with artwork

### Advanced Features
- **Mobile App**: React Native mobile application
- **API v2**: GraphQL API for efficient data fetching
- **Webhooks**: External service integrations
- **Analytics**: Usage statistics and reporting
- **Backup/Restore**: Library backup and migration tools

## 📝 Contributing

### Development Guidelines
- Follow Go best practices and effective Go
- Use pgx v5 for all database operations
- Implement proper error handling with pgx.ErrNoRows
- Write comprehensive tests for new features
- Update documentation for API changes
- Add Bruno tests for new API endpoints
- Include integration tests for complex features

### Code Style
- Follow existing code formatting
- Use meaningful variable and function names
- Add comments for complex business logic
- Ensure type safety with proper error handling

## 📄 License

GPL-3.0 - See [LICENSE](LICENSE) file for details.

---

**Built with ❤️ using Go, PostgreSQL, HTMX, and Tailwind CSS**