bookhoard/docs/contributing/development.md

Bookhoard Development Guide

This guide is for developers contributing to Bookhoard or setting up a development environment.

🏗 Architecture

Directory Structure

bookhoard/
├── cmd/server/                    # Application entry point
│   ├── main.go                   # Server initialization, route registration
│   └── tests/                    # Integration tests (30+ test files)
├── internal/
│   ├── config/                   # Configuration management
│   ├── database/                 # Database layer (SQLC generated)
│   ├── handlers/                 # HTTP request handlers (18 files)
│   ├── middleware/              # HTTP middleware (9 files)
│   ├── services/                # Business logic (7 files)
│   ├── sync/                    # Sync framework (5 files)
│   ├── opds/                    # OPDS feed generation
│   └── utils/                   # Utility functions
├── templates/                   # UI templates (17 .templ files)
├── web/src/                     # Frontend TypeScript
├── database/schema/             # Database schema
├── docs/                        # Documentation
└── bruno/                       # API test collections

Backend Components

Handlers (internal/handlers/):

• auth.go - Authentication & user management
• library.go - Library CRUD operations
• scanner.go - Media scanning operations
• media.go - Media downloads, shelves
• koreader.go - KOReader sync protocol
• kobo.go - Kobo sync protocol
• collections.go - Collection management
• devices.go - Device registration/management
• conflicts.go - Sync conflict resolution
• queue.go - Sync queue management
• progress.go - Reading progress tracking
• analytics.go - Usage analytics
• opds.go - OPDS feed generation
• websocket.go - WebSocket connections
• sync.go - Sync orchestration
• book_matching.go - Book linking/matching
• sidecar.go - Sidecar file handling
• refresh_token.go - Token refresh logic
• context.go - Handler context utilities

Middleware (internal/middleware/):

• device_auth.go - Device authentication
• device_rate_limiter.go - Device-specific rate limiting
• error_handler.go - Global error handling
• login_attempts.go - Login attempt tracking
• password_validator.go - Password complexity validation
• rate_limiter.go - IP-based rate limiting
• request_tracing.go - Request ID tracking
• security.go - Security headers
• transaction.go - Database transaction middleware

Services (internal/services/):

• library_service.go - Library operations
• media_scanner.go - File scanning, metadata extraction, and real-time file watching
• worker.go - Job queue worker pool
• collection_service.go - Collection rules processing
• conversion_service.go - EPUB→KEPUB conversion
• book_matching.go - Book matching algorithms

Sync Framework (internal/sync/):

• queue.go - Sync queue processor
• progress.go - Universal progress format
• websocket.go - Real-time sync broadcast
• offline.go - Offline sync support
• format.go - Format group conversion

Database Schema

Core Tables:

• users - User accounts with authentication and settings
• libraries - Library definitions
• library_types - Media type definitions (ebooks, comics, manga)
• library_folders - Multiple folders per library
• library_visibility - User-specific library access control
• media_items - Universal media storage (replaces ebooks table)
• media_ratings - User ratings (1-10 scale for half-star precision)
• media_notes - User annotations
• media_highlights - User highlights with color customization
• reading_progress - Universal progress tracking across devices
• devices - Device registry for sync
• sync_queue - Offline sync support
• sync_conflicts - Conflict resolution tracking
• collections - Device-neutral collections
• collection_items - Books in collections
• device_shelf_mappings - Map collections to device-specific shelves
• device_catalogs - Track OPDS downloads and ContentId mappings
• kobo_shelves - Kobo-specific shelf management
• reading_history - Reading session tracking
• unlinked_books - Track books that couldn't be auto-matched
• media_item_formats - Track all format versions with hashes
• device_file_aliases - Track file paths per device
• refresh_tokens - JWT refresh token storage

Database Functions:

• normalize_isbn() - ISBN format normalization
• detect_format_group() - Detect format group (reflowable, fixed_layout, comic_archive)
• convert_progress() - Convert progress between format groups
• detect_conflict() - Detect sync conflicts
• merge_progress() - Merge progress from multiple sources

Technology Stack

Backend:

• Go 1.25+
• Echo v4 - HTTP framework
• pgx v5 - PostgreSQL driver
• SQLC - SQL code generation
• jwt-go - JWT authentication
• bcrypt - Password hashing

Frontend:

• Templ - HTML templating with Go
• HTMX - Dynamic interactions
• Tailwind CSS - Styling
• TypeScript - Frontend logic

Database:

• PostgreSQL 15+
• 30+ tables
• 50+ indexes
• JSONB for complex data

Testing:

• Testify - Testing framework
• Bruno - API testing
• 30+ integration test files

🚀 Local Development

Prerequisites

• Go 1.25+
• Node.js 18+ (for frontend build)
• Podman or Docker
• PostgreSQL 15+ (or use Podman)

Setup

# 1. Clone repository
git clone https://github.com/yourusername/bookhoard.git
cd bookhoard

# 2. Install Go dependencies
go mod download

# 3. Install build tools
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
go install github.com/a-h/templ/cmd/templ@latest

# 4. Set up environment
cp .env.example .env
# Edit .env with your settings

# 5. Generate database code
cd internal/database
sqlc generate

# 6. Generate templates
cd ../../templates
templ generate

# 7. Build frontend
cd ../web
npm install
npm run build

# 8. Run tests
cd ..
go test ./... -v

Running Locally

# Option 1: Using containers (recommended)
podman-compose up --build

# Option 2: Direct Go run (requires local PostgreSQL)
# Generate secure passwords (no special characters):
#   JWT_SECRET: openssl rand -hex 32
#   DBPASS: openssl rand -hex 16
export JWT_SECRET="your-dev-secret"
export DBPASS="your-db-password"
go run cmd/server/main.go

Development Workflow

Backend Development:

# Watch mode for Go (requires air or similar)
air

# Or manual rebuild
go build -o bookhoard cmd/server/main.go
./bookhoard

Frontend Development:

cd web
npm run dev  # Watch mode for TypeScript/CSS

Database Changes:

1. Edit database/schema/schema.sql
2. Edit internal/database/queries/queries.sql
3. Run: cd internal/database && sqlc generate
4. Restart server

Template Changes:

1. Edit templates/*.templ
2. Run: cd templates && templ generate
3. Restart server (templates auto-reload in dev mode)

🧪 Testing

Unit Tests

# Run all unit tests
go test ./... -v

# Run specific package tests
go test ./internal/handlers/... -v

# Run with coverage
go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out

Integration Tests

# Run integration tests (may hit rate limits)
go test ./cmd/server/tests -v

# Run with test mode (recommended)
TEST_MODE=true RATE_LIMIT_ENABLED=false go test ./cmd/server/tests -v

# Run specific test
go test ./cmd/server/tests -run TestAuth -v

API Testing with Bruno OpenCollection YAML

# Install Bruno CLI
npm install -g @usebruno/cli

# Run all tests
bruno run

# Run specific collection
bruno run bruno/user/
bruno run bruno/sync-kobo/

Test Configuration

Environment variables for testing:

• TEST_MODE=true - Enable test mode (disables rate limiting)
• RATE_LIMIT_ENABLED=false - Disable rate limiting
• REQUESTS_PER_MINUTE=1000 - Increase rate limit

⚠️ WARNING: Never enable these in production!

Test Library Naming Convention

Integration tests automatically clean up libraries with "test" in the name (case-insensitive).

⚠️ IMPORTANT: Do not use "test" in library names if you want to keep them!

• Libraries containing "test" (e.g., "My Test Library", "Test Library 1") will be deleted by test cleanup
• Use names like "Development Library", "Staging Books", or "Personal" for libraries you want to keep
• This ensures your manual test data persists between test runs

📝 Code Style Guidelines

Go Code

• Follow Effective Go guidelines
• Use gofmt for formatting
• Use golangci-lint for linting
• Use meaningful variable and function names
• Add comments for complex business logic
• Handle errors explicitly - don't ignore them

Database Operations

• Always use sqlc-generated code - No raw SQL in handlers
• Use transactions for multi-step operations
• Handle pgx.ErrNoRows explicitly
• Use pgtype.UUID for UUID parameters
• Validate inputs before database operations

Error Handling

// Good - Explicit error handling
user, err := h.db.GetUser(c.Request().Context(), userID)
if err != nil {
    if errors.Is(err, pgx.ErrNoRows) {
        return c.JSON(http.StatusNotFound, map[string]string{"error": "user not found"})
    }
    return c.JSON(http.StatusInternalServerError, map[string]string{"error": "database error"})
}

// Bad - Ignoring errors
user, _ := h.db.GetUser(c.Request().Context(), userID)

Adding New Features

1. Database First: Add tables/columns to schema.sql
2. Generate Queries: Add to queries.sql and run sqlc generate
3. Handler: Implement in internal/handlers/
4. Routes: Register in cmd/server/main.go
5. Tests: Add integration test in cmd/server/tests/
6. Bruno: Add API test in bruno/
7. Docs: Update relevant documentation

API Design Principles

• RESTful naming conventions
• Consistent error responses
• Proper HTTP status codes
• JWT authentication on protected routes
• Input validation with struct tags
• Use echo.Context for request/response

🐳 Deployment

Building for Production

# Using Makefile
make build-force

# Or manually
podman-compose build --no-cache

Environment Variables

Required for production:

• JWT_SECRET - 64-byte random string (generate: openssl rand -hex 32)
• DBPASS - Strong database password (generate: openssl rand -hex 16)
• BASE_URL - Public URL (e.g., https://bookhoard.example.com)

Optional:

• HTTPS_PROXY - If behind reverse proxy

Note: Conversion service, rate limiting, and other operational settings have defaults in docker-compose.yml and can be overridden via .env if needed.

Performance Tuning

PostgreSQL Settings:

-- In postgresql.conf
shared_buffers = 256MB
effective_cache_size = 1GB
maintenance_work_mem = 64MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 2621kB
min_wal_size = 1GB
max_wal_size = 4GB

Go Settings:

• GOMAXPROCS = number of CPU cores
• Worker pool concurrency: 3 (configurable in services/worker.go)

🔍 Debugging

Enable Debug Logging

# Set environment variable
export DEBUG=true

# Or in .env
DEBUG=true

Common Issues

Database Connection Errors:

• Check PostgreSQL is running
• Verify DATABASE_HOST and DATABASE_PORT
• Check firewall settings

Rate Limiting During Development:

• Enable test mode: TEST_MODE=true RATE_LIMIT_ENABLED=false
• Or increase limit: REQUESTS_PER_MINUTE=1000

Template Not Updating:

• Run templ generate in templates/ directory
• Restart server

Database Queries Not Working:

• Run sqlc generate in internal/database/
• Check generated code in queries.sql.go
• Verify SQL syntax in queries.sql

📚 Additional Resources

• Project Guidelines - Development rules and standards
• API Reference - Complete API documentation
• Troubleshooting - Deployment issues
• Go Documentation
• Echo Framework
• pgx Documentation

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Follow code style guidelines
4. Add tests for new features
5. Ensure all tests pass
6. Commit with clear messages
7. Push to branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Pull Request Checklist

• Code follows style guidelines
• Tests added/updated
• Documentation updated
• All tests passing
• No new warnings
• Commit messages are clear

---

Happy Coding! 🚀