bookhoard/docs/contributing/development.md

Bookhoard Development Guide

This guide is for developers contributing to Bookhoard or setting up a development environment.

🏗 Architecture

Directory Structure

bookhoard/
├── cmd/server/                    # Application entry point
│   ├── main.go                   # Server initialization, route registration
│   └── tests/                    # Integration tests (30+ test files)
├── internal/
│   ├── config/                   # Configuration management
│   ├── database/                 # Database layer (SQLC generated)
│   ├── handlers/                 # HTTP request handlers (18 files)
│   ├── middleware/              # HTTP middleware (9 files)
│   ├── services/                # Business logic (7 files)
│   ├── sync/                    # Sync framework (5 files)
│   ├── opds/                    # OPDS feed generation
│   └── utils/                   # Utility functions
├── templates/                   # UI templates (17 .templ files)
├── web/src/                     # Frontend TypeScript
├── database/schema/             # Database schema
├── docs/                        # Documentation
└── bruno/                       # API test collections

Backend Components

Handlers (internal/handlers/):

• auth.go - Authentication & user management
• library.go - Library CRUD operations
• scanner.go - Media scanning operations
• media.go - Media downloads, shelves
• koreader.go - KOReader sync protocol
• kobo.go - Kobo sync protocol
• collections.go - Collection management
• devices.go - Device registration/management
• conflicts.go - Sync conflict resolution
• queue.go - Sync queue management
• progress.go - Reading progress tracking
• analytics.go - Usage analytics
• opds.go - OPDS feed generation
• websocket.go - WebSocket connections
• sync.go - Sync orchestration
• book_matching.go - Book linking/matching
• sidecar.go - Sidecar file handling
• refresh_token.go - Token refresh logic
• context.go - Handler context utilities

Middleware (internal/middleware/):

• device_auth.go - Device authentication
• device_rate_limiter.go - Device-specific rate limiting
• error_handler.go - Global error handling
• login_attempts.go - Login attempt tracking
• password_validator.go - Password complexity validation
• rate_limiter.go - IP-based rate limiting
• request_tracing.go - Request ID tracking
• security.go - Security headers
• transaction.go - Database transaction middleware

Services (internal/services/):

• library_service.go - Library operations
• media_scanner.go - File scanning, metadata extraction, and real-time file watching
• worker.go - Job queue worker pool
• collection_service.go - Collection rules processing
• conversion_service.go - EPUB→KEPUB conversion
• book_matching.go - Book matching algorithms

Sync Framework (internal/sync/):

• queue.go - Sync queue processor
• progress.go - Universal progress format
• websocket.go - Real-time sync broadcast
• offline.go - Offline sync support
• format.go - Format group conversion

Database Schema

Core Tables:

• users - User accounts with authentication and settings
• libraries - Library definitions
• library_types - Media type definitions (ebooks, comics, manga)
• library_folders - Multiple folders per library
• library_visibility - User-specific library access control
• media_items - Universal media storage (replaces ebooks table)
• media_ratings - User ratings (1-10 scale for half-star precision)
• media_notes - User annotations
• media_highlights - User highlights with color customization
• reading_progress - Universal progress tracking across devices
• devices - Device registry for sync
• sync_queue - Offline sync support
• sync_conflicts - Conflict resolution tracking
• collections - Device-neutral collections
• collection_items - Books in collections
• device_shelf_mappings - Map collections to device-specific shelves
• device_catalogs - Track OPDS downloads and ContentId mappings
• kobo_shelves - Kobo-specific shelf management
• reading_history - Reading session tracking
• unlinked_books - Track books that couldn't be auto-matched
• media_item_formats - Track all format versions with hashes
• device_file_aliases - Track file paths per device
• refresh_tokens - JWT refresh token storage

Database Functions:

• normalize_isbn() - ISBN format normalization
• detect_format_group() - Detect format group (reflowable, fixed_layout, comic_archive)
• convert_progress() - Convert progress between format groups
• detect_conflict() - Detect sync conflicts
• merge_progress() - Merge progress from multiple sources

Technology Stack

Backend:

• Go 1.25+
• Echo v4 - HTTP framework
• pgx v5 - PostgreSQL driver
• SQLC - SQL code generation
• jwt-go - JWT authentication
• bcrypt - Password hashing

Frontend:

• Templ - HTML templating with Go
• HTMX - Dynamic interactions
• Tailwind CSS - Styling
• TypeScript - Frontend logic

Database:

• PostgreSQL 15+
• 30+ tables
• 50+ indexes
• JSONB for complex data

Testing:

• Testify - Testing framework
• Bruno - API testing
• 30+ integration test files

🚀 Local Development

Prerequisites

• Go 1.25+
• Node.js 18+ (for frontend build)
• Podman or Docker
• PostgreSQL 15+ (or use Podman)

Setup

# 1. Clone repository
git clone https://github.com/yourusername/bookhoard.git
cd bookhoard

# 2. Install Go dependencies
go mod download

# 3. Install build tools
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
go install github.com/a-h/templ/cmd/templ@latest

# 4. Set up environment
cp .env.example .env
# Edit .env with your settings

# 5. Generate database code
cd internal/database
sqlc generate

# 6. Generate templates
cd ../../templates
templ generate

# 7. Build frontend
cd ../web
npm install
npm run build

# 8. Run tests
cd ..
go test ./... -v

Running Locally

# Option 1: Using containers (recommended)
podman-compose up --build

# Option 2: Direct Go run (requires local PostgreSQL)
# Generate secure passwords (no special characters):
#   JWT_SECRET: openssl rand -hex 32
#   DBPASS: openssl rand -hex 16
export JWT_SECRET="your-dev-secret"
export DBPASS="your-db-password"
go run cmd/server/main.go

Development Workflow

Backend Development:

# Watch mode for Go (requires air or similar)
air

# Or manual rebuild
go build -o bookhoard cmd/server/main.go
./bookhoard

Frontend Development:

cd web
npm run dev  # Watch mode for TypeScript/CSS

Database Changes:

1. Edit database/schema/schema.sql
2. Edit internal/database/queries/queries.sql
3. Run: cd internal/database && sqlc generate
4. Restart server

Template Changes:

1. Edit templates/*.templ
2. Run: cd templates && templ generate
3. Restart server (templates auto-reload in dev mode)

🧪 Testing

Unit Tests

# Run all unit tests
go test ./... -v

# Run specific package tests
go test ./internal/handlers/... -v

# Run with coverage
go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out

Integration Tests

Integration tests hit the LIVE dev database and bind-mounted paths, both of which exist only inside the Docker stack. A bare host go test ./cmd/server/tests fails with database-auth errors on every machine that isn't running the compose stack — the container is the only supported runner.

# Run the whole integration suite (pre-commit validation — NOT for
# iterating on a change; some tests are coupled to the shared dev DB's
# live state and this environment)
make test-integration

# Iterate on YOUR change: run only your own tests, in the container
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile tests run --rm tests \
  go test ./cmd/server/tests -run 'TestYourPrefix' -v -timeout 600s

Run only your own tests while iterating. A full-suite run rebuilds the test image and takes minutes; more importantly, several tests read live state (the dev DB's system_settings, real library folders, a periodic backup scan broadcasting over websocket) and can fail on timing or on data your change never touched. Filtered runs are cheap, deterministic for your scope, and don't fight the live reader for the database.

Test-created data rules (harness cleanup depends on them): users use @tests.bookhoard.internal / @example.com emails, libraries and collections carry "test"-prefixed names, and cleanups delete test libraries (cascades handle the rest) rather than table-wide deletes.

Media Fixtures (audio/books)

Never commit binary media. The Go toolchain and Docker prod builds ignore internal/services/testdata/ entirely (gitignored by policy — see .gitignore), and the production image never copies internal/. Generate what a test needs at runtime with ffmpeg (shipped in both the test-runner and runtime images):

# The five audio containers (audiobook library accepts all of these)
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a libmp3lame -b:a 32k plain.mp3
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a flac       book.flac
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a libopus  -b:a 24k book.opus
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a libvorbis -q:a 1 book.ogg
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a aac      -b:a 32k book.aac
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a pcm_s16le      book.wav

# Tags (album=book title, album_artist=author, composer=narrator —
# Audible's own convention — plus disc/track ordering keys)
ffmpeg -i in.mp3 -codec copy \
  -metadata title="Chapter Three" -metadata album="The Book" \
  -metadata album_artist="Author" -metadata composer="Narrator" \
  -metadata track="3/9" -metadata disc="2/3" \
  -metadata grouping="Series" -metadata genre="Audiobook" -metadata date=2020 out.mp3

# Embedded cover art (mp3 APIC / m4b covr): the picture rides as the
# only video stream
ffmpeg -i cover.jpg -i audio-only.mp3 -map 1:a -map 0:v -c:v copy \
  -disposition:v attached_pic tagged.mp3

# Embedded m4b chapters: an ffmetadata sidecar file
printf ';FFMETADATA1\n[CHAPTER]\nTIMEBASE=1/1000\nSTART=0\nEND=500\ntitle=Part One\n' > chapters.txt
ffmpeg -i in.m4b -i chapters.txt -map 0:a -map 1 -map_metadata 1 -codec copy -f ipod chaptered.m4b

CUE sheets and m3u playlists are plain text — write them inline in the test. The one thing ffmpeg cannot synthesize: Audible freeform atoms (----:com.apple.iTunes:SERIES, :SUBTITLE, :ASIN, :ABRIDGED) — those ingestion paths are covered by unit tests using synthetic ffprobeReport values instead of binary files.

Manual end-to-end testing against the live system: drop fixture books into uploads/Audiobooks/Ingestion Fixtures/ — one book per subfolder (a directory with ≥2 audio files ingests as ONE folder-of-tracks book) — and run a library scan from the admin UI.

API Testing with Bruno OpenCollection YAML

# Install Bruno CLI
npm install -g @usebruno/cli

# Run all tests
bruno run

# Run specific collection
bruno run bruno/user/
bruno run bruno/sync-kobo/

Test Configuration

Environment variables for testing:

• TEST_MODE=true - Enable test mode (disables rate limiting)
• RATE_LIMIT_ENABLED=false - Disable rate limiting
• REQUESTS_PER_MINUTE=1000 - Increase rate limit

⚠️ WARNING: Never enable these in production!

Test Library Naming Convention

Integration tests automatically clean up libraries with "test" in the name (case-insensitive).

⚠️ IMPORTANT: Do not use "test" in library names if you want to keep them!

• Libraries containing "test" (e.g., "My Test Library", "Test Library 1") will be deleted by test cleanup
• Use names like "Development Library", "Staging Books", or "Personal" for libraries you want to keep
• This ensures your manual test data persists between test runs

📝 Code Style Guidelines

Go Code

• Follow Effective Go guidelines
• Use gofmt for formatting
• Use golangci-lint for linting
• Use meaningful variable and function names
• Add comments for complex business logic
• Handle errors explicitly - don't ignore them

Database Operations

• Always use sqlc-generated code - No raw SQL in handlers
• Use transactions for multi-step operations
• Handle pgx.ErrNoRows explicitly
• Use pgtype.UUID for UUID parameters
• Validate inputs before database operations

Error Handling

// Good - Explicit error handling
user, err := h.db.GetUser(c.Request().Context(), userID)
if err != nil {
    if errors.Is(err, pgx.ErrNoRows) {
        return c.JSON(http.StatusNotFound, map[string]string{"error": "user not found"})
    }
    return c.JSON(http.StatusInternalServerError, map[string]string{"error": "database error"})
}

// Bad - Ignoring errors
user, _ := h.db.GetUser(c.Request().Context(), userID)

Adding New Features

1. Database First: Add tables/columns to schema.sql
2. Generate Queries: Add to queries.sql and run sqlc generate
3. Handler: Implement in internal/handlers/
4. Routes: Register in cmd/server/main.go
5. Tests: Add integration test in cmd/server/tests/
6. Bruno: Add API test in bruno/
7. Docs: Update relevant documentation

API Design Principles

• RESTful naming conventions
• Consistent error responses
• Proper HTTP status codes
• JWT authentication on protected routes
• Input validation with struct tags
• Use echo.Context for request/response

🐳 Deployment

Building for Production

# Using Makefile
make build-force

# Or manually
podman-compose build --no-cache

Environment Variables

Required for production:

• JWT_SECRET - 64-byte random string (generate: openssl rand -hex 32)
• DBPASS - Strong database password (generate: openssl rand -hex 16)
• BASE_URL - Public URL (e.g., https://bookhoard.example.com)

Optional:

• HTTPS_PROXY - If behind reverse proxy

Note: Conversion service, rate limiting, and other operational settings have defaults in docker-compose.yml and can be overridden via .env if needed.

Performance Tuning

PostgreSQL Settings:

-- In postgresql.conf
shared_buffers = 256MB
effective_cache_size = 1GB
maintenance_work_mem = 64MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 2621kB
min_wal_size = 1GB
max_wal_size = 4GB

Go Settings:

• GOMAXPROCS = number of CPU cores
• Worker pool concurrency: 3 (configurable in services/worker.go)

🔍 Debugging

Enable Debug Logging

# Set environment variable
export DEBUG=true

# Or in .env
DEBUG=true

Common Issues

Database Connection Errors:

• Check PostgreSQL is running
• Verify DATABASE_HOST and DATABASE_PORT
• Check firewall settings

Rate Limiting During Development:

• Enable test mode: TEST_MODE=true RATE_LIMIT_ENABLED=false
• Or increase limit: REQUESTS_PER_MINUTE=1000

Template Not Updating:

• Run templ generate in templates/ directory
• Restart server

Database Queries Not Working:

• Run sqlc generate in internal/database/
• Check generated code in queries.sql.go
• Verify SQL syntax in queries.sql

📚 Additional Resources

• AGENTS.md - Agent/development rules
• API Reference - Complete API documentation
• Troubleshooting - Deployment issues
• Go Documentation
• Echo Framework
• pgx Documentation

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Follow code style guidelines
4. Add tests for new features
5. Ensure all tests pass
6. Commit with clear messages
7. Push to branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Pull Request Checklist

• Code follows style guidelines
• Tests added/updated
• Documentation updated
• All tests passing
• No new warnings
• Commit messages are clear

---

Happy Coding! 🚀