bookhoard/docs/contributing/development.md

# Bookhoard Development Guide

This guide is for developers contributing to Bookhoard or setting up a development environment.

## 🏗 Architecture

### Directory Structure

```
bookhoard/
├── cmd/server/                    # Application entry point
│   ├── main.go                   # Server initialization, route registration
│   └── tests/                    # Integration tests (30+ test files)
├── internal/
│   ├── config/                   # Configuration management
│   ├── database/                 # Database layer (SQLC generated)
│   ├── handlers/                 # HTTP request handlers (18 files)
│   ├── middleware/              # HTTP middleware (9 files)
│   ├── services/                # Business logic (7 files)
│   ├── sync/                    # Sync framework (5 files)
│   ├── opds/                    # OPDS feed generation
│   └── utils/                   # Utility functions
├── templates/                   # UI templates (17 .templ files)
├── web/src/                     # Frontend TypeScript
├── database/schema/             # Database schema
├── docs/                        # Documentation
└── bruno/                       # API test collections
```

### Backend Components

**Handlers** (`internal/handlers/`):

- `auth.go` - Authentication & user management
- `library.go` - Library CRUD operations
- `scanner.go` - Media scanning operations
- `media.go` - Media downloads, shelves
- `koreader.go` - KOReader sync protocol
- `kobo.go` - Kobo sync protocol
- `collections.go` - Collection management
- `devices.go` - Device registration/management
- `conflicts.go` - Sync conflict resolution
- `queue.go` - Sync queue management
- `progress.go` - Reading progress tracking
- `analytics.go` - Usage analytics
- `opds.go` - OPDS feed generation
- `websocket.go` - WebSocket connections
- `sync.go` - Sync orchestration
- `book_matching.go` - Book linking/matching
- `sidecar.go` - Sidecar file handling
- `refresh_token.go` - Token refresh logic
- `context.go` - Handler context utilities

**Middleware** (`internal/middleware/`):

- `device_auth.go` - Device authentication
- `device_rate_limiter.go` - Device-specific rate limiting
- `error_handler.go` - Global error handling
- `login_attempts.go` - Login attempt tracking
- `password_validator.go` - Password complexity validation
- `rate_limiter.go` - IP-based rate limiting
- `request_tracing.go` - Request ID tracking
- `security.go` - Security headers
- `transaction.go` - Database transaction middleware

**Services** (`internal/services/`):

- `library_service.go` - Library operations
- `media_scanner.go` - File scanning, metadata extraction, and real-time file watching
- `worker.go` - Job queue worker pool
- `collection_service.go` - Collection rules processing
- `conversion_service.go` - EPUB→KEPUB conversion
- `book_matching.go` - Book matching algorithms

**Sync Framework** (`internal/sync/`):

- `queue.go` - Sync queue processor
- `progress.go` - Universal progress format
- `websocket.go` - Real-time sync broadcast
- `offline.go` - Offline sync support
- `format.go` - Format group conversion

### Database Schema

**Core Tables**:

- `users` - User accounts with authentication and settings
- `libraries` - Library definitions
- `library_types` - Media type definitions (ebooks, comics, manga)
- `library_folders` - Multiple folders per library
- `library_visibility` - User-specific library access control
- `media_items` - Universal media storage (replaces ebooks table)
- `media_ratings` - User ratings (1-10 scale for half-star precision)
- `media_notes` - User annotations
- `media_highlights` - User highlights with color customization
- `reading_progress` - Universal progress tracking across devices
- `devices` - Device registry for sync
- `sync_queue` - Offline sync support
- `sync_conflicts` - Conflict resolution tracking
- `collections` - Device-neutral collections
- `collection_items` - Books in collections
- `device_shelf_mappings` - Map collections to device-specific shelves
- `device_catalogs` - Track OPDS downloads and ContentId mappings
- `kobo_shelves` - Kobo-specific shelf management
- `reading_history` - Reading session tracking
- `unlinked_books` - Track books that couldn't be auto-matched
- `media_item_formats` - Track all format versions with hashes
- `device_file_aliases` - Track file paths per device
- `refresh_tokens` - JWT refresh token storage

**Database Functions**:

- `normalize_isbn()` - ISBN format normalization
- `detect_format_group()` - Detect format group (reflowable, fixed_layout, comic_archive)
- `convert_progress()` - Convert progress between format groups
- `detect_conflict()` - Detect sync conflicts
- `merge_progress()` - Merge progress from multiple sources

### Technology Stack

**Backend**:

- Go 1.25+
- Echo v4 - HTTP framework
- pgx v5 - PostgreSQL driver
- SQLC - SQL code generation
- jwt-go - JWT authentication
- bcrypt - Password hashing

**Frontend**:

- Templ - HTML templating with Go
- HTMX - Dynamic interactions
- Tailwind CSS - Styling
- TypeScript - Frontend logic

**Database**:

- PostgreSQL 15+
- 30+ tables
- 50+ indexes
- JSONB for complex data

**Testing**:

- Testify - Testing framework
- Bruno - API testing
- 30+ integration test files

## 🚀 Local Development

### Prerequisites

- Go 1.25+
- Node.js 18+ (for frontend build)
- Podman or Docker
- PostgreSQL 15+ (or use Podman)

### Setup

```bash
# 1. Clone repository
git clone https://github.com/yourusername/bookhoard.git
cd bookhoard

# 2. Install Go dependencies
go mod download

# 3. Install build tools
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
go install github.com/a-h/templ/cmd/templ@latest

# 4. Set up environment
cp .env.example .env
# Edit .env with your settings

# 5. Generate database code
cd internal/database
sqlc generate

# 6. Generate templates
cd ../../templates
templ generate

# 7. Build frontend
cd ../web
npm install
npm run build

# 8. Run tests
cd ..
go test ./... -v
```

### Running Locally

```bash
# Option 1: Using containers (recommended)
podman-compose up --build

# Option 2: Direct Go run (requires local PostgreSQL)
# Generate secure passwords (no special characters):
#   JWT_SECRET: openssl rand -hex 32
#   DBPASS: openssl rand -hex 16
export JWT_SECRET="your-dev-secret"
export DBPASS="your-db-password"
go run cmd/server/main.go
```

### Development Workflow

**Backend Development**:

```bash
# Watch mode for Go (requires air or similar)
air

# Or manual rebuild
go build -o bookhoard cmd/server/main.go
./bookhoard
```

**Frontend Development**:

```bash
cd web
npm run dev  # Watch mode for TypeScript/CSS
```

**Database Changes**:

1. Edit `database/schema/schema.sql`
2. Edit `internal/database/queries/queries.sql`
3. Run: `cd internal/database && sqlc generate`
4. Restart server

**Template Changes**:

1. Edit `templates/*.templ`
2. Run: `cd templates && templ generate`
3. Restart server (templates auto-reload in dev mode)

## 🧪 Testing

### Unit Tests

```bash
# Run all unit tests
go test ./... -v

# Run specific package tests
go test ./internal/handlers/... -v

# Run with coverage
go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out
```

### Integration Tests

Integration tests hit the LIVE dev database and bind-mounted paths, both
of which exist only inside the Docker stack. A bare host `go test
./cmd/server/tests` fails with database-auth errors on every machine
that isn't running the compose stack — the container is the only
supported runner.

```bash
# Run the whole integration suite (pre-commit validation — NOT for
# iterating on a change; some tests are coupled to the shared dev DB's
# live state and this environment)
make test-integration

# Iterate on YOUR change: run only your own tests, in the container
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile tests run --rm tests \
  go test ./cmd/server/tests -run 'TestYourPrefix' -v -timeout 600s
```

**Run only your own tests while iterating.** A full-suite run rebuilds
the test image and takes minutes; more importantly, several tests read
live state (the dev DB's `system_settings`, real library folders, a
periodic backup scan broadcasting over websocket) and can fail on
timing or on data your change never touched. Filtered runs are cheap,
deterministic for your scope, and don't fight the live reader for the
database.

Test-created data rules (harness cleanup depends on them): users use
`@tests.bookhoard.internal` / `@example.com` emails, libraries and
collections carry "test"-prefixed names, and cleanups delete test
libraries (cascades handle the rest) rather than table-wide deletes.
Devices attach to the per-run regular user when it doesn't matter, or
self-delete via `TestDeviceSetup.CreateDevice` / `createContractDevice`
/ `cleanupDeviceByIdentifier` when they attach to the shared admin —
names lacking "test" (device-type names like "koreader", the
"Reinstall Device" label) escaped the name sweeps for weeks and piled
476 rows onto the live dev account; every suite now cleans its own
rows at test end, with the harness sweep's identifier/token patterns
(`contract-%`, `opds-folders-%`, `repro-reinstall-%`,
`dev_contract_*`) as the drift net.

### Media Fixtures (audio/books)

**Never commit binary media.** The Go toolchain and Docker prod builds
ignore `internal/services/testdata/` entirely (gitignored by policy —
see `.gitignore`), and the production image never copies `internal/`.
Generate what a test needs at runtime with ffmpeg (shipped in both the
test-runner and runtime images):

```bash
# The five audio containers (audiobook library accepts all of these)
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a libmp3lame -b:a 32k plain.mp3
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a flac       book.flac
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a libopus  -b:a 24k book.opus
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a libvorbis -q:a 1 book.ogg
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a aac      -b:a 32k book.aac
ffmpeg -f lavfi -i sine=frequency=440:duration=0.3 -codec:a pcm_s16le      book.wav

# Tags (album=book title, album_artist=author, composer=narrator —
# Audible's own convention — plus disc/track ordering keys)
ffmpeg -i in.mp3 -codec copy \
  -metadata title="Chapter Three" -metadata album="The Book" \
  -metadata album_artist="Author" -metadata composer="Narrator" \
  -metadata track="3/9" -metadata disc="2/3" \
  -metadata grouping="Series" -metadata genre="Audiobook" -metadata date=2020 out.mp3

# Embedded cover art (mp3 APIC / m4b covr): the picture rides as the
# only video stream
ffmpeg -i cover.jpg -i audio-only.mp3 -map 1:a -map 0:v -c:v copy \
  -disposition:v attached_pic tagged.mp3

# Embedded m4b chapters: an ffmetadata sidecar file
printf ';FFMETADATA1\n[CHAPTER]\nTIMEBASE=1/1000\nSTART=0\nEND=500\ntitle=Part One\n' > chapters.txt
ffmpeg -i in.m4b -i chapters.txt -map 0:a -map 1 -map_metadata 1 -codec copy -f ipod chaptered.m4b
```

CUE sheets and m3u playlists are plain text — write them inline in the
test. **The one thing ffmpeg cannot synthesize: Audible freeform atoms**
(`----:com.apple.iTunes:SERIES`, `:SUBTITLE`, `:ASIN`, `:ABRIDGED`) —
those ingestion paths are covered by unit tests using synthetic
`ffprobeReport` values instead of binary files.

Manual end-to-end testing against the live system: drop fixture books
into `uploads/Audiobooks/Ingestion Fixtures/` — **one book per
subfolder** (a directory with ≥2 audio files ingests as ONE
folder-of-tracks book) — and run a library scan from the admin UI.

### API Testing with Bruno OpenCollection YAML

```bash
# Install Bruno CLI
npm install -g @usebruno/cli

# Run all tests
bruno run

# Run specific collection
bruno run bruno/user/
bruno run bruno/sync-kobo/
```

### Test Configuration

Environment variables for testing:

- `TEST_MODE=true` - Enable test mode (disables rate limiting)
- `RATE_LIMIT_ENABLED=false` - Disable rate limiting
- `REQUESTS_PER_MINUTE=1000` - Increase rate limit

**⚠️ WARNING**: Never enable these in production!

### Test Library Naming Convention

Integration tests automatically clean up libraries with "test" in the name (case-insensitive).

**⚠️ IMPORTANT**: Do not use "test" in library names if you want to keep them!

- Libraries containing "test" (e.g., "My Test Library", "Test Library 1") will be deleted by test cleanup
- Use names like "Development Library", "Staging Books", or "Personal" for libraries you want to keep
- This ensures your manual test data persists between test runs

## 📝 Code Style Guidelines

### Go Code

- Follow [Effective Go](https://go.dev/doc/effective_go) guidelines
- Use `gofmt` for formatting
- Use `golangci-lint` for linting
- Use meaningful variable and function names
- Add comments for complex business logic
- Handle errors explicitly - don't ignore them

### Database Operations

- **Always use sqlc-generated code** - No raw SQL in handlers
- Use transactions for multi-step operations
- Handle `pgx.ErrNoRows` explicitly
- Use `pgtype.UUID` for UUID parameters
- Validate inputs before database operations

### Error Handling

```go
// Good - Explicit error handling
user, err := h.db.GetUser(c.Request().Context(), userID)
if err != nil {
    if errors.Is(err, pgx.ErrNoRows) {
        return c.JSON(http.StatusNotFound, map[string]string{"error": "user not found"})
    }
    return c.JSON(http.StatusInternalServerError, map[string]string{"error": "database error"})
}

// Bad - Ignoring errors
user, _ := h.db.GetUser(c.Request().Context(), userID)
```

### Adding New Features

1. **Database First**: Add tables/columns to `schema.sql`
2. **Generate Queries**: Add to `queries.sql` and run `sqlc generate`
3. **Handler**: Implement in `internal/handlers/`
4. **Routes**: Register in `cmd/server/main.go`
5. **Tests**: Add integration test in `cmd/server/tests/`
6. **Bruno**: Add API test in `bruno/`
7. **Docs**: Update relevant documentation

### API Design Principles

- RESTful naming conventions
- Consistent error responses
- Proper HTTP status codes
- JWT authentication on protected routes
- Input validation with struct tags
- Use echo.Context for request/response

## 🐳 Deployment

### Building for Production

```bash
# Using Makefile
make build-force

# Or manually
podman-compose build --no-cache
```

### Environment Variables

Required for production:

- `JWT_SECRET` - 64-byte random string (generate: `openssl rand -hex 32`)
- `DBPASS` - Strong database password (generate: `openssl rand -hex 16`)
- `BASE_URL` - Public URL (e.g., https://bookhoard.example.com)

Optional:

- `HTTPS_PROXY` - If behind reverse proxy

**Note**: Conversion service, rate limiting, and other operational settings have defaults in `docker-compose.yml` and can be overridden via `.env` if needed.

### Performance Tuning

**PostgreSQL Settings**:

```sql
-- In postgresql.conf
shared_buffers = 256MB
effective_cache_size = 1GB
maintenance_work_mem = 64MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 2621kB
min_wal_size = 1GB
max_wal_size = 4GB
```

**Go Settings**:

- GOMAXPROCS = number of CPU cores
- Worker pool concurrency: 3 (configurable in services/worker.go)

## 🔍 Debugging

### Enable Debug Logging

```bash
# Set environment variable
export DEBUG=true

# Or in .env
DEBUG=true
```

### Common Issues

**Database Connection Errors**:

- Check PostgreSQL is running
- Verify DATABASE_HOST and DATABASE_PORT
- Check firewall settings

**Rate Limiting During Development**:

- Enable test mode: `TEST_MODE=true RATE_LIMIT_ENABLED=false`
- Or increase limit: `REQUESTS_PER_MINUTE=1000`

**Template Not Updating**:

- Run `templ generate` in templates/ directory
- Restart server

**Database Queries Not Working**:

- Run `sqlc generate` in internal/database/
- Check generated code in `queries.sql.go`
- Verify SQL syntax in `queries.sql`

## 📚 Additional Resources

- [Development rules and standards](contributing.md) - Contributing portal
- [API Reference](../developer/api/api-reference.md) - Split endpoint docs, indexed from its overview
- [Troubleshooting](../operations/troubleshooting.md) - Deployment issues
- [Go Documentation](https://go.dev/doc/)
- [Echo Framework](https://echo.labstack.com/docs)
- [pgx Documentation](https://pgx.github.io/pgx/)

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Follow code style guidelines
4. Add tests for new features
5. Ensure all tests pass
6. Commit with clear messages
7. Push to branch (`git push origin feature/amazing-feature`)
8. Open a Pull Request

### Pull Request Checklist

- [ ] Code follows style guidelines
- [ ] Tests added/updated
- [ ] Documentation updated
- [ ] All tests passing
- [ ] No new warnings
- [ ] Commit messages are clear

---

**Happy Coding!** 🚀