bookhoard/docs/contributing/development.md

# Bookhoard Development Guide

This guide is for developers contributing to Bookhoard or setting up a development environment.

## 🏗 Architecture

### Directory Structure

```
bookhoard/
├── cmd/server/                    # Application entry point
│   ├── main.go                   # Server initialization, route registration
│   └── tests/                    # Integration tests (30+ test files)
├── internal/
│   ├── config/                   # Configuration management
│   ├── database/                 # Database layer (SQLC generated)
│   ├── handlers/                 # HTTP request handlers (18 files)
│   ├── middleware/              # HTTP middleware (9 files)
│   ├── services/                # Business logic (7 files)
│   ├── sync/                    # Sync framework (5 files)
│   ├── opds/                    # OPDS feed generation
│   └── utils/                   # Utility functions
├── templates/                   # UI templates (17 .templ files)
├── web/src/                     # Frontend TypeScript
├── database/schema/             # Database schema
├── docs/                        # Documentation
└── bruno/                       # API test collections
```

### Backend Components

**Handlers** (`internal/handlers/`):

- `auth.go` - Authentication & user management
- `library.go` - Library CRUD operations
- `scanner.go` - Media scanning operations
- `media.go` - Media downloads, shelves
- `koreader.go` - KOReader sync protocol
- `kobo.go` - Kobo sync protocol
- `collections.go` - Collection management
- `devices.go` - Device registration/management
- `conflicts.go` - Sync conflict resolution
- `queue.go` - Sync queue management
- `progress.go` - Reading progress tracking
- `analytics.go` - Usage analytics
- `opds.go` - OPDS feed generation
- `websocket.go` - WebSocket connections
- `sync.go` - Sync orchestration
- `book_matching.go` - Book linking/matching
- `sidecar.go` - Sidecar file handling
- `refresh_token.go` - Token refresh logic
- `context.go` - Handler context utilities

**Middleware** (`internal/middleware/`):

- `device_auth.go` - Device authentication
- `device_rate_limiter.go` - Device-specific rate limiting
- `error_handler.go` - Global error handling
- `login_attempts.go` - Login attempt tracking
- `password_validator.go` - Password complexity validation
- `rate_limiter.go` - IP-based rate limiting
- `request_tracing.go` - Request ID tracking
- `security.go` - Security headers
- `transaction.go` - Database transaction middleware

**Services** (`internal/services/`):

- `library_service.go` - Library operations
- `media_scanner.go` - File scanning, metadata extraction, and real-time file watching
- `worker.go` - Job queue worker pool
- `collection_service.go` - Collection rules processing
- `conversion_service.go` - EPUB→KEPUB conversion
- `book_matching.go` - Book matching algorithms

**Sync Framework** (`internal/sync/`):

- `queue.go` - Sync queue processor
- `progress.go` - Universal progress format
- `websocket.go` - Real-time sync broadcast
- `offline.go` - Offline sync support
- `format.go` - Format group conversion

### Database Schema

**Core Tables**:

- `users` - User accounts with authentication and settings
- `libraries` - Library definitions
- `library_types` - Media type definitions (ebooks, comics, manga)
- `library_folders` - Multiple folders per library
- `library_visibility` - User-specific library access control
- `media_items` - Universal media storage (replaces ebooks table)
- `media_ratings` - User ratings (1-10 scale for half-star precision)
- `media_notes` - User annotations
- `media_highlights` - User highlights with color customization
- `reading_progress` - Universal progress tracking across devices
- `devices` - Device registry for sync
- `sync_queue` - Offline sync support
- `sync_conflicts` - Conflict resolution tracking
- `collections` - Device-neutral collections
- `collection_items` - Books in collections
- `device_shelf_mappings` - Map collections to device-specific shelves
- `device_catalogs` - Track OPDS downloads and ContentId mappings
- `kobo_shelves` - Kobo-specific shelf management
- `reading_history` - Reading session tracking
- `unlinked_books` - Track books that couldn't be auto-matched
- `media_item_formats` - Track all format versions with hashes
- `device_file_aliases` - Track file paths per device
- `refresh_tokens` - JWT refresh token storage

**Database Functions**:

- `normalize_isbn()` - ISBN format normalization
- `detect_format_group()` - Detect format group (reflowable, fixed_layout, comic_archive)
- `convert_progress()` - Convert progress between format groups
- `detect_conflict()` - Detect sync conflicts
- `merge_progress()` - Merge progress from multiple sources

### Technology Stack

**Backend**:

- Go 1.25+
- Echo v4 - HTTP framework
- pgx v5 - PostgreSQL driver
- SQLC - SQL code generation
- jwt-go - JWT authentication
- bcrypt - Password hashing

**Frontend**:

- Templ - HTML templating with Go
- HTMX - Dynamic interactions
- Tailwind CSS - Styling
- TypeScript - Frontend logic

**Database**:

- PostgreSQL 15+
- 30+ tables
- 50+ indexes
- JSONB for complex data

**Testing**:

- Testify - Testing framework
- Bruno - API testing
- 30+ integration test files

## 🚀 Local Development

### Prerequisites

- Go 1.25+
- Node.js 18+ (for frontend build)
- Podman or Docker
- PostgreSQL 15+ (or use Podman)

### Setup

```bash
# 1. Clone repository
git clone https://github.com/yourusername/bookhoard.git
cd bookhoard

# 2. Install Go dependencies
go mod download

# 3. Install build tools
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
go install github.com/a-h/templ/cmd/templ@latest

# 4. Set up environment
cp .env.example .env
# Edit .env with your settings

# 5. Generate database code
cd internal/database
sqlc generate

# 6. Generate templates
cd ../../templates
templ generate

# 7. Build frontend
cd ../web
npm install
npm run build

# 8. Run tests
cd ..
go test ./... -v
```

### Running Locally

```bash
# Option 1: Using containers (recommended)
podman-compose up --build

# Option 2: Direct Go run (requires local PostgreSQL)
# Generate secure passwords (no special characters):
#   JWT_SECRET: openssl rand -hex 32
#   DBPASS: openssl rand -hex 16
export JWT_SECRET="your-dev-secret"
export DBPASS="your-db-password"
go run cmd/server/main.go
```

### Development Workflow

**Backend Development**:

```bash
# Watch mode for Go (requires air or similar)
air

# Or manual rebuild
go build -o bookhoard cmd/server/main.go
./bookhoard
```

**Frontend Development**:

```bash
cd web
npm run dev  # Watch mode for TypeScript/CSS
```

**Database Changes**:

1. Edit `database/schema/schema.sql`
2. Edit `internal/database/queries/queries.sql`
3. Run: `cd internal/database && sqlc generate`
4. Restart server

**Template Changes**:

1. Edit `templates/*.templ`
2. Run: `cd templates && templ generate`
3. Restart server (templates auto-reload in dev mode)

## 🧪 Testing

### Unit Tests

```bash
# Run all unit tests
go test ./... -v

# Run specific package tests
go test ./internal/handlers/... -v

# Run with coverage
go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out
```

### Integration Tests

```bash
# Run integration tests (may hit rate limits)
go test ./cmd/server/tests -v

# Run with test mode (recommended)
TEST_MODE=true RATE_LIMIT_ENABLED=false go test ./cmd/server/tests -v

# Run specific test
go test ./cmd/server/tests -run TestAuth -v
```

### API Testing with Bruno OpenCollection YAML

```bash
# Install Bruno CLI
npm install -g @usebruno/cli

# Run all tests
bruno run

# Run specific collection
bruno run bruno/user/
bruno run bruno/sync-kobo/
```

### Test Configuration

Environment variables for testing:

- `TEST_MODE=true` - Enable test mode (disables rate limiting)
- `RATE_LIMIT_ENABLED=false` - Disable rate limiting
- `REQUESTS_PER_MINUTE=1000` - Increase rate limit

**⚠️ WARNING**: Never enable these in production!

### Test Library Naming Convention

Integration tests automatically clean up libraries with "test" in the name (case-insensitive).

**⚠️ IMPORTANT**: Do not use "test" in library names if you want to keep them!

- Libraries containing "test" (e.g., "My Test Library", "Test Library 1") will be deleted by test cleanup
- Use names like "Development Library", "Staging Books", or "Personal" for libraries you want to keep
- This ensures your manual test data persists between test runs

## 📝 Code Style Guidelines

### Go Code

- Follow [Effective Go](https://go.dev/doc/effective_go) guidelines
- Use `gofmt` for formatting
- Use `golangci-lint` for linting
- Use meaningful variable and function names
- Add comments for complex business logic
- Handle errors explicitly - don't ignore them

### Database Operations

- **Always use sqlc-generated code** - No raw SQL in handlers
- Use transactions for multi-step operations
- Handle `pgx.ErrNoRows` explicitly
- Use `pgtype.UUID` for UUID parameters
- Validate inputs before database operations

### Error Handling

```go
// Good - Explicit error handling
user, err := h.db.GetUser(c.Request().Context(), userID)
if err != nil {
    if errors.Is(err, pgx.ErrNoRows) {
        return c.JSON(http.StatusNotFound, map[string]string{"error": "user not found"})
    }
    return c.JSON(http.StatusInternalServerError, map[string]string{"error": "database error"})
}

// Bad - Ignoring errors
user, _ := h.db.GetUser(c.Request().Context(), userID)
```

### Adding New Features

1. **Database First**: Add tables/columns to `schema.sql`
2. **Generate Queries**: Add to `queries.sql` and run `sqlc generate`
3. **Handler**: Implement in `internal/handlers/`
4. **Routes**: Register in `cmd/server/main.go`
5. **Tests**: Add integration test in `cmd/server/tests/`
6. **Bruno**: Add API test in `bruno/`
7. **Docs**: Update relevant documentation

### API Design Principles

- RESTful naming conventions
- Consistent error responses
- Proper HTTP status codes
- JWT authentication on protected routes
- Input validation with struct tags
- Use echo.Context for request/response

## 🐳 Deployment

### Building for Production

```bash
# Using Makefile
make build-force

# Or manually
podman-compose build --no-cache
```

### Environment Variables

Required for production:

- `JWT_SECRET` - 64-byte random string (generate: `openssl rand -hex 32`)
- `DBPASS` - Strong database password (generate: `openssl rand -hex 16`)
- `BASE_URL` - Public URL (e.g., https://bookhoard.example.com)

Optional:

- `HTTPS_PROXY` - If behind reverse proxy

**Note**: Conversion service, rate limiting, and other operational settings have defaults in `docker-compose.yml` and can be overridden via `.env` if needed.

### Performance Tuning

**PostgreSQL Settings**:

```sql
-- In postgresql.conf
shared_buffers = 256MB
effective_cache_size = 1GB
maintenance_work_mem = 64MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 2621kB
min_wal_size = 1GB
max_wal_size = 4GB
```

**Go Settings**:

- GOMAXPROCS = number of CPU cores
- Worker pool concurrency: 3 (configurable in services/worker.go)

## 🔍 Debugging

### Enable Debug Logging

```bash
# Set environment variable
export DEBUG=true

# Or in .env
DEBUG=true
```

### Common Issues

**Database Connection Errors**:

- Check PostgreSQL is running
- Verify DATABASE_HOST and DATABASE_PORT
- Check firewall settings

**Rate Limiting During Development**:

- Enable test mode: `TEST_MODE=true RATE_LIMIT_ENABLED=false`
- Or increase limit: `REQUESTS_PER_MINUTE=1000`

**Template Not Updating**:

- Run `templ generate` in templates/ directory
- Restart server

**Database Queries Not Working**:

- Run `sqlc generate` in internal/database/
- Check generated code in `queries.sql.go`
- Verify SQL syntax in `queries.sql`

## 📚 Additional Resources

- [Project Guidelines](../../PROJECT_GUIDELINES.md) - Development rules and standards
- [API Reference](../API_REFERENCE.md) - Complete API documentation
- [Troubleshooting](../TROUBLESHOOTING.md) - Deployment issues
- [Go Documentation](https://go.dev/doc/)
- [Echo Framework](https://echo.labstack.com/docs)
- [pgx Documentation](https://pgx.github.io/pgx/)

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Follow code style guidelines
4. Add tests for new features
5. Ensure all tests pass
6. Commit with clear messages
7. Push to branch (`git push origin feature/amazing-feature`)
8. Open a Pull Request

### Pull Request Checklist

- [ ] Code follows style guidelines
- [ ] Tests added/updated
- [ ] Documentation updated
- [ ] All tests passing
- [ ] No new warnings
- [ ] Commit messages are clear

---

**Happy Coding!** 🚀