docs(schema): Consolidate schema initialization plans with corrections
Remove obsolete planning documents: - KOBO_IMPLEMENTATION_PLAN.md (replaced by refined plan) - SCHEMA_INITIALIZATION_PLAN.md (replaced by refined plan) Update REFINED_SCHEMA_PLAN.md with critical corrections: - Fix FNV-1a hash constant (7804706162000639061, was 582394759234) - Correct table/index counts (27 tables, 82 indexes, not 36/102) - Change approach: embed existing schema.sql (no duplication) - Add local database update step after schema changes - Add documentation requirements section This consolidates three planning documents into one accurate, actionable plan for implementing automatic schema initialization with idempotent migrations.
This commit is contained in:
+83
-51
@@ -16,11 +16,15 @@
|
||||
|
||||
## 📊 Technical Decisions
|
||||
|
||||
### 1. File Structure: Single Cohesive Package
|
||||
### 1. File Structure: Single Source of Truth
|
||||
|
||||
**Decision:** Single `internal/database/schema/schema.go` file containing all schema logic.
|
||||
**Decision:** Single `internal/database/schema.go` file referencing existing `database/schema/schema.sql`.
|
||||
|
||||
**Rationale:** Follows Go ecosystem standards and PostgreSQL/pgx patterns.
|
||||
**Rationale:**
|
||||
- **No duplication**: Single schema file remains the source of truth
|
||||
- **Clear separation**: `/database/` for schema definitions, `/internal/database/` for Go logic
|
||||
- **Maintainability**: Only one schema file to maintain
|
||||
- **Follows existing patterns**: Database definitions stay where they belong
|
||||
|
||||
### 2. Function Location: All in One Package
|
||||
|
||||
@@ -71,44 +75,69 @@
|
||||
## 📁 Implementation File Structure
|
||||
|
||||
```
|
||||
internal/database/schema/
|
||||
internal/database/
|
||||
├── schema.go # Main initialization logic
|
||||
├── schema.sql # Embedded idempotent schema copy
|
||||
└── verification.go # REMOVED - consolidated into schema.go
|
||||
├── db.go # Existing sqlc-generated code
|
||||
├── models.go # Existing sqlc-generated models
|
||||
├── queries.sql.go # Existing sqlc-generated queries
|
||||
└── ...
|
||||
|
||||
database/
|
||||
└── schema/
|
||||
└── schema.sql # Existing single source of truth schema
|
||||
```
|
||||
|
||||
**Single Responsibility:** `schema.go` handles all database initialization logic.
|
||||
**Single Responsibility:** `schema.go` handles all database initialization logic, referencing the existing schema file.
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Implementation Plan
|
||||
|
||||
### Phase 1: Cleanup (Start Fresh)
|
||||
### Phase 1: Schema File Verification
|
||||
|
||||
#### Step 1.1: Remove Duplicate Files
|
||||
#### Step 1.1: Ensure Schema is Idempotent
|
||||
```bash
|
||||
rm -f /home/nymusicman/Code/bookhoard/internal/database/schema/verification.go
|
||||
# Convert remaining statements to idempotent form
|
||||
# Current state: 9/27 tables, 20/82 indexes, 1/3 inserts are idempotent
|
||||
|
||||
# Tables needing IF NOT EXISTS (18):
|
||||
# - library_types, users, system_settings, refresh_tokens, libraries
|
||||
# - library_folders, library_visibility, media_items, reading_progress
|
||||
# - media_ratings, media_notes, media_highlights, devices, sync_queue
|
||||
# - sync_conflicts, kobo_shelves, kobo_entitlements, reading_history
|
||||
|
||||
# Indexes: Convert remaining 62 CREATE INDEX to CREATE INDEX IF NOT EXISTS
|
||||
|
||||
# Inserts needing ON CONFLICT (2):
|
||||
# - Line 14: library_types (name)
|
||||
# - Line 44: system_settings (setting_key)
|
||||
```
|
||||
|
||||
#### Step 1.2: Clean Up Schema File
|
||||
#### Step 1.2: Verify Schema Syntax
|
||||
```bash
|
||||
# Ensure schema.sql is fully idempotent
|
||||
grep "CREATE TABLE IF NOT EXISTS" database/schema/schema.sql | wc -l # Should be 36
|
||||
grep "CREATE INDEX IF NOT EXISTS" database/schema/schema.sql | wc -l # Should be 102
|
||||
grep "ON CONFLICT" database/schema/schema.sql | wc -l # Should be 3
|
||||
# Test schema.sql syntax validity
|
||||
psql -h localhost -U postgres -d postgres -f database/schema/schema.sql --echo-errors --quiet
|
||||
```
|
||||
|
||||
#### Step 1.3: Update Local Database
|
||||
```bash
|
||||
# CRITICAL: This is pre-production, update local DB after schema.sql changes
|
||||
podman compose down -v # WARNING: loses all data
|
||||
podman compose up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Create Single Schema Package
|
||||
### Phase 2: Create Schema Initialization Logic
|
||||
|
||||
#### Step 2.1: Create `internal/database/schema/schema.go`
|
||||
#### Step 2.1: Create `internal/database/schema.go`
|
||||
|
||||
**Implementation Strategy:**
|
||||
```go
|
||||
package schema
|
||||
package database
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"embed"
|
||||
"fmt"
|
||||
@@ -121,13 +150,13 @@ import (
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
//go:embed schema.sql
|
||||
//go:embed ../../database/schema/schema.sql
|
||||
var SchemaFile string
|
||||
|
||||
const (
|
||||
// Pre-computed FNV-1a hash of "bookhoard:schema:init"
|
||||
// Generated using: generateLockID("bookhoard:schema:init") = 582394759234
|
||||
schemaInitLockID = 582394759234
|
||||
// Generated using: generateLockID("bookhoard:schema:init")
|
||||
schemaInitLockID = 7804706162000639061
|
||||
)
|
||||
|
||||
// Hash calculation function (for reference/testing)
|
||||
@@ -301,15 +330,14 @@ func Initialize(ctx context.Context, db *pgxpool.Pool) error {
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 2.2: Copy schema.sql to Package
|
||||
#### Step 2.2: Verify Embed Path
|
||||
```bash
|
||||
cp database/schema/schema.sql internal/database/schema/schema.sql
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
diff database/schema/schema.sql internal/database/schema/schema.sql
|
||||
# Should produce no output
|
||||
# Test that the embed path resolves correctly
|
||||
cd /home/nymusicman/Code/bookhoard/internal/database
|
||||
go test -c -o /tmp/test_embed .
|
||||
# Or simply verify the relative path exists:
|
||||
ls -la ../../database/schema/schema.sql
|
||||
# Should show the file exists at the correct relative location
|
||||
```
|
||||
|
||||
---
|
||||
@@ -324,7 +352,7 @@ diff database/schema/schema.sql internal/database/schema/schema.sql
|
||||
```go
|
||||
import (
|
||||
// ... existing imports ...
|
||||
"bookhoard/internal/database/schema"
|
||||
"bookhoard/internal/database"
|
||||
)
|
||||
|
||||
func main() {
|
||||
@@ -341,7 +369,7 @@ func main() {
|
||||
// ===== NEW: Schema Initialization =====
|
||||
log.Println("🔧 Ensuring database schema is initialized...")
|
||||
ctx := context.Background()
|
||||
if err := schema.Initialize(ctx, dbPool); err != nil {
|
||||
if err := database.Initialize(ctx, dbPool); err != nil {
|
||||
log.Fatal("❌ Database schema initialization failed:", err)
|
||||
}
|
||||
log.Println("✅ Database schema initialized and verified, starting server...")
|
||||
@@ -358,25 +386,27 @@ func main() {
|
||||
### Before Declaring Complete
|
||||
|
||||
**Phase 1 Verification:**
|
||||
- [ ] Cleaned up duplicate verification.go file
|
||||
- [ ] Verified all CREATE TABLE use IF NOT EXISTS (36 total)
|
||||
- [ ] Verified all CREATE INDEX use IF NOT EXISTS (102 total)
|
||||
- [ ] Verified all INSERT have ON CONFLICT (3 total)
|
||||
- [ ] Converted 18 CREATE TABLE → CREATE TABLE IF NOT EXISTS (27 total)
|
||||
- [ ] Converted 62 CREATE INDEX → CREATE INDEX IF NOT EXISTS (82 total)
|
||||
- [ ] Added ON CONFLICT to 2 INSERT statements (3 total)
|
||||
- [ ] Verified schema.sql syntax is valid
|
||||
- [ ] Confirmed single schema.sql exists as source of truth
|
||||
|
||||
**Phase 2 Verification:**
|
||||
- [ ] `internal/database/schema/schema.go` compiles without errors
|
||||
- [ ] Single cohesive package with all related functions
|
||||
- [ ] Imports only what's used (embed, pgx v5)
|
||||
- [ ] `internal/database/schema.go` compiles without errors
|
||||
- [ ] Single file with all schema initialization functions
|
||||
- [ ] Imports only what's used (bufio, embed, pgx v5)
|
||||
- [ ] Uses *pgxpool.Pool concrete type
|
||||
- [ ] Pre-computed constant for lock ID (582394759234)
|
||||
- [ ] Pre-computed constant for lock ID (7804706162000639061)
|
||||
- [ ] Stream scanning implementation for performance
|
||||
- [ ] schema.sql successfully embedded and copied
|
||||
- [ ] Successfully embeds existing schema.sql file
|
||||
- [ ] No schema file duplication (single source of truth)
|
||||
|
||||
**Phase 3 Verification:**
|
||||
- [ ] `cmd/server/main.go` compiles without errors
|
||||
- [ ] Schema initialization call added before handler creation
|
||||
- [ ] Proper error handling with Fatal on failure
|
||||
- [ ] Uses correct import path (`database.Initialize`)
|
||||
|
||||
**Integration Testing:**
|
||||
- [ ] Fresh database initializes correctly
|
||||
@@ -425,18 +455,12 @@ func main() {
|
||||
|
||||
If critical errors occur:
|
||||
|
||||
### Option 1: Restore Single File
|
||||
### Option 1: Restore Schema File
|
||||
```bash
|
||||
git checkout HEAD -- internal/database/schema/schema.go
|
||||
git checkout HEAD -- internal/database/schema.go
|
||||
```
|
||||
|
||||
### Option 2: Restore Package
|
||||
```bash
|
||||
rm -rf internal/database/schema
|
||||
git checkout HEAD -- internal/database/schema/
|
||||
```
|
||||
|
||||
### Option 3: Restore Integration
|
||||
### Option 2: Restore Integration
|
||||
```bash
|
||||
git checkout HEAD -- cmd/server/main.go
|
||||
```
|
||||
@@ -458,7 +482,7 @@ The app will initialize the database automatically on first startup.
|
||||
|
||||
**Monitor logs for:**
|
||||
```
|
||||
✅ Database schema initialized and verified (36 tables verified)
|
||||
✅ Database schema initialized and verified (27 tables verified)
|
||||
```
|
||||
|
||||
### Subsequent Deployments
|
||||
@@ -494,7 +518,7 @@ podman exec bookhoard_db psql -U postgres -d bookhoard -c "\dt"
|
||||
### Check advisory locks:
|
||||
```bash
|
||||
podman exec bookhoard_db psql -U postgres -d bookhoard -c "
|
||||
SELECT * FROM pg_locks WHERE objid = 582394759234;
|
||||
SELECT * FROM pg_locks WHERE objid = 7804706162000639061;
|
||||
"
|
||||
```
|
||||
|
||||
@@ -506,4 +530,12 @@ podman compose up -d
|
||||
|
||||
---
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
Create documentation explaining the automatic schema initialization:
|
||||
- **docs/contributing/database-schema.md** - How schema initialization works, how to modify schema safely
|
||||
- **README.md** - Add "Database Initialization" section documenting first-run behavior
|
||||
|
||||
---
|
||||
|
||||
**End of Refined Implementation Plan**
|
||||
|
||||
Reference in New Issue
Block a user