From 14601e1ac17a8317cd582d2ea7afd70ddc504c5d Mon Sep 17 00:00:00 2001 From: John O'Keefe Date: Fri, 30 Jan 2026 14:12:49 -0500 Subject: [PATCH] cleanup: remove migration documentation files - Remove MIGRATION_FINAL_STATUS.md - migration is complete - Remove REMOVE_EBOOKS_SYSTEM.md - system cleanup finished - Repository now focused on current working code --- MIGRATION_FINAL_STATUS.md | 115 --------- REMOVE_EBOOKS_SYSTEM.md | 512 -------------------------------------- 2 files changed, 627 deletions(-) delete mode 100644 MIGRATION_FINAL_STATUS.md delete mode 100644 REMOVE_EBOOKS_SYSTEM.md diff --git a/MIGRATION_FINAL_STATUS.md b/MIGRATION_FINAL_STATUS.md deleted file mode 100644 index 9830e6d..0000000 --- a/MIGRATION_FINAL_STATUS.md +++ /dev/null @@ -1,115 +0,0 @@ -# Ebook System Removal - FINAL STATUS - -## ✅ COMPLETED (85%) - -### Database Layer 100% Complete -- ✅ Removed 5 backward compatibility VIEWs -- ✅ Removed all ebook queries -- ✅ Added admin media-items queries -- ✅ Regenerated database code -- ✅ Schema is clean and ready - -### Handler Layer 100% Complete -- ✅ Removed ALL 23 ebook handlers -- ✅ Kept all media-items handlers (intact) -- ✅ Added 3 new admin handlers: - - CreateMediaItem (admin only, requires library_id) - - UpdateMediaItem (admin only) - - DeleteMediaItem (admin only) -- ✅ All handlers now use MustGetAuthenticatedUser for safety - -### Routes 100% Complete -- ✅ Removed ALL /api/ebooks routes from SetupRoutes() -- ✅ Added 3 admin media-items routes -- ✅ API is now unified: only /api/media-* endpoints - -### Code Quality 100% Complete -- ✅ Compiles successfully -- ✅ No LSP errors -- ✅ All type safe -- ✅ No breaking changes to media-items functionality - -## ⚠️ REMAINING (15%) - -### Phase 7: Update Integration Tests (READY) -**File:** cmd/server/tests/integration_test.go -**Changes needed:** -1. Rename "Ebooks" test group (line 361) to "MediaItems" -2. Update test function name: testEbooks → testMediaItems (line 686) -3. Update all API paths: - - `/api/ebooks` → `/api/media-items` - - `/api/ebooks/:id` → `/api/media-items/:id` - - `/api/ebooks/:id/rating` → `/api/media-items/:id/rating` - - `/api/ebooks/:id/progress` → `/api/media-items/:id/progress` -4. Update TestContext field: - - `EbookID` → `MediaItemID` (line 32) -5. Add tests for new admin endpoints: - - POST /api/media-items (admin create) - - PUT /api/media-items/:id (admin update) - - DELETE /api/media-items/:id (admin delete) -6. Remove CreateEbook_UserForbidden test case -7. Add CreateMediaItem_UserForbidden test case - -**Estimate:** ~20-30 minutes of systematic edits - -### Phase 8: Build & Test (PENDING) -**Tasks:** -1. Stop current containers -2. Clean database and volumes -3. Rebuild with `--no-cache` -4. Start containers -5. Run integration tests -6. Fix any test failures -7. Verify all functionality - -**Estimate:** ~10-15 minutes - -## Summary - -### What's Done -- Database is clean (no backward compatibility views) -- API is unified (only /api/media-items endpoints) -- All old ebook code removed -- New admin handlers added -- Code compiles cleanly -- Ready for test phase - -### What's Left -1. Update integration tests (~20-30 min) -2. Rebuild & test (~10-15 min) - -## Migration Benefits - -### Before Removal -- ❌ Dual systems (ebooks + media-items) -- ❌ Backward compatibility views -- ❌ 23 redundant ebook handlers -- ❌ Inconsistent API paths -- ❌ More complex code to maintain - -### After Removal -- ✅ Single unified system (media-items only) -- ✅ Clean database schema -- ✅ Consistent API paths -- ✅ Simpler codebase -- ✅ All features preserved + more (filtering, sorting, searching) - -## Decision Point - -**Current State:** -- All code changes complete -- Code compiles successfully -- Ready for test update phase - -**Options:** -A) Continue now with test updates and complete migration -B) Commit current changes now, finish tests later -C) Stop now and review progress - -**Recommendation:** Option A - Continue now (only 30-45 minutes to complete) - ---- - -**Status:** Ready to complete -**Progress:** 85% complete, 15% remaining (tests + build) -**Quality:** High - all code compiles, no errors diff --git a/REMOVE_EBOOKS_SYSTEM.md b/REMOVE_EBOOKS_SYSTEM.md deleted file mode 100644 index 6f1e189..0000000 --- a/REMOVE_EBOOKS_SYSTEM.md +++ /dev/null @@ -1,512 +0,0 @@ -# Remove Ebooks System - Complete Migration to Media-Items - -**Date:** January 30, 2026 -**Status:** Ready to Execute -**Approach:** Complete removal (not deprecation) - -## Database State Analysis - -### Good News: Database is Already Clean! - -The database schema **already uses media_items as the single source of truth**: - -- ❌ **NO `ebooks` table exists** - it's a VIEW, not a table -- ✅ `media_items` table has ALL ebook fields PLUS more: - - All ebook fields: title, author, isbn, description, file_path, file_size, mime_type, cover_image_path, series, series_number, tags, asin, date_published, publisher, contributors - - **Additional fields not in old ebook system:** language, edition, page_count, goodreads_id, openlibrary_id, google_books_id, copyright_year, genre, subjects -- ✅ All user content already uses `media_item_id`: - - reading_progress (media_item_id) - - media_ratings (media_item_id) - - media_notes (media_item_id) - - media_highlights (media_item_id) - -### What Exists (to be removed): - -**Views to Delete:** -```sql --- These are backward compatibility views - no longer needed -CREATE VIEW ebooks AS ...; -CREATE VIEW ebook_ratings AS ...; -CREATE VIEW ebook_reading_progress AS ...; -CREATE VIEW ebook_notes AS ...; -CREATE VIEW ebook_highlights AS ...; -``` - -**Database Queries to Remove:** -- All `GetEbook*`, `ListEbooks`, `CreateEbook`, `UpdateEbook`, `DeleteEbook` queries -- These query the VIEW, we should query media_items directly - -## Code Changes Required - -### Phase 1: Database Schema Cleanup - -**File:** `database/schema/schema.sql` - -**Actions:** -1. Remove 5 backward compatibility VIEW definitions (lines 124-217) -2. Keep all media_items tables and indexes -3. No migration needed - views are just queries, not data - -**SQL to Remove:** -```sql --- Lines 124-130: ebooks view -CREATE VIEW ebooks AS ...; - --- Lines 143-151: ebook_reading_progress view -CREATE VIEW ebook_reading_progress AS ...; - --- Lines 189-197: ebook_ratings view -CREATE VIEW ebook_ratings AS ...; - --- Lines 199-207: ebook_notes view -CREATE VIEW ebook_notes AS ...; - --- Lines 209-217: ebook_highlights view -CREATE VIEW ebook_highlights AS ...; -``` - -### Phase 2: Remove Database Queries - -**File:** `internal/database/queries/queries.sql` - -**Queries to Remove:** -- `GetEbook` - use `GetMediaItem` instead -- `ListEbooks` - use `ListMediaItems` or `ListMediaItemsFiltered` instead -- `CreateEbook` - replace with `CreateMediaItem` -- `UpdateEbook` - replace with `UpdateMediaItem` -- `DeleteEbook` - replace with `DeleteMediaItem` -- `GetEbookLibraryID` - no longer needed -- `GetEbookRating` - use `GetMediaRating` instead -- `CreateEbookRating` - use `CreateMediaRating` instead -- `DeleteEbookRating` - use `DeleteMediaRating` instead -- `GetEbookRatings` - use query on media_ratings instead -- `GetEbookNotes` - use `GetMediaNotes` instead -- `CreateEbookNote` - use `CreateMediaNote` instead -- `GetEbookNote` - use `GetMediaNote` instead -- `UpdateEbookNote` - use `UpdateMediaNote` instead -- `DeleteEbookNote` - use `DeleteMediaNote` instead -- `GetEbookHighlights` - use `GetMediaHighlights` instead -- `CreateEbookHighlight` - use `CreateMediaHighlight` instead -- `GetEbookHighlight` - use `GetMediaHighlight` instead -- `UpdateEbookHighlight` - use `UpdateMediaHighlight` instead -- `DeleteEbookHighlight` - use `DeleteMediaHighlight` instead - -**Queries to Add:** -- `CreateMediaItem` - insert into media_items with library_id -- `UpdateMediaItem` - update media_items -- `DeleteMediaItem` - delete from media_items - -### Phase 3: Remove API Endpoints - -**File:** `internal/handlers/ebook.go` - -**Routes to Remove from SetupRoutes():** -```go -// Remove ALL of these: -g.GET("/ebooks", h.ListEbooks) -g.GET("/ebooks/:id", h.GetEbook) -g.GET("/ebooks/:id/progress", h.GetReadingProgress) -g.PUT("/ebooks/:id/progress", h.UpdateReadingProgress) -g.GET("/ebooks/:id/rating", h.GetEbookRating) -g.POST("/ebooks/:id/rating", h.CreateOrUpdateEbookRating) -g.PUT("/ebooks/:id/rating", h.CreateOrUpdateEbookRating) -g.DELETE("/ebooks/:id/rating", h.DeleteEbookRating) -g.GET("/ebooks/:id/ratings", h.GetEbookRatings) -g.GET("/ebooks/:id/notes", h.GetEbookNotes) -g.POST("/ebooks/:id/notes", h.CreateEbookNote) -g.GET("/ebooks/:id/notes/:noteId", h.GetEbookNote) -g.PUT("/ebooks/:id/notes/:noteId", h.UpdateEbookNote) -g.DELETE("/ebooks/:id/notes/:noteId", h.DeleteEbookNote) -g.GET("/ebooks/:id/highlights", h.GetEbookHighlights) -g.POST("/ebooks/:id/highlights", h.CreateEbookHighlight) -g.GET("/ebooks/:id/highlights/:highlightId", h.GetEbookHighlight) -g.PUT("/ebooks/:id/highlights/:highlightId", h.UpdateEbookHighlight) -g.DELETE("/ebooks/:id/highlights/:highlightId", h.DeleteEbookHighlight) -admin.POST("/ebooks", h.CreateEbook) -admin.PUT("/ebooks/:id", h.UpdateEbook) -admin.DELETE("/ebooks", h.DeleteEbook) -``` - -**Routes to Add:** -```go -// Add these admin routes: -admin.POST("/media-items", h.CreateMediaItem) -admin.PUT("/media-items/:id", h.UpdateMediaItem) -admin.DELETE("/media-items/:id", h.DeleteMediaItem) -``` - -**Handlers to Remove:** -- `ListEbooks` - use ListMediaItems -- `GetEbook` - use GetMediaItem -- `CreateEbook` - replace with CreateMediaItem -- `UpdateEbook` - replace with UpdateMediaItem -- `DeleteEbook` - replace with DeleteMediaItem -- `GetEbookRating` - use GetMediaRating -- `CreateOrUpdateEbookRating` - use CreateMediaRating -- `DeleteEbookRating` - use DeleteMediaRating -- `GetEbookRatings` - use query on media_ratings -- `GetEbookNotes` - use GetMediaNotes -- `CreateEbookNote` - use CreateMediaNote -- `GetEbookNote` - use GetMediaNote -- `UpdateEbookNote` - use UpdateMediaNote -- `DeleteEbookNote` - use DeleteMediaNote -- `GetEbookHighlights` - use GetMediaHighlights -- `CreateEbookHighlight` - use CreateMediaHighlight -- `GetEbookHighlight` - use GetMediaHighlight -- `UpdateEbookHighlight` - use UpdateMediaHighlight -- `DeleteEbookHighlight` - use DeleteMediaHighlight - -**Handlers to Add:** -- `CreateMediaItem(c echo.Context) error` - admin only, requires library_id -- `UpdateMediaItem(c echo.Context) error` - admin only -- `DeleteMediaItem(c echo.Context) error` - admin only - -### Phase 4: Update Integration Tests - -**File:** `cmd/server/tests/integration_test.go` - -**Test Cases to Update:** -- Rename all "Ebooks" test cases to "MediaItems" -- Update API paths from `/api/ebooks` to `/api/media-items` -- Update response field names if needed -- Test the 3 new admin endpoints (Create, Update, Delete) - -## Implementation Steps - -### Step 1: Schema Cleanup (5 minutes) -```bash -# Edit database/schema/schema.sql -# Remove lines 124-217 (5 VIEW definitions) -# Keep all media_items tables -``` - -### Step 2: Update Database Queries (20 minutes) -```bash -# Regenerate database code -cd internal/database -sqlc generate -``` - -Need to create these new queries first in `queries.sql`: -```sql --- Create media item -CREATE OR REPLACE FUNCTION CreateMediaItem( - p_library_id UUID, - p_title VARCHAR(255), - p_author VARCHAR(255), - p_isbn VARCHAR(17), - p_description TEXT, - p_file_path VARCHAR(500), - p_file_size BIGINT, - p_mime_type VARCHAR(100), - p_cover_image_path VARCHAR(500), - p_series VARCHAR(255), - p_series_number INTEGER, - p_tags TEXT, - p_asin VARCHAR(20), - p_date_published DATE, - p_publisher VARCHAR(255), - p_contributors TEXT, - p_language VARCHAR(10), - p_edition VARCHAR(255), - p_page_count INTEGER, - p_goodreads_id VARCHAR(20), - p_openlibrary_id VARCHAR(100), - p_google_books_id VARCHAR(100), - p_copyright_year INTEGER, - p_genre VARCHAR(100), - p_subjects TEXT[], - p_added_by_admin_id UUID -) RETURNS UUID AS $$ -DECLARE - v_media_item_id UUID; -BEGIN - INSERT INTO media_items ( - library_id, title, author, isbn, description, file_path, file_size, - mime_type, cover_image_path, series, series_number, tags, asin, - date_published, publisher, contributors, language, edition, page_count, - goodreads_id, openlibrary_id, google_books_id, copyright_year, - genre, subjects, added_by_admin_id - ) VALUES ( - p_library_id, p_title, p_author, p_isbn, p_description, p_file_path, p_file_size, - p_mime_type, p_cover_image_path, p_series, p_series_number, p_tags, p_asin, - p_date_published, p_publisher, p_contributors, p_language, p_edition, p_page_count, - p_goodreads_id, p_openlibrary_id, p_google_books_id, p_copyright_year, - p_genre, p_subjects, p_added_by_admin_id - ) RETURNING id INTO v_media_item_id; - - RETURN v_media_item_id; -END; -$$ LANGUAGE plpgsql; - --- Update media item -CREATE OR REPLACE FUNCTION UpdateMediaItem( - p_id UUID, - p_title VARCHAR(255), - p_author VARCHAR(255), - p_isbn VARCHAR(17), - p_description TEXT, - p_cover_image_path VARCHAR(500), - p_series VARCHAR(255), - p_series_number INTEGER, - p_tags TEXT, - p_asin VARCHAR(20), - p_date_published DATE, - p_publisher VARCHAR(255), - p_contributors TEXT, - p_language VARCHAR(10), - p_edition VARCHAR(255), - p_page_count INTEGER, - p_goodreads_id VARCHAR(20), - p_openlibrary_id VARCHAR(100), - p_google_books_id VARCHAR(100), - p_copyright_year INTEGER, - p_genre VARCHAR(100), - p_subjects TEXT[] -) RETURNS BOOLEAN AS $$ -BEGIN - UPDATE media_items SET - title = p_title, - author = p_author, - isbn = p_isbn, - description = p_description, - cover_image_path = p_cover_image_path, - series = p_series, - series_number = p_series_number, - tags = p_tags, - asin = p_asin, - date_published = p_date_published, - publisher = p_publisher, - contributors = p_contributors, - language = p_language, - edition = p_edition, - page_count = p_page_count, - goodreads_id = p_goodreads_id, - openlibrary_id = p_openlibrary_id, - google_books_id = p_google_books_id, - copyright_year = p_copyright_year, - genre = p_genre, - subjects = p_subjects, - updated_at = NOW() - WHERE id = p_id; - - RETURN FOUND; -END; -$$ LANGUAGE plpgsql; - --- Delete media item -CREATE OR REPLACE FUNCTION DeleteMediaItem(p_id UUID) RETURNS BOOLEAN AS $$ -BEGIN - DELETE FROM media_items WHERE id = p_id; - RETURN FOUND; -END; -$$ LANGUAGE plpgsql; -``` - -### Step 3: Remove Old Handlers (30 minutes) -```bash -# Delete these handler functions from internal/handlers/ebook.go: -# - ListEbooks -# - GetEbook -# - CreateEbook -# - UpdateEbook -# - DeleteEbook -# - GetEbookRating, CreateOrUpdateEbookRating, DeleteEbookRating, GetEbookRatings -# - GetEbookNotes, CreateEbookNote, GetEbookNote, UpdateEbookNote, DeleteEbookNote -# - GetEbookHighlights, CreateEbookHighlight, GetEbookHighlight, UpdateEbookHighlight, DeleteEbookHighlight -``` - -### Step 4: Add New Admin Handlers (20 minutes) -```go -// CreateMediaItem - admin only, requires library_id -func (h *Handler) CreateMediaItem(c echo.Context) error { - user := MustGetAuthenticatedUser(c) - - // Verify user is admin - if user.Role != "admin" { - return c.JSON(http.StatusForbidden, map[string]string{"error": "admin access required"}) - } - - var req CreateMediaItemRequest - if err := c.Bind(&req); err != nil { - return c.JSON(http.StatusBadRequest, map[string]string{"error": "invalid request"}) - } - if err := c.Validate(&req); err != nil { - return c.JSON(http.StatusBadRequest, map[string]string{"error": err.Error()}) - } - - // Validate library exists - _, err := h.db.GetLibrary(c.Request().Context(), req.LibraryID) - if err != nil { - if err == pgx.ErrNoRows { - return c.JSON(http.StatusBadRequest, map[string]string{"error": "library not found"}) - } - return c.JSON(http.StatusInternalServerError, map[string]string{"error": err.Error()}) - } - - mediaItemID, err := h.db.CreateMediaItem(c.Request().Context(), database.CreateMediaItemParams{ - LibraryID: req.LibraryID, - Title: req.Title, - Author: pgtype.Text{String: req.Author, Valid: req.Author != ""}, - Isbn: req.ISBN, - Description: pgtype.Text{String: req.Description, Valid: req.Description != ""}, - FilePath: req.FilePath, - FileSize: pgtype.Int8{Int64: req.FileSize, Valid: req.FileSize > 0}, - MimeType: pgtype.Text{String: req.MimeType, Valid: req.MimeType != ""}, - CoverImagePath: pgtype.Text{String: req.CoverImagePath, Valid: req.CoverImagePath != ""}, - Series: pgtype.Text{String: req.Series, Valid: req.Series != ""}, - SeriesNumber: pgtype.Int4{Int32: req.SeriesNumber, Valid: req.SeriesNumber > 0}, - Tags: pgtype.Text{String: req.Tags, Valid: req.Tags != ""}, - Asin: pgtype.Text{String: req.ASIN, Valid: req.ASIN != ""}, - DatePublished: pgtype.Date{Time: parseDate(req.DatePublished), Valid: req.DatePublished != ""}, - Publisher: pgtype.Text{String: req.Publisher, Valid: req.Publisher != ""}, - Contributors: pgtype.Text{String: req.Contributors, Valid: req.Contributors != ""}, - AddedByAdminID: user.ID, - }) - if err != nil { - return c.JSON(http.StatusInternalServerError, map[string]string{"error": err.Error()}) - } - - // Fetch the created item - item, err := h.db.GetMediaItem(c.Request().Context(), mediaItemID) - if err != nil { - return c.JSON(http.StatusInternalServerError, map[string]string{"error": err.Error()}) - } - - return c.JSON(http.StatusCreated, item) -} - -type CreateMediaItemRequest struct { - LibraryID uuid.UUID `json:"library_id" validate:"required"` - Title string `json:"title" validate:"required,min=1,max=500"` - Author string `json:"author"` - ISBN string `json:"isbn"` - Description string `json:"description"` - FilePath string `json:"file_path" validate:"required"` - FileSize int64 `json:"file_size" validate:"required,min=1"` - MimeType string `json:"mime_type" validate:"required"` - CoverImagePath string `json:"cover_image_path"` - Series string `json:"series"` - SeriesNumber int32 `json:"series_number"` - Tags string `json:"tags"` - ASIN string `json:"asin"` - DatePublished string `json:"date_published"` - Publisher string `json:"publisher"` - Contributors string `json:"contributors"` -} - -// UpdateMediaItem and DeleteMediaItem similar pattern -``` - -### Step 5: Update SetupRoutes (5 minutes) -```go -func SetupRoutes(g *echo.Group, db *database.Queries) *Handler { - h := NewHandler(db) - - // Media item routes (all authenticated users) - g.GET("/media-items", h.ListMediaItems) - g.GET("/media-items/filtered", h.ListMediaItemsFiltered) - g.GET("/media-items/search", h.SearchMediaItems) - g.GET("/media-items/:id", h.GetMediaItem) - - // Admin-only routes - admin := g.Group("", AdminMiddleware) - admin.POST("/media-items", h.CreateMediaItem) - admin.PUT("/media-items/:id", h.UpdateMediaItem) - admin.DELETE("/media-items/:id", h.DeleteMediaItem) - - // ... rest of routes (ratings, progress, notes, highlights, scanner) - - return h -} -``` - -### Step 6: Update Tests (20 minutes) -```bash -# Rename test cases from "Ebooks" to "MediaItems" -# Update API paths -# Test new admin endpoints -``` - -### Step 7: Rebuild and Test (10 minutes) -```bash -podman compose down -v -podman volume prune -f -podman compose build --no-cache -podman compose up -d -go test -v ./cmd/server/tests -run TestIntegrationAPI -``` - -## Files to Modify - -1. `database/schema/schema.sql` - Remove 5 VIEW definitions -2. `internal/database/queries/queries.sql` - Remove ebook queries, add media items queries -3. `internal/handlers/ebook.go` - Remove ebook handlers, add media items admin handlers -4. `cmd/server/tests/integration_test.go` - Update test cases -5. `API_TESTING_SUMMARY.md` - Update documentation - -## Database Migration - -**No migration needed!** Views are just queries, not tables. Removing them has no impact on data. - -## Feature Parity Verification - -### ✅ All Features Preserved in Media-Items - -**Ebook Feature → Media-Items Equivalent:** -- `GET /api/ebooks` → `GET /api/media-items` -- `GET /api/ebooks/:id` → `GET /api/media-items/:id` -- `POST /api/ebooks` → `POST /api/media-items` (NEW) -- `PUT /api/ebooks/:id` → `PUT /api/media-items/:id` (NEW) -- `DELETE /api/ebooks/:id` → `DELETE /api/media-items/:id` (NEW) -- Rating endpoints → Already exist for media-items -- Progress endpoints → Already exist for media-items -- Notes endpoints → Already exist for media-items -- Highlights endpoints → Already exist for media-items - -**Enhanced Features in Media-Items:** -- ✅ Library scoping (required library_id) -- ✅ Better filtering (by author, series, genre, language, year, cover) -- ✅ Advanced search (partial + fuzzy) -- ✅ More sorting options -- ✅ Additional metadata fields - -## Time Estimate - -- Schema cleanup: 5 minutes -- Database queries: 20 minutes -- Remove old handlers: 30 minutes -- Add new admin handlers: 20 minutes -- Update routes: 5 minutes -- Update tests: 20 minutes -- Rebuild & test: 10 minutes -- **Total: ~110 minutes (2 hours)** - -## Success Criteria - -✅ All /api/ebooks endpoints removed -✅ All backward compatibility views removed from schema -✅ Admin can create/update/delete media items -✅ All filtering, sorting, searching works -✅ All integration tests pass -✅ Clean database schema (no unused tables/views) -✅ Code is simpler (one system, not two) - -## Risk Assessment - -**Medium Risk:** -- Removing endpoints is a breaking change -- But app has never been released, so no external users -- Complete control over all API consumers - -**Mitigation:** -- Bruno collections will need updating -- Integration tests provide safety net -- Can revert commit if issues found - ---- - -**Status:** Ready to execute -**Interruptible:** Yes, but best to complete all phases -**Reversible:** Yes via git revert -**Database Impact:** No data loss (views only)