From 8cc593af99d254a2c27cf038c67fd9cac9f43d41 Mon Sep 17 00:00:00 2001 From: John O'Keefe Date: Wed, 25 Mar 2026 18:02:57 -0400 Subject: [PATCH] docs: add tags filter implementation plan Add comprehensive implementation plan for replacing genre_filter with tags_filter throughout the application. This document outlines the approach to leverage Calibre's tag-based categorization instead of the NULL genre field for imported books. Key decisions documented: - Keep genre_filter in API for backward compatibility - Filter tags instead of genre to work with existing Calibre data - Avoid database migration by using populated tags field Includes detailed implementation phases, technical specifications, testing strategy, and commit structure guidance for future work. Related to tags-based filtering enhancement --- IMPLEMENTATION_TAGS_FILTER.md | 714 ++++++++++++++++++++++++++++++++++ 1 file changed, 714 insertions(+) create mode 100644 IMPLEMENTATION_TAGS_FILTER.md diff --git a/IMPLEMENTATION_TAGS_FILTER.md b/IMPLEMENTATION_TAGS_FILTER.md new file mode 100644 index 0000000..7c1055e --- /dev/null +++ b/IMPLEMENTATION_TAGS_FILTER.md @@ -0,0 +1,714 @@ +# Implementation Plan: Replace Genre Filter with Tags Filter + +## Status: ๐Ÿ“‹ Planning Phase +**Created:** 2026-03-25 +**Type:** Full-Stack Task +**Priority:** Medium +**Estimated Time:** 2-3 hours + +--- + +## ๐ŸŽฏ Overview + +Replace the `genre_filter` with `tags_filter` throughout the application. The `genre` field in the database is currently NULL for all Calibre imports because Calibre uses tags for genre classification. By filtering tags instead, we leverage existing data without requiring a schema migration. + +**Key Decision:** Keep `genre_filter` in the API for backward compatibility but remove it from the frontend UI. + +--- + +## ๐Ÿ“Š Current State Analysis + +### Database State +```sql +-- Current data (from investigation): +SELECT COUNT(*) as total, COUNT(genre) as with_genre, COUNT(tags) as with_tags FROM media_items; +-- Result: 17 total books, 0 with genre, 17 with tags +``` + +### Problem +- **Genre field:** NULL for all imported books (Calibre doesn't populate it) +- **Tags field:** Populated with Calibre tags (including genre-like tags) +- **Filter behavior:** genre_filter returns 0 results (no matches on NULL) +- **User impact:** Cannot filter books by genre/category + +### Root Cause +Calibre treats tags as genre classification. Example tags: +``` +["Science Fiction", "Adventure", "Dystopian", "Favorites"] +``` + +The current genre filter expects a single `genre` text field, which is always NULL. + +--- + +## ๐Ÿ”„ Proposed Solution + +### Option B: Filter Tags Instead (Selected) + +**Why this approach:** +- โœ… No database migration required (zero downtime) +- โœ… Works immediately with existing Calibre data +- โœ… Low risk (isolated changes) +- โœ… Calibre users already understand this model +- โœ… Faster to implement + +**Trade-offs:** +- โš ๏ธ Tags may contain non-genre values (e.g., "Favorites", "To Read") +- โš ๏ธ User filters by "Tags" instead of "Genre" in UI + +**Why acceptable:** +- Calibre has the same "con" and it's not a problem there +- Users can filter their own tags as well (feature, not bug) +- API backward compatibility maintained (genre_filter still works) + +--- + +## ๐Ÿ“‹ Implementation Checklist + +### Phase 1: Backend - SQL Layer (15 minutes) +- [ ] Add `tags_filter` parameter to `SearchMediaItemsUnified` query +- [ ] Update filter logic from `word_similarity()` to `= ANY()` for array matching +- [ ] Keep `genre_filter` for backward compatibility +- [ ] Regenerate Go code with `sqlc generate` + +### Phase 2: Backend - Service Layer (20 minutes) +- [ ] Add `TagsFilter string` to `SearchParams` struct +- [ ] Update `dbParams` building to include tags_filter +- [ ] Update handler to extract `tags_filter` query parameter +- [ ] Keep existing `genre_filter` extraction (backward compat) + +### Phase 3: Backend - Tags Autocomplete API (30 minutes) +- [ ] Add `SearchTagsValues` SQL query +- [ ] Add "tags" case to `SearchFieldValues` service +- [ ] Add tags route to handler (autocomplete support) +- [ ] Test tags autocomplete endpoint + +### Phase 4: Frontend - TypeScript (20 minutes) +- [ ] Rename `fetchGenreValues()` โ†’ `fetchTagValues()` in bookshelf.ts +- [ ] Update field id from "genre" โ†’ "tags" in custom-section-builder.ts +- [ ] Update collection rules from "genre" โ†’ "tags" in collection-rules.ts +- [ ] No other TS files need changes (search.ts is header search only) + +### Phase 5: Frontend - Templates (15 minutes) +- [ ] Update bookshelf template (genre โ†’ tags labels/inputs) +- [ ] Change datalist id from "genre-datalist" โ†’ "tags-datalist" +- [ ] Update @input handler from `fetchGenreValues` โ†’ `fetchTagValues` +- [ ] Regenerate template Go files with `templ generate` + +### Phase 6: Bruno Collection Updates (15 minutes) +- [ ] Create `Fuzzy Tags Filter.yml` (rename/update genre filter test) +- [ ] Create `Field Values Search - Tags.yml` (rename genre autocomplete) +- [ ] Update `Combined Search and Filters.yml` (genre โ†’ tags_filter) +- [ ] Add documentation for tags filter behavior + +### Phase 7: Documentation (20 minutes) +- [ ] Create `docs/developer/api/media-items/search_media_items.md` (or update existing) +- [ ] Document tags_filter parameter +- [ ] Document backward compatibility with genre_filter +- [ ] Add examples of tags filtering +- [ ] Update any relevant user documentation + +### Phase 8: Integration Tests (30 minutes) +- [ ] Create `tags_filter_test.go` in `cmd/server/tests/` +- [ ] Test with `setupTestServer()` helper from `test_helpers_test.go` +- [ ] Test tags filter returns matching books +- [ ] Test tags autocomplete endpoint +- [ ] Test backward compatibility (genre_filter still works) +- [ ] Test combined filters (tags + author + series) + +### Phase 9: Verification & Testing (20 minutes) +- [ ] Run full test suite: `go test ./... -v` +- [ ] Test tags filter in browser (manual QA) +- [ ] Verify tags autocomplete works +- [ ] Verify backward compatibility (genre_filter) +- [ ] Check no regressions in existing filters + +--- + +## ๐Ÿ”ง Technical Changes + +### 1. SQL Query Changes + +**File:** `internal/database/queries/queries.sql` + +**Location:** Line 435-437 (SearchMediaItemsUnified query) + +**Change:** +```sql +-- BEFORE: +-- Fuzzy genre filter +AND (sqlc.narg('genre_filter') = '' OR word_similarity(sqlc.narg('genre_filter'), COALESCE(mi.genre, '')) > 0.3) + +-- AFTER: +-- Fuzzy genre filter (kept for backward compatibility) +AND (sqlc.narg('genre_filter') = '' OR word_similarity(sqlc.narg('genre_filter'), COALESCE(mi.genre, '')) > 0.3) +-- Tags filter (NEW - matches against tags array) +AND (sqlc.narg('tags_filter') = '' OR sqlc.narg('tags_filter') = ANY(mi.tags_search)) +``` + +**New Query:** Add `SearchTagsValues` for autocomplete +```sql +-- name: SearchTagsValues :many +SELECT + unnest(mi.tags_search) as value, + COUNT(*) as count, + word_similarity(sqlc.narg('search_query'), unnest(mi.tags_search))::float8 as score +FROM media_items mi +JOIN libraries l ON mi.library_id = l.id +LEFT JOIN library_visibility lv ON l.id = lv.library_id AND lv.user_id = sqlc.narg('user_id') +WHERE COALESCE(lv.is_visible, true) = true + AND (sqlc.narg('library_id')::uuid IS NULL OR mi.library_id = sqlc.narg('library_id')::uuid) + AND unnest(mi.tags_search) IS NOT NULL + AND word_similarity(sqlc.narg('search_query'), unnest(mi.tags_search)) > 0.3 +GROUP BY unnest(mi.tags_search), word_similarity(sqlc.narg('search_query'), unnest(mi.tags_search))::float8 +ORDER BY word_similarity(sqlc.narg('search_query'), unnest(mi.tags_search))::float8 DESC, COUNT DESC +LIMIT sqlc.narg('limit') OFFSET sqlc.narg('offset'); +``` + +### 2. Service Layer Changes + +**File:** `internal/services/search.go` + +**Change 1:** Update SearchParams struct (line 22-36) +```go +type SearchParams struct { + UserID pgtype.UUID + LibraryID pgtype.UUID + AuthorFilter string + SeriesFilter string + GenreFilter string // KEPT for backward compatibility + TagsFilter string // NEW + LanguageFilter string + YearMin int + YearMax int + HasCover bool + SearchQuery string + Sort string + Limit int + Offset int +} +``` + +**Change 2:** Update dbParams building (line 67-82) +```go +dbParams := database.SearchMediaItemsUnifiedParams{ + UserID: params.UserID, + LibraryID: params.LibraryID, + AuthorFilter: pgtype.Text{String: params.AuthorFilter, Valid: params.AuthorFilter != ""}, + SeriesFilter: pgtype.Text{String: params.SeriesFilter, Valid: params.SeriesFilter != ""}, + GenreFilter: pgtype.Text{String: params.GenreFilter, Valid: params.GenreFilter != ""}, // KEPT + TagsFilter: pgtype.Text{String: params.TagsFilter, Valid: params.TagsFilter != ""}, // NEW + LanguageFilter: pgtype.Text{String: params.LanguageFilter, Valid: params.LanguageFilter != ""}, + // ... rest of params +} +``` + +**Change 3:** Add tags case to SearchFieldValues (after line 151) +```go +case "tags": + results, err := s.db.SearchTagsValues(ctx, database.SearchTagsValuesParams{ + SearchQuery: pgtype.Text{String: params.SearchQuery, Valid: true}, + UserID: params.UserID, + LibraryID: params.LibraryID, + Limit: pgtype.Int4{Int32: int32(params.Limit), Valid: true}, + Offset: pgtype.Int4{Int32: int32(params.Offset), Valid: true}, + }) + if err != nil { + return nil, err + } + fieldValues := make([]FieldValue, len(results)) + for i, r := range results { + fieldValues[i] = FieldValue{Value: r.Value.String, Count: r.Count, Score: r.Score} + } + return fieldValues, nil +``` + +### 3. Handler Changes + +**File:** `internal/handlers/media.go` + +**Change 1:** Extract tags_filter parameter (after line 1414) +```go +// Extract filter parameters +authorFilter := c.QueryParam("author_filter") +seriesFilter := c.QueryParam("series_filter") +genreFilter := c.QueryParam("genre_filter") // KEPT for backward compatibility +tagsFilter := c.QueryParam("tags_filter") // NEW +languageFilter := c.QueryParam("language_filter") +``` + +**Change 2:** Add to SearchParams (line 1426-1440) +```go +params := services.SearchParams{ + UserID: userID.ID, + LibraryID: libUUID, + SearchQuery: query, + AuthorFilter: authorFilter, + SeriesFilter: seriesFilter, + GenreFilter: genreFilter, // KEPT + TagsFilter: tagsFilter, // NEW + LanguageFilter: languageFilter, + // ... rest of params +} +``` + +**Change 3:** Add tags autocomplete route (after line 1379) +```go +if tags := c.QueryParam("tags"); tags != "" { + return mh.handleFieldValuesSearch(c, userID.ID, "tags", tags) +} +``` + +### 4. Frontend TypeScript Changes + +**File:** `web/src/bookshelf.ts` + +**Change:** Line 297-300 +```typescript +// AFTER (add new function, keep old commented out): +async fetchTagValues(input: HTMLInputElement): Promise { + await this.fetchFieldValues("tags", input.value, "tags-datalist"); +} + +// TODO: Uncomment if genre field is populated in future +// async fetchGenreValues(input: HTMLInputElement): Promise { +// await this.fetchFieldValues("genre", input.value, "genre-datalist"); +// } +``` + +**File:** `web/src/custom-section-builder.ts` + +**Change:** Line 49-50 +```typescript +// AFTER (add new field, keep old commented out): +[ + { id: "tags", label: "Tags" }, + // TODO: Uncomment if genre field is populated in future + // { id: "genre", label: "Genre" }, + // ... other fields +] +``` + +**File:** `web/src/collection-rules.ts` + +**Change:** Line 45 +```typescript +// AFTER (add new field, keep old commented out): +const ruleFields = { + tags: "Tags", + // TODO: Uncomment if genre field is populated in future + // genre: "Genre", + // ... other fields +}; +``` + +### 5. Template Changes + +**File:** `templates/bookshelf.templ` (source file) + +**Locate:** Find the genre filter input section (around line 131 in generated Go file) + +**Change:** +```html + + +
+ + + +
+ + +{{- /* Genre Filter with Autocomplete */} +{{- /*
*/} +{{- /* */} +{{- /* */}} +{{- /* */}} +{{- /*
*/}} +``` + +**Then regenerate:** `templ generate` + +**Note:** The genre filter code is preserved in the template as a comment, making it easy to re-enable in the future if the genre field is populated. + +--- + +## ๐Ÿงช Testing Strategy + +### Integration Tests + +**File:** `cmd/server/tests/tags_filter_test.go` (NEW) + +**Test Structure:** +```go +package main + +import ( + "testing" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestTagsFilter(t *testing.T) { + setup := setupTestServer(t) // โœ… Using test_helpers + defer setup.Server.Close() + + libraryID := setup.CreateLibrary(t, "Test Tags Library", "ebooks") + _ = setup.CreateDevice(t, "Test Device", "koreader", "tags-test-123") + + client := &http.Client{} + + // Add folder to library + addFolderToLibrary(t, setup, libraryID, "/app/uploads") + + // Helper to create book with tags + createBook := func(title, author string, tags []string) { + bookReq := map[string]interface{}{ + "library_id": libraryID, + "title": title, + "author": author, + "tags": tags, + "file_path": "/tmp/test.epub", + "file_size": 1024, + "mime_type": "application/epub+zip", + } + // ... create book + } + + // Create test books with different tags + createBook("Foundation", "Asimov", []string{"Science Fiction", "Adventure"}) + createBook("Dune", "Herbert", []string{"Science Fiction", "Dystopian"}) + createBook("Neuromancer", "Gibson", []string{"Cyberpunk", "Science Fiction"}) + createBook("The Hobbit", "Tolkien", []string{"Fantasy", "Adventure"}) + + t.Run("Tags filter - Science Fiction", func(t *testing.T) { + req := httptest.NewRequest("GET", "/api/media-items/search?library_id="+libraryID+"&tags_filter=Science+Foundation", nil) + req.Header.Set("Authorization", "Bearer "+setup.UserToken) + // ... test implementation + // Should return Foundation, Dune, Neuromancer (all have "Science Fiction" tag) + }) + + t.Run("Tags filter - Fantasy", func(t *testing.T) { + // Should return only The Hobbit + }) + + t.Run("Tags autocomplete", func(t *testing.T) { + req := httptest.NewRequest("GET", "/api/media-items/search?tags=Sci&library_id="+libraryID, nil) + req.Header.Set("Authorization", "Bearer "+setup.UserToken) + // Should return "Science Fiction" in autocomplete results + }) + + t.Run("Backward compatibility - genre_filter still works", func(t *testing.T) { + // Verify genre_filter doesn't break (returns 0 results on NULL genre) + }) + + t.Run("Combined filters - tags + author", func(t *testing.T) { + // tags_filter=Science+Fiction&author_filter=Asimov + // Should return only Foundation + }) +} +``` + +### Manual QA Checklist + +- [ ] Tags filter returns books with matching tags +- [ ] Tags autocomplete shows tag values as you type +- [ ] Multiple tags in book are all searchable +- [ ] Combined filters work (tags + author + series) +- [ ] Empty tags_filter returns all books (no filter) +- [ ] Backward compatibility: genre_filter still works (returns 0 on NULL) +- [ ] Browser console shows no errors +- [ ] HTMX requests complete successfully +- [ ] UI updates without page reload + +--- + +## ๐Ÿ“ Documentation + +### API Documentation + +**File:** `docs/developer/api/media-items/search_media_items.md` (CREATE or UPDATE) + +**Content:** +```markdown +# Search Media Items + +## GET /api/media-items/search + +Search and filter media items with fuzzy matching and tag-based filtering. + +### Query Parameters + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `q` | string | No | Search query (matches title, author, series, tags) | +| `library_id` | string | No | Filter by library UUID | +| `author_filter` | string | No | Fuzzy match author field | +| `series_filter` | string | No | Fuzzy match series field | +| `tags_filter` | string | No | **Exact match tags array** (NEW) | +| `language_filter` | string | No | Fuzzy match language field | +| `year_min` | integer | No | Minimum copyright year | +| `year_max` | integer | No | Maximum copyright year | +| `has_cover` | boolean | No | Filter by cover existence | +| `sort` | string | No | Sort order (e.g., "title ASC") | +| `limit` | integer | No | Max results (default: 50) | +| `offset` | integer | No | Results offset (default: 0) | + +### Tags Filter Behavior + +**New in v1.x:** The `tags_filter` parameter filters books by their tags array. + +- **Matching:** Exact match (not fuzzy) - the filter value must equal a tag +- **Calibre Integration:** Works seamlessly with Calibre tags (which include genre) +- **Example Tags:** `["Science Fiction", "Adventure", "Dystopian"]` + +**Examples:** + +```bash +# Filter by "Science Fiction" tag +GET /api/media-items/search?tags_filter=Science+Fiction + +# Combine tags with author +GET /api/media-items/search?tags_filter=Science+Fiction&author_filter=Asimov + +# Filter by multiple tags (returns books with ANY matching tag) +GET /api/media-items/search?tags_filter=Cyberpunk +``` + +### Backward Compatibility + +The `genre_filter` parameter is **deprecated but still supported** for backward compatibility. It will return 0 results for books imported from Calibre (genre field is NULL). + +**Migration Guide:** +```javascript +// OLD (deprecated): +genre_filter=Science+Fiction + +// NEW (recommended): +tags_filter=Science+Fiction +``` + +### Response + +**Status Codes:** +- `200 OK` - Results found +- `404 Not Found` - No results match filters +- `401 Unauthorized` - Missing or invalid JWT +- `500 Internal Server Error` - Server error + +**Response Body:** +```json +[ + { + "id": "uuid", + "title": "Foundation", + "author": "Isaac Asimov", + "tags": ["Science Fiction", "Adventure"], + "library_id": "uuid", + "library_name": "E-Books" + } +] +``` +``` + +### User Documentation + +**File:** `docs/user/searching-and-filtering.md` (UPDATE or CREATE) + +**Content:** +```markdown +# Searching and Filtering Books + +## Tag-Based Filtering + +Bookhoard uses Calibre's tag system for categorization. When you import books from Calibre, their tags are automatically available for filtering. + +### How Tags Work + +Tags are keywords or categories assigned to books, such as: +- **Genres:** Science Fiction, Fantasy, Mystery, Romance +- **Categories:** Favorites, To Read, Read +- **Metadata:** Ebook, Owned, Borrowed + +### Filtering by Tags + +1. Navigate to the **Bookshelf** page +2. Use the **Tags** filter input +3. Start typing to see autocomplete suggestions +4. Select a tag or press Enter to filter + +**Example:** Typing "Sci" will suggest "Science Fiction" + +### Combining Filters + +You can combine tags with other filters: +- **Tags + Author:** Find all Science Fiction by Asimov +- **Tags + Series:** Find all Fantasy in the "Wheel of Time" series +- **Tags + Year:** Find all Dystopian fiction published after 2000 + +### Tips + +- Tags from Calibre are automatically imported +- You can add custom tags when editing book metadata +- Tags are case-insensitive for searching +- Autocomplete shows existing tags as you type +``` + +--- + +## ๐ŸŽฏ Git Commit Structure + +### Commit 1: Backend SQL Layer +``` +feat: add tags_filter to search query + +- Add tags_filter parameter to SearchMediaItemsUnified +- Add SearchTagsValues query for autocomplete +- Keep genre_filter for backward compatibility + +Relates to # +``` + +### Commit 2: Backend Service Layer +``` +feat: implement tags filter in service layer + +- Add TagsFilter to SearchParams struct +- Update dbParams building to include tags_filter +- Add tags case to SearchFieldValues service + +Relates to # +``` + +### Commit 3: Backend Handler Layer +``` +feat: add tags filter and autocomplete endpoints + +- Extract tags_filter query parameter in handler +- Add tags autocomplete route +- Keep genre_filter for backward compatibility + +Relates to # +``` + +### Commit 4: Frontend TypeScript +``` +refactor: replace genre with tags in frontend TypeScript + +- Add fetchTagValues function (keep fetchGenreValues commented out) +- Update custom-section-builder field id to "tags" (keep "genre" commented) +- Update collection-rules field mappings to "tags" (keep "genre" commented) +- Genre code preserved for easy restoration if field is populated later + +Relates to # +``` + +### Commit 5: Frontend Templates +``` +refactor: replace genre with tags in bookshelf UI + +- Add Tags filter input (keep Genre input commented out) +- Update input name to tags_filter (keep genre_filter in comment) +- Update datalist and Alpine.js bindings +- Genre HTML preserved in template comments for future use + +Relates to # +``` + +### Commit 6: Bruno Collection +``` +test: update Bruno requests for tags filter + +- Add Fuzzy Tags Filter.yml +- Add Field Values Search - Tags.yml +- Update Combined Search and Filters.yml + +Relates to # +``` + +### Commit 7: Documentation +``` +docs: document tags filter API and usage + +- Add search_media_items API documentation +- Add user guide for tag-based filtering +- Document backward compatibility with genre_filter + +Relates to # +``` + +### Commit 8: Integration Tests +``` +test: add integration tests for tags filter + +- Create tags_filter_test.go +- Test tags filter functionality +- Test tags autocomplete endpoint +- Test backward compatibility +- Test combined filters + +Uses test_helpers.setupTestServer() +Relates to # +``` + +--- + +## โœ… Verification Checklist + +### Before Completing Task + +- [ ] All code compiles: `go build ./...` +- [ ] All tests pass: `go test ./... -v` +- [ ] Guidelines verified: `bash scripts/verify-guidelines.sh` (0 errors) +- [ ] Documentation renders at `/docs` endpoint +- [ ] Bruno requests execute successfully +- [ ] Manual QA completed in browser +- [ ] No regressions in existing filters +- [ ] Git commits are logical and well-structured +- [ ] Only intended changes in `git diff` + +### Post-Deployment Verification + +- [ ] Tags filter works in production +- [ ] Tags autocomplete returns results +- [ ] Backward compatibility maintained (genre_filter) +- [ ] No performance issues +- [ ] User feedback collected + +--- + +## ๐Ÿ“š References + +- **Project Guidelines:** `PROJECT_GUIDELINES.md` +- **Service Layer Pattern:** `internal/services/` +- **Test Helpers:** `cmd/server/tests/test_helpers_test.go` +- **Bruno Collection:** `bruno/media-items/` +- **API Documentation:** `docs/developer/api/` +- **User Documentation:** `docs/user/` + +--- + +## ๐Ÿš€ Next Steps + +1. **Review this plan** and approve approach +2. **Implement Phase 1-3** (Backend changes) +3. **Implement Phase 4-5** (Frontend changes) +4. **Implement Phase 6-7** (Bruno + Docs) +5. **Implement Phase 8-9** (Tests + Verification) +6. **Create git commits** following commit structure +7. **Push changes** and verify in production + +--- + +**Last Updated:** 2026-03-25 +**Status:** Ready for implementation