Commit Graph
58 Commits
Author SHA1 Message Date
john-okeefe 0667cad8a9 feat: add reader infrastructure - Phase 0 database schema and queries
Implement Phase 0 prerequisites for reader functionality including
database schema, SQL queries, and frontend dependencies.

## Database Schema (5 New Tables + 1 Column Addition)

### New Tables Added:
1. **panel_data** - Comic/manga panel detection results
   - Stores detected panel boundaries (x, y, width, height)
   - Supports grid, ML, and manual detection methods
   - JSONB storage for flexible panel structures

2. **reading_speed** - User reading speed statistics
   - Tracks pages per minute and total reading time
   - Per-user per-media-item tracking
   - Enables progress estimation and analytics

3. **dictionary_cache** - Offline dictionary word definitions
   - Caches external dictionary lookups
   - Reduces API calls and improves performance
   - Supports offline reading functionality

4. **reader_settings** - User reader preferences (per-user)
   - Stores typography, theme, and display settings
   - JSONB storage for flexible configuration
   - Per-user customization (fonts, margins, themes)

5. **media_bookmarks** - Enhanced bookmarks with chapter/CFI support
   - Unified bookmarking for ebooks, comics, manga, PDFs
   - Supports page_number, chapter_number, and epubcfi_position
   - Includes notes field for annotations
   - Unique constraint on (media_item_id, user_id, title)

### Column Addition:
- **media_items.chapter_metadata** (JSONB) - Caches detected chapter structure
  - Stores TOC/chapter detection results
  - Prevents re-parsing files on every read
  - Populated by ReaderService.DetectChapters()

## Database Queries (12 New Queries)

Added queries for all reader functionality:
- Panel data: GetPanelData, UpsertPanelData
- Reading speed: GetReadingSpeed, CreateReadingSpeed, UpdateReadingSpeed
- Dictionary: GetDictionaryEntry, CreateDictionaryEntry, UpdateDictionaryAccessed
- Settings: GetReaderSettings, UpsertReaderSettings
- Bookmarks: GetMediaBookmarks, CreateMediaBookmark, DeleteMediaBookmark, UpdateMediaBookmark

## Frontend Dependencies

Added to package.json:
- jszip@^3.10.1 - EPUB/comic archive parsing (client-side)
- pdfjs-dist@^3.11.174 - PDF rendering library (Mozilla PDF.js)

## Generated Code

Ran `sqlc generate` to regenerate:
- models.go - Go structs for new tables (55 lines added)
- querier.go - Database query methods (14 lines added)
- queries.sql.go - Compiled SQL queries (504 lines added)

## Implementation Status

Phase 0 prerequisites now complete:
 Database schema (5 tables + 1 column)
 SQL queries (12 queries)
 Frontend dependencies (2 packages)
 Generated Go code (sqlc)
 Database recreated with new schema

Ready for Phase 1: Infrastructure & Basic Reader implementation.

Related to: Universal web reader for ebooks, comics, manga, PDFs
2026-04-02 21:01:45 -04:00
john-okeefe dd81dc08a2 db: update SQL queries and regenerate models for comic metadata
Phase 1 implementation: Update CreateMediaItem query to support 14 new comic metadata fields.

Changes:
- Add 14 new columns to CreateMediaItem INSERT statement
- Regenerate sqlc models with new fields
- CommunityRating now maps to pgtype.Float8 (was pgtype.Numeric)
- All new comic and universal metadata fields included

New fields supported:
- Reading direction: manga_type, reading_direction
- Universal: series_count, volume, imprint, age_rating, web_url
- Comic-specific: story_arc, is_black_and_white, metadata_notes,
  community_rating, alternate_info, scan_information, summary

Generated models verified:
- MediaItems struct includes all 14 new fields
- CreateMediaItemParams has correct parameter count (42 total)
- CommunityRating is pgtype.Float8 (not pgtype.Numeric)

Relates to: Phase 4.1-4.2 database layer implementation
2026-03-29 21:12:20 -04:00
john-okeefe 0e11c9263c fix: add unique constraint for saved filter names and fix search sort ordering
Database changes:
- Add unique index on saved_filters(user_id, name, resource_type)
  Prevents duplicate filter names while allowing same name across
  different users or different resource types

Search functionality fix:
- Remove DISTINCT ON (mi.id) from SearchMediaItemsUnified query
- Remove mi.id from ORDER BY clause (was required by DISTINCT ON)
- This allows user-selected sort field to be primary sort criteria
- Previously results were always sorted by ID first, making sort
  dropdown ineffective
- Relevance score and title remain as fallback sorts

This fixes the sort dropdown functionality on the bookshelf page
where changing the sort option appeared to have no effect.
2026-03-28 00:46:03 -04:00
john-okeefe 36ae781765 fix: implement proper 3-state boolean logic for has_cover filter
Fixed the SearchMediaItemsUnified query to properly handle the has_cover
parameter in three states:
- NULL (not specified): Show all books
- TRUE: Show only books with cover images
- FALSE: Show only books without cover images

Changes:
- Added explicit boolean casting (::bool) to sqlc.narg('has_cover')
  to resolve PostgreSQL type inference error (SQLSTATE 42P08)
- Replaced single AND condition with OR'd logic to handle all three
  states without mutual exclusion
- Used IS NULL check to detect when parameter is not specified
- Used IS TRUE/IS FALSE to explicitly check boolean states

The previous implementation had mutually exclusive AND conditions that
prevented any records from matching when has_cover was explicitly set
to TRUE or FALSE, causing the filter to block all searches.

This fix resolves the issue where searches were returning 0 results
regardless of other filter parameters when has_cover was included in
the query.
2026-03-27 18:07:41 -04:00
john-okeefe be31cc88f1 feat: enhance search with date-prioritized year filtering and true exact matching
Improve media item search functionality with two key enhancements:

1. Date-prioritized year filtering:
   - Prioritize date_published over copyright_year for year range queries
   - Fall back to copyright_year when date_published is NULL
   - Extract year from date_published timestamp for comparison

2. True exact search matching:
   - Replace ILIKE pattern matching with exact equality for quoted queries
   - Use search_query directly instead of wildcard pattern for exact matches
   - Remove SearchPattern parameter and related wildcard logic
   - Add COALESCE handling for author/series NULL values in exact matches

These changes make year filtering more accurate with published dates
and provide genuine exact matching when users wrap queries in quotes.

Refs internal/database/queries/queries.sql:475, internal/services/search.go:62
2026-03-26 15:35:31 -04:00
john-okeefe 0f70f74f12 fix: correct tag alias references in SearchTagsValues query
- Change all tag.value references to tag in SearchTagsValues query
- Fix PostgreSQL error: "column tag.value does not exist"
- CROSS JOIN LATERAL unnest() creates alias 'tag', not 'tag.value'
- Updates SELECT, WHERE, GROUP BY, and ORDER BY clauses
- Regenerate Go code with sqlc generate

When using CROSS JOIN LATERAL unnest(mi.tags_search) AS tag,
PostgreSQL creates 'tag' as the column alias, not 'tag.value'.
This fix aligns all references to use just 'tag', matching the
actual column name created by the LATERAL join.

Resolves tags autocomplete SQLSTATE 42703 error.

Relates to TestTagsFilter tags autocomplete test
2026-03-25 20:57:14 -04:00
john-okeefe c4ebbd990c fix: resolve tags autocomplete SQL error with CROSS JOIN LATERAL
- Fix SearchTagsValues query to use CROSS JOIN LATERAL instead of unnest() in WHERE clause
- PostgreSQL error: "set-returning functions are not allowed in WHERE"
- Change from direct unnest() calls to a proper lateral join pattern
- References: tag.value instead of repeated unnest(mi.tags_search) calls
- Regenerate Go code with sqlc generate

This fixes the tags autocomplete functionality which was failing with
SQLSTATE 0A000 error. The CROSS JOIN LATERAL approach properly expands
the tags array before filtering, allowing set-returning functions to
work correctly in the query.

Relates to TestTagsFilter tags autocomplete test
2026-03-25 20:51:21 -04:00
john-okeefe 00840c2fe1 feat: add fuzzy tags_filter to search query
- Add tags_filter parameter to SearchMediaItemsUnified
- Add EXISTS clause with word_similarity() for fuzzy tag matching
- Add tag similarity scoring to ORDER BY clause (GREATEST function)
- Add SearchTagsValues query for autocomplete with ::TEXT cast
- Keep genre_filter for backward compatibility
- Regenerate Go code with sqlc generate

This enables filtering books by tags (from Calibre) instead of genre,
which is always NULL for imported books. Uses fuzzy matching consistent
with author/series filters, with best matches sorted first.

Relates to IMPLEMENTATION_TAGS_FILTER.md Phase 1
2026-03-25 20:38:08 -04:00
john-okeefe 5571a47830 fix: eliminate duplicate search results from library visibility LEFT JOIN
Problem:
The search API was returning duplicate media items when searching across
libraries. For example, searching for "Harry" with 2 books would return
4-8 results instead of 2, depending on how many users had library visibility
entries.

Root Cause:
The SearchMediaItemsUnified query uses a LEFT JOIN with library_visibility:

  LEFT JOIN library_visibility lv ON l.id = lv.library_id AND lv.user_id = $1

When multiple library_visibility entries exist for the same library
(e.g., one per user during testing), the LEFT JOIN can create duplicate
rows for each media_item. The query didn't have a DISTINCT clause to
eliminate these duplicates.

Solution:
Added DISTINCT ON (mi.id) clause with mi.id as the first ORDER BY expression:

  SELECT DISTINCT ON (mi.id) mi.*, ...
  FROM media_items mi
  ...
  ORDER BY mi.id, <other_sort_criteria>

This ensures that even if the LEFT JOIN produces multiple rows per
media_item, only one row per mi.id is returned, preserving the first
occurrence based on the relevance sorting.

Impact:
- Search results now correctly return unique media items
- Test TestCollectionSearchLibraryFilter will pass after database cleanup
- No API changes required - this is purely a query optimization

Note: After deploying this change, residual test data should be cleaned up
with: docker-compose down -v && docker-compose up -d

Files changed:
- internal/database/queries/queries.sql: Added DISTINCT ON clause
- internal/database/queries.sql.go: Regenerated from sqlc
2026-03-24 20:25:13 -04:00
john-okeefe fe8a65af84 feat: enable cross-library search in unified search query
Updates SearchMediaItemsUnified query to support searching across all
libraries when library_id parameter is not provided. Changes SQL from
requiring library_id to checking for NULL:

  AND (sqlc.narg('library_id')::uuid IS NULL
      OR mi.library_id = sqlc.narg('library_id')::uuid)

The explicit ::uuid cast ensures PostgreSQL handles type inference
correctly when comparing UUID columns with nullable parameters.

Regenerates Go database code including queries.sql.go and querier.go
to reflect the updated SQL schema.

This enables the /api/media-items/search endpoint to search all libraries
by omitting the library_id query parameter, matching the behavior of
the OPDS search endpoint.
2026-03-24 16:47:23 -04:00
john-okeefe 43a6d843a3 feat: add unified search SQL queries with fuzzy filters
- Add SearchMediaItemsUnified query combining search + filters
- Add 4 field value search queries (author, genre, series, language) for autocomplete
- Support fuzzy text matching via pg_trgm (threshold: 0.3 similarity)
- Support exact match with quotes detection for search queries
- Add sort parameter support (title ASC/DESC, author ASC/DESC, created_at ASC/DESC, page_count ASC/DESC)
- Primary sort by relevance score when searching, secondary by user-specified sort
- Combine search query with all filter types in single optimized query
- Uses 4 separate simple queries instead of 1 complex query due to sqlc v1.30.0 limitation with CASE in GROUP BY

This consolidates the deprecated /filtered and /search endpoints into one unified endpoint.
2026-03-23 22:37:37 -04:00
john-okeefe 26c81c8793 feat: add unified search queries with fuzzy matching
Add comprehensive search queries supporting both fuzzy and exact matching:

1. SearchMediaItemsUnified - Main search query with:
   - Fuzzy matching on author, series, genre, language filters
   - Fuzzy search on title, author, series, tags, contributors
   - Exact matching with quotes (is_exact_search flag)
   - Year range and boolean filters
   - Relevance-based ordering using word_similarity scores

2. Field-specific autocomplete queries:
   - SearchAuthorValues, SearchGenreValues, SearchSeriesValues, SearchLanguageValues
   - Each returns distinct values with counts and similarity scores
   - Threshold of 0.3 for word_similarity filter
   - Ordered by relevance (score DESC, count DESC)

Note: Using 4 separate field value queries instead of 1 complex query
due to sqlc v1.30.0 limitation with CASE expressions in GROUP BY clauses.
2026-03-22 20:35:04 -04:00
john-okeefe 85964ec932 fix(api): enforce user isolation on saved filters delete operation
Fix critical security issue where admin users could delete other users'
saved filters due to incorrect error handling in DELETE query.

Database Schema Changes:
- Change DeleteSavedFilter from :exec to :one (queries.sql:1747-1750)
- Add RETURNING * to return deleted row for proper error detection
- Regenerate querier.go and queries.sql.go with updated signature

Service Layer (internal/services/filters.go):
- Update DeleteSavedFilter to capture returned row (using _ to discard)
- Properly propagate pgx.ErrNoRows when no rows are deleted
- Error wrapping preserves original error for handler detection

Handler Layer (internal/handlers/filters.go):
- Add errors.Is() check for pgx.ErrNoRows (line 148)
- Return 404 Not Found when filter doesn't exist or belongs to different user
- Return 500 Internal Server Error for other database errors
- Add "errors" import (line 8)

Security Fix Details:
Before: Admin could delete user's filter → 204 No Content (SUCCESS)
After:  Admin tries to delete user's filter → 404 Not Found (DENIED)

The DELETE query uses WHERE id = @id AND user_id = @user_id, which matches
0 rows when attempting to delete another user's filter. The old :exec query
didn't return row count, so 0 affected rows looked like success. The new :one
query with RETURNING * returns pgx.ErrNoRows when no rows match, allowing
the handler to return proper 404 error.

Test Impact:
- TestSavedFilters/User_cannot_access_another_user's_filter now passes
- All 6 integration tests pass with proper user isolation enforcement

Pattern Consistency:
- Matches DeleteLibraryFolder pattern (line 99 in queries.sql)
- Uses same error handling as media handlers (errors.Is + pgx.ErrNoRows)
- Follows user-scoping pattern used throughout codebase

Related: Saved filters implementation user isolation
Security: Prevents unauthorized deletion of user data
2026-03-21 01:24:15 -04:00
john-okeefe e17a96123f feat(db): add saved_filters table and CRUD operations
Add database schema and SQL queries for generic saved filters system
that allows users to save custom filter presets for any resource type.

Database Schema:
- Add saved_filters table with user_id, name, resource_type, filters (JSONB)
- Create composite index on (user_id, resource_type) for efficient lookups
- Create index on (user_id, name) for future name search feature
- Add update_updated_at_column() trigger to auto-update timestamps
- Make trigger creation idempotent with DROP TRIGGER IF EXISTS

SQL Queries (5 new queries):
- GetSavedFilters: List all filters for user + resource type
- GetSavedFilterByID: Retrieve single filter by ID
- CreateSavedFilter: Create new saved filter
- UpdateSavedFilter: Update filter name/criteria
- DeleteSavedFilter: Remove saved filter

Design Decisions:
- Generic resource_type field supports any resource (media-items, collections, devices)
- JSONB filters field allows flexible schema without migrations
- User-scoped via JWT (user_id foreign key with CASCADE delete)
- Automatic updated_at timestamp via database trigger

Generated Code:
- database.SavedFilters model (10 fields including JSONB filters)
- All 5 CRUD query functions with proper parameter types
- pgtype.UUID wrappers for UUID parameters

Part of: Saved Filters Implementation (Phase 1: Database)
Related: #saved-filters-feature
2026-03-21 00:16:05 -04:00
john-okeefe 79690751c8 fix: improve type safety in media item search queries
- Change library_id parameter from interface{} to pgtype.UUID
- Add explicit UUID type casting in SQL queries
- Fix SearchMediaItemsParams to use strongly-typed UUID
- Prevents potential type assertion errors and improves type safety
- Ensures proper NULL handling for optional library_id filter
2026-03-06 01:52:33 -05:00
john-okeefe 9b3d8cc949 feat: implement collection library filter with WebSocket improvements and test coverage
This commit adds comprehensive functionality for filtering collections by library,
improves WebSocket real-time updates with user activity detection, and adds
extensive test coverage.

## Core Features

### Collection Library Filter
- Added library_id parameter to media-items search API
- Collections can now be filtered by specific library
- Toggle UI component for enabling/disabling library filter
- Default state is "checked" when library_id is present
- Consistent behavior across partial and fuzzy search modes

### WebSocket Auto-Reload Mitigation
- Added user activity detection to prevent disruptive page reloads
- Checks if user is actively typing in INPUT/TEXTAREA/SELECT elements
- Skips auto-reload when user is interacting with form elements
- Toast notifications still show for awareness
- Prevents data loss during editing operations

## Implementation Changes

### Backend
- internal/database/queries.sql.go: Added library filter support to search queries
- internal/handlers/media.go: Enhanced search with library_id parameter validation
- internal/handlers/collections.go: Updated collection handlers with library filtering
- internal/sync/websocket.go: Improved broadcast mechanism with user-scoped updates
- internal/router/frontend.go: Pass libraryID to collection templates

### Frontend
- templates/collections.templ: Added library filter toggle UI component
- web/src/collections.ts: TypeScript implementation with WebSocket integration
- templates/collections_templ.go: Generated template code

### Testing
- cmd/server/tests/search_test.go: Added TestCollectionSearchLibraryFilter
- cmd/server/tests/websocket_test.go: Added TestWebSocketUserScopedBroadcast
- New helper functions for creating libraries and media items via API
- Comprehensive test coverage for library filtering and user-scoped broadcasts

## API Documentation Updates

### Bruno Tests (Comprehensive Documentation)
- bruno/collections/*: Added detailed API documentation for all collection endpoints
- bruno/devices/*: Added device management and sync API documentation
- bruno/devices/kobo/api.yml: Kobo-specific sync protocol docs
- bruno/devices/koreader/api.yml: KOReader-specific sync protocol docs
- bruno/opds/*: Added OPDS feed and download endpoint documentation
- bruno/library/browse-folders.yml: Library folder browsing API docs

### New Bruno Tests
- bruno/media-items/Search All Libraries.yml: Test search without library filter
- bruno/media-items/Search Specific Library.yml: Test search with library filter
- bruno/media-items/Search Invalid Library ID.yml: Test error handling

## Documentation

- docs/developer/api/media-items/search_media_items.md: Updated with library_id parameter
- IMPLEMENTATION_COLLECTION_FIX.md: Comprehensive implementation guide with test scenarios

## Testing

### Integration Tests
- Library filter tests verify correct filtering across multiple libraries
- Invalid library_id tests ensure proper error handling
- WebSocket tests verify user-scoped broadcast behavior
- User A no longer receives User B's collection updates

### Manual Testing Scenarios
- Open collection in multiple tabs - updates propagate correctly
- Type in search box while another tab adds books - no disruptive reload
- Add/remove books from collection - toast notifications appear
- Toggle library filter - results update dynamically

## Technical Details

- WebSocket broadcasts are now user-scoped for privacy
- Active element detection uses tagName and contenteditable attributes
- Library ID validation uses UUID format checking
- Progressive enhancement maintained - page works without JavaScript
- All changes follow PROJECT_GUIDELINES.md conventions
- TypeScript only for frontend logic
- TailwindCSS only for styling
- Procedural programming style throughout

## Breaking Changes

None - all changes are additive and backward compatible.
2026-03-04 22:37:47 -05:00
john-okeefe eb2da1e05b fix: Change library ordering to oldest-first
Change library ordering in dropdown from DESC to ASC to display
libraries in creation order (oldest first).

Database changes in internal/database/queries/queries.sql:
- Modify GetUserLibraries query ORDER BY clause
  - Change from ORDER BY l.created_at DESC to ASC
  - Displays oldest libraries first in dropdown

This provides a more intuitive ordering where users see their
first-created libraries at the top of the list.
2026-03-01 00:29:27 -05:00
john-okeefe 209e9f2a3c feat: implement relative path storage and URL resolution for media files
- Add libraryService dependency to CollectionHandler and OPDSHandler for centralized path resolution
- Create internal/utils/mediaurl.go with ResolveMediaURL() function as single source of truth
- Update GetMediaItem and ListMediaItems handlers to return resolved URLs in API responses
- Update collection handlers (GetCollection, TestRules, PreviewCollection) to use resolved cover URLs
- Update progress handler (GetAllProgress) to use resolved cover URLs
- Add library_id to GetCollectionItems SQL query to enable URL resolution
- Refactor media scanner to store relative paths instead of absolute filesystem paths
- Add ResolveMediaPath() to LibraryService for resolving relative paths to absolute paths
- Add ServeFile endpoint at /uploads/library-:id/* for authenticated file serving
- Add MimeTypes map to library_service.go for consistent MIME type handling
- Update DownloadBook handler to use resolved filesystem paths
- Add getRelativePath() helper to MediaScanner for converting absolute to relative paths
- Use strings.EqualFold for case-insensitive path comparisons in zip extraction

This change enables the application to work with relative paths stored in the
database, making it portable across different server environments while
maintaining backward compatibility with existing absolute paths.
2026-02-27 16:51:44 -05:00
john-okeefe d802236874 scanner: fix library isolation, file mtime, force rescan, and deletion handling
Fix 1 - File modification time for created_at:
- Get file.ModTime() in processMediaFile and pass to CreateMediaItem
- Modified SQL INSERT to include created_at column

Fix 2 - Force rescan UPDATE instead of DELETE+INSERT:
- Changed force rescan logic to call updateMediaItem instead of delete + create
- Preserves created_at timestamp on force rescan

Fix 3 - GetMediaItemByFilePath filters by library_id:
- Added library_id to WHERE clause in SQL query
- Created GetMediaItemByFilePathAnyLibrary for cross-library lookups (KOReader)
- Added SetLibraryID method to MediaScanner
- Updated handler to call SetLibraryID for watch mode

Fix 4 - File deletion handling with persistent logging:
- Added fsnotify.Remove handler in WatchChanges
- Added orphan cleanup in ScanFolders after scan completes
- Created scanner_logger.go with daily log rotation (7 days)
- Logs to /app/logs/scanner-deletes-YYYY-MM-DD.log and scanner-errors-YYYY-MM-DD.log
- Individual deletes with enhanced safety logging

Note: Integration tests can now safely scan /app/uploads because
GetMediaItemByFilePath now filters by library_id, preventing
cross-library interference.
2026-02-26 16:39:42 -05:00
john-okeefe e3a3aa124f feat(api): consolidate user profile update endpoints
Add delete_user, reset_user_password, and update_user endpoints to replace
individual update operations. Update database schema to include deleted_at
column for soft deletion. Add DeleteUser, ResetUserPassword, and
UpdateUserAdmin queries. Update Querier with new methods for user management.
2026-02-22 01:57:42 -05:00
john-okeefe 1f80f6acfd feat(dashboard): add Phase 3 database queries for Carousel-style dashboard
Add SQL queries for dashboard functionality and system collections:

Dashboard Preferences Queries:
- GetDashboardPreferences: Fetch user preferences for a library
- UpsertDashboardPreferences: Create or update user dashboard preferences
- UpdateDashboardPreferences: Update existing preferences

Dashboard Collections Queries:
- GetSystemCollectionsForDashboard: Fetch system collections (user_id IS NULL)
- GetUserCollectionsForDashboard: Fetch user collections marked for dashboard
- DeleteUserSystemCollection: Delete user's copy of a system collection

System Collection Smart Queries:
- GetContinueReadingItems: Books with 0 < progress < 1
- GetRecentlyAddedItems: Newly added items to library
- GetRecentlyReadItems: Books with progress >= 1
- GetNotStartedItems: Books with progress = 0 or no record

Collection Management Queries:
- GetCollectionItemsForDashboard: Fetch collection items with excluded flag
- GetLibraryItems: Fetch all items in a library

These queries support the unified collections architecture where system
defaults and user-created sections are both collections with user_id
NULL for system-owned and NOT NULL for user-created.
2026-02-19 20:55:54 -05:00
john-okeefe 557f057621 fix(db): cast status to varchar in sync queue update for proper enum comparison 2026-02-14 21:37:38 -05:00
john-okeefe 2706ae52c1 refactor: remove Phase X terminology from source code comments
Remove planning document phase references from code comments:

app_test.go:
- Remove Phase 5 references from 8 test function comments

querier.go & queries.sql.go:
- Remove Phase 1, 2, 3, 4, 6 references from section headers
- Clean up week numbers (Weeks 5-6, Week 3-4, etc.)

queries.sql:
- Remove Phase 4 references from Kobo queries

kobo.go:
- Remove Phase 6 references from ContentId mapping comments

progress.go:
- Remove Phase 1 reference from route comment

media_scanner.go & media_scanner_library_type_test.go:
- Remove Phase 2 references from library type scanning comments

schema.sql:
- Remove Phase 1, 2, 3, 4, 5, 7 references from table/section comments
- Clean up: Format Detection, Progress Tracking, Device Registry,
  Sync Queue, Conflict Resolution, Reading History, Indexes, etc.

test_helpers.go:
- Remove Phase 6 reference from handler setup comment

These phase numbers were from internal planning documents and have no
meaning in the codebase. Removing them makes the code self-documenting.
2026-02-13 21:50:29 -05:00
john-okeefe 8321149957 test: add Bruno API test collections for device authentication
- Device token regeneration tests (success, forbidden, not found, unauthorized)
- OPDS authentication tests (Bearer token, query token)
- Kobo sync tests with token authentication
- Test various authentication methods and error cases
2026-02-13 12:12:17 -05:00
john-okeefe 363e747cb3 feat(db): add system settings queries and enhance user queries
System Settings Migration:
- Add GetSystemSetting query for single setting retrieval
- Add UpdateSystemSetting query for updating settings
- Add GetAllSystemSettings query for all settings
- Remove UpdateScanSettings and GetScanSettings (per-user queries)

User Query Enhancement:
- Add max_devices field to GetUser query
- Add device_count computed field to GetUser query
- Add max_devices field to ListUsers query
- Add device_count computed field to ListUsers query

These changes support:
1. System-wide scan settings instead of per-user settings
2. Users can now see their device count and limits
3. Admins can monitor device usage across all users
2026-02-09 20:09:48 -05:00
john-okeefe eed1ef37dc Change UpdateUserMaxDevices to return updated user record
- Change query from :exec to :one with RETURNING *
- Allows handler to detect when user doesn't exist
- Follows pattern established by UpdateMediaItem
- Required for 404 response on non-existent user

Related: Fix for TestUpdateUserMaxDevicesNonExistentUser
2026-02-09 15:46:57 -05:00
john-okeefe 701815c1bc docs: update documentation for media scanner naming
- Update development.md with new file names (scanner.go, media_scanner.go)
- Update API reference: "Book/ebook operations" -> "Media item operations"
- Remove historical migration comments from schema:
  - Simplify media_items table comment
  - Remove backward compatibility comments for reading_progress and media_ratings
  - Remove note about backward compatibility views
- Remove historical comment from queries.sql about ebook folders
2026-02-08 14:55:06 -05:00
john-okeefe 9e166c9670 chore(database): remove unused EbookNote backward compatibility functions
- Remove CreateEbookNote, UpdateEbookNote, DeleteEbookNote queries
- These were marked as backward compatibility but never used
- API uses CreateMediaNote, UpdateMediaNote, DeleteMediaNote instead
- Remove misleading backward compatibility comments
- Regenerate sqlc code
2026-02-08 14:28:08 -05:00
john-okeefe c606a7ffcc feat: add tags_search and contributors_search fields to database
Schema changes:
- Add tags_search TEXT[] column for case-insensitive, punctuation-free search
- Add contributors_search TEXT[] column for case-insensitive, punctuation-free search
- Create GIN indexes for fast array searches on both search fields

Query updates:
- CreateMediaItem: Include tags_search and contributors_search parameters
- UpdateMediaItem: Include tags_search and contributors_search parameters
- SearchMediaItems: Search against tags_search instead of tags
- SearchMediaItems: Search against contributors_search instead of contributors
- SearchMediaItemsFuzzy: Use tags_search and contributors_search for fuzzy matching
- Update ranking and priority logic to use search fields

Benefits:
- Case-insensitive search: "acme corp" finds "ACME CORP."
- Punctuation-agnostic search: "oreilly" finds "O'Reilly Media"
- Better UX: Users don't need to match exact casing or punctuation
- Improved performance with dedicated GIN indexes

Relates to Tags & Contributors Migration Phases 5 & 8
2026-02-08 11:05:18 -05:00
john-okeefe 516cec5a7f feat: migrate tags and contributors from TEXT to TEXT[] arrays
Convert tags and contributors columns from comma-separated strings to PostgreSQL
TEXT[] arrays for better data normalization and query performance.

Database Changes:
- schema.sql: Change tags/contributors from TEXT to TEXT[]
- schema.sql: Add GIN indexes for fast array searches
- queries.sql: Update search queries to use ANY() operator
- queries.sql: Update fuzzy search with unnest() for arrays

Generated Code (sqlc):
- models.go: Auto-generated with []string types for tags/contributors
- queries.sql.go: Auto-generated with proper array handling

Handler Changes:
- media.go: Update request structs to use []string for tags/contributors
- media.go: Remove pgtype.Text wrapping, use direct array assignment
- media.go: Add tag normalization in CreateMediaItemHandler
- collections.go: Update tags evaluation to join arrays for comparison
- collections.go: Add strings import for Join() function

Service Changes:
- ebook_scanner.go: Update EbookMetadata struct to use []string
- ebook_scanner.go: Remove string Join(), assign arrays directly
- collection_service.go: Update tags rule evaluation to join arrays
- collection_service.go: Add strings import

New Utilities:
- internal/utils/tags.go: Create NormalizeTags(), JoinTags(), SplitTags()
- Normalizes tags by trimming, lowercasing, removing duplicates/empties

API Documentation:
- bruno/media-items/Create Media Item.bru: Update examples to use arrays
- bruno/media-items/Update Media Item.bru: Update examples to use arrays
- Update docs: tags/contributors now array of string

Breaking Change:
- JSON format changes from "tags": "tag1,tag2" to "tags": ["tag1", "tag2"]
- Tests already use array format (no changes needed)

Benefits:
- GIN indexes enable faster array searches
- Normalization prevents data quality issues (case, duplicates)
- Array operations use PostgreSQL native operators (ANY, &&, unnest)
- Better separation of concerns (no string parsing in application)
2026-02-07 22:53:12 -05:00
john-okeefe 153b330dd1 Update database schema and queries: Bookmann → Bookhoard
Database changes:
- schema.sql: Update column name bookmann_uuid → bookhoard_uuid
- schema.sql: Update index names and example URLs
- queries.sql: Update all SQL queries to use bookhoard_uuid
- Update example configuration values

Part of project rename to Bookhoard.
2026-02-01 16:21:03 -05:00
john-okeefe d43e0105eb Rename project module and database references: Bookmann → Bookhoard
Core infrastructure changes:
- Update Go module name: bookmann → bookhoard
- Rename database schema references and comments
- Update database column names: bookmann_uuid → bookhoard_uuid
- Rename SQL query functions: GetDeviceCatalogByBookmannUUID → GetDeviceCatalogByBookhoardUUID
- Update configuration defaults

This is part 1 of the project rename to Bookhoard.
2026-02-01 16:11:47 -05:00
john-okeefe f23b6d35e9 feat(database): add analytics and book matching queries
Add analytics queries:
- GetUserReadingHistory: detailed reading history with device info
- GetUserDeviceUsage: device usage statistics (sync count, time spent)
- GetPopularBooks: most read books with completion rates

Add book matching queries:
- GetUnlinkedBookByID: fetch single unlinked book
- DeleteUnlinkedBook: remove resolved unlinked book
- ListUnresolvedUnlinkedBooks: paginated list of unresolved books
2026-02-01 12:15:33 -05:00
john-okeefe 63008001c6 feat(db): Add kobo_shelves and device catalog tables
- Add kobo_shelves table for Kobo device shelves management
- Add device_shelf_mappings for collection<->device shelf mappings
- Update device_catalogs with better ContentId tracking
- Add indexes for performance
2026-01-31 22:32:04 -05:00
john-okeefe 030d30a225 Add queue management API and database queries
- Add 7 new database queries for queue management
- GetStuckSyncQueueItems - Detect stuck items
- GetSyncQueueStats - Queue statistics
- GetNextRetryTime - Exponential backoff calc
- ListAllSyncQueueItems - Admin view
- IncrementSyncQueueAttempts - Retry counter
- Add queue handler with 7 REST endpoints
- GET /api/queue/devices/:id/stats - Queue statistics
- GET /api/queue/devices/:id/items - List device queue
- POST /api/queue/items/:id/retry - Retry failed item
- DELETE /api/queue/items/:id - Delete queue item
- DELETE /api/queue/devices/:id/clear - Clear device queue
- GET /api/queue/items - List all items (admin)
- Add full user/admin access control
2026-01-31 13:06:12 -05:00
john-okeefe 2d2d643873 Add sync conflict detection and resolution system
Implement conflict detection for concurrent reading progress updates from different devices. Adds conflict management endpoints for listing, viewing, and resolving conflicts.

- Add ConflictHandler with CRUD endpoints for conflict management
- Implement automatic conflict detection in KOReader progress updates
- Add WebSocket broadcast for real-time conflict notifications
- Add database query for listing user conflicts by status
- Add integration tests and Bruno API test collection
2026-01-31 11:45:52 -05:00
john-okeefe cd240d3054 feat: add Kobo shelf and entitlement SQL queries 2026-01-31 00:29:38 -05:00
john-okeefe 4c01c0d12e Phase 3 Week 7: Add KOReader bulk sync function to database schema
- Add bulk_update_progress_from_koreader() function for batch processing
- Handles progress, annotations, and conflict detection
- Returns success/failure status for each book
- Supports device matching by UUID, file path, or title/author
- Implements automatic conflict detection for concurrent syncs
- Part of Phase 3 KOReader Integration implementation
2026-01-30 20:54:51 -05:00
john-okeefe 23ad70158c Phase 2 Week 5: Device Registration & Management
Implement device registration and management system for universal sync.

Database Changes:
- Add device queries to queries.sql (CRUD operations, registration, auth)
- Add sync queue management queries
- Add conflict resolution queries
- Regenerate sqlc models with new device-related types

Device Handler (devices.go):
- InitiateRegistration: Start device registration with auth URL and QR code
- CheckRegistrationStatus: Poll for registration approval
- ListDevices: Get all devices for current user
- GetDevice: Get specific device details
- UpdateDevice: Update device settings (name, sync settings, frequency)
- DeleteDevice: Remove device from account
- ApproveDevice: User approves device registration via web
- RejectDevice: Reject pending device registration
- ListPendingRegistrations: Show all pending registrations
- generateDeviceToken: Generate secure Bearer token for devices

Device Authentication Middleware (device_auth.go):
- Authenticate: Validate device Bearer tokens
- RequirePermission: Check device permissions by type
- hasPermission: Define permissions per device type
- UpdateLastSeen: Auto-update device last_seen timestamp

Configuration:
- Add BaseURL field to Config for device setup URLs

API Endpoints:
POST /api/devices/register - Initiate device registration
POST /api/devices/register/status - Check registration status
GET /api/devices/approve/:id - Approve device (web UI)
POST /api/devices/reject/:id - Reject device
GET /api/devices - List user's devices
GET /api/devices/:id - Get device details
PUT /api/devices/:id - Update device settings
DELETE /api/devices/:id - Delete device
GET /api/devices/pending - List pending registrations

Bruno API Collection:
- Initiate Device Registration
- Check Registration Status
- List Devices
- Get Device
- Update Device
- Delete Device

Dependencies:
- github.com/skip2/go-qrcode for QR code generation

Device Types Supported:
- koreader: Calibre-compatible sync
- kobo: Kobo sync protocol
- web: Web interface
- mobile: Mobile apps

Device Permissions:
- sync:progress
- sync:annotations
- sync:metadata
- device:manage (web only)
2026-01-30 16:45:10 -05:00
john-okeefe bb3c32c59f Phase 1 Week 2: Format detection and progress conversion engine
- Add internal/sync package with format detection
- FormatGroup types: reflowable, fixed_layout, comic_archive
- DetectFormatGroup() function based on mimetype and file extension
- MimeType mappings for common ebook formats
- Progress conversion engine with:
  - ConvertProgress() between format groups
  - Extract percentage from various progress formats
  - PageToPercentage / PercentageToPage helpers
  - CharacterToPercentage / PercentageToCharacter helpers
  - MergeProgress() with 'max progress wins' strategy
  - FormatProgressForDisplay() for UI rendering
- Add sqlc queries for format detection and progress updates
- BulkUpdateFormatGroups query for auto-format detection
- GetUniversalProgress query with all location references
- UpdateUniversalProgress query with device sync metadata
- ReadingHistory queries for session tracking
2026-01-30 16:07:00 -05:00
john-okeefe 37b8380533 fix: sync database queries with enhanced media-items schema
- Add missing enhanced fields to CreateMediaItem INSERT statement
- Add missing enhanced fields to UpdateMediaItem UPDATE statement
- Include language, edition, page_count, genre, copyright_year
- Include integration fields: goodreads_id, openlibrary_id, google_books_id
- Add corresponding indexes for enhanced fields
- Add column comments for better documentation
- Resolves schema-query mismatch causing field removal cycles
2026-01-30 14:11:52 -05:00
john-okeefe fc71c2ef76 fix: database queries and schema sync
- Remove references to non-existent columns (language, edition, page_count, etc.)
- Fix CreateMediaItem and UpdateMediaItem queries
- Remove normalize_isbn() function calls (moved to Go code)
- Regenerate database code with sqlc generate
- Add GetLibraryByFolder method to queries

Fixes compiler error: s.db.GetLibraryByFolder undefined
2026-01-30 13:51:44 -05:00
john-okeefe f96044b6c7 refactor: remove ebook system, unify on media-items
Phase 1-3: Database layer cleanup
- Remove 5 backward compatibility VIEWs (ebooks, ebook_ratings, etc.)
- Remove all ebook-specific database queries
- Add new admin media-items queries (Create, Update, Delete)
- Fix sqlc.yaml to point to schema.sql file
- Regenerate database code successfully

Phase 4: Remove old ebook handlers
- Remove all 23 ebook handler functions:
  * ListEbooks, GetEbook, CreateEbook, UpdateEbook, DeleteEbook
  * GetEbookRating, CreateOrUpdateEbookRating, DeleteEbookRating, GetEbookRatings
  * GetEbookNotes, CreateEbookNote, GetEbookNote, UpdateEbookNote, DeleteEbookNote
  * GetEbookHighlights, CreateEbookHighlight, GetEbookHighlight, UpdateEbookHighlight, DeleteEbookHighlight
  * GetReadingProgress, UpdateReadingProgress
- Remove ebook request types (CreateEbookRequest, UpdateEbookRequest, etc.)

Phase 5: Add new admin media-items handlers
- CreateMediaItem (admin only, requires library_id)
- UpdateMediaItem (admin only)
- DeleteMediaItem (admin only)
- Add CreateMediaItemRequest, UpdateMediaItemRequest types
- All use MustGetAuthenticatedUser for safe context access
- Validate admin role before allowing operations
- Validate library exists before creating items

Phase 6: Update routes
- Remove ALL /api/ebooks routes from SetupRoutes()
- Remove ebook progress, rating, notes, highlights routes
- Add admin.POST/PUT/DELETE /api/media-items routes
- Keep all media-items, scanner, and watch mode routes intact

Result: Unified API with only /api/media-items endpoints
- All features preserved (filtering, sorting, searching)
- Better features than old ebook system (more fields, library scoping)
- Cleaner codebase with single system
- All code compiles successfully

Breaking Change: /api/ebooks endpoints removed (use /api/media-items instead)
Status: 85% complete (Phases 1-6 done, Phases 7-8 pending: tests + rebuild)

Tests: Need update (rename Ebooks → MediaItems, update API paths)
Build: Need rebuild with clean cache
2026-01-30 10:03:13 -05:00
john-okeefe 3b2075fc70 Phase 1: Add enhanced database fields and sorting
- Add 9 new fields to media_items table (language, edition, page_count, goodreads_id, openlibrary_id, google_books_id, copyright_year, genre, subjects)
- Add indexes for new fields (language, genre, page_count, copyright_year, series_order, date_published)
- Add ListMediaItemsSorted SQL query for dynamic sorting
- Update ListMediaItems handler to process sort parameter
- Support 16 sorting options (title, author, created_at, date_published, copyright_year, page_count, genre, series)
- Add /api/media-items/filtered endpoint for advanced filtering
- Register new filtered endpoint in routes
2026-01-30 08:32:49 -05:00
john-okeefe 3145f64a66 feat: add search queries with partial matching and fuzzy fallback
- Add SearchMediaItems query with ILIKE partial matching
- Add SearchMediaItemsFuzzy query with word_similarity()
- Use sqlc.narg() for named parameters (search_pattern, search_query)
- Rank results by relevance: title > author > series > tags
- Fuzzy threshold set to 0.3 for word_similarity
- Generated Go models with proper parameter types
2026-01-29 20:20:30 -05:00
john-okeefe 66f1eb11a0 feat(ebooks): add ISBN normalization and graceful library requirement handling
- Increase ISBN column from VARCHAR(13) to VARCHAR(17) to support ISBN-13 with hyphens
- Add normalize_isbn() database function to automatically remove hyphens and spaces
- Create trigger to auto-normalize ISBNs on INSERT/UPDATE operations
- Update all Ebook and MediaItem queries to use ISBN normalization
- Add GetEbookLibraryID query to check for existing ebook libraries
- Add graceful error handling when no ebook library exists
- Return helpful error message: 'no ebook library found. Please create an ebook library first'
- Create comprehensive tests for ISBN normalization and library selection
- Add Bruno test files for various ISBN formats and error scenarios
- Update documentation with ISBN normalization details
2026-01-29 10:52:14 -05:00
john-okeefe 311361a2ed feat(security): add password complexity validator
- Implement strict password requirements:
  - Minimum 8 characters
  - At least one uppercase letter
  - At least one lowercase letter
  - At least one number
  - At least one special character
- Add custom validator for Echo integration
- Add GetPasswordRequirements helper function
- Add ValidatePassword function for manual validation
2026-01-29 09:23:34 -05:00
john-okeefe 29669e2fa3 feat: add database models and queries for annotations
- Add MediaNotes and MediaHighlights model structs with pgx v5 types
- Add EbookNotes and EbookHighlights for backward compatibility
- Add complete CRUD SQL queries for notes and highlights
- Add database connection pool function using pgx v5
- Generate sqlc code for new annotation functionality
2026-01-28 15:42:34 -05:00
john-okeefe 57e545cbcb fix: correct SQL syntax for ebook rating creation
- Fix VALUES clause in CreateEbookRating query
- Remove invalid SELECT that caused SQL syntax error
- Use proper INSERT VALUES (, , ) syntax for pgx v5
- Ensure compatibility with existing code generation

Resolves database syntax error while maintaining backward compatibility
2026-01-28 11:28:20 -05:00
john-okeefe dbc3590cad feat: implement library system database schema
- Add library_types table with ebooks, comics, manga types
- Add libraries table for multiple library support
- Add library_folders table for multi-folder libraries
- Add library_visibility table for user access control
- Add media_items table replacing ebooks for broader media support
- Create backward compatibility views for existing API
- Implement library service with type validation and file extension handling
- Support modular extension for future media types

Manga type includes cbz/cbr archives as requested
2026-01-28 11:00:06 -05:00