60 Commits
Author SHA1 Message Date
john-okeefe 5c5593644d fix(media): gate file serving by library visibility; proper download URL
ServeFile previously authenticated only ("any logged-in user") and never
checked that the user can actually see the library owning the file, so
knowing a library UUID + path was enough to fetch content from hidden
libraries. Library visibility is the permission model - the library is
what grants access to its media.

- ServeFile now resolves two URL forms through one flow:
  /uploads/library-{id}/{path}   (covers, reader files)
  /api/media-items/{id}/download (explicit book download, new)
  The item form looks up the media item, derives its library and file
  path, and adds a Content-Disposition attachment header.
- Both forms enforce GetUserVisibleLibraries for the authenticated
  user, mirroring the OPDS download handler (403 when not visible).
- Deleted the dead MediaHandler.DownloadBook handler (never routed).

Also widen media_highlights.start_position/end_position from
VARCHAR(100) to TEXT: the API handlers validate up to 1000 characters
(full Readium locators, KOReader CRE xpointers) but the column rejected
anything longer at the database layer. Metadata-only change applied
idempotently at startup; existing rows are untouched.

Verified against the running server: download 200 + attachment headers
+ epub bytes, unauthenticated 401, user hidden from the library 403 on
both URL forms, visible user 200, covers unchanged, and a 334-char
locator JSON now round-trips through the highlights API.
2026-08-30 11:44:36 -04:00
john-okeefe 0c39e04e4a feat(db): hash_conflicts table and backfill/conflict queries
Content duplicates (same library + file_sha256 at different paths,
e.g. the same book imported twice under two names on a preexisting
database) cannot be auto-collapsed the way path duplicates were:
keeping both copies may be intentional. Surface them for an explicit
admin decision instead.

Schema:
- new hash_conflicts table keyed (library_id, file_sha256) with a
  status/resolution lifecycle: 'pending' until an admin resolves via
  'keep_all' or 'kept:<uuid>' (which copy was kept after merging)
- resolution is VARCHAR(50) - 'kept:<uuid>' is 41 chars; include a
  widening ALTER for databases created with the initial 30-char width
- resolution/resolved_by/resolved_at record who decided what and when

Queries:
- ListMediaItemsMissingHash: items imported before hashing existed
  (file_sha256 IS NULL), ordered oldest-first for the backfill pass
- FindHashConflictGroups: the content-duplicate group detection
  (GROUP BY library_id, file_sha256 HAVING COUNT(*) > 1)
- ListMediaItemsBySHA256AndLibrary: full membership of one group
- CreateHashConflict: upsert with DO NOTHING so already-tracked groups
  are untouched - critical behavior: a group an admin resolved as
  'keep both' is never re-flagged by later sweeps
- ListPendingHashConflicts: admin listing with library name and live
  item counts (items may have been deleted since flagging)
- GetHashConflict / ResolveHashConflict: lifecycle
- GetMediaItemUsageCounts: per-item progress/highlight/bookmark/note/
  collection counts so the admin can make an informed keep choice
- ReparentMediaItemChildren: sqlc binding for the existing
  reparent_media_item_children() migration function, used to merge a
  losing copy's child rows into the kept copy
2026-08-14 08:52:05 -04:00
john-okeefe 9b171a0060 fix(scanner): prevent duplicate media item imports
A read-then-write race in processMediaFile allowed the same file to be
imported twice: two concurrent scan jobs (startup scan, fsnotify dirty-
directory scan, periodic backup poll, or a manual scan each run on
separate worker goroutines with separate MediaScanner instances) could
both SELECT 'not found' and both INSERT. There was no transaction, no
row lock, no unique constraint on (library_id, file_path), and no
ON CONFLICT clause, so nothing stopped the double insert. Observed in
production as two identical 'Head First SQL' rows created in the same
second (same sha256, size, path, library).

Database enforcement:
- schema.sql: add UNIQUE(library_id, file_path) constraint, guarded so
  re-runs don't error
- schema.sql: add self-healing migration that runs on every startup -
  dedup_media_items_by_path() collapses existing path-duplicates and
  reparent_media_item_children() moves all child rows (progress,
  highlights, bookmarks, notes, collections, formats, aliases, kobo
  entitlements, etc.) onto a survivor before deleting losers, so the
  constraint applies cleanly on already-duplicated servers without
  losing reading history. Survivor picks the row with the most user
  data, ties broken by lowest id
- CreateMediaItem: upsert via ON CONFLICT (library_id, file_path) DO
  UPDATE so concurrent inserts collapse to one row and return it
- CreateMediaItemFormat: upsert via ON CONFLICT (media_item_id,
  format_type), closing the same race on format rows

Application-level guards:
- media_scanner processMediaFile: after computing the file hash, check
  GetMediaItemBySHA256AndLibrary (new query) and treat the file as
  existing when identical content is already in the library under a
  different path (content dedup, library-scoped so multi-library
  setups still work)

Ops tooling:
- scripts/dedup_media_items.sql: standalone idempotent maintenance
  script with a dry-run report (path + content duplicate groups, child
  row counts) and transactional cleanup, for servers that prefer to
  dedup manually before upgrading

Verified against the live database: the duplicate pair was collapsed
(reading_progress preserved on the survivor), schema.sql re-runs are a
no-op, and the constraint is in place with 62 unique books remaining.
2026-08-14 08:18:36 -04:00
john-okeefe bc47450653 feat(db): typed tunable system settings + SettingsRegistry
Add a typed, cached registry over the system_settings table so that
values which used to be hardcoded Go literals can be changed at runtime.

Schema (database/schema/schema.sql):
- Extend system_settings with setting_type, min_value, max_value,
  requires_restart, and category columns (all ADD COLUMN IF NOT EXISTS,
  nullable for backward compat with the original three rows).
- Seed rows for every tunable: session duration, password rules,
  login lockout, auth/device rate limits, OPDS page size, tombstone TTL,
  conversion cache TTL, sync queue interval/batch, and worker pool
  size/cap. Seed values equal the previous hardcoded literals, so
  behavior is unchanged on upgrade. ON CONFLICT DO NOTHING preserves
  any admin-modified values.

Queries (queries.sql):
- Add UpsertSystemSetting (RETURNING *) so new keys without a seed row
  can still be written through the API.
- Add GetSystemSettingFull + GetAllSystemSettingsFull returning the
  full typed row.
- Refactor CleanupExpiredRefreshTokens to take the retention window as
  a parameter (make_interval(secs => $1)) instead of the INTERVAL '7
  days' literal, so it can follow a configurable session duration.

Registry (internal/database/settings_registry.go):
- SettingsRegistry holds an in-memory cache of all known settings,
  populated by Load at startup and refreshed by Reload on writes.
- Typed domain getters (SessionDuration, PasswordRules, DeviceRateLimits,
  TombstoneTTL, OpdsPageSize, ConversionCacheTTL, SyncQueueConfig,
  WorkerPoolConfig, LoginLockout, AuthRateLimit, ...) with compiled-in
  fallback defaults and min/max clamping, so a corrupt or missing row
  can never break the app.
- SettingDefaults is the single source of truth for keys, types, bounds,
  and human descriptions; All() exposes metadata + current values for
  the admin UI/API.

The registry lives in the database package (rather than its own
internal/settings package) because a quirk in this custom go1.26.5
toolchain prevented the large handlers package from importing any
newly-created package; every consumer already imports database.

Tests: settings_registry_test.go covers default validity per type,
int clamping at both bounds, garbage-value fallback, and unknown-key
lookup.
2026-08-10 08:00:52 -04:00
john-okeefe 4716790564 fix: OPDS base_url placeholder bug + setup gate requires base_url
Three bugs fixed:

1. Schema seeded base_url with fake placeholder 'bookhoard.example.com'.
   Removed seed; startup now seeds from BASE_URL env var only if DB row
   is empty (admin changes persist across restarts). One-time UPDATE
   clears the placeholder in existing installs.

2. config.GetBaseURL() had a broken type assertion (local SystemConfigRow
   vs database.SystemConfig) that always failed, returning . Admin panel
   showed env var fallback instead of actual DB value. Fixed with a
   function-type getter that properly wraps the DB query.

3. OPDS handler read base_url only from DB with no fallback. When DB had
   the placeholder, all feed links pointed to an unreachable domain,
   breaking KOReader search/download. Added deriveBaseURL() helper that
   falls back to the request Host/scheme when DB value is empty.

Setup gate improvements:
- isSetupComplete now requires both admin user AND non-empty base_url
- Setup middleware no longer exempts all /api/ routes; only allows
  /api/auth/register, /api/auth/login, /api/system/config before setup
  is complete. All other API routes get 503.
- Cache invalidated when base_url is saved via admin settings

Dev workflow:
- New bruno/NewDevDBSetup/SetBaseUrl.yml for dev DB setup
- NewDB.sh runs SetBaseUrl between RegisterUser and CreateEbookLibrary
2026-08-06 13:02:35 -04:00
john-okeefe 2a15effc3e feat(db): add annotation sync schema, queries, and tombstone support
Add migration columns to media_highlights, media_notes, and media_bookmarks
for cross-device annotation sync:

- dedup_key: SHA-1 of normalized selection text + bucketed position, used
  as the stable cross-device identity for annotations
- last_modified_at / last_modified_source: edit clock for LWW resolution
  and cross-source conflict detection
- deleted / deleted_at: sticky tombstone columns for delete-wins semantics
  with a 30-day TTL before rows are physically purged
- note_text on highlights: stores attached notes from KOReader entries that
  have both selected text and a user note
- Location columns on bookmarks (cfi_position, percentage_location,
  epubcfi_location, chapter_reference, paragraph_reference)
- device_sync_data JSONB on all three tables: stores per-device native
  identifiers (e.g. KOReader pos0/datetime, Kobo bookmark_id) so each
  device can locate and manipulate its own copy of an annotation

Add partial unique indexes on (user_id, media_item_id, dedup_key) where
deleted = FALSE to enforce one active annotation per dedup key.

Add tombstone purge indexes on (deleted, deleted_at) for efficient GC.

New queries:
- GetByDedupKey for all three tables (returns active or most-recent tombstone)
- CreateFull / UpdateForSync for all three tables (populate sync columns)
- TombstoneByDedupKey / TombstoneByID for all three tables
- PurgeExpired* for all three tables (GC past TTL)
- GetActiveAnnotationsForBook (filtered union of highlights + notes)
- GetTombstonedAnnotationsForBook (union of all 3 deleted within TTL)
- GetMediaBookmarks with deleted filter
- GetMediaBookmark (singular) with deleted filter
- Added deleted=FALSE filter to GetMediaHighlights, GetAnnotationsForBook
- CreateAutoResolvedSyncConflict (INSERT with resolution_status='auto_resolved')
2026-07-29 14:48:30 -04:00
john-okeefe 980aaee0d9 refactor(setup): derive setup-complete status from admin user count
Setup completion was previously tracked by a manually-flipped setup_complete row in system_settings, written via a JWT-protected PUT /api/setup/complete endpoint. This meant any admin user created outside the setup wizard (future CLI, seed scripts, direct DB inserts) would not flip the switch, leaving the app stuck redirecting to /setup.

The trigger is now derived from real data: setup is complete iff at least one admin user exists. This is self-correcting regardless of how users are created, and re-engages setup automatically if all admins are ever removed.

Changes:
- Add internal/setupstatus package with IsSetupComplete() (queries CountAdmins, 10s in-memory cache, fails open on DB error) and Invalidate() to clear the cache. Uses an AdminCounter interface to avoid importing the database package.
- Add CountAdmins sqlc query (SELECT COUNT(*) FROM users WHERE role = 'admin') and regenerate.
- Rewire router/setup.go isSetupComplete() to delegate to setupstatus; drop the old setup_complete setting read, cache vars, and the PUT /api/setup/complete route.
- Call setupstatus.Invalidate() in the auth handler after CreateUser, UpdateUserRole, and DeleteUser so the cache reflects admin-count changes immediately.
- Align first-user promotion in Register to key off !adminExists instead of len(users) == 0, so the two checks cannot diverge.
- Remove the now-dead SetSetupComplete/GetSetupStatus handlers.
- Drop the setup_complete seed row from schema.sql.
- Remove the apiPut('/setup/complete') call from the setup wizard finishSetup(); the admin account created in submitAdmin already marks setup complete server-side.
2026-07-29 11:08:18 -04:00
john-okeefe cea8e64da2 feat(api): add setup_complete system setting and status handlers
Add 'setup_complete' boolean to the system_settings seed data (defaults
to false) so fresh databases start in the unconfigured state.

Add two new handlers to SystemSettingsHandler:
- SetSetupComplete: marks setup_complete=true in the database
- GetSetupStatus: reads the current setup_complete value, returns
  {setup_complete: bool} JSON response, defaults to false if the
  setting row is missing or unparseable
2026-06-06 00:03:52 -04:00
john-okeefe c82f20c3f2 feat(db): add context_text column to reading_progress
Stores surrounding text (~100 chars) at the reader's current position.
Used as fallback for CFI resolution when converting between epubcfi
and CREngine XPointer formats.

Updates:
- schema.sql: add context_text TEXT column, update stored procedure
- queries.sql: add context_text to GetUniversalProgress and
  UpdateUniversalProgress queries
- Regenerate sqlc Go code (models.go, querier.go, queries.sql.go)
2026-06-02 19:44:31 -04:00
john-okeefe d0460885ff feat(db): add imported_at column to media_items for accurate "Recently Added" sorting
The created_at column stores file modification time (intentional for preserving
original metadata), but this makes 'Recently Added' sorting unreliable for
imported files. Add imported_at column that records the actual database insert
timestamp.

Changes:
- Add imported_at TIMESTAMPTZ column to media_items (nullable)
- Update GetRecentlyAddedItems to sort by imported_at DESC NULLS LAST first
- Add ReassignLibraries and ReassignMediaItems queries for admin deletion handover
- Add SyncLibraryTypeExtensions query for startup extension sync
- Update all media_items SELECT queries to include imported_at column
2026-05-16 19:30:27 -04:00
john-okeefe 607ce8ce3a Merge branch 'main' of ssh://git.linuxhg.com:2222/Bookhoard/bookhoard 2026-05-01 14:31:54 -04:00
john-okeefe ff0517d038 fix(library): sync allowed extensions across service, schema, and tests
Add .epub and .pdf to comics type, add .pdf to manga type, and ensure
avif/tiff/tif extensions are consistently included in manga across all
layers. Update test fixtures to match the canonical extension lists.
2026-05-01 14:31:17 -04:00
john-okeefe caf50ade31 Add timezone support to database schema and queries
- Add timezone column (VARCHAR(50) DEFAULT 'UTC') to users table
- Add default_timezone row to system_settings seed data
- Add idx_users_timezone index for user timezone lookups
- Add UpdateUserTimezone and GetSystemTimezone queries
- Regenerate sqlc code (models, querier, queries.sql.go)
- Reuse existing UpdateSystemSetting for system timezone updates
  instead of creating a redundant UpdateSystemTimezone query
2026-04-27 21:30:37 -04:00
john-okeefe b912a037ff fix(database): correct mismatched parentheses in detect_fixed_layout_epub
The string_to_array call in detect_fixed_layout_epub() had an extra
closing parenthesis after the '<img' delimiter, causing a SQL syntax
error that prevented the database container from initializing:

  IF array_length(string_to_array(opf_content, '<img')), 1) - 1 > 50

Fixed to:

  IF array_length(string_to_array(opf_content, '<img'), 1) - 1 > 50
2026-04-19 14:27:04 -04:00
john-okeefe 7ac86dafa9 feat(schema): Add processing_issues table for tracking media validation problems
- Add processing_issues table to track media items that cannot be properly processed in their assigned library
- Include fields for issue type, description, severity, and resolution status
- Add indexes for efficient querying by library and severity
- Support tracking format mismatches and other processing problems
- Unique constraint on media_item_id and issue_type to prevent duplicates
2026-04-12 20:43:42 -04:00
john-okeefe cf8c690474 Add library_type_name column to media_items with auto-population trigger
- Add library_type_name VARCHAR(50) column to media_items table in schema.sql
- Create PostgreSQL trigger 'set_library_type_name_on_insert' that automatically
  populates library_type_name by joining libraries table with library_types on insert
- Add library_type_name parameter to CreateMediaItem SQL query
- Update Go models (models.go) to include LibraryTypeName field
- Add UpdateMediaItemChapterMetadata query method to querier.go
- Regenerate queries.sql.go with sqlc

This allows media items to store their library type (e.g., 'Books', 'Comics', 'Manga')
at the database level, enabling filtering and display without needing additional joins.
2026-04-04 22:49:07 -04:00
john-okeefe dd621956b9 database: change reading_speed columns from DECIMAL to REAL
- Changed words_per_minute, pages_per_minute, total_reading_minutes from DECIMAL(6,2) / DECIMAL(8,2) to REAL
- REAL (pgtype.Float4) is sufficient for reading statistics and simplifies Go code
- No precision loss for typical reading speed values (200-400 wpm, 0.5-3.0 pages/min)
2026-04-03 16:52:53 -04:00
john-okeefe 0667cad8a9 feat: add reader infrastructure - Phase 0 database schema and queries
Implement Phase 0 prerequisites for reader functionality including
database schema, SQL queries, and frontend dependencies.

## Database Schema (5 New Tables + 1 Column Addition)

### New Tables Added:
1. **panel_data** - Comic/manga panel detection results
   - Stores detected panel boundaries (x, y, width, height)
   - Supports grid, ML, and manual detection methods
   - JSONB storage for flexible panel structures

2. **reading_speed** - User reading speed statistics
   - Tracks pages per minute and total reading time
   - Per-user per-media-item tracking
   - Enables progress estimation and analytics

3. **dictionary_cache** - Offline dictionary word definitions
   - Caches external dictionary lookups
   - Reduces API calls and improves performance
   - Supports offline reading functionality

4. **reader_settings** - User reader preferences (per-user)
   - Stores typography, theme, and display settings
   - JSONB storage for flexible configuration
   - Per-user customization (fonts, margins, themes)

5. **media_bookmarks** - Enhanced bookmarks with chapter/CFI support
   - Unified bookmarking for ebooks, comics, manga, PDFs
   - Supports page_number, chapter_number, and epubcfi_position
   - Includes notes field for annotations
   - Unique constraint on (media_item_id, user_id, title)

### Column Addition:
- **media_items.chapter_metadata** (JSONB) - Caches detected chapter structure
  - Stores TOC/chapter detection results
  - Prevents re-parsing files on every read
  - Populated by ReaderService.DetectChapters()

## Database Queries (12 New Queries)

Added queries for all reader functionality:
- Panel data: GetPanelData, UpsertPanelData
- Reading speed: GetReadingSpeed, CreateReadingSpeed, UpdateReadingSpeed
- Dictionary: GetDictionaryEntry, CreateDictionaryEntry, UpdateDictionaryAccessed
- Settings: GetReaderSettings, UpsertReaderSettings
- Bookmarks: GetMediaBookmarks, CreateMediaBookmark, DeleteMediaBookmark, UpdateMediaBookmark

## Frontend Dependencies

Added to package.json:
- jszip@^3.10.1 - EPUB/comic archive parsing (client-side)
- pdfjs-dist@^3.11.174 - PDF rendering library (Mozilla PDF.js)

## Generated Code

Ran `sqlc generate` to regenerate:
- models.go - Go structs for new tables (55 lines added)
- querier.go - Database query methods (14 lines added)
- queries.sql.go - Compiled SQL queries (504 lines added)

## Implementation Status

Phase 0 prerequisites now complete:
 Database schema (5 tables + 1 column)
 SQL queries (12 queries)
 Frontend dependencies (2 packages)
 Generated Go code (sqlc)
 Database recreated with new schema

Ready for Phase 1: Infrastructure & Basic Reader implementation.

Related to: Universal web reader for ebooks, comics, manga, PDFs
2026-04-02 21:01:45 -04:00
john-okeefe 54559a6e91 schema: change community_rating to DOUBLE PRECISION for simpler type mapping
Change community_rating from DECIMAL(3,1) to DOUBLE PRECISION to:
- Eliminate awkward pgtype.Numeric conversion in Go code
- Enable direct pgtype.Float8 mapping from ComicInfo.xml float64
- Simplify code by matching natural types (XML float64 → PostgreSQL DOUBLE PRECISION → Go pgtype.Float8)
- Remove need for string formatting and Scan() method calls

The floating-point precision error (< 0.00001%) is negligible for 0-10 rating scale.
This simplifies Phase 4 implementation significantly.

Column comment updated to reflect DOUBLE PRECISION type.

Relates to: Phase 1 database schema changes for comic metadata support
2026-03-29 21:12:18 -04:00
john-okeefe 2ffd8e032b schema: add comic metadata and reading direction support
Add 14 new columns to media_items table for comprehensive comic and manga
metadata support, including reading direction fields and universal metadata
that applies to all media formats.

New Columns:
- Reading direction: manga_type (raw ComicInfo.xml field), reading_direction (computed)
- Universal series: series_count, volume (apply to ebooks, audiobooks, comics)
- Publisher info: imprint, age_rating (all formats)
- Comic-specific: story_arc, is_black_and_white, alternate_info, scan_information, summary
- Additional metadata: metadata_notes, community_rating, web_url

Constraints:
- manga_type CHECK: unknown, no, yes, yes_and_right_to_left
- reading_direction CHECK: auto, ltr, rtl, vertical

Indexes (8 new):
- idx_media_items_reading_direction, idx_media_items_manga_type
- idx_media_items_story_arc, idx_media_items_imprint
- idx_media_items_age_rating, idx_media_items_series_count, idx_media_items_volume
- idx_media_items_alternate_info_gin (GIN index for JSONB queries)

Documentation:
- Added COMMENT ON COLUMN for all 14 new fields
- Distinctions between comic-specific and universal fields clearly documented

This supports the ComicInfo.xml v2.0 standard with 29 fields and enables
proper reading direction detection for manga, webtoons, and Western comics.

Part of Phase 1: Database Schema Changes
Implementation: IMPLEMENTATION_PLAN_MERGE_METADATA_READING_DIRECTION.md
2026-03-29 19:11:48 -04:00
john-okeefe 0e11c9263c fix: add unique constraint for saved filter names and fix search sort ordering
Database changes:
- Add unique index on saved_filters(user_id, name, resource_type)
  Prevents duplicate filter names while allowing same name across
  different users or different resource types

Search functionality fix:
- Remove DISTINCT ON (mi.id) from SearchMediaItemsUnified query
- Remove mi.id from ORDER BY clause (was required by DISTINCT ON)
- This allows user-selected sort field to be primary sort criteria
- Previously results were always sorted by ID first, making sort
  dropdown ineffective
- Relevance score and title remain as fallback sorts

This fixes the sort dropdown functionality on the bookshelf page
where changing the sort option appeared to have no effect.
2026-03-28 00:46:03 -04:00
john-okeefe e6bca1457c fix: add pg_trgm extension to database schema
Adds pg_trgm extension to enable GIN indexes for fuzzy text
search functionality. This extension provides trigram matching
required by word_similarity() function used in unified search.

Resolves container startup failures when GIN indexes with gin_trgm_ops
are created without the extension being loaded.
2026-03-24 16:47:16 -04:00
john-okeefe 7b49ab253f perf: add GIN indexes for pg_trgm fuzzy search optimization
Add GIN indexes with gin_trgm_ops for text fields used in fuzzy search:
- author, title, series, genre, language fields

These indexes significantly improve performance of word_similarity()
queries used in the unified search implementation. pg_trgm extension
must already be enabled for these indexes to function.

Performance impact: O(n) sequential scans become O(log n) index scans
for fuzzy text search operations.
2026-03-22 20:35:01 -04:00
john-okeefe e17a96123f feat(db): add saved_filters table and CRUD operations
Add database schema and SQL queries for generic saved filters system
that allows users to save custom filter presets for any resource type.

Database Schema:
- Add saved_filters table with user_id, name, resource_type, filters (JSONB)
- Create composite index on (user_id, resource_type) for efficient lookups
- Create index on (user_id, name) for future name search feature
- Add update_updated_at_column() trigger to auto-update timestamps
- Make trigger creation idempotent with DROP TRIGGER IF EXISTS

SQL Queries (5 new queries):
- GetSavedFilters: List all filters for user + resource type
- GetSavedFilterByID: Retrieve single filter by ID
- CreateSavedFilter: Create new saved filter
- UpdateSavedFilter: Update filter name/criteria
- DeleteSavedFilter: Remove saved filter

Design Decisions:
- Generic resource_type field supports any resource (media-items, collections, devices)
- JSONB filters field allows flexible schema without migrations
- User-scoped via JWT (user_id foreign key with CASCADE delete)
- Automatic updated_at timestamp via database trigger

Generated Code:
- database.SavedFilters model (10 fields including JSONB filters)
- All 5 CRUD query functions with proper parameter types
- pgtype.UUID wrappers for UUID parameters

Part of: Saved Filters Implementation (Phase 1: Database)
Related: #saved-filters-feature
2026-03-21 00:16:05 -04:00
john-okeefe 877fccbb52 refactor(api): rename scan_frequency_minutes to scan_poll_interval_seconds
- Update API field from scan_frequency_minutes to scan_poll_interval_seconds
- Update database schema default value key
- Update Bruno API collection requests and documentation
- Update OpenAPI documentation examples and field descriptions
2026-02-28 12:56:53 -05:00
john-okeefe e3a3aa124f feat(api): consolidate user profile update endpoints
Add delete_user, reset_user_password, and update_user endpoints to replace
individual update operations. Update database schema to include deleted_at
column for soft deletion. Add DeleteUser, ResetUserPassword, and
UpdateUserAdmin queries. Update Querier with new methods for user management.
2026-02-22 01:57:42 -05:00
john-okeefe 3af2fb0ba4 schema(dashboard): implement Phase 1 unified collections architecture
Add support for Carousel-style dashboard with unified collections architecture:

Database Schema Changes:
- Add user_dashboard_preferences table:
  - hidden_collections: TEXT[] for managing section visibility
  - collection_order: TEXT[] for custom ordering
  - items_per_section: INT for limiting items per section
- Update collections table:
  - user_id: Make nullable to support system-owned collections (NULL = system)
  - show_on_dashboard: BOOLEAN for controlling visibility
  - query_type: TEXT for different query types (continue-reading, recently-added, etc.)
  - priority: INT for display order (lower = higher priority)
  - is_system_collection: BOOLEAN for flagging system defaults
- Update collection_items table:
  - Add excluded BOOLEAN for user overrides of auto-assigned items

Indexes:
- idx_collections_dashboard: (user_id, show_on_dashboard, priority) WHERE show_on_dashboard = true
- idx_dashboard_prefs_user_library: (user_id, library_id)
- idx_collection_items_excluded: (collection_id, excluded) WHERE excluded = true

System Collections (pre-seeded defaults):
- continue-reading: Books with 0 < progress < 1
- recently-added: Newly added items to library
- recently-read: Books with progress >= 1
- not-started: Books with progress = 0 or no record

This implements Phase 1 of the Carousel-style dashboard redesign plan.
2026-02-19 20:52:21 -05:00
john-okeefe 2706ae52c1 refactor: remove Phase X terminology from source code comments
Remove planning document phase references from code comments:

app_test.go:
- Remove Phase 5 references from 8 test function comments

querier.go & queries.sql.go:
- Remove Phase 1, 2, 3, 4, 6 references from section headers
- Clean up week numbers (Weeks 5-6, Week 3-4, etc.)

queries.sql:
- Remove Phase 4 references from Kobo queries

kobo.go:
- Remove Phase 6 references from ContentId mapping comments

progress.go:
- Remove Phase 1 reference from route comment

media_scanner.go & media_scanner_library_type_test.go:
- Remove Phase 2 references from library type scanning comments

schema.sql:
- Remove Phase 1, 2, 3, 4, 5, 7 references from table/section comments
- Clean up: Format Detection, Progress Tracking, Device Registry,
  Sync Queue, Conflict Resolution, Reading History, Indexes, etc.

test_helpers.go:
- Remove Phase 6 reference from handler setup comment

These phase numbers were from internal planning documents and have no
meaning in the codebase. Removing them makes the code self-documenting.
2026-02-13 21:50:29 -05:00
john-okeefe 7e7945fbb7 fix: make trigger creation idempotent
- Add DROP TRIGGER IF EXISTS before CREATE TRIGGER
- Fixes 'trigger already exists' error during schema initialization
- Allows schema to run multiple times safely
2026-02-10 16:54:01 -05:00
john-okeefe 6ffb2ef6bc schema: make all schema statements idempotent (Phase 1)
- Convert 18 CREATE TABLE → CREATE TABLE IF NOT EXISTS (27 total)
- Convert 62 CREATE INDEX → CREATE INDEX IF NOT EXISTS (82 total)
- Add ON CONFLICT to 2 INSERT statements (3 total)
- Verify 8 ALTER TABLE already have IF NOT EXISTS
- Verify 6 CREATE FUNCTION use OR REPLACE

Schema is now fully idempotent and safe for automatic initialization on every startup.
2026-02-10 16:47:18 -05:00
john-okeefe e360ec2c30 feat(db): add system_settings table and remove per-user scan settings
- Add system_settings table with id, setting_key, setting_value, description, updated_at
- Insert default settings: scan_frequency_minutes=60, auto_scan_enabled=true
- Remove scan_frequency_minutes and auto_scan_enabled from users table
- Migrate from per-user scan settings to system-wide settings

This change enables centralized scan configuration for all libraries
while removing individual user scan preferences.

Database schema changes require container recreation:
  podman compose down -v
  podman compose up -d
2026-02-09 20:09:29 -05:00
john-okeefe 701815c1bc docs: update documentation for media scanner naming
- Update development.md with new file names (scanner.go, media_scanner.go)
- Update API reference: "Book/ebook operations" -> "Media item operations"
- Remove historical migration comments from schema:
  - Simplify media_items table comment
  - Remove backward compatibility comments for reading_progress and media_ratings
  - Remove note about backward compatibility views
- Remove historical comment from queries.sql about ebook folders
2026-02-08 14:55:06 -05:00
john-okeefe cedea0f0eb feat(deps): add golang.org/x/text dependency
Added golang.org/x/text v0.33.0 for proper titlecasing support in tag
normalization. Required for dual-field normalization to display tags in
title case (e.g., "Science Fiction", "Non-Fiction") while maintaining
search fields in lowercase without punctuation.
2026-02-08 00:43:12 -05:00
john-okeefe 516cec5a7f feat: migrate tags and contributors from TEXT to TEXT[] arrays
Convert tags and contributors columns from comma-separated strings to PostgreSQL
TEXT[] arrays for better data normalization and query performance.

Database Changes:
- schema.sql: Change tags/contributors from TEXT to TEXT[]
- schema.sql: Add GIN indexes for fast array searches
- queries.sql: Update search queries to use ANY() operator
- queries.sql: Update fuzzy search with unnest() for arrays

Generated Code (sqlc):
- models.go: Auto-generated with []string types for tags/contributors
- queries.sql.go: Auto-generated with proper array handling

Handler Changes:
- media.go: Update request structs to use []string for tags/contributors
- media.go: Remove pgtype.Text wrapping, use direct array assignment
- media.go: Add tag normalization in CreateMediaItemHandler
- collections.go: Update tags evaluation to join arrays for comparison
- collections.go: Add strings import for Join() function

Service Changes:
- ebook_scanner.go: Update EbookMetadata struct to use []string
- ebook_scanner.go: Remove string Join(), assign arrays directly
- collection_service.go: Update tags rule evaluation to join arrays
- collection_service.go: Add strings import

New Utilities:
- internal/utils/tags.go: Create NormalizeTags(), JoinTags(), SplitTags()
- Normalizes tags by trimming, lowercasing, removing duplicates/empties

API Documentation:
- bruno/media-items/Create Media Item.bru: Update examples to use arrays
- bruno/media-items/Update Media Item.bru: Update examples to use arrays
- Update docs: tags/contributors now array of string

Breaking Change:
- JSON format changes from "tags": "tag1,tag2" to "tags": ["tag1", "tag2"]
- Tests already use array format (no changes needed)

Benefits:
- GIN indexes enable faster array searches
- Normalization prevents data quality issues (case, duplicates)
- Array operations use PostgreSQL native operators (ANY, &&, unnest)
- Better separation of concerns (no string parsing in application)
2026-02-07 22:53:12 -05:00
john-okeefe 153b330dd1 Update database schema and queries: Bookmann → Bookhoard
Database changes:
- schema.sql: Update column name bookmann_uuid → bookhoard_uuid
- schema.sql: Update index names and example URLs
- queries.sql: Update all SQL queries to use bookhoard_uuid
- Update example configuration values

Part of project rename to Bookhoard.
2026-02-01 16:21:03 -05:00
john-okeefe d43e0105eb Rename project module and database references: Bookmann → Bookhoard
Core infrastructure changes:
- Update Go module name: bookmann → bookhoard
- Rename database schema references and comments
- Update database column names: bookmann_uuid → bookhoard_uuid
- Rename SQL query functions: GetDeviceCatalogByBookmannUUID → GetDeviceCatalogByBookhoardUUID
- Update configuration defaults

This is part 1 of the project rename to Bookhoard.
2026-02-01 16:11:47 -05:00
john-okeefe d0d86f19d6 refactor: remove all backward compatibility database views
Removed all 5 backward compatibility views from database schema:

1. ebooks view (lines 122-128)
   - Old API: /api/ebooks
   - Replaced by: media_items table with library_type filter

2. ebook_reading_progress view (lines 160-168)
   - Old API: mapped ebook_id alias
   - All progress routes now use reading_progress table directly

3. ebook_ratings view (lines 323-331)
   - Old API: /api/ebooks/:id/rating
   - Replaced by: media_ratings table

4. ebook_notes view (lines 333-341)
   - Old API: /api/ebooks/:id/notes
   - Replaced by: media_notes table

5. ebook_highlights view (lines 343-351)
   - Old API: /api/ebooks/:id/highlights
   - Replaced by: media_highlights table

Verification:
- No code queries these views directly (all queries use base tables)
- No handlers reference these views
- Bruno API tests use /api/media-items endpoints
- Application compiles successfully

This completes Phase 4 of legacy code cleanup.
All backward compatibility layers removed, codebase uses base tables directly.
2026-02-01 15:18:09 -05:00
john-okeefe 19d12c3a6b docs: remove migration comments from database schema
- Removed comment about user_ebook_folders table replacement
- Removed comment about library system transition
- Historical migration documentation removed

This is part of legacy code cleanup Phase 1.
Phase 1: Documentation Cleanup
2026-02-01 14:09:33 -05:00
john-okeefe 63008001c6 feat(db): Add kobo_shelves and device catalog tables
- Add kobo_shelves table for Kobo device shelves management
- Add device_shelf_mappings for collection<->device shelf mappings
- Update device_catalogs with better ContentId tracking
- Add indexes for performance
2026-01-31 22:32:04 -05:00
john-okeefe 7ead4109ef Add performance optimizations (Phase 7)
- Add 12 composite database indexes for sync operations
- Composite indexes for sync_queue (device/status/priority)
- Composite indexes for reading_progress (user/media timestamps)
- Composite indexes for devices (user/sync_enabled)
- Composite indexes for annotations (user/media)
- Comment out ALTER SYSTEM commands for sqlc compatibility
- PostgreSQL tuning recommendations included for manual application
2026-01-31 13:06:17 -05:00
john-okeefe 835e78898a Convert refresh_tokens.token from VARCHAR to UUID
- Change token column type from VARCHAR(255) to UUID
- Add gen_random_uuid() as default value for token
- Improves type safety and performance for token storage
2026-01-31 11:44:23 -05:00
john-okeefe c17b42bf83 feat: add Kobo shelf and entitlements tables to schema 2026-01-31 00:29:36 -05:00
john-okeefe 25057cf33a Phase 3 Week 7: Add KOReader bulk sync function to database schema
- Add bulk_update_progress_from_koreader() function for batch processing
- Handles progress, annotations, and conflict detection
- Returns success/failure status for each book
- Supports device matching by UUID, file path, or title/author
- Implements automatic conflict detection for concurrent syncs
- Part of Phase 3 KOReader Integration implementation
2026-01-30 20:55:50 -05:00
john-okeefe bddec2942b Phase 3 Week 7: Add KOReader database queries and generated code
- Add GetMediaItemByFilePathForSync for file path matching
- Add GetUserProgressForBooks for bulk progress retrieval
- Add GetAnnotationsForBook for annotation sync
- Add UpdateDeviceSyncTimestamp for device tracking
- Add GetUserMediaItemsForSync for library sync
- Add CheckForProgressConflicts for conflict detection
- Regenerate sqlc code for all new queries
- Part of Phase 3 KOReader Integration implementation
2026-01-30 20:54:51 -05:00
john-okeefe 4c01c0d12e Phase 3 Week 7: Add KOReader bulk sync function to database schema
- Add bulk_update_progress_from_koreader() function for batch processing
- Handles progress, annotations, and conflict detection
- Returns success/failure status for each book
- Supports device matching by UUID, file path, or title/author
- Implements automatic conflict detection for concurrent syncs
- Part of Phase 3 KOReader Integration implementation
2026-01-30 20:54:51 -05:00
john-okeefe fa4a9c35bb Phase 1 Week 1: Database schema for universal sync system
- Add format_group columns to media_items table
- Add universal progress tracking to reading_progress (percentage, epubcfi, chapter, etc.)
- Add device sync metadata (last_sync_device, conflict tracking)
- Add location enhancements to media_notes and media_highlights
- Create devices table for device registry
- Create sync_queue table for offline support
- Create sync_conflicts table for conflict resolution
- Create reading_history table for session tracking
- Add 15 new indexes for performance
- Create update_updated_at_column trigger function
- Add SQL helper functions: detect_format_group, convert_progress, detect_conflict, merge_progress

Schema grew from 272 to 598 lines (+326 lines)
Verified with sqlc generate
2026-01-30 16:04:06 -05:00
john-okeefe 37b8380533 fix: sync database queries with enhanced media-items schema
- Add missing enhanced fields to CreateMediaItem INSERT statement
- Add missing enhanced fields to UpdateMediaItem UPDATE statement
- Include language, edition, page_count, genre, copyright_year
- Include integration fields: goodreads_id, openlibrary_id, google_books_id
- Add corresponding indexes for enhanced fields
- Add column comments for better documentation
- Resolves schema-query mismatch causing field removal cycles
2026-01-30 14:11:52 -05:00
john-okeefe 758c5874eb test: rename Ebooks group to MediaItems and update API paths
Major changes:
- Rename testEbooks() function to testMediaItems()
- Remove all old ebook test cases
- Update all /api/ebooks paths to /api/media-items
- Update TestContext: remove EbookID, add MediaItemID field
- Add admin media-items tests (Create, Update, Delete)
- Fix compilation errors and missing imports

Tests updated to use new API structure while maintaining test coverage.

Breaking change: /api/ebooks endpoints removed (use /api/media-items instead)
2026-01-30 10:22:07 -05:00
john-okeefe f96044b6c7 refactor: remove ebook system, unify on media-items
Phase 1-3: Database layer cleanup
- Remove 5 backward compatibility VIEWs (ebooks, ebook_ratings, etc.)
- Remove all ebook-specific database queries
- Add new admin media-items queries (Create, Update, Delete)
- Fix sqlc.yaml to point to schema.sql file
- Regenerate database code successfully

Phase 4: Remove old ebook handlers
- Remove all 23 ebook handler functions:
  * ListEbooks, GetEbook, CreateEbook, UpdateEbook, DeleteEbook
  * GetEbookRating, CreateOrUpdateEbookRating, DeleteEbookRating, GetEbookRatings
  * GetEbookNotes, CreateEbookNote, GetEbookNote, UpdateEbookNote, DeleteEbookNote
  * GetEbookHighlights, CreateEbookHighlight, GetEbookHighlight, UpdateEbookHighlight, DeleteEbookHighlight
  * GetReadingProgress, UpdateReadingProgress
- Remove ebook request types (CreateEbookRequest, UpdateEbookRequest, etc.)

Phase 5: Add new admin media-items handlers
- CreateMediaItem (admin only, requires library_id)
- UpdateMediaItem (admin only)
- DeleteMediaItem (admin only)
- Add CreateMediaItemRequest, UpdateMediaItemRequest types
- All use MustGetAuthenticatedUser for safe context access
- Validate admin role before allowing operations
- Validate library exists before creating items

Phase 6: Update routes
- Remove ALL /api/ebooks routes from SetupRoutes()
- Remove ebook progress, rating, notes, highlights routes
- Add admin.POST/PUT/DELETE /api/media-items routes
- Keep all media-items, scanner, and watch mode routes intact

Result: Unified API with only /api/media-items endpoints
- All features preserved (filtering, sorting, searching)
- Better features than old ebook system (more fields, library scoping)
- Cleaner codebase with single system
- All code compiles successfully

Breaking Change: /api/ebooks endpoints removed (use /api/media-items instead)
Status: 85% complete (Phases 1-6 done, Phases 7-8 pending: tests + rebuild)

Tests: Need update (rename Ebooks → MediaItems, update API paths)
Build: Need rebuild with clean cache
2026-01-30 10:03:13 -05:00
john-okeefe 3b2075fc70 Phase 1: Add enhanced database fields and sorting
- Add 9 new fields to media_items table (language, edition, page_count, goodreads_id, openlibrary_id, google_books_id, copyright_year, genre, subjects)
- Add indexes for new fields (language, genre, page_count, copyright_year, series_order, date_published)
- Add ListMediaItemsSorted SQL query for dynamic sorting
- Update ListMediaItems handler to process sort parameter
- Support 16 sorting options (title, author, created_at, date_published, copyright_year, page_count, genre, series)
- Add /api/media-items/filtered endpoint for advanced filtering
- Register new filtered endpoint in routes
2026-01-30 08:32:49 -05:00