A read-then-write race in processMediaFile allowed the same file to be
imported twice: two concurrent scan jobs (startup scan, fsnotify dirty-
directory scan, periodic backup poll, or a manual scan each run on
separate worker goroutines with separate MediaScanner instances) could
both SELECT 'not found' and both INSERT. There was no transaction, no
row lock, no unique constraint on (library_id, file_path), and no
ON CONFLICT clause, so nothing stopped the double insert. Observed in
production as two identical 'Head First SQL' rows created in the same
second (same sha256, size, path, library).
Database enforcement:
- schema.sql: add UNIQUE(library_id, file_path) constraint, guarded so
re-runs don't error
- schema.sql: add self-healing migration that runs on every startup -
dedup_media_items_by_path() collapses existing path-duplicates and
reparent_media_item_children() moves all child rows (progress,
highlights, bookmarks, notes, collections, formats, aliases, kobo
entitlements, etc.) onto a survivor before deleting losers, so the
constraint applies cleanly on already-duplicated servers without
losing reading history. Survivor picks the row with the most user
data, ties broken by lowest id
- CreateMediaItem: upsert via ON CONFLICT (library_id, file_path) DO
UPDATE so concurrent inserts collapse to one row and return it
- CreateMediaItemFormat: upsert via ON CONFLICT (media_item_id,
format_type), closing the same race on format rows
Application-level guards:
- media_scanner processMediaFile: after computing the file hash, check
GetMediaItemBySHA256AndLibrary (new query) and treat the file as
existing when identical content is already in the library under a
different path (content dedup, library-scoped so multi-library
setups still work)
Ops tooling:
- scripts/dedup_media_items.sql: standalone idempotent maintenance
script with a dry-run report (path + content duplicate groups, child
row counts) and transactional cleanup, for servers that prefer to
dedup manually before upgrading
Verified against the live database: the duplicate pair was collapsed
(reading_progress preserved on the survivor), schema.sql re-runs are a
no-op, and the constraint is in place with 62 unique books remaining.
Add a typed, cached registry over the system_settings table so that
values which used to be hardcoded Go literals can be changed at runtime.
Schema (database/schema/schema.sql):
- Extend system_settings with setting_type, min_value, max_value,
requires_restart, and category columns (all ADD COLUMN IF NOT EXISTS,
nullable for backward compat with the original three rows).
- Seed rows for every tunable: session duration, password rules,
login lockout, auth/device rate limits, OPDS page size, tombstone TTL,
conversion cache TTL, sync queue interval/batch, and worker pool
size/cap. Seed values equal the previous hardcoded literals, so
behavior is unchanged on upgrade. ON CONFLICT DO NOTHING preserves
any admin-modified values.
Queries (queries.sql):
- Add UpsertSystemSetting (RETURNING *) so new keys without a seed row
can still be written through the API.
- Add GetSystemSettingFull + GetAllSystemSettingsFull returning the
full typed row.
- Refactor CleanupExpiredRefreshTokens to take the retention window as
a parameter (make_interval(secs => $1)) instead of the INTERVAL '7
days' literal, so it can follow a configurable session duration.
Registry (internal/database/settings_registry.go):
- SettingsRegistry holds an in-memory cache of all known settings,
populated by Load at startup and refreshed by Reload on writes.
- Typed domain getters (SessionDuration, PasswordRules, DeviceRateLimits,
TombstoneTTL, OpdsPageSize, ConversionCacheTTL, SyncQueueConfig,
WorkerPoolConfig, LoginLockout, AuthRateLimit, ...) with compiled-in
fallback defaults and min/max clamping, so a corrupt or missing row
can never break the app.
- SettingDefaults is the single source of truth for keys, types, bounds,
and human descriptions; All() exposes metadata + current values for
the admin UI/API.
The registry lives in the database package (rather than its own
internal/settings package) because a quirk in this custom go1.26.5
toolchain prevented the large handlers package from importing any
newly-created package; every consumer already imports database.
Tests: settings_registry_test.go covers default validity per type,
int clamping at both bounds, garbage-value fallback, and unknown-key
lookup.
Add GetVisibleLibraryMediaCounts, which returns the media item count for
each library visible to a given user in a single GROUP BY query over
media_items. It mirrors the visibility logic in GetUserVisibleLibraries
(libraries default to visible unless an explicit false row exists) so
counts can be resolved in one round-trip instead of N per-library
lookups.
Regenerated sqlc bindings (querier.go, queries.sql.go).
Add migration columns to media_highlights, media_notes, and media_bookmarks
for cross-device annotation sync:
- dedup_key: SHA-1 of normalized selection text + bucketed position, used
as the stable cross-device identity for annotations
- last_modified_at / last_modified_source: edit clock for LWW resolution
and cross-source conflict detection
- deleted / deleted_at: sticky tombstone columns for delete-wins semantics
with a 30-day TTL before rows are physically purged
- note_text on highlights: stores attached notes from KOReader entries that
have both selected text and a user note
- Location columns on bookmarks (cfi_position, percentage_location,
epubcfi_location, chapter_reference, paragraph_reference)
- device_sync_data JSONB on all three tables: stores per-device native
identifiers (e.g. KOReader pos0/datetime, Kobo bookmark_id) so each
device can locate and manipulate its own copy of an annotation
Add partial unique indexes on (user_id, media_item_id, dedup_key) where
deleted = FALSE to enforce one active annotation per dedup key.
Add tombstone purge indexes on (deleted, deleted_at) for efficient GC.
New queries:
- GetByDedupKey for all three tables (returns active or most-recent tombstone)
- CreateFull / UpdateForSync for all three tables (populate sync columns)
- TombstoneByDedupKey / TombstoneByID for all three tables
- PurgeExpired* for all three tables (GC past TTL)
- GetActiveAnnotationsForBook (filtered union of highlights + notes)
- GetTombstonedAnnotationsForBook (union of all 3 deleted within TTL)
- GetMediaBookmarks with deleted filter
- GetMediaBookmark (singular) with deleted filter
- Added deleted=FALSE filter to GetMediaHighlights, GetAnnotationsForBook
- CreateAutoResolvedSyncConflict (INSERT with resolution_status='auto_resolved')
Setup completion was previously tracked by a manually-flipped setup_complete row in system_settings, written via a JWT-protected PUT /api/setup/complete endpoint. This meant any admin user created outside the setup wizard (future CLI, seed scripts, direct DB inserts) would not flip the switch, leaving the app stuck redirecting to /setup.
The trigger is now derived from real data: setup is complete iff at least one admin user exists. This is self-correcting regardless of how users are created, and re-engages setup automatically if all admins are ever removed.
Changes:
- Add internal/setupstatus package with IsSetupComplete() (queries CountAdmins, 10s in-memory cache, fails open on DB error) and Invalidate() to clear the cache. Uses an AdminCounter interface to avoid importing the database package.
- Add CountAdmins sqlc query (SELECT COUNT(*) FROM users WHERE role = 'admin') and regenerate.
- Rewire router/setup.go isSetupComplete() to delegate to setupstatus; drop the old setup_complete setting read, cache vars, and the PUT /api/setup/complete route.
- Call setupstatus.Invalidate() in the auth handler after CreateUser, UpdateUserRole, and DeleteUser so the cache reflects admin-count changes immediately.
- Align first-user promotion in Register to key off !adminExists instead of len(users) == 0, so the two checks cannot diverge.
- Remove the now-dead SetSetupComplete/GetSetupStatus handlers.
- Drop the setup_complete seed row from schema.sql.
- Remove the apiPut('/setup/complete') call from the setup wizard finishSetup(); the admin account created in submitAdmin already marks setup complete server-side.
Stores surrounding text (~100 chars) at the reader's current position.
Used as fallback for CFI resolution when converting between epubcfi
and CREngine XPointer formats.
Updates:
- schema.sql: add context_text TEXT column, update stored procedure
- queries.sql: add context_text to GetUniversalProgress and
UpdateUniversalProgress queries
- Regenerate sqlc Go code (models.go, querier.go, queries.sql.go)
Regenerated database code after sqlc version upgrade from v1.30.0 to
v1.31.1. No functional changes — only the version header in generated
files was updated.
Files: db.go, models.go, querier.go, queries.sql.go
Convert 12 SQL queries to use sqlc.narg('library_id') instead of
direct @library_id parameters. This allows passing a NULL/invalid
pgtype.UUID to mean "no library filter" (i.e., All Libraries),
making the SQL layer correctly handle the optional filter via:
(sqlc.narg('library_id')::uuid IS NULL
OR mi.library_id = sqlc.narg('library_id')::uuid)
Also remove the library_id filter from GetSeriesBooks entirely —
a series is a series regardless of library.
Queries affected:
- GetDashboardSections, GetRecentlyAdded, GetInProgress
- GetHighestRated, GetMostRead, GetAbandonedBooks
- GetLeastRead, GetBooksByTag, GetCollectionItemsForDashboard
- SearchMediaItemsUnified, GetSeriesCardsData
Generated code (queries.sql.go, querier.go) regenerated via sqlc.
The created_at column stores file modification time (intentional for preserving
original metadata), but this makes 'Recently Added' sorting unreliable for
imported files. Add imported_at column that records the actual database insert
timestamp.
Changes:
- Add imported_at TIMESTAMPTZ column to media_items (nullable)
- Update GetRecentlyAddedItems to sort by imported_at DESC NULLS LAST first
- Add ReassignLibraries and ReassignMediaItems queries for admin deletion handover
- Add SyncLibraryTypeExtensions query for startup extension sync
- Update all media_items SELECT queries to include imported_at column
Add five new sqlc queries to support the series browse page and
continue-series dashboard collection:
- GetDistinctSeries: list unique series with book counts, sorted by
most recent entry, with pagination
- GetDistinctSeriesCount: total distinct series count for pagination
- GetSeriesCovers: fetch up to N cover image paths for a series,
ordered by series_number
- GetSeriesBooks: fetch all books in a series ordered by series_number
- GetContinueSeriesItems: CTE-based query using DISTINCT ON to find
the next unread book per series for a given user/library, sorted
by most recent last_read_at
- Add timezone column (VARCHAR(50) DEFAULT 'UTC') to users table
- Add default_timezone row to system_settings seed data
- Add idx_users_timezone index for user timezone lookups
- Add UpdateUserTimezone and GetSystemTimezone queries
- Regenerate sqlc code (models, querier, queries.sql.go)
- Reuse existing UpdateSystemSetting for system timezone updates
instead of creating a redundant UpdateSystemTimezone query
DismissAllResolved was calling ListSyncConflictsByUser which filters to
'unresolved' conflicts only, so it could never find the user_resolved or
bulk_resolved conflicts it was trying to delete. The query always returned
an empty set, making dismiss-all a no-op.
Fix the leading space in three SQL query name annotations (ListConflictsByUser,
ListAllConflictsByUserAndStatus, CheckForProgressConflicts) that prevented
sqlc from generating their Go functions. Regenerate the query code and swap
DismissAllResolved to use ListConflictsByUser (no status filter) — the
existing Go loop already filters by resolution_status.
- Add ProcessingIssues model struct
- Update Querier interface with processing issues methods
- Add generated query implementations for CreateProcessingIssue, ListProcessingIssuesByLibrary, GetProcessingIssueStats, ResolveProcessingIssue, DeleteProcessingIssue
- Add GetLibraryWithType query for fetching library with type information
- Add library_type_name VARCHAR(50) column to media_items table in schema.sql
- Create PostgreSQL trigger 'set_library_type_name_on_insert' that automatically
populates library_type_name by joining libraries table with library_types on insert
- Add library_type_name parameter to CreateMediaItem SQL query
- Update Go models (models.go) to include LibraryTypeName field
- Add UpdateMediaItemChapterMetadata query method to querier.go
- Regenerate queries.sql.go with sqlc
This allows media items to store their library type (e.g., 'Books', 'Comics', 'Manga')
at the database level, enabling filtering and display without needing additional joins.
- Add tags_filter parameter to SearchMediaItemsUnified
- Add EXISTS clause with word_similarity() for fuzzy tag matching
- Add tag similarity scoring to ORDER BY clause (GREATEST function)
- Add SearchTagsValues query for autocomplete with ::TEXT cast
- Keep genre_filter for backward compatibility
- Regenerate Go code with sqlc generate
This enables filtering books by tags (from Calibre) instead of genre,
which is always NULL for imported books. Uses fuzzy matching consistent
with author/series filters, with best matches sorted first.
Relates to IMPLEMENTATION_TAGS_FILTER.md Phase 1
Updates SearchMediaItemsUnified query to support searching across all
libraries when library_id parameter is not provided. Changes SQL from
requiring library_id to checking for NULL:
AND (sqlc.narg('library_id')::uuid IS NULL
OR mi.library_id = sqlc.narg('library_id')::uuid)
The explicit ::uuid cast ensures PostgreSQL handles type inference
correctly when comparing UUID columns with nullable parameters.
Regenerates Go database code including queries.sql.go and querier.go
to reflect the updated SQL schema.
This enables the /api/media-items/search endpoint to search all libraries
by omitting the library_id query parameter, matching the behavior of
the OPDS search endpoint.
- Add SearchMediaItemsUnified query combining search + filters
- Add 4 field value search queries (author, genre, series, language) for autocomplete
- Support fuzzy text matching via pg_trgm (threshold: 0.3 similarity)
- Support exact match with quotes detection for search queries
- Add sort parameter support (title ASC/DESC, author ASC/DESC, created_at ASC/DESC, page_count ASC/DESC)
- Primary sort by relevance score when searching, secondary by user-specified sort
- Combine search query with all filter types in single optimized query
- Uses 4 separate simple queries instead of 1 complex query due to sqlc v1.30.0 limitation with CASE in GROUP BY
This consolidates the deprecated /filtered and /search endpoints into one unified endpoint.
Run sqlc generate to create Go code for new search queries:
Added methods to Querier interface:
- SearchMediaItemsUnified - Main unified search with fuzzy/exact matching
- SearchAuthorValues - Author field autocomplete
- SearchGenreValues - Genre field autocomplete
- SearchSeriesValues - Series field autocomplete
- SearchLanguageValues - Language field autocomplete
Generated parameter structs and row types for all new queries.
All queries include proper library visibility checks.
Fix critical security issue where admin users could delete other users'
saved filters due to incorrect error handling in DELETE query.
Database Schema Changes:
- Change DeleteSavedFilter from :exec to :one (queries.sql:1747-1750)
- Add RETURNING * to return deleted row for proper error detection
- Regenerate querier.go and queries.sql.go with updated signature
Service Layer (internal/services/filters.go):
- Update DeleteSavedFilter to capture returned row (using _ to discard)
- Properly propagate pgx.ErrNoRows when no rows are deleted
- Error wrapping preserves original error for handler detection
Handler Layer (internal/handlers/filters.go):
- Add errors.Is() check for pgx.ErrNoRows (line 148)
- Return 404 Not Found when filter doesn't exist or belongs to different user
- Return 500 Internal Server Error for other database errors
- Add "errors" import (line 8)
Security Fix Details:
Before: Admin could delete user's filter → 204 No Content (SUCCESS)
After: Admin tries to delete user's filter → 404 Not Found (DENIED)
The DELETE query uses WHERE id = @id AND user_id = @user_id, which matches
0 rows when attempting to delete another user's filter. The old :exec query
didn't return row count, so 0 affected rows looked like success. The new :one
query with RETURNING * returns pgx.ErrNoRows when no rows match, allowing
the handler to return proper 404 error.
Test Impact:
- TestSavedFilters/User_cannot_access_another_user's_filter now passes
- All 6 integration tests pass with proper user isolation enforcement
Pattern Consistency:
- Matches DeleteLibraryFolder pattern (line 99 in queries.sql)
- Uses same error handling as media handlers (errors.Is + pgx.ErrNoRows)
- Follows user-scoping pattern used throughout codebase
Related: Saved filters implementation user isolation
Security: Prevents unauthorized deletion of user data
Add database schema and SQL queries for generic saved filters system
that allows users to save custom filter presets for any resource type.
Database Schema:
- Add saved_filters table with user_id, name, resource_type, filters (JSONB)
- Create composite index on (user_id, resource_type) for efficient lookups
- Create index on (user_id, name) for future name search feature
- Add update_updated_at_column() trigger to auto-update timestamps
- Make trigger creation idempotent with DROP TRIGGER IF EXISTS
SQL Queries (5 new queries):
- GetSavedFilters: List all filters for user + resource type
- GetSavedFilterByID: Retrieve single filter by ID
- CreateSavedFilter: Create new saved filter
- UpdateSavedFilter: Update filter name/criteria
- DeleteSavedFilter: Remove saved filter
Design Decisions:
- Generic resource_type field supports any resource (media-items, collections, devices)
- JSONB filters field allows flexible schema without migrations
- User-scoped via JWT (user_id foreign key with CASCADE delete)
- Automatic updated_at timestamp via database trigger
Generated Code:
- database.SavedFilters model (10 fields including JSONB filters)
- All 5 CRUD query functions with proper parameter types
- pgtype.UUID wrappers for UUID parameters
Part of: Saved Filters Implementation (Phase 1: Database)
Related: #saved-filters-feature
Fix 1 - File modification time for created_at:
- Get file.ModTime() in processMediaFile and pass to CreateMediaItem
- Modified SQL INSERT to include created_at column
Fix 2 - Force rescan UPDATE instead of DELETE+INSERT:
- Changed force rescan logic to call updateMediaItem instead of delete + create
- Preserves created_at timestamp on force rescan
Fix 3 - GetMediaItemByFilePath filters by library_id:
- Added library_id to WHERE clause in SQL query
- Created GetMediaItemByFilePathAnyLibrary for cross-library lookups (KOReader)
- Added SetLibraryID method to MediaScanner
- Updated handler to call SetLibraryID for watch mode
Fix 4 - File deletion handling with persistent logging:
- Added fsnotify.Remove handler in WatchChanges
- Added orphan cleanup in ScanFolders after scan completes
- Created scanner_logger.go with daily log rotation (7 days)
- Logs to /app/logs/scanner-deletes-YYYY-MM-DD.log and scanner-errors-YYYY-MM-DD.log
- Individual deletes with enhanced safety logging
Note: Integration tests can now safely scan /app/uploads because
GetMediaItemByFilePath now filters by library_id, preventing
cross-library interference.
Add delete_user, reset_user_password, and update_user endpoints to replace
individual update operations. Update database schema to include deleted_at
column for soft deletion. Add DeleteUser, ResetUserPassword, and
UpdateUserAdmin queries. Update Querier with new methods for user management.
Auto-generated changes from running 'sqlc generate' after query updates:
- models.go: updated with SystemSettings struct, removed scan fields from Users
- querier.go: updated interface with new system settings methods
- queries.sql.go: regenerated with new query methods
Generated via: cd internal/database && sqlc generate
- Remove CreateEbookNote, UpdateEbookNote, DeleteEbookNote queries
- These were marked as backward compatibility but never used
- API uses CreateMediaNote, UpdateMediaNote, DeleteMediaNote instead
- Remove misleading backward compatibility comments
- Regenerate sqlc code
Changes:
- Update comments: "Bookmann UUID" → "Bookhoard UUID"
- Rename sidecar struct field: Bookmann → Bookhoard
- Update type names: SidecarBookmannConfig → SidecarBookhoardConfig
- Fix test database name in queue_test.go
- Fix uppercase env var examples in KOBO_SETUP.md
Internal Go variable names (BookmannUuid, bookmannUUID) left unchanged
as they're implementation details that don't affect functionality.
Part of project rename to Bookhoard.
Add analytics queries:
- GetUserReadingHistory: detailed reading history with device info
- GetUserDeviceUsage: device usage statistics (sync count, time spent)
- GetPopularBooks: most read books with completion rates
Add book matching queries:
- GetUnlinkedBookByID: fetch single unlinked book
- DeleteUnlinkedBook: remove resolved unlinked book
- ListUnresolvedUnlinkedBooks: paginated list of unresolved books
- Add bulk_update_progress_from_koreader() function for batch processing
- Handles progress, annotations, and conflict detection
- Returns success/failure status for each book
- Supports device matching by UUID, file path, or title/author
- Implements automatic conflict detection for concurrent syncs
- Part of Phase 3 KOReader Integration implementation
- Add internal/sync package with format detection
- FormatGroup types: reflowable, fixed_layout, comic_archive
- DetectFormatGroup() function based on mimetype and file extension
- MimeType mappings for common ebook formats
- Progress conversion engine with:
- ConvertProgress() between format groups
- Extract percentage from various progress formats
- PageToPercentage / PercentageToPage helpers
- CharacterToPercentage / PercentageToCharacter helpers
- MergeProgress() with 'max progress wins' strategy
- FormatProgressForDisplay() for UI rendering
- Add sqlc queries for format detection and progress updates
- BulkUpdateFormatGroups query for auto-format detection
- GetUniversalProgress query with all location references
- UpdateUniversalProgress query with device sync metadata
- ReadingHistory queries for session tracking
- Add SearchMediaItems query with ILIKE partial matching
- Add SearchMediaItemsFuzzy query with word_similarity()
- Use sqlc.narg() for named parameters (search_pattern, search_query)
- Rank results by relevance: title > author > series > tags
- Fuzzy threshold set to 0.3 for word_similarity
- Generated Go models with proper parameter types
- Increase ISBN column from VARCHAR(13) to VARCHAR(17) to support ISBN-13 with hyphens
- Add normalize_isbn() database function to automatically remove hyphens and spaces
- Create trigger to auto-normalize ISBNs on INSERT/UPDATE operations
- Update all Ebook and MediaItem queries to use ISBN normalization
- Add GetEbookLibraryID query to check for existing ebook libraries
- Add graceful error handling when no ebook library exists
- Return helpful error message: 'no ebook library found. Please create an ebook library first'
- Create comprehensive tests for ISBN normalization and library selection
- Add Bruno test files for various ISBN formats and error scenarios
- Update documentation with ISBN normalization details
- Regenerate queries.sql.go with refresh token queries
- Update models.go with RefreshTokens type
- Update querier.go with new query methods
- Update db.go with generated code
- Add library_types table with ebooks, comics, manga types
- Add libraries table for multiple library support
- Add library_folders table for multi-folder libraries
- Add library_visibility table for user access control
- Add media_items table replacing ebooks for broader media support
- Create backward compatibility views for existing API
- Implement library service with type validation and file extension handling
- Support modular extension for future media types
Manga type includes cbz/cbr archives as requested
- Move migrations/ to database/schema/ for clarity on database schema definitions
- Move sqlc.yaml to internal/database/ to group with database code
- Move static/ to cmd/server/static/ to co-locate with server
- Update all configuration files and documentation
- Follow Go project conventions for better organization
- Update CreateUser query to explicitly select only existing columns
- Update all user SELECT queries to explicitly select columns to avoid scan_frequency_minutes column issues
- Add GetUserForLogin query that includes password_hash for authentication
- Update login handler to use GetUserForLogin instead of GetUserByEmailOrUsername
- This prevents errors when migration hasn't been applied yet, allowing user registration to work
- Add admin dashboard at /admin route consolidating all user settings
- Move user preferences (username, email, password, theme) from separate page
- Add ebook library preferences section with folder management
- Add scan settings (frequency and auto-scan toggle) with database persistence
- Create database migration for scan_frequency_minutes and auto_scan_enabled columns
- Add API endpoints for scan settings management:
- PUT /api/library/scan-settings - Update scan preferences
- GET /api/library/scan-settings - Get current scan settings
- Update dashboard navigation to link to admin dashboard
- Remove old preferences.html template (functionality moved to admin)
- Create Bruno API testing files for library endpoints
- Add real-time folder loading and management in admin interface
- Implement scan settings persistence and retrieval from database
john-okeefe