Complete the annotation sync pipeline across all ingest and serve paths.
Previously, annotations sent inline with KOReader progress pushes were
silently discarded, and no annotations were ever served back to devices.
INGEST (device → server):
KOReader (koreader.go):
- Add processBookAnnotations helper that processes inline highlights,
notes, and bookmarks from every progress push (immediate + checkpoint)
- Highlights get CRE→CFI position conversion before SaveHighlight
- KOReader 'notes' (text + notes) stored as highlights with NoteText
to ensure correct round-trip classification
- Bookmarks routed through SaveBookmark with device sync data
- Called from both updateProgressForBook and handleCheckpointSync
Kobo (kobo.go):
- Markup handler: annotations and bookmarks route through
AnnotationService (SaveHighlight/SaveBookmark)
- Bookmark handler: same routing with device sync data
- SyncFromServer handler: same routing
- All handlers fall back to direct DB calls when annotationSvc == nil
Web reader (media.go):
- CreateMediaHighlight → SaveHighlight (Source="web", ModifiedAt=now)
- CreateMediaNote → SaveNote (Source="web")
- DeleteMediaHighlight → TombstoneHighlightByID
- DeleteMediaNote → TombstoneNoteByID (was hard delete, now tombstone)
- All fall back to old behavior when annotationSvc == nil
SERVE (server → device):
KOReader GetMetadata (koreader.go):
- Query and serve bookmarks from media_bookmarks table (was missing)
- Serve deleted_highlights and deleted_bookmarks arrays containing
device_sync_data + dedup_key for client-side deletion
- Highlights/notes already served with reverse CFI conversion
Kobo Markup handler (kobo.go):
- Track processed books during sync
- Query tombstones per book, extract bookmark_id from device_sync_data
- Return DeletedAnnotations array in KoboSyncStatus response
Conflict resolution (conflicts.go):
- Enable annotation conflict types in ResolveConflict handler
- Add applyAnnotationResolution dispatching to:
applyHighlightResolution / applyBookmarkResolution / applyNoteResolution
- Each looks up by dedup_key and applies winner's fields
- Allow manual override of auto_resolved conflicts
(changed check from != "unresolved" to == "user_resolved")
Infrastructure:
- AnnotationService field + SetAnnotationService in router Config
- Inject AnnotationService into KOReader, Kobo, Media handlers
- Start tombstone purger goroutine in main.go (24h interval)
- Test helpers: construct AnnotationService in test setup
Tests were deleting the development admin user, causing ON DELETE SET NULL
to cascade and set created_by_admin_id to NULL on all libraries.
- test_helpers: skip deletion of testuser@tests.bookhoard.internal
- sync_integration_test: use test-sync% prefix for isolated test data
Unit tests:
- series_service_test.go: test continueSeriesRowToMediaItems conversion
with valid fields, null fields, and comprehensive field mapping
- series_test.go: test handler initialization and textToString helper
Integration tests (series_integration_test.go):
- GET /api/series: requires library_id, rejects invalid UUID, returns
empty array for empty library, pagination params, limit clamped to
100, response structure validation, special characters in names
- GET /api/series/books: requires library_id and name, handles
nonexistent series, unauthorized access
- Restore Continue Series system collection
- Dashboard sections include all 5 collections (including continue-series)
Update test helpers:
- Add SeriesHandler to setupTestServer router config
- Add 5th Continue Series collection to createDefaultCollectionsForUser
- Update dashboard integration test for 5 collections
Adds 30 integration tests across 7 test functions covering all progress
endpoints with real HTTP requests and database verification:
- AuthContexts (8 tests): unauthenticated PUT/GET return 401, regular
user and admin both get 200, invalid UUID returns 400, nonexistent
item returns 200 with empty data.
- MergePreservesFields (2 tests): second PUT with only percentage
preserves epubcfi and chapter from first save via GET verification;
web save preserves koreader character_offset via DB query.
- EnrichmentComputesFields (2 tests): character_offset computed from
percentage when total_characters is set on media item; GET returns
enriched format_group and total_characters.
- ConflictDetection (3 tests): different sources with >1% diff within
5 minutes creates sync_conflicts record; same-source rapid saves
create no conflict; <1% diff creates no conflict.
- KoboIntegration (3 tests): ReadingSync then last-read-place preserves
percentage via DB; standalone last-read-place sets epubcfi/chapter;
unauthenticated returns 401.
- KOReaderIntegration (2 tests): Bearer token auth with proper request
body returns 202 Accepted; unauthenticated returns 401.
- DeleteProgress (2 tests): DELETE clears progress; unauthenticated
returns 401.
- EdgeCases (4 tests): empty body succeeds, 0.0% and 1.0% boundaries,
all fields with full DB verification of each column.
Updates test_helpers to create ProgressService in setupTestServer and
inject into all handlers. Fixes previous tests that used testing.Short()
(which caused all tests to be skipped in the container) and assertions
against wrong JSON format (pgtype serializes as plain values, not
wrapped objects).
The setupTestServer() helper in test_helpers_test.go was not creating
a ProcessingIssuesHandler and not passing one to the router config,
causing a nil pointer dereference when any processing issues route was
hit during tests. Add handler creation and wire it into routerConfig
to match how cmd/server/main.go does it.
Replace all unhandled resp.Body.Close() calls throughout the test suite:
- Deferred calls: replace 'defer VAR.Body.Close()' with a closure that explicitly
discards the error via 'defer func(Body io.ReadCloser) { _ = Body.Close() }(VAR.Body)'
- Immediate calls: replace 'VAR.Body.Close()' with '_ = VAR.Body.Close()'
Replace all unhandled json.NewDecoder(VAR.Body).Decode(&x) calls with error capture
and require.NoError assertion. Files using httptest.ResponseRecorder (collections_preview,
processing_issues) use 'err :=' declaration; suite-style tests (scanner_integration,
dashboard_integration) use s.T() instead of t.
Problem:
When running `make test-integration`, the last test to run would leave its
"test" libraries in the database. This happened because:
1. setupTestServer() cleaned up old "test" libraries at the START
2. Tests created their own libraries
3. When tests finished, t.Cleanup() called setup.Close() which only closed
connections but did NOT delete libraries
4. The LAST test's libraries persisted because no subsequent test cleaned them
For example, "Job Status Test Library" from TestWorker_JobStatusTracking
would remain in the database after all tests completed, visible when logging
into the UI.
Root Cause:
The cleanup logic only ran at the START of each test (in setupTestServer),
not at the END. This worked for intermediate tests (each test cleaned up
the previous test's libraries), but the final test had no cleanup.
Solution:
Added library cleanup to the t.Cleanup() function in setupTestServer(). Now
each test deletes its own "test" libraries when it completes, ensuring:
- Clean state after `make test-integration` finishes
- No residual test data in the database
- Safe for tests with subtests (cleanup runs after all subtests finish)
Note on Test Structure:
Tests like TestOPDSEndpoints and TestCollectionSearchLibraryFilter create
libraries once and share them across all subtests. The t.Cleanup() function
runs AFTER all subtests complete, so this change is safe and doesn't
interfere with subtest resource sharing.
Files changed:
- cmd/server/tests/test_helpers_test.go: Added library cleanup to t.Cleanup()
Add complete backend implementation for saved filters CRUD operations
with proper service layer architecture and RESTful API endpoints.
Service Layer (internal/services/filters.go):
- NewFiltersService() constructor following project patterns
- GetSavedFilters(): Retrieve all filters for user + resource type
- CreateSavedFilter(): Create filter with duplicate name validation
- UpdateSavedFilter(): Update filter with ownership verification
- DeleteSavedFilter(): Delete filter with user scoping
Business Logic:
- Filter name uniqueness enforced per user + resource type
- User ownership validation on all operations (JWT user_id)
- JSONB marshaling/unmarshaling for flexible filter storage
- Proper error wrapping with context messages
Handler Layer (internal/handlers/filters.go):
- NewFiltersHandler() constructor (receives db.Queries)
- GetSavedFilters: GET /api/saved-filters?resource_type=X
- CreateSavedFilter: POST /api/saved-filters
- UpdateSavedFilter: PUT /api/saved-filters/:id
- DeleteSavedFilter: DELETE /api/saved-filters/:id
Content Negotiation:
- Supports both JSON (API clients) and HTML (HTMX) responses
- wantsHTML() helper checks Accept header
- HX-Redirect header for HTMX form submissions
- Proper status codes (200, 201, 204, 400, 401, 404, 409)
Router Configuration:
- registerFiltersRoutes() function in internal/router/filters.go
- JWT middleware protection on all endpoints
- RESTful route structure: /api/saved-filters
- Registered in main router.go RegisterRoutes() function
- Added FiltersHandler to router.Config struct
Test Infrastructure:
- Added FiltersHandler to test server setup (test_helpers_test.go)
- FiltersHandler initialized in setupTestServer() function
- Router.Config includes FiltersHandler for integration tests
Code Quality:
- Follows PROJECT_GUIDELINES.md service layer patterns
- Uses database models (not custom domain models)
- JSONB returned as []byte (matches collections pattern)
- All errors wrapped with context using fmt.Errorf
- Handlers create services internally (not dependency injection)
Part of: Saved Filters Implementation (Phase 2: Backend)
Related: #saved-filters-feature
Replace default CORS middleware with explicit CORS configuration in test
server setup to align with production security settings. This ensures test
environment matches production behavior and prevents potential CORS-related
test failures.
Changes:
- Replace echomiddleware.CORS() with echomiddleware.CORSWithConfig()
- Configure allowed origins, methods, and headers explicitly
- Set AllowCredentials to false for test environment
- Add ExposeHeaders for Content-Length
This maintains consistency with the CORS configuration applied to the main
server in commit fb05c49.
Rename test helper files from .go to _test.go suffix to comply with
Go testing conventions. This ensures proper test file recognition by
the Go toolchain and improves build organization.
- library_test_comprehensive.go → library_test_comprehensive_test.go
- test_helpers.go → test_helpers_test.go
- test_helpers_db.go → test_helpers_db_test.go
john-okeefe