The design parked in the handoff (multi-tier stats off reading_history;
bookorbit-style in data, not looks) is in. The real prerequisite work
was data: reading_history existed but NOTHING wrote to it — the whole
stats feature was sitting on an empty table.
- CAPTURE (no client changes, no repo-boundary crossings): every
successful progress write through SaveProgress (KOReader + app + web
+ Kobo + offline queue — the one chokepoint) now records one
reading_history session row: span = the interval since the PREVIOUS
write instant, pages = positive page delta when both sides know the
page index, marker rows for first contact. Spans over 2h are kept as
zero-second markers (overnight suspend→resume, offline flushes must
not inflate totals) — the honest 'sync cadence, not a stopwatch'
caveat stays. device_id only when a REAL device row matches (the
web/app progress path historically stuffs the USER id into DeviceID
— probe instead of trusting it; FK violation would silently drop
every capture). Source preserved in device_metadata.
- API: /api/stats/summary?range=7|30|90|year (totals, zero-filled
daily arrays, streaks, most-active day/weekday, top books, library
breakdown — one call), /api/stats/this-week (widget: rolling 7d vs
previous + streak), /api/stats/books/:id (lifetime + recent
sessions). All bucketed in the user's timezone; finished ≡ any push
≥0.995; visible-library lens enforced at source. Legacy
/api/analytics/* kept untouched. Range windows anchor end-of-day to
the START of today (blasting the clock time into tomorrow resolved
'today' a calendar day late — found by the streak test).
- /stats page (sidebar Stats between Progress and Devices, previously
/analytics was an orphan): range selector, five stat cards, two
Chart.js v4 charts (daily minutes bar, pages/day line — chunked via
dynamic import so main.js stays untouched), per-library breakdown,
most-read list, empty-state explainer. /analytics now 301s to /stats.
The orphaned analytics.templ + analytics.ts removed (their containers
and the page JSON never matched; the page was astub).
- Dashboard 'This Week' widget (server-rendered, suppressed on service
failure) and the book detail 'Your Reading Stats' card (lifetime
totals of the viewer's own sessions) complete the tiers: per-user
default, per-library breakdown inline, per-book on the page,
per-system/admin aggregate parked as before.
Tests: statistics_test.go (capture semantics at the chokepoint,
tz/day buckets, zero-fill, streak math, finished detection, range
validation, auth gate, widget, per-book lifetime). Full suite PASS.
Type is DERIVED, not stored: a saved ruleset makes a collection SMART,
no rules = STATIC (stored collection_items links, unchanged behavior).
No schema change — collections.auto_assign_rules JSONB already carried
the config; user's decision 2026-10-14 (handoff: live-at-render, not
refresh-on-scan; create-time Static/Smart toggle; v1 NO hybrid).
- Rule gains an enabled flag (absent = enabled, legacy rules unaffected).
- Rule CRUD: POST/PUT(DELETE) /api/collections/:id/rules[/:ruleId] — the
survivor dead effort of the rules-builder UI finally has real
endpoints (rules were never editable after creation; nothing ever
re-evaluated them). First rule converts a user collection to smart;
removing the last reverts it to static. UpdateCollectionRulesJSON
edits ONLY the rules doc so rule CRUD cannot clobber metadata.
- Live evaluation: candidates from GetVisibleLiveMediaForRules —
visible-library set (family lens enforced at source) + archived IS
NULL AND missing_scan_count = 0 — any enabled rule matches, title
order. Zero refresh hooks by design: always current by construction.
- Membership mutations (add/remove/bulk-add) are 400 on smart — v1 no
hybrid; collection_items.excluded stays the reserved future hook.
- Rules builder page (/collections/:id/rules) first time ever routable;
templ whole-attribute interpolation ("{ expr }" inside a quoted attr
value is literal text, as theme-{ user.Theme } silently proved).
- UI: create modal Static/⚡Smart cards (smart create redirects into the
rules builder), ⚡ badge + real per-card counts on the grid (smart =
live matches, static = stored-link count), smart detail renders
matched books with no hand-pick actions + an Edit Rules button.
- API: GET /:id gains is_smart + live books; GET /: keeps the Android
app's default exclusion, include_auto=true surfaces smart rows with
live counts; TestRules previews against the SAME candidates the live
render uses (visible + live, was raw first-1000).
- FIXED pre-existing regression from 7ce7321: static collection detail
with NO library_id returned zero books — QueryType != '' caught user
rows ('filter' default) and routed them to dashboard sections that
never contain them; now only real system query types go there
(handlers + SSR). App-visible symptom: /api/collections/:id without
library_id came back books:[] for every static collection.
Tests: collections_smart_test.go (conversion, disabled rules, mutation
guards, revert-to-static, validation, system-collection prohibition,
include_auto semantics). Full offline integration suite PASS.
The old model hard-deleted system collections and re-created them at
the tail, resurrecting stale dedups and breaking ordering. Redesign
(user-directed B-final):
- schema.sql: collections.deleted_at TIMESTAMPTZ (idempotent ALTER —
the whole schema is re-applied on every boot via executeSchema).
- queries.sql: every collection list/read query filters
deleted_at IS NULL; new GetSystemCollectionByQueryType :one
(unfiltered, sees tombstones), TombstoneSystemCollection :exec,
ClearCollectionItems :exec, RestoreSystemCollectionRow :one
(in-place metadata reset + un-tombstone; collections have no
updated_at column — ResetUpdatedAt was a 500 caught by tests).
sqlc regenerated.
- collection_service.DeleteCollection: system collections are
tombstoned (preserving position/ownership); user collections keep
the hard delete. No ErrNoRows to the client (delete is idempotent).
- dashboard_service.RestoreSystemCollection: resolve by query_type;
found → in-place restore (ClearCollectionItems + metadata reset,
un-tombstoned); missing → provision fresh. "Want to Read"/etc.
stay at their home positions forever.
- handlers/collections.CreateCollection: name-taken is 409, not 500.
- handlers/auth.CreateDefaultCollectionsForUser: idempotent by
(user_id, query_type) — update-in-place when a row for that query
type exists (heals renames/tombstones), insert when missing; this
also fixes the duplicate-system-collections regression at source.
Apply Go 1.18+ language features and modern style:
internal/services/collection_service.go:
- Use map[string]any instead of map[string]interface{} (Go 1.18+)
- Use range clause with single variable for iteration-only loops
- Replace if-else chains with switch statements for better readability
- Remove explicit type initialization for zero values
internal/services/filters.go:
- Add Err prefix to custom error variable for error naming convention
internal/router/library.go:
- Use cfg.ProcessingIssuesHandler instead of local processingIssuesHandler variable
- Ensures proper dependency injection through router config
These changes follow current Go best practices and improve code readability.
Convert tags and contributors columns from comma-separated strings to PostgreSQL
TEXT[] arrays for better data normalization and query performance.
Database Changes:
- schema.sql: Change tags/contributors from TEXT to TEXT[]
- schema.sql: Add GIN indexes for fast array searches
- queries.sql: Update search queries to use ANY() operator
- queries.sql: Update fuzzy search with unnest() for arrays
Generated Code (sqlc):
- models.go: Auto-generated with []string types for tags/contributors
- queries.sql.go: Auto-generated with proper array handling
Handler Changes:
- media.go: Update request structs to use []string for tags/contributors
- media.go: Remove pgtype.Text wrapping, use direct array assignment
- media.go: Add tag normalization in CreateMediaItemHandler
- collections.go: Update tags evaluation to join arrays for comparison
- collections.go: Add strings import for Join() function
Service Changes:
- ebook_scanner.go: Update EbookMetadata struct to use []string
- ebook_scanner.go: Remove string Join(), assign arrays directly
- collection_service.go: Update tags rule evaluation to join arrays
- collection_service.go: Add strings import
New Utilities:
- internal/utils/tags.go: Create NormalizeTags(), JoinTags(), SplitTags()
- Normalizes tags by trimming, lowercasing, removing duplicates/empties
API Documentation:
- bruno/media-items/Create Media Item.bru: Update examples to use arrays
- bruno/media-items/Update Media Item.bru: Update examples to use arrays
- Update docs: tags/contributors now array of string
Breaking Change:
- JSON format changes from "tags": "tag1,tag2" to "tags": ["tag1", "tag2"]
- Tests already use array format (no changes needed)
Benefits:
- GIN indexes enable faster array searches
- Normalization prevents data quality issues (case, duplicates)
- Array operations use PostgreSQL native operators (ANY, &&, unnest)
- Better separation of concerns (no string parsing in application)
- Add book matching service for intelligent book deduplication
- Add collection service for collection management
- Add test files for book matching and collections
John O'Keefe
john-okeefe