A read-then-write race in processMediaFile allowed the same file to be
imported twice: two concurrent scan jobs (startup scan, fsnotify dirty-
directory scan, periodic backup poll, or a manual scan each run on
separate worker goroutines with separate MediaScanner instances) could
both SELECT 'not found' and both INSERT. There was no transaction, no
row lock, no unique constraint on (library_id, file_path), and no
ON CONFLICT clause, so nothing stopped the double insert. Observed in
production as two identical 'Head First SQL' rows created in the same
second (same sha256, size, path, library).
Database enforcement:
- schema.sql: add UNIQUE(library_id, file_path) constraint, guarded so
re-runs don't error
- schema.sql: add self-healing migration that runs on every startup -
dedup_media_items_by_path() collapses existing path-duplicates and
reparent_media_item_children() moves all child rows (progress,
highlights, bookmarks, notes, collections, formats, aliases, kobo
entitlements, etc.) onto a survivor before deleting losers, so the
constraint applies cleanly on already-duplicated servers without
losing reading history. Survivor picks the row with the most user
data, ties broken by lowest id
- CreateMediaItem: upsert via ON CONFLICT (library_id, file_path) DO
UPDATE so concurrent inserts collapse to one row and return it
- CreateMediaItemFormat: upsert via ON CONFLICT (media_item_id,
format_type), closing the same race on format rows
Application-level guards:
- media_scanner processMediaFile: after computing the file hash, check
GetMediaItemBySHA256AndLibrary (new query) and treat the file as
existing when identical content is already in the library under a
different path (content dedup, library-scoped so multi-library
setups still work)
Ops tooling:
- scripts/dedup_media_items.sql: standalone idempotent maintenance
script with a dry-run report (path + content duplicate groups, child
row counts) and transactional cleanup, for servers that prefer to
dedup manually before upgrading
Verified against the live database: the duplicate pair was collapsed
(reading_progress preserved on the survivor), schema.sql re-runs are a
no-op, and the constraint is in place with 62 unique books remaining.
Migrate from git submodule to npm package management for better
developer experience and simplified deployment.
Changes:
- Add @bookhoard/foliate-js from GitHub fork
(john-okeefe/foliate-js#bookhoard-panel-detection)
- Update vite alias to point to node_modules instead of vendor
- Delete .gitmodules (no submodules tracked)
- Remove scripts/setup-git-hooks.sh (no longer needed)
- Delete web/vendor/foliate-js/ submodule directory
- Remove sc-commit git alias (submodule-specific)
Benefits:
- Standard npm workflow (npm install / npm update)
- No authentication issues for end users (public GitHub)
- Simpler deployment (npm ci in containers)
- foliate-js protected in node_modules (AI won't rewrite)
- Independent project management
- Cleaner git history
Technical details:
- Import remains unchanged: import "foliate-js/view.js"
- Vite alias maps "foliate-js" to "/node_modules/@bookhoard/foliate-js"
- Build verified working (reader.js includes foliate-js)
- Package installed from git branch: bookhoard-panel-detection
- Add chmod +x to ensure pre-push hook is executable after creation
- Add global git alias 'sc-commit' for committing to all submodules at once
- Improve user feedback with detailed explanation of installed components
- Better code organization with clearer comments
This makes the setup script more robust by ensuring the hook has proper permissions and provides a convenient command for bulk submodule commits.
This script installs a pre-push hook that prevents pushing commits when submodules have uncommitted changes, helping avoid accidental commits with dirty submodule states.
The hook checks all submodules for uncommitted changes before allowing a push, protecting against pushing incomplete work that includes submodule modifications.
- Update verification script to check for /static/style.css (local build)
- Reject cdn.tailwindcss.com usage (violates production-ready requirement)
- Local builds are faster, have no external dependencies, and are self-contained
- Changes verification from WARNING to ERROR when CDN is detected
- Now passes all 26 checks with 0 warnings, 0 errors
The verification script was incorrectly flagging .env files that exist locally
but are properly gitignored. Now checks if files are tracked by git using
'git ls-files' instead of just checking file existence.
This prevents false positives when .env is in .gitignore and exists locally
for development but is not committed to the repository.
- Add smart device content detection based on mention thresholds
- Check for README.md files in bruno directory (error)
- Update docs structure checks to match new paths (docs/developer/api, docs/user/devices)
- Add INFO-level warnings for moderate device mentions in docs
- Exclude README.md from device content placement checks
- Improve error recovery with better variable sanitization
- Add complete enhanced output examples for all error/warning types
- Document verification script architecture and design principles
- Include content detection methods and data collection strategies
- Add comprehensive usage scenarios for development, CI/CD, AI workflows
- Provide advanced troubleshooting with debugging techniques
- Include performance optimization and edge case handling
- Demonstrate integration patterns for hooks and pipelines
This creates definitive documentation for verification system that
covers all enhanced outputs, usage patterns, and integration
methods, making scripts fully understandable and actionable for
both human developers and AI assistance.
- Document detailed error/warning output patterns
- Show examples of enhanced information provided
- Explain specific content found for each violation type
- Include examples for both errors and warnings
- Demonstrate improved user experience with actionable details
This documents the latest enhancement where verification scripts now
provide specific file paths, line numbers, and content details
instead of generic error messages, making issues easier to understand
and resolve for both humans and AI assistance.
Enhanced Error/Warning Details:
- API content warnings now show actual lines found
- Build failures now display error logs
- Missing files now show expected locations
- Dockerfile warnings list found files
- Large commits now show specific commit details
- Git ignore issues display current contents
verify-guidelines.sh Comprehensive Documentation Validation:
- Check 12: Content placement validation (API patterns, device setup)
- Check 13: Structure validation (required directories, file counts)
- Check 14: Bruno API tests validation (coverage comparison)
- Full integration of comprehensive documentation checks
- Maintains AI behavior protocol across all operations
User Experience Improvement:
- All errors/warnings now include specific details
- Clear guidance on what was found and why it's an issue
- Verbose output helps identify false positives vs real violations
- Enables faster issue resolution without guesswork
This addresses user request for detailed error/warning information
instead of generic messages, making verification results actionable and
understandable for both humans and AI assistance.
Makefile Integration:
- Add make verify-quick target for critical-only checks
- Fix verify-guidelines target to call comprehensive script
- Clear separation of usage patterns
AI Behavior Protocol:
- Add comprehensive AI instructions to both scripts
- Enhanced error/warning functions with AI reminders
- Multi-layered safeguards prevent automatic fixing
- Protocol applies to ALL file modifications
verify-quick.sh Enhancements:
- Basic documentation structure validation
- API content placement detection in README.md
- Maintains fast performance for development
Documentation:
- Comprehensive scripts/README.md with usage guidelines
- Bruno API tests validation explained
- Troubleshooting and compliance sections
- Clear AI protocol instructions and examples
This provides dual-script approach: fast critical checks during development,
comprehensive validation for pre-commit/CI, with AI safety across all operations.
Check 14: Bruno API Tests Validation
- Count and verify Bruno .bru test files presence
- Compare API documentation vs Bruno test coverage
- Flag insufficient test coverage for human review
Check 15: Recent Documentation Changes Analysis
- Analyze recent commits for documentation compliance
- Flag code commits without corresponding documentation updates
- Verify proper commit message format (docs: prefix)
- Ensure documentation stays synchronized with code changes
These checks provide comprehensive validation of API testing coverage
and ensure documentation follows proper git commit conventions
per PROJECT_GUIDELINES.md requirements.
- Add comprehensive AI instructions at script start and end
- Enhance error/warning functions with AI reminders
- Multi-layered safeguards prevent automatic fixing
- Clear protocol: explain -> propose -> ask permission -> await response
- Instructions apply to ALL file modifications, not just verification issues
These safeguards ensure AI always asks permission before fixing any issues
found by the verification script, preventing automatic corrections of
potential false positives.
Updated PROJECT_GUIDELINES.md and verification script to clarify:
1. OOP restriction applies to FRONTEND (TypeScript) only
2. Go methods are fine and encouraged
3. Avoid classes, inheritance, and OOP bloat in TypeScript
Changed verification script:
- Removed Go struct methods check (was incorrect)
- Added TypeScript class declaration check instead
- Now checks for 'class ' keyword in web/*.ts files
This clarifies the guideline was never about Go backend code,
only about avoiding OOP patterns in TypeScript frontend code.
Verification now shows: 13/13 checks passing, 0 warnings
Only 1 error remains: 12 legacy templates with custom CSS.
Updated verify-quick.sh to follow PROJECT_GUIDELINES.md structure:
- Added comments for each check showing which guideline it verifies
- Reordered checks to match guideline document order
- Expanded from 5 checks to 13 comprehensive checks
New checks added:
- Backend & Database: migration files, pgx v5 driver version
- Frontend & Styling: OOP pattern detection, TailwindCSS usage
- General: git history for secrets, Dockerfile proliferation
- Build & Deployment: code compilation (post-edit verification)
- Configuration: .env.example, .gitignore validation
- Code Modification Safety: commit quality check (no large commits)
Updated scripts/README.md to document all 13 checks with their
corresponding guidelines.
Current status: 12/13 checks passing
- Only 1 error: 12 legacy templates with custom CSS (need Tailwind conversion)
- 1 warning: some Go files have >10 methods (potential OOP, needs manual review)
Exclude web/static/ from verification checks:
- These are TypeScript compiled output files
- Already in .gitignore (web/static/*.js)
- Similar to node_modules/ - build artifacts, not source
Updated verify-quick.sh to exclude:
- web/static/*.js (TypeScript → JS compilation)
- web/static/*.css (TailwindCSS → CSS compilation)
Also removed ./bookhoard binary from repository.
Verification now shows only 1 error: 12 legacy templates with custom CSS.
Docs templates already comply (converted in Phase 4).
Created comprehensive verification script to check codebase against PROJECT_GUIDELINES.md
Features:
- Checks for custom CSS (TailwindCSS requirement)
- Detects JavaScript files that should be TypeScript
- Verifies no secrets committed (.env, credentials.json)
- Validates code compiles (go build)
- Finds local binaries (should use container builds)
- Quick checks with clear pass/fail/warning output
Usage:
make verify-guidelines
./scripts/verify-quick.sh
Current codebase status:
- 12 templates with custom CSS (need Tailwind conversion)
- 2 .js files in web/static/ (need TypeScript conversion)
- 1 binary file (./bookhoard)
This addresses the trust issue: AI now has a tool to prove guideline compliance
john-okeefe