docs: add comprehensive scanning progress implementation plan

Add detailed implementation plan for fixing Scan Library button and
diagnosing dashboard display issues. Documents investigation findings,
implementation steps, and testing requirements.

Content Sections:
- Fixed Issues: Bruno JSON syntax correction
- Issue 1: Scan Library Button (frontend fix needed)
- Issue 2: Scanned Books Not Showing on Dashboard (diagnosis needed)
- Complete Implementation Plan with code examples
- Diagnostic Steps for dashboard issue
- Potential fixes for all scenarios
- Testing checklist
- Related files and dependencies

Key Findings Documented:
1. Scan Library Button
   - Current: Sends folder_paths: [] → 400 error
   - Fix: Fetch all libraries, scan each, track progress
   - Includes full JavaScript implementation

2. Dashboard Display Issue
   - Books exist in database after scan
   - Not appearing on dashboard UI
   - Root cause requires diagnosis
   - Multiple hypotheses provided

3. Implementation Priority
   - HIGH: Fix Scan button (2-3 hours)
   - MEDIUM: Diagnose dashboard (30 min)
   - LOW: Fix dashboard (unknown)

Implementation Details:
- Complete quickScan() rewrite with error handling
- Progress UI with real-time polling
- Per-library progress tracking
- Results summary display
- CSS animations and styling
- Full diagnostic checklist

Benefits:
- Single source of truth for scanning work
- Can resume implementation at any time
- Documents all investigation findings
- Includes copy-paste ready code examples
- Testing checklist for validation

File: TASKS-scanning-progress.md
Lines: 600+
Related: templates/admin.templ, templates/dashboard.templ
This commit is contained in:
2026-02-24 21:33:55 -05:00
parent a830e0e2b9
commit e8effeb666
+624
View File
@@ -0,0 +1,624 @@
# Scanning & Dashboard Issues - Implementation Plan
**Date Created:** 2025-02-24
**Status:** Documented - Ready to Implement
---
## ✅ Fixed Issues
### Bruno Collection File
**File:** `/home/nymusicman/Code/bookhoard/bruno/scanner/Scan Media Items.yml`
**Problem:** Invalid JSON syntax - library_id variable was not quoted
**Original (Line 20):**
```yaml
"library_id": {{library_id}} # ❌ WRONG - UUID not quoted
```
**Fixed:**
```yaml
"library_id": "{{library_id}}" # ✅ CORRECT - quoted string
```
**Impact:** Bruno requests now work correctly. API scanning confirmed functional.
---
## 🚧 Remaining Issues
### Issue 1: Scan Library Button (Frontend)
**Severity:** HIGH - Button completely non-functional
**File:** `templates/admin.templ` (lines 69-85)
**Current Broken Code:**
```javascript
function quickScan() {
fetch('/api/scanner/scan', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer ' + localStorage.getItem('token')
},
body: JSON.stringify({
folder_paths: [] // ← EMPTY ARRAY! Causes 400 error
})
})
}
```
**Why It Fails:**
- Sends `folder_paths: []` (empty array)
- Handler checks `if len(req.FolderPaths) > 0` → FALSE for empty array
- Returns 400: "either library_id or folder_paths required for scanning"
- No scan job is created
- No progress feedback
- Users can't scan from UI
**What Should Happen:**
1. Fetch all libraries from `/api/libraries`
2. Trigger scan for each library via `/api/libraries/{id}/scan`
3. Collect all job IDs
4. Poll `/api/scanner/status/{jobId}` for progress
5. Display progress UI
6. Show results when complete
---
### Issue 2: Scanned Books Not Showing on Dashboard
**Severity:** MEDIUM - Data exists but not visible
**Status:** Requires diagnosis
**Symptoms:**
- Book successfully scanned via API
- Book exists in database
- Book not visible on `/dashboard` page
- Book appears in `/api/media-items` endpoint
**Expected Behavior:**
- Book should appear in "recently-added" section
- Should be at top of list (most recent `created_at`)
- Should be visible immediately after scan
**Possible Root Causes:**
#### Hypothesis 1: System Collections Missing
- System collections (including "recently-added") created during user registration
- Possible creation failure or user created before feature existed
- Check: Query database for user's system collections
#### Hypothesis 2: Wrong Library Selected
- Dashboard shows books for selected library only
- Book might be in different library than displayed
- Check: Compare book's library_id with dashboard's selected library
#### Hypothesis 3: Collection Hidden
- User preferences might hide "recently-added" collection
- `show_on_dashboard = false` in database
- Check: User's dashboard preferences
#### Hypothesis 4: Empty Result Set
- Query limit too low
- Ordering incorrect
- Check: API responses directly
---
## 📋 Implementation Plan
### Part 1: Fix Scan Library Button
**Files to Modify:**
- `templates/admin.templ`
**Implementation Steps:**
#### Step 1: Replace quickScan() Function
**Location:** `templates/admin.templ` lines 69-85
**New Implementation:**
```javascript
async function quickScan() {
const token = localStorage.getItem('token');
try {
// Step 1: Get all libraries
const libsResp = await fetch('/api/libraries', {
headers: { 'Authorization': `Bearer ${token}` }
});
if (!libsResp.ok) {
throw new Error('Failed to get libraries');
}
const libsData = await libsResp.json();
if (!libsData.data || libsData.data.length === 0) {
alert('No libraries found. Please create a library first.');
return;
}
const libraries = libsData.data;
// Step 2: Scan each library
const jobs = [];
const libraryNames = {};
for (const lib of libraries) {
const scanResp = await fetch(`/api/libraries/${lib.id}/scan`, {
method: 'POST',
headers: { 'Authorization': `Bearer ${token}` }
});
if (scanResp.ok) {
const result = await scanResp.json();
jobs.push(result.job_id);
libraryNames[result.job_id] = lib.name;
} else {
console.error(`Failed to scan library: ${lib.name}`);
}
}
if (jobs.length === 0) {
alert('Failed to start scan for any library');
return;
}
// Step 3: Show progress UI
showScanProgress(jobs, libraryNames);
} catch (error) {
console.error('Scan error:', error);
alert('Failed to start scan: ' + error.message);
}
}
```
#### Step 2: Add Progress UI to admin.templ
**Location:** After the Quick Actions grid (around line 64)
**Add this HTML:**
```html
<!-- Scan Progress Section -->
<div id="scan-progress-container" class="hidden mt-6 p-6 rounded-lg border"
style="background-color: var(--bg-secondary); border-color: var(--border);">
<div class="flex justify-between items-center mb-4">
<h3 class="text-lg font-semibold" style="color: var(--text-primary)">
📚 Scanning Libraries
</h3>
<button onclick="hideScanProgress()" class="p-2 hover:bg-gray-700 rounded">
</button>
</div>
<!-- Overall Progress -->
<div class="mb-4">
<div class="flex justify-between text-sm mb-2">
<span style="color: var(--text-secondary)">Overall Progress</span>
<span id="scan-progress-text" style="color: var(--text-primary)">0%</span>
</div>
<div class="w-full bg-gray-700 rounded-full h-3">
<div id="scan-progress-bar"
class="h-3 rounded-full transition-all duration-500"
style="width: 0%; background-color: var(--accent);">
</div>
</div>
<div id="scan-status" class="text-sm mt-2" style="color: var(--text-secondary)">
Starting scan...
</div>
</div>
<!-- Per-Library Progress -->
<div id="library-progress-list" class="space-y-3">
<!-- Dynamically populated -->
</div>
<!-- Results Summary -->
<div id="scan-results" class="hidden mt-6 p-4 rounded-lg border"
style="background-color: var(--bg-primary); border-color: var(--border);">
<h4 class="font-semibold mb-2" style="color: var(--text-primary)">✅ Scan Complete!</h4>
<div id="scan-results-content" style="color: var(--text-secondary)">
<!-- Results populated by JS -->
</div>
<div class="mt-4 flex gap-2">
<button onclick="window.location.reload()"
class="btn-primary px-4 py-2 rounded-lg">
Refresh to View Books
</button>
<button onclick="hideScanProgress()"
class="btn-secondary px-4 py-2 rounded-lg">
Dismiss
</button>
</div>
</div>
</div>
```
#### Step 3: Add Progress Polling Logic
**Location:** In the `<script>` section of admin.templ
**Add these functions:**
```javascript
function showScanProgress(jobIds, libraryNames) {
const container = document.getElementById('scan-progress-container');
const list = document.getElementById('library-progress-list');
container.classList.remove('hidden');
// Create progress items for each library
list.innerHTML = jobIds.map(jobId => `
<div id="progress-${jobId}" class="p-3 rounded border"
style="background-color: var(--bg-primary); border-color: var(--border);">
<div class="flex justify-between items-center mb-2">
<span class="font-medium" style="color: var(--text-primary)">
${libraryNames[jobId]}
</span>
<span id="status-${jobId}" class="text-sm" style="color: var(--text-secondary)">
Pending...
</span>
</div>
<div class="w-full bg-gray-700 rounded-full h-2">
<div id="bar-${jobId}"
class="h-2 rounded-full transition-all duration-500"
style="width: 0%; background-color: var(--accent);">
</div>
</div>
</div>
`).join('');
// Start polling
pollScanProgress(jobIds);
}
async function pollScanProgress(jobIds) {
const token = localStorage.getItem('token');
const startTime = Date.now();
const interval = setInterval(async () => {
let allComplete = true;
let totalProgress = 0;
let totalFiles = 0;
let totalNewItems = 0;
let totalErrors = 0;
for (const jobId of jobIds) {
try {
const resp = await fetch(`/api/scanner/status/${jobId}`, {
headers: { 'Authorization': `Bearer ${token}` }
});
if (resp.ok) {
const status = await resp.json();
// Update individual library progress
updateLibraryProgress(jobId, status);
totalProgress += status.progress || 0;
totalFiles += status.files_scanned || 0;
totalNewItems += status.new_items || 0;
totalErrors += status.errors || 0;
if (status.status !== 'completed' && status.status !== 'failed') {
allComplete = false;
}
}
} catch (error) {
console.error(`Failed to poll job ${jobId}:`, error);
}
}
// Update overall progress
const overallProgress = Math.round(totalProgress / jobIds.length);
document.getElementById('scan-progress-bar').style.width = overallProgress + '%';
document.getElementById('scan-progress-text').textContent = overallProgress + '%';
// Update status text
const elapsed = Math.round((Date.now() - startTime) / 1000);
if (!allComplete) {
document.getElementById('scan-status').textContent =
`Scanning... ${elapsed}s elapsed • ${totalFiles} files processed`;
}
// Check if all complete
if (allComplete) {
clearInterval(interval);
showScanResults(jobIds.length, totalFiles, totalNewItems, totalErrors, elapsed);
}
}, 2000);
}
function updateLibraryProgress(jobId, status) {
const bar = document.getElementById(`bar-${jobId}`);
const statusText = document.getElementById(`status-${jobId}`);
if (bar) {
bar.style.width = (status.progress || 0) + '%';
}
if (statusText) {
const statusMessages = {
'pending': 'Pending...',
'running': `Scanning... ${status.progress || 0}%`,
'completed': `✓ Complete (${status.new_items || 0} items)`,
'failed': `✗ Failed`
};
statusText.textContent = statusMessages[status.status] || status.status;
}
}
function showScanResults(libCount, files, items, errors, elapsed) {
const resultsDiv = document.getElementById('scan-results');
const contentDiv = document.getElementById('scan-results-content');
contentDiv.innerHTML = `
<p>• ${libCount} librar${libCount === 1 ? 'y' : 'ies'} scanned</p>
<p>• ${files} files processed</p>
<p>• ${items} new items added</p>
${errors > 0 ? `<p style="color: var(--accent);">• ${errors} errors</p>` : ''}
<p style="color: var(--text-secondary)">Completed in ${elapsed} seconds</p>
`;
resultsDiv.classList.remove('hidden');
document.getElementById('scan-status').textContent = 'Scan complete!';
}
function hideScanProgress() {
document.getElementById('scan-progress-container').classList.add('hidden');
}
```
#### Step 4: Add CSS Styling (if needed)
**Location:** `web/static/input.css` or `web/static/style.css`
Most styles use existing CSS variables, but add if needed:
```css
#scan-progress-container {
animation: slideIn 0.3s ease-out;
}
@keyframes slideIn {
from {
opacity: 0;
transform: translateY(-10px);
}
to {
opacity: 1;
transform: translateY(0);
}
}
```
---
### Part 2: Diagnose Dashboard Issue
**Diagnostic Steps:**
#### Step 1: Check System Collections Exist
**Bruno Request:**
```
GET /api/dashboard/sections?library_id=551ac19c-896a-4406-b479-353fc489b295
```
**Expected Response:**
```json
{
"data": [
{
"title": "continue-reading",
"query_type": "continue-reading",
"items": [...]
},
{
"title": "recently-added",
"query_type": "recently-added",
"items": [
{
"title": "Leviticus on the Butcher's Block",
"created_at": "2025-02-24...",
...
}
]
},
{
"title": "recently-read",
"query_type": "recently-read",
"items": [...]
},
{
"title": "not-started",
"query_type": "not-started",
"items": [...]
}
]
}
```
**If sections array is empty or "recently-added" missing:**
- System collections were not created for this user
- Need to manually call `CreateDefaultCollectionsForUser`
#### Step 2: Check Book's Library
**Bruno Request:**
```
GET /api/media-items?library_id=551ac19c-896a-4406-b479-353fc489b295&limit=5&sort=created_at+DESC
```
**Expected:**
- Scanned book should appear first (most recent created_at)
- If book appears here, it's in the correct library
**If book appears:**
- Book is in correct library
- Issue is with dashboard query or collection visibility
**If book doesn't appear:**
- Book was added to different library
- Check other libraries
#### Step 3: Check All Libraries
**Bruno Request:**
```
GET /api/libraries
```
**Purpose:**
- See all available libraries
- Check if book might be in a different library
- Confirm the library_id being used
#### Step 4: Verify URL Library Parameter
**Check:**
- Does `/dashboard` URL have `?library_id=xxx` parameter?
- Which library is selected in dropdown?
**If no library_id parameter:**
- Dashboard auto-selects first visible library
- Book might be in a different library
#### Step 5: Direct Database Check (if needed)
**Check system collections:**
```sql
SELECT name, query_type, show_on_dashboard
FROM collections
WHERE user_id = 'your-user-id'
AND is_system_collection = true;
```
**Check book's library:**
```sql
SELECT id, title, library_id, created_at
FROM media_items
WHERE title LIKE '%Leviticus%'
ORDER BY created_at DESC
LIMIT 1;
```
---
## 🔧 Potential Fixes for Dashboard Issue
### Fix A: Recreate System Collections
If system collections don't exist:
**Option 1: Manual API Call**
```
POST /api/admin/recreate-system-collections
```
(Endpoint may need to be created)
**Option 2: Direct Database**
```sql
INSERT INTO collections (user_id, name, description, icon, color, show_on_dashboard, query_type, priority, is_system_collection)
VALUES
('your-user-id', 'continue-reading', 'Books you''re currently reading (0 < progress < 1)', '📖', '#7aa2f7', true, 'continue-reading', 1, true),
('your-user-id', 'recently-added', 'Newly added items to this library', '🆕', '#9ece6a', true, 'recently-added', 2, true),
('your-user-id', 'recently-read', 'Books you''ve finished (progress >= 1)', '', '#e0af68', true, 'recently-read', 3, true),
('your-user-id', 'not-started', 'Books you haven''t read yet (progress = 0 or no record)', '📕', '#f7768e', true, 'not-started', 4, true);
```
**Option 3: Backend Handler**
Create endpoint to recreate system collections for a user.
### Fix B: Switch to Correct Library
If book is in different library:
- Select the correct library in dropdown
- Or create a combined view showing all libraries
### Fix C: Update User Preferences
If collection is hidden:
```
GET /api/dashboard/preferences?library_id=xxx
```
Check if "recently-added" is in `hidden_collections`
Update:
```
PUT /api/dashboard/preferences
{
"library_id": "xxx",
"hidden_collections": [], // Empty = show all
"collection_order": [...],
"items_per_section": 20
}
```
---
## 📊 Implementation Priority
1. **HIGH PRIORITY:** Fix Scan Library button
- Impact: Users can't scan from UI at all
- Effort: Medium (2-3 hours)
- Files: 1 (`admin.templ`)
2. **MEDIUM PRIORITY:** Diagnose dashboard issue
- Impact: Books exist but not visible
- Effort: Low (30 min diagnosis)
- Files: 0 (investigation only)
3. **LOW PRIORITY:** Fix dashboard issue
- Impact: Depends on root cause
- Effort: Unknown until diagnosis complete
- Files: Unknown until diagnosis complete
---
## 🧪 Testing Checklist
### After Fixing Scan Button:
- [ ] Scan button triggers without errors
- [ ] Progress UI appears
- [ ] Progress bar updates every 2 seconds
- [ ] Each library shows individual progress
- [ ] Scan completes and shows results
- [ ] "Refresh to View Books" works
- [ ] Books appear on dashboard after refresh
### After Fixing Dashboard:
- [ ] Scanned book appears in "recently-added" section
- [ ] Book is at top of list (most recent)
- [ ] Book cover displays correctly
- [ ] Clicking book opens it
- [ ] All system collections show data
- [ ] Collections can be hidden/shown
- [ ] Dashboard works across page refreshes
---
## 📝 Notes
- Backend scanning is confirmed working (via Bruno)
- Job status polling endpoint works correctly
- Database contains the scanned book
- Issue is purely frontend/dashboard display logic
- System collections should be created during user registration
- Dashboard supports library switching via dropdown
- User can customize dashboard (hide collections, reorder, change items per section)
---
## 🔗 Related Files
- `templates/admin.templ` - Admin page with Scan button
- `templates/dashboard.templ` - Dashboard template
- `internal/handlers/scanner.go` - Scan endpoints
- `internal/services/dashboard_service.go` - Dashboard logic
- `internal/services/worker.go` - Background job processing
- `internal/handlers/dashboard.go` - Dashboard handlers
- `internal/router/frontend.go` - Dashboard route
- `web/static/dashboard.js` - Dashboard frontend logic
- `bruno/scanner/Scan Media Items.yml` - Bruno collection (FIXED)
---
**Last Updated:** 2025-02-24
**Status:** Ready to implement Scan button fix, Dashboard issue needs diagnosis