bookhoard/TASKS-scanning-progress.md

Scanning & Dashboard Issues - Implementation Plan

Date Created: 2025-02-24
Status: Documented - Ready to Implement

---

✅ Fixed Issues

Bruno Collection File

File: /home/nymusicman/Code/bookhoard/bruno/scanner/Scan Media Items.yml

Problem: Invalid JSON syntax - library_id variable was not quoted

Original (Line 20):

"library_id": {{library_id}}   # ❌ WRONG - UUID not quoted

Fixed:

"library_id": "{{library_id}}"   # ✅ CORRECT - quoted string

Impact: Bruno requests now work correctly. API scanning confirmed functional.

---

🚧 Remaining Issues

Issue 1: Scan Library Button (Frontend)

Severity: HIGH - Button completely non-functional
File: templates/admin.templ (lines 69-85)

Current Broken Code:

function quickScan() {
    fetch('/api/scanner/scan', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + localStorage.getItem('token')
        },
        body: JSON.stringify({
            folder_paths: []  // ← EMPTY ARRAY! Causes 400 error
        })
    })
}

Why It Fails:

• Sends folder_paths: [] (empty array)
• Handler checks if len(req.FolderPaths) > 0 → FALSE for empty array
• Returns 400: "either library_id or folder_paths required for scanning"
• No scan job is created
• No progress feedback
• Users can't scan from UI

What Should Happen:

1. Fetch all libraries from /api/libraries
2. Trigger scan for each library via /api/libraries/{id}/scan
3. Collect all job IDs
4. Poll /api/scanner/status/{jobId} for progress
5. Display progress UI
6. Show results when complete

---

Issue 2: Scanned Books Not Showing on Dashboard

Severity: MEDIUM - Data exists but not visible
Status: Requires diagnosis

Symptoms:

• Book successfully scanned via API
• Book exists in database
• Book not visible on /dashboard page
• Book appears in /api/media-items endpoint

Expected Behavior:

• Book should appear in "recently-added" section
• Should be at top of list (most recent created_at)
• Should be visible immediately after scan

Possible Root Causes:

Hypothesis 1: System Collections Missing

• System collections (including "recently-added") created during user registration
• Possible creation failure or user created before feature existed
• Check: Query database for user's system collections

Hypothesis 2: Wrong Library Selected

• Dashboard shows books for selected library only
• Book might be in different library than displayed
• Check: Compare book's library_id with dashboard's selected library

Hypothesis 3: Collection Hidden

• User preferences might hide "recently-added" collection
• show_on_dashboard = false in database
• Check: User's dashboard preferences

Hypothesis 4: Empty Result Set

• Query limit too low
• Ordering incorrect
• Check: API responses directly

---

📋 Implementation Plan

Part 1: Fix Scan Library Button

Files to Modify:

• templates/admin.templ

Implementation Steps:

Step 1: Replace quickScan() Function

Location: templates/admin.templ lines 69-85

New Implementation:

async function quickScan() {
    const token = localStorage.getItem('token');
    
    try {
        // Step 1: Get all libraries
        const libsResp = await fetch('/api/libraries', {
            headers: { 'Authorization': `Bearer ${token}` }
        });
        
        if (!libsResp.ok) {
            throw new Error('Failed to get libraries');
        }
        
        const libsData = await libsResp.json();
        
        if (!libsData.data || libsData.data.length === 0) {
            alert('No libraries found. Please create a library first.');
            return;
        }
        
        const libraries = libsData.data;
        
        // Step 2: Scan each library
        const jobs = [];
        const libraryNames = {};
        
        for (const lib of libraries) {
            const scanResp = await fetch(`/api/libraries/${lib.id}/scan`, {
                method: 'POST',
                headers: { 'Authorization': `Bearer ${token}` }
            });
            
            if (scanResp.ok) {
                const result = await scanResp.json();
                jobs.push(result.job_id);
                libraryNames[result.job_id] = lib.name;
            } else {
                console.error(`Failed to scan library: ${lib.name}`);
            }
        }
        
        if (jobs.length === 0) {
            alert('Failed to start scan for any library');
            return;
        }
        
        // Step 3: Show progress UI
        showScanProgress(jobs, libraryNames);
        
    } catch (error) {
        console.error('Scan error:', error);
        alert('Failed to start scan: ' + error.message);
    }
}

Step 2: Add Progress UI to admin.templ

Location: After the Quick Actions grid (around line 64)

Add this HTML:

<!-- Scan Progress Section -->
<div id="scan-progress-container" class="hidden mt-6 p-6 rounded-lg border"
     style="background-color: var(--bg-secondary); border-color: var(--border);">
    <div class="flex justify-between items-center mb-4">
        <h3 class="text-lg font-semibold" style="color: var(--text-primary)">
            📚 Scanning Libraries
        </h3>
        <button onclick="hideScanProgress()" class="p-2 hover:bg-gray-700 rounded">
            ✕
        </button>
    </div>
    
    <!-- Overall Progress -->
    <div class="mb-4">
        <div class="flex justify-between text-sm mb-2">
            <span style="color: var(--text-secondary)">Overall Progress</span>
            <span id="scan-progress-text" style="color: var(--text-primary)">0%</span>
        </div>
        <div class="w-full bg-gray-700 rounded-full h-3">
            <div id="scan-progress-bar" 
                 class="h-3 rounded-full transition-all duration-500"
                 style="width: 0%; background-color: var(--accent);">
            </div>
        </div>
        <div id="scan-status" class="text-sm mt-2" style="color: var(--text-secondary)">
            Starting scan...
        </div>
    </div>
    
    <!-- Per-Library Progress -->
    <div id="library-progress-list" class="space-y-3">
        <!-- Dynamically populated -->
    </div>
    
    <!-- Results Summary -->
    <div id="scan-results" class="hidden mt-6 p-4 rounded-lg border"
         style="background-color: var(--bg-primary); border-color: var(--border);">
        <h4 class="font-semibold mb-2" style="color: var(--text-primary)">✅ Scan Complete!</h4>
        <div id="scan-results-content" style="color: var(--text-secondary)">
            <!-- Results populated by JS -->
        </div>
        <div class="mt-4 flex gap-2">
            <button onclick="window.location.reload()" 
                    class="btn-primary px-4 py-2 rounded-lg">
                Refresh to View Books
            </button>
            <button onclick="hideScanProgress()" 
                    class="btn-secondary px-4 py-2 rounded-lg">
                Dismiss
            </button>
        </div>
    </div>
</div>

Step 3: Add Progress Polling Logic

Location: In the <script> section of admin.templ

Add these functions:

function showScanProgress(jobIds, libraryNames) {
    const container = document.getElementById('scan-progress-container');
    const list = document.getElementById('library-progress-list');
    
    container.classList.remove('hidden');
    
    // Create progress items for each library
    list.innerHTML = jobIds.map(jobId => `
        <div id="progress-${jobId}" class="p-3 rounded border"
             style="background-color: var(--bg-primary); border-color: var(--border);">
            <div class="flex justify-between items-center mb-2">
                <span class="font-medium" style="color: var(--text-primary)">
                    ${libraryNames[jobId]}
                </span>
                <span id="status-${jobId}" class="text-sm" style="color: var(--text-secondary)">
                    Pending...
                </span>
            </div>
            <div class="w-full bg-gray-700 rounded-full h-2">
                <div id="bar-${jobId}" 
                     class="h-2 rounded-full transition-all duration-500"
                     style="width: 0%; background-color: var(--accent);">
                </div>
            </div>
        </div>
    `).join('');
    
    // Start polling
    pollScanProgress(jobIds);
}

async function pollScanProgress(jobIds) {
    const token = localStorage.getItem('token');
    const startTime = Date.now();
    
    const interval = setInterval(async () => {
        let allComplete = true;
        let totalProgress = 0;
        let totalFiles = 0;
        let totalNewItems = 0;
        let totalErrors = 0;
        
        for (const jobId of jobIds) {
            try {
                const resp = await fetch(`/api/scanner/status/${jobId}`, {
                    headers: { 'Authorization': `Bearer ${token}` }
                });
                
                if (resp.ok) {
                    const status = await resp.json();
                    
                    // Update individual library progress
                    updateLibraryProgress(jobId, status);
                    
                    totalProgress += status.progress || 0;
                    totalFiles += status.files_scanned || 0;
                    totalNewItems += status.new_items || 0;
                    totalErrors += status.errors || 0;
                    
                    if (status.status !== 'completed' && status.status !== 'failed') {
                        allComplete = false;
                    }
                }
            } catch (error) {
                console.error(`Failed to poll job ${jobId}:`, error);
            }
        }
        
        // Update overall progress
        const overallProgress = Math.round(totalProgress / jobIds.length);
        document.getElementById('scan-progress-bar').style.width = overallProgress + '%';
        document.getElementById('scan-progress-text').textContent = overallProgress + '%';
        
        // Update status text
        const elapsed = Math.round((Date.now() - startTime) / 1000);
        if (!allComplete) {
            document.getElementById('scan-status').textContent = 
                `Scanning... ${elapsed}s elapsed • ${totalFiles} files processed`;
        }
        
        // Check if all complete
        if (allComplete) {
            clearInterval(interval);
            showScanResults(jobIds.length, totalFiles, totalNewItems, totalErrors, elapsed);
        }
    }, 2000);
}

function updateLibraryProgress(jobId, status) {
    const bar = document.getElementById(`bar-${jobId}`);
    const statusText = document.getElementById(`status-${jobId}`);
    
    if (bar) {
        bar.style.width = (status.progress || 0) + '%';
    }
    
    if (statusText) {
        const statusMessages = {
            'pending': 'Pending...',
            'running': `Scanning... ${status.progress || 0}%`,
            'completed': `✓ Complete (${status.new_items || 0} items)`,
            'failed': `✗ Failed`
        };
        statusText.textContent = statusMessages[status.status] || status.status;
    }
}

function showScanResults(libCount, files, items, errors, elapsed) {
    const resultsDiv = document.getElementById('scan-results');
    const contentDiv = document.getElementById('scan-results-content');
    
    contentDiv.innerHTML = `
        <p>• ${libCount} librar${libCount === 1 ? 'y' : 'ies'} scanned</p>
        <p>• ${files} files processed</p>
        <p>• ${items} new items added</p>
        ${errors > 0 ? `<p style="color: var(--accent);">• ${errors} errors</p>` : ''}
        <p style="color: var(--text-secondary)">Completed in ${elapsed} seconds</p>
    `;
    
    resultsDiv.classList.remove('hidden');
    document.getElementById('scan-status').textContent = 'Scan complete!';
}

function hideScanProgress() {
    document.getElementById('scan-progress-container').classList.add('hidden');
}

Step 4: Add CSS Styling (if needed)

Location: web/static/input.css or web/static/style.css

Most styles use existing CSS variables, but add if needed:

#scan-progress-container {
    animation: slideIn 0.3s ease-out;
}

@keyframes slideIn {
    from {
        opacity: 0;
        transform: translateY(-10px);
    }
    to {
        opacity: 1;
        transform: translateY(0);
    }
}

---

Part 2: Diagnose Dashboard Issue

Diagnostic Steps:

Step 1: Check System Collections Exist

Bruno Request:

GET /api/dashboard/sections?library_id=551ac19c-896a-4406-b479-353fc489b295

Expected Response:

{
  "data": [
    {
      "title": "continue-reading",
      "query_type": "continue-reading",
      "items": [...]
    },
    {
      "title": "recently-added",
      "query_type": "recently-added",
      "items": [
        {
          "title": "Leviticus on the Butcher's Block",
          "created_at": "2025-02-24...",
          ...
        }
      ]
    },
    {
      "title": "recently-read",
      "query_type": "recently-read",
      "items": [...]
    },
    {
      "title": "not-started",
      "query_type": "not-started",
      "items": [...]
    }
  ]
}

If sections array is empty or "recently-added" missing:

• System collections were not created for this user
• Need to manually call CreateDefaultCollectionsForUser

Step 2: Check Book's Library

Bruno Request:

GET /api/media-items?library_id=551ac19c-896a-4406-b479-353fc489b295&limit=5&sort=created_at+DESC

Expected:

• Scanned book should appear first (most recent created_at)
• If book appears here, it's in the correct library

If book appears:

• Book is in correct library
• Issue is with dashboard query or collection visibility

If book doesn't appear:

• Book was added to different library
• Check other libraries

Step 3: Check All Libraries

Bruno Request:

GET /api/libraries

Purpose:

• See all available libraries
• Check if book might be in a different library
• Confirm the library_id being used

Step 4: Verify URL Library Parameter

Check:

• Does /dashboard URL have ?library_id=xxx parameter?
• Which library is selected in dropdown?

If no library_id parameter:

• Dashboard auto-selects first visible library
• Book might be in a different library

Step 5: Direct Database Check (if needed)

Check system collections:

SELECT name, query_type, show_on_dashboard 
FROM collections 
WHERE user_id = 'your-user-id' 
  AND is_system_collection = true;

Check book's library:

SELECT id, title, library_id, created_at 
FROM media_items 
WHERE title LIKE '%Leviticus%'
ORDER BY created_at DESC 
LIMIT 1;

---

🔧 Potential Fixes for Dashboard Issue

Fix A: Recreate System Collections

If system collections don't exist:

Option 1: Manual API Call

POST /api/admin/recreate-system-collections

(Endpoint may need to be created)

Option 2: Direct Database

INSERT INTO collections (user_id, name, description, icon, color, show_on_dashboard, query_type, priority, is_system_collection)
VALUES 
  ('your-user-id', 'continue-reading', 'Books you''re currently reading (0 < progress < 1)', '📖', '#7aa2f7', true, 'continue-reading', 1, true),
  ('your-user-id', 'recently-added', 'Newly added items to this library', '🆕', '#9ece6a', true, 'recently-added', 2, true),
  ('your-user-id', 'recently-read', 'Books you''ve finished (progress >= 1)', '✅', '#e0af68', true, 'recently-read', 3, true),
  ('your-user-id', 'not-started', 'Books you haven''t read yet (progress = 0 or no record)', '📕', '#f7768e', true, 'not-started', 4, true);

Option 3: Backend Handler Create endpoint to recreate system collections for a user.

Fix B: Switch to Correct Library

If book is in different library:

• Select the correct library in dropdown
• Or create a combined view showing all libraries

Fix C: Update User Preferences

If collection is hidden:

GET /api/dashboard/preferences?library_id=xxx

Check if "recently-added" is in hidden_collections

Update:

PUT /api/dashboard/preferences
{
  "library_id": "xxx",
  "hidden_collections": [],  // Empty = show all
  "collection_order": [...],
  "items_per_section": 20
}

---

📊 Implementation Priority

1. HIGH PRIORITY: Fix Scan Library button

   • Impact: Users can't scan from UI at all
   • Effort: Medium (2-3 hours)
   • Files: 1 (admin.templ)

2. MEDIUM PRIORITY: Diagnose dashboard issue

   • Impact: Books exist but not visible
   • Effort: Low (30 min diagnosis)
   • Files: 0 (investigation only)

3. LOW PRIORITY: Fix dashboard issue

   • Impact: Depends on root cause
   • Effort: Unknown until diagnosis complete
   • Files: Unknown until diagnosis complete

---

🧪 Testing Checklist

After Fixing Scan Button:

• Scan button triggers without errors
• Progress UI appears
• Progress bar updates every 2 seconds
• Each library shows individual progress
• Scan completes and shows results
• "Refresh to View Books" works
• Books appear on dashboard after refresh

After Fixing Dashboard:

• Scanned book appears in "recently-added" section
• Book is at top of list (most recent)
• Book cover displays correctly
• Clicking book opens it
• All system collections show data
• Collections can be hidden/shown
• Dashboard works across page refreshes

---

📝 Notes

• Backend scanning is confirmed working (via Bruno)
• Job status polling endpoint works correctly
• Database contains the scanned book
• Issue is purely frontend/dashboard display logic
• System collections should be created during user registration
• Dashboard supports library switching via dropdown
• User can customize dashboard (hide collections, reorder, change items per section)

---

🔗 Related Files

• templates/admin.templ - Admin page with Scan button
• templates/dashboard.templ - Dashboard template
• internal/handlers/scanner.go - Scan endpoints
• internal/services/dashboard_service.go - Dashboard logic
• internal/services/worker.go - Background job processing
• internal/handlers/dashboard.go - Dashboard handlers
• internal/router/frontend.go - Dashboard route
• web/static/dashboard.js - Dashboard frontend logic
• bruno/scanner/Scan Media Items.yml - Bruno collection (FIXED)

---

Last Updated: 2025-02-24
Status: Ready to implement Scan button fix, Dashboard issue needs diagnosis