bookhoard/TASKS-scanning-progress.md

# Scanning & Dashboard Issues - Implementation Plan

**Date Created:** 2025-02-24  
**Status:** Documented - Ready to Implement

---

## ✅ Fixed Issues

### Bruno Collection File
**File:** `/home/nymusicman/Code/bookhoard/bruno/scanner/Scan Media Items.yml`

**Problem:** Invalid JSON syntax - library_id variable was not quoted

**Original (Line 20):**
```yaml
"library_id": {{library_id}}   # ❌ WRONG - UUID not quoted
```

**Fixed:**
```yaml
"library_id": "{{library_id}}"   # ✅ CORRECT - quoted string
```

**Impact:** Bruno requests now work correctly. API scanning confirmed functional.

---

## 🚧 Remaining Issues

### Issue 1: Scan Library Button (Frontend)
**Severity:** HIGH - Button completely non-functional  
**File:** `templates/admin.templ` (lines 69-85)

**Current Broken Code:**
```javascript
function quickScan() {
    fetch('/api/scanner/scan', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + localStorage.getItem('token')
        },
        body: JSON.stringify({
            folder_paths: []  // ← EMPTY ARRAY! Causes 400 error
        })
    })
}
```

**Why It Fails:**
- Sends `folder_paths: []` (empty array)
- Handler checks `if len(req.FolderPaths) > 0` → FALSE for empty array
- Returns 400: "either library_id or folder_paths required for scanning"
- No scan job is created
- No progress feedback
- Users can't scan from UI

**What Should Happen:**
1. Fetch all libraries from `/api/libraries`
2. Trigger scan for each library via `/api/libraries/{id}/scan`
3. Collect all job IDs
4. Poll `/api/scanner/status/{jobId}` for progress
5. Display progress UI
6. Show results when complete

---

### Issue 2: Scanned Books Not Showing on Dashboard
**Severity:** MEDIUM - Data exists but not visible  
**Status:** Requires diagnosis

**Symptoms:**
- Book successfully scanned via API
- Book exists in database
- Book not visible on `/dashboard` page
- Book appears in `/api/media-items` endpoint

**Expected Behavior:**
- Book should appear in "recently-added" section
- Should be at top of list (most recent `created_at`)
- Should be visible immediately after scan

**Possible Root Causes:**

#### Hypothesis 1: System Collections Missing
- System collections (including "recently-added") created during user registration
- Possible creation failure or user created before feature existed
- Check: Query database for user's system collections

#### Hypothesis 2: Wrong Library Selected
- Dashboard shows books for selected library only
- Book might be in different library than displayed
- Check: Compare book's library_id with dashboard's selected library

#### Hypothesis 3: Collection Hidden
- User preferences might hide "recently-added" collection
- `show_on_dashboard = false` in database
- Check: User's dashboard preferences

#### Hypothesis 4: Empty Result Set
- Query limit too low
- Ordering incorrect
- Check: API responses directly

---

## 📋 Implementation Plan

### Part 1: Fix Scan Library Button

**Files to Modify:**
- `templates/admin.templ`

**Implementation Steps:**

#### Step 1: Replace quickScan() Function

**Location:** `templates/admin.templ` lines 69-85

**New Implementation:**
```javascript
async function quickScan() {
    const token = localStorage.getItem('token');
    
    try {
        // Step 1: Get all libraries
        const libsResp = await fetch('/api/libraries', {
            headers: { 'Authorization': `Bearer ${token}` }
        });
        
        if (!libsResp.ok) {
            throw new Error('Failed to get libraries');
        }
        
        const libsData = await libsResp.json();
        
        if (!libsData.data || libsData.data.length === 0) {
            alert('No libraries found. Please create a library first.');
            return;
        }
        
        const libraries = libsData.data;
        
        // Step 2: Scan each library
        const jobs = [];
        const libraryNames = {};
        
        for (const lib of libraries) {
            const scanResp = await fetch(`/api/libraries/${lib.id}/scan`, {
                method: 'POST',
                headers: { 'Authorization': `Bearer ${token}` }
            });
            
            if (scanResp.ok) {
                const result = await scanResp.json();
                jobs.push(result.job_id);
                libraryNames[result.job_id] = lib.name;
            } else {
                console.error(`Failed to scan library: ${lib.name}`);
            }
        }
        
        if (jobs.length === 0) {
            alert('Failed to start scan for any library');
            return;
        }
        
        // Step 3: Show progress UI
        showScanProgress(jobs, libraryNames);
        
    } catch (error) {
        console.error('Scan error:', error);
        alert('Failed to start scan: ' + error.message);
    }
}
```

#### Step 2: Add Progress UI to admin.templ

**Location:** After the Quick Actions grid (around line 64)

**Add this HTML:**
```html
<!-- Scan Progress Section -->
<div id="scan-progress-container" class="hidden mt-6 p-6 rounded-lg border"
     style="background-color: var(--bg-secondary); border-color: var(--border);">
    <div class="flex justify-between items-center mb-4">
        <h3 class="text-lg font-semibold" style="color: var(--text-primary)">
            📚 Scanning Libraries
        </h3>
        <button onclick="hideScanProgress()" class="p-2 hover:bg-gray-700 rounded">
            ✕
        </button>
    </div>
    
    <!-- Overall Progress -->
    <div class="mb-4">
        <div class="flex justify-between text-sm mb-2">
            <span style="color: var(--text-secondary)">Overall Progress</span>
            <span id="scan-progress-text" style="color: var(--text-primary)">0%</span>
        </div>
        <div class="w-full bg-gray-700 rounded-full h-3">
            <div id="scan-progress-bar" 
                 class="h-3 rounded-full transition-all duration-500"
                 style="width: 0%; background-color: var(--accent);">
            </div>
        </div>
        <div id="scan-status" class="text-sm mt-2" style="color: var(--text-secondary)">
            Starting scan...
        </div>
    </div>
    
    <!-- Per-Library Progress -->
    <div id="library-progress-list" class="space-y-3">
        <!-- Dynamically populated -->
    </div>
    
    <!-- Results Summary -->
    <div id="scan-results" class="hidden mt-6 p-4 rounded-lg border"
         style="background-color: var(--bg-primary); border-color: var(--border);">
        <h4 class="font-semibold mb-2" style="color: var(--text-primary)">✅ Scan Complete!</h4>
        <div id="scan-results-content" style="color: var(--text-secondary)">
            <!-- Results populated by JS -->
        </div>
        <div class="mt-4 flex gap-2">
            <button onclick="window.location.reload()" 
                    class="btn-primary px-4 py-2 rounded-lg">
                Refresh to View Books
            </button>
            <button onclick="hideScanProgress()" 
                    class="btn-secondary px-4 py-2 rounded-lg">
                Dismiss
            </button>
        </div>
    </div>
</div>
```

#### Step 3: Add Progress Polling Logic

**Location:** In the `<script>` section of admin.templ

**Add these functions:**
```javascript
function showScanProgress(jobIds, libraryNames) {
    const container = document.getElementById('scan-progress-container');
    const list = document.getElementById('library-progress-list');
    
    container.classList.remove('hidden');
    
    // Create progress items for each library
    list.innerHTML = jobIds.map(jobId => `
        <div id="progress-${jobId}" class="p-3 rounded border"
             style="background-color: var(--bg-primary); border-color: var(--border);">
            <div class="flex justify-between items-center mb-2">
                <span class="font-medium" style="color: var(--text-primary)">
                    ${libraryNames[jobId]}
                </span>
                <span id="status-${jobId}" class="text-sm" style="color: var(--text-secondary)">
                    Pending...
                </span>
            </div>
            <div class="w-full bg-gray-700 rounded-full h-2">
                <div id="bar-${jobId}" 
                     class="h-2 rounded-full transition-all duration-500"
                     style="width: 0%; background-color: var(--accent);">
                </div>
            </div>
        </div>
    `).join('');
    
    // Start polling
    pollScanProgress(jobIds);
}

async function pollScanProgress(jobIds) {
    const token = localStorage.getItem('token');
    const startTime = Date.now();
    
    const interval = setInterval(async () => {
        let allComplete = true;
        let totalProgress = 0;
        let totalFiles = 0;
        let totalNewItems = 0;
        let totalErrors = 0;
        
        for (const jobId of jobIds) {
            try {
                const resp = await fetch(`/api/scanner/status/${jobId}`, {
                    headers: { 'Authorization': `Bearer ${token}` }
                });
                
                if (resp.ok) {
                    const status = await resp.json();
                    
                    // Update individual library progress
                    updateLibraryProgress(jobId, status);
                    
                    totalProgress += status.progress || 0;
                    totalFiles += status.files_scanned || 0;
                    totalNewItems += status.new_items || 0;
                    totalErrors += status.errors || 0;
                    
                    if (status.status !== 'completed' && status.status !== 'failed') {
                        allComplete = false;
                    }
                }
            } catch (error) {
                console.error(`Failed to poll job ${jobId}:`, error);
            }
        }
        
        // Update overall progress
        const overallProgress = Math.round(totalProgress / jobIds.length);
        document.getElementById('scan-progress-bar').style.width = overallProgress + '%';
        document.getElementById('scan-progress-text').textContent = overallProgress + '%';
        
        // Update status text
        const elapsed = Math.round((Date.now() - startTime) / 1000);
        if (!allComplete) {
            document.getElementById('scan-status').textContent = 
                `Scanning... ${elapsed}s elapsed • ${totalFiles} files processed`;
        }
        
        // Check if all complete
        if (allComplete) {
            clearInterval(interval);
            showScanResults(jobIds.length, totalFiles, totalNewItems, totalErrors, elapsed);
        }
    }, 2000);
}

function updateLibraryProgress(jobId, status) {
    const bar = document.getElementById(`bar-${jobId}`);
    const statusText = document.getElementById(`status-${jobId}`);
    
    if (bar) {
        bar.style.width = (status.progress || 0) + '%';
    }
    
    if (statusText) {
        const statusMessages = {
            'pending': 'Pending...',
            'running': `Scanning... ${status.progress || 0}%`,
            'completed': `✓ Complete (${status.new_items || 0} items)`,
            'failed': `✗ Failed`
        };
        statusText.textContent = statusMessages[status.status] || status.status;
    }
}

function showScanResults(libCount, files, items, errors, elapsed) {
    const resultsDiv = document.getElementById('scan-results');
    const contentDiv = document.getElementById('scan-results-content');
    
    contentDiv.innerHTML = `
        <p>• ${libCount} librar${libCount === 1 ? 'y' : 'ies'} scanned</p>
        <p>• ${files} files processed</p>
        <p>• ${items} new items added</p>
        ${errors > 0 ? `<p style="color: var(--accent);">• ${errors} errors</p>` : ''}
        <p style="color: var(--text-secondary)">Completed in ${elapsed} seconds</p>
    `;
    
    resultsDiv.classList.remove('hidden');
    document.getElementById('scan-status').textContent = 'Scan complete!';
}

function hideScanProgress() {
    document.getElementById('scan-progress-container').classList.add('hidden');
}
```

#### Step 4: Add CSS Styling (if needed)

**Location:** `web/static/input.css` or `web/static/style.css`

Most styles use existing CSS variables, but add if needed:
```css
#scan-progress-container {
    animation: slideIn 0.3s ease-out;
}

@keyframes slideIn {
    from {
        opacity: 0;
        transform: translateY(-10px);
    }
    to {
        opacity: 1;
        transform: translateY(0);
    }
}
```

---

### Part 2: Diagnose Dashboard Issue

**Diagnostic Steps:**

#### Step 1: Check System Collections Exist
**Bruno Request:**
```
GET /api/dashboard/sections?library_id=551ac19c-896a-4406-b479-353fc489b295
```

**Expected Response:**
```json
{
  "data": [
    {
      "title": "continue-reading",
      "query_type": "continue-reading",
      "items": [...]
    },
    {
      "title": "recently-added",
      "query_type": "recently-added",
      "items": [
        {
          "title": "Leviticus on the Butcher's Block",
          "created_at": "2025-02-24...",
          ...
        }
      ]
    },
    {
      "title": "recently-read",
      "query_type": "recently-read",
      "items": [...]
    },
    {
      "title": "not-started",
      "query_type": "not-started",
      "items": [...]
    }
  ]
}
```

**If sections array is empty or "recently-added" missing:**
- System collections were not created for this user
- Need to manually call `CreateDefaultCollectionsForUser`

#### Step 2: Check Book's Library
**Bruno Request:**
```
GET /api/media-items?library_id=551ac19c-896a-4406-b479-353fc489b295&limit=5&sort=created_at+DESC
```

**Expected:**
- Scanned book should appear first (most recent created_at)
- If book appears here, it's in the correct library

**If book appears:**
- Book is in correct library
- Issue is with dashboard query or collection visibility

**If book doesn't appear:**
- Book was added to different library
- Check other libraries

#### Step 3: Check All Libraries
**Bruno Request:**
```
GET /api/libraries
```

**Purpose:**
- See all available libraries
- Check if book might be in a different library
- Confirm the library_id being used

#### Step 4: Verify URL Library Parameter
**Check:**
- Does `/dashboard` URL have `?library_id=xxx` parameter?
- Which library is selected in dropdown?

**If no library_id parameter:**
- Dashboard auto-selects first visible library
- Book might be in a different library

#### Step 5: Direct Database Check (if needed)

**Check system collections:**
```sql
SELECT name, query_type, show_on_dashboard 
FROM collections 
WHERE user_id = 'your-user-id' 
  AND is_system_collection = true;
```

**Check book's library:**
```sql
SELECT id, title, library_id, created_at 
FROM media_items 
WHERE title LIKE '%Leviticus%'
ORDER BY created_at DESC 
LIMIT 1;
```

---

## 🔧 Potential Fixes for Dashboard Issue

### Fix A: Recreate System Collections
If system collections don't exist:

**Option 1: Manual API Call**
```
POST /api/admin/recreate-system-collections
```
(Endpoint may need to be created)

**Option 2: Direct Database**
```sql
INSERT INTO collections (user_id, name, description, icon, color, show_on_dashboard, query_type, priority, is_system_collection)
VALUES 
  ('your-user-id', 'continue-reading', 'Books you''re currently reading (0 < progress < 1)', '📖', '#7aa2f7', true, 'continue-reading', 1, true),
  ('your-user-id', 'recently-added', 'Newly added items to this library', '🆕', '#9ece6a', true, 'recently-added', 2, true),
  ('your-user-id', 'recently-read', 'Books you''ve finished (progress >= 1)', '✅', '#e0af68', true, 'recently-read', 3, true),
  ('your-user-id', 'not-started', 'Books you haven''t read yet (progress = 0 or no record)', '📕', '#f7768e', true, 'not-started', 4, true);
```

**Option 3: Backend Handler**
Create endpoint to recreate system collections for a user.

### Fix B: Switch to Correct Library
If book is in different library:
- Select the correct library in dropdown
- Or create a combined view showing all libraries

### Fix C: Update User Preferences
If collection is hidden:
```
GET /api/dashboard/preferences?library_id=xxx
```
Check if "recently-added" is in `hidden_collections`

Update:
```
PUT /api/dashboard/preferences
{
  "library_id": "xxx",
  "hidden_collections": [],  // Empty = show all
  "collection_order": [...],
  "items_per_section": 20
}
```

---

## 📊 Implementation Priority

1. **HIGH PRIORITY:** Fix Scan Library button
   - Impact: Users can't scan from UI at all
   - Effort: Medium (2-3 hours)
   - Files: 1 (`admin.templ`)

2. **MEDIUM PRIORITY:** Diagnose dashboard issue
   - Impact: Books exist but not visible
   - Effort: Low (30 min diagnosis)
   - Files: 0 (investigation only)

3. **LOW PRIORITY:** Fix dashboard issue
   - Impact: Depends on root cause
   - Effort: Unknown until diagnosis complete
   - Files: Unknown until diagnosis complete

---

## 🧪 Testing Checklist

### After Fixing Scan Button:

- [ ] Scan button triggers without errors
- [ ] Progress UI appears
- [ ] Progress bar updates every 2 seconds
- [ ] Each library shows individual progress
- [ ] Scan completes and shows results
- [ ] "Refresh to View Books" works
- [ ] Books appear on dashboard after refresh

### After Fixing Dashboard:

- [ ] Scanned book appears in "recently-added" section
- [ ] Book is at top of list (most recent)
- [ ] Book cover displays correctly
- [ ] Clicking book opens it
- [ ] All system collections show data
- [ ] Collections can be hidden/shown
- [ ] Dashboard works across page refreshes

---

## 📝 Notes

- Backend scanning is confirmed working (via Bruno)
- Job status polling endpoint works correctly
- Database contains the scanned book
- Issue is purely frontend/dashboard display logic
- System collections should be created during user registration
- Dashboard supports library switching via dropdown
- User can customize dashboard (hide collections, reorder, change items per section)

---

## 🔗 Related Files

- `templates/admin.templ` - Admin page with Scan button
- `templates/dashboard.templ` - Dashboard template
- `internal/handlers/scanner.go` - Scan endpoints
- `internal/services/dashboard_service.go` - Dashboard logic
- `internal/services/worker.go` - Background job processing
- `internal/handlers/dashboard.go` - Dashboard handlers
- `internal/router/frontend.go` - Dashboard route
- `web/static/dashboard.js` - Dashboard frontend logic
- `bruno/scanner/Scan Media Items.yml` - Bruno collection (FIXED)

---

**Last Updated:** 2025-02-24  
**Status:** Ready to implement Scan button fix, Dashboard issue needs diagnosis