bookhoard/TASKS-scanning-progress.md

# Scanning & Dashboard Issues - Implementation Plan

**Date Created:** 2025-02-24  
**Status:** Documented - Ready to Implement

---

## ✅ Fixed Issues

### Bruno Collection File
**File:** `/home/nymusicman/Code/bookhoard/bruno/scanner/Scan Media Items.yml`

**Problem:** Invalid JSON syntax - library_id variable was not quoted

**Original (Line 20):**
```yaml
"library_id": {{library_id}}   # ❌ WRONG - UUID not quoted
```

**Fixed:**
```yaml
"library_id": "{{library_id}}"   # ✅ CORRECT - quoted string
```

**Impact:** Bruno requests now work correctly. API scanning confirmed functional.

---

## 🚧 Remaining Issues

### Issue 1: Scan Library Button (Frontend)
**Severity:** HIGH - Button completely non-functional  
**File:** `templates/admin.templ` (lines 69-85)

**Current Broken Code:**
```javascript
function quickScan() {
    fetch('/api/scanner/scan', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + localStorage.getItem('token')
        },
        body: JSON.stringify({
            folder_paths: []  // ← EMPTY ARRAY! Causes 400 error
        })
    })
}
```

**Why It Fails:**
- Sends `folder_paths: []` (empty array)
- Handler checks `if len(req.FolderPaths) > 0` → FALSE for empty array
- Returns 400: "either library_id or folder_paths required for scanning"
- No scan job is created
- No progress feedback
- Users can't scan from UI

**What Should Happen:**
1. Fetch all libraries from `/api/libraries`
2. Trigger scan for each library via `/api/libraries/{id}/scan`
3. Collect all job IDs
4. Poll `/api/scanner/status/{jobId}` for progress
5. Display progress UI
6. Show results when complete

---

### Issue 2: Scanned Books Not Showing on Dashboard
**Severity:** MEDIUM - Data exists but not visible  
**Status:** Requires diagnosis

**Symptoms:**
- Book successfully scanned via API
- Book exists in database
- Book not visible on `/dashboard` page
- Book appears in `/api/media-items` endpoint

**Expected Behavior:**
- Book should appear in "recently-added" section
- Should be at top of list (most recent `created_at`)
- Should be visible immediately after scan

**Possible Root Causes:**

#### Hypothesis 1: System Collections Missing
- System collections (including "recently-added") created during user registration
- Possible creation failure or user created before feature existed
- Check: Query database for user's system collections

#### Hypothesis 2: Wrong Library Selected
- Dashboard shows books for selected library only
- Book might be in different library than displayed
- Check: Compare book's library_id with dashboard's selected library

#### Hypothesis 3: Collection Hidden
- User preferences might hide "recently-added" collection
- `show_on_dashboard = false` in database
- Check: User's dashboard preferences

#### Hypothesis 4: Empty Result Set
- Query limit too low
- Ordering incorrect
- Check: API responses directly

---

## 📋 Implementation Plan

### Part 1: Fix Scan Library Button

**NOTE - Major Changes to Original Plan:**
- **Switched from inline JavaScript to TypeScript** (follows PROJECT_GUIDELINES.md: "convert all JavaScript to TypeScript")
- **Uses existing `web/src/admin.ts` infrastructure** instead of adding new inline code
- **No custom CSS** - uses TailwindCSS transition classes for animation (follows "TailwindCSS classes only" rule)
- **Inline CSS with variables retained** - follows existing pattern in admin.templ for theme support

**Rationale:**
- Project already has `web/src/admin.ts` with TypeScript scanning functions
- Inline JavaScript in templates makes code harder to maintain
- TypeScript provides better type safety and code organization
- TailwindCSS transitions are sufficient for UI animation

**Files to Modify:**
- `web/src/admin.ts` (extend TypeScript scanning functions)
- `templates/admin.templ` (add progress UI, include admin.js)

**Implementation Steps:**

#### Step 1: Extend TypeScript in web/src/admin.ts

**Location:** Add new functions after `loadSystemStats()` (around line 103)

**Add these functions:**
```typescript
async function scanAllLibraries(): Promise<void> {
    const token = localStorage.getItem('token');
    if (!token) return;

    try {
        // Step 1: Get all libraries
        const libsResp = await fetch('/api/libraries', {
            headers: { 'Authorization': `Bearer ${token}` }
        });

        if (!libsResp.ok) {
            throw new Error('Failed to get libraries');
        }

        const libsData = await libsResp.json();

        if (!libsData.data || libsData.data.length === 0) {
            if ((window as any).showToast?.error) {
                (window as any).showToast.error('No libraries found. Please create a library first.');
            }
            return;
        }

        const libraries = libsData.data;

        // Step 2: Scan each library
        const jobs: string[] = [];
        const libraryNames: Record<string, string> = {};

        for (const lib of libraries) {
            const scanResp = await fetch(`/api/libraries/${lib.id}/scan`, {
                method: 'POST',
                headers: { 'Authorization': `Bearer ${token}` }
            });

            if (scanResp.ok) {
                const result = await scanResp.json();
                jobs.push(result.job_id);
                libraryNames[result.job_id] = lib.name;
            } else {
                console.error(`Failed to scan library: ${lib.name}`);
            }
        }

        if (jobs.length === 0) {
            if ((window as any).showToast?.error) {
                (window as any).showToast.error('Failed to start scan for any library');
            }
            return;
        }

        // Step 3: Show progress UI
        showScanProgress(jobs, libraryNames);

    } catch (error) {
        console.error('Scan error:', error);
        if ((window as any).showToast?.error) {
            (window as any).showToast.error('Failed to start scan: ' + (error as Error).message);
        }
    }
}

function showScanProgress(jobIds: string[], libraryNames: Record<string, string>): void {
    const container = document.getElementById('scan-progress-container') as HTMLElement;
    const list = document.getElementById('library-progress-list') as HTMLElement;

    if (!container || !list) return;

    container.classList.remove('hidden');
    // Trigger slide-in animation by removing opacity and transform classes
    container.classList.remove('opacity-0', '-translate-y-2.5');

    // Create progress items for each library
    list.innerHTML = jobIds.map(jobId => `
        <div id="progress-${jobId}" class="p-3 rounded border"
             style="background-color: var(--bg-primary); border-color: var(--border);">
            <div class="flex justify-between items-center mb-2">
                <span class="font-medium" style="color: var(--text-primary)">
                    ${libraryNames[jobId]}
                </span>
                <span id="status-${jobId}" class="text-sm" style="color: var(--text-secondary)">
                    Pending...
                </span>
            </div>
            <div class="w-full bg-gray-700 rounded-full h-2">
                <div id="bar-${jobId}"
                     class="h-2 rounded-full transition-all duration-500"
                     style="width: 0%; background-color: var(--accent);">
                </div>
            </div>
        </div>
    `).join('');

    // Start polling
    pollScanProgress(jobIds, libraryNames);
}

function pollScanProgress(jobIds: string[], libraryNames: Record<string, string>): void {
    const token = localStorage.getItem('token');
    const startTime = Date.now();

    const interval = setInterval(async () => {
        let allComplete = true;
        let totalProgress = 0;
        let totalFiles = 0;
        let totalNewItems = 0;
        let totalErrors = 0;

        for (const jobId of jobIds) {
            try {
                const resp = await fetch(`/api/scanner/status/${jobId}`, {
                    headers: { 'Authorization': `Bearer ${token}` }
                });

                if (resp.ok) {
                    const status = await resp.json();

                    // Update individual library progress
                    updateLibraryProgress(jobId, status);

                    totalProgress += status.progress || 0;
                    totalFiles += status.files_scanned || 0;
                    totalNewItems += status.new_items || 0;
                    totalErrors += status.errors || 0;

                    if (status.status !== 'completed' && status.status !== 'failed') {
                        allComplete = false;
                    }
                }
            } catch (error) {
                console.error(`Failed to poll job ${jobId}:`, error);
            }
        }

        // Update overall progress
        const overallProgress = Math.round(totalProgress / jobIds.length);
        const progressBar = document.getElementById('scan-progress-bar') as HTMLElement;
        const progressText = document.getElementById('scan-progress-text') as HTMLElement;
        const statusText = document.getElementById('scan-status') as HTMLElement;

        if (progressBar) progressBar.style.width = overallProgress + '%';
        if (progressText) progressText.textContent = overallProgress + '%';

        // Update status text
        const elapsed = Math.round((Date.now() - startTime) / 1000);
        if (!allComplete && statusText) {
            statusText.textContent = `Scanning... ${elapsed}s elapsed • ${totalFiles} files processed`;
        }

        // Check if all complete
        if (allComplete) {
            clearInterval(interval);
            showScanResults(jobIds.length, totalFiles, totalNewItems, totalErrors, elapsed);
        }
    }, 2000);
}

function updateLibraryProgress(jobId: string, status: any): void {
    const bar = document.getElementById(`bar-${jobId}`) as HTMLElement;
    const statusText = document.getElementById(`status-${jobId}`) as HTMLElement;

    if (bar) {
        bar.style.width = (status.progress || 0) + '%';
    }

    if (statusText) {
        const statusMessages: Record<string, string> = {
            'pending': 'Pending...',
            'running': `Scanning... ${status.progress || 0}%`,
            'completed': `✓ Complete (${status.new_items || 0} items)`,
            'failed': `✗ Failed`
        };
        statusText.textContent = statusMessages[status.status] || status.status;
    }
}

function showScanResults(libCount: number, files: number, items: number, errors: number, elapsed: number): void {
    const resultsDiv = document.getElementById('scan-results') as HTMLElement;
    const contentDiv = document.getElementById('scan-results-content') as HTMLElement;

    if (!resultsDiv || !contentDiv) return;

    contentDiv.innerHTML = `
        <p>• ${libCount} librar${libCount === 1 ? 'y' : 'ies'} scanned</p>
        <p>• ${files} files processed</p>
        <p>• ${items} new items added</p>
        ${errors > 0 ? `<p style="color: var(--accent);">• ${errors} errors</p>` : ''}
        <p style="color: var(--text-secondary)">Completed in ${elapsed} seconds</p>
    `;

    resultsDiv.classList.remove('hidden');

    const statusText = document.getElementById('scan-status') as HTMLElement;
    if (statusText) statusText.textContent = 'Scan complete!';
}

function hideScanProgress(): void {
    const container = document.getElementById('scan-progress-container') as HTMLElement;
    if (container) container.classList.add('hidden');
}

// Export to window
(window as any).scanAllLibraries = scanAllLibraries;
(window as any).hideScanProgress = hideScanProgress;
```

#### Step 2: Update admin.templ to Include admin.js

**Location:** `templates/admin.templ` lines 7-12 (head section)

**Add admin.js script tag:**
```html
<script src="/static/htmx.min.js"></script>
<script src="/static/toast.js"></script>
<script src="/static/admin.js"></script>  <!-- ADD THIS LINE -->
<link href="/static/style.css" rel="stylesheet">
```

#### Step 3: Update Scan Button onClick Handler

**Location:** `templates/admin.templ` line 54

**Change from:**
```html
<button onclick="quickScan()" class="btn-primary p-4 rounded-lg text-left">
```

**Change to:**
```html
<button onclick="scanAllLibraries()" class="btn-primary p-4 rounded-lg text-left">
```

#### Step 3.5: Remove Old Inline JavaScript Function

**Location:** `templates/admin.templ` lines 69-92

**Action:** DELETE the entire `quickScan()` function from the `<script>` section

**What to remove:**
```html
<script>
    function quickScan() {
        fetch('/api/scanner/scan', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Authorization': 'Bearer ' + localStorage.getItem('token')
            },
            body: JSON.stringify({
                folder_paths: []
            })
        }).then(res => res.json()).then(data => {
            alert(data.message || 'Scan completed successfully!');
        }).catch(err => {
            console.error('Scan error:', err);
            alert('Scan failed. Please check your folder configuration.');
        });
    }

    function logout() {
        localStorage.removeItem('token');
        localStorage.removeItem('user');
        window.location.href = '/';
    }

    document.addEventListener('DOMContentLoaded', function() {
        loadTheme();
    });
</script>
```

**Replace with:**
```html
<script>
    function logout() {
        localStorage.removeItem('token');
        localStorage.removeItem('user');
        window.location.href = '/';
    }

    document.addEventListener('DOMContentLoaded', function() {
        loadTheme();
    });
</script>
```

**Note:** Keep `logout()` and `DOMContentLoaded` handlers - only remove `quickScan()`.

#### Step 4: Add Progress UI to admin.templ

**Location:** After the Quick Actions card (after line 64)

**Add this HTML after line 64:**
```html
<!-- Scan Progress Section -->
<div id="scan-progress-container" class="hidden mt-6 p-6 rounded-lg border opacity-0 -translate-y-2.5 transition-all duration-300 ease-out"
     style="background-color: var(--bg-secondary); border-color: var(--border);">
    <div class="flex justify-between items-center mb-4">
        <h3 class="text-lg font-semibold" style="color: var(--text-primary)">
            📚 Scanning Libraries
        </h3>
        <button onclick="hideScanProgress()" class="p-2 hover:bg-gray-700 rounded">
            ✕
        </button>
    </div>

    <!-- Overall Progress -->
    <div class="mb-4">
        <div class="flex justify-between text-sm mb-2">
            <span style="color: var(--text-secondary)">Overall Progress</span>
            <span id="scan-progress-text" style="color: var(--text-primary)">0%</span>
        </div>
        <div class="w-full bg-gray-700 rounded-full h-3">
            <div id="scan-progress-bar"
                 class="h-3 rounded-full transition-all duration-500"
                 style="width: 0%; background-color: var(--accent);">
            </div>
        </div>
        <div id="scan-status" class="text-sm mt-2" style="color: var(--text-secondary)">
            Starting scan...
        </div>
    </div>
    
    <!-- Per-Library Progress -->
    <div id="library-progress-list" class="space-y-3">
        <!-- Dynamically populated -->
    </div>
    
    <!-- Results Summary -->
    <div id="scan-results" class="hidden mt-6 p-4 rounded-lg border"
         style="background-color: var(--bg-primary); border-color: var(--border);">
        <h4 class="font-semibold mb-2" style="color: var(--text-primary)">✅ Scan Complete!</h4>
        <div id="scan-results-content" style="color: var(--text-secondary)">
            <!-- Results populated by JS -->
        </div>
        <div class="mt-4 flex gap-2">
            <button onclick="window.location.reload()" 
                    class="btn-primary px-4 py-2 rounded-lg">
                Refresh to View Books
            </button>
            <button onclick="hideScanProgress()" 
                    class="btn-secondary px-4 py-2 rounded-lg">
                Dismiss
            </button>
        </div>
    </div>
</div>
```

#### Step 5: Build TypeScript to JavaScript

**Build command:**
```bash
npm run build:ts
```

**What this does:**
- Compiles `web/src/admin.ts` to `web/static/admin.js`
- TypeScript compiler (`tsc`) handles the conversion
- Output file `admin.js` will be loaded by the script tag added in Step 2

**Note:** This build step runs automatically in the Docker container during image build. For local development, run it manually after editing TypeScript files.

#### Step 6: Animation Implementation Note

**How the slide-in animation works:**

1. **Initial state** (Step 4 HTML): Container has classes `hidden opacity-0 -translate-y-2.5 transition-all duration-300 ease-out`
2. **When scan starts** (Step 1 TypeScript): `showScanProgress()` function:
   ```typescript
   container.classList.remove('hidden');           // Makes element visible
   container.classList.remove('opacity-0', '-translate-y-2.5');  // Triggers animation
   ```
3. **Result:** Browser transitions from `opacity-0` to `opacity-1` and `-translate-y-2.5` to `translate-y-0` over 300ms

**No custom CSS needed** - follows PROJECT_GUIDELINES.md "TailwindCSS classes only" rule.

---

### Part 2: Diagnose Dashboard Issue

**Diagnostic Steps:**

#### Step 1: Check System Collections Exist
**Bruno Request:**
```
GET /api/dashboard/sections?library_id=849151fb-564e-4b24-89e3-d11360789576
```

**Expected Response:**
```json
{
  "data": [
    {
      "title": "continue-reading",
      "query_type": "continue-reading",
      "items": [...]
    },
    {
      "title": "recently-added",
      "query_type": "recently-added",
      "items": [
        {
          "title": "Leviticus on the Butcher's Block",
          "created_at": "2025-02-24...",
          ...
        }
      ]
    },
    {
      "title": "recently-read",
      "query_type": "recently-read",
      "items": [...]
    },
    {
      "title": "not-started",
      "query_type": "not-started",
      "items": [...]
    }
  ]
}
```

**If sections array is empty or "recently-added" missing:**
- System collections were not created for this user
- Need to manually call `CreateDefaultCollectionsForUser`

#### Step 2: Check Book's Library
**Bruno Request:**
```
GET /api/media-items?library_id=849151fb-564e-4b24-89e3-d11360789576&limit=5&sort=created_at+DESC
```

**Expected:**
- Scanned book should appear first (most recent created_at)
- If book appears here, it's in the correct library

**If book appears:**
- Book is in correct library
- Issue is with dashboard query or collection visibility

**If book doesn't appear:**
- Book was added to different library
- Check other libraries

#### Step 3: Check All Libraries
**Bruno Request:**
```
GET /api/libraries
```

**Purpose:**
- See all available libraries
- Check if book might be in a different library
- Confirm the library_id being used

#### Step 4: Verify URL Library Parameter
**Check:**
- Does `/dashboard` URL have `?library_id=xxx` parameter?
- Which library is selected in dropdown?

**If no library_id parameter:**
- Dashboard auto-selects first visible library
- Book might be in a different library

#### Step 5: Direct Database Check (if needed)

**Check system collections:**
```sql
SELECT name, query_type, show_on_dashboard 
FROM collections 
WHERE user_id = 'your-user-id' 
  AND is_system_collection = true;
```

**Check book's library:**
```sql
SELECT id, title, library_id, created_at 
FROM media_items 
WHERE title LIKE '%Leviticus%'
ORDER BY created_at DESC 
LIMIT 1;
```

---

## 🔧 Potential Fixes for Dashboard Issue

### Fix A: Recreate System Collections
If system collections don't exist:

**Option 1: Manual API Call**
```
POST /api/admin/recreate-system-collections
```
(Endpoint may need to be created)

**Option 2: Direct Database**
```sql
INSERT INTO collections (user_id, name, description, icon, color, show_on_dashboard, query_type, priority, is_system_collection)
VALUES 
  ('your-user-id', 'continue-reading', 'Books you''re currently reading (0 < progress < 1)', '📖', '#7aa2f7', true, 'continue-reading', 1, true),
  ('your-user-id', 'recently-added', 'Newly added items to this library', '🆕', '#9ece6a', true, 'recently-added', 2, true),
  ('your-user-id', 'recently-read', 'Books you''ve finished (progress >= 1)', '✅', '#e0af68', true, 'recently-read', 3, true),
  ('your-user-id', 'not-started', 'Books you haven''t read yet (progress = 0 or no record)', '📕', '#f7768e', true, 'not-started', 4, true);
```

**Option 3: Backend Handler**
Create endpoint to recreate system collections for a user.

### Fix B: Switch to Correct Library
If book is in different library:
- Select the correct library in dropdown
- Or create a combined view showing all libraries

### Fix C: Update User Preferences
If collection is hidden:
```
GET /api/dashboard/preferences?library_id=xxx
```
Check if "recently-added" is in `hidden_collections`

Update:
```
PUT /api/dashboard/preferences
{
  "library_id": "xxx",
  "hidden_collections": [],  // Empty = show all
  "collection_order": [...],
  "items_per_section": 20
}
```

---

## 📊 Implementation Priority

1. **HIGH PRIORITY:** Fix Scan Library button
   - Impact: Users can't scan from UI at all
   - Effort: Medium (2-3 hours)
   - Files: 1 (`admin.templ`)

2. **MEDIUM PRIORITY:** Diagnose dashboard issue
   - Impact: Books exist but not visible
   - Effort: Low (30 min diagnosis)
   - Files: 0 (investigation only)

3. **LOW PRIORITY:** Fix dashboard issue
   - Impact: Depends on root cause
   - Effort: Unknown until diagnosis complete
   - Files: Unknown until diagnosis complete

---

## 🧪 Testing Checklist

### After Fixing Scan Button:

- [ ] Scan button triggers without errors
- [ ] Progress UI appears
- [ ] Progress bar updates every 2 seconds
- [ ] Each library shows individual progress
- [ ] Scan completes and shows results
- [ ] "Refresh to View Books" works
- [ ] Books appear on dashboard after refresh

### After Fixing Dashboard:

- [ ] Scanned book appears in "recently-added" section
- [ ] Book is at top of list (most recent)
- [ ] Book cover displays correctly
- [ ] Clicking book opens it
- [ ] All system collections show data
- [ ] Collections can be hidden/shown
- [ ] Dashboard works across page refreshes

---

## 📝 Notes

- Backend scanning is confirmed working (via Bruno)
- Job status polling endpoint works correctly
- Database contains the scanned book
- Issue is purely frontend/dashboard display logic
- System collections should be created during user registration
- Dashboard supports library switching via dropdown
- User can customize dashboard (hide collections, reorder, change items per section)

---

## 🔗 Related Files

- `templates/admin.templ` - Admin page with Scan button
- `templates/dashboard.templ` - Dashboard template
- `internal/handlers/scanner.go` - Scan endpoints
- `internal/services/dashboard_service.go` - Dashboard logic
- `internal/services/worker.go` - Background job processing
- `internal/handlers/dashboard.go` - Dashboard handlers
- `internal/router/frontend.go` - Dashboard route
- `web/static/dashboard.js` - Dashboard frontend logic
- `bruno/scanner/Scan Media Items.yml` - Bruno collection (FIXED)

---

**Last Updated:** 2025-02-24  
**Status:** Ready to implement Scan button fix, Dashboard issue needs diagnosis