bookhoard/TASKS-scanning-progress.md

Scanning & Dashboard Issues - Implementation Plan

Date Created: 2025-02-24
Status: Documented - Ready to Implement

---

✅ Fixed Issues

Bruno Collection File

File: /home/nymusicman/Code/bookhoard/bruno/scanner/Scan Media Items.yml

Problem: Invalid JSON syntax - library_id variable was not quoted

Original (Line 20):

"library_id": {{library_id}}   # ❌ WRONG - UUID not quoted

Fixed:

"library_id": "{{library_id}}"   # ✅ CORRECT - quoted string

Impact: Bruno requests now work correctly. API scanning confirmed functional.

---

🚧 Remaining Issues

Issue 1: Scan Library Button (Frontend)

Severity: HIGH - Button completely non-functional
File: templates/admin.templ (lines 69-85)

Current Broken Code:

function quickScan() {
    fetch('/api/scanner/scan', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer ' + localStorage.getItem('token')
        },
        body: JSON.stringify({
            folder_paths: []  // ← EMPTY ARRAY! Causes 400 error
        })
    })
}

Why It Fails:

• Sends folder_paths: [] (empty array)
• Handler checks if len(req.FolderPaths) > 0 → FALSE for empty array
• Returns 400: "either library_id or folder_paths required for scanning"
• No scan job is created
• No progress feedback
• Users can't scan from UI

What Should Happen:

1. Fetch all libraries from /api/libraries
2. Trigger scan for each library via /api/libraries/{id}/scan
3. Collect all job IDs
4. Poll /api/scanner/status/{jobId} for progress
5. Display progress UI
6. Show results when complete

---

Issue 2: Scanned Books Not Showing on Dashboard

Severity: MEDIUM - Data exists but not visible
Status: Requires diagnosis

Symptoms:

• Book successfully scanned via API
• Book exists in database
• Book not visible on /dashboard page
• Book appears in /api/media-items endpoint

Expected Behavior:

• Book should appear in "recently-added" section
• Should be at top of list (most recent created_at)
• Should be visible immediately after scan

Possible Root Causes:

Hypothesis 1: System Collections Missing

• System collections (including "recently-added") created during user registration
• Possible creation failure or user created before feature existed
• Check: Query database for user's system collections

Hypothesis 2: Wrong Library Selected

• Dashboard shows books for selected library only
• Book might be in different library than displayed
• Check: Compare book's library_id with dashboard's selected library

Hypothesis 3: Collection Hidden

• User preferences might hide "recently-added" collection
• show_on_dashboard = false in database
• Check: User's dashboard preferences

Hypothesis 4: Empty Result Set

• Query limit too low
• Ordering incorrect
• Check: API responses directly

---

📋 Implementation Plan

Part 1: Fix Scan Library Button

NOTE - Major Changes to Original Plan:

• Switched from inline JavaScript to TypeScript (follows PROJECT_GUIDELINES.md: "convert all JavaScript to TypeScript")
• Uses existing web/src/admin.ts infrastructure instead of adding new inline code
• No custom CSS - uses TailwindCSS transition classes for animation (follows "TailwindCSS classes only" rule)
• Inline CSS with variables retained - follows existing pattern in admin.templ for theme support

Rationale:

• Project already has web/src/admin.ts with TypeScript scanning functions
• Inline JavaScript in templates makes code harder to maintain
• TypeScript provides better type safety and code organization
• TailwindCSS transitions are sufficient for UI animation

Files to Modify:

• web/src/admin.ts (extend TypeScript scanning functions)
• templates/admin.templ (add progress UI, include admin.js)

Implementation Steps:

Step 1: Extend TypeScript in web/src/admin.ts

Location: Add new functions after loadSystemStats() (around line 103)

Add these functions:

async function scanAllLibraries(): Promise<void> {
    const token = localStorage.getItem('token');
    if (!token) return;

    try {
        // Step 1: Get all libraries
        const libsResp = await fetch('/api/libraries', {
            headers: { 'Authorization': `Bearer ${token}` }
        });

        if (!libsResp.ok) {
            throw new Error('Failed to get libraries');
        }

        const libsData = await libsResp.json();

        if (!libsData.data || libsData.data.length === 0) {
            if ((window as any).showToast?.error) {
                (window as any).showToast.error('No libraries found. Please create a library first.');
            }
            return;
        }

        const libraries = libsData.data;

        // Step 2: Scan each library
        const jobs: string[] = [];
        const libraryNames: Record<string, string> = {};

        for (const lib of libraries) {
            const scanResp = await fetch(`/api/libraries/${lib.id}/scan`, {
                method: 'POST',
                headers: { 'Authorization': `Bearer ${token}` }
            });

            if (scanResp.ok) {
                const result = await scanResp.json();
                jobs.push(result.job_id);
                libraryNames[result.job_id] = lib.name;
            } else {
                console.error(`Failed to scan library: ${lib.name}`);
            }
        }

        if (jobs.length === 0) {
            if ((window as any).showToast?.error) {
                (window as any).showToast.error('Failed to start scan for any library');
            }
            return;
        }

        // Step 3: Show progress UI
        showScanProgress(jobs, libraryNames);

    } catch (error) {
        console.error('Scan error:', error);
        if ((window as any).showToast?.error) {
            (window as any).showToast.error('Failed to start scan: ' + (error as Error).message);
        }
    }
}

function showScanProgress(jobIds: string[], libraryNames: Record<string, string>): void {
    const container = document.getElementById('scan-progress-container') as HTMLElement;
    const list = document.getElementById('library-progress-list') as HTMLElement;

    if (!container || !list) return;

    container.classList.remove('hidden');
    // Trigger slide-in animation by removing opacity and transform classes
    container.classList.remove('opacity-0', '-translate-y-2.5');

    // Create progress items for each library
    list.innerHTML = jobIds.map(jobId => `
        <div id="progress-${jobId}" class="p-3 rounded border"
             style="background-color: var(--bg-primary); border-color: var(--border);">
            <div class="flex justify-between items-center mb-2">
                <span class="font-medium" style="color: var(--text-primary)">
                    ${libraryNames[jobId]}
                </span>
                <span id="status-${jobId}" class="text-sm" style="color: var(--text-secondary)">
                    Pending...
                </span>
            </div>
            <div class="w-full bg-gray-700 rounded-full h-2">
                <div id="bar-${jobId}"
                     class="h-2 rounded-full transition-all duration-500"
                     style="width: 0%; background-color: var(--accent);">
                </div>
            </div>
        </div>
    `).join('');

    // Start polling
    pollScanProgress(jobIds, libraryNames);
}

function pollScanProgress(jobIds: string[], libraryNames: Record<string, string>): void {
    const token = localStorage.getItem('token');
    const startTime = Date.now();

    const interval = setInterval(async () => {
        let allComplete = true;
        let totalProgress = 0;
        let totalFiles = 0;
        let totalNewItems = 0;
        let totalErrors = 0;

        for (const jobId of jobIds) {
            try {
                const resp = await fetch(`/api/scanner/status/${jobId}`, {
                    headers: { 'Authorization': `Bearer ${token}` }
                });

                if (resp.ok) {
                    const status = await resp.json();

                    // Update individual library progress
                    updateLibraryProgress(jobId, status);

                    totalProgress += status.progress || 0;
                    totalFiles += status.files_scanned || 0;
                    totalNewItems += status.new_items || 0;
                    totalErrors += status.errors || 0;

                    if (status.status !== 'completed' && status.status !== 'failed') {
                        allComplete = false;
                    }
                }
            } catch (error) {
                console.error(`Failed to poll job ${jobId}:`, error);
            }
        }

        // Update overall progress
        const overallProgress = Math.round(totalProgress / jobIds.length);
        const progressBar = document.getElementById('scan-progress-bar') as HTMLElement;
        const progressText = document.getElementById('scan-progress-text') as HTMLElement;
        const statusText = document.getElementById('scan-status') as HTMLElement;

        if (progressBar) progressBar.style.width = overallProgress + '%';
        if (progressText) progressText.textContent = overallProgress + '%';

        // Update status text
        const elapsed = Math.round((Date.now() - startTime) / 1000);
        if (!allComplete && statusText) {
            statusText.textContent = `Scanning... ${elapsed}s elapsed • ${totalFiles} files processed`;
        }

        // Check if all complete
        if (allComplete) {
            clearInterval(interval);
            showScanResults(jobIds.length, totalFiles, totalNewItems, totalErrors, elapsed);
        }
    }, 2000);
}

function updateLibraryProgress(jobId: string, status: any): void {
    const bar = document.getElementById(`bar-${jobId}`) as HTMLElement;
    const statusText = document.getElementById(`status-${jobId}`) as HTMLElement;

    if (bar) {
        bar.style.width = (status.progress || 0) + '%';
    }

    if (statusText) {
        const statusMessages: Record<string, string> = {
            'pending': 'Pending...',
            'running': `Scanning... ${status.progress || 0}%`,
            'completed': `✓ Complete (${status.new_items || 0} items)`,
            'failed': `✗ Failed`
        };
        statusText.textContent = statusMessages[status.status] || status.status;
    }
}

function showScanResults(libCount: number, files: number, items: number, errors: number, elapsed: number): void {
    const resultsDiv = document.getElementById('scan-results') as HTMLElement;
    const contentDiv = document.getElementById('scan-results-content') as HTMLElement;

    if (!resultsDiv || !contentDiv) return;

    contentDiv.innerHTML = `
        <p>• ${libCount} librar${libCount === 1 ? 'y' : 'ies'} scanned</p>
        <p>• ${files} files processed</p>
        <p>• ${items} new items added</p>
        ${errors > 0 ? `<p style="color: var(--accent);">• ${errors} errors</p>` : ''}
        <p style="color: var(--text-secondary)">Completed in ${elapsed} seconds</p>
    `;

    resultsDiv.classList.remove('hidden');

    const statusText = document.getElementById('scan-status') as HTMLElement;
    if (statusText) statusText.textContent = 'Scan complete!';
}

function hideScanProgress(): void {
    const container = document.getElementById('scan-progress-container') as HTMLElement;
    if (container) container.classList.add('hidden');
}

// Export to window
(window as any).scanAllLibraries = scanAllLibraries;
(window as any).hideScanProgress = hideScanProgress;

Step 2: Update admin.templ to Include admin.js

Location: templates/admin.templ lines 7-12 (head section)

Add admin.js script tag:

<script src="/static/htmx.min.js"></script>
<script src="/static/toast.js"></script>
<script src="/static/admin.js"></script>  <!-- ADD THIS LINE -->
<link href="/static/style.css" rel="stylesheet">

Step 3: Update Scan Button onClick Handler

Location: templates/admin.templ line 54

Change from:

<button onclick="quickScan()" class="btn-primary p-4 rounded-lg text-left">

Change to:

<button onclick="scanAllLibraries()" class="btn-primary p-4 rounded-lg text-left">

Step 3.5: Remove Old Inline JavaScript Function

Location: templates/admin.templ lines 69-92

Action: DELETE the entire quickScan() function from the <script> section

What to remove:

<script>
    function quickScan() {
        fetch('/api/scanner/scan', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Authorization': 'Bearer ' + localStorage.getItem('token')
            },
            body: JSON.stringify({
                folder_paths: []
            })
        }).then(res => res.json()).then(data => {
            alert(data.message || 'Scan completed successfully!');
        }).catch(err => {
            console.error('Scan error:', err);
            alert('Scan failed. Please check your folder configuration.');
        });
    }

    function logout() {
        localStorage.removeItem('token');
        localStorage.removeItem('user');
        window.location.href = '/';
    }

    document.addEventListener('DOMContentLoaded', function() {
        loadTheme();
    });
</script>

Replace with:

<script>
    function logout() {
        localStorage.removeItem('token');
        localStorage.removeItem('user');
        window.location.href = '/';
    }

    document.addEventListener('DOMContentLoaded', function() {
        loadTheme();
    });
</script>

Note: Keep logout() and DOMContentLoaded handlers - only remove quickScan().

Step 4: Add Progress UI to admin.templ

Location: After the Quick Actions card (after line 64)

Add this HTML after line 64:

<!-- Scan Progress Section -->
<div id="scan-progress-container" class="hidden mt-6 p-6 rounded-lg border opacity-0 -translate-y-2.5 transition-all duration-300 ease-out"
     style="background-color: var(--bg-secondary); border-color: var(--border);">
    <div class="flex justify-between items-center mb-4">
        <h3 class="text-lg font-semibold" style="color: var(--text-primary)">
            📚 Scanning Libraries
        </h3>
        <button onclick="hideScanProgress()" class="p-2 hover:bg-gray-700 rounded">
            ✕
        </button>
    </div>

    <!-- Overall Progress -->
    <div class="mb-4">
        <div class="flex justify-between text-sm mb-2">
            <span style="color: var(--text-secondary)">Overall Progress</span>
            <span id="scan-progress-text" style="color: var(--text-primary)">0%</span>
        </div>
        <div class="w-full bg-gray-700 rounded-full h-3">
            <div id="scan-progress-bar"
                 class="h-3 rounded-full transition-all duration-500"
                 style="width: 0%; background-color: var(--accent);">
            </div>
        </div>
        <div id="scan-status" class="text-sm mt-2" style="color: var(--text-secondary)">
            Starting scan...
        </div>
    </div>
    
    <!-- Per-Library Progress -->
    <div id="library-progress-list" class="space-y-3">
        <!-- Dynamically populated -->
    </div>
    
    <!-- Results Summary -->
    <div id="scan-results" class="hidden mt-6 p-4 rounded-lg border"
         style="background-color: var(--bg-primary); border-color: var(--border);">
        <h4 class="font-semibold mb-2" style="color: var(--text-primary)">✅ Scan Complete!</h4>
        <div id="scan-results-content" style="color: var(--text-secondary)">
            <!-- Results populated by JS -->
        </div>
        <div class="mt-4 flex gap-2">
            <button onclick="window.location.reload()" 
                    class="btn-primary px-4 py-2 rounded-lg">
                Refresh to View Books
            </button>
            <button onclick="hideScanProgress()" 
                    class="btn-secondary px-4 py-2 rounded-lg">
                Dismiss
            </button>
        </div>
    </div>
</div>

Step 5: Build TypeScript to JavaScript

Build command:

npm run build:ts

What this does:

• Compiles web/src/admin.ts to web/static/admin.js
• TypeScript compiler (tsc) handles the conversion
• Output file admin.js will be loaded by the script tag added in Step 2

Note: This build step runs automatically in the Docker container during image build. For local development, run it manually after editing TypeScript files.

Step 6: Animation Implementation Note

How the slide-in animation works:

1. Initial state (Step 4 HTML): Container has classes hidden opacity-0 -translate-y-2.5 transition-all duration-300 ease-out
2. When scan starts (Step 1 TypeScript): showScanProgress() function:

   container.classList.remove('hidden');           // Makes element visible
   container.classList.remove('opacity-0', '-translate-y-2.5');  // Triggers animation
   

3. Result: Browser transitions from opacity-0 to opacity-1 and -translate-y-2.5 to translate-y-0 over 300ms

No custom CSS needed - follows PROJECT_GUIDELINES.md "TailwindCSS classes only" rule.

---

Part 2: Diagnose Dashboard Issue

Diagnostic Steps:

Step 1: Check System Collections Exist

Bruno Request:

GET /api/dashboard/sections?library_id=849151fb-564e-4b24-89e3-d11360789576

Expected Response:

{
  "data": [
    {
      "title": "continue-reading",
      "query_type": "continue-reading",
      "items": [...]
    },
    {
      "title": "recently-added",
      "query_type": "recently-added",
      "items": [
        {
          "title": "Leviticus on the Butcher's Block",
          "created_at": "2025-02-24...",
          ...
        }
      ]
    },
    {
      "title": "recently-read",
      "query_type": "recently-read",
      "items": [...]
    },
    {
      "title": "not-started",
      "query_type": "not-started",
      "items": [...]
    }
  ]
}

If sections array is empty or "recently-added" missing:

• System collections were not created for this user
• Need to manually call CreateDefaultCollectionsForUser

Step 2: Check Book's Library

Bruno Request:

GET /api/media-items?library_id=849151fb-564e-4b24-89e3-d11360789576&limit=5&sort=created_at+DESC

Expected:

• Scanned book should appear first (most recent created_at)
• If book appears here, it's in the correct library

If book appears:

• Book is in correct library
• Issue is with dashboard query or collection visibility

If book doesn't appear:

• Book was added to different library
• Check other libraries

Step 3: Check All Libraries

Bruno Request:

GET /api/libraries

Purpose:

• See all available libraries
• Check if book might be in a different library
• Confirm the library_id being used

Step 4: Verify URL Library Parameter

Check:

• Does /dashboard URL have ?library_id=xxx parameter?
• Which library is selected in dropdown?

If no library_id parameter:

• Dashboard auto-selects first visible library
• Book might be in a different library

Step 5: Direct Database Check (if needed)

Check system collections:

SELECT name, query_type, show_on_dashboard 
FROM collections 
WHERE user_id = 'your-user-id' 
  AND is_system_collection = true;

Check book's library:

SELECT id, title, library_id, created_at 
FROM media_items 
WHERE title LIKE '%Leviticus%'
ORDER BY created_at DESC 
LIMIT 1;

---

🔧 Potential Fixes for Dashboard Issue

Fix A: Recreate System Collections

If system collections don't exist:

Option 1: Manual API Call

POST /api/admin/recreate-system-collections

(Endpoint may need to be created)

Option 2: Direct Database

INSERT INTO collections (user_id, name, description, icon, color, show_on_dashboard, query_type, priority, is_system_collection)
VALUES 
  ('your-user-id', 'continue-reading', 'Books you''re currently reading (0 < progress < 1)', '📖', '#7aa2f7', true, 'continue-reading', 1, true),
  ('your-user-id', 'recently-added', 'Newly added items to this library', '🆕', '#9ece6a', true, 'recently-added', 2, true),
  ('your-user-id', 'recently-read', 'Books you''ve finished (progress >= 1)', '✅', '#e0af68', true, 'recently-read', 3, true),
  ('your-user-id', 'not-started', 'Books you haven''t read yet (progress = 0 or no record)', '📕', '#f7768e', true, 'not-started', 4, true);

Option 3: Backend Handler Create endpoint to recreate system collections for a user.

Fix B: Switch to Correct Library

If book is in different library:

• Select the correct library in dropdown
• Or create a combined view showing all libraries

Fix C: Update User Preferences

If collection is hidden:

GET /api/dashboard/preferences?library_id=xxx

Check if "recently-added" is in hidden_collections

Update:

PUT /api/dashboard/preferences
{
  "library_id": "xxx",
  "hidden_collections": [],  // Empty = show all
  "collection_order": [...],
  "items_per_section": 20
}

---

📊 Implementation Priority

1. HIGH PRIORITY: Fix Scan Library button

   • Impact: Users can't scan from UI at all
   • Effort: Medium (2-3 hours)
   • Files: 1 (admin.templ)

2. MEDIUM PRIORITY: Diagnose dashboard issue

   • Impact: Books exist but not visible
   • Effort: Low (30 min diagnosis)
   • Files: 0 (investigation only)

3. LOW PRIORITY: Fix dashboard issue

   • Impact: Depends on root cause
   • Effort: Unknown until diagnosis complete
   • Files: Unknown until diagnosis complete

---

🧪 Testing Checklist

After Fixing Scan Button:

• Scan button triggers without errors
• Progress UI appears
• Progress bar updates every 2 seconds
• Each library shows individual progress
• Scan completes and shows results
• "Refresh to View Books" works
• Books appear on dashboard after refresh

After Fixing Dashboard:

• Scanned book appears in "recently-added" section
• Book is at top of list (most recent)
• Book cover displays correctly
• Clicking book opens it
• All system collections show data
• Collections can be hidden/shown
• Dashboard works across page refreshes

---

📝 Notes

• Backend scanning is confirmed working (via Bruno)
• Job status polling endpoint works correctly
• Database contains the scanned book
• Issue is purely frontend/dashboard display logic
• System collections should be created during user registration
• Dashboard supports library switching via dropdown
• User can customize dashboard (hide collections, reorder, change items per section)

---

🔗 Related Files

• templates/admin.templ - Admin page with Scan button
• templates/dashboard.templ - Dashboard template
• internal/handlers/scanner.go - Scan endpoints
• internal/services/dashboard_service.go - Dashboard logic
• internal/services/worker.go - Background job processing
• internal/handlers/dashboard.go - Dashboard handlers
• internal/router/frontend.go - Dashboard route
• web/static/dashboard.js - Dashboard frontend logic
• bruno/scanner/Scan Media Items.yml - Bruno collection (FIXED)

---

Last Updated: 2025-02-24
Status: Ready to implement Scan button fix, Dashboard issue needs diagnosis