Remove obsolete task tracking and implementation plan files
These files were used during development tracking but are no longer needed: - IMPLEMENTATION_PLAN_COVER_PDF.md: PDF cover extraction plan (completed) - TASKS-backend-progress-tracking.md: Backend progress tracking tasks (completed) - TASKS-scanning-progress.md: Scanning progress tasks (completed) The functionality from these planning documents has been fully implemented: - EPUB and PDF cover extraction with metadata support - Backend progress tracking for scanning operations - Enhanced scanning progress UI with real-time updates Project housekeeping: remove temporary tracking files now that features are complete.
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -1,799 +0,0 @@
|
||||
# Backend Scan Progress Tracking
|
||||
|
||||
**Date Created:** 2025-02-25
|
||||
**Status:** Ready to Implement
|
||||
**Priority:** HIGH - Required for accurate progress UI
|
||||
|
||||
---
|
||||
|
||||
## 🚨 Problem
|
||||
|
||||
The scan job status endpoint returns minimal data:
|
||||
```json
|
||||
{
|
||||
"job_id": "...",
|
||||
"status": "completed",
|
||||
"progress": 1.0,
|
||||
"result": {"message": "scan completed", "library_id": "..."}
|
||||
}
|
||||
```
|
||||
|
||||
**Missing fields needed by frontend:**
|
||||
- `files_scanned` - total files processed
|
||||
- `new_items` - books added to database
|
||||
- `errors` - scan errors encountered
|
||||
- Real-time `progress` updates (0% → 100% during scan)
|
||||
|
||||
**Current behavior:**
|
||||
- Progress jumps from 0% to 100% when scan completes
|
||||
- No file counts during scanning
|
||||
- No error tracking
|
||||
|
||||
---
|
||||
|
||||
## 📋 Implementation Plan
|
||||
|
||||
### Overview
|
||||
|
||||
Add progress tracking to scan jobs by:
|
||||
1. Extending `JobResult` to include scan statistics
|
||||
2. Adding progress update mechanism to worker
|
||||
3. Tracking statistics during `ScanFolders()` (with batching every 10 files)
|
||||
4. Updating progress as files are processed
|
||||
5. Adding integration tests to verify behavior
|
||||
|
||||
---
|
||||
|
||||
### Step 1: Extend JobResult Structure
|
||||
|
||||
**File:** `internal/services/worker.go`
|
||||
|
||||
**Current JobResult:**
|
||||
```go
|
||||
type JobResult struct {
|
||||
JobID string
|
||||
Status JobStatus
|
||||
Error string
|
||||
Result interface{}
|
||||
Progress float64
|
||||
}
|
||||
```
|
||||
|
||||
**Add scan statistics:**
|
||||
```go
|
||||
type JobResult struct {
|
||||
JobID string
|
||||
Status JobStatus
|
||||
Error string
|
||||
Result interface{}
|
||||
Progress float64
|
||||
FilesScanned int // NEW
|
||||
NewItems int // NEW
|
||||
Errors int // NEW
|
||||
}
|
||||
```
|
||||
|
||||
**Update `processScanJob()` to return stats:**
|
||||
```go
|
||||
return map[string]interface{}{
|
||||
"message": "scan completed",
|
||||
"library_id": libraryID,
|
||||
"files_scanned": totalFiles,
|
||||
"new_items": newItems,
|
||||
"errors": errors,
|
||||
}, nil
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 2: Add Progress Update Callback to Job
|
||||
|
||||
**File:** `internal/services/worker.go`
|
||||
|
||||
**Add callback to Job struct:**
|
||||
```go
|
||||
type Job struct {
|
||||
ID string
|
||||
Type JobType
|
||||
Params map[string]interface{}
|
||||
Status JobStatus
|
||||
CreatedAt time.Time
|
||||
StartedAt *time.Time
|
||||
CompletedAt *time.Time
|
||||
Error error
|
||||
Result interface{}
|
||||
Context context.Context
|
||||
ProgressCallback func(progress float64, filesScanned, newItems, errors int) // NEW
|
||||
}
|
||||
```
|
||||
|
||||
**Add update method:**
|
||||
```go
|
||||
func (j *Job) UpdateProgress(progress float64, filesScanned, newItems, errors int) {
|
||||
if j.ProgressCallback != nil {
|
||||
j.ProgressCallback(progress, filesScanned, newItems, errors)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 3: Set Up Callback and Pass Job to Scanner
|
||||
|
||||
**File:** `internal/services/worker.go`
|
||||
|
||||
**Update processScanJob() to set up callback:**
|
||||
```go
|
||||
func (w *Worker) processScanJob(job *Job) (interface{}, error) {
|
||||
libraryID, ok := job.Params["library_id"].(string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("library_id required")
|
||||
}
|
||||
|
||||
folders, ok := job.Params["folders"].([]string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("folders required")
|
||||
}
|
||||
|
||||
adminID, ok := job.Params["admin_id"].(string)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("admin_id required")
|
||||
}
|
||||
|
||||
db, ok := job.Params["db"].(*database.Queries)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("database queries required")
|
||||
}
|
||||
|
||||
scanner := NewMediaScanner(db)
|
||||
scanner.job = job // NEW: Pass job reference for progress updates
|
||||
|
||||
// NEW: Set up progress callback to update JobResult in real-time
|
||||
job.ProgressCallback = func(progress float64, filesScanned, newItems, errors int) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
if result, exists := w.results[job.ID]; exists {
|
||||
result.Progress = progress
|
||||
result.FilesScanned = filesScanned
|
||||
result.NewItems = newItems
|
||||
result.Errors = errors
|
||||
}
|
||||
}
|
||||
|
||||
if err := scanner.SetFolders(folders); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var adminUUID pgtype.UUID
|
||||
if err := adminUUID.Scan(adminID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
scanner.SetAdminID(adminUUID)
|
||||
|
||||
if err := scanner.ScanFolders(job.Context); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
totalFiles, newItems, errors := scanner.GetStats()
|
||||
|
||||
return map[string]interface{}{
|
||||
"message": "scan completed",
|
||||
"library_id": libraryID,
|
||||
"files_scanned": totalFiles,
|
||||
"new_items": newItems,
|
||||
"errors": errors,
|
||||
}, nil
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4: Update Worker Job Completion Tracking
|
||||
|
||||
**File:** `internal/services/worker.go`
|
||||
|
||||
**Modify job processing loop:**
|
||||
```go
|
||||
case JobTypeScan:
|
||||
result, err = w.processScanJob(job)
|
||||
|
||||
// After job completes, update JobResult with stats
|
||||
w.mu.Lock()
|
||||
status := JobStatusCompleted
|
||||
if err != nil {
|
||||
status = JobStatusFailed
|
||||
}
|
||||
if job.Context != nil && job.Context.Err() != nil {
|
||||
status = JobStatusCancelled
|
||||
}
|
||||
|
||||
// Extract stats from result if available
|
||||
var filesScanned, newItems, errors int
|
||||
if result != nil {
|
||||
if stats, ok := result.(map[string]interface{}); ok {
|
||||
filesScanned = int(stats["files_scanned"].(float64))
|
||||
newItems = int(stats["new_items"].(float64))
|
||||
errors = int(stats["errors"].(float64))
|
||||
}
|
||||
}
|
||||
|
||||
w.results[job.ID] = &JobResult{
|
||||
JobID: job.ID,
|
||||
Status: status,
|
||||
Error: func() string { if err != nil { return err.Error() } else { return "" } }(),
|
||||
Result: result,
|
||||
Progress: 1.0,
|
||||
FilesScanned: filesScanned, // NEW
|
||||
NewItems: newItems, // NEW
|
||||
Errors: errors, // NEW
|
||||
}
|
||||
w.mu.Unlock()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 5: Track Statistics During Scan
|
||||
|
||||
**File:** `internal/services/media_scanner.go`
|
||||
|
||||
**Add counter fields to MediaScanner:**
|
||||
```go
|
||||
type MediaScanner struct {
|
||||
db *database.Queries
|
||||
watcher *fsnotify.Watcher
|
||||
folders []string
|
||||
adminID pgtype.UUID
|
||||
defaultLibraryID pgtype.UUID
|
||||
libraryTypes map[string][]string
|
||||
|
||||
// NEW: Scan statistics
|
||||
totalFiles int
|
||||
newItems int
|
||||
errors int
|
||||
job *Job // Reference to job for progress updates
|
||||
}
|
||||
```
|
||||
|
||||
**Add getter method:**
|
||||
```go
|
||||
func (s *MediaScanner) GetStats() (int, int, int) {
|
||||
return s.totalFiles, s.newItems, s.errors
|
||||
}
|
||||
```
|
||||
|
||||
**Update ScanFolders() to track stats:**
|
||||
```go
|
||||
func (s *MediaScanner) ScanFolders(ctx context.Context) error {
|
||||
if len(s.folders) == 0 {
|
||||
return fmt.Errorf("no folders set")
|
||||
}
|
||||
|
||||
// Reset counters
|
||||
s.totalFiles = 0
|
||||
s.newItems = 0
|
||||
s.errors = 0
|
||||
|
||||
// First pass: count total files
|
||||
for _, folder := range s.folders {
|
||||
filepath.WalkDir(folder, func(path string, d fs.DirEntry, err error) error {
|
||||
if !d.IsDir() && s.isScannableFile(path) {
|
||||
s.totalFiles++
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
fmt.Printf("Starting scan of %d folders: %v (%d files to scan)\n", len(s.folders), s.folders, s.totalFiles)
|
||||
|
||||
processedFiles := 0
|
||||
mediaFiles := 0
|
||||
|
||||
for _, folder := range s.folders {
|
||||
fmt.Printf("Scanning folder: %s\n", folder)
|
||||
|
||||
if _, err := os.Stat(folder); os.IsNotExist(err) {
|
||||
fmt.Printf("Folder does not exist: %s\n", folder)
|
||||
s.errors++
|
||||
continue
|
||||
}
|
||||
|
||||
err := filepath.WalkDir(folder, func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
fmt.Printf("Error accessing path %s: %v\n", path, err)
|
||||
s.errors++
|
||||
return err
|
||||
}
|
||||
|
||||
if d.IsDir() {
|
||||
if err := s.watcher.Add(path); err != nil {
|
||||
fmt.Printf("Warning: failed to watch subdirectory %s: %v\n", path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
if s.isScannableFile(path) {
|
||||
mediaFiles++
|
||||
processedFiles++
|
||||
|
||||
// Update progress (batch every 10 files to reduce mutex contention)
|
||||
if processedFiles%10 == 0 && s.totalFiles > 0 {
|
||||
progress := float64(processedFiles) / float64(s.totalFiles)
|
||||
if s.job != nil {
|
||||
s.job.UpdateProgress(progress, processedFiles, s.newItems, s.errors)
|
||||
}
|
||||
}
|
||||
|
||||
wasNew, err := s.processMediaFile(ctx, path)
|
||||
if err != nil {
|
||||
fmt.Printf("Error processing media file %s: %v\n", path, err)
|
||||
s.errors++
|
||||
} else {
|
||||
fmt.Printf("Successfully processed media file: %s\n", path)
|
||||
// newItems already incremented in processMediaFile if wasNew
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
s.errors++
|
||||
return fmt.Errorf("failed to scan folder %s: %v", folder, err)
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf("Scan completed: %d total files scanned, %d media files found, %d new items, %d errors\n",
|
||||
processedFiles, mediaFiles, s.newItems, s.errors)
|
||||
|
||||
// Final progress update to ensure we report 100%
|
||||
if s.job != nil && s.totalFiles > 0 {
|
||||
s.job.UpdateProgress(1.0, processedFiles, s.newItems, s.errors)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 6: Update processMediaFile to Track New Items
|
||||
|
||||
**File:** `internal/services/media_scanner.go`
|
||||
|
||||
**Modify return value:**
|
||||
```go
|
||||
// CURRENT: func (s *MediaScanner) processMediaFile(ctx context.Context, path string) error
|
||||
// NEW: Returns (bool, error) where bool indicates if item was newly created
|
||||
|
||||
func (s *MediaScanner) processMediaFile(ctx context.Context, path string) (bool, error) {
|
||||
// ... existing file processing code ...
|
||||
|
||||
// Check if media item already exists
|
||||
existingItem, err := s.getMediaItemByFilePath(ctx, path)
|
||||
if err == nil && existingItem.FileSize.Int64 == info.Size() {
|
||||
fmt.Printf("Media item already exists with same size, skipping: %s\n", path)
|
||||
return false, nil // FALSE = not a new item (already exists)
|
||||
}
|
||||
|
||||
// If item exists but different size, it's an update - still not "new"
|
||||
if err == nil {
|
||||
fmt.Printf("Updating existing media item: %s\n", path)
|
||||
// ... update logic ...
|
||||
return false, nil // FALSE = not a new item (was an update)
|
||||
}
|
||||
|
||||
// ... rest of processing for new item ...
|
||||
|
||||
// Create media item in database
|
||||
createdItem, err := s.db.CreateMediaItem(ctx, database.CreateMediaItemParams{...})
|
||||
if err != nil {
|
||||
return false, err // FALSE = error, false means not created
|
||||
}
|
||||
|
||||
s.newItems++ // NEW: Track new items
|
||||
return true, nil // TRUE = new item created
|
||||
}
|
||||
```
|
||||
|
||||
**Update ScanFolders() to use return value:**
|
||||
```go
|
||||
wasNew, err := s.processMediaFile(ctx, path)
|
||||
if err != nil {
|
||||
s.errors++
|
||||
} else if wasNew {
|
||||
// newItems already incremented in processMediaFile
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 7: Update GetScanStatus Handler
|
||||
|
||||
**File:** `internal/handlers/scanner.go`
|
||||
|
||||
**Update response to include new fields:**
|
||||
```go
|
||||
func (h *Handler) GetScanStatus(c echo.Context) error {
|
||||
jobID := c.Param("jobId")
|
||||
|
||||
result, exists := h.worker.GetJobStatus(jobID)
|
||||
if !exists {
|
||||
return c.JSON(http.StatusNotFound, map[string]string{"error": "job not found"})
|
||||
}
|
||||
|
||||
return c.JSON(http.StatusOK, map[string]interface{}{
|
||||
"job_id": result.JobID,
|
||||
"status": result.Status,
|
||||
"error": result.Error,
|
||||
"result": result.Result,
|
||||
"progress": result.Progress,
|
||||
"files_scanned": result.FilesScanned, // NEW
|
||||
"new_items": result.NewItems, // NEW
|
||||
"errors": result.Errors, // NEW
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 8: Add Integration Tests
|
||||
|
||||
**File:** `cmd/server/tests/scanner_integration_test.go` (new file)
|
||||
|
||||
**Create new integration test file:**
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/stretchr/testify/suite"
|
||||
)
|
||||
|
||||
type ScannerIntegrationTestSuite struct {
|
||||
suite.Suite
|
||||
setup *TestServerSetup
|
||||
}
|
||||
|
||||
func (s *ScannerIntegrationTestSuite) SetupSuite() {
|
||||
s.setup = setupTestServer(s.T())
|
||||
}
|
||||
|
||||
func (s *ScannerIntegrationTestSuite) TearDownSuite() {
|
||||
s.setup.Close()
|
||||
}
|
||||
|
||||
func (s *ScannerIntegrationTestSuite) TestScanProgress_TracksStatistics() {
|
||||
token := s.setup.Token
|
||||
|
||||
// Create test library
|
||||
libraryID := s.setup.CreateLibrary(s.T(), "Scan Test Library", "ebooks")
|
||||
|
||||
// Add folder to library
|
||||
folderURL := fmt.Sprintf("%s/api/libraries/%s/folders", s.setup.Server.URL, libraryID)
|
||||
folderReq := map[string]interface{}{
|
||||
"folder_path": "/app/uploads",
|
||||
}
|
||||
folderBody, _ := json.Marshal(folderReq)
|
||||
req, _ := http.NewRequest("POST", folderURL, bytes.NewBuffer(folderBody))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
client := &http.Client{}
|
||||
resp, err := client.Do(req)
|
||||
require.NoError(s.T(), err)
|
||||
resp.Body.Close()
|
||||
require.Equal(s.T(), http.StatusCreated, resp.StatusCode, "Folder creation should succeed")
|
||||
|
||||
// Start scan
|
||||
scanURL := fmt.Sprintf("%s/api/libraries/%s/scan", s.setup.Server.URL, libraryID)
|
||||
scanReq, _ := http.NewRequest("POST", scanURL, nil)
|
||||
scanReq.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
scanResp, err := client.Do(scanReq)
|
||||
require.NoError(s.T(), err)
|
||||
require.Equal(s.T(), http.StatusAccepted, scanResp.StatusCode)
|
||||
|
||||
var scanResponse map[string]interface{}
|
||||
err = json.NewDecoder(scanResp.Body).Decode(&scanResponse)
|
||||
require.NoError(s.T(), err)
|
||||
scanResp.Body.Close()
|
||||
|
||||
jobID, ok := scanResponse["job_id"].(string)
|
||||
require.True(s.T(), ok, "job_id should be string")
|
||||
require.NotEmpty(s.T(), jobID, "job_id should not be empty")
|
||||
|
||||
// Poll for progress updates
|
||||
var lastProgress float64
|
||||
var lastFilesScanned, lastNewItems, lastErrors int
|
||||
|
||||
for i := 0; i < 30; i++ { // Poll for up to 30 seconds
|
||||
time.Sleep(1 * time.Second)
|
||||
|
||||
statusURL := fmt.Sprintf("%s/api/scanner/status/%s", s.setup.Server.URL, jobID)
|
||||
statusReq, _ := http.NewRequest("GET", statusURL, nil)
|
||||
statusReq.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
statusResp, err := client.Do(statusReq)
|
||||
require.NoError(s.T(), err)
|
||||
|
||||
var status map[string]interface{}
|
||||
err = json.NewDecoder(statusResp.Body).Decode(&status)
|
||||
statusResp.Body.Close()
|
||||
require.NoError(s.T(), err)
|
||||
|
||||
// Verify new fields exist
|
||||
assert.Contains(s.T(), status, "files_scanned")
|
||||
assert.Contains(s.T(), status, "new_items")
|
||||
assert.Contains(s.T(), status, "errors")
|
||||
|
||||
// Track progress with safe type assertions
|
||||
progressFloat, ok := status["progress"].(float64)
|
||||
require.True(s.T(), ok, "progress should be float64")
|
||||
progress := progressFloat
|
||||
|
||||
filesScannedFloat, ok := status["files_scanned"].(float64)
|
||||
require.True(s.T(), ok, "files_scanned should be float64")
|
||||
filesScanned := int(filesScannedFloat)
|
||||
|
||||
newItemsFloat, ok := status["new_items"].(float64)
|
||||
require.True(s.T(), ok, "new_items should be float64")
|
||||
newItems := int(newItemsFloat)
|
||||
|
||||
errorsFloat, ok := status["errors"].(float64)
|
||||
require.True(s.T(), ok, "errors should be float64")
|
||||
errors := int(errorsFloat)
|
||||
|
||||
// Progress should be non-decreasing
|
||||
assert.GreaterOrEqual(s.T(), progress, lastProgress)
|
||||
lastProgress = progress
|
||||
|
||||
// Files scanned should be non-decreasing
|
||||
assert.GreaterOrEqual(s.T(), filesScanned, lastFilesScanned)
|
||||
lastFilesScanned = filesScanned
|
||||
|
||||
// Items/errors should be non-decreasing
|
||||
assert.GreaterOrEqual(s.T(), newItems, lastNewItems)
|
||||
assert.GreaterOrEqual(s.T(), errors, lastErrors)
|
||||
|
||||
// Break if scan complete
|
||||
if status["status"] == "completed" || status["status"] == "failed" {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Verify final state
|
||||
assert.Equal(s.T(), 1.0, lastProgress)
|
||||
assert.GreaterOrEqual(s.T(), lastFilesScanned, 0)
|
||||
}
|
||||
|
||||
func (s *ScannerIntegrationTestSuite) TestScanProgress_BatchingWorks() {
|
||||
token := s.setup.Token
|
||||
|
||||
// Create library with folder
|
||||
libraryID := s.setup.CreateLibrary(s.T(), "Batch Test Library", "ebooks")
|
||||
|
||||
folderURL := fmt.Sprintf("%s/api/libraries/%s/folders", s.setup.Server.URL, libraryID)
|
||||
folderReq := map[string]interface{}{
|
||||
"folder_path": "/app/uploads",
|
||||
}
|
||||
folderBody, _ := json.Marshal(folderReq)
|
||||
req, _ := http.NewRequest("POST", folderURL, bytes.NewBuffer(folderBody))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
client := &http.Client{}
|
||||
resp, err := client.Do(req)
|
||||
require.NoError(s.T(), err)
|
||||
resp.Body.Close()
|
||||
|
||||
// Start scan
|
||||
scanURL := fmt.Sprintf("%s/api/libraries/%s/scan", s.setup.Server.URL, libraryID)
|
||||
scanReq, _ := http.NewRequest("POST", scanURL, nil)
|
||||
scanReq.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
scanResp, err := client.Do(scanReq)
|
||||
require.NoError(s.T(), err)
|
||||
|
||||
var scanResponse map[string]interface{}
|
||||
json.NewDecoder(scanResp.Body).Decode(&scanResponse)
|
||||
scanResp.Body.Close()
|
||||
|
||||
jobID, ok := scanResponse["job_id"].(string)
|
||||
require.True(s.T(), ok, "job_id should be string")
|
||||
require.NotEmpty(s.T(), jobID, "job_id should not be empty")
|
||||
|
||||
// Poll and verify we don't get updates on EVERY file
|
||||
updateCount := 0
|
||||
previousFilesScanned := -1
|
||||
|
||||
for i := 0; i < 20; i++ {
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
|
||||
statusURL := fmt.Sprintf("%s/api/scanner/status/%s", s.setup.Server.URL, jobID)
|
||||
statusReq, _ := http.NewRequest("GET", statusURL, nil)
|
||||
statusReq.Header.Set("Authorization", "Bearer "+token)
|
||||
|
||||
statusResp, _ := client.Do(statusReq)
|
||||
|
||||
var status map[string]interface{}
|
||||
err = json.NewDecoder(statusResp.Body).Decode(&status)
|
||||
require.NoError(s.T(), err)
|
||||
statusResp.Body.Close()
|
||||
|
||||
filesScannedFloat, ok := status["files_scanned"].(float64)
|
||||
require.True(s.T(), ok, "files_scanned should be float64")
|
||||
filesScanned := int(filesScannedFloat)
|
||||
|
||||
// Only count as update if files_scanned changed
|
||||
if filesScanned != previousFilesScanned {
|
||||
updateCount++
|
||||
previousFilesScanned = filesScanned
|
||||
}
|
||||
|
||||
if status["status"] == "completed" || status["status"] == "failed" {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// With batching every 10 files, we should have FEWER updates than files
|
||||
// This is a weak assertion, but verifies batching is working
|
||||
// Threshold of 50 assumes test library has < 500 files - adjust based on actual test data
|
||||
assert.Less(s.T(), updateCount, 50)
|
||||
}
|
||||
|
||||
func TestScannerIntegrationTestSuite(t *testing.T) {
|
||||
suite.Run(t, new(ScannerIntegrationTestSuite))
|
||||
}
|
||||
```
|
||||
|
||||
**Note:** The integration test requires `encoding/json` import (already included in the import list above).
|
||||
|
||||
**Update existing unit tests:** `internal/services/worker_test.go`
|
||||
|
||||
**Add test for new JobResult fields:**
|
||||
```go
|
||||
func TestWorker_JobResult_HasStatsFields(t *testing.T) {
|
||||
// Test that JobResult properly stores scan statistics
|
||||
worker := NewWorker(1)
|
||||
defer worker.Shutdown()
|
||||
|
||||
jobID := "test-job-stats"
|
||||
|
||||
// Simulate job completion with stats
|
||||
worker.mu.Lock()
|
||||
worker.results[jobID] = &JobResult{
|
||||
JobID: jobID,
|
||||
Status: JobStatusCompleted,
|
||||
Progress: 1.0,
|
||||
FilesScanned: 42,
|
||||
NewItems: 5,
|
||||
Errors: 1,
|
||||
}
|
||||
worker.mu.Unlock()
|
||||
|
||||
// Verify stats are retrievable
|
||||
result, exists := worker.GetJobStatus(jobID)
|
||||
require.True(t, exists, "Job result should exist")
|
||||
require.NotNil(t, result, "Result should not be nil")
|
||||
|
||||
assert.Equal(t, jobID, result.JobID)
|
||||
assert.Equal(t, JobStatusCompleted, result.Status)
|
||||
assert.Equal(t, 1.0, result.Progress)
|
||||
assert.Equal(t, 42, result.FilesScanned, "FilesScanned should be 42")
|
||||
assert.Equal(t, 5, result.NewItems, "NewItems should be 5")
|
||||
assert.Equal(t, 1, result.Errors, "Errors should be 1")
|
||||
}
|
||||
|
||||
func TestWorker_ProgressCallback_UpdatesJobResult(t *testing.T) {
|
||||
// Test that progress callback updates JobResult in real-time
|
||||
worker := NewWorker(1)
|
||||
defer worker.Shutdown()
|
||||
|
||||
job := &Job{
|
||||
ID: "test-progress",
|
||||
Type: JobTypeScan,
|
||||
Status: JobStatusInProgress,
|
||||
Context: context.Background(),
|
||||
}
|
||||
|
||||
// Set up progress callback
|
||||
job.ProgressCallback = func(progress float64, filesScanned, newItems, errors int) {
|
||||
worker.mu.Lock()
|
||||
defer worker.mu.Unlock()
|
||||
|
||||
if result, exists := worker.results[job.ID]; exists {
|
||||
result.Progress = progress
|
||||
result.FilesScanned = filesScanned
|
||||
result.NewItems = newItems
|
||||
result.Errors = errors
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize result
|
||||
worker.mu.Lock()
|
||||
worker.results[job.ID] = &JobResult{
|
||||
JobID: job.ID,
|
||||
Status: JobStatusInProgress,
|
||||
}
|
||||
worker.mu.Unlock()
|
||||
|
||||
// Simulate progress updates
|
||||
job.UpdateProgress(0.5, 10, 2, 0)
|
||||
|
||||
result, exists := worker.GetJobStatus(job.ID)
|
||||
require.True(t, exists)
|
||||
assert.Equal(t, 0.5, result.Progress)
|
||||
assert.Equal(t, 10, result.FilesScanned)
|
||||
assert.Equal(t, 2, result.NewItems)
|
||||
assert.Equal(t, 0, result.Errors)
|
||||
|
||||
// Simulate completion
|
||||
job.UpdateProgress(1.0, 20, 5, 1)
|
||||
|
||||
result, exists = worker.GetJobStatus(job.ID)
|
||||
require.True(t, exists)
|
||||
assert.Equal(t, 1.0, result.Progress)
|
||||
assert.Equal(t, 20, result.FilesScanned)
|
||||
assert.Equal(t, 5, result.NewItems)
|
||||
assert.Equal(t, 1, result.Errors)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Checklist
|
||||
|
||||
After implementation:
|
||||
|
||||
- [ ] Unit tests pass: `go test ./internal/services/...`
|
||||
- [ ] TestWorker_JobResult_HasStatsFields verifies stats are stored
|
||||
- [ ] TestWorker_ProgressCallback_UpdatesJobResult verifies real-time updates
|
||||
- [ ] Integration tests pass: `go test ./cmd/server/tests/...`
|
||||
- [ ] TestScanProgress_TracksStatistics verifies all new fields exist and increment
|
||||
- [ ] TestScanProgress_BatchingWorks verifies batching reduces update frequency
|
||||
- [ ] Scan a library with multiple files
|
||||
- [ ] Poll `/api/scanner/status/{jobId}` during scan
|
||||
- [ ] Verify `progress` increases from 0% to 100% gradually
|
||||
- [ ] Verify `files_scanned` count increases during scan
|
||||
- [ ] Verify `new_items` count shows books added
|
||||
- [ ] Verify `errors` count shows scan errors
|
||||
- [ ] Check final status has accurate totals
|
||||
- [ ] Test with empty library (no files)
|
||||
- [ ] Test with library containing only non-media files
|
||||
- [ ] Test with library causing scan errors
|
||||
- [ ] Verify batching reduces update frequency (integration test)
|
||||
|
||||
---
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- **Thread Safety:** JobResult updates are thread-safe via worker's mutex (w.mu.Lock())
|
||||
- **Performance:** Progress updates are batched every 10 files to reduce mutex contention
|
||||
- **Memory:** Stats tracking uses 3 int fields (24 bytes) per MediaScanner instance
|
||||
- **Backwards Compatibility:** Frontend already uses `|| 0` fallbacks, so safe to deploy
|
||||
- **Testing:** Integration tests use `setupTestServer()` from `test_helpers.go`
|
||||
- **Database Pool Configuration:** `setupTestServer()` already sets `max_conns=1` (test_helpers.go:428), preventing connection pool exhaustion during test runs
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Related Files
|
||||
|
||||
- `internal/services/worker.go` - Job processing and result tracking
|
||||
- `internal/services/media_scanner.go` - Scan logic and statistics
|
||||
- `internal/handlers/scanner.go` - Status API endpoint
|
||||
- `cmd/server/tests/scanner_integration_test.go` - Integration tests (NEW)
|
||||
- `internal/services/worker_test.go` - Unit tests to update
|
||||
- `TASKS-scanning-progress.md` - Frontend implementation that depends on this
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2025-02-25
|
||||
**Status:** Ready for review and implementation
|
||||
@@ -1,741 +0,0 @@
|
||||
# Scanning & Dashboard Issues - Implementation Plan
|
||||
|
||||
**Date Created:** 2025-02-24
|
||||
**Status:** Documented - Ready to Implement
|
||||
|
||||
---
|
||||
|
||||
## ✅ Fixed Issues
|
||||
|
||||
### Bruno Collection File
|
||||
**File:** `/home/nymusicman/Code/bookhoard/bruno/scanner/Scan Media Items.yml`
|
||||
|
||||
**Problem:** Invalid JSON syntax - library_id variable was not quoted
|
||||
|
||||
**Original (Line 20):**
|
||||
```yaml
|
||||
"library_id": {{library_id}} # ❌ WRONG - UUID not quoted
|
||||
```
|
||||
|
||||
**Fixed:**
|
||||
```yaml
|
||||
"library_id": "{{library_id}}" # ✅ CORRECT - quoted string
|
||||
```
|
||||
|
||||
**Impact:** Bruno requests now work correctly. API scanning confirmed functional.
|
||||
|
||||
---
|
||||
|
||||
## 🚧 Remaining Issues
|
||||
|
||||
### Issue 1: Scan Library Button (Frontend)
|
||||
**Severity:** HIGH - Button completely non-functional
|
||||
**File:** `templates/admin.templ` (lines 69-85)
|
||||
|
||||
**Current Broken Code:**
|
||||
```javascript
|
||||
function quickScan() {
|
||||
fetch('/api/scanner/scan', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer ' + localStorage.getItem('token')
|
||||
},
|
||||
body: JSON.stringify({
|
||||
folder_paths: [] // ← EMPTY ARRAY! Causes 400 error
|
||||
})
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
**Why It Fails:**
|
||||
- Sends `folder_paths: []` (empty array)
|
||||
- Handler checks `if len(req.FolderPaths) > 0` → FALSE for empty array
|
||||
- Returns 400: "either library_id or folder_paths required for scanning"
|
||||
- No scan job is created
|
||||
- No progress feedback
|
||||
- Users can't scan from UI
|
||||
|
||||
**What Should Happen:**
|
||||
1. Fetch all libraries from `/api/libraries`
|
||||
2. Trigger scan for each library via `/api/libraries/{id}/scan`
|
||||
3. Collect all job IDs
|
||||
4. Poll `/api/scanner/status/{jobId}` for progress
|
||||
5. Display progress UI
|
||||
6. Show results when complete
|
||||
|
||||
---
|
||||
|
||||
### Issue 2: Scanned Books Not Showing on Dashboard
|
||||
**Severity:** MEDIUM - Data exists but not visible
|
||||
**Status:** Requires diagnosis
|
||||
|
||||
**Symptoms:**
|
||||
- Book successfully scanned via API
|
||||
- Book exists in database
|
||||
- Book not visible on `/dashboard` page
|
||||
- Book appears in `/api/media-items` endpoint
|
||||
|
||||
**Expected Behavior:**
|
||||
- Book should appear in "recently-added" section
|
||||
- Should be at top of list (most recent `created_at`)
|
||||
- Should be visible immediately after scan
|
||||
|
||||
**Possible Root Causes:**
|
||||
|
||||
#### Hypothesis 1: System Collections Missing
|
||||
- System collections (including "recently-added") created during user registration
|
||||
- Possible creation failure or user created before feature existed
|
||||
- Check: Query database for user's system collections
|
||||
|
||||
#### Hypothesis 2: Wrong Library Selected
|
||||
- Dashboard shows books for selected library only
|
||||
- Book might be in different library than displayed
|
||||
- Check: Compare book's library_id with dashboard's selected library
|
||||
|
||||
#### Hypothesis 3: Collection Hidden
|
||||
- User preferences might hide "recently-added" collection
|
||||
- `show_on_dashboard = false` in database
|
||||
- Check: User's dashboard preferences
|
||||
|
||||
#### Hypothesis 4: Empty Result Set
|
||||
- Query limit too low
|
||||
- Ordering incorrect
|
||||
- Check: API responses directly
|
||||
|
||||
---
|
||||
|
||||
## 📋 Implementation Plan
|
||||
|
||||
### Part 1: Fix Scan Library Button
|
||||
|
||||
**NOTE - Major Changes to Original Plan:**
|
||||
- **Switched from inline JavaScript to TypeScript** (follows PROJECT_GUIDELINES.md: "convert all JavaScript to TypeScript")
|
||||
- **Uses existing `web/src/admin.ts` infrastructure** instead of adding new inline code
|
||||
- **No custom CSS** - uses TailwindCSS transition classes for animation (follows "TailwindCSS classes only" rule)
|
||||
- **Inline CSS with variables retained** - follows existing pattern in admin.templ for theme support
|
||||
|
||||
**Rationale:**
|
||||
- Project already has `web/src/admin.ts` with TypeScript scanning functions
|
||||
- Inline JavaScript in templates makes code harder to maintain
|
||||
- TypeScript provides better type safety and code organization
|
||||
- TailwindCSS transitions are sufficient for UI animation
|
||||
|
||||
**Files to Modify:**
|
||||
- `web/src/admin.ts` (extend TypeScript scanning functions)
|
||||
- `templates/admin.templ` (add progress UI, include admin.js)
|
||||
|
||||
**Implementation Steps:**
|
||||
|
||||
#### Step 1: Extend TypeScript in web/src/admin.ts
|
||||
|
||||
**Location:** Add new functions after `loadSystemStats()` (around line 103)
|
||||
|
||||
**Add these functions:**
|
||||
```typescript
|
||||
async function scanAllLibraries(): Promise<void> {
|
||||
const token = localStorage.getItem('token');
|
||||
if (!token) return;
|
||||
|
||||
try {
|
||||
// Step 1: Get all libraries
|
||||
const libsResp = await fetch('/api/libraries', {
|
||||
headers: { 'Authorization': `Bearer ${token}` }
|
||||
});
|
||||
|
||||
if (!libsResp.ok) {
|
||||
throw new Error('Failed to get libraries');
|
||||
}
|
||||
|
||||
const libsData = await libsResp.json();
|
||||
|
||||
if (!libsData.data || libsData.data.length === 0) {
|
||||
if ((window as any).showToast?.error) {
|
||||
(window as any).showToast.error('No libraries found. Please create a library first.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const libraries = libsData.data;
|
||||
|
||||
// Step 2: Scan each library
|
||||
const jobs: string[] = [];
|
||||
const libraryNames: Record<string, string> = {};
|
||||
|
||||
for (const lib of libraries) {
|
||||
const scanResp = await fetch(`/api/libraries/${lib.id}/scan`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${token}` }
|
||||
});
|
||||
|
||||
if (scanResp.ok) {
|
||||
const result = await scanResp.json();
|
||||
jobs.push(result.job_id);
|
||||
libraryNames[result.job_id] = lib.name;
|
||||
} else {
|
||||
console.error(`Failed to scan library: ${lib.name}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (jobs.length === 0) {
|
||||
if ((window as any).showToast?.error) {
|
||||
(window as any).showToast.error('Failed to start scan for any library');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Step 3: Show progress UI
|
||||
showScanProgress(jobs, libraryNames);
|
||||
|
||||
} catch (error) {
|
||||
console.error('Scan error:', error);
|
||||
if ((window as any).showToast?.error) {
|
||||
(window as any).showToast.error('Failed to start scan: ' + (error as Error).message);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function showScanProgress(jobIds: string[], libraryNames: Record<string, string>): void {
|
||||
const container = document.getElementById('scan-progress-container') as HTMLElement;
|
||||
const list = document.getElementById('library-progress-list') as HTMLElement;
|
||||
|
||||
if (!container || !list) return;
|
||||
|
||||
container.classList.remove('hidden');
|
||||
// Trigger slide-in animation by removing opacity and transform classes
|
||||
container.classList.remove('opacity-0', '-translate-y-2.5');
|
||||
|
||||
// Create progress items for each library
|
||||
list.innerHTML = jobIds.map(jobId => `
|
||||
<div id="progress-${jobId}" class="p-3 rounded border"
|
||||
style="background-color: var(--bg-primary); border-color: var(--border);">
|
||||
<div class="flex justify-between items-center mb-2">
|
||||
<span class="font-medium" style="color: var(--text-primary)">
|
||||
${libraryNames[jobId]}
|
||||
</span>
|
||||
<span id="status-${jobId}" class="text-sm" style="color: var(--text-secondary)">
|
||||
Pending...
|
||||
</span>
|
||||
</div>
|
||||
<div class="w-full bg-gray-700 rounded-full h-2">
|
||||
<div id="bar-${jobId}"
|
||||
class="h-2 rounded-full transition-all duration-500"
|
||||
style="width: 0%; background-color: var(--accent);">
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
`).join('');
|
||||
|
||||
// Start polling
|
||||
pollScanProgress(jobIds, libraryNames);
|
||||
}
|
||||
|
||||
function pollScanProgress(jobIds: string[], libraryNames: Record<string, string>): void {
|
||||
const token = localStorage.getItem('token');
|
||||
const startTime = Date.now();
|
||||
|
||||
const interval = setInterval(async () => {
|
||||
let allComplete = true;
|
||||
let totalProgress = 0;
|
||||
let totalFiles = 0;
|
||||
let totalNewItems = 0;
|
||||
let totalErrors = 0;
|
||||
|
||||
for (const jobId of jobIds) {
|
||||
try {
|
||||
const resp = await fetch(`/api/scanner/status/${jobId}`, {
|
||||
headers: { 'Authorization': `Bearer ${token}` }
|
||||
});
|
||||
|
||||
if (resp.ok) {
|
||||
const status = await resp.json();
|
||||
|
||||
// Update individual library progress
|
||||
updateLibraryProgress(jobId, status);
|
||||
|
||||
totalProgress += status.progress || 0;
|
||||
totalFiles += status.files_scanned || 0;
|
||||
totalNewItems += status.new_items || 0;
|
||||
totalErrors += status.errors || 0;
|
||||
|
||||
if (status.status !== 'completed' && status.status !== 'failed') {
|
||||
allComplete = false;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`Failed to poll job ${jobId}:`, error);
|
||||
}
|
||||
}
|
||||
|
||||
// Update overall progress
|
||||
const overallProgress = Math.round(totalProgress / jobIds.length);
|
||||
const progressBar = document.getElementById('scan-progress-bar') as HTMLElement;
|
||||
const progressText = document.getElementById('scan-progress-text') as HTMLElement;
|
||||
const statusText = document.getElementById('scan-status') as HTMLElement;
|
||||
|
||||
if (progressBar) progressBar.style.width = overallProgress + '%';
|
||||
if (progressText) progressText.textContent = overallProgress + '%';
|
||||
|
||||
// Update status text
|
||||
const elapsed = Math.round((Date.now() - startTime) / 1000);
|
||||
if (!allComplete && statusText) {
|
||||
statusText.textContent = `Scanning... ${elapsed}s elapsed • ${totalFiles} files processed`;
|
||||
}
|
||||
|
||||
// Check if all complete
|
||||
if (allComplete) {
|
||||
clearInterval(interval);
|
||||
showScanResults(jobIds.length, totalFiles, totalNewItems, totalErrors, elapsed);
|
||||
}
|
||||
}, 2000);
|
||||
}
|
||||
|
||||
function updateLibraryProgress(jobId: string, status: any): void {
|
||||
const bar = document.getElementById(`bar-${jobId}`) as HTMLElement;
|
||||
const statusText = document.getElementById(`status-${jobId}`) as HTMLElement;
|
||||
|
||||
if (bar) {
|
||||
bar.style.width = (status.progress || 0) + '%';
|
||||
}
|
||||
|
||||
if (statusText) {
|
||||
const statusMessages: Record<string, string> = {
|
||||
'pending': 'Pending...',
|
||||
'running': `Scanning... ${status.progress || 0}%`,
|
||||
'completed': `✓ Complete (${status.new_items || 0} items)`,
|
||||
'failed': `✗ Failed`
|
||||
};
|
||||
statusText.textContent = statusMessages[status.status] || status.status;
|
||||
}
|
||||
}
|
||||
|
||||
function showScanResults(libCount: number, files: number, items: number, errors: number, elapsed: number): void {
|
||||
const resultsDiv = document.getElementById('scan-results') as HTMLElement;
|
||||
const contentDiv = document.getElementById('scan-results-content') as HTMLElement;
|
||||
|
||||
if (!resultsDiv || !contentDiv) return;
|
||||
|
||||
contentDiv.innerHTML = `
|
||||
<p>• ${libCount} librar${libCount === 1 ? 'y' : 'ies'} scanned</p>
|
||||
<p>• ${files} files processed</p>
|
||||
<p>• ${items} new items added</p>
|
||||
${errors > 0 ? `<p style="color: var(--accent);">• ${errors} errors</p>` : ''}
|
||||
<p style="color: var(--text-secondary)">Completed in ${elapsed} seconds</p>
|
||||
`;
|
||||
|
||||
resultsDiv.classList.remove('hidden');
|
||||
|
||||
const statusText = document.getElementById('scan-status') as HTMLElement;
|
||||
if (statusText) statusText.textContent = 'Scan complete!';
|
||||
}
|
||||
|
||||
function hideScanProgress(): void {
|
||||
const container = document.getElementById('scan-progress-container') as HTMLElement;
|
||||
if (container) container.classList.add('hidden');
|
||||
}
|
||||
|
||||
// Export to window
|
||||
(window as any).scanAllLibraries = scanAllLibraries;
|
||||
(window as any).hideScanProgress = hideScanProgress;
|
||||
```
|
||||
|
||||
#### Step 2: Update admin.templ to Include admin.js
|
||||
|
||||
**Location:** `templates/admin.templ` lines 7-12 (head section)
|
||||
|
||||
**Add admin.js script tag:**
|
||||
```html
|
||||
<script src="/static/htmx.min.js"></script>
|
||||
<script src="/static/toast.js"></script>
|
||||
<script src="/static/admin.js"></script> <!-- ADD THIS LINE -->
|
||||
<link href="/static/style.css" rel="stylesheet">
|
||||
```
|
||||
|
||||
#### Step 3: Update Scan Button onClick Handler
|
||||
|
||||
**Location:** `templates/admin.templ` line 54
|
||||
|
||||
**Change from:**
|
||||
```html
|
||||
<button onclick="quickScan()" class="btn-primary p-4 rounded-lg text-left">
|
||||
```
|
||||
|
||||
**Change to:**
|
||||
```html
|
||||
<button onclick="scanAllLibraries()" class="btn-primary p-4 rounded-lg text-left">
|
||||
```
|
||||
|
||||
#### Step 3.5: Remove Old Inline JavaScript Function
|
||||
|
||||
**Location:** `templates/admin.templ` lines 69-92
|
||||
|
||||
**Action:** DELETE the entire `quickScan()` function from the `<script>` section
|
||||
|
||||
**What to remove:**
|
||||
```html
|
||||
<script>
|
||||
function quickScan() {
|
||||
fetch('/api/scanner/scan', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer ' + localStorage.getItem('token')
|
||||
},
|
||||
body: JSON.stringify({
|
||||
folder_paths: []
|
||||
})
|
||||
}).then(res => res.json()).then(data => {
|
||||
alert(data.message || 'Scan completed successfully!');
|
||||
}).catch(err => {
|
||||
console.error('Scan error:', err);
|
||||
alert('Scan failed. Please check your folder configuration.');
|
||||
});
|
||||
}
|
||||
|
||||
function logout() {
|
||||
localStorage.removeItem('token');
|
||||
localStorage.removeItem('user');
|
||||
window.location.href = '/';
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded', function() {
|
||||
loadTheme();
|
||||
});
|
||||
</script>
|
||||
```
|
||||
|
||||
**Replace with:**
|
||||
```html
|
||||
<script>
|
||||
function logout() {
|
||||
localStorage.removeItem('token');
|
||||
localStorage.removeItem('user');
|
||||
window.location.href = '/';
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded', function() {
|
||||
loadTheme();
|
||||
});
|
||||
</script>
|
||||
```
|
||||
|
||||
**Note:** Keep `logout()` and `DOMContentLoaded` handlers - only remove `quickScan()`.
|
||||
|
||||
#### Step 4: Add Progress UI to admin.templ
|
||||
|
||||
**Location:** After the Quick Actions card (after line 64)
|
||||
|
||||
**Add this HTML after line 64:**
|
||||
```html
|
||||
<!-- Scan Progress Section -->
|
||||
<div id="scan-progress-container" class="hidden mt-6 p-6 rounded-lg border opacity-0 -translate-y-2.5 transition-all duration-300 ease-out"
|
||||
style="background-color: var(--bg-secondary); border-color: var(--border);">
|
||||
<div class="flex justify-between items-center mb-4">
|
||||
<h3 class="text-lg font-semibold" style="color: var(--text-primary)">
|
||||
📚 Scanning Libraries
|
||||
</h3>
|
||||
<button onclick="hideScanProgress()" class="p-2 hover:bg-gray-700 rounded">
|
||||
✕
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<!-- Overall Progress -->
|
||||
<div class="mb-4">
|
||||
<div class="flex justify-between text-sm mb-2">
|
||||
<span style="color: var(--text-secondary)">Overall Progress</span>
|
||||
<span id="scan-progress-text" style="color: var(--text-primary)">0%</span>
|
||||
</div>
|
||||
<div class="w-full bg-gray-700 rounded-full h-3">
|
||||
<div id="scan-progress-bar"
|
||||
class="h-3 rounded-full transition-all duration-500"
|
||||
style="width: 0%; background-color: var(--accent);">
|
||||
</div>
|
||||
</div>
|
||||
<div id="scan-status" class="text-sm mt-2" style="color: var(--text-secondary)">
|
||||
Starting scan...
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Per-Library Progress -->
|
||||
<div id="library-progress-list" class="space-y-3">
|
||||
<!-- Dynamically populated -->
|
||||
</div>
|
||||
|
||||
<!-- Results Summary -->
|
||||
<div id="scan-results" class="hidden mt-6 p-4 rounded-lg border"
|
||||
style="background-color: var(--bg-primary); border-color: var(--border);">
|
||||
<h4 class="font-semibold mb-2" style="color: var(--text-primary)">✅ Scan Complete!</h4>
|
||||
<div id="scan-results-content" style="color: var(--text-secondary)">
|
||||
<!-- Results populated by JS -->
|
||||
</div>
|
||||
<div class="mt-4 flex gap-2">
|
||||
<button onclick="window.location.reload()"
|
||||
class="btn-primary px-4 py-2 rounded-lg">
|
||||
Refresh to View Books
|
||||
</button>
|
||||
<button onclick="hideScanProgress()"
|
||||
class="btn-secondary px-4 py-2 rounded-lg">
|
||||
Dismiss
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
#### Step 5: Build TypeScript to JavaScript
|
||||
|
||||
**Build command:**
|
||||
```bash
|
||||
npm run build:ts
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
- Compiles `web/src/admin.ts` to `web/static/admin.js`
|
||||
- TypeScript compiler (`tsc`) handles the conversion
|
||||
- Output file `admin.js` will be loaded by the script tag added in Step 2
|
||||
|
||||
**Note:** This build step runs automatically in the Docker container during image build. For local development, run it manually after editing TypeScript files.
|
||||
|
||||
#### Step 6: Animation Implementation Note
|
||||
|
||||
**How the slide-in animation works:**
|
||||
|
||||
1. **Initial state** (Step 4 HTML): Container has classes `hidden opacity-0 -translate-y-2.5 transition-all duration-300 ease-out`
|
||||
2. **When scan starts** (Step 1 TypeScript): `showScanProgress()` function:
|
||||
```typescript
|
||||
container.classList.remove('hidden'); // Makes element visible
|
||||
container.classList.remove('opacity-0', '-translate-y-2.5'); // Triggers animation
|
||||
```
|
||||
3. **Result:** Browser transitions from `opacity-0` to `opacity-1` and `-translate-y-2.5` to `translate-y-0` over 300ms
|
||||
|
||||
**No custom CSS needed** - follows PROJECT_GUIDELINES.md "TailwindCSS classes only" rule.
|
||||
|
||||
---
|
||||
|
||||
### Part 2: Diagnose Dashboard Issue
|
||||
|
||||
**Diagnostic Steps:**
|
||||
|
||||
#### Step 1: Check System Collections Exist
|
||||
**Bruno Request:**
|
||||
```
|
||||
GET /api/dashboard/sections?library_id=849151fb-564e-4b24-89e3-d11360789576
|
||||
```
|
||||
|
||||
**Expected Response:**
|
||||
```json
|
||||
{
|
||||
"data": [
|
||||
{
|
||||
"title": "continue-reading",
|
||||
"query_type": "continue-reading",
|
||||
"items": [...]
|
||||
},
|
||||
{
|
||||
"title": "recently-added",
|
||||
"query_type": "recently-added",
|
||||
"items": [
|
||||
{
|
||||
"title": "Leviticus on the Butcher's Block",
|
||||
"created_at": "2025-02-24...",
|
||||
...
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "recently-read",
|
||||
"query_type": "recently-read",
|
||||
"items": [...]
|
||||
},
|
||||
{
|
||||
"title": "not-started",
|
||||
"query_type": "not-started",
|
||||
"items": [...]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**If sections array is empty or "recently-added" missing:**
|
||||
- System collections were not created for this user
|
||||
- Need to manually call `CreateDefaultCollectionsForUser`
|
||||
|
||||
#### Step 2: Check Book's Library
|
||||
**Bruno Request:**
|
||||
```
|
||||
GET /api/media-items?library_id=849151fb-564e-4b24-89e3-d11360789576&limit=5&sort=created_at+DESC
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
- Scanned book should appear first (most recent created_at)
|
||||
- If book appears here, it's in the correct library
|
||||
|
||||
**If book appears:**
|
||||
- Book is in correct library
|
||||
- Issue is with dashboard query or collection visibility
|
||||
|
||||
**If book doesn't appear:**
|
||||
- Book was added to different library
|
||||
- Check other libraries
|
||||
|
||||
#### Step 3: Check All Libraries
|
||||
**Bruno Request:**
|
||||
```
|
||||
GET /api/libraries
|
||||
```
|
||||
|
||||
**Purpose:**
|
||||
- See all available libraries
|
||||
- Check if book might be in a different library
|
||||
- Confirm the library_id being used
|
||||
|
||||
#### Step 4: Verify URL Library Parameter
|
||||
**Check:**
|
||||
- Does `/dashboard` URL have `?library_id=xxx` parameter?
|
||||
- Which library is selected in dropdown?
|
||||
|
||||
**If no library_id parameter:**
|
||||
- Dashboard auto-selects first visible library
|
||||
- Book might be in a different library
|
||||
|
||||
#### Step 5: Direct Database Check (if needed)
|
||||
|
||||
**Check system collections:**
|
||||
```sql
|
||||
SELECT name, query_type, show_on_dashboard
|
||||
FROM collections
|
||||
WHERE user_id = 'your-user-id'
|
||||
AND is_system_collection = true;
|
||||
```
|
||||
|
||||
**Check book's library:**
|
||||
```sql
|
||||
SELECT id, title, library_id, created_at
|
||||
FROM media_items
|
||||
WHERE title LIKE '%Leviticus%'
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 1;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Potential Fixes for Dashboard Issue
|
||||
|
||||
### Fix A: Recreate System Collections
|
||||
If system collections don't exist:
|
||||
|
||||
**Option 1: Manual API Call**
|
||||
```
|
||||
POST /api/admin/recreate-system-collections
|
||||
```
|
||||
(Endpoint may need to be created)
|
||||
|
||||
**Option 2: Direct Database**
|
||||
```sql
|
||||
INSERT INTO collections (user_id, name, description, icon, color, show_on_dashboard, query_type, priority, is_system_collection)
|
||||
VALUES
|
||||
('your-user-id', 'continue-reading', 'Books you''re currently reading (0 < progress < 1)', '📖', '#7aa2f7', true, 'continue-reading', 1, true),
|
||||
('your-user-id', 'recently-added', 'Newly added items to this library', '🆕', '#9ece6a', true, 'recently-added', 2, true),
|
||||
('your-user-id', 'recently-read', 'Books you''ve finished (progress >= 1)', '✅', '#e0af68', true, 'recently-read', 3, true),
|
||||
('your-user-id', 'not-started', 'Books you haven''t read yet (progress = 0 or no record)', '📕', '#f7768e', true, 'not-started', 4, true);
|
||||
```
|
||||
|
||||
**Option 3: Backend Handler**
|
||||
Create endpoint to recreate system collections for a user.
|
||||
|
||||
### Fix B: Switch to Correct Library
|
||||
If book is in different library:
|
||||
- Select the correct library in dropdown
|
||||
- Or create a combined view showing all libraries
|
||||
|
||||
### Fix C: Update User Preferences
|
||||
If collection is hidden:
|
||||
```
|
||||
GET /api/dashboard/preferences?library_id=xxx
|
||||
```
|
||||
Check if "recently-added" is in `hidden_collections`
|
||||
|
||||
Update:
|
||||
```
|
||||
PUT /api/dashboard/preferences
|
||||
{
|
||||
"library_id": "xxx",
|
||||
"hidden_collections": [], // Empty = show all
|
||||
"collection_order": [...],
|
||||
"items_per_section": 20
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Implementation Priority
|
||||
|
||||
1. **HIGH PRIORITY:** Fix Scan Library button
|
||||
- Impact: Users can't scan from UI at all
|
||||
- Effort: Medium (2-3 hours)
|
||||
- Files: 1 (`admin.templ`)
|
||||
|
||||
2. **MEDIUM PRIORITY:** Diagnose dashboard issue
|
||||
- Impact: Books exist but not visible
|
||||
- Effort: Low (30 min diagnosis)
|
||||
- Files: 0 (investigation only)
|
||||
|
||||
3. **LOW PRIORITY:** Fix dashboard issue
|
||||
- Impact: Depends on root cause
|
||||
- Effort: Unknown until diagnosis complete
|
||||
- Files: Unknown until diagnosis complete
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Checklist
|
||||
|
||||
### After Fixing Scan Button:
|
||||
|
||||
- [ ] Scan button triggers without errors
|
||||
- [ ] Progress UI appears
|
||||
- [ ] Progress bar updates every 2 seconds
|
||||
- [ ] Each library shows individual progress
|
||||
- [ ] Scan completes and shows results
|
||||
- [ ] "Refresh to View Books" works
|
||||
- [ ] Books appear on dashboard after refresh
|
||||
|
||||
### After Fixing Dashboard:
|
||||
|
||||
- [ ] Scanned book appears in "recently-added" section
|
||||
- [ ] Book is at top of list (most recent)
|
||||
- [ ] Book cover displays correctly
|
||||
- [ ] Clicking book opens it
|
||||
- [ ] All system collections show data
|
||||
- [ ] Collections can be hidden/shown
|
||||
- [ ] Dashboard works across page refreshes
|
||||
|
||||
---
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- Backend scanning is confirmed working (via Bruno)
|
||||
- Job status polling endpoint works correctly
|
||||
- Database contains the scanned book
|
||||
- Issue is purely frontend/dashboard display logic
|
||||
- System collections should be created during user registration
|
||||
- Dashboard supports library switching via dropdown
|
||||
- User can customize dashboard (hide collections, reorder, change items per section)
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Related Files
|
||||
|
||||
- `templates/admin.templ` - Admin page with Scan button
|
||||
- `templates/dashboard.templ` - Dashboard template
|
||||
- `internal/handlers/scanner.go` - Scan endpoints
|
||||
- `internal/services/dashboard_service.go` - Dashboard logic
|
||||
- `internal/services/worker.go` - Background job processing
|
||||
- `internal/handlers/dashboard.go` - Dashboard handlers
|
||||
- `internal/router/frontend.go` - Dashboard route
|
||||
- `web/static/dashboard.js` - Dashboard frontend logic
|
||||
- `bruno/scanner/Scan Media Items.yml` - Bruno collection (FIXED)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2025-02-24
|
||||
**Status:** Ready to implement Scan button fix, Dashboard issue needs diagnosis
|
||||
Reference in New Issue
Block a user