# Implementation Plan: Docs Search API ## Overview Add backend full-text search for 152 documentation files using a new `/api/docs/search` endpoint, replacing the planned build-time Lunr approach with runtime server-side search. **Type**: Full-stack task (backend API + frontend changes) **Status**: Planned **Priority**: Medium **Estimated effort**: 3-4 hours --- ## Table of Contents 1. [Architecture Decision](#architecture-decision) 2. [Backend Implementation](#backend-implementation) 3. [Frontend Changes](#frontend-changes) 4. [Testing Strategy](#testing-strategy) 5. [Dependencies to Remove](#dependencies-to-remove) 6. [Documentation Updates](#documentation-updates) 7. [Implementation Checklist](#implementation-checklist) --- ## Architecture Decision ### Why Backend API Instead of Build-time Lunr? | Aspect | Build-time Lunr (Original Plan) | Backend API (This Plan) | |--------|----------------------------------|-------------------------| | Build step | ✅ Required | ❌ None | | Network request | ❌ Client-side only | ✅ API call | | Server load | ❌ None | ✅ Minimal (152 docs) | | Offline support | ✅ Yes | ❌ No | | Search speed | ⚡ Instant | 🌐 Network latency | | Bundle size | +50KB | No change | | Always current | ❌ Need rebuild | ✅ Yes | | Consistency | New pattern | Matches existing `/api/media-items/search` | **Decision**: Backend API is simpler and more consistent with existing architecture. --- ## Backend Implementation ### 1. Add Search Method to DocsHandler **File**: `internal/docs/search.go` (new file) ```go package docs import ( "bufio" "fmt" "io/fs" "path/filepath" "strings" "unicode" ) // SearchResult represents a single search result type SearchResult struct { Path string `json:"path"` Title string `json:"title"` Section string `json:"section"` Snippet string `json:"snippet,omitempty"` } // SearchDocuments performs full-text search across all markdown files func (h *DocsHandler) SearchDocuments(query string) ([]SearchResult, error) { // Validate query query = strings.TrimSpace(query) if len(query) < 2 { return []SearchResult{}, nil } // Convert to lowercase for case-insensitive search queryLower := strings.ToLower(query) var results []SearchResult // Walk through all markdown files err := fs.WalkDir(h.docsFS, ".", func(path string, d fs.DirEntry, err error) error { if err != nil { return err } // Skip directories and non-markdown files if d.IsDir() || !strings.HasSuffix(path, ".md") { return nil } // Read file content content, err := fs.ReadFile(h.docsFS, path) if err != nil { // Log but continue - don't fail entire search fmt.Printf("Warning: failed to read %s: %v\n", path, err) return nil } // Extract title and check for matches title := h.extractTitle(content) section := h.getSectionFromPath(path) textContent := h.markdownToText(content) // Check if query matches if h.matchesQuery(textContent, queryLower) { result := SearchResult{ Path: path, Title: title, Section: section, } // Generate snippet with highlighted match snippet := h.extractSnippet(textContent, queryLower) if snippet != "" { result.Snippet = snippet } results = append(results, result) } return nil }) if err != nil { return nil, fmt.Errorf("failed to walk docs directory: %w", err) } return results, nil } // getSectionFromPath derives section name from file path func (h *DocsHandler) getSectionFromPath(path string) string { // Remove .md extension path = strings.TrimSuffix(path, ".md") // Split by directory separators parts := strings.Split(path, string(filepath.Separator)) // Filter out empty strings and index.md var filtered []string for _, part := range parts { if part != "" && part != "index.md" && part != "index" { // Convert kebab-case to Title Case filtered = append(filtered, kebabToTitle(part)) } } // Join with " > " return strings.Join(filtered, " > ") } // kebabToTitle converts kebab-case to Title Case func kebabToTitle(s string) string { words := strings.Split(s, "-") for i, word := range words { if len(word) > 0 { // Capitalize first letter runes := []rune(word) runes[0] = unicode.ToUpper(runes[0]) words[i] = string(runes) } } return strings.Join(words, " ") } // markdownToText converts markdown content to plain text for searching func (h *DocsHandler) markdownToText(content []byte) string { // Remove markdown syntax but keep text lines := strings.Split(string(content), "\n") var textLines []string for _, line := range lines { line = strings.TrimSpace(line) // Skip code blocks if strings.HasPrefix(line, "```") { continue } // Remove markdown headers line = strings.TrimLeft(line, "#") // Remove bold/italic markers line = strings.ReplaceAll(line, "**", "") line = strings.ReplaceAll(line, "*", "") line = strings.ReplaceAll(line, "__", "") line = strings.ReplaceAll(line, "_", "") // Remove links but keep text line = h.removeMarkdownLinks(line) // Remove code inline line = strings.ReplaceAll(line, "`", "") if line != "" { textLines = append(textLines, line) } } return strings.Join(textLines, " ") } // removeMarkdownLinks removes markdown link syntax func (h *DocsHandler) removeMarkdownLinks(line string) string { // Simple approach: remove [text](url) patterns result := line for { start := strings.Index(result, "[") if start == -1 { break } end := strings.Index(result[start:], "](") if end == -1 { break } end += start + 2 // include "](" // Find closing paren closeParen := strings.Index(result[end:], ")") if closeParen == -1 { break } closeParen += end // Extract link text linkText := result[start+1 : start+end-start-2] // Replace entire link with just text result = result[:start] + linkText + result[closeParen+1:] } return result } // matchesQuery checks if content matches the search query func (h *DocsHandler) matchesQuery(content, queryLower string) bool { contentLower := strings.ToLower(content) return strings.Contains(contentLower, queryLower) } // extractSnippet extracts a snippet around the matched term func (h *DocsHandler) extractSnippet(content, queryLower string) string { contentLower := strings.ToLower(content) // Find first match idx := strings.Index(contentLower, queryLower) if idx == -1 { return "" } // Extract context around match (100 chars before and after) start := idx - 50 if start < 0 { start = 0 } end := idx + len(queryLower) + 50 if end > len(content) { end = len(content) } snippet := content[start:end] // Add ellipsis if truncated if start > 0 { snippet = "..." + snippet } if end < len(content) { snippet = snippet + "..." } return snippet } ``` ### 2. Add HTTP Handler **File**: `internal/docs/http_handler.go` (existing file - append to it) ```go package docs import ( "encoding/json" "net/http" "strings" "github.com/go-chi/chi/v5" ) // HandleSearch handles documentation search requests func (h *DocsHandler) HandleSearch(w http.ResponseWriter, r *http.Request) { // Extract query parameter query := r.URL.Query().Get("q") query = strings.TrimSpace(query) // Validate query if query == "" { w.WriteHeader(http.StatusBadRequest) json.NewEncoder(w).Encode(map[string]interface{}{ "error": "query parameter 'q' is required", }) return } if len(query) < 2 { w.WriteHeader(http.StatusBadRequest) json.NewEncoder(w).Encode(map[string]interface{}{ "error": "query must be at least 2 characters", }) return } // Perform search results, err := h.SearchDocuments(query) if err != nil { w.WriteHeader(http.StatusInternalServerError) json.NewEncoder(w).Encode(map[string]interface{}{ "error": "failed to search documents", }) return } // Return results w.Header().Set("Content-Type", "application/json") json.NewEncoder(w).Encode(map[string]interface{}{ "query": query, "count": len(results), "results": results, }) } // RegisterDocsSearchRoutes registers search endpoints func RegisterDocsSearchRoutes(r chi.Router, handler *DocsHandler) { r.Get("/api/docs/search", handler.HandleSearch) } ``` ### 3. Register Route **File**: `internal/router/docs.go` (modify existing) Add the search route registration: ```go func RegisterDocsRoutes(r chi.Router, docsPath string) { handler := docs.NewDocsHandler(docsPath) // Existing routes... r.Get("/docs", docsHandler) r.Get("/docs/*", docsHandler) // NEW: Search route docs.RegisterDocsSearchRoutes(r, handler) } ``` --- ## Frontend Changes ### 1. Update docs.ts to Remove Lunr **File**: `web/src/docs.ts` **Remove broken imports** (lines 1-4): ```typescript // DELETE THESE LINES: import { docs } from "../data/docs.json"; import searchIndex from "../data/search_index.json"; import * as lunr from "lunr"; import hljs from "highlight.js"; ``` **Replace search function** (lines 46-88): ```typescript // BEFORE: function performDocsSearch(query: string): void { const searchResults = document.getElementById("docs-search-results"); if (!searchResults) return; try { const idx = lunr.Builder.loadJs(searchIndex); // ... lunr search logic } catch (error) { console.error("Search error:", error); } } // AFTER: async function performDocsSearch(query: string): Promise { const searchResults = document.getElementById("docs-search-results"); if (!searchResults) return; try { // Show loading state searchResults.innerHTML = '

Searching...

'; searchResults.classList.remove("hidden"); // Fetch from backend API const response = await fetch(`/api/docs/search?q=${encodeURIComponent(query)}`); if (!response.ok) { throw new Error(`Search failed: ${response.statusText}`); } const data = await response.json(); if (!data.results || data.results.length === 0) { searchResults.innerHTML = '

No results found

'; } else { searchResults.innerHTML = data.results .slice(0, 10) .map((result: { path: string; title: string; section: string; snippet?: string }) => { return `

${result.title || result.path}

${result.section ? `

${result.section}

` : ""} ${result.snippet ? `

${result.snippet}

` : ""}
`; }) .join(""); } searchResults.classList.remove("hidden"); } catch (error) { console.error("Search error:", error); searchResults.innerHTML = '

Search error

'; searchResults.classList.remove("hidden"); } } ``` **Keep the rest** - toggleSidebar, initializeDocsSearch, and Alpine registration remain the same. ### 2. Remove Lunr Dependency **File**: `package.json` **Remove from dependencies** (line 16): ```json "lunr": "^2.3.9", // DELETE THIS LINE ``` **Run**: ```bash npm uninstall lunr ``` --- ## Testing Strategy ### 1. Unit Tests (No API Required) **File**: `internal/docs/search_test.go` (new file) ```go package docs import ( "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) func TestMarkdownToText(t *testing.T) { handler := &DocsHandler{} testCases := []struct { name string input string expected string }{ { name: "Simple text", input: "Hello world", expected: "Hello world", }, { name: "Remove headers", input: "# Title\n\nSome content", expected: "Title Some content", }, { name: "Remove bold", input: "This is **bold** text", expected: "This is bold text", }, { name: "Remove links", input: "See [the docs](/docs/index.md) for more", expected: "See the docs for more", }, { name: "Remove code blocks", input: "```go\nfunc test() {}\n```", expected: "", }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { result := handler.markdownToText([]byte(tc.input)) assert.Equal(t, tc.expected, result) }) } } func TestGetSectionFromPath(t *testing.T) { handler := &DocsHandler{} testCases := []struct { name string path string expected string }{ { name: "API endpoint", path: "developer/api/auth/login.md", expected: "Developer > Api > Auth > Login", }, { name: "User guide", path: "user/devices/kobo-setup.md", expected: "User > Devices > Kobo Setup", }, { name: "Index file", path: "index.md", expected: "", }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { result := handler.getSectionFromPath(tc.path) assert.Equal(t, tc.expected, result) }) } } func TestMatchesQuery(t *testing.T) { handler := &DocsHandler{} testCases := []struct { name string content string query string expected bool }{ { name: "Exact match", content: "Kobo setup guide for e-readers", query: "kobo", expected: true, }, { name: "Case insensitive", content: "Kobo Setup Guide", query: "KOBO", expected: true, }, { name: "No match", content: "iPhone setup guide", query: "kobo", expected: false, }, { name: "Partial match", content: "Configure your Kobo device", query: "kobo device", expected: true, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { result := handler.matchesQuery(tc.content, tc.query) assert.Equal(t, tc.expected, result) }) } } func TestExtractSnippet(t *testing.T) { handler := &DocsHandler{} testCases := []struct { name string content string query string expectSnippet bool }{ { name: "Match in middle", content: "This is some long text with Kobo device configuration options", query: "kobo device", expectSnippet: true, }, { name: "No match", content: "This is some text without the term", query: "kobo", expectSnippet: false, }, { name: "Match at start", content: "Kobo setup guide for e-readers with detailed instructions", query: "kobo", expectSnippet: true, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { result := handler.extractSnippet(tc.content, tc.query) if tc.expectSnippet { assert.NotEmpty(t, result) assert.Contains(t, result, "...") } else { assert.Empty(t, result) } }) } } func TestKebabToTitle(t *testing.T) { testCases := []struct { input string expected string }{ {"kobo-setup", "Kobo Setup"}, {"api-reference", "Api Reference"}, {"user-guide", "User Guide"}, {"auth", "Auth"}, } for _, tc := range testCases { t.Run(tc.input, func(t *testing.T) { result := kebabToTitle(tc.input) assert.Equal(t, tc.expected, result) }) } } ``` ### 2. Integration Tests (with API) **File**: `cmd/server/tests/docs_search_test.go` (new file) ```go package main import ( "encoding/json" "net/http" "net/http/httptest" "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) func TestDocsSearchAPI(t *testing.T) { // Setup test server ts := setupTestServer(t) defer ts.cleanup(t) testCases := []struct { name string query string expectedStatus int expectResults bool description string }{ { name: "Empty query", query: "", expectedStatus: http.StatusBadRequest, expectResults: false, description: "Should reject empty query", }, { name: "Single character", query: "a", expectedStatus: http.StatusBadRequest, expectResults: false, description: "Should reject queries < 2 chars", }, { name: "Valid search - kobo", query: "kobo", expectedStatus: http.StatusOK, expectResults: true, description: "Should find Kobo documentation", }, { name: "Valid search - API", query: "API", expectedStatus: http.StatusOK, expectResults: true, description: "Should find API documentation (case insensitive)", }, { name: "Valid search - no results", query: "xyznonexistent", expectedStatus: http.StatusOK, expectResults: false, description: "Should return empty results for non-existent term", }, { name: "Valid search - sync", query: "sync", expectedStatus: http.StatusOK, expectResults: true, description: "Should find sync documentation", }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { // Create request req := httptest.NewRequest("GET", "/api/docs/search?q="+tc.query, nil) rr := httptest.NewRecorder() // Serve request ts.handler.ServeHTTP(rr, req) // Check status assert.Equal(t, tc.expectedStatus, rr.Code, tc.description) // Parse response var response map[string]interface{} err := json.Unmarshal(rr.Body.Bytes(), &response) require.NoError(t, err) // Validate response structure if tc.expectedStatus == http.StatusOK { assert.Contains(t, response, "query") assert.Contains(t, response, "count") assert.Contains(t, response, "results") results, ok := response["results"].([]interface{}) require.True(t, ok, "results should be an array") if tc.expectResults { assert.Greater(t, len(results), 0, "Should have results") // Check first result structure if len(results) > 0 { firstResult, ok := results[0].(map[string]interface{}) require.True(t, ok, "result should be an object") assert.Contains(t, firstResult, "path") assert.Contains(t, firstResult, "title") assert.Contains(t, firstResult, "section") } } else { assert.Equal(t, 0, len(results), "Should have no results") } } else { assert.Contains(t, response, "error") } }) } } func TestDocsSearchResultStructure(t *testing.T) { ts := setupTestServer(t) defer ts.cleanup(t) // Search for a known term req := httptest.NewRequest("GET", "/api/docs/search?q=kobo", nil) rr := httptest.NewRecorder() ts.handler.ServeHTTP(rr, req) assert.Equal(t, http.StatusOK, rr.Code) var response map[string]interface{} err := json.Unmarshal(rr.Body.Bytes(), &response) require.NoError(t, err) results, ok := response["results"].([]interface{}) require.True(t, ok) require.Greater(t, len(results), 0, "Should have at least one result for 'kobo'") // Validate first result has all required fields firstResult := results[0].(map[string]interface{}) assert.Contains(t, firstResult, "path", "Should have path") assert.Contains(t, firstResult, "title", "Should have title") assert.Contains(t, firstResult, "section", "Should have section") // Check types assert.IsType(t, "", firstResult["path"]) assert.IsType(t, "", firstResult["title"]) assert.IsType(t, "", firstResult["section"]) // Snippet is optional if snippet, ok := firstResult["snippet"]; ok { assert.IsType(t, "", snippet) } } ``` **Note**: `setupTestServer()` helper should be called once at the test function level (not per subtest). If test_helpers.go doesn't exist yet, create it following existing test patterns in `cmd/server/tests/`. ### 3. Bruno OpenCollection YAML **File**: `bruno/docs/Search Docs.yml` (new file) ```yaml info: name: Search Docs type: http seq: 1 http: method: GET url: '{{base_url}}/api/docs/search?q=kobo' auth: none docs: |- ## Search Documentation Performs full-text search across all documentation files. **Method:** GET **Endpoint:** /api/docs/search **Query Parameters:** - `q` (string, required): Search query (minimum 2 characters) **Authentication:** Not required (public documentation) **Response:** ```json { "query": "kobo", "count": 5, "results": [ { "path": "user/devices/kobo-setup.md", "title": "Kobo Setup Guide", "section": "User > Devices > Kobo Setup", "snippet": "Complete Kobo e-reader configuration instructions..." } ] } ``` **Status Codes:** - 200: Success (results may be empty) - 400: Bad request (missing or invalid query) - 500: Internal server error **Examples:** ```bash # Search for "kobo" GET /api/docs/search?q=kobo # Search for "API authentication" GET /api/docs/search?q=api%20authentication # Case insensitive GET /api/docs/search?q=KOBO # Minimum 2 characters GET /api/docs/search?q=a # Returns 400 Bad Request ``` --- ## Dependencies to Remove ### 1. Lunr (npm package) **File**: `package.json` **Before** (line 16): ```json "lunr": "^2.3.9", ``` **After**: Delete this line **Run**: ```bash npm uninstall lunr ``` ### 2. Clean Up docs.ts **File**: `web/src/docs.ts` **Remove imports** (lines 1-4): ```typescript import { docs } from "../data/docs.json"; import searchIndex from "../data/search_index.json"; import * as lunr from "lunr"; import hljs from "highlight.js"; ``` **Note**: `hljs` import can also be removed if not used elsewhere in the file. --- ## Documentation Updates ### 1. API Documentation **File**: `docs/developer/api/docs/search.md` (new file) ```markdown # Search Documentation Search across all documentation files with full-text search. ## Endpoint `GET /api/docs/search` ## Authentication Not required (public documentation). ## Query Parameters | Parameter | Type | Required | Description | |-----------|--------|----------|----------------------------------| | q | string | Yes | Search query (min 2 characters) | ## Response ### Success (200 OK) ```json { "query": "kobo", "count": 5, "results": [ { "path": "user/devices/kobo-setup.md", "title": "Kobo Setup Guide", "section": "User > Devices > Kobo Setup", "snippet": "...Complete Kobo e-reader configuration..." } ] } ``` ### Bad Request (400) Missing or invalid query: ```json { "error": "query parameter 'q' is required" } ``` Query too short: ```json { "error": "query must be at least 2 characters" } ``` ## Examples ```bash # Search for Kobo documentation curl "https://bookhoard.com/api/docs/search?q=kobo" # Search for API endpoints curl "https://bookhoard.com/api/docs/search?q=api%20authentication" # Multi-word search curl "https://bookhoard.com/api/docs/search?q=device%20sync" ``` ## Notes - Search is case-insensitive - Searches across title, section, and content - Returns up to all matches (frontend can limit) - Snippets include context around matched terms - Results are not ranked (order depends on file system traversal) ``` ### 2. Update Development Documentation **File**: `docs/contributing/development.md` Add section about the new search endpoint (if there's a section about API development or docs system). ### 3. Update README **File**: `README.md` If there's a section about documentation features, mention the new search functionality. --- ## Implementation Checklist ### Phase 1: Backend (1.5 hours) - [ ] Create `internal/docs/search.go` with SearchDocuments method - [ ] Add search handler to `internal/docs/http_handler.go` - [ ] Register search route in `internal/router/docs.go` - [ ] Write unit tests for search logic (`internal/docs/search_test.go`) - [ ] Run unit tests: `go test ./internal/docs/... -v` - [ ] Verify compilation: `go build ./...` ### Phase 2: Integration Testing (1 hour) - [ ] Create `cmd/server/tests/docs_search_test.go` - [ ] Implement table-driven integration tests - [ ] Test all three contexts (no auth, user, admin) - though docs search doesn't require auth - [ ] Run integration tests: `go test ./cmd/server/tests/... -v` - [ ] Verify tests pass ### Phase 3: Frontend (1 hour) - [ ] Update `web/src/docs.ts` to remove Lunr imports - [ ] Replace performDocsSearch with async API call - [ ] Test search in browser (manual testing) - [ ] Verify no 404 errors for missing JSON files ### Phase 4: Cleanup (30 minutes) - [ ] Remove Lunr from `package.json` - [ ] Run `npm uninstall lunr` - [ ] Rebuild frontend: `npm run build:ts` - [ ] Verify no TypeScript errors - [ ] Remove any references to `/web/src/data/` directory ### Phase 5: Documentation & Bruno (30 minutes) - [ ] Create `bruno/docs/Search Docs.yml` - [ ] Test endpoint with Bruno - [ ] Create `docs/developer/api/docs/search.md` - [ ] Update `docs/contributing/development.md` if needed - [ ] Verify docs render at `/docs` endpoint - [ ] Test docs search finds new API documentation ### Phase 6: Final Verification (30 minutes) - [ ] Run full test suite: `go test ./... -v` - [ ] Run verification script: `bash scripts/verify-guidelines.sh` - [ ] Build entire project: `go build ./...` - [ ] Test search in running application - [ ] Verify all three test cases pass: - Empty query → 400 error - Valid search → results - No results → empty array - [ ] Check git diff for unintended changes - [ ] Review all modified files --- ## Git Commit Strategy Follow PROJECT_GUIDELINES.md - make multiple logical commits: 1. **Backend implementation** (search.go, http_handler.go, router changes) 2. **Unit tests** (search_test.go) 3. **Integration tests** (docs_search_test.go) 4. **Frontend changes** (docs.ts updates, remove Lunr) 5. **Cleanup** (package.json, remove data directory references) 6. **Documentation** (API docs, Bruno YAML, contributing guide) --- ## Success Criteria ✅ Backend API `/api/docs/search` returns 200 with results ✅ Empty/short queries return 400 Bad Request ✅ Unit tests pass for all search helper functions ✅ Integration tests pass with table-driven tests ✅ Bruno YAML validates successfully ✅ Frontend search works without errors ✅ No 404s for missing JSON files ✅ Lunr dependency removed from package.json ✅ API documentation created and renders ✅ Full test suite passes ✅ Verification script passes (0 errors) --- ## Rollback Plan If issues arise: 1. **Backend**: Revert search.go, http_handler.go changes 2. **Tests**: Remove new test files 3. **Frontend**: Restore original docs.ts with Lunr code 4. **Dependencies**: Reinstall Lunr: `npm install lunr@^2.3.9` Use `git diff` to identify changes and `git checkout` to revert specific files if needed.