prev_plan.txt

Files Page Refactor - Tree-Based File Explorer
Summary
Transform the Files page into a professional tree-based file explorer (like Google Drive/Windows Explorer) with intelligent file grouping and keyboard navigation.
Design Overview
Left Sidebar - Hierarchical Tree View
Live Servers Section: Expandable tree of providers with lazy-loaded folder hierarchy
Scans Section: Compact one-line scan items with expandable folder trees
Background Tasks: Compact progress indicators
Search Box: Filter tree items with keyboard shortcut (/)
Center Panel - Simplified Browser
Remove server dropdown (redundant with tree)
Remove Live/Snapshot toggle (implicit from tree selection)
Breadcrumb navigation (synced with tree)
File table
Contextual actions (Scan/Commit based on selection)
Right Panel - Smart Metadata
Server selected: Provider info, connection status, mount points, scan button
Scan selected: ID, path, timestamps, commit status, commit/delete buttons
File selected: File properties, size, type, modified date
Folder selected: Folder info, item count, scan button
Answers to Your Questions
1. Tree Lazy-Loading ✅
YES - Fetch children when expanding (▶ → ▼)
Performance: Only load what user explores
Implementation: API call on expand, cache results
2. Tree Depth Limit
Industry Standard Approach:
Initial load: Show 1 level (immediate children only)
Lazy expansion: User controls depth by expanding nodes
Viewport management: Virtual scrolling for long lists (>100 items)
Max visible depth: No hard limit, but scroll container prevents clutter
Collapse all button: Quick reset if tree gets too deep
Implementation: Infinite depth with lazy-loading + virtual scrolling (like VS Code file explorer)
3. Scan Tree - Intelligent File Grouping
Show full file tree BUT with smart clustering:
Pattern Detection:
📁 imaging_stack/
  ├─ 📄 header.xml
  └─ 📂 stack_001.tiff ... stack_1000.tiff (1000 files)  ← Clustered
      └─ Click to expand individual files
Grouping Logic:
Detect sequential patterns:
file_001.tiff, file_002.tiff, ... file_999.tiff
Common prefix + number sequence + common extension
Group threshold:
If >10 files match pattern → create cluster node
Show as: 📂 file_001...999.tiff (999 files)
Expandable clusters:
Click cluster → shows all individual files (paginated if >100)
Or shows sub-ranges: 001-100, 101-200, etc.
Fallback:
If folder has >50 files with no pattern → show first 10 + "... and 40 more" with "Load More" button
Implementation:
function detectFilePatterns(files) {
  // Group by: commonPrefix + sequential numbers + commonExtension
  // Return: { type: 'cluster', pattern: 'stack_*.tiff', count: 1000, files: [...] }
}
4. Icons - Professional Icon Font ✅
Switch to Bootstrap Icons (already in project):
📁 → <i class="bi bi-folder"></i>
📄 → <i class="bi bi-file-earmark"></i>
💻 → <i class="bi bi-hdd"></i>
🌐 → <i class="bi bi-cloud"></i>
📸 → <i class="bi bi-camera"></i>
▶/▼ → <i class="bi bi-chevron-right"></i> / <i class="bi bi-chevron-down"></i>
5. Tree Search ✅
Search/Filter Box Above Tree:
Input box at top of sidebar
Live filtering as you type
Highlights matches in tree
Expands parent nodes to show matches
Clear button (X) when text entered
Keyboard shortcut: / to focus
6. Keyboard Shortcuts ✅
Full Keyboard Navigation:
↑/↓: Navigate tree items
←/→: Collapse/expand current node
Enter: Open folder in center panel
/: Focus search box
Escape: Clear search / deselect
Space: Select/deselect item
Ctrl+F: Alternative search focus
Implementation Plan
Phase 1: Tree Component Foundation (30% of work)
Files to modify: scidk/ui/templates/datasets.html
1.1 Tree HTML Structure
<div class="tree-search">
  <input type="text" placeholder="Search folders... (/)" />
</div>

<div class="tree-section">
  <div class="tree-section-header">LIVE SERVERS</div>
  <div class="tree-node" data-type="server" data-id="local_fs">
    <i class="bi bi-chevron-right toggle"></i>
    <i class="bi bi-hdd icon"></i>
    <span class="label">Local Filesystem</span>
  </div>
  <!-- Children loaded here on expand -->
</div>

<div class="tree-section">
  <div class="tree-section-header">SCANS</div>
  <div class="tree-node" data-type="scan" data-id="123">
    <i class="bi bi-chevron-right toggle"></i>
    <i class="bi bi-camera icon"></i>
    <span class="label">#123 /home/data (42 files)</span>
  </div>
</div>
1.2 Tree CSS
Indentation: padding-left: calc(level * 20px)
Hover states
Selection highlight
Expand/collapse animation
Icon spacing and sizing
1.3 Tree JavaScript Class
class FileTree {
  constructor(containerId) {
    this.container = document.getElementById(containerId);
    this.selectedNode = null;
    this.expandedNodes = new Set();
    this.nodeCache = new Map();
  }
  
  async expandNode(nodeId, type) {
    // Lazy-load children
    const children = await this.fetchChildren(nodeId, type);
    this.renderChildren(nodeId, children);
    this.expandedNodes.add(nodeId);
  }
  
  collapseNode(nodeId) {
    // Hide children
    this.expandedNodes.delete(nodeId);
  }
  
  selectNode(nodeId, type) {
    // Update selection, notify listeners
    this.selectedNode = { id: nodeId, type };
    this.emit('select', { id: nodeId, type });
  }
  
  filterTree(query) {
    // Filter and expand matching nodes
  }
}
Phase 2: Lazy Loading & API Integration (20% of work)
2.1 API Endpoints (already exist)
/api/providers → servers list
/api/provider_roots?provider_id=X → server roots
/api/browse?provider_id=X&path=Y → folder contents (for tree)
/api/scans → scans list
/api/scans/{id}/browse?path=Y → scan folder contents
2.2 Fetch Strategy
async function fetchTreeChildren(nodeType, nodeId, path) {
  const cacheKey = `${nodeType}:${nodeId}:${path}`;
  if (this.nodeCache.has(cacheKey)) {
    return this.nodeCache.get(cacheKey);
  }
  
  let data;
  if (nodeType === 'server') {
    const r = await fetch(`/api/browse?provider_id=${nodeId}&path=${path}`);
    data = await r.json();
  } else if (nodeType === 'scan') {
    const r = await fetch(`/api/scans/${nodeId}/browse?path=${path}`);
    data = await r.json();
  }
  
  const children = (data.entries || []).filter(e => e.type === 'folder');
  this.nodeCache.set(cacheKey, children);
  return children;
}
2.3 Virtual Scrolling (for large lists)
Use IntersectionObserver for viewport detection
Only render visible tree nodes (if folder has 1000+ items)
Render buffer: 20 items above/below viewport
Phase 3: Intelligent File Clustering (25% of work)
3.1 Pattern Detection Algorithm
function detectSequentialPatterns(files) {
  const groups = {};
  
  files.forEach(file => {
    // Extract: prefix, number, extension
    const match = file.name.match(/^(.+?)(\d+)(\.\w+)$/);
    if (match) {
      const [_, prefix, num, ext] = match;
      const key = `${prefix}*${ext}`;
      if (!groups[key]) groups[key] = [];
      groups[key].push({ name: file.name, num: parseInt(num), ...file });
    }
  });
  
  // Identify sequential groups (>10 files)
  return Object.entries(groups)
    .filter(([_, items]) => items.length > 10)
    .map(([pattern, items]) => {
      items.sort((a, b) => a.num - b.num);
      const min = items[0].num;
      const max = items[items.length - 1].num;
      return {
        type: 'cluster',
        pattern: pattern,
        range: `${min}-${max}`,
        count: items.length,
        files: items
      };
    });
}
3.2 Cluster Node Rendering
function renderCluster(cluster) {
  return `
    <div class="tree-node tree-cluster" data-type="cluster">
      <i class="bi bi-chevron-right toggle"></i>
      <i class="bi bi-files icon"></i>
      <span class="label">${cluster.pattern.replace('*', cluster.range)} (${cluster.count} files)</span>
    </div>
  `;
}

// On expand: show sub-ranges or individual files
function expandCluster(cluster) {
  if (cluster.count > 100) {
    // Create sub-ranges: 1-100, 101-200, etc.
    return createSubRanges(cluster.files, 100);
  } else {
    // Show all individual files
    return cluster.files.map(f => renderFileNode(f));
  }
}
3.3 Fallback for Unstructured Folders
function renderLargeFolder(files) {
  if (files.length > 50) {
    const visible = files.slice(0, 10);
    const remaining = files.length - 10;
    return [
      ...visible.map(renderFileNode),
      `<div class="tree-node tree-more">
        <span>... and ${remaining} more</span>
        <button class="btn btn-sm">Load More</button>
      </div>`
    ];
  }
  return files.map(renderFileNode);
}
Phase 4: Smart Right Panel (15% of work)
4.1 Metadata Views
function updateMetadataPanel(selection) {
  const panel = document.getElementById('file-metadata');
  
  switch (selection.type) {
    case 'server':
      panel.innerHTML = renderServerMetadata(selection.id);
      break;
    case 'scan':
      panel.innerHTML = renderScanMetadata(selection.id);
      break;
    case 'folder':
      panel.innerHTML = renderFolderMetadata(selection);
      break;
    case 'file':
      panel.innerHTML = renderFileMetadata(selection);
      break;
  }
  
  // Auto-expand right panel if collapsed
  if (detailsPanel.classList.contains('collapsed')) {
    expandDetailsPanel();
  }
}
4.2 Contextual Actions
function renderServerMetadata(serverId) {
  return `
    <h4><i class="bi bi-hdd"></i> Server Details</h4>
    <table class="table table-sm">
      <tr><td>Type:</td><td>${server.type}</td></tr>
      <tr><td>Status:</td><td>${server.connected ? '✓ Connected' : '✗ Disconnected'}</td></tr>
    </table>
    <button class="btn btn-primary w-100" onclick="scanCurrentFolder()">
      <i class="bi bi-search"></i> Scan Current Folder
    </button>
  `;
}

function renderScanMetadata(scanId) {
  return `
    <h4><i class="bi bi-camera"></i> Scan #${scan.id}</h4>
    <table class="table table-sm">
      <tr><td>Path:</td><td>${scan.path}</td></tr>
      <tr><td>Files:</td><td>${scan.file_count}</td></tr>
      <tr><td>Started:</td><td>${formatTime(scan.started)}</td></tr>
      <tr><td>Committed:</td><td>${scan.committed ? 'Yes' : 'No'}</td></tr>
    </table>
    <button class="btn btn-success w-100 mb-2" onclick="commitScan(${scanId})">
      <i class="bi bi-check-circle"></i> Commit to Graph
    </button>
    <button class="btn btn-outline-danger w-100" onclick="deleteScan(${scanId})">
      <i class="bi bi-trash"></i> Delete Scan
    </button>
  `;
}
Phase 5: Keyboard Navigation (5% of work)
5.1 Keyboard Event Handlers
document.addEventListener('keydown', (e) => {
  // / to focus search
  if (e.key === '/' && !isInputFocused()) {
    e.preventDefault();
    focusSearch();
  }
  
  // Arrow navigation when tree focused
  if (treeHasFocus()) {
    switch(e.key) {
      case 'ArrowDown':
        e.preventDefault();
        navigateDown();
        break;
      case 'ArrowUp':
        e.preventDefault();
        navigateUp();
        break;
      case 'ArrowRight':
        e.preventDefault();
        expandCurrentNode();
        break;
      case 'ArrowLeft':
        e.preventDefault();
        collapseCurrentNode();
        break;
      case 'Enter':
        e.preventDefault();
        selectCurrentNode();
        break;
      case 'Escape':
        clearSelection();
        break;
    }
  }
});
Phase 6: Polish & Integration (5% of work)
6.1 Tree Animations
.tree-node.expanding {
  animation: slideDown 0.2s ease-out;
}

.tree-node.collapsing {
  animation: slideUp 0.2s ease-out;
}

@keyframes slideDown {
  from { opacity: 0; max-height: 0; }
  to { opacity: 1; max-height: 500px; }
}
6.2 Loading States
function showTreeLoading(nodeId) {
  const node = getTreeNode(nodeId);
  node.querySelector('.toggle').innerHTML = '<span class="spinner-border spinner-border-sm"></span>';
}

function hideTreeLoading(nodeId) {
  const node = getTreeNode(nodeId);
  node.querySelector('.toggle').innerHTML = '<i class="bi bi-chevron-down"></i>';
}
6.3 Remember Expanded State
// Save to localStorage
function saveTreeState() {
  localStorage.setItem('filesTreeExpanded', JSON.stringify([...expandedNodes]));
}

// Restore on page load
function restoreTreeState() {
  const saved = JSON.parse(localStorage.getItem('filesTreeExpanded') || '[]');
  saved.forEach(nodeId => expandNode(nodeId));
}
File Structure
Modified Files:
scidk/ui/templates/datasets.html (complete rewrite ~1200 lines)
No New Files Needed:
All functionality in single template
Uses existing Bootstrap Icons
Uses existing API endpoints
Testing Plan
Manual Testing:
Tree Navigation: Expand/collapse servers and scans
Lazy Loading: Verify children load only on expand
File Clustering: Test with imaging stack folder (1000+ files)
Search: Filter tree, verify matches highlight
Keyboard Nav: Test all shortcuts (arrows, /, Enter, Esc)
Metadata Panel: Verify correct info for server/scan/file/folder
Actions: Test scan folder, commit scan, delete scan
Breadcrumb Sync: Click tree node → verify breadcrumb updates
Mode Switching: Live server → scan → verify UI adapts
E2E Tests:
Update tests/test_files_page_e2e.py or create new E2E test
Test tree expansion, file selection, scanning, commit
Estimated Effort
Total: ~8-10 hours
Phase 1 (Tree Foundation): 3 hours
Phase 2 (Lazy Loading): 2 hours
Phase 3 (File Clustering): 2.5 hours
Phase 4 (Metadata Panel): 1.5 hours
Phase 5 (Keyboard Nav): 0.5 hours
Phase 6 (Polish): 0.5 hours
Risk Mitigation
Performance Risks:
Large folders (10k+ files): Use virtual scrolling + pagination
Deep trees: Lazy-loading prevents loading entire tree upfront
Slow API calls: Show loading spinners, cache results
UX Risks:
Confusing clustering: Provide "Show all files" option to bypass grouping
Lost in deep tree: Add "Collapse All" button, breadcrumb navigation
Keyboard conflicts: Document shortcuts, use non-conflicting keys
Success Criteria
✅ Functional:
Tree expands/collapses smoothly
Lazy-loading works for both live and scan trees
File clustering works for sequential patterns (1000+ files)
Metadata panel shows correct info based on selection
All keyboard shortcuts work
Search filters tree in real-time
✅ Visual:
Professional icon font (Bootstrap Icons)
Consistent indentation and spacing
Smooth animations
Clear visual hierarchy
✅ Performance:
Page loads in <2s
Tree expansion <500ms
Search filtering <100ms
No UI freezing with large folders
Ready to implement?