diff --git a/docs/analytics/phase5-export-and-reporting.md b/docs/analytics/phase5-export-and-reporting.md new file mode 100644 index 0000000..42dd7ef --- /dev/null +++ b/docs/analytics/phase5-export-and-reporting.md @@ -0,0 +1,177 @@ +# Phase 5: Export & Reporting + +## PR +[#30](https://github.com/Leddger-AI/LedgerAI/pull/30) — `feature/analytics-phase5` — **Open** + +## Overview + +Phase 5 adds CSV and JSON export capabilities to the analytics feature. Users can download overview analytics (all templates, KPIs, trends) or per-template detail analytics (field stats, raw submissions) as CSV or JSON files directly from the UI. + +## Backend + +### exportUtils.js (`server/utils/exportUtils.js`) + +Utility module with 5 export functions + 2 CSV helpers. + +#### `escapeCSV(value)` +Escapes a value for safe CSV inclusion: +- Wraps in quotes if value contains comma, quote, newline, or carriage return +- Doubles internal quotes per RFC 4180 + +#### `arrayToCSV(rows)` +Converts array of objects to CSV string: +- First object's keys become headers +- Each row's values are escaped and comma-joined +- Returns header line + data lines separated by `\n` + +#### `exportOverviewCSV(ownerUid)` +Generates a multi-section CSV with 4 sections: +1. **Overview KPIs** — TotalTemplates, ActiveLinks, TotalSubmissions, AvgFieldsPerTemplate +2. **Templates** — All templates with submission counts and dates +3. **Submission Trends** — Last 30 days of daily submission counts +4. **Template Type Distribution** — Type counts for donut chart data + +Sections are separated by blank lines and `# Section Name` headers. + +#### `exportOverviewJSON(ownerUid)` +Generates structured JSON with: +```json +{ + "generatedAt": "2024-08-14T18:00:00Z", + "overview": { "totalTemplates": 12, "activeLinks": 5, ... }, + "templates": [ { "draftId": "...", "title": "...", ... } ], + "trends": [ { "date": "2024-08-01", "count": 3 }, ... ], + "typeDistribution": [ { "type": "student", "count": 8 }, ... ] +} +``` + +#### `exportTemplateDetailCSV(ownerUid, draftId)` +Generates a 3-section CSV for a specific template: +1. **Template Info** — Title, Type, Status, TotalSubmissions, EnabledFields, CreatedAt, ExpiresAt +2. **Field Statistics** — Per-field: Filled, CompletionRate, Average, Min, Max +3. **Raw Submissions** — All submissions flattened with submittedData keys as columns + +Returns `null` if template not found (caller returns 404). + +#### `exportTemplateDetailJSON(ownerUid, draftId)` +Generates structured JSON: +```json +{ + "generatedAt": "2024-08-14T18:00:00Z", + "template": { + "draftId": "...", "title": "...", "templateType": "...", + "status": "...", "totalSubmissions": 23, + "enabledFields": ["name", "experience", ...], + "fieldStats": { "experience": { "avg": 3.5, ... } } + }, + "submissions": [ { "submissionId": "...", "submittedData": {...}, ... } ] +} +``` + +#### `exportAllSubmissionsCSV(ownerUid, draftId)` +Generates a flat CSV of all submissions for a template. Dynamic columns from `submittedData` keys. Returns "No submissions found" if empty. + +--- + +### API Endpoints (`server/index.js`) + +5 new endpoints, all behind `verifyToken` and scoped to `req.user.uid`. + +| Endpoint | Format | Description | +|---|---|---| +| `GET /api/analytics/export/overview.csv` | CSV | Multi-section overview export | +| `GET /api/analytics/export/overview.json` | JSON | Structured JSON overview | +| `GET /api/analytics/templates/:draftId/export.csv` | CSV | Template detail with field stats + submissions | +| `GET /api/analytics/templates/:draftId/export.json` | JSON | Structured JSON template detail | +| `GET /api/analytics/templates/:draftId/submissions/export.csv` | CSV | Flat submissions-only CSV | + +**Response headers:** +``` +Content-Type: text/csv (or application/json) +Content-Disposition: attachment; filename="analytics-overview-1723646400000.csv" +``` + +**Error handling:** +- 404 if template not found (CSV/JSON detail exports) +- 500 for server errors + +--- + +## Frontend + +### AnalyticsPage.jsx — Export Buttons + +Added to the header actions area, next to the date range selector and Sync Data button. + +**New state:** `exporting` (boolean) — disables buttons during download + +**`handleExport(format)`** function: +1. Gets auth token via `getAuthToken()` +2. Fetches `${API_BASE_URL}/api/analytics/export/overview.${format}` +3. Converts response to Blob +4. Creates object URL and temporary `` element +5. Triggers download with filename `analytics-overview-{timestamp}.{format}` +6. Cleans up object URL + +**UI:** Two buttons in `.analytics-export-group`: +- **CSV** button with Download icon +- **JSON** button with Download icon +- Both show spinner (Loader2) when `exporting` is true + +### TemplateDetailAnalytics.jsx — Export Buttons + +Added to the detail header, next to the template title and metadata badges. + +**New state:** `exporting` (boolean) + +**`handleExport(format)`** function: +- Same blob download pattern as AnalyticsPage +- Fetches `${API_BASE_URL}/api/analytics/templates/${draftId}/export.${format}` +- Downloads as `template-{draftId}-{timestamp}.{format}` + +**UI:** Two buttons in `.analytics-export-group` in the detail header: +- **CSV** button with Download icon +- **JSON** button with Download icon + +--- + +## CSS (`src/pages/AnalyticsPage.css`) + +| Class | Description | +|---|---| +| `.analytics-export-group` | Flex container with 6px gap for export buttons | +| `.analytics-export-btn` | Outlined button: 6px 12px padding, 12px font, 600 weight | +| `.analytics-export-btn:hover` | Blue-tinted background and border on hover | +| `.analytics-export-btn:disabled` | 50% opacity, not-allowed cursor | +| `.analytics-detail-header` | Flex space-between for title + export buttons, wraps on mobile | + +--- + +## File Summary + +| File | Change | Lines | +|---|---|---| +| `server/utils/exportUtils.js` | **New** | 166 | +| `server/index.js` | Modified — import + 5 endpoints | +93 | +| `src/pages/AnalyticsPage.jsx` | Modified — import, state, handler, buttons | +37 | +| `src/pages/TemplateDetailAnalytics.jsx` | Modified — import, state, handler, buttons | +54 | +| `src/pages/AnalyticsPage.css` | Modified — export button styles | +40 | + +--- + +## Build Verification + +- Vite production build: **PASS** (2.14s) +- `exportUtils.js` module load: **OK** — all 7 functions exported +- No new npm dependencies added — uses only existing `mongoose` and `analyticsUtils` + +--- + +## Commits + +| # | Message | +|---|---| +| 1 | `feat(export): add exportUtils.js with CSV and JSON export generators` | +| 2 | `feat(api): add 5 analytics export endpoints (CSV + JSON)` | +| 3 | `feat(frontend): add CSV/JSON export buttons to AnalyticsPage and TemplateDetailAnalytics` | +| 4 | `style(export): add CSS for export button group and detail header layout` | diff --git a/server/index.js b/server/index.js index d50ccea..e05e5a0 100644 --- a/server/index.js +++ b/server/index.js @@ -26,6 +26,13 @@ const { getTemplateTypeDistribution, } = require('./utils/analyticsUtils'); const { analyzeTemplateGitHub } = require('./utils/githubAnalyzer'); +const { + exportOverviewCSV, + exportOverviewJSON, + exportTemplateDetailCSV, + exportTemplateDetailJSON, + exportAllSubmissionsCSV, +} = require('./utils/exportUtils'); const { encrypt, decrypt } = require('./utils/crypto'); const { sendFormSubmissionEmail } = require('./utils/emailService'); const { scheduleCampaign, cancelScheduledCampaign, stopAgenda, scheduleDraftActivation, cancelDraftActivation } = require('./scheduler'); @@ -2214,6 +2221,92 @@ app.get('/api/analytics/trends', verifyToken, async (req, res) => { } }); +// ========================================== +// ANALYTICS EXPORT ENDPOINTS +// ========================================== + +/** + * GET /api/analytics/export/overview.csv + * Export overview analytics as CSV + */ +app.get('/api/analytics/export/overview.csv', verifyToken, async (req, res) => { + try { + const csv = await exportOverviewCSV(req.user.uid); + res.setHeader('Content-Type', 'text/csv'); + res.setHeader('Content-Disposition', `attachment; filename="analytics-overview-${Date.now()}.csv"`); + res.send(csv); + } catch (error) { + console.error('Error exporting overview CSV:', error); + res.status(500).json({ error: 'Failed to export CSV' }); + } +}); + +/** + * GET /api/analytics/export/overview.json + * Export overview analytics as JSON + */ +app.get('/api/analytics/export/overview.json', verifyToken, async (req, res) => { + try { + const json = await exportOverviewJSON(req.user.uid); + res.setHeader('Content-Type', 'application/json'); + res.setHeader('Content-Disposition', `attachment; filename="analytics-overview-${Date.now()}.json"`); + res.send(json); + } catch (error) { + console.error('Error exporting overview JSON:', error); + res.status(500).json({ error: 'Failed to export JSON' }); + } +}); + +/** + * GET /api/analytics/templates/:draftId/export.csv + * Export template detail analytics as CSV + */ +app.get('/api/analytics/templates/:draftId/export.csv', verifyToken, async (req, res) => { + try { + const csv = await exportTemplateDetailCSV(req.user.uid, req.params.draftId); + if (!csv) return res.status(404).json({ error: 'Template not found' }); + res.setHeader('Content-Type', 'text/csv'); + res.setHeader('Content-Disposition', `attachment; filename="template-${req.params.draftId}-${Date.now()}.csv"`); + res.send(csv); + } catch (error) { + console.error('Error exporting template CSV:', error); + res.status(500).json({ error: 'Failed to export CSV' }); + } +}); + +/** + * GET /api/analytics/templates/:draftId/export.json + * Export template detail analytics as JSON + */ +app.get('/api/analytics/templates/:draftId/export.json', verifyToken, async (req, res) => { + try { + const json = await exportTemplateDetailJSON(req.user.uid, req.params.draftId); + if (!json) return res.status(404).json({ error: 'Template not found' }); + res.setHeader('Content-Type', 'application/json'); + res.setHeader('Content-Disposition', `attachment; filename="template-${req.params.draftId}-${Date.now()}.json"`); + res.send(json); + } catch (error) { + console.error('Error exporting template JSON:', error); + res.status(500).json({ error: 'Failed to export JSON' }); + } +}); + +/** + * GET /api/analytics/templates/:draftId/submissions/export.csv + * Export all submissions for a template as CSV + */ +app.get('/api/analytics/templates/:draftId/submissions/export.csv', verifyToken, async (req, res) => { + try { + const csv = await exportAllSubmissionsCSV(req.user.uid, req.params.draftId); + res.setHeader('Content-Type', 'text/csv'); + res.setHeader('Content-Disposition', `attachment; filename="submissions-${req.params.draftId}-${Date.now()}.csv"`); + res.send(csv); + } catch (error) { + console.error('Error exporting submissions CSV:', error); + res.status(500).json({ error: 'Failed to export submissions' }); + } +}); + if (require.main === module) { app.listen(PORT, "0.0.0.0", async () => { await mongoConnectPromise; diff --git a/server/utils/exportUtils.js b/server/utils/exportUtils.js new file mode 100644 index 0000000..8bffddd --- /dev/null +++ b/server/utils/exportUtils.js @@ -0,0 +1,166 @@ +const TemplateData = require('../models/TemplateData'); +const TemplateSubmission = require('../models/TemplateSubmission'); +const { + getOverviewStats, + getTemplatesWithStats, + getTemplateDetail, + getTemplateSubmissions, + getSubmissionTrends, + getTemplateTypeDistribution, +} = require('./analyticsUtils'); + +function escapeCSV(value) { + if (value === null || value === undefined) return ''; + const str = String(value); + if (str.includes(',') || str.includes('"') || str.includes('\n') || str.includes('\r')) { + return `"${str.replace(/"/g, '""')}"`; + } + return str; +} + +function arrayToCSV(rows) { + if (!rows || rows.length === 0) return ''; + const headers = Object.keys(rows[0]); + const headerLine = headers.map(escapeCSV).join(','); + const dataLines = rows.map(row => + headers.map(h => escapeCSV(row[h])).join(',') + ); + return [headerLine, ...dataLines].join('\n'); +} + +async function exportOverviewCSV(ownerUid) { + const stats = await getOverviewStats(ownerUid); + const templates = await getTemplatesWithStats(ownerUid); + const trends = await getSubmissionTrends(ownerUid, 30); + const typeDist = await getTemplateTypeDistribution(ownerUid); + + const sections = []; + + sections.push('# Overview KPIs'); + sections.push(arrayToCSV([{ + TotalTemplates: stats.totalTemplates, + ActiveLinks: stats.activeLinks, + TotalSubmissions: stats.totalSubmissions, + AvgFieldsPerTemplate: stats.avgFieldsPerTemplate, + }])); + + sections.push('\n# Templates'); + sections.push(arrayToCSV(templates)); + + sections.push('\n# Submission Trends (Last 30 Days)'); + sections.push(arrayToCSV(trends)); + + sections.push('\n# Template Type Distribution'); + sections.push(arrayToCSV(typeDist)); + + return sections.join('\n'); +} + +async function exportOverviewJSON(ownerUid) { + const [stats, templates, trends, typeDist] = await Promise.all([ + getOverviewStats(ownerUid), + getTemplatesWithStats(ownerUid), + getSubmissionTrends(ownerUid, 30), + getTemplateTypeDistribution(ownerUid), + ]); + + return JSON.stringify({ + generatedAt: new Date().toISOString(), + overview: stats, + templates, + trends, + typeDistribution: typeDist, + }, null, 2); +} + +async function exportTemplateDetailCSV(ownerUid, draftId) { + const detail = await getTemplateDetail(ownerUid, draftId); + if (!detail) return null; + + const submissions = await getTemplateSubmissions(ownerUid, draftId, 1, 10000); + const sections = []; + + sections.push('# Template Info'); + sections.push(arrayToCSV([{ + Title: detail.title, + Type: detail.templateType, + Status: detail.status, + TotalSubmissions: detail.totalSubmissions, + EnabledFields: detail.enabledFields.join('; '), + CreatedAt: detail.createdAt, + ExpiresAt: detail.expiresAt || 'N/A', + }])); + + sections.push('\n# Field Statistics'); + const fieldRows = detail.enabledFields.map(field => ({ + Field: field, + Filled: detail.fieldStats[field]?.totalFilled || 0, + CompletionRate: `${detail.fieldStats[field]?.completionRate || 0}%`, + Average: detail.fieldStats[field]?.avg || 'N/A', + Min: detail.fieldStats[field]?.min || 'N/A', + Max: detail.fieldStats[field]?.max || 'N/A', + })); + sections.push(arrayToCSV(fieldRows)); + + sections.push('\n# Raw Submissions'); + if (submissions.submissions.length > 0) { + const flatSubs = submissions.submissions.map(s => { + const row = { SubmissionId: s.submissionId, SubmittedAt: s.submittedAt }; + const data = s.submittedData || {}; + Object.keys(data).forEach(key => { row[key] = data[key]; }); + return row; + }); + sections.push(arrayToCSV(flatSubs)); + } else { + sections.push('No submissions'); + } + + return sections.join('\n'); +} + +async function exportTemplateDetailJSON(ownerUid, draftId) { + const detail = await getTemplateDetail(ownerUid, draftId); + if (!detail) return null; + + const submissions = await getTemplateSubmissions(ownerUid, draftId, 1, 10000); + + return JSON.stringify({ + generatedAt: new Date().toISOString(), + template: { + draftId: detail.draftId, + title: detail.title, + templateType: detail.templateType, + status: detail.status, + createdAt: detail.createdAt, + expiresAt: detail.expiresAt, + totalSubmissions: detail.totalSubmissions, + enabledFields: detail.enabledFields, + fieldStats: detail.fieldStats, + }, + submissions: submissions.submissions, + }, null, 2); +} + +async function exportAllSubmissionsCSV(ownerUid, draftId) { + const result = await getTemplateSubmissions(ownerUid, draftId, 1, 10000); + if (result.submissions.length === 0) return 'No submissions found'; + + const flatSubs = result.submissions.map(s => { + const row = { SubmissionId: s.submissionId, SubmittedAt: s.submittedAt }; + const data = s.submittedData || {}; + Object.keys(data).forEach(key => { row[key] = data[key]; }); + return row; + }); + + return arrayToCSV(flatSubs); +} + +module.exports = { + exportOverviewCSV, + exportOverviewJSON, + exportTemplateDetailCSV, + exportTemplateDetailJSON, + exportAllSubmissionsCSV, + arrayToCSV, + escapeCSV, +}; diff --git a/src/pages/AnalyticsPage.css b/src/pages/AnalyticsPage.css index b9ae0fa..2d81ae0 100644 --- a/src/pages/AnalyticsPage.css +++ b/src/pages/AnalyticsPage.css @@ -582,3 +582,43 @@ .analytics-type-badge.data { background: rgba(34, 197, 94, 0.15); color: #22C55E; } .analytics-type-badge.mobile { background: rgba(99, 102, 241, 0.15); color: #6366F1; } .analytics-type-badge.unknown { background: rgba(128, 128, 128, 0.15); color: #888; } + +/* ===== Export Buttons ===== */ +.analytics-export-group { + display: flex; + gap: 6px; +} + +.analytics-export-btn { + display: inline-flex; + align-items: center; + gap: 5px; + padding: 6px 12px; + border: 1px solid rgba(255, 255, 255, 0.12); + border-radius: 6px; + background: rgba(255, 255, 255, 0.04); + color: var(--text-secondary, #a8b3cf); + font-size: 12px; + font-weight: 600; + cursor: pointer; + transition: all 0.15s ease; +} + +.analytics-export-btn:hover:not(:disabled) { + background: rgba(59, 130, 246, 0.12); + border-color: rgba(59, 130, 246, 0.3); + color: #3B82F6; +} + +.analytics-export-btn:disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.analytics-detail-header { + display: flex; + justify-content: space-between; + align-items: flex-start; + gap: 16px; + flex-wrap: wrap; +} diff --git a/src/pages/AnalyticsPage.jsx b/src/pages/AnalyticsPage.jsx index 15ed836..49f85b7 100644 --- a/src/pages/AnalyticsPage.jsx +++ b/src/pages/AnalyticsPage.jsx @@ -1,7 +1,7 @@ import React, { useState, useEffect, useCallback } from 'react'; import { BarChart3, TrendingUp, Users, FileText, RefreshCw, ChevronDown, - Loader2, AlertCircle, ArrowLeft, Clock, CheckCircle2, Eye, + Loader2, AlertCircle, ArrowLeft, Clock, CheckCircle2, Eye, Download, } from 'lucide-react'; import { ResponsiveContainer, AreaChart, Area, XAxis, YAxis, CartesianGrid, Tooltip, @@ -110,6 +110,33 @@ export default function AnalyticsPage({ user }) { } }; + const [exporting, setExporting] = useState(false); + + const handleExport = async (format) => { + setExporting(true); + try { + const token = await getAuthToken(); + if (!token) return; + const res = await fetch(`${API_BASE_URL}/api/analytics/export/overview.${format}`, { + headers: { Authorization: `Bearer ${token}` }, + }); + if (!res.ok) throw new Error('Export failed'); + const blob = await res.blob(); + const url = window.URL.createObjectURL(blob); + const a = document.createElement('a'); + a.href = url; + a.download = `analytics-overview-${Date.now()}.${format}`; + document.body.appendChild(a); + a.click(); + document.body.removeChild(a); + window.URL.revokeObjectURL(url); + } catch (err) { + setError(err.message); + } finally { + setExporting(false); + } + }; + if (selectedDraftId) { return ( : } {syncing ? 'Syncing...' : 'Sync Data'} +
+ + +
diff --git a/src/pages/TemplateDetailAnalytics.jsx b/src/pages/TemplateDetailAnalytics.jsx index 6cd3eb8..90e4b68 100644 --- a/src/pages/TemplateDetailAnalytics.jsx +++ b/src/pages/TemplateDetailAnalytics.jsx @@ -1,7 +1,7 @@ import React, { useState, useEffect, useCallback } from 'react'; import { ArrowLeft, Loader2, AlertCircle, FileText, Users, Clock, - BarChart3, ChevronLeft, ChevronRight, GitBranch, Star, + BarChart3, ChevronLeft, ChevronRight, GitBranch, Star, Download, } from 'lucide-react'; import { ResponsiveContainer, BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip, @@ -25,6 +25,7 @@ export default function TemplateDetailAnalytics({ draftId, onBack }) { const [githubData, setGithubData] = useState(null); const [githubLoading, setGithubLoading] = useState(false); const [githubError, setGithubError] = useState(null); + const [exporting, setExporting] = useState(false); const fetchDetail = useCallback(async () => { try { @@ -60,6 +61,31 @@ export default function TemplateDetailAnalytics({ draftId, onBack }) { } }, [draftId]); + const handleExport = useCallback(async (format) => { + setExporting(true); + try { + const token = await getAuthToken(); + if (!token) return; + const res = await fetch(`${API_BASE_URL}/api/analytics/templates/${draftId}/export.${format}`, { + headers: { Authorization: `Bearer ${token}` }, + }); + if (!res.ok) throw new Error('Export failed'); + const blob = await res.blob(); + const url = window.URL.createObjectURL(blob); + const a = document.createElement('a'); + a.href = url; + a.download = `template-${draftId}-${Date.now()}.${format}`; + document.body.appendChild(a); + a.click(); + document.body.removeChild(a); + window.URL.revokeObjectURL(url); + } catch (err) { + console.error('Export error:', err); + } finally { + setExporting(false); + } + }, [draftId]); + const fetchGithubAnalytics = useCallback(async () => { setGithubLoading(true); setGithubError(null); @@ -154,6 +180,24 @@ export default function TemplateDetailAnalytics({ draftId, onBack }) { +
+ + +
{/* KPI Cards for field stats */}