Skip to content

feat: add @ottabase/ottaport data import/export package with admin UI#135

Draft
Copilot wants to merge 12 commits intomainfrom
copilot/add-data-import-export-package
Draft

feat: add @ottabase/ottaport data import/export package with admin UI#135
Copilot wants to merge 12 commits intomainfrom
copilot/add-data-import-export-package

Conversation

Copy link
Copy Markdown

Copilot AI commented Feb 28, 2026

New package for bulk data import/export across OttaORM models — CSV/JSON/TSV parsing, field mapping with auto-match, batched upserts on a unique field, filtered exports, and job history tracking.

@ottabase/ottaport package

  • Parsers — Edge-runtime CSV/TSV (character-by-character state machine, no ReDoS) and JSON array parser with heterogeneous object support
  • Import handler — Batched find-then-create/update keyed on a configurable unique field (e.g. email). Returns per-batch and aggregate stats
  • Export handler — Query model with optional search, date range, field filters → format as CSV/JSON/TSV download
  • PortJob model — Fat model tracking import/export history (direction, status, row counts, filename, R2 key, user, duration)
  • portJobsTable schema — SQLite table for job metadata, always-on via CORE_PACKAGES

OttaORM enhancement

Added BaseModel.bulkUpsert(records, uniqueField, options?) — batched find-then-create/update returning { created, updated, failed, errors }.

TanStack app wiring

  • 6 admin API endpoints under /api/admin/ottaport/* (models list, import parse, import execute, export download, export preview, job history)
  • All endpoints gated by requireAdminAccess({ scope: 'system' })
  • Filename sanitization on R2 keys to prevent path traversal
  • Field mapping validation against actual model table columns
  • AuditLog integration for all import/export operations

Admin UI (/admin/ottaport)

  • Import Wizard — drag-drop upload → auto-map columns to model fields → review config → execute with progress → results with error detail
  • Export Page — model picker, format selector, search + date range filters, server-side paginated preview table, blob download
  • History Page — filterable job log with status badges and pagination
// Server-side usage
import { processImport, processExport } from '@ottabase/ottaport/server';

const result = await processImport(parsedRows, {
  modelEntity: 'users',
  fieldMappings: [{ sourceColumn: 'Email', targetField: 'email' }],
  uniqueField: 'email',
  batchSize: 50,
});
// { status: 'completed', totalCreated: 95, totalUpdated: 5, totalFailed: 0 }

Config migrations refactor

Extracted CORE_PACKAGES array and isCorePackage() helper to DRY the three places that check for always-enabled packages (brandEngine, ottaport).

Security fix

Updated storybook catalog entries from ^10.2.3^10.2.10 (resolves to 10.2.13) to patch WebSocket Hijacking CVE.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 11 commits February 28, 2026 22:52
Create new package for CSV/JSON/TSV parsing, field mapping, bulk
upserts, and export with filters for OttaORM models.

- CSV/TSV parser with quoted field support
- JSON array parser with heterogeneous object support
- Batched import handler with upsert logic
- Export handler with date range, search, and field filters
- PortJob model for tracking import/export job history
- Format output as CSV, TSV, or JSON
- 30 tests covering parsers and import handler

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…andlers, and bulkUpsert on BaseModel

Co-authored-by: thinkdj <688055+thinkdj@users.noreply.github.com>
…tration, and API routes

Co-authored-by: thinkdj <688055+thinkdj@users.noreply.github.com>
Create admin pages for the @ottabase/ottaport data import/export feature:
- AdminOttaportPage: Main hub with Import/Export/History tabs
- OttaportImportWizard: Multi-step wizard (upload → map → preview → execute)
- OttaportExportPage: Model selection, filters, preview, and download
- OttaportHistoryPage: Import/export job history with filtering
- Barrel export index.ts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use Date.parse() for string values instead of redundant passthrough.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ter integration

Co-authored-by: thinkdj <688055+thinkdj@users.noreply.github.com>
…ypes

Co-authored-by: thinkdj <688055+thinkdj@users.noreply.github.com>
…SIZE constant, fix inline styles

Co-authored-by: thinkdj <688055+thinkdj@users.noreply.github.com>
…inst model, encode Content-Disposition

Co-authored-by: thinkdj <688055+thinkdj@users.noreply.github.com>
… Hijacking CVE

Co-authored-by: thinkdj <688055+thinkdj@users.noreply.github.com>
Copilot AI changed the title [WIP] Add new package for data import/export functionality feat: add @ottabase/ottaport data import/export package with admin UI Feb 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants