Skip to content

feat: BROWSE per-cell validation by declared column type (#45) #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DDecoene merged 1 commit into from
Jul 9, 2026
Merged

feat: BROWSE per-cell validation by declared column type (#45) #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,25 @@ Versions follow [Semantic Versioning](https://semver.org/) — minor bump per su
week containing the year's first Thursday. Early-January dates correctly report the
previous year's week 52/53, and late-December dates week 1 of the next year. Accepts
ISO `YYYY-MM-DD` or `MM/DD/YY`; invalid input returns 0. (#44)
- `BROWSE` now validates each cell edit against its column's declared type before
committing. An invalid edit keeps the cell in edit mode, outlines it in red and shows
why (`HH:MM`, `multiple of 15`, `at most 2 decimal place(s)`, `not a real date`, …);
the error clears as soon as the value becomes valid. `DATE`, `TIME`/`TIME(n)`,
`NUM(p,s)`, `INT` and `LOGICAL` are checked; `CHAR`/`MEMO` stay unconstrained. The rules
live in `src/shared/cellValidation.ts` and run on both the client (instant feedback) and
the server (`grid-edit` is now validated authoritatively — previously it wrote straight
to SQLite with no check at all). (#45)

### Fixed
- `CREATE TABLE t (price NUM(8,2))` silently created a **phantom column named `2`** of
type `)`: the parser read the precision, then treated the scale as the next column
definition. The shipped `PRODUCTS`, `DEALS` and `SALES` demo tables all carried this
stray column. `NUM(p,s)` now parses correctly. Tables created before this fix keep the
stray column until recreated. (#45)
- Column type metadata is now scoped per database. Two databases holding same-named
tables previously shared (and overwrote) one another's declared column types, so a
`TIME(15)` column in one database could be validated against another database's
`CHAR(20)` declaration of the same name. (#45)

---

Expand Down
35 changes: 28 additions & 7 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ server/
ServerDatabaseBridge.ts IDatabaseBridge impl wrapping better-sqlite3
ProgramStore.ts .prg program storage in data/system.sqlite3
IndexStore.ts Index metadata + active index in data/system.sqlite3
ColumnMetaStore.ts Declared column types (TIME, TIME(n)) in data/system.sqlite3 — SQLite affinity can't distinguish TIME from CHAR/DATE
ColumnMetaStore.ts Declared column types per (db, table, column) in data/system.sqlite3 — SQLite affinity can't distinguish TIME/DATE/CHAR, LOGICAL/INT, or recover NUM(p,s)
ReportStore.ts Report definition storage in data/system.sqlite3 (reports table)
ReportRunner.ts ASCII and HTML report rendering, group breaks, subtotals, grand totals
DemoSeeder.ts Seeds demos/*.prg into the program store and demos/reports/*.json into the report store at startup (demos win)
Expand All @@ -68,7 +68,7 @@ src/
Terminal.ts REPL UI — command history, multi-line block accumulation

ui/
Grid.ts BROWSE spreadsheet — inline cell editing, keyboard nav
Grid.ts BROWSE spreadsheet — inline cell editing with per-column type validation, keyboard nav
FormLayout.ts @ SAY GET form engine — character-cell coordinates
ProgramEditor.ts .prg source editor UI
ReportPreview.ts iframe-based HTML report preview panel (Esc to close, Ctrl+P to print)
Expand All @@ -81,7 +81,8 @@ src/
WsClient.ts Browser WebSocket client — sends commands, receives messages

shared/
types.ts Shared TS types (IDatabaseBridge, IIndexStore, WS message shapes)
types.ts Shared TS types (IDatabaseBridge, IIndexStore, IColumnMetaStore, WS message shapes)
cellValidation.ts Declared-type cell validation, shared by Grid.ts (inline UX) and Session (authoritative)

main.ts Boot: connect WS → wire terminal/grid/form/editor

Expand Down Expand Up @@ -111,6 +112,9 @@ tests/
ProgramStore.test.ts
AlterTable.test.ts ALTER TABLE + MODIFY STRUCTURE integration tests
TimeType.test.ts TIME / TIME(n) columns — creation, structure, write validation
ColumnMeta.test.ts NUM(p,s) parsing, declared types in LIST STRUCTURE, grid-open columnTypes, server-side grid-edit validation
ColumnMetaStore.test.ts Per-(db,table,column) type metadata + legacy-schema migration
CellValidation.test.ts Shared per-type cell validation rules
Print.test.ts `?` / `??` print command
Aggregate.test.ts `SUM` / `AVERAGE`
Builtins.test.ts / BuiltinsParse.test.ts built-in functions (direct + through the parser)
Expand Down Expand Up @@ -181,7 +185,24 @@ WebBase-III supports **unlimited work areas** (no DOS 10-area limit). Cross-area
| `DATE` | | `TEXT` |
| `TIME` / `TIME(n)` | | `TEXT` |

`TIME` stores `HH:MM` (24-hour). The optional `TIME(n)` qualifier (only via `CREATE TABLE` — not yet carried through `ALTER TABLE`) requires minutes to be a multiple of `n`, e.g. `TIME(15)` only accepts `:00`/`:15`/`:30`/`:45`. `APPEND RECORD` leaves new fields `NULL` (unvalidated); `REPLACE ... WITH` rejects a malformed or off-granularity `TIME` value with `** Error: ...` and does not write it. `LIST STRUCTURE` prints the declared type (`TIME`, `TIME(15)`) rather than the raw SQLite storage class.
`TIME` stores `HH:MM` (24-hour). The optional `TIME(n)` qualifier (only via `CREATE TABLE` — not carried through `ALTER TABLE`) requires minutes to be a multiple of `n`, e.g. `TIME(15)` only accepts `:00`/`:15`/`:30`/`:45`. `APPEND RECORD` leaves new fields `NULL` (unvalidated); `REPLACE ... WITH` rejects a malformed or off-granularity `TIME` value with `** Error: ...` and does not write it. `LIST STRUCTURE` prints the declared type (`TIME`, `NUM(8,2)`, `TIME(15)`) rather than the raw SQLite storage class.

Declared types are recorded per `(database, table, column)` in `server/ColumnMetaStore.ts`, because SQLite only keeps a storage affinity: `TIME`/`DATE`/`CHAR` are all `TEXT`, `LOGICAL`/`INT` are both `INTEGER`, and a `NUM(p,s)` qualifier is lost entirely.

#### Cell validation (`BROWSE`)

`src/shared/cellValidation.ts` holds the rules and runs on **both** sides: `Grid.ts` checks before commit (an invalid edit keeps the cell in edit mode, outlined red, with the reason shown; the error clears as the value becomes valid), and `Session`'s `grid-edit` handler re-checks authoritatively before writing — a WS message can reach the server without passing through the grid.

| Type | Accepted |
|---|---|
| `DATE` | `YYYY-MM-DD`, a real calendar date (rejects `2023-02-29`) |
| `TIME` / `TIME(n)` | `HH:MM`; minutes a multiple of `n` when set |
| `NUM(p,s)` | numeric; ≤ `s` decimals, ≤ `p - s` integer digits |
| `INT` | a whole number |
| `LOGICAL` | `.T.`/`.F.`/`.TRUE.`/`.FALSE.`/`T`/`F`/`TRUE`/`FALSE`/`1`/`0` |
| `CHAR` / `MEMO` | anything (length is not enforced) |

An empty value is always allowed (clears the cell). Columns with no recorded declared type are unconstrained. `REPLACE` enforces only `TIME` — widening it would change the semantics of existing programs.

### Indexing & search
| Command | What it does |
Expand Down Expand Up @@ -293,7 +314,7 @@ line.
- ~~`TIME` column type~~ — `TIME`/`TIME(n)` columns storing `HH:MM`, with a minute-granularity
qualifier validated on write; declared types tracked in `server/ColumnMetaStore.ts` (#43) ✅
- ~~`WEEK()` built-in~~ — ISO-8601 week number (#44) ✅
- BROWSE per-cell validation — grid rejects invalid edits per column type (#45)
- ~~BROWSE per-cell validation~~ — grid rejects invalid edits per column type, validated on both client and server via `src/shared/cellValidation.ts` (#45) ✅
- `demos/overtime.prg` — overtime tracker showcasing all three of the above (#46)

## Boolean literals
Expand All @@ -303,11 +324,11 @@ Both styles accepted: `TRUE`/`FALSE` and `.T.`/`.TRUE.`/`.F.`/`.FALSE.` (dBASE I
## Testing

```bash
npm test # Vitest unit + integration (281 tests)
npm test # Vitest unit + integration (316 tests)
npx playwright test # E2E browser tests — requires dev server on :5173/:3000
```

Playwright suites (75 tests): `tests/integration.spec.ts` (20 tests — full REPL scenario), `tests/assistant.spec.ts` (21 tests — sidebar, wizards, report designer, MODIFY STRUCTURE round-trip, `TIME(15)` column + REPLACE validation, program run, CSV/SORT/SUM-AVERAGE/REINDEX/PACK actions, demo launchers), `tests/inventory.spec.ts` (8 tests — INVENTORY.prg menu + valuation/low-stock report/sort/CSV/JOIN), `tests/crm.spec.ts` (6 tests — CRM demo menu, pipeline summary, sort, report, CSV, JOIN), `tests/parity-commands.spec.ts` (5 tests — `?`/`??`, built-in functions, `WEEK()`, `SUM`/`AVERAGE`, `SORT ON … TO`), `tests/multiarea.spec.ts` (4 tests — multi-work-area, relations, alias.field), `tests/demos.spec.ts` (4 tests — demo program + report seeding), `tests/copycsv.spec.ts` (2 tests — COPY TO download + APPEND FROM upload), `tests/splash.spec.ts` (2 tests — version banner + demo discoverability), `tests/join.spec.ts` (1 test — JOIN materialization), `tests/propagation.spec.ts` (1 test — live multiuser refresh), `tests/program-side-effects.spec.ts` (1 test — CSV/report side-effects fire from inside a program block).
Playwright suites (79 tests): `tests/assistant.spec.ts` (22 tests — sidebar, wizards, report designer, MODIFY STRUCTURE round-trip, `TIME(15)` column + REPLACE validation, Browse-action grid validation, program run, CSV/SORT/SUM-AVERAGE/REINDEX/PACK actions, demo launchers), `tests/integration.spec.ts` (20 tests — full REPL scenario), `tests/inventory.spec.ts` (8 tests — INVENTORY.prg menu + valuation/low-stock report/sort/CSV/JOIN), `tests/crm.spec.ts` (6 tests — CRM demo menu, pipeline summary, sort, report, CSV, JOIN), `tests/parity-commands.spec.ts` (5 tests — `?`/`??`, built-in functions, `WEEK()`, `SUM`/`AVERAGE`, `SORT ON … TO`), `tests/multiarea.spec.ts` (4 tests — multi-work-area, relations, alias.field), `tests/demos.spec.ts` (4 tests — demo program + report seeding), `tests/grid-validation.spec.ts` (3 tests — BROWSE per-cell validation: TIME(15), NUM(p,s)/DATE, Esc abandons), `tests/copycsv.spec.ts` (2 tests — COPY TO download + APPEND FROM upload), `tests/splash.spec.ts` (2 tests — version banner + demo discoverability), `tests/join.spec.ts` (1 test — JOIN materialization), `tests/propagation.spec.ts` (1 test — live multiuser refresh), `tests/program-side-effects.spec.ts` (1 test — CSV/report side-effects fire from inside a program block).

## Definition of done

Expand Down
4 changes: 3 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ WebBase-III supports **unlimited work areas** — each independently holding a t

> Column ops that can invalidate an index (DROP, RENAME, ALTER type) drop all of the table's indexes and warn you to rebuild with `INDEX ON`.

**Column types**: `CHAR(n)` (aliases `CHARACTER`/`VARCHAR`/`STRING`/`MEMO`), `NUM` (`NUMERIC`/`FLOAT`/`DOUBLE`/`DECIMAL`), `INT`/`INTEGER`, `LOGICAL`/`BOOLEAN`, `DATE`, and `TIME`/`TIME(n)`. `TIME` stores `HH:MM` (24-hour); the optional `TIME(n)` qualifier (e.g. `TIME(15)`) requires minutes to be a multiple of `n`. `REPLACE ... WITH` rejects a malformed or off-granularity `TIME` value instead of silently coercing it.
**Column types**: `CHAR(n)` (aliases `CHARACTER`/`VARCHAR`/`STRING`/`MEMO`), `NUM`/`NUM(p,s)` (`NUMERIC`/`FLOAT`/`DOUBLE`/`DECIMAL`), `INT`/`INTEGER`, `LOGICAL`/`BOOLEAN`, `DATE`, and `TIME`/`TIME(n)`. `TIME` stores `HH:MM` (24-hour); the optional `TIME(n)` qualifier (e.g. `TIME(15)`) requires minutes to be a multiple of `n`. `REPLACE ... WITH` rejects a malformed or off-granularity `TIME` value instead of silently coercing it, and `LIST STRUCTURE` prints the declared type (`NUM(8,2)`, `TIME(15)`) rather than SQLite's storage class.

> **CSV format (`COPY TO` / `APPEND FROM`):** Unlike dBASE III's headerless,
> positional `DELIMITED`/`SDF` formats, WebBase-III uses modern **header-based CSV**
Expand Down Expand Up @@ -380,6 +380,8 @@ Logical operators are accepted in both styles too: `NOT` / `.NOT.`, `AND` / `.AN
| F5 | Refresh from DB |
| Esc | Exit grid, return to terminal |

> **Cell validation.** An edit is checked against the column's declared type before it commits. An invalid value keeps the cell in edit mode, outlined in red, with the reason shown (`HH:MM`, `multiple of 15`, `at most 2 decimal place(s)`, `not a real date`); the error clears as soon as you fix it, and `Esc` abandons the edit. `DATE`, `TIME`/`TIME(n)`, `NUM(p,s)`, `INT` and `LOGICAL` are validated; `CHAR`/`MEMO` are not. The server re-checks every edit independently.

---

## Architecture
Expand Down
72 changes: 52 additions & 20 deletions server/ColumnMetaStore.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,16 @@ import type { IColumnMetaStore, ColumnTypeInfo } from '../src/shared/types.js';
const DATA_DIR = path.join(process.cwd(), 'data');
const DB_PATH = path.join(DATA_DIR, 'system.sqlite3');

/**
* Declared column types, keyed by (database, table, column).
*
* SQLite only records a storage affinity (TEXT/REAL/INTEGER), which cannot tell
* TIME from DATE from CHAR, LOGICAL from INT, or recover a NUM(p,s) qualifier.
* The grid and REPLACE need the declared type to validate writes.
*
* Scoping by database matters: two databases may each hold a table of the same
* name with different column types.
*/
export class ColumnMetaStore implements IColumnMetaStore {
private db: Database.Database;

Expand All @@ -15,51 +25,73 @@ export class ColumnMetaStore implements IColumnMetaStore {
this.db.pragma('journal_mode = WAL');
this.db.exec(`
CREATE TABLE IF NOT EXISTS column_types (
db_name TEXT NOT NULL,
table_name TEXT NOT NULL,
col_name TEXT NOT NULL,
base_type TEXT NOT NULL,
qualifier INTEGER,
PRIMARY KEY (table_name, col_name)
scale INTEGER,
PRIMARY KEY (db_name, table_name, col_name)
);
`);
// v1.2.0 dev migration: the first cut of this table (#43) had neither db_name
// nor scale. The rows only cache what CREATE TABLE re-records, so rebuilding
// is cheaper and safer than back-filling an unscoped key.
const cols = this.db.prepare('PRAGMA table_info(column_types)').all() as { name: string }[];
if (!cols.some(c => c.name === 'db_name') || !cols.some(c => c.name === 'scale')) {
this.db.exec(`
DROP TABLE column_types;
CREATE TABLE column_types (
db_name TEXT NOT NULL,
table_name TEXT NOT NULL,
col_name TEXT NOT NULL,
base_type TEXT NOT NULL,
qualifier INTEGER,
scale INTEGER,
PRIMARY KEY (db_name, table_name, col_name)
);
`);
}
}

setColumnType(tableName: string, colName: string, baseType: string, qualifier: number | null): void {
setColumnType(dbName: string, tableName: string, colName: string, baseType: string, qualifier: number | null, scale: number | null): void {
this.db.prepare(`
INSERT INTO column_types (table_name, col_name, base_type, qualifier)
VALUES (?, ?, ?, ?)
ON CONFLICT(table_name, col_name) DO UPDATE SET base_type = excluded.base_type, qualifier = excluded.qualifier
`).run(tableName, colName, baseType, qualifier);
INSERT INTO column_types (db_name, table_name, col_name, base_type, qualifier, scale)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(db_name, table_name, col_name) DO UPDATE SET
base_type = excluded.base_type, qualifier = excluded.qualifier, scale = excluded.scale
`).run(dbName, tableName, colName, baseType, qualifier, scale);
}

getColumnType(tableName: string, colName: string): ColumnTypeInfo | null {
getColumnType(dbName: string, tableName: string, colName: string): ColumnTypeInfo | null {
const row = this.db.prepare(
'SELECT base_type AS baseType, qualifier FROM column_types WHERE table_name = ? AND col_name = ?'
).get(tableName, colName) as ColumnTypeInfo | undefined;
'SELECT base_type AS baseType, qualifier, scale FROM column_types WHERE db_name = ? AND table_name = ? AND col_name = ?'
).get(dbName, tableName, colName) as ColumnTypeInfo | undefined;
return row ?? null;
}

listColumnTypes(tableName: string): Record<string, ColumnTypeInfo> {
listColumnTypes(dbName: string, tableName: string): Record<string, ColumnTypeInfo> {
const rows = this.db.prepare(
'SELECT col_name AS colName, base_type AS baseType, qualifier FROM column_types WHERE table_name = ?'
).all(tableName) as Array<ColumnTypeInfo & { colName: string }>;
'SELECT col_name AS colName, base_type AS baseType, qualifier, scale FROM column_types WHERE db_name = ? AND table_name = ?'
).all(dbName, tableName) as Array<ColumnTypeInfo & { colName: string }>;
const out: Record<string, ColumnTypeInfo> = {};
for (const r of rows) out[r.colName] = { baseType: r.baseType, qualifier: r.qualifier };
for (const r of rows) out[r.colName] = { baseType: r.baseType, qualifier: r.qualifier, scale: r.scale };
return out;
}

renameColumn(tableName: string, oldName: string, newName: string): void {
renameColumn(dbName: string, tableName: string, oldName: string, newName: string): void {
this.db.prepare(
'UPDATE column_types SET col_name = ? WHERE table_name = ? AND col_name = ?'
).run(newName, tableName, oldName);
'UPDATE column_types SET col_name = ? WHERE db_name = ? AND table_name = ? AND col_name = ?'
).run(newName, dbName, tableName, oldName);
}

dropColumn(tableName: string, colName: string): void {
this.db.prepare('DELETE FROM column_types WHERE table_name = ? AND col_name = ?').run(tableName, colName);
dropColumn(dbName: string, tableName: string, colName: string): void {
this.db.prepare('DELETE FROM column_types WHERE db_name = ? AND table_name = ? AND col_name = ?')
.run(dbName, tableName, colName);
}

dropTable(tableName: string): void {
this.db.prepare('DELETE FROM column_types WHERE table_name = ?').run(tableName);
dropTable(dbName: string, tableName: string): void {
this.db.prepare('DELETE FROM column_types WHERE db_name = ? AND table_name = ?').run(dbName, tableName);
}
}

Expand Down
13 changes: 12 additions & 1 deletion server/Session.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { programStore } from './ProgramStore.js';
import { reportStore } from './ReportStore.js';
import { indexStore } from './IndexStore.js';
import { columnMetaStore } from './ColumnMetaStore.js';
import { validateCellValue } from '../src/shared/cellValidation.js';
import type { ClientMessage, ServerMessage, ColInfo } from '../src/shared/types.js';

export class Session {
Expand Down Expand Up @@ -67,6 +68,15 @@ export class Session {
const { rowid, col, value } = msg;
const table = this.executor.area.table;
if (table) {
// Authoritative check — the grid validates client-side for fast
// feedback, but a message can reach here without passing through it.
const db = this.executor.area.db ?? '';
const err = validateCellValue(col, value, columnMetaStore.getColumnType(db, table, col));
if (err) {
this.send({ type: 'output', lines: [{ text: `** ${err}`, cls: 'error' }] });
await this.sendGridData();
break;
}
await this.bridge.exec(
`UPDATE ${q(table)} SET ${q(col)} = ? WHERE rowid = ?`,
[value, rowid]
Expand Down Expand Up @@ -346,8 +356,9 @@ export class Session {
return;
}
const columns = await this.bridge.getStructure(area.table);
const columnTypes = columnMetaStore.listColumnTypes(area.db ?? '', area.table);
const rows = await this.executor.getOrderedRowsWithIds(2000);
this.send({ type: 'grid-open', table: area.table, filter: area.filter, columns, rows });
this.send({ type: 'grid-open', table: area.table, filter: area.filter, columns, columnTypes, rows });
}

private sendStatus(): void {
Expand Down
Loading
Loading