Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ The **Schema Change** page works on a saved live connection, not on the diagram

1. **Find the tables (optional).** You do not have to choose any. The catalog (names, columns, keys, estimated rows, sizes) is read in one round trip. The words of your request are matched against table, schema and column names, weighted by how rare each word is in *your* database, and the foreign-key neighbours of the best matches are added. The scope is shown with the reason for each table and you can add or remove tables. It matches names, not meanings: a request that calls `clients` "customers" finds nothing, and the plan then goes ahead with no table in scope. That is how a request for **new tables** works: the model is told the names of the existing tables (not their contents), creates the new ones with `create_table`, and any existing table its foreign keys point at is read afterwards so the keys are checked against that table's real columns. Names already taken in the database are refused.
2. **Give the planner facts.** Only the tables in scope are sent in full (the others by name): columns, keys, existing indexes with how often each was scanned (`pg_stat_user_indexes`), row estimates, data sizes, dependent views, and two aggregate statistics per column (NULL fraction, distinct-value estimate). Never rows, default expressions, index predicates or credentials.
3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check`, `drop_constraint` and `add_primary_key`. An existing integer column can become an identity column (`alter_column` with `autoIncrement`): the sequence is started after the column's largest value in the same transaction, which needs an index on the column unless the table is small, and a serial column's old default is dropped in the same change. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`).
3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check`, `drop_constraint`, `add_primary_key`, `create_enum` and `add_enum_value`. `create_enum` makes a type (a column can use it later in the same plan, with a default that is one of its labels) and `add_enum_value` adds a label, optionally before or after another: neither locks any table, and a label can never be removed again. An existing integer column can become an identity column (`alter_column` with `autoIncrement`): the sequence is started after the column's largest value in the same transaction, which needs an index on the column unless the table is small, and a serial column's old default is dropped in the same change. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`).
4. **Check the database, then run it.** *Check the database first* reads, without writing, who holds or waits for the tables (found through `pg_locks`, so even a role that may not read other roles' sessions sees them), transactions open for 30 seconds or more, strong locks, prepared transactions, replicas that are behind, replication slots holding WAL, vacuum running on the tables, and the timeouts the role runs with; each check says "can't tell" instead of "ok" when it cannot read something. Free disk space cannot be read from a connection, so that is for you to check. *Run it here, one step at a time* then runs each step on its own in a SQL session of its own (no transaction, because `CONCURRENTLY` and the batched backfills need that): only the next step can run, the first failure stops the run, a step can be run again or stopped, and the most recent finished step can be undone with the statement the compiler wrote for it. A step that needs a strong lock on an existing table waits at most 2 seconds per attempt and tries up to 8 times, backing off between tries; if the table never frees up it stops with the sessions holding it and changes nothing. After the last step the catalog is read again and compared, table by table, with the state the compiler said it would leave (columns, types, NOT NULL, keys, unique and check constraints, foreign keys, indexes, views); every difference is listed. Optional clean-up steps (dropping an old column, a compatibility view) are never run for you. The runbook can also be opened in the SQL Editor with the transaction switch off, or copied.
5. **Health.** A read-only report from catalog queries: tables without a primary key, foreign keys without an index, duplicate, redundant, unused and invalid indexes, constraints that were never validated, wide tables, missing or stale statistics, tables with many dead rows, sequences past half their range. A model may be asked to order and explain the findings; it can only point at findings that exist.
6. **Governance.** The model is called through the OpenAI API with your key (see above; GoForge does not store it), an allow-list of model ids, optional redaction of schema, table, column, index and constraint names into placeholders before anything leaves the server (the answer is translated back), and an audit log of every request with exactly what was sent, every plan, every runbook opened and every settings change, exportable as CSV or JSON. The audit log records that a runbook was *opened*; what you run in the SQL Editor is not tracked.
Expand Down
30 changes: 29 additions & 1 deletion scripts/schema-change-pg.ts
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,30 @@ const SCENARIOS: Scenario[] = [
checks.equal((await one(client, "INSERT INTO ledger2 (memo) VALUES ('new') RETURNING id"))?.id, 300001, "the next row gets the number after the largest one");
},
},
{
name: "create an enum, add a column of that type with a default, then add a label after one",
tables: ["public.orders"],
operations: [
{ type: "create_enum", typeName: "status_kind", values: ["new", "open"] },
{ type: "add_column", tableName: "orders", field: { name: "kind", dataType: "status_kind", nullable: false, defaultValue: "'new'" } },
{ type: "add_enum_value", typeName: "status_kind", value: "closed", after: "open" },
],
async verify(client, checks) {
checks.equal((await client.query("SELECT enumlabel AS l FROM pg_enum WHERE enumtypid = 'status_kind'::regtype ORDER BY enumsortorder")).rows.map((row) => row.l), ["new", "open", "closed"], "the labels, in order");
checks.equal(Number((await one(client, "SELECT count(*)::int AS n FROM orders WHERE kind = 'new'"))?.n), 400000, "every existing row reads the default");
await client.query("UPDATE orders SET kind = 'closed' WHERE id = 1");
checks.ok(true, "the added label can be used");
},
},
{
name: "add a label before another to an enum that exists",
setup: "CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');",
tables: [],
operations: [{ type: "add_enum_value", typeName: "mood", value: "meh", before: "ok" }],
async verify(client, checks) {
checks.equal((await client.query("SELECT enumlabel AS l FROM pg_enum WHERE enumtypid = 'mood'::regtype ORDER BY enumsortorder")).rows.map((row) => row.l), ["sad", "meh", "ok", "happy"], "the label is where it was asked to be");
},
},
{
name: "refuses NOT NULL without a default on a table with rows",
tables: ["public.orders"],
Expand Down Expand Up @@ -484,7 +508,7 @@ async function rewritesSection(checks: Checks) {
await client.query(`INSERT INTO ${table} SELECT g, ${sample}${from.startsWith("char(") ? "::" + from : ""} FROM generate_series(1, 100) g`);
const files = async () => (await client.query(`SELECT string_agg(c.relname || '=' || pg_relation_filenode(c.oid)::text, ',' ORDER BY c.relname) AS f FROM pg_class c WHERE c.relname IN ('${table}', '${table}_v')`)).rows[0].f as string;
const before = await files();
const probes = await probe(asRunner(client), { types: [], typeChanges: [{ from, to }], functions: [], defaults: [], checks: [], relations: [], schemas: [] });
const probes = await probe(asRunner(client), { types: [], typeChanges: [{ from, to }], functions: [], defaults: [], checks: [], enums: [], relations: [], schemas: [] });
const cast = probes.casts.get(castKey(from, to));
let verdict = classifyTypeChange(await columnType(client, table), to, cast);
try {
Expand Down Expand Up @@ -1278,6 +1302,7 @@ const sections: Record<string, Section> = {
{ expression: "no_such_function()", type: "text" },
{ expression: "active", type: "text" },
],
enums: [{ schema: "public", name: "mood" }, { schema: "public", name: "no_such_type" }, { schema: "public", name: "orders" }],
checks: [
{ table: "public.orders", expression: "total >= 0 AND status <> 'x'" },
{ table: "public.orders", expression: "no_such_column > 0" },
Expand All @@ -1288,6 +1313,9 @@ const sections: Record<string, Section> = {
});
checks.equal(probes.defaults.get(defaultKey("'new'", "varchar(20)")), undefined, "PostgreSQL accepts a text default");
checks.equal(probes.defaults.get(defaultKey("now()", "timestamptz")), undefined, "and a function default");
checks.equal(probes.enums.get("public.mood"), { exists: true, kind: "e", labels: ["ok", "bad"] }, "an enum type is found with its labels in order");
checks.equal(probes.enums.get("public.no_such_type")?.exists, false, "a type that is not there is not found");
checks.equal(probes.enums.get("public.orders")?.kind, "c", "a table's row type is found, and is not an enum");
checks.equal(probes.checks.get(checkKey("public.orders", "total >= 0 AND status <> 'x'")), undefined, "PostgreSQL accepts a CHECK over the table's columns");
checks.match(probes.checks.get(checkKey("public.orders", "no_such_column > 0")), /column "no_such_column" does not exist/, "rejects a CHECK on a column the table does not have, with its own message");
checks.match(probes.checks.get(checkKey("public.orders", "total + 1")), /must be type boolean/, "and one that is not boolean");
Expand Down
4 changes: 4 additions & 0 deletions server/src/ai/plan-normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,10 @@ const normalizeOperation = (value: unknown): unknown => {
return { type, tableName, constraintName: operation.constraintName ?? operation.name, expression: operation.expression ?? operation.condition ?? operation.check };
case "drop_constraint":
return { type, tableName, constraintName: operation.constraintName ?? operation.name };
case "create_enum":
return { type, typeName: operation.typeName ?? operation.name ?? operation.enumName, values: operation.values ?? operation.labels };
case "add_enum_value":
return { type, typeName: operation.typeName ?? operation.name ?? operation.enumName, value: operation.value ?? operation.label, before: operation.before, after: operation.after };
case "add_primary_key":
return { type, tableName, constraintName: operation.constraintName ?? operation.name, columns: operation.columns ?? (operation.column !== undefined ? [operation.column] : undefined) };
case "add_relationship":
Expand Down
2 changes: 2 additions & 0 deletions server/src/ai/schema-plan.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ function makeOperationSchema<Changes extends z.ZodTypeAny>(changes: Changes) {
z.object({ type: z.literal("add_unique"), tableName: identifier, constraintName: identifier.nullable().optional(), columns: z.array(identifier).min(1).max(32) }).strict(),
z.object({ type: z.literal("add_check"), tableName: identifier, constraintName: identifier.nullable().optional(), expression: z.string().trim().min(1).max(400) }).strict(),
z.object({ type: z.literal("drop_constraint"), tableName: identifier, constraintName: identifier }).strict(),
z.object({ type: z.literal("create_enum"), typeName: identifier, values: z.array(z.string().min(1).max(63)).min(1).max(100) }).strict(),
z.object({ type: z.literal("add_enum_value"), typeName: identifier, value: z.string().min(1).max(63), before: z.string().min(1).max(63).nullable().optional(), after: z.string().min(1).max(63).nullable().optional() }).strict(),
z.object({ type: z.literal("add_primary_key"), tableName: identifier, constraintName: identifier.nullable().optional(), columns: z.array(identifier).min(1).max(32) }).strict(),
]);
}
Expand Down
2 changes: 1 addition & 1 deletion server/src/ai/structured.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ describe("the schemas the model is given", () => {
expect(properties).toBeLessThan(500);
expect(JSON.stringify(schema).length).toBeLessThan(60_000);
const operations = schema.properties.operations.items.anyOf as Json[];
expect(operations.map((operation) => operation.properties.type.const).sort()).toEqual(["add_check", "add_column", "add_index", "add_primary_key", "add_relationship", "add_unique", "alter_column", "create_table", "drop_column", "drop_constraint", "drop_index", "drop_relationship", "drop_table", "rename_table"]);
expect(operations.map((operation) => operation.properties.type.const).sort()).toEqual(["add_check", "add_column", "add_enum_value", "add_index", "add_primary_key", "add_relationship", "add_unique", "alter_column", "create_enum", "create_table", "drop_column", "drop_constraint", "drop_index", "drop_relationship", "drop_table", "rename_table"]);
});

it("the model's plan schema has the same top-level keys as the plan validator", () => {
Expand Down
Loading
Loading