Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ The **Schema Change** page works on a saved live connection, not on the diagram

1. **Find the tables (optional).** You do not have to choose any. The catalog (names, columns, keys, estimated rows, sizes) is read in one round trip. The words of your request are matched against table, schema and column names, weighted by how rare each word is in *your* database, and the foreign-key neighbours of the best matches are added. The scope is shown with the reason for each table and you can add or remove tables. It matches names, not meanings: a request that calls `clients` "customers" finds nothing, and the plan then goes ahead with no table in scope. That is how a request for **new tables** works: the model is told the names of the existing tables (not their contents), creates the new ones with `create_table`, and any existing table its foreign keys point at is read afterwards so the keys are checked against that table's real columns. Names already taken in the database are refused.
2. **Give the planner facts.** Only the tables in scope are sent in full (the others by name): columns, keys, existing indexes with how often each was scanned (`pg_stat_user_indexes`), row estimates, data sizes, dependent views, and two aggregate statistics per column (NULL fraction, distinct-value estimate). Never rows, default expressions, index predicates or credentials.
3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check` and `drop_constraint`. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`).
3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check`, `drop_constraint` and `add_primary_key`. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`).
4. **Check the database, then run it.** *Check the database first* reads, without writing, who holds or waits for the tables (found through `pg_locks`, so even a role that may not read other roles' sessions sees them), transactions open for 30 seconds or more, strong locks, prepared transactions, replicas that are behind, replication slots holding WAL, vacuum running on the tables, and the timeouts the role runs with; each check says "can't tell" instead of "ok" when it cannot read something. Free disk space cannot be read from a connection, so that is for you to check. *Run it here, one step at a time* then runs each step on its own in a SQL session of its own (no transaction, because `CONCURRENTLY` and the batched backfills need that): only the next step can run, the first failure stops the run, a step can be run again or stopped, and the most recent finished step can be undone with the statement the compiler wrote for it. A step that needs a strong lock on an existing table waits at most 2 seconds per attempt and tries up to 8 times, backing off between tries; if the table never frees up it stops with the sessions holding it and changes nothing. After the last step the catalog is read again and compared, table by table, with the state the compiler said it would leave (columns, types, NOT NULL, keys, unique and check constraints, foreign keys, indexes, views); every difference is listed. Optional clean-up steps (dropping an old column, a compatibility view) are never run for you. The runbook can also be opened in the SQL Editor with the transaction switch off, or copied.
5. **Health.** A read-only report from catalog queries: tables without a primary key, foreign keys without an index, duplicate, redundant, unused and invalid indexes, constraints that were never validated, wide tables, missing or stale statistics, tables with many dead rows, sequences past half their range. A model may be asked to order and explain the findings; it can only point at findings that exist.
6. **Governance.** The model is called through the OpenAI API with your key (see above; GoForge does not store it), an allow-list of model ids, optional redaction of schema, table, column, index and constraint names into placeholders before anything leaves the server (the answer is translated back), and an audit log of every request with exactly what was sent, every plan, every runbook opened and every settings change, exportable as CSV or JSON. The audit log records that a runbook was *opened*; what you run in the SQL Editor is not tracked.
Expand Down
18 changes: 17 additions & 1 deletion scripts/schema-change-pg.ts
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ interface Scenario {
/** The runbook is expected to compile with this error code instead of running. */
refused?: string;
includeOptional?: boolean;
/** SQL run on the scratch database before the plan is compiled (for what the shared fixture does not have). */
setup?: string;
}

const one = async (client: PgClient, sql: string): Promise<Record<string, unknown> | undefined> => (await client.query(sql)).rows[0];
Expand Down Expand Up @@ -264,6 +266,18 @@ const SCENARIOS: Scenario[] = [
checks.equal((await one(client, "SELECT convalidated AS v FROM pg_constraint WHERE conname = 'orders_total_small'"))?.v, false, "added NOT VALID, enforced for new rows, validation failed");
},
},
{
name: "add a primary key to a table without one (NOT NULL first, unique index CONCURRENTLY, USING INDEX)",
setup: "CREATE TABLE orders_nopk AS SELECT id, customer_id, status FROM orders; ANALYZE orders_nopk;",
tables: ["public.orders_nopk"],
operations: [{ type: "add_primary_key", tableName: "orders_nopk", columns: ["id"] }],
async verify(client, checks) {
checks.equal(await one(client, "SELECT contype::text AS t, (SELECT attnotnull FROM pg_attribute WHERE attrelid = 'orders_nopk'::regclass AND attname = 'id') AS nn FROM pg_constraint WHERE conrelid = 'orders_nopk'::regclass AND contype = 'p'"), { t: "p", nn: true }, "a primary key on a NOT NULL column");
checks.equal((await one(client, "SELECT indisprimary AS p, indisvalid AS v FROM pg_index WHERE indexrelid = 'orders_nopk_pkey'::regclass"))?.p, true, "the index built for it is the primary key's");
await client.query("INSERT INTO orders_nopk VALUES (1, 1, 'dup')").then(() => checks.ok(false, "a duplicate key is refused"), () => checks.ok(true, "a duplicate key is refused"));
checks.equal((await one(client, "SELECT count(*)::int AS n FROM pg_constraint WHERE conrelid = 'orders_nopk'::regclass AND contype = 'c'"))?.n, 0, "the helper CHECK is gone");
},
},
{
name: "refuses NOT NULL without a default on a table with rows",
tables: ["public.orders"],
Expand All @@ -289,6 +303,7 @@ async function compileSection(checks: Checks, only?: string) {
for (const scenario of SCENARIOS) {
if (only && !scenario.name.includes(only)) continue;
await withDatabase(async (client) => {
if (scenario.setup) await client.query(scenario.setup);
const plan = parsePlan(scenario.operations);
const { runbook } = await compileOn(client, plan, scenario.tables);
if (scenario.refused) {
Expand Down Expand Up @@ -348,6 +363,7 @@ async function locksSection(checks: Checks) {
for (const scenario of SCENARIOS) {
if (scenario.refused || scenario.name.includes("(fails") || scenario.name.includes("has NULLs")) continue;
await withDatabase(async (client) => {
if (scenario.setup) await client.query(scenario.setup);
const { runbook } = await compileOn(client, parsePlan(scenario.operations), scenario.tables);
for (const step of runbook.steps) {
if (step.optional && !scenario.includeOptional) continue;
Expand Down Expand Up @@ -960,7 +976,7 @@ async function modelSection(checks: Checks) {
];
let compared = 0;
try {
const scenarios = [...SCENARIOS.filter((scenario) => !(scenario.refused || scenario.name.includes("(fails") || scenario.name.includes("has NULLs"))).map((scenario) => ({ name: scenario.name, tables: scenario.tables, operations: scenario.operations, setup: undefined as string | undefined })), ...own];
const scenarios = [...SCENARIOS.filter((scenario) => !(scenario.refused || scenario.name.includes("(fails") || scenario.name.includes("has NULLs"))).map((scenario) => ({ name: scenario.name, tables: scenario.tables, operations: scenario.operations, setup: scenario.setup })), ...own];
for (const scenario of scenarios) {
await withDatabase(async (client) => {
if (scenario.setup) await client.query(scenario.setup);
Expand Down
2 changes: 2 additions & 0 deletions server/src/ai/plan-normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ const normalizeOperation = (value: unknown): unknown => {
return { type, tableName, constraintName: operation.constraintName ?? operation.name, expression: operation.expression ?? operation.condition ?? operation.check };
case "drop_constraint":
return { type, tableName, constraintName: operation.constraintName ?? operation.name };
case "add_primary_key":
return { type, tableName, constraintName: operation.constraintName ?? operation.name, columns: operation.columns ?? (operation.column !== undefined ? [operation.column] : undefined) };
case "add_relationship":
return {
type,
Expand Down
1 change: 1 addition & 0 deletions server/src/ai/schema-plan.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ function makeOperationSchema<Changes extends z.ZodTypeAny>(changes: Changes) {
z.object({ type: z.literal("add_unique"), tableName: identifier, constraintName: identifier.nullable().optional(), columns: z.array(identifier).min(1).max(32) }).strict(),
z.object({ type: z.literal("add_check"), tableName: identifier, constraintName: identifier.nullable().optional(), expression: z.string().trim().min(1).max(400) }).strict(),
z.object({ type: z.literal("drop_constraint"), tableName: identifier, constraintName: identifier }).strict(),
z.object({ type: z.literal("add_primary_key"), tableName: identifier, constraintName: identifier.nullable().optional(), columns: z.array(identifier).min(1).max(32) }).strict(),
]);
}

Expand Down
2 changes: 1 addition & 1 deletion server/src/ai/structured.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ describe("the schemas the model is given", () => {
expect(properties).toBeLessThan(500);
expect(JSON.stringify(schema).length).toBeLessThan(60_000);
const operations = schema.properties.operations.items.anyOf as Json[];
expect(operations.map((operation) => operation.properties.type.const).sort()).toEqual(["add_check", "add_column", "add_index", "add_relationship", "add_unique", "alter_column", "create_table", "drop_column", "drop_constraint", "drop_index", "drop_relationship", "drop_table", "rename_table"]);
expect(operations.map((operation) => operation.properties.type.const).sort()).toEqual(["add_check", "add_column", "add_index", "add_primary_key", "add_relationship", "add_unique", "alter_column", "create_table", "drop_column", "drop_constraint", "drop_index", "drop_relationship", "drop_table", "rename_table"]);
});

it("the model's plan schema has the same top-level keys as the plan validator", () => {
Expand Down
44 changes: 44 additions & 0 deletions server/src/schema-change/compile.ts
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,7 @@ class Compiler {
case "add_unique": return this.addUnique(operation);
case "add_check": return this.addCheck(operation);
case "drop_constraint": return this.dropConstraint(operation);
case "add_primary_key": return this.addPrimaryKey(operation);
default:
this.fail("unsupported-operation", `Operation ${index + 1} has a type this compiler does not know.`);
}
Expand Down Expand Up @@ -1302,6 +1303,49 @@ class Compiler {
table.checks.push({ name, kind: "check", definition: `CHECK ((${expression}))`, columns: used, validated: true });
}

private addPrimaryKey(operation: Extract<Operation, { type: "add_primary_key" }>): void {
const table = this.findTable(operation.tableName);
const label = table.qualified;
if (table.primaryKey.length > 0) this.fail("has-primary-key", `${label} already has a primary key (${table.primaryKey.join(", ")}). Drop it first, or change the key some other way.`, table.qualified);
if (table.partitioned) this.fail("unsupported", `${label} is partitioned. A primary key on a partitioned table has to include the partition key and is built partition by partition; that is not compiled.`, table.qualified);
const columns = operation.columns.map((name) => this.findColumn(table, name));
if (new Set(columns.map((column) => lower(column.name))).size !== columns.length) this.fail("duplicate-column", `The primary key of ${label} names a column twice.`, table.qualified);
const names = columns.map((column) => column.name);
const target = qualifiedIdent(table.schema, table.name);
const given = operation.constraintName ?? undefined;
const name = given ?? limitName(`${table.name}_pkey`, this.names(table));
this.claimRelation(table.schema, name, "Primary key");
if (this.names(table).has(name)) this.fail("name-taken", `${label} already has a constraint called ${name}.`, table.qualified);

// A primary key makes its columns NOT NULL. Doing that first, through the validated CHECK, keeps ADD PRIMARY KEY from having to read the table under its lock.
for (const column of columns) {
if (column.notNull) continue;
if (column.nullFraction !== null && column.nullFraction > 0) this.note("warning", "has-nulls", `${label}.${column.name} has NULLs according to the table statistics (about ${(column.nullFraction * 100).toFixed(1)}% of rows). A primary key cannot have any: the validation step will fail until they are filled.`, table.qualified);
if (column.generated === null && column.default === null) this.note("info", "null-key", `${label}.${column.name} has no default, so rows inserted without it would fail once it is NOT NULL.`, table.qualified);
this.notNullSteps(table, column.name);
column.notNull = true;
}

const list = names.map(ident).join(", ");
if (sizeClass(table) === "new") {
this.add({ title: `Add the primary key ${name} to ${label}`, sql: `ALTER TABLE ${target} ADD CONSTRAINT ${ident(name)} PRIMARY KEY (${list});`, mode: "short", lock: "ACCESS EXCLUSIVE", work: "none", table });
} else {
this.concurrentIndexStep(table, name, names, true, `Build a unique index for the primary key of ${label}(${names.join(", ")})`);
this.add({
title: `Make the index the primary key of ${label}`,
sql: `ALTER TABLE ${target} ADD CONSTRAINT ${ident(name)} PRIMARY KEY USING INDEX ${ident(name)};`,
mode: "short",
lock: "ACCESS EXCLUSIVE",
work: "metadata",
table,
notes: ["The columns are already NOT NULL and the index is built and valid, so this only changes the catalog.", "A unique index fails to build if the table already has duplicates: find them with SELECT " + list + ", count(*) FROM " + target + " GROUP BY " + list + " HAVING count(*) > 1 LIMIT 10;"],
undo: `ALTER TABLE ${target} DROP CONSTRAINT ${ident(name)};`,
});
}
table.primaryKey = names;
table.indexes.push({ name, definition: "", columns: names, uses: names, method: "btree", unique: true, primary: true, valid: true, partial: false, constraint: name, bytes: 0, scans: null });
}

private dropConstraint(operation: Extract<Operation, { type: "drop_constraint" }>): void {
const table = this.findTable(operation.tableName);
const name = operation.constraintName;
Expand Down
Loading
Loading