Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ The **Schema Change** page works on a saved live connection, not on the diagram

1. **Find the tables (optional).** You do not have to choose any. The catalog (names, columns, keys, estimated rows, sizes) is read in one round trip. The words of your request are matched against table, schema and column names, weighted by how rare each word is in *your* database, and the foreign-key neighbours of the best matches are added. The scope is shown with the reason for each table and you can add or remove tables. It matches names, not meanings: a request that calls `clients` "customers" finds nothing, and the plan then goes ahead with no table in scope. That is how a request for **new tables** works: the model is told the names of the existing tables (not their contents), creates the new ones with `create_table`, and any existing table its foreign keys point at is read afterwards so the keys are checked against that table's real columns. Names already taken in the database are refused.
2. **Give the planner facts.** Only the tables in scope are sent in full (the others by name): columns, keys, existing indexes with how often each was scanned (`pg_stat_user_indexes`), row estimates, data sizes, dependent views, and two aggregate statistics per column (NULL fraction, distinct-value estimate). Never rows, default expressions, index predicates or credentials.
3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check`, `drop_constraint` and `add_primary_key`. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`).
3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check`, `drop_constraint` and `add_primary_key`. An existing integer column can become an identity column (`alter_column` with `autoIncrement`): the sequence is started after the column's largest value in the same transaction, which needs an index on the column unless the table is small, and a serial column's old default is dropped in the same change. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`).
4. **Check the database, then run it.** *Check the database first* reads, without writing, who holds or waits for the tables (found through `pg_locks`, so even a role that may not read other roles' sessions sees them), transactions open for 30 seconds or more, strong locks, prepared transactions, replicas that are behind, replication slots holding WAL, vacuum running on the tables, and the timeouts the role runs with; each check says "can't tell" instead of "ok" when it cannot read something. Free disk space cannot be read from a connection, so that is for you to check. *Run it here, one step at a time* then runs each step on its own in a SQL session of its own (no transaction, because `CONCURRENTLY` and the batched backfills need that): only the next step can run, the first failure stops the run, a step can be run again or stopped, and the most recent finished step can be undone with the statement the compiler wrote for it. A step that needs a strong lock on an existing table waits at most 2 seconds per attempt and tries up to 8 times, backing off between tries; if the table never frees up it stops with the sessions holding it and changes nothing. After the last step the catalog is read again and compared, table by table, with the state the compiler said it would leave (columns, types, NOT NULL, keys, unique and check constraints, foreign keys, indexes, views); every difference is listed. Optional clean-up steps (dropping an old column, a compatibility view) are never run for you. The runbook can also be opened in the SQL Editor with the transaction switch off, or copied.
5. **Health.** A read-only report from catalog queries: tables without a primary key, foreign keys without an index, duplicate, redundant, unused and invalid indexes, constraints that were never validated, wide tables, missing or stale statistics, tables with many dead rows, sequences past half their range. A model may be asked to order and explain the findings; it can only point at findings that exist.
6. **Governance.** The model is called through the OpenAI API with your key (see above; GoForge does not store it), an allow-list of model ids, optional redaction of schema, table, column, index and constraint names into placeholders before anything leaves the server (the answer is translated back), and an audit log of every request with exactly what was sent, every plan, every runbook opened and every settings change, exportable as CSV or JSON. The audit log records that a runbook was *opened*; what you run in the SQL Editor is not tracked.
Expand Down
25 changes: 24 additions & 1 deletion scripts/schema-change-pg.ts
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,28 @@ const SCENARIOS: Scenario[] = [
checks.equal((await one(client, "SELECT count(*)::int AS n FROM pg_constraint WHERE conrelid = 'orders_nopk'::regclass AND contype = 'c'"))?.n, 0, "the helper CHECK is gone");
},
},
{
name: "make an integer column an identity column (the sequence starts after the largest value)",
setup: "CREATE TABLE ledger (id integer NOT NULL PRIMARY KEY, memo text); INSERT INTO ledger SELECT g, repeat('x', 40) FROM generate_series(1, 300000) g; ANALYZE ledger;",
tables: ["public.ledger"],
operations: [{ type: "alter_column", tableName: "ledger", columnName: "id", changes: { autoIncrement: true } }],
async verify(client, checks) {
checks.equal((await one(client, "SELECT attidentity::text AS i FROM pg_attribute WHERE attrelid = 'ledger'::regclass AND attname = 'id'"))?.i, "d", "GENERATED BY DEFAULT AS IDENTITY");
checks.equal((await one(client, "INSERT INTO ledger (memo) VALUES ('new') RETURNING id"))?.id, 300001, "the next row gets the number after the largest one");
await client.query("INSERT INTO ledger (id, memo) VALUES (-5, 'explicit')");
checks.ok(true, "a row that gives its own id is still accepted");
},
},
{
name: "turn a serial column into an identity column (drop the default, then identity)",
setup: "CREATE TABLE ledger2 (id serial PRIMARY KEY, memo text); INSERT INTO ledger2 (memo) SELECT repeat('x', 40) FROM generate_series(1, 300000); ANALYZE ledger2;",
tables: ["public.ledger2"],
operations: [{ type: "alter_column", tableName: "ledger2", columnName: "id", changes: { defaultValue: null, autoIncrement: true } }],
async verify(client, checks) {
checks.equal(await one(client, "SELECT attidentity::text AS i, pg_get_expr(d.adbin, d.adrelid) AS d FROM pg_attribute a LEFT JOIN pg_attrdef d ON d.adrelid = a.attrelid AND d.adnum = a.attnum WHERE a.attrelid = 'ledger2'::regclass AND a.attname = 'id'"), { i: "d", d: null }, "an identity column and no default");
checks.equal((await one(client, "INSERT INTO ledger2 (memo) VALUES ('new') RETURNING id"))?.id, 300001, "the next row gets the number after the largest one");
},
},
{
name: "refuses NOT NULL without a default on a table with rows",
tables: ["public.orders"],
Expand Down Expand Up @@ -760,7 +782,8 @@ async function lockWaitSection(checks: Checks) {
checks.ok(message.includes(`pid ${holderPid}`), "names the session holding the table", message);
checks.ok(message.includes("long report"), "shows what that session is running", message);
checks.ok(!(await columns()).includes("note_b"), "and changed nothing");
checks.ok(elapsed < 12000, `and it stopped (${elapsed} ms)`);
// that it stops is what is checked, not how fast: a loaded machine makes the attempts slow
checks.ok(elapsed < 180000, `and it stopped (${elapsed} ms)`);
} finally {
await holder.query("ROLLBACK").catch(() => undefined);
await Promise.all([holder.end(), runner.end()].map((end) => end.catch(() => undefined)));
Expand Down
55 changes: 54 additions & 1 deletion server/src/schema-change/compile.ts
Original file line number Diff line number Diff line change
Expand Up @@ -785,7 +785,7 @@ class Compiler {
const ref = ident(column.name);

if (changes.primaryKey !== undefined) this.fail("unsupported", `Changing whether ${label} is part of the primary key is not compiled; the key and every foreign key that points at it have to change together.`, table.qualified);
if (changes.autoIncrement === true && column.generated === null) this.fail("unsupported", `Making the existing column ${label} an identity column is not compiled: its sequence has to start after the largest existing value.`, table.qualified);
if (changes.autoIncrement === true && column.generated === "stored") this.fail("generated-column", `${label} is a generated column and cannot also be an identity column.`, table.qualified);

const wantsType = changes.dataType !== undefined || changes.maxLength !== undefined || changes.precision !== undefined || changes.scale !== undefined;
let changed = false;
Expand Down Expand Up @@ -834,6 +834,13 @@ class Compiler {
changed = true;
}

if (changes.autoIncrement === true && column.generated === null) {
this.identitySteps(table, column);
changed = true;
} else if (changes.autoIncrement === true) {
this.note("info", "no-change", `${label} is already an identity column.`, table.qualified);
}

if (changes.autoIncrement === false && column.generated !== null) {
this.add({ title: `Remove the identity from ${label}`, sql: `ALTER TABLE ${target} ALTER COLUMN ${ref} DROP IDENTITY;`, mode: "short", lock: "ACCESS EXCLUSIVE", work: "metadata", table });
column.generated = null;
Expand Down Expand Up @@ -891,6 +898,52 @@ class Compiler {
}
}

/**
* An existing column becomes `GENERATED BY DEFAULT AS IDENTITY`, with its sequence started after the largest value it holds. Both happen in one
* transaction, so no insert can be handed a number that is already taken. The largest value is read through an index on the column: without one it
* would be a scan of the table under the lock, so a table that is not small needs one first.
*/
private identitySteps(table: TableFacts, column: ColumnFacts): void {
const label = `${table.qualified}.${column.name}`;
if (!isIntegerType(column.type)) this.fail("unsupported", `${label} is ${column.type}; an identity column has to be smallint, integer or bigint.`, table.qualified);
if (!column.notNull) this.fail("not-null-required", `${label} allows NULL. An identity column has to be NOT NULL: make it NOT NULL in the same change (nullable: false), or first.`, table.qualified);
if (column.default !== null) this.fail("has-default", `${label} has a default (${column.default}), and an identity column cannot have one. Remove it in the same change (defaultValue: null). If it is a serial column, its old sequence stays, owned by the column, and is dropped with it.`, table.qualified);
const small = sizeClass(table) === "new" || sizeClass(table) === "small" || !this.hasRows(table);
const indexed = table.indexes.some((index) => index.valid && !index.partial && index.method === "btree" && index.columns[0] === column.name);
if (!small && !indexed) this.fail("index-needed", `${label} has no index to find its largest value with, and finding it would read all of ${table.qualified} under a lock. Add an index on the column first (add_index), then change it to an identity column.`, table.qualified);

const target = qualifiedIdent(table.schema, table.name);
const ref = ident(column.name);
// Before PostgreSQL 13 a column that owns the sequence of an old serial default and also has an identity makes every INSERT fail ("more than
// one owned sequence found"), so the old sequence is detached first. It stays, for anything that calls nextval on it by name.
const detach = this.facts.serverVersionNum < 130000
? `EXECUTE COALESCE((SELECT string_agg(format('ALTER SEQUENCE %s OWNED BY NONE', d.objid::regclass), '; ') FROM pg_depend d JOIN pg_attribute a ON a.attrelid = d.refobjid AND a.attnum = d.refobjsubid WHERE d.refobjid = ${literal(target)}::regclass AND a.attname = ${literal(column.name)} AND d.classid = 'pg_class'::regclass AND d.deptype = 'a'), 'SELECT 1');`
: undefined;
const alter = `ALTER TABLE ${target} ALTER COLUMN ${ref} ADD GENERATED BY DEFAULT AS IDENTITY;`;
// The identity's own sequence is the one the column depends on internally. pg_get_serial_sequence would be ambiguous for a column that still
// owns the sequence of an old serial default, and would start the wrong one.
const sequence = `(SELECT d.objid FROM pg_depend d JOIN pg_attribute a ON a.attrelid = d.refobjid AND a.attnum = d.refobjsubid WHERE d.refobjid = ${literal(target)}::regclass AND a.attname = ${literal(column.name)} AND d.classid = 'pg_class'::regclass AND d.deptype = 'i')::regclass`;
const start = `setval(${sequence}, (SELECT COALESCE(max(${ref}), 0) + 1 FROM ${target}), false)`;
this.add({
title: `Make ${label} an identity column, starting after its largest value`,
// one transaction either way: inside the lock-retry block when there is one, otherwise as a DO block of its own
sql: this.settings.lockAttempts >= 2
? `${detach ? `${detach}\n` : ""}${alter}\nPERFORM ${start};`
: `DO $identity$\nBEGIN\n SET LOCAL lock_timeout = '${this.settings.lockTimeoutSeconds}s';\n ${detach ? `${detach}\n ` : ""}${alter}\n PERFORM ${start};\nEND\n$identity$;`,
mode: "short",
lock: "ACCESS EXCLUSIVE",
work: "metadata",
table,
notes: [
"BY DEFAULT: an insert that gives the column a value still works, and one that leaves it out gets the next number.",
`The sequence is started at the largest value the column holds plus one, in the same transaction, so a new row cannot be given a number that is already used.${small ? "" : " The largest value is read through an index."}`,
...(detach ? ["This server is older than PostgreSQL 13, which cannot have an identity and the sequence of an old serial default on one column, so any such old sequence is detached from the column first. It stays, and is no longer dropped with the column."] : []),
],
undo: `ALTER TABLE ${target} ALTER COLUMN ${ref} DROP IDENTITY;`,
});
column.generated = "default-identity";
}

/** ALTER COLUMN ... TYPE: in place when PostgreSQL only has to change the catalog, otherwise the expand/contract sequence. */
private typeChange(table: TableFacts, column: ColumnFacts, to: string): void {
const target = qualifiedIdent(table.schema, table.name);
Expand Down
Loading
Loading