diff --git a/README.md b/README.md index fe98f65..7a4fe69 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ The **Schema Change** page works on a saved live connection, not on the diagram 1. **Find the tables (optional).** You do not have to choose any. The catalog (names, columns, keys, estimated rows, sizes) is read in one round trip. The words of your request are matched against table, schema and column names, weighted by how rare each word is in *your* database, and the foreign-key neighbours of the best matches are added. The scope is shown with the reason for each table and you can add or remove tables. It matches names, not meanings: a request that calls `clients` "customers" finds nothing, and the plan then goes ahead with no table in scope. That is how a request for **new tables** works: the model is told the names of the existing tables (not their contents), creates the new ones with `create_table`, and any existing table its foreign keys point at is read afterwards so the keys are checked against that table's real columns. Names already taken in the database are refused. 2. **Give the planner facts.** Only the tables in scope are sent in full (the others by name): columns, keys, existing indexes with how often each was scanned (`pg_stat_user_indexes`), row estimates, data sizes, dependent views, and two aggregate statistics per column (NULL fraction, distinct-value estimate). Never rows, default expressions, index predicates or credentials. -3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check`, `drop_constraint` and `add_primary_key`. An existing integer column can become an identity column (`alter_column` with `autoIncrement`): the sequence is started after the column's largest value in the same transaction, which needs an index on the column unless the table is small, and a serial column's old default is dropped in the same change. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`). +3. **A model proposes operations; fixed rules write the SQL.** Each operation is checked against the real catalog (names exist, types are real, a foreign key points at a unique column, a NOT NULL column has a default when the table has rows, an index name is free, a default is accepted by PostgreSQL) and compiled into a runbook: `CREATE INDEX CONCURRENTLY`, foreign keys and checks `NOT VALID` then `VALIDATE`, `NOT NULL` through a validated `CHECK`, a `lock_timeout` on every step, batched committed backfills, and for a type change that would rewrite the table an expand/contract sequence (new column, trigger, batched copy, swap in one `DO` block, the old column dropped in a step that is commented out). Every step states its lock, the work it does, the size of the table and a risk class that follows from those three by a fixed rule, plus how to undo it where that is possible. The operations are `create_table`, `rename_table`, `drop_table`, `add_column`, `alter_column`, `drop_column`, `add_relationship`, `drop_relationship`, `add_index`, `drop_index`, `add_unique`, `add_check`, `drop_constraint`, `add_primary_key`, `create_enum` and `add_enum_value`. `create_enum` makes a type (a column can use it later in the same plan, with a default that is one of its labels) and `add_enum_value` adds a label, optionally before or after another: neither locks any table, and a label can never be removed again. An existing integer column can become an identity column (`alter_column` with `autoIncrement`): the sequence is started after the column's largest value in the same transaction, which needs an index on the column unless the table is small, and a serial column's old default is dropped in the same change. A drop or a type change is refused, naming what depends on the column or table (views, triggers, policies, generated columns, SQL-body functions, publications), exactly where PostgreSQL would refuse it without `CASCADE`. Renaming an existing table keeps the old name working as a view (with the table's grants copied; from PostgreSQL 15 the view runs with the caller's own rights, and before that a table with row-level security is refused) until an optional last step drops it. The lock levels and the "rewrites the table" predictions are checked against real servers from PostgreSQL 12 to 18 (`npm run test:schema-pg:versions`). 4. **Check the database, then run it.** *Check the database first* reads, without writing, who holds or waits for the tables (found through `pg_locks`, so even a role that may not read other roles' sessions sees them), transactions open for 30 seconds or more, strong locks, prepared transactions, replicas that are behind, replication slots holding WAL, vacuum running on the tables, and the timeouts the role runs with; each check says "can't tell" instead of "ok" when it cannot read something. Free disk space cannot be read from a connection, so that is for you to check. *Run it here, one step at a time* then runs each step on its own in a SQL session of its own (no transaction, because `CONCURRENTLY` and the batched backfills need that): only the next step can run, the first failure stops the run, a step can be run again or stopped, and the most recent finished step can be undone with the statement the compiler wrote for it. A step that needs a strong lock on an existing table waits at most 2 seconds per attempt and tries up to 8 times, backing off between tries; if the table never frees up it stops with the sessions holding it and changes nothing. After the last step the catalog is read again and compared, table by table, with the state the compiler said it would leave (columns, types, NOT NULL, keys, unique and check constraints, foreign keys, indexes, views); every difference is listed. Optional clean-up steps (dropping an old column, a compatibility view) are never run for you. The runbook can also be opened in the SQL Editor with the transaction switch off, or copied. 5. **Health.** A read-only report from catalog queries: tables without a primary key, foreign keys without an index, duplicate, redundant, unused and invalid indexes, constraints that were never validated, wide tables, missing or stale statistics, tables with many dead rows, sequences past half their range. A model may be asked to order and explain the findings; it can only point at findings that exist. 6. **Governance.** The model is called through the OpenAI API with your key (see above; GoForge does not store it), an allow-list of model ids, optional redaction of schema, table, column, index and constraint names into placeholders before anything leaves the server (the answer is translated back), and an audit log of every request with exactly what was sent, every plan, every runbook opened and every settings change, exportable as CSV or JSON. The audit log records that a runbook was *opened*; what you run in the SQL Editor is not tracked. diff --git a/scripts/schema-change-pg.ts b/scripts/schema-change-pg.ts index f4a360d..1f0f2e3 100644 --- a/scripts/schema-change-pg.ts +++ b/scripts/schema-change-pg.ts @@ -300,6 +300,30 @@ const SCENARIOS: Scenario[] = [ checks.equal((await one(client, "INSERT INTO ledger2 (memo) VALUES ('new') RETURNING id"))?.id, 300001, "the next row gets the number after the largest one"); }, }, + { + name: "create an enum, add a column of that type with a default, then add a label after one", + tables: ["public.orders"], + operations: [ + { type: "create_enum", typeName: "status_kind", values: ["new", "open"] }, + { type: "add_column", tableName: "orders", field: { name: "kind", dataType: "status_kind", nullable: false, defaultValue: "'new'" } }, + { type: "add_enum_value", typeName: "status_kind", value: "closed", after: "open" }, + ], + async verify(client, checks) { + checks.equal((await client.query("SELECT enumlabel AS l FROM pg_enum WHERE enumtypid = 'status_kind'::regtype ORDER BY enumsortorder")).rows.map((row) => row.l), ["new", "open", "closed"], "the labels, in order"); + checks.equal(Number((await one(client, "SELECT count(*)::int AS n FROM orders WHERE kind = 'new'"))?.n), 400000, "every existing row reads the default"); + await client.query("UPDATE orders SET kind = 'closed' WHERE id = 1"); + checks.ok(true, "the added label can be used"); + }, + }, + { + name: "add a label before another to an enum that exists", + setup: "CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');", + tables: [], + operations: [{ type: "add_enum_value", typeName: "mood", value: "meh", before: "ok" }], + async verify(client, checks) { + checks.equal((await client.query("SELECT enumlabel AS l FROM pg_enum WHERE enumtypid = 'mood'::regtype ORDER BY enumsortorder")).rows.map((row) => row.l), ["sad", "meh", "ok", "happy"], "the label is where it was asked to be"); + }, + }, { name: "refuses NOT NULL without a default on a table with rows", tables: ["public.orders"], @@ -484,7 +508,7 @@ async function rewritesSection(checks: Checks) { await client.query(`INSERT INTO ${table} SELECT g, ${sample}${from.startsWith("char(") ? "::" + from : ""} FROM generate_series(1, 100) g`); const files = async () => (await client.query(`SELECT string_agg(c.relname || '=' || pg_relation_filenode(c.oid)::text, ',' ORDER BY c.relname) AS f FROM pg_class c WHERE c.relname IN ('${table}', '${table}_v')`)).rows[0].f as string; const before = await files(); - const probes = await probe(asRunner(client), { types: [], typeChanges: [{ from, to }], functions: [], defaults: [], checks: [], relations: [], schemas: [] }); + const probes = await probe(asRunner(client), { types: [], typeChanges: [{ from, to }], functions: [], defaults: [], checks: [], enums: [], relations: [], schemas: [] }); const cast = probes.casts.get(castKey(from, to)); let verdict = classifyTypeChange(await columnType(client, table), to, cast); try { @@ -1278,6 +1302,7 @@ const sections: Record = { { expression: "no_such_function()", type: "text" }, { expression: "active", type: "text" }, ], + enums: [{ schema: "public", name: "mood" }, { schema: "public", name: "no_such_type" }, { schema: "public", name: "orders" }], checks: [ { table: "public.orders", expression: "total >= 0 AND status <> 'x'" }, { table: "public.orders", expression: "no_such_column > 0" }, @@ -1288,6 +1313,9 @@ const sections: Record = { }); checks.equal(probes.defaults.get(defaultKey("'new'", "varchar(20)")), undefined, "PostgreSQL accepts a text default"); checks.equal(probes.defaults.get(defaultKey("now()", "timestamptz")), undefined, "and a function default"); + checks.equal(probes.enums.get("public.mood"), { exists: true, kind: "e", labels: ["ok", "bad"] }, "an enum type is found with its labels in order"); + checks.equal(probes.enums.get("public.no_such_type")?.exists, false, "a type that is not there is not found"); + checks.equal(probes.enums.get("public.orders")?.kind, "c", "a table's row type is found, and is not an enum"); checks.equal(probes.checks.get(checkKey("public.orders", "total >= 0 AND status <> 'x'")), undefined, "PostgreSQL accepts a CHECK over the table's columns"); checks.match(probes.checks.get(checkKey("public.orders", "no_such_column > 0")), /column "no_such_column" does not exist/, "rejects a CHECK on a column the table does not have, with its own message"); checks.match(probes.checks.get(checkKey("public.orders", "total + 1")), /must be type boolean/, "and one that is not boolean"); diff --git a/server/src/ai/plan-normalize.ts b/server/src/ai/plan-normalize.ts index aa7abe3..cee3539 100644 --- a/server/src/ai/plan-normalize.ts +++ b/server/src/ai/plan-normalize.ts @@ -85,6 +85,10 @@ const normalizeOperation = (value: unknown): unknown => { return { type, tableName, constraintName: operation.constraintName ?? operation.name, expression: operation.expression ?? operation.condition ?? operation.check }; case "drop_constraint": return { type, tableName, constraintName: operation.constraintName ?? operation.name }; + case "create_enum": + return { type, typeName: operation.typeName ?? operation.name ?? operation.enumName, values: operation.values ?? operation.labels }; + case "add_enum_value": + return { type, typeName: operation.typeName ?? operation.name ?? operation.enumName, value: operation.value ?? operation.label, before: operation.before, after: operation.after }; case "add_primary_key": return { type, tableName, constraintName: operation.constraintName ?? operation.name, columns: operation.columns ?? (operation.column !== undefined ? [operation.column] : undefined) }; case "add_relationship": diff --git a/server/src/ai/schema-plan.ts b/server/src/ai/schema-plan.ts index 5b24773..80d7278 100644 --- a/server/src/ai/schema-plan.ts +++ b/server/src/ai/schema-plan.ts @@ -97,6 +97,8 @@ function makeOperationSchema(changes: Changes) { z.object({ type: z.literal("add_unique"), tableName: identifier, constraintName: identifier.nullable().optional(), columns: z.array(identifier).min(1).max(32) }).strict(), z.object({ type: z.literal("add_check"), tableName: identifier, constraintName: identifier.nullable().optional(), expression: z.string().trim().min(1).max(400) }).strict(), z.object({ type: z.literal("drop_constraint"), tableName: identifier, constraintName: identifier }).strict(), + z.object({ type: z.literal("create_enum"), typeName: identifier, values: z.array(z.string().min(1).max(63)).min(1).max(100) }).strict(), + z.object({ type: z.literal("add_enum_value"), typeName: identifier, value: z.string().min(1).max(63), before: z.string().min(1).max(63).nullable().optional(), after: z.string().min(1).max(63).nullable().optional() }).strict(), z.object({ type: z.literal("add_primary_key"), tableName: identifier, constraintName: identifier.nullable().optional(), columns: z.array(identifier).min(1).max(32) }).strict(), ]); } diff --git a/server/src/ai/structured.test.ts b/server/src/ai/structured.test.ts index dc8bfb8..0eb7834 100644 --- a/server/src/ai/structured.test.ts +++ b/server/src/ai/structured.test.ts @@ -94,7 +94,7 @@ describe("the schemas the model is given", () => { expect(properties).toBeLessThan(500); expect(JSON.stringify(schema).length).toBeLessThan(60_000); const operations = schema.properties.operations.items.anyOf as Json[]; - expect(operations.map((operation) => operation.properties.type.const).sort()).toEqual(["add_check", "add_column", "add_index", "add_primary_key", "add_relationship", "add_unique", "alter_column", "create_table", "drop_column", "drop_constraint", "drop_index", "drop_relationship", "drop_table", "rename_table"]); + expect(operations.map((operation) => operation.properties.type.const).sort()).toEqual(["add_check", "add_column", "add_enum_value", "add_index", "add_primary_key", "add_relationship", "add_unique", "alter_column", "create_enum", "create_table", "drop_column", "drop_constraint", "drop_index", "drop_relationship", "drop_table", "rename_table"]); }); it("the model's plan schema has the same top-level keys as the plan validator", () => { diff --git a/server/src/schema-change/compile.ts b/server/src/schema-change/compile.ts index 1d652f0..6435e1f 100644 --- a/server/src/schema-change/compile.ts +++ b/server/src/schema-change/compile.ts @@ -77,10 +77,12 @@ class Compiler { private readonly uncompiled: number[] = []; private tables: Map; private readonly settings: CompileSettings; - private readonly needed: ProbeRequest = { types: [], typeChanges: [], functions: [], defaults: [], checks: [], relations: [], schemas: [] }; + private readonly needed: ProbeRequest = { types: [], typeChanges: [], functions: [], defaults: [], checks: [], enums: [], relations: [], schemas: [] }; private current = -1; /** Names this plan has claimed so far, so two operations can't both create `orders_status_idx`. */ private readonly claimedRelations = new Set(); + /** Enum types this plan creates, and the labels they have so far (key: lower-case `schema.name`), so a later operation can use them before they exist. */ + private createdEnums = new Map(); constructor( private readonly plan: AiSchemaPlan, @@ -103,6 +105,7 @@ class Compiler { const keepSteps = this.steps.length; const keepTables = structuredClone(this.tables); const keepClaims = new Set(this.claimedRelations); + const keepEnums = new Map([...this.createdEnums].map(([key, labels]) => [key, [...labels]])); try { this.compileOperation(operation, index); } catch (error) { @@ -111,6 +114,7 @@ class Compiler { this.tables = keepTables; this.claimedRelations.clear(); keepClaims.forEach((claim) => this.claimedRelations.add(claim)); + this.createdEnums = keepEnums; this.uncompiled.push(index); } }); @@ -183,6 +187,7 @@ class Compiler { private type(spec: { dataType: string; maxLength?: number | null; precision?: number | null; scale?: number | null }, label: string): string { const text = renderType(spec); + if (this.createdEnum(text)) return text; if (this.discovery) { this.needed.types.push(text); return text; @@ -215,6 +220,13 @@ class Compiler { private defaultExpression(expression: string, type: string, label: string): { expression: string; volatile: boolean } { const problem = checkDefaultExpression(expression); if (problem) this.fail("bad-default", `Default for ${label}: ${problem}`); + const created = this.createdEnum(type); + if (created) { + // the type does not exist yet, so PostgreSQL cannot be asked: the default has to be one of its labels + const literalLabel = /^'((?:[^']|'')*)'(?:\s*::\s*[A-Za-z_][\w."]*)?$/.exec(expression.trim()); + if (!literalLabel || !created.includes(literalLabel[1].replace(/''/g, "'"))) this.fail("bad-default", `The default for ${label} has to be one of the labels of ${type}: ${created.map((value) => `'${value.replace(/'/g, "''")}'`).join(", ")}.`); + return { expression, volatile: false }; + } const functions = functionsIn(expression); if (this.discovery) { this.needed.functions.push(...functions); @@ -300,6 +312,8 @@ class Compiler { case "add_check": return this.addCheck(operation); case "drop_constraint": return this.dropConstraint(operation); case "add_primary_key": return this.addPrimaryKey(operation); + case "create_enum": return this.createEnum(operation); + case "add_enum_value": return this.addEnumValue(operation); default: this.fail("unsupported-operation", `Operation ${index + 1} has a type this compiler does not know.`); } @@ -365,7 +379,7 @@ class Compiler { table.uniques.push({ name: unique, kind: "unique", definition: `UNIQUE (${ident(field.name)})`, columns: [field.name], validated: true }); table.indexes.push({ name: unique, definition: "", columns: [field.name], uses: [field.name], method: "btree", unique: true, primary: false, valid: true, partial: false, constraint: unique, bytes: 0, scans: null }); } - if (field.values && field.values.length > 0) this.note("warning", "values-ignored", `"values" for ${qualified}.${field.name} was ignored: create an enum type or a CHECK constraint separately.`, qualified); + if (field.values && field.values.length > 0) this.note("warning", "values-ignored", `"values" for ${qualified}.${field.name} was ignored: create the enum with create_enum (or a CHECK constraint) and give the column its type.`, qualified); if (field.unsigned) this.note("warning", "unsigned-ignored", `PostgreSQL has no unsigned types; "unsigned" for ${qualified}.${field.name} was ignored.`, qualified); } @@ -551,7 +565,7 @@ class Compiler { this.fail("not-null-needs-default", `${label} is NOT NULL with no default, and ${table.qualified} has rows (${table.estimatedRows === null ? "size unknown" : `about ${formatCount(table.estimatedRows)}`}): PostgreSQL would reject it. Give it a default, or add it as nullable, fill it, then make it NOT NULL.`, table.qualified); } if (spec.unique && withDefault && rows) this.fail("unique-with-default", `${label} is unique and has a default, so every existing row would get the same value.`, table.qualified); - if (spec.values?.length) this.note("warning", "values-ignored", `"values" for ${label} was ignored: create an enum type or a CHECK constraint separately.`, table.qualified); + if (spec.values?.length) this.note("warning", "values-ignored", `"values" for ${label} was ignored: create the enum with create_enum (or a CHECK constraint) and give the column its type.`, table.qualified); if (spec.unsigned) this.note("warning", "unsigned-ignored", `PostgreSQL has no unsigned types; "unsigned" for ${label} was ignored.`, table.qualified); if (table.partitioned && spec.unique) this.fail("unsupported", `A unique column on the partitioned table ${table.qualified} is not compiled: the key must include the partition key.`, table.qualified); @@ -1356,6 +1370,90 @@ class Compiler { table.checks.push({ name, kind: "check", definition: `CHECK ((${expression}))`, columns: used, validated: true }); } + // ---- enums ------------------------------------------------------------------------------------------------------------ + + private enumName(name: string): { schema: string; name: string; key: string; sql: string } { + const { schema, name: bare } = splitQualified(name.trim()); + const resolved = schema ?? this.settings.defaultSchema; + return { schema: resolved, name: bare, key: lower(`${resolved}.${bare}`), sql: qualifiedIdent(resolved, bare) }; + } + + /** The labels of an enum this plan creates, when `type` (a column's type, arrays included) names one. */ + private createdEnum(type: string): string[] | undefined { + const bare = type.replace(/\s*\[.*$/, "").replace(/"/g, "").trim(); + if (!bare || /[()]/.test(bare)) return undefined; + return this.createdEnums.get(this.enumName(bare).key); + } + + private createEnum(operation: Extract): void { + const type = this.enumName(operation.typeName); + this.checkNameLength(type.name, "Type"); + const values = operation.values.map((value) => value); + if (new Set(values).size !== values.length) this.fail("duplicate-value", `The enum ${type.schema}.${type.name} lists a label twice.`); + for (const value of values) { + if (Buffer.byteLength(value) > 63) this.fail("name-too-long", `The label "${value.slice(0, 30)}…" of ${type.schema}.${type.name} is longer than the 63 bytes PostgreSQL allows.`); + if (value.includes("\0")) this.fail("bad-value", `A label of ${type.schema}.${type.name} contains a null byte.`); + } + if (this.discovery) this.needed.enums.push({ schema: type.schema, name: type.name }); + // a type and a table of the same name cannot both exist, so the relation namespace is checked as well as the types + this.claimRelation(type.schema, type.name, "Type"); + if (!this.discovery) { + const known = this.probes.enums.get(relationKey(type.schema, type.name)); + if (!known) this.fail("type-unverified", `The type ${type.schema}.${type.name} was not looked for in the database.`); + if (known.exists || this.createdEnums.has(type.key)) this.fail("name-taken", `A type called "${type.schema}.${type.name}" already exists.`); + } + this.add({ + title: `Create the enum ${type.schema}.${type.name} (${values.join(", ")})`, + sql: `CREATE TYPE ${type.sql} AS ENUM (${values.map(literal).join(", ")});`, + mode: "short", + lock: "none", + work: "none", + notes: ["A new type: nothing existing is locked or rewritten.", "Labels can be added later but never removed or reordered without recreating the type."], + undo: `DROP TYPE ${type.sql};`, + }); + this.createdEnums.set(type.key, [...values]); + } + + private addEnumValue(operation: Extract): void { + const type = this.enumName(operation.typeName); + const value = operation.value; + if (Buffer.byteLength(value) > 63) this.fail("name-too-long", `The label "${value.slice(0, 30)}…" is longer than the 63 bytes PostgreSQL allows.`); + if (value.includes("\0")) this.fail("bad-value", `The label contains a null byte.`); + if (operation.before && operation.after) this.fail("bad-position", `Say BEFORE or AFTER a label, not both.`); + if (this.discovery) { + this.needed.enums.push({ schema: type.schema, name: type.name }); + return; + } + let labels = this.createdEnums.get(type.key); + if (!labels) { + const known = this.probes.enums.get(relationKey(type.schema, type.name)); + if (!known) this.fail("type-unverified", `The type ${type.schema}.${type.name} was not looked for in the database.`); + if (!known.exists) this.fail("unknown-type", `PostgreSQL has no type called ${type.schema}.${type.name}.`); + if (known.kind !== "e") this.fail("not-an-enum", `${type.schema}.${type.name} is not an enum type.`); + labels = [...known.labels]; + } + const neighbour = operation.before ?? operation.after ?? undefined; + if (neighbour !== undefined && !labels.includes(neighbour)) this.fail("unknown-value", `${type.schema}.${type.name} has no label "${neighbour}" to put the new one ${operation.before ? "before" : "after"}.`); + if (labels.includes(value)) { + this.note("info", "no-change", `${type.schema}.${type.name} already has the label "${value}".`); + return; + } + this.add({ + title: `Add the label ${value} to the enum ${type.schema}.${type.name}`, + sql: `ALTER TYPE ${type.sql} ADD VALUE IF NOT EXISTS ${literal(value)}${operation.before ? ` BEFORE ${literal(operation.before)}` : operation.after ? ` AFTER ${literal(operation.after)}` : ""};`, + mode: "short", + lock: "none", + work: "none", + notes: [ + "Changes only the type: no table that uses it is locked, scanned or rewritten.", + "A label can never be removed from an enum again (only by recreating the type), and it cannot be used in the same transaction that adds it: this step runs on its own, so later steps can use it.", + ], + }); + const at = neighbour === undefined ? labels.length : labels.indexOf(neighbour) + (operation.after ? 1 : 0); + labels.splice(at, 0, value); + this.createdEnums.set(type.key, labels); + } + private addPrimaryKey(operation: Extract): void { const table = this.findTable(operation.tableName); const label = table.qualified; diff --git a/server/src/schema-change/enums.test.ts b/server/src/schema-change/enums.test.ts new file mode 100644 index 0000000..9f3d07d --- /dev/null +++ b/server/src/schema-change/enums.test.ts @@ -0,0 +1,70 @@ +import { describe, expect, it } from "vitest"; +import { compile, facts, MIB, plan, table } from "./test-fixtures.js"; +import type { Runbook } from "./runbook.js"; + +const errors = (runbook: Runbook) => runbook.findings.filter((finding) => finding.severity === "error").map((finding) => finding.code); +const orders = () => table({ name: "orders", rows: 5_000_000, bytes: 400 * MIB, columns: ["id:bigint:notnull", "note:text"] }); +const mood = { "public.mood": { labels: ["sad", "ok", "happy"] } }; + +describe("create_enum", () => { + it("creates the type in one step that locks nothing, with the labels quoted, and says how to undo it", () => { + const runbook = compile(plan({ type: "create_enum", typeName: "mood", values: ["sad", "it's fine", "happy"] }), facts()); + expect(errors(runbook)).toEqual([]); + expect(runbook.steps).toHaveLength(1); + expect(runbook.steps[0]).toMatchObject({ lock: "none", work: "none", risk: "low", undo: "DROP TYPE public.mood;" }); + expect(runbook.steps[0].sql).toContain("CREATE TYPE public.mood AS ENUM ('sad', 'it''s fine', 'happy');"); + }); + + it("refuses a name that is taken by a type or by a table, repeated labels and labels that are too long", () => { + expect(compile(plan({ type: "create_enum", typeName: "mood", values: ["a"] }), facts(), { enums: mood }).findings.map((finding) => finding.code)).toContain("name-taken"); + expect(compile(plan({ type: "create_enum", typeName: "orders", values: ["a"] }), facts(orders()), { existing: ["public.orders"] }).findings.map((finding) => finding.code)).toContain("name-taken"); + expect(errors(compile(plan({ type: "create_enum", typeName: "mood", values: ["a", "b", "a"] }), facts()))).toEqual(["duplicate-value"]); + expect(errors(compile(plan({ type: "create_enum", typeName: "mood", values: ["é".repeat(40)] }), facts()))).toEqual(["name-too-long"]); + expect(errors(compile(plan({ type: "create_enum", typeName: "nope.mood", values: ["a"] }), facts(), { missingSchemas: ["nope"] }))).toEqual(["unknown-schema"]); + }); + + it("can be used by a column later in the same plan, whose default has to be one of its labels", () => { + const ok = compile(plan({ type: "create_enum", typeName: "status_kind", values: ["new", "open"] }, { type: "add_column", tableName: "orders", field: { name: "kind", dataType: "status_kind", nullable: false, defaultValue: "'new'" } }), facts(orders())); + expect(errors(ok)).toEqual([]); + expect(ok.steps.map((step) => step.sql).join("\n")).toMatch(/ADD COLUMN kind status_kind (NOT NULL DEFAULT 'new'|DEFAULT 'new' NOT NULL)/); + const wrongDefault = compile(plan({ type: "create_enum", typeName: "status_kind", values: ["new", "open"] }, { type: "add_column", tableName: "orders", field: { name: "kind", dataType: "status_kind", defaultValue: "'closed'" } }), facts(orders())); + expect(errors(wrongDefault)).toEqual(["bad-default"]); + expect(wrongDefault.findings.find((finding) => finding.code === "bad-default")?.message).toContain("'new', 'open'"); + const array = compile(plan({ type: "create_enum", typeName: "status_kind", values: ["new"] }, { type: "add_column", tableName: "orders", field: { name: "kinds", dataType: "public.status_kind[]" } }), facts(orders())); + expect(errors(array)).toEqual([]); + // without the create_enum the same column is refused: the database has no such type + expect(errors(compile(plan({ type: "add_column", tableName: "orders", field: { name: "kind", dataType: "status_kind" } }), facts(orders()), { unknownTypes: ["status_kind"] }))).toEqual(["unknown-type"]); + }); +}); + +describe("add_enum_value", () => { + const add = (operation: Record, enums = mood) => compile(plan({ type: "add_enum_value", typeName: "mood", ...operation }), facts(), { enums }); + + it("adds the label at the end, before or after another, without locking any table", () => { + const end = add({ value: "ecstatic" }); + expect(errors(end)).toEqual([]); + expect(end.steps[0]).toMatchObject({ lock: "none", work: "none" }); + expect(end.steps[0].sql).toContain("ALTER TYPE public.mood ADD VALUE IF NOT EXISTS 'ecstatic';"); + expect(add({ value: "meh", before: "ok" }).steps[0].sql).toContain("ADD VALUE IF NOT EXISTS 'meh' BEFORE 'ok';"); + expect(add({ value: "fine", after: "ok" }).steps[0].sql).toContain("ADD VALUE IF NOT EXISTS 'fine' AFTER 'ok';"); + expect(end.steps[0].notes.join(" ")).toMatch(/never be removed/); + }); + + it("says so when the label is there, and refuses an unknown type, a type that is not an enum, an unknown neighbour and both positions", () => { + const present = add({ value: "ok" }); + expect(present.steps).toHaveLength(0); + expect(present.findings.map((finding) => finding.code)).toContain("no-change"); + expect(errors(add({ value: "x" }, {}))).toEqual(["unknown-type"]); + expect(errors(add({ value: "x" }, { "public.mood": { kind: "c", labels: [] } }))).toEqual(["not-an-enum"]); + expect(errors(add({ value: "x", before: "nothing" }))).toEqual(["unknown-value"]); + expect(errors(add({ value: "x", before: "ok", after: "sad" }))).toEqual(["bad-position"]); + }); + + it("sees the labels an earlier step of the same plan added", () => { + const runbook = compile(plan({ type: "add_enum_value", typeName: "mood", value: "ecstatic" }, { type: "add_enum_value", typeName: "mood", value: "ecstatic" }, { type: "add_enum_value", typeName: "mood", value: "calm", after: "ecstatic" }), facts(), { enums: mood }); + expect(errors(runbook)).toEqual([]); + expect(runbook.steps).toHaveLength(2); + expect(runbook.findings.map((finding) => finding.code)).toContain("no-change"); + expect(runbook.steps[1].sql).toContain("'calm' AFTER 'ecstatic'"); + }); +}); diff --git a/server/src/schema-change/facts.ts b/server/src/schema-change/facts.ts index 7e8c560..4a23542 100644 --- a/server/src/schema-change/facts.ts +++ b/server/src/schema-change/facts.ts @@ -447,11 +447,20 @@ export interface ProbeRequest { defaults: { expression: string; type: string }[]; /** CHECK expressions with the (quoted, schema-qualified) table they are for: PostgreSQL is asked to plan `SELECT 1 FROM table WHERE (expression)`. */ checks: { table: string; expression: string }[]; + /** Types the plan creates or adds values to (schema qualified): do they exist, are they enums, and which labels do they have? */ + enums: { schema: string; name: string }[]; /** Quoted, schema-qualified names the plan wants to create: are they free? */ relations: { schema: string; name: string }[]; schemas: string[]; } +export interface EnumProbe { + exists: boolean; + /** pg_type.typtype: `e` for an enum. */ + kind: string | null; + labels: string[]; +} + export interface Probes { /** Canonical name of a type PostgreSQL knows; undefined for one it doesn't. */ types: Map; @@ -463,6 +472,8 @@ export interface Probes { defaults: Map; /** PostgreSQL's message when it rejects a CHECK expression on a table; undefined when it accepts it. */ checks: Map; + /** What PostgreSQL has under each requested type name, keyed `schema.name`. */ + enums: Map; /** Whether a table, index or sequence already has this schema-qualified name. */ relationExists: Map; schemaExists: Map; @@ -473,7 +484,7 @@ export const defaultKey = (expression: string, type: string): string => `${type} export const checkKey = (table: string, expression: string): string => `${table}\u0000${expression}`; export const relationKey = (schema: string, name: string): string => `${schema}.${name}`; -export const emptyProbes = (): Probes => ({ types: new Map(), casts: new Map(), volatility: new Map(), defaults: new Map(), checks: new Map(), relationExists: new Map(), schemaExists: new Map() }); +export const emptyProbes = (): Probes => ({ types: new Map(), casts: new Map(), volatility: new Map(), defaults: new Map(), checks: new Map(), enums: new Map(), relationExists: new Map(), schemaExists: new Map() }); const quoteName = (name: string): string => `"${name.replace(/"/g, '""')}"`; @@ -501,8 +512,18 @@ export async function probe(client: Queryable, request: ProbeRequest): Promise

[relationKey(type.schema, type.name), type])).values()]; + const enumIndexes = enumTypes.map((type) => { + const literal = quoteLiteral(`${quoteName(type.schema)}.${quoteName(type.name)}`); + return statements.push(`SELECT to_regtype(${literal}) IS NOT NULL AS exists, (SELECT typtype::text FROM pg_type WHERE oid = to_regtype(${literal})) AS kind, COALESCE((SELECT json_agg(e.enumlabel ORDER BY e.enumsortorder) FROM pg_enum e WHERE e.enumtypid = to_regtype(${literal})), '[]'::json) AS labels`) - 1; + }); + if (statements.length > 0) { const results = await runReadOnly(client, statements); + enumTypes.forEach((type, i) => { + const row = results[enumIndexes[i]].rows[0] as Row; + probes.enums.set(relationKey(type.schema, type.name), { exists: row.exists === true, kind: (row.kind as string | null) ?? null, labels: (row.labels as string[]) ?? [] }); + }); if (typeIndex >= 0) { const row = results[typeIndex].rows[0] as Row; diff --git a/server/src/schema-change/planner.ts b/server/src/schema-change/planner.ts index 99ef2ad..a2badf7 100644 --- a/server/src/schema-change/planner.ts +++ b/server/src/schema-change/planner.ts @@ -79,7 +79,8 @@ export function referencedTableNames(plan: AiSchemaPlan): string[] { }; const names: string[] = []; for (const operation of plan.operations) { - if (operation.type === "create_table") continue; + // a type is not a table + if (operation.type === "create_table" || operation.type === "create_enum" || operation.type === "add_enum_value") continue; if (operation.type === "add_relationship") names.push(operation.relationship.sourceTable, operation.relationship.targetTable); else if (operation.type === "drop_relationship") names.push(operation.sourceTable, operation.targetTable); else names.push(operation.tableName); diff --git a/server/src/schema-change/prompt.ts b/server/src/schema-change/prompt.ts index 093964b..2a9fb7e 100644 --- a/server/src/schema-change/prompt.ts +++ b/server/src/schema-change/prompt.ts @@ -8,12 +8,12 @@ export function plannerInstructions(): string { "You are GoForge's schema change planner for a large, live PostgreSQL database. You decide WHAT should change; a separate compiler turns your operations into online-safe SQL and checks them against the real catalog.", "Return JSON only. Do not use markdown fences or prose outside the JSON object.", "The JSON object must contain: summary (string), assumptions (string[]), warnings (string[]), clarifyingQuestions (string[]), operations (array).", - "Supported operation types: create_table, rename_table, drop_table, add_column, alter_column, drop_column, add_relationship, drop_relationship, add_index, drop_index, add_unique, add_check, drop_constraint, add_primary_key.", + "Supported operation types: create_table, rename_table, drop_table, add_column, alter_column, drop_column, add_relationship, drop_relationship, add_index, drop_index, add_unique, add_check, drop_constraint, add_primary_key, create_enum, add_enum_value.", "create_table.table contains name, optional note, fields, and optional indexes. A field contains name, dataType, and optional nullable, primaryKey, unique, autoIncrement, defaultValue, maxLength, precision, scale, note.", "add_column has tableName and field. alter_column has tableName, columnName and changes (any of newName, dataType, maxLength, precision, scale, nullable, unique, defaultValue, autoIncrement). drop_column has tableName and columnName.", "Every key of the answer's shape is always written: give null for any that does not apply (a field with no default has defaultValue null). To remove an existing column's default, set changes.dropDefault to true; to leave the default alone, leave it null.", "An index contains name, columns (existing column names, in order) and optional unique. add_index has tableName and index; drop_index has tableName and indexName.", - "add_unique has tableName, optional constraintName and columns (existing column names). add_check has tableName, optional constraintName and expression: ONE boolean SQL expression over the table's columns exactly as it follows CHECK, using only immutable functions and no subqueries. drop_constraint has tableName and constraintName of an existing unique or check constraint (use drop_relationship for a foreign key; the primary key is not dropped this way). add_primary_key has tableName, optional constraintName and columns, for a table that has no primary key yet.", + "add_unique has tableName, optional constraintName and columns (existing column names). add_check has tableName, optional constraintName and expression: ONE boolean SQL expression over the table's columns exactly as it follows CHECK, using only immutable functions and no subqueries. drop_constraint has tableName and constraintName of an existing unique or check constraint (use drop_relationship for a foreign key; the primary key is not dropped this way). add_primary_key has tableName, optional constraintName and columns, for a table that has no primary key yet. create_enum has typeName (schema-qualified like tables) and values (the labels, in order); a column can use the new type later in the same plan (dataType is its name). add_enum_value has typeName, value and optionally before OR after (an existing label); an enum value can never be removed again.", "add_relationship has relationship: sourceTable and sourceColumn are the REFERENCED (parent) side, targetTable and targetColumn are the column that HOLDS the foreign key (child side); optional name, onDelete, onUpdate (no_action, cascade, set_null, set_default, restrict).", "drop_relationship has sourceTable, sourceColumn, targetTable, targetColumn, with the same meaning of source and target.", "Table names are exactly as given in the facts, schema-qualified (schema.table). Use existing table, column and index names exactly. New object names may be anything valid.", diff --git a/server/src/schema-change/test-fixtures.ts b/server/src/schema-change/test-fixtures.ts index 76433cb..316e739 100644 --- a/server/src/schema-change/test-fixtures.ts +++ b/server/src/schema-change/test-fixtures.ts @@ -106,13 +106,18 @@ const STABLE = new Set(["now", "current_timestamp", "current_date"]); const IMMUTABLE = new Set(["lower", "upper", "abs", "coalesce"]); /** What the database would say, for tests that don't have one: every type is known, defaults are accepted unless they name a function nobody has, and nothing exists under a new name. */ -export function answer(request: ProbeRequest, overrides: { unknownTypes?: string[]; existing?: string[]; missingSchemas?: string[]; rejectedDefaults?: Record; rejectedChecks?: Record } = {}): Probes { +export function answer(request: ProbeRequest, overrides: { unknownTypes?: string[]; existing?: string[]; missingSchemas?: string[]; rejectedDefaults?: Record; rejectedChecks?: Record; enums?: Record } = {}): Probes { const probes = emptyProbes(); for (const type of [...request.types, ...request.typeChanges.flatMap((c) => [c.from, c.to])]) probes.types.set(type, overrides.unknownTypes?.includes(type) ? undefined : type); for (const change of request.typeChanges) probes.casts.set(castKey(change.from, change.to), CASTS[castKey(change.from, change.to)]); for (const name of request.functions) probes.volatility.set(name, VOLATILE.has(name) ? "v" : STABLE.has(name) ? "s" : IMMUTABLE.has(name) ? "i" : undefined); for (const { expression, type } of request.defaults) probes.defaults.set(defaultKey(expression, type), overrides.rejectedDefaults?.[expression]); for (const { table, expression } of request.checks) probes.checks.set(checkKey(table, expression), overrides.rejectedChecks?.[expression]); + for (const type of request.enums ?? []) { + const key = relationKey(type.schema, type.name); + const known = overrides.enums?.[key]; + probes.enums.set(key, known ? { exists: true, kind: known.kind ?? "e", labels: known.labels } : { exists: false, kind: null, labels: [] }); + } for (const relation of request.relations) probes.relationExists.set(relationKey(relation.schema, relation.name), overrides.existing?.includes(relationKey(relation.schema, relation.name)) ?? false); for (const schema of request.schemas) probes.schemaExists.set(schema, !overrides.missingSchemas?.includes(schema)); return probes; diff --git a/server/src/schema-change/type-change.ts b/server/src/schema-change/type-change.ts index 4c6c798..5e3e8be 100644 --- a/server/src/schema-change/type-change.ts +++ b/server/src/schema-change/type-change.ts @@ -49,7 +49,14 @@ export function parseType(text: string): ParsedType { export const sameType = (a: string, b: string): boolean => { const left = parseType(a); const right = parseType(b); - return left.base === right.base && left.array === right.array && left.args.join(",") === right.args.join(","); + return sameName(left.base, right.base) && left.array === right.array && left.args.join(",") === right.args.join(","); +}; + +/** `public.mood` and `mood` are the same type: PostgreSQL leaves the schema out of a type that is on the search path. */ +const sameName = (a: string, b: string): boolean => { + if (a === b) return true; + const [left, right] = [a.split("."), b.split(".")]; + return (left.length === 1 || right.length === 1) && left[left.length - 1] === right[right.length - 1]; }; export interface TypeChangeVerdict { diff --git a/server/src/schema-change/units.test.ts b/server/src/schema-change/units.test.ts index 57345b6..4d196c5 100644 --- a/server/src/schema-change/units.test.ts +++ b/server/src/schema-change/units.test.ts @@ -68,6 +68,11 @@ describe("type changes", () => { expect(parseType("int4").base).toBe("integer"); expect(sameType("varchar(20)", "character varying(20)")).toBe(true); expect(sameType("varchar(20)", "character varying(21)")).toBe(false); + // a type on the search path is reported without its schema + expect(sameType("public.mood", "mood")).toBe(true); + expect(sameType("mood[]", "public.mood[]")).toBe(true); + expect(sameType("billing.mood", "public.mood")).toBe(false); + expect(sameType("public.mood", "public.moods")).toBe(false); }); it("lists the changes that are catalog-only and treats everything else as a rewrite", () => { diff --git a/server/src/services/schema-plan-client.ts b/server/src/services/schema-plan-client.ts index 29f04f1..c7ab035 100644 --- a/server/src/services/schema-plan-client.ts +++ b/server/src/services/schema-plan-client.ts @@ -15,7 +15,7 @@ function buildInstructions(request: AiSchemaPlanRequest): string { "If a missing requirement could materially change table structure, keys, cardinality, tenancy, or deletion behavior, ask concise clarifying questions and return no operations.", "Use assumptions only for low-risk details that do not materially change the design.", "The JSON object must contain: summary (string), assumptions (string[]), warnings (string[]), clarifyingQuestions (string[]), operations (array).", - "Supported operation types: create_table, rename_table, drop_table, add_column, alter_column, drop_column, add_relationship, drop_relationship, add_index, drop_index, add_unique, add_check, drop_constraint, add_primary_key.", + "Supported operation types: create_table, rename_table, drop_table, add_column, alter_column, drop_column, add_relationship, drop_relationship, add_index, drop_index, add_unique, add_check, drop_constraint, add_primary_key, create_enum, add_enum_value.", "Use the exact nested operation shapes described below; do not put field attributes at the operation root.", "Every key of the answer's shape is always written: give null for any that does not apply. To remove an existing column's default, set changes.dropDefault to true; to leave the default alone, leave it null.", "create_table.table contains name, optional note, fields, and optional indexes.", diff --git a/src/features/schema-change/components/plan-panel.tsx b/src/features/schema-change/components/plan-panel.tsx index 13d0ad1..d8f2f5b 100644 --- a/src/features/schema-change/components/plan-panel.tsx +++ b/src/features/schema-change/components/plan-panel.tsx @@ -155,6 +155,10 @@ function describeOperation(operation: PlanResponse["plan"]["operations"][number] return `check${operation.constraintName ? ` ${get("constraintName")}` : ""} on ${get("tableName")}: ${get("expression")}`; case "drop_constraint": return `drop constraint ${get("constraintName")} on ${get("tableName")}`; + case "create_enum": + return `enum type ${get("typeName")} (${((operation.values as string[]) ?? []).join(", ")})`; + case "add_enum_value": + return `add "${get("value")}" to enum ${get("typeName")}${operation.before ? ` before "${get("before")}"` : operation.after ? ` after "${get("after")}"` : ""}`; case "add_primary_key": return `primary key${operation.constraintName ? ` ${get("constraintName")}` : ""} on ${get("tableName")} (${((operation.columns as string[]) ?? []).join(", ")})`; default: