Switch to using goose for migrations

[iplay88keys]

Warning

This is an intentional clean slate for 1.0:

• Existing 0.10.x databases require a new PostgreSQL database; there is no migration bridge.
• Downgrade from 1.0 to 0.10.x is unsupported.
• Starting with 1.0, migrations are append-only and future minor releases retain compatibility with the previous release line.

Summary

Replace golang-migrate with Goose and establish a clean PostgreSQL migration baseline for Kagent 1.0.

• Capture the current core and optional vector schemas as direct version 1 baselines.
• Commit each schema change and migration record atomically.
• Simplify the database migration CLI around Goose.
• Preserve Helm-controlled vector migrations and out-of-band migration management.
• Add upgrade coverage that begins enforcing release-line compatibility after 1.0 is published.

Why

golang-migrate records a dirty version separately from the migration transaction. A failure between the schema commit and version update can leave committed schema marked dirty and require an operator to force a version.

Goose records both in the same PostgreSQL transaction, so a failed migration rolls back cleanly and startup can retry it. Substrate PR #1196 made the same change; this review thread describes the failure mode.

Changes

Migration engine and baseline

• Replace github.com/golang-migrate/migrate with github.com/pressly/goose/v3.
• Replace the replay of split core and vector migrations with direct final-schema 000001_initial.sql baselines.
• Require Up and Down sections and reject NO TRANSACTION migrations.
• Validate every source before applying any migration.
• Reject legacy golang-migrate ledgers with a clear fresh-database error.
• Permit non-destructive startup when the database is ahead of the binary for rolling compatibility.
• Update migration immutability CI to allow this one-time cutover, then protect merged Goose migrations.

Controller and CLI

• DATABASE_VECTOR_ENABLED=true includes vector migrations and registers pgvector types on database connections.
• SKIP_MIGRATIONS=true verifies migration state without applying migrations.
• Register both settings in the existing environment-variable registry so kagent env --component database documents them and the CLI reuses the same vector-setting name.
• Replace the CLI's dirty-state and force behavior with Goose-backed up, down, goto, status, and version commands.
• Reject destructive CLI operations when the database is newer than the CLI's embedded migrations.

Upgrade coverage

• Update normal and rolling upgrade suites for Goose state, data survival, previous-release behavior against target migrations, clean-install schema equivalence, and application/schema rollback.
• Use the API v2 AgentInstance interaction E2E; document that its second request verifies the Actor wakes after the first terminal task quiesces it.
• Add adjacent and prev-stable matrix legs. The setup skips 0.10.x because it predates Goose; there will be no 0.11.x. Once a 1.0 prerelease or final tag exists, the previous-stable leg automatically begins enforcing compatibility with 1.0.

Verification

• Applied current origin/main migrations and the Goose baselines to separate fresh databases; normalized schema-only dumps matched exactly after excluding their intentionally different migration ledgers.
• make -C go sqlc-generate
• From go/: go test ./core/pkg/migrations ./core/cli/internal/db/migrate ./core/test/upgrade ./core/internal/database ./core/pkg/app ./core/cmd/controller
• From go/: focused short, race, vet, CLI, and E2E compile checks.
• make -C go lint
• bash scripts/version-resolution_test.sh