fix(validators): stop dbt false alarms and concurrent altimate-dbt package corruption

.opencode/skills/dbt-develop/SKILL.md

@@ -168,7 +168,9 @@ altimate-dbt info                                            # confirm every req
 
 **Diff column shape against the spec — use the `dbt-schema-verify` skill.**
 For each model the task touched, run `altimate-dbt schema-verify --model
-<name>` and treat any `mismatch` verdict as "not done." Full procedure,
+<name>` and treat a `mismatch` verdict (an enforced contract the table does
+not match, or a declared column with tests that the table lacks) as "not
+done." Columns the YAML does not list are not a mismatch. Full procedure,
 output interpretation, and fallback (when `altimate-dbt` is missing) live
 in the dedicated **dbt-schema-verify** skill, which auto-loads alongside
 this one.
@@ -204,7 +206,7 @@ Use `altimate-dbt children` and `altimate-dbt parents` to verify the DAG is inta
 5. **Fix ALL errors, not just yours.** After creating/modifying models, run a full `dbt build`. If ANY model fails — even pre-existing ones you didn't touch — fix them. Your job is to leave the project in a fully working state.
 6. **Verify transformation correctness, not just mechanics.** For non-trivial models, generate and run dbt unit tests as part of the validate step (use the `dbt-unit-tests` skill). Passing `dbt build` only proves the SQL is syntactically valid — it doesn't prove the *values* are right.
 7. **Enumerate deliverables, then check them off.** The task is not done until every model, column, test, and config change explicitly requested exists on disk and in the manifest. Re-read the prompt at the end and verify each requested item — don't trust your own intermediate "done" feeling.
-8. **Match the column spec exactly — and verify it mechanically, not by inspection.** Use the dedicated **dbt-schema-verify** skill. Before declaring any model task done, run `altimate-dbt schema-verify --model <name>` and treat any `mismatch` verdict as "not done." Adding "helpful" extras (rank breakdowns, name-resolved fields, lineage metadata), reordering columns "more logically", or substituting synonyms (`supplier_id` for `supplier_company`, `transaction_type_name` for `transaction_type`) all break equality tests. The contract is what the spec says, not what you think would be useful.
+8. **Produce the columns the task asks for — and check declared columns mechanically, not by inspection.** Use the dedicated **dbt-schema-verify** skill. Before declaring any model task done, run `altimate-dbt schema-verify --model <name>` and treat a `mismatch` verdict as "not done." Adding "helpful" extras (rank breakdowns, name-resolved fields, lineage metadata) or substituting synonyms (`supplier_id` for `supplier_company`) can break the task's own expectations: follow the columns the task and the existing consumers of the model call for. YAML that lists only some columns does not mean the others should be removed, and a YAML column nothing tests or enforces may be stale rather than required.
 
 ## Common Pitfalls in Transformation Logic
 

.opencode/skills/dbt-schema-verify/SKILL.md

@@ -4,21 +4,17 @@ applyPaths:
   - "dbt_project.yml"
   - "**/dbt_project.yml"
 description: |
-  REQUIRED after building or modifying ANY dbt model that has columns declared
-  in `schema.yml` / `_models.yml`. Run `altimate-dbt schema-verify --model
-  <name>` to diff actual columns against the spec, and treat any `mismatch`
-  verdict as "not done."
-
-  The most common reason "the build is green but the tests still fail" is
-  that the model produces the right *data values* in the wrong *column
-  shape* — extra columns, missing columns, wrong order, wrong types. Many
-  dbt equality tests grade the column tuple `(name, type, position)`
-  exactly, and the agent's prior bias is to add "helpful" extras
-  (`p1`/`p2`/`p3` rank breakdowns, name-resolved variants, lineage
-  metadata) or reorder columns "more logically." Both break the contract.
-
-  This skill enforces the mechanical check that catches those bugs before
-  declaring done. Use it before declaring any model task complete.
+  Use after building or modifying a dbt model whose columns are declared in
+  YAML (`schema.yml` / `_models.yml`). Run `altimate-dbt schema-verify --model
+  <name>` to compare the built table with the declared columns, and treat a
+  `mismatch` verdict (a non-empty `findings` list) as "not done" until you have
+  decided whether the model or the YAML is wrong.
+
+  YAML `columns:` is documentation plus the anchor for column tests. It is not
+  an exhaustive list of what a model may produce, so columns the YAML does not
+  mention are not an error unless a contract is enforced. What does count: an
+  enforced contract the table does not match, and a declared column that has
+  tests attached but is not produced.
 ---
 
 # dbt schema-verify
@@ -59,6 +55,15 @@ Returns a structured JSON result:
 {
   "model": "int_asana__project_user_agg",
   "verdict": "mismatch",
+  "spec": { "declared_in": "models/schema.yml", "package": "asana", "contract_enforced": false },
+  "findings": [
+    {
+      "kind": "tested-column-missing",
+      "columns": ["number_of_users_involved"],
+      "evidence": "models/schema.yml declares number_of_users_involved with tests (not_null_...), but the model does not produce it"
+    }
+  ],
+  "notes": [],
   "expected_columns": ["project_id", "users", "number_of_users_involved"],
   "actual_columns": ["project_id", "users"],
   "columns_extra": [],
@@ -72,42 +77,49 @@ Returns a structured JSON result:
 
 | verdict | meaning | what to do |
 |---|---|---|
-| `match` | actual columns match the spec exactly (case-insensitive on names) | DONE — proceed |
-| `mismatch` | one or more of `columns_extra`, `columns_missing`, `columns_reordered`, `type_mismatches` is non-empty | NOT DONE — read the diff, fix the model SQL, rebuild, re-run schema-verify |
-| `no-spec` | the model has no columns declared in `schema.yml` | DONE for shape-fidelity purposes — no contract to verify against |
-
-## How to act on a `mismatch`
-
-For each non-empty list, the fix is mechanical:
-
-| Field | What it means | What to change in the model SQL |
-|---|---|---|
-| `columns_extra` | columns in your model NOT in the spec | REMOVE them from the `SELECT` |
-| `columns_missing` | columns in the spec NOT in your model | ADD them to the `SELECT` (compute them, or rename an existing column if you used a synonym) |
-| `columns_reordered` | columns present in both but at different positions | REORDER the columns in your `SELECT` to match the spec's order |
-| `type_mismatches` | declared `data_type` in spec disagrees with the warehouse's reported type | CAST in the `SELECT` or change the upstream source |
-
-Then run `altimate-dbt build --model <name>` again, then re-run
-`altimate-dbt schema-verify --model <name>` until verdict is `match`.
+| `match` | nothing dbt treats as an error was found (`notes` may still hold true observations) | proceed |
+| `mismatch` | `findings` is non-empty: an enforced contract the table does not match, or a declared column with tests attached that the table lacks | NOT DONE until you decide which side is wrong |
+| `no-spec` | the model has no columns declared in YAML | nothing to compare |
+
+`spec` says which YAML file declares the columns, in which package, and
+whether a contract is enforced. Each entry in `findings` carries the
+evidence. `columns_extra` / `columns_missing` / `columns_reordered` /
+`type_mismatches` are the raw diff against the YAML, not instructions:
+`columns_extra` only means "not listed in the YAML".
+
+## How to act on a finding
+
+A finding says two things disagree. It does not say which one is wrong.
+
+- **Contract enforced.** dbt will refuse to build a model whose columns differ
+  from its contract. Either the model should produce exactly the contract's
+  columns, or the contract is out of date. Decide from the task.
+- **Declared column with tests, not produced.** The tests read a column that
+  is not there and will fail or error. Either the model should produce the
+  column, or the YAML entry (and its tests) is stale. Decide from the task.
+- **Never** delete or invent a column just because the YAML does or does not
+  list it. Undocumented columns are normal, and a declared column with no
+  tests and no contract (a `note`) may simply be stale. If the task states
+  which columns the output must have, follow the task.
+
+Then run `altimate-dbt build --model <name>` again and re-run
+`altimate-dbt schema-verify --model <name>`.
 
 ## Iron Rules
 
 1. **The verdict is the source of truth, not your inspection.** Reading the
    columns yourself and concluding "looks right to me" does not count.
    Run the command and read its output.
 2. **A `mismatch` is "not done", even if the build is green.** dbt build
-   only proves the SQL compiled and ran without errors. It does not prove
-   the column shape is correct. Equality tests grade shape AND values.
-3. **Do not reinterpret the spec to make the model right.** The spec is
-   the contract. If the spec lists `supplier_company` and your model has
-   `supplier_id`, the answer is to fix your model, not to argue that
-   `supplier_id` is more useful.
+   does not validate the full declared column list unless a contract is enforced;
+   attached tests still run and check the columns they reference.
+3. **Do not change the model to satisfy YAML that only documents some
+   columns.** The YAML is evidence about intent, not the spec of the whole
+   table. The task and the existing consumers of the model decide which
+   columns are wanted.
 4. **Run schema-verify on every model touched, not just the last one.**
-   The most common "almost-pass" is N-1 models passing and the Nth one
-   silently failing on column shape. Walk the list.
-5. **Skip only on `no-spec`.** Do not skip on the grounds that the model
-   is small, or trivial, or "obvious." The spec is small only because
-   the dbt project author already curated it.
+5. **`no-spec` and `match` with notes need no action.** Read the notes; act
+   on them only if they match what the task asked for.
 
 ## Fallback when altimate-dbt is unavailable
 
@@ -123,10 +135,9 @@ cat models/**/*.yml | grep -A 50 "name: <name>"   # or: yq eval '...' models/**/
 dbt show --select <name> --limit 0
 ```
 
-Compare the two ordered lists. Produce the same four-bucket diff
-(`columns_extra`, `columns_missing`, `columns_reordered`,
-`type_mismatches`) in your head, and apply the same fix logic. The
-mechanics don't change; only the tool name does.
+Compare the two lists and apply the same rules: only an enforced
+contract, or a declared column with tests that the table lacks, is a
+problem. Columns the YAML does not list are not.
 
 ## What this skill does NOT cover
 

docs/docs/data-engineering/tools/dbt-tools.md

@@ -168,6 +168,8 @@ All commands provide friendly error diagnostics with actionable fix suggestions
 
 > **Tip:** In builder mode, the agent prefers `altimate-dbt` over the raw `dbt_run` tool for better error handling and auto-detection.
 
+**Packages.** If the project declares packages (`packages.yml`, `dependencies.yml`), `altimate-dbt` installs them when an adapter-backed command starts (not `doctor`), but only when they are missing or out of date (a package directory is absent or has no `dbt_project.yml`, or the declarations changed since the last install). Installation is serialised across processes with a lock under `target/.altimate-dbt/`, so several `altimate-dbt` commands started together install at most once and cannot corrupt `dbt_packages/`. A lock left by a crashed process is broken automatically (owner process gone, or no heartbeat for 30 seconds) and the interrupted install is repeated. `altimate-dbt deps` always runs `dbt deps`, under the same lock. Because every command re-parses the project and dbt rewrites `target/manifest.json` in place, a command that reads the manifest while a sibling rewrites it re-reads it (up to six times, with backoff) before reporting it missing.
+
 ---
 
 ## dbt Skills

docs/docs/data-engineering/validators.md

@@ -53,24 +53,71 @@ deterministic-validators work and are currently soak-tested in shadow mode
 
 After the agent declares done, runs `altimate-dbt test --model <name>`
 against every dbt model the agent modified during this session. Refuses
-to terminate if any model's tests fail or error.
+to terminate if a model's tests fail, or if tests could not be run
+because dbt itself errored.
 
 **Catches**: row-data correctness errors (`relationships`, `unique`,
 `not_null`, `accepted_values`, `AUTO_*_equality` tests) — the kind of
 bug that column-shape verification cannot detect because the schema can
 be green while the SELECT logic produces wrong values or wrong row
 counts.
 
+**What counts as what**
+
+| dbt result for the model | Validator |
+|---|---|
+| Tests ran and all passed | passes |
+| Tests ran and some failed or errored | **fails**, naming captured failing tests when available |
+| dbt selected no tests (it prints `Nothing to do` and no summary) | passes, and sends nothing. A model with no tests has nothing to check. Recorded as `no_tests` in telemetry. |
+| The selector matched no enabled node | passes (`unmatched_selector_models` in telemetry) |
+| dbt aborted (connection, lock, compile or runtime error) before or while running tests | **fails**, with dbt's error text |
+
+An error from a run that happened in parallel with other `altimate-dbt`
+processes is retried once, alone, before it is reported, because several
+processes sharing a single-writer warehouse such as DuckDB can fail each
+other without anything being wrong with the model (`retried_serially` in
+telemetry). Timeouts and spawn failures are not retried.
+
 ### `dbt-schema-verify`
 
 After the agent declares done, runs `altimate-dbt schema-verify --model
-<name>` on every modified model. Reports a mismatch if the produced
-column shape diverges from the `schema.yml` spec (extra, missing,
-reordered, or type-mismatched columns).
-
-**Catches**: column-level drift that wouldn't be caught by `dbt build`
-alone — equality tests against the spec would fail later but the
-agent has already declared done.
+<name>` on every modified model and compares the built table with the
+columns its YAML declares. It reports only what dbt's own semantics
+establish as a problem, and says which file declares what:
+
+- **Enforced contract.** The model has `contract: {enforced: true}` and
+  the built table has columns the contract does not list, or lacks columns
+  it lists. dbt itself rejects such a model. Only column names are compared
+  here; `dbt build` is what checks a contract's data types.
+- **Declared column with tests, not produced.** The YAML declares a column
+  that has at least one test attached, but the built table does not have it.
+  The test reads a column that does not exist.
+
+It does **not** report, and never tells the agent to remove, add or
+reorder anything because of:
+
+- columns the model produces that the YAML does not list. YAML commonly
+  documents only some columns (those with tests or descriptions); that is
+  not an exhaustive list unless a contract is enforced;
+- a declared column with no tests and no contract that the model lacks.
+  This may be a stale or aspirational entry, so it is only a note in the
+  `altimate-dbt schema-verify` output;
+- column order, or a declared `data_type` that differs from the warehouse
+  type outside a contract (dbt does not enforce either, and adapters spell
+  types differently);
+- a model with no YAML entry or no declared columns (`no-spec`).
+
+The message states the evidence and leaves the decision with the agent
+(change the model if the YAML is right, or the YAML if its entry is
+stale). The full classification appears in `altimate-dbt schema-verify`
+and `altimate-dbt build --model` as `findings` (established problems),
+`notes` (true observations) and `spec` (declaring file, package, contract
+flag). The project-wide `altimate-dbt build` returns these detailed fields
+only for models with a mismatch; matching models are summarised as a count.
+`columns_extra`, `columns_missing`, `columns_reordered` and
+`type_mismatches` remain as the raw diff against the YAML and are not
+instructions. A tool error is still a
+failure (drift cannot be ruled out) and now includes the first error text.
 
 ## Completion gates (shadow mode)
 
@@ -234,6 +281,10 @@ When a validator runs, it returns:
     // validator-specific extras:
     // dbt-tests-pass:
     passed?: number
+    no_tests?: number
+    no_tests_models?: string[]
+    unmatched_selector_models?: string[]
+    retried_serially?: number
     failed?: number
     errored?: number
     spawn_failures?: number
@@ -244,13 +295,14 @@ When a validator runs, it returns:
     match?: number
     mismatch?: number
     no_spec?: number
+    diff_not_established?: number
     mismatch_models?: string[]
   }
 }
 ```
 
 `reason` names the failing models inline (e.g. `"2 of 3 models you
-edited have a column-shape mismatch against schema.yml: foo, bar"`).
+edited contradict the YAML that declares their columns: foo, bar"`).
 
 ## Phased rollout plan
 
@@ -284,9 +336,11 @@ default-on. Track progress in
   deeper (workspace layouts) are not detected.
 - Multiple `dbt_project.yml` candidates pick the alphabetically-first
   match deterministically.
-- The validator surfaces "schema mismatch" even when the real cause
-  is "model never materialized" — distinguishing these requires
-  changes inside `altimate-dbt`.
+- A model that was never materialized is reported as a tool error
+  ("has no warehouse table. Build it first"), not as a column finding.
+- Retries consume the shared validator budget (three by default) whether
+  or not the failure text changed. A failure that is re-sent verbatim on
+  every retry spends the budget without giving the agent new information.
 
 ## Writing custom validators
 

packages/dbt-tools/package.json

@@ -9,11 +9,12 @@
   "scripts": {
     "build": "bun build src/index.ts --outdir dist --target node --format esm && bun run script/copy-python.ts",
     "typecheck": "tsc --noEmit",
-    "test": "bun test test/cli.test.ts test/config.test.ts test/dbt-cli.test.ts test/dbt-resolve.test.ts test/build-integrity.test.ts --timeout 30000",
+    "test": "bun test test/cli.test.ts test/config.test.ts test/dbt-cli.test.ts test/dbt-resolve.test.ts test/build-integrity.test.ts test/packages.test.ts test/manifest-retry.test.ts test/schema-verify.test.ts test/build.test.ts --timeout 30000",
     "test:e2e": "bun test test/e2e/ --timeout 300000"
   },
   "dependencies": {
-    "@altimateai/dbt-integration": "^0.2.2"
+    "@altimateai/dbt-integration": "^0.2.2",
+    "yaml": "2.8.3"
   },
   "devDependencies": {
     "@tsconfig/bun": "catalog:",