Fixes #31857, #31858: preserve table constraints and column custom properties on recursive CSV import

[sonika-shah]

Problem

A recursive CSV import of a Database Service / Database / Database Schema silently loses two kinds of metadata on every table it touches — even though neither is represented in the CSV:

1. All table constraints — PRIMARY KEY, UNIQUE, and FOREIGN KEY (incl. the referenced-table linkage). Fixes CSV recursive import wipes table constraints (primary key / unique / foreign key) #31857.
2. Column-level custom properties (column.extension). Fixes CSV import at schema/database/service level wipes column-level custom properties (column.extension) #31858.

Both are user-reachable from the UI: Manage (⋮) → Export/Import on a Service / Database / Schema page issues importAsync?...&recursive=true, so a normal parent-level re-import in the UI drops them. Reproduced on 1.12.14 and on current main / 2.0.

Root cause

Both come from the field set EntityCsv.createTableEntity(...) uses to load the existing table before the recursive createEntity(...) → repository.createOrUpdate(...):

table = getEntityWithDependencyResolution(
    TABLE, tableFqn, "owners,tags,domains,extension", Include.NON_DELETED);

• tableConstraints/tablePartition were not loaded, so createOrUpdate persisted them as null → constraints dropped.
• column.extension is hydrated by TableRepository.setFields only when both columns and extension are requested; columns was absent, so the persisted columns had extension = null → column custom properties dropped (before the column rows patched back).

The recursive CSV has no column for constraints or column custom properties, so neither is re-added.

Fix

Load tableConstraints, tablePartition, and columns in the createTableEntity fetch so all three are carried through the createOrUpdate unchanged.

Because the recursive import updates tables through the single-entity createOrUpdate path (→ TableUpdater.updateTableConstraints, which add/deletes FK RELATED_TO edges only on a diff) — not the bulk clearEntitySpecificRelationshipsForMany path — an unchanged constraint set leaves the FK relationship edge intact too.

Tests (DatabaseServiceResourceIT)

Added two recursive export→re-import ITs alongside the existing test_importExportRecursive_withColumnTagsAndGlossaryTerms:

• test_importExportRecursive_preservesTableConstraints — a referenced table + a constrained table with PRIMARY_KEY, UNIQUE, and a FOREIGN_KEY (with referredColumns); asserts all three survive, including the FK linkage.
• test_importExportRecursive_preservesColumnCustomProperties — columns carrying extension; asserts the values survive.

All three recursive ITs pass together (constraints+FK, column custom properties, and the pre-existing tags/glossary regression guard): Tests run: 3, Failures: 0, Errors: 0.

Remaining known gap (out of scope, not a normal scenario): a Tier tag applied directly to a column is dropped on a table-level import — but Tier is an asset/table-level concept and is not normally placed on columns.

Greptile Summary

The PR expands recursive CSV table hydration to preserve constraints, partitions, columns, and column extensions across re-imports.

• Loads table constraints, partition metadata, and columns before updating existing tables.
• Adds recursive round-trip integration coverage for table constraints and column custom properties.

Confidence Score: 4/5

The PR is not yet safe to merge because recursive imports still remove the relationship edge representing a preserved foreign key.

The normal existing-table import path uses the bulk updater, which clears table-to-table RELATED_TO relationships and restores only schema containment; preserving tableConstraints JSON does not recreate the foreign-key edge, and the new test does not inspect that relationship.

Files Needing Attention: openmetadata-service/src/main/java/org/openmetadata/csv/EntityCsv.java; openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java; openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/DatabaseServiceResourceIT.java

Important Files Changed

Filename	Overview
openmetadata-service/src/main/java/org/openmetadata/csv/EntityCsv.java	Expands existing-table hydration during recursive CSV imports so metadata absent from the CSV is carried into persistence.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/DatabaseServiceResourceIT.java	Adds recursive import/export coverage for primary, unique, and foreign-key constraint data and column custom properties.

_{Reviews (3): Last reviewed commit: "Fixes #31858: preserve column custom pro..." | Re-trigger Greptile}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue CSV recursive import wipes table constraints (primary key / unique / foreign key) #31857 is not added to the Shipping project.
• Linked issue CSV import at schema/database/service level wipes column-level custom properties (column.extension) #31858 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Expands recursive CSV import table hydration to preserve table constraints, partitions, and column custom properties. No issues found.

✅ 1 resolved

✅ Quality: FK test asserts JSON only, not the RELATED_TO edge

📄 openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/DatabaseServiceResourceIT.java:814 📄 openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/DatabaseServiceResourceIT.java:888-898 📄 openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/DatabaseServiceResourceIT.java:871-885
The comment at line 814 states this table exercises "the table-to-table RELATED_TO edge path," but the assertions (lines 894-898) only verify the FOREIGN_KEY constraint's referredColumns JSON survives — they never confirm the actual table-to-table relationship edge is recreated after a recursive import. Per the PR discussion, recursive batch updates can delete the RELATED_TO edge without recreating it, so this test could pass while the relationship remains broken. Consider asserting the relationship edge exists (e.g. fetch the referenced table and check its inbound/related relationships, or query the relationships field) so the test matches its stated intent.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit 940093e09f88135b8b2e4cbb5a5480d4ec3f1032 in Playwright run 32459274580, attempt 1.

✅ 553 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 48m 19s

⏱️ Max setup 4m 5s · max shard execution 14m 35s · max shard-job elapsed before upload 17m 55s · reporting 3s

🌐 215.41 requests/attempt · 2.83 app boots/UI scenario · 16.00% common-shard skew

Optimization targets still in progress:

• Common shard skew was 16% (convergence target: at most 15%).
• Browser traffic was 215.41 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 2.83 per UI scenario (1622 boots / 574 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	103	0	0	0	0	0
✅ Shard chromium-02	103	0	0	0	0	0
✅ Shard chromium-03	91	0	0	0	0	0
✅ Shard chromium-04	102	0	0	0	0	0
✅ Shard data-asset-rules-01	61	0	0	0	0	0
✅ Shard domain-isolation-01	16	0	0	0	0	0
✅ Shard global-state-01	34	0	0	0	0	0
✅ Shard ingestion-01	1	0	0	0	0	0
✅ Shard reindex-01	2	0	0	0	0	0
✅ Shard search-01	11	0	0	0	0	0
✅ Shard search-rbac-01	29	0	0	0	0	0

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace