diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index dc552fbb..416baefe 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -148,7 +148,7 @@ content was changed. The numbered section list above describes the # Open Issues -**At a glance** — 14 open pyegeria items, 7 awaiting re-test, 5 open Egeria Server issues (re-sorted 2026-10-28). +**At a glance** — 2 open pyegeria items, 8 awaiting re-test, 7 open Egeria Server issues (checked against merged PRs and the code 2026-10-29; ISSUE-159, 132 and 85 live-verified the same day). **Egeria team note, 2026-10-18 (Mandy Chessell).** Saved verbatim at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`. Each entry below has a dated @@ -156,7 +156,7 @@ content was changed. The numbered section list above describes the were re-sorted: fixes released since 6.2.2 moved to the Appendix, and the re-test candidates moved into *Awaiting re-test* below the open sections. - **Fixed on Egeria's `oak2026` branch, pending merge to `main`:** ISSUE-147, - 143, 130, 132, 128, 38. Re-test once it is merged and a platform is built + 143, 130, 128, 38 (ISSUE-132 verified live and closed 2026-10-29). Re-test once it is merged and a platform is built from `main`. - **Fixed on Egeria `main`, re-test against a platform built from `main`:** ISSUE-125, 124, 117, 112, 95, 90, 89. @@ -166,27 +166,17 @@ re-test candidates moved into *Awaiting re-test* below the open sections. copied into each entry). - **pyegeria follow-ons** (class renames, `cascadedDelete`, `pushDown` default, new `DataValueAssignment`/`CrowdSourcingRole` values, view-service - fixes that pyegeria worked around): ISSUE-154, proposed and not yet made. + fixes that pyegeria worked around): ISSUE-154, fixed 2026-10-29 for 6.2.15. - **Gaps against Egeria `main` since 6.2** (omvs_audit plus source diff): - ISSUE-155, a ranked list for the owner to pick from. + ISSUE-155. Items 1, 5 and 6 are done; items 4 (`typeName` on + create-from-template) and 7 (`viewElementGUID`) are still open. **Open pyegeria items** -- ISSUE-85: No REST endpoint exists to create a `SolutionPort` element — only attach/detach/delegation relationships are exposed -- ISSUE-159: no archive wrappers on the project or collection managers, and no restore wrapper at all, although Egeria offers both -- ISSUE-158: `GOV_DEF_PROPERTIES_LIST` lacks `DataQualityRuleProperties`, and Dr.Egeria's governance create fallback silently drops every type-specific attribute -- ISSUE-157: Dr.Egeria `Create Service Level Objective` sends `entitlements`/`obligations`/`restrictions` as empty lists; Egeria types them as maps -- ISSUE-156: `AutomatedCuration.add_catalog_target` posts a flat `"class": "CatalogTargetRequestBody"`, which Egeria does not have; the endpoint takes `NewRelationshipRequestBody` - ISSUE-155: pyegeria gaps against Egeria `main` since 6.2 (omvs_audit + source diff, V6.2..05a5e982) -- ISSUE-154: pyegeria follow-ons from the Egeria team's 2026-10-18 note: renamed property classes, `cascadedDelete`, the `pushDown` default, new `DataValueAssignment`/`CrowdSourcingRole` values -- ISSUE-152: five update methods pass the body into `_async_update_element_body_request`'s `prop` slot, so nothing is sent - ISSUE-150: Dr.Egeria commands for an Analytic Function's Egeria declaration: a GovernanceActionType that Resource Explorer runs as an engine host, plus a synchronous path that bypasses Egeria -- ISSUE-146: Deleting a journal entry fails with OMAG-REPOSITORY-HANDLER-400-008, and pyegeria's `remove_note` reaches the delete only through the generic Asset Maker service -- ISSUE-132: `DeleteElementRequestBody` serializes the cascade flag as `cascadeDelete`; Egeria's field is `cascadedDelete` and unknown fields are ignored, so no pyegeria cascade delete has ever cascaded -- ISSUE-127: `CollectionManager.delete_collection(cascade=True)` silently never sends `cascadeDelete` -- same shape as ISSUE-62 -- ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` -- ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55) **Awaiting re-test** +- ISSUE-146: Deleting a journal entry fails with OMAG-REPOSITORY-HANDLER-400-008, and pyegeria's `remove_note` reaches the delete only through the generic Asset Maker service - ISSUE-125: `deleteEnumDef` answers 500 "unknown TypeDef" for an enum the server lists -- an enum added through the API cannot be removed - ISSUE-124: `updateTypeDef` (Metadata Expert `open-metadata-types/update`) can never succeed -- the server builds the patch without `updatedBy`, which the repository services require - ISSUE-117: Cascade delete that takes the soft-delete (Memento) path fails partway with `OMAG-REPOSITORY-HANDLER-400-010` unless `forLineage=true` — leaves the asset live and an anchored element already soft-deleted @@ -201,7 +191,8 @@ re-test candidates moved into *Awaiting re-test* below the open sections. - ISSUE-130: The YAML secrets store's user directory caches a user forever, so editing an existing user needs a platform restart - ISSUE-128: Retention cannot be set on Egeria 6.2: the server registers the Jackson subtype as `RetentionProperties.class` (typo), and pyegeria only allows `RetentionClassificationProperties` - ISSUE-108: newly created relationships stay invisible to related-element queries for up to ~20 minutes after creation - +- ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` +- ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55) --- @@ -214,138 +205,10 @@ into the entry now, so it isn't rediscovered from scratch later. --- -### ISSUE-85: No REST endpoint exists to create a `SolutionPort` element — only attach/detach/delegation relationships are exposed - -**Status:** open, now a pyegeria gap (Egeria added the endpoints 2026-09-06) -consumer (trellis/Resource Explorer, designing a `PortMaterializer` for -architecture-recovery proposals) hitting the same shape of gap ISSUE-84 -turned out not to be: this time the `.http` ground truth's absence really -does match the live server, not a stale/incomplete `.http` file. - -**Confirmed at every layer:** -- `pyegeria/omvs/solution_architect.py`: `link_solution_component_port` - (attach an *existing* port to a component), `link_solution_port_delegation` - (port-to-port delegation), and both detach counterparts exist. No - `create_solution_port`/`_async_create_solution_port` anywhere. -- `Egeria-api-solution-architect.http`: only `.../solution-components/ - {guid}/solution-ports/{guid}/attach|detach` and `.../solution-ports/ - {guid}/port-delegations/{guid}/attach|detach`. No `POST .../solution-ports` - create path. -- `md_processing/data/compact_commands/commands_solution_architect_compact.json`: - only `Link Solution Component Port`/`Link Solution Port Delegation`. No - `Create Solution Port` command. -- `pyegeria/models/models.py`: no `SolutionPortProperties` model at all — - so even the generic fallback route has nothing typed to build a body - against. - -**Live-verified not just a stale-`.http`-file case** (the precedent this -repo already has for that shape of false alarm — see the multi-link audit -above, where 3 of 4 similarly-unwrapped relationship types turned out to -have zero real endpoint on live-server inspection while others did): -`dwolfson-1b`/trellis fetched the full live OpenAPI spec directly -(`GET /v3/api-docs`, 2.39MB) and searched every path containing "port" — -30 POST paths matched, every one of them attach/detach (component-port, -port-delegation, even a generic process-ports attach under a different -`{urlMarker}` service). Zero creation endpoints anywhere in the spec. -This confirms the `.http` ground truth exactly; not a documentation gap. - -**The generic fallback exists but isn't usable without guessing:** -`/servers/{serverName}/api/open-metadata/{urlMarker}/metadata-elements` -(`MetadataExpert.create_metadata_element`, -`NewOpenMetadataElementRequestBody`) is real and live-confirmed present, -with a top-level `type_name` field that could carry `"SolutionPort"`. But -per this file's own `pyegeria/models/models.py` gotcha, that endpoint's -`properties` field needs the verbose nested `ElementProperties`/ -`propertyValueMap` shape server-side, not the flat dict every bespoke -wrapper uses — and with no `SolutionPortProperties` model to check -property names against, building that body means hand-guessing what -properties a `SolutionPort` actually takes. Consistent with this repo's -established practice (see the multi-link audit's "left unbuilt rather -than guessing at undocumented URLs or building against the awkward -generic path speculatively") — not attempted. - -**Nothing to fix in pyegeria until Egeria adds a real create endpoint.** -Once one exists, the follow-on work is small and well-precedented: a -`create_solution_port`/`_async_create_solution_port` wrapper in -`SolutionArchitect` following the exact pattern ISSUE-84 just corrected -(`NewElementRequestBody` + `contentStatus` inside `properties`, no -fictional separate request-body class), plus a `Create Solution Port` -Dr.Egeria compact command — the family's registration plumbing in -`register_solution_architect_processors()` (`md_processing/dr_egeria.py`) -already handles this shape of addition with no new routing logic needed. - -**Not blocking anything urgent** on the trellis/Resource Explorer side — -they're holding real port materialization until this exists server-side -(tracked as their own Backlog item 8), or until a workaround surfaces. - - -**Egeria side addressed (2026-10-05 check, carried over from the retired `docs/egeria-open-bugs-2026-10-05.md`):** at `449ad06894`, Solution Architect has `createSolutionPort` and `createSolutionPortFromTemplate`, with endpoints `POST /solution-ports`, `/solution-ports/from-template`, `/solution-ports/{guid}/update`, `.../delete`, `/by-name`, `/by-search-string` and `.../retrieve`. They first appear in commit `571034df11` (2026-09-06). The running platform answers `POST /solution-ports/by-name` with 200. **Remaining (pyegeria):** the `.http` snapshot lists none of these, and `pyegeria/omvs/solution_architect.py` has no `create_solution_port` (or update, delete, find). Add them, ideally with Dr.Egeria support. ---- - - -### ISSUE-159: no archive wrappers on the project or collection managers, and no restore wrapper at all, although Egeria offers both - -**Layer:** Pyegeria · **Status:** open, logged only (no fix without the owner's word) · **Found:** 2026-10-20, raised by the Resource Explorer expansion-architecture session: RE's investigation-delete ruling needs a Project archived through the project manager and restored by GUID. Facts confirmed here against Egeria `origin/main` 05a5e982 and pyegeria upstream `main`, read-only. - -**Egeria:** `OpenMetadataAPIGenericHandler.java` (common-services/generic-handlers): -- `archiveBeanInRepository` (`:3289`) classifies the element Memento, renames each affected Referenceable's qualifiedName to `_archivedOn_` (marker `archivedQualifiedNameMarker = "_archivedOn_"`, `:51`), and records one `lastOperationId` per operation. -- `restoreBeanInRepository` (`:4170`) restores an element and everything the same delete or archive changed, given any GUID that operation touched. - -The OMF store's delete (`OpenMetadataStoreRESTServices`, around `:4118`) also routes `deleteMethod: ARCHIVE` to `archiveMetadataElementInStore`, and resolves `LOOK_FOR_LINEAGE` to ARCHIVE when lineage relationships exist. - -**pyegeria:** -- Archive exists only as `MetadataExpert.archive_metadata_element` (`metadata-expert/metadata-elements/{guid}/archive`). -- There is no `archive_project`, `archive_collection` or other manager-level archive. -- There is no `restore_*` method anywhere. ISSUE-155 item 5 already lists Metadata Expert `metadata-elements/{guid}/restore` and `related-elements/{guid}/restore` as missing. - -**Possible route today (unverified):** `ProjectManager.delete_project(guid, body={"class": "DeleteElementRequestBody", "deleteMethod": "ARCHIVE"})`. pyegeria passes `deleteMethod` through, but whether the project-manager delete path honours ARCHIVE as the OMF store delete does was not traced. - -**Proposed (not made):** -1. ~~Add `MetadataExpert.restore_metadata_element` / `restore_related_elements` (and undo), per ISSUE-155 item 5.~~ Done 2026-10-23 (branch `fix/role-appointment-class-names`). Items 2 and 3 are still open. -2. Add `ProjectManager.archive_project` and `CollectionManager.archive_collection`. These can be thin wrappers that send `deleteMethod: ARCHIVE` once that path is confirmed live, or call Metadata Expert archive. -3. Live-test archive then restore of a throwaway Project, including the `_archivedOn_` rename and its reversal. - ---- - -### ISSUE-158: `GOV_DEF_PROPERTIES_LIST` lacks `DataQualityRuleProperties`, and Dr.Egeria's governance create fallback silently drops every type-specific attribute - -**Layer:** Pyegeria · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, raised by the TestGen evaluation session while drafting a DataQualityRule convenience-API spec (`~/localGit/egeria-v6/wt-testgen-notes/packages/resource-explorer/docs/design-notes/SPEC-PYEGERIA-DATA-QUALITY-RULE.md`, uncommitted, pending owner review). Both points confirmed in source here, read-only; not run live. - -1. **Whitelist gap.** `GOV_DEF_PROPERTIES_LIST` (`pyegeria/omvs/governance_officer.py:37`) is the client-side property-class whitelist for `create_governance_definition` (`:426`) and `update_governance_definition` (`:955`). It has no `DataQualityRuleProperties`, although Egeria `main` has that class (`frameworks/.../properties/governance/DataQualityRuleProperties.java`). **Corrected 2026-10-19:** the list is enforced only on update. `validate_new_element_request` (`pyegeria/core/_server_client.py:6115`) has its class check commented out, so a create with `DataQualityRuleProperties` reaches Egeria unchanged. `validate_update_element_request` (`:6252`) does check a dict body's class against the list, so `update_governance_definition` refuses a DataQualityRule body client-side ("unexpected property class name"). No Dr.Egeria command creates DataQualityRules today. **Proposed:** add `DataQualityRuleProperties`, and drop the list's duplicates (`RegulationProperties`, `GovernanceControlProperties` and `GovernanceApproachProperties` each appear twice). Better still, check the list against Egeria's `GovernanceDefinitionProperties` `@JsonSubTypes` so it can't drift again. -2. **Silent fallback.** In `GovernanceProcessor`'s create path (`md_processing/v2/governance.py:203-221`), any non-SecurityGroup create that Egeria refuses with `OMAG-COMMON-400-029` (property class not accepted for `createGovernanceDefinition`) is retried with `_to_base_governance_definition_properties` (`:96`). That keeps only the cross-type fields (qualifiedName, displayName, description, category, identifier, contentStatus, versionIdentifier, effective dates, additional/extended properties). The element is created, but every type-specific attribute is dropped (e.g. a DataQualityRule's quality fields, a metric's `measurement`/`target`, a control's `implementationDescription`). The only sign is a `logger.warning`, and Dr.Egeria reports success. **Proposed:** remove the generic fallback, or keep it only for the server-compat case it was written for. Otherwise make it fail visibly, naming the dropped fields in the command's result. Live-test which current types, if any, still hit 400-029 on Egeria `main` before deciding. - ---- - -### ISSUE-157: Dr.Egeria `Create Service Level Objective` sends `entitlements`/`obligations`/`restrictions` as empty lists; Egeria types them as maps - -**Fixed 2026-10-19 (owner approved) on branch `fix/issue-156-157-catalog-target-slo` (956b199c, off upstream main 770dfe31; not pushed, not released).** `set_gov_prop_body` now defaults the three fields to `None`. **Cause confirmed live** on Egeria `main` 05a5e982: the old code got a 400 `com.fasterxml.jackson.databind.exc.MismatchedInputException`, "Cannot deserialize value of type `java.util.LinkedHashMap` from Array value ... ServiceLevelObjectiveProperties[\"entitlements\"]", and created nothing. The fixed code created `ServiceLevelObjective::pyeg_rt_slo` (`6258d428`) with `implementationDescription` intact; it was deleted and verified gone by GUID and by name. Micro-tests: `tests/micro-tests/test_issue_156_157_catalog_target_slo.py`; the full micro-test suite (1,394) passes in a clean python:3.13-slim container. - -**Layer:** Pyegeria (Dr.Egeria) · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, reported by the Egeria team against pyegeria 6.2.5: the command "sends a field the platform rejects", so their hourly refreshes use the content pack's existing within-an-hour SLO instead. The cause below is from reading source; it was not reproduced live, since a wrong guess would create an SLO on the shared platform. The Egeria team's error text would confirm it. - -**Likely cause:** `set_gov_prop_body` (`md_processing/md_processing_utils/common_md_utils.py:588`) always sets `entitlements`, `obligations` and `restrictions` to `attributes...get('value', [])`, which is `[]` when the command has no such attributes. `Service Level Objective` is in `GOVERNANCE_CONTROLS` (`common_md_utils.py:50`), so `update_gov_body_for_type` takes the controls branch, adds `implementationDescription`, and leaves the three `[]` values. Only the `TermsAndConditions`/`CertificationType`/`LicenseType` branch replaces them with maps or `None`. `body_slimmer` keeps empty lists, so `[]` is sent. In Egeria `main` (05a5e982), `ServiceLevelObjectiveProperties extends TermsAndConditionsProperties`, whose `entitlements`/`restrictions`/`obligations` are `Map`. Jackson cannot read a JSON array into a map, so the request is refused. Other governance types do not have these fields, so they silently ignore them (`ignoreUnknown=true`), which is why only SLO fails. - -**Proposed fix (not made):** in `set_gov_prop_body`, default the three to `None` rather than `[]`, or set them only in the TermsAndConditions branch. Route `ServiceLevelObjective` through that branch (it is a TermsAndConditions subtype, not a plain control), keeping `implementationDescription`. Optionally add `Entitlements`/`Restrictions`/`Obligations` (dict-valued) to the SLO command through the Spec Editor API. Then live-test one `Create Service Level Objective` on a throwaway. - ---- - -### ISSUE-156: `AutomatedCuration.add_catalog_target` posts a flat `"class": "CatalogTargetRequestBody"`, which Egeria does not have; the endpoint takes `NewRelationshipRequestBody` - -**Fixed 2026-10-19 (owner approved) on branch `fix/issue-156-157-catalog-target-slo` (956b199c; not pushed, not released).** `add_catalog_target`/`update_catalog_target` keep their signatures but now send `NewRelationshipRequestBody`/`UpdateRelationshipRequestBody` with `CatalogTargetProperties` through the shared relationship helpers. `template_properties` now maps to Egeria's `templates` field; the old body had misnamed it `templateProperties`. **Live on Egeria `main` 05a5e982** (two runs, throwaway undeployed IntegrationConnector `pyeg_rt_ic` and a Collection target, never linked to an integration group): the old flat body was refused; the fixed add returned a relationship GUID and read back name/connection/config; update read back `pyeg_rt_target_v2` and `FROM_THIRD_PARTY` with `connectionName` kept; remove made the relationship unreadable; all throwaways verified gone. The `hey_egeria` `add_catalog_target` command uses this method, so it is fixed too. - -**Layer:** Pyegeria · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, reported by the Egeria team against pyegeria 6.2.5: `add_catalog_target` "sends a request format the platform rejects"; their notebook now works around it. Confirmed from source; not re-run live. - -**What:** pyegeria has two `add_catalog_target` methods for the same URL, `.../integration-connectors/{connectorGUID}/catalog-targets/{elementGUID}`: -- `AutomatedCuration.add_catalog_target(integ_connector_guid, metadata_element_guid, catalog_target_name, connection_name=..., metadata_src_qual_name=..., config_properties=..., template_properties=..., permitted_sync=..., delete_method=...)` (`pyegeria/omvs/automated_curation.py:4120-4190`) builds `{"class": "CatalogTargetRequestBody", "catalogTargetName": ..., "connectionName": ..., ...}` with the properties at the top level. **Broken:** no `CatalogTargetRequestBody` class exists in Egeria `main`. -- `AssetMaker.add_catalog_target(integration_connector_guid, metadata_element_guid, body)` (`pyegeria/omvs/asset_maker.py:1095`) sends `NewRelationshipRequestBody` with `properties.class = "CatalogTargetProperties"`. **Correct.** - -Egeria (`AssetMakerResource.addCatalogTarget`, served under every generic URL marker including `automated-curation`) takes `@RequestBody(required=false) NewRelationshipRequestBody`. `CatalogTargetProperties` has `catalogTargetName`, `metadataCollectionQualifiedName`, `metadataSourceQualifiedName`, `connectionName`, `configurationProperties`, `templates`, `permittedSynchronization` and `deleteMethod`. The `hey_egeria` CLI command `add_catalog_target` (`commands/ops/gov_server_actions.py:45`, registered in `commands/cli/egeria.py` and `egeria_ops.py`) uses the broken one. - -**Proposed fix (not made):** keep `AutomatedCuration.add_catalog_target`'s signature but build `{"class": "NewRelationshipRequestBody", "properties": {"class": "CatalogTargetProperties", "catalogTargetName": ..., "connectionName": ..., "metadataSourceQualifiedName": ..., "configurationProperties": ..., "templates": ..., "permittedSynchronization": ..., "deleteMethod": ...}}` and send it through `_async_new_relationship_request`, as AssetMaker does. Add a micro-test asserting the body shape. `AutomatedCuration._async_update_catalog_target` (`automated_curation.py:4248`, body at `:4301`) has the same flat `CatalogTargetRequestBody` shape against `catalog-targets/{relationshipGUID}/update`. Egeria's `updateCatalogTarget` takes `UpdateRelationshipRequestBody`, so it needs the same fix. - ---- - ### ISSUE-155: pyegeria gaps against Egeria `main` since 6.2 (omvs_audit + source diff, V6.2..05a5e982) +**Status 2026-10-29:** items 1 (Governance Engine moves, #460), 5 (undo/restore) and 6 (usage-activity and popularity classifications) are done and released, and item 2 is fixed under ISSUE-154. Still open: item 4 (`typeName`, which only `OpenMetadataTemplateRequestBody` carries) and item 7 (`GetOptions.viewElementGUID`, low priority). + **Layer:** Pyegeria · **Status:** open, proposed list for the owner to pick from; nothing changed · **Found:** 2026-10-18, Resource Explorer ops item 7. File-only audit, no Egeria calls. **How:** `scripts/omvs_audit.py --quiet --http-dir ` was run twice against this branch's `pyegeria/omvs`: once with the `.http` files from Egeria tag `V6.2` (bc08cf55a8, 2026-10-06), once with those from `origin/main` (05a5e9822e, 2026-10-18). Each was restricted to the collections `pyegeria/http-clients/` already carries, plus the new `Egeria-api-governance-engine.http`. Totals: V6.2 OK=965, mismatch 4, missing 3. Main OK=959, mismatch 5, missing 15. The `.http` files lag the Java, so the view-service `*Resource.java` and request-DTO diffs from V6.2 to `main` were also read, for PRs #9365-#9374 and #9321. The checkouts were not touched; the files were read with `git show`. Ranked: risk to existing callers first, then new capability. @@ -367,67 +230,6 @@ Egeria (`AssetMakerResource.addCatalogTarget`, served under every generic URL ma --- -### ISSUE-154: pyegeria follow-ons from the Egeria team's 2026-10-18 note: renamed property classes, `cascadedDelete`, the `pushDown` default, new `DataValueAssignment`/`CrowdSourcingRole` values - -**Layer:** Pyegeria · **Status:** open, proposed, not made (owner approval needed, per this repo's policy) · **Found:** 2026-10-18, from the Egeria team's note (`~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`) and read-only greps of this checkout. - -Egeria says none of its changes break existing requests: the old class names and spellings are still accepted. So nothing here is urgent, and every change should keep accepting the old names so pyegeria still works against older platforms. Ranked by risk, highest first. - -**1. `cascadeDelete` now cascades (ISSUE-132).** Egeria accepts `cascadeDelete` as an alias for `cascadedDelete`, so any pyegeria call that sends `true` will start cascading on a platform with `oak2026`. Checked: no production caller in `pyegeria/`, `md_processing/` or `commands/` passes `True`; every `cascade`/`cascade_delete` default is `False`; `commands/tech/element_actions.py`'s `--cascade` flag (default True) is not forwarded. Only live tests pass `True`, on purpose. Proposed: serialize `DeleteElementRequestBody.cascade_delete` as `cascadedDelete` (accept both on input) and change the ~25 `"cascadeDelete"` literals (`_server_client.py` ×3, `models.py` ×7, `actor_manager.py` ×8, `metadata_expert.py`, `product_manager.py` ×2, `reference_data.py` ×2, `valid_metadata.py` ×2; several are docstring samples). Then live-test one cascade and re-test ISSUE-127. - -**2. Metadata Expert count default flipped (ISSUE-38).** Egeria's `pushDown` default is now `false` (list-matching, slower); `true` is the fast repository count. `metadata_expert.py:3698` and `:4013` document omission as `pushDown=True`. `pyegeria/view/overview_metrics.py` calls both counts without `push_down`, so its statistics silently become the slower kind. Proposed: fix both docstrings; owner to choose whether `overview_metrics` sends `push_down=True` (Egeria's advice for statistics) or keeps the list-matching default (which matches its own `len(find)` fallback). - -**3. Renamed property classes.** Responses now carry the new names in `class`. Checked: no pyegeria response parsing compares a response's `class` with any of the renamed names. The only matches are request-side `prop=[...]` whitelists, which refuse a caller-built body that uses the *new* name: - -| Old → new | Where pyegeria uses the old name | Proposed | -|---|---|---| -| `RetentionClassificationProperties` → `RetentionProperties` | `omvs/classification_explorer.py:8851,8878` (whitelist),`8919`; `md_processing/v2/processors.py:220-237` (comment says the opposite); `md_processing/v2/curation.py:136` | Send `RetentionProperties`, accept both in the whitelist, rewrite the comment (ISSUE-128) | -| `SecurityGroupMembershipProperties` → `SecurityListMembershipProperties` | `omvs/actor_manager.py:3602,3617` (whitelist),`3654,3703,3753` | Send the new name, accept both | -| `AssetSchemaTypeProperties` → `SchemaProperties` | not used; `schema_maker.py:822` already whitelists `SchemaProperties` | none | -| `AssociatedSecurityGroupProperties` → `AssociatedSecurityListProperties` | not used; `md_processing/v2/governance.py:620` already sends the new name | none | -| `ReferenceCodeMappingProperties` → `ReferenceCodeMappingTableProperties` | not used | none | -| `GovernanceClassificationBase` → `GovernedDataClassificationBase` | only in a description string, `commands_curation_compact.json:2256` | optional wording fix through the Spec Editor API | - -`tests/functional-tests/test_classification_manager.py:1233` already sends `RetentionProperties`. - -**4. `DataValueAssignment` now accepts `label` and `description`.** Dr.Egeria's `Assign Data Value Specification` (`md_processing/v2/data_designer.py:1026-1037`) replaces `properties` with a fixed field list, and the `Data Value Assignment Base` bundle has no Label/Description. Proposed (feature, not a fix): add `Label` and `Description` to that bundle through the Spec Editor API, map them in the processor, run `refresh_specs`. - -**5. `CrowdSourcingRole` values now work** (PROPOSER, REVIEWER, SUPPORTER, APPROVER, OTHER). pyegeria's `link_crowd_sourcing_contribution` (`omvs/community_matters_omvs.py:479`) passes the body through, and no Dr.Egeria command sets a role. Proposed: list the values in the docstring; nothing else. - -**6. ISSUE-147's 400-011 workaround:** none exists. Nothing to drop. - -**7. View-service fixes pyegeria may have worked around.** The note lists 48 view-service defects found by Egeria's new view-services FVT. A read-only sweep of `pyegeria/`, `md_processing/` and `commands/` (2026-10-18) found almost nothing worked around, and no issues-file entry for any of them. The detach, Classification Explorer, Metadata Expert, search-option, history-order and delete/purge rows have no client-side workaround: the wrappers call the endpoint once and pass options straight through. So pyegeria users met those defects as calls that "succeeded" but did nothing, and the fixes need no pyegeria change. Candidates to review: -- **Project hierarchy:** `commands/cat/get_project_structure.py:105-116` (and `get_project_dependencies.py:107`) walk the tree with `get_linked_projects()` and keep only `ProjectHierarchy` links, instead of calling `ProjectManager.get_project_hierarchy`. Not commented as a workaround. Optional: switch to `get_project_hierarchy` after re-testing it on a platform built from `main`. -- **Term classifications with no body:** `glossary_manager.py:2686-2692` (is-question) and `4059-4062`/`4116`/`4170` (prime word, modifier, class word) always send a default body. The comment blames pyegeria's own helper (ISSUE-31), not the server, so keep them; harmless either way. -- **`linkSolutionLinkingWire` with no body:** `md_processing/v2/solution_architect.py:856-860` always sends a body and skips recording the GUID when none returns. Keep it; no change needed. -- **Asset from template:** Dr.Egeria's generic `Create Element` (`md_processing/v2/asset_maker.py:57-75`) uses Automated Curation `catalog-templates/new-element`, not Asset Maker `assets/from-template`, so it never hit the "always an Asset" bug. No change. -- **Missing connector now refused** (Security Officer, Runtime Manager): calls that used to report success with nothing done will now raise. Callers should expect the error. ISSUE-112 is the same shape for client-side secrets. -- **Delete with `forLineage`:** a delete that needs `forLineage=true` (Memento) is now refused whole instead of half-done. pyegeria's editing deletes already default to `forLineage=true` via `lineage_visible()`, so no change. - ---- - -### ISSUE-152: five update methods pass the body into `_async_update_element_body_request`'s `prop` slot, so nothing is sent - -**Layer:** Pyegeria · **Status:** open (found 2026-10-15 by Resource Explorer's ClassificationAnnotation backfill; not fixed, per this repo's policy) - -**Fixed 2026-10-20 (owner approved) on branch `fix/issue-152-update-body-slot`, not yet merged or released.** All five call sites now pass `body=body`. `tests/micro-tests/test_issue_152_update_body_slot.py` sends a dict body and an `UpdateElementRequestBody` through each method and checks that the body reaches the HTTP layer. All 10 cases fail on the old lines and pass on the fix, and the full micro-test suite (1,404) passes in a clean python:3.13-slim container. Not live-tested. Hit again 2026-10-20 by the TestGen evaluation session (RE #791, `update_annotation`), which carries its own workaround until a release ships this. - -`_async_update_element_body_request(self, url, prop=None, body=None)` (`pyegeria/core/_server_client.py:7317`) takes the body as its **third** parameter. Five callers pass it positionally as the second, so it lands in `prop` and `body` is `None`; the call fails before any request goes out (the builder observed an AttributeError): - -- `pyegeria/omvs/data_discovery.py:299` (`_async_update_annotation`, behind `update_annotation`) -- `pyegeria/omvs/time_keeper.py:342` -- `pyegeria/omvs/schema_maker.py:131` and `:174` -- `pyegeria/omvs/subject_area.py:302` - -Checked at egeria-python `f1b06cdd`; the installed wheel in trellis has the same line in `data_discovery.py`. - -**Fix (one token each):** `await self._async_update_element_body_request(url, body=body)`. A test per method that the body reaches the HTTP layer would catch a repeat. - -**Workaround in Resource Explorer:** `scripts/backfill_classification_annotations.py` calls `_async_update_element_body_request(url, body=body)` directly, and pins this bug with a test that fails once pyegeria is fixed. - ---- - - ### ISSUE-150: Dr.Egeria commands for an Analytic Function's Egeria declaration: a GovernanceActionType that Resource Explorer runs as an engine host, plus a synchronous path that bypasses Egeria **Layer:** Dr.Egeria (commands over existing Action Author / governance commands) · **Status:** open, direction agreed with the owner 2026-10-13 and **confirmed by Mandy 2026-10-14**, not yet built · **Found:** 2026-10-13, handed off by the Resource Explorer PR/CI session. Source: trellis `docs/extensibility-walking-skeleton-brief.md` §2c, §3a, §8. Egeria references: [Governance Action Processes (0462)](https://egeria-project.org/types/4/0462-Governance-Action-Processes/), [Survey Action Engine](https://egeria-project.org/concepts/survey-action-engine/). @@ -514,266 +316,393 @@ Checked at egeria-python `f1b06cdd`; the installed wheel in trellis has the same -### ISSUE-146: Deleting a journal entry fails with OMAG-REPOSITORY-HANDLER-400-008, and pyegeria's `remove_note` reaches the delete only through the generic Asset Maker service +## Open Egeria Server issues (not fixable in pyegeria) -**Egeria reset 2026-10-23 ~21:10-23:54Z** (egeria-workspaces-fs, owner-approved): the quickstart database was wiped and Egeria recreated on build 81beb8e5 (odpi/egeria 4486d1a7), then Coco content was reloaded. Every GUID recorded before then is gone. The test JournalEntries `fef83ed5` and `8bec72e4` on Erin's profile no longer exist, so the pending `remove_note` re-test below needs a NEW entry made with `journal_my_activity` (and, for the 400-008 case, one owned by an external source). The 2026-10-18 Egeria analysis (works as designed; 400-008 is the provenance check) still applies. -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: works as designed (Egeria).** Asset Maker's `deleteAsset` is the right call: a JournalEntry is a Notification, so an asset, and no separate note delete is needed. Egeria's new FVT creates entries with My Profile's `journal-my-activity` and deletes them with `deleteAsset`, with no external source; every case passes on in-memory and PostgreSQL repositories. **400-008 is the provenance check:** it refuses a change made without an external source to an element an external source owns. No external source → 400-008; a different external source → 400-007; the matching external source → success. So an entry that returns 400-008 was not created locally, and its delete needs that entry's `externalSourceGUID` and `externalSourceName`. If 400-008 appears on an entry created through My Profile, that is a bug and Egeria wants: (1) the full error message (it names the type, GUID, provenance and metadata collection of the refused instance); (2) how the entry was created (which request, and which external source if any). Also new: purge applies the same provenance check. **pyegeria:** the `remove_note` clean-up below stays (route + docstring). Re-test on `8bec72e4` against a platform with the fixes; if 400-008 recurs, capture the full message for Egeria, since that entry was created locally through `journal-my-activity`. +These need an Egeria Server-side fix, an upstream bug report, or a +deployment/config change — nothing here can be resolved by editing this +repo. Where an entry has a known pyegeria/Dr.Egeria follow-on once the +Egeria side lands, it's noted inline (also cross-referenced from the +"Open pyegeria items" section below where the follow-on is substantial +enough to track there too). -**Layer:** Egeria Server (pyegeria clean-up) · **Status:** open · **Found:** 2026-10-13, reported by the egeria-workspaces-fs Resource Browser session from a live private-journal test. +--- -**What:** as erinoverview, `_async_remove_note` (`pyegeria/core/_server_client.py`) -was called on a JournalEntry created by `MyProfile.journal_my_activity`. It -failed with `OMAG-REPOSITORY-HANDLER-400-008` (`LOCAL_CANNOT_CHANGE_EXTERNAL`): -`deleteMetadataElementInStore` cannot modify Erin's UserIdentity (`ff983b64`), -which has Content Pack provenance ("Coco Pharmaceuticals Combination"). The -peer verified that nothing was removed. The entry keeps its `actionSource` and -`presentInNoteLogs` relationships, and Erin's Person still has 1 identity, -1 contact detail, 8 roles and 15 scopes. Test entry `fef83ed5` is still on -Erin's profile, waiting for the owner's decision. +### ISSUE-147: Every valid-metadata clear call answers OPEN-METADATA-400-011 "not found", even after a successful delete; clear-map-value deletes nothing; clear-map-name leaves the map's value set behind -**The route is right:** in 6.2, notes are Actions. The type chain is -JournalEntry → Notification → Action → Process → Asset. Egeria main -(`af400039c2`) has no note, notification or action delete endpoint. -`AssetHandler.createNoteLogEntry` has no matching remove, and My Profile has -`journal-my-activity` with no matching remove. The only delete for an Action is -the generic `/assets/{assetGUID}/delete` (`AssetMakerResource`), which is what -`remove_note`'s `feedback-manager/assets/{guid}/delete` reaches, because -generic view services answer under any URL marker. Feedback Manager can only -remove a whole note log (`note-logs/{noteLogGUID}/remove`). +**Egeria reset 2026-10-23 ~21:10-23:54Z** (egeria-workspaces-fs, owner-approved): the quickstart database was wiped and Egeria recreated on build 81beb8e5 (odpi/egeria 4486d1a7), then Coco content was reloaded. Every GUID recorded before then is gone. The leftover `pyegeriaClearRetest*` test elements described under "Leftovers on qs-view-server" are gone with it; nothing to clean up. The fix had already passed its 2026-10-19 re-test. -**Why it fails (likely, not verified):** `AssetHandler.createAction` anchors the -entry to its action source, linked by `ActionRequester`. For a journal entry -that is the caller's profile. Deleting an anchored element updates its anchor, -and Erin's identity is owned by the content pack, so the local delete is -refused. If so, no journal entry for a content-pack user can be deleted. That -is for the Egeria team. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. A successful clear now returns success; clear-map-value now uses the map name; clear-map-name also removes that map name's values. Egeria asks us to drop any workaround that treats 400-011 as success. **pyegeria:** there is none (checked 2026-10-18: no 400-011 handling in `md_processing/v2/reference_data.py` or `pyegeria/omvs/valid_metadata.py`), so the follow-on proposed below is withdrawn. Nothing to change; after the re-test, remove the leftover `pyegeriaClearRetest*` test elements on qs-view-server. -**Anchor read (peer, read-only):** JournalEntry `fef83ed5` has anchorGUID -`10dc7da9`, Erin's Person profile. Its `actionSource` is ActionRequester → -Person `10dc7da9`, and `presentInNoteLogs` is AttachedNoteLogEntry → NoteLog -`f780511a` (MyJournal). No relationship from the entry reaches UserIdentity -`ff983b64`, which the error names. That identity is anchored to the same -Person. So the refusal comes from the delete touching the anchor's graph, most -likely the LatestChange update on the anchor, or a provenance check across -elements that share the anchor, not from a direct link. +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** On throwaway properties `pyegRtClear`/`pyegRtClearMap` (typeName Asset), `clear_valid_metadata_value` (alpha, beta), `clear_valid_metadata_map_value` (mapOne/v1) and `clear_valid_metadata_map_name` (mapTwo, mapOne) all return success. A second clear of an already-removed value still answers 400-011 "not found", which is correct. Values and map names were gone afterwards. One oddity: `get_valid_metadata_values` read in the same run straight after the clears still listed alpha/beta/mapOne/mapTwo, but a read a minute later was empty. Possibly a short server-side cache; not pursued. Setup creates per-property container sets (`Egeria:ValidMetadataValue::pyegRtClear-`, `...pyegRtClearMap-`) that the clears do not remove; deleted by GUID. **Not cleaned (outside what peers cleared):** the 2026-10-13 leftovers under `pyegeriaClearRetest*`, which the platform has since turned into DataProduct/DigitalProduct elements (e.g. `98de81ed`, `4b937012`). -**Delete-path trace (2026-10-13, Egeria source, read-only):** the asset delete -(`AssetMakerResource.deleteAsset` → `AssetHandler.deleteAsset`) and the Metadata -Expert delete (`MetadataExpertResource` → `MetadataExpertRESTServices`) both call -`OpenMetadataClient.deleteMetadataElementInStore`, then the same -`MetadataElementHandler` → `OpenMetadataAPIGenericHandler.deleteBeanInRepository` -code. Neither does any extra checking, so switching routes cannot avoid 400-008. -The error is thrown only by `RepositoryErrorHandler.validateProvenance`, when -the delete removes an entity or relationship whose provenance is not -LOCAL_COHORT and the request carries no `externalSourceGUID`. That happens in -`removeRelationship` for each relationship on the note and in `removeEntity` -for the note itself. No anchor (LatestChange) update was found on the delete -path, which corrects "Why it fails" above. But the error names UserIdentity -`ff983b64`, which is not linked to the note, so the exact instance is still -unexplained. Untried options are to pass the content pack's metadata-collection -id as `externalSourceGUID`, or to use `deleteMethod: ARCHIVE`, which goes -through a different handler. +**Layer:** Egeria Server (`ValidMetadataValueHandler`) · **Status:** open · **Found:** 2026-10-13. The first live run of the Valid Metadata commands showed it (the error text wasn't captured then). Re-tested the same day with the full responses captured. -**Note-privacy findings (egeria-workspaces-fs session, 2026-10-13, with owner -approval):** a second test JournalEntry, `8bec72e4-6da6-40a4-943e-d81e2173ec9b`, -was created on Erin's profile (`10dc7da9`) and cannot be deleted either until a -working delete exists. -- `POST my-profile/journal-my-activity` ignores `initialClassifications` and - stamps only ZoneMembership `['erinoverview']`. -- `POST classification-explorer/elements/{guid}/zone-membership` with - `['resource-explorer-private','erinoverview']` makes the entry private. - erinoverview can read it. peterprofile and garygeeke get 403 - `OPEN-METADATA-SECURITY-403-020` by GUID, and 0 hits from - `feedback-manager/assets/by-search-string`. -- The journal's NoteLog (`f780511a`) stays at `['erinoverview']`, so other - users can see that the log exists but not the hidden entries. -- The owner chose a temporary workaround for the Resource Locator: call - `add_zone_membership(['resource-explorer-private', user])` straight after - `journal-my-activity`. Review it once Egeria's journal fix lands. +**Re-test:** run once on qs-view-server as erinoverview, with all four live sessions clearing it first. It used made-up properties `pyegeriaClearRetest` and `pyegeriaClearRetestMap` (typeName `Asset`) and called the SDK methods in `pyegeria/omvs/valid_metadata.py`. The Dr.Egeria `Clear Valid Metadata *` commands call the same methods (`md_processing/v2/reference_data.py:198-242`). -**pyegeria clean-up:** make the `remove_note` route explicit and describe it as -an Action delete in the docstring. The owner chose the Metadata Expert delete -(`metadata-expert/metadata-elements/{guid}/delete`) for now. Per the trace -above, this is a clarity change, not a fix for 400-008. +| Call | Response | Actual effect | +|---|---|---| +| `clear_valid_metadata_value(alpha)`, set up just before | 400-011 "value alpha ... is not found" | **alpha removed** (gone from the list afterwards) | +| same call again (alpha already gone) | 400-011 | nothing to remove | +| `clear_valid_metadata_value(beta)` | 400-011 | **beta removed** | +| `clear_valid_metadata_map_value(mapOne, v1)` | 400-011 "value v1 ... is not found" | **nothing removed**: `get_valid_metadata_map_value` still returns v1 | +| `clear_valid_metadata_map_name(mapTwo)` | 400-011 | **mapTwo removed** | +| `clear_valid_metadata_map_name(mapOne)`, with v1 under it | 400-011 | the mapOne name value is removed, but the map's value set "mapOne" (holding v1) stays in `get_valid_metadata_values` | -**Re-test after Egeria's delete fixes are published:** the owner said -(2026-10-13) that Egeria is working on delete issues now. Once that work is -published, re-test `remove_note` on `8bec72e4` (plain delete, then `ARCHIVE` if -needed). Tell the egeria-workspaces-fs session before deleting anything on -Erin's profile. `fef83ed5` stays. +Request ids: `ec9052e7` and `3975182d` (alpha), `38759736` (beta), `475e68f1` (map value), `407c0e17` and `07886f29` (map names). pyegeria surfaces each one as `SERVER_ERROR_500` wrapping `relatedHTTPCode` 400 `InvalidParameterException`. ---- +**Cause (Egeria source, `af400039c2`, `frameworks/open-metadata-framework/.../handlers/ValidMetadataValueHandler.java`):** +1. **No `return` after the delete (lines ~541-680).** In all three `clear*` methods, `if (validValueGUID != null) deleteMetadataElementInStore(...)` is followed unconditionally by the audit log and `throw new InvalidParameterException(VALID_METADATA_MISSING ...)`. So every call throws, whether or not it deleted anything. The "not found" error is the method's normal exit. +2. **Map name dropped in `clearValidMetadataMapValue`.** It calls `getValidValueGUID(userId, typeName, propertyName, null, preferredValue)`, passing `null` where `mapName` belongs. It therefore searches for `v1` as a plain value of the property, finds nothing, and deletes nothing. +3. **`clearValidMetadataMapName` deletes only the name's own value.** It doesn't touch the per-map value set created by `setup-map-value` (the "Set of valid values for the mapOne map name ..." element) or its members. Whether that should cascade is Egeria's call. +**Effect on Dr.Egeria:** `Clear Valid Metadata Value` and `Clear Valid Metadata Map Name` report failure although the clear worked. `Clear Valid Metadata Map Value` reports failure and really did fail. +**Leftovers on qs-view-server:** the map value set "mapOne" with member v1 under `pyegeriaClearRetestMap`/`Asset`, plus whatever per-property container elements setup creates (`Egeria:ValidMetadataValue::pyegeriaClearRetest-Asset` and `...pyegeriaClearRetestMap-Asset`). These are test elements, not content-pack elements. RE's blueprint materializer treats `Egeria:ValidMetadataValue:` qualifiedNames as content-pack origin, but only for SolutionComponents, so they don't affect it. +**pyegeria follow-on (not done, needs the owner's go-ahead):** Dr.Egeria's clear processor could treat 400-011 from `clear-value`/`clear-map-name` as success when a follow-up read confirms the value is gone. Otherwise, wait for the Egeria fix and re-test (see the BACKLOG.md delete re-test item). -### ISSUE-132: `DeleteElementRequestBody` serializes the cascade flag as `cascadeDelete`; Egeria's field is `cascadedDelete` and unknown fields are ignored, so no pyegeria cascade delete has ever cascaded +--- -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. `cascadeDelete` is now accepted as an alias for `cascadedDelete`, so existing pyegeria requests will start to cascade; Egeria still recommends switching to `cascadedDelete`. **Risk check, 2026-10-18:** no production caller in `pyegeria/`, `md_processing/` or `commands/` passes cascade `True`; every `cascade`/`cascade_delete` default is `False`; `commands/tech/element_actions.py`'s `--cascade` flag (default True) is not forwarded. Live functional/scenario tests that pass `cascade=True` (collection, glossary, governance officer, `remove_note_log`) will now really cascade, which is what they intend. **pyegeria (proposed, not done):** change `DeleteElementRequestBody`'s serialization alias to `cascadedDelete` (keep both accepted on input) and the ~25 `"cascadeDelete"` literals (`_server_client.py`, `models.py`, `actor_manager.py`, `metadata_expert.py`, `product_manager.py`, `reference_data.py`, `valid_metadata.py`), then live-test one cascade and re-test ISSUE-127. No longer urgent, since the old spelling now works. +### ISSUE-143: `includeOnlyRelationships: ["PersonRoleAppointment"]` returns no relationships for a Person, although the unfiltered read has its role appointments -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS (alias).** A throwaway DataStructure with a linked member DataField, deleted with `delete_data_structure(guid, cascade_delete=True)` (pyegeria still sends `cascadeDelete`), succeeded. Both elements were gone by GUID before any cleanup, so the cascade reached the server. The ISSUE-154 item 1 rename to `cascadedDelete` is still recommended but no longer urgent. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. The `includeOnlyRelationships` filter now leaves already-retrieved relationships in place, so `performsRoles` survives `["PersonRoleAppointment"]`. **pyegeria:** no code change; tell the Resource Browser (egeria-workspaces-fs) it can use the filter again after the re-test. -**Layer:** pyegeria · **Status:** open, on hold: an Egeria fix for cascade delete is coming (owner, 2026-10-10). Egeria `main` checked 2026-10-10 (head `af400039c2`, 2026-10-06): no cascade change yet. Wait -for it, live-test cascade delete against it, then decide the pyegeria change; the fix below may change once the -Egeria side lands. Confirmed from Egeria source, not yet live-tested · **Found:** -2026-10-10, while checking ISSUE-129's server contract against Egeria source instead of the `.http` files. +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** Erin's Person `10dc7da9` read with `includeOnlyRelationships: ["PersonRoleAppointment"]` returns `performsRoles` 8 (unfiltered: performsRoles 8, assignmentScope 15, requestedActions 13, ...; `["ContactThrough"]` still returns contactDetails 1). Read-only. Tell the Resource Browser it can use the filter again. -**What:** `pyegeria/models/models.py` `DeleteElementRequestBody.cascade_delete` accepts both spellings on input but -serializes as `serialization_alias="cascadeDelete"`. The comment there records that the `.http` files were split -about 50/50 and that the choice was unverified (ISSUE-62). Egeria source settles it (`egeria-v6/egeria` at -`af400039c2`, 2026-10-06): `DeleteElementRequestBody` -> `DeleteRequestBody` -> `DeleteOptions` -(`open-metadata-framework/.../search/DeleteOptions.java`), whose field is `cascadedDelete` with -`getCascadedDelete()`/`setCascadedDelete()`. It is unchanged since 2025-08-04. The class is -`@JsonAutoDetect(getterVisibility=PUBLIC_ONLY, fieldVisibility=NONE)` and `@JsonIgnoreProperties(ignoreUnknown=true)`, -with no `@JsonProperty`/`@JsonAlias`, so the JSON name is `cascadedDelete` and `cascadeDelete` is silently ignored. +**Layer:** Egeria Server (not pyegeria) · **Status:** open, not yet reported to the Egeria team · **Found:** 2026-10-13, reported by the egeria-workspaces-fs Resource Browser session; reproduced read-only the same day. -**Effect:** every pyegeria delete that uses `DeleteElementRequestBody` with cascade on (the ISSUE-62 fix, -`validate_delete_element_request`, ISSUE-127's fix in #429, `MetadataExpert.delete_metadata_element(cascade_delete=True)`) -sends a flag Egeria discards, so it deletes without cascading. A caller-supplied dict using the correct -`cascadedDelete` key is also re-serialized as `cascadeDelete`, so even that does not work. The models module's own -`DeleteRequestBody.cascaded_delete` already uses the right name. +**What:** reading a Person with a `GetRequestBody` whose +`includeOnlyRelationships` is `["PersonRoleAppointment"]` returns the element +with no relationships at all. The same read with no filter returns 8 +`performsRoles` entries, each with relationship type `PersonRoleAppointment`. +Other filters on the same Person work. -**Fix (not made):** change the serialization alias to `cascadedDelete` (keep both spellings accepted on input), and -fix the 19 `"cascadeDelete"` references in `_server_client.py`, `product_manager.py`, `actor_manager.py`, -`valid_metadata.py`, `metadata_expert.py` and `reference_data.py` (several are docstring samples). Then live-test one -cascade delete, and re-test ISSUE-127's `DataStructure` refusal. The `.http` files using `cascadedDelete` are correct; -the ones using `cascadeDelete` are the ones in error (worth reporting to Egeria). +**Reproduction** (Person `10dc7da9-e9e1-471f-9afd-46a7ff414cb7`, Erin Overview, +as erinoverview on qs-view-server; +`ClassificationExplorer.get_element_by_guid(guid, graph_query_depth=1, +output_format="JSON", body={"class": "GetRequestBody", "graphQueryDepth": 1, +"includeOnlyRelationships": [...]})`): -**Lesson:** where the `.http` files disagree with each other, read the Java DTO (getter/setter names plus Jackson -annotations); do not choose by majority or by what the code already sends. +| Filter | Relationship keys returned | +|---|---| +| none | performsRoles 8, assignmentScope 15, requestedActions 11, contactDetails 1, userIdentities 1, contributionRecord 1 | +| `["ContactThrough"]` | contactDetails 1 | +| `["AssignmentScope"]` | assignmentScope 15 | +| `["PersonRoleAppointment"]` | none | + +A spy on the request confirmed pyegeria sends the body exactly as given, so +the filtering happens in Egeria. + +**Likely cause (not confirmed):** Egeria's relationship filter +(`OpenMetadataHandlerBase`, around line 478) keeps a relationship when +`propertyHelper.isTypeOf(rel, name)`, which should match. But `performsRoles` is +filled in a separate step (`OpenMetadataHandlerBase`, around line 1282), so the +role appointments may be dropped or never fetched once a filter is set. + +**Workaround:** read the Person without `includeOnlyRelationships` and take +`performsRoles` from the result. The Resource Browser already does this. --- -### ISSUE-127: `CollectionManager.delete_collection(cascade=True)` silently never sends `cascadeDelete` -- same shape as ISSUE-62 +### ISSUE-130: The YAML secrets store's user directory caches a user forever, so editing an existing user needs a platform restart -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** The `DataStructure` refusal (`validateNoMemberDataFields`) did not recur: a cascade delete of a structure with a member field removed both. See ISSUE-132's re-test. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. The YAML secrets store rebuilds its user cache each time it re-reads the file (`refreshTimeInterval`), so editing an existing user no longer needs a platform restart. **pyegeria:** no code change. -**Layer:** pyegeria · **Status:** partly fixed in #429 (`68279835`), released in 6.1.29; cascade still blocked by ISSUE-132 · **Found:** 2026-10-05, found by capturing the request body while diagnosing a -failed cleanup. +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **Not re-testable here.** Needs an edit to a user in the platform's YAML secrets file and a wait for `refreshTimeInterval`. egeria-workspaces-fs is putting that to the owner as a separate step. -**Fix + scope (2026-10-05):** a scan of all 42 callers of `_async_delete_element_request` found exactly two -that pre-fill a default body *and* pass a cascade flag: `CollectionManager._async_delete_collection` and -`ProductManager._async_delete_digital_product` (so `delete_digital_product(cascade=True)` was never cascading -either -- the live check's own product cleanup included). Both now pass `None` so the helper applies the flag. -`delete_digital_product`'s docstring sample also used the wrong key (`cascadedDelete`); corrected. The helper -itself is unchanged: a caller that passes an explicit body dict *and* `cascade=True` still gets the body's -value, which is documented behaviour ("body supersedes"). +**Re-test 2026-10-19 by egeria-workspaces-fs (reported to this session, not independently re-run here):** **PASS.** A throwaway user `pyegrt130` was added to the live `coco-user-directory.omsecrets` and logged in (200, 13:19:48Z), which cached it. It was then set to `userAccountStatus: DISABLED` in place with no restart (13:19:56Z). Logins worked until 13:20:32Z and were refused with 401 at 13:21:02Z, so the cache was rebuilt about a minute after the edit (`refreshTimeInterval: 10`). The file was restored byte-for-byte afterwards. Ready to move to the Appendix. -**What:** `_async_delete_collection` always pre-fills `body = {"class": "DeleteElementRequestBody"}` when none is -given and hands that dict to `_async_delete_element_request(url, body, cascade)`. -`validate_delete_element_request` applies `cascade_delete` only in its *no body* branch, so with that pre-filled dict -the flag is discarded. Captured: `delete_collection("g", cascade=True)` posts -`{'class': 'DeleteElementRequestBody', 'forLineage': False, 'forDuplicateProcessing': False}` with no -`cascadeDelete`. `MetadataExpert.delete_metadata_element(cascade_delete=True)` does send it. +**Layer:** Egeria server (`YAMLSecretsStoreConnector`) · **Status:** open, reported to the Egeria lead · **Found:** 2026-10-09, while adding Resource Explorer's own account (`resourceexplorernpa`). -**Fixed in #429, but cascade still does not reach Egeria -- see ISSUE-132.** #429 made the flag reach the request -body, but under the name `cascadeDelete`. Egeria's real field is `cascadedDelete`, and it ignores unknown fields, so -the flag is still dropped, now on the server side. #429's docstring "correction" from `cascadedDelete` to -`cascadeDelete` was also the wrong way round. The `DataStructure` refusal described below is probably this bug -rather than an Egeria quirk; re-test once ISSUE-132 is fixed. +**What:** `getUser(userId)` returns the entry from `userAccountMap` if it is already there. `refreshSecrets()` re-reads the YAML file every `refreshTimeInterval` minutes, but `userAccountMap` (and `securityAccessControlMap`) are created only on the first load (when `secretsStoreFile == null`) and are never cleared afterwards. Re-reading the file only helps users the cache has never seen; the comment says "this may be a new user". -**Related, an Egeria quirk found alongside:** even with `cascadeDelete: true`, Egeria refused to delete a -`DataStructure` that still had a member `DataField` (`OMAG-GENERIC-HANDLERS-403-005 ... validateNoMemberDataFields`), -so a data contract imported by `import_data_contract*` has to be removed child-first: DataField, DataStructure, -Agreement. +**Seen:** the account was first added without its `secrets` block, and the first token request cached it. After the password was added (and matched the servers file), `/api/token` still answered 401 for more than 10 minutes. It worked straight after a platform restart. + +**Effect:** any change to an existing user (password, status, named-list membership) needs a platform restart. + +**Fix (Egeria):** clear or rebuild `userAccountMap` and `securityAccessControlMap` in `refreshSecrets()`. --- -### ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` +### ISSUE-128: Retention cannot be set on Egeria 6.2: the server registers the Jackson subtype as `RetentionProperties.class` (typo), and pyegeria only allows `RetentionClassificationProperties` -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: needs information from us.** Egeria cannot reproduce it. The error is now a clear "no asset connector" message, not a NullPointerException. Cause: the surveyed FileFolder had no ResourceConnection; the likely route is that the template create returned an existing FileFolder with the same qualified name (template create allows an existing element to be returned). They ask for: -- [ ] The request used to create the FileFolder: its template GUID and placeholder values, and whether `deepCopy` or `allowRetrieve` was set. -- [ ] The GUID of the surveyed asset, and whether that GUID existed before the create call. -- [ ] Whether that asset has a ResourceConnection, and which process first catalogued it (for example a files integration connector). +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. Fixed on `main` 13 Oct (`RetentionProperties`); `oak2026` also accepts `RetentionClassificationProperties` again. Responses now carry `RetentionProperties` in `class`. **pyegeria (proposed, not done):** per the owner's 2026-10-09 decision, switch the whitelist, docstrings and Dr.Egeria pins to `RetentionProperties`: `pyegeria/omvs/classification_explorer.py:8851,8878,8919`, `md_processing/v2/processors.py:220-237` (its comment says the opposite and must be rewritten), `md_processing/v2/curation.py:136`. Accept both names in the whitelist so older platforms keep working. Not urgent: the old name is accepted again. -**pyegeria:** this fits the 2026-10-06 live check below (with `deepCopy` no longer sent, the new folder has a ResourceConnection). `create_folder_element_from_template` hardcodes `"allowRetrieve": True` (`pyegeria/omvs/automated_curation.py:1081`), so a create whose qualified name already exists returns the old, possibly connection-less folder, which matches Egeria's theory. Proposed (not done): let callers pass `allow_retrieve`, or log when the returned GUID was pre-existing. The answers above can come from the 2026-10-06 `live_79.py` run's request plus a read-only lookup of the original surveyed GUID. +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** On a throwaway collection, `classification-explorer/elements/{guid}/retention` accepted both `"class": "RetentionClassificationProperties"` and `"RetentionProperties"`; each read back `retentionBasis` 2 and its notes under `elementHeader.retention`. pyegeria's own `set_retention_classification` (old name) also succeeded. ISSUE-154 item 3's switch to the new name stays optional. -**Live check 2026-10-06 (about 02:00Z, current pyegeria 6.1.29 code, rebuilt platform): the `deepCopy:false` default WAS the cause of the missing connection.** `create_folder_element_from_template` now sends a request with no `deepCopy` key (keys sent: allowRetrieve, class, forDuplicateProcessing, forLineage, isOwnAnchor, parentAtEnd1, placeholderPropertyValues, templateGUID, templateSubstitute), and the new folder read back through the classification explorer **has 1 connection** (`ResourceConnection` present). Folder deleted (cascade) and verified gone by GUID. **Not run: the survey itself** (a survey creates elements owned by the survey identity that an ordinary user may not be able to delete), so "the survey now completes" is still unproven; the original `assetConnector`-null failure was the symptom of no connection, which is now shown not to occur. Whether the Egeria-side repair (`e01426db86`) also matters is untested. Script: `live_79.py` stage A (stage B `--survey` is opt-in). +**Status:** open, found 2026-10-09 by Resource Explorer's first Retention publish on Egeria 6.2. **Layer:** Egeria server (a type registration typo). **Decision (owner, 2026-10-09):** wait for the Egeria fix; no pyegeria workaround. Retention publishing stays blocked until then. -**Update 2026-08-30, from the Egeria team (Mandy Chessell).** The entry -stays in "Open Egeria Server issues" — the reported failure is not -explained and not fixed. What has changed is that the cause recorded in -this entry, and in Resource Explorer's `re-as-engine-host-plan.md`, is now -known to be wrong, and the error it produces has been made legible. +**What (all proven live on 6.2, with raw HTTP POSTs to `classification-explorer/elements/{guid}/retention` on a throwaway asset):** +- `"class": "RetentionClassificationProperties"` is refused with `InvalidTypeIdException`. This is what pyegeria documents and whitelists, and it worked on the pre-6.2 server (processors.py comment, 2026-08-03). +- `"class": "RetentionProperties"` (the `.http` ground truth) is also refused with `InvalidTypeIdException`. +- `"class": "RetentionProperties.class"` gets **HTTP 200**, and the classification reads back with `retentionBasis` 2 and the notes. +- The server's known-type list in the error has 96 ids. The only Retention entry is literally `RetentionProperties.class`. That looks like an `@JsonSubTypes.Type(name = "RetentionProperties.class")` typo in Egeria's `ClassificationProperties`. +- pyegeria's `validate_new_classification_request` whitelist (`prop=["RetentionClassificationProperties"]`) refuses anything else client-side, so no Retention can be set through pyegeria on 6.2. -**Status: still open (Egeria Server)** — but the recorded cause is -disproved, and this entry and `re-as-engine-host-plan.md` should both be -corrected. Investigated 2026-08-30 from the Egeria side, including a new -FVT suite built to run this exact scenario. +(An earlier version of this entry said 6.2 knows `RetentionProperties`. That was a misread of the error text, corrected the same day.) -**The premise is false.** This entry, and the design doc's hold, rest on -"template-based asset creation never wires up a working Connection." That -is not true, in either environment it was checked against: +**Where seen:** Resource Explorer's Curate classifications step on two repository assets. Both were refused, so nothing was written. Confidentiality, Criticality, Confidence and Impact property classes are registered normally. -- **On the quickstart that produced the report.** Every element created - from a template that itself carries a `ResourceConnection` got one — 10 - of 10. Every `FileFolder` in that repository has one, including all - three that were created from a template (`/deployments/content-packs`, - `/deployments/loading-bay/sample-data`, `/deployments/secrets`). -- **In a clean environment built from the current archives.** A folder - catalogued through `FileGovernance::create-file-folder` came back with a - `ResourceConnection` attached. -- **The `FilesContentPack` template itself is also well formed:** - `FileFolder` template `fbdd8efd-...` carries an `Anchors`-classified - `Connection` with both a `ConnectToEndpoint` and a - `ConnectionConnectorType`. +**Candidate fix:** +1. **Egeria:** correct the subtype name to `RetentionProperties`, matching the `.http` file. Report it to Mandy. +2. **pyegeria, after Egeria's fix:** change the whitelist and docstring to `RetentionProperties`, the `.http` name. No workaround for the typo, by the owner's decision. +3. Re-check `md_processing/v2/processors.py` and `v2/curation.py`, which both pin `RetentionClassificationProperties`. -**The folder survey works.** A new `files-fvt` suite runs the reported -scenario end to end — catalogue a folder from the `FileFolder` template, -then survey it natively — and it passes: -``` -OMES-SURVEY-ACTION-0019 The survey action service folder-survey-service has completed the -analysis of asset 7050dc50-... with request type survey-folder in 1927 milliseconds; the -results are stored in survey report 7e7892e2-... -``` -The report carried six annotations. The test asserts the survey -*completed*, not merely that it ran: a survey that dies on a null -connector still produces an engine action, and the waiter throws on any -terminal status other than `COMPLETED`. +--- -**A second hypothesis was also eliminated.** While investigating, a real -defect was found in the PostgreSQL repository: a `TypeDefPatch` that -reparents a type does not update the denormalised supertype chain stored -on existing rows, so instances written before the patch become invisible -to searches for the new supertype. That would have hidden connections -exactly as reported, since `AssetConnection` was reparented under -`ResourceConnection` on 2026-08-27. It is fixed — but it was never the -cause here: the quickstart repository stores 202 rows as -`:ResourceConnection:LabeledRelationship:` and none as -`:AssetConnection:`, because it was reloaded after the type change. +### ISSUE-108: newly created relationships stay invisible to related-element queries for up to ~20 minutes after creation -**What has been fixed, and what it does and does not buy.** The NPE -itself is now a clean error. `getConnectorForAsset()` returns `null` by -contract when an asset has no connection; `FolderSurveyService` cast that -null and dereferenced it, and the framework's own -`performCheckAssetAnalysisStep()` had the same hole on its error path. -Both now report `OPEN-SURVEY-400-008` / `OPEN-SURVEY-0009` -(`NO_ASSET_CONNECTOR`), naming the asset. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: needs information from us.** Egeria cannot reproduce it, and the new FVT does not show it. Their best lead is still timing: a related-element query hides a relationship whose `effectiveFrom` is later than the server's current time. For one link still missing about 12 minutes after a bulk load, they ask for: +- [ ] The relationship's GUID, `effectiveFrom` and `createTime`, and the server clock at that moment. +- [ ] Whether the link is returned when fetched directly by its GUID while it is missing from the related-element query. +- [ ] Whether the loader (Dr.Egeria or pyegeria) sets `effectiveFrom` or `effectiveTime` on that run. +- [ ] A timed run with `MemberDataField` links at 200 or more, polling every minute, to show whether the delay grows with load size or stays fixed. -This does not make such a survey succeed — an asset with no connection -describes a resource nothing can open. It makes the failure say *which* -asset, which is what the original report was missing: the NPE named -`BasicFolderConnector`, a class the caller had never heard of, and pointed -away from the asset. +**pyegeria:** Dr.Egeria's body builders send `effectiveFrom` only when the author supplies `Effective From` (`common_md_utils.py:342` and siblings), so a default load sets none; still confirm `effectiveTime` is not stamped from the client clock on link requests (a client clock ahead of the server would explain the delay). The timed run needs Egeria writes, so it needs the owner's go-ahead. -**Why that matters for how this entry reads.** Four further defects were -found in the same connectors, all on paths nothing else exercises: both -file survey services failed on every invocation (annotations created -without a qualified name, in three places), the CSV survey dereferenced a -null root schema type, wrote a relationship property to an entity, and -produced colliding annotation qualified names. The file survey path had, -on this evidence, never run to completion. Someone hitting an NPE in this -family had every reason to suspect the asset — the errors never named -anything else. That is context for the original diagnosis, not a -criticism of it. +**Layer:** Egeria Server · **Status:** open · **Found:** 2026-09-20, reported by +the user from a Dr.Egeria bulk load session (1,027 expected field-link +relationships). -**The naming trap, which is directly actionable on the pyegeria side.** -This entry already records that `initiate_file_folder_survey`'s default -(`FileSurveys:survey-folder`) and the `.http` example -(`AssetSurvey:survey-folder`) are both wrong. Confirmed and now -understood: a governance action type is registered under -`::` — two colons — so the real name is +**What:** a catalog walk run immediately after a load found only 876 of 1,027 +expected field links. A second walk 12 minutes later still showed 82 +structures short. A third walk, 20 minutes after the load, showed every link +present. No errors were raised at any point in the load itself — the +relationships were created successfully, they simply did not show up in +related-element queries for a long window afterward, then all became visible +at once with no further action taken. + +**Not yet root-caused.** This looks like the same family of symptom as the +2026-08-15 Postgres-checkpoint/background-connector-load entry above (server +falling behind under load, catching up later) rather than a paging defect +like ISSUE-54 (which would not self-heal purely by waiting — ISSUE-54's +bug is about tie-ordering across separately-executed pages, not about data +not existing yet). But this has not been directly correlated with connector +log activity or checkpoint timing the way that entry was — this is a fresh +report, not yet independently reproduced or diagnosed by a pyegeria session. + +**Impact:** any workflow that verifies a bulk write immediately after +issuing it (a post-load catalog walk, an integration test, an automated +Dr.Egeria "did it work" check) can get a false-negative "missing" result +that resolves itself given enough time — worth knowing before treating an +immediate post-write verification gap as evidence of a real data-loss bug. + +**Candidate next step (not yet done):** reproduce with a smaller, timed +sample (e.g. create N relationships, poll related-element queries at 1-minute +intervals, log when each becomes visible) to get a real latency +distribution instead of the three anecdotal checkpoints here, and check +whether it correlates with the same background-connector/checkpoint load +already documented above. Worth raising with the Egeria team directly, per +the original report. + +**Update 2026-09-20 (peer check, no upstream issue filed yet):** checked +`odpi/egeria`'s issue tracker directly — nothing filed for this symptom, so +no response from the Egeria team via GitHub. Asked three peer sessions +working against the same dev Egeria server: +- **egeria-workspaces-fs-4a:** hasn't seen this specific symptom, no word + from the Egeria team either. Raised a speculative, explicitly-unconfirmed + lead worth checking: a chronic, still-open issue on `egeria-shared-kafka` + where the `GroupCoordinator` repeatedly times out writing to + `__consumer_offsets` (5000ms) and triggers a consumer-group rebalance + roughly every 90s (confirmed via broker logs 2026-09-03; root cause still + open, CPU/GC/Postgres-restarts/virtiofs I/O already ruled out; historically + "Information" severity, self-heals within seconds, not chased further + since it wasn't visibly causing downstream harm). If `qs-view-server`'s + related-element queries are backed by Kafka-driven indexing/event + propagation rather than a direct repository read, a chronic rebalance + storm could plausibly manifest as exactly this "relationship exists but + isn't visible to queries yet" lag. **Explicitly not a confirmed + correlation** — the timescales don't obviously line up (a recurring ~90s + self-healing blip vs. a one-shot ~20-minute resolution) — and the peer + offered to check Kafka broker logs around the original bulk-load window + for the same rebalance pattern if useful. +- **Resource Explorer experimental UI:** no direct evidence either way — + today's two live writes against the same server (a Governance Action + Process Step upsert, and a native-survey `SurveyReport` lookup) both + checked either *immediately* after the write or only *after* polling to a + terminal status, never in the 1-20 minute gap this entry describes, so + neither confirms nor rules it out. Flagged that a background agent on + their side doing a related connection-refresh fix will trigger fresh + engine actions/relationships shortly and will report back if it surfaces + the same lag. +- **Egeria-trellis backlog review:** also nothing from the Egeria team (no + channel to them; checked RE docs/Backlog/design notes for this exact + shape first — a checked "no", not a guess). Offered a different, measured + precedent on this same platform as a *pattern* to distinguish causes, not + a diagnosis: `packages/resource-explorer/docs/investigation-classification-and-zoning-design.md` + (~line 830) documents a privately-zoned element still readable by a + non-owner *six minutes* after the control was written (denial began at + seven), caused by the security connector's `refreshTimeInterval` (10 min) + reload — "read it back immediately, so it must be fine" was exactly the + misleading evidence there too. Their key diagnostic point: a + **fixed-wall-clock cause** (everything becomes visible at once, at a time + unrelated to load size — test with 50 links instead of 1,027 and see if + the delay is still ~20 min) and a **volume-proportional cause** (async + indexing/event-driven materialisation catching up, arriving in a stream) + look identical from a single observation but have different fixes; the + 876→945(implied)→1,027 partial-recovery-at-12-minutes data point *mildly* + favours volume-proportional, but "two points don't make a curve." Also + worth checking directly: fetch one of the missing links by its own GUID + during the window — if that resolves but the relationship is still absent + from the related-element query, the write landed fine and only the + traversal/index path lags, narrowing which service is actually behind. If + this gets filed upstream, the volume-vs-fixed measurement is what turns + it into a real bug report instead of getting bounced as "eventual + consistency, working as intended." + +Two independent, unconfirmed leads now on record (Kafka rebalance storms; +a refresh-interval-style cache) alongside a concrete diagnostic method +(fixed-wall-clock vs. volume-proportional, plus the direct-GUID-fetch +isolation check) for telling them apart before assuming either. Next real +step is still reproducing with a smaller, timed, instrumented load using +that method — not yet done. + +**Reproduction attempt, 2026-09-20 (small end of the curve, not reproduced):** +after clearing with all 5 live peer sessions first (per `coordinate-shared-writes`), +ran a timed, instrumented small-scale reproduction against `qs-view-server`: +20 throwaway `Collection` elements created and each linked to a parent via +`CollectionMembership` (not the original `MemberDataField`/"field link" type +— a different relationship type, noted as a limitation below), polling +`get_collection_members` every 30s for up to 20 minutes, with a direct-GUID +isolation check ready for anything still missing at the deadline. + +**Result: all 20 relationships were visible on the very first poll, ~1.1s +after the last one was created — no lag at all.** Cleanup (delete all 21 +fixture elements) verified two independent ways: a direct-GUID fetch on +each (`404`/`OMAG-REPOSITORY-HANDLER-404-007`, confirmed real soft-delete, +not an ISSUE-63-style silent no-op) and a broad `find_collections` search +for the `test-fixture-issue108` prefix (0 results). The script itself was +run from an ephemeral per-session scratch directory, not committed anywhere +— reproduce via the same approach (create N Collections linked to a parent +via `_async_add_to_collection`, poll `_async_get_collection_members`) if +needed later; the approach, not the script file, is what's worth keeping. + +**Per Egeria-trellis backlog review's own framework, this is one point on +the curve, not a refutation.** It's consistent with either "no bug under +current load" or "the lag is genuinely volume-proportional and 20 elements +is nowhere near the ~1,027-link scale that surfaced it originally" — the +one data point this run *can't* distinguish those two. Also a real +methodology gap versus the original report: this used `CollectionMembership` +on plain `Collection` elements, not the `MemberDataField` "field link" +relationship the original 1,027-link load actually exercised (ISSUE-102 is +in the same `MemberDataField` neighborhood, worth checking together). Server +load conditions also weren't recorded this run (no `ListAgents`/`:8810` +snapshot taken alongside the timings, despite the earlier plan to do so — +an execution gap, not a design one). + +**Candidate next step, still open:** a same-relationship-type +(`MemberDataField`), same-order-of-magnitude (~200+) run, following the +same peer-clearance and independent-verification discipline, is what would +actually test the volume-proportional hypothesis — this run didn't attempt +that scale on its own initiative, since committing to it wasn't part of +what peers were asked to clear. + +**Egeria team response, relayed 2026-10-05 (source `oak2026` fb3d6fce53; **fixes are in two unmerged PRs, so no build contains them, and nothing was tested against a live platform**):** not validated; no cache found. Best lead: a client clock ahead of the server's, with `effectiveFrom` set from it (related-element queries silently hide a relationship whose `effectiveFrom` is later than the server's "now"). They need, for one link still missing at ~12 minutes: its `effectiveFrom` and `createTime`, the server clock then, and whether the loader sets `effectiveFrom`. **Our facts:** pyegeria sets no `effectiveFrom`/`effectiveTo`/`effectiveTime` by default (model defaults are `None`; no clock call is tied to any `effective*` field; Dr.Egeria forwards an "Effective From" only if typed). Host and container clocks agree within 1 s on 2026-10-05; the 2026-09-20 load's values are unavailable. + +**Update 2026-10-10:** the `oak2026` source (`fb3d6fce53`) the team examined is now in Egeria `main` (merged 2026-10-06, odpi/egeria#9363). No fix was made for this issue; the team's questions above still stand. + +--- + + + + +### ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` + +**Status 2026-10-29: moved to the Egeria Server section; Egeria needs information from us (checklist below).** + +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: needs information from us.** Egeria cannot reproduce it. The error is now a clear "no asset connector" message, not a NullPointerException. Cause: the surveyed FileFolder had no ResourceConnection; the likely route is that the template create returned an existing FileFolder with the same qualified name (template create allows an existing element to be returned). They ask for: +- [ ] The request used to create the FileFolder: its template GUID and placeholder values, and whether `deepCopy` or `allowRetrieve` was set. +- [ ] The GUID of the surveyed asset, and whether that GUID existed before the create call. +- [ ] Whether that asset has a ResourceConnection, and which process first catalogued it (for example a files integration connector). + +**pyegeria:** this fits the 2026-10-06 live check below (with `deepCopy` no longer sent, the new folder has a ResourceConnection). `create_folder_element_from_template` hardcodes `"allowRetrieve": True` (`pyegeria/omvs/automated_curation.py:1081`), so a create whose qualified name already exists returns the old, possibly connection-less folder, which matches Egeria's theory. Proposed (not done): let callers pass `allow_retrieve`, or log when the returned GUID was pre-existing. The answers above can come from the 2026-10-06 `live_79.py` run's request plus a read-only lookup of the original surveyed GUID. + +**Live check 2026-10-06 (about 02:00Z, current pyegeria 6.1.29 code, rebuilt platform): the `deepCopy:false` default WAS the cause of the missing connection.** `create_folder_element_from_template` now sends a request with no `deepCopy` key (keys sent: allowRetrieve, class, forDuplicateProcessing, forLineage, isOwnAnchor, parentAtEnd1, placeholderPropertyValues, templateGUID, templateSubstitute), and the new folder read back through the classification explorer **has 1 connection** (`ResourceConnection` present). Folder deleted (cascade) and verified gone by GUID. **Not run: the survey itself** (a survey creates elements owned by the survey identity that an ordinary user may not be able to delete), so "the survey now completes" is still unproven; the original `assetConnector`-null failure was the symptom of no connection, which is now shown not to occur. Whether the Egeria-side repair (`e01426db86`) also matters is untested. Script: `live_79.py` stage A (stage B `--survey` is opt-in). + +**Update 2026-08-30, from the Egeria team (Mandy Chessell).** The entry +stays in "Open Egeria Server issues" — the reported failure is not +explained and not fixed. What has changed is that the cause recorded in +this entry, and in Resource Explorer's `re-as-engine-host-plan.md`, is now +known to be wrong, and the error it produces has been made legible. + +**Status: still open (Egeria Server)** — but the recorded cause is +disproved, and this entry and `re-as-engine-host-plan.md` should both be +corrected. Investigated 2026-08-30 from the Egeria side, including a new +FVT suite built to run this exact scenario. + +**The premise is false.** This entry, and the design doc's hold, rest on +"template-based asset creation never wires up a working Connection." That +is not true, in either environment it was checked against: + +- **On the quickstart that produced the report.** Every element created + from a template that itself carries a `ResourceConnection` got one — 10 + of 10. Every `FileFolder` in that repository has one, including all + three that were created from a template (`/deployments/content-packs`, + `/deployments/loading-bay/sample-data`, `/deployments/secrets`). +- **In a clean environment built from the current archives.** A folder + catalogued through `FileGovernance::create-file-folder` came back with a + `ResourceConnection` attached. +- **The `FilesContentPack` template itself is also well formed:** + `FileFolder` template `fbdd8efd-...` carries an `Anchors`-classified + `Connection` with both a `ConnectToEndpoint` and a + `ConnectionConnectorType`. + +**The folder survey works.** A new `files-fvt` suite runs the reported +scenario end to end — catalogue a folder from the `FileFolder` template, +then survey it natively — and it passes: +``` +OMES-SURVEY-ACTION-0019 The survey action service folder-survey-service has completed the +analysis of asset 7050dc50-... with request type survey-folder in 1927 milliseconds; the +results are stored in survey report 7e7892e2-... +``` +The report carried six annotations. The test asserts the survey +*completed*, not merely that it ran: a survey that dies on a null +connector still produces an engine action, and the waiter throws on any +terminal status other than `COMPLETED`. + +**A second hypothesis was also eliminated.** While investigating, a real +defect was found in the PostgreSQL repository: a `TypeDefPatch` that +reparents a type does not update the denormalised supertype chain stored +on existing rows, so instances written before the patch become invisible +to searches for the new supertype. That would have hidden connections +exactly as reported, since `AssetConnection` was reparented under +`ResourceConnection` on 2026-08-27. It is fixed — but it was never the +cause here: the quickstart repository stores 202 rows as +`:ResourceConnection:LabeledRelationship:` and none as +`:AssetConnection:`, because it was reloaded after the type change. + +**What has been fixed, and what it does and does not buy.** The NPE +itself is now a clean error. `getConnectorForAsset()` returns `null` by +contract when an asset has no connection; `FolderSurveyService` cast that +null and dereferenced it, and the framework's own +`performCheckAssetAnalysisStep()` had the same hole on its error path. +Both now report `OPEN-SURVEY-400-008` / `OPEN-SURVEY-0009` +(`NO_ASSET_CONNECTOR`), naming the asset. + +This does not make such a survey succeed — an asset with no connection +describes a resource nothing can open. It makes the failure say *which* +asset, which is what the original report was missing: the NPE named +`BasicFolderConnector`, a class the caller had never heard of, and pointed +away from the asset. + +**Why that matters for how this entry reads.** Four further defects were +found in the same connectors, all on paths nothing else exercises: both +file survey services failed on every invocation (annotations created +without a qualified name, in three places), the CSV survey dereferenced a +null root schema type, wrote a relationship property to an entity, and +produced colliding annotation qualified names. The file survey path had, +on this evidence, never run to completion. Someone hitting an NPE in this +family had every reason to suspect the asset — the errors never named +anything else. That is context for the original diagnosis, not a +criticism of it. + +**The naming trap, which is directly actionable on the pyegeria side.** +This entry already records that `initiate_file_folder_survey`'s default +(`FileSurveys:survey-folder`) and the `.http` example +(`AssetSurvey:survey-folder`) are both wrong. Confirmed and now +understood: a governance action type is registered under +`::` — two colons — so the real name is `FileSurvey::survey-folder`. `AssetSurvey` is not an engine that exists at all. The `.http` examples have been corrected upstream (`FileSurvey::survey-folder`, `ApacheKafkaSurvey::survey-kafka-server`, @@ -923,6 +852,8 @@ evidence for both dates — not yet deleted. ### ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55) +**Status 2026-10-29: moved to the Egeria Server section; the next step is Egeria's (merge of `oak2026`), then a re-test.** + **Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. Metadata Expert counts now take `pushDown`, **default `false`** (count matches the list); `true` gives the fast repository count. Relationship counts with end GUIDs and no criteria used to return 0; fixed. Egeria recommends `pushDown=false` for user-facing counts and `pushDown=true` for statistics. **pyegeria (proposed, not done):** `MetadataExpert.count_metadata_elements`/`count_relationships_between_elements` docstrings (`metadata_expert.py:3698`, `:4013`) say omitting `push_down` means `pushDown=True`; that is now wrong. Decide what `pyegeria/view/overview_metrics.py` (which omits `push_down`) should send, since its counts will become the slower list-matching kind once the platform has this. Verification below still needs running on a platform with the fix. **Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** Counts against the paged `ClassificationExplorer.get_relationships` total: `Exception` gives push_down omitted 146, false 146, true 148, list 146; `SemanticAssignment` gives 563, 563, 583, list 563. The default is now `false` and matches the list. The fast count is higher by what erinoverview cannot see (2 and 20). Read-only. The docstring fix in ISSUE-154 item 2 is now confirmed needed. @@ -1140,330 +1071,39 @@ from pyegeria import MetadataExpert, ClassificationExplorer me = MetadataExpert(view_server="qs-view-server", platform_url="https://localhost:9443", user_id="erinoverview", user_pwd="secret"); me.create_egeria_bearer_token() ce = ClassificationExplorer(view_server="qs-view-server", platform_url="https://localhost:9443", - user_id="erinoverview", user_pwd="secret"); ce.create_egeria_bearer_token() - -me.count_relationships_between_elements({"class":"FindRelationshipRequestBody","relationshipTypeName":"Exception"}) -# -> 276 -len(ce.get_relationships(relationship_type="Exception", output_format="JSON", start_from=0, page_size=5000)) -# -> 55 (all 55 have exact typeName "Exception"; no effectivity dates) -``` - -**What it is NOT:** -- Not the type filter — `count("SemanticAssignment")` = 397 = - `get_relationships` = 397; `License` 2 = 2; `Certification` 0 = 0; - `AttachedRating` 0 = 0. Every other tested type matches; only `Exception` - diverges. -- Not status/effectivity — `count("Exception")` is 276 with - `limitResultsByStatus=[ACTIVE]`, with `effectiveTime=`, and with - neither; the 55 `get_relationships` results carry no - `effectiveFromTime`/`effectiveToTime`. -- `count(no relationshipTypeName)` = 31857 (all relationships), so 276 is - a genuine type-scoped subset, not "count ignores the filter". - -**Open question for Egeria:** what does the metadata-expert count include -for `Exception` that the classification-explorer traversal excludes -(subtypes counted under the supertype? relationships to non-visible/ -anchored/dangling ends? access/zone filtering that differs between the two -OMVS)? Whichever is "true", the two APIs should agree for a given type — -or the difference should be documented. - -**Impact / workaround:** egeria-workspaces-fs's Overview dashboard keeps -**relationship** counts on `ClassificationExplorer.get_relationships` (so -"Open Exceptions" stays consistent with the Audit app at 55) and uses -native counting only for **element** counts. - ---- - -## Open Egeria Server issues (not fixable in pyegeria) - - -These need an Egeria Server-side fix, an upstream bug report, or a -deployment/config change — nothing here can be resolved by editing this -repo. Where an entry has a known pyegeria/Dr.Egeria follow-on once the -Egeria side lands, it's noted inline (also cross-referenced from the -"Open pyegeria items" section below where the follow-on is substantial -enough to track there too). - ---- - -### ISSUE-147: Every valid-metadata clear call answers OPEN-METADATA-400-011 "not found", even after a successful delete; clear-map-value deletes nothing; clear-map-name leaves the map's value set behind - -**Egeria reset 2026-10-23 ~21:10-23:54Z** (egeria-workspaces-fs, owner-approved): the quickstart database was wiped and Egeria recreated on build 81beb8e5 (odpi/egeria 4486d1a7), then Coco content was reloaded. Every GUID recorded before then is gone. The leftover `pyegeriaClearRetest*` test elements described under "Leftovers on qs-view-server" are gone with it; nothing to clean up. The fix had already passed its 2026-10-19 re-test. - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. A successful clear now returns success; clear-map-value now uses the map name; clear-map-name also removes that map name's values. Egeria asks us to drop any workaround that treats 400-011 as success. **pyegeria:** there is none (checked 2026-10-18: no 400-011 handling in `md_processing/v2/reference_data.py` or `pyegeria/omvs/valid_metadata.py`), so the follow-on proposed below is withdrawn. Nothing to change; after the re-test, remove the leftover `pyegeriaClearRetest*` test elements on qs-view-server. - -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** On throwaway properties `pyegRtClear`/`pyegRtClearMap` (typeName Asset), `clear_valid_metadata_value` (alpha, beta), `clear_valid_metadata_map_value` (mapOne/v1) and `clear_valid_metadata_map_name` (mapTwo, mapOne) all return success. A second clear of an already-removed value still answers 400-011 "not found", which is correct. Values and map names were gone afterwards. One oddity: `get_valid_metadata_values` read in the same run straight after the clears still listed alpha/beta/mapOne/mapTwo, but a read a minute later was empty. Possibly a short server-side cache; not pursued. Setup creates per-property container sets (`Egeria:ValidMetadataValue::pyegRtClear-`, `...pyegRtClearMap-`) that the clears do not remove; deleted by GUID. **Not cleaned (outside what peers cleared):** the 2026-10-13 leftovers under `pyegeriaClearRetest*`, which the platform has since turned into DataProduct/DigitalProduct elements (e.g. `98de81ed`, `4b937012`). - -**Layer:** Egeria Server (`ValidMetadataValueHandler`) · **Status:** open · **Found:** 2026-10-13. The first live run of the Valid Metadata commands showed it (the error text wasn't captured then). Re-tested the same day with the full responses captured. - -**Re-test:** run once on qs-view-server as erinoverview, with all four live sessions clearing it first. It used made-up properties `pyegeriaClearRetest` and `pyegeriaClearRetestMap` (typeName `Asset`) and called the SDK methods in `pyegeria/omvs/valid_metadata.py`. The Dr.Egeria `Clear Valid Metadata *` commands call the same methods (`md_processing/v2/reference_data.py:198-242`). - -| Call | Response | Actual effect | -|---|---|---| -| `clear_valid_metadata_value(alpha)`, set up just before | 400-011 "value alpha ... is not found" | **alpha removed** (gone from the list afterwards) | -| same call again (alpha already gone) | 400-011 | nothing to remove | -| `clear_valid_metadata_value(beta)` | 400-011 | **beta removed** | -| `clear_valid_metadata_map_value(mapOne, v1)` | 400-011 "value v1 ... is not found" | **nothing removed**: `get_valid_metadata_map_value` still returns v1 | -| `clear_valid_metadata_map_name(mapTwo)` | 400-011 | **mapTwo removed** | -| `clear_valid_metadata_map_name(mapOne)`, with v1 under it | 400-011 | the mapOne name value is removed, but the map's value set "mapOne" (holding v1) stays in `get_valid_metadata_values` | - -Request ids: `ec9052e7` and `3975182d` (alpha), `38759736` (beta), `475e68f1` (map value), `407c0e17` and `07886f29` (map names). pyegeria surfaces each one as `SERVER_ERROR_500` wrapping `relatedHTTPCode` 400 `InvalidParameterException`. - -**Cause (Egeria source, `af400039c2`, `frameworks/open-metadata-framework/.../handlers/ValidMetadataValueHandler.java`):** -1. **No `return` after the delete (lines ~541-680).** In all three `clear*` methods, `if (validValueGUID != null) deleteMetadataElementInStore(...)` is followed unconditionally by the audit log and `throw new InvalidParameterException(VALID_METADATA_MISSING ...)`. So every call throws, whether or not it deleted anything. The "not found" error is the method's normal exit. -2. **Map name dropped in `clearValidMetadataMapValue`.** It calls `getValidValueGUID(userId, typeName, propertyName, null, preferredValue)`, passing `null` where `mapName` belongs. It therefore searches for `v1` as a plain value of the property, finds nothing, and deletes nothing. -3. **`clearValidMetadataMapName` deletes only the name's own value.** It doesn't touch the per-map value set created by `setup-map-value` (the "Set of valid values for the mapOne map name ..." element) or its members. Whether that should cascade is Egeria's call. - -**Effect on Dr.Egeria:** `Clear Valid Metadata Value` and `Clear Valid Metadata Map Name` report failure although the clear worked. `Clear Valid Metadata Map Value` reports failure and really did fail. - -**Leftovers on qs-view-server:** the map value set "mapOne" with member v1 under `pyegeriaClearRetestMap`/`Asset`, plus whatever per-property container elements setup creates (`Egeria:ValidMetadataValue::pyegeriaClearRetest-Asset` and `...pyegeriaClearRetestMap-Asset`). These are test elements, not content-pack elements. RE's blueprint materializer treats `Egeria:ValidMetadataValue:` qualifiedNames as content-pack origin, but only for SolutionComponents, so they don't affect it. - -**pyegeria follow-on (not done, needs the owner's go-ahead):** Dr.Egeria's clear processor could treat 400-011 from `clear-value`/`clear-map-name` as success when a follow-up read confirms the value is gone. Otherwise, wait for the Egeria fix and re-test (see the BACKLOG.md delete re-test item). - ---- - -### ISSUE-143: `includeOnlyRelationships: ["PersonRoleAppointment"]` returns no relationships for a Person, although the unfiltered read has its role appointments - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. The `includeOnlyRelationships` filter now leaves already-retrieved relationships in place, so `performsRoles` survives `["PersonRoleAppointment"]`. **pyegeria:** no code change; tell the Resource Browser (egeria-workspaces-fs) it can use the filter again after the re-test. - -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** Erin's Person `10dc7da9` read with `includeOnlyRelationships: ["PersonRoleAppointment"]` returns `performsRoles` 8 (unfiltered: performsRoles 8, assignmentScope 15, requestedActions 13, ...; `["ContactThrough"]` still returns contactDetails 1). Read-only. Tell the Resource Browser it can use the filter again. - -**Layer:** Egeria Server (not pyegeria) · **Status:** open, not yet reported to the Egeria team · **Found:** 2026-10-13, reported by the egeria-workspaces-fs Resource Browser session; reproduced read-only the same day. - -**What:** reading a Person with a `GetRequestBody` whose -`includeOnlyRelationships` is `["PersonRoleAppointment"]` returns the element -with no relationships at all. The same read with no filter returns 8 -`performsRoles` entries, each with relationship type `PersonRoleAppointment`. -Other filters on the same Person work. - -**Reproduction** (Person `10dc7da9-e9e1-471f-9afd-46a7ff414cb7`, Erin Overview, -as erinoverview on qs-view-server; -`ClassificationExplorer.get_element_by_guid(guid, graph_query_depth=1, -output_format="JSON", body={"class": "GetRequestBody", "graphQueryDepth": 1, -"includeOnlyRelationships": [...]})`): - -| Filter | Relationship keys returned | -|---|---| -| none | performsRoles 8, assignmentScope 15, requestedActions 11, contactDetails 1, userIdentities 1, contributionRecord 1 | -| `["ContactThrough"]` | contactDetails 1 | -| `["AssignmentScope"]` | assignmentScope 15 | -| `["PersonRoleAppointment"]` | none | - -A spy on the request confirmed pyegeria sends the body exactly as given, so -the filtering happens in Egeria. - -**Likely cause (not confirmed):** Egeria's relationship filter -(`OpenMetadataHandlerBase`, around line 478) keeps a relationship when -`propertyHelper.isTypeOf(rel, name)`, which should match. But `performsRoles` is -filled in a separate step (`OpenMetadataHandlerBase`, around line 1282), so the -role appointments may be dropped or never fetched once a filter is set. - -**Workaround:** read the Person without `includeOnlyRelationships` and take -`performsRoles` from the result. The Resource Browser already does this. - ---- - -### ISSUE-130: The YAML secrets store's user directory caches a user forever, so editing an existing user needs a platform restart - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. The YAML secrets store rebuilds its user cache each time it re-reads the file (`refreshTimeInterval`), so editing an existing user no longer needs a platform restart. **pyegeria:** no code change. - -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **Not re-testable here.** Needs an edit to a user in the platform's YAML secrets file and a wait for `refreshTimeInterval`. egeria-workspaces-fs is putting that to the owner as a separate step. - -**Re-test 2026-10-19 by egeria-workspaces-fs (reported to this session, not independently re-run here):** **PASS.** A throwaway user `pyegrt130` was added to the live `coco-user-directory.omsecrets` and logged in (200, 13:19:48Z), which cached it. It was then set to `userAccountStatus: DISABLED` in place with no restart (13:19:56Z). Logins worked until 13:20:32Z and were refused with 401 at 13:21:02Z, so the cache was rebuilt about a minute after the edit (`refreshTimeInterval: 10`). The file was restored byte-for-byte afterwards. Ready to move to the Appendix. - -**Layer:** Egeria server (`YAMLSecretsStoreConnector`) · **Status:** open, reported to the Egeria lead · **Found:** 2026-10-09, while adding Resource Explorer's own account (`resourceexplorernpa`). - -**What:** `getUser(userId)` returns the entry from `userAccountMap` if it is already there. `refreshSecrets()` re-reads the YAML file every `refreshTimeInterval` minutes, but `userAccountMap` (and `securityAccessControlMap`) are created only on the first load (when `secretsStoreFile == null`) and are never cleared afterwards. Re-reading the file only helps users the cache has never seen; the comment says "this may be a new user". - -**Seen:** the account was first added without its `secrets` block, and the first token request cached it. After the password was added (and matched the servers file), `/api/token` still answered 401 for more than 10 minutes. It worked straight after a platform restart. - -**Effect:** any change to an existing user (password, status, named-list membership) needs a platform restart. - -**Fix (Egeria):** clear or rebuild `userAccountMap` and `securityAccessControlMap` in `refreshSecrets()`. - ---- - -### ISSUE-128: Retention cannot be set on Egeria 6.2: the server registers the Jackson subtype as `RetentionProperties.class` (typo), and pyegeria only allows `RetentionClassificationProperties` - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. Fixed on `main` 13 Oct (`RetentionProperties`); `oak2026` also accepts `RetentionClassificationProperties` again. Responses now carry `RetentionProperties` in `class`. **pyegeria (proposed, not done):** per the owner's 2026-10-09 decision, switch the whitelist, docstrings and Dr.Egeria pins to `RetentionProperties`: `pyegeria/omvs/classification_explorer.py:8851,8878,8919`, `md_processing/v2/processors.py:220-237` (its comment says the opposite and must be rewritten), `md_processing/v2/curation.py:136`. Accept both names in the whitelist so older platforms keep working. Not urgent: the old name is accepted again. - -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** On a throwaway collection, `classification-explorer/elements/{guid}/retention` accepted both `"class": "RetentionClassificationProperties"` and `"RetentionProperties"`; each read back `retentionBasis` 2 and its notes under `elementHeader.retention`. pyegeria's own `set_retention_classification` (old name) also succeeded. ISSUE-154 item 3's switch to the new name stays optional. - -**Status:** open, found 2026-10-09 by Resource Explorer's first Retention publish on Egeria 6.2. **Layer:** Egeria server (a type registration typo). **Decision (owner, 2026-10-09):** wait for the Egeria fix; no pyegeria workaround. Retention publishing stays blocked until then. - -**What (all proven live on 6.2, with raw HTTP POSTs to `classification-explorer/elements/{guid}/retention` on a throwaway asset):** -- `"class": "RetentionClassificationProperties"` is refused with `InvalidTypeIdException`. This is what pyegeria documents and whitelists, and it worked on the pre-6.2 server (processors.py comment, 2026-08-03). -- `"class": "RetentionProperties"` (the `.http` ground truth) is also refused with `InvalidTypeIdException`. -- `"class": "RetentionProperties.class"` gets **HTTP 200**, and the classification reads back with `retentionBasis` 2 and the notes. -- The server's known-type list in the error has 96 ids. The only Retention entry is literally `RetentionProperties.class`. That looks like an `@JsonSubTypes.Type(name = "RetentionProperties.class")` typo in Egeria's `ClassificationProperties`. -- pyegeria's `validate_new_classification_request` whitelist (`prop=["RetentionClassificationProperties"]`) refuses anything else client-side, so no Retention can be set through pyegeria on 6.2. - -(An earlier version of this entry said 6.2 knows `RetentionProperties`. That was a misread of the error text, corrected the same day.) - -**Where seen:** Resource Explorer's Curate classifications step on two repository assets. Both were refused, so nothing was written. Confidentiality, Criticality, Confidence and Impact property classes are registered normally. - -**Candidate fix:** -1. **Egeria:** correct the subtype name to `RetentionProperties`, matching the `.http` file. Report it to Mandy. -2. **pyegeria, after Egeria's fix:** change the whitelist and docstring to `RetentionProperties`, the `.http` name. No workaround for the typo, by the owner's decision. -3. Re-check `md_processing/v2/processors.py` and `v2/curation.py`, which both pin `RetentionClassificationProperties`. - ---- - -### ISSUE-108: newly created relationships stay invisible to related-element queries for up to ~20 minutes after creation - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: needs information from us.** Egeria cannot reproduce it, and the new FVT does not show it. Their best lead is still timing: a related-element query hides a relationship whose `effectiveFrom` is later than the server's current time. For one link still missing about 12 minutes after a bulk load, they ask for: -- [ ] The relationship's GUID, `effectiveFrom` and `createTime`, and the server clock at that moment. -- [ ] Whether the link is returned when fetched directly by its GUID while it is missing from the related-element query. -- [ ] Whether the loader (Dr.Egeria or pyegeria) sets `effectiveFrom` or `effectiveTime` on that run. -- [ ] A timed run with `MemberDataField` links at 200 or more, polling every minute, to show whether the delay grows with load size or stays fixed. - -**pyegeria:** Dr.Egeria's body builders send `effectiveFrom` only when the author supplies `Effective From` (`common_md_utils.py:342` and siblings), so a default load sets none; still confirm `effectiveTime` is not stamped from the client clock on link requests (a client clock ahead of the server would explain the delay). The timed run needs Egeria writes, so it needs the owner's go-ahead. - -**Layer:** Egeria Server · **Status:** open · **Found:** 2026-09-20, reported by -the user from a Dr.Egeria bulk load session (1,027 expected field-link -relationships). - -**What:** a catalog walk run immediately after a load found only 876 of 1,027 -expected field links. A second walk 12 minutes later still showed 82 -structures short. A third walk, 20 minutes after the load, showed every link -present. No errors were raised at any point in the load itself — the -relationships were created successfully, they simply did not show up in -related-element queries for a long window afterward, then all became visible -at once with no further action taken. - -**Not yet root-caused.** This looks like the same family of symptom as the -2026-08-15 Postgres-checkpoint/background-connector-load entry above (server -falling behind under load, catching up later) rather than a paging defect -like ISSUE-54 (which would not self-heal purely by waiting — ISSUE-54's -bug is about tie-ordering across separately-executed pages, not about data -not existing yet). But this has not been directly correlated with connector -log activity or checkpoint timing the way that entry was — this is a fresh -report, not yet independently reproduced or diagnosed by a pyegeria session. - -**Impact:** any workflow that verifies a bulk write immediately after -issuing it (a post-load catalog walk, an integration test, an automated -Dr.Egeria "did it work" check) can get a false-negative "missing" result -that resolves itself given enough time — worth knowing before treating an -immediate post-write verification gap as evidence of a real data-loss bug. - -**Candidate next step (not yet done):** reproduce with a smaller, timed -sample (e.g. create N relationships, poll related-element queries at 1-minute -intervals, log when each becomes visible) to get a real latency -distribution instead of the three anecdotal checkpoints here, and check -whether it correlates with the same background-connector/checkpoint load -already documented above. Worth raising with the Egeria team directly, per -the original report. - -**Update 2026-09-20 (peer check, no upstream issue filed yet):** checked -`odpi/egeria`'s issue tracker directly — nothing filed for this symptom, so -no response from the Egeria team via GitHub. Asked three peer sessions -working against the same dev Egeria server: -- **egeria-workspaces-fs-4a:** hasn't seen this specific symptom, no word - from the Egeria team either. Raised a speculative, explicitly-unconfirmed - lead worth checking: a chronic, still-open issue on `egeria-shared-kafka` - where the `GroupCoordinator` repeatedly times out writing to - `__consumer_offsets` (5000ms) and triggers a consumer-group rebalance - roughly every 90s (confirmed via broker logs 2026-09-03; root cause still - open, CPU/GC/Postgres-restarts/virtiofs I/O already ruled out; historically - "Information" severity, self-heals within seconds, not chased further - since it wasn't visibly causing downstream harm). If `qs-view-server`'s - related-element queries are backed by Kafka-driven indexing/event - propagation rather than a direct repository read, a chronic rebalance - storm could plausibly manifest as exactly this "relationship exists but - isn't visible to queries yet" lag. **Explicitly not a confirmed - correlation** — the timescales don't obviously line up (a recurring ~90s - self-healing blip vs. a one-shot ~20-minute resolution) — and the peer - offered to check Kafka broker logs around the original bulk-load window - for the same rebalance pattern if useful. -- **Resource Explorer experimental UI:** no direct evidence either way — - today's two live writes against the same server (a Governance Action - Process Step upsert, and a native-survey `SurveyReport` lookup) both - checked either *immediately* after the write or only *after* polling to a - terminal status, never in the 1-20 minute gap this entry describes, so - neither confirms nor rules it out. Flagged that a background agent on - their side doing a related connection-refresh fix will trigger fresh - engine actions/relationships shortly and will report back if it surfaces - the same lag. -- **Egeria-trellis backlog review:** also nothing from the Egeria team (no - channel to them; checked RE docs/Backlog/design notes for this exact - shape first — a checked "no", not a guess). Offered a different, measured - precedent on this same platform as a *pattern* to distinguish causes, not - a diagnosis: `packages/resource-explorer/docs/investigation-classification-and-zoning-design.md` - (~line 830) documents a privately-zoned element still readable by a - non-owner *six minutes* after the control was written (denial began at - seven), caused by the security connector's `refreshTimeInterval` (10 min) - reload — "read it back immediately, so it must be fine" was exactly the - misleading evidence there too. Their key diagnostic point: a - **fixed-wall-clock cause** (everything becomes visible at once, at a time - unrelated to load size — test with 50 links instead of 1,027 and see if - the delay is still ~20 min) and a **volume-proportional cause** (async - indexing/event-driven materialisation catching up, arriving in a stream) - look identical from a single observation but have different fixes; the - 876→945(implied)→1,027 partial-recovery-at-12-minutes data point *mildly* - favours volume-proportional, but "two points don't make a curve." Also - worth checking directly: fetch one of the missing links by its own GUID - during the window — if that resolves but the relationship is still absent - from the related-element query, the write landed fine and only the - traversal/index path lags, narrowing which service is actually behind. If - this gets filed upstream, the volume-vs-fixed measurement is what turns - it into a real bug report instead of getting bounced as "eventual - consistency, working as intended." - -Two independent, unconfirmed leads now on record (Kafka rebalance storms; -a refresh-interval-style cache) alongside a concrete diagnostic method -(fixed-wall-clock vs. volume-proportional, plus the direct-GUID-fetch -isolation check) for telling them apart before assuming either. Next real -step is still reproducing with a smaller, timed, instrumented load using -that method — not yet done. - -**Reproduction attempt, 2026-09-20 (small end of the curve, not reproduced):** -after clearing with all 5 live peer sessions first (per `coordinate-shared-writes`), -ran a timed, instrumented small-scale reproduction against `qs-view-server`: -20 throwaway `Collection` elements created and each linked to a parent via -`CollectionMembership` (not the original `MemberDataField`/"field link" type -— a different relationship type, noted as a limitation below), polling -`get_collection_members` every 30s for up to 20 minutes, with a direct-GUID -isolation check ready for anything still missing at the deadline. - -**Result: all 20 relationships were visible on the very first poll, ~1.1s -after the last one was created — no lag at all.** Cleanup (delete all 21 -fixture elements) verified two independent ways: a direct-GUID fetch on -each (`404`/`OMAG-REPOSITORY-HANDLER-404-007`, confirmed real soft-delete, -not an ISSUE-63-style silent no-op) and a broad `find_collections` search -for the `test-fixture-issue108` prefix (0 results). The script itself was -run from an ephemeral per-session scratch directory, not committed anywhere -— reproduce via the same approach (create N Collections linked to a parent -via `_async_add_to_collection`, poll `_async_get_collection_members`) if -needed later; the approach, not the script file, is what's worth keeping. - -**Per Egeria-trellis backlog review's own framework, this is one point on -the curve, not a refutation.** It's consistent with either "no bug under -current load" or "the lag is genuinely volume-proportional and 20 elements -is nowhere near the ~1,027-link scale that surfaced it originally" — the -one data point this run *can't* distinguish those two. Also a real -methodology gap versus the original report: this used `CollectionMembership` -on plain `Collection` elements, not the `MemberDataField` "field link" -relationship the original 1,027-link load actually exercised (ISSUE-102 is -in the same `MemberDataField` neighborhood, worth checking together). Server -load conditions also weren't recorded this run (no `ListAgents`/`:8810` -snapshot taken alongside the timings, despite the earlier plan to do so — -an execution gap, not a design one). - -**Candidate next step, still open:** a same-relationship-type -(`MemberDataField`), same-order-of-magnitude (~200+) run, following the -same peer-clearance and independent-verification discipline, is what would -actually test the volume-proportional hypothesis — this run didn't attempt -that scale on its own initiative, since committing to it wasn't part of -what peers were asked to clear. - -**Egeria team response, relayed 2026-10-05 (source `oak2026` fb3d6fce53; **fixes are in two unmerged PRs, so no build contains them, and nothing was tested against a live platform**):** not validated; no cache found. Best lead: a client clock ahead of the server's, with `effectiveFrom` set from it (related-element queries silently hide a relationship whose `effectiveFrom` is later than the server's "now"). They need, for one link still missing at ~12 minutes: its `effectiveFrom` and `createTime`, the server clock then, and whether the loader sets `effectiveFrom`. **Our facts:** pyegeria sets no `effectiveFrom`/`effectiveTo`/`effectiveTime` by default (model defaults are `None`; no clock call is tied to any `effective*` field; Dr.Egeria forwards an "Effective From" only if typed). Host and container clocks agree within 1 s on 2026-10-05; the 2026-09-20 load's values are unavailable. + user_id="erinoverview", user_pwd="secret"); ce.create_egeria_bearer_token() -**Update 2026-10-10:** the `oak2026` source (`fb3d6fce53`) the team examined is now in Egeria `main` (merged 2026-10-06, odpi/egeria#9363). No fix was made for this issue; the team's questions above still stand. +me.count_relationships_between_elements({"class":"FindRelationshipRequestBody","relationshipTypeName":"Exception"}) +# -> 276 +len(ce.get_relationships(relationship_type="Exception", output_format="JSON", start_from=0, page_size=5000)) +# -> 55 (all 55 have exact typeName "Exception"; no effectivity dates) +``` ---- +**What it is NOT:** +- Not the type filter — `count("SemanticAssignment")` = 397 = + `get_relationships` = 397; `License` 2 = 2; `Certification` 0 = 0; + `AttachedRating` 0 = 0. Every other tested type matches; only `Exception` + diverges. +- Not status/effectivity — `count("Exception")` is 276 with + `limitResultsByStatus=[ACTIVE]`, with `effectiveTime=`, and with + neither; the 55 `get_relationships` results carry no + `effectiveFromTime`/`effectiveToTime`. +- `count(no relationshipTypeName)` = 31857 (all relationships), so 276 is + a genuine type-scoped subset, not "count ignores the filter". +**Open question for Egeria:** what does the metadata-expert count include +for `Exception` that the classification-explorer traversal excludes +(subtypes counted under the supertype? relationships to non-visible/ +anchored/dangling ends? access/zone filtering that differs between the two +OMVS)? Whichever is "true", the two APIs should agree for a given type — +or the difference should be documented. +**Impact / workaround:** egeria-workspaces-fs's Overview dashboard keeps +**relationship** counts on `ClassificationExplorer.get_relationships` (so +"Open Exceptions" stays consistent with the Audit app at 55) and uses +native counting only for **element** counts. +--- ## Awaiting re-test (fix landed, not yet verified live) @@ -1481,6 +1121,99 @@ sessions first. --- +### ISSUE-146: Deleting a journal entry fails with OMAG-REPOSITORY-HANDLER-400-008, and pyegeria's `remove_note` reaches the delete only through the generic Asset Maker service + +**Status 2026-10-29: the pyegeria part (`remove_note` routing) was fixed in #451, released in 6.2.3; Egeria says the 400-008 is the provenance check working as designed.** What's left is the `remove_note` re-test on a NEW journal entry (below). + +**Egeria reset 2026-10-23 ~21:10-23:54Z** (egeria-workspaces-fs, owner-approved): the quickstart database was wiped and Egeria recreated on build 81beb8e5 (odpi/egeria 4486d1a7), then Coco content was reloaded. Every GUID recorded before then is gone. The test JournalEntries `fef83ed5` and `8bec72e4` on Erin's profile no longer exist, so the pending `remove_note` re-test below needs a NEW entry made with `journal_my_activity` (and, for the 400-008 case, one owned by an external source). The 2026-10-18 Egeria analysis (works as designed; 400-008 is the provenance check) still applies. + +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: works as designed (Egeria).** Asset Maker's `deleteAsset` is the right call: a JournalEntry is a Notification, so an asset, and no separate note delete is needed. Egeria's new FVT creates entries with My Profile's `journal-my-activity` and deletes them with `deleteAsset`, with no external source; every case passes on in-memory and PostgreSQL repositories. **400-008 is the provenance check:** it refuses a change made without an external source to an element an external source owns. No external source → 400-008; a different external source → 400-007; the matching external source → success. So an entry that returns 400-008 was not created locally, and its delete needs that entry's `externalSourceGUID` and `externalSourceName`. If 400-008 appears on an entry created through My Profile, that is a bug and Egeria wants: (1) the full error message (it names the type, GUID, provenance and metadata collection of the refused instance); (2) how the entry was created (which request, and which external source if any). Also new: purge applies the same provenance check. **pyegeria:** the `remove_note` clean-up below stays (route + docstring). Re-test on `8bec72e4` against a platform with the fixes; if 400-008 recurs, capture the full message for Egeria, since that entry was created locally through `journal-my-activity`. + +**Layer:** Egeria Server (pyegeria clean-up) · **Status:** open · **Found:** 2026-10-13, reported by the egeria-workspaces-fs Resource Browser session from a live private-journal test. + +**What:** as erinoverview, `_async_remove_note` (`pyegeria/core/_server_client.py`) +was called on a JournalEntry created by `MyProfile.journal_my_activity`. It +failed with `OMAG-REPOSITORY-HANDLER-400-008` (`LOCAL_CANNOT_CHANGE_EXTERNAL`): +`deleteMetadataElementInStore` cannot modify Erin's UserIdentity (`ff983b64`), +which has Content Pack provenance ("Coco Pharmaceuticals Combination"). The +peer verified that nothing was removed. The entry keeps its `actionSource` and +`presentInNoteLogs` relationships, and Erin's Person still has 1 identity, +1 contact detail, 8 roles and 15 scopes. Test entry `fef83ed5` is still on +Erin's profile, waiting for the owner's decision. + +**The route is right:** in 6.2, notes are Actions. The type chain is +JournalEntry → Notification → Action → Process → Asset. Egeria main +(`af400039c2`) has no note, notification or action delete endpoint. +`AssetHandler.createNoteLogEntry` has no matching remove, and My Profile has +`journal-my-activity` with no matching remove. The only delete for an Action is +the generic `/assets/{assetGUID}/delete` (`AssetMakerResource`), which is what +`remove_note`'s `feedback-manager/assets/{guid}/delete` reaches, because +generic view services answer under any URL marker. Feedback Manager can only +remove a whole note log (`note-logs/{noteLogGUID}/remove`). + +**Why it fails (likely, not verified):** `AssetHandler.createAction` anchors the +entry to its action source, linked by `ActionRequester`. For a journal entry +that is the caller's profile. Deleting an anchored element updates its anchor, +and Erin's identity is owned by the content pack, so the local delete is +refused. If so, no journal entry for a content-pack user can be deleted. That +is for the Egeria team. + +**Anchor read (peer, read-only):** JournalEntry `fef83ed5` has anchorGUID +`10dc7da9`, Erin's Person profile. Its `actionSource` is ActionRequester → +Person `10dc7da9`, and `presentInNoteLogs` is AttachedNoteLogEntry → NoteLog +`f780511a` (MyJournal). No relationship from the entry reaches UserIdentity +`ff983b64`, which the error names. That identity is anchored to the same +Person. So the refusal comes from the delete touching the anchor's graph, most +likely the LatestChange update on the anchor, or a provenance check across +elements that share the anchor, not from a direct link. + +**Delete-path trace (2026-10-13, Egeria source, read-only):** the asset delete +(`AssetMakerResource.deleteAsset` → `AssetHandler.deleteAsset`) and the Metadata +Expert delete (`MetadataExpertResource` → `MetadataExpertRESTServices`) both call +`OpenMetadataClient.deleteMetadataElementInStore`, then the same +`MetadataElementHandler` → `OpenMetadataAPIGenericHandler.deleteBeanInRepository` +code. Neither does any extra checking, so switching routes cannot avoid 400-008. +The error is thrown only by `RepositoryErrorHandler.validateProvenance`, when +the delete removes an entity or relationship whose provenance is not +LOCAL_COHORT and the request carries no `externalSourceGUID`. That happens in +`removeRelationship` for each relationship on the note and in `removeEntity` +for the note itself. No anchor (LatestChange) update was found on the delete +path, which corrects "Why it fails" above. But the error names UserIdentity +`ff983b64`, which is not linked to the note, so the exact instance is still +unexplained. Untried options are to pass the content pack's metadata-collection +id as `externalSourceGUID`, or to use `deleteMethod: ARCHIVE`, which goes +through a different handler. + +**Note-privacy findings (egeria-workspaces-fs session, 2026-10-13, with owner +approval):** a second test JournalEntry, `8bec72e4-6da6-40a4-943e-d81e2173ec9b`, +was created on Erin's profile (`10dc7da9`) and cannot be deleted either until a +working delete exists. +- `POST my-profile/journal-my-activity` ignores `initialClassifications` and + stamps only ZoneMembership `['erinoverview']`. +- `POST classification-explorer/elements/{guid}/zone-membership` with + `['resource-explorer-private','erinoverview']` makes the entry private. + erinoverview can read it. peterprofile and garygeeke get 403 + `OPEN-METADATA-SECURITY-403-020` by GUID, and 0 hits from + `feedback-manager/assets/by-search-string`. +- The journal's NoteLog (`f780511a`) stays at `['erinoverview']`, so other + users can see that the log exists but not the hidden entries. +- The owner chose a temporary workaround for the Resource Locator: call + `add_zone_membership(['resource-explorer-private', user])` straight after + `journal-my-activity`. Review it once Egeria's journal fix lands. + +**pyegeria clean-up:** make the `remove_note` route explicit and describe it as +an Action delete in the docstring. The owner chose the Metadata Expert delete +(`metadata-expert/metadata-elements/{guid}/delete`) for now. Per the trace +above, this is a clarity change, not a fix for 400-008. + +**Re-test after Egeria's delete fixes are published:** the owner said +(2026-10-13) that Egeria is working on delete issues now. Once that work is +published, re-test `remove_note` on `8bec72e4` (plain delete, then `ARCHIVE` if +needed). Tell the egeria-workspaces-fs session before deleting anything on +Erin's profile. `fef83ed5` stays. + +--- + ### ISSUE-125: `deleteEnumDef` answers 500 "unknown TypeDef" for an enum the server lists -- an enum added through the API cannot be removed **Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). `deleteEnumDef` removes an enum added through the API. **pyegeria:** no code change; the re-test should also remove the stray enum left on the dev platform. @@ -2195,84 +1928,196 @@ The improved audit flags all eight on the pre-fix code. - `_get_subclient` passes the user and token to every sub-client, and there is no token cache. Intended precedence (CLAUDE.md): explicit `--userid` > OS env > `.env` > `config.json` > default. -**Real issues (pyegeria):** -1. **Inconsistent resolution between entry points:** `md_processing/dr_egeria.py:93` (module constants) and `:923` (its own `__main__` path) read `os.environ.get("EGERIA_USER", "erinoverview")` / `EGERIA_USER_PASSWORD` directly, bypassing `.env` and `config.json`. `md_processing/md_processing_utils/common_md_utils.py:29-30` does the same. So `python -m md_processing.dr_egeria` and the `dr_egeria` CLI disagree whenever the user is set only in `.env` or `config.json`. -2. **Import-time defaults:** `commands/cat/dr_egeria.py:32-33` computes the `--userid` / `--user_pass` defaults when the module is imported. Code that imports it, then sets `EGERIA_USER` and invokes the command in the same process, gets the earlier user. This doesn't affect normal CLI use, where each run is a new process. -3. **Help text:** "Overrides EGERIA_USER (env var or .env file)" doesn't mention `config.json`, or that the default is fixed at import. +**Real issues (pyegeria):** +1. **Inconsistent resolution between entry points:** `md_processing/dr_egeria.py:93` (module constants) and `:923` (its own `__main__` path) read `os.environ.get("EGERIA_USER", "erinoverview")` / `EGERIA_USER_PASSWORD` directly, bypassing `.env` and `config.json`. `md_processing/md_processing_utils/common_md_utils.py:29-30` does the same. So `python -m md_processing.dr_egeria` and the `dr_egeria` CLI disagree whenever the user is set only in `.env` or `config.json`. +2. **Import-time defaults:** `commands/cat/dr_egeria.py:32-33` computes the `--userid` / `--user_pass` defaults when the module is imported. Code that imports it, then sets `EGERIA_USER` and invokes the command in the same process, gets the earlier user. This doesn't affect normal CLI use, where each run is a new process. +3. **Help text:** "Overrides EGERIA_USER (env var or .env file)" doesn't mention `config.json`, or that the default is fixed at import. + +**Fix (2026-10-21):** a new `pyegeria.core.config.resolve_egeria_credentials(user_id=None, user_pwd=None)` resolves, when called: an explicit value > the `EGERIA_USER` / `EGERIA_USER_PASSWORD` environment variables > the settings User Profile (`config.json` / `.env`) > "erinoverview" / "secret". +- The `dr_egeria` and `dr_egeria_folder` CLIs now default `--userid` / `--user_pass` to `None` and call it when they run. Their help text says so. +- `md_processing/dr_egeria.py` (module constants and its `__main__` path) and `common_md_utils.py` constants now use it too. The module-level `EGERIA_USER` names are kept for importers. +- Tests: `tests/micro-tests/test_issue_161_dr_egeria_credentials.py`. Five cases: precedence; env set after import, for both CLIs; config-only user; explicit flag wins. The three behaviour cases fail with the old caller code. The full micro-test suite (1,425) passes in a clean python:3.13-slim container. +- **Not changed:** other `commands/cat` tools (`list_glossaries`, `get_asset_graph`, `run_report`, `get_collection_tree`, `get_project_dependencies`, `list_user_ids`, `view_schema_tree`) still read `EGERIA_USER_PASSWORD` straight from the environment. They could adopt the helper later. + +**Originally proposed:** +- Resolve credentials through `settings.User_Profile` in all three places. +- Look them up when the command runs (click `default=None`, then resolve inside the command). +- Add a micro-test that both entry points pick up a user set only in `config.json` or `.env`. + +--- + + +### ISSUE-160: `AutomatedCuration`'s template builders miss placeholders Egeria's templates now use, so created elements keep literal `~{name}~` values (folder endpoint `~{directoryAddress}~` breaks folder surveys) + +**Layer:** Pyegeria · **Status:** fixed in #474 and the follow-up #475 (released in 6.2.8), with the Dr.Egeria Postgres follow-up in #478 (released in 6.2.9). The items originally listed as not fixed are kept below for the record. **Found:** 2026-10-21, live by the RE coordinator against Egeria `main` 05a5e982 with pyegeria 6.2.7. Two FileFolders made with `create_folder_element_from_template` had Endpoint `networkAddress` = `~{directoryAddress}~`. Egeria's Folder Survey then failed with `BASIC-FILE-CONNECTOR-404-001` "The folder named ~{directoryAddress}~ ... does not exist" (engine action `c1d924b5`). The same folders also kept `versionIdentifier` = `~{versionIdentifier}~` and `deployedImplementationType` = `~{deployedImplementationType}~` (record `~/.re-gate/repair-folder-endpoints-20261021T143415Z.json`). + +**Cause (Egeria `origin/main` d8f72cbe25, read-only):** no placeholder was renamed (`git log -S` on `PlaceholderProperty.java`). Egeria **added** placeholders that pyegeria never sends: +- **`directoryAddress` / `fileAddress`:** added by `156650efd6` (2026-07-19, "file system control for file cataloging"). They are now the Endpoint address in the folder and file templates (`DataAssetTemplateDefinition` `FILE_FOLDER_TEMPLATE`, `DATA_FOLDER_TEMPLATE`, `CSV_FILE_TEMPLATE`, ...). The templates still use `directoryPathName` / `filePathName` in the qualifiedName, so both are needed. +- **`deployedImplementationType`:** in every folder/file placeholder list (`getFolderPlaceholderPropertyTypes`, `getDataFilesPlaceholderPropertyTypes`). +- **`databaseDescription`:** the PostgreSQL Relational Database template's description placeholder (`PostgresPlaceholderProperty.DATABASE_DESCRIPTION`); pyegeria sent `description`. + +An unsent placeholder is stored as the literal `~{name}~`. `body_slimmer` drops `None` values, so an optional argument left unset has the same effect. + +**Fixed on the branch** (`pyegeria/omvs/automated_curation.py`; signatures unchanged): +| Method | Added | +|---|---| +| `create_folder_element_from_template` | `directoryAddress` (= `path_name`), `deployedImplementationType: "File System Directory"` | +| `create_csv_data_file_element_from_template`, `get_create_csv_data_file_element_from_template` | `fileAddress` (= `file_path_name`), `deployedImplementationType: "CSV Data File"` | +| `create_secrets_store_element_from_template` | `fileAddress` (= `file_path_name`), `deployedImplementationType: "YAML File Secrets Collection"` | +| `create_postgres_database_element_from_template` | `databaseDescription` (= `description`; `description` still sent) | + +Tests: `tests/micro-tests/test_issue_160_template_placeholders.py` (5 cases; all fail on the old code, all pass on the fix). The full micro-test suite (1,409) passes in a clean python:3.13-slim container. + +**Live check 2026-10-21 14:52Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; the fixed code on this branch): `create_folder_element_from_template("/tmp/pyeg_rt_folder_20261021T145254Z", ..., version="1.0")` created FileFolder `e9a45c7f` with `deployedImplementationType` "File System Directory" and `versionIdentifier` "1.0". Its anchored Endpoint `8f003544` had **`networkAddress` = `/tmp/pyeg_rt_folder_20261021T145254Z`**. The folder, Connection `18ef2ee9` and Endpoint had no `~{...}~` left in their own properties. A metadata-expert delete (`deleteMethod: PURGE`, `cascadedDelete: true`) removed all three; each was verified gone by GUID. A first run (`a3475d3e`) confirmed the folder fields but its script missed the Endpoint. Exact `networkAddress` searches (forLineage) afterwards found no leftover for that run either. The survey itself was not run. + +**Follow-up fix 2026-10-21 (owner: "fix them all in a new pr")** on branch `fix/issue-160-remaining-placeholders`, built on #474. All 13 hard-coded builders now wrap their values in `_placeholder_values`, which sends `""` for any placeholder left `None`, so no `~{name}~` literal is stored. `body_slimmer` drops only `None`. A folder created without a file system therefore gets qualified name `FileFolder:::/path`. +- **Postgres server and database:** new optional `secrets_store_path_name` / `secrets_collection_name`, sent as `secretsStorePathName` / `secretsCollectionName`, plus `version` (and `resource_name` on the server, defaulting to the server name). `db_user` / `db_pwd` are now optional, deprecated and no longer sent; passing either raises a `DeprecationWarning`. +- **Kafka server:** `resourceName` (defaults to the server name) and `versionIdentifier`. +- **Unity Catalog:** + - UC server: `resourceName`, defaulting to the server name; + - UC catalog: `ucServerQualifiedName`; + - UC table and function: `rootSchemaTypeQualifiedName`; + - UC volume: `directoryPathName` and `localStorageLocation`. + Each is a new optional parameter on the async method and its sync wrapper. +- **Item 4 needed no change:** "Keystore File" is a real template (`DataAssetTemplateDefinition.KEYSTORE_FILE_TEMPLATE`), and live, both "YAML File Secrets Collection" and "Keystore File" resolve to template `fbcfcc0c`. + +Tests: `tests/micro-tests/test_issue_160_remaining_placeholders.py`, 11 cases, all failing on #474's code. The full micro-test suite (1,420) passes in a clean python:3.13-slim container. **Dr.Egeria follow-up, done 2026-10-21** (after 6.2.8; branch `claude/elegant-chatterjee-d280de`): the `Create Postgres Server Element` / `Create Postgres Database Element` commands no longer offer `Database User Id` / `Database Password`. Both attributes were deleted from the Asset Maker compact spec, since no other command used them. The commands now offer `Secrets Store Path Name`, `Secrets Collection Name` and `Version Identifier`, and the server also offers `Resource Name` (the canonical Governance Officer definition, copied). `AssetMakerProcessor` passes them as `secrets_store_path_name` / `secrets_collection_name` / `version` / `resource_name` and no longer passes `db_user` / `db_pwd`. Templates, help and report specs were regenerated, and the templates were copied to egeria-workspaces and trellis on local branches. Test: `tests/micro-tests/test_postgres_element_secrets.py`. Not run against a live server. + +**Follow-up live check 2026-10-21 15:16Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; #475 code): +- **(a)** `create_postgres_server_element_from_template("pyeg_rt_pg_…", "pyeg-rt.invalid", "5432", secrets_store_path_name=…, secrets_collection_name=…)`, with no `db_user` / `db_pwd`, created SoftwareServer `b6b2ebff`: `resourceName` = the server name, `versionIdentifier` and `description` = `""`, and no `~{...}~` in its own properties. Its Connection `55686913` and Endpoint `59014065` (`jdbc:postgresql://pyeg-rt.invalid:5432/postgres`) had no literals either. +- The server had **no CatalogTarget link** (its relationships were sourcedFromTemplate, serverEndpoints, capabilities, connections), so the JDBC integration connector never saw it. +- I did not find where Egeria stored the secrets-store values. They were not in the Connection's own properties, and no literal was left. Unconfirmed. +- **(b)** `create_folder_element_from_template("/tmp/pyeg_rt_folder2_…", …, None)`, with no file system, version or description: qualifiedName `FileFolder:::/tmp/pyeg_rt_folder2_…`, `resourceName` `:/tmp/…`, `versionIdentifier` / `description` = `""`, `deployedImplementationType` "File System Directory", and Endpoint `networkAddress` = the path. No literals on the folder, Connection or Endpoint. +- Both were purged (metadata-expert delete, `PURGE`, `cascadedDelete`), and all six GUIDs were verified gone. Name, host, stamp and secrets-file searches afterwards found nothing, which covers the server's capabilities too. + +**Originally listed as not fixed** (kept for the record): +1. **Postgres server and database credentials:** both templates now take `secretsStorePathName` and `secretsCollectionName` (credentials come from a secrets store). pyegeria never sends them, so the connection keeps literals. pyegeria still sends `databaseUserId` / `databasePassword`, which the templates no longer list (`312c172639`, `56fa898e43`), so those are dead. Proposed: add optional `secrets_store_path_name` / `secrets_collection_name` parameters, and deprecate `db_user` / `db_pwd`. +2. **Optional arguments left unset:** `versionIdentifier`, `description`, `fileSystemName` and the like are dropped when `None`, so they stay `~{...}~` (RE's `versionIdentifier` case). Options: send `""` for unset optional placeholders, or document that callers must pass them. +3. **Unsent template placeholders in the Unity Catalog and Kafka builders:** + - UC catalog: `ucServerQualifiedName`; + - UC table and function: `rootSchemaTypeQualifiedName`; + - UC volume: `directoryPathName`, `localStorageLocation`; + - UC server and Kafka server: `resourceName` (Kafka also `versionIdentifier`). + Add parameters for them where a caller can know the value. +4. **The secrets-store builder's fallback** to a "Keystore File" technology type (`_async_get_template_guid_for_technology_type`) matches no template on `origin/main`. + +Builders with no hard-coded keys (`create_elem_from_template`, Dr.Egeria's "Placeholder Property Values" attribute) are unaffected. The UC schema builder sends all its placeholders. + +--- + + +### ISSUE-159: no archive wrappers on the project or collection managers, and no restore wrapper at all, although Egeria offers both + +**Live-verified 2026-10-29 (Egeria `main` 8de9572c quickstart, qs-view-server, as peterprofile; throwaways prefixed `b2-live-509::`, deleted and verified gone by search and by GUID; peers cleared first): PASS.** `archive_project` and `archive_collection` each left the element hidden from normal reads but readable with `forLineage`, and `restore_metadata_element` brought both back live. Fixed in #509, for 6.2.15. + +**Status 2026-10-29: item 2 done on branch `fix/issues-154-158-159-85-tracker-cleanup`, for 6.2.15; item 3 (live test) still to do.** `ProjectManager.archive_project` and `CollectionManager.archive_collection` send each manager's existing delete endpoint with `deleteMethod: ARCHIVE` (shared helper `ServerClient._archive_delete_body`). Traced in Egeria main 8de9572c: `ProjectHandler.deleteProject` and `CollectionHandler.deleteCollection` both pass the `DeleteOptions` to `openMetadataClient.deleteMetadataElementInStore`, and the store routes ARCHIVE to `archiveMetadataElementInStore`. Restore: `MetadataExpert.restore_metadata_element`. + +**Layer:** Pyegeria · **Status:** open, logged only (no fix without the owner's word) · **Found:** 2026-10-20, raised by the Resource Explorer expansion-architecture session: RE's investigation-delete ruling needs a Project archived through the project manager and restored by GUID. Facts confirmed here against Egeria `origin/main` 05a5e982 and pyegeria upstream `main`, read-only. + +**Egeria:** `OpenMetadataAPIGenericHandler.java` (common-services/generic-handlers): +- `archiveBeanInRepository` (`:3289`) classifies the element Memento, renames each affected Referenceable's qualifiedName to `_archivedOn_` (marker `archivedQualifiedNameMarker = "_archivedOn_"`, `:51`), and records one `lastOperationId` per operation. +- `restoreBeanInRepository` (`:4170`) restores an element and everything the same delete or archive changed, given any GUID that operation touched. + +The OMF store's delete (`OpenMetadataStoreRESTServices`, around `:4118`) also routes `deleteMethod: ARCHIVE` to `archiveMetadataElementInStore`, and resolves `LOOK_FOR_LINEAGE` to ARCHIVE when lineage relationships exist. + +**pyegeria:** +- Archive exists only as `MetadataExpert.archive_metadata_element` (`metadata-expert/metadata-elements/{guid}/archive`). +- There is no `archive_project`, `archive_collection` or other manager-level archive. +- There is no `restore_*` method anywhere. ISSUE-155 item 5 already lists Metadata Expert `metadata-elements/{guid}/restore` and `related-elements/{guid}/restore` as missing. + +**Possible route today (unverified):** `ProjectManager.delete_project(guid, body={"class": "DeleteElementRequestBody", "deleteMethod": "ARCHIVE"})`. pyegeria passes `deleteMethod` through, but whether the project-manager delete path honours ARCHIVE as the OMF store delete does was not traced. + +**Proposed (not made):** +1. ~~Add `MetadataExpert.restore_metadata_element` / `restore_related_elements` (and undo), per ISSUE-155 item 5.~~ Done 2026-10-23 (branch `fix/role-appointment-class-names`). Items 2 and 3 are still open. +2. Add `ProjectManager.archive_project` and `CollectionManager.archive_collection`. These can be thin wrappers that send `deleteMethod: ARCHIVE` once that path is confirmed live, or call Metadata Expert archive. +3. Live-test archive then restore of a throwaway Project, including the `_archivedOn_` rename and its reversal. + +--- + +### ISSUE-158: `GOV_DEF_PROPERTIES_LIST` lacks `DataQualityRuleProperties`, and Dr.Egeria's governance create fallback silently drops every type-specific attribute + +**Status 2026-10-29: fixed on branch `fix/issues-154-158-159-85-tracker-cleanup`, for 6.2.15.** 1. `GOV_DEF_PROPERTIES_LIST` is now `GovernanceDefinitionProperties` plus its whole `@JsonSubTypes` tree from Egeria main 8de9572c (44 classes, no duplicates), so it includes `DataQualityRuleProperties`, `RiskProperties`, `GovernanceProcedureProperties` and the others the hand-kept list missed. 2. `GovernanceProcessor`'s base-class retry now fails the command when it would drop a set type-specific field, naming the fields (a field moved into `additionalProperties`, as SecurityGroup's `distinguishedName` is, counts as kept). Not run live: whether any type still hits 400-029 on Egeria main is still unknown. + +**Layer:** Pyegeria · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, raised by the TestGen evaluation session while drafting a DataQualityRule convenience-API spec (`~/localGit/egeria-v6/wt-testgen-notes/packages/resource-explorer/docs/design-notes/SPEC-PYEGERIA-DATA-QUALITY-RULE.md`, uncommitted, pending owner review). Both points confirmed in source here, read-only; not run live. + +1. **Whitelist gap.** `GOV_DEF_PROPERTIES_LIST` (`pyegeria/omvs/governance_officer.py:37`) is the client-side property-class whitelist for `create_governance_definition` (`:426`) and `update_governance_definition` (`:955`). It has no `DataQualityRuleProperties`, although Egeria `main` has that class (`frameworks/.../properties/governance/DataQualityRuleProperties.java`). **Corrected 2026-10-19:** the list is enforced only on update. `validate_new_element_request` (`pyegeria/core/_server_client.py:6115`) has its class check commented out, so a create with `DataQualityRuleProperties` reaches Egeria unchanged. `validate_update_element_request` (`:6252`) does check a dict body's class against the list, so `update_governance_definition` refuses a DataQualityRule body client-side ("unexpected property class name"). No Dr.Egeria command creates DataQualityRules today. **Proposed:** add `DataQualityRuleProperties`, and drop the list's duplicates (`RegulationProperties`, `GovernanceControlProperties` and `GovernanceApproachProperties` each appear twice). Better still, check the list against Egeria's `GovernanceDefinitionProperties` `@JsonSubTypes` so it can't drift again. +2. **Silent fallback.** In `GovernanceProcessor`'s create path (`md_processing/v2/governance.py:203-221`), any non-SecurityGroup create that Egeria refuses with `OMAG-COMMON-400-029` (property class not accepted for `createGovernanceDefinition`) is retried with `_to_base_governance_definition_properties` (`:96`). That keeps only the cross-type fields (qualifiedName, displayName, description, category, identifier, contentStatus, versionIdentifier, effective dates, additional/extended properties). The element is created, but every type-specific attribute is dropped (e.g. a DataQualityRule's quality fields, a metric's `measurement`/`target`, a control's `implementationDescription`). The only sign is a `logger.warning`, and Dr.Egeria reports success. **Proposed:** remove the generic fallback, or keep it only for the server-compat case it was written for. Otherwise make it fail visibly, naming the dropped fields in the command's result. Live-test which current types, if any, still hit 400-029 on Egeria `main` before deciding. + +--- + +### ISSUE-157: Dr.Egeria `Create Service Level Objective` sends `entitlements`/`obligations`/`restrictions` as empty lists; Egeria types them as maps + +**Status 2026-10-29: fixed in #464, released in 6.2.6.** Moved here from the open list during the 2026-10-29 tracker check. + +**Fixed 2026-10-19 (owner approved) on branch `fix/issue-156-157-catalog-target-slo` (956b199c, off upstream main 770dfe31; not pushed, not released).** `set_gov_prop_body` now defaults the three fields to `None`. **Cause confirmed live** on Egeria `main` 05a5e982: the old code got a 400 `com.fasterxml.jackson.databind.exc.MismatchedInputException`, "Cannot deserialize value of type `java.util.LinkedHashMap` from Array value ... ServiceLevelObjectiveProperties[\"entitlements\"]", and created nothing. The fixed code created `ServiceLevelObjective::pyeg_rt_slo` (`6258d428`) with `implementationDescription` intact; it was deleted and verified gone by GUID and by name. Micro-tests: `tests/micro-tests/test_issue_156_157_catalog_target_slo.py`; the full micro-test suite (1,394) passes in a clean python:3.13-slim container. + +**Layer:** Pyegeria (Dr.Egeria) · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, reported by the Egeria team against pyegeria 6.2.5: the command "sends a field the platform rejects", so their hourly refreshes use the content pack's existing within-an-hour SLO instead. The cause below is from reading source; it was not reproduced live, since a wrong guess would create an SLO on the shared platform. The Egeria team's error text would confirm it. + +**Likely cause:** `set_gov_prop_body` (`md_processing/md_processing_utils/common_md_utils.py:588`) always sets `entitlements`, `obligations` and `restrictions` to `attributes...get('value', [])`, which is `[]` when the command has no such attributes. `Service Level Objective` is in `GOVERNANCE_CONTROLS` (`common_md_utils.py:50`), so `update_gov_body_for_type` takes the controls branch, adds `implementationDescription`, and leaves the three `[]` values. Only the `TermsAndConditions`/`CertificationType`/`LicenseType` branch replaces them with maps or `None`. `body_slimmer` keeps empty lists, so `[]` is sent. In Egeria `main` (05a5e982), `ServiceLevelObjectiveProperties extends TermsAndConditionsProperties`, whose `entitlements`/`restrictions`/`obligations` are `Map`. Jackson cannot read a JSON array into a map, so the request is refused. Other governance types do not have these fields, so they silently ignore them (`ignoreUnknown=true`), which is why only SLO fails. + +**Proposed fix (not made):** in `set_gov_prop_body`, default the three to `None` rather than `[]`, or set them only in the TermsAndConditions branch. Route `ServiceLevelObjective` through that branch (it is a TermsAndConditions subtype, not a plain control), keeping `implementationDescription`. Optionally add `Entitlements`/`Restrictions`/`Obligations` (dict-valued) to the SLO command through the Spec Editor API. Then live-test one `Create Service Level Objective` on a throwaway. + +--- + +### ISSUE-156: `AutomatedCuration.add_catalog_target` posts a flat `"class": "CatalogTargetRequestBody"`, which Egeria does not have; the endpoint takes `NewRelationshipRequestBody` + +**Status 2026-10-29: fixed in #464, released in 6.2.6.** Moved here from the open list during the 2026-10-29 tracker check. + +**Fixed 2026-10-19 (owner approved) on branch `fix/issue-156-157-catalog-target-slo` (956b199c; not pushed, not released).** `add_catalog_target`/`update_catalog_target` keep their signatures but now send `NewRelationshipRequestBody`/`UpdateRelationshipRequestBody` with `CatalogTargetProperties` through the shared relationship helpers. `template_properties` now maps to Egeria's `templates` field; the old body had misnamed it `templateProperties`. **Live on Egeria `main` 05a5e982** (two runs, throwaway undeployed IntegrationConnector `pyeg_rt_ic` and a Collection target, never linked to an integration group): the old flat body was refused; the fixed add returned a relationship GUID and read back name/connection/config; update read back `pyeg_rt_target_v2` and `FROM_THIRD_PARTY` with `connectionName` kept; remove made the relationship unreadable; all throwaways verified gone. The `hey_egeria` `add_catalog_target` command uses this method, so it is fixed too. + +**Layer:** Pyegeria · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, reported by the Egeria team against pyegeria 6.2.5: `add_catalog_target` "sends a request format the platform rejects"; their notebook now works around it. Confirmed from source; not re-run live. + +**What:** pyegeria has two `add_catalog_target` methods for the same URL, `.../integration-connectors/{connectorGUID}/catalog-targets/{elementGUID}`: +- `AutomatedCuration.add_catalog_target(integ_connector_guid, metadata_element_guid, catalog_target_name, connection_name=..., metadata_src_qual_name=..., config_properties=..., template_properties=..., permitted_sync=..., delete_method=...)` (`pyegeria/omvs/automated_curation.py:4120-4190`) builds `{"class": "CatalogTargetRequestBody", "catalogTargetName": ..., "connectionName": ..., ...}` with the properties at the top level. **Broken:** no `CatalogTargetRequestBody` class exists in Egeria `main`. +- `AssetMaker.add_catalog_target(integration_connector_guid, metadata_element_guid, body)` (`pyegeria/omvs/asset_maker.py:1095`) sends `NewRelationshipRequestBody` with `properties.class = "CatalogTargetProperties"`. **Correct.** -**Fix (2026-10-21):** a new `pyegeria.core.config.resolve_egeria_credentials(user_id=None, user_pwd=None)` resolves, when called: an explicit value > the `EGERIA_USER` / `EGERIA_USER_PASSWORD` environment variables > the settings User Profile (`config.json` / `.env`) > "erinoverview" / "secret". -- The `dr_egeria` and `dr_egeria_folder` CLIs now default `--userid` / `--user_pass` to `None` and call it when they run. Their help text says so. -- `md_processing/dr_egeria.py` (module constants and its `__main__` path) and `common_md_utils.py` constants now use it too. The module-level `EGERIA_USER` names are kept for importers. -- Tests: `tests/micro-tests/test_issue_161_dr_egeria_credentials.py`. Five cases: precedence; env set after import, for both CLIs; config-only user; explicit flag wins. The three behaviour cases fail with the old caller code. The full micro-test suite (1,425) passes in a clean python:3.13-slim container. -- **Not changed:** other `commands/cat` tools (`list_glossaries`, `get_asset_graph`, `run_report`, `get_collection_tree`, `get_project_dependencies`, `list_user_ids`, `view_schema_tree`) still read `EGERIA_USER_PASSWORD` straight from the environment. They could adopt the helper later. +Egeria (`AssetMakerResource.addCatalogTarget`, served under every generic URL marker including `automated-curation`) takes `@RequestBody(required=false) NewRelationshipRequestBody`. `CatalogTargetProperties` has `catalogTargetName`, `metadataCollectionQualifiedName`, `metadataSourceQualifiedName`, `connectionName`, `configurationProperties`, `templates`, `permittedSynchronization` and `deleteMethod`. The `hey_egeria` CLI command `add_catalog_target` (`commands/ops/gov_server_actions.py:45`, registered in `commands/cli/egeria.py` and `egeria_ops.py`) uses the broken one. -**Originally proposed:** -- Resolve credentials through `settings.User_Profile` in all three places. -- Look them up when the command runs (click `default=None`, then resolve inside the command). -- Add a micro-test that both entry points pick up a user set only in `config.json` or `.env`. +**Proposed fix (not made):** keep `AutomatedCuration.add_catalog_target`'s signature but build `{"class": "NewRelationshipRequestBody", "properties": {"class": "CatalogTargetProperties", "catalogTargetName": ..., "connectionName": ..., "metadataSourceQualifiedName": ..., "configurationProperties": ..., "templates": ..., "permittedSynchronization": ..., "deleteMethod": ...}}` and send it through `_async_new_relationship_request`, as AssetMaker does. Add a micro-test asserting the body shape. `AutomatedCuration._async_update_catalog_target` (`automated_curation.py:4248`, body at `:4301`) has the same flat `CatalogTargetRequestBody` shape against `catalog-targets/{relationshipGUID}/update`. Egeria's `updateCatalogTarget` takes `UpdateRelationshipRequestBody`, so it needs the same fix. --- +### ISSUE-154: pyegeria follow-ons from the Egeria team's 2026-10-18 note: renamed property classes, `cascadedDelete`, the `pushDown` default, new `DataValueAssignment`/`CrowdSourcingRole` values -### ISSUE-160: `AutomatedCuration`'s template builders miss placeholders Egeria's templates now use, so created elements keep literal `~{name}~` values (folder endpoint `~{directoryAddress}~` breaks folder surveys) +**Status 2026-10-29: fixed on branch `fix/issues-154-158-159-85-tracker-cleanup`, for 6.2.15.** Item 1: `DeleteElementRequestBody` now serializes `cascadedDelete` (both spellings still accepted on input), and the no-body path and the docstring samples use it; see ISSUE-132 for the live re-test. Item 2: the Metadata Expert count docstrings now say the server default is `false` on Egeria main, `true` on older servers; `overview_metrics` is unchanged (it keeps the server default). Item 3: the Retention and SecurityGroupMembership whitelists accept both the old and the new class name. pyegeria still SENDS the old names: Egeria main registers them as aliases of the new ones, and 6.2 knows only the old ones, so the old names work everywhere. Item 4: `Assign Data Value Specification` takes `Label` and `Description` (command attributes via the Spec Editor API, mapped in `DataValueAssignmentProcessor`). Item 5: `link_crowd_sourcing_contribution` documents the `roleType` values. Items 6-7 needed no change. Tests: `tests/micro-tests/test_issues_154_158_159_85.py`. -**Layer:** Pyegeria · **Status:** fixed in #474 and the follow-up #475 (released in 6.2.8), with the Dr.Egeria Postgres follow-up in #478 (released in 6.2.9). The items originally listed as not fixed are kept below for the record. **Found:** 2026-10-21, live by the RE coordinator against Egeria `main` 05a5e982 with pyegeria 6.2.7. Two FileFolders made with `create_folder_element_from_template` had Endpoint `networkAddress` = `~{directoryAddress}~`. Egeria's Folder Survey then failed with `BASIC-FILE-CONNECTOR-404-001` "The folder named ~{directoryAddress}~ ... does not exist" (engine action `c1d924b5`). The same folders also kept `versionIdentifier` = `~{versionIdentifier}~` and `deployedImplementationType` = `~{deployedImplementationType}~` (record `~/.re-gate/repair-folder-endpoints-20261021T143415Z.json`). +**Layer:** Pyegeria · **Status:** open, proposed, not made (owner approval needed, per this repo's policy) · **Found:** 2026-10-18, from the Egeria team's note (`~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`) and read-only greps of this checkout. -**Cause (Egeria `origin/main` d8f72cbe25, read-only):** no placeholder was renamed (`git log -S` on `PlaceholderProperty.java`). Egeria **added** placeholders that pyegeria never sends: -- **`directoryAddress` / `fileAddress`:** added by `156650efd6` (2026-07-19, "file system control for file cataloging"). They are now the Endpoint address in the folder and file templates (`DataAssetTemplateDefinition` `FILE_FOLDER_TEMPLATE`, `DATA_FOLDER_TEMPLATE`, `CSV_FILE_TEMPLATE`, ...). The templates still use `directoryPathName` / `filePathName` in the qualifiedName, so both are needed. -- **`deployedImplementationType`:** in every folder/file placeholder list (`getFolderPlaceholderPropertyTypes`, `getDataFilesPlaceholderPropertyTypes`). -- **`databaseDescription`:** the PostgreSQL Relational Database template's description placeholder (`PostgresPlaceholderProperty.DATABASE_DESCRIPTION`); pyegeria sent `description`. +Egeria says none of its changes break existing requests: the old class names and spellings are still accepted. So nothing here is urgent, and every change should keep accepting the old names so pyegeria still works against older platforms. Ranked by risk, highest first. -An unsent placeholder is stored as the literal `~{name}~`. `body_slimmer` drops `None` values, so an optional argument left unset has the same effect. +**1. `cascadeDelete` now cascades (ISSUE-132).** Egeria accepts `cascadeDelete` as an alias for `cascadedDelete`, so any pyegeria call that sends `true` will start cascading on a platform with `oak2026`. Checked: no production caller in `pyegeria/`, `md_processing/` or `commands/` passes `True`; every `cascade`/`cascade_delete` default is `False`; `commands/tech/element_actions.py`'s `--cascade` flag (default True) is not forwarded. Only live tests pass `True`, on purpose. Proposed: serialize `DeleteElementRequestBody.cascade_delete` as `cascadedDelete` (accept both on input) and change the ~25 `"cascadeDelete"` literals (`_server_client.py` ×3, `models.py` ×7, `actor_manager.py` ×8, `metadata_expert.py`, `product_manager.py` ×2, `reference_data.py` ×2, `valid_metadata.py` ×2; several are docstring samples). Then live-test one cascade and re-test ISSUE-127. -**Fixed on the branch** (`pyegeria/omvs/automated_curation.py`; signatures unchanged): -| Method | Added | -|---|---| -| `create_folder_element_from_template` | `directoryAddress` (= `path_name`), `deployedImplementationType: "File System Directory"` | -| `create_csv_data_file_element_from_template`, `get_create_csv_data_file_element_from_template` | `fileAddress` (= `file_path_name`), `deployedImplementationType: "CSV Data File"` | -| `create_secrets_store_element_from_template` | `fileAddress` (= `file_path_name`), `deployedImplementationType: "YAML File Secrets Collection"` | -| `create_postgres_database_element_from_template` | `databaseDescription` (= `description`; `description` still sent) | +**2. Metadata Expert count default flipped (ISSUE-38).** Egeria's `pushDown` default is now `false` (list-matching, slower); `true` is the fast repository count. `metadata_expert.py:3698` and `:4013` document omission as `pushDown=True`. `pyegeria/view/overview_metrics.py` calls both counts without `push_down`, so its statistics silently become the slower kind. Proposed: fix both docstrings; owner to choose whether `overview_metrics` sends `push_down=True` (Egeria's advice for statistics) or keeps the list-matching default (which matches its own `len(find)` fallback). -Tests: `tests/micro-tests/test_issue_160_template_placeholders.py` (5 cases; all fail on the old code, all pass on the fix). The full micro-test suite (1,409) passes in a clean python:3.13-slim container. +**3. Renamed property classes.** Responses now carry the new names in `class`. Checked: no pyegeria response parsing compares a response's `class` with any of the renamed names. The only matches are request-side `prop=[...]` whitelists, which refuse a caller-built body that uses the *new* name: -**Live check 2026-10-21 14:52Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; the fixed code on this branch): `create_folder_element_from_template("/tmp/pyeg_rt_folder_20261021T145254Z", ..., version="1.0")` created FileFolder `e9a45c7f` with `deployedImplementationType` "File System Directory" and `versionIdentifier` "1.0". Its anchored Endpoint `8f003544` had **`networkAddress` = `/tmp/pyeg_rt_folder_20261021T145254Z`**. The folder, Connection `18ef2ee9` and Endpoint had no `~{...}~` left in their own properties. A metadata-expert delete (`deleteMethod: PURGE`, `cascadedDelete: true`) removed all three; each was verified gone by GUID. A first run (`a3475d3e`) confirmed the folder fields but its script missed the Endpoint. Exact `networkAddress` searches (forLineage) afterwards found no leftover for that run either. The survey itself was not run. +| Old → new | Where pyegeria uses the old name | Proposed | +|---|---|---| +| `RetentionClassificationProperties` → `RetentionProperties` | `omvs/classification_explorer.py:8851,8878` (whitelist),`8919`; `md_processing/v2/processors.py:220-237` (comment says the opposite); `md_processing/v2/curation.py:136` | Send `RetentionProperties`, accept both in the whitelist, rewrite the comment (ISSUE-128) | +| `SecurityGroupMembershipProperties` → `SecurityListMembershipProperties` | `omvs/actor_manager.py:3602,3617` (whitelist),`3654,3703,3753` | Send the new name, accept both | +| `AssetSchemaTypeProperties` → `SchemaProperties` | not used; `schema_maker.py:822` already whitelists `SchemaProperties` | none | +| `AssociatedSecurityGroupProperties` → `AssociatedSecurityListProperties` | not used; `md_processing/v2/governance.py:620` already sends the new name | none | +| `ReferenceCodeMappingProperties` → `ReferenceCodeMappingTableProperties` | not used | none | +| `GovernanceClassificationBase` → `GovernedDataClassificationBase` | only in a description string, `commands_curation_compact.json:2256` | optional wording fix through the Spec Editor API | -**Follow-up fix 2026-10-21 (owner: "fix them all in a new pr")** on branch `fix/issue-160-remaining-placeholders`, built on #474. All 13 hard-coded builders now wrap their values in `_placeholder_values`, which sends `""` for any placeholder left `None`, so no `~{name}~` literal is stored. `body_slimmer` drops only `None`. A folder created without a file system therefore gets qualified name `FileFolder:::/path`. -- **Postgres server and database:** new optional `secrets_store_path_name` / `secrets_collection_name`, sent as `secretsStorePathName` / `secretsCollectionName`, plus `version` (and `resource_name` on the server, defaulting to the server name). `db_user` / `db_pwd` are now optional, deprecated and no longer sent; passing either raises a `DeprecationWarning`. -- **Kafka server:** `resourceName` (defaults to the server name) and `versionIdentifier`. -- **Unity Catalog:** - - UC server: `resourceName`, defaulting to the server name; - - UC catalog: `ucServerQualifiedName`; - - UC table and function: `rootSchemaTypeQualifiedName`; - - UC volume: `directoryPathName` and `localStorageLocation`. - Each is a new optional parameter on the async method and its sync wrapper. -- **Item 4 needed no change:** "Keystore File" is a real template (`DataAssetTemplateDefinition.KEYSTORE_FILE_TEMPLATE`), and live, both "YAML File Secrets Collection" and "Keystore File" resolve to template `fbcfcc0c`. +`tests/functional-tests/test_classification_manager.py:1233` already sends `RetentionProperties`. -Tests: `tests/micro-tests/test_issue_160_remaining_placeholders.py`, 11 cases, all failing on #474's code. The full micro-test suite (1,420) passes in a clean python:3.13-slim container. **Dr.Egeria follow-up, done 2026-10-21** (after 6.2.8; branch `claude/elegant-chatterjee-d280de`): the `Create Postgres Server Element` / `Create Postgres Database Element` commands no longer offer `Database User Id` / `Database Password`. Both attributes were deleted from the Asset Maker compact spec, since no other command used them. The commands now offer `Secrets Store Path Name`, `Secrets Collection Name` and `Version Identifier`, and the server also offers `Resource Name` (the canonical Governance Officer definition, copied). `AssetMakerProcessor` passes them as `secrets_store_path_name` / `secrets_collection_name` / `version` / `resource_name` and no longer passes `db_user` / `db_pwd`. Templates, help and report specs were regenerated, and the templates were copied to egeria-workspaces and trellis on local branches. Test: `tests/micro-tests/test_postgres_element_secrets.py`. Not run against a live server. +**4. `DataValueAssignment` now accepts `label` and `description`.** Dr.Egeria's `Assign Data Value Specification` (`md_processing/v2/data_designer.py:1026-1037`) replaces `properties` with a fixed field list, and the `Data Value Assignment Base` bundle has no Label/Description. Proposed (feature, not a fix): add `Label` and `Description` to that bundle through the Spec Editor API, map them in the processor, run `refresh_specs`. -**Follow-up live check 2026-10-21 15:16Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; #475 code): -- **(a)** `create_postgres_server_element_from_template("pyeg_rt_pg_…", "pyeg-rt.invalid", "5432", secrets_store_path_name=…, secrets_collection_name=…)`, with no `db_user` / `db_pwd`, created SoftwareServer `b6b2ebff`: `resourceName` = the server name, `versionIdentifier` and `description` = `""`, and no `~{...}~` in its own properties. Its Connection `55686913` and Endpoint `59014065` (`jdbc:postgresql://pyeg-rt.invalid:5432/postgres`) had no literals either. -- The server had **no CatalogTarget link** (its relationships were sourcedFromTemplate, serverEndpoints, capabilities, connections), so the JDBC integration connector never saw it. -- I did not find where Egeria stored the secrets-store values. They were not in the Connection's own properties, and no literal was left. Unconfirmed. -- **(b)** `create_folder_element_from_template("/tmp/pyeg_rt_folder2_…", …, None)`, with no file system, version or description: qualifiedName `FileFolder:::/tmp/pyeg_rt_folder2_…`, `resourceName` `:/tmp/…`, `versionIdentifier` / `description` = `""`, `deployedImplementationType` "File System Directory", and Endpoint `networkAddress` = the path. No literals on the folder, Connection or Endpoint. -- Both were purged (metadata-expert delete, `PURGE`, `cascadedDelete`), and all six GUIDs were verified gone. Name, host, stamp and secrets-file searches afterwards found nothing, which covers the server's capabilities too. +**5. `CrowdSourcingRole` values now work** (PROPOSER, REVIEWER, SUPPORTER, APPROVER, OTHER). pyegeria's `link_crowd_sourcing_contribution` (`omvs/community_matters_omvs.py:479`) passes the body through, and no Dr.Egeria command sets a role. Proposed: list the values in the docstring; nothing else. -**Originally listed as not fixed** (kept for the record): -1. **Postgres server and database credentials:** both templates now take `secretsStorePathName` and `secretsCollectionName` (credentials come from a secrets store). pyegeria never sends them, so the connection keeps literals. pyegeria still sends `databaseUserId` / `databasePassword`, which the templates no longer list (`312c172639`, `56fa898e43`), so those are dead. Proposed: add optional `secrets_store_path_name` / `secrets_collection_name` parameters, and deprecate `db_user` / `db_pwd`. -2. **Optional arguments left unset:** `versionIdentifier`, `description`, `fileSystemName` and the like are dropped when `None`, so they stay `~{...}~` (RE's `versionIdentifier` case). Options: send `""` for unset optional placeholders, or document that callers must pass them. -3. **Unsent template placeholders in the Unity Catalog and Kafka builders:** - - UC catalog: `ucServerQualifiedName`; - - UC table and function: `rootSchemaTypeQualifiedName`; - - UC volume: `directoryPathName`, `localStorageLocation`; - - UC server and Kafka server: `resourceName` (Kafka also `versionIdentifier`). - Add parameters for them where a caller can know the value. -4. **The secrets-store builder's fallback** to a "Keystore File" technology type (`_async_get_template_guid_for_technology_type`) matches no template on `origin/main`. +**6. ISSUE-147's 400-011 workaround:** none exists. Nothing to drop. -Builders with no hard-coded keys (`create_elem_from_template`, Dr.Egeria's "Placeholder Property Values" attribute) are unaffected. The UC schema builder sends all its placeholders. +**7. View-service fixes pyegeria may have worked around.** The note lists 48 view-service defects found by Egeria's new view-services FVT. A read-only sweep of `pyegeria/`, `md_processing/` and `commands/` (2026-10-18) found almost nothing worked around, and no issues-file entry for any of them. The detach, Classification Explorer, Metadata Expert, search-option, history-order and delete/purge rows have no client-side workaround: the wrappers call the endpoint once and pass options straight through. So pyegeria users met those defects as calls that "succeeded" but did nothing, and the fixes need no pyegeria change. Candidates to review: +- **Project hierarchy:** `commands/cat/get_project_structure.py:105-116` (and `get_project_dependencies.py:107`) walk the tree with `get_linked_projects()` and keep only `ProjectHierarchy` links, instead of calling `ProjectManager.get_project_hierarchy`. Not commented as a workaround. Optional: switch to `get_project_hierarchy` after re-testing it on a platform built from `main`. +- **Term classifications with no body:** `glossary_manager.py:2686-2692` (is-question) and `4059-4062`/`4116`/`4170` (prime word, modifier, class word) always send a default body. The comment blames pyegeria's own helper (ISSUE-31), not the server, so keep them; harmless either way. +- **`linkSolutionLinkingWire` with no body:** `md_processing/v2/solution_architect.py:856-860` always sends a body and skips recording the GUID when none returns. Keep it; no change needed. +- **Asset from template:** Dr.Egeria's generic `Create Element` (`md_processing/v2/asset_maker.py:57-75`) uses Automated Curation `catalog-templates/new-element`, not Asset Maker `assets/from-template`, so it never hit the "always an Asset" bug. No change. +- **Missing connector now refused** (Security Officer, Runtime Manager): calls that used to report success with nothing done will now raise. Callers should expect the error. ISSUE-112 is the same shape for client-side secrets. +- **Delete with `forLineage`:** a delete that needs `forLineage=true` (Memento) is now refused whole instead of half-done. pyegeria's editing deletes already default to `forLineage=true` via `lineage_visible()`, so no change. --- - ### ISSUE-153: engine-action claim/status/completion calls still use Automated Curation paths that Egeria moved to the Governance Engine view service **Found 2026-10-16 by the Resource Explorer operations session, after the quickstart was redeployed on Egeria main (85d6c48a, includes odpi/egeria#9371 "Add Governance Engine API").** @@ -2286,6 +2131,29 @@ Builders with no hard-coded keys (`create_elem_from_template`, Dr.Egeria's "Plac --- +### ISSUE-152: five update methods pass the body into `_async_update_element_body_request`'s `prop` slot, so nothing is sent + +**Status 2026-10-29: fixed in #471, released in 6.2.7.** Moved here from the open list during the 2026-10-29 tracker check. + +**Layer:** Pyegeria · **Status:** open (found 2026-10-15 by Resource Explorer's ClassificationAnnotation backfill; not fixed, per this repo's policy) + +**Fixed 2026-10-20 (owner approved) on branch `fix/issue-152-update-body-slot`, not yet merged or released.** All five call sites now pass `body=body`. `tests/micro-tests/test_issue_152_update_body_slot.py` sends a dict body and an `UpdateElementRequestBody` through each method and checks that the body reaches the HTTP layer. All 10 cases fail on the old lines and pass on the fix, and the full micro-test suite (1,404) passes in a clean python:3.13-slim container. Not live-tested. Hit again 2026-10-20 by the TestGen evaluation session (RE #791, `update_annotation`), which carries its own workaround until a release ships this. + +`_async_update_element_body_request(self, url, prop=None, body=None)` (`pyegeria/core/_server_client.py:7317`) takes the body as its **third** parameter. Five callers pass it positionally as the second, so it lands in `prop` and `body` is `None`; the call fails before any request goes out (the builder observed an AttributeError): + +- `pyegeria/omvs/data_discovery.py:299` (`_async_update_annotation`, behind `update_annotation`) +- `pyegeria/omvs/time_keeper.py:342` +- `pyegeria/omvs/schema_maker.py:131` and `:174` +- `pyegeria/omvs/subject_area.py:302` + +Checked at egeria-python `f1b06cdd`; the installed wheel in trellis has the same line in `data_discovery.py`. + +**Fix (one token each):** `await self._async_update_element_body_request(url, body=body)`. A test per method that the body reaches the HTTP layer would catch a repeat. + +**Workaround in Resource Explorer:** `scripts/backfill_classification_annotations.py` calls `_async_update_element_body_request(url, body=body)` directly, and pins this bug with a test that fails once pyegeria is fixed. + +--- + ### ISSUE-149: `AnalyticFunctionSpec` lacks the manifest fields trellis-analytics needs (`kind`, `source`, `version`, `envelope`, `annotation_type`, `resource_types`), so trellis subclasses carry them **Layer:** pyegeria (`pyegeria/view/analytic_registry.py`) · **Status:** fixed 2026-10-13 · **Found:** 2026-10-13, handed off by the Resource Explorer PR/CI session on the owner's instruction. Source: trellis brief §2a and `packages/trellis-analytics/src/trellis_analytics/manifest.py`. RE runs pyegeria 6.2.2. @@ -2739,6 +2607,47 @@ three dead `subComponents` fallbacks are removed. Micro-tests in `tests/micro-te --- +### ISSUE-132: `DeleteElementRequestBody` serializes the cascade flag as `cascadeDelete`; Egeria's field is `cascadedDelete` and unknown fields are ignored, so no pyegeria cascade delete has ever cascaded + +**Live-verified 2026-10-29 (Egeria `main` 8de9572c quickstart, qs-view-server, as peterprofile; peers cleared first): PASS.** A throwaway Collection with an anchored member Collection: `delete_collection(cascade=False)` was refused with `OMAG-GENERIC-HANDLERS-403-005` ("not permitted because it still has a dependent Collection element … use the cascaded delete option"); `delete_collection(cascade=True)`, now sending `cascadedDelete`, removed both. Fixed in #509, for 6.2.15. + +**Status 2026-10-29: pyegeria side fixed on branch `fix/issues-154-158-159-85-tracker-cleanup`, for 6.2.15 (ISSUE-154 item 1): cascade deletes now send `cascadedDelete`, which every Egeria version reads.** What's left is a live cascade test on a platform built from Egeria main. + +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. `cascadeDelete` is now accepted as an alias for `cascadedDelete`, so existing pyegeria requests will start to cascade; Egeria still recommends switching to `cascadedDelete`. **Risk check, 2026-10-18:** no production caller in `pyegeria/`, `md_processing/` or `commands/` passes cascade `True`; every `cascade`/`cascade_delete` default is `False`; `commands/tech/element_actions.py`'s `--cascade` flag (default True) is not forwarded. Live functional/scenario tests that pass `cascade=True` (collection, glossary, governance officer, `remove_note_log`) will now really cascade, which is what they intend. **pyegeria (proposed, not done):** change `DeleteElementRequestBody`'s serialization alias to `cascadedDelete` (keep both accepted on input) and the ~25 `"cascadeDelete"` literals (`_server_client.py`, `models.py`, `actor_manager.py`, `metadata_expert.py`, `product_manager.py`, `reference_data.py`, `valid_metadata.py`), then live-test one cascade and re-test ISSUE-127. No longer urgent, since the old spelling now works. + +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS (alias).** A throwaway DataStructure with a linked member DataField, deleted with `delete_data_structure(guid, cascade_delete=True)` (pyegeria still sends `cascadeDelete`), succeeded. Both elements were gone by GUID before any cleanup, so the cascade reached the server. The ISSUE-154 item 1 rename to `cascadedDelete` is still recommended but no longer urgent. + +**Layer:** pyegeria · **Status:** open, on hold: an Egeria fix for cascade delete is coming (owner, 2026-10-10). Egeria `main` checked 2026-10-10 (head `af400039c2`, 2026-10-06): no cascade change yet. Wait +for it, live-test cascade delete against it, then decide the pyegeria change; the fix below may change once the +Egeria side lands. Confirmed from Egeria source, not yet live-tested · **Found:** +2026-10-10, while checking ISSUE-129's server contract against Egeria source instead of the `.http` files. + +**What:** `pyegeria/models/models.py` `DeleteElementRequestBody.cascade_delete` accepts both spellings on input but +serializes as `serialization_alias="cascadeDelete"`. The comment there records that the `.http` files were split +about 50/50 and that the choice was unverified (ISSUE-62). Egeria source settles it (`egeria-v6/egeria` at +`af400039c2`, 2026-10-06): `DeleteElementRequestBody` -> `DeleteRequestBody` -> `DeleteOptions` +(`open-metadata-framework/.../search/DeleteOptions.java`), whose field is `cascadedDelete` with +`getCascadedDelete()`/`setCascadedDelete()`. It is unchanged since 2025-08-04. The class is +`@JsonAutoDetect(getterVisibility=PUBLIC_ONLY, fieldVisibility=NONE)` and `@JsonIgnoreProperties(ignoreUnknown=true)`, +with no `@JsonProperty`/`@JsonAlias`, so the JSON name is `cascadedDelete` and `cascadeDelete` is silently ignored. + +**Effect:** every pyegeria delete that uses `DeleteElementRequestBody` with cascade on (the ISSUE-62 fix, +`validate_delete_element_request`, ISSUE-127's fix in #429, `MetadataExpert.delete_metadata_element(cascade_delete=True)`) +sends a flag Egeria discards, so it deletes without cascading. A caller-supplied dict using the correct +`cascadedDelete` key is also re-serialized as `cascadeDelete`, so even that does not work. The models module's own +`DeleteRequestBody.cascaded_delete` already uses the right name. + +**Fix (not made):** change the serialization alias to `cascadedDelete` (keep both spellings accepted on input), and +fix the 19 `"cascadeDelete"` references in `_server_client.py`, `product_manager.py`, `actor_manager.py`, +`valid_metadata.py`, `metadata_expert.py` and `reference_data.py` (several are docstring samples). Then live-test one +cascade delete, and re-test ISSUE-127's `DataStructure` refusal. The `.http` files using `cascadedDelete` are correct; +the ones using `cascadeDelete` are the ones in error (worth reporting to Egeria). + +**Lesson:** where the `.http` files disagree with each other, read the Java DTO (getter/setter names plus Jackson +annotations); do not choose by majority or by what the code already sends. + +--- + ### ISSUE-131: 13 `*_by_name` wrappers pass `metadata_element_type=`, but `_async_get_name_request` reads `metadata_element_type_name`, so the type filter is silently dropped **Layer:** pyegeria · **Status:** fixed in #439, released in 6.2.1 (2026-10-10) · **Found:** 2026-10-10, while building Resource Explorer's actor-graph lookups (`get_user_identities_by_name`, `get_actor_profiles_by_name`). @@ -2812,6 +2721,43 @@ overwriting would silently change callers who set it in the body. --- +### ISSUE-127: `CollectionManager.delete_collection(cascade=True)` silently never sends `cascadeDelete` -- same shape as ISSUE-62 + +**Status 2026-10-29: fixed in #429 (released in 6.1.29) and re-tested live 2026-10-19 (PASS, below).** Moved here from the open list during the 2026-10-29 tracker check. + +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** The `DataStructure` refusal (`validateNoMemberDataFields`) did not recur: a cascade delete of a structure with a member field removed both. See ISSUE-132's re-test. + +**Layer:** pyegeria · **Status:** partly fixed in #429 (`68279835`), released in 6.1.29; cascade still blocked by ISSUE-132 · **Found:** 2026-10-05, found by capturing the request body while diagnosing a +failed cleanup. + +**Fix + scope (2026-10-05):** a scan of all 42 callers of `_async_delete_element_request` found exactly two +that pre-fill a default body *and* pass a cascade flag: `CollectionManager._async_delete_collection` and +`ProductManager._async_delete_digital_product` (so `delete_digital_product(cascade=True)` was never cascading +either -- the live check's own product cleanup included). Both now pass `None` so the helper applies the flag. +`delete_digital_product`'s docstring sample also used the wrong key (`cascadedDelete`); corrected. The helper +itself is unchanged: a caller that passes an explicit body dict *and* `cascade=True` still gets the body's +value, which is documented behaviour ("body supersedes"). + +**What:** `_async_delete_collection` always pre-fills `body = {"class": "DeleteElementRequestBody"}` when none is +given and hands that dict to `_async_delete_element_request(url, body, cascade)`. +`validate_delete_element_request` applies `cascade_delete` only in its *no body* branch, so with that pre-filled dict +the flag is discarded. Captured: `delete_collection("g", cascade=True)` posts +`{'class': 'DeleteElementRequestBody', 'forLineage': False, 'forDuplicateProcessing': False}` with no +`cascadeDelete`. `MetadataExpert.delete_metadata_element(cascade_delete=True)` does send it. + +**Fixed in #429, but cascade still does not reach Egeria -- see ISSUE-132.** #429 made the flag reach the request +body, but under the name `cascadeDelete`. Egeria's real field is `cascadedDelete`, and it ignores unknown fields, so +the flag is still dropped, now on the server side. #429's docstring "correction" from `cascadedDelete` to +`cascadeDelete` was also the wrong way round. The `DataStructure` refusal described below is probably this bug +rather than an Egeria quirk; re-test once ISSUE-132 is fixed. + +**Related, an Egeria quirk found alongside:** even with `cascadeDelete: true`, Egeria refused to delete a +`DataStructure` that still had a member `DataField` (`OMAG-GENERIC-HANDLERS-403-005 ... validateNoMemberDataFields`), +so a data contract imported by `import_data_contract*` has to be removed child-first: DataField, DataStructure, +Agreement. + +--- + ### ISSUE-126: `ProductManager` and `DigitalBusiness` never set `collection_command_root`, so every inherited `CollectionManager` method that uses it raises `AttributeError` **Layer:** pyegeria · **Status:** fixed in #429 (`68279835`), released in 6.1.29 · **Found:** 2026-10-05, live cleanup of a throwaway data contract (PR #427's @@ -4915,6 +4861,79 @@ the builder. Backward compatible: `user`/`user_pass` keep working when no token --- +### ISSUE-85: No REST endpoint exists to create a `SolutionPort` element — only attach/detach/delegation relationships are exposed + +**Live-verified 2026-10-29 (Egeria `main` 8de9572c quickstart, qs-view-server, as peterprofile; throwaways prefixed `b2-live-509::`, deleted and verified gone; peers cleared first): PASS.** A port created anchored to a SolutionComponent (parent relationship `SolutionComponentPort`), updated, found by search string and by name, read by GUID, seen from the component, copied with create-from-template, and deleted. The live test caught wrong `direction` values in the new docstrings (they said IN/OUT/OUTIN/OTHER; Egeria's `SolutionPortDirection` is UNKNOWN, INPUT, OUTPUT, INOUT, and `IN` is a 400); corrected before merge. Fixed in #509, for 6.2.15. Still not done: a Dr.Egeria `Create Solution Port` command. + +**Status 2026-10-29: SDK wrappers added on branch `fix/issues-154-158-159-85-tracker-cleanup`, for 6.2.15; awaiting a live test.** `SolutionArchitect` now has `create_solution_port`, `create_solution_port_from_template`, `update_solution_port`, `delete_solution_port`, `find_solution_ports`, `get_solution_ports_by_name` and `get_solution_port_by_guid`, matching `SolutionArchitectResource.java` on Egeria main (endpoints and body classes; `omvs_audit` Java check passes). Not done: a Dr.Egeria `Create Solution Port` command. + +**Status:** open, now a pyegeria gap (Egeria added the endpoints 2026-09-06) +consumer (trellis/Resource Explorer, designing a `PortMaterializer` for +architecture-recovery proposals) hitting the same shape of gap ISSUE-84 +turned out not to be: this time the `.http` ground truth's absence really +does match the live server, not a stale/incomplete `.http` file. + +**Confirmed at every layer:** +- `pyegeria/omvs/solution_architect.py`: `link_solution_component_port` + (attach an *existing* port to a component), `link_solution_port_delegation` + (port-to-port delegation), and both detach counterparts exist. No + `create_solution_port`/`_async_create_solution_port` anywhere. +- `Egeria-api-solution-architect.http`: only `.../solution-components/ + {guid}/solution-ports/{guid}/attach|detach` and `.../solution-ports/ + {guid}/port-delegations/{guid}/attach|detach`. No `POST .../solution-ports` + create path. +- `md_processing/data/compact_commands/commands_solution_architect_compact.json`: + only `Link Solution Component Port`/`Link Solution Port Delegation`. No + `Create Solution Port` command. +- `pyegeria/models/models.py`: no `SolutionPortProperties` model at all — + so even the generic fallback route has nothing typed to build a body + against. + +**Live-verified not just a stale-`.http`-file case** (the precedent this +repo already has for that shape of false alarm — see the multi-link audit +above, where 3 of 4 similarly-unwrapped relationship types turned out to +have zero real endpoint on live-server inspection while others did): +`dwolfson-1b`/trellis fetched the full live OpenAPI spec directly +(`GET /v3/api-docs`, 2.39MB) and searched every path containing "port" — +30 POST paths matched, every one of them attach/detach (component-port, +port-delegation, even a generic process-ports attach under a different +`{urlMarker}` service). Zero creation endpoints anywhere in the spec. +This confirms the `.http` ground truth exactly; not a documentation gap. + +**The generic fallback exists but isn't usable without guessing:** +`/servers/{serverName}/api/open-metadata/{urlMarker}/metadata-elements` +(`MetadataExpert.create_metadata_element`, +`NewOpenMetadataElementRequestBody`) is real and live-confirmed present, +with a top-level `type_name` field that could carry `"SolutionPort"`. But +per this file's own `pyegeria/models/models.py` gotcha, that endpoint's +`properties` field needs the verbose nested `ElementProperties`/ +`propertyValueMap` shape server-side, not the flat dict every bespoke +wrapper uses — and with no `SolutionPortProperties` model to check +property names against, building that body means hand-guessing what +properties a `SolutionPort` actually takes. Consistent with this repo's +established practice (see the multi-link audit's "left unbuilt rather +than guessing at undocumented URLs or building against the awkward +generic path speculatively") — not attempted. + +**Nothing to fix in pyegeria until Egeria adds a real create endpoint.** +Once one exists, the follow-on work is small and well-precedented: a +`create_solution_port`/`_async_create_solution_port` wrapper in +`SolutionArchitect` following the exact pattern ISSUE-84 just corrected +(`NewElementRequestBody` + `contentStatus` inside `properties`, no +fictional separate request-body class), plus a `Create Solution Port` +Dr.Egeria compact command — the family's registration plumbing in +`register_solution_architect_processors()` (`md_processing/dr_egeria.py`) +already handles this shape of addition with no new routing logic needed. + +**Not blocking anything urgent** on the trellis/Resource Explorer side — +they're holding real port materialization until this exists server-side +(tracked as their own Backlog item 8), or until a workaround surfaces. + + +**Egeria side addressed (2026-10-05 check, carried over from the retired `docs/egeria-open-bugs-2026-10-05.md`):** at `449ad06894`, Solution Architect has `createSolutionPort` and `createSolutionPortFromTemplate`, with endpoints `POST /solution-ports`, `/solution-ports/from-template`, `/solution-ports/{guid}/update`, `.../delete`, `/by-name`, `/by-search-string` and `.../retrieve`. They first appear in commit `571034df11` (2026-09-06). The running platform answers `POST /solution-ports/by-name` with 200. **Remaining (pyegeria):** the `.http` snapshot lists none of these, and `pyegeria/omvs/solution_architect.py` has no `create_solution_port` (or update, delete, find). Add them, ideally with Dr.Egeria support. + +--- + ### ISSUE-84: `SolutionArchitect.create_solution_blueprint`'s own docstring documents a `NewSolutionElementRequestBody` body (with `initialStatus`) for Draft-status creation — that class does not exist as a pydantic model, so the documented shape fails client-side validation before any HTTP call **Status:** fixed and live-verified 2026-09-03 — docstring-only fix (see diff --git a/md_processing/data/compact_commands/commands_data_designer.json b/md_processing/data/compact_commands/commands_data_designer.json index 25bc6723..5d0580c4 100644 --- a/md_processing/data/compact_commands/commands_data_designer.json +++ b/md_processing/data/compact_commands/commands_data_designer.json @@ -3946,7 +3946,9 @@ "bundle": "Data Value Assignment Base", "custom_attributes": [ "Data Value Specification", - "Element Id" + "Element Id", + "Label", + "Description" ], "Journal Entry": "" }, diff --git a/md_processing/v2/curation.py b/md_processing/v2/curation.py index 2258654a..6f846f9a 100644 --- a/md_processing/v2/curation.py +++ b/md_processing/v2/curation.py @@ -132,6 +132,8 @@ class ClassificationSpec: "Criticality": ClassificationSpec( "_async_set_criticality_classification", "_async_clear_criticality_classification", "CriticalityProperties", level_field="criticalityLevel", fields=dict(_GOVERNANCE_SHARED_FIELDS)), + # Old class name on purpose: Egeria main (RetentionProperties) accepts it as an alias; older servers + # know only this one (ISSUE-154). "Retention": ClassificationSpec( "_async_set_retention_classification", "_async_clear_retention_classification", "RetentionClassificationProperties", level_field="retentionBasis", # "Retention Basis" attribute holds an enum, reuse level_field slot diff --git a/md_processing/v2/data_designer.py b/md_processing/v2/data_designer.py index 53646e81..b616fbf7 100644 --- a/md_processing/v2/data_designer.py +++ b/md_processing/v2/data_designer.py @@ -1051,6 +1051,9 @@ async def apply_changes(self) -> str: body = set_rel_request_body("DataValueAssignment", attributes) body["properties"] = { "class": "DataValueAssignmentProperties", + # label/description come from LabeledRelationshipProperties (ISSUE-154 item 4). + "label": attributes.get("Label", {}).get("value"), + "description": attributes.get("Description", {}).get("value"), "assignmentStatus": attributes.get("Assignment Status", {}).get("value"), "confidence": attributes.get("Confidence", {}).get("value"), "method": attributes.get("Method", {}).get("value"), diff --git a/md_processing/v2/governance.py b/md_processing/v2/governance.py index da94c815..75b2a87d 100644 --- a/md_processing/v2/governance.py +++ b/md_processing/v2/governance.py @@ -92,6 +92,16 @@ def _requires_base_governance_definition_properties(exc: Exception) -> bool: and "governancedefinitionproperties" in msg ) + @staticmethod + def _fields_dropped_by_base_fallback(create_props: Dict[str, Any], fallback_props: Dict[str, Any]) -> list[str]: + """The set (non-empty) properties the base GovernanceDefinitionProperties retry would not send. + A field the fallback moves into additionalProperties (SecurityGroup's distinguishedName) is kept.""" + carried = fallback_props.get("additionalProperties") or {} + return sorted( + k for k, v in create_props.items() + if k not in fallback_props and k not in carried and k != "class" and v not in (None, "", [], {}) + ) + @staticmethod def _to_base_governance_definition_properties(properties: Dict[str, Any], object_type: str) -> Dict[str, Any]: source = dict(properties) @@ -213,6 +223,14 @@ async def apply_changes(self) -> str: raise retry_body = dict(body) retry_body["properties"] = self._to_base_governance_definition_properties(dict(create_props), object_type) + # The base-class retry keeps only cross-type fields. Creating the element without the + # type-specific ones and reporting success lost them silently (ISSUE-158), so refuse instead. + dropped = self._fields_dropped_by_base_fallback(create_props, retry_body["properties"]) + if dropped: + raise ValueError( + f"Egeria would only create this {object_type} as a plain governance definition " + f"(OMAG-COMMON-400-029), which would drop: {', '.join(dropped)}. Not created." + ) from exc logger.debug( f"Create SecurityGroup fallback payload properties: {json.dumps(retry_body.get('properties', {}), sort_keys=True)}" ) diff --git a/md_processing/v2/processors.py b/md_processing/v2/processors.py index 3fa98524..239cff00 100644 --- a/md_processing/v2/processors.py +++ b/md_processing/v2/processors.py @@ -217,14 +217,10 @@ def add_related_result(self, label: str, guid: Optional[str] = None, status: str {"UNCLASSIFIED": 0, "MARGINAL": 1, "IMPORTANT": 2, "CRITICAL": 3, "CATASTROPHIC": 4, "OTHER": 99}, ), "Retention Classification": ( - # "RetentionClassificationProperties" (the server's registered Jackson - # subtype ID for this classification - confirmed live via its own - # InvalidTypeIdException error listing all valid ids) is the correct - # class name to send, despite the real Java properties class's own - # simple name being "RetentionProperties" - do not "fix" this to - # "RetentionProperties", that's the wrong direction (confirmed live - # 2026-08-03: sending "RetentionProperties" gets rejected by Jackson - # outright as an unrecognized subtype id). + # Send "RetentionClassificationProperties". Servers before 2026-10 know + # only that subtype id (sending "RetentionProperties" was rejected live + # on 2026-08-03). Egeria main renamed it "RetentionProperties" and keeps + # the old name as an alias, so the old name works on both (ISSUE-154). # # Previously blocked server-side (see BACKLOG.md history) by # OMRS-REPOSITORY-400-028 ("a property called statusIdentifier ... diff --git a/pyegeria/core/_server_client.py b/pyegeria/core/_server_client.py index b319d73c..875835a5 100644 --- a/pyegeria/core/_server_client.py +++ b/pyegeria/core/_server_client.py @@ -48,6 +48,7 @@ # Main Client Imports from pyegeria.models.models import ( + DeleteMethod, NewElementRequestBody, NewAttachmentRequestBody, UpdateElementRequestBody, @@ -6202,6 +6203,18 @@ def validate_new_classification_request(self, body: dict | NewClassificationRequ return validated_body + @staticmethod + def _archive_delete_body(body: Optional[dict | DeleteElementRequestBody] = None) -> dict | DeleteElementRequestBody: + """Return a copy of a delete request body with deleteMethod ARCHIVE (ISSUE-159). Egeria routes an + ARCHIVE delete to its archive operation: the element is classified Memento and its qualifiedName + gets an `_archivedOn_` suffix, and MetadataExpert.restore_metadata_element reverses it.""" + if isinstance(body, DeleteElementRequestBody): + return body.model_copy(update={"delete_method": DeleteMethod.ARCHIVE}) + body = dict(body or {}) + body.setdefault("class", "DeleteElementRequestBody") + body["deleteMethod"] = DeleteMethod.ARCHIVE.value + return body + @dynamic_catch def validate_delete_element_request(self, body: Optional[dict | DeleteElementRequestBody] = None, cascade_delete: bool = False) -> DeleteElementRequestBody | None: @@ -6212,7 +6225,7 @@ def validate_delete_element_request(self, body: Optional[dict | DeleteElementReq else: # handle case where body not provided body = { "class": "DeleteElementRequestBody", - "cascadeDelete": cascade_delete + "cascadedDelete": cascade_delete } validated_body = self._validate_body(DeleteElementRequestBody.model_validate, body) return validated_body @@ -6227,7 +6240,7 @@ def validate_delete_relationship_request(self, body: dict | DeleteRelationshipRe else: # handle case where body not provided # body = { # "class": "DeleteRelationshipRequestBody", - # "cascadeDelete": cascade_delete + # "cascadedDelete": cascade_delete # } # validated_body = self._validate_body(DeleteRelationshipRequestBody.model_validate, body) return None @@ -6243,7 +6256,7 @@ def validate_delete_classification_request(self, body: dict | DeleteClassificati else: # handle case where body not provided body = { "class": "DeleteClassificationRequestBody", - "cascadeDelete": cascade_delete + "cascadedDelete": cascade_delete } validated_body = self._validate_body(DeleteClassificationRequestBody.model_validate, body) return validated_body diff --git a/pyegeria/models/models.py b/pyegeria/models/models.py index 09916546..6aae331c 100644 --- a/pyegeria/models/models.py +++ b/pyegeria/models/models.py @@ -413,22 +413,14 @@ class DeleteElementRequestBody(RequestBody): # DeleteElementRequestBody bodies carry "deleteMethod" consistently -- # no spelling ambiguity found for this field. delete_method: Optional[DeleteMethod] = None - # cascadeDelete: ground-truth .http files are genuinely split roughly - # 50/50 between "cascadeDelete" and "cascadedDelete" for this same class - # -- even within a single file (Egeria-api-actor-manager.http uses - # both). Not resolved here; "cascadeDelete" was chosen as the - # OUTGOING/serialized name because it's what this repo's own code has - # already been trying to send since before this fix - # (validate_delete_element_request's no-body-provided branch). Both - # spellings are accepted on input via validation_alias so a - # caller-supplied dict using either one still populates correctly. - # Flagged in PYEGERIA_ISSUES.md ISSUE-62 for live verification -- if a - # given delete endpoint actually expects "cascadedDelete" instead, this - # choice would need revisiting for that specific endpoint. + # cascadedDelete is Egeria's field (DeleteOptions.java). pyegeria sent "cascadeDelete" from the + # .http files until 6.2.15; Egeria ignored it until odpi/egeria#9365 added it as a @JsonAlias, so + # no cascade reached an older server (ISSUE-132). Sending cascadedDelete works on old and new + # servers (ISSUE-154 item 1). Both spellings are accepted on input. cascade_delete: Optional[bool] = Field( default=None, validation_alias=AliasChoices("cascadeDelete", "cascadedDelete"), - serialization_alias="cascadeDelete", + serialization_alias="cascadedDelete", ) archive_date: datetime | None = None archive_process: str | None = None diff --git a/pyegeria/omvs/actor_manager.py b/pyegeria/omvs/actor_manager.py index 109dbfc8..7b560ef5 100644 --- a/pyegeria/omvs/actor_manager.py +++ b/pyegeria/omvs/actor_manager.py @@ -3614,7 +3614,10 @@ async def _async_add_security_group_membership(self, user_identity_guid: str, se """ url = (f"{self.command_root}/user-identities/{user_identity_guid}/security-group-memberships/classify") - await self._async_new_classification_request(url, ["SecurityGroupMembershipProperties"], body) + # Egeria renamed the class SecurityListMembershipProperties (main, 2026-10); it still accepts the + # old name, which is the only one older servers know, so accept both (ISSUE-154). + await self._async_new_classification_request( + url, ["SecurityGroupMembershipProperties", "SecurityListMembershipProperties"], body) logger.debug(f"Classifying User Identity {user_identity_guid} with Security Groups {security_groups}") @dynamic_catch @@ -4773,7 +4776,7 @@ async def _async_delete_contribution_record(self, contribution_record_guid: str, example: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime" : "{{$isoTimestamp}}", @@ -4814,7 +4817,7 @@ def delete_contribution_record(self, contribution_record_guid: str, body: Option example: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime" : "{{$isoTimestamp}}", @@ -5767,7 +5770,7 @@ async def _async_delete_contact_details(self, contact_details_guid: str, body: O example: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime" : "{{$isoTimestamp}}", @@ -5808,7 +5811,7 @@ def delete_contact_details(self, contact_details_guid: str, body: Optional[dict example: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime" : "{{$isoTimestamp}}", @@ -6761,7 +6764,7 @@ async def _async_delete_perspective(self, perspective_guid: str, body: Optional[ example: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "deleteMethod" : "LOOK_FOR_LINEAGE", "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", @@ -6803,7 +6806,7 @@ def delete_perspective(self, perspective_guid: str, body: Optional[dict | Delete example: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "deleteMethod" : "LOOK_FOR_LINEAGE", "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", @@ -7567,7 +7570,7 @@ async def _async_delete_skill(self, skill_guid: str, body: Optional[dict | Delet example: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "deleteMethod" : "LOOK_FOR_LINEAGE", "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", @@ -7609,7 +7612,7 @@ def delete_skill(self, skill_guid: str, body: Optional[dict | DeleteElementReque example: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "deleteMethod" : "LOOK_FOR_LINEAGE", "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", diff --git a/pyegeria/omvs/classification_explorer.py b/pyegeria/omvs/classification_explorer.py index 3a613cbc..533a51c1 100644 --- a/pyegeria/omvs/classification_explorer.py +++ b/pyegeria/omvs/classification_explorer.py @@ -8874,8 +8874,10 @@ async def _async_set_retention_classification( f"{self.classification_command_root}/elements/{element_guid}/retention" ) + # Egeria renamed the class RetentionProperties (main, 2026-10); it still accepts the old name, + # which is the only one older servers know, so accept both (ISSUE-154). await self._async_new_classification_request( - url, prop=["RetentionClassificationProperties"], body=body + url, prop=["RetentionClassificationProperties", "RetentionProperties"], body=body ) def set_retention_classification( diff --git a/pyegeria/omvs/collection_manager.py b/pyegeria/omvs/collection_manager.py index 1d59ff8d..9a179908 100644 --- a/pyegeria/omvs/collection_manager.py +++ b/pyegeria/omvs/collection_manager.py @@ -6602,6 +6602,43 @@ def delete_collection(self, collection_guid: str, body: Optional[dict | DeleteEl loop = asyncio.get_event_loop() loop.run_until_complete(self._async_delete_collection(collection_guid, body, cascade)) + @dynamic_catch + async def _async_archive_collection(self, collection_guid: str, + body: Optional[dict | DeleteElementRequestBody] = None) -> None: + """Archive a collection rather than delete it (ISSUE-159). Sends the collection delete with deleteMethod + ARCHIVE: Egeria classifies the collection Memento, so it stays readable for lineage, and renames its + qualifiedName with an `_archivedOn_` suffix. Reverse it with + MetadataExpert.restore_metadata_element. Async version. + + Parameters + ---------- + collection_guid: str + The guid of the collection to archive. + body: dict | DeleteElementRequestBody, optional + May set archiveDate, archiveProcess and archiveProperties; deleteMethod is always ARCHIVE. + """ + url = f"{self.collection_command_root}/{collection_guid}/delete" + await self._async_delete_element_request(url, self._archive_delete_body(body)) + logger.info(f"Archived collection {collection_guid}") + + @dynamic_catch + def archive_collection(self, collection_guid: str, + body: Optional[dict | DeleteElementRequestBody] = None) -> None: + """Archive a collection rather than delete it (ISSUE-159). Sends the collection delete with deleteMethod + ARCHIVE: Egeria classifies the collection Memento, so it stays readable for lineage, and renames its + qualifiedName with an `_archivedOn_` suffix. Reverse it with + MetadataExpert.restore_metadata_element. + + Parameters + ---------- + collection_guid: str + The guid of the collection to archive. + body: dict | DeleteElementRequestBody, optional + May set archiveDate, archiveProcess and archiveProperties; deleteMethod is always ARCHIVE. + """ + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_archive_collection(collection_guid, body)) + @dynamic_catch async def _async_add_to_collection(self, collection_guid: str, element_guid: str, diff --git a/pyegeria/omvs/community_matters_omvs.py b/pyegeria/omvs/community_matters_omvs.py index d39101a4..8676a061 100644 --- a/pyegeria/omvs/community_matters_omvs.py +++ b/pyegeria/omvs/community_matters_omvs.py @@ -478,14 +478,18 @@ def get_community_by_guid( @dynamic_catch async def _async_link_crowd_sourcing_contribution(self, element_guid: str, contributor_guid: str, body: Optional[dict | NewRelationshipRequestBody] = None) -> None: - """Attach a contributor to an element they crowd-sourced content for (CrowdSourcingContribution relationship). Async version.""" + """Attach a contributor to an element they crowd-sourced content for (CrowdSourcingContribution relationship). + The relationship properties' `roleType` is a CrowdSourcingRole: PROPOSER, REVIEWER, SUPPORTER, APPROVER + or OTHER. Async version.""" url = f"{self.community_command_base}/elements/{element_guid}/crowd-sourcing-contributions/{contributor_guid}/attach" await self._async_new_relationship_request(url, ["CrowdSourcingContributionProperties"], body) @dynamic_catch def link_crowd_sourcing_contribution(self, element_guid: str, contributor_guid: str, body: Optional[dict | NewRelationshipRequestBody] = None) -> None: - """Attach a contributor to an element they crowd-sourced content for (CrowdSourcingContribution relationship).""" + """Attach a contributor to an element they crowd-sourced content for (CrowdSourcingContribution relationship). + The relationship properties' `roleType` is a CrowdSourcingRole: PROPOSER, REVIEWER, SUPPORTER, APPROVER + or OTHER.""" loop = asyncio.get_event_loop() loop.run_until_complete(self._async_link_crowd_sourcing_contribution(element_guid, contributor_guid, body)) diff --git a/pyegeria/omvs/governance_officer.py b/pyegeria/omvs/governance_officer.py index 30cfbdb8..e654c9c1 100644 --- a/pyegeria/omvs/governance_officer.py +++ b/pyegeria/omvs/governance_officer.py @@ -34,17 +34,26 @@ NewClassificationRequestBody, DeleteClassificationRequestBody) from pyegeria.core.utils import dynamic_catch -GOV_DEF_PROPERTIES_LIST = ["GovernanceDefinitionProperties", "GovernanceStrategyProperties", "RegulationProperties", - "GovernanceControlProperties", "SecurityAccessControlProperties", "NamingStandardRuleProperties", - "CertificationTypeProperties", "LicenseTypeProperties", "GovernanceApproachProperties", - "DataProcessingPurposeProperties","RegulationProperties","GovernanceControlProperties", - "BusinessImperativeProperties", "RegulationArticleProperties", "ThreatProperties", - "DataLensProperties","NotificationTypeProperties", "ServiceAccessControlProperties", - "GovernanceZoneProperties", "GovernancePolicyProperties", "GovernancePrincipleProperties", - "GovernanceObligationProperties", "GovernanceApproachProperties", "RequirementProperties", - "GovernanceActionProcessProperties", "GovernanceActionTypeProperties","ResearchQuestionProperties", - "TermsAndConditionsProperties", "GovernanceRuleProperties","GovernanceMetricProperties", - "ServiceLevelObjectiveProperties", "MethodologyProperties"] +# GovernanceDefinitionProperties and every subclass in its @JsonSubTypes tree, from Egeria main +# (8de9572c, 2026-10-26). Regenerate from the Java when Egeria adds a type; a hand-kept list missed +# DataQualityRuleProperties and others, so update_governance_definition refused them (ISSUE-158). +GOV_DEF_PROPERTIES_LIST = [ + "AnalyticalActionProcessProperties", "BusinessImperativeProperties", "CataloguingActionProcessProperties", + "CertificationTypeProperties", "DataLensProperties", "DataProcessingPurposeProperties", + "DataQualityRuleProperties", "DeletingActionProcessProperties", "ExceptionTypeProperties", + "ExploringActionProcessProperties", "GovernanceActionProcessProperties", "GovernanceActionProcessStepProperties", + "GovernanceActionProperties", "GovernanceActionTypeProperties", "GovernanceApproachProperties", + "GovernanceControlProperties", "GovernanceDefinitionProperties", "GovernanceDriverProperties", + "GovernanceMetricProperties", "GovernanceObligationProperties", "GovernancePolicyProperties", + "GovernancePrincipleProperties", "GovernanceProcedureProperties", "GovernanceResponsibilityProperties", + "GovernanceRuleProperties", "GovernanceStrategyProperties", "GovernanceZoneProperties", + "LicenseTypeProperties", "MethodologyProperties", "NamingStandardRuleProperties", + "NotificationTypeProperties", "ProvisioningActionProcessProperties", "RegulationArticleProperties", + "RegulationProperties", "RequirementProperties", "ResearchQuestionProperties", + "RiskProperties", "SecurityAccessControlProperties", "ServiceAccessControlProperties", + "ServiceLevelObjectiveProperties", "SubscribingActionProcessProperties", "SurveyingActionProcessProperties", + "TermsAndConditionsProperties", "ThreatProperties", +] GOV_BASIC_TYPES = ["BusinessImperative", "RegulationArticle", "Threat", "GovernancePrinciple", "GovernanceObligation", "GovernanceApproach", "GovernanceProcessingPurpose"] diff --git a/pyegeria/omvs/metadata_expert.py b/pyegeria/omvs/metadata_expert.py index b22d3ec7..bc013c5b 100644 --- a/pyegeria/omvs/metadata_expert.py +++ b/pyegeria/omvs/metadata_expert.py @@ -516,7 +516,7 @@ async def _async_delete_metadata_element(self, metadata_element_guid: str, body: "class" : "DeleteElementRequestBody", "externalSourceGUID" : "", "externalSourceName" : "", - "cascadeDelete" : false, + "cascadedDelete" : false, "deleteMethod" : "SOFT_DELETE", "forLineage" : false, "forDuplicateProcessing" : false, @@ -3881,15 +3881,16 @@ async def _async_count_metadata_elements( - A `FindRequestBody` search structure (see `find_metadata_elements`). push_down: bool, optional, default = None - ISSUE-38 (PY-18): controls whether the repository counts matching - rows itself (`pushDown=True`, the server default when omitted — - fast, but does not apply the per-element visibility check + rows itself (`pushDown=True` — fast, but does not apply the + per-element visibility check `find_metadata_elements` does, so it can disagree with a list built from the same search) or whether the server retrieves and counts the matching elements the same way a find would (`pushDown=False` — agrees with a corresponding `find_metadata_elements` call by construction, at the cost of reading every one of them server-side). `None` omits the query - parameter entirely, i.e. today's existing behaviour. + parameter, and the server default applies: `false` on Egeria + main since the 2026-10-18 fixes, `true` on older servers. timeout: int, default = default_timeout - http request timeout for this request @@ -3946,7 +3947,7 @@ def count_metadata_elements( - A `FindRequestBody` search structure (see `find_metadata_elements`). push_down: bool, optional, default = None - ISSUE-38 (PY-18): see `_async_count_metadata_elements` for the full - explanation. `None` omits the query parameter (today's behaviour). + explanation. `None` omits the query parameter (server default). timeout: int, default = default_timeout - http request timeout for this request @@ -4196,8 +4197,8 @@ async def _async_count_relationships_between_elements( (see `find_relationships_between_elements`). push_down: bool, optional, default = None - ISSUE-38 (PY-18): controls whether the repository counts matching - rows itself (`pushDown=True`, the server default when omitted — - fast, but skips the per-relationship-end visibility check + rows itself (`pushDown=True` — fast, but skips the + per-relationship-end visibility check `find_relationships_between_elements` applies, so it can disagree with a list built from the same search — this is exactly the `count_relationships_between_elements("Exception")` vs. @@ -4206,8 +4207,9 @@ async def _async_count_relationships_between_elements( matching relationships the same way a find would (`pushDown=False` — agrees with a corresponding find by construction, at the cost of reading every one of them - server-side). `None` omits the query parameter entirely, i.e. - today's existing behaviour. + server-side). `None` omits the query parameter, and the server + default applies: `false` on Egeria main since the 2026-10-18 + fixes, `true` on older servers. timeout: int, default = default_timeout - http request timeout for this request @@ -4267,7 +4269,7 @@ def count_relationships_between_elements( push_down: bool, optional, default = None - ISSUE-38 (PY-18): see `_async_count_relationships_between_elements` for the full explanation. `None` omits the query parameter - (today's behaviour). + (server default). timeout: int, default = default_timeout - http request timeout for this request diff --git a/pyegeria/omvs/product_manager.py b/pyegeria/omvs/product_manager.py index e66809a7..492ef1c9 100644 --- a/pyegeria/omvs/product_manager.py +++ b/pyegeria/omvs/product_manager.py @@ -322,7 +322,7 @@ async def _async_delete_digital_product( ```json { "class": "DeleteElementRequestBody", - "cascadeDelete": false + "cascadedDelete": false } ``` """ @@ -364,7 +364,7 @@ def delete_digital_product( ```json { "class": "DeleteElementRequestBody", - "cascadeDelete": false + "cascadedDelete": false } ``` """ diff --git a/pyegeria/omvs/project_manager.py b/pyegeria/omvs/project_manager.py index 0a4cc024..1667aec6 100644 --- a/pyegeria/omvs/project_manager.py +++ b/pyegeria/omvs/project_manager.py @@ -1782,6 +1782,45 @@ def delete_project( loop = asyncio.get_event_loop() loop.run_until_complete(self._async_delete_project(project_guid, cascade, body)) + @dynamic_catch + async def _async_archive_project(self, project_guid: str, + body: Optional[dict | DeleteElementRequestBody] = None) -> None: + """Archive a project rather than delete it (ISSUE-159). Sends the project delete with deleteMethod + ARCHIVE: Egeria classifies the project Memento, so it stays readable for lineage, and renames its + qualifiedName with an `_archivedOn_` suffix. Reverse it with + MetadataExpert.restore_metadata_element. Async version. + + Parameters + ---------- + project_guid: str + The guid of the project to archive. + body: dict | DeleteElementRequestBody, optional + May set archiveDate, archiveProcess and archiveProperties; deleteMethod is always ARCHIVE. + """ + url = ( + f"{self.platform_url}/servers/{self.view_server}/api/open-metadata/project-manager/projects/" + f"{project_guid}/delete" + ) + await self._async_delete_element_request(url, self._archive_delete_body(body)) + logger.info(f"Archived project {project_guid}") + + @dynamic_catch + def archive_project(self, project_guid: str, body: Optional[dict | DeleteElementRequestBody] = None) -> None: + """Archive a project rather than delete it (ISSUE-159). Sends the project delete with deleteMethod + ARCHIVE: Egeria classifies the project Memento, so it stays readable for lineage, and renames its + qualifiedName with an `_archivedOn_` suffix. Reverse it with + MetadataExpert.restore_metadata_element. + + Parameters + ---------- + project_guid: str + The guid of the project to archive. + body: dict | DeleteElementRequestBody, optional + May set archiveDate, archiveProcess and archiveProperties; deleteMethod is always ARCHIVE. + """ + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_archive_project(project_guid, body)) + @dynamic_catch async def _async_set_project_dependency(self, project_guid: str, upstream_project_guid: str, diff --git a/pyegeria/omvs/reference_data.py b/pyegeria/omvs/reference_data.py index b46581ba..237c0c95 100644 --- a/pyegeria/omvs/reference_data.py +++ b/pyegeria/omvs/reference_data.py @@ -2260,7 +2260,7 @@ async def _async_delete_valid_value_definition( Sample body: { "class" : "DeleteElementRequestBody", - "cascadeDelete" : false, + "cascadedDelete" : false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime" : "{{$isoTimestamp}}", @@ -2304,7 +2304,7 @@ def delete_valid_value_definition( Sample body: { "class": "DeleteElementRequestBody", - "cascadeDelete": false, + "cascadedDelete": false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime": "{{$isoTimestamp}}", diff --git a/pyegeria/omvs/solution_architect.py b/pyegeria/omvs/solution_architect.py index 80250314..92b8cdbe 100644 --- a/pyegeria/omvs/solution_architect.py +++ b/pyegeria/omvs/solution_architect.py @@ -25,7 +25,7 @@ NestedDesignPatternProperties, SpecializedDesignPatternProperties, \ RelatedDesignPatternProperties, SolutionDesignProperties, \ SolutionComponentActorProperties, SolutionLinkingWireProperties, GetRequestBody, \ - UpdateRelationshipRequestBody + UpdateRelationshipRequestBody, FilterRequestBody from pyegeria.view.output_formatter import extract_mermaid_only, \ populate_common_columns from pyegeria.core.utils import body_slimmer, dynamic_catch @@ -6920,6 +6920,146 @@ def detach_concept_bead_extension(self, concept_bead_guid: str, extension_concep loop = asyncio.get_event_loop() loop.run_until_complete(self._async_detach_concept_bead_extension(concept_bead_guid, extension_concept_bead_guid, body)) + # + # Solution ports (ISSUE-85; Egeria added these endpoints 2026-09-06) + # + @dynamic_catch + async def _async_create_solution_port(self, body: dict | NewElementRequestBody) -> str: + """Create a solution port, with SolutionPortProperties (which adds `direction`: UNKNOWN, + INPUT, OUTPUT or INOUT to the design model element properties). Attach it to its component + with link_solution_component_port. Async version.""" + url = f"{self.solution_architect_command_root}/solution-ports" + return await self._async_create_element_body_request(url, ["SolutionPortProperties"], body) + + @dynamic_catch + def create_solution_port(self, body: dict | NewElementRequestBody) -> str: + """Create a solution port, with SolutionPortProperties (which adds `direction`: UNKNOWN, + INPUT, OUTPUT or INOUT to the design model element properties). Attach it to its component + with link_solution_component_port.""" + loop = asyncio.get_event_loop() + return loop.run_until_complete(self._async_create_solution_port(body)) + + @dynamic_catch + async def _async_create_solution_port_from_template(self, body: dict | TemplateRequestBody) -> str: + """Create a solution port from a template. Async version.""" + url = f"{self.solution_architect_command_root}/solution-ports/from-template" + return await self._async_create_element_from_template(url, body) + + @dynamic_catch + def create_solution_port_from_template(self, body: dict | TemplateRequestBody) -> str: + """Create a solution port from a template.""" + loop = asyncio.get_event_loop() + return loop.run_until_complete(self._async_create_solution_port_from_template(body)) + + @dynamic_catch + async def _async_update_solution_port(self, solution_port_guid: str, + body: dict | UpdateElementRequestBody) -> None: + """Update the properties of a solution port. Async version.""" + url = f"{self.solution_architect_command_root}/solution-ports/{solution_port_guid}/update" + await self._async_update_element_body_request(url, ["SolutionPortProperties"], body) + + @dynamic_catch + def update_solution_port(self, solution_port_guid: str, body: dict | UpdateElementRequestBody) -> None: + """Update the properties of a solution port.""" + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_update_solution_port(solution_port_guid, body)) + + @dynamic_catch + async def _async_delete_solution_port(self, solution_port_guid: str, cascade_delete: bool = False, + body: Optional[dict | DeleteElementRequestBody] = None) -> None: + """Delete a solution port. Async version.""" + url = f"{self.solution_architect_command_root}/solution-ports/{solution_port_guid}/delete" + await self._async_delete_element_request(url, body, cascade_delete=cascade_delete) + + @dynamic_catch + def delete_solution_port(self, solution_port_guid: str, cascade_delete: bool = False, + body: Optional[dict | DeleteElementRequestBody] = None) -> None: + """Delete a solution port.""" + loop = asyncio.get_event_loop() + loop.run_until_complete(self._async_delete_solution_port(solution_port_guid, cascade_delete, body)) + + @dynamic_catch + async def _async_find_solution_ports(self, search_string: str = "*", + body: Optional[dict | SearchStringRequestBody] = None, + starts_with: bool = True, ends_with: bool = False, + ignore_case: bool = False, start_from: int = 0, + page_size: int = 100, graph_query_depth: int = 3, + output_format: str = "JSON", report_spec: str | dict = None, + **kwargs) -> list[dict] | str: + """Retrieve the solution ports whose properties contain the search string. Async version.""" + url = f"{self.solution_architect_command_root}/solution-ports/by-search-string" + return await self._async_find_request(url, _type="SolutionPort", + _gen_output=self._generate_referenceable_output, + search_string=search_string, body=body, + starts_with=starts_with, ends_with=ends_with, + ignore_case=ignore_case, start_from=start_from, + page_size=page_size, graph_query_depth=graph_query_depth, + output_format=output_format, report_spec=report_spec, **kwargs) + + @dynamic_catch + def find_solution_ports(self, search_string: str = "*", + body: Optional[dict | SearchStringRequestBody] = None, + starts_with: bool = True, ends_with: bool = False, + ignore_case: bool = False, start_from: int = 0, + page_size: int = 100, graph_query_depth: int = 3, + output_format: str = "JSON", report_spec: str | dict = None, + **kwargs) -> list[dict] | str: + """Retrieve the solution ports whose properties contain the search string.""" + loop = asyncio.get_event_loop() + return loop.run_until_complete( + self._async_find_solution_ports(search_string, body, starts_with, ends_with, ignore_case, + start_from, page_size, graph_query_depth, output_format, + report_spec, **kwargs)) + + @dynamic_catch + async def _async_get_solution_ports_by_name(self, name: str | None = None, + body: Optional[dict | FilterRequestBody] = None, + start_from: int = 0, page_size: int = 0, + graph_query_depth: int = 3, output_format: str = "JSON", + report_spec: str | dict = None, **kwargs) -> list | str: + """Retrieve the solution ports with a particular name. Async version.""" + url = f"{self.solution_architect_command_root}/solution-ports/by-name" + return await self._async_get_name_request(url, _type="SolutionPort", + _gen_output=self._generate_referenceable_output, + filter_string=name, graph_query_depth=graph_query_depth, + start_from=start_from, page_size=page_size, + output_format=output_format, report_spec=report_spec, + body=body, **kwargs) + + @dynamic_catch + def get_solution_ports_by_name(self, name: str | None = None, + body: Optional[dict | FilterRequestBody] = None, + start_from: int = 0, page_size: int = 0, + graph_query_depth: int = 3, output_format: str = "JSON", + report_spec: str | dict = None, **kwargs) -> list | str: + """Retrieve the solution ports with a particular name.""" + loop = asyncio.get_event_loop() + return loop.run_until_complete( + self._async_get_solution_ports_by_name(name, body, start_from, page_size, graph_query_depth, + output_format, report_spec, **kwargs)) + + @dynamic_catch + async def _async_get_solution_port_by_guid(self, solution_port_guid: str, body: Optional[dict] = None, + graph_query_depth: int = 3, output_format: str = "JSON", + report_spec: str | dict = None, **kwargs) -> dict | str: + """Return the properties of a specific solution port. Async version.""" + url = f"{self.solution_architect_command_root}/solution-ports/{solution_port_guid}/retrieve" + return await self._async_get_guid_request(url, _type="SolutionPort", + _gen_output=self._generate_referenceable_output, + graph_query_depth=graph_query_depth, + output_format=output_format, report_spec=report_spec, + body=body, **kwargs) + + @dynamic_catch + def get_solution_port_by_guid(self, solution_port_guid: str, body: Optional[dict] = None, + graph_query_depth: int = 3, output_format: str = "JSON", + report_spec: str | dict = None, **kwargs) -> dict | str: + """Return the properties of a specific solution port.""" + loop = asyncio.get_event_loop() + return loop.run_until_complete( + self._async_get_solution_port_by_guid(solution_port_guid, body, graph_query_depth, + output_format, report_spec, **kwargs)) + @dynamic_catch async def _async_link_solution_component_port(self, solution_component_guid: str, solution_port_guid: str, body: Optional[dict | NewRelationshipRequestBody] = None) -> None: diff --git a/pyegeria/omvs/valid_metadata.py b/pyegeria/omvs/valid_metadata.py index 68fde9a6..c02ca843 100644 --- a/pyegeria/omvs/valid_metadata.py +++ b/pyegeria/omvs/valid_metadata.py @@ -2685,7 +2685,7 @@ async def _async_delete_specification_property( Example payload structures: { "class": "DeleteElementRequestBody", - "cascadeDelete": false, + "cascadedDelete": false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime": "{{$isoTimestamp}}", @@ -2727,7 +2727,7 @@ def delete_specification_property(self, spec_property_guid: str, body: dict | De Example payload structures: { "class": "DeleteElementRequestBody", - "cascadeDelete": false, + "cascadedDelete": false, "externalSourceGUID": "add guid here", "externalSourceName": "add qualified name here", "effectiveTime": "{{$isoTimestamp}}", diff --git a/sample-data/templates/advanced/Data Designer/Assign_Data_Value_Specification.md b/sample-data/templates/advanced/Data Designer/Assign_Data_Value_Specification.md index c9fb5b65..b68a80ca 100644 --- a/sample-data/templates/advanced/Data Designer/Assign_Data_Value_Specification.md +++ b/sample-data/templates/advanced/Data Designer/Assign_Data_Value_Specification.md @@ -51,6 +51,24 @@ ___ > **Description**: Threshold for assignment. +### Label +> **Input Required**: False + +> **Attribute Type**: Simple + +> **Description**: A label used to identify or categorise a relationship link. + +> **Alternative Labels**: Wire Label + + +### Description +> **Input Required**: False + +> **Attribute Type**: Simple + +> **Description**: A description. + + ### Confidence > **Input Required**: False diff --git a/sample-data/templates/basic/Data Designer/Assign_Data_Value_Specification.md b/sample-data/templates/basic/Data Designer/Assign_Data_Value_Specification.md index 8dff49e2..e014f310 100644 --- a/sample-data/templates/basic/Data Designer/Assign_Data_Value_Specification.md +++ b/sample-data/templates/basic/Data Designer/Assign_Data_Value_Specification.md @@ -51,4 +51,22 @@ ___ > **Description**: Threshold for assignment. +### Label +> **Input Required**: False + +> **Attribute Type**: Simple + +> **Description**: A label used to identify or categorise a relationship link. + +> **Alternative Labels**: Wire Label + + +### Description +> **Input Required**: False + +> **Attribute Type**: Simple + +> **Description**: A description. + + ___ diff --git a/tests/micro-tests/test_collection_manager_subclasses.py b/tests/micro-tests/test_collection_manager_subclasses.py index c037da0e..db2135c9 100644 --- a/tests/micro-tests/test_collection_manager_subclasses.py +++ b/tests/micro-tests/test_collection_manager_subclasses.py @@ -94,7 +94,7 @@ async def test_cascade_true_reaches_the_server(cls, method, tail, monkeypatch): await getattr(client, method)("g1", cascade=True) assert calls[0]["url"].endswith(tail) - assert calls[0]["body"]["cascadeDelete"] is True + assert calls[0]["body"]["cascadedDelete"] is True assert calls[0]["body"]["class"] == "DeleteElementRequestBody" @@ -105,7 +105,7 @@ async def test_cascade_default_does_not_cascade(method, monkeypatch): await getattr(client, method)("g1") - assert calls[0]["body"].get("cascadeDelete") in (None, False) + assert calls[0]["body"].get("cascadedDelete") in (None, False) @pytest.mark.asyncio @@ -115,4 +115,4 @@ async def test_an_explicit_body_is_still_honoured(monkeypatch): await client._async_delete_collection("g1", body=body) - assert calls[0]["body"]["cascadeDelete"] is True and calls[0]["body"]["deleteMethod"] == "PURGE" + assert calls[0]["body"]["cascadedDelete"] is True and calls[0]["body"]["deleteMethod"] == "PURGE" diff --git a/tests/micro-tests/test_delete_element_request_body.py b/tests/micro-tests/test_delete_element_request_body.py index 8c61b5bf..0770e672 100644 --- a/tests/micro-tests/test_delete_element_request_body.py +++ b/tests/micro-tests/test_delete_element_request_body.py @@ -60,12 +60,12 @@ def test_delete_method_survives_validation(): def test_cascade_delete_and_delete_method_serialize_with_canonical_names(): b = DeleteElementRequestBody.model_validate( - {"class": "DeleteElementRequestBody", "cascadedDelete": True, "deleteMethod": "ARCHIVE"}) + {"class": "DeleteElementRequestBody", "cascadeDelete": True, "deleteMethod": "ARCHIVE"}) dumped = b.model_dump_json(exclude_none=True) - assert '"cascadeDelete":true' in dumped + # Egeria's field is cascadedDelete; older servers ignore cascadeDelete (ISSUE-132, ISSUE-154). + assert '"cascadedDelete":true' in dumped assert '"deleteMethod":"ARCHIVE"' in dumped - # confirms the alt spelling never leaks into the outgoing body - assert "cascadedDelete" not in dumped + assert '"cascadeDelete"' not in dumped async def test_delete_element_request_no_body_path_sends_cascade_delete(): @@ -79,7 +79,7 @@ async def test_delete_element_request_no_body_path_sends_cascade_delete(): ) assert cap["body"] is not None - assert '"cascadeDelete": true' in cap["body"] + assert '"cascadedDelete": true' in cap["body"] async def test_delete_element_request_forwards_caller_dict_body(): diff --git a/tests/micro-tests/test_issues_154_158_159_85.py b/tests/micro-tests/test_issues_154_158_159_85.py new file mode 100644 index 00000000..d3dcdaf0 --- /dev/null +++ b/tests/micro-tests/test_issues_154_158_159_85.py @@ -0,0 +1,151 @@ +""" +ISSUE-85: SolutionArchitect solution-port create/update/delete/find/get wrappers. +ISSUE-154: cascade deletes send Egeria's cascadedDelete; renamed property classes are accepted +under both names. +ISSUE-158: GOV_DEF_PROPERTIES_LIST covers DataQualityRuleProperties; the governance create fallback +names the fields it would drop. +ISSUE-159: archive_project / archive_collection send the delete with deleteMethod ARCHIVE. +""" +import json + +import pytest + +from md_processing.v2.governance import GovernanceProcessor +from pyegeria.core._server_client import ServerClient +from pyegeria.models import DeleteElementRequestBody +from pyegeria.omvs.actor_manager import ActorManager +from pyegeria.omvs.classification_explorer import ClassificationExplorer +from pyegeria.omvs.collection_manager import CollectionManager +from pyegeria.omvs.governance_officer import GOV_DEF_PROPERTIES_LIST +from pyegeria.omvs.project_manager import ProjectManager +from pyegeria.omvs.solution_architect import SolutionArchitect + + +class _Response: + def json(self): + return {"relatedHTTPCode": 200, "guid": "new-guid", "elements": [{"elementHeader": {"guid": "p1"}}]} + + +def _capture(cls, monkeypatch): + monkeypatch.setattr(cls, "check_connection", lambda self: "test") + client = cls("view-server", "https://localhost:1", "user", "pwd") + calls: list = [] + + async def fake_make_request(method, url, payload=None, *args, **kwargs): + calls.append({"method": method, "url": url, + "body": json.loads(payload) if isinstance(payload, str) else payload}) + return _Response() + + client._async_make_request = fake_make_request + return client, calls + + +# --- ISSUE-85 ------------------------------------------------------------------------------- + +SA_ROOT = "/api/open-metadata/solution-architect" + + +async def test_create_solution_port(monkeypatch): + client, calls = _capture(SolutionArchitect, monkeypatch) + guid = await client._async_create_solution_port({ + "class": "NewElementRequestBody", + "properties": {"class": "SolutionPortProperties", "qualifiedName": "SolutionPort::p", "direction": "INPUT"}, + }) + assert guid == "new-guid" + assert calls[0]["url"].endswith(f"{SA_ROOT}/solution-ports") + assert calls[0]["body"]["properties"]["class"] == "SolutionPortProperties" + assert calls[0]["body"]["properties"]["direction"] == "INPUT" + + +@pytest.mark.parametrize("method, args, tail, body_class", [ + ("_async_create_solution_port_from_template", + ({"class": "TemplateRequestBody", "templateGUID": "t1"},), "/solution-ports/from-template", + "TemplateRequestBody"), + ("_async_update_solution_port", + ("p1", {"class": "UpdateElementRequestBody", + "properties": {"class": "SolutionPortProperties", "direction": "OUTPUT"}}), + "/solution-ports/p1/update", "UpdateElementRequestBody"), + ("_async_delete_solution_port", ("p1",), "/solution-ports/p1/delete", "DeleteElementRequestBody"), + ("_async_get_solution_ports_by_name", ("port",), "/solution-ports/by-name", "FilterRequestBody"), + ("_async_find_solution_ports", ("port",), "/solution-ports/by-search-string", "SearchStringRequestBody"), + ("_async_get_solution_port_by_guid", ("p1",), "/solution-ports/p1/retrieve", "GetRequestBody"), +]) +async def test_solution_port_endpoints(monkeypatch, method, args, tail, body_class): + client, calls = _capture(SolutionArchitect, monkeypatch) + await getattr(client, method)(*args) + assert calls[0]["method"] == "POST" + assert calls[0]["url"].endswith(SA_ROOT + tail) + assert calls[0]["body"]["class"] == body_class + + +# --- ISSUE-159 ------------------------------------------------------------------------------ + +@pytest.mark.parametrize("cls, method, tail", [ + (ProjectManager, "_async_archive_project", "/project-manager/projects/g1/delete"), + (CollectionManager, "_async_archive_collection", "/collection-manager/collections/g1/delete"), +]) +async def test_archive_sends_delete_with_archive_method(monkeypatch, cls, method, tail): + client, calls = _capture(cls, monkeypatch) + await getattr(client, method)("g1") + assert calls[0]["url"].endswith(tail) + assert calls[0]["body"]["class"] == "DeleteElementRequestBody" + assert calls[0]["body"]["deleteMethod"] == "ARCHIVE" + assert not calls[0]["body"].get("cascadedDelete") + + +async def test_archive_keeps_caller_fields_but_forces_archive(monkeypatch): + client, calls = _capture(ProjectManager, monkeypatch) + caller_body = {"class": "DeleteElementRequestBody", "deleteMethod": "PURGE", "archiveProcess": "RE close-out"} + await client._async_archive_project("g1", caller_body) + assert calls[0]["body"]["deleteMethod"] == "ARCHIVE" + assert calls[0]["body"]["archiveProcess"] == "RE close-out" + assert caller_body["deleteMethod"] == "PURGE" # the caller's dict is not changed + + +def test_archive_body_from_model(): + body = DeleteElementRequestBody.model_validate({"class": "DeleteElementRequestBody", "deleteMethod": "PURGE"}) + archived = ServerClient._archive_delete_body(body) + assert archived.delete_method == "ARCHIVE" + assert body.delete_method == "PURGE" + + +# --- ISSUE-154 ------------------------------------------------------------------------------ + +@pytest.mark.parametrize("name", ["RetentionClassificationProperties", "RetentionProperties"]) +async def test_retention_accepts_old_and_new_class_names(monkeypatch, name): + client, calls = _capture(ClassificationExplorer, monkeypatch) + await client._async_set_retention_classification( + "e1", {"class": "NewClassificationRequestBody", "properties": {"class": name, "retentionBasis": 2}}) + assert calls[0]["body"]["properties"]["class"] == name + + +@pytest.mark.parametrize("name", ["SecurityGroupMembershipProperties", "SecurityListMembershipProperties"]) +async def test_security_membership_accepts_old_and_new_class_names(monkeypatch, name): + client, calls = _capture(ActorManager, monkeypatch) + await client._async_add_security_group_membership( + "u1", body={"class": "NewClassificationRequestBody", "properties": {"class": name, "groups": ["g"]}}) + assert calls[0]["body"]["properties"]["class"] == name + + +# --- ISSUE-158 ------------------------------------------------------------------------------ + +def test_governance_definition_list_covers_data_quality_rule_without_duplicates(): + assert "DataQualityRuleProperties" in GOV_DEF_PROPERTIES_LIST + assert len(GOV_DEF_PROPERTIES_LIST) == len(set(GOV_DEF_PROPERTIES_LIST)) + + +def test_base_fallback_reports_type_specific_fields(): + create_props = { + "class": "GovernanceMetricProperties", "typeName": "GovernanceMetric", + "qualifiedName": "GovernanceMetric::m", "displayName": "m", + "measurement": "count of open issues", "target": "", "domainIdentifier": None, + } + fallback = GovernanceProcessor._to_base_governance_definition_properties(create_props, "Governance Metric") + assert GovernanceProcessor._fields_dropped_by_base_fallback(create_props, fallback) == ["measurement"] + + +def test_base_fallback_with_only_cross_type_fields_drops_nothing(): + create_props = {"class": "GovernanceStrategyProperties", "typeName": "GovernanceStrategy", + "qualifiedName": "GovernanceStrategy::s", "displayName": "s", "description": "d"} + fallback = GovernanceProcessor._to_base_governance_definition_properties(create_props, "Governance Strategy") + assert GovernanceProcessor._fields_dropped_by_base_fallback(create_props, fallback) == [] diff --git a/tests/micro-tests/test_metadata_expert_delete_methods.py b/tests/micro-tests/test_metadata_expert_delete_methods.py index 85edcd90..3da61030 100644 --- a/tests/micro-tests/test_metadata_expert_delete_methods.py +++ b/tests/micro-tests/test_metadata_expert_delete_methods.py @@ -70,7 +70,7 @@ def test_delete_metadata_element_uses_delete_element_request_body(): assert cap["url"].endswith("/metadata-elements/element-guid/delete") assert '"deleteMethod": "PURGE"' in cap["body"] - assert '"cascadeDelete": true' in cap["body"] + assert '"cascadedDelete": true' in cap["body"] def test_delete_metadata_element_cascade_delete_param_flows_through_no_body(): @@ -81,7 +81,7 @@ def test_delete_metadata_element_cascade_delete_param_flows_through_no_body(): me.delete_metadata_element("element-guid", cascade_delete=True) assert cap["url"].endswith("/metadata-elements/element-guid/delete") - assert '"cascadeDelete": true' in cap["body"] + assert '"cascadedDelete": true' in cap["body"] def test_delete_relationship_request_body_still_preserves_delete_method(): @@ -98,4 +98,4 @@ def test_delete_element_request_body_still_preserves_delete_method_and_cascade() {"class": "DeleteElementRequestBody", "deleteMethod": "PURGE", "cascadeDelete": True}) dumped = b.model_dump(by_alias=True, exclude_none=True) assert dumped["deleteMethod"] == "PURGE" - assert dumped["cascadeDelete"] is True + assert dumped["cascadedDelete"] is True