diff --git a/EGERIA_OPEN_ISSUES.md b/EGERIA_OPEN_ISSUES.md index bfaa195b..2b1dc3e3 100644 --- a/EGERIA_OPEN_ISSUES.md +++ b/EGERIA_OPEN_ISSUES.md @@ -2,7 +2,7 @@ A standalone extract of the **Egeria-Server-side** issues in [`PYEGERIA_ISSUES.md`](PYEGERIA_ISSUES.md), for whoever owns the Egeria side. None of them can be fixed by editing this repo: each needs a server-side fix, an upstream bug report, or a deployment change. -**`PYEGERIA_ISSUES.md` is the source of record.** This file was **re-derived from it on 2026-10-13** (its "Open Egeria Server issues" and "Awaiting re-test" index lists, plus the Egeria-side entries filed under pyegeria items). Don't edit the two independently. When an entry changes status, change it in `PYEGERIA_ISSUES.md`, then re-derive this file. +**`PYEGERIA_ISSUES.md` is the source of record.** This file was **re-derived from it on 2026-10-13**, and updated on 2026-10-28 after the re-sort (ISSUE-166 removed as fixed in Egeria 6.3; ISSUE-79 and ISSUE-38 moved out of *Awaiting re-test*, matching their 2026-10-18 Egeria-team status) (its "Open Egeria Server issues" and "Awaiting re-test" index lists, plus the Egeria-side entries filed under pyegeria items). Don't edit the two independently. When an entry changes status, change it in `PYEGERIA_ISSUES.md`, then re-derive this file. It replaces the 2026-10-05 snapshot `docs/egeria-open-bugs-2026-10-05.md` (deleted; it's in git history). That snapshot's Egeria-team verdicts and fix evidence are now in the individual `PYEGERIA_ISSUES.md` entries. @@ -11,8 +11,8 @@ It replaces the 2026-10-05 snapshot `docs/egeria-open-bugs-2026-10-05.md` (delet | Group | Issues | |---|---| | **Open on the Egeria side** (full entries below) | ISSUE-147, ISSUE-143, ISSUE-130, ISSUE-128, ISSUE-108 | -| **Awaiting re-test:** a fix landed (mostly Egeria `main`, odpi/egeria#9363, merged 2026-10-06), not yet verified live | ISSUE-125, ISSUE-124, ISSUE-117, ISSUE-112, ISSUE-95, ISSUE-90, ISSUE-89, ISSUE-79, ISSUE-38 | -| **Egeria-side, but tracked as pyegeria items** (a pyegeria change waits on Egeria) | ISSUE-146, ISSUE-132 | +| **Awaiting re-test:** a fix landed (mostly Egeria `main`, odpi/egeria#9363, merged 2026-10-06), not yet verified live | ISSUE-125, ISSUE-124, ISSUE-117, ISSUE-112, ISSUE-95, ISSUE-90, ISSUE-89 | +| **Egeria-side, but tracked as pyegeria items** (a pyegeria change waits on Egeria, or Egeria needs information from us) | ISSUE-146, ISSUE-132, ISSUE-79, ISSUE-38 | ## Awaiting re-test @@ -25,48 +25,16 @@ It replaces the 2026-10-05 snapshot `docs/egeria-open-bugs-2026-10-05.md` (delet | ISSUE-95: No catalog template registered for the "Apache Kafka Server" technology type — `Create Kafka Server Element` (Asset Maker) fails with a 400 on `qs-view-server` | awaiting re-test (addressed on the Egeria side, confirmed live 2026-10-05) | | ISSUE-90: [fixed?] `qs-engine-host` retries `startMissedEngineActions` forever when one incomplete engine action's anchor is unreadable by the engine-host user | open | | ISSUE-89: No configurable bearer-token lifetime in `application.properties` — platform tokens are fixed at 3600 s, and the signing key is random per restart | awaiting re-test (addressed in Egeria source 2026-09-19; not verified on the dev platform) | -| ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` | open (Egeria Server), reproduced live 2026-08-27, **re-checked | -| ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55) | re-confirmed 2026-08-18, byte-for-byte identical to the | ## Egeria-side issues tracked as pyegeria items - **ISSUE-146: Deleting a journal entry fails with OMAG-REPOSITORY-HANDLER-400-008, and pyegeria's `remove_note` reaches the delete only through the generic Asset Maker service**: Egeria has no note/action delete, and the only route (the asset delete) fails with 400-008 for journal entries; awaiting Egeria's delete fixes. Full entry in `PYEGERIA_ISSUES.md`. - **ISSUE-132: `DeleteElementRequestBody` serializes the cascade flag as `cascadeDelete`; Egeria's field is `cascadedDelete` and unknown fields are ignored, so no pyegeria cascade delete has ever cascaded**: Egeria's DeleteOptions field is `cascadedDelete`; the pyegeria fix is on hold until Egeria's cascade-delete fix lands. Full entry in `PYEGERIA_ISSUES.md`. +- **ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()`**: Egeria cannot reproduce it and needs information from us (Egeria team, 2026-10-18; checklist in the entry). Full entry in `PYEGERIA_ISSUES.md`. +- **ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55)**: fixed on Egeria's `oak2026` branch, pending its merge to `main` (Egeria team, 2026-10-18). Full entry in `PYEGERIA_ISSUES.md`. ## Open on the Egeria side -### ISSUE-166: Security annotation properties are Java primitives, so an unset CVSS score, exploit/patch flag or posture count reads back as 0 / false - -**Layer:** Egeria Server (type model / properties classes) · **Status:** fixed (Egeria 6.3, odpi/egeria#9389) · **Found:** 2026-10-26, while adding the 6.3 security annotation types to pyegeria (odpi/egeria-workspaces#702 hand-off, step 3; PR #495). - -**FIXED 2026-10-27: odpi/egeria#9389, merged as 12b1aaae.** Verified live the same day on egeria-platform c2ca054c (throwaway VulnerabilityAnnotation ed152edb, deleted, NotFound by GUID): unset cvssScore/exploitAvailable/patchAvailable were omitted; a merge update that left out 8.6 / true / false kept them; an explicit false came back. The pyegeria workarounds below are removed on branch `feature/boxed-properties-audit`. It boxes 217 number and flag fields across 86 property beans, not only the security ones. An unset value is no longer stored and is left out of the response, and a merge update keeps a value it does not mention. Defaults move to `OpenMetadataProperty.defaultValue`, published in the Core Content Pack as `additionalProperties.defaultValue` on each property's valid metadata value. `allowsDuplicateValues` now defaults to true everywhere, including SchemaAttribute (was false). The PR does not migrate stored data: elements written by earlier servers keep the 0 / false they were stored with. **pyegeria:** fixes that work with either server are on branch `feature/boxed-properties-audit`. The client-side workarounds stay until the PR merges and pyegeria targets that server. - -**What happens:** in `SecurityFindingAnnotationProperties`, `VulnerabilityAnnotationProperties` and `SecurityPostureAnnotationProperties` (`frameworks/open-metadata-framework/.../properties/surveyreports/`), these fields are Java primitives with a 0 / false default: - -| Class | Fields | -|---|---| -| `SecurityFindingAnnotationProperties` | `float cvssScore`, `boolean exploitAvailable` | -| `VulnerabilityAnnotationProperties` | `boolean patchAvailable` | -| `SecurityPostureAnnotationProperties` | `long resourcesInScopeCount`, `resourcesTestedCount`, `newFindingCount`, `resolvedFindingCount` | -| `AnnotationProperties` (inherited by all four) | `long sampleSize`, `int samplePercent`, `int confidence`, `long absoluteUncertainty`, `relativeUncertainty` | - -So "never set" and "set to 0 / false" come back identical. Confirmed live on 2026-10-26 (egeria-platform:latest quickstart, as erinoverview on qs-view-server, throwaway `SecTest*` elements, since deleted and verified gone by GUID): a `VulnerabilityAnnotation` created with only `severity: "Medium"` read back with `cvssScore: 0.0`, `exploitAvailable: false` and `patchAvailable: false`. A `SecurityPostureAnnotation` created with only `findingCountsBySeverity` and `newFindingCount` read back with `resourcesInScopeCount: 0`, `resourcesTestedCount: 0` and `resolvedFindingCount: 0`. - -**Why it matters:** each of these values is a reassuring answer. A findings table that shows them as returned tells a reader "CVSS 0, no public exploit, no patch" about a finding nobody assessed, and "0 findings resolved" about a run that never recorded it. `exploitAvailable: false` is the most dangerous, since it is the field a triager sorts on. - -**Question for the Egeria team (Mandy):** could these become boxed types (`Float`, `Boolean`, `Long`, `Integer`) with a `null` default, so that an unset value is omitted from the response? `cvssScore` matters most: a CVSS score of 0.0 is a real rating (severity "None"). - -**Addendum 2026-10-26 (later the same day): a merge update wipes these fields.** While live-testing the Dr.Egeria finding commands (step 4), a `PenetrationTestFindingAnnotation` created with `cvssScore: 8.6` and `exploitAvailable: true` was updated with `mergeUpdate: true` and only `reproductionSteps` in the properties. Afterwards it held `cvssScore: 0.0` and `exploitAvailable: false`. Egeria cannot tell a primitive that was left out of a merge update from one set to 0 / false, so it stores the default over the current value. So any merge update of a security annotation resets the CVSS score, the exploit and patch flags, and the posture counts it doesn't mention. This raises the question above from "an unset value reads as 0" to "an update loses data". **pyegeria workaround (step 4 PR):** Dr.Egeria's `SecurityFindingProcessor` copies those fields' current values into every merge update unless the command sets them (verified live: the same update then kept 8.6 / true). Callers of `DataDiscovery.update_annotation` must do the same until Egeria changes. - -**pyegeria workaround (PR #495, until then):** the security report rows (`DataDiscovery._security_row`, behind the `Security-Findings`, `Penetration-Test-Findings` and `Security-Posture` report specs) leave a value blank where a 0 or false would read as an answer nobody gave: -- a CVSS score of 0.0 is shown only when the severity is "None"; -- `exploitAvailable` and `patchAvailable` are shown only when true; -- 0 resources in scope means not recorded, so both resource counts are blank. - -`newFindingCount` / `resolvedFindingCount` of 0 cannot be told apart from unrecorded on the client at all, and the `Security-Posture` description says so. Once Egeria returns `null` for unset values, the workaround can go, apart from the "None" severity rule, which stays true either way. - ---- - ### ISSUE-147: Every valid-metadata clear call answers OPEN-METADATA-400-011 "not found", even after a successful delete; clear-map-value deletes nothing; clear-map-name leaves the map's value set behind **Layer:** Egeria Server (`ValidMetadataValueHandler`) · **Status:** open · **Found:** 2026-10-13. The first live run of the Valid Metadata commands showed it (the error text wasn't captured then). Re-tested the same day with the full responses captured. diff --git a/PYEGERIA_ISSUES.md b/PYEGERIA_ISSUES.md index d2561a9d..edf2b128 100644 --- a/PYEGERIA_ISSUES.md +++ b/PYEGERIA_ISSUES.md @@ -148,12 +148,13 @@ content was changed. The numbered section list above describes the # Open Issues -**At a glance** — 7 open pyegeria items, 7 awaiting re-test, 8 open Egeria Server issues. +**At a glance** — 14 open pyegeria items, 7 awaiting re-test, 5 open Egeria Server issues (re-sorted 2026-10-28). **Egeria team note, 2026-10-18 (Mandy Chessell).** Saved verbatim at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`. Each entry below has a dated -"Egeria team, 2026-10-18" paragraph under its header; entries have not moved -between sections yet, so the lists here still show the 2026-10-12 layout. +"Egeria team, 2026-10-18" paragraph under its header. On 2026-10-28 entries +were re-sorted: fixes released since 6.2.2 moved to the Appendix, and the +re-test candidates moved into *Awaiting re-test* below the open sections. - **Fixed on Egeria's `oak2026` branch, pending merge to `main`:** ISSUE-147, 143, 130, 132, 128, 38. Re-test once it is merged and a platform is built from `main`. @@ -170,31 +171,38 @@ between sections yet, so the lists here still show the 2026-10-12 layout. ISSUE-155, a ranked list for the owner to pick from. **Open pyegeria items** -- ISSUE-150: Dr.Egeria commands for an Analytic Function's Egeria declaration: a GovernanceActionType Resource Explorer runs as an engine host, plus a synchronous path that bypasses Egeria (direction agreed, not built) +- ISSUE-85: No REST endpoint exists to create a `SolutionPort` element — only attach/detach/delegation relationships are exposed +- ISSUE-159: no archive wrappers on the project or collection managers, and no restore wrapper at all, although Egeria offers both +- ISSUE-158: `GOV_DEF_PROPERTIES_LIST` lacks `DataQualityRuleProperties`, and Dr.Egeria's governance create fallback silently drops every type-specific attribute +- ISSUE-157: Dr.Egeria `Create Service Level Objective` sends `entitlements`/`obligations`/`restrictions` as empty lists; Egeria types them as maps +- ISSUE-156: `AutomatedCuration.add_catalog_target` posts a flat `"class": "CatalogTargetRequestBody"`, which Egeria does not have; the endpoint takes `NewRelationshipRequestBody` +- ISSUE-155: pyegeria gaps against Egeria `main` since 6.2 (omvs_audit + source diff, V6.2..05a5e982) +- ISSUE-154: pyegeria follow-ons from the Egeria team's 2026-10-18 note: renamed property classes, `cascadedDelete`, the `pushDown` default, new `DataValueAssignment`/`CrowdSourcingRole` values +- ISSUE-152: five update methods pass the body into `_async_update_element_body_request`'s `prop` slot, so nothing is sent +- ISSUE-150: Dr.Egeria commands for an Analytic Function's Egeria declaration: a GovernanceActionType that Resource Explorer runs as an engine host, plus a synchronous path that bypasses Egeria - ISSUE-146: Deleting a journal entry fails with OMAG-REPOSITORY-HANDLER-400-008, and pyegeria's `remove_note` reaches the delete only through the generic Asset Maker service - ISSUE-132: `DeleteElementRequestBody` serializes the cascade flag as `cascadeDelete`; Egeria's field is `cascadedDelete` and unknown fields are ignored, so no pyegeria cascade delete has ever cascaded - ISSUE-127: `CollectionManager.delete_collection(cascade=True)` silently never sends `cascadeDelete` -- same shape as ISSUE-62 -- ISSUE-85: No REST endpoint exists to create a `SolutionPort` element — only attach/detach/delegation relationships are exposed +- ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` +- ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55) **Awaiting re-test** - ISSUE-125: `deleteEnumDef` answers 500 "unknown TypeDef" for an enum the server lists -- an enum added through the API cannot be removed - ISSUE-124: `updateTypeDef` (Metadata Expert `open-metadata-types/update`) can never succeed -- the server builds the patch without `updatedBy`, which the repository services require - ISSUE-117: Cascade delete that takes the soft-delete (Memento) path fails partway with `OMAG-REPOSITORY-HANDLER-400-010` unless `forLineage=true` — leaves the asset live and an anchored element already soft-deleted - ISSUE-112: `AutomatedCurationRESTServices.saveClientSideSecret`/`deleteClientSideSecret` return a plain success `VoidResponse` when the resolved connector isn't a `YAMLSecretsFileConnector` — no error, no write, no indication anything was skipped -- ISSUE-95: No catalog template registered for the "Apache Kafka Server" technology type — `Create Kafka Server Element` (Asset Maker) fails with a 400 on `qs-view-server` - ISSUE-90: [fixed?] `qs-engine-host` retries `startMissedEngineActions` forever when one incomplete engine action's anchor is unreadable by the engine-host user +- ISSUE-95: No catalog template registered for the "Apache Kafka Server" technology type — `Create Kafka Server Element` (Asset Maker) fails with a 400 on `qs-view-server` - ISSUE-89: No configurable bearer-token lifetime in `application.properties` — platform tokens are fixed at 3600 s, and the signing key is random per restart -- ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` -- ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55) **Open Egeria Server issues** -- ISSUE-166: Security annotation properties are Java primitives, so an unset CVSS score, exploit/patch flag or posture count reads back as 0 / false - ISSUE-147: Every valid-metadata clear call answers OPEN-METADATA-400-011 "not found", even after a successful delete; clear-map-value deletes nothing; clear-map-name leaves the map's value set behind - ISSUE-143: `includeOnlyRelationships: ["PersonRoleAppointment"]` returns no relationships for a Person, although the unfiltered read has its role appointments - ISSUE-130: The YAML secrets store's user directory caches a user forever, so editing an existing user needs a platform restart - ISSUE-128: Retention cannot be set on Egeria 6.2: the server registers the Jackson subtype as `RetentionProperties.class` (typo), and pyegeria only allows `RetentionClassificationProperties` - ISSUE-108: newly created relationships stay invisible to related-element queries for up to ~20 minutes after creation + --- ## Open pyegeria items (including follow-ons blocked on an Egeria fix) @@ -275,745 +283,1204 @@ they're holding real port materialization until this exists server-side --- -### ISSUE-153: engine-action claim/status/completion calls still use Automated Curation paths that Egeria moved to the Governance Engine view service +### ISSUE-159: no archive wrappers on the project or collection managers, and no restore wrapper at all, although Egeria offers both -**Found 2026-10-16 by the Resource Explorer operations session, after the quickstart was redeployed on Egeria main (85d6c48a, includes odpi/egeria#9371 "Add Governance Engine API").** +**Layer:** Pyegeria · **Status:** open, logged only (no fix without the owner's word) · **Found:** 2026-10-20, raised by the Resource Explorer expansion-architecture session: RE's investigation-delete ruling needs a Project archived through the project manager and restored by GUID. Facts confirmed here against Egeria `origin/main` 05a5e982 and pyegeria upstream `main`, read-only. -- Egeria #9371 removed these from `AutomatedCurationResource` and added them to the new `GovernanceEngineResource` (`/servers/{server}/api/open-metadata/governance-engine/...`): `GET /governance-engines/{guid}/engine-actions/approved` (new), `POST /engine-actions/{guid}/claim`, `GET /governance-engines/{guid}/engine-actions/active-claimed`, `POST /engine-actions/{guid}/status/update`, `POST /engine-actions/action-targets/update`, `POST /engine-actions/{guid}/completion-status`. `GET /engine-actions/active` stays on Automated Curation (urlMarker form). #9371 also adds `POST /governance-action-process-instances/{guid}/cancel` and makes cancel skip finished engine actions. -- pyegeria v6.2.4 still builds `{ref_curation_command_base}/governance-engines/{guid}/engine-actions/active-claimed` (automated_curation.py:2614), and the same base for claim, status update and completion. Live result on the redeployed platform: the old active-claimed path returns 404. RE's engine host completion pass has failed every minute since about 15:44, and a claim would fail the same way. -- **Also blocking, and not pyegeria:** on the redeployed quickstart, the new paths return 404 too (`/governance-engine/governance-engines/{guid}/engine-actions/active-claimed` and `.../approved`). `qs-view-server` does not have the new Governance Engine view service configured yet, so that's a quickstart config change for egeria-workspaces. -- **Needed in pyegeria:** a `GovernanceEngine` client (or methods moved off `AutomatedCuration`) for approved, claim, active-claimed, status update, action-targets update and completion-status on the new base, plus `cancel_governance_action_process`. Keep `get_active_engine_actions` on Automated Curation. -- Status: FIXED on branch fix/issue-153-governance-engine (9d412207, signed, pushed to dwolfson/egeria-python; owner approved 2026-10-16). New GovernanceEngine client (6 calls incl. get_approved_engine_actions), AutomatedCuration.cancel_governance_action_process, five deprecated AutomatedCuration wrappers with a one-time 404 fallback to the old path. 20 micro-tests (whole file fails to collect on upstream/main). Audit: governance-engine OK=6, automated-curation OK=9. Not yet released; not verified live (qs-view-server lacks the service). +**Egeria:** `OpenMetadataAPIGenericHandler.java` (common-services/generic-handlers): +- `archiveBeanInRepository` (`:3289`) classifies the element Memento, renames each affected Referenceable's qualifiedName to `_archivedOn_` (marker `archivedQualifiedNameMarker = "_archivedOn_"`, `:51`), and records one `lastOperationId` per operation. +- `restoreBeanInRepository` (`:4170`) restores an element and everything the same delete or archive changed, given any GUID that operation touched. + +The OMF store's delete (`OpenMetadataStoreRESTServices`, around `:4118`) also routes `deleteMethod: ARCHIVE` to `archiveMetadataElementInStore`, and resolves `LOOK_FOR_LINEAGE` to ARCHIVE when lineage relationships exist. + +**pyegeria:** +- Archive exists only as `MetadataExpert.archive_metadata_element` (`metadata-expert/metadata-elements/{guid}/archive`). +- There is no `archive_project`, `archive_collection` or other manager-level archive. +- There is no `restore_*` method anywhere. ISSUE-155 item 5 already lists Metadata Expert `metadata-elements/{guid}/restore` and `related-elements/{guid}/restore` as missing. + +**Possible route today (unverified):** `ProjectManager.delete_project(guid, body={"class": "DeleteElementRequestBody", "deleteMethod": "ARCHIVE"})`. pyegeria passes `deleteMethod` through, but whether the project-manager delete path honours ARCHIVE as the OMF store delete does was not traced. + +**Proposed (not made):** +1. ~~Add `MetadataExpert.restore_metadata_element` / `restore_related_elements` (and undo), per ISSUE-155 item 5.~~ Done 2026-10-23 (branch `fix/role-appointment-class-names`). Items 2 and 3 are still open. +2. Add `ProjectManager.archive_project` and `CollectionManager.archive_collection`. These can be thin wrappers that send `deleteMethod: ARCHIVE` once that path is confirmed live, or call Metadata Expert archive. +3. Live-test archive then restore of a throwaway Project, including the `_archivedOn_` rename and its reversal. --- -### ISSUE-165: 19 SDK methods called endpoints Egeria doesn't serve: 13 removed upstream, 6 never existed (2 since repointed) +### ISSUE-158: `GOV_DEF_PROPERTIES_LIST` lacks `DataQualityRuleProperties`, and Dr.Egeria's governance create fallback silently drops every type-specific attribute -**Layer:** Pyegeria · **Status:** resolved: replacements implemented as deprecated wrappers and verified live (#485); `get_platform_organization` repointed (2026-10-24) · **Found:** 2026-10-23 by `scripts/omvs_audit.py` JAVA-MISSING (ISSUE-164). +**Layer:** Pyegeria · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, raised by the TestGen evaluation session while drafting a DataQualityRule convenience-API spec (`~/localGit/egeria-v6/wt-testgen-notes/packages/resource-explorer/docs/design-notes/SPEC-PYEGERIA-DATA-QUALITY-RULE.md`, uncommitted, pending owner review). Both points confirmed in source here, read-only; not run live. -**Validation.** Each URL was checked three ways: -1. **No Spring mapping** in Egeria `main` 4486d1a7f1 (2026-10-23), the checkout `omvs_audit.py` reads. -2. **Absent from the live platform's OpenAPI doc** (`GET https://localhost:9443/v3/api-docs`, which `omvs_audit.py --live` reads; it marks these rows "[LIVE: absent, so a confirmed 404]"). -3. **Git history:** `git log -S` on the path fragment, to find the commit that removed it, or to show it was never in Egeria's Java. -4. **Not moved to another module** (owner's question): the same path tail is not served under any other view service, in Java or live. The mappings added by each removing commit show no renamed replacement, with one exception. `d299b120f4` moved the governance-action-process **step-link** endpoints to Action Author; pyegeria already calls them there (`action_author.py`, `ActionProcessStepLinkProcessor`). Process search was not re-added anywhere. The audit now tags a JAVA-MISSING row "MOVED?" when the tail is served elsewhere; there are 0 such rows today. +1. **Whitelist gap.** `GOV_DEF_PROPERTIES_LIST` (`pyegeria/omvs/governance_officer.py:37`) is the client-side property-class whitelist for `create_governance_definition` (`:426`) and `update_governance_definition` (`:955`). It has no `DataQualityRuleProperties`, although Egeria `main` has that class (`frameworks/.../properties/governance/DataQualityRuleProperties.java`). **Corrected 2026-10-19:** the list is enforced only on update. `validate_new_element_request` (`pyegeria/core/_server_client.py:6115`) has its class check commented out, so a create with `DataQualityRuleProperties` reaches Egeria unchanged. `validate_update_element_request` (`:6252`) does check a dict body's class against the list, so `update_governance_definition` refuses a DataQualityRule body client-side ("unexpected property class name"). No Dr.Egeria command creates DataQualityRules today. **Proposed:** add `DataQualityRuleProperties`, and drop the list's duplicates (`RegulationProperties`, `GovernanceControlProperties` and `GovernanceApproachProperties` each appear twice). Better still, check the list against Egeria's `GovernanceDefinitionProperties` `@JsonSubTypes` so it can't drift again. +2. **Silent fallback.** In `GovernanceProcessor`'s create path (`md_processing/v2/governance.py:203-221`), any non-SecurityGroup create that Egeria refuses with `OMAG-COMMON-400-029` (property class not accepted for `createGovernanceDefinition`) is retried with `_to_base_governance_definition_properties` (`:96`). That keeps only the cross-type fields (qualifiedName, displayName, description, category, identifier, contentStatus, versionIdentifier, effective dates, additional/extended properties). The element is created, but every type-specific attribute is dropped (e.g. a DataQualityRule's quality fields, a metric's `measurement`/`target`, a control's `implementationDescription`). The only sign is a `logger.warning`, and Dr.Egeria reports success. **Proposed:** remove the generic fallback, or keep it only for the server-compat case it was written for. Otherwise make it fail visibly, naming the dropped fields in the command's result. Live-test which current types, if any, still hit 400-029 on Egeria `main` before deciding. -Three other JAVA-MISSING rows are served live, so they are Java-scan false positives and are excluded: `get_classified_elements_by` (runtime `by-{}` segment), `get_search_keyword_by_guid` and `new_subscription_type`. +--- -**Removed from Egeria:** +### ISSUE-157: Dr.Egeria `Create Service Level Objective` sends `entitlements`/`obligations`/`restrictions` as empty lists; Egeria types them as maps -| SDK method (file:line) | Calls | Removed by Egeria | Replacement in Egeria now | -|---|---|---|---| -| `data_discovery`: `create/update/delete_analysis_report`, `get_analysis_reports_by_name`, `find_analysis_reports`, `get_analysis_report_by_guid`, `link/detach_analysis_report_to/from_asset` (`data_discovery.py:1692-1773`, 8 methods) | `data-discovery/analysis-reports/...` | `e17cc09d44`, 2025-08-04 "Radical simplification of platform" | None as a CRUD API. Data Discovery now serves only `survey-reports/{g}/new-annotations[/{g}/attach\|detach]`; survey reports are produced by survey engines. The `AnalysisReportProperties` class these methods use doesn't exist either. | -| `classification_explorer.add_element_to_subject_area` / `remove_element_from_subject_area` (`classification_explorer.py:12293/12405`) | `classification-explorer/elements/{g}/subject-area-member[/remove]` | `6a49a3e23f`, 2026-01-09, which also removed `elements/by-subject-area-membership` | Not confirmed. Likely collection membership on the subject area (`{urlMarker}/collections/{g}/members/{g}/attach\|detach`, served by the generic CollectionManager resource); ask the Egeria team. `SubjectAreaMemberProperties` doesn't exist either. | -| `governance_officer.find_governance_action_processes` (`governance_officer.py:2310`) | `governance-officer/governance-action-processes/by-search-string` | `d299b120f4`, 2025-10-16, which removed all of Governance Officer's process create/update/publish/withdraw/remove/search endpoints | Not confirmed; a generic search by type `GovernanceActionProcess` is the likely route. `initiate` and `cancel` live on Automated Curation / Governance Engine (ISSUE-153). | -| `platform_services.get_platform_organization` (`platform_services.py:1083`) | `GET platform-services/server-platform/organization-name` | `816076ab50`, 2026-06-05 "Improve OMAGServerPlatform properties…" | **Fixed 2026-10-24:** Mandy Chessell confirmed the value is now the `organizationName` field of `GET /api/public/app/info`, and the method reads it from there. | +**Fixed 2026-10-19 (owner approved) on branch `fix/issue-156-157-catalog-target-slo` (956b199c, off upstream main 770dfe31; not pushed, not released).** `set_gov_prop_body` now defaults the three fields to `None`. **Cause confirmed live** on Egeria `main` 05a5e982: the old code got a 400 `com.fasterxml.jackson.databind.exc.MismatchedInputException`, "Cannot deserialize value of type `java.util.LinkedHashMap` from Array value ... ServiceLevelObjectiveProperties[\"entitlements\"]", and created nothing. The fixed code created `ServiceLevelObjective::pyeg_rt_slo` (`6258d428`) with `implementationDescription` intact; it was deleted and verified gone by GUID and by name. Micro-tests: `tests/micro-tests/test_issue_156_157_catalog_target_slo.py`; the full micro-test suite (1,394) passes in a clean python:3.13-slim container. -**Never in Egeria's Java (the SDK URL was a guess):** +**Layer:** Pyegeria (Dr.Egeria) · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, reported by the Egeria team against pyegeria 6.2.5: the command "sends a field the platform rejects", so their hourly refreshes use the content pack's existing within-an-hour SLO instead. The cause below is from reading source; it was not reproduced live, since a wrong guess would create an SLO on the shared platform. The Egeria team's error text would confirm it. -| SDK method (file:line) | Calls | What Egeria has | -|---|---|---| -| `data_designer.create_data_classification`, `update_data_classification`, `link_data_classification_to_data_field`, `detach_data_classification_from_data_field` (`data_designer.py:3298/3465/3601/3709`) | `data-designer/data-classifications/...` | Only `data-classes/{g}/nested-data-classes/{g}/attach\|detach`; there is no "data classification" element API. The `DataFieldClassificationProperties` class doesn't exist either. | -| `subject_area.link_subject_area_hierarchy` / `detach_subject_area_hierarchy` (`subject_area.py:408/495`) | `subject-area/collections/{g}/collection-hierarchies/{g}/attach\|detach` | `subject-area/subject-areas/{parent}/subject-area-hierarchies/{nested}/attach\|detach`. **Fixed 2026-10-23** (owner approved): repointed. The same module already had `link_subject_areas` / `detach_subject_areas` on the correct URLs, so these were broken duplicates; both names are kept for existing callers. | -| `governance_officer.get_gov_def_in_context` (`governance_officer.py:2796`) | `governance-officer/governance-definitions/{g}/retrieve-context` | Only `governance-definitions/{g}/retrieve` (plus update/delete and the peer/supporting links). | +**Likely cause:** `set_gov_prop_body` (`md_processing/md_processing_utils/common_md_utils.py:588`) always sets `entitlements`, `obligations` and `restrictions` to `attributes...get('value', [])`, which is `[]` when the command has no such attributes. `Service Level Objective` is in `GOVERNANCE_CONTROLS` (`common_md_utils.py:50`), so `update_gov_body_for_type` takes the controls branch, adds `implementationDescription`, and leaves the three `[]` values. Only the `TermsAndConditions`/`CertificationType`/`LicenseType` branch replaces them with maps or `None`. `body_slimmer` keeps empty lists, so `[]` is sent. In Egeria `main` (05a5e982), `ServiceLevelObjectiveProperties extends TermsAndConditionsProperties`, whose `entitlements`/`restrictions`/`obligations` are `Map`. Jackson cannot read a JSON array into a map, so the request is refused. Other governance types do not have these fields, so they silently ignore them (`ignoreUnknown=true`), which is why only SLO fails. -**Known consumer:** Dr.Egeria's `View Governance Definition Context` command (`GovernanceContextProcessor`, `md_processing/v2/governance.py:775`, registered in `dr_egeria.py:207`) calls `get_gov_def_in_context`, so that command 404s too. No other Dr.Egeria or CLI code calls these methods. +**Proposed fix (not made):** in `set_gov_prop_body`, default the three to `None` rather than `[]`, or set them only in the TermsAndConditions branch. Route `ServiceLevelObjective` through that branch (it is a TermsAndConditions subtype, not a plain control), keeping `implementationDescription`. Optionally add `Entitlements`/`Restrictions`/`Obligations` (dict-valued) to the SLO command through the Spec Editor API. Then live-test one `Create Service Level Objective` on a throwaway. -**Every call above returns a 404 today.** Before ISSUE-164, pyegeria reported a 404 as "unable to connect to the Egeria platform", so a caller could not tell "this API is gone" from "the server is down". +--- -**Egeria team's answer (Mandy Chessell, 2026-10-23), and what pyegeria now does** (branch `fix/issue-165-replacements`). Each obsolete method is now a deprecated wrapper (`DeprecationWarning`) that delegates to the replacement, following the ISSUE-153 pattern; removal comes in a later release: +### ISSUE-156: `AutomatedCuration.add_catalog_target` posts a flat `"class": "CatalogTargetRequestBody"`, which Egeria does not have; the endpoint takes `NewRelationshipRequestBody` -| Obsolete | Why (Mandy) | Now delegates to | -|---|---|---| -| analysis-report create/update/delete/get/find | SurveyReport now inherits from Report, so it is an Asset | `AssetMaker.create_asset(["SurveyReportProperties"])` / `update_asset` / `delete_asset` / `get_asset_by_guid` / `get_assets_by_name` / `find_assets`, with type `SurveyReport`. A body naming `AnalysisReportProperties` is rewritten to `SurveyReportProperties`. | -| `link/detach_analysis_report_to/from_asset` | use the ReportSubject relationship, `/elements/{subjectGUID}/reports/{reportGUID}/attach` | `AssetMaker.link_report_subject` / `unlink_report_subject(asset_guid, report_guid)` | -| `add/remove_element_to/from_subject_area` | SubjectArea now inherits from Collection; use collection membership | `CollectionManager.add_to_collection` / `remove_from_collection(subject_area_guid, element_guid)`. The old methods never took the subject area, so they gain an optional `subject_area_guid`; without it they raise an error naming the replacement. | -| `find_governance_action_processes` | GovernanceActionProcess now inherits from GovernanceDefinition | `find_governance_definitions(..., metadata_element_type="GovernanceActionProcess")` | -| `get_gov_def_in_context` | no longer needed: OpenMetadataRootElement returns the context by default | `get_governance_definition_by_guid`. Dr.Egeria's `View Governance Definition Context` calls it directly. | -| data-classification create/update/link/detach | "probably the old data class operations", now data value specifications (the common root of DataClass and DataGrain); **Mandy was not certain** | `create/update_data_value_specification` (a body naming `DataClassificationProperties` is rewritten to `DataClassProperties`) and `link/detach_data_value_assignment(data_field_guid, spec_guid)` (`DataValueAssignmentProperties`). A best-effort mapping. | +**Fixed 2026-10-19 (owner approved) on branch `fix/issue-156-157-catalog-target-slo` (956b199c; not pushed, not released).** `add_catalog_target`/`update_catalog_target` keep their signatures but now send `NewRelationshipRequestBody`/`UpdateRelationshipRequestBody` with `CatalogTargetProperties` through the shared relationship helpers. `template_properties` now maps to Egeria's `templates` field; the old body had misnamed it `templateProperties`. **Live on Egeria `main` 05a5e982** (two runs, throwaway undeployed IntegrationConnector `pyeg_rt_ic` and a Collection target, never linked to an integration group): the old flat body was refused; the fixed add returned a relationship GUID and read back name/connection/config; update read back `pyeg_rt_target_v2` and `FROM_THIRD_PARTY` with `connectionName` kept; remove made the relationship unreadable; all throwaways verified gone. The `hey_egeria` `add_catalog_target` command uses this method, so it is fixed too. -`get_platform_organization` (removed in `816076ab50`): Mandy confirmed on 2026-10-24 that the same value is returned by `GET /api/public/app/info`. The method now returns that response's `organizationName`, not a deprecated wrapper, because its signature and meaning are unchanged. That response has no `relatedHTTPCode` envelope, so it is requested with `is_json=False`. Verified live: it returns `'Coco Pharmaceuticals'` on quickstart 9443. Tests: `tests/micro-tests/test_issue165_replacements.py` (9). +**Layer:** Pyegeria · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, reported by the Egeria team against pyegeria 6.2.5: `add_catalog_target` "sends a request format the platform rejects"; their notebook now works around it. Confirmed from source; not re-run live. -**Live on quickstart 9443, 2026-10-24** (Egeria 81beb8e5 after the reset; one peer-cleared run as `peterprofile` with throwaway `pyeg-test::issue165::` elements, all 5 deleted and read back as NotFound by GUID, and the prefix find was empty afterwards). Each wrapper reached its replacement and the replacement had the intended effect: -- `find_governance_action_processes("*")` returned 100 GovernanceActionProcess-family elements (GovernanceActionProcess plus its Provisioning/Deleting/Subscribing subtypes). -- `get_gov_def_in_context` returned the same element as `get_governance_definition_by_guid`, including `relatedActions`, `associatedGovernanceActions` and `subscribers`, which confirms Mandy's point that the context now comes by default. -- `add_element_to_subject_area` without `subject_area_guid` raised the guidance error. With it, the term was listed in the subject area's collection members; after `remove_element_from_subject_area` the subject area had no members. -- `create_analysis_report` created a `SurveyReport`; `get_analysis_report_by_guid` and `find_analysis_reports` returned it. After `link_analysis_report_to_asset(report, term)` the report showed the term under `reportSubjects`; after the detach it didn't. -- `create_data_classification` created a `DataClass`. After `link_data_classification_to_data_field` the field showed it under `assignedDataValueSpecifications`; after the detach it didn't. +**What:** pyegeria has two `add_catalog_target` methods for the same URL, `.../integration-connectors/{connectorGUID}/catalog-targets/{elementGUID}`: +- `AutomatedCuration.add_catalog_target(integ_connector_guid, metadata_element_guid, catalog_target_name, connection_name=..., metadata_src_qual_name=..., config_properties=..., template_properties=..., permitted_sync=..., delete_method=...)` (`pyegeria/omvs/automated_curation.py:4120-4190`) builds `{"class": "CatalogTargetRequestBody", "catalogTargetName": ..., "connectionName": ..., ...}` with the properties at the top level. **Broken:** no `CatalogTargetRequestBody` class exists in Egeria `main`. +- `AssetMaker.add_catalog_target(integration_connector_guid, metadata_element_guid, body)` (`pyegeria/omvs/asset_maker.py:1095`) sends `NewRelationshipRequestBody` with `properties.class = "CatalogTargetProperties"`. **Correct.** -**Proposed (not made):** -1. ~~Repoint `link/detach_subject_area_hierarchy`~~ (done). -2. ~~Mark the obsolete methods deprecated~~ (done, as delegating wrappers rather than errors, per the Egeria team's answer). -3. ~~Ask the Egeria team for the intended replacements~~ (answered, except the platform organization name). +Egeria (`AssetMakerResource.addCatalogTarget`, served under every generic URL marker including `automated-curation`) takes `@RequestBody(required=false) NewRelationshipRequestBody`. `CatalogTargetProperties` has `catalogTargetName`, `metadataCollectionQualifiedName`, `metadataSourceQualifiedName`, `connectionName`, `configurationProperties`, `templates`, `permittedSynchronization` and `deleteMethod`. The `hey_egeria` CLI command `add_catalog_target` (`commands/ops/gov_server_actions.py:45`, registered in `commands/cli/egeria.py` and `egeria_ops.py`) uses the broken one. + +**Proposed fix (not made):** keep `AutomatedCuration.add_catalog_target`'s signature but build `{"class": "NewRelationshipRequestBody", "properties": {"class": "CatalogTargetProperties", "catalogTargetName": ..., "connectionName": ..., "metadataSourceQualifiedName": ..., "configurationProperties": ..., "templates": ..., "permittedSynchronization": ..., "deleteMethod": ...}}` and send it through `_async_new_relationship_request`, as AssetMaker does. Add a micro-test asserting the body shape. `AutomatedCuration._async_update_catalog_target` (`automated_curation.py:4248`, body at `:4301`) has the same flat `CatalogTargetRequestBody` shape against `catalog-targets/{relationshipGUID}/update`. Egeria's `updateCatalogTarget` takes `UpdateRelationshipRequestBody`, so it needs the same fix. --- -### ISSUE-164: `omvs_audit` now checks against Egeria's Java source. The first sweep finds 6 always-400 calls, 4 more 400s with a body, and 24 endpoints with no Spring mapping +### ISSUE-155: pyegeria gaps against Egeria `main` since 6.2 (omvs_audit + source diff, V6.2..05a5e982) -**Layer:** Pyegeria (+ Dr.Egeria, CLI) · **Status:** audit improved, and the findings were fixed on branch `fix/role-appointment-class-names` (owner approved 2026-10-23; not yet released), except the items under *Not fixed* at the end · **Found:** 2026-10-23, sweep requested by the owner before the next release, after ISSUE-163. +**Layer:** Pyegeria · **Status:** open, proposed list for the owner to pick from; nothing changed · **Found:** 2026-10-18, Resource Explorer ops item 7. File-only audit, no Egeria calls. -**Fixed 2026-10-23, in the owner's order:** -1. **Anchor scope and find keys** (3bb233c1): `anchorScopeGUIDs` serialisation on the New/Relationship/Template models; the `_async_find_request` keys. -2. **Always-400 deletes and wrong class names** (b8b4567b): `remove_is_abstract_concept`, `remove_activity_description`, `detach_cohort_member`, `remove_search_keyword`, `initiate_engine_action` (`InitiateEngineActionRequestBody`, `actionSourceGUIDs`, `startDate`), `DataValueMeaningProperties`, `SolutionActorRoleProperties`, `DataProcessingPurposeProperties`, and the CLI todo typo. -3. **Model gaps and Dr.Egeria/CLI keys** (13c17e37): `ActionRequestBody` anchor fields and `newActionTargets`; `TemplateRequestBody` `parentRelationshipProperties` and `replacementClassifications`; `endsWith`; Dr.Egeria/CLI `anchorScopeGUIDs`; the CLI update `properties`. +**How:** `scripts/omvs_audit.py --quiet --http-dir ` was run twice against this branch's `pyegeria/omvs`: once with the `.http` files from Egeria tag `V6.2` (bc08cf55a8, 2026-10-06), once with those from `origin/main` (05a5e9822e, 2026-10-18). Each was restricted to the collections `pyegeria/http-clients/` already carries, plus the new `Egeria-api-governance-engine.http`. Totals: V6.2 OK=965, mismatch 4, missing 3. Main OK=959, mismatch 5, missing 15. The `.http` files lag the Java, so the view-service `*Resource.java` and request-DTO diffs from V6.2 to `main` were also read, for PRs #9365-#9374 and #9321. The checkouts were not touched; the files were read with `git show`. Ranked: risk to existing callers first, then new capability. -Tests: `tests/micro-tests/test_issue164_silent_field_drops.py` (15 tests). +1. **Engine-action calls moved to the Governance Engine view service (PR #9373).** Automated Curation no longer has claim, status update, action-targets update, completion-status or active-claimed. On `main` the audit reports them ELSEWHERE, plus `getApprovedEngineActions` and `cancelGovernanceActionProcess` MISSING. **Already fixed on branch `fix/issue-153-governance-engine` (ISSUE-153), not yet merged or released.** The only action is to merge and release that branch. +2. **`cascadeDelete` alias (PR #9365, `DeleteOptions.setCascadedDelete` now has `@JsonAlias("cascadeDelete")`).** pyegeria's cascade flag now reaches the server, so ISSUE-154 item 1 applies: switch to `cascadedDelete` and re-test cascades. +3. **Zones fail closed (PR #9367).** A GovernanceZone with no security access control now admits no-one; before, it was ignored. This is not a REST change. The new `MISSING_ZONE_ACCESS_CONTROL` exception type is raised by the platform cataloguer, not on a REST path. Callers will see 403s or empty results for elements whose only zones are unsecured. **pyegeria:** no code change. Expect more 403s on quickstart content, as with the digital-products zone (2026-10-14). The 403 error message already reads clearly since ISSUE-142. +4. **`createMetadataElementFromTemplate` body class.** The `.http` now shows `OpenMetadataTemplateRequestBody`. The Java parameter type was already that at V6.2, so only the sample changed. pyegeria sends its `TemplateRequestBody` model (`metadata_expert.py:159-207`, `models.py:526`). That model has no `typeName`, `replacementClassifications` or `parentRelationshipProperties`, so the PyegeriaModel `extra='ignore'` rule silently drops those fields if a caller passes them. **Proposed:** add an `OpenMetadataTemplateRequestBody` model with those fields and use it here. Medium value: it is the only way to create a subtype from a template through Metadata Expert. **Partly done 2026-10-23:** pyegeria's `TemplateRequestBody` now sends `replacementClassifications` and `parentRelationshipProperties` (ISSUE-164). `typeName` (on `OpenMetadataTemplateRequestBody` only) is still open. +5. **Undo and restore (PR #9365/#9373, new).** Metadata Expert `POST /metadata-elements/{guid}/undo` restores the previous version and fails if there is none. `/metadata-elements/{guid}/restore` reverses a delete or archive, including what that delete cascaded, but not a purge. `related-elements/{guid}/undo` and `/restore` do the same for relationships. Each takes an optional `MetadataSourceRequestBody`. **Proposed:** four methods on `MetadataExpert` via the existing `_async_metadata_source_body_request` (`_server_client.py:6395`). This pairs well with ISSUE-117 and RE's archive walk: restore is the safety net for a mistaken archive. **Done 2026-10-23 (ISSUE-164 follow-up, branch `fix/role-appointment-class-names`):** `MetadataExpert.undo_metadata_element_update`, `restore_metadata_element`, `undo_related_elements_update`, `restore_related_elements`. +6. **Usage Activity and Member Popularity Profile classifications (Classification Explorer, new, probably for the Lovelace Popularity Profiler in #9369).** Each has add (`POST /elements/{guid}/usage-activity`), `/update` and `/remove`, and the same for `/member-popularity-profile`. The bodies are New/Update/DeleteClassificationRequestBody with `UsageActivityProperties` or `MemberPopularityProfileProperties`. **Proposed:** six methods, and the property names in `element_status_fields.py` if Dr.Egeria should show them. Low priority until something writes them. **Done 2026-10-23 (same branch):** `ClassificationExplorer.add/update/clear_usage_activity` and `add/update/clear_member_popularity_profile`. +7. **`GetOptions.viewElementGUID` (new).** It identifies the element whose view a request is assembling, so related elements fetched for that view are not reported as "found" (usage tracking). **Proposed:** an optional field on the get-request models, for the Resource Browser and the explorers. Low priority. +8. **`relationshipPurpose` on RelationshipDef (PR #9365).** Type definitions now carry a relationship purpose used for anchor processing. Nothing in pyegeria reads it. **Proposed:** none now. Possibly show it in the Egeria Explorer type views later. +9. **`finalAnalysisStep` (PR #9366).** This is internal to the survey connectors (a provider/service API) and not a REST field. **pyegeria:** none. +10. **Property-class renames (PR #9365):** covered by ISSUE-154 item 3. -**Live on quickstart 9443, 2026-10-23** (three peer-cleared runs, throwaway `pyeg-test::issue164::` glossary/terms/solution role; every GUID read back as gone and the prefix find was empty after each run): -- **Anchor scope stored:** a term created with `anchorScopeGUIDs: []` reads back with `elementHeader.anchor.classificationProperties.anchorScopeGUIDs == []`; a term created without it has none. -- **Classification add/remove round trips:** AbstractConcept, ContextDefinition, ActivityDescription and DataValue (stored as `DataValueMeaning`, matching the corrected class) each appear in `elementHeader.glossaryTermKinds` after the add and are gone after the remove. Before the fix, `remove_is_abstract_concept` was a 400 on every call. -- **Solution role:** `create_solution_role` with `SolutionActorRoleProperties` created, then deleted. -- **Not verified live:** - - The find zone filter: an unzoned element matches any zone filter, and zoning a throwaway element could make it undeletable for the test user. Verified at the wire level only. - - `remove_search_keyword`: my test body was wrong (`keyword`; Egeria's field is `displayName`, as the docstring says). - - `detach_cohort_member`, `initiate_engine_action`: not exercised, by design. +**Audit noise (pre-existing, not gaps):** `getActorRoleByGUID` and `getSchemaAttributeByGUID` are reported as mismatches only because their `.http` URLs end in a stray `"}` (an Egeria `.http` typo; worth telling Egeria). `detachSolutionLinkingWire` is a false positive: Egeria has both `.../wired-to/{guid}/detach` and `.../wires/{relationshipGUID}/detach`, and pyegeria implements both (`solution_architect.py:4338`, `:4505`). `removeNote` is ISSUE-146. The three `product-catalog` MISSING rows are examples of one find call. `Egeria-valid-metadata-lists.http` exists only in the local copy, not on Egeria `main`. -**Egeria 6 element shape (for anyone reading classifications back; RE's read-backs too).** The view-service `get_*_by_guid(output_format="JSON")` calls (glossary, collection, actor manager, …) return the element with `elementHeader` and `properties`, and that header **has no `classifications` list**. The lower-level Metadata Expert / OMF shape (`get_metadata_element_by_guid`, an OpenMetadataElement) is different: it still has a top-level `classifications` list (Thursday Coordinator's read-only probe, 2026-10-23). Readers must know which shape they hold. Each classification is a named camelCase slot on `elementHeader` (`anchor`, `zoneMembership`, `retention`, `glossaryTermKinds`, `projectClassification`, `memento`, … per Java `ElementHeader`), and anything unslotted is in `elementHeader.otherClassifications`. A slot holds an `ElementClassification`, or a list of them for the `…Kinds` slots, each with `classificationName` and `classificationProperties`. Trimmed raw sample from this run: -```json -"elementHeader": { - "guid": "97581996-622d-46fe-82e0-0feaeea17132", - "anchor": {"class": "ElementClassification", "classificationName": "Anchors", - "classificationProperties": {"class": "AnchorsProperties", "anchorTypeName": "GlossaryTerm", - "anchorDomainName": "AuthoredReferenceable", - "anchorScopeGUIDs": ["057d4bd3-62bc-4591-8777-410f7528d867"]}}, - "glossaryTermKinds": [{"class": "ElementClassification", "classificationName": "AbstractConcept", "...": "..."}] -} -``` -Note that the anchor property is the plural `anchorScopeGUIDs`, a list. Code that looks for a generic `classifications` list, or for a singular `anchorScopeGUID`, finds nothing. That is how this run's first two read-backs reported false absences. +**Also worth doing:** refresh the gitignored `pyegeria/http-clients/` from Egeria `main`. The local copy predates `Egeria-api-governance-engine.http` and the classification-explorer and metadata-expert additions. -**Correction to the triage below:** `RetentionClassificationProperties` and `SecurityGroupMembershipProperties` are **not** bugs. Egeria registers them as `@JsonSubTypes` aliases (`names = {...}`): `RetentionProperties` is also accepted as `"RetentionClassificationProperties"`, and `processors.py` records a 2026-08-03 live 400 for `RetentionProperties` on an older build. Left unchanged; the audit now reads registered names and aliases. +--- -**Audit changes** (`scripts/omvs_audit.py`, docs in CLAUDE.md and `scripts/README.md`, tests in `tests/micro-tests/test_omvs_audit_java_truth.py`): -- **RENAMED matches** now get the verb/body checks too; they used to be skipped. -- **Java ground truth** (`--java-dir`, `$EGERIA_JAVA_DIR`, default `~/localGit/egeria-v6/egeria`; `--no-java` skips it). The audit indexes 1,587 Spring endpoints → `@RequestBody` type. - - **JAVA-BODY REJECTED** (defect): fires only when the declared type sits under a `@JsonTypeInfo(property="class")` base. Among request bodies that's only `DeleteRequestBody` → `Delete{Element,Relationship,Classification}RequestBody`, where a sibling class is a 400. - - **CLASS-IGNORED** (review): everywhere else Egeria's DTOs are plain `@JsonIgnoreProperties(ignoreUnknown=true)`, so `"class"` is ignored and fields map by name. - - **UNKNOWN-CLASS** (defect): any `"…Properties"`/`"…RequestBody"` string in `pyegeria/omvs`, `md_processing` or `commands` that is no Egeria class and no pyegeria class. The properties bases (`RelationshipBeanProperties`, `ClassificationBeanProperties`, …) are polymorphic, so an unknown name there is a 400. - - **JAVA-MISSING** (review): an SDK verb+path with no Spring mapping. - - **HTTP-vs-JAVA** (info): 21 `.http` entries name a body class Java doesn't accept. When Java accepts what the SDK sends, the `.http` BODY mismatch is no longer counted. -- **Regression check:** on the pre-ISSUE-163 code the audit flags all eight ISSUE-163 bugs: 5 REJECTED and 3 UNKNOWN-CLASS, the detaches via the RENAMED path. +### ISSUE-154: pyegeria follow-ons from the Egeria team's 2026-10-18 note: renamed property classes, `cascadedDelete`, the `pushDown` default, new `DataValueAssignment`/`CrowdSourcingRole` values -**Findings, current branch**, against Egeria `main` 4486d1a7f1 (2026-10-23). `python scripts/omvs_audit.py --quiet` reports java_ok=951, rejected=4, class_ignored=25, unknown_class=16, java_missing=22, field_drop=3, client_drop=13, model_gaps=27, http_vs_java=21. An earlier run against a 2026-10-06 checkout listed 24 JAVA-MISSING; `cancel_governance_action_process` and `get_approved_engine_actions` (ISSUE-153's Governance Engine paths) now resolve. +**Layer:** Pyegeria · **Status:** open, proposed, not made (owner approval needed, per this repo's policy) · **Found:** 2026-10-18, from the Egeria team's note (`~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`) and read-only greps of this checkout. -*Always fails: the SDK always sends this, so every call is a 400 or a client-side ValidationError.* +Egeria says none of its changes break existing requests: the old class names and spellings are still accepted. So nothing here is urgent, and every change should keep accepting the old names so pyegeria still works against older platforms. Ranked by risk, highest first. -| Where | What | Fix (from Java) | -|---|---|---| -| `glossary_manager.remove_is_abstract_concept` | element helper always sends `DeleteElementRequestBody`; endpoint takes `DeleteClassificationRequestBody` | `_async_delete_classification_request` | -| `runtime_manager.detach_cohort_member` | `_async_metadata_source_body_request` always sends `{"class":"MetadataSourceRequestBody"}`; endpoint takes `DeleteRelationshipRequestBody` | `_async_delete_relationship_request` | -| Dr.Egeria `Retention` classify (`md_processing/v2/curation.py:136`, `processors.py:237`) | body `class` is `RetentionClassificationProperties` | `RetentionProperties` | -| CLI todo status update (`commands/my/todo_actions.py:201`) | typo `UpdataElementRequestBody`; pyegeria's Literal validator rejects it before sending | `UpdateElementRequestBody` | -| `data_discovery` analysis-report methods (8) | `analysis-reports/...` has no mapping in `DataDiscoveryResource`; class `AnalysisReportProperties` doesn't exist | confirm whether Egeria still has the endpoint | -| `classification_explorer.add/remove_element_to/from_subject_area` | `elements/{g}/subject-area-member` has no mapping; class `SubjectAreaMemberProperties` doesn't exist | confirm | +**1. `cascadeDelete` now cascades (ISSUE-132).** Egeria accepts `cascadeDelete` as an alias for `cascadedDelete`, so any pyegeria call that sends `true` will start cascading on a platform with `oak2026`. Checked: no production caller in `pyegeria/`, `md_processing/` or `commands/` passes `True`; every `cascade`/`cascade_delete` default is `False`; `commands/tech/element_actions.py`'s `--cascade` flag (default True) is not forwarded. Only live tests pass `True`, on purpose. Proposed: serialize `DeleteElementRequestBody.cascade_delete` as `cascadedDelete` (accept both on input) and change the ~25 `"cascadeDelete"` literals (`_server_client.py` ×3, `models.py` ×7, `actor_manager.py` ×8, `metadata_expert.py`, `product_manager.py` ×2, `reference_data.py` ×2, `valid_metadata.py` ×2; several are docstring samples). Then live-test one cascade and re-test ISSUE-127. -*Fails only when a no-body default or a docstring-style body is used. With the caller's own correct body these work.* +**2. Metadata Expert count default flipped (ISSUE-38).** Egeria's `pushDown` default is now `false` (list-matching, slower); `true` is the fast repository count. `metadata_expert.py:3698` and `:4013` document omission as `pushDown=True`. `pyegeria/view/overview_metrics.py` calls both counts without `push_down`, so its statistics silently become the slower kind. Proposed: fix both docstrings; owner to choose whether `overview_metrics` sends `push_down=True` (Egeria's advice for statistics) or keeps the list-matching default (which matches its own `len(find)` fallback). -| Where | What | Fix | +**3. Renamed property classes.** Responses now carry the new names in `class`. Checked: no pyegeria response parsing compares a response's `class` with any of the renamed names. The only matches are request-side `prop=[...]` whitelists, which refuse a caller-built body that uses the *new* name: + +| Old → new | Where pyegeria uses the old name | Proposed | |---|---|---| -| `feedback_manager.remove_search_keyword` | relationship body to an element `/remove` | `_async_delete_element_request` | -| `glossary_manager.remove_activity_description` | relationship body; endpoint takes `DeleteClassificationRequestBody` | `_async_delete_classification_request` | -| `actor_manager.add_security_group_membership` default class | `SecurityGroupMembershipProperties` | `SecurityListMembershipProperties` | -| `classification_explorer.set_retention_classification` default class | `RetentionClassificationProperties` | `RetentionProperties` | -| `glossary_manager.add_is_data_value` default class | `DataValueProperties` | `DataValueMeaningProperties` | -| `solution_architect.create/update_solution_role` default class | `SolutionRoleProperties` | `SolutionActorRoleProperties` (extends `ActorRoleProperties`; the endpoint is `createActorRole`) | -| `data_designer.link_data_classification_to_data_field` | `data-classifications/.../data-fields/{g}/attach` has no mapping; `DataFieldClassificationProperties` doesn't exist. The create/update/detach data-classification methods are also unmapped | confirm | -| `GOV_DEF_PROPERTIES_LIST` (`governance_officer.py:40`) | `GovernanceProcessingPurposeProperties` | `DataProcessingPurposeProperties`; probably one of ISSUE-154's renames | +| `RetentionClassificationProperties` → `RetentionProperties` | `omvs/classification_explorer.py:8851,8878` (whitelist),`8919`; `md_processing/v2/processors.py:220-237` (comment says the opposite); `md_processing/v2/curation.py:136` | Send `RetentionProperties`, accept both in the whitelist, rewrite the comment (ISSUE-128) | +| `SecurityGroupMembershipProperties` → `SecurityListMembershipProperties` | `omvs/actor_manager.py:3602,3617` (whitelist),`3654,3703,3753` | Send the new name, accept both | +| `AssetSchemaTypeProperties` → `SchemaProperties` | not used; `schema_maker.py:822` already whitelists `SchemaProperties` | none | +| `AssociatedSecurityGroupProperties` → `AssociatedSecurityListProperties` | not used; `md_processing/v2/governance.py:620` already sends the new name | none | +| `ReferenceCodeMappingProperties` → `ReferenceCodeMappingTableProperties` | not used | none | +| `GovernanceClassificationBase` → `GovernedDataClassificationBase` | only in a description string, `commands_curation_compact.json:2256` | optional wording fix through the Spec Editor API | -*Harmless on the wire: an unknown top-level `class` that Egeria ignores.* `GovernanceActionRequestBody` (`automated_curation.initiate_engine_action`), `IntegrationServiceRequestBody` (`full_omag_server_config`), `AnyTimeRequestBody` (`metadata_expert.get_relationship_by_guid`, three Solution Architect retrieves) and `FindProperties` (`classification_explorer.retrieve_instance_for_guid`). Worth renaming for clarity only. +`tests/functional-tests/test_classification_manager.py:1233` already sends `RetentionProperties`. -*JAVA-MISSING (meaning no such URL in Egeria; it is not about classes), other entries to confirm.* `get_search_keyword_by_guid`, `find_governance_action_processes`, `get_gov_def_in_context`, `get_platform_organization`, `new_subscription_type`, and `subject_area.link/detach_subject_area_hierarchy`. The last calls `collections/{g}/collection-hierarchies`, while Java has `subject-areas/{g}/subject-area-hierarchies/{g}/attach|detach`, so it is likely a real 404. `get_classified_elements_by` (`by-{}`) is a static-analysis false positive. +**4. `DataValueAssignment` now accepts `label` and `description`.** Dr.Egeria's `Assign Data Value Specification` (`md_processing/v2/data_designer.py:1026-1037`) replaces `properties` with a fixed field list, and the `Data Value Assignment Base` bundle has no Label/Description. Proposed (feature, not a fix): add `Label` and `Description` to that bundle through the Spec Editor API, map them in the processor, run `refresh_specs`. -*CLASS-IGNORED (25), review only.* Mostly Filter-vs-Results and MetadataSource-vs-DeleteClassification pairs that are harmless. Worth a field-level look: `metadata_expert.create_metadata_element_from_template` (Java `OpenMetadataTemplateRequestBody`) and `metadata_expert.classify_metadata_element` (Java `NewOpenMetadataClassificationRequestBody`). Both Java classes use typed `ElementProperties`, so the flat SDK shape may silently lose fields; see CLAUDE.md's note on `MetadataExpert` create. Also `project_manager.create_project_task` (Java `NewAttachmentRequestBody`). +**5. `CrowdSourcingRole` values now work** (PROPOSER, REVIEWER, SUPPORTER, APPROVER, OTHER). pyegeria's `link_crowd_sourcing_contribution` (`omvs/community_matters_omvs.py:479`) passes the body through, and no Dr.Egeria command sets a role. Proposed: list the values in the docstring; nothing else. -**Field-level findings.** These calls succeed but silently ignore an option. Egeria's DTOs are `ignoreUnknown=true`, and pyegeria's models are `extra='ignore'`, so a misnamed field disappears on either side with no error. +**6. ISSUE-147's 400-011 workaround:** none exists. Nothing to drop. -*CLIENT-DROP: pyegeria's own model discards the key before sending.* +**7. View-service fixes pyegeria may have worked around.** The note lists 48 view-service defects found by Egeria's new view-services FVT. A read-only sweep of `pyegeria/`, `md_processing/` and `commands/` (2026-10-18) found almost nothing worked around, and no issues-file entry for any of them. The detach, Classification Explorer, Metadata Expert, search-option, history-order and delete/purge rows have no client-side workaround: the wrappers call the endpoint once and pass options straight through. So pyegeria users met those defects as calls that "succeeded" but did nothing, and the fixes need no pyegeria change. Candidates to review: +- **Project hierarchy:** `commands/cat/get_project_structure.py:105-116` (and `get_project_dependencies.py:107`) walk the tree with `get_linked_projects()` and keep only `ProjectHierarchy` links, instead of calling `ProjectManager.get_project_hierarchy`. Not commented as a workaround. Optional: switch to `get_project_hierarchy` after re-testing it on a platform built from `main`. +- **Term classifications with no body:** `glossary_manager.py:2686-2692` (is-question) and `4059-4062`/`4116`/`4170` (prime word, modifier, class word) always send a default body. The comment blames pyegeria's own helper (ISSUE-31), not the server, so keep them; harmless either way. +- **`linkSolutionLinkingWire` with no body:** `md_processing/v2/solution_architect.py:856-860` always sends a body and skips recording the GUID when none returns. Keep it; no change needed. +- **Asset from template:** Dr.Egeria's generic `Create Element` (`md_processing/v2/asset_maker.py:57-75`) uses Automated Curation `catalog-templates/new-element`, not Asset Maker `assets/from-template`, so it never hit the "always an Asset" bug. No change. +- **Missing connector now refused** (Security Officer, Runtime Manager): calls that used to report success with nothing done will now raise. Callers should expect the error. ISSUE-112 is the same shape for client-side secrets. +- **Delete with `forLineage`:** a delete that needs `forLineage=true` (Memento) is now refused whole instead of half-done. pyegeria's editing deletes already default to `forLineage=true` via `lineage_visible()`, so no change. -| Where | Key sent | Model/Java field | Effect | -|---|---|---|---| -| `_async_find_request` (`_server_client.py:6543`, every `find_*` without property names) | `zoneFilter`, `relationshipPageSize`, `anchorScopeGuid`, `anchorTypeName` | `governanceZoneFilter`, `relationshipsPageSize`, `anchorScopeGUID`; `anchorTypeName` is not a field at all | `governance_zone_filter`, `relationship_page_size` and `anchor_scope_guid` never take effect on any find | -| My Profile to-do / meeting / review create (`my_profile.py:1314/1431/1523`) | `isOwnAnchor` | Java `ActionRequestBody` (via `AnchorOptions`) has it; **pyegeria's `ActionRequestBody` model lacks it**, and also `anchorGUID`, `anchorScopeGUIDs` and `initialClassifications`. It names Java's `newActionTargets` `actionTargetGUIDs` | not anchored as intended | -| Dr.Egeria `common_md_utils.py:263` | `endWith` | `endsWith` | the "ends with" search option is ignored | -| Dr.Egeria `common_md_utils.py:292/323/737`, `v2/schema_maker.py:28`, `v2/asset_maker.py:57`, CLI `glossary_actions.py:123` | `anchorScopeGUID(s)`, `governanceZoneFilter` on create/update bodies | the model reads `anchorScopeGuids` (see below); create/update bodies have no zone filter | anchor scope is dropped | -| Dr.Egeria `v2/asset_maker.py:57` | `parentRelationshipProperties`, `typeName` | Java `TemplateRequestBody` has `parentRelationshipProperties` (and `replacementClassifications`); **the pyegeria model lacks both** | parent-relationship properties are dropped | -| CLI `glossary_actions.py:178` | `elementProperties` | `properties` | update has no properties | -| `validate_delete_classification_request` default (`_server_client.py:6244`) | `cascadeDelete` | not on the model, and Java spells it `cascadedDelete` | ISSUE-132 | +--- -*Model spelling that Egeria discards. Confirmed: anchor scope cannot be set through pyegeria at all.* The `NewElementRequestBody`, `NewRelationshipRequestBody` and `TemplateRequestBody` models alias `anchor_scope_guids` as `anchorScopeGuids`, while Java's `MakeAnchorOptions` reads `anchorScopeGUIDs`. A dict using the documented `"anchorScopeGUIDs"` (as in `collection_manager.py`'s docstrings) is dropped client-side. `anchorScopeGuids` / `anchor_scope_guids` are sent, then dropped server-side. Checked by `model_validate` → `model_dump_json`. This affects 47 create, 157 link and 26 from-template methods; one alias fixes it. +### ISSUE-152: five update methods pass the body into `_async_update_element_body_request`'s `prop` slot, so nothing is sent -*FIELD-DROP: the method passes it, Egeria's class lacks it.* -- `get_tech_types_for_open_metadata_type(tech_name=...)` and `get_endpoints_for_asset(name=...)` send `filter` in a `FilterRequestBody`. The endpoints declare `ResultsRequestBody`, which has no `filter`, so they return everything unfiltered. -- `find_software_capabilities(property_names=...)` sends `propertyNames` where Java declares `SearchStringRequestBody`, so the property restriction is lost. +**Layer:** Pyegeria · **Status:** open (found 2026-10-15 by Resource Explorer's ClassificationAnnotation backfill; not fixed, per this repo's policy) -**New APIs (2026-10-23, from the refreshed `.http`; ISSUE-155 items 5 and 6), live on 9443.** One peer-cleared run with throwaway `pyeg-test::newapis::` elements; all GUIDs read back as gone and the prefix find was empty. Raw dumps were analysed afterwards: -- **`add/update/clear_usage_activity`:** add stored `explicitGetCount=1, findCount=2`; update set `findCount=9`; clear removed the `elementHeader.usageActivity` slot. **Egeria behaviour:** that update also reset `explicitGetCount` to 0 despite `mergeUpdate: true`. The count fields are primitive `long`s (Java `UsageActivityProperties`), so an omitted count deserialises as 0. The docstrings say to send every count. -- **`add/update/clear_member_popularity_profile`** (on a glossary): the total and the per-member map were stored; update set `findCountTotal=4` and kept the map; clear removed it. -- **`undo_metadata_element_update`:** a term description `v2` went back to `v1`. **`undo_related_elements_update`:** a RelatedTerm description `r2` went back to `r1`. -- **`restore_related_elements`:** a removed RelatedTerm came back with its properties. -- **`restore_metadata_element`:** a soft-deleted term came back. **An archived (Memento) term also came back, with the same GUID, the Memento removed, and its qualifiedName restored** from `…t3_archivedOn_Fri Oct 23 19:29:39 GMT 2026` to the original. This answers ISSUE-159's open question: restore is not soft-delete-only. Anchor-scoped restore (anchored descendants) was not tested. A plain get of the archived term (no `forLineage`) returned HTTP 500, not 404. -- **`link/detach_subject_area_hierarchy`** (repointed, ISSUE-165): the child was linked, then gone after detach. -- **`ProductCatalog.find_digital_product_catalogs`:** a named search returned 2 results in 6.2s (`ProductManager`'s product-manager-marker version returned 13 in 41s). An unfiltered "*" search timed out at the helper's 90s. +**Fixed 2026-10-20 (owner approved) on branch `fix/issue-152-update-body-slot`, not yet merged or released.** All five call sites now pass `body=body`. `tests/micro-tests/test_issue_152_update_body_slot.py` sends a dict body and an `UpdateElementRequestBody` through each method and checks that the body reaches the HTTP layer. All 10 cases fail on the old lines and pass on the fix, and the full micro-test suite (1,404) passes in a clean python:3.13-slim container. Not live-tested. Hit again 2026-10-20 by the TestGen evaluation session (RE #791, `update_annotation`), which carries its own workaround until a release ships this. -**Not fixed (owner's call or Egeria's):** -- **19 endpoints Egeria doesn't serve**, documented in full with provenance in **ISSUE-165**. Confirmed absent from the live platform's OpenAPI doc (`omvs_audit.py --live`; GET `/v3/api-docs` on 9443) and from Egeria `main` 4486d1a7f1: the 8 Data Discovery analysis-report methods, data-classification create/update/link/detach, subject-area member add/remove, subject-area hierarchy link/detach (the SDK uses `collections/{g}/collection-hierarchies`; Egeria has `subject-areas/{g}/subject-area-hierarchies`), `find_governance_action_processes`, `get_gov_def_in_context` and `get_platform_organization`. Raise with the Egeria team, or remove from the SDK. Three other JAVA-MISSING entries are served live, so they were Java-scan false positives. -- **No server-side filter exists:** `get_tech_types_for_open_metadata_type(tech_name=)` and `get_endpoints_for_asset(name=)`, whose endpoints take `ResultsRequestBody` (no `filter`), and `find_software_capabilities(property_names=)` (`SearchStringRequestBody`). Options: filter client-side, or drop and document the argument. -- **Dr.Egeria Asset Maker template `typeName`:** Egeria's nearest field is `TemplateOptions.metadataElementSubtypeName` ("create an element of a subtype to the template type"). That isn't clearly the same as the spec's "type name of the element to create", so it needs a live check before mapping. Part of ISSUE-155 item 4 (Metadata Expert create-from-template) is covered by the `TemplateRequestBody` fields added above. -- **Harmless top-level names Egeria ignores:** `FindProperties`, `AnyTimeRequestBody` (Java: `GetRequestBody`), `IntegrationServiceRequestBody` (legacy admin-services config). -- `cascadeDelete` stays as it is: ISSUE-132, on hold for Egeria. -- **New `.http` coverage, not part of this sweep:** the refreshed collections document usage-activity, member-popularity-profile and metadata undo/restore, which pyegeria lacks (ISSUE-155 style). -- **Dead legacy path:** `pyegeria/view/md_processing_utils.py:1539` (behind the `dr_egeria_jupyter` CLI) still sends the sunset `initialStatus`, but it calls `create_controlled_glossary_term`, which no longer exists. +`_async_update_element_body_request(self, url, prop=None, body=None)` (`pyegeria/core/_server_client.py:7317`) takes the body as its **third** parameter. Five callers pass it positionally as the second, so it lands in `prop` and `body` is `None`; the call fails before any request goes out (the builder observed an AttributeError): -*MODEL-FIELD gaps (review; 27 model→Java pairs, with users, in the report).* -- Most are benign. For example, `ResultsRequestBody → GetRequestBody` drops paging on 51 `get_*_by_guid` calls, where paging means nothing. -- Status-filter models carry both `activityStatus` and `activityStatusList`; only the `…List` form exists in Java. No current caller uses the singular form in a filter (My Profile's uses are inside the nested `…Properties`, where it is a real field). -- `UpdatePropertiesRequestBody.replaceProperties` (Java `UpdateOptions` has only `mergeUpdate`) is used by three Metadata Expert updates. +- `pyegeria/omvs/data_discovery.py:299` (`_async_update_annotation`, behind `update_annotation`) +- `pyegeria/omvs/time_keeper.py:342` +- `pyegeria/omvs/schema_maker.py:131` and `:174` +- `pyegeria/omvs/subject_area.py:302` + +Checked at egeria-python `f1b06cdd`; the installed wheel in trellis has the same line in `data_discovery.py`. + +**Fix (one token each):** `await self._async_update_element_body_request(url, body=body)`. A test per method that the body reaches the HTTP layer would catch a repeat. + +**Workaround in Resource Explorer:** `scripts/backfill_classification_annotations.py` calls `_async_update_element_body_request(url, body=body)` directly, and pins this bug with a test that fails once pyegeria is fixed. --- -### ISSUE-163: Actor Manager role-appointment links send non-existent `*AppointmentProperties` classes, and five delete/detach methods send the wrong request-body class -**Fixed 2026-10-23 on branch `fix/role-appointment-class-names` (not yet released).** **Live on quickstart 9443** (two runs, throwaway `pyeg-test::role-appt::` elements, all read back as gone by GUID afterwards, and a find on the prefix returned nothing). The old person-link class was refused with a 400. All three fixed links, called with no body, read back the relationship to the profile. The old IT detach body was refused with a 400; the fixed detach made the link unreadable. The fixed `delete_actor_role` deleted its role. +### ISSUE-150: Dr.Egeria commands for an Analytic Function's Egeria declaration: a GovernanceActionType that Resource Explorer runs as an engine host, plus a synchronous path that bypasses Egeria -**Layer:** Pyegeria (+ Dr.Egeria) · **Status:** fixed on branch · **Found:** 2026-10-23, reported by the owner ("pyegeria is using the wrong class name when linking a role to a profile"). +**Layer:** Dr.Egeria (commands over existing Action Author / governance commands) · **Status:** open, direction agreed with the owner 2026-10-13 and **confirmed by Mandy 2026-10-14**, not yet built · **Found:** 2026-10-13, handed off by the Resource Explorer PR/CI session. Source: trellis `docs/extensibility-walking-skeleton-brief.md` §2c, §3a, §8. Egeria references: [Governance Action Processes (0462)](https://egeria-project.org/types/4/0462-Governance-Action-Processes/), [Survey Action Engine](https://egeria-project.org/concepts/survey-action-engine/). -**Root cause:** each method below either passes a properties class that doesn't exist to `_async_new_relationship_request`, or calls the shared delete helper for the wrong body class. Neither is caught client-side. A wrong `prop` list only matters when the caller passes no body: the helper then builds `{"class": "NewRelationshipRequestBody", "properties": {"class": prop[0]}}` itself. My Profile's "add role" calls `link_person_role_to_profile` that way. The ground truth is `ActorManagerResource.java` / `GlossaryManagerResource.java` (`@RequestBody` types) and `open-metadata-framework .../properties/actors/*RoleAppointmentProperties.java`. **The `.http` file is wrong for `actor-roles/{guid}/delete`:** it says `DeleteRelationshipRequestBody`, but Egeria declares `DeleteElementRequestBody`. +**Background (Egeria's model):** +- A `GovernanceActionType` is a template for **one** action. Initiating it creates one Engine Action. +- A `GovernanceActionProcessStep` is a subtype of `GovernanceActionType` that exists only inside a process. Its flow links (`GovernanceActionProcessFlow`, `NextGovernanceActionProcessStep`) carry guards and mandatory guards. Each step that fires becomes its own Engine Action. +- Both reach their code through `GovernanceActionExecutor` (a governance engine plus a `requestType`). +- Surveys use the `SurveyingActionProcess` subtype. Survey action services write annotations into a Survey Report linked to the asset. -| Method | Sent | Egeria expects | Live before fix | -|---|---|---|---| -| `link_person_role_to_profile` | `PersonAppointmentProperties` | `PersonRoleAppointmentProperties` | 400 | -| `link_team_role_to_profile` | `TeamAppointmentProperties` | `TeamRoleAppointmentProperties` | (same shape) | -| `link_it_profile_role_to_it_profile` | `ITProfileAppointmentProperties` | `ITProfileRoleAppointmentProperties` | (same shape) | -| `detach_it_profile_role_from_it_profile` | `DeleteElementRequestBody` | `DeleteRelationshipRequestBody` | 400 | -| `detach_identity_from_profile` (`user-identities/{g}/profile-identity/{g}/detach`) | `DeleteElementRequestBody` | `DeleteRelationshipRequestBody` | not run | -| `delete_actor_role` | `DeleteRelationshipRequestBody` | `DeleteElementRequestBody` | accepted (tolerated) | -| `delete_contribution_record` | `DeleteRelationshipRequestBody` | `DeleteElementRequestBody` | not run | -| `GlossaryManager.remove_is_context_definition` | `DeleteRelationshipRequestBody` | `DeleteClassificationRequestBody` | not run | +**Direction (owner, 2026-10-13):** use Egeria's designed governance-action infrastructure, not a bypass; it may need changing, but it should be used. Resource Explorer **is an engine host** and can present itself to Egeria as one. The owner's concern is timing: running an action or process is asynchronous and possibly long-running, while an analytic function often has to answer synchronously during a screen draw or report generation. So declaring a function is kept separate from running it: -**IT detach request and response (before the fix), Egeria quickstart 9443, 2026-10-23:** -``` -POST .../actor-manager/actor-roles/a6e578d4-…/it-profile-role-appointments/91cc35e6-…/detach -{"forLineage":false,"forDuplicateProcessing":false,"class":"DeleteElementRequestBody","cascadeDelete":false} -→ HTTP 400 (pyegeria reports it as CLIENT_ERROR_400, "The client is unable to connect to the Egeria platform", which is misleading for a 400) -``` +1. **Declaration (in Egeria):** a `GovernanceActionType` per function. + - A `GovernanceActionExecutor` links it to Resource Explorer's governance engine, with the function's name as the `requestType`. + - It carries the ISSUE-149 manifest fields: `kind`, `version`, `source`, `resource_types`, `annotation_type`, `family`, `tags`. + - It joins its family `Collection`. + - Retirement is a status change, never a delete. -**Dr.Egeria:** `ActorManagerLinkProcessor` (`md_processing/v2/actor_manager.py`) called `_async_link_it_profile_role_to_profile` for `Link IT Profile Role Appointment`. No such method exists (the real one is `_async_link_it_profile_role_to_it_profile`), so that command raised `AttributeError` on every Link. + Within a survey it is used as a `GovernanceActionProcessStep` in a `SurveyingActionProcess`. A single run can be wrapped as a one-step process if a uniform "everything runs as a process" shape is wanted. +2. **Asynchronous run (through Egeria):** initiate the type, or a process containing it, and an Engine Action is created. Resource Explorer, as an engine host, claims it and runs it, using `engine-actions/{guid}/claim`, `status/update`, `completion-status` and the active-claimed listing. Results are stored as annotations carrying their envelope, and Egeria keeps the run record. This path is for surveys and anything slow or scheduled. Mandy's per-engine "approved, unclaimed" engine-action listing (2026-10-05, governance-context-service only so far; see ISSUE-151) is what such an engine host polls. +3. **Synchronous run (no Egeria round-trip):** a screen or report calls the installed implementation directly, as a report spec's `analytic_function` already does through `exec_report_spec` (ISSUE-137/138/139/140/149 made this path work for trellis routines). + - Egeria is not on the hot path; the declaration only identifies which function and version is meant. + - **Synchronous runs are not recorded in Egeria** (owner: lineage doesn't make sense for what is drawn on a screen). Diagnostic logging at most; still to discuss. -**Fix:** corrected the three `prop` lists; switched the two detaches to `_async_delete_relationship_request`, `delete_actor_role` and `delete_contribution_record` to `_async_delete_element_request`, and `remove_is_context_definition` to `_async_delete_classification_request`; aligned the type hints and docstrings; corrected the Dr.Egeria method name. Tests: `tests/micro-tests/test_role_appointment_class_names.py` (11 tests; 9 of them fail on the old code; the two that pass cover the person and team detaches, which were already correct). +**How a screen uses it:** for each function, one of three choices: +- compute now (cheap functions); +- read the last stored result (expensive functions; the envelope's `measured_at` says how old it is); +- request an asynchronous refresh that the next draw picks up. -**Checked and left alone:** `MetadataExpert.delete_related_elements` sends `DeleteRelationshipRequestBody`, while Egeria declares `OpenMetadataDeleteRequestBody`. That's deliberate (ISSUE-63): the Egeria class can't carry `deleteMethod`. Decertify and unlicense (`GovernanceOfficerResource`: `DeleteRelationshipRequestBody`) are already correct. **Why the audit missed them:** `scripts/omvs_audit.py --service actor-manager` on the old code reported 59 OK and 1 unrelated mismatch. Three blind spots, all fixed in ISSUE-164: -- Both detaches matched only by verb+path (RENAMED), and RENAMED matches skipped the body check. -- Nothing compared the inner `properties` class, which is where the three link bugs were. -- For `delete_actor_role`, `delete_contribution_record` and `is-context-definition`, the `.http` file named the same wrong class the SDK sent. +"Routine versus step" therefore becomes a property of each function (is it cheap enough to compute on read?), possibly a manifest field, not two kinds of element. -The improved audit flags all eight on the pre-fix code. +**Keeping the two paths honest:** the code run synchronously must be the code the declaration names, at the same version, that Resource Explorer's engine runs. trellis's planned `trellis-analytics drift` (installed vs declared) checks this. The envelope's `producer_version` lets readers tell versions apart. ---- +**Dr.Egeria work this implies (to be confirmed with the RE sessions):** +- Mostly convenience over existing commands: `Create Governance Action Type`, `Link Action to Action Executor` (`GovernanceActionExecutor`), `Link First/Next Process Step`, and family-collection membership. +- Plus the manifest attributes. Where these live on the element (`additionalProperties`, or real properties if Egeria adds them) is still to decide. +- Possibly a `Create/Update/Retire Analytic Function` alias family that sets all of this in one document, which `trellis-analytics declare` would generate. And `View Analytic Functions [family]`. +- The Action Author processors (`ActionExecutorTargetLinkProcessor`, `ActionProcessStepLinkProcessor`) already handle the links. `register_governance_processors()` is family-name-gated, so a new family name must be added there. -### ISSUE-162: Dr.Egeria "Update Glossary Term" resets isAbstractConcept / isActivityDescription / isContext / isDataValue to False whenever the document doesn't mention them +**Open points:** +- Whether Egeria needs changes, for example to let a synchronous-safe flag or the manifest fields be first-class properties. +- Whether a single run is always wrapped as a one-step process. +- Whether routines that are only ever computed synchronously need a declaration at all, beyond the Report element (`Create Report`). +- Diagnostic logging for synchronous runs. -**Layer:** Pyegeria (Dr.Egeria) · **Status:** fixed on branch `fix/issue-162-update-resets-booleans` (owner approved 2026-10-23), not yet released · **Found:** 2026-10-23 by RE's question-term update batch builder, reading `md_processing/v2` (RE term batch fix round 3). Reported via the Thursday Coordinator; confirmed here in source on `main` (6.2.9), not run live. -**What:** `TermProcessor.apply_changes` (`md_processing/v2/glossary.py:145-148`) builds the term properties with `attributes.get('Is Abstract Concept', {}).get('value', False)`, and likewise for `Is Activity Description`, `Is Context` and `Is Data Value`. On update, the body goes through `filter_update_properties` (`md_processing/v2/processors.py:1957`). For a merge update that drops only `None` values, so the `False` defaults are always sent. Every "Update Glossary Term", even one that changes only the description, therefore writes `False` to all four flags and clears any that were `True`. +**Resource Explorer's answers (RE PR/CI session, 2026-10-13):** +- **(a) The direction matches RE's engine-host design** (trellis branch `re/design-engine-host`, `packages/resource-explorer/docs/design-notes/DESIGN-RE-ENGINE-HOST-BY-EQUIVALENCE.md`). Its first slice (E1) is exactly the declaration above: an RE governance engine, a `SupportedGovernanceService` whose requestType names the trellis step, and a `GovernanceActionType` with a `GovernanceActionExecutor` to that engine plus `producedGuards`. RE claims and completes Engine Actions. + - Owner decisions recorded there: the engine runs as RE's service account. An action on a resource in a private investigation is left unclaimed by the unattended host and run attended, as the owner. RE may claim and hold future-dated actions. + - RE polls `get_active_engine_actions()` today, filtered client-side to its engine and APPROVED (or `find_engine_actions`, see ISSUE-151). Mandy's per-engine "approved, unclaimed" listing exists only on the metadata server's governance-context service, so RE is asking her to expose it on the View Server. + - Questions open with Mandy: the engine subtype (`SurveyActionEngine`?); whether a GovernanceService whose connector class nothing can load is acceptable; and where guards come from (standard survey guards, or declared by the step from the envelope). +- **(b) RE's survey definitions today** are Dr.Egeria `Create Governance Action Process` (`GovActionProcess::`) plus `Create Governance Action Process Step` (`GovActionProcessStep::::`), linked with First/Next Process Step. They are plain `GovernanceActionProcess`, not `SurveyingActionProcess`: Dr.Egeria's command has `OM_TYPE` `GovernanceActionProcess` and offers no subtype choice. The steps carry **no executor**; RE's own executor reads `executes_at`/`re_analysis_step`/`supported_technology_type` from `additionalProperties`. That is why Egeria can't dispatch any RE step today. Moving the steps onto an executor is part of RE's engine-host slice. +- **(c) Manifest fields go in `additionalProperties` for now,** with stable keys named exactly like the `AnalyticFunctionSpec` fields (`kind`, `source`, `version`, `envelope`, `annotation_type`, `resource_types`, `family`, `tags`). This matches the owner's interim ruling for annotations. Asking Egeria for real properties is a later item, once a few functions exist. +- **Synchronous runs not recorded:** agreed. RE's OpenLineage run events (branch `re/openlineage-run-events`, off by default) cover only step runs inside RE's survey executor, so a screen-draw run never reaches them. -**Same pattern, likely affected (not confirmed):** `md_processing/v2/data_designer.py:54-55` (`DataValueSpecificationProcessor`) and `:547-549` (`DataClassProcessor`) default `allowsDuplicateValues` / `isNullable` to `True`. Both processors appear to build one property body for create and update, so an unrelated update would reset those to `True`. Other `.get('value', )` uses checked in `md_processing` are request options (`forLineage`, `startsWith`, ...) or create-only values, not stored properties. +**Dr.Egeria gaps this exposes (part of this issue's work):** +1. `Create Governance Action Process` should be able to create the process subtypes (`SurveyingActionProcess`, `CataloguingActionProcess`, ...), e.g. via a `Process Type` attribute validated against the 0462 subtypes. Today it always creates the plain supertype. +2. A process step needs its `GovernanceActionExecutor` link and its manifest `additionalProperties`. Check that `Link Action to Action Executor` accepts a `GovernanceActionProcessStep` (a subtype of `GovernanceActionType`) as its subject. It should, but this hasn't been live-verified (see the Action Author note in CLAUDE.md). +3. Declaring the engine side may need `SupportedGovernanceService` / governance-engine commands, if Dr.Egeria doesn't already have them. Check before building. -**Impact:** a term with any of the four flags set loses it on an unrelated update. RE's 96-term question update batch will check the terms before applying. +**Owner's answers on these gaps (2026-10-13):** +1. Process subtypes: create new Dr.Egeria commands for the other process types where needed. +2. Executor on a process step: yes, it needs live testing. +3. Engine-side commands: yes, likely needed. -**Fix (2026-10-23):** -- **Glossary terms:** the four flags now default to `None`, so a merge update omits them unless the document sets them. Create still sets them to `False` as before. -- **Data Designer: confirmed and wider than first logged.** Both `DataValueSpecificationProcessor` and `DataClassProcessor` updates send their property body straight to `_async_update_element_body_request`, where `body_slimmer` drops only `None`. So an unrelated update also reset `matchPropertyNames`, `sampleValues` and `dataPatterns` to `[]`, `matchThreshold` to `0`, and `specificationDetails` and `additionalProperties` to `{}`, besides `allowsDuplicateValues` / `isNullable` to `True`. All of these now default to `None`, and the old defaults are applied only on Create (`_with_create_defaults`). Egeria's `mergeUpdate` defaults to `true` (`UpdateOptions.java:23`), so omitted properties keep their values. -- Tests: `tests/micro-tests/test_issue_162_update_does_not_reset_properties.py`, 7 cases. The 4 update cases fail on the old code; the 3 create cases guard the unchanged create defaults. The full micro-test suite (1,436) passes in a clean python:3.13-slim container. Not live-tested. +**Requirements proposed by RE's expansion-design session (2026-10-13; it found no contradiction with the direction). Owner's review, the same day: 1 and 2 accepted; 3 kept as an informational key only:** +1. **One declaration, idempotent.** The manifest's qualifiers must live on the `GovernanceActionType` itself (`supportedRequestParameters` or `additionalProperties`), so Egeria, the Portal and RE all read the same declaration. That means question ids, perspectives, `purpose` (from the existing `VALID_PURPOSES` set) and `family`. Declaring is idempotent by qualifiedName: declaring twice updates, and never creates a second type. Dr.Egeria's upsert (Create→Update by qualified name) already behaves this way. +2. **Label synchronous values differently from recorded ones.** On a screen, a value computed now must be labelled "computed now · not recorded", and a stored result "last recorded ". A reader must never mistake a synchronous draw for a governed measurement. (The envelope gives the state and `measured_at`; the "recorded or not" label is the caller's, from which path it used.) +3. **State the credential tier.** User-written steps stay on the resource's read-only tier (owner ruling, 2026-10-07). A `GovernanceActionType` declared for one must say so, with a property naming the tier (e.g. `credential_tier: read_only`). **Owner: an informational key, not a control.** Egeria schedules Engine Actions the same way regardless, and never hands a step its resource credentials. The engine host does: RE's wrapper enforces read-only from the step's manifest (`StepManifest.credential_tier`). The key on the declaration is for governance visibility (whoever reviews or approves the type can see it), and for any other engine host that might run the step later. The design session confirmed this when asked (2026-10-13). It withdrew its "control" reading: the only reader other than RE's wrapper would be a second engine host, which enforces the tier itself. An Egeria-side approval step or governance rule that reads the tier would be new design nobody has asked for. -**Originally proposed:** -- Default these attributes to `None` (`.get('value')`), so a merge update omits them unless the document sets them. Keep `False` / `True` only on the create path, or leave creation to Egeria's own defaults. -- Check the Data Designer cases the same way. -- Add a micro-test: an update with no flag attributes sends none of the four keys. +**Mandy's answer (2026-10-14, via the owner and the RE PR/CI session); the declaration direction is confirmed, no longer provisional:** +- **Engine type:** `SurveyActionEngine` for surveys. RE may also expose services through a `GovernanceActionEngine` (general) or a `WatchdogActionEngine` (monitoring). +- **What to create:** a governance service definition for each service, of the subtype matching the engine: a **`SurveyActionService`** for a `SurveyActionEngine` (survey steps), a `GovernanceActionService` for a `GovernanceActionEngine`. Plus the `GovernanceActionExecutor` relationship to the engine element. (Owner's correction, 2026-10-14: not a plain `GovernanceService`.) +- **requestType:** the value the engine advertises, and the value it's passed when asked to execute. RE dispatches on it. +- **Guards:** standard survey guards are best practice, not mandatory. +- **Coming in a few days:** more engine-host internal APIs, including the per-engine approved-actions listing, will be exposed as View Server REST calls once her current fixes are done. pyegeria will need wrappers for them (BACKLOG.md). +- Refs: [Survey Action Engine](https://egeria-project.org/concepts/survey-action-engine/), [0461 Governance Engines](https://egeria-project.org/types/4/0461-Governance-Engines/), [0462 Governance Action Processes](https://egeria-project.org/types/4/0462-Governance-Action-Processes/). +**What exists today for the engine side (gap 3, checked 2026-10-14):** +- The SDK can link a service to an engine: `AssetMaker` `governance-engines/{guid}/supported-governance-services/{guid}/attach`, plus `update`/`detach` (`asset_maker.py` ~1693-1813 and ~6586-6803). +- Dr.Egeria has **no** commands to create a `GovernanceEngine` (or its `SurveyActionEngine`/`GovernanceActionEngine`/`WatchdogActionEngine` subtypes) or a governance service (`SurveyActionService`/`GovernanceActionService`/...), and none for the supported-service link. Only `Link Action to Action Executor` exists (Action Author). +- These are the Dr.Egeria commands to add: `Create Governance Engine` (with an engine-type attribute), `Create Governance Service` (with a service-type attribute matching the engine: `SurveyActionService` for survey steps), and `Link Governance Service to Engine` (with requestType). Check the 0461 types and the `.http` files for the create endpoints before building. --- -### ISSUE-161: Dr.Egeria's two entry points resolve the Egeria user differently, and the CLI fixes its credential defaults at import time -**Layer:** Pyegeria · **Status:** fixed on branch `fix/issue-161-dr-egeria-credentials` (owner approved 2026-10-21), not yet released · **Found:** 2026-10-21, while checking an RE report that `dr_egeria --process` ignored `EGERIA_USER`. **That report is withdrawn; RE was the cause:** RE's own `config.py:316` defaulted its daemon identity (`EGERIA_USER_ID`) to "erinoverview", so RE handed the child process erinoverview. The CLI correctly used it. RE is removing that default. -**What was checked:** with `EGERIA_USER` set, the CLI path resolves and uses it throughout on 6.2.8: -- `config.py:423` (`os.getenv("EGERIA_USER", config.json / .env value)`) feeds `settings.User_Profile.user_name`, the `--userid` default; -- `EgeriaTech` / `BasePlatformClient` keep that `user_id`; -- `create_egeria_bearer_token()` with no arguments uses `self.user_id`; -- `_get_subclient` passes the user and token to every sub-client, and there is no token cache. -Intended precedence (CLAUDE.md): explicit `--userid` > OS env > `.env` > `config.json` > default. +### ISSUE-146: Deleting a journal entry fails with OMAG-REPOSITORY-HANDLER-400-008, and pyegeria's `remove_note` reaches the delete only through the generic Asset Maker service -**Real issues (pyegeria):** -1. **Inconsistent resolution between entry points:** `md_processing/dr_egeria.py:93` (module constants) and `:923` (its own `__main__` path) read `os.environ.get("EGERIA_USER", "erinoverview")` / `EGERIA_USER_PASSWORD` directly, bypassing `.env` and `config.json`. `md_processing/md_processing_utils/common_md_utils.py:29-30` does the same. So `python -m md_processing.dr_egeria` and the `dr_egeria` CLI disagree whenever the user is set only in `.env` or `config.json`. -2. **Import-time defaults:** `commands/cat/dr_egeria.py:32-33` computes the `--userid` / `--user_pass` defaults when the module is imported. Code that imports it, then sets `EGERIA_USER` and invokes the command in the same process, gets the earlier user. This doesn't affect normal CLI use, where each run is a new process. -3. **Help text:** "Overrides EGERIA_USER (env var or .env file)" doesn't mention `config.json`, or that the default is fixed at import. +**Egeria reset 2026-10-23 ~21:10-23:54Z** (egeria-workspaces-fs, owner-approved): the quickstart database was wiped and Egeria recreated on build 81beb8e5 (odpi/egeria 4486d1a7), then Coco content was reloaded. Every GUID recorded before then is gone. The test JournalEntries `fef83ed5` and `8bec72e4` on Erin's profile no longer exist, so the pending `remove_note` re-test below needs a NEW entry made with `journal_my_activity` (and, for the 400-008 case, one owned by an external source). The 2026-10-18 Egeria analysis (works as designed; 400-008 is the provenance check) still applies. -**Fix (2026-10-21):** a new `pyegeria.core.config.resolve_egeria_credentials(user_id=None, user_pwd=None)` resolves, when called: an explicit value > the `EGERIA_USER` / `EGERIA_USER_PASSWORD` environment variables > the settings User Profile (`config.json` / `.env`) > "erinoverview" / "secret". -- The `dr_egeria` and `dr_egeria_folder` CLIs now default `--userid` / `--user_pass` to `None` and call it when they run. Their help text says so. -- `md_processing/dr_egeria.py` (module constants and its `__main__` path) and `common_md_utils.py` constants now use it too. The module-level `EGERIA_USER` names are kept for importers. -- Tests: `tests/micro-tests/test_issue_161_dr_egeria_credentials.py`. Five cases: precedence; env set after import, for both CLIs; config-only user; explicit flag wins. The three behaviour cases fail with the old caller code. The full micro-test suite (1,425) passes in a clean python:3.13-slim container. -- **Not changed:** other `commands/cat` tools (`list_glossaries`, `get_asset_graph`, `run_report`, `get_collection_tree`, `get_project_dependencies`, `list_user_ids`, `view_schema_tree`) still read `EGERIA_USER_PASSWORD` straight from the environment. They could adopt the helper later. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: works as designed (Egeria).** Asset Maker's `deleteAsset` is the right call: a JournalEntry is a Notification, so an asset, and no separate note delete is needed. Egeria's new FVT creates entries with My Profile's `journal-my-activity` and deletes them with `deleteAsset`, with no external source; every case passes on in-memory and PostgreSQL repositories. **400-008 is the provenance check:** it refuses a change made without an external source to an element an external source owns. No external source → 400-008; a different external source → 400-007; the matching external source → success. So an entry that returns 400-008 was not created locally, and its delete needs that entry's `externalSourceGUID` and `externalSourceName`. If 400-008 appears on an entry created through My Profile, that is a bug and Egeria wants: (1) the full error message (it names the type, GUID, provenance and metadata collection of the refused instance); (2) how the entry was created (which request, and which external source if any). Also new: purge applies the same provenance check. **pyegeria:** the `remove_note` clean-up below stays (route + docstring). Re-test on `8bec72e4` against a platform with the fixes; if 400-008 recurs, capture the full message for Egeria, since that entry was created locally through `journal-my-activity`. -**Originally proposed:** -- Resolve credentials through `settings.User_Profile` in all three places. -- Look them up when the command runs (click `default=None`, then resolve inside the command). -- Add a micro-test that both entry points pick up a user set only in `config.json` or `.env`. +**Layer:** Egeria Server (pyegeria clean-up) · **Status:** open · **Found:** 2026-10-13, reported by the egeria-workspaces-fs Resource Browser session from a live private-journal test. ---- +**What:** as erinoverview, `_async_remove_note` (`pyegeria/core/_server_client.py`) +was called on a JournalEntry created by `MyProfile.journal_my_activity`. It +failed with `OMAG-REPOSITORY-HANDLER-400-008` (`LOCAL_CANNOT_CHANGE_EXTERNAL`): +`deleteMetadataElementInStore` cannot modify Erin's UserIdentity (`ff983b64`), +which has Content Pack provenance ("Coco Pharmaceuticals Combination"). The +peer verified that nothing was removed. The entry keeps its `actionSource` and +`presentInNoteLogs` relationships, and Erin's Person still has 1 identity, +1 contact detail, 8 roles and 15 scopes. Test entry `fef83ed5` is still on +Erin's profile, waiting for the owner's decision. -### ISSUE-160: `AutomatedCuration`'s template builders miss placeholders Egeria's templates now use, so created elements keep literal `~{name}~` values (folder endpoint `~{directoryAddress}~` breaks folder surveys) +**The route is right:** in 6.2, notes are Actions. The type chain is +JournalEntry → Notification → Action → Process → Asset. Egeria main +(`af400039c2`) has no note, notification or action delete endpoint. +`AssetHandler.createNoteLogEntry` has no matching remove, and My Profile has +`journal-my-activity` with no matching remove. The only delete for an Action is +the generic `/assets/{assetGUID}/delete` (`AssetMakerResource`), which is what +`remove_note`'s `feedback-manager/assets/{guid}/delete` reaches, because +generic view services answer under any URL marker. Feedback Manager can only +remove a whole note log (`note-logs/{noteLogGUID}/remove`). -**Layer:** Pyegeria · **Status:** partly fixed on branch `fix/issue-160-template-placeholders` (owner approved 2026-10-21); not yet merged or released. The rest is listed below for the owner. **Found:** 2026-10-21, live by the RE coordinator against Egeria `main` 05a5e982 with pyegeria 6.2.7. Two FileFolders made with `create_folder_element_from_template` had Endpoint `networkAddress` = `~{directoryAddress}~`. Egeria's Folder Survey then failed with `BASIC-FILE-CONNECTOR-404-001` "The folder named ~{directoryAddress}~ ... does not exist" (engine action `c1d924b5`). The same folders also kept `versionIdentifier` = `~{versionIdentifier}~` and `deployedImplementationType` = `~{deployedImplementationType}~` (record `~/.re-gate/repair-folder-endpoints-20261021T143415Z.json`). +**Why it fails (likely, not verified):** `AssetHandler.createAction` anchors the +entry to its action source, linked by `ActionRequester`. For a journal entry +that is the caller's profile. Deleting an anchored element updates its anchor, +and Erin's identity is owned by the content pack, so the local delete is +refused. If so, no journal entry for a content-pack user can be deleted. That +is for the Egeria team. -**Cause (Egeria `origin/main` d8f72cbe25, read-only):** no placeholder was renamed (`git log -S` on `PlaceholderProperty.java`). Egeria **added** placeholders that pyegeria never sends: -- **`directoryAddress` / `fileAddress`:** added by `156650efd6` (2026-07-19, "file system control for file cataloging"). They are now the Endpoint address in the folder and file templates (`DataAssetTemplateDefinition` `FILE_FOLDER_TEMPLATE`, `DATA_FOLDER_TEMPLATE`, `CSV_FILE_TEMPLATE`, ...). The templates still use `directoryPathName` / `filePathName` in the qualifiedName, so both are needed. -- **`deployedImplementationType`:** in every folder/file placeholder list (`getFolderPlaceholderPropertyTypes`, `getDataFilesPlaceholderPropertyTypes`). -- **`databaseDescription`:** the PostgreSQL Relational Database template's description placeholder (`PostgresPlaceholderProperty.DATABASE_DESCRIPTION`); pyegeria sent `description`. +**Anchor read (peer, read-only):** JournalEntry `fef83ed5` has anchorGUID +`10dc7da9`, Erin's Person profile. Its `actionSource` is ActionRequester → +Person `10dc7da9`, and `presentInNoteLogs` is AttachedNoteLogEntry → NoteLog +`f780511a` (MyJournal). No relationship from the entry reaches UserIdentity +`ff983b64`, which the error names. That identity is anchored to the same +Person. So the refusal comes from the delete touching the anchor's graph, most +likely the LatestChange update on the anchor, or a provenance check across +elements that share the anchor, not from a direct link. -An unsent placeholder is stored as the literal `~{name}~`. `body_slimmer` drops `None` values, so an optional argument left unset has the same effect. +**Delete-path trace (2026-10-13, Egeria source, read-only):** the asset delete +(`AssetMakerResource.deleteAsset` → `AssetHandler.deleteAsset`) and the Metadata +Expert delete (`MetadataExpertResource` → `MetadataExpertRESTServices`) both call +`OpenMetadataClient.deleteMetadataElementInStore`, then the same +`MetadataElementHandler` → `OpenMetadataAPIGenericHandler.deleteBeanInRepository` +code. Neither does any extra checking, so switching routes cannot avoid 400-008. +The error is thrown only by `RepositoryErrorHandler.validateProvenance`, when +the delete removes an entity or relationship whose provenance is not +LOCAL_COHORT and the request carries no `externalSourceGUID`. That happens in +`removeRelationship` for each relationship on the note and in `removeEntity` +for the note itself. No anchor (LatestChange) update was found on the delete +path, which corrects "Why it fails" above. But the error names UserIdentity +`ff983b64`, which is not linked to the note, so the exact instance is still +unexplained. Untried options are to pass the content pack's metadata-collection +id as `externalSourceGUID`, or to use `deleteMethod: ARCHIVE`, which goes +through a different handler. -**Fixed on the branch** (`pyegeria/omvs/automated_curation.py`; signatures unchanged): -| Method | Added | -|---|---| -| `create_folder_element_from_template` | `directoryAddress` (= `path_name`), `deployedImplementationType: "File System Directory"` | -| `create_csv_data_file_element_from_template`, `get_create_csv_data_file_element_from_template` | `fileAddress` (= `file_path_name`), `deployedImplementationType: "CSV Data File"` | -| `create_secrets_store_element_from_template` | `fileAddress` (= `file_path_name`), `deployedImplementationType: "YAML File Secrets Collection"` | -| `create_postgres_database_element_from_template` | `databaseDescription` (= `description`; `description` still sent) | +**Note-privacy findings (egeria-workspaces-fs session, 2026-10-13, with owner +approval):** a second test JournalEntry, `8bec72e4-6da6-40a4-943e-d81e2173ec9b`, +was created on Erin's profile (`10dc7da9`) and cannot be deleted either until a +working delete exists. +- `POST my-profile/journal-my-activity` ignores `initialClassifications` and + stamps only ZoneMembership `['erinoverview']`. +- `POST classification-explorer/elements/{guid}/zone-membership` with + `['resource-explorer-private','erinoverview']` makes the entry private. + erinoverview can read it. peterprofile and garygeeke get 403 + `OPEN-METADATA-SECURITY-403-020` by GUID, and 0 hits from + `feedback-manager/assets/by-search-string`. +- The journal's NoteLog (`f780511a`) stays at `['erinoverview']`, so other + users can see that the log exists but not the hidden entries. +- The owner chose a temporary workaround for the Resource Locator: call + `add_zone_membership(['resource-explorer-private', user])` straight after + `journal-my-activity`. Review it once Egeria's journal fix lands. -Tests: `tests/micro-tests/test_issue_160_template_placeholders.py` (5 cases; all fail on the old code, all pass on the fix). The full micro-test suite (1,409) passes in a clean python:3.13-slim container. +**pyegeria clean-up:** make the `remove_note` route explicit and describe it as +an Action delete in the docstring. The owner chose the Metadata Expert delete +(`metadata-expert/metadata-elements/{guid}/delete`) for now. Per the trace +above, this is a clarity change, not a fix for 400-008. -**Live check 2026-10-21 14:52Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; the fixed code on this branch): `create_folder_element_from_template("/tmp/pyeg_rt_folder_20261021T145254Z", ..., version="1.0")` created FileFolder `e9a45c7f` with `deployedImplementationType` "File System Directory" and `versionIdentifier` "1.0". Its anchored Endpoint `8f003544` had **`networkAddress` = `/tmp/pyeg_rt_folder_20261021T145254Z`**. The folder, Connection `18ef2ee9` and Endpoint had no `~{...}~` left in their own properties. A metadata-expert delete (`deleteMethod: PURGE`, `cascadedDelete: true`) removed all three; each was verified gone by GUID. A first run (`a3475d3e`) confirmed the folder fields but its script missed the Endpoint. Exact `networkAddress` searches (forLineage) afterwards found no leftover for that run either. The survey itself was not run. +**Re-test after Egeria's delete fixes are published:** the owner said +(2026-10-13) that Egeria is working on delete issues now. Once that work is +published, re-test `remove_note` on `8bec72e4` (plain delete, then `ARCHIVE` if +needed). Tell the egeria-workspaces-fs session before deleting anything on +Erin's profile. `fef83ed5` stays. -**Follow-up fix 2026-10-21 (owner: "fix them all in a new pr")** on branch `fix/issue-160-remaining-placeholders`, built on #474. All 13 hard-coded builders now wrap their values in `_placeholder_values`, which sends `""` for any placeholder left `None`, so no `~{name}~` literal is stored. `body_slimmer` drops only `None`. A folder created without a file system therefore gets qualified name `FileFolder:::/path`. -- **Postgres server and database:** new optional `secrets_store_path_name` / `secrets_collection_name`, sent as `secretsStorePathName` / `secretsCollectionName`, plus `version` (and `resource_name` on the server, defaulting to the server name). `db_user` / `db_pwd` are now optional, deprecated and no longer sent; passing either raises a `DeprecationWarning`. -- **Kafka server:** `resourceName` (defaults to the server name) and `versionIdentifier`. -- **Unity Catalog:** - - UC server: `resourceName`, defaulting to the server name; - - UC catalog: `ucServerQualifiedName`; - - UC table and function: `rootSchemaTypeQualifiedName`; - - UC volume: `directoryPathName` and `localStorageLocation`. - Each is a new optional parameter on the async method and its sync wrapper. -- **Item 4 needed no change:** "Keystore File" is a real template (`DataAssetTemplateDefinition.KEYSTORE_FILE_TEMPLATE`), and live, both "YAML File Secrets Collection" and "Keystore File" resolve to template `fbcfcc0c`. +--- -Tests: `tests/micro-tests/test_issue_160_remaining_placeholders.py`, 11 cases, all failing on #474's code. The full micro-test suite (1,420) passes in a clean python:3.13-slim container. **Dr.Egeria follow-up, done 2026-10-21** (after 6.2.8; branch `claude/elegant-chatterjee-d280de`): the `Create Postgres Server Element` / `Create Postgres Database Element` commands no longer offer `Database User Id` / `Database Password`. Both attributes were deleted from the Asset Maker compact spec, since no other command used them. The commands now offer `Secrets Store Path Name`, `Secrets Collection Name` and `Version Identifier`, and the server also offers `Resource Name` (the canonical Governance Officer definition, copied). `AssetMakerProcessor` passes them as `secrets_store_path_name` / `secrets_collection_name` / `version` / `resource_name` and no longer passes `db_user` / `db_pwd`. Templates, help and report specs were regenerated, and the templates were copied to egeria-workspaces and trellis on local branches. Test: `tests/micro-tests/test_postgres_element_secrets.py`. Not run against a live server. -**Follow-up live check 2026-10-21 15:16Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; #475 code): -- **(a)** `create_postgres_server_element_from_template("pyeg_rt_pg_…", "pyeg-rt.invalid", "5432", secrets_store_path_name=…, secrets_collection_name=…)`, with no `db_user` / `db_pwd`, created SoftwareServer `b6b2ebff`: `resourceName` = the server name, `versionIdentifier` and `description` = `""`, and no `~{...}~` in its own properties. Its Connection `55686913` and Endpoint `59014065` (`jdbc:postgresql://pyeg-rt.invalid:5432/postgres`) had no literals either. -- The server had **no CatalogTarget link** (its relationships were sourcedFromTemplate, serverEndpoints, capabilities, connections), so the JDBC integration connector never saw it. -- I did not find where Egeria stored the secrets-store values. They were not in the Connection's own properties, and no literal was left. Unconfirmed. -- **(b)** `create_folder_element_from_template("/tmp/pyeg_rt_folder2_…", …, None)`, with no file system, version or description: qualifiedName `FileFolder:::/tmp/pyeg_rt_folder2_…`, `resourceName` `:/tmp/…`, `versionIdentifier` / `description` = `""`, `deployedImplementationType` "File System Directory", and Endpoint `networkAddress` = the path. No literals on the folder, Connection or Endpoint. -- Both were purged (metadata-expert delete, `PURGE`, `cascadedDelete`), and all six GUIDs were verified gone. Name, host, stamp and secrets-file searches afterwards found nothing, which covers the server's capabilities too. -**Originally listed as not fixed** (kept for the record): -1. **Postgres server and database credentials:** both templates now take `secretsStorePathName` and `secretsCollectionName` (credentials come from a secrets store). pyegeria never sends them, so the connection keeps literals. pyegeria still sends `databaseUserId` / `databasePassword`, which the templates no longer list (`312c172639`, `56fa898e43`), so those are dead. Proposed: add optional `secrets_store_path_name` / `secrets_collection_name` parameters, and deprecate `db_user` / `db_pwd`. -2. **Optional arguments left unset:** `versionIdentifier`, `description`, `fileSystemName` and the like are dropped when `None`, so they stay `~{...}~` (RE's `versionIdentifier` case). Options: send `""` for unset optional placeholders, or document that callers must pass them. -3. **Unsent template placeholders in the Unity Catalog and Kafka builders:** - - UC catalog: `ucServerQualifiedName`; - - UC table and function: `rootSchemaTypeQualifiedName`; - - UC volume: `directoryPathName`, `localStorageLocation`; - - UC server and Kafka server: `resourceName` (Kafka also `versionIdentifier`). - Add parameters for them where a caller can know the value. -4. **The secrets-store builder's fallback** to a "Keystore File" technology type (`_async_get_template_guid_for_technology_type`) matches no template on `origin/main`. -Builders with no hard-coded keys (`create_elem_from_template`, Dr.Egeria's "Placeholder Property Values" attribute) are unaffected. The UC schema builder sends all its placeholders. +### ISSUE-132: `DeleteElementRequestBody` serializes the cascade flag as `cascadeDelete`; Egeria's field is `cascadedDelete` and unknown fields are ignored, so no pyegeria cascade delete has ever cascaded ---- +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. `cascadeDelete` is now accepted as an alias for `cascadedDelete`, so existing pyegeria requests will start to cascade; Egeria still recommends switching to `cascadedDelete`. **Risk check, 2026-10-18:** no production caller in `pyegeria/`, `md_processing/` or `commands/` passes cascade `True`; every `cascade`/`cascade_delete` default is `False`; `commands/tech/element_actions.py`'s `--cascade` flag (default True) is not forwarded. Live functional/scenario tests that pass `cascade=True` (collection, glossary, governance officer, `remove_note_log`) will now really cascade, which is what they intend. **pyegeria (proposed, not done):** change `DeleteElementRequestBody`'s serialization alias to `cascadedDelete` (keep both accepted on input) and the ~25 `"cascadeDelete"` literals (`_server_client.py`, `models.py`, `actor_manager.py`, `metadata_expert.py`, `product_manager.py`, `reference_data.py`, `valid_metadata.py`), then live-test one cascade and re-test ISSUE-127. No longer urgent, since the old spelling now works. -### ISSUE-159: no archive wrappers on the project or collection managers, and no restore wrapper at all, although Egeria offers both +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS (alias).** A throwaway DataStructure with a linked member DataField, deleted with `delete_data_structure(guid, cascade_delete=True)` (pyegeria still sends `cascadeDelete`), succeeded. Both elements were gone by GUID before any cleanup, so the cascade reached the server. The ISSUE-154 item 1 rename to `cascadedDelete` is still recommended but no longer urgent. -**Layer:** Pyegeria · **Status:** open, logged only (no fix without the owner's word) · **Found:** 2026-10-20, raised by the Resource Explorer expansion-architecture session: RE's investigation-delete ruling needs a Project archived through the project manager and restored by GUID. Facts confirmed here against Egeria `origin/main` 05a5e982 and pyegeria upstream `main`, read-only. +**Layer:** pyegeria · **Status:** open, on hold: an Egeria fix for cascade delete is coming (owner, 2026-10-10). Egeria `main` checked 2026-10-10 (head `af400039c2`, 2026-10-06): no cascade change yet. Wait +for it, live-test cascade delete against it, then decide the pyegeria change; the fix below may change once the +Egeria side lands. Confirmed from Egeria source, not yet live-tested · **Found:** +2026-10-10, while checking ISSUE-129's server contract against Egeria source instead of the `.http` files. -**Egeria:** `OpenMetadataAPIGenericHandler.java` (common-services/generic-handlers): -- `archiveBeanInRepository` (`:3289`) classifies the element Memento, renames each affected Referenceable's qualifiedName to `_archivedOn_` (marker `archivedQualifiedNameMarker = "_archivedOn_"`, `:51`), and records one `lastOperationId` per operation. -- `restoreBeanInRepository` (`:4170`) restores an element and everything the same delete or archive changed, given any GUID that operation touched. +**What:** `pyegeria/models/models.py` `DeleteElementRequestBody.cascade_delete` accepts both spellings on input but +serializes as `serialization_alias="cascadeDelete"`. The comment there records that the `.http` files were split +about 50/50 and that the choice was unverified (ISSUE-62). Egeria source settles it (`egeria-v6/egeria` at +`af400039c2`, 2026-10-06): `DeleteElementRequestBody` -> `DeleteRequestBody` -> `DeleteOptions` +(`open-metadata-framework/.../search/DeleteOptions.java`), whose field is `cascadedDelete` with +`getCascadedDelete()`/`setCascadedDelete()`. It is unchanged since 2025-08-04. The class is +`@JsonAutoDetect(getterVisibility=PUBLIC_ONLY, fieldVisibility=NONE)` and `@JsonIgnoreProperties(ignoreUnknown=true)`, +with no `@JsonProperty`/`@JsonAlias`, so the JSON name is `cascadedDelete` and `cascadeDelete` is silently ignored. -The OMF store's delete (`OpenMetadataStoreRESTServices`, around `:4118`) also routes `deleteMethod: ARCHIVE` to `archiveMetadataElementInStore`, and resolves `LOOK_FOR_LINEAGE` to ARCHIVE when lineage relationships exist. - -**pyegeria:** -- Archive exists only as `MetadataExpert.archive_metadata_element` (`metadata-expert/metadata-elements/{guid}/archive`). -- There is no `archive_project`, `archive_collection` or other manager-level archive. -- There is no `restore_*` method anywhere. ISSUE-155 item 5 already lists Metadata Expert `metadata-elements/{guid}/restore` and `related-elements/{guid}/restore` as missing. +**Effect:** every pyegeria delete that uses `DeleteElementRequestBody` with cascade on (the ISSUE-62 fix, +`validate_delete_element_request`, ISSUE-127's fix in #429, `MetadataExpert.delete_metadata_element(cascade_delete=True)`) +sends a flag Egeria discards, so it deletes without cascading. A caller-supplied dict using the correct +`cascadedDelete` key is also re-serialized as `cascadeDelete`, so even that does not work. The models module's own +`DeleteRequestBody.cascaded_delete` already uses the right name. -**Possible route today (unverified):** `ProjectManager.delete_project(guid, body={"class": "DeleteElementRequestBody", "deleteMethod": "ARCHIVE"})`. pyegeria passes `deleteMethod` through, but whether the project-manager delete path honours ARCHIVE as the OMF store delete does was not traced. +**Fix (not made):** change the serialization alias to `cascadedDelete` (keep both spellings accepted on input), and +fix the 19 `"cascadeDelete"` references in `_server_client.py`, `product_manager.py`, `actor_manager.py`, +`valid_metadata.py`, `metadata_expert.py` and `reference_data.py` (several are docstring samples). Then live-test one +cascade delete, and re-test ISSUE-127's `DataStructure` refusal. The `.http` files using `cascadedDelete` are correct; +the ones using `cascadeDelete` are the ones in error (worth reporting to Egeria). -**Proposed (not made):** -1. ~~Add `MetadataExpert.restore_metadata_element` / `restore_related_elements` (and undo), per ISSUE-155 item 5.~~ Done 2026-10-23 (branch `fix/role-appointment-class-names`). Items 2 and 3 are still open. -2. Add `ProjectManager.archive_project` and `CollectionManager.archive_collection`. These can be thin wrappers that send `deleteMethod: ARCHIVE` once that path is confirmed live, or call Metadata Expert archive. -3. Live-test archive then restore of a throwaway Project, including the `_archivedOn_` rename and its reversal. +**Lesson:** where the `.http` files disagree with each other, read the Java DTO (getter/setter names plus Jackson +annotations); do not choose by majority or by what the code already sends. --- -### ISSUE-158: `GOV_DEF_PROPERTIES_LIST` lacks `DataQualityRuleProperties`, and Dr.Egeria's governance create fallback silently drops every type-specific attribute - -**Layer:** Pyegeria · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, raised by the TestGen evaluation session while drafting a DataQualityRule convenience-API spec (`~/localGit/egeria-v6/wt-testgen-notes/packages/resource-explorer/docs/design-notes/SPEC-PYEGERIA-DATA-QUALITY-RULE.md`, uncommitted, pending owner review). Both points confirmed in source here, read-only; not run live. - -1. **Whitelist gap.** `GOV_DEF_PROPERTIES_LIST` (`pyegeria/omvs/governance_officer.py:37`) is the client-side property-class whitelist for `create_governance_definition` (`:426`) and `update_governance_definition` (`:955`). It has no `DataQualityRuleProperties`, although Egeria `main` has that class (`frameworks/.../properties/governance/DataQualityRuleProperties.java`). **Corrected 2026-10-19:** the list is enforced only on update. `validate_new_element_request` (`pyegeria/core/_server_client.py:6115`) has its class check commented out, so a create with `DataQualityRuleProperties` reaches Egeria unchanged. `validate_update_element_request` (`:6252`) does check a dict body's class against the list, so `update_governance_definition` refuses a DataQualityRule body client-side ("unexpected property class name"). No Dr.Egeria command creates DataQualityRules today. **Proposed:** add `DataQualityRuleProperties`, and drop the list's duplicates (`RegulationProperties`, `GovernanceControlProperties` and `GovernanceApproachProperties` each appear twice). Better still, check the list against Egeria's `GovernanceDefinitionProperties` `@JsonSubTypes` so it can't drift again. -2. **Silent fallback.** In `GovernanceProcessor`'s create path (`md_processing/v2/governance.py:203-221`), any non-SecurityGroup create that Egeria refuses with `OMAG-COMMON-400-029` (property class not accepted for `createGovernanceDefinition`) is retried with `_to_base_governance_definition_properties` (`:96`). That keeps only the cross-type fields (qualifiedName, displayName, description, category, identifier, contentStatus, versionIdentifier, effective dates, additional/extended properties). The element is created, but every type-specific attribute is dropped (e.g. a DataQualityRule's quality fields, a metric's `measurement`/`target`, a control's `implementationDescription`). The only sign is a `logger.warning`, and Dr.Egeria reports success. **Proposed:** remove the generic fallback, or keep it only for the server-compat case it was written for. Otherwise make it fail visibly, naming the dropped fields in the command's result. Live-test which current types, if any, still hit 400-029 on Egeria `main` before deciding. +### ISSUE-127: `CollectionManager.delete_collection(cascade=True)` silently never sends `cascadeDelete` -- same shape as ISSUE-62 ---- +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** The `DataStructure` refusal (`validateNoMemberDataFields`) did not recur: a cascade delete of a structure with a member field removed both. See ISSUE-132's re-test. -### ISSUE-157: Dr.Egeria `Create Service Level Objective` sends `entitlements`/`obligations`/`restrictions` as empty lists; Egeria types them as maps +**Layer:** pyegeria · **Status:** partly fixed in #429 (`68279835`), released in 6.1.29; cascade still blocked by ISSUE-132 · **Found:** 2026-10-05, found by capturing the request body while diagnosing a +failed cleanup. -**Fixed 2026-10-19 (owner approved) on branch `fix/issue-156-157-catalog-target-slo` (956b199c, off upstream main 770dfe31; not pushed, not released).** `set_gov_prop_body` now defaults the three fields to `None`. **Cause confirmed live** on Egeria `main` 05a5e982: the old code got a 400 `com.fasterxml.jackson.databind.exc.MismatchedInputException`, "Cannot deserialize value of type `java.util.LinkedHashMap` from Array value ... ServiceLevelObjectiveProperties[\"entitlements\"]", and created nothing. The fixed code created `ServiceLevelObjective::pyeg_rt_slo` (`6258d428`) with `implementationDescription` intact; it was deleted and verified gone by GUID and by name. Micro-tests: `tests/micro-tests/test_issue_156_157_catalog_target_slo.py`; the full micro-test suite (1,394) passes in a clean python:3.13-slim container. +**Fix + scope (2026-10-05):** a scan of all 42 callers of `_async_delete_element_request` found exactly two +that pre-fill a default body *and* pass a cascade flag: `CollectionManager._async_delete_collection` and +`ProductManager._async_delete_digital_product` (so `delete_digital_product(cascade=True)` was never cascading +either -- the live check's own product cleanup included). Both now pass `None` so the helper applies the flag. +`delete_digital_product`'s docstring sample also used the wrong key (`cascadedDelete`); corrected. The helper +itself is unchanged: a caller that passes an explicit body dict *and* `cascade=True` still gets the body's +value, which is documented behaviour ("body supersedes"). -**Layer:** Pyegeria (Dr.Egeria) · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, reported by the Egeria team against pyegeria 6.2.5: the command "sends a field the platform rejects", so their hourly refreshes use the content pack's existing within-an-hour SLO instead. The cause below is from reading source; it was not reproduced live, since a wrong guess would create an SLO on the shared platform. The Egeria team's error text would confirm it. +**What:** `_async_delete_collection` always pre-fills `body = {"class": "DeleteElementRequestBody"}` when none is +given and hands that dict to `_async_delete_element_request(url, body, cascade)`. +`validate_delete_element_request` applies `cascade_delete` only in its *no body* branch, so with that pre-filled dict +the flag is discarded. Captured: `delete_collection("g", cascade=True)` posts +`{'class': 'DeleteElementRequestBody', 'forLineage': False, 'forDuplicateProcessing': False}` with no +`cascadeDelete`. `MetadataExpert.delete_metadata_element(cascade_delete=True)` does send it. -**Likely cause:** `set_gov_prop_body` (`md_processing/md_processing_utils/common_md_utils.py:588`) always sets `entitlements`, `obligations` and `restrictions` to `attributes...get('value', [])`, which is `[]` when the command has no such attributes. `Service Level Objective` is in `GOVERNANCE_CONTROLS` (`common_md_utils.py:50`), so `update_gov_body_for_type` takes the controls branch, adds `implementationDescription`, and leaves the three `[]` values. Only the `TermsAndConditions`/`CertificationType`/`LicenseType` branch replaces them with maps or `None`. `body_slimmer` keeps empty lists, so `[]` is sent. In Egeria `main` (05a5e982), `ServiceLevelObjectiveProperties extends TermsAndConditionsProperties`, whose `entitlements`/`restrictions`/`obligations` are `Map`. Jackson cannot read a JSON array into a map, so the request is refused. Other governance types do not have these fields, so they silently ignore them (`ignoreUnknown=true`), which is why only SLO fails. +**Fixed in #429, but cascade still does not reach Egeria -- see ISSUE-132.** #429 made the flag reach the request +body, but under the name `cascadeDelete`. Egeria's real field is `cascadedDelete`, and it ignores unknown fields, so +the flag is still dropped, now on the server side. #429's docstring "correction" from `cascadedDelete` to +`cascadeDelete` was also the wrong way round. The `DataStructure` refusal described below is probably this bug +rather than an Egeria quirk; re-test once ISSUE-132 is fixed. -**Proposed fix (not made):** in `set_gov_prop_body`, default the three to `None` rather than `[]`, or set them only in the TermsAndConditions branch. Route `ServiceLevelObjective` through that branch (it is a TermsAndConditions subtype, not a plain control), keeping `implementationDescription`. Optionally add `Entitlements`/`Restrictions`/`Obligations` (dict-valued) to the SLO command through the Spec Editor API. Then live-test one `Create Service Level Objective` on a throwaway. +**Related, an Egeria quirk found alongside:** even with `cascadeDelete: true`, Egeria refused to delete a +`DataStructure` that still had a member `DataField` (`OMAG-GENERIC-HANDLERS-403-005 ... validateNoMemberDataFields`), +so a data contract imported by `import_data_contract*` has to be removed child-first: DataField, DataStructure, +Agreement. --- -### ISSUE-156: `AutomatedCuration.add_catalog_target` posts a flat `"class": "CatalogTargetRequestBody"`, which Egeria does not have; the endpoint takes `NewRelationshipRequestBody` - -**Fixed 2026-10-19 (owner approved) on branch `fix/issue-156-157-catalog-target-slo` (956b199c; not pushed, not released).** `add_catalog_target`/`update_catalog_target` keep their signatures but now send `NewRelationshipRequestBody`/`UpdateRelationshipRequestBody` with `CatalogTargetProperties` through the shared relationship helpers. `template_properties` now maps to Egeria's `templates` field; the old body had misnamed it `templateProperties`. **Live on Egeria `main` 05a5e982** (two runs, throwaway undeployed IntegrationConnector `pyeg_rt_ic` and a Collection target, never linked to an integration group): the old flat body was refused; the fixed add returned a relationship GUID and read back name/connection/config; update read back `pyeg_rt_target_v2` and `FROM_THIRD_PARTY` with `connectionName` kept; remove made the relationship unreadable; all throwaways verified gone. The `hey_egeria` `add_catalog_target` command uses this method, so it is fixed too. +### ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` -**Layer:** Pyegeria · **Status:** open, not fixed (owner approval needed) · **Found:** 2026-10-19, reported by the Egeria team against pyegeria 6.2.5: `add_catalog_target` "sends a request format the platform rejects"; their notebook now works around it. Confirmed from source; not re-run live. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: needs information from us.** Egeria cannot reproduce it. The error is now a clear "no asset connector" message, not a NullPointerException. Cause: the surveyed FileFolder had no ResourceConnection; the likely route is that the template create returned an existing FileFolder with the same qualified name (template create allows an existing element to be returned). They ask for: +- [ ] The request used to create the FileFolder: its template GUID and placeholder values, and whether `deepCopy` or `allowRetrieve` was set. +- [ ] The GUID of the surveyed asset, and whether that GUID existed before the create call. +- [ ] Whether that asset has a ResourceConnection, and which process first catalogued it (for example a files integration connector). -**What:** pyegeria has two `add_catalog_target` methods for the same URL, `.../integration-connectors/{connectorGUID}/catalog-targets/{elementGUID}`: -- `AutomatedCuration.add_catalog_target(integ_connector_guid, metadata_element_guid, catalog_target_name, connection_name=..., metadata_src_qual_name=..., config_properties=..., template_properties=..., permitted_sync=..., delete_method=...)` (`pyegeria/omvs/automated_curation.py:4120-4190`) builds `{"class": "CatalogTargetRequestBody", "catalogTargetName": ..., "connectionName": ..., ...}` with the properties at the top level. **Broken:** no `CatalogTargetRequestBody` class exists in Egeria `main`. -- `AssetMaker.add_catalog_target(integration_connector_guid, metadata_element_guid, body)` (`pyegeria/omvs/asset_maker.py:1095`) sends `NewRelationshipRequestBody` with `properties.class = "CatalogTargetProperties"`. **Correct.** +**pyegeria:** this fits the 2026-10-06 live check below (with `deepCopy` no longer sent, the new folder has a ResourceConnection). `create_folder_element_from_template` hardcodes `"allowRetrieve": True` (`pyegeria/omvs/automated_curation.py:1081`), so a create whose qualified name already exists returns the old, possibly connection-less folder, which matches Egeria's theory. Proposed (not done): let callers pass `allow_retrieve`, or log when the returned GUID was pre-existing. The answers above can come from the 2026-10-06 `live_79.py` run's request plus a read-only lookup of the original surveyed GUID. -Egeria (`AssetMakerResource.addCatalogTarget`, served under every generic URL marker including `automated-curation`) takes `@RequestBody(required=false) NewRelationshipRequestBody`. `CatalogTargetProperties` has `catalogTargetName`, `metadataCollectionQualifiedName`, `metadataSourceQualifiedName`, `connectionName`, `configurationProperties`, `templates`, `permittedSynchronization` and `deleteMethod`. The `hey_egeria` CLI command `add_catalog_target` (`commands/ops/gov_server_actions.py:45`, registered in `commands/cli/egeria.py` and `egeria_ops.py`) uses the broken one. +**Live check 2026-10-06 (about 02:00Z, current pyegeria 6.1.29 code, rebuilt platform): the `deepCopy:false` default WAS the cause of the missing connection.** `create_folder_element_from_template` now sends a request with no `deepCopy` key (keys sent: allowRetrieve, class, forDuplicateProcessing, forLineage, isOwnAnchor, parentAtEnd1, placeholderPropertyValues, templateGUID, templateSubstitute), and the new folder read back through the classification explorer **has 1 connection** (`ResourceConnection` present). Folder deleted (cascade) and verified gone by GUID. **Not run: the survey itself** (a survey creates elements owned by the survey identity that an ordinary user may not be able to delete), so "the survey now completes" is still unproven; the original `assetConnector`-null failure was the symptom of no connection, which is now shown not to occur. Whether the Egeria-side repair (`e01426db86`) also matters is untested. Script: `live_79.py` stage A (stage B `--survey` is opt-in). -**Proposed fix (not made):** keep `AutomatedCuration.add_catalog_target`'s signature but build `{"class": "NewRelationshipRequestBody", "properties": {"class": "CatalogTargetProperties", "catalogTargetName": ..., "connectionName": ..., "metadataSourceQualifiedName": ..., "configurationProperties": ..., "templates": ..., "permittedSynchronization": ..., "deleteMethod": ...}}` and send it through `_async_new_relationship_request`, as AssetMaker does. Add a micro-test asserting the body shape. `AutomatedCuration._async_update_catalog_target` (`automated_curation.py:4248`, body at `:4301`) has the same flat `CatalogTargetRequestBody` shape against `catalog-targets/{relationshipGUID}/update`. Egeria's `updateCatalogTarget` takes `UpdateRelationshipRequestBody`, so it needs the same fix. +**Update 2026-08-30, from the Egeria team (Mandy Chessell).** The entry +stays in "Open Egeria Server issues" — the reported failure is not +explained and not fixed. What has changed is that the cause recorded in +this entry, and in Resource Explorer's `re-as-engine-host-plan.md`, is now +known to be wrong, and the error it produces has been made legible. ---- +**Status: still open (Egeria Server)** — but the recorded cause is +disproved, and this entry and `re-as-engine-host-plan.md` should both be +corrected. Investigated 2026-08-30 from the Egeria side, including a new +FVT suite built to run this exact scenario. -### ISSUE-155: pyegeria gaps against Egeria `main` since 6.2 (omvs_audit + source diff, V6.2..05a5e982) +**The premise is false.** This entry, and the design doc's hold, rest on +"template-based asset creation never wires up a working Connection." That +is not true, in either environment it was checked against: -**Layer:** Pyegeria · **Status:** open, proposed list for the owner to pick from; nothing changed · **Found:** 2026-10-18, Resource Explorer ops item 7. File-only audit, no Egeria calls. +- **On the quickstart that produced the report.** Every element created + from a template that itself carries a `ResourceConnection` got one — 10 + of 10. Every `FileFolder` in that repository has one, including all + three that were created from a template (`/deployments/content-packs`, + `/deployments/loading-bay/sample-data`, `/deployments/secrets`). +- **In a clean environment built from the current archives.** A folder + catalogued through `FileGovernance::create-file-folder` came back with a + `ResourceConnection` attached. +- **The `FilesContentPack` template itself is also well formed:** + `FileFolder` template `fbdd8efd-...` carries an `Anchors`-classified + `Connection` with both a `ConnectToEndpoint` and a + `ConnectionConnectorType`. -**How:** `scripts/omvs_audit.py --quiet --http-dir ` was run twice against this branch's `pyegeria/omvs`: once with the `.http` files from Egeria tag `V6.2` (bc08cf55a8, 2026-10-06), once with those from `origin/main` (05a5e9822e, 2026-10-18). Each was restricted to the collections `pyegeria/http-clients/` already carries, plus the new `Egeria-api-governance-engine.http`. Totals: V6.2 OK=965, mismatch 4, missing 3. Main OK=959, mismatch 5, missing 15. The `.http` files lag the Java, so the view-service `*Resource.java` and request-DTO diffs from V6.2 to `main` were also read, for PRs #9365-#9374 and #9321. The checkouts were not touched; the files were read with `git show`. Ranked: risk to existing callers first, then new capability. +**The folder survey works.** A new `files-fvt` suite runs the reported +scenario end to end — catalogue a folder from the `FileFolder` template, +then survey it natively — and it passes: +``` +OMES-SURVEY-ACTION-0019 The survey action service folder-survey-service has completed the +analysis of asset 7050dc50-... with request type survey-folder in 1927 milliseconds; the +results are stored in survey report 7e7892e2-... +``` +The report carried six annotations. The test asserts the survey +*completed*, not merely that it ran: a survey that dies on a null +connector still produces an engine action, and the waiter throws on any +terminal status other than `COMPLETED`. -1. **Engine-action calls moved to the Governance Engine view service (PR #9373).** Automated Curation no longer has claim, status update, action-targets update, completion-status or active-claimed. On `main` the audit reports them ELSEWHERE, plus `getApprovedEngineActions` and `cancelGovernanceActionProcess` MISSING. **Already fixed on branch `fix/issue-153-governance-engine` (ISSUE-153), not yet merged or released.** The only action is to merge and release that branch. -2. **`cascadeDelete` alias (PR #9365, `DeleteOptions.setCascadedDelete` now has `@JsonAlias("cascadeDelete")`).** pyegeria's cascade flag now reaches the server, so ISSUE-154 item 1 applies: switch to `cascadedDelete` and re-test cascades. -3. **Zones fail closed (PR #9367).** A GovernanceZone with no security access control now admits no-one; before, it was ignored. This is not a REST change. The new `MISSING_ZONE_ACCESS_CONTROL` exception type is raised by the platform cataloguer, not on a REST path. Callers will see 403s or empty results for elements whose only zones are unsecured. **pyegeria:** no code change. Expect more 403s on quickstart content, as with the digital-products zone (2026-10-14). The 403 error message already reads clearly since ISSUE-142. -4. **`createMetadataElementFromTemplate` body class.** The `.http` now shows `OpenMetadataTemplateRequestBody`. The Java parameter type was already that at V6.2, so only the sample changed. pyegeria sends its `TemplateRequestBody` model (`metadata_expert.py:159-207`, `models.py:526`). That model has no `typeName`, `replacementClassifications` or `parentRelationshipProperties`, so the PyegeriaModel `extra='ignore'` rule silently drops those fields if a caller passes them. **Proposed:** add an `OpenMetadataTemplateRequestBody` model with those fields and use it here. Medium value: it is the only way to create a subtype from a template through Metadata Expert. **Partly done 2026-10-23:** pyegeria's `TemplateRequestBody` now sends `replacementClassifications` and `parentRelationshipProperties` (ISSUE-164). `typeName` (on `OpenMetadataTemplateRequestBody` only) is still open. -5. **Undo and restore (PR #9365/#9373, new).** Metadata Expert `POST /metadata-elements/{guid}/undo` restores the previous version and fails if there is none. `/metadata-elements/{guid}/restore` reverses a delete or archive, including what that delete cascaded, but not a purge. `related-elements/{guid}/undo` and `/restore` do the same for relationships. Each takes an optional `MetadataSourceRequestBody`. **Proposed:** four methods on `MetadataExpert` via the existing `_async_metadata_source_body_request` (`_server_client.py:6395`). This pairs well with ISSUE-117 and RE's archive walk: restore is the safety net for a mistaken archive. **Done 2026-10-23 (ISSUE-164 follow-up, branch `fix/role-appointment-class-names`):** `MetadataExpert.undo_metadata_element_update`, `restore_metadata_element`, `undo_related_elements_update`, `restore_related_elements`. -6. **Usage Activity and Member Popularity Profile classifications (Classification Explorer, new, probably for the Lovelace Popularity Profiler in #9369).** Each has add (`POST /elements/{guid}/usage-activity`), `/update` and `/remove`, and the same for `/member-popularity-profile`. The bodies are New/Update/DeleteClassificationRequestBody with `UsageActivityProperties` or `MemberPopularityProfileProperties`. **Proposed:** six methods, and the property names in `element_status_fields.py` if Dr.Egeria should show them. Low priority until something writes them. **Done 2026-10-23 (same branch):** `ClassificationExplorer.add/update/clear_usage_activity` and `add/update/clear_member_popularity_profile`. -7. **`GetOptions.viewElementGUID` (new).** It identifies the element whose view a request is assembling, so related elements fetched for that view are not reported as "found" (usage tracking). **Proposed:** an optional field on the get-request models, for the Resource Browser and the explorers. Low priority. -8. **`relationshipPurpose` on RelationshipDef (PR #9365).** Type definitions now carry a relationship purpose used for anchor processing. Nothing in pyegeria reads it. **Proposed:** none now. Possibly show it in the Egeria Explorer type views later. -9. **`finalAnalysisStep` (PR #9366).** This is internal to the survey connectors (a provider/service API) and not a REST field. **pyegeria:** none. -10. **Property-class renames (PR #9365):** covered by ISSUE-154 item 3. +**A second hypothesis was also eliminated.** While investigating, a real +defect was found in the PostgreSQL repository: a `TypeDefPatch` that +reparents a type does not update the denormalised supertype chain stored +on existing rows, so instances written before the patch become invisible +to searches for the new supertype. That would have hidden connections +exactly as reported, since `AssetConnection` was reparented under +`ResourceConnection` on 2026-08-27. It is fixed — but it was never the +cause here: the quickstart repository stores 202 rows as +`:ResourceConnection:LabeledRelationship:` and none as +`:AssetConnection:`, because it was reloaded after the type change. -**Audit noise (pre-existing, not gaps):** `getActorRoleByGUID` and `getSchemaAttributeByGUID` are reported as mismatches only because their `.http` URLs end in a stray `"}` (an Egeria `.http` typo; worth telling Egeria). `detachSolutionLinkingWire` is a false positive: Egeria has both `.../wired-to/{guid}/detach` and `.../wires/{relationshipGUID}/detach`, and pyegeria implements both (`solution_architect.py:4338`, `:4505`). `removeNote` is ISSUE-146. The three `product-catalog` MISSING rows are examples of one find call. `Egeria-valid-metadata-lists.http` exists only in the local copy, not on Egeria `main`. +**What has been fixed, and what it does and does not buy.** The NPE +itself is now a clean error. `getConnectorForAsset()` returns `null` by +contract when an asset has no connection; `FolderSurveyService` cast that +null and dereferenced it, and the framework's own +`performCheckAssetAnalysisStep()` had the same hole on its error path. +Both now report `OPEN-SURVEY-400-008` / `OPEN-SURVEY-0009` +(`NO_ASSET_CONNECTOR`), naming the asset. -**Also worth doing:** refresh the gitignored `pyegeria/http-clients/` from Egeria `main`. The local copy predates `Egeria-api-governance-engine.http` and the classification-explorer and metadata-expert additions. +This does not make such a survey succeed — an asset with no connection +describes a resource nothing can open. It makes the failure say *which* +asset, which is what the original report was missing: the NPE named +`BasicFolderConnector`, a class the caller had never heard of, and pointed +away from the asset. ---- +**Why that matters for how this entry reads.** Four further defects were +found in the same connectors, all on paths nothing else exercises: both +file survey services failed on every invocation (annotations created +without a qualified name, in three places), the CSV survey dereferenced a +null root schema type, wrote a relationship property to an entity, and +produced colliding annotation qualified names. The file survey path had, +on this evidence, never run to completion. Someone hitting an NPE in this +family had every reason to suspect the asset — the errors never named +anything else. That is context for the original diagnosis, not a +criticism of it. -### ISSUE-154: pyegeria follow-ons from the Egeria team's 2026-10-18 note: renamed property classes, `cascadedDelete`, the `pushDown` default, new `DataValueAssignment`/`CrowdSourcingRole` values +**The naming trap, which is directly actionable on the pyegeria side.** +This entry already records that `initiate_file_folder_survey`'s default +(`FileSurveys:survey-folder`) and the `.http` example +(`AssetSurvey:survey-folder`) are both wrong. Confirmed and now +understood: a governance action type is registered under +`::` — two colons — so the real name is +`FileSurvey::survey-folder`. `AssetSurvey` is not an engine that exists at +all. The `.http` examples have been corrected upstream +(`FileSurvey::survey-folder`, `ApacheKafkaSurvey::survey-kafka-server`, +`PostgreSQLSurvey::survey-postgres-server`, +`PostgreSQLSurvey::survey-postgres-database`) with a note stating the +shape, because the error a wrong name produces — +`OMAG-GENERIC-HANDLERS-400-013`, "the name is not recognized" — does not +say what it wanted instead. `initiate_file_folder_survey`'s default still +needs fixing in pyegeria. -**Layer:** Pyegeria · **Status:** open, proposed, not made (owner approval needed, per this repo's policy) · **Found:** 2026-10-18, from the Egeria team's note (`~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`) and read-only greps of this checkout. +**What is left, and what would settle it.** The remaining candidate is +the deployment rather than the cataloguing or the survey: which server +and userId the engine host's connected-asset client resolves +`SurveyAssetStore.getConnectorToAsset()` against. That cannot be tested +from the Egeria side — it needs the engine host that produced the +failure. Suggested, in order: -Egeria says none of its changes break existing requests: the old class names and spellings are still accepted. So nothing here is urgent, and every change should keep accepting the old names so pyegeria still works against older platforms. Ranked by risk, highest first. +1. Re-run the original repro against a build carrying the NPE fix. If the + asset genuinely has no connection the run now fails with + `OPEN-SURVEY-400-008` naming it, which turns the question from "why is + this connector null" into "why does this asset have no connection that + this userId can see". If instead the survey completes, the deployment + has moved on and this entry can be closed. +2. Check the folder asset at the moment of failure for a + `ResourceConnection`, using the same userId the engine host runs as — + not an admin one. If the relationship is there but the survey cannot + see it, that is a visibility or server-routing problem, and the userId + is where to look next. +3. Compare against `files-fvt`, which is a known-good baseline for this + exact scenario. Pointing the same two actions at a + differently-configured engine host isolates the variable. -**1. `cascadeDelete` now cascades (ISSUE-132).** Egeria accepts `cascadeDelete` as an alias for `cascadedDelete`, so any pyegeria call that sends `true` will start cascading on a platform with `oak2026`. Checked: no production caller in `pyegeria/`, `md_processing/` or `commands/` passes `True`; every `cascade`/`cascade_delete` default is `False`; `commands/tech/element_actions.py`'s `--cascade` flag (default True) is not forwarded. Only live tests pass `True`, on purpose. Proposed: serialize `DeleteElementRequestBody.cascade_delete` as `cascadedDelete` (accept both on input) and change the ~25 `"cascadeDelete"` literals (`_server_client.py` ×3, `models.py` ×7, `actor_manager.py` ×8, `metadata_expert.py`, `product_manager.py` ×2, `reference_data.py` ×2, `valid_metadata.py` ×2; several are docstring samples). Then live-test one cascade and re-test ISSUE-127. +Also worth correcting: this entry's earlier note about leftover probe +elements (`FileFolder::localhost:/tmp/re-e1-probe-*`, +`FileFolder::localhost:/tmp/re-issue79-recheck-*`) is out of date — the +quickstart repository has been reloaded since and holds none of them, so +that evidence is gone. -**2. Metadata Expert count default flipped (ISSUE-38).** Egeria's `pushDown` default is now `false` (list-matching, slower); `true` is the fast repository count. `metadata_expert.py:3698` and `:4013` document omission as `pushDown=True`. `pyegeria/view/overview_metrics.py` calls both counts without `push_down`, so its statistics silently become the slower kind. Proposed: fix both docstrings; owner to choose whether `overview_metrics` sends `push_down=True` (Egeria's advice for statistics) or keeps the list-matching default (which matches its own `len(find)` fallback). +**Status:** open (Egeria Server), reproduced live 2026-08-27, **re-checked +2026-08-28 — still reproduces, identical failure.** Same class/method +(`BasicFolderConnector.getFile()`, `assetConnector` null), same error IDs +(`OMES-SURVEY-ACTION-0018`/`OPEN-SURVEY-500-001`), same +`java.lang.NullPointerException`, against a freshly created `FileFolder` +and a fresh survey run — not a one-off. Filed by content, not by number — +see "why this entry exists" below. -**3. Renamed property classes.** Responses now carry the new names in `class`. Checked: no pyegeria response parsing compares a response's `class` with any of the renamed names. The only matches are request-side `prop=[...]` whitelists, which refuse a caller-built body that uses the *new* name: +**Layer:** Egeria Server (survey action framework / `BasicFolderConnector`), +not pyegeria. `create_folder_element_from_template` and the survey-initiate +call both succeed and are correctly formed; the failure happens entirely +inside the server's survey action service. -| Old → new | Where pyegeria uses the old name | Proposed | -|---|---|---| -| `RetentionClassificationProperties` → `RetentionProperties` | `omvs/classification_explorer.py:8851,8878` (whitelist),`8919`; `md_processing/v2/processors.py:220-237` (comment says the opposite); `md_processing/v2/curation.py:136` | Send `RetentionProperties`, accept both in the whitelist, rewrite the comment (ISSUE-128) | -| `SecurityGroupMembershipProperties` → `SecurityListMembershipProperties` | `omvs/actor_manager.py:3602,3617` (whitelist),`3654,3703,3753` | Send the new name, accept both | -| `AssetSchemaTypeProperties` → `SchemaProperties` | not used; `schema_maker.py:822` already whitelists `SchemaProperties` | none | -| `AssociatedSecurityGroupProperties` → `AssociatedSecurityListProperties` | not used; `md_processing/v2/governance.py:620` already sends the new name | none | -| `ReferenceCodeMappingProperties` → `ReferenceCodeMappingTableProperties` | not used | none | -| `GovernanceClassificationBase` → `GovernedDataClassificationBase` | only in a description string, `commands_curation_compact.json:2256` | optional wording fix through the Spec Editor API | +**Why this entry exists.** Resource Explorer's `re-as-engine-host-plan.md` +is ON HOLD citing this exact symptom ("connector is null" on a native +survey against a template-created asset) under the name `PYEGERIA_ISSUES.md` +**ISSUE-51** — but this file was renumbered on 2026-08-15, and today's +ISSUE-51 is an unrelated, already-fixed `fetch_element()` shape problem. +The blocker's own content had no surviving entry anywhere in this file. +Investigated fresh rather than guessed at, per the plan's own citation +being untrustworthy. -`tests/functional-tests/test_classification_manager.py:1233` already sends `RetentionProperties`. +**Two candidate bugs were in play, and they are not the same bug:** -**4. `DataValueAssignment` now accepts `label` and `description`.** Dr.Egeria's `Assign Data Value Specification` (`md_processing/v2/data_designer.py:1026-1037`) replaces `properties` with a fixed field list, and the `Data Value Assignment Base` bundle has no Label/Description. Proposed (feature, not a fix): add `Label` and `Description` to that bundle through the Spec Editor API, map them in the processor, run `refresh_specs`. +1. **Resource Explorer's superseded tracker, entry E1** (`packages/ + resource-explorer/docs/egeria-pyegeria-issues.md`) describes a *different* + symptom: a 500 from the PostgreSQL repository connector on + `createMetadataElementFromTemplate` (`metadata_collection_guid` null + while saving a classification) — a failure to *create* the element at + all. **Re-tested live 2026-08-27 and does not reproduce**: + `create_folder_element_from_template(path_name=..., folder_name=..., + file_system="localhost")` against `qs-view-server` returned a real GUID + with no error. Whatever this was, it looks fixed. +2. **The design doc's actual blocker** — a template-created asset surveyed + natively fails with a null connector — **does still reproduce**, and is + what this entry tracks. -**5. `CrowdSourcingRole` values now work** (PROPOSER, REVIEWER, SUPPORTER, APPROVER, OTHER). pyegeria's `link_crowd_sourcing_contribution` (`omvs/community_matters_omvs.py:479`) passes the body through, and no Dr.Egeria command sets a role. Proposed: list the values in the docstring; nothing else. +**Reproduction** (against `qs-view-server`, no pyegeria bug involved): +```python +from pyegeria import AutomatedCuration +c = AutomatedCuration("qs-view-server", "https://localhost:9443", "erinoverview", "secret") +c.create_egeria_bearer_token() -**6. ISSUE-147's 400-011 workaround:** none exists. Nothing to drop. - -**7. View-service fixes pyegeria may have worked around.** The note lists 48 view-service defects found by Egeria's new view-services FVT. A read-only sweep of `pyegeria/`, `md_processing/` and `commands/` (2026-10-18) found almost nothing worked around, and no issues-file entry for any of them. The detach, Classification Explorer, Metadata Expert, search-option, history-order and delete/purge rows have no client-side workaround: the wrappers call the endpoint once and pass options straight through. So pyegeria users met those defects as calls that "succeeded" but did nothing, and the fixes need no pyegeria change. Candidates to review: -- **Project hierarchy:** `commands/cat/get_project_structure.py:105-116` (and `get_project_dependencies.py:107`) walk the tree with `get_linked_projects()` and keep only `ProjectHierarchy` links, instead of calling `ProjectManager.get_project_hierarchy`. Not commented as a workaround. Optional: switch to `get_project_hierarchy` after re-testing it on a platform built from `main`. -- **Term classifications with no body:** `glossary_manager.py:2686-2692` (is-question) and `4059-4062`/`4116`/`4170` (prime word, modifier, class word) always send a default body. The comment blames pyegeria's own helper (ISSUE-31), not the server, so keep them; harmless either way. -- **`linkSolutionLinkingWire` with no body:** `md_processing/v2/solution_architect.py:856-860` always sends a body and skips recording the GUID when none returns. Keep it; no change needed. -- **Asset from template:** Dr.Egeria's generic `Create Element` (`md_processing/v2/asset_maker.py:57-75`) uses Automated Curation `catalog-templates/new-element`, not Asset Maker `assets/from-template`, so it never hit the "always an Asset" bug. No change. -- **Missing connector now refused** (Security Officer, Runtime Manager): calls that used to report success with nothing done will now raise. Callers should expect the error. ISSUE-112 is the same shape for client-side secrets. -- **Delete with `forLineage`:** a delete that needs `forLineage=true` (Memento) is now refused whole instead of half-done. pyegeria's editing deletes already default to `forLineage=true` via `lineage_visible()`, so no change. - ---- - -### ISSUE-152: five update methods pass the body into `_async_update_element_body_request`'s `prop` slot, so nothing is sent - -**Layer:** Pyegeria · **Status:** open (found 2026-10-15 by Resource Explorer's ClassificationAnnotation backfill; not fixed, per this repo's policy) +folder_guid = c.create_folder_element_from_template( + path_name="/tmp/some-new-path", folder_name="probe-folder", file_system="localhost") -**Fixed 2026-10-20 (owner approved) on branch `fix/issue-152-update-body-slot`, not yet merged or released.** All five call sites now pass `body=body`. `tests/micro-tests/test_issue_152_update_body_slot.py` sends a dict body and an `UpdateElementRequestBody` through each method and checks that the body reaches the HTTP layer. All 10 cases fail on the old lines and pass on the fix, and the full micro-test suite (1,404) passes in a clean python:3.13-slim container. Not live-tested. Hit again 2026-10-20 by the TestGen evaluation session (RE #791, `update_annotation`), which carries its own workaround until a release ships this. +# Note: initiate_file_folder_survey's own default survey_name +# ("FileSurveys:survey-folder") and the .http ground truth's example +# ("AssetSurvey:survey-folder") are BOTH wrong on this server -- neither +# GovernanceActionType is registered (OMAG-GENERIC-HANDLERS-400-013). The +# real registered qualifiedName, confirmed via +# `EgeriaTech.get_elements("GovernanceActionType")`, is "FileSurvey::survey-folder". +action_guid = c.initiate_file_folder_survey(folder_guid, survey_name="FileSurvey::survey-folder") +``` -`_async_update_element_body_request(self, url, prop=None, body=None)` (`pyegeria/core/_server_client.py:7317`) takes the body as its **third** parameter. Five callers pass it positionally as the second, so it lands in `prop` and `body` is `None`; the call fails before any request goes out (the builder observed an AttributeError): +Poll the resulting `EngineAction` (`EgeriaTech.get_elements("EngineAction")`, +match on `elementHeader.guid`) — it completes (fast, ~1s) with +`activityStatus: "FAILED"` and `completionGuards: ["survey-failed"]`. Full +`completionMessage`: +``` +OMES-SURVEY-ACTION-0018 The survey action service folder-survey-service +threw a org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectorCheckedException +exception during the generation of survey report ... for asset + during request type survey-folder in survey action engine +FileSurvey (guid=4168abb9-6c60-46fb-b9c0-b44180d19500). The error message +was OPEN-SURVEY-500-001 Unexpected exception in survey action service +folder-survey-service of type java.lang.NullPointerException detected by +method start. The error message was Cannot invoke +"org.odpi.openmetadata.adapters.connectors.datastore.basicfile.BasicFolderConnector.getFile()" +because "assetConnector" is null +``` -- `pyegeria/omvs/data_discovery.py:299` (`_async_update_annotation`, behind `update_annotation`) -- `pyegeria/omvs/time_keeper.py:342` -- `pyegeria/omvs/schema_maker.py:131` and `:174` -- `pyegeria/omvs/subject_area.py:302` +**Analysis.** `create_folder_element_from_template` creates the `FileFolder` +element and it looks entirely normal (confirmed via `get_elements`: proper +`pathName`, `resourceName`, `Anchors` classification) — but no working +`Connection`/`Connector` got wired to it, so when `folder-survey-service` +tries to open the asset via `BasicFolderConnector.getFile()`, the +connector reference is null and the survey action service crashes with an +NPE rather than a clean error. This matches the design doc's +characterization exactly ("template-based asset creation never wires up a +working Connection, so any native survey against a template-created asset +fails server-side with a NullPointerException") — just with the precise +class/method now on record, which the design doc didn't have. -Checked at egeria-python `f1b06cdd`; the installed wheel in trellis has the same line in `data_discovery.py`. +**Impact:** blocks Resource Explorer's engine-host participation design +(`re-as-engine-host-plan.md`, cases 1/2/4 — anything needing a native +survey to actually *complete*, not just be triggered; case 3, RE-local +surveying, is unaffected). The hold should be re-evaluated against this +entry specifically, not against the old ISSUE-51 number. As of 2026-08-28 +this is confirmed the *only* thing still blocking that work on the +pyegeria side — ISSUE-78's engine-host method trio shipped in 6.1.5 and RE +has upgraded and verified it end to end, and the two other gaps +identified alongside it (no REST way to create a `GovernanceEngine`/ +`GovernanceService` element; no per-engine "claimable work" listing, +`get_active_engine_actions()` filtered client-side is the only option) are +both genuine server-side gaps, not pyegeria ones. This entry is the real +holdup. -**Fix (one token each):** `await self._async_update_element_body_request(url, body=body)`. A test per method that the body reaches the HTTP layer would catch a repeat. +**Leftover from this investigation:** probe `FileFolder` elements +(`qualifiedName: "FileFolder::localhost:/tmp/re-e1-probe-*"` from +2026-08-27, `"FileFolder::localhost:/tmp/re-issue79-recheck-*"` from the +2026-08-28 re-check) on `qs-metadata-store`, plus their failed +`SurveyReport`/`EngineAction` pairs, left in place as reproduction +evidence for both dates — not yet deleted. -**Workaround in Resource Explorer:** `scripts/backfill_classification_annotations.py` calls `_async_update_element_body_request(url, body=body)` directly, and pins this bug with a test that fails once pyegeria is fixed. +**Update 2026-10-05 -- probably a pyegeria bug, fixed on our side:** the Egeria team asked us to check that pyegeria did not send `deepCopy:false`. It did: `TemplateRequestBody.deep_copy` defaulted to `False` from 2026-01-09 until fixed on 2026-09-29 (#399, first released in v6.1.23), the model is serialised with `exclude_none` (which keeps `False`), and `create_folder_element_from_template` builds its body without `deepCopy`. Egeria then copies only the top-level element and none of the template's anchored attachments, so no `ResourceConnection`, which is exactly a null `assetConnector`. The Egeria team also points to `e01426db86` (2026-08-30, PostgreSQL supertype-chain repair after the 08-27 `ResourceConnection` re-parenting). **To confirm:** create a folder from the template with current pyegeria, check it has a `ResourceConnection`, then survey it. --- +### ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55) -### ISSUE-150: Dr.Egeria commands for an Analytic Function's Egeria declaration: a GovernanceActionType that Resource Explorer runs as an engine host, plus a synchronous path that bypasses Egeria +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. Metadata Expert counts now take `pushDown`, **default `false`** (count matches the list); `true` gives the fast repository count. Relationship counts with end GUIDs and no criteria used to return 0; fixed. Egeria recommends `pushDown=false` for user-facing counts and `pushDown=true` for statistics. **pyegeria (proposed, not done):** `MetadataExpert.count_metadata_elements`/`count_relationships_between_elements` docstrings (`metadata_expert.py:3698`, `:4013`) say omitting `push_down` means `pushDown=True`; that is now wrong. Decide what `pyegeria/view/overview_metrics.py` (which omits `push_down`) should send, since its counts will become the slower list-matching kind once the platform has this. Verification below still needs running on a platform with the fix. -**Layer:** Dr.Egeria (commands over existing Action Author / governance commands) · **Status:** open, direction agreed with the owner 2026-10-13 and **confirmed by Mandy 2026-10-14**, not yet built · **Found:** 2026-10-13, handed off by the Resource Explorer PR/CI session. Source: trellis `docs/extensibility-walking-skeleton-brief.md` §2c, §3a, §8. Egeria references: [Governance Action Processes (0462)](https://egeria-project.org/types/4/0462-Governance-Action-Processes/), [Survey Action Engine](https://egeria-project.org/concepts/survey-action-engine/). +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** Counts against the paged `ClassificationExplorer.get_relationships` total: `Exception` gives push_down omitted 146, false 146, true 148, list 146; `SemanticAssignment` gives 563, 563, 583, list 563. The default is now `false` and matches the list. The fast count is higher by what erinoverview cannot see (2 and 20). Read-only. The docstring fix in ISSUE-154 item 2 is now confirmed needed. -**Background (Egeria's model):** -- A `GovernanceActionType` is a template for **one** action. Initiating it creates one Engine Action. -- A `GovernanceActionProcessStep` is a subtype of `GovernanceActionType` that exists only inside a process. Its flow links (`GovernanceActionProcessFlow`, `NextGovernanceActionProcessStep`) carry guards and mandatory guards. Each step that fires becomes its own Engine Action. -- Both reach their code through `GovernanceActionExecutor` (a governance engine plus a `requestType`). -- Surveys use the `SurveyingActionProcess` subtype. Survey action services write annotations into a Survey Report linked to the asset. +**Update 2026-08-30, from the Egeria team (Mandy Chessell).** Leaving this +entry in "Open Egeria Server issues" until the verification below has been +run against real data — the Egeria side is done, but this entry has been +re-measured and re-confirmed several times, and it shouldn't move to +Fixed/Resolved on the strength of a change nobody has yet pointed at the +data that produced the original report. -**Direction (owner, 2026-10-13):** use Egeria's designed governance-action infrastructure, not a bypass; it may need changing, but it should be used. Resource Explorer **is an engine host** and can present itself to Egeria as one. The owner's concern is timing: running an action or process is asynchronous and possibly long-running, while an analytic function often has to answer synchronously during a screen draw or report generation. So declaring a function is kept separate from running it: +**Status: resolved on the Egeria side 2026-08-30** (`omf-metadata-spring`, +`omf-metadata-server`, `generic-handlers`, and the OMF Java client) — +**awaiting verification from the pyegeria side**, see below. The two +numbers are not going to be made equal, because they are answers to two +different questions and both are wanted; what has changed is that the +caller now chooses which one they get. -1. **Declaration (in Egeria):** a `GovernanceActionType` per function. - - A `GovernanceActionExecutor` links it to Resource Explorer's governance engine, with the function's name as the `requestType`. - - It carries the ISSUE-149 manifest fields: `kind`, `version`, `source`, `resource_types`, `annotation_type`, `family`, `tags`. - - It joins its family `Collection`. - - Retirement is a status change, never a delete. +**What the difference actually was.** `findAttachmentLinks()` puts every +relationship it retrieves through a visibility check on the entities at +both ends, and silently drops the ones whose anchor the caller cannot +read. `countAttachmentLinks()` skipped that check — deliberately, and +documented as such in its own javadoc, because applying it means fetching +every matching relationship, which is the cost the count exists to avoid +— and also ignored `effectiveTime`. So the count was of what matches the +search and the list was of what the caller may see. That answers this +entry's open question ("what does the metadata-expert count include for +`Exception` that the classification-explorer traversal excludes?"): +relationships with an end whose anchor is not readable by the calling +user, plus anything outside the effectivity window. It also explains why +only `Exception` diverged among the types tested — it is the type whose +ends are most likely to have been deleted or anchored to something the +caller cannot see — and why the divergence tracked the demo data rather +than staying at a fixed size. - Within a survey it is used as a `GovernanceActionProcessStep` in a `SurveyingActionProcess`. A single run can be wrapped as a one-step process if a uniform "everything runs as a process" shape is wanted. -2. **Asynchronous run (through Egeria):** initiate the type, or a process containing it, and an Engine Action is created. Resource Explorer, as an engine host, claims it and runs it, using `engine-actions/{guid}/claim`, `status/update`, `completion-status` and the active-claimed listing. Results are stored as annotations carrying their envelope, and Egeria keeps the run record. This path is for surveys and anything slow or scheduled. Mandy's per-engine "approved, unclaimed" engine-action listing (2026-10-05, governance-context-service only so far; see ISSUE-151) is what such an engine host polls. -3. **Synchronous run (no Egeria round-trip):** a screen or report calls the installed implementation directly, as a report spec's `analytic_function` already does through `exec_report_spec` (ISSUE-137/138/139/140/149 made this path work for trellis routines). - - Egeria is not on the hot path; the declaration only identifies which function and version is meant. - - **Synchronous runs are not recorded in Egeria** (owner: lineage doesn't make sense for what is drawn on a screen). Diagnostic logging at most; still to discuss. +**The new option.** Both counting endpoints take a `pushDown` query +parameter, `required=false`, default `true`: +``` +POST .../users/{userId}/relationships/by-search-conditions/count?pushDown=false +POST .../users/{userId}/metadata-elements/by-search-conditions/count?pushDown=false +``` +- `pushDown=true` (the default, and exactly the existing behaviour) — the + repository counts the matching rows itself. Fast, and counts what + matches the search. +- `pushDown=false` — the relationships are retrieved and counted, so the + answer agrees with the list by construction, at the cost of reading + every one of them. -**How a screen uses it:** for each function, one of three choices: -- compute now (cheap functions); -- read the last stored result (expensive functions; the envelope's `measured_at` says how old it is); -- request an asynchronous refresh that the next draw picks up. +Nothing changes for a caller that does not send the parameter. The same +choice is on the Java client (`countMetadataElements(..., pushDown)` and +`countRelationshipsBetweenMetadataElements(..., pushDown)` on +`OpenMetadataStore` and `OpenMetadataClient`), with the existing +signatures kept and defaulting to `true`. -"Routine versus step" therefore becomes a property of each function (is it cheap enough to compute on read?), possibly a manifest field, not two kinds of element. +**The test the pyegeria side should run to prove it.** Against the +environment that produced the original numbers, using whatever the +current counts are rather than the 58/57 recorded here — the demo data +has moved several times during this entry's life, and what matters is the +relationship between the numbers, not their values: -**Keeping the two paths honest:** the code run synchronously must be the code the declaration names, at the same version, that Resource Explorer's engine runs. trellis's planned `trellis-analytics drift` (installed vs declared) checks this. The envelope's `producer_version` lets readers tell versions apart. +1. **The accurate count should equal the list.** For + `relationshipTypeName: "Exception"`, the count with `pushDown=false` + should equal `len(ClassificationExplorer.get_relationships("Exception"))` + exactly. This is the assertion that closes the entry — it is the + discrepancy this entry is about, stated as an equality. +2. **The fast count should be unchanged.** The same call with + `pushDown=true`, and with the parameter omitted altogether, should both + return what the count returns today. If either moves, something + regressed for every existing caller. +3. **A type that already agreed should still agree both ways.** + `SemanticAssignment` matched at 401=401 when this entry was last + measured. Both `pushDown` values should return that same number — the + retrieval route must not lose or double-count anything on a type where + there was never a discrepancy to resolve. -**Dr.Egeria work this implies (to be confirmed with the RE sessions):** -- Mostly convenience over existing commands: `Create Governance Action Type`, `Link Action to Action Executor` (`GovernanceActionExecutor`), `Link First/Next Process Step`, and family-collection membership. -- Plus the manifest attributes. Where these live on the element (`additionalProperties`, or real properties if Egeria adds them) is still to decide. -- Possibly a `Create/Update/Retire Analytic Function` alias family that sets all of this in one document, which `trellis-analytics declare` would generate. And `View Analytic Functions [family]`. -- The Action Author processors (`ActionExecutorTargetLinkProcessor`, `ActionProcessStepLinkProcessor`) already handle the links. `register_governance_processors()` is family-name-gated, so a new family name must be added there. +Worth capturing while verifying: the difference between the two counts is +the number of matching relationships the calling user cannot fully see. +If it is not the 1 recorded here, that is not a failure — it is a +measurement of the current data — but it is worth writing down, because a +large difference on a type that should be fully visible would be a +separate finding. If test 1 comes out unequal, the remaining difference is +not the visibility check and this entry should stay open with both +numbers recorded. -**Open points:** -- Whether Egeria needs changes, for example to let a synchronous-safe flag or the manifest fields be first-class properties. -- Whether a single run is always wrapped as a one-step process. -- Whether routines that are only ever computed synchronously need a declaration at all, beyond the Report element (`Create Report`). -- Diagnostic logging for synchronous runs. +**Follow-on for pyegeria, once verified:** the Overview dashboard's +current workaround — keeping relationship counts on +`ClassificationExplorer.get_relationships` while using native counting +only for element counts — can be revisited. +`count_relationships_between_elements(..., pushDown=False)` gives the same +number as the traversal without materialising the list client-side. Keep +in mind that `pushDown=false` reads every matching relationship +server-side, so it is the right choice for a number a user is going to +compare against a list they can also see, and the wrong choice for a +headline figure over a large type where speed is what matters. That +trade-off is now the caller's to make, which is what this entry was +ultimately asking for. +**Egeria-side test cover:** `CountPushDownFVT` in +`open-metadata-test/open-metadata-fvt/query-fvt` checks that both routes, +over REST and through the Java client, agree when every matching element +is visible, and that omitting the parameter changes nothing. It +deliberately does not attempt the case where they differ — that needs an +element the calling user cannot read, which that suite cannot arrange. +Which is why the verification above, against real data, is the part that +actually proves the fix. -**Resource Explorer's answers (RE PR/CI session, 2026-10-13):** -- **(a) The direction matches RE's engine-host design** (trellis branch `re/design-engine-host`, `packages/resource-explorer/docs/design-notes/DESIGN-RE-ENGINE-HOST-BY-EQUIVALENCE.md`). Its first slice (E1) is exactly the declaration above: an RE governance engine, a `SupportedGovernanceService` whose requestType names the trellis step, and a `GovernanceActionType` with a `GovernanceActionExecutor` to that engine plus `producedGuards`. RE claims and completes Engine Actions. - - Owner decisions recorded there: the engine runs as RE's service account. An action on a resource in a private investigation is left unclaimed by the unattended host and run attended, as the owner. RE may claim and hold future-dated actions. - - RE polls `get_active_engine_actions()` today, filtered client-side to its engine and APPROVED (or `find_engine_actions`, see ISSUE-151). Mandy's per-engine "approved, unclaimed" listing exists only on the metadata server's governance-context service, so RE is asking her to expose it on the View Server. - - Questions open with Mandy: the engine subtype (`SurveyActionEngine`?); whether a GovernanceService whose connector class nothing can load is acceptable; and where guards come from (standard survey guards, or declared by the step from the envelope). -- **(b) RE's survey definitions today** are Dr.Egeria `Create Governance Action Process` (`GovActionProcess::`) plus `Create Governance Action Process Step` (`GovActionProcessStep::::`), linked with First/Next Process Step. They are plain `GovernanceActionProcess`, not `SurveyingActionProcess`: Dr.Egeria's command has `OM_TYPE` `GovernanceActionProcess` and offers no subtype choice. The steps carry **no executor**; RE's own executor reads `executes_at`/`re_analysis_step`/`supported_technology_type` from `additionalProperties`. That is why Egeria can't dispatch any RE step today. Moving the steps onto an executor is part of RE's engine-host slice. -- **(c) Manifest fields go in `additionalProperties` for now,** with stable keys named exactly like the `AnalyticFunctionSpec` fields (`kind`, `source`, `version`, `envelope`, `annotation_type`, `resource_types`, `family`, `tags`). This matches the owner's interim ruling for annotations. Asking Egeria for real properties is a later item, once a few functions exist. -- **Synchronous runs not recorded:** agreed. RE's OpenLineage run events (branch `re/openlineage-run-events`, off by default) cover only step runs inside RE's survey executor, so a screen-draw run never reaches them. +**Update 2026-09-02 — pyegeria side implemented, verification attempted +and blocked (not failed) by a stale deployed platform image.** -**Dr.Egeria gaps this exposes (part of this issue's work):** -1. `Create Governance Action Process` should be able to create the process subtypes (`SurveyingActionProcess`, `CataloguingActionProcess`, ...), e.g. via a `Process Type` attribute validated against the 0462 subtypes. Today it always creates the plain supertype. -2. A process step needs its `GovernanceActionExecutor` link and its manifest `additionalProperties`. Check that `Link Action to Action Executor` accepts a `GovernanceActionProcessStep` (a subtype of `GovernanceActionType`) as its subject. It should, but this hasn't been live-verified (see the Action Author note in CLAUDE.md). -3. Declaring the engine side may need `SupportedGovernanceService` / governance-engine commands, if Dr.Egeria doesn't already have them. Check before building. +`push_down: Optional[bool] = None` added to +`MetadataExpert.count_metadata_elements`/`_async_count_metadata_elements` +and `count_relationships_between_elements`/ +`_async_count_relationships_between_elements` (`pyegeria/omvs/ +metadata_expert.py`) — `None` omits the `pushDown` query param entirely +(today's existing behaviour, zero change for any existing caller); `True`/ +`False` append `?pushDown=true`/`?pushDown=false` explicitly, following the +same `if : url += f"&{param}=..."` conditional-append convention +already used elsewhere in this codebase (e.g. `valid_metadata.py`'s +`typeName` handling, ISSUE-82/PY-25). -**Owner's answers on these gaps (2026-10-13):** -1. Process subtypes: create new Dr.Egeria commands for the other process types where needed. -2. Executor on a process step: yes, it needs live testing. -3. Engine-side commands: yes, likely needed. +Ran this entry's own 3-test verification plan live-patched into +`quickstart-pyegeria-web` (site-packages `docker cp` + restart, diff- +confirmed byte-identical), against `qs-metadata-store`: -**Requirements proposed by RE's expansion-design session (2026-10-13; it found no contradiction with the direction). Owner's review, the same day: 1 and 2 accepted; 3 kept as an informational key only:** -1. **One declaration, idempotent.** The manifest's qualifiers must live on the `GovernanceActionType` itself (`supportedRequestParameters` or `additionalProperties`), so Egeria, the Portal and RE all read the same declaration. That means question ids, perspectives, `purpose` (from the existing `VALID_PURPOSES` set) and `family`. Declaring is idempotent by qualifiedName: declaring twice updates, and never creates a second type. Dr.Egeria's upsert (Create→Update by qualified name) already behaves this way. -2. **Label synchronous values differently from recorded ones.** On a screen, a value computed now must be labelled "computed now · not recorded", and a stored result "last recorded ". A reader must never mistake a synchronous draw for a governed measurement. (The envelope gives the state and `measured_at`; the "recorded or not" label is the caller's, from which path it used.) -3. **State the credential tier.** User-written steps stay on the resource's read-only tier (owner ruling, 2026-10-07). A `GovernanceActionType` declared for one must say so, with a property naming the tier (e.g. `credential_tier: read_only`). **Owner: an informational key, not a control.** Egeria schedules Engine Actions the same way regardless, and never hands a step its resource credentials. The engine host does: RE's wrapper enforces read-only from the step's manifest (`StepManifest.credential_tier`). The key on the declaration is for governance visibility (whoever reviews or approves the type can see it), and for any other engine host that might run the step later. The design session confirmed this when asked (2026-10-13). It withdrew its "control" reading: the only reader other than RE's wrapper would be a second engine host, which enforces the tier itself. An Egeria-side approval step or governance rule that reads the tier would be new design nobody has asked for. +``` +count (pushDown omitted): 58 +count (pushDown=true): 58 +count (pushDown=false): 58 +ClassificationExplorer.get_relationships("Exception") length: 57 +``` -**Mandy's answer (2026-10-14, via the owner and the RE PR/CI session); the declaration direction is confirmed, no longer provisional:** -- **Engine type:** `SurveyActionEngine` for surveys. RE may also expose services through a `GovernanceActionEngine` (general) or a `WatchdogActionEngine` (monitoring). -- **What to create:** a governance service definition for each service, of the subtype matching the engine: a **`SurveyActionService`** for a `SurveyActionEngine` (survey steps), a `GovernanceActionService` for a `GovernanceActionEngine`. Plus the `GovernanceActionExecutor` relationship to the engine element. (Owner's correction, 2026-10-14: not a plain `GovernanceService`.) -- **requestType:** the value the engine advertises, and the value it's passed when asked to execute. RE dispatches on it. -- **Guards:** standard survey guards are best practice, not mandatory. -- **Coming in a few days:** more engine-host internal APIs, including the per-engine approved-actions listing, will be exposed as View Server REST calls once her current fixes are done. pyegeria will need wrappers for them (BACKLOG.md). -- Refs: [Survey Action Engine](https://egeria-project.org/concepts/survey-action-engine/), [0461 Governance Engines](https://egeria-project.org/types/4/0461-Governance-Engines/), [0462 Governance Action Processes](https://egeria-project.org/types/4/0462-Governance-Action-Processes/). +Test 1 (accurate count == list) came out **unequal** (58 vs 57) — but +before treating that as "the visibility check isn't it," checked whether +the platform under test actually has the 2026-08-30 fix at all. First +pass looked at the running image's `build-date` label (2026-08-24, six +days before the merge) and Docker Hub's manifest digest for `:latest`, +which matched what was already deployed — reasonable evidence, but not +airtight, since Docker Hub then showed the `latest` tag re-pushed +2026-08-31 (label alone can't tell "same app, new base layer" from "new +app build that forgot to bump the label"). Settled it properly: pulled +that exact 2026-08-31 digest +(`sha256:8ff341b7cf1b440f7aa4db991258afd70758f15e803c43db9bd40b41f3a0b9e5`) +and diffed the actual server jar against the one already deployed — +`deployments/omag-server-platform-6.2-SNAPSHOT.jar`, **identical +SHA-256 in both images** (`38d3512e1ab66c0d66c94bebf09e121176fa8643 +b6d5fe2fc16400e26d8afda0`). The 2026-08-31 push is a container-image +rebuild (base OS/security layer) around the byte-for-byte same, +still-pre-fix application jar — not a new Egeria build. So: confirmed, +not just inferred from a label — **no published image yet includes the +2026-08-30 server-side merge.** The server accepts `?pushDown=false` +(200, no error — an unrecognized query param is simply ignored, not +rejected) but it has no effect: all three counts come back identical, +matching the exact pre-fix numbers this entry already had on record. +Test 3 (`SemanticAssignment`, an already-agreeing type) still returned +170=170 either way, as expected regardless of whether the platform has +the fix, so that one doesn't help distinguish "not deployed yet" from +"deployed and still broken." + +**This is a blocked verification, not a failed one** — per this entry's +own "if test 1 comes out unequal" clause, the honest reading given the +platform-date mismatch is "cannot yet tell," not "the fix doesn't work." +Leaving this entry in Open Egeria Server issues, as before. **Re-run once +a platform image built after 2026-08-30 is available** — the pyegeria-side +`push_down` parameter is ready and requires no further client-side work; +only the deployed Egeria server needs to catch up. + +**Status:** re-confirmed 2026-08-18, byte-for-byte identical to the +2026-08-15 numbers below — `count_relationships_between_elements("Exception")` +still returns 58, `get_relationships("Exception")` still returns 57, same +off-by-one. No change. + +**Status:** re-investigated 2026-08-15, narrowed and re-confirmed +**Egeria server, not pyegeria-fixable** — this is a genuine, real bug, just +smaller than originally measured (demo data has evidently changed since +2026-07-24: `count` is now 58 and `get_relationships` is now 57, not +276/55). Cross-checked against a **second, independent pyegeria list-based +method** — `MetadataExpert.find_relationships_between_elements` (a +different OMVS client, different endpoint, +`.../relationships/by-search-conditions` rather than +`ClassificationExplorer`'s `.../relationships/{type}`) — with the same +`relationshipTypeName: "Exception"` filter: it also returns exactly 57, +and its 57 relationship GUIDs are the **identical set** to +`get_relationships`'s 57 (zero difference either direction). Two +independently-implemented pyegeria code paths through two different +endpoints agree exactly with each other; only the server's native +`COUNT(*)` endpoint (`.../relationships/by-search-conditions/count`) +disagrees, by exactly 1. Also re-confirmed the original "not the type +filter" finding still holds with current data — re-ran `count` vs. +`get_relationships` for `SemanticAssignment` (401=401), `License` (2=2), +`Certification` (0=0), `AttachedRating` (0=0): all match; only `Exception` +diverges. Also checked whether the count includes non-`ACTIVE`/soft-deleted +relationships the list endpoints filter out — no: `limitResultsByStatus: +["ACTIVE"]` and `["ACTIVE","DELETED"]` both still return 58 from `count` +(no status-filter difference). Since both retrieval paths agree with each +other and only the count differs, there's genuinely nothing to fix on the +pyegeria side — this is squarely a server-side discrepancy between the +native COUNT(*) query and the list-materialization logic for this one +relationship type. No further pyegeria action needed; still needs +Egeria-side investigation to say what specifically triggers it for +`Exception`. + +**Original status:** open (Egeria server) — needs Egeria-side investigation. Found +2026-07-24 while wiring the Egeria Overview dashboard to native counting +(odpi/egeria#9168). Consolidated in from `egeria-workspaces-fs/ +PYEGERIA_ISSUES.md` 2026-08-05. + +**Summary:** the OMF metadata-expert native relationship count and the +classification-explorer `get_relationships` return materially different +totals for the `Exception` relationship type — and *only* that type, among +those tested. + +**How to trigger:** +```python +from pyegeria import MetadataExpert, ClassificationExplorer +me = MetadataExpert(view_server="qs-view-server", platform_url="https://localhost:9443", + user_id="erinoverview", user_pwd="secret"); me.create_egeria_bearer_token() +ce = ClassificationExplorer(view_server="qs-view-server", platform_url="https://localhost:9443", + user_id="erinoverview", user_pwd="secret"); ce.create_egeria_bearer_token() + +me.count_relationships_between_elements({"class":"FindRelationshipRequestBody","relationshipTypeName":"Exception"}) +# -> 276 +len(ce.get_relationships(relationship_type="Exception", output_format="JSON", start_from=0, page_size=5000)) +# -> 55 (all 55 have exact typeName "Exception"; no effectivity dates) +``` + +**What it is NOT:** +- Not the type filter — `count("SemanticAssignment")` = 397 = + `get_relationships` = 397; `License` 2 = 2; `Certification` 0 = 0; + `AttachedRating` 0 = 0. Every other tested type matches; only `Exception` + diverges. +- Not status/effectivity — `count("Exception")` is 276 with + `limitResultsByStatus=[ACTIVE]`, with `effectiveTime=`, and with + neither; the 55 `get_relationships` results carry no + `effectiveFromTime`/`effectiveToTime`. +- `count(no relationshipTypeName)` = 31857 (all relationships), so 276 is + a genuine type-scoped subset, not "count ignores the filter". + +**Open question for Egeria:** what does the metadata-expert count include +for `Exception` that the classification-explorer traversal excludes +(subtypes counted under the supertype? relationships to non-visible/ +anchored/dangling ends? access/zone filtering that differs between the two +OMVS)? Whichever is "true", the two APIs should agree for a given type — +or the difference should be documented. + +**Impact / workaround:** egeria-workspaces-fs's Overview dashboard keeps +**relationship** counts on `ClassificationExplorer.get_relationships` (so +"Open Exceptions" stays consistent with the Audit app at 55) and uses +native counting only for **element** counts. -**What exists today for the engine side (gap 3, checked 2026-10-14):** -- The SDK can link a service to an engine: `AssetMaker` `governance-engines/{guid}/supported-governance-services/{guid}/attach`, plus `update`/`detach` (`asset_maker.py` ~1693-1813 and ~6586-6803). -- Dr.Egeria has **no** commands to create a `GovernanceEngine` (or its `SurveyActionEngine`/`GovernanceActionEngine`/`WatchdogActionEngine` subtypes) or a governance service (`SurveyActionService`/`GovernanceActionService`/...), and none for the supported-service link. Only `Link Action to Action Executor` exists (Action Author). -- These are the Dr.Egeria commands to add: `Create Governance Engine` (with an engine-type attribute), `Create Governance Service` (with a service-type attribute matching the engine: `SurveyActionService` for survey steps), and `Link Governance Service to Engine` (with requestType). Check the 0461 types and the `.http` files for the create endpoints before building. --- +## Open Egeria Server issues (not fixable in pyegeria) -### ISSUE-146: Deleting a journal entry fails with OMAG-REPOSITORY-HANDLER-400-008, and pyegeria's `remove_note` reaches the delete only through the generic Asset Maker service +These need an Egeria Server-side fix, an upstream bug report, or a +deployment/config change — nothing here can be resolved by editing this +repo. Where an entry has a known pyegeria/Dr.Egeria follow-on once the +Egeria side lands, it's noted inline (also cross-referenced from the +"Open pyegeria items" section below where the follow-on is substantial +enough to track there too). -**Egeria reset 2026-10-23 ~21:10-23:54Z** (egeria-workspaces-fs, owner-approved): the quickstart database was wiped and Egeria recreated on build 81beb8e5 (odpi/egeria 4486d1a7), then Coco content was reloaded. Every GUID recorded before then is gone. The test JournalEntries `fef83ed5` and `8bec72e4` on Erin's profile no longer exist, so the pending `remove_note` re-test below needs a NEW entry made with `journal_my_activity` (and, for the 400-008 case, one owned by an external source). The 2026-10-18 Egeria analysis (works as designed; 400-008 is the provenance check) still applies. +--- -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: works as designed (Egeria).** Asset Maker's `deleteAsset` is the right call: a JournalEntry is a Notification, so an asset, and no separate note delete is needed. Egeria's new FVT creates entries with My Profile's `journal-my-activity` and deletes them with `deleteAsset`, with no external source; every case passes on in-memory and PostgreSQL repositories. **400-008 is the provenance check:** it refuses a change made without an external source to an element an external source owns. No external source → 400-008; a different external source → 400-007; the matching external source → success. So an entry that returns 400-008 was not created locally, and its delete needs that entry's `externalSourceGUID` and `externalSourceName`. If 400-008 appears on an entry created through My Profile, that is a bug and Egeria wants: (1) the full error message (it names the type, GUID, provenance and metadata collection of the refused instance); (2) how the entry was created (which request, and which external source if any). Also new: purge applies the same provenance check. **pyegeria:** the `remove_note` clean-up below stays (route + docstring). Re-test on `8bec72e4` against a platform with the fixes; if 400-008 recurs, capture the full message for Egeria, since that entry was created locally through `journal-my-activity`. +### ISSUE-147: Every valid-metadata clear call answers OPEN-METADATA-400-011 "not found", even after a successful delete; clear-map-value deletes nothing; clear-map-name leaves the map's value set behind -**Layer:** Egeria Server (pyegeria clean-up) · **Status:** open · **Found:** 2026-10-13, reported by the egeria-workspaces-fs Resource Browser session from a live private-journal test. +**Egeria reset 2026-10-23 ~21:10-23:54Z** (egeria-workspaces-fs, owner-approved): the quickstart database was wiped and Egeria recreated on build 81beb8e5 (odpi/egeria 4486d1a7), then Coco content was reloaded. Every GUID recorded before then is gone. The leftover `pyegeriaClearRetest*` test elements described under "Leftovers on qs-view-server" are gone with it; nothing to clean up. The fix had already passed its 2026-10-19 re-test. -**What:** as erinoverview, `_async_remove_note` (`pyegeria/core/_server_client.py`) -was called on a JournalEntry created by `MyProfile.journal_my_activity`. It -failed with `OMAG-REPOSITORY-HANDLER-400-008` (`LOCAL_CANNOT_CHANGE_EXTERNAL`): -`deleteMetadataElementInStore` cannot modify Erin's UserIdentity (`ff983b64`), -which has Content Pack provenance ("Coco Pharmaceuticals Combination"). The -peer verified that nothing was removed. The entry keeps its `actionSource` and -`presentInNoteLogs` relationships, and Erin's Person still has 1 identity, -1 contact detail, 8 roles and 15 scopes. Test entry `fef83ed5` is still on -Erin's profile, waiting for the owner's decision. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. A successful clear now returns success; clear-map-value now uses the map name; clear-map-name also removes that map name's values. Egeria asks us to drop any workaround that treats 400-011 as success. **pyegeria:** there is none (checked 2026-10-18: no 400-011 handling in `md_processing/v2/reference_data.py` or `pyegeria/omvs/valid_metadata.py`), so the follow-on proposed below is withdrawn. Nothing to change; after the re-test, remove the leftover `pyegeriaClearRetest*` test elements on qs-view-server. -**The route is right:** in 6.2, notes are Actions. The type chain is -JournalEntry → Notification → Action → Process → Asset. Egeria main -(`af400039c2`) has no note, notification or action delete endpoint. -`AssetHandler.createNoteLogEntry` has no matching remove, and My Profile has -`journal-my-activity` with no matching remove. The only delete for an Action is -the generic `/assets/{assetGUID}/delete` (`AssetMakerResource`), which is what -`remove_note`'s `feedback-manager/assets/{guid}/delete` reaches, because -generic view services answer under any URL marker. Feedback Manager can only -remove a whole note log (`note-logs/{noteLogGUID}/remove`). +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** On throwaway properties `pyegRtClear`/`pyegRtClearMap` (typeName Asset), `clear_valid_metadata_value` (alpha, beta), `clear_valid_metadata_map_value` (mapOne/v1) and `clear_valid_metadata_map_name` (mapTwo, mapOne) all return success. A second clear of an already-removed value still answers 400-011 "not found", which is correct. Values and map names were gone afterwards. One oddity: `get_valid_metadata_values` read in the same run straight after the clears still listed alpha/beta/mapOne/mapTwo, but a read a minute later was empty. Possibly a short server-side cache; not pursued. Setup creates per-property container sets (`Egeria:ValidMetadataValue::pyegRtClear-`, `...pyegRtClearMap-`) that the clears do not remove; deleted by GUID. **Not cleaned (outside what peers cleared):** the 2026-10-13 leftovers under `pyegeriaClearRetest*`, which the platform has since turned into DataProduct/DigitalProduct elements (e.g. `98de81ed`, `4b937012`). -**Why it fails (likely, not verified):** `AssetHandler.createAction` anchors the -entry to its action source, linked by `ActionRequester`. For a journal entry -that is the caller's profile. Deleting an anchored element updates its anchor, -and Erin's identity is owned by the content pack, so the local delete is -refused. If so, no journal entry for a content-pack user can be deleted. That -is for the Egeria team. +**Layer:** Egeria Server (`ValidMetadataValueHandler`) · **Status:** open · **Found:** 2026-10-13. The first live run of the Valid Metadata commands showed it (the error text wasn't captured then). Re-tested the same day with the full responses captured. -**Anchor read (peer, read-only):** JournalEntry `fef83ed5` has anchorGUID -`10dc7da9`, Erin's Person profile. Its `actionSource` is ActionRequester → -Person `10dc7da9`, and `presentInNoteLogs` is AttachedNoteLogEntry → NoteLog -`f780511a` (MyJournal). No relationship from the entry reaches UserIdentity -`ff983b64`, which the error names. That identity is anchored to the same -Person. So the refusal comes from the delete touching the anchor's graph, most -likely the LatestChange update on the anchor, or a provenance check across -elements that share the anchor, not from a direct link. +**Re-test:** run once on qs-view-server as erinoverview, with all four live sessions clearing it first. It used made-up properties `pyegeriaClearRetest` and `pyegeriaClearRetestMap` (typeName `Asset`) and called the SDK methods in `pyegeria/omvs/valid_metadata.py`. The Dr.Egeria `Clear Valid Metadata *` commands call the same methods (`md_processing/v2/reference_data.py:198-242`). -**Delete-path trace (2026-10-13, Egeria source, read-only):** the asset delete -(`AssetMakerResource.deleteAsset` → `AssetHandler.deleteAsset`) and the Metadata -Expert delete (`MetadataExpertResource` → `MetadataExpertRESTServices`) both call -`OpenMetadataClient.deleteMetadataElementInStore`, then the same -`MetadataElementHandler` → `OpenMetadataAPIGenericHandler.deleteBeanInRepository` -code. Neither does any extra checking, so switching routes cannot avoid 400-008. -The error is thrown only by `RepositoryErrorHandler.validateProvenance`, when -the delete removes an entity or relationship whose provenance is not -LOCAL_COHORT and the request carries no `externalSourceGUID`. That happens in -`removeRelationship` for each relationship on the note and in `removeEntity` -for the note itself. No anchor (LatestChange) update was found on the delete -path, which corrects "Why it fails" above. But the error names UserIdentity -`ff983b64`, which is not linked to the note, so the exact instance is still -unexplained. Untried options are to pass the content pack's metadata-collection -id as `externalSourceGUID`, or to use `deleteMethod: ARCHIVE`, which goes -through a different handler. +| Call | Response | Actual effect | +|---|---|---| +| `clear_valid_metadata_value(alpha)`, set up just before | 400-011 "value alpha ... is not found" | **alpha removed** (gone from the list afterwards) | +| same call again (alpha already gone) | 400-011 | nothing to remove | +| `clear_valid_metadata_value(beta)` | 400-011 | **beta removed** | +| `clear_valid_metadata_map_value(mapOne, v1)` | 400-011 "value v1 ... is not found" | **nothing removed**: `get_valid_metadata_map_value` still returns v1 | +| `clear_valid_metadata_map_name(mapTwo)` | 400-011 | **mapTwo removed** | +| `clear_valid_metadata_map_name(mapOne)`, with v1 under it | 400-011 | the mapOne name value is removed, but the map's value set "mapOne" (holding v1) stays in `get_valid_metadata_values` | -**Note-privacy findings (egeria-workspaces-fs session, 2026-10-13, with owner -approval):** a second test JournalEntry, `8bec72e4-6da6-40a4-943e-d81e2173ec9b`, -was created on Erin's profile (`10dc7da9`) and cannot be deleted either until a -working delete exists. -- `POST my-profile/journal-my-activity` ignores `initialClassifications` and - stamps only ZoneMembership `['erinoverview']`. -- `POST classification-explorer/elements/{guid}/zone-membership` with - `['resource-explorer-private','erinoverview']` makes the entry private. - erinoverview can read it. peterprofile and garygeeke get 403 - `OPEN-METADATA-SECURITY-403-020` by GUID, and 0 hits from - `feedback-manager/assets/by-search-string`. -- The journal's NoteLog (`f780511a`) stays at `['erinoverview']`, so other - users can see that the log exists but not the hidden entries. -- The owner chose a temporary workaround for the Resource Locator: call - `add_zone_membership(['resource-explorer-private', user])` straight after - `journal-my-activity`. Review it once Egeria's journal fix lands. +Request ids: `ec9052e7` and `3975182d` (alpha), `38759736` (beta), `475e68f1` (map value), `407c0e17` and `07886f29` (map names). pyegeria surfaces each one as `SERVER_ERROR_500` wrapping `relatedHTTPCode` 400 `InvalidParameterException`. -**pyegeria clean-up:** make the `remove_note` route explicit and describe it as -an Action delete in the docstring. The owner chose the Metadata Expert delete -(`metadata-expert/metadata-elements/{guid}/delete`) for now. Per the trace -above, this is a clarity change, not a fix for 400-008. +**Cause (Egeria source, `af400039c2`, `frameworks/open-metadata-framework/.../handlers/ValidMetadataValueHandler.java`):** +1. **No `return` after the delete (lines ~541-680).** In all three `clear*` methods, `if (validValueGUID != null) deleteMetadataElementInStore(...)` is followed unconditionally by the audit log and `throw new InvalidParameterException(VALID_METADATA_MISSING ...)`. So every call throws, whether or not it deleted anything. The "not found" error is the method's normal exit. +2. **Map name dropped in `clearValidMetadataMapValue`.** It calls `getValidValueGUID(userId, typeName, propertyName, null, preferredValue)`, passing `null` where `mapName` belongs. It therefore searches for `v1` as a plain value of the property, finds nothing, and deletes nothing. +3. **`clearValidMetadataMapName` deletes only the name's own value.** It doesn't touch the per-map value set created by `setup-map-value` (the "Set of valid values for the mapOne map name ..." element) or its members. Whether that should cascade is Egeria's call. -**Re-test after Egeria's delete fixes are published:** the owner said -(2026-10-13) that Egeria is working on delete issues now. Once that work is -published, re-test `remove_note` on `8bec72e4` (plain delete, then `ARCHIVE` if -needed). Tell the egeria-workspaces-fs session before deleting anything on -Erin's profile. `fef83ed5` stays. +**Effect on Dr.Egeria:** `Clear Valid Metadata Value` and `Clear Valid Metadata Map Name` report failure although the clear worked. `Clear Valid Metadata Map Value` reports failure and really did fail. + +**Leftovers on qs-view-server:** the map value set "mapOne" with member v1 under `pyegeriaClearRetestMap`/`Asset`, plus whatever per-property container elements setup creates (`Egeria:ValidMetadataValue::pyegeriaClearRetest-Asset` and `...pyegeriaClearRetestMap-Asset`). These are test elements, not content-pack elements. RE's blueprint materializer treats `Egeria:ValidMetadataValue:` qualifiedNames as content-pack origin, but only for SolutionComponents, so they don't affect it. + +**pyegeria follow-on (not done, needs the owner's go-ahead):** Dr.Egeria's clear processor could treat 400-011 from `clear-value`/`clear-map-name` as success when a follow-up read confirms the value is gone. Otherwise, wait for the Egeria fix and re-test (see the BACKLOG.md delete re-test item). --- +### ISSUE-143: `includeOnlyRelationships: ["PersonRoleAppointment"]` returns no relationships for a Person, although the unfiltered read has its role appointments +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. The `includeOnlyRelationships` filter now leaves already-retrieved relationships in place, so `performsRoles` survives `["PersonRoleAppointment"]`. **pyegeria:** no code change; tell the Resource Browser (egeria-workspaces-fs) it can use the filter again after the re-test. +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** Erin's Person `10dc7da9` read with `includeOnlyRelationships: ["PersonRoleAppointment"]` returns `performsRoles` 8 (unfiltered: performsRoles 8, assignmentScope 15, requestedActions 13, ...; `["ContactThrough"]` still returns contactDetails 1). Read-only. Tell the Resource Browser it can use the filter again. -### ISSUE-132: `DeleteElementRequestBody` serializes the cascade flag as `cascadeDelete`; Egeria's field is `cascadedDelete` and unknown fields are ignored, so no pyegeria cascade delete has ever cascaded +**Layer:** Egeria Server (not pyegeria) · **Status:** open, not yet reported to the Egeria team · **Found:** 2026-10-13, reported by the egeria-workspaces-fs Resource Browser session; reproduced read-only the same day. -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. `cascadeDelete` is now accepted as an alias for `cascadedDelete`, so existing pyegeria requests will start to cascade; Egeria still recommends switching to `cascadedDelete`. **Risk check, 2026-10-18:** no production caller in `pyegeria/`, `md_processing/` or `commands/` passes cascade `True`; every `cascade`/`cascade_delete` default is `False`; `commands/tech/element_actions.py`'s `--cascade` flag (default True) is not forwarded. Live functional/scenario tests that pass `cascade=True` (collection, glossary, governance officer, `remove_note_log`) will now really cascade, which is what they intend. **pyegeria (proposed, not done):** change `DeleteElementRequestBody`'s serialization alias to `cascadedDelete` (keep both accepted on input) and the ~25 `"cascadeDelete"` literals (`_server_client.py`, `models.py`, `actor_manager.py`, `metadata_expert.py`, `product_manager.py`, `reference_data.py`, `valid_metadata.py`), then live-test one cascade and re-test ISSUE-127. No longer urgent, since the old spelling now works. +**What:** reading a Person with a `GetRequestBody` whose +`includeOnlyRelationships` is `["PersonRoleAppointment"]` returns the element +with no relationships at all. The same read with no filter returns 8 +`performsRoles` entries, each with relationship type `PersonRoleAppointment`. +Other filters on the same Person work. -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS (alias).** A throwaway DataStructure with a linked member DataField, deleted with `delete_data_structure(guid, cascade_delete=True)` (pyegeria still sends `cascadeDelete`), succeeded. Both elements were gone by GUID before any cleanup, so the cascade reached the server. The ISSUE-154 item 1 rename to `cascadedDelete` is still recommended but no longer urgent. +**Reproduction** (Person `10dc7da9-e9e1-471f-9afd-46a7ff414cb7`, Erin Overview, +as erinoverview on qs-view-server; +`ClassificationExplorer.get_element_by_guid(guid, graph_query_depth=1, +output_format="JSON", body={"class": "GetRequestBody", "graphQueryDepth": 1, +"includeOnlyRelationships": [...]})`): -**Layer:** pyegeria · **Status:** open, on hold: an Egeria fix for cascade delete is coming (owner, 2026-10-10). Egeria `main` checked 2026-10-10 (head `af400039c2`, 2026-10-06): no cascade change yet. Wait -for it, live-test cascade delete against it, then decide the pyegeria change; the fix below may change once the -Egeria side lands. Confirmed from Egeria source, not yet live-tested · **Found:** -2026-10-10, while checking ISSUE-129's server contract against Egeria source instead of the `.http` files. +| Filter | Relationship keys returned | +|---|---| +| none | performsRoles 8, assignmentScope 15, requestedActions 11, contactDetails 1, userIdentities 1, contributionRecord 1 | +| `["ContactThrough"]` | contactDetails 1 | +| `["AssignmentScope"]` | assignmentScope 15 | +| `["PersonRoleAppointment"]` | none | -**What:** `pyegeria/models/models.py` `DeleteElementRequestBody.cascade_delete` accepts both spellings on input but -serializes as `serialization_alias="cascadeDelete"`. The comment there records that the `.http` files were split -about 50/50 and that the choice was unverified (ISSUE-62). Egeria source settles it (`egeria-v6/egeria` at -`af400039c2`, 2026-10-06): `DeleteElementRequestBody` -> `DeleteRequestBody` -> `DeleteOptions` -(`open-metadata-framework/.../search/DeleteOptions.java`), whose field is `cascadedDelete` with -`getCascadedDelete()`/`setCascadedDelete()`. It is unchanged since 2025-08-04. The class is -`@JsonAutoDetect(getterVisibility=PUBLIC_ONLY, fieldVisibility=NONE)` and `@JsonIgnoreProperties(ignoreUnknown=true)`, -with no `@JsonProperty`/`@JsonAlias`, so the JSON name is `cascadedDelete` and `cascadeDelete` is silently ignored. +A spy on the request confirmed pyegeria sends the body exactly as given, so +the filtering happens in Egeria. -**Effect:** every pyegeria delete that uses `DeleteElementRequestBody` with cascade on (the ISSUE-62 fix, -`validate_delete_element_request`, ISSUE-127's fix in #429, `MetadataExpert.delete_metadata_element(cascade_delete=True)`) -sends a flag Egeria discards, so it deletes without cascading. A caller-supplied dict using the correct -`cascadedDelete` key is also re-serialized as `cascadeDelete`, so even that does not work. The models module's own -`DeleteRequestBody.cascaded_delete` already uses the right name. +**Likely cause (not confirmed):** Egeria's relationship filter +(`OpenMetadataHandlerBase`, around line 478) keeps a relationship when +`propertyHelper.isTypeOf(rel, name)`, which should match. But `performsRoles` is +filled in a separate step (`OpenMetadataHandlerBase`, around line 1282), so the +role appointments may be dropped or never fetched once a filter is set. + +**Workaround:** read the Person without `includeOnlyRelationships` and take +`performsRoles` from the result. The Resource Browser already does this. + +--- + +### ISSUE-130: The YAML secrets store's user directory caches a user forever, so editing an existing user needs a platform restart + +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. The YAML secrets store rebuilds its user cache each time it re-reads the file (`refreshTimeInterval`), so editing an existing user no longer needs a platform restart. **pyegeria:** no code change. + +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **Not re-testable here.** Needs an edit to a user in the platform's YAML secrets file and a wait for `refreshTimeInterval`. egeria-workspaces-fs is putting that to the owner as a separate step. + +**Re-test 2026-10-19 by egeria-workspaces-fs (reported to this session, not independently re-run here):** **PASS.** A throwaway user `pyegrt130` was added to the live `coco-user-directory.omsecrets` and logged in (200, 13:19:48Z), which cached it. It was then set to `userAccountStatus: DISABLED` in place with no restart (13:19:56Z). Logins worked until 13:20:32Z and were refused with 401 at 13:21:02Z, so the cache was rebuilt about a minute after the edit (`refreshTimeInterval: 10`). The file was restored byte-for-byte afterwards. Ready to move to the Appendix. + +**Layer:** Egeria server (`YAMLSecretsStoreConnector`) · **Status:** open, reported to the Egeria lead · **Found:** 2026-10-09, while adding Resource Explorer's own account (`resourceexplorernpa`). + +**What:** `getUser(userId)` returns the entry from `userAccountMap` if it is already there. `refreshSecrets()` re-reads the YAML file every `refreshTimeInterval` minutes, but `userAccountMap` (and `securityAccessControlMap`) are created only on the first load (when `secretsStoreFile == null`) and are never cleared afterwards. Re-reading the file only helps users the cache has never seen; the comment says "this may be a new user". + +**Seen:** the account was first added without its `secrets` block, and the first token request cached it. After the password was added (and matched the servers file), `/api/token` still answered 401 for more than 10 minutes. It worked straight after a platform restart. + +**Effect:** any change to an existing user (password, status, named-list membership) needs a platform restart. + +**Fix (Egeria):** clear or rebuild `userAccountMap` and `securityAccessControlMap` in `refreshSecrets()`. + +--- + +### ISSUE-128: Retention cannot be set on Egeria 6.2: the server registers the Jackson subtype as `RetentionProperties.class` (typo), and pyegeria only allows `RetentionClassificationProperties` + +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. Fixed on `main` 13 Oct (`RetentionProperties`); `oak2026` also accepts `RetentionClassificationProperties` again. Responses now carry `RetentionProperties` in `class`. **pyegeria (proposed, not done):** per the owner's 2026-10-09 decision, switch the whitelist, docstrings and Dr.Egeria pins to `RetentionProperties`: `pyegeria/omvs/classification_explorer.py:8851,8878,8919`, `md_processing/v2/processors.py:220-237` (its comment says the opposite and must be rewritten), `md_processing/v2/curation.py:136`. Accept both names in the whitelist so older platforms keep working. Not urgent: the old name is accepted again. + +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** On a throwaway collection, `classification-explorer/elements/{guid}/retention` accepted both `"class": "RetentionClassificationProperties"` and `"RetentionProperties"`; each read back `retentionBasis` 2 and its notes under `elementHeader.retention`. pyegeria's own `set_retention_classification` (old name) also succeeded. ISSUE-154 item 3's switch to the new name stays optional. + +**Status:** open, found 2026-10-09 by Resource Explorer's first Retention publish on Egeria 6.2. **Layer:** Egeria server (a type registration typo). **Decision (owner, 2026-10-09):** wait for the Egeria fix; no pyegeria workaround. Retention publishing stays blocked until then. + +**What (all proven live on 6.2, with raw HTTP POSTs to `classification-explorer/elements/{guid}/retention` on a throwaway asset):** +- `"class": "RetentionClassificationProperties"` is refused with `InvalidTypeIdException`. This is what pyegeria documents and whitelists, and it worked on the pre-6.2 server (processors.py comment, 2026-08-03). +- `"class": "RetentionProperties"` (the `.http` ground truth) is also refused with `InvalidTypeIdException`. +- `"class": "RetentionProperties.class"` gets **HTTP 200**, and the classification reads back with `retentionBasis` 2 and the notes. +- The server's known-type list in the error has 96 ids. The only Retention entry is literally `RetentionProperties.class`. That looks like an `@JsonSubTypes.Type(name = "RetentionProperties.class")` typo in Egeria's `ClassificationProperties`. +- pyegeria's `validate_new_classification_request` whitelist (`prop=["RetentionClassificationProperties"]`) refuses anything else client-side, so no Retention can be set through pyegeria on 6.2. + +(An earlier version of this entry said 6.2 knows `RetentionProperties`. That was a misread of the error text, corrected the same day.) + +**Where seen:** Resource Explorer's Curate classifications step on two repository assets. Both were refused, so nothing was written. Confidentiality, Criticality, Confidence and Impact property classes are registered normally. + +**Candidate fix:** +1. **Egeria:** correct the subtype name to `RetentionProperties`, matching the `.http` file. Report it to Mandy. +2. **pyegeria, after Egeria's fix:** change the whitelist and docstring to `RetentionProperties`, the `.http` name. No workaround for the typo, by the owner's decision. +3. Re-check `md_processing/v2/processors.py` and `v2/curation.py`, which both pin `RetentionClassificationProperties`. + +--- + +### ISSUE-108: newly created relationships stay invisible to related-element queries for up to ~20 minutes after creation + +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: needs information from us.** Egeria cannot reproduce it, and the new FVT does not show it. Their best lead is still timing: a related-element query hides a relationship whose `effectiveFrom` is later than the server's current time. For one link still missing about 12 minutes after a bulk load, they ask for: +- [ ] The relationship's GUID, `effectiveFrom` and `createTime`, and the server clock at that moment. +- [ ] Whether the link is returned when fetched directly by its GUID while it is missing from the related-element query. +- [ ] Whether the loader (Dr.Egeria or pyegeria) sets `effectiveFrom` or `effectiveTime` on that run. +- [ ] A timed run with `MemberDataField` links at 200 or more, polling every minute, to show whether the delay grows with load size or stays fixed. + +**pyegeria:** Dr.Egeria's body builders send `effectiveFrom` only when the author supplies `Effective From` (`common_md_utils.py:342` and siblings), so a default load sets none; still confirm `effectiveTime` is not stamped from the client clock on link requests (a client clock ahead of the server would explain the delay). The timed run needs Egeria writes, so it needs the owner's go-ahead. + +**Layer:** Egeria Server · **Status:** open · **Found:** 2026-09-20, reported by +the user from a Dr.Egeria bulk load session (1,027 expected field-link +relationships). + +**What:** a catalog walk run immediately after a load found only 876 of 1,027 +expected field links. A second walk 12 minutes later still showed 82 +structures short. A third walk, 20 minutes after the load, showed every link +present. No errors were raised at any point in the load itself — the +relationships were created successfully, they simply did not show up in +related-element queries for a long window afterward, then all became visible +at once with no further action taken. + +**Not yet root-caused.** This looks like the same family of symptom as the +2026-08-15 Postgres-checkpoint/background-connector-load entry above (server +falling behind under load, catching up later) rather than a paging defect +like ISSUE-54 (which would not self-heal purely by waiting — ISSUE-54's +bug is about tie-ordering across separately-executed pages, not about data +not existing yet). But this has not been directly correlated with connector +log activity or checkpoint timing the way that entry was — this is a fresh +report, not yet independently reproduced or diagnosed by a pyegeria session. + +**Impact:** any workflow that verifies a bulk write immediately after +issuing it (a post-load catalog walk, an integration test, an automated +Dr.Egeria "did it work" check) can get a false-negative "missing" result +that resolves itself given enough time — worth knowing before treating an +immediate post-write verification gap as evidence of a real data-loss bug. + +**Candidate next step (not yet done):** reproduce with a smaller, timed +sample (e.g. create N relationships, poll related-element queries at 1-minute +intervals, log when each becomes visible) to get a real latency +distribution instead of the three anecdotal checkpoints here, and check +whether it correlates with the same background-connector/checkpoint load +already documented above. Worth raising with the Egeria team directly, per +the original report. + +**Update 2026-09-20 (peer check, no upstream issue filed yet):** checked +`odpi/egeria`'s issue tracker directly — nothing filed for this symptom, so +no response from the Egeria team via GitHub. Asked three peer sessions +working against the same dev Egeria server: +- **egeria-workspaces-fs-4a:** hasn't seen this specific symptom, no word + from the Egeria team either. Raised a speculative, explicitly-unconfirmed + lead worth checking: a chronic, still-open issue on `egeria-shared-kafka` + where the `GroupCoordinator` repeatedly times out writing to + `__consumer_offsets` (5000ms) and triggers a consumer-group rebalance + roughly every 90s (confirmed via broker logs 2026-09-03; root cause still + open, CPU/GC/Postgres-restarts/virtiofs I/O already ruled out; historically + "Information" severity, self-heals within seconds, not chased further + since it wasn't visibly causing downstream harm). If `qs-view-server`'s + related-element queries are backed by Kafka-driven indexing/event + propagation rather than a direct repository read, a chronic rebalance + storm could plausibly manifest as exactly this "relationship exists but + isn't visible to queries yet" lag. **Explicitly not a confirmed + correlation** — the timescales don't obviously line up (a recurring ~90s + self-healing blip vs. a one-shot ~20-minute resolution) — and the peer + offered to check Kafka broker logs around the original bulk-load window + for the same rebalance pattern if useful. +- **Resource Explorer experimental UI:** no direct evidence either way — + today's two live writes against the same server (a Governance Action + Process Step upsert, and a native-survey `SurveyReport` lookup) both + checked either *immediately* after the write or only *after* polling to a + terminal status, never in the 1-20 minute gap this entry describes, so + neither confirms nor rules it out. Flagged that a background agent on + their side doing a related connection-refresh fix will trigger fresh + engine actions/relationships shortly and will report back if it surfaces + the same lag. +- **Egeria-trellis backlog review:** also nothing from the Egeria team (no + channel to them; checked RE docs/Backlog/design notes for this exact + shape first — a checked "no", not a guess). Offered a different, measured + precedent on this same platform as a *pattern* to distinguish causes, not + a diagnosis: `packages/resource-explorer/docs/investigation-classification-and-zoning-design.md` + (~line 830) documents a privately-zoned element still readable by a + non-owner *six minutes* after the control was written (denial began at + seven), caused by the security connector's `refreshTimeInterval` (10 min) + reload — "read it back immediately, so it must be fine" was exactly the + misleading evidence there too. Their key diagnostic point: a + **fixed-wall-clock cause** (everything becomes visible at once, at a time + unrelated to load size — test with 50 links instead of 1,027 and see if + the delay is still ~20 min) and a **volume-proportional cause** (async + indexing/event-driven materialisation catching up, arriving in a stream) + look identical from a single observation but have different fixes; the + 876→945(implied)→1,027 partial-recovery-at-12-minutes data point *mildly* + favours volume-proportional, but "two points don't make a curve." Also + worth checking directly: fetch one of the missing links by its own GUID + during the window — if that resolves but the relationship is still absent + from the related-element query, the write landed fine and only the + traversal/index path lags, narrowing which service is actually behind. If + this gets filed upstream, the volume-vs-fixed measurement is what turns + it into a real bug report instead of getting bounced as "eventual + consistency, working as intended." + +Two independent, unconfirmed leads now on record (Kafka rebalance storms; +a refresh-interval-style cache) alongside a concrete diagnostic method +(fixed-wall-clock vs. volume-proportional, plus the direct-GUID-fetch +isolation check) for telling them apart before assuming either. Next real +step is still reproducing with a smaller, timed, instrumented load using +that method — not yet done. + +**Reproduction attempt, 2026-09-20 (small end of the curve, not reproduced):** +after clearing with all 5 live peer sessions first (per `coordinate-shared-writes`), +ran a timed, instrumented small-scale reproduction against `qs-view-server`: +20 throwaway `Collection` elements created and each linked to a parent via +`CollectionMembership` (not the original `MemberDataField`/"field link" type +— a different relationship type, noted as a limitation below), polling +`get_collection_members` every 30s for up to 20 minutes, with a direct-GUID +isolation check ready for anything still missing at the deadline. + +**Result: all 20 relationships were visible on the very first poll, ~1.1s +after the last one was created — no lag at all.** Cleanup (delete all 21 +fixture elements) verified two independent ways: a direct-GUID fetch on +each (`404`/`OMAG-REPOSITORY-HANDLER-404-007`, confirmed real soft-delete, +not an ISSUE-63-style silent no-op) and a broad `find_collections` search +for the `test-fixture-issue108` prefix (0 results). The script itself was +run from an ephemeral per-session scratch directory, not committed anywhere +— reproduce via the same approach (create N Collections linked to a parent +via `_async_add_to_collection`, poll `_async_get_collection_members`) if +needed later; the approach, not the script file, is what's worth keeping. + +**Per Egeria-trellis backlog review's own framework, this is one point on +the curve, not a refutation.** It's consistent with either "no bug under +current load" or "the lag is genuinely volume-proportional and 20 elements +is nowhere near the ~1,027-link scale that surfaced it originally" — the +one data point this run *can't* distinguish those two. Also a real +methodology gap versus the original report: this used `CollectionMembership` +on plain `Collection` elements, not the `MemberDataField` "field link" +relationship the original 1,027-link load actually exercised (ISSUE-102 is +in the same `MemberDataField` neighborhood, worth checking together). Server +load conditions also weren't recorded this run (no `ListAgents`/`:8810` +snapshot taken alongside the timings, despite the earlier plan to do so — +an execution gap, not a design one). + +**Candidate next step, still open:** a same-relationship-type +(`MemberDataField`), same-order-of-magnitude (~200+) run, following the +same peer-clearance and independent-verification discipline, is what would +actually test the volume-proportional hypothesis — this run didn't attempt +that scale on its own initiative, since committing to it wasn't part of +what peers were asked to clear. -**Fix (not made):** change the serialization alias to `cascadedDelete` (keep both spellings accepted on input), and -fix the 19 `"cascadeDelete"` references in `_server_client.py`, `product_manager.py`, `actor_manager.py`, -`valid_metadata.py`, `metadata_expert.py` and `reference_data.py` (several are docstring samples). Then live-test one -cascade delete, and re-test ISSUE-127's `DataStructure` refusal. The `.http` files using `cascadedDelete` are correct; -the ones using `cascadeDelete` are the ones in error (worth reporting to Egeria). +**Egeria team response, relayed 2026-10-05 (source `oak2026` fb3d6fce53; **fixes are in two unmerged PRs, so no build contains them, and nothing was tested against a live platform**):** not validated; no cache found. Best lead: a client clock ahead of the server's, with `effectiveFrom` set from it (related-element queries silently hide a relationship whose `effectiveFrom` is later than the server's "now"). They need, for one link still missing at ~12 minutes: its `effectiveFrom` and `createTime`, the server clock then, and whether the loader sets `effectiveFrom`. **Our facts:** pyegeria sets no `effectiveFrom`/`effectiveTo`/`effectiveTime` by default (model defaults are `None`; no clock call is tied to any `effective*` field; Dr.Egeria forwards an "Effective From" only if typed). Host and container clocks agree within 1 s on 2026-10-05; the 2026-09-20 load's values are unavailable. -**Lesson:** where the `.http` files disagree with each other, read the Java DTO (getter/setter names plus Jackson -annotations); do not choose by majority or by what the code already sends. +**Update 2026-10-10:** the `oak2026` source (`fb3d6fce53`) the team examined is now in Egeria `main` (merged 2026-10-06, odpi/egeria#9363). No fix was made for this issue; the team's questions above still stand. --- -### ISSUE-127: `CollectionManager.delete_collection(cascade=True)` silently never sends `cascadeDelete` -- same shape as ISSUE-62 - -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** The `DataStructure` refusal (`validateNoMemberDataFields`) did not recur: a cascade delete of a structure with a member field removed both. See ISSUE-132's re-test. -**Layer:** pyegeria · **Status:** partly fixed in #429 (`68279835`), released in 6.1.29; cascade still blocked by ISSUE-132 · **Found:** 2026-10-05, found by capturing the request body while diagnosing a -failed cleanup. -**Fix + scope (2026-10-05):** a scan of all 42 callers of `_async_delete_element_request` found exactly two -that pre-fill a default body *and* pass a cascade flag: `CollectionManager._async_delete_collection` and -`ProductManager._async_delete_digital_product` (so `delete_digital_product(cascade=True)` was never cascading -either -- the live check's own product cleanup included). Both now pass `None` so the helper applies the flag. -`delete_digital_product`'s docstring sample also used the wrong key (`cascadedDelete`); corrected. The helper -itself is unchanged: a caller that passes an explicit body dict *and* `cascade=True` still gets the body's -value, which is documented behaviour ("body supersedes"). -**What:** `_async_delete_collection` always pre-fills `body = {"class": "DeleteElementRequestBody"}` when none is -given and hands that dict to `_async_delete_element_request(url, body, cascade)`. -`validate_delete_element_request` applies `cascade_delete` only in its *no body* branch, so with that pre-filled dict -the flag is discarded. Captured: `delete_collection("g", cascade=True)` posts -`{'class': 'DeleteElementRequestBody', 'forLineage': False, 'forDuplicateProcessing': False}` with no -`cascadeDelete`. `MetadataExpert.delete_metadata_element(cascade_delete=True)` does send it. +## Awaiting re-test (fix landed, not yet verified live) -**Fixed in #429, but cascade still does not reach Egeria -- see ISSUE-132.** #429 made the flag reach the request -body, but under the name `cascadeDelete`. Egeria's real field is `cascadedDelete`, and it ignores unknown fields, so -the flag is still dropped, now on the server side. #429's docstring "correction" from `cascadedDelete` to -`cascadeDelete` was also the wrong way round. The `DataStructure` refusal described below is probably this bug -rather than an Egeria quirk; re-test once ISSUE-132 is fixed. +Each of these has a fix in place — usually Egeria `main` (odpi/egeria#9363, +merged 2026-10-06), sometimes on the pyegeria side — but nobody has yet +re-run the original repro against a platform that contains it. The next +action for every entry here is the same: re-test, then move it to the +Appendix or back to an open section with what was found. -**Related, an Egeria quirk found alongside:** even with `cascadeDelete: true`, Egeria refused to delete a -`DataStructure` that still had a member `DataField` (`OMAG-GENERIC-HANDLERS-403-005 ... validateNoMemberDataFields`), -so a data contract imported by `import_data_contract*` has to be removed child-first: DataField, DataStructure, -Agreement. +**2026-10-28:** the local quickstart now runs Egeria `main` `abe31272`, which +contains odpi/egeria#9363, so these can be re-tested there. Several re-tests +create and delete elements in the shared Egeria: check with the other +sessions first. --- + ### ISSUE-125: `deleteEnumDef` answers 500 "unknown TypeDef" for an enum the server lists -- an enum added through the API cannot be removed **Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). `deleteEnumDef` removes an enum added through the API. **pyegeria:** no code change; the re-test should also remove the stray enum left on the dev platform. @@ -1053,6 +1520,7 @@ explains: the failure does not depend on what was loaded at startup. --- + ### ISSUE-124: `updateTypeDef` (Metadata Expert `open-metadata-types/update`) can never succeed -- the server builds the patch without `updatedBy`, which the repository services require **Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). `updateTypeDef` builds the patch with `updatedBy`, so updates succeed. **pyegeria:** no code change. @@ -1090,6 +1558,7 @@ then deleted and verified gone. **Confirmed still present on the rebuilt platfor --- + ### ISSUE-117: Cascade delete that takes the soft-delete (Memento) path fails partway with `OMAG-REPOSITORY-HANDLER-400-010` unless `forLineage=true` — leaves the asset live and an anchored element already soft-deleted **Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). Cascade deletes no longer fail partway. Related: a delete that is refused no longer leaves the job half done (deleting an element linked to a deleted action, a Memento, still needs `forLineage=true`; without it the delete is refused and nothing is removed), and the default delete method now honours `effectiveTime`/`forLineage`. **pyegeria:** no code change; keep sending `forLineage=true` on editing deletes. Resource Explorer's archive/delete block stays on until a throwaway re-proves the archive walk (owner's call). @@ -1162,6 +1631,7 @@ reproduction is to run a subscription process with a fresh asset as --- + ### ISSUE-112: `AutomatedCurationRESTServices.saveClientSideSecret`/`deleteClientSideSecret` return a plain success `VoidResponse` when the resolved connector isn't a `YAMLSecretsFileConnector` — no error, no write, no indication anything was skipped **Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). Client-side secret save and delete now refuse a secrets store that cannot be updated instead of reporting success. **pyegeria:** no code change; callers must now expect an error where they used to get a silent success. @@ -1251,591 +1721,182 @@ downstream investigation. **Egeria team response, relayed 2026-10-05 (source `oak2026` fb3d6fce53; **fixes are in two unmerged PRs, so no build contains them, and nothing was tested against a live platform**):** valid. A store whose connector cannot write secrets collections will get a 400 `OMAG-COMMON-400-034` naming the operation, asset and connector class; only `YAMLSecretsFileConnector` supports save and delete (`SecretsStoreConnector` gains `isSecretsCollectionUpdateSupported()`, others reject with `OCF-CONNECTOR-400-012`). -**Update 2026-10-10:** the `oak2026` source (`fb3d6fce53`) is now in Egeria `main`, merged 2026-10-06 (odpi/egeria#9363), so a platform built from current `main` has the fix. Not yet re-tested. - ---- - -### ISSUE-90: [fixed?] `qs-engine-host` retries `startMissedEngineActions` forever when one incomplete engine action's anchor is unreadable by the engine-host user - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). One unreadable engine action no longer blocks the restart sweep. **pyegeria:** no code change; re-test is on the engine-host log (and RE's engine-host restart sweep). - -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **No recurrence.** Read-only: no `ENGINE-HOST-SERVICES-2002` (or other qs-engine-host error) in the egeria-main log since the 13:01Z restart; the engine host initialized normally. The original trigger, an incomplete engine action whose anchor the engine-host user can't read, may simply not exist on this data, so this is not proof. RE also re-tests its engine-host restart sweep. - -**Layer:** Egeria Server (possibly quickstart content) · **Status:** open · **Found:** 2026-09-04 (trevor fresh quickstart) - -On a fresh quickstart repository (6.2-SNAPSHOT builds of 2026-08-31 and 2026-09-04, all standard -content packs, stock `coco-user-directory.omsecrets`) the engine host logs, every few seconds from first -start and without end: - -``` -qs-engine-host Error ENGINE-HOST-SERVICES-2002 startMissedEngineActions caught an exception -UserNotAuthorizedException while restarting incomplete engine actions; the error message was -OMAG-SERVER-SECURITY-403-007 User generalnpa is not authorized to issue operation Read on -a0baa4da-5f85-446c-b655-88ae81455966 anchor element DigitalProductFamily -``` - -Always the same element GUID; 196 refusals in ten minutes, >2,000 in ninety. A second quickstart on -another machine, with the identical platform build, identical server configuration documents (only the -generated `localServerId` differs), identical `coco-user-directory.omsecrets`, `DEMO_MODE=false` on both, -and its repository also reloaded the same day, shows zero refusals in 24 h. The one thing that differs is -history: the affected platform was stopped and restarted several times during its first hour (a -configuration copy, a Kafka log-directory reset, a rebuild) while content packs and the nanny daemon's -integration groups were still initialising; the healthy one was loaded once and left alone. The refused -element cannot be read through the view server by `garygeeke`, `peterprofile` or `erinoverview` either -(401), so it is not merely a `generalnpa` permission gap — the anchor looks unreadable by everyone, as -if left behind by an interrupted initialisation. - -**What it means:** an `EngineAction` exists whose anchor is a `DigitalProductFamily`; the engine host's own -identity (`generalnpa`) may not read that anchor under the security connector; the "restart incomplete -engine actions" pass treats the authorization failure as transient, aborts on it, and retries at its -normal cadence, so one permanently unreadable action becomes a permanent loop. Cost: platform container -at 200-330 % CPU and its Postgres at 500-750 % on an idle box, from `EngineAction` entity searches and -classification reads; everything else on the platform is slower. - -**Ask (either helps):** (1) engine-host-services: treat `UserNotAuthorizedException` for a specific engine -action as terminal for that action — log once, skip it, continue with the rest, do not retry it until the -security context changes; (2) quickstart content: give `generalnpa` read access to the digital-product -elements its engine actions anchor to, or anchor those actions to elements the engine-host identity can -read. Full draft: trellis session scratch `egeria-issue-engine-host-403-loop.md`. - -**Re-tested 2026-10-05 on the rebuilt platform** (image built 14:19Z from Egeria main `449ad06894`; -`egeria-main` restarted 14:39Z; checked 32 minutes later, read-only): - -- **Not reproduced.** Zero `ENGINE-HOST-SERVICES-2002`, zero `startMissedEngineActions` and zero - `OMAG-SERVER-SECURITY-403-007` in the platform log since the restart; the engine host is refreshing its governance - engines normally. CPU is calm: the platform container about 50% of one core and Postgres about 30%, against 200-330% - and 500-750% in the original report. -- **But the trigger was absent, so this proves little.** Only two engine actions are active: a `REQUESTED` PostgreSQL - survey and the `IN_PROGRESS` `EgeriaWatchdog`. `startMissedEngineActions` only processes `APPROVED` actions (read in - `GovernanceEngineHandler`), so it had nothing to retry. As this entry already said, the loop depends on start-up - history; a clean run cannot distinguish "fixed" from "not triggered". -- **The code is unchanged where it matters.** Between 2026-09-15 and `449ad06894` the only change to - `startMissedEngineActions` is `logException` becoming `logMessage` in the per-action `catch`. The outer `catch` that - aborts the whole pass on a failed page fetch is untouched, so an `APPROVED` action with an unreadable anchor would - still loop. -- **Unreadable anchors do exist on this platform.** 381 `OPEN-METADATA-SECURITY-0011` "not authorized to issue operation - Read" messages in 32 minutes, all for user `erinoverview`, across 11 elements anchored to a `DigitalProduct` (6) or a - `DigitalProductFamily` (5), arriving in bursts of 20 that match page-sized searches, not a steady background loop. - None were for `generalnpa`, and the element from the original report (`a0baa4da-...`) is not among them. So the - quickstart content does carry elements an ordinary user cannot read, which is the precondition this entry - hypothesised; whether any engine action anchors to one is not shown. -- **Status:** still open, latent. Not reproduced here; not shown fixed. - -**Egeria team response, relayed 2026-10-05 (source `oak2026` fb3d6fce53; **fixes are in two unmerged PRs, so no build contains them, and nothing was tested against a live platform**):** valid. `startMissedEngineActions` runs every 5 s per engine; `getActiveEngineActions` reads every action target and requester, so one target anchored to an element the engine's user cannot read (zone) throws, the outer `catch` ends the sweep, and newest-first paging brings the bad page back. The sweep is also unfiltered by engine, a likely contributor to the idle CPU (unmeasured). Fixes: unreadable actions filtered out, one-time error logging, anchor message argument order fixed, an engine-scoped sweep (new `.../governance-engines/{guid}/approved-engine-actions`), and anchor-refused search results filtered instead of logged as unauthorized. - -**Update 2026-10-10:** the `oak2026` source (`fb3d6fce53`) is now in Egeria `main`, merged 2026-10-06 (odpi/egeria#9363), so a platform built from current `main` has the fix. Not yet re-tested. - ---- - -### ISSUE-79: native survey against a template-created `FileFolder` asset fails server-side — `assetConnector` is null in `BasicFolderConnector.getFile()` - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: needs information from us.** Egeria cannot reproduce it. The error is now a clear "no asset connector" message, not a NullPointerException. Cause: the surveyed FileFolder had no ResourceConnection; the likely route is that the template create returned an existing FileFolder with the same qualified name (template create allows an existing element to be returned). They ask for: -- [ ] The request used to create the FileFolder: its template GUID and placeholder values, and whether `deepCopy` or `allowRetrieve` was set. -- [ ] The GUID of the surveyed asset, and whether that GUID existed before the create call. -- [ ] Whether that asset has a ResourceConnection, and which process first catalogued it (for example a files integration connector). - -**pyegeria:** this fits the 2026-10-06 live check below (with `deepCopy` no longer sent, the new folder has a ResourceConnection). `create_folder_element_from_template` hardcodes `"allowRetrieve": True` (`pyegeria/omvs/automated_curation.py:1081`), so a create whose qualified name already exists returns the old, possibly connection-less folder, which matches Egeria's theory. Proposed (not done): let callers pass `allow_retrieve`, or log when the returned GUID was pre-existing. The answers above can come from the 2026-10-06 `live_79.py` run's request plus a read-only lookup of the original surveyed GUID. - -**Live check 2026-10-06 (about 02:00Z, current pyegeria 6.1.29 code, rebuilt platform): the `deepCopy:false` default WAS the cause of the missing connection.** `create_folder_element_from_template` now sends a request with no `deepCopy` key (keys sent: allowRetrieve, class, forDuplicateProcessing, forLineage, isOwnAnchor, parentAtEnd1, placeholderPropertyValues, templateGUID, templateSubstitute), and the new folder read back through the classification explorer **has 1 connection** (`ResourceConnection` present). Folder deleted (cascade) and verified gone by GUID. **Not run: the survey itself** (a survey creates elements owned by the survey identity that an ordinary user may not be able to delete), so "the survey now completes" is still unproven; the original `assetConnector`-null failure was the symptom of no connection, which is now shown not to occur. Whether the Egeria-side repair (`e01426db86`) also matters is untested. Script: `live_79.py` stage A (stage B `--survey` is opt-in). - -**Update 2026-08-30, from the Egeria team (Mandy Chessell).** The entry -stays in "Open Egeria Server issues" — the reported failure is not -explained and not fixed. What has changed is that the cause recorded in -this entry, and in Resource Explorer's `re-as-engine-host-plan.md`, is now -known to be wrong, and the error it produces has been made legible. - -**Status: still open (Egeria Server)** — but the recorded cause is -disproved, and this entry and `re-as-engine-host-plan.md` should both be -corrected. Investigated 2026-08-30 from the Egeria side, including a new -FVT suite built to run this exact scenario. - -**The premise is false.** This entry, and the design doc's hold, rest on -"template-based asset creation never wires up a working Connection." That -is not true, in either environment it was checked against: - -- **On the quickstart that produced the report.** Every element created - from a template that itself carries a `ResourceConnection` got one — 10 - of 10. Every `FileFolder` in that repository has one, including all - three that were created from a template (`/deployments/content-packs`, - `/deployments/loading-bay/sample-data`, `/deployments/secrets`). -- **In a clean environment built from the current archives.** A folder - catalogued through `FileGovernance::create-file-folder` came back with a - `ResourceConnection` attached. -- **The `FilesContentPack` template itself is also well formed:** - `FileFolder` template `fbdd8efd-...` carries an `Anchors`-classified - `Connection` with both a `ConnectToEndpoint` and a - `ConnectionConnectorType`. - -**The folder survey works.** A new `files-fvt` suite runs the reported -scenario end to end — catalogue a folder from the `FileFolder` template, -then survey it natively — and it passes: -``` -OMES-SURVEY-ACTION-0019 The survey action service folder-survey-service has completed the -analysis of asset 7050dc50-... with request type survey-folder in 1927 milliseconds; the -results are stored in survey report 7e7892e2-... -``` -The report carried six annotations. The test asserts the survey -*completed*, not merely that it ran: a survey that dies on a null -connector still produces an engine action, and the waiter throws on any -terminal status other than `COMPLETED`. - -**A second hypothesis was also eliminated.** While investigating, a real -defect was found in the PostgreSQL repository: a `TypeDefPatch` that -reparents a type does not update the denormalised supertype chain stored -on existing rows, so instances written before the patch become invisible -to searches for the new supertype. That would have hidden connections -exactly as reported, since `AssetConnection` was reparented under -`ResourceConnection` on 2026-08-27. It is fixed — but it was never the -cause here: the quickstart repository stores 202 rows as -`:ResourceConnection:LabeledRelationship:` and none as -`:AssetConnection:`, because it was reloaded after the type change. - -**What has been fixed, and what it does and does not buy.** The NPE -itself is now a clean error. `getConnectorForAsset()` returns `null` by -contract when an asset has no connection; `FolderSurveyService` cast that -null and dereferenced it, and the framework's own -`performCheckAssetAnalysisStep()` had the same hole on its error path. -Both now report `OPEN-SURVEY-400-008` / `OPEN-SURVEY-0009` -(`NO_ASSET_CONNECTOR`), naming the asset. - -This does not make such a survey succeed — an asset with no connection -describes a resource nothing can open. It makes the failure say *which* -asset, which is what the original report was missing: the NPE named -`BasicFolderConnector`, a class the caller had never heard of, and pointed -away from the asset. - -**Why that matters for how this entry reads.** Four further defects were -found in the same connectors, all on paths nothing else exercises: both -file survey services failed on every invocation (annotations created -without a qualified name, in three places), the CSV survey dereferenced a -null root schema type, wrote a relationship property to an entity, and -produced colliding annotation qualified names. The file survey path had, -on this evidence, never run to completion. Someone hitting an NPE in this -family had every reason to suspect the asset — the errors never named -anything else. That is context for the original diagnosis, not a -criticism of it. - -**The naming trap, which is directly actionable on the pyegeria side.** -This entry already records that `initiate_file_folder_survey`'s default -(`FileSurveys:survey-folder`) and the `.http` example -(`AssetSurvey:survey-folder`) are both wrong. Confirmed and now -understood: a governance action type is registered under -`::` — two colons — so the real name is -`FileSurvey::survey-folder`. `AssetSurvey` is not an engine that exists at -all. The `.http` examples have been corrected upstream -(`FileSurvey::survey-folder`, `ApacheKafkaSurvey::survey-kafka-server`, -`PostgreSQLSurvey::survey-postgres-server`, -`PostgreSQLSurvey::survey-postgres-database`) with a note stating the -shape, because the error a wrong name produces — -`OMAG-GENERIC-HANDLERS-400-013`, "the name is not recognized" — does not -say what it wanted instead. `initiate_file_folder_survey`'s default still -needs fixing in pyegeria. - -**What is left, and what would settle it.** The remaining candidate is -the deployment rather than the cataloguing or the survey: which server -and userId the engine host's connected-asset client resolves -`SurveyAssetStore.getConnectorToAsset()` against. That cannot be tested -from the Egeria side — it needs the engine host that produced the -failure. Suggested, in order: - -1. Re-run the original repro against a build carrying the NPE fix. If the - asset genuinely has no connection the run now fails with - `OPEN-SURVEY-400-008` naming it, which turns the question from "why is - this connector null" into "why does this asset have no connection that - this userId can see". If instead the survey completes, the deployment - has moved on and this entry can be closed. -2. Check the folder asset at the moment of failure for a - `ResourceConnection`, using the same userId the engine host runs as — - not an admin one. If the relationship is there but the survey cannot - see it, that is a visibility or server-routing problem, and the userId - is where to look next. -3. Compare against `files-fvt`, which is a known-good baseline for this - exact scenario. Pointing the same two actions at a - differently-configured engine host isolates the variable. - -Also worth correcting: this entry's earlier note about leftover probe -elements (`FileFolder::localhost:/tmp/re-e1-probe-*`, -`FileFolder::localhost:/tmp/re-issue79-recheck-*`) is out of date — the -quickstart repository has been reloaded since and holds none of them, so -that evidence is gone. - -**Status:** open (Egeria Server), reproduced live 2026-08-27, **re-checked -2026-08-28 — still reproduces, identical failure.** Same class/method -(`BasicFolderConnector.getFile()`, `assetConnector` null), same error IDs -(`OMES-SURVEY-ACTION-0018`/`OPEN-SURVEY-500-001`), same -`java.lang.NullPointerException`, against a freshly created `FileFolder` -and a fresh survey run — not a one-off. Filed by content, not by number — -see "why this entry exists" below. +**Update 2026-10-10:** the `oak2026` source (`fb3d6fce53`) is now in Egeria `main`, merged 2026-10-06 (odpi/egeria#9363), so a platform built from current `main` has the fix. Not yet re-tested. -**Layer:** Egeria Server (survey action framework / `BasicFolderConnector`), -not pyegeria. `create_folder_element_from_template` and the survey-initiate -call both succeed and are correctly formed; the failure happens entirely -inside the server's survey action service. +--- -**Why this entry exists.** Resource Explorer's `re-as-engine-host-plan.md` -is ON HOLD citing this exact symptom ("connector is null" on a native -survey against a template-created asset) under the name `PYEGERIA_ISSUES.md` -**ISSUE-51** — but this file was renumbered on 2026-08-15, and today's -ISSUE-51 is an unrelated, already-fixed `fetch_element()` shape problem. -The blocker's own content had no surviving entry anywhere in this file. -Investigated fresh rather than guessed at, per the plan's own citation -being untrustworthy. -**Two candidate bugs were in play, and they are not the same bug:** +### ISSUE-90: [fixed?] `qs-engine-host` retries `startMissedEngineActions` forever when one incomplete engine action's anchor is unreadable by the engine-host user -1. **Resource Explorer's superseded tracker, entry E1** (`packages/ - resource-explorer/docs/egeria-pyegeria-issues.md`) describes a *different* - symptom: a 500 from the PostgreSQL repository connector on - `createMetadataElementFromTemplate` (`metadata_collection_guid` null - while saving a classification) — a failure to *create* the element at - all. **Re-tested live 2026-08-27 and does not reproduce**: - `create_folder_element_from_template(path_name=..., folder_name=..., - file_system="localhost")` against `qs-view-server` returned a real GUID - with no error. Whatever this was, it looks fixed. -2. **The design doc's actual blocker** — a template-created asset surveyed - natively fails with a null connector — **does still reproduce**, and is - what this entry tracks. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). One unreadable engine action no longer blocks the restart sweep. **pyegeria:** no code change; re-test is on the engine-host log (and RE's engine-host restart sweep). -**Reproduction** (against `qs-view-server`, no pyegeria bug involved): -```python -from pyegeria import AutomatedCuration -c = AutomatedCuration("qs-view-server", "https://localhost:9443", "erinoverview", "secret") -c.create_egeria_bearer_token() +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **No recurrence.** Read-only: no `ENGINE-HOST-SERVICES-2002` (or other qs-engine-host error) in the egeria-main log since the 13:01Z restart; the engine host initialized normally. The original trigger, an incomplete engine action whose anchor the engine-host user can't read, may simply not exist on this data, so this is not proof. RE also re-tests its engine-host restart sweep. -folder_guid = c.create_folder_element_from_template( - path_name="/tmp/some-new-path", folder_name="probe-folder", file_system="localhost") +**Layer:** Egeria Server (possibly quickstart content) · **Status:** open · **Found:** 2026-09-04 (trevor fresh quickstart) -# Note: initiate_file_folder_survey's own default survey_name -# ("FileSurveys:survey-folder") and the .http ground truth's example -# ("AssetSurvey:survey-folder") are BOTH wrong on this server -- neither -# GovernanceActionType is registered (OMAG-GENERIC-HANDLERS-400-013). The -# real registered qualifiedName, confirmed via -# `EgeriaTech.get_elements("GovernanceActionType")`, is "FileSurvey::survey-folder". -action_guid = c.initiate_file_folder_survey(folder_guid, survey_name="FileSurvey::survey-folder") -``` +On a fresh quickstart repository (6.2-SNAPSHOT builds of 2026-08-31 and 2026-09-04, all standard +content packs, stock `coco-user-directory.omsecrets`) the engine host logs, every few seconds from first +start and without end: -Poll the resulting `EngineAction` (`EgeriaTech.get_elements("EngineAction")`, -match on `elementHeader.guid`) — it completes (fast, ~1s) with -`activityStatus: "FAILED"` and `completionGuards: ["survey-failed"]`. Full -`completionMessage`: ``` -OMES-SURVEY-ACTION-0018 The survey action service folder-survey-service -threw a org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectorCheckedException -exception during the generation of survey report ... for asset - during request type survey-folder in survey action engine -FileSurvey (guid=4168abb9-6c60-46fb-b9c0-b44180d19500). The error message -was OPEN-SURVEY-500-001 Unexpected exception in survey action service -folder-survey-service of type java.lang.NullPointerException detected by -method start. The error message was Cannot invoke -"org.odpi.openmetadata.adapters.connectors.datastore.basicfile.BasicFolderConnector.getFile()" -because "assetConnector" is null +qs-engine-host Error ENGINE-HOST-SERVICES-2002 startMissedEngineActions caught an exception +UserNotAuthorizedException while restarting incomplete engine actions; the error message was +OMAG-SERVER-SECURITY-403-007 User generalnpa is not authorized to issue operation Read on +a0baa4da-5f85-446c-b655-88ae81455966 anchor element DigitalProductFamily ``` -**Analysis.** `create_folder_element_from_template` creates the `FileFolder` -element and it looks entirely normal (confirmed via `get_elements`: proper -`pathName`, `resourceName`, `Anchors` classification) — but no working -`Connection`/`Connector` got wired to it, so when `folder-survey-service` -tries to open the asset via `BasicFolderConnector.getFile()`, the -connector reference is null and the survey action service crashes with an -NPE rather than a clean error. This matches the design doc's -characterization exactly ("template-based asset creation never wires up a -working Connection, so any native survey against a template-created asset -fails server-side with a NullPointerException") — just with the precise -class/method now on record, which the design doc didn't have. - -**Impact:** blocks Resource Explorer's engine-host participation design -(`re-as-engine-host-plan.md`, cases 1/2/4 — anything needing a native -survey to actually *complete*, not just be triggered; case 3, RE-local -surveying, is unaffected). The hold should be re-evaluated against this -entry specifically, not against the old ISSUE-51 number. As of 2026-08-28 -this is confirmed the *only* thing still blocking that work on the -pyegeria side — ISSUE-78's engine-host method trio shipped in 6.1.5 and RE -has upgraded and verified it end to end, and the two other gaps -identified alongside it (no REST way to create a `GovernanceEngine`/ -`GovernanceService` element; no per-engine "claimable work" listing, -`get_active_engine_actions()` filtered client-side is the only option) are -both genuine server-side gaps, not pyegeria ones. This entry is the real -holdup. - -**Leftover from this investigation:** probe `FileFolder` elements -(`qualifiedName: "FileFolder::localhost:/tmp/re-e1-probe-*"` from -2026-08-27, `"FileFolder::localhost:/tmp/re-issue79-recheck-*"` from the -2026-08-28 re-check) on `qs-metadata-store`, plus their failed -`SurveyReport`/`EngineAction` pairs, left in place as reproduction -evidence for both dates — not yet deleted. +Always the same element GUID; 196 refusals in ten minutes, >2,000 in ninety. A second quickstart on +another machine, with the identical platform build, identical server configuration documents (only the +generated `localServerId` differs), identical `coco-user-directory.omsecrets`, `DEMO_MODE=false` on both, +and its repository also reloaded the same day, shows zero refusals in 24 h. The one thing that differs is +history: the affected platform was stopped and restarted several times during its first hour (a +configuration copy, a Kafka log-directory reset, a rebuild) while content packs and the nanny daemon's +integration groups were still initialising; the healthy one was loaded once and left alone. The refused +element cannot be read through the view server by `garygeeke`, `peterprofile` or `erinoverview` either +(401), so it is not merely a `generalnpa` permission gap — the anchor looks unreadable by everyone, as +if left behind by an interrupted initialisation. -**Update 2026-10-05 -- probably a pyegeria bug, fixed on our side:** the Egeria team asked us to check that pyegeria did not send `deepCopy:false`. It did: `TemplateRequestBody.deep_copy` defaulted to `False` from 2026-01-09 until fixed on 2026-09-29 (#399, first released in v6.1.23), the model is serialised with `exclude_none` (which keeps `False`), and `create_folder_element_from_template` builds its body without `deepCopy`. Egeria then copies only the top-level element and none of the template's anchored attachments, so no `ResourceConnection`, which is exactly a null `assetConnector`. The Egeria team also points to `e01426db86` (2026-08-30, PostgreSQL supertype-chain repair after the 08-27 `ResourceConnection` re-parenting). **To confirm:** create a folder from the template with current pyegeria, check it has a `ResourceConnection`, then survey it. +**What it means:** an `EngineAction` exists whose anchor is a `DigitalProductFamily`; the engine host's own +identity (`generalnpa`) may not read that anchor under the security connector; the "restart incomplete +engine actions" pass treats the authorization failure as transient, aborts on it, and retries at its +normal cadence, so one permanently unreadable action becomes a permanent loop. Cost: platform container +at 200-330 % CPU and its Postgres at 500-750 % on an idle box, from `EngineAction` entity searches and +classification reads; everything else on the platform is slower. ---- +**Ask (either helps):** (1) engine-host-services: treat `UserNotAuthorizedException` for a specific engine +action as terminal for that action — log once, skip it, continue with the rest, do not retry it until the +security context changes; (2) quickstart content: give `generalnpa` read access to the digital-product +elements its engine actions anchor to, or anchor those actions to elements the engine-host identity can +read. Full draft: trellis session scratch `egeria-issue-engine-host-403-loop.md`. -### ISSUE-38 (PY-18): `count_relationships_between_elements("Exception")` (276) disagrees with `ClassificationExplorer.get_relationships("Exception")` (55) +**Re-tested 2026-10-05 on the rebuilt platform** (image built 14:19Z from Egeria main `449ad06894`; +`egeria-main` restarted 14:39Z; checked 32 minutes later, read-only): -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. Metadata Expert counts now take `pushDown`, **default `false`** (count matches the list); `true` gives the fast repository count. Relationship counts with end GUIDs and no criteria used to return 0; fixed. Egeria recommends `pushDown=false` for user-facing counts and `pushDown=true` for statistics. **pyegeria (proposed, not done):** `MetadataExpert.count_metadata_elements`/`count_relationships_between_elements` docstrings (`metadata_expert.py:3698`, `:4013`) say omitting `push_down` means `pushDown=True`; that is now wrong. Decide what `pyegeria/view/overview_metrics.py` (which omits `push_down`) should send, since its counts will become the slower list-matching kind once the platform has this. Verification below still needs running on a platform with the fix. +- **Not reproduced.** Zero `ENGINE-HOST-SERVICES-2002`, zero `startMissedEngineActions` and zero + `OMAG-SERVER-SECURITY-403-007` in the platform log since the restart; the engine host is refreshing its governance + engines normally. CPU is calm: the platform container about 50% of one core and Postgres about 30%, against 200-330% + and 500-750% in the original report. +- **But the trigger was absent, so this proves little.** Only two engine actions are active: a `REQUESTED` PostgreSQL + survey and the `IN_PROGRESS` `EgeriaWatchdog`. `startMissedEngineActions` only processes `APPROVED` actions (read in + `GovernanceEngineHandler`), so it had nothing to retry. As this entry already said, the loop depends on start-up + history; a clean run cannot distinguish "fixed" from "not triggered". +- **The code is unchanged where it matters.** Between 2026-09-15 and `449ad06894` the only change to + `startMissedEngineActions` is `logException` becoming `logMessage` in the per-action `catch`. The outer `catch` that + aborts the whole pass on a failed page fetch is untouched, so an `APPROVED` action with an unreadable anchor would + still loop. +- **Unreadable anchors do exist on this platform.** 381 `OPEN-METADATA-SECURITY-0011` "not authorized to issue operation + Read" messages in 32 minutes, all for user `erinoverview`, across 11 elements anchored to a `DigitalProduct` (6) or a + `DigitalProductFamily` (5), arriving in bursts of 20 that match page-sized searches, not a steady background loop. + None were for `generalnpa`, and the element from the original report (`a0baa4da-...`) is not among them. So the + quickstart content does carry elements an ordinary user cannot read, which is the precondition this entry + hypothesised; whether any engine action anchors to one is not shown. +- **Status:** still open, latent. Not reproduced here; not shown fixed. -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** Counts against the paged `ClassificationExplorer.get_relationships` total: `Exception` gives push_down omitted 146, false 146, true 148, list 146; `SemanticAssignment` gives 563, 563, 583, list 563. The default is now `false` and matches the list. The fast count is higher by what erinoverview cannot see (2 and 20). Read-only. The docstring fix in ISSUE-154 item 2 is now confirmed needed. +**Egeria team response, relayed 2026-10-05 (source `oak2026` fb3d6fce53; **fixes are in two unmerged PRs, so no build contains them, and nothing was tested against a live platform**):** valid. `startMissedEngineActions` runs every 5 s per engine; `getActiveEngineActions` reads every action target and requester, so one target anchored to an element the engine's user cannot read (zone) throws, the outer `catch` ends the sweep, and newest-first paging brings the bad page back. The sweep is also unfiltered by engine, a likely contributor to the idle CPU (unmeasured). Fixes: unreadable actions filtered out, one-time error logging, anchor message argument order fixed, an engine-scoped sweep (new `.../governance-engines/{guid}/approved-engine-actions`), and anchor-refused search results filtered instead of logged as unauthorized. -**Update 2026-08-30, from the Egeria team (Mandy Chessell).** Leaving this -entry in "Open Egeria Server issues" until the verification below has been -run against real data — the Egeria side is done, but this entry has been -re-measured and re-confirmed several times, and it shouldn't move to -Fixed/Resolved on the strength of a change nobody has yet pointed at the -data that produced the original report. +**Update 2026-10-10:** the `oak2026` source (`fb3d6fce53`) is now in Egeria `main`, merged 2026-10-06 (odpi/egeria#9363), so a platform built from current `main` has the fix. Not yet re-tested. -**Status: resolved on the Egeria side 2026-08-30** (`omf-metadata-spring`, -`omf-metadata-server`, `generic-handlers`, and the OMF Java client) — -**awaiting verification from the pyegeria side**, see below. The two -numbers are not going to be made equal, because they are answers to two -different questions and both are wanted; what has changed is that the -caller now chooses which one they get. +--- -**What the difference actually was.** `findAttachmentLinks()` puts every -relationship it retrieves through a visibility check on the entities at -both ends, and silently drops the ones whose anchor the caller cannot -read. `countAttachmentLinks()` skipped that check — deliberately, and -documented as such in its own javadoc, because applying it means fetching -every matching relationship, which is the cost the count exists to avoid -— and also ignored `effectiveTime`. So the count was of what matches the -search and the list was of what the caller may see. That answers this -entry's open question ("what does the metadata-expert count include for -`Exception` that the classification-explorer traversal excludes?"): -relationships with an end whose anchor is not readable by the calling -user, plus anything outside the effectivity window. It also explains why -only `Exception` diverged among the types tested — it is the type whose -ends are most likely to have been deleted or anchored to something the -caller cannot see — and why the divergence tracked the demo data rather -than staying at a fixed size. -**The new option.** Both counting endpoints take a `pushDown` query -parameter, `required=false`, default `true`: -``` -POST .../users/{userId}/relationships/by-search-conditions/count?pushDown=false -POST .../users/{userId}/metadata-elements/by-search-conditions/count?pushDown=false -``` -- `pushDown=true` (the default, and exactly the existing behaviour) — the - repository counts the matching rows itself. Fast, and counts what - matches the search. -- `pushDown=false` — the relationships are retrieved and counted, so the - answer agrees with the list by construction, at the cost of reading - every one of them. +### ISSUE-95: No catalog template registered for the "Apache Kafka Server" technology type — `Create Kafka Server Element` (Asset Maker) fails with a 400 on `qs-view-server` -Nothing changes for a caller that does not send the parameter. The same -choice is on the Java client (`countMetadataElements(..., pushDown)` and -`countRelationshipsBetweenMetadataElements(..., pushDown)` on -`OpenMetadataStore` and `OpenMetadataClient`), with the existing -signatures kept and defaulting to `true`. +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). The Apache Kafka Server catalog template is registered. **pyegeria:** no code change; re-run `Create Kafka Server Element`. -**The test the pyegeria side should run to prove it.** Against the -environment that produced the original numbers, using whatever the -current counts are rather than the 58/57 recorded here — the demo data -has moved several times during this entry's life, and what matters is the -relationship between the numbers, not their values: +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** `get_template_guid_for_technology_type("Apache Kafka Server")` returns `5e1ff810-5418-43f7-b7c4-e6e062f9aff7`. Read-only. Ready to move to the Appendix. -1. **The accurate count should equal the list.** For - `relationshipTypeName: "Exception"`, the count with `pushDown=false` - should equal `len(ClassificationExplorer.get_relationships("Exception"))` - exactly. This is the assertion that closes the entry — it is the - discrepancy this entry is about, stated as an equality. -2. **The fast count should be unchanged.** The same call with - `pushDown=true`, and with the parameter omitted altogether, should both - return what the count returns today. If either moves, something - regressed for every existing caller. -3. **A type that already agreed should still agree both ways.** - `SemanticAssignment` matched at 401=401 when this entry was last - measured. Both `pushDown` values should return that same number — the - retrieval route must not lose or double-count anything on a type where - there was never a discrepancy to resolve. +**Layer:** Egeria Server (deployment/archive content) · **Status:** awaiting re-test (addressed on the Egeria side, confirmed live 2026-10-05) · **Found:** 2026-09-11, live-verifying the new Asset Maker Dr.Egeria family (#354) -Worth capturing while verifying: the difference between the two counts is -the number of matching relationships the calling user cannot fully see. -If it is not the 1 recorded here, that is not a failure — it is a -measurement of the current data — but it is worth writing down, because a -large difference on a type that should be fully visible would be a -separate finding. If test 1 comes out unequal, the remaining difference is -not the visibility check and this entry should stay open with both -numbers recorded. +`AutomatedCuration._async_get_template_guid_for_technology_type("Apache Kafka Server")` +returns `None` on `qs-view-server` — no catalog template is registered for +that technology type on this server — so `Create Kafka Server Element`'s +underlying `_async_create_kafka_server_element_from_template` posts a body +with `templateGUID: null` and gets back `VALIDATION_ERROR_1` ("Request body +failed validation"). Not a pyegeria code defect: every other Asset Maker +technology type tried resolved a real template GUID on the same server +(CSV Data File, PostgreSQL Server, File System Directory, YAML File Secrets +Collection, Unity Catalog Server) — this one is specific to the Kafka +technology type's archive content on this deployment. `Create Kafka Server +Element` itself is otherwise correct (parses/validates cleanly; the failure +is entirely server-side at execution). -**Follow-on for pyegeria, once verified:** the Overview dashboard's -current workaround — keeping relationship counts on -`ClassificationExplorer.get_relationships` while using native counting -only for element counts — can be revisited. -`count_relationships_between_elements(..., pushDown=False)` gives the same -number as the traversal without materialising the list client-side. Keep -in mind that `pushDown=false` reads every matching relationship -server-side, so it is the right choice for a number a user is going to -compare against a list they can also see, and the wrong choice for a -headline figure over a large type where speed is what matters. That -trade-off is now the caller's to make, which is what this entry was -ultimately asking for. +**Ask:** confirm whether the Apache Kafka Server catalog template is +expected to ship with quickstart's default archive content, and if so, why +it's missing here — or add it to whichever archive/content-pack is meant to +carry it. -**Egeria-side test cover:** `CountPushDownFVT` in -`open-metadata-test/open-metadata-fvt/query-fvt` checks that both routes, -over REST and through the Java client, agree when every matching element -is visible, and that omitting the parameter changes nothing. It -deliberately does not attempt the case where they differ — that needs an -element the calling user cannot read, which that suite cannot arrange. -Which is why the verification above, against real data, is the part that -actually proves the fix. -**Update 2026-09-02 — pyegeria side implemented, verification attempted -and blocked (not failed) by a stale deployed platform image.** +**Addressed on the Egeria side (2026-10-05 check, carried over from the retired `docs/egeria-open-bugs-2026-10-05.md`):** the running platform resolves a template. `get_template_guid_for_technology_type("Apache Kafka Server")` returns `5e1ff810-5418-43f7-b7c4-e6e062f9aff7`, and Egeria source `449ad06894` ships `ApacheKafkaContentPack.omarchive`. It looks like a content-pack loading matter, not a code defect. **Remaining:** re-test `Create Kafka Server Element` live. -`push_down: Optional[bool] = None` added to -`MetadataExpert.count_metadata_elements`/`_async_count_metadata_elements` -and `count_relationships_between_elements`/ -`_async_count_relationships_between_elements` (`pyegeria/omvs/ -metadata_expert.py`) — `None` omits the `pushDown` query param entirely -(today's existing behaviour, zero change for any existing caller); `True`/ -`False` append `?pushDown=true`/`?pushDown=false` explicitly, following the -same `if : url += f"&{param}=..."` conditional-append convention -already used elsewhere in this codebase (e.g. `valid_metadata.py`'s -`typeName` handling, ISSUE-82/PY-25). +--- -Ran this entry's own 3-test verification plan live-patched into -`quickstart-pyegeria-web` (site-packages `docker cp` + restart, diff- -confirmed byte-identical), against `qs-metadata-store`: -``` -count (pushDown omitted): 58 -count (pushDown=true): 58 -count (pushDown=false): 58 -ClassificationExplorer.get_relationships("Exception") length: 57 -``` +### ISSUE-89: No configurable bearer-token lifetime in `application.properties` — platform tokens are fixed at 3600 s, and the signing key is random per restart -Test 1 (accurate count == list) came out **unequal** (58 vs 57) — but -before treating that as "the visibility check isn't it," checked whether -the platform under test actually has the 2026-08-30 fix at all. First -pass looked at the running image's `build-date` label (2026-08-24, six -days before the merge) and Docker Hub's manifest digest for `:latest`, -which matched what was already deployed — reasonable evidence, but not -airtight, since Docker Hub then showed the `latest` tag re-pushed -2026-08-31 (label alone can't tell "same app, new base layer" from "new -app build that forgot to bump the label"). Settled it properly: pulled -that exact 2026-08-31 digest -(`sha256:8ff341b7cf1b440f7aa4db991258afd70758f15e803c43db9bd40b41f3a0b9e5`) -and diffed the actual server jar against the one already deployed — -`deployments/omag-server-platform-6.2-SNAPSHOT.jar`, **identical -SHA-256 in both images** (`38d3512e1ab66c0d66c94bebf09e121176fa8643 -b6d5fe2fc16400e26d8afda0`). The 2026-08-31 push is a container-image -rebuild (base OS/security layer) around the byte-for-byte same, -still-pre-fix application jar — not a new Egeria build. So: confirmed, -not just inferred from a label — **no published image yet includes the -2026-08-30 server-side merge.** The server accepts `?pushDown=false` -(200, no error — an unrecognized query param is simply ignored, not -rejected) but it has no effect: all three counts come back identical, -matching the exact pre-fix numbers this entry already had on record. -Test 3 (`SemanticAssignment`, an already-agreeing type) still returned -170=170 either way, as expected regardless of whether the platform has -the fix, so that one doesn't help distinguish "not deployed yet" from -"deployed and still broken." +**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). Bearer-token lifetime is configurable in `application.properties`. **pyegeria:** no code change; re-test means setting the property and decoding a token's `exp - iat`. -**This is a blocked verification, not a failed one** — per this entry's -own "if test 1 comes out unequal" clause, the honest reading given the -platform-date mismatch is "cannot yet tell," not "the fix doesn't work." -Leaving this entry in Open Egeria Server issues, as before. **Re-run once -a platform image built after 2026-08-30 is available** — the pyegeria-side -`push_down` parameter is ready and requires no further client-side work; -only the deployed Egeria server needs to catch up. +**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **Not re-testable yet.** Tokens still decode to `exp - iat = 3600`, as expected: Egeria `main` has `bearerTokenTimeout` (hours, default 1, `TokenService`), but the quickstart's `/deployments/application.properties` does not set it. Proving it needs that property set and an egeria-main restart. egeria-workspaces-fs is putting that to the owner as a separate step. -**Status:** re-confirmed 2026-08-18, byte-for-byte identical to the -2026-08-15 numbers below — `count_relationships_between_elements("Exception")` -still returns 58, `get_relationships("Exception")` still returns 57, same -off-by-one. No change. +**Re-test 2026-10-19, after the config change:** **PASS.** egeria-workspaces-fs set `bearerTokenTimeout=2` in the quickstart's `/deployments/application.properties` (the owner's choice) and restarted egeria-main. A fresh erinoverview token now decodes to `exp - iat = 7200`, re-checked from this session. The property is in hours. Ready to move to the Appendix. -**Status:** re-investigated 2026-08-15, narrowed and re-confirmed -**Egeria server, not pyegeria-fixable** — this is a genuine, real bug, just -smaller than originally measured (demo data has evidently changed since -2026-07-24: `count` is now 58 and `get_relationships` is now 57, not -276/55). Cross-checked against a **second, independent pyegeria list-based -method** — `MetadataExpert.find_relationships_between_elements` (a -different OMVS client, different endpoint, -`.../relationships/by-search-conditions` rather than -`ClassificationExplorer`'s `.../relationships/{type}`) — with the same -`relationshipTypeName: "Exception"` filter: it also returns exactly 57, -and its 57 relationship GUIDs are the **identical set** to -`get_relationships`'s 57 (zero difference either direction). Two -independently-implemented pyegeria code paths through two different -endpoints agree exactly with each other; only the server's native -`COUNT(*)` endpoint (`.../relationships/by-search-conditions/count`) -disagrees, by exactly 1. Also re-confirmed the original "not the type -filter" finding still holds with current data — re-ran `count` vs. -`get_relationships` for `SemanticAssignment` (401=401), `License` (2=2), -`Certification` (0=0), `AttachedRating` (0=0): all match; only `Exception` -diverges. Also checked whether the count includes non-`ACTIVE`/soft-deleted -relationships the list endpoints filter out — no: `limitResultsByStatus: -["ACTIVE"]` and `["ACTIVE","DELETED"]` both still return 58 from `count` -(no status-filter difference). Since both retrieval paths agree with each -other and only the count differs, there's genuinely nothing to fix on the -pyegeria side — this is squarely a server-side discrepancy between the -native COUNT(*) query and the list-materialization logic for this one -relationship type. No further pyegeria action needed; still needs -Egeria-side investigation to say what specifically triggers it for -`Exception`. +**Layer:** Egeria Server · **Status:** awaiting re-test (addressed in Egeria source 2026-09-19; not verified on the dev platform) · **Found:** 2026-09-04 (trellis-auth token contract) -**Original status:** open (Egeria server) — needs Egeria-side investigation. Found -2026-07-24 while wiring the Egeria Overview dashboard to native counting -(odpi/egeria#9168). Consolidated in from `egeria-workspaces-fs/ -PYEGERIA_ISSUES.md` 2026-08-05. +With `authentication.mode=token`, every token minted by the platform decodes to `exp - iat = 3600` +(observed on `omag-server-platform-6.2-SNAPSHOT`, quickstart, `authentication.source=platform`). +`application.properties` exposes `authentication.mode`, `authentication.source` and `rsa.key-id`, and +nothing that controls the lifetime; with `rsa.key-id` empty the signing key is regenerated at start, so +every token also dies on a platform restart. Client applications that hold a platform token on the +user's behalf (Portal, Egeria Advisor, Resource Explorer — which now carry the Egeria bearer token in +their session and never the password) cannot offer a session longer than an hour without either +re-prompting the user or caching a password, which is what a bearer token exists to avoid. -**Summary:** the OMF metadata-expert native relationship count and the -classification-explorer `get_relationships` return materially different -totals for the `Exception` relationship type — and *only* that type, among -those tested. +**Ask:** a property such as `authentication.token.lifetime=3600` (seconds, default unchanged), read at +start and applied when a token is issued; optionally a refresh operation that returns a new token for a +valid unexpired one. Full draft issue text: trellis session scratch `egeria-issue-token-lifetime.md`. -**How to trigger:** -```python -from pyegeria import MetadataExpert, ClassificationExplorer -me = MetadataExpert(view_server="qs-view-server", platform_url="https://localhost:9443", - user_id="erinoverview", user_pwd="secret"); me.create_egeria_bearer_token() -ce = ClassificationExplorer(view_server="qs-view-server", platform_url="https://localhost:9443", - user_id="erinoverview", user_pwd="secret"); ce.create_egeria_bearer_token() +**Correction, 2026-10-05 (Egeria team; confirmed in `RSAGenerator`):** the statement above that an empty `rsa.key-id` is why the signing key is regenerated is wrong. `RSAGenerator.generateRSAKeyPair(keyId)` always makes a fresh key pair and only uses the id as a label, so **no setting makes tokens survive a restart**; persisting the key is planned for the next release. Also: the lifetime property now exists, `bearerTokenTimeout` in HOURS (default 1), added in `a1f228d4a9` (2026-09-19); there is still no refresh operation. -me.count_relationships_between_elements({"class":"FindRelationshipRequestBody","relationshipTypeName":"Exception"}) -# -> 276 -len(ce.get_relationships(relationship_type="Exception", output_format="JSON", start_from=0, page_size=5000)) -# -> 55 (all 55 have exact typeName "Exception"; no effectivity dates) -``` -**What it is NOT:** -- Not the type filter — `count("SemanticAssignment")` = 397 = - `get_relationships` = 397; `License` 2 = 2; `Certification` 0 = 0; - `AttachedRating` 0 = 0. Every other tested type matches; only `Exception` - diverges. -- Not status/effectivity — `count("Exception")` is 276 with - `limitResultsByStatus=[ACTIVE]`, with `effectiveTime=`, and with - neither; the 55 `get_relationships` results carry no - `effectiveFromTime`/`effectiveToTime`. -- `count(no relationshipTypeName)` = 31857 (all relationships), so 276 is - a genuine type-scoped subset, not "count ignores the filter". +**Addressed in Egeria source (2026-10-05 check, carried over from the retired `docs/egeria-open-bugs-2026-10-05.md`):** commit `a1f228d4a9` (2026-09-19, "Add control of bearer token timeout") added `bearerTokenTimeout`, read by `TokenService`, in **hours** (default 1; the request asked for seconds). It is in `application.properties` and `container.application.properties`. **Not covered:** a token refresh operation. The signing key also changes on every restart: `rsa.key-id` does **not** prevent that (confirmed in `RSAGenerator.generateRSAKeyPair`: it always makes a fresh key pair and uses the id only as a label), and the Egeria team plans to persist the signing key in the next release. **Remaining:** verify on a platform built after 2026-09-19. -**Open question for Egeria:** what does the metadata-expert count include -for `Exception` that the classification-explorer traversal excludes -(subtypes counted under the supertype? relationships to non-visible/ -anchored/dangling ends? access/zone filtering that differs between the two -OMVS)? Whichever is "true", the two APIs should agree for a given type — -or the difference should be documented. +--- -**Impact / workaround:** egeria-workspaces-fs's Overview dashboard keeps -**relationship** counts on `ClassificationExplorer.get_relationships` (so -"Open Exceptions" stays consistent with the Audit app at 55) and uses -native counting only for **element** counts. + +# Quick reference: which OMVS client class for which purpose + +| Need | Class | Notes | +|---|---|---| +| Business reference data (country/currency codes) | `ReferenceDataManager` | Does **not** cover specification properties (ISSUE-19, docs-only) | +| Valid metadata values for a property name | `ReferenceDataManager` or `MetadataExpert` | `get_valid_metadata_values` lives on shared `ServerClient` base; no `as_of_time` support — Egeria endpoint doesn't expose it (ISSUE-18) | +| Specification properties (placeholders, guards, action targets, etc.) | `SpecificationProperties` | `get_specification_property_by_type` now works with either PascalCase or `SCREAMING_SNAKE_CASE` input (ISSUE-17, fixed 2026-08-15); `find_specification_property` with `graph_query_depth=0` also available (ISSUE-15); `get_specification_property_by_guid` works too, `NameError` fixed (ISSUE-28, fixed 2026-08-05, re-verified 2026-08-15) | +| `DataGrain` / `DataClass` listing | `find_data_value_specifications` / `get_data_value_specifications_by_name("*")` | Both fixed (ISSUE-1, ISSUE-2) | +| `DataSpec` (Collection subtype) | `CollectionManager.find_collections(metadata_element_type="DataSpec")` | | +| `DataStructure` / `DataField` | `DataDesigner.find_data_structures` / `find_data_fields` | | +| Solution blueprints/components (any pyegeria version) | `SolutionArchitect.find_solution_blueprints/components(search_string="*")` | Avoid `find_all_*` variants on old versions (ISSUE-11) | +| Note logs (list) | `find_note_logs("*", graph_query_depth=0)` | ISSUE-15 | +| Note logs (entries) | `get_notes_for_note_log(guid, page_size=100)` | ISSUE-3 — never pass `metadata_element_type_name="NoteLog"` | +| Collection members | `get_collection_members(collection_guid)` | ISSUE-8 — now returns members of any type, not just the collection's own type | +| Comparing results across two runs/environments that don't match | — | Check whether the same user's credentials were used in both — governance zone visibility can legitimately change results per-user (ISSUE-29) before assuming a pyegeria bug | +| Multi-classification search (`matchClassifications`, 2+ conditions) | `MetadataExpert.find_metadata_elements` | Fixed in Egeria server (ISSUE-35) | +| Paging a `find_metadata_elements` result | Set `"startFrom"`/`"pageSize"` **in the body dict** | Fixed (ISSUE-34) — these are NOT separate parameters on this method anymore; passing them as kwargs is silently a no-op. Same for `"graphQueryDepth"`. | +| Relationships for a single element by guid | `MetadataExpert.get_all_related_elements(guid)` | **Not** `get_metadata_element_by_guid` — that call never returns relationships, by design (ISSUE-37, not a bug) | +| Project parent/child hierarchy (any linked project, not just hierarchy) | `ProjectManager.get_linked_projects(guid)` | Fixed (ISSUE-42) — was silently returning "No elements found" regardless of real data | +| Checking whether a `get_guid_for_name` lookup found anything | `guid_or_none(client.get_guid_for_name(name))` | ISSUE-114 — a miss returns a truthy sentinel string ("No elements found"), so `if not guid` reads it as found; the helper returns `None` instead | --- -## Open Egeria Server issues (not fixable in pyegeria) +# Appendix: Closed / Not-a-bug entries +## Fixed / Resolved -These need an Egeria Server-side fix, an upstream bug report, or a -deployment/config change — nothing here can be resolved by editing this -repo. Where an entry has a known pyegeria/Dr.Egeria follow-on once the -Egeria side lands, it's noted inline (also cross-referenced from the -"Open pyegeria items" section below where the follow-on is substantial -enough to track there too). +Sorted newest-first by `ISSUE-#` (2026-10-12). ISSUE-91 has two entries +(the same defect reported twice); both are kept, next to each other. --- ### ISSUE-166: Security annotation properties are Java primitives, so an unset CVSS score, exploit/patch flag or posture count reads back as 0 / false -**Layer:** Egeria Server (type model / properties classes) · **Status:** fixed (Egeria 6.3, odpi/egeria#9389) · **Found:** 2026-10-26, while adding the 6.3 security annotation types to pyegeria (odpi/egeria-workspaces#702 hand-off, step 3; PR #495). +**Layer:** Egeria Server (type model / properties classes) · **Status:** fixed (Egeria 6.3, odpi/egeria#9389); pyegeria adapted in #498, released in 6.2.13 · **Found:** 2026-10-26, while adding the 6.3 security annotation types to pyegeria (odpi/egeria-workspaces#702 hand-off, step 3; PR #495). **FIXED 2026-10-27: odpi/egeria#9389, merged as 12b1aaae.** Verified live the same day on egeria-platform c2ca054c (throwaway VulnerabilityAnnotation ed152edb, deleted, NotFound by GUID): unset cvssScore/exploitAvailable/patchAvailable were omitted; a merge update that left out 8.6 / true / false kept them; an explicit false came back. The pyegeria workarounds below are removed on branch `feature/boxed-properties-audit`. It boxes 217 number and flag fields across 86 property beans, not only the security ones. An unset value is no longer stored and is left out of the response, and a merge update keeps a value it does not mention. Defaults move to `OpenMetadataProperty.defaultValue`, published in the Core Content Pack as `additionalProperties.defaultValue` on each property's valid metadata value. `allowsDuplicateValues` now defaults to true everywhere, including SchemaAttribute (was false). The PR does not migrate stored data: elements written by earlier servers keep the 0 / false they were stored with. **pyegeria:** fixes that work with either server are on branch `feature/boxed-properties-audit`. The client-side workarounds stay until the PR merges and pyegeria targets that server. @@ -1863,319 +1924,358 @@ So "never set" and "set to 0 / false" come back identical. Confirmed live on 202 `newFindingCount` / `resolvedFindingCount` of 0 cannot be told apart from unrecorded on the client at all, and the `Security-Posture` description says so. Once Egeria returns `null` for unset values, the workaround can go, apart from the "None" severity rule, which stays true either way. ---- +--- + + +### ISSUE-165: 19 SDK methods called endpoints Egeria doesn't serve: 13 removed upstream, 6 never existed (2 since repointed) + +**Layer:** Pyegeria · **Status:** resolved: replacements implemented as deprecated wrappers and verified live (#485, released in 6.2.11); `get_platform_organization` repointed (2026-10-24) · **Found:** 2026-10-23 by `scripts/omvs_audit.py` JAVA-MISSING (ISSUE-164). + +**Validation.** Each URL was checked three ways: +1. **No Spring mapping** in Egeria `main` 4486d1a7f1 (2026-10-23), the checkout `omvs_audit.py` reads. +2. **Absent from the live platform's OpenAPI doc** (`GET https://localhost:9443/v3/api-docs`, which `omvs_audit.py --live` reads; it marks these rows "[LIVE: absent, so a confirmed 404]"). +3. **Git history:** `git log -S` on the path fragment, to find the commit that removed it, or to show it was never in Egeria's Java. +4. **Not moved to another module** (owner's question): the same path tail is not served under any other view service, in Java or live. The mappings added by each removing commit show no renamed replacement, with one exception. `d299b120f4` moved the governance-action-process **step-link** endpoints to Action Author; pyegeria already calls them there (`action_author.py`, `ActionProcessStepLinkProcessor`). Process search was not re-added anywhere. The audit now tags a JAVA-MISSING row "MOVED?" when the tail is served elsewhere; there are 0 such rows today. + +Three other JAVA-MISSING rows are served live, so they are Java-scan false positives and are excluded: `get_classified_elements_by` (runtime `by-{}` segment), `get_search_keyword_by_guid` and `new_subscription_type`. + +**Removed from Egeria:** + +| SDK method (file:line) | Calls | Removed by Egeria | Replacement in Egeria now | +|---|---|---|---| +| `data_discovery`: `create/update/delete_analysis_report`, `get_analysis_reports_by_name`, `find_analysis_reports`, `get_analysis_report_by_guid`, `link/detach_analysis_report_to/from_asset` (`data_discovery.py:1692-1773`, 8 methods) | `data-discovery/analysis-reports/...` | `e17cc09d44`, 2025-08-04 "Radical simplification of platform" | None as a CRUD API. Data Discovery now serves only `survey-reports/{g}/new-annotations[/{g}/attach\|detach]`; survey reports are produced by survey engines. The `AnalysisReportProperties` class these methods use doesn't exist either. | +| `classification_explorer.add_element_to_subject_area` / `remove_element_from_subject_area` (`classification_explorer.py:12293/12405`) | `classification-explorer/elements/{g}/subject-area-member[/remove]` | `6a49a3e23f`, 2026-01-09, which also removed `elements/by-subject-area-membership` | Not confirmed. Likely collection membership on the subject area (`{urlMarker}/collections/{g}/members/{g}/attach\|detach`, served by the generic CollectionManager resource); ask the Egeria team. `SubjectAreaMemberProperties` doesn't exist either. | +| `governance_officer.find_governance_action_processes` (`governance_officer.py:2310`) | `governance-officer/governance-action-processes/by-search-string` | `d299b120f4`, 2025-10-16, which removed all of Governance Officer's process create/update/publish/withdraw/remove/search endpoints | Not confirmed; a generic search by type `GovernanceActionProcess` is the likely route. `initiate` and `cancel` live on Automated Curation / Governance Engine (ISSUE-153). | +| `platform_services.get_platform_organization` (`platform_services.py:1083`) | `GET platform-services/server-platform/organization-name` | `816076ab50`, 2026-06-05 "Improve OMAGServerPlatform properties…" | **Fixed 2026-10-24:** Mandy Chessell confirmed the value is now the `organizationName` field of `GET /api/public/app/info`, and the method reads it from there. | + +**Never in Egeria's Java (the SDK URL was a guess):** + +| SDK method (file:line) | Calls | What Egeria has | +|---|---|---| +| `data_designer.create_data_classification`, `update_data_classification`, `link_data_classification_to_data_field`, `detach_data_classification_from_data_field` (`data_designer.py:3298/3465/3601/3709`) | `data-designer/data-classifications/...` | Only `data-classes/{g}/nested-data-classes/{g}/attach\|detach`; there is no "data classification" element API. The `DataFieldClassificationProperties` class doesn't exist either. | +| `subject_area.link_subject_area_hierarchy` / `detach_subject_area_hierarchy` (`subject_area.py:408/495`) | `subject-area/collections/{g}/collection-hierarchies/{g}/attach\|detach` | `subject-area/subject-areas/{parent}/subject-area-hierarchies/{nested}/attach\|detach`. **Fixed 2026-10-23** (owner approved): repointed. The same module already had `link_subject_areas` / `detach_subject_areas` on the correct URLs, so these were broken duplicates; both names are kept for existing callers. | +| `governance_officer.get_gov_def_in_context` (`governance_officer.py:2796`) | `governance-officer/governance-definitions/{g}/retrieve-context` | Only `governance-definitions/{g}/retrieve` (plus update/delete and the peer/supporting links). | + +**Known consumer:** Dr.Egeria's `View Governance Definition Context` command (`GovernanceContextProcessor`, `md_processing/v2/governance.py:775`, registered in `dr_egeria.py:207`) calls `get_gov_def_in_context`, so that command 404s too. No other Dr.Egeria or CLI code calls these methods. + +**Every call above returns a 404 today.** Before ISSUE-164, pyegeria reported a 404 as "unable to connect to the Egeria platform", so a caller could not tell "this API is gone" from "the server is down". + +**Egeria team's answer (Mandy Chessell, 2026-10-23), and what pyegeria now does** (branch `fix/issue-165-replacements`). Each obsolete method is now a deprecated wrapper (`DeprecationWarning`) that delegates to the replacement, following the ISSUE-153 pattern; removal comes in a later release: + +| Obsolete | Why (Mandy) | Now delegates to | +|---|---|---| +| analysis-report create/update/delete/get/find | SurveyReport now inherits from Report, so it is an Asset | `AssetMaker.create_asset(["SurveyReportProperties"])` / `update_asset` / `delete_asset` / `get_asset_by_guid` / `get_assets_by_name` / `find_assets`, with type `SurveyReport`. A body naming `AnalysisReportProperties` is rewritten to `SurveyReportProperties`. | +| `link/detach_analysis_report_to/from_asset` | use the ReportSubject relationship, `/elements/{subjectGUID}/reports/{reportGUID}/attach` | `AssetMaker.link_report_subject` / `unlink_report_subject(asset_guid, report_guid)` | +| `add/remove_element_to/from_subject_area` | SubjectArea now inherits from Collection; use collection membership | `CollectionManager.add_to_collection` / `remove_from_collection(subject_area_guid, element_guid)`. The old methods never took the subject area, so they gain an optional `subject_area_guid`; without it they raise an error naming the replacement. | +| `find_governance_action_processes` | GovernanceActionProcess now inherits from GovernanceDefinition | `find_governance_definitions(..., metadata_element_type="GovernanceActionProcess")` | +| `get_gov_def_in_context` | no longer needed: OpenMetadataRootElement returns the context by default | `get_governance_definition_by_guid`. Dr.Egeria's `View Governance Definition Context` calls it directly. | +| data-classification create/update/link/detach | "probably the old data class operations", now data value specifications (the common root of DataClass and DataGrain); **Mandy was not certain** | `create/update_data_value_specification` (a body naming `DataClassificationProperties` is rewritten to `DataClassProperties`) and `link/detach_data_value_assignment(data_field_guid, spec_guid)` (`DataValueAssignmentProperties`). A best-effort mapping. | + +`get_platform_organization` (removed in `816076ab50`): Mandy confirmed on 2026-10-24 that the same value is returned by `GET /api/public/app/info`. The method now returns that response's `organizationName`, not a deprecated wrapper, because its signature and meaning are unchanged. That response has no `relatedHTTPCode` envelope, so it is requested with `is_json=False`. Verified live: it returns `'Coco Pharmaceuticals'` on quickstart 9443. Tests: `tests/micro-tests/test_issue165_replacements.py` (9). + +**Live on quickstart 9443, 2026-10-24** (Egeria 81beb8e5 after the reset; one peer-cleared run as `peterprofile` with throwaway `pyeg-test::issue165::` elements, all 5 deleted and read back as NotFound by GUID, and the prefix find was empty afterwards). Each wrapper reached its replacement and the replacement had the intended effect: +- `find_governance_action_processes("*")` returned 100 GovernanceActionProcess-family elements (GovernanceActionProcess plus its Provisioning/Deleting/Subscribing subtypes). +- `get_gov_def_in_context` returned the same element as `get_governance_definition_by_guid`, including `relatedActions`, `associatedGovernanceActions` and `subscribers`, which confirms Mandy's point that the context now comes by default. +- `add_element_to_subject_area` without `subject_area_guid` raised the guidance error. With it, the term was listed in the subject area's collection members; after `remove_element_from_subject_area` the subject area had no members. +- `create_analysis_report` created a `SurveyReport`; `get_analysis_report_by_guid` and `find_analysis_reports` returned it. After `link_analysis_report_to_asset(report, term)` the report showed the term under `reportSubjects`; after the detach it didn't. +- `create_data_classification` created a `DataClass`. After `link_data_classification_to_data_field` the field showed it under `assignedDataValueSpecifications`; after the detach it didn't. + +**Proposed (not made):** +1. ~~Repoint `link/detach_subject_area_hierarchy`~~ (done). +2. ~~Mark the obsolete methods deprecated~~ (done, as delegating wrappers rather than errors, per the Egeria team's answer). +3. ~~Ask the Egeria team for the intended replacements~~ (answered, except the platform organization name). + +--- + + +### ISSUE-164: `omvs_audit` now checks against Egeria's Java source. The first sweep finds 6 always-400 calls, 4 more 400s with a body, and 24 endpoints with no Spring mapping -### ISSUE-147: Every valid-metadata clear call answers OPEN-METADATA-400-011 "not found", even after a successful delete; clear-map-value deletes nothing; clear-map-name leaves the map's value set behind +**Layer:** Pyegeria (+ Dr.Egeria, CLI) · **Status:** audit improved, and the findings were fixed in #483, released in 6.2.10. The items under *Not fixed* at the end are tracked in ISSUE-165, ISSUE-132 and ISSUE-155, or are the owner's call · **Found:** 2026-10-23, sweep requested by the owner before the next release, after ISSUE-163. -**Egeria reset 2026-10-23 ~21:10-23:54Z** (egeria-workspaces-fs, owner-approved): the quickstart database was wiped and Egeria recreated on build 81beb8e5 (odpi/egeria 4486d1a7), then Coco content was reloaded. Every GUID recorded before then is gone. The leftover `pyegeriaClearRetest*` test elements described under "Leftovers on qs-view-server" are gone with it; nothing to clean up. The fix had already passed its 2026-10-19 re-test. +**Fixed 2026-10-23, in the owner's order:** +1. **Anchor scope and find keys** (3bb233c1): `anchorScopeGUIDs` serialisation on the New/Relationship/Template models; the `_async_find_request` keys. +2. **Always-400 deletes and wrong class names** (b8b4567b): `remove_is_abstract_concept`, `remove_activity_description`, `detach_cohort_member`, `remove_search_keyword`, `initiate_engine_action` (`InitiateEngineActionRequestBody`, `actionSourceGUIDs`, `startDate`), `DataValueMeaningProperties`, `SolutionActorRoleProperties`, `DataProcessingPurposeProperties`, and the CLI todo typo. +3. **Model gaps and Dr.Egeria/CLI keys** (13c17e37): `ActionRequestBody` anchor fields and `newActionTargets`; `TemplateRequestBody` `parentRelationshipProperties` and `replacementClassifications`; `endsWith`; Dr.Egeria/CLI `anchorScopeGUIDs`; the CLI update `properties`. -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. A successful clear now returns success; clear-map-value now uses the map name; clear-map-name also removes that map name's values. Egeria asks us to drop any workaround that treats 400-011 as success. **pyegeria:** there is none (checked 2026-10-18: no 400-011 handling in `md_processing/v2/reference_data.py` or `pyegeria/omvs/valid_metadata.py`), so the follow-on proposed below is withdrawn. Nothing to change; after the re-test, remove the leftover `pyegeriaClearRetest*` test elements on qs-view-server. +Tests: `tests/micro-tests/test_issue164_silent_field_drops.py` (15 tests). -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** On throwaway properties `pyegRtClear`/`pyegRtClearMap` (typeName Asset), `clear_valid_metadata_value` (alpha, beta), `clear_valid_metadata_map_value` (mapOne/v1) and `clear_valid_metadata_map_name` (mapTwo, mapOne) all return success. A second clear of an already-removed value still answers 400-011 "not found", which is correct. Values and map names were gone afterwards. One oddity: `get_valid_metadata_values` read in the same run straight after the clears still listed alpha/beta/mapOne/mapTwo, but a read a minute later was empty. Possibly a short server-side cache; not pursued. Setup creates per-property container sets (`Egeria:ValidMetadataValue::pyegRtClear-`, `...pyegRtClearMap-`) that the clears do not remove; deleted by GUID. **Not cleaned (outside what peers cleared):** the 2026-10-13 leftovers under `pyegeriaClearRetest*`, which the platform has since turned into DataProduct/DigitalProduct elements (e.g. `98de81ed`, `4b937012`). +**Live on quickstart 9443, 2026-10-23** (three peer-cleared runs, throwaway `pyeg-test::issue164::` glossary/terms/solution role; every GUID read back as gone and the prefix find was empty after each run): +- **Anchor scope stored:** a term created with `anchorScopeGUIDs: []` reads back with `elementHeader.anchor.classificationProperties.anchorScopeGUIDs == []`; a term created without it has none. +- **Classification add/remove round trips:** AbstractConcept, ContextDefinition, ActivityDescription and DataValue (stored as `DataValueMeaning`, matching the corrected class) each appear in `elementHeader.glossaryTermKinds` after the add and are gone after the remove. Before the fix, `remove_is_abstract_concept` was a 400 on every call. +- **Solution role:** `create_solution_role` with `SolutionActorRoleProperties` created, then deleted. +- **Not verified live:** + - The find zone filter: an unzoned element matches any zone filter, and zoning a throwaway element could make it undeletable for the test user. Verified at the wire level only. + - `remove_search_keyword`: my test body was wrong (`keyword`; Egeria's field is `displayName`, as the docstring says). + - `detach_cohort_member`, `initiate_engine_action`: not exercised, by design. -**Layer:** Egeria Server (`ValidMetadataValueHandler`) · **Status:** open · **Found:** 2026-10-13. The first live run of the Valid Metadata commands showed it (the error text wasn't captured then). Re-tested the same day with the full responses captured. +**Egeria 6 element shape (for anyone reading classifications back; RE's read-backs too).** The view-service `get_*_by_guid(output_format="JSON")` calls (glossary, collection, actor manager, …) return the element with `elementHeader` and `properties`, and that header **has no `classifications` list**. The lower-level Metadata Expert / OMF shape (`get_metadata_element_by_guid`, an OpenMetadataElement) is different: it still has a top-level `classifications` list (Thursday Coordinator's read-only probe, 2026-10-23). Readers must know which shape they hold. Each classification is a named camelCase slot on `elementHeader` (`anchor`, `zoneMembership`, `retention`, `glossaryTermKinds`, `projectClassification`, `memento`, … per Java `ElementHeader`), and anything unslotted is in `elementHeader.otherClassifications`. A slot holds an `ElementClassification`, or a list of them for the `…Kinds` slots, each with `classificationName` and `classificationProperties`. Trimmed raw sample from this run: +```json +"elementHeader": { + "guid": "97581996-622d-46fe-82e0-0feaeea17132", + "anchor": {"class": "ElementClassification", "classificationName": "Anchors", + "classificationProperties": {"class": "AnchorsProperties", "anchorTypeName": "GlossaryTerm", + "anchorDomainName": "AuthoredReferenceable", + "anchorScopeGUIDs": ["057d4bd3-62bc-4591-8777-410f7528d867"]}}, + "glossaryTermKinds": [{"class": "ElementClassification", "classificationName": "AbstractConcept", "...": "..."}] +} +``` +Note that the anchor property is the plural `anchorScopeGUIDs`, a list. Code that looks for a generic `classifications` list, or for a singular `anchorScopeGUID`, finds nothing. That is how this run's first two read-backs reported false absences. -**Re-test:** run once on qs-view-server as erinoverview, with all four live sessions clearing it first. It used made-up properties `pyegeriaClearRetest` and `pyegeriaClearRetestMap` (typeName `Asset`) and called the SDK methods in `pyegeria/omvs/valid_metadata.py`. The Dr.Egeria `Clear Valid Metadata *` commands call the same methods (`md_processing/v2/reference_data.py:198-242`). +**Correction to the triage below:** `RetentionClassificationProperties` and `SecurityGroupMembershipProperties` are **not** bugs. Egeria registers them as `@JsonSubTypes` aliases (`names = {...}`): `RetentionProperties` is also accepted as `"RetentionClassificationProperties"`, and `processors.py` records a 2026-08-03 live 400 for `RetentionProperties` on an older build. Left unchanged; the audit now reads registered names and aliases. -| Call | Response | Actual effect | -|---|---|---| -| `clear_valid_metadata_value(alpha)`, set up just before | 400-011 "value alpha ... is not found" | **alpha removed** (gone from the list afterwards) | -| same call again (alpha already gone) | 400-011 | nothing to remove | -| `clear_valid_metadata_value(beta)` | 400-011 | **beta removed** | -| `clear_valid_metadata_map_value(mapOne, v1)` | 400-011 "value v1 ... is not found" | **nothing removed**: `get_valid_metadata_map_value` still returns v1 | -| `clear_valid_metadata_map_name(mapTwo)` | 400-011 | **mapTwo removed** | -| `clear_valid_metadata_map_name(mapOne)`, with v1 under it | 400-011 | the mapOne name value is removed, but the map's value set "mapOne" (holding v1) stays in `get_valid_metadata_values` | +**Audit changes** (`scripts/omvs_audit.py`, docs in CLAUDE.md and `scripts/README.md`, tests in `tests/micro-tests/test_omvs_audit_java_truth.py`): +- **RENAMED matches** now get the verb/body checks too; they used to be skipped. +- **Java ground truth** (`--java-dir`, `$EGERIA_JAVA_DIR`, default `~/localGit/egeria-v6/egeria`; `--no-java` skips it). The audit indexes 1,587 Spring endpoints → `@RequestBody` type. + - **JAVA-BODY REJECTED** (defect): fires only when the declared type sits under a `@JsonTypeInfo(property="class")` base. Among request bodies that's only `DeleteRequestBody` → `Delete{Element,Relationship,Classification}RequestBody`, where a sibling class is a 400. + - **CLASS-IGNORED** (review): everywhere else Egeria's DTOs are plain `@JsonIgnoreProperties(ignoreUnknown=true)`, so `"class"` is ignored and fields map by name. + - **UNKNOWN-CLASS** (defect): any `"…Properties"`/`"…RequestBody"` string in `pyegeria/omvs`, `md_processing` or `commands` that is no Egeria class and no pyegeria class. The properties bases (`RelationshipBeanProperties`, `ClassificationBeanProperties`, …) are polymorphic, so an unknown name there is a 400. + - **JAVA-MISSING** (review): an SDK verb+path with no Spring mapping. + - **HTTP-vs-JAVA** (info): 21 `.http` entries name a body class Java doesn't accept. When Java accepts what the SDK sends, the `.http` BODY mismatch is no longer counted. +- **Regression check:** on the pre-ISSUE-163 code the audit flags all eight ISSUE-163 bugs: 5 REJECTED and 3 UNKNOWN-CLASS, the detaches via the RENAMED path. -Request ids: `ec9052e7` and `3975182d` (alpha), `38759736` (beta), `475e68f1` (map value), `407c0e17` and `07886f29` (map names). pyegeria surfaces each one as `SERVER_ERROR_500` wrapping `relatedHTTPCode` 400 `InvalidParameterException`. +**Findings, current branch**, against Egeria `main` 4486d1a7f1 (2026-10-23). `python scripts/omvs_audit.py --quiet` reports java_ok=951, rejected=4, class_ignored=25, unknown_class=16, java_missing=22, field_drop=3, client_drop=13, model_gaps=27, http_vs_java=21. An earlier run against a 2026-10-06 checkout listed 24 JAVA-MISSING; `cancel_governance_action_process` and `get_approved_engine_actions` (ISSUE-153's Governance Engine paths) now resolve. -**Cause (Egeria source, `af400039c2`, `frameworks/open-metadata-framework/.../handlers/ValidMetadataValueHandler.java`):** -1. **No `return` after the delete (lines ~541-680).** In all three `clear*` methods, `if (validValueGUID != null) deleteMetadataElementInStore(...)` is followed unconditionally by the audit log and `throw new InvalidParameterException(VALID_METADATA_MISSING ...)`. So every call throws, whether or not it deleted anything. The "not found" error is the method's normal exit. -2. **Map name dropped in `clearValidMetadataMapValue`.** It calls `getValidValueGUID(userId, typeName, propertyName, null, preferredValue)`, passing `null` where `mapName` belongs. It therefore searches for `v1` as a plain value of the property, finds nothing, and deletes nothing. -3. **`clearValidMetadataMapName` deletes only the name's own value.** It doesn't touch the per-map value set created by `setup-map-value` (the "Set of valid values for the mapOne map name ..." element) or its members. Whether that should cascade is Egeria's call. +*Always fails: the SDK always sends this, so every call is a 400 or a client-side ValidationError.* -**Effect on Dr.Egeria:** `Clear Valid Metadata Value` and `Clear Valid Metadata Map Name` report failure although the clear worked. `Clear Valid Metadata Map Value` reports failure and really did fail. +| Where | What | Fix (from Java) | +|---|---|---| +| `glossary_manager.remove_is_abstract_concept` | element helper always sends `DeleteElementRequestBody`; endpoint takes `DeleteClassificationRequestBody` | `_async_delete_classification_request` | +| `runtime_manager.detach_cohort_member` | `_async_metadata_source_body_request` always sends `{"class":"MetadataSourceRequestBody"}`; endpoint takes `DeleteRelationshipRequestBody` | `_async_delete_relationship_request` | +| Dr.Egeria `Retention` classify (`md_processing/v2/curation.py:136`, `processors.py:237`) | body `class` is `RetentionClassificationProperties` | `RetentionProperties` | +| CLI todo status update (`commands/my/todo_actions.py:201`) | typo `UpdataElementRequestBody`; pyegeria's Literal validator rejects it before sending | `UpdateElementRequestBody` | +| `data_discovery` analysis-report methods (8) | `analysis-reports/...` has no mapping in `DataDiscoveryResource`; class `AnalysisReportProperties` doesn't exist | confirm whether Egeria still has the endpoint | +| `classification_explorer.add/remove_element_to/from_subject_area` | `elements/{g}/subject-area-member` has no mapping; class `SubjectAreaMemberProperties` doesn't exist | confirm | -**Leftovers on qs-view-server:** the map value set "mapOne" with member v1 under `pyegeriaClearRetestMap`/`Asset`, plus whatever per-property container elements setup creates (`Egeria:ValidMetadataValue::pyegeriaClearRetest-Asset` and `...pyegeriaClearRetestMap-Asset`). These are test elements, not content-pack elements. RE's blueprint materializer treats `Egeria:ValidMetadataValue:` qualifiedNames as content-pack origin, but only for SolutionComponents, so they don't affect it. +*Fails only when a no-body default or a docstring-style body is used. With the caller's own correct body these work.* -**pyegeria follow-on (not done, needs the owner's go-ahead):** Dr.Egeria's clear processor could treat 400-011 from `clear-value`/`clear-map-name` as success when a follow-up read confirms the value is gone. Otherwise, wait for the Egeria fix and re-test (see the BACKLOG.md delete re-test item). +| Where | What | Fix | +|---|---|---| +| `feedback_manager.remove_search_keyword` | relationship body to an element `/remove` | `_async_delete_element_request` | +| `glossary_manager.remove_activity_description` | relationship body; endpoint takes `DeleteClassificationRequestBody` | `_async_delete_classification_request` | +| `actor_manager.add_security_group_membership` default class | `SecurityGroupMembershipProperties` | `SecurityListMembershipProperties` | +| `classification_explorer.set_retention_classification` default class | `RetentionClassificationProperties` | `RetentionProperties` | +| `glossary_manager.add_is_data_value` default class | `DataValueProperties` | `DataValueMeaningProperties` | +| `solution_architect.create/update_solution_role` default class | `SolutionRoleProperties` | `SolutionActorRoleProperties` (extends `ActorRoleProperties`; the endpoint is `createActorRole`) | +| `data_designer.link_data_classification_to_data_field` | `data-classifications/.../data-fields/{g}/attach` has no mapping; `DataFieldClassificationProperties` doesn't exist. The create/update/detach data-classification methods are also unmapped | confirm | +| `GOV_DEF_PROPERTIES_LIST` (`governance_officer.py:40`) | `GovernanceProcessingPurposeProperties` | `DataProcessingPurposeProperties`; probably one of ISSUE-154's renames | ---- +*Harmless on the wire: an unknown top-level `class` that Egeria ignores.* `GovernanceActionRequestBody` (`automated_curation.initiate_engine_action`), `IntegrationServiceRequestBody` (`full_omag_server_config`), `AnyTimeRequestBody` (`metadata_expert.get_relationship_by_guid`, three Solution Architect retrieves) and `FindProperties` (`classification_explorer.retrieve_instance_for_guid`). Worth renaming for clarity only. -### ISSUE-143: `includeOnlyRelationships: ["PersonRoleAppointment"]` returns no relationships for a Person, although the unfiltered read has its role appointments +*JAVA-MISSING (meaning no such URL in Egeria; it is not about classes), other entries to confirm.* `get_search_keyword_by_guid`, `find_governance_action_processes`, `get_gov_def_in_context`, `get_platform_organization`, `new_subscription_type`, and `subject_area.link/detach_subject_area_hierarchy`. The last calls `collections/{g}/collection-hierarchies`, while Java has `subject-areas/{g}/subject-area-hierarchies/{g}/attach|detach`, so it is likely a real 404. `get_classified_elements_by` (`by-{}`) is a static-analysis false positive. -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. The `includeOnlyRelationships` filter now leaves already-retrieved relationships in place, so `performsRoles` survives `["PersonRoleAppointment"]`. **pyegeria:** no code change; tell the Resource Browser (egeria-workspaces-fs) it can use the filter again after the re-test. +*CLASS-IGNORED (25), review only.* Mostly Filter-vs-Results and MetadataSource-vs-DeleteClassification pairs that are harmless. Worth a field-level look: `metadata_expert.create_metadata_element_from_template` (Java `OpenMetadataTemplateRequestBody`) and `metadata_expert.classify_metadata_element` (Java `NewOpenMetadataClassificationRequestBody`). Both Java classes use typed `ElementProperties`, so the flat SDK shape may silently lose fields; see CLAUDE.md's note on `MetadataExpert` create. Also `project_manager.create_project_task` (Java `NewAttachmentRequestBody`). -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** Erin's Person `10dc7da9` read with `includeOnlyRelationships: ["PersonRoleAppointment"]` returns `performsRoles` 8 (unfiltered: performsRoles 8, assignmentScope 15, requestedActions 13, ...; `["ContactThrough"]` still returns contactDetails 1). Read-only. Tell the Resource Browser it can use the filter again. +**Field-level findings.** These calls succeed but silently ignore an option. Egeria's DTOs are `ignoreUnknown=true`, and pyegeria's models are `extra='ignore'`, so a misnamed field disappears on either side with no error. -**Layer:** Egeria Server (not pyegeria) · **Status:** open, not yet reported to the Egeria team · **Found:** 2026-10-13, reported by the egeria-workspaces-fs Resource Browser session; reproduced read-only the same day. +*CLIENT-DROP: pyegeria's own model discards the key before sending.* -**What:** reading a Person with a `GetRequestBody` whose -`includeOnlyRelationships` is `["PersonRoleAppointment"]` returns the element -with no relationships at all. The same read with no filter returns 8 -`performsRoles` entries, each with relationship type `PersonRoleAppointment`. -Other filters on the same Person work. +| Where | Key sent | Model/Java field | Effect | +|---|---|---|---| +| `_async_find_request` (`_server_client.py:6543`, every `find_*` without property names) | `zoneFilter`, `relationshipPageSize`, `anchorScopeGuid`, `anchorTypeName` | `governanceZoneFilter`, `relationshipsPageSize`, `anchorScopeGUID`; `anchorTypeName` is not a field at all | `governance_zone_filter`, `relationship_page_size` and `anchor_scope_guid` never take effect on any find | +| My Profile to-do / meeting / review create (`my_profile.py:1314/1431/1523`) | `isOwnAnchor` | Java `ActionRequestBody` (via `AnchorOptions`) has it; **pyegeria's `ActionRequestBody` model lacks it**, and also `anchorGUID`, `anchorScopeGUIDs` and `initialClassifications`. It names Java's `newActionTargets` `actionTargetGUIDs` | not anchored as intended | +| Dr.Egeria `common_md_utils.py:263` | `endWith` | `endsWith` | the "ends with" search option is ignored | +| Dr.Egeria `common_md_utils.py:292/323/737`, `v2/schema_maker.py:28`, `v2/asset_maker.py:57`, CLI `glossary_actions.py:123` | `anchorScopeGUID(s)`, `governanceZoneFilter` on create/update bodies | the model reads `anchorScopeGuids` (see below); create/update bodies have no zone filter | anchor scope is dropped | +| Dr.Egeria `v2/asset_maker.py:57` | `parentRelationshipProperties`, `typeName` | Java `TemplateRequestBody` has `parentRelationshipProperties` (and `replacementClassifications`); **the pyegeria model lacks both** | parent-relationship properties are dropped | +| CLI `glossary_actions.py:178` | `elementProperties` | `properties` | update has no properties | +| `validate_delete_classification_request` default (`_server_client.py:6244`) | `cascadeDelete` | not on the model, and Java spells it `cascadedDelete` | ISSUE-132 | -**Reproduction** (Person `10dc7da9-e9e1-471f-9afd-46a7ff414cb7`, Erin Overview, -as erinoverview on qs-view-server; -`ClassificationExplorer.get_element_by_guid(guid, graph_query_depth=1, -output_format="JSON", body={"class": "GetRequestBody", "graphQueryDepth": 1, -"includeOnlyRelationships": [...]})`): +*Model spelling that Egeria discards. Confirmed: anchor scope cannot be set through pyegeria at all.* The `NewElementRequestBody`, `NewRelationshipRequestBody` and `TemplateRequestBody` models alias `anchor_scope_guids` as `anchorScopeGuids`, while Java's `MakeAnchorOptions` reads `anchorScopeGUIDs`. A dict using the documented `"anchorScopeGUIDs"` (as in `collection_manager.py`'s docstrings) is dropped client-side. `anchorScopeGuids` / `anchor_scope_guids` are sent, then dropped server-side. Checked by `model_validate` → `model_dump_json`. This affects 47 create, 157 link and 26 from-template methods; one alias fixes it. -| Filter | Relationship keys returned | -|---|---| -| none | performsRoles 8, assignmentScope 15, requestedActions 11, contactDetails 1, userIdentities 1, contributionRecord 1 | -| `["ContactThrough"]` | contactDetails 1 | -| `["AssignmentScope"]` | assignmentScope 15 | -| `["PersonRoleAppointment"]` | none | +*FIELD-DROP: the method passes it, Egeria's class lacks it.* +- `get_tech_types_for_open_metadata_type(tech_name=...)` and `get_endpoints_for_asset(name=...)` send `filter` in a `FilterRequestBody`. The endpoints declare `ResultsRequestBody`, which has no `filter`, so they return everything unfiltered. +- `find_software_capabilities(property_names=...)` sends `propertyNames` where Java declares `SearchStringRequestBody`, so the property restriction is lost. -A spy on the request confirmed pyegeria sends the body exactly as given, so -the filtering happens in Egeria. +**New APIs (2026-10-23, from the refreshed `.http`; ISSUE-155 items 5 and 6), live on 9443.** One peer-cleared run with throwaway `pyeg-test::newapis::` elements; all GUIDs read back as gone and the prefix find was empty. Raw dumps were analysed afterwards: +- **`add/update/clear_usage_activity`:** add stored `explicitGetCount=1, findCount=2`; update set `findCount=9`; clear removed the `elementHeader.usageActivity` slot. **Egeria behaviour:** that update also reset `explicitGetCount` to 0 despite `mergeUpdate: true`. The count fields are primitive `long`s (Java `UsageActivityProperties`), so an omitted count deserialises as 0. The docstrings say to send every count. +- **`add/update/clear_member_popularity_profile`** (on a glossary): the total and the per-member map were stored; update set `findCountTotal=4` and kept the map; clear removed it. +- **`undo_metadata_element_update`:** a term description `v2` went back to `v1`. **`undo_related_elements_update`:** a RelatedTerm description `r2` went back to `r1`. +- **`restore_related_elements`:** a removed RelatedTerm came back with its properties. +- **`restore_metadata_element`:** a soft-deleted term came back. **An archived (Memento) term also came back, with the same GUID, the Memento removed, and its qualifiedName restored** from `…t3_archivedOn_Fri Oct 23 19:29:39 GMT 2026` to the original. This answers ISSUE-159's open question: restore is not soft-delete-only. Anchor-scoped restore (anchored descendants) was not tested. A plain get of the archived term (no `forLineage`) returned HTTP 500, not 404. +- **`link/detach_subject_area_hierarchy`** (repointed, ISSUE-165): the child was linked, then gone after detach. +- **`ProductCatalog.find_digital_product_catalogs`:** a named search returned 2 results in 6.2s (`ProductManager`'s product-manager-marker version returned 13 in 41s). An unfiltered "*" search timed out at the helper's 90s. -**Likely cause (not confirmed):** Egeria's relationship filter -(`OpenMetadataHandlerBase`, around line 478) keeps a relationship when -`propertyHelper.isTypeOf(rel, name)`, which should match. But `performsRoles` is -filled in a separate step (`OpenMetadataHandlerBase`, around line 1282), so the -role appointments may be dropped or never fetched once a filter is set. +**Not fixed (owner's call or Egeria's):** +- **19 endpoints Egeria doesn't serve**, documented in full with provenance in **ISSUE-165**. Confirmed absent from the live platform's OpenAPI doc (`omvs_audit.py --live`; GET `/v3/api-docs` on 9443) and from Egeria `main` 4486d1a7f1: the 8 Data Discovery analysis-report methods, data-classification create/update/link/detach, subject-area member add/remove, subject-area hierarchy link/detach (the SDK uses `collections/{g}/collection-hierarchies`; Egeria has `subject-areas/{g}/subject-area-hierarchies`), `find_governance_action_processes`, `get_gov_def_in_context` and `get_platform_organization`. Raise with the Egeria team, or remove from the SDK. Three other JAVA-MISSING entries are served live, so they were Java-scan false positives. +- **No server-side filter exists:** `get_tech_types_for_open_metadata_type(tech_name=)` and `get_endpoints_for_asset(name=)`, whose endpoints take `ResultsRequestBody` (no `filter`), and `find_software_capabilities(property_names=)` (`SearchStringRequestBody`). Options: filter client-side, or drop and document the argument. +- **Dr.Egeria Asset Maker template `typeName`:** Egeria's nearest field is `TemplateOptions.metadataElementSubtypeName` ("create an element of a subtype to the template type"). That isn't clearly the same as the spec's "type name of the element to create", so it needs a live check before mapping. Part of ISSUE-155 item 4 (Metadata Expert create-from-template) is covered by the `TemplateRequestBody` fields added above. +- **Harmless top-level names Egeria ignores:** `FindProperties`, `AnyTimeRequestBody` (Java: `GetRequestBody`), `IntegrationServiceRequestBody` (legacy admin-services config). +- `cascadeDelete` stays as it is: ISSUE-132, on hold for Egeria. +- **New `.http` coverage, not part of this sweep:** the refreshed collections document usage-activity, member-popularity-profile and metadata undo/restore, which pyegeria lacks (ISSUE-155 style). +- **Dead legacy path:** `pyegeria/view/md_processing_utils.py:1539` (behind the `dr_egeria_jupyter` CLI) still sends the sunset `initialStatus`, but it calls `create_controlled_glossary_term`, which no longer exists. -**Workaround:** read the Person without `includeOnlyRelationships` and take -`performsRoles` from the result. The Resource Browser already does this. +*MODEL-FIELD gaps (review; 27 model→Java pairs, with users, in the report).* +- Most are benign. For example, `ResultsRequestBody → GetRequestBody` drops paging on 51 `get_*_by_guid` calls, where paging means nothing. +- Status-filter models carry both `activityStatus` and `activityStatusList`; only the `…List` form exists in Java. No current caller uses the singular form in a filter (My Profile's uses are inside the nested `…Properties`, where it is a real field). +- `UpdatePropertiesRequestBody.replaceProperties` (Java `UpdateOptions` has only `mergeUpdate`) is used by three Metadata Expert updates. --- -### ISSUE-130: The YAML secrets store's user directory caches a user forever, so editing an existing user needs a platform restart -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. The YAML secrets store rebuilds its user cache each time it re-reads the file (`refreshTimeInterval`), so editing an existing user no longer needs a platform restart. **pyegeria:** no code change. +### ISSUE-163: Actor Manager role-appointment links send non-existent `*AppointmentProperties` classes, and five delete/detach methods send the wrong request-body class -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **Not re-testable here.** Needs an edit to a user in the platform's YAML secrets file and a wait for `refreshTimeInterval`. egeria-workspaces-fs is putting that to the owner as a separate step. +**Fixed 2026-10-23 in #483, released in 6.2.10.** **Live on quickstart 9443** (two runs, throwaway `pyeg-test::role-appt::` elements, all read back as gone by GUID afterwards, and a find on the prefix returned nothing). The old person-link class was refused with a 400. All three fixed links, called with no body, read back the relationship to the profile. The old IT detach body was refused with a 400; the fixed detach made the link unreadable. The fixed `delete_actor_role` deleted its role. -**Re-test 2026-10-19 by egeria-workspaces-fs (reported to this session, not independently re-run here):** **PASS.** A throwaway user `pyegrt130` was added to the live `coco-user-directory.omsecrets` and logged in (200, 13:19:48Z), which cached it. It was then set to `userAccountStatus: DISABLED` in place with no restart (13:19:56Z). Logins worked until 13:20:32Z and were refused with 401 at 13:21:02Z, so the cache was rebuilt about a minute after the edit (`refreshTimeInterval: 10`). The file was restored byte-for-byte afterwards. Ready to move to the Appendix. +**Layer:** Pyegeria (+ Dr.Egeria) · **Status:** fixed in #483, released in 6.2.10 · **Found:** 2026-10-23, reported by the owner ("pyegeria is using the wrong class name when linking a role to a profile"). -**Layer:** Egeria server (`YAMLSecretsStoreConnector`) · **Status:** open, reported to the Egeria lead · **Found:** 2026-10-09, while adding Resource Explorer's own account (`resourceexplorernpa`). +**Root cause:** each method below either passes a properties class that doesn't exist to `_async_new_relationship_request`, or calls the shared delete helper for the wrong body class. Neither is caught client-side. A wrong `prop` list only matters when the caller passes no body: the helper then builds `{"class": "NewRelationshipRequestBody", "properties": {"class": prop[0]}}` itself. My Profile's "add role" calls `link_person_role_to_profile` that way. The ground truth is `ActorManagerResource.java` / `GlossaryManagerResource.java` (`@RequestBody` types) and `open-metadata-framework .../properties/actors/*RoleAppointmentProperties.java`. **The `.http` file is wrong for `actor-roles/{guid}/delete`:** it says `DeleteRelationshipRequestBody`, but Egeria declares `DeleteElementRequestBody`. -**What:** `getUser(userId)` returns the entry from `userAccountMap` if it is already there. `refreshSecrets()` re-reads the YAML file every `refreshTimeInterval` minutes, but `userAccountMap` (and `securityAccessControlMap`) are created only on the first load (when `secretsStoreFile == null`) and are never cleared afterwards. Re-reading the file only helps users the cache has never seen; the comment says "this may be a new user". +| Method | Sent | Egeria expects | Live before fix | +|---|---|---|---| +| `link_person_role_to_profile` | `PersonAppointmentProperties` | `PersonRoleAppointmentProperties` | 400 | +| `link_team_role_to_profile` | `TeamAppointmentProperties` | `TeamRoleAppointmentProperties` | (same shape) | +| `link_it_profile_role_to_it_profile` | `ITProfileAppointmentProperties` | `ITProfileRoleAppointmentProperties` | (same shape) | +| `detach_it_profile_role_from_it_profile` | `DeleteElementRequestBody` | `DeleteRelationshipRequestBody` | 400 | +| `detach_identity_from_profile` (`user-identities/{g}/profile-identity/{g}/detach`) | `DeleteElementRequestBody` | `DeleteRelationshipRequestBody` | not run | +| `delete_actor_role` | `DeleteRelationshipRequestBody` | `DeleteElementRequestBody` | accepted (tolerated) | +| `delete_contribution_record` | `DeleteRelationshipRequestBody` | `DeleteElementRequestBody` | not run | +| `GlossaryManager.remove_is_context_definition` | `DeleteRelationshipRequestBody` | `DeleteClassificationRequestBody` | not run | -**Seen:** the account was first added without its `secrets` block, and the first token request cached it. After the password was added (and matched the servers file), `/api/token` still answered 401 for more than 10 minutes. It worked straight after a platform restart. +**IT detach request and response (before the fix), Egeria quickstart 9443, 2026-10-23:** +``` +POST .../actor-manager/actor-roles/a6e578d4-…/it-profile-role-appointments/91cc35e6-…/detach +{"forLineage":false,"forDuplicateProcessing":false,"class":"DeleteElementRequestBody","cascadeDelete":false} +→ HTTP 400 (pyegeria reports it as CLIENT_ERROR_400, "The client is unable to connect to the Egeria platform", which is misleading for a 400) +``` -**Effect:** any change to an existing user (password, status, named-list membership) needs a platform restart. +**Dr.Egeria:** `ActorManagerLinkProcessor` (`md_processing/v2/actor_manager.py`) called `_async_link_it_profile_role_to_profile` for `Link IT Profile Role Appointment`. No such method exists (the real one is `_async_link_it_profile_role_to_it_profile`), so that command raised `AttributeError` on every Link. -**Fix (Egeria):** clear or rebuild `userAccountMap` and `securityAccessControlMap` in `refreshSecrets()`. +**Fix:** corrected the three `prop` lists; switched the two detaches to `_async_delete_relationship_request`, `delete_actor_role` and `delete_contribution_record` to `_async_delete_element_request`, and `remove_is_context_definition` to `_async_delete_classification_request`; aligned the type hints and docstrings; corrected the Dr.Egeria method name. Tests: `tests/micro-tests/test_role_appointment_class_names.py` (11 tests; 9 of them fail on the old code; the two that pass cover the person and team detaches, which were already correct). + +**Checked and left alone:** `MetadataExpert.delete_related_elements` sends `DeleteRelationshipRequestBody`, while Egeria declares `OpenMetadataDeleteRequestBody`. That's deliberate (ISSUE-63): the Egeria class can't carry `deleteMethod`. Decertify and unlicense (`GovernanceOfficerResource`: `DeleteRelationshipRequestBody`) are already correct. **Why the audit missed them:** `scripts/omvs_audit.py --service actor-manager` on the old code reported 59 OK and 1 unrelated mismatch. Three blind spots, all fixed in ISSUE-164: +- Both detaches matched only by verb+path (RENAMED), and RENAMED matches skipped the body check. +- Nothing compared the inner `properties` class, which is where the three link bugs were. +- For `delete_actor_role`, `delete_contribution_record` and `is-context-definition`, the `.http` file named the same wrong class the SDK sent. + +The improved audit flags all eight on the pre-fix code. --- -### ISSUE-128: Retention cannot be set on Egeria 6.2: the server registers the Jackson subtype as `RetentionProperties.class` (typo), and pyegeria only allows `RetentionClassificationProperties` -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria's `oak2026` branch, pending its merge to `main`** (each fix has an FVT or unit test). Re-test once `oak2026` is merged and a platform is built from `main`. Fixed on `main` 13 Oct (`RetentionProperties`); `oak2026` also accepts `RetentionClassificationProperties` again. Responses now carry `RetentionProperties` in `class`. **pyegeria (proposed, not done):** per the owner's 2026-10-09 decision, switch the whitelist, docstrings and Dr.Egeria pins to `RetentionProperties`: `pyegeria/omvs/classification_explorer.py:8851,8878,8919`, `md_processing/v2/processors.py:220-237` (its comment says the opposite and must be rewritten), `md_processing/v2/curation.py:136`. Accept both names in the whitelist so older platforms keep working. Not urgent: the old name is accepted again. +### ISSUE-162: Dr.Egeria "Update Glossary Term" resets isAbstractConcept / isActivityDescription / isContext / isDataValue to False whenever the document doesn't mention them -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** On a throwaway collection, `classification-explorer/elements/{guid}/retention` accepted both `"class": "RetentionClassificationProperties"` and `"RetentionProperties"`; each read back `retentionBasis` 2 and its notes under `elementHeader.retention`. pyegeria's own `set_retention_classification` (old name) also succeeded. ISSUE-154 item 3's switch to the new name stays optional. +**Layer:** Pyegeria (Dr.Egeria) · **Status:** fixed in #482, released in 6.2.10 · **Found:** 2026-10-23 by RE's question-term update batch builder, reading `md_processing/v2` (RE term batch fix round 3). Reported via the Thursday Coordinator; confirmed here in source on `main` (6.2.9), not run live. -**Status:** open, found 2026-10-09 by Resource Explorer's first Retention publish on Egeria 6.2. **Layer:** Egeria server (a type registration typo). **Decision (owner, 2026-10-09):** wait for the Egeria fix; no pyegeria workaround. Retention publishing stays blocked until then. +**What:** `TermProcessor.apply_changes` (`md_processing/v2/glossary.py:145-148`) builds the term properties with `attributes.get('Is Abstract Concept', {}).get('value', False)`, and likewise for `Is Activity Description`, `Is Context` and `Is Data Value`. On update, the body goes through `filter_update_properties` (`md_processing/v2/processors.py:1957`). For a merge update that drops only `None` values, so the `False` defaults are always sent. Every "Update Glossary Term", even one that changes only the description, therefore writes `False` to all four flags and clears any that were `True`. -**What (all proven live on 6.2, with raw HTTP POSTs to `classification-explorer/elements/{guid}/retention` on a throwaway asset):** -- `"class": "RetentionClassificationProperties"` is refused with `InvalidTypeIdException`. This is what pyegeria documents and whitelists, and it worked on the pre-6.2 server (processors.py comment, 2026-08-03). -- `"class": "RetentionProperties"` (the `.http` ground truth) is also refused with `InvalidTypeIdException`. -- `"class": "RetentionProperties.class"` gets **HTTP 200**, and the classification reads back with `retentionBasis` 2 and the notes. -- The server's known-type list in the error has 96 ids. The only Retention entry is literally `RetentionProperties.class`. That looks like an `@JsonSubTypes.Type(name = "RetentionProperties.class")` typo in Egeria's `ClassificationProperties`. -- pyegeria's `validate_new_classification_request` whitelist (`prop=["RetentionClassificationProperties"]`) refuses anything else client-side, so no Retention can be set through pyegeria on 6.2. +**Same pattern, likely affected (not confirmed):** `md_processing/v2/data_designer.py:54-55` (`DataValueSpecificationProcessor`) and `:547-549` (`DataClassProcessor`) default `allowsDuplicateValues` / `isNullable` to `True`. Both processors appear to build one property body for create and update, so an unrelated update would reset those to `True`. Other `.get('value', )` uses checked in `md_processing` are request options (`forLineage`, `startsWith`, ...) or create-only values, not stored properties. -(An earlier version of this entry said 6.2 knows `RetentionProperties`. That was a misread of the error text, corrected the same day.) +**Impact:** a term with any of the four flags set loses it on an unrelated update. RE's 96-term question update batch will check the terms before applying. -**Where seen:** Resource Explorer's Curate classifications step on two repository assets. Both were refused, so nothing was written. Confidentiality, Criticality, Confidence and Impact property classes are registered normally. +**Fix (2026-10-23):** +- **Glossary terms:** the four flags now default to `None`, so a merge update omits them unless the document sets them. Create still sets them to `False` as before. +- **Data Designer: confirmed and wider than first logged.** Both `DataValueSpecificationProcessor` and `DataClassProcessor` updates send their property body straight to `_async_update_element_body_request`, where `body_slimmer` drops only `None`. So an unrelated update also reset `matchPropertyNames`, `sampleValues` and `dataPatterns` to `[]`, `matchThreshold` to `0`, and `specificationDetails` and `additionalProperties` to `{}`, besides `allowsDuplicateValues` / `isNullable` to `True`. All of these now default to `None`, and the old defaults are applied only on Create (`_with_create_defaults`). Egeria's `mergeUpdate` defaults to `true` (`UpdateOptions.java:23`), so omitted properties keep their values. +- Tests: `tests/micro-tests/test_issue_162_update_does_not_reset_properties.py`, 7 cases. The 4 update cases fail on the old code; the 3 create cases guard the unchanged create defaults. The full micro-test suite (1,436) passes in a clean python:3.13-slim container. Not live-tested. -**Candidate fix:** -1. **Egeria:** correct the subtype name to `RetentionProperties`, matching the `.http` file. Report it to Mandy. -2. **pyegeria, after Egeria's fix:** change the whitelist and docstring to `RetentionProperties`, the `.http` name. No workaround for the typo, by the owner's decision. -3. Re-check `md_processing/v2/processors.py` and `v2/curation.py`, which both pin `RetentionClassificationProperties`. +**Originally proposed:** +- Default these attributes to `None` (`.get('value')`), so a merge update omits them unless the document sets them. Keep `False` / `True` only on the create path, or leave creation to Egeria's own defaults. +- Check the Data Designer cases the same way. +- Add a micro-test: an update with no flag attributes sends none of the four keys. --- -### ISSUE-108: newly created relationships stay invisible to related-element queries for up to ~20 minutes after creation - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: needs information from us.** Egeria cannot reproduce it, and the new FVT does not show it. Their best lead is still timing: a related-element query hides a relationship whose `effectiveFrom` is later than the server's current time. For one link still missing about 12 minutes after a bulk load, they ask for: -- [ ] The relationship's GUID, `effectiveFrom` and `createTime`, and the server clock at that moment. -- [ ] Whether the link is returned when fetched directly by its GUID while it is missing from the related-element query. -- [ ] Whether the loader (Dr.Egeria or pyegeria) sets `effectiveFrom` or `effectiveTime` on that run. -- [ ] A timed run with `MemberDataField` links at 200 or more, polling every minute, to show whether the delay grows with load size or stays fixed. -**pyegeria:** Dr.Egeria's body builders send `effectiveFrom` only when the author supplies `Effective From` (`common_md_utils.py:342` and siblings), so a default load sets none; still confirm `effectiveTime` is not stamped from the client clock on link requests (a client clock ahead of the server would explain the delay). The timed run needs Egeria writes, so it needs the owner's go-ahead. +### ISSUE-161: Dr.Egeria's two entry points resolve the Egeria user differently, and the CLI fixes its credential defaults at import time -**Layer:** Egeria Server · **Status:** open · **Found:** 2026-09-20, reported by -the user from a Dr.Egeria bulk load session (1,027 expected field-link -relationships). +**Layer:** Pyegeria · **Status:** fixed in #477, released in 6.2.9 · **Found:** 2026-10-21, while checking an RE report that `dr_egeria --process` ignored `EGERIA_USER`. **That report is withdrawn; RE was the cause:** RE's own `config.py:316` defaulted its daemon identity (`EGERIA_USER_ID`) to "erinoverview", so RE handed the child process erinoverview. The CLI correctly used it. RE is removing that default. -**What:** a catalog walk run immediately after a load found only 876 of 1,027 -expected field links. A second walk 12 minutes later still showed 82 -structures short. A third walk, 20 minutes after the load, showed every link -present. No errors were raised at any point in the load itself — the -relationships were created successfully, they simply did not show up in -related-element queries for a long window afterward, then all became visible -at once with no further action taken. +**What was checked:** with `EGERIA_USER` set, the CLI path resolves and uses it throughout on 6.2.8: +- `config.py:423` (`os.getenv("EGERIA_USER", config.json / .env value)`) feeds `settings.User_Profile.user_name`, the `--userid` default; +- `EgeriaTech` / `BasePlatformClient` keep that `user_id`; +- `create_egeria_bearer_token()` with no arguments uses `self.user_id`; +- `_get_subclient` passes the user and token to every sub-client, and there is no token cache. +Intended precedence (CLAUDE.md): explicit `--userid` > OS env > `.env` > `config.json` > default. -**Not yet root-caused.** This looks like the same family of symptom as the -2026-08-15 Postgres-checkpoint/background-connector-load entry above (server -falling behind under load, catching up later) rather than a paging defect -like ISSUE-54 (which would not self-heal purely by waiting — ISSUE-54's -bug is about tie-ordering across separately-executed pages, not about data -not existing yet). But this has not been directly correlated with connector -log activity or checkpoint timing the way that entry was — this is a fresh -report, not yet independently reproduced or diagnosed by a pyegeria session. +**Real issues (pyegeria):** +1. **Inconsistent resolution between entry points:** `md_processing/dr_egeria.py:93` (module constants) and `:923` (its own `__main__` path) read `os.environ.get("EGERIA_USER", "erinoverview")` / `EGERIA_USER_PASSWORD` directly, bypassing `.env` and `config.json`. `md_processing/md_processing_utils/common_md_utils.py:29-30` does the same. So `python -m md_processing.dr_egeria` and the `dr_egeria` CLI disagree whenever the user is set only in `.env` or `config.json`. +2. **Import-time defaults:** `commands/cat/dr_egeria.py:32-33` computes the `--userid` / `--user_pass` defaults when the module is imported. Code that imports it, then sets `EGERIA_USER` and invokes the command in the same process, gets the earlier user. This doesn't affect normal CLI use, where each run is a new process. +3. **Help text:** "Overrides EGERIA_USER (env var or .env file)" doesn't mention `config.json`, or that the default is fixed at import. -**Impact:** any workflow that verifies a bulk write immediately after -issuing it (a post-load catalog walk, an integration test, an automated -Dr.Egeria "did it work" check) can get a false-negative "missing" result -that resolves itself given enough time — worth knowing before treating an -immediate post-write verification gap as evidence of a real data-loss bug. +**Fix (2026-10-21):** a new `pyegeria.core.config.resolve_egeria_credentials(user_id=None, user_pwd=None)` resolves, when called: an explicit value > the `EGERIA_USER` / `EGERIA_USER_PASSWORD` environment variables > the settings User Profile (`config.json` / `.env`) > "erinoverview" / "secret". +- The `dr_egeria` and `dr_egeria_folder` CLIs now default `--userid` / `--user_pass` to `None` and call it when they run. Their help text says so. +- `md_processing/dr_egeria.py` (module constants and its `__main__` path) and `common_md_utils.py` constants now use it too. The module-level `EGERIA_USER` names are kept for importers. +- Tests: `tests/micro-tests/test_issue_161_dr_egeria_credentials.py`. Five cases: precedence; env set after import, for both CLIs; config-only user; explicit flag wins. The three behaviour cases fail with the old caller code. The full micro-test suite (1,425) passes in a clean python:3.13-slim container. +- **Not changed:** other `commands/cat` tools (`list_glossaries`, `get_asset_graph`, `run_report`, `get_collection_tree`, `get_project_dependencies`, `list_user_ids`, `view_schema_tree`) still read `EGERIA_USER_PASSWORD` straight from the environment. They could adopt the helper later. -**Candidate next step (not yet done):** reproduce with a smaller, timed -sample (e.g. create N relationships, poll related-element queries at 1-minute -intervals, log when each becomes visible) to get a real latency -distribution instead of the three anecdotal checkpoints here, and check -whether it correlates with the same background-connector/checkpoint load -already documented above. Worth raising with the Egeria team directly, per -the original report. +**Originally proposed:** +- Resolve credentials through `settings.User_Profile` in all three places. +- Look them up when the command runs (click `default=None`, then resolve inside the command). +- Add a micro-test that both entry points pick up a user set only in `config.json` or `.env`. -**Update 2026-09-20 (peer check, no upstream issue filed yet):** checked -`odpi/egeria`'s issue tracker directly — nothing filed for this symptom, so -no response from the Egeria team via GitHub. Asked three peer sessions -working against the same dev Egeria server: -- **egeria-workspaces-fs-4a:** hasn't seen this specific symptom, no word - from the Egeria team either. Raised a speculative, explicitly-unconfirmed - lead worth checking: a chronic, still-open issue on `egeria-shared-kafka` - where the `GroupCoordinator` repeatedly times out writing to - `__consumer_offsets` (5000ms) and triggers a consumer-group rebalance - roughly every 90s (confirmed via broker logs 2026-09-03; root cause still - open, CPU/GC/Postgres-restarts/virtiofs I/O already ruled out; historically - "Information" severity, self-heals within seconds, not chased further - since it wasn't visibly causing downstream harm). If `qs-view-server`'s - related-element queries are backed by Kafka-driven indexing/event - propagation rather than a direct repository read, a chronic rebalance - storm could plausibly manifest as exactly this "relationship exists but - isn't visible to queries yet" lag. **Explicitly not a confirmed - correlation** — the timescales don't obviously line up (a recurring ~90s - self-healing blip vs. a one-shot ~20-minute resolution) — and the peer - offered to check Kafka broker logs around the original bulk-load window - for the same rebalance pattern if useful. -- **Resource Explorer experimental UI:** no direct evidence either way — - today's two live writes against the same server (a Governance Action - Process Step upsert, and a native-survey `SurveyReport` lookup) both - checked either *immediately* after the write or only *after* polling to a - terminal status, never in the 1-20 minute gap this entry describes, so - neither confirms nor rules it out. Flagged that a background agent on - their side doing a related connection-refresh fix will trigger fresh - engine actions/relationships shortly and will report back if it surfaces - the same lag. -- **Egeria-trellis backlog review:** also nothing from the Egeria team (no - channel to them; checked RE docs/Backlog/design notes for this exact - shape first — a checked "no", not a guess). Offered a different, measured - precedent on this same platform as a *pattern* to distinguish causes, not - a diagnosis: `packages/resource-explorer/docs/investigation-classification-and-zoning-design.md` - (~line 830) documents a privately-zoned element still readable by a - non-owner *six minutes* after the control was written (denial began at - seven), caused by the security connector's `refreshTimeInterval` (10 min) - reload — "read it back immediately, so it must be fine" was exactly the - misleading evidence there too. Their key diagnostic point: a - **fixed-wall-clock cause** (everything becomes visible at once, at a time - unrelated to load size — test with 50 links instead of 1,027 and see if - the delay is still ~20 min) and a **volume-proportional cause** (async - indexing/event-driven materialisation catching up, arriving in a stream) - look identical from a single observation but have different fixes; the - 876→945(implied)→1,027 partial-recovery-at-12-minutes data point *mildly* - favours volume-proportional, but "two points don't make a curve." Also - worth checking directly: fetch one of the missing links by its own GUID - during the window — if that resolves but the relationship is still absent - from the related-element query, the write landed fine and only the - traversal/index path lags, narrowing which service is actually behind. If - this gets filed upstream, the volume-vs-fixed measurement is what turns - it into a real bug report instead of getting bounced as "eventual - consistency, working as intended." +--- -Two independent, unconfirmed leads now on record (Kafka rebalance storms; -a refresh-interval-style cache) alongside a concrete diagnostic method -(fixed-wall-clock vs. volume-proportional, plus the direct-GUID-fetch -isolation check) for telling them apart before assuming either. Next real -step is still reproducing with a smaller, timed, instrumented load using -that method — not yet done. -**Reproduction attempt, 2026-09-20 (small end of the curve, not reproduced):** -after clearing with all 5 live peer sessions first (per `coordinate-shared-writes`), -ran a timed, instrumented small-scale reproduction against `qs-view-server`: -20 throwaway `Collection` elements created and each linked to a parent via -`CollectionMembership` (not the original `MemberDataField`/"field link" type -— a different relationship type, noted as a limitation below), polling -`get_collection_members` every 30s for up to 20 minutes, with a direct-GUID -isolation check ready for anything still missing at the deadline. +### ISSUE-160: `AutomatedCuration`'s template builders miss placeholders Egeria's templates now use, so created elements keep literal `~{name}~` values (folder endpoint `~{directoryAddress}~` breaks folder surveys) -**Result: all 20 relationships were visible on the very first poll, ~1.1s -after the last one was created — no lag at all.** Cleanup (delete all 21 -fixture elements) verified two independent ways: a direct-GUID fetch on -each (`404`/`OMAG-REPOSITORY-HANDLER-404-007`, confirmed real soft-delete, -not an ISSUE-63-style silent no-op) and a broad `find_collections` search -for the `test-fixture-issue108` prefix (0 results). The script itself was -run from an ephemeral per-session scratch directory, not committed anywhere -— reproduce via the same approach (create N Collections linked to a parent -via `_async_add_to_collection`, poll `_async_get_collection_members`) if -needed later; the approach, not the script file, is what's worth keeping. +**Layer:** Pyegeria · **Status:** fixed in #474 and the follow-up #475 (released in 6.2.8), with the Dr.Egeria Postgres follow-up in #478 (released in 6.2.9). The items originally listed as not fixed are kept below for the record. **Found:** 2026-10-21, live by the RE coordinator against Egeria `main` 05a5e982 with pyegeria 6.2.7. Two FileFolders made with `create_folder_element_from_template` had Endpoint `networkAddress` = `~{directoryAddress}~`. Egeria's Folder Survey then failed with `BASIC-FILE-CONNECTOR-404-001` "The folder named ~{directoryAddress}~ ... does not exist" (engine action `c1d924b5`). The same folders also kept `versionIdentifier` = `~{versionIdentifier}~` and `deployedImplementationType` = `~{deployedImplementationType}~` (record `~/.re-gate/repair-folder-endpoints-20261021T143415Z.json`). -**Per Egeria-trellis backlog review's own framework, this is one point on -the curve, not a refutation.** It's consistent with either "no bug under -current load" or "the lag is genuinely volume-proportional and 20 elements -is nowhere near the ~1,027-link scale that surfaced it originally" — the -one data point this run *can't* distinguish those two. Also a real -methodology gap versus the original report: this used `CollectionMembership` -on plain `Collection` elements, not the `MemberDataField` "field link" -relationship the original 1,027-link load actually exercised (ISSUE-102 is -in the same `MemberDataField` neighborhood, worth checking together). Server -load conditions also weren't recorded this run (no `ListAgents`/`:8810` -snapshot taken alongside the timings, despite the earlier plan to do so — -an execution gap, not a design one). +**Cause (Egeria `origin/main` d8f72cbe25, read-only):** no placeholder was renamed (`git log -S` on `PlaceholderProperty.java`). Egeria **added** placeholders that pyegeria never sends: +- **`directoryAddress` / `fileAddress`:** added by `156650efd6` (2026-07-19, "file system control for file cataloging"). They are now the Endpoint address in the folder and file templates (`DataAssetTemplateDefinition` `FILE_FOLDER_TEMPLATE`, `DATA_FOLDER_TEMPLATE`, `CSV_FILE_TEMPLATE`, ...). The templates still use `directoryPathName` / `filePathName` in the qualifiedName, so both are needed. +- **`deployedImplementationType`:** in every folder/file placeholder list (`getFolderPlaceholderPropertyTypes`, `getDataFilesPlaceholderPropertyTypes`). +- **`databaseDescription`:** the PostgreSQL Relational Database template's description placeholder (`PostgresPlaceholderProperty.DATABASE_DESCRIPTION`); pyegeria sent `description`. -**Candidate next step, still open:** a same-relationship-type -(`MemberDataField`), same-order-of-magnitude (~200+) run, following the -same peer-clearance and independent-verification discipline, is what would -actually test the volume-proportional hypothesis — this run didn't attempt -that scale on its own initiative, since committing to it wasn't part of -what peers were asked to clear. +An unsent placeholder is stored as the literal `~{name}~`. `body_slimmer` drops `None` values, so an optional argument left unset has the same effect. -**Egeria team response, relayed 2026-10-05 (source `oak2026` fb3d6fce53; **fixes are in two unmerged PRs, so no build contains them, and nothing was tested against a live platform**):** not validated; no cache found. Best lead: a client clock ahead of the server's, with `effectiveFrom` set from it (related-element queries silently hide a relationship whose `effectiveFrom` is later than the server's "now"). They need, for one link still missing at ~12 minutes: its `effectiveFrom` and `createTime`, the server clock then, and whether the loader sets `effectiveFrom`. **Our facts:** pyegeria sets no `effectiveFrom`/`effectiveTo`/`effectiveTime` by default (model defaults are `None`; no clock call is tied to any `effective*` field; Dr.Egeria forwards an "Effective From" only if typed). Host and container clocks agree within 1 s on 2026-10-05; the 2026-09-20 load's values are unavailable. +**Fixed on the branch** (`pyegeria/omvs/automated_curation.py`; signatures unchanged): +| Method | Added | +|---|---| +| `create_folder_element_from_template` | `directoryAddress` (= `path_name`), `deployedImplementationType: "File System Directory"` | +| `create_csv_data_file_element_from_template`, `get_create_csv_data_file_element_from_template` | `fileAddress` (= `file_path_name`), `deployedImplementationType: "CSV Data File"` | +| `create_secrets_store_element_from_template` | `fileAddress` (= `file_path_name`), `deployedImplementationType: "YAML File Secrets Collection"` | +| `create_postgres_database_element_from_template` | `databaseDescription` (= `description`; `description` still sent) | -**Update 2026-10-10:** the `oak2026` source (`fb3d6fce53`) the team examined is now in Egeria `main` (merged 2026-10-06, odpi/egeria#9363). No fix was made for this issue; the team's questions above still stand. +Tests: `tests/micro-tests/test_issue_160_template_placeholders.py` (5 cases; all fail on the old code, all pass on the fix). The full micro-test suite (1,409) passes in a clean python:3.13-slim container. ---- +**Live check 2026-10-21 14:52Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; the fixed code on this branch): `create_folder_element_from_template("/tmp/pyeg_rt_folder_20261021T145254Z", ..., version="1.0")` created FileFolder `e9a45c7f` with `deployedImplementationType` "File System Directory" and `versionIdentifier` "1.0". Its anchored Endpoint `8f003544` had **`networkAddress` = `/tmp/pyeg_rt_folder_20261021T145254Z`**. The folder, Connection `18ef2ee9` and Endpoint had no `~{...}~` left in their own properties. A metadata-expert delete (`deleteMethod: PURGE`, `cascadedDelete: true`) removed all three; each was verified gone by GUID. A first run (`a3475d3e`) confirmed the folder fields but its script missed the Endpoint. Exact `networkAddress` searches (forLineage) afterwards found no leftover for that run either. The survey itself was not run. +**Follow-up fix 2026-10-21 (owner: "fix them all in a new pr")** on branch `fix/issue-160-remaining-placeholders`, built on #474. All 13 hard-coded builders now wrap their values in `_placeholder_values`, which sends `""` for any placeholder left `None`, so no `~{name}~` literal is stored. `body_slimmer` drops only `None`. A folder created without a file system therefore gets qualified name `FileFolder:::/path`. +- **Postgres server and database:** new optional `secrets_store_path_name` / `secrets_collection_name`, sent as `secretsStorePathName` / `secretsCollectionName`, plus `version` (and `resource_name` on the server, defaulting to the server name). `db_user` / `db_pwd` are now optional, deprecated and no longer sent; passing either raises a `DeprecationWarning`. +- **Kafka server:** `resourceName` (defaults to the server name) and `versionIdentifier`. +- **Unity Catalog:** + - UC server: `resourceName`, defaulting to the server name; + - UC catalog: `ucServerQualifiedName`; + - UC table and function: `rootSchemaTypeQualifiedName`; + - UC volume: `directoryPathName` and `localStorageLocation`. + Each is a new optional parameter on the async method and its sync wrapper. +- **Item 4 needed no change:** "Keystore File" is a real template (`DataAssetTemplateDefinition.KEYSTORE_FILE_TEMPLATE`), and live, both "YAML File Secrets Collection" and "Keystore File" resolve to template `fbcfcc0c`. +Tests: `tests/micro-tests/test_issue_160_remaining_placeholders.py`, 11 cases, all failing on #474's code. The full micro-test suite (1,420) passes in a clean python:3.13-slim container. **Dr.Egeria follow-up, done 2026-10-21** (after 6.2.8; branch `claude/elegant-chatterjee-d280de`): the `Create Postgres Server Element` / `Create Postgres Database Element` commands no longer offer `Database User Id` / `Database Password`. Both attributes were deleted from the Asset Maker compact spec, since no other command used them. The commands now offer `Secrets Store Path Name`, `Secrets Collection Name` and `Version Identifier`, and the server also offers `Resource Name` (the canonical Governance Officer definition, copied). `AssetMakerProcessor` passes them as `secrets_store_path_name` / `secrets_collection_name` / `version` / `resource_name` and no longer passes `db_user` / `db_pwd`. Templates, help and report specs were regenerated, and the templates were copied to egeria-workspaces and trellis on local branches. Test: `tests/micro-tests/test_postgres_element_secrets.py`. Not run against a live server. +**Follow-up live check 2026-10-21 15:16Z** (Egeria `main` 05a5e982, erinoverview, after a peer round; #475 code): +- **(a)** `create_postgres_server_element_from_template("pyeg_rt_pg_…", "pyeg-rt.invalid", "5432", secrets_store_path_name=…, secrets_collection_name=…)`, with no `db_user` / `db_pwd`, created SoftwareServer `b6b2ebff`: `resourceName` = the server name, `versionIdentifier` and `description` = `""`, and no `~{...}~` in its own properties. Its Connection `55686913` and Endpoint `59014065` (`jdbc:postgresql://pyeg-rt.invalid:5432/postgres`) had no literals either. +- The server had **no CatalogTarget link** (its relationships were sourcedFromTemplate, serverEndpoints, capabilities, connections), so the JDBC integration connector never saw it. +- I did not find where Egeria stored the secrets-store values. They were not in the Connection's own properties, and no literal was left. Unconfirmed. +- **(b)** `create_folder_element_from_template("/tmp/pyeg_rt_folder2_…", …, None)`, with no file system, version or description: qualifiedName `FileFolder:::/tmp/pyeg_rt_folder2_…`, `resourceName` `:/tmp/…`, `versionIdentifier` / `description` = `""`, `deployedImplementationType` "File System Directory", and Endpoint `networkAddress` = the path. No literals on the folder, Connection or Endpoint. +- Both were purged (metadata-expert delete, `PURGE`, `cascadedDelete`), and all six GUIDs were verified gone. Name, host, stamp and secrets-file searches afterwards found nothing, which covers the server's capabilities too. -# Quick reference: which OMVS client class for which purpose +**Originally listed as not fixed** (kept for the record): +1. **Postgres server and database credentials:** both templates now take `secretsStorePathName` and `secretsCollectionName` (credentials come from a secrets store). pyegeria never sends them, so the connection keeps literals. pyegeria still sends `databaseUserId` / `databasePassword`, which the templates no longer list (`312c172639`, `56fa898e43`), so those are dead. Proposed: add optional `secrets_store_path_name` / `secrets_collection_name` parameters, and deprecate `db_user` / `db_pwd`. +2. **Optional arguments left unset:** `versionIdentifier`, `description`, `fileSystemName` and the like are dropped when `None`, so they stay `~{...}~` (RE's `versionIdentifier` case). Options: send `""` for unset optional placeholders, or document that callers must pass them. +3. **Unsent template placeholders in the Unity Catalog and Kafka builders:** + - UC catalog: `ucServerQualifiedName`; + - UC table and function: `rootSchemaTypeQualifiedName`; + - UC volume: `directoryPathName`, `localStorageLocation`; + - UC server and Kafka server: `resourceName` (Kafka also `versionIdentifier`). + Add parameters for them where a caller can know the value. +4. **The secrets-store builder's fallback** to a "Keystore File" technology type (`_async_get_template_guid_for_technology_type`) matches no template on `origin/main`. -| Need | Class | Notes | -|---|---|---| -| Business reference data (country/currency codes) | `ReferenceDataManager` | Does **not** cover specification properties (ISSUE-19, docs-only) | -| Valid metadata values for a property name | `ReferenceDataManager` or `MetadataExpert` | `get_valid_metadata_values` lives on shared `ServerClient` base; no `as_of_time` support — Egeria endpoint doesn't expose it (ISSUE-18) | -| Specification properties (placeholders, guards, action targets, etc.) | `SpecificationProperties` | `get_specification_property_by_type` now works with either PascalCase or `SCREAMING_SNAKE_CASE` input (ISSUE-17, fixed 2026-08-15); `find_specification_property` with `graph_query_depth=0` also available (ISSUE-15); `get_specification_property_by_guid` works too, `NameError` fixed (ISSUE-28, fixed 2026-08-05, re-verified 2026-08-15) | -| `DataGrain` / `DataClass` listing | `find_data_value_specifications` / `get_data_value_specifications_by_name("*")` | Both fixed (ISSUE-1, ISSUE-2) | -| `DataSpec` (Collection subtype) | `CollectionManager.find_collections(metadata_element_type="DataSpec")` | | -| `DataStructure` / `DataField` | `DataDesigner.find_data_structures` / `find_data_fields` | | -| Solution blueprints/components (any pyegeria version) | `SolutionArchitect.find_solution_blueprints/components(search_string="*")` | Avoid `find_all_*` variants on old versions (ISSUE-11) | -| Note logs (list) | `find_note_logs("*", graph_query_depth=0)` | ISSUE-15 | -| Note logs (entries) | `get_notes_for_note_log(guid, page_size=100)` | ISSUE-3 — never pass `metadata_element_type_name="NoteLog"` | -| Collection members | `get_collection_members(collection_guid)` | ISSUE-8 — now returns members of any type, not just the collection's own type | -| Comparing results across two runs/environments that don't match | — | Check whether the same user's credentials were used in both — governance zone visibility can legitimately change results per-user (ISSUE-29) before assuming a pyegeria bug | -| Multi-classification search (`matchClassifications`, 2+ conditions) | `MetadataExpert.find_metadata_elements` | Fixed in Egeria server (ISSUE-35) | -| Paging a `find_metadata_elements` result | Set `"startFrom"`/`"pageSize"` **in the body dict** | Fixed (ISSUE-34) — these are NOT separate parameters on this method anymore; passing them as kwargs is silently a no-op. Same for `"graphQueryDepth"`. | -| Relationships for a single element by guid | `MetadataExpert.get_all_related_elements(guid)` | **Not** `get_metadata_element_by_guid` — that call never returns relationships, by design (ISSUE-37, not a bug) | -| Project parent/child hierarchy (any linked project, not just hierarchy) | `ProjectManager.get_linked_projects(guid)` | Fixed (ISSUE-42) — was silently returning "No elements found" regardless of real data | -| Checking whether a `get_guid_for_name` lookup found anything | `guid_or_none(client.get_guid_for_name(name))` | ISSUE-114 — a miss returns a truthy sentinel string ("No elements found"), so `if not guid` reads it as found; the helper returns `None` instead | +Builders with no hard-coded keys (`create_elem_from_template`, Dr.Egeria's "Placeholder Property Values" attribute) are unaffected. The UC schema builder sends all its placeholders. --- -# Appendix: Closed / Not-a-bug entries -## Fixed / Resolved +### ISSUE-153: engine-action claim/status/completion calls still use Automated Curation paths that Egeria moved to the Governance Engine view service -Sorted newest-first by `ISSUE-#` (2026-10-12). ISSUE-91 has two entries -(the same defect reported twice); both are kept, next to each other. +**Found 2026-10-16 by the Resource Explorer operations session, after the quickstart was redeployed on Egeria main (85d6c48a, includes odpi/egeria#9371 "Add Governance Engine API").** + +- Egeria #9371 removed these from `AutomatedCurationResource` and added them to the new `GovernanceEngineResource` (`/servers/{server}/api/open-metadata/governance-engine/...`): `GET /governance-engines/{guid}/engine-actions/approved` (new), `POST /engine-actions/{guid}/claim`, `GET /governance-engines/{guid}/engine-actions/active-claimed`, `POST /engine-actions/{guid}/status/update`, `POST /engine-actions/action-targets/update`, `POST /engine-actions/{guid}/completion-status`. `GET /engine-actions/active` stays on Automated Curation (urlMarker form). #9371 also adds `POST /governance-action-process-instances/{guid}/cancel` and makes cancel skip finished engine actions. +- pyegeria v6.2.4 still builds `{ref_curation_command_base}/governance-engines/{guid}/engine-actions/active-claimed` (automated_curation.py:2614), and the same base for claim, status update and completion. Live result on the redeployed platform: the old active-claimed path returns 404. RE's engine host completion pass has failed every minute since about 15:44, and a claim would fail the same way. +- **Also blocking, and not pyegeria:** on the redeployed quickstart, the new paths return 404 too (`/governance-engine/governance-engines/{guid}/engine-actions/active-claimed` and `.../approved`). `qs-view-server` does not have the new Governance Engine view service configured yet, so that's a quickstart config change for egeria-workspaces. +- **Needed in pyegeria:** a `GovernanceEngine` client (or methods moved off `AutomatedCuration`) for approved, claim, active-claimed, status update, action-targets update and completion-status on the new base, plus `cancel_governance_action_process`. Keep `get_active_engine_actions` on Automated Curation. +- Status: FIXED on branch fix/issue-153-governance-engine (9d412207, signed, pushed to dwolfson/egeria-python; owner approved 2026-10-16). New GovernanceEngine client (6 calls incl. get_approved_engine_actions), AutomatedCuration.cancel_governance_action_process, five deprecated AutomatedCuration wrappers with a one-time 404 fallback to the old path. 20 micro-tests (whole file fails to collect on upstream/main). Audit: governance-engine OK=6, automated-curation OK=9. Merged in #460, released in 6.2.5; not verified live (qs-view-server lacks the service). --- + ### ISSUE-149: `AnalyticFunctionSpec` lacks the manifest fields trellis-analytics needs (`kind`, `source`, `version`, `envelope`, `annotation_type`, `resource_types`), so trellis subclasses carry them **Layer:** pyegeria (`pyegeria/view/analytic_registry.py`) · **Status:** fixed 2026-10-13 · **Found:** 2026-10-13, handed off by the Resource Explorer PR/CI session on the owner's instruction. Source: trellis brief §2a and `packages/trellis-analytics/src/trellis_analytics/manifest.py`. RE runs pyegeria 6.2.2. @@ -2486,7 +2586,7 @@ asks for a client parameter. ### ISSUE-136: `get_notes_for_note_log` filters by type `Action`, so a NoteLog's Notes don't come back; `get_note_by_guid` calls an endpoint that doesn't exist -**Layer:** pyegeria · **Status:** fixed 2026-10-12 on branch `docs/issue-136` (PR pending), not yet released · **Found:** 2026-10-11, reported by the Resource Explorer session (6.2.1, +**Layer:** pyegeria · **Status:** fixed 2026-10-12 in #446, released in 6.2.2 · **Found:** 2026-10-11, reported by the Resource Explorer session (6.2.1, from a live read and the source); `get_note_by_guid` found while checking the sibling methods. **Fixed 2026-10-12, and the diagnosis below was wrong in two places.** @@ -2537,7 +2637,7 @@ Dr.Egeria callers to match. Add a micro-test that checks the URL and the request ### ISSUE-135: 14 `*_by_name` wrappers required `name` even though they accept `body`, so a body-only call raised `TypeError` -**Layer:** pyegeria · **Status:** fixed in #445 (merged 2026-10-11), not yet released · **Found:** +**Layer:** pyegeria · **Status:** fixed in #445 (merged 2026-10-11), released in 6.2.2 · **Found:** 2026-10-10, reported by the Resource Explorer session (`ActorManager.get_actor_profiles_by_name` on 6.1.15; still present in 6.2.1). @@ -2579,7 +2679,7 @@ mishandled a body-only call: ### ISSUE-134: With `PYEGERIA_ENABLE_LOGGER_CATCH` on, `@dynamic_catch` turns every failure into a `None` return, indistinguishable from "nothing found" -**Layer:** pyegeria · **Status:** fixed 2026-10-12 on branch `docs/issue-136` (PR pending), not yet released · **Found:** 2026-10-10, reported by the Resource Explorer +**Layer:** pyegeria · **Status:** fixed 2026-10-12 in #446, released in 6.2.2 · **Found:** 2026-10-10, reported by the Resource Explorer session; confirmed from code. **Fixed 2026-10-12:** `dynamic_catch` now applies @@ -2654,7 +2754,7 @@ three dead `subComponents` fallbacks are removed. Micro-tests in `tests/micro-te ### ISSUE-129: `MetadataExpert.get_related_metadata_elements` (and two sibling relationship queries) accept paging, direction and lineage arguments but never send them -**Layer:** pyegeria · **Status:** fixed 2026-10-12 on branch `docs/issue-136` (PR pending), not yet released · **Found:** 2026-10-09, in a code review of Resource Explorer's +**Layer:** pyegeria · **Status:** fixed 2026-10-12 in #446, released in 6.2.2 · **Found:** 2026-10-09, in a code review of Resource Explorer's 6.2 composition read (`blueprint_materializer.sub_component_guids`). **Fixed 2026-10-12:** `_async_get_related_metadata_elements` sends @@ -3093,15 +3193,6 @@ would make `if not guid` correct instead of a trap. Not proposing this without the owner's approval, since existing callers across pyegeria and its consumers may already depend on the string return value. ---- - -## Awaiting re-test (fix landed, not yet verified live) - -Each of these has a fix in place — usually Egeria `main` (odpi/egeria#9363, -merged 2026-10-06), sometimes on the pyegeria side — but nobody has yet -re-run the original repro against a platform that contains it. The next -action for every entry here is the same: re-test, then move it to the -Appendix or back to an open section with what was found. **Resolved 2026-10-12 (convention kept, helper added):** the owner chose to keep the sentinel return, since changing it would break callers across @@ -3122,68 +3213,9 @@ Also added to the Quick reference table. Test: --- -### ISSUE-95: No catalog template registered for the "Apache Kafka Server" technology type — `Create Kafka Server Element` (Asset Maker) fails with a 400 on `qs-view-server` - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). The Apache Kafka Server catalog template is registered. **pyegeria:** no code change; re-run `Create Kafka Server Element`. - -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **PASS.** `get_template_guid_for_technology_type("Apache Kafka Server")` returns `5e1ff810-5418-43f7-b7c4-e6e062f9aff7`. Read-only. Ready to move to the Appendix. - -**Layer:** Egeria Server (deployment/archive content) · **Status:** awaiting re-test (addressed on the Egeria side, confirmed live 2026-10-05) · **Found:** 2026-09-11, live-verifying the new Asset Maker Dr.Egeria family (#354) - -`AutomatedCuration._async_get_template_guid_for_technology_type("Apache Kafka Server")` -returns `None` on `qs-view-server` — no catalog template is registered for -that technology type on this server — so `Create Kafka Server Element`'s -underlying `_async_create_kafka_server_element_from_template` posts a body -with `templateGUID: null` and gets back `VALIDATION_ERROR_1` ("Request body -failed validation"). Not a pyegeria code defect: every other Asset Maker -technology type tried resolved a real template GUID on the same server -(CSV Data File, PostgreSQL Server, File System Directory, YAML File Secrets -Collection, Unity Catalog Server) — this one is specific to the Kafka -technology type's archive content on this deployment. `Create Kafka Server -Element` itself is otherwise correct (parses/validates cleanly; the failure -is entirely server-side at execution). - -**Ask:** confirm whether the Apache Kafka Server catalog template is -expected to ship with quickstart's default archive content, and if so, why -it's missing here — or add it to whichever archive/content-pack is meant to -carry it. - - -**Addressed on the Egeria side (2026-10-05 check, carried over from the retired `docs/egeria-open-bugs-2026-10-05.md`):** the running platform resolves a template. `get_template_guid_for_technology_type("Apache Kafka Server")` returns `5e1ff810-5418-43f7-b7c4-e6e062f9aff7`, and Egeria source `449ad06894` ships `ApacheKafkaContentPack.omarchive`. It looks like a content-pack loading matter, not a code defect. **Remaining:** re-test `Create Kafka Server Element` live. ---- - -### ISSUE-89: No configurable bearer-token lifetime in `application.properties` — platform tokens are fixed at 3600 s, and the signing key is random per restart - -**Egeria team, 2026-10-18 (Mandy Chessell, note saved at `~/.re-gate/EGERIA-TEAM-NOTE-2026-10-18.md`):** **Status now: fixed on Egeria `main`; re-test against a platform built from current `main`** (the new view-services FVT exercises this request). Bearer-token lifetime is configurable in `application.properties`. **pyegeria:** no code change; re-test means setting the property and decoding a token's `exp - iat`. - -**Re-test 2026-10-19 (quickstart recreated 13:02Z on Egeria `main` 05a5e982, image 6.3-SNAPSHOT built 2026-10-18 16:13Z; as erinoverview on qs-view-server; throwaways named `pyeg_rt_*`, each deleted and verified gone by GUID; peers cleared first):** **Not re-testable yet.** Tokens still decode to `exp - iat = 3600`, as expected: Egeria `main` has `bearerTokenTimeout` (hours, default 1, `TokenService`), but the quickstart's `/deployments/application.properties` does not set it. Proving it needs that property set and an egeria-main restart. egeria-workspaces-fs is putting that to the owner as a separate step. - -**Re-test 2026-10-19, after the config change:** **PASS.** egeria-workspaces-fs set `bearerTokenTimeout=2` in the quickstart's `/deployments/application.properties` (the owner's choice) and restarted egeria-main. A fresh erinoverview token now decodes to `exp - iat = 7200`, re-checked from this session. The property is in hours. Ready to move to the Appendix. - -**Layer:** Egeria Server · **Status:** awaiting re-test (addressed in Egeria source 2026-09-19; not verified on the dev platform) · **Found:** 2026-09-04 (trellis-auth token contract) - -With `authentication.mode=token`, every token minted by the platform decodes to `exp - iat = 3600` -(observed on `omag-server-platform-6.2-SNAPSHOT`, quickstart, `authentication.source=platform`). -`application.properties` exposes `authentication.mode`, `authentication.source` and `rsa.key-id`, and -nothing that controls the lifetime; with `rsa.key-id` empty the signing key is regenerated at start, so -every token also dies on a platform restart. Client applications that hold a platform token on the -user's behalf (Portal, Egeria Advisor, Resource Explorer — which now carry the Egeria bearer token in -their session and never the password) cannot offer a session longer than an hour without either -re-prompting the user or caching a password, which is what a bearer token exists to avoid. - -**Ask:** a property such as `authentication.token.lifetime=3600` (seconds, default unchanged), read at -start and applied when a token is issued; optionally a refresh operation that returns a new token for a -valid unexpired one. Full draft issue text: trellis session scratch `egeria-issue-token-lifetime.md`. - -**Correction, 2026-10-05 (Egeria team; confirmed in `RSAGenerator`):** the statement above that an empty `rsa.key-id` is why the signing key is regenerated is wrong. `RSAGenerator.generateRSAKeyPair(keyId)` always makes a fresh key pair and only uses the id as a label, so **no setting makes tokens survive a restart**; persisting the key is planned for the next release. Also: the lifetime property now exists, `bearerTokenTimeout` in HOURS (default 1), added in `a1f228d4a9` (2026-09-19); there is still no refresh operation. - - -**Addressed in Egeria source (2026-10-05 check, carried over from the retired `docs/egeria-open-bugs-2026-10-05.md`):** commit `a1f228d4a9` (2026-09-19, "Add control of bearer token timeout") added `bearerTokenTimeout`, read by `TokenService`, in **hours** (default 1; the request asked for seconds). It is in `application.properties` and `container.application.properties`. **Not covered:** a token refresh operation. The signing key also changes on every restart: `rsa.key-id` does **not** prevent that (confirmed in `RSAGenerator.generateRSAKeyPair`: it always makes a fresh key pair and uses the id only as a label), and the Egeria team plans to persist the signing key in the next release. **Remaining:** verify on a platform built after 2026-09-19. ---- - ### ISSUE-113: `NewElementRequestBody` has no `initial_status` field, so `initialStatus` is silently dropped on every create routed through it -**Layer:** pyegeria · **Status:** fixed 2026-10-12 on branch `docs/issue-136` (PR pending), not yet released; reclassified: not a missing model field · **Found:** 2026-09-21, verifying +**Layer:** pyegeria · **Status:** fixed 2026-10-12 in #446, released in 6.2.2; reclassified: not a missing model field · **Found:** 2026-09-21, verifying Resource Explorer's multi-resource design doc's probe 4 (`egeria-support- for-multi-resource.md` §9, trellis repo): whether a client can create a governance element (DataClass, DataGrain, ValidValueSet, ...) directly in @@ -3317,7 +3349,7 @@ in the trellis repo. ### ISSUE-111: `ConnectionMaker.link_connection_connector_type`/`link_connection_endpoint`/`link_asset_to_connection` silently create no relationship when `body` defaults to `None` -**Layer:** Pyegeria · **Status:** fixed 2026-10-12 on branch `docs/issue-136` (PR pending), not yet released · **Found:** 2026-09-21, reported by a +**Layer:** Pyegeria · **Status:** fixed 2026-10-12 in #446, released in 6.2.2 · **Found:** 2026-09-21, reported by a downstream user (Resource Explorer) while building a client-side secrets store: three separate `link_*` calls "succeeded" (no exception, no error response) yet none of the relationships existed afterward, discovered only @@ -9710,9 +9742,8 @@ correctly coerces to a real `int`. Ran `tests/micro-tests/test_overview_metrics.py`, `test_gen_report_specs.py`, `test_base_report_formats_mermaid.py` (34 tests) — all pass, no regressions. Cleaned up the throwaway Report -afterward. Not yet released to PyPI — still pinned at `6.0.17.8` in -egeria-workspaces-fs's `requirements.txt`; needs a version bump when the -next release goes out. +afterward. Released since (it was not yet released when this was written, with +egeria-workspaces-fs then pinned at `6.0.17.8`). ---