Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ admin UI actually stores — which differs from the seed files.
Keys seen: `apiKey`, `locale`, `authConfig`, `rateLimits`, `sdmxConfig` (`id`, `url`, `name`,
`headers` per resource type, `supports`, `versions`, `dataContentType`), `authEnabled`, `sdmx1Source`,
`apiKeyHeader`, `attributesUrl`, `annotationsUrl`, `dataExplorerUrl`, `providerDiscovery`. The
connector (`SDMX21` / `QH_SDMX21` / `PROXY_SDMX30`) is chosen in the wizard's Properties step.
connector (`SDMX21` / `QH_SDMX21` / `PROXY_SDMX30`) is chosen in the wizard's Properties step. Keep this framing
when rewriting the connector list: **`PROXY_SDMX30` is recommended** (one data source covers many providers),
**`QH_SDMX21` is only for QuantHub-based registries**, and **`SDMX21` is deprecated**.

### Dataset config — the **`dimensions` map** is the headline schema

Expand Down Expand Up @@ -79,6 +81,11 @@ Adjust to scope. Each list/menu screen filtered to sample content.

- `ds-list` (Add, filter, row ⋯) · `ds-add-properties` (Name, connector, Next) ·
`ds-add-config` (editor, Finish) · `ds-configure` (editor, Save)
- **Not captured yet** (`<!-- SCREENSHOT (pending) -->` markers sit at the intended spots in
`admin-guide.md`): `ds-add-config-proxy` (Step 2 editor for a `PROXY_SDMX30` source, scrolled to
`configUrl` + head of `proxyConfig`) · `ds-configure-proxy` (Configure editor with a populated
`proxyConfig`). Both need a **sample-only** `PROXY_SDMX30` data source on the environment — the
sample seed configs have none, and creating one on a shared environment is out of bounds.
- `datasets-list` (Add, filter, row ⋯) · `datasets-row-menu` (Edit/Delete) ·
`dataset-add-source` / `dataset-add-provider` / `dataset-add-dataflow` (select + Next) ·
`dataset-add-config` (the `dimensions` block, Finish) · `dataset-edit-config` (editor, Save)
Expand Down
23 changes: 14 additions & 9 deletions architecture/services.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,13 +145,14 @@ invoke channel tools inside their own reasoning loops.

#### Dependencies

| Service | Purpose |
|---------------------------|---------------------------------------------|
| **PostgreSQL + pgvector** | Configuration and metadata storage |
| Service | Purpose |
|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
| **PostgreSQL + pgvector** | Configuration and metadata storage |
| **StatGPT SDMX Proxy** | Unified SDMX 3.0 facade in front of upstream SDMX registries (IMF, BIS, …). Admin Backend reads metadata for dataset onboarding through the proxy. |
| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) |
| **AI DIAL** | Content storage (files, archives) |
| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) |
| **SDMX Proxy Config Server** | Owns the proxy's registry configuration. |
| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) |
| **AI DIAL** | Content storage (files, archives) |
| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) |

### 🔁 StatGPT SDMX Proxy

Expand All @@ -174,15 +175,19 @@ invoke channel tools inside their own reasoning loops.
**Primary Function**: Expose a single, unified
[SDMX 3.0 REST API](https://github.com/sdmx-twg/sdmx-rest/tree/master/doc) in front of multiple upstream SDMX
registries (IMF, BIS, …) so that StatGPT components are decoupled from per-registry version, format, auth, and
quirk differences.
quirk differences. It is the recommended path for reaching a registry: a `PROXY_SDMX30` data source covers every
registry the proxy routes, so new ones are onboarded by configuration instead of one data source per provider. The
direct `SDMX21` connector is deprecated; `QH_SDMX21` remains in use for QuantHub-based registries.

**Key Responsibilities**:

- Protocol translation between SDMX 2.1 and SDMX 3.0
- Format conversion across SDMX-JSON, SDMX-ML (XML), and SDMX-CSV
- Agency-based routing (including sub-agency wildcarding and synthetic AgencyScheme discovery)
- Caching, circuit breaking, retries, rate limiting, and per-registry response patching
- Configuration-driven onboarding of new registries (no code change required)
- Configuration-driven onboarding of new registries, without a code change or a redeploy: the configuration is
served by the config server and editable from the StatGPT Admin Portal as the `proxyConfig` of a
`PROXY_SDMX30` data source (see the [proxy registry guide](../guides/sdmx-proxy-registry-guide.md))

#### Authentication & Authorization

Expand All @@ -198,7 +203,7 @@ network.
|----------------------|--------------------------------------------------------------------------------------------------------|
| **SDMX Registries** | Upstream statistical data sources (IMF, BIS, and any registry added via configuration) |
| **Redis** (optional) | Distributed cache (`CACHE_MODE=REDIS`). Falls back to in-memory Caffeine when unavailable. |
| **Config Server** (optional) | `sdmx-proxy-config-server` module — runtime-managed registry configuration when the bundled defaults aren't enough. |
| **Config Server** | `sdmx-proxy-config-server` module — the authoritative source of registry and agency-routing configuration at runtime. Admin Backend reads and writes it as the `proxyConfig` of a `PROXY_SDMX30` data source, so registries are onboarded without redeploying the proxy. Schema: [`sdmx-proxy-config/README.md`](https://github.com/epam/statgpt-sdmx-proxy/blob/development/sdmx-proxy-config/README.md). |

### 🕹️ StatGPT Admin Frontend

Expand Down
1 change: 1 addition & 0 deletions guides/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,6 @@ This directory contains guides on StatGPT components.
## Available Guides

- [Administrator Guide](./admin-guide.md) - Instructions for StatGPT configuration management.
- [Configuring SDMX Registries on a Proxy Data Source](./sdmx-proxy-registry-guide.md) - Instructions for configuring the SDMX Proxy connected to a data source.
- [GTDC Portal Guide](./gtdc-portal-guide.md) - Instructions for using the GTDC (Global Trusted Data Commons) Portal.
- [StatGPT Admin Azure Auth Guide](./admin-azure-auth-guide.md) - Entra ID auth for StatGPT Admin Frontend and StatGPT Admin Backend.
87 changes: 83 additions & 4 deletions guides/admin-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,10 @@ StatGPT uses several key concepts that are referenced throughout this document:

1. **Data Source** — A source of data that can be queried using the SDMX protocol. Examples include IMF, Eurostat,
World Bank, etc. Each data source has a **connector** that determines how StatGPT talks to it
(`SDMX21`, `QH_SDMX21`, or `PROXY_SDMX30`).
(`PROXY_SDMX30`, `QH_SDMX21`, or the deprecated `SDMX21`). `PROXY_SDMX30` is the **recommended** connector: it
points at the StatGPT SDMX Proxy, which fronts several upstream registries, so one source serves many providers
and new registries are added by configuration rather than by adding sources — see
[Configuring SDMX Registries on a Proxy Data Source](./sdmx-proxy-registry-guide.md).
2. **Dataset** — A direct representation of an SDMX dataflow in a data source, plus StatGPT-specific configuration
(dimension roles, default queries, indexing options, citation, etc.).
3. **Channel** — A representation of the StatGPT application for end users. Each channel has its own configuration,
Expand Down Expand Up @@ -92,9 +95,11 @@ Adding a source is a two-step wizard.

1. **Name** — a unique identifier for the source (e.g. `IMF_SDMX21`). An optional description can be added below.
2. **Connector** — choose how StatGPT connects to the registry:
- `SDMX21` — a standard SDMX 2.1 REST endpoint.
- `QH_SDMX21` — an SDMX 2.1 endpoint served through QuantHub (used by the IMF sample).
- `PROXY_SDMX30` — an SDMX 3.0 source accessed through the StatGPT SDMX Proxy.
- `PROXY_SDMX30` — **recommended.** Access through the StatGPT SDMX Proxy. One source covers many registries, and
new ones are added by editing its configuration in this UI — no code change and no redeploy.
- `QH_SDMX21` — only for QuantHub-based registries (used by the IMF sample).
- `SDMX21` — a direct connection to a standard SDMX 2.1 REST endpoint. **Deprecated:** existing sources keep
working, but use `PROXY_SDMX30` for new ones.
3. **Next** — continue to the configuration editor.

**Step 2 — Configuration.** Provide the connector configuration as YAML, then click **Finish**.
Expand Down Expand Up @@ -146,6 +151,63 @@ dataExplorerUrl: https://data.imf.org/en/Data-Explorer # link surfaced to end u
providerDiscovery: dataflows # how providers/agencies are discovered
```

<!-- SCREENSHOT (pending): ds-add-properties.png — re-capture only if the Connector dropdown changed
when PROXY_SDMX30 configuration landed. Badges: 1 Name, 2 Connector, 3 Next. -->

A source for the StatGPT SDMX Proxy (connector `PROXY_SDMX30`, the recommended one) differs in two ways:
`sdmxConfig.url` points at the **proxy**, not at a registry, and two extra keys appear. The values below are
illustrative — substitute the proxy URL and ID of your deployment:

```yaml
locale: en
sdmxConfig:
id: STATGPT_SDMX_PROXY # unique data source id
name: StatGPT SDMX Proxy
url: http://statgpt-sdmx-proxy:8050 # the proxy, never an upstream registry
providerDiscovery: agencyscheme # required for this connector; `dataflows` is rejected
configUrl: $env:{SDMX_PROXY_CONFIG_SERVER_HOST}/statgpt/sdmx-proxy-config-server/api/v0/config
proxyConfig: # the proxy's registry and agency-routing configuration
structureFanOutEnabled: true
configs:
- name: OECD
description: Organisation for Economic Co-operation and Development
versions:
SDMX_2_1:
sdmxVersion: SDMX_2_1
structureEndpointConfig:
url: https://sdmx.oecd.org/public/rest/
supportedFormats: [XML_STRUCTURE_2_1]
defaultFormat: XML_STRUCTURE_2_1
supportedStructures: [datastructure, dataflow, codelist, conceptscheme] # abbreviated
# ...dataEndpointConfig, availabilityEndpointConfig, resilienceConfig
agencies:
- name: OECD
primaryRegistry: OECD
allowSubAgencies: true
```

**About `proxyConfig`.** It is the SDMX Proxy's registry configuration: which upstream registries exist, how each is
queried, and which agency routes to which registry.

- **The config server owns it, not StatGPT's database.** StatGPT reads it from the SDMX Proxy config server when the
data source is read and pushes it back when the source is created or saved. `configUrl` locates that server and
resolves the `SDMX_PROXY_CONFIG_SERVER_HOST` environment variable.
- **Omitting the key does not clear the value.** A save with no `proxyConfig` block leaves the stored registry
configuration untouched, so the other fields can be edited while the config server is unavailable.
- **`422`** — the config server rejected the configuration; the message names the field at fault. **`502`** — the
config server could not be reached while the change was pushed. The save fails; the change is not dropped.
- **A missing `proxyConfig` is a symptom.** Absent means the config server could not be read. Present but `null` means
it holds no configuration yet.

<!-- SCREENSHOT (pending): ds-add-config-proxy.png — the Step 2 Configuration editor with a
PROXY_SDMX30 source selected, scrolled so `configUrl` and the head of `proxyConfig` are visible.
Badges: 1 Configuration editor, 2 the proxyConfig block, 3 Finish. Requires a sample-only
PROXY_SDMX30 data source on the environment; none exists in the sample seed configs today. -->

> **Onboarding a registry.** The minimum block, the fields whose defaults amount to a refusal, per-registry quirks,
> fixtures, and resilience are covered in
> [Configuring SDMX Registries on a Proxy Data Source](./sdmx-proxy-registry-guide.md).

### Editing a data source

To change connection parameters later, open the row **⋯** menu and choose **Configure**. This reopens the YAML editor.
Expand All @@ -155,6 +217,14 @@ To change connection parameters later, open the row **⋯** menu and choose **Co
1. **Configuration editor** — edit the YAML in place.
2. **Save** — apply the changes.

For a `PROXY_SDMX30` source, the same editor changes the proxy's registry configuration. Edit `proxyConfig` and
**Save**, and the change is pushed to the config server; the proxy is not redeployed. That configuration is not stored
in StatGPT's database, so it is re-read every time the source is opened.

<!-- SCREENSHOT (pending): ds-configure-proxy.png — the Configure editor of a PROXY_SDMX30 source
showing a populated proxyConfig. Badges: 1 Configuration editor, 2 Save. Same prerequisite as
ds-add-config-proxy.png. -->

> See [Module 05 — Data Sources & Channel Configuration](../learning/administration/05-data-sources-and-channels.md)
> of the learning course for connector details and source-discovery options, and
> [SDMX Compatibility & Requirements](../architecture/sdmx-compatibility.md) for the technical requirements a source
Expand Down Expand Up @@ -600,6 +670,13 @@ import or export runs as a background **job** linked to the channel.
- `Update data sources` — update data sources to the versions in the archive.
3. **Import** — start the import job.

> **`PROXY_SDMX30` sources in an archive.** An export reads `proxyConfig` from the config server and writes it into the
> archive with the rest of the data source configuration, so registry configuration travels with the channel. On
> import, `Update data sources` pushes it to the *target* environment's config server, which can change how every
> `PROXY_SDMX30` source there queries its registries. Two exceptions: an archived source whose `details` match the
> existing ones is skipped, and a source exported while the config server was unreachable carries no `proxyConfig`, so
> importing it leaves the target untouched.

**Jobs.** Open the channel **⋯** menu → **Jobs** to see the import/export history for the channel, review job status,
and download artifacts.

Expand Down Expand Up @@ -629,5 +706,7 @@ Each entry also records who initiated the change (the **Initiated** column, reda
## Related resources

- [Admin Learning Course](../learning/administration/README.md) — end-to-end dataset onboarding methodology.
- [Configuring SDMX Registries on a Proxy Data Source](./sdmx-proxy-registry-guide.md) — the `proxyConfig` block of a
`PROXY_SDMX30` data source.
- [Architecture Overview](../architecture/overview.md) and [Agent design](../architecture/agent.md).
- [SDMX Compatibility & Requirements](../architecture/sdmx-compatibility.md).
Loading