From 1025fc96fca5faba9ba3c14562b119f46f0c06c7 Mon Sep 17 00:00:00 2001 From: Iliia_Veselov Date: Wed, 5 Aug 2026 11:39:05 +0300 Subject: [PATCH] feat: Add sdmx-proxy configuration guide --- .../references/config-and-structure.md | 9 +- architecture/services.md | 23 +- guides/README.md | 1 + guides/admin-guide.md | 87 +++++++- guides/sdmx-proxy-registry-guide.md | 201 ++++++++++++++++++ .../05-data-sources-and-channels.md | 53 ++++- learning/administration/quick-reference.md | 2 +- 7 files changed, 359 insertions(+), 17 deletions(-) create mode 100644 guides/sdmx-proxy-registry-guide.md diff --git a/.claude/skills/statgpt-admin-fe-guide-macos/references/config-and-structure.md b/.claude/skills/statgpt-admin-fe-guide-macos/references/config-and-structure.md index 8e7d891..cd6d5d2 100644 --- a/.claude/skills/statgpt-admin-fe-guide-macos/references/config-and-structure.md +++ b/.claude/skills/statgpt-admin-fe-guide-macos/references/config-and-structure.md @@ -20,7 +20,9 @@ admin UI actually stores — which differs from the seed files. Keys seen: `apiKey`, `locale`, `authConfig`, `rateLimits`, `sdmxConfig` (`id`, `url`, `name`, `headers` per resource type, `supports`, `versions`, `dataContentType`), `authEnabled`, `sdmx1Source`, `apiKeyHeader`, `attributesUrl`, `annotationsUrl`, `dataExplorerUrl`, `providerDiscovery`. The -connector (`SDMX21` / `QH_SDMX21` / `PROXY_SDMX30`) is chosen in the wizard's Properties step. +connector (`SDMX21` / `QH_SDMX21` / `PROXY_SDMX30`) is chosen in the wizard's Properties step. Keep this framing +when rewriting the connector list: **`PROXY_SDMX30` is recommended** (one data source covers many providers), +**`QH_SDMX21` is only for QuantHub-based registries**, and **`SDMX21` is deprecated**. ### Dataset config — the **`dimensions` map** is the headline schema @@ -79,6 +81,11 @@ Adjust to scope. Each list/menu screen filtered to sample content. - `ds-list` (Add, filter, row ⋯) · `ds-add-properties` (Name, connector, Next) · `ds-add-config` (editor, Finish) · `ds-configure` (editor, Save) +- **Not captured yet** (`` markers sit at the intended spots in + `admin-guide.md`): `ds-add-config-proxy` (Step 2 editor for a `PROXY_SDMX30` source, scrolled to + `configUrl` + head of `proxyConfig`) · `ds-configure-proxy` (Configure editor with a populated + `proxyConfig`). Both need a **sample-only** `PROXY_SDMX30` data source on the environment — the + sample seed configs have none, and creating one on a shared environment is out of bounds. - `datasets-list` (Add, filter, row ⋯) · `datasets-row-menu` (Edit/Delete) · `dataset-add-source` / `dataset-add-provider` / `dataset-add-dataflow` (select + Next) · `dataset-add-config` (the `dimensions` block, Finish) · `dataset-edit-config` (editor, Save) diff --git a/architecture/services.md b/architecture/services.md index fac2136..84c976b 100644 --- a/architecture/services.md +++ b/architecture/services.md @@ -145,13 +145,14 @@ invoke channel tools inside their own reasoning loops. #### Dependencies -| Service | Purpose | -|---------------------------|---------------------------------------------| -| **PostgreSQL + pgvector** | Configuration and metadata storage | +| Service | Purpose | +|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------| +| **PostgreSQL + pgvector** | Configuration and metadata storage | | **StatGPT SDMX Proxy** | Unified SDMX 3.0 facade in front of upstream SDMX registries (IMF, BIS, …). Admin Backend reads metadata for dataset onboarding through the proxy. | -| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) | -| **AI DIAL** | Content storage (files, archives) | -| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) | +| **SDMX Proxy Config Server** | Owns the proxy's registry configuration. | +| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) | +| **AI DIAL** | Content storage (files, archives) | +| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) | ### 🔁 StatGPT SDMX Proxy @@ -174,7 +175,9 @@ invoke channel tools inside their own reasoning loops. **Primary Function**: Expose a single, unified [SDMX 3.0 REST API](https://github.com/sdmx-twg/sdmx-rest/tree/master/doc) in front of multiple upstream SDMX registries (IMF, BIS, …) so that StatGPT components are decoupled from per-registry version, format, auth, and -quirk differences. +quirk differences. It is the recommended path for reaching a registry: a `PROXY_SDMX30` data source covers every +registry the proxy routes, so new ones are onboarded by configuration instead of one data source per provider. The +direct `SDMX21` connector is deprecated; `QH_SDMX21` remains in use for QuantHub-based registries. **Key Responsibilities**: @@ -182,7 +185,9 @@ quirk differences. - Format conversion across SDMX-JSON, SDMX-ML (XML), and SDMX-CSV - Agency-based routing (including sub-agency wildcarding and synthetic AgencyScheme discovery) - Caching, circuit breaking, retries, rate limiting, and per-registry response patching -- Configuration-driven onboarding of new registries (no code change required) +- Configuration-driven onboarding of new registries, without a code change or a redeploy: the configuration is + served by the config server and editable from the StatGPT Admin Portal as the `proxyConfig` of a + `PROXY_SDMX30` data source (see the [proxy registry guide](../guides/sdmx-proxy-registry-guide.md)) #### Authentication & Authorization @@ -198,7 +203,7 @@ network. |----------------------|--------------------------------------------------------------------------------------------------------| | **SDMX Registries** | Upstream statistical data sources (IMF, BIS, and any registry added via configuration) | | **Redis** (optional) | Distributed cache (`CACHE_MODE=REDIS`). Falls back to in-memory Caffeine when unavailable. | -| **Config Server** (optional) | `sdmx-proxy-config-server` module — runtime-managed registry configuration when the bundled defaults aren't enough. | +| **Config Server** | `sdmx-proxy-config-server` module — the authoritative source of registry and agency-routing configuration at runtime. Admin Backend reads and writes it as the `proxyConfig` of a `PROXY_SDMX30` data source, so registries are onboarded without redeploying the proxy. Schema: [`sdmx-proxy-config/README.md`](https://github.com/epam/statgpt-sdmx-proxy/blob/development/sdmx-proxy-config/README.md). | ### 🕹️ StatGPT Admin Frontend diff --git a/guides/README.md b/guides/README.md index 55ded5a..5e21318 100644 --- a/guides/README.md +++ b/guides/README.md @@ -5,5 +5,6 @@ This directory contains guides on StatGPT components. ## Available Guides - [Administrator Guide](./admin-guide.md) - Instructions for StatGPT configuration management. + - [Configuring SDMX Registries on a Proxy Data Source](./sdmx-proxy-registry-guide.md) - Instructions for configuring the SDMX Proxy connected to a data source. - [GTDC Portal Guide](./gtdc-portal-guide.md) - Instructions for using the GTDC (Global Trusted Data Commons) Portal. - [StatGPT Admin Azure Auth Guide](./admin-azure-auth-guide.md) - Entra ID auth for StatGPT Admin Frontend and StatGPT Admin Backend. diff --git a/guides/admin-guide.md b/guides/admin-guide.md index 7f48028..0a6ea95 100644 --- a/guides/admin-guide.md +++ b/guides/admin-guide.md @@ -52,7 +52,10 @@ StatGPT uses several key concepts that are referenced throughout this document: 1. **Data Source** — A source of data that can be queried using the SDMX protocol. Examples include IMF, Eurostat, World Bank, etc. Each data source has a **connector** that determines how StatGPT talks to it - (`SDMX21`, `QH_SDMX21`, or `PROXY_SDMX30`). + (`PROXY_SDMX30`, `QH_SDMX21`, or the deprecated `SDMX21`). `PROXY_SDMX30` is the **recommended** connector: it + points at the StatGPT SDMX Proxy, which fronts several upstream registries, so one source serves many providers + and new registries are added by configuration rather than by adding sources — see + [Configuring SDMX Registries on a Proxy Data Source](./sdmx-proxy-registry-guide.md). 2. **Dataset** — A direct representation of an SDMX dataflow in a data source, plus StatGPT-specific configuration (dimension roles, default queries, indexing options, citation, etc.). 3. **Channel** — A representation of the StatGPT application for end users. Each channel has its own configuration, @@ -92,9 +95,11 @@ Adding a source is a two-step wizard. 1. **Name** — a unique identifier for the source (e.g. `IMF_SDMX21`). An optional description can be added below. 2. **Connector** — choose how StatGPT connects to the registry: - - `SDMX21` — a standard SDMX 2.1 REST endpoint. - - `QH_SDMX21` — an SDMX 2.1 endpoint served through QuantHub (used by the IMF sample). - - `PROXY_SDMX30` — an SDMX 3.0 source accessed through the StatGPT SDMX Proxy. + - `PROXY_SDMX30` — **recommended.** Access through the StatGPT SDMX Proxy. One source covers many registries, and + new ones are added by editing its configuration in this UI — no code change and no redeploy. + - `QH_SDMX21` — only for QuantHub-based registries (used by the IMF sample). + - `SDMX21` — a direct connection to a standard SDMX 2.1 REST endpoint. **Deprecated:** existing sources keep + working, but use `PROXY_SDMX30` for new ones. 3. **Next** — continue to the configuration editor. **Step 2 — Configuration.** Provide the connector configuration as YAML, then click **Finish**. @@ -146,6 +151,63 @@ dataExplorerUrl: https://data.imf.org/en/Data-Explorer # link surfaced to end u providerDiscovery: dataflows # how providers/agencies are discovered ``` + + +A source for the StatGPT SDMX Proxy (connector `PROXY_SDMX30`, the recommended one) differs in two ways: +`sdmxConfig.url` points at the **proxy**, not at a registry, and two extra keys appear. The values below are +illustrative — substitute the proxy URL and ID of your deployment: + +```yaml +locale: en +sdmxConfig: + id: STATGPT_SDMX_PROXY # unique data source id + name: StatGPT SDMX Proxy + url: http://statgpt-sdmx-proxy:8050 # the proxy, never an upstream registry +providerDiscovery: agencyscheme # required for this connector; `dataflows` is rejected +configUrl: $env:{SDMX_PROXY_CONFIG_SERVER_HOST}/statgpt/sdmx-proxy-config-server/api/v0/config +proxyConfig: # the proxy's registry and agency-routing configuration + structureFanOutEnabled: true + configs: + - name: OECD + description: Organisation for Economic Co-operation and Development + versions: + SDMX_2_1: + sdmxVersion: SDMX_2_1 + structureEndpointConfig: + url: https://sdmx.oecd.org/public/rest/ + supportedFormats: [XML_STRUCTURE_2_1] + defaultFormat: XML_STRUCTURE_2_1 + supportedStructures: [datastructure, dataflow, codelist, conceptscheme] # abbreviated + # ...dataEndpointConfig, availabilityEndpointConfig, resilienceConfig + agencies: + - name: OECD + primaryRegistry: OECD + allowSubAgencies: true +``` + +**About `proxyConfig`.** It is the SDMX Proxy's registry configuration: which upstream registries exist, how each is +queried, and which agency routes to which registry. + +- **The config server owns it, not StatGPT's database.** StatGPT reads it from the SDMX Proxy config server when the + data source is read and pushes it back when the source is created or saved. `configUrl` locates that server and + resolves the `SDMX_PROXY_CONFIG_SERVER_HOST` environment variable. +- **Omitting the key does not clear the value.** A save with no `proxyConfig` block leaves the stored registry + configuration untouched, so the other fields can be edited while the config server is unavailable. +- **`422`** — the config server rejected the configuration; the message names the field at fault. **`502`** — the + config server could not be reached while the change was pushed. The save fails; the change is not dropped. +- **A missing `proxyConfig` is a symptom.** Absent means the config server could not be read. Present but `null` means + it holds no configuration yet. + + + +> **Onboarding a registry.** The minimum block, the fields whose defaults amount to a refusal, per-registry quirks, +> fixtures, and resilience are covered in +> [Configuring SDMX Registries on a Proxy Data Source](./sdmx-proxy-registry-guide.md). + ### Editing a data source To change connection parameters later, open the row **⋯** menu and choose **Configure**. This reopens the YAML editor. @@ -155,6 +217,14 @@ To change connection parameters later, open the row **⋯** menu and choose **Co 1. **Configuration editor** — edit the YAML in place. 2. **Save** — apply the changes. +For a `PROXY_SDMX30` source, the same editor changes the proxy's registry configuration. Edit `proxyConfig` and +**Save**, and the change is pushed to the config server; the proxy is not redeployed. That configuration is not stored +in StatGPT's database, so it is re-read every time the source is opened. + + + > See [Module 05 — Data Sources & Channel Configuration](../learning/administration/05-data-sources-and-channels.md) > of the learning course for connector details and source-discovery options, and > [SDMX Compatibility & Requirements](../architecture/sdmx-compatibility.md) for the technical requirements a source @@ -600,6 +670,13 @@ import or export runs as a background **job** linked to the channel. - `Update data sources` — update data sources to the versions in the archive. 3. **Import** — start the import job. +> **`PROXY_SDMX30` sources in an archive.** An export reads `proxyConfig` from the config server and writes it into the +> archive with the rest of the data source configuration, so registry configuration travels with the channel. On +> import, `Update data sources` pushes it to the *target* environment's config server, which can change how every +> `PROXY_SDMX30` source there queries its registries. Two exceptions: an archived source whose `details` match the +> existing ones is skipped, and a source exported while the config server was unreachable carries no `proxyConfig`, so +> importing it leaves the target untouched. + **Jobs.** Open the channel **⋯** menu → **Jobs** to see the import/export history for the channel, review job status, and download artifacts. @@ -629,5 +706,7 @@ Each entry also records who initiated the change (the **Initiated** column, reda ## Related resources - [Admin Learning Course](../learning/administration/README.md) — end-to-end dataset onboarding methodology. +- [Configuring SDMX Registries on a Proxy Data Source](./sdmx-proxy-registry-guide.md) — the `proxyConfig` block of a + `PROXY_SDMX30` data source. - [Architecture Overview](../architecture/overview.md) and [Agent design](../architecture/agent.md). - [SDMX Compatibility & Requirements](../architecture/sdmx-compatibility.md). diff --git a/guides/sdmx-proxy-registry-guide.md b/guides/sdmx-proxy-registry-guide.md new file mode 100644 index 0000000..c5551c9 --- /dev/null +++ b/guides/sdmx-proxy-registry-guide.md @@ -0,0 +1,201 @@ +# Configuring SDMX Registries on a Proxy Data Source + +This document covers the `proxyConfig` block of a `PROXY_SDMX30` data source — the **registries** the StatGPT SDMX +Proxy fronts, the **per-registry query settings** that make each one answer correctly, and the **agency routing** that +decides which registry serves which agency. + +A `PROXY_SDMX30` data source points at the proxy instead of connecting to a registry directly. It is the recommended +connector: one data source covers many providers, and new registries are added by editing `proxyConfig` from the Admin +Portal. + +> This guide covers the settings you need to add a registry and get correct answers out of it. The +> [full field reference](https://github.com/epam/statgpt-sdmx-proxy/blob/development/sdmx-proxy-config/README.md) lists +> every field. This guide extends the [Administrator Guide](./admin-guide.md#adding-a-data-source), which covers adding +> and editing the data source itself. + +## Table of Contents + +- [The model](#the-model) +- [Adding a registry](#adding-a-registry) + - [Minimum block](#minimum-block) +- [Registry compatibility settings](#registry-compatibility-settings) +- [Fixtures](#fixtures) +- [Formats](#formats) +- [Resilience](#resilience) +- [Operational notes](#operational-notes) +- [Full field reference](#full-field-reference) +- [Related resources](#related-resources) + + +## The model + +The configuration has three levels: + +1. **Data source** (`PROXY_SDMX30`) — StatGPT's handle on the proxy. `sdmxConfig.url` points at the proxy. +2. **Registries** (`proxyConfig.configs`) — one entry per upstream API. Each is configured per SDMX version, because a + registry can behave differently on its 2.1 and 3.0 endpoints. A version configures up to three endpoints — + structure, data, availability. +3. **Agency routing** (`proxyConfig.agencies`) — which registry answers for an agency ID as it appears in SDMX URLs. + `primaryRegistry` names the registry. `allowSubAgencies: true` extends it to dot-separated descendants, so `IMF.STA` + is served by the registry configured for `IMF`. + +**Routing is explicit.** An agency with no `agencies` entry is rejected, even if a registry of the same name exists. A +registry in `configs` with no agency entry is unreachable. + +**`structureFanOutEnabled` is a root-level switch.** When true, a structure query that names no agency (`agencyID=*`) +is sent to every registry that supports the requested structure type and the results are merged. When false, that query +is rejected. A comma-separated list of agencies (`agencyID=BIS,IMF`) is rejected either way. The schema default is +`false`; the configuration shipped with the proxy sets it to `true`. + +--- + +## Adding a registry + +### Minimum block + +The IMF entry from the shipped configuration: + +```yaml +proxyConfig: + configs: + - name: IMF # routing key; must match agencies[].primaryRegistry + description: International Monetary Fund + versions: + SDMX_3_0: # or SDMX_2_1 — key and sdmxVersion must agree + sdmxVersion: SDMX_3_0 + structureEndpointConfig: + url: https://api.imf.org/external/sdmx/3.0/structure/ + supportedFormats: [JSON_STRUCTURE_2_0_0] + defaultFormat: JSON_STRUCTURE_2_0_0 + supportedStructures: # unset: every structure query is rejected + - datastructure + - conceptscheme + - codelist + - dataflow + - hierarchy + - hierarchyassociation + - metadatastructure + - metadataflow + - metadataprovisionagreement + dataEndpointConfig: + url: https://api.imf.org/external/sdmx/3.0/data/ + supportedFormats: [JSON_DATA_2_0_0, CSV_DATA_2_0_0] + defaultFormat: JSON_DATA_2_0_0 + replaceEmptyDimensionsWithWildcard: true # IMF wants explicit wildcards in a key + availabilityEndpointConfig: + url: https://api.imf.org/external/sdmx/3.0/availability/ + supportedFormats: [JSON_STRUCTURE_2_0_0] + defaultFormat: JSON_STRUCTURE_2_0_0 + availabilityEnabled: true # unset: availability queries are rejected + # per-endpoint `fixtures` and `resilienceConfig` omitted here — see the sections below + agencies: + - name: IMF # agency ID as it appears in SDMX URLs + primaryRegistry: IMF # without this entry the registry is unreachable + allowSubAgencies: true # IMF.STA, IMF.RES, ... route here too +``` + +**Four things to set explicitly:** + +- **An `agencies` entry** — without one the registry serves nothing. +- **`supportedStructures`** — structure types outside this set are rejected. Unset means the empty set, so every + structure query fails. Values used in the shipped configuration: `agencyscheme`, `categoryscheme`, `codelist`, + `conceptscheme`, `dataflow`, `datastructure`, `hierarchy`, `hierarchyassociation`, `metadataflow`, + `metadataprovisionagreement`, `metadatastructure`. +- **`availabilityEnabled`** — defaults to `false`. Availability is used in dataset onboarding and querying, so leaving it off makes the registry unusable. +- **`defaultFormat`** — the format requested from the registry. + +--- + +## Registry compatibility settings + +The settings below let the proxy adapt a request to what a particular registry expects. + +| Symptom | Setting | Endpoint | +|---------|---------|----------| +| A key with empty positions (`..L_T.P_F3`) returns nothing, while explicit wildcards work | `replaceEmptyDimensionsWithWildcard: true` | data | +| A key of all wildcards (`*.*.*.*`) returns an empty response, while a single `*` works | `mergeAllWildcardKey: true` | data, availability | +| Availability expects the dimensions listed rather than `*` as the component ID | `unwrapStarComponentId: true` | availability | +| Narrowing a query barely changes the result, and availability queries get slow | `convertKeyToFilters: true` | data, availability | +| The `limit` parameter has no effect, so broad queries return everything | `supportsLimit: false` | data | +| Availability filters return nothing, while the registry answers the same filters expressed another way | `unwrapFilterParameters: true` | availability | + +`unwrapFilterParameters` is not needed by any registry in the shipped configuration — `convertKeyToFilters` is the +better fit for the one case where it might apply. + +> **⚠️ SDMX 3.0 only.** `convertKeyToFilters` and `unwrapStarComponentId` apply to SDMX 3.0 only. Setting either inside +> an `SDMX_2_1` block makes the configuration invalid, and the save is rejected. + +**Emulating `limit`.** `supportsLimit: false` lets the proxy cap the response size itself. Two settings tune that: +`limitEmulationTolerance` (default `1.2`, range `1.0`–`10.0`) allows the result to overshoot the requested limit by that +factor, and `limitEmulationProbeBudget` (default `8`, range `1`–`64`) limits how much work goes into hitting it. Raising +the budget slows every request. + +--- + +## Fixtures + +Where a registry's response differs from what StatGPT expects, a fixture adjusts it on the way through. Add one when +the matching symptom appears. Each is scoped to an endpoint, and several can be listed for the same endpoint. + +| Endpoint | Type | Adjusts | +|----------|------|---------| +| structure | `DSD_ATTRIBUTE_ATTACHMENT_LEVEL` | Attribute attachment levels on a data structure | +| structure | `VERSION_WILDCARD` | Wildcard versions in structure responses | +| structure | `ANNOTATION_VALUE_TO_TEXT` | Annotation values that would otherwise not reach StatGPT | +| structure | `PRESERVE_METADATA_ATTRIBUTE_USAGES` | Metadata attribute usages on a data structure | +| availability | `MOVE_CUBE_REGION_COMPONENTS_TO_KEY_VALUES` | Where components appear in an availability response | +| data | `TIME_PERIOD_MONTHLY_NORMALIZATION` | Monthly periods returned as `2024-03` into the `2024-M03` form StatGPT uses to recognise monthly data | +| data | `PRESERVE_METADATA_ATTRIBUTES` | Metadata attribute values in data responses | + +--- + +## Formats + +Each endpoint declares which formats the registry can return (`supportedFormats`) and which one to ask it for +(`defaultFormat`). The proxy converts the response to whatever the client asked for. The +[reference](https://github.com/epam/statgpt-sdmx-proxy/blob/development/sdmx-proxy-config/README.md#enum-sdmxformat) +lists the available format values. + +`bypassEnabled` is off by default, and every shipped registry leaves it off. Turning it on lets a response go straight +to the client whenever the registry already returns the requested format. That is faster, but such a response keeps any +defects the [fixtures](#fixtures) above would have corrected. + +--- + +## Resilience + +`resilienceConfig` is set per registry per version. All durations are in milliseconds; connect and read timeouts default +to `30000`. Four independent blocks sit below it: + +- **`circuitBreaker`** — pause calls to a registry that is not responding, and resume after a wait. +- **`retry`** — server and network errors. +- **`rateLimit`** — cap outbound requests. +- **`rateLimitRetry`** — wait and retry when a registry answers "too many requests". + +--- + +## Operational notes + +**`rateLimitRetry` changes need a proxy restart** before they take effect. Every other setting applies without one. + +--- + +## Full field reference + +Every field, with required/default/allowed values, is maintained next to the code by the proxy team: + +- [Registry configuration schema](https://github.com/epam/statgpt-sdmx-proxy/blob/development/sdmx-proxy-config/README.md) + — registries, per-version and per-endpoint settings, resilience, agencies, fixtures, and formats. +- [Config server](https://github.com/epam/statgpt-sdmx-proxy/blob/development/sdmx-proxy-config-server/README.md) + — storage backends, environment variables, and the forced-reseed flow. + +--- + +## Related resources + +- [Administrator Guide](./admin-guide.md#adding-a-data-source) — adding and editing data sources, datasets, channels. +- [Module 05 — Data Sources & Channel Configuration](../learning/administration/05-data-sources-and-channels.md) — data + source configuration in the onboarding course. +- [Services](../architecture/services.md#-statgpt-sdmx-proxy) — where the proxy and its config server sit. +- [SDMX Compatibility & Requirements](../architecture/sdmx-compatibility.md) — what an upstream registry must provide. +- [statgpt-sdmx-proxy](https://github.com/epam/statgpt-sdmx-proxy) — the proxy itself. diff --git a/learning/administration/05-data-sources-and-channels.md b/learning/administration/05-data-sources-and-channels.md index 733ef44..5913252 100644 --- a/learning/administration/05-data-sources-and-channels.md +++ b/learning/administration/05-data-sources-and-channels.md @@ -20,10 +20,25 @@ A Data Source defines the connection to an external SDMX data provider. You must ### Steps in the Admin UI 1. Navigate to **Data Sources** tab and click **Add** -2. Select the data source type (e.g., SDMX 2.1) +2. Select the data source type — `PROXY_SDMX30` is recommended (see below) 3. Fill in the configuration YAML 4. Save +### Choosing a connector + +| Connector | Use it for | Status | +|-----------|------------|--------| +| `PROXY_SDMX30` | Access through the StatGPT SDMX Proxy | **Recommended** — the default choice | +| `QH_SDMX21` | QuantHub-based registries only | Supported for that case | +| `SDMX21` | A direct connection to a standard SDMX 2.1 REST endpoint | **Deprecated** | + +`PROXY_SDMX30` is recommended over a direct connection because the proxy absorbs the per-provider work: + +- **One Data Source covers many providers.** New registries are added by editing that source's `proxyConfig` in the Admin UI. +- **Registry differences are handled for you.** The proxy deals with the SDMX version and response format each + registry speaks, and with known defects in their responses, so StatGPT does not have to. +- **Timeouts, retries, and rate limits are configured per registry** in the same place. + ### Data Source Configuration Here is the configuration YAML for connecting to the IMF SDMX 2.1 API: @@ -52,6 +67,33 @@ sdmxConfig: # All other features default to true ``` +A `PROXY_SDMX30` Data Source — the recommended kind — reads through the StatGPT SDMX Proxy instead of a +registry, and adds two keys. The values are illustrative; use the proxy URL and ID of your deployment: + +```yaml +sdmxConfig: + id: STATGPT_SDMX_PROXY + name: StatGPT SDMX Proxy + url: http://statgpt-sdmx-proxy:8050 # The proxy, not an upstream registry +providerDiscovery: agencyscheme # Required here: `dataflows` is rejected for this connector +configUrl: $env:{SDMX_PROXY_CONFIG_SERVER_HOST}/statgpt/sdmx-proxy-config-server/api/v0/config +proxyConfig: # The proxy's registries and agency routing + structureFanOutEnabled: true + configs: + - name: OECD + versions: + SDMX_2_1: { } # Endpoints, formats, and quirk flags — abbreviated + agencies: + - name: OECD + primaryRegistry: OECD + allowSubAgencies: true +``` + +The SDMX Proxy config server owns `proxyConfig`, not StatGPT's database: it is read when the Data Source +is read and pushed back on save. The minimum block per registry, the fields whose defaults amount to a +refusal, and the per-registry quirk flags are covered in +[Configuring SDMX Registries on a Proxy Data Source](../../guides/sdmx-proxy-registry-guide.md). + ### Key Fields | Field | Description | @@ -65,6 +107,8 @@ sdmxConfig: | `annotationsUrl` | Optional URL for fetching SDMX annotations (some providers serve annotations from a different endpoint) | | `attributesUrl` | Optional URL for fetching SDMX attributes from a different endpoint | | `dataExplorerUrl` | Optional URL to the provider's web-based data explorer (shown in citations) | +| `configUrl` | `PROXY_SDMX30` only. URL of the SDMX Proxy config server. Defaults to `$env:{SDMX_PROXY_CONFIG_SERVER_HOST}` plus the server's standard path — override only for a non-standard deployment | +| `proxyConfig` | `PROXY_SDMX30` only. The proxy's registry and agency-routing configuration. **Owned by the config server**, never stored in StatGPT's database: read when the Data Source is read, pushed back on save. Omitting it on save leaves the stored value untouched. See the [proxy registry guide](../../guides/sdmx-proxy-registry-guide.md) | ### Authentication @@ -74,13 +118,18 @@ When `authEnabled: true`, provide credentials: ### What Data Sources Exist For -Each data provider you want to query needs its own Data Source. For example: +With a direct connection, each data provider needs its own Data Source. For example: - An IMF data source for IMF datasets (WEO, CPI, BOP, etc.) - A Eurostat data source for Eurostat datasets (NAMA_10_GDP, etc.) - A World Bank data source for WDI datasets A single channel can include datasets from multiple data sources. +> **With `PROXY_SDMX30`, one Data Source covers many providers** — which is the main reason it is the +> recommended connector. The proxy fronts several upstream registries and routes each agency to one of +> them, so a new provider means a new entry in that source's `proxyConfig`, not a new Data Source. See +> [Configuring SDMX Registries on a Proxy Data Source](../../guides/sdmx-proxy-registry-guide.md). + --- ## Creating a Channel diff --git a/learning/administration/quick-reference.md b/learning/administration/quick-reference.md index ae0c7c2..f652032 100644 --- a/learning/administration/quick-reference.md +++ b/learning/administration/quick-reference.md @@ -157,7 +157,7 @@ See [Module 04](04-dataset-configuration.md) for field-by-field details. ### Phase 5: Data Source & Channel ([Module 05](05-data-sources-and-channels.md)) -- [ ] Data Source exists for the provider +- [ ] Data Source exists for the provider (`PROXY_SDMX30` recommended) - [ ] Dataset linked to channel - [ ] Channel Named Entity types updated if new NON_INDICATOR dimensions added