Skip to content

feat: Add SDMX Proxy to documentation #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
iliia-veselov merged 1 commit into from
May 18, 2026
Merged

feat: Add SDMX Proxy to documentation #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions architecture/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ Stateless services design enables horizontal scaling to handle varying loads eff
- [StatGPT Backend](https://github.com/epam/statgpt-backend) - Core services implementation
- [StatGPT Admin Frontend](https://github.com/epam/statgpt-admin-frontend) - Administrative interface
- [StatGPT Portal Frontend](https://github.com/epam/statgpt-portal-frontend) - User interface library
- [StatGPT SDMX Proxy](https://github.com/epam/statgpt-sdmx-proxy) - Unified SDMX 3.0 facade in front of upstream registries
- [StatGPT Helm](https://github.com/epam/statgpt-helm) - Kubernetes deployment charts

### External Documentation
Expand Down
2 changes: 1 addition & 1 deletion architecture/content/architecture_detailed.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
68 changes: 40 additions & 28 deletions architecture/content/diagrams.drawio
View file
Open in desktop

Large diffs are not rendered by default.

79 changes: 63 additions & 16 deletions architecture/services.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,13 +88,13 @@ with [DIAL API Specification](https://dialx.ai/dial_api#operation/sendChatComple

#### Dependencies

| Service | Purpose |
|---------------------------|------------------------------------------------------|
| **AI DIAL** | Platform services and model access |
| **PostgreSQL + pgvector** | Data storage and vector operations |
| **ElasticSearch** | Search and indexing |
| **External AI Models** | LLM capabilities (e.g., Azure OpenAI) |
| **SDMX Providers** | Statistical data sources (IMF, World Bank, Eurostat) |
| Service | Purpose |
|---------------------------|-----------------------------------------------------------------------------|
| **AI DIAL** | Platform services and model access |
| **PostgreSQL + pgvector** | Data storage and vector operations |
| **ElasticSearch** | Search and indexing |
| **External AI Models** | LLM capabilities (e.g., Azure OpenAI) |
| **StatGPT SDMX Proxy** | Unified SDMX 3.0 facade in front of upstream SDMX registries (IMF, BIS, …). |

#### MCP Endpoint

Expand Down Expand Up @@ -147,11 +147,58 @@ invoke channel tools inside their own reasoning loops.

| Service | Purpose |
|---------------------------|---------------------------------------------|
| **PostgreSQL + pgvector** | Configuration and metadata storage |
| **SDMX Providers** | Dataset sources (IMF, World Bank, Eurostat) |
| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) |
| **AI DIAL** | Content storage (files, archives) |
| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) |
| **PostgreSQL + pgvector** | Configuration and metadata storage |
| **StatGPT SDMX Proxy** | Unified SDMX 3.0 facade in front of upstream SDMX registries (IMF, BIS, …). Admin Backend reads metadata for dataset onboarding through the proxy. |
| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) |
| **AI DIAL** | Content storage (files, archives) |
| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) |

### 🔁 StatGPT SDMX Proxy

#### Technology Stack

**Type**: Standalone Spring Boot service ([repository](https://github.com/epam/statgpt-sdmx-proxy))

**Core Frameworks**:

| Framework | Purpose |
|-------------------------------------------------------------------|--------------------------------------------------|
| [Spring Boot 4.0](https://spring.io/projects/spring-boot) | Application framework |
| [OpenFeign](https://github.com/OpenFeign/feign) | Declarative HTTP clients for upstream registries |
| [Resilience4j](https://resilience4j.readme.io/) | Circuit breaker, retry, rate limiting |
| [Spring Data Redis](https://spring.io/projects/spring-data-redis) | Distributed caching backend |
| [OpenTelemetry](https://opentelemetry.io/) | Observability (traces, metrics, logs) |

#### Overview

**Primary Function**: Expose a single, unified
[SDMX 3.0 REST API](https://github.com/sdmx-twg/sdmx-rest/tree/master/doc) in front of multiple upstream SDMX
registries (IMF, BIS, …) so that StatGPT components are decoupled from per-registry version, format, auth, and
quirk differences.

**Key Responsibilities**:

- Protocol translation between SDMX 2.1 and SDMX 3.0
- Format conversion across SDMX-JSON, SDMX-ML (XML), and SDMX-CSV
- Agency-based routing (including sub-agency wildcarding and synthetic AgencyScheme discovery)
- Caching, circuit breaking, retries, rate limiting, and per-registry response patching
- Configuration-driven onboarding of new registries (no code change required)

#### Authentication & Authorization

**Outbound** (to upstream SDMX registries): currently the proxy does not send credentials to upstream registries.

**Inbound**: the proxy itself is not directly exposed to the internet. External traffic reaches it through DIAL
Core, which is responsible for authorizing requests. StatGPT backends call the proxy over the internal cluster
network.

#### Dependencies

| Service | Purpose |
|----------------------|--------------------------------------------------------------------------------------------------------|
| **SDMX Registries** | Upstream statistical data sources (IMF, BIS, and any registry added via configuration) |
| **Redis** (optional) | Distributed cache (`CACHE_MODE=REDIS`). Falls back to in-memory Caffeine when unavailable. |
| **Config Server** (optional) | `sdmx-proxy-config-server` module — runtime-managed registry configuration when the bundled defaults aren't enough. |

### 🕹️ StatGPT Admin Frontend

Expand Down Expand Up @@ -239,7 +286,7 @@ invoke channel tools inside their own reasoning loops.

| Service | Purpose |
|--------------------------|-------------------------------------------------------|
| **DIAL Platform** | Authentication, rate limits, file storage, API access |
| **StatGPT Chat Backend** | Core chat and data query services |
| **Identity Provider** | User authentication and SSO |
| **SDMX Providers** | Direct metadata and data queries |
| **DIAL Platform** | Authentication, rate limits, file storage, API access |
| **StatGPT Chat Backend** | Core chat and data query services |
| **Identity Provider** | User authentication and SSO |
| **StatGPT SDMX Proxy** | Metadata and data queries from the Advanced View editor — Portal does not call upstream SDMX registries directly. |
Loading