diff --git a/architecture/README.md b/architecture/README.md index ee715de..bc5f31f 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -61,6 +61,7 @@ Stateless services design enables horizontal scaling to handle varying loads eff - [StatGPT Backend](https://github.com/epam/statgpt-backend) - Core services implementation - [StatGPT Admin Frontend](https://github.com/epam/statgpt-admin-frontend) - Administrative interface - [StatGPT Portal Frontend](https://github.com/epam/statgpt-portal-frontend) - User interface library +- [StatGPT SDMX Proxy](https://github.com/epam/statgpt-sdmx-proxy) - Unified SDMX 3.0 facade in front of upstream registries - [StatGPT Helm](https://github.com/epam/statgpt-helm) - Kubernetes deployment charts ### External Documentation diff --git a/architecture/content/architecture_detailed.svg b/architecture/content/architecture_detailed.svg index df57389..790ed3c 100644 --- a/architecture/content/architecture_detailed.svg +++ b/architecture/content/architecture_detailed.svg @@ -1,4 +1,4 @@ -
Common Services (**)
Container Orchestration (Kubernetes, OCP, etc.)
Azure
AI DIAL
StatGPT
Icon-identity-230 BusinessUser
HTTP
HTTP
StatGPT App Backend
TCP
HTTP
HTTP
StatGPT Admin Backend
PostgreSQL (*)
Elasticsearch (*)
Request flow
TCP
HTTP
HTTP
DIAL Core
HTTP
DIAL OpenAI Adapter
HTTP
DIAL Realtime Analytics
HTTP
Reads
Logs
HTTP
Vector
InfluxDBRedis (*)
End user Communication Channels
DIAL Chat StatGPT PortalMCP Client
TCP
HTTP
DIAL Admin Backend
Admin Communication Channels
Grafana DIALAdmin FE StatGPTAdmin FE
HTTP
HTTP
HTTP
HTTP
Azure OpenAIBlob Storage Icon-identity-230 AdminUser
Logging
Monitoring
(OTEL)
External Services
Secret 
management
SSO
(*) - cloud-managed versions of services can be used
(**) - Common Services can be located either in Container orchestration or in different Clouds
SDMX API (HTTP)
SDMX Registries
(External)
SDMX API (HTTP)
StatGPT App MCP
HTTP
HTTP
+
Common Services (**)
Container Orchestration (Kubernetes, OCP, etc.)
Azure
AI DIAL
StatGPT
Icon-identity-230 BusinessUser
HTTP
HTTP
StatGPT App Backend
TCP
HTTP
HTTP
StatGPT Admin Backend
PostgreSQL (*)
Elasticsearch (*)
Request flow
TCP
HTTP
HTTP
DIAL Core
HTTP
DIAL OpenAI Adapter
HTTP
DIAL Realtime Analytics
HTTP
Reads
Logs
HTTP
Vector
InfluxDBRedis (*)
End user Communication Channels
DIAL Chat StatGPT PortalMCP Client
TCP
HTTP
DIAL Admin Backend
Admin Communication Channels
Grafana DIALAdmin FE StatGPTAdmin FE
HTTP
HTTP
HTTP
HTTP
Azure OpenAIBlob Storage Icon-identity-230 AdminUser
Logging
Monitoring
(OTEL)
External Services
Secret 
management
SSO
(*) - cloud-managed versions of services can be used
(**) - Common Services can be located either in Container orchestration or in different Clouds
SDMX API (HTTP)
StatGPT SDMX Proxy
SDMX API (HTTP)
StatGPT App MCP
HTTP
HTTP
SDMX Registries
HTTP
diff --git a/architecture/content/diagrams.drawio b/architecture/content/diagrams.drawio index 26ee1bf..f99ecd0 100644 --- a/architecture/content/diagrams.drawio +++ b/architecture/content/diagrams.drawio @@ -134,18 +134,18 @@ - + - + - + @@ -154,7 +154,7 @@ - + @@ -326,19 +326,19 @@ - + - + - + - + - + @@ -353,7 +353,7 @@ - + @@ -364,9 +364,7 @@ - - - + @@ -381,7 +379,7 @@ - + @@ -390,7 +388,7 @@ - + @@ -401,27 +399,22 @@ - - - + + - - + - - + + - - + + - - + - - @@ -448,6 +441,25 @@ + + + + + + + + + + + + + + + + + + + diff --git a/architecture/services.md b/architecture/services.md index f1c9a17..b844393 100644 --- a/architecture/services.md +++ b/architecture/services.md @@ -88,13 +88,13 @@ with [DIAL API Specification](https://dialx.ai/dial_api#operation/sendChatComple #### Dependencies -| Service | Purpose | -|---------------------------|------------------------------------------------------| -| **AI DIAL** | Platform services and model access | -| **PostgreSQL + pgvector** | Data storage and vector operations | -| **ElasticSearch** | Search and indexing | -| **External AI Models** | LLM capabilities (e.g., Azure OpenAI) | -| **SDMX Providers** | Statistical data sources (IMF, World Bank, Eurostat) | +| Service | Purpose | +|---------------------------|-----------------------------------------------------------------------------| +| **AI DIAL** | Platform services and model access | +| **PostgreSQL + pgvector** | Data storage and vector operations | +| **ElasticSearch** | Search and indexing | +| **External AI Models** | LLM capabilities (e.g., Azure OpenAI) | +| **StatGPT SDMX Proxy** | Unified SDMX 3.0 facade in front of upstream SDMX registries (IMF, BIS, â€Ļ). | #### MCP Endpoint @@ -147,11 +147,58 @@ invoke channel tools inside their own reasoning loops. | Service | Purpose | |---------------------------|---------------------------------------------| -| **PostgreSQL + pgvector** | Configuration and metadata storage | -| **SDMX Providers** | Dataset sources (IMF, World Bank, Eurostat) | -| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) | -| **AI DIAL** | Content storage (files, archives) | -| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) | +| **PostgreSQL + pgvector** | Configuration and metadata storage | +| **StatGPT SDMX Proxy** | Unified SDMX 3.0 facade in front of upstream SDMX registries (IMF, BIS, â€Ļ). Admin Backend reads metadata for dataset onboarding through the proxy. | +| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) | +| **AI DIAL** | Content storage (files, archives) | +| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) | + +### 🔁 StatGPT SDMX Proxy + +#### Technology Stack + +**Type**: Standalone Spring Boot service ([repository](https://github.com/epam/statgpt-sdmx-proxy)) + +**Core Frameworks**: + +| Framework | Purpose | +|-------------------------------------------------------------------|--------------------------------------------------| +| [Spring Boot 4.0](https://spring.io/projects/spring-boot) | Application framework | +| [OpenFeign](https://github.com/OpenFeign/feign) | Declarative HTTP clients for upstream registries | +| [Resilience4j](https://resilience4j.readme.io/) | Circuit breaker, retry, rate limiting | +| [Spring Data Redis](https://spring.io/projects/spring-data-redis) | Distributed caching backend | +| [OpenTelemetry](https://opentelemetry.io/) | Observability (traces, metrics, logs) | + +#### Overview + +**Primary Function**: Expose a single, unified +[SDMX 3.0 REST API](https://github.com/sdmx-twg/sdmx-rest/tree/master/doc) in front of multiple upstream SDMX +registries (IMF, BIS, â€Ļ) so that StatGPT components are decoupled from per-registry version, format, auth, and +quirk differences. + +**Key Responsibilities**: + +- Protocol translation between SDMX 2.1 and SDMX 3.0 +- Format conversion across SDMX-JSON, SDMX-ML (XML), and SDMX-CSV +- Agency-based routing (including sub-agency wildcarding and synthetic AgencyScheme discovery) +- Caching, circuit breaking, retries, rate limiting, and per-registry response patching +- Configuration-driven onboarding of new registries (no code change required) + +#### Authentication & Authorization + +**Outbound** (to upstream SDMX registries): currently the proxy does not send credentials to upstream registries. + +**Inbound**: the proxy itself is not directly exposed to the internet. External traffic reaches it through DIAL +Core, which is responsible for authorizing requests. StatGPT backends call the proxy over the internal cluster +network. + +#### Dependencies + +| Service | Purpose | +|----------------------|--------------------------------------------------------------------------------------------------------| +| **SDMX Registries** | Upstream statistical data sources (IMF, BIS, and any registry added via configuration) | +| **Redis** (optional) | Distributed cache (`CACHE_MODE=REDIS`). Falls back to in-memory Caffeine when unavailable. | +| **Config Server** (optional) | `sdmx-proxy-config-server` module — runtime-managed registry configuration when the bundled defaults aren't enough. | ### đŸ•šī¸ StatGPT Admin Frontend @@ -239,7 +286,7 @@ invoke channel tools inside their own reasoning loops. | Service | Purpose | |--------------------------|-------------------------------------------------------| -| **DIAL Platform** | Authentication, rate limits, file storage, API access | -| **StatGPT Chat Backend** | Core chat and data query services | -| **Identity Provider** | User authentication and SSO | -| **SDMX Providers** | Direct metadata and data queries | +| **DIAL Platform** | Authentication, rate limits, file storage, API access | +| **StatGPT Chat Backend** | Core chat and data query services | +| **Identity Provider** | User authentication and SSO | +| **StatGPT SDMX Proxy** | Metadata and data queries from the Advanced View editor — Portal does not call upstream SDMX registries directly. |