diff --git a/architecture/README.md b/architecture/README.md index ee715de..bc5f31f 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -61,6 +61,7 @@ Stateless services design enables horizontal scaling to handle varying loads eff - [StatGPT Backend](https://github.com/epam/statgpt-backend) - Core services implementation - [StatGPT Admin Frontend](https://github.com/epam/statgpt-admin-frontend) - Administrative interface - [StatGPT Portal Frontend](https://github.com/epam/statgpt-portal-frontend) - User interface library +- [StatGPT SDMX Proxy](https://github.com/epam/statgpt-sdmx-proxy) - Unified SDMX 3.0 facade in front of upstream registries - [StatGPT Helm](https://github.com/epam/statgpt-helm) - Kubernetes deployment charts ### External Documentation diff --git a/architecture/content/architecture_detailed.svg b/architecture/content/architecture_detailed.svg index df57389..790ed3c 100644 --- a/architecture/content/architecture_detailed.svg +++ b/architecture/content/architecture_detailed.svg @@ -1,4 +1,4 @@ -Common Services (**)Container Orchestration (Kubernetes, OCP, etc.)AzureAI DIALStatGPT Icon-identity-230 BusinessUserHTTPHTTPStatGPT App BackendTCPHTTPHTTPStatGPT Admin BackendPostgreSQL (*)Elasticsearch (*)Request flowTCPHTTPHTTPDIAL CoreHTTPDIAL OpenAI AdapterHTTPDIAL Realtime AnalyticsHTTPReadsLogsHTTPVectorInfluxDBRedis (*)End user Communication Channels DIAL Chat StatGPT PortalMCP ClientTCPHTTPDIAL Admin BackendAdmin Communication ChannelsGrafana DIALAdmin FE StatGPTAdmin FEHTTPHTTPHTTPHTTPAzure OpenAIBlob Storage Icon-identity-230 AdminUserLogging Monitoring(OTEL)External ServicesSecret managementSSO(*) - cloud-managed versions of services can be used(**) - Common Services can be located either in Container orchestration or in different CloudsSDMX API (HTTP)SDMX Registries(External)SDMX API (HTTP)StatGPT App MCPHTTPHTTP +Common Services (**)Container Orchestration (Kubernetes, OCP, etc.)AzureAI DIALStatGPT Icon-identity-230 BusinessUserHTTPHTTPStatGPT App BackendTCPHTTPHTTPStatGPT Admin BackendPostgreSQL (*)Elasticsearch (*)Request flowTCPHTTPHTTPDIAL CoreHTTPDIAL OpenAI AdapterHTTPDIAL Realtime AnalyticsHTTPReadsLogsHTTPVectorInfluxDBRedis (*)End user Communication Channels DIAL Chat StatGPT PortalMCP ClientTCPHTTPDIAL Admin BackendAdmin Communication ChannelsGrafana DIALAdmin FE StatGPTAdmin FEHTTPHTTPHTTPHTTPAzure OpenAIBlob Storage Icon-identity-230 AdminUserLogging Monitoring(OTEL)External ServicesSecret managementSSO(*) - cloud-managed versions of services can be used(**) - Common Services can be located either in Container orchestration or in different CloudsSDMX API (HTTP)StatGPT SDMX ProxySDMX API (HTTP)StatGPT App MCPHTTPHTTPSDMX RegistriesHTTP diff --git a/architecture/content/diagrams.drawio b/architecture/content/diagrams.drawio index 26ee1bf..f99ecd0 100644 --- a/architecture/content/diagrams.drawio +++ b/architecture/content/diagrams.drawio @@ -134,18 +134,18 @@ - + - + - + @@ -154,7 +154,7 @@ - + @@ -326,19 +326,19 @@ - + - + - + - + - + @@ -353,7 +353,7 @@ - + @@ -364,9 +364,7 @@ - - - + @@ -381,7 +379,7 @@ - + @@ -390,7 +388,7 @@ - + @@ -401,27 +399,22 @@ - - - + + - - + - - + + - - + + - - + - - @@ -448,6 +441,25 @@ + + + + + + + + + + + + + + + + + + + diff --git a/architecture/services.md b/architecture/services.md index f1c9a17..b844393 100644 --- a/architecture/services.md +++ b/architecture/services.md @@ -88,13 +88,13 @@ with [DIAL API Specification](https://dialx.ai/dial_api#operation/sendChatComple #### Dependencies -| Service | Purpose | -|---------------------------|------------------------------------------------------| -| **AI DIAL** | Platform services and model access | -| **PostgreSQL + pgvector** | Data storage and vector operations | -| **ElasticSearch** | Search and indexing | -| **External AI Models** | LLM capabilities (e.g., Azure OpenAI) | -| **SDMX Providers** | Statistical data sources (IMF, World Bank, Eurostat) | +| Service | Purpose | +|---------------------------|-----------------------------------------------------------------------------| +| **AI DIAL** | Platform services and model access | +| **PostgreSQL + pgvector** | Data storage and vector operations | +| **ElasticSearch** | Search and indexing | +| **External AI Models** | LLM capabilities (e.g., Azure OpenAI) | +| **StatGPT SDMX Proxy** | Unified SDMX 3.0 facade in front of upstream SDMX registries (IMF, BIS, âĻ). | #### MCP Endpoint @@ -147,11 +147,58 @@ invoke channel tools inside their own reasoning loops. | Service | Purpose | |---------------------------|---------------------------------------------| -| **PostgreSQL + pgvector** | Configuration and metadata storage | -| **SDMX Providers** | Dataset sources (IMF, World Bank, Eurostat) | -| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) | -| **AI DIAL** | Content storage (files, archives) | -| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) | +| **PostgreSQL + pgvector** | Configuration and metadata storage | +| **StatGPT SDMX Proxy** | Unified SDMX 3.0 facade in front of upstream SDMX registries (IMF, BIS, âĻ). Admin Backend reads metadata for dataset onboarding through the proxy. | +| **Identity Provider** | Authentication (Azure Entra ID, Keycloak) | +| **AI DIAL** | Content storage (files, archives) | +| **External AI Models** | Dataset indexing (e.g., Azure OpenAI) | + +### đ StatGPT SDMX Proxy + +#### Technology Stack + +**Type**: Standalone Spring Boot service ([repository](https://github.com/epam/statgpt-sdmx-proxy)) + +**Core Frameworks**: + +| Framework | Purpose | +|-------------------------------------------------------------------|--------------------------------------------------| +| [Spring Boot 4.0](https://spring.io/projects/spring-boot) | Application framework | +| [OpenFeign](https://github.com/OpenFeign/feign) | Declarative HTTP clients for upstream registries | +| [Resilience4j](https://resilience4j.readme.io/) | Circuit breaker, retry, rate limiting | +| [Spring Data Redis](https://spring.io/projects/spring-data-redis) | Distributed caching backend | +| [OpenTelemetry](https://opentelemetry.io/) | Observability (traces, metrics, logs) | + +#### Overview + +**Primary Function**: Expose a single, unified +[SDMX 3.0 REST API](https://github.com/sdmx-twg/sdmx-rest/tree/master/doc) in front of multiple upstream SDMX +registries (IMF, BIS, âĻ) so that StatGPT components are decoupled from per-registry version, format, auth, and +quirk differences. + +**Key Responsibilities**: + +- Protocol translation between SDMX 2.1 and SDMX 3.0 +- Format conversion across SDMX-JSON, SDMX-ML (XML), and SDMX-CSV +- Agency-based routing (including sub-agency wildcarding and synthetic AgencyScheme discovery) +- Caching, circuit breaking, retries, rate limiting, and per-registry response patching +- Configuration-driven onboarding of new registries (no code change required) + +#### Authentication & Authorization + +**Outbound** (to upstream SDMX registries): currently the proxy does not send credentials to upstream registries. + +**Inbound**: the proxy itself is not directly exposed to the internet. External traffic reaches it through DIAL +Core, which is responsible for authorizing requests. StatGPT backends call the proxy over the internal cluster +network. + +#### Dependencies + +| Service | Purpose | +|----------------------|--------------------------------------------------------------------------------------------------------| +| **SDMX Registries** | Upstream statistical data sources (IMF, BIS, and any registry added via configuration) | +| **Redis** (optional) | Distributed cache (`CACHE_MODE=REDIS`). Falls back to in-memory Caffeine when unavailable. | +| **Config Server** (optional) | `sdmx-proxy-config-server` module â runtime-managed registry configuration when the bundled defaults aren't enough. | ### đšī¸ StatGPT Admin Frontend @@ -239,7 +286,7 @@ invoke channel tools inside their own reasoning loops. | Service | Purpose | |--------------------------|-------------------------------------------------------| -| **DIAL Platform** | Authentication, rate limits, file storage, API access | -| **StatGPT Chat Backend** | Core chat and data query services | -| **Identity Provider** | User authentication and SSO | -| **SDMX Providers** | Direct metadata and data queries | +| **DIAL Platform** | Authentication, rate limits, file storage, API access | +| **StatGPT Chat Backend** | Core chat and data query services | +| **Identity Provider** | User authentication and SSO | +| **StatGPT SDMX Proxy** | Metadata and data queries from the Advanced View editor â Portal does not call upstream SDMX registries directly. |