diff --git a/guides/admin-guide.md b/guides/admin-guide.md index 53a1df5..ac11717 100644 --- a/guides/admin-guide.md +++ b/guides/admin-guide.md @@ -1,483 +1,629 @@ # Administrator Guide -This document covers the StatGPT Admin Frontend, its features, and StatGPT configurations. +This document covers the StatGPT Admin Frontend — its features, screens, and the StatGPT configurations you manage +through it: **data sources**, **datasets**, and **channels**. + +> Screenshots in this guide are taken from the built-in **StatGPT Sample** channel and its IMF sample resources. +> Numbered badges in each screenshot correspond to the numbered steps in the surrounding text. + +## Table of Contents + +- [Administrator App](#administrator-app) +- [Concepts](#concepts) +- [Data Sources](#data-sources) + - [Browsing data sources](#browsing-data-sources) + - [Adding a data source](#adding-a-data-source) + - [Editing a data source](#editing-a-data-source) +- [Datasets](#datasets) + - [Browsing datasets](#browsing-datasets) + - [Adding a dataset](#adding-a-dataset) + - [Editing a dataset](#editing-a-dataset) +- [Channels](#channels) + - [Browsing channels](#browsing-channels) + - [Configuring a channel](#configuring-a-channel) + - [The channel datasets page](#the-channel-datasets-page) + - [Per-dataset actions](#per-dataset-actions) + - [Versions](#versions) + - [Auto update jobs](#auto-update-jobs) + - [Indexing operations](#indexing-operations) + - [Glossary of Terms](#glossary-of-terms) + - [Import / Export & Jobs](#import--export--jobs) +- [Audit Logs](#audit-logs) +- [Related resources](#related-resources) ## Administrator App StatGPT Admin is an application that focuses solely on managing StatGPT configurations, including data sources, -datasets, -and channels. The application does not cover user management, authentication, authorization, monitoring, or cost -control, -which are handled by the DIAL platform. More information on this topic can be found -in the [architecture overview article](../architecture/overview.md#-core-requirements). +datasets, and channels. The application does not cover user management, authentication, authorization, monitoring, or +cost control, which are handled by the DIAL platform. More information on this topic can be found in the +[architecture overview article](../architecture/overview.md#-core-requirements). + +The left-hand navigation gives access to the main sections: + +- **Data Sources** — SDMX endpoints StatGPT can read from. +- **Datasets** — individual dataflows onboarded from those sources, with StatGPT-specific configuration. +- **Documents** — publication/RAG content (outside the scope of this guide). +- **Channels** — end-user StatGPT applications, each with its own datasets, glossary, index, and agent configuration. +- **Audit Logs** — a chronological record of configuration changes. ## Concepts -StatGPT uses several key concepts that will be referenced throughout this document: +StatGPT uses several key concepts that are referenced throughout this document: + +1. **Data Source** — A source of data that can be queried using the SDMX protocol. Examples include IMF, Eurostat, + World Bank, etc. Each data source has a **connector** that determines how StatGPT talks to it + (`SDMX21`, `QH_SDMX21`, or `PROXY_SDMX30`). +2. **Dataset** — A direct representation of an SDMX dataflow in a data source, plus StatGPT-specific configuration + (dimension roles, default queries, indexing options, citation, etc.). +3. **Channel** — A representation of the StatGPT application for end users. Each channel has its own configuration, + datasets, and index. Having multiple channels allows experimentation with different LLM / index / dataset + configurations. Each channel is exposed as a separate DIAL Application. +4. **Index version** — Each dataset, *within a channel*, is indexed as an immutable **version**. A new version is + created whenever the dataset configuration changes, a manual reindex is triggered, or a channel is imported from a + zip. The channel always serves data from the latest **completed** version. +5. **Auto-update** — When enabled, StatGPT periodically checks the source for new data and, if something changed, + builds a fresh index version automatically (see [Auto update jobs](#auto-update-jobs)). + +> **Tip — going deeper on dataset onboarding.** This guide focuses on the Admin UI. For a field-by-field methodology +> (assessing datasets, classifying dimensions, configuring indicators, indexing, and validation), follow the +> [Admin Learning Course](../learning/administration/README.md). + +--- + +## Data Sources + +### Browsing data sources + +The **Data Sources** screen lists every configured source. Use the column filters to narrow the list. -1. **Data Source** - A source of data that can be queried using the SDMX protocol. Examples of data sources include IMF, - Eurostat, - World Bank, etc. -2. **Dataset** - A direct representation of an SDMX dataflow in a datasource, with the addition of StatGPT-specific - configurations. -3. **Channel** - A representation of the StatGPT application for end users. Each channel has its own - configuration, datasets, and index. Having multiple channels allows experimentation with different LLM/index/dataset - configurations. - Each channel is represented as a separate DIAL Application. +![Data Sources list](./content/admin-guide/ds-list.png) -## Add Data Source +1. **+ Add** — start the *Add Data Source* wizard. +2. **Column filters** — filter by Name, Description, or Connection Type. +3. **Row menu (⋯)** — **Configure** or **Delete** an existing source. -To add a new data source: +### Adding a data source -1. Navigate to the "Data Sources" tab and click the "Add" button - ![DataSource List](./content/admin-guide/datasource-list.png) -2. Fill in the required fields in the form, including selecting the datasource type: - ![Add Data Source](./content/admin-guide/datasource-type.png) -3. Fill the configuration fields for the selected datasource type and save the datasource: - ![Add Data Source Config](./content/admin-guide/datasource-config.png) +Adding a source is a two-step wizard. -Configuration for an IMF SDMX 2.1 datasource with English locale is as follows: +**Step 1 — Properties.** + +![Add Data Source — Properties](./content/admin-guide/ds-add-properties.png) + +1. **Name** — a unique identifier for the source (e.g. `IMF_SDMX21`). An optional description can be added below. +2. **Connector** — choose how StatGPT connects to the registry: + - `SDMX21` — a standard SDMX 2.1 REST endpoint. + - `QH_SDMX21` — an SDMX 2.1 endpoint served through QuantHub (used by the IMF sample). + - `PROXY_SDMX30` — an SDMX 3.0 source accessed through the StatGPT SDMX Proxy. +3. **Next** — continue to the configuration editor. + +**Step 2 — Configuration.** Provide the connector configuration as YAML, then click **Finish**. + +![Add Data Source — Configuration](./content/admin-guide/ds-add-config.png) + +1. **Configuration editor** — the YAML configuration for the selected connector. +2. **Finish** — validate and create the data source. + +A configuration for the IMF SDMX 2.1 source (connector `QH_SDMX21`) looks like this: ```yaml -locale: en # Locale for the datasource, e.g., 'en' for English. Used for indexing and querying. +apiKey: "" # subscription/API key, if the registry requires one +locale: en # locale used for indexing and querying +authConfig: null +rateLimits: # optional concurrency limits for outgoing requests + structureRequestsConcurrency: null + availabilityAndDataRequestsConcurrency: null sdmxConfig: - id: IMF_SDMX21 # Unique identifier for the datasource - url: https://api.imf.org/external/sdmx/2.1 # IMF SDMX 2.1 REST API endpoint - name: Public IMF SDMX 21 Registry # Human-readable name for the datasource - headers: # HTTP headers for specific structures and data requests + id: IMF_SDMX21 # unique data source id + url: https://api.imf.org/external/sdmx/2.1 # SDMX 2.1 REST endpoint + name: Public IMF SDMX 21 Registry + headers: # Accept header to send per SDMX resource type data: accept: Text/JSON codelist: accept: application/xml dataflow: accept: application/xml - agencyscheme: - accept: application/xml - conceptscheme: - accept: application/xml datastructure: accept: application/xml - categoryscheme: - accept: application/xml - provisionagreement: - accept: application/xml - availableconstraint: - accept: application/xml - hierarchicalcodelist: - accept: application/xml - supports: # Supported SDMX features + # ...also: agencyscheme, conceptscheme, categoryscheme, provisionagreement, + # availableconstraint, hierarchicalcodelist (all application/xml) + supports: # which SDMX resource types the registry supports data: true - preview: true codelist: true dataflow: true - agencyscheme: true - conceptscheme: true datastructure: true - categoryscheme: true - provisionagreement: true - availableconstraint: true - hierarchicalcodelist: true - data_content_type: JSON -authEnabled: false # Whether authentication is required to access the datasource -description: "" # Optional description of the datasource -annotationsUrl: https://api.imf.org/external/sdmx/3.0 # Optional URL for annotations + # ...(remaining resource types set to true) + versions: + - "2.1" + dataContentType: JSON +authEnabled: false # whether the source requires authentication +sdmx1Source: IMF_DATA # sdmx1 library source profile +apiKeyHeader: Ocp-Apim-Subscription-Key +attributesUrl: https://api.imf.org/external/sdmx/3.0 # SDMX 3.0 endpoint for attributes +annotationsUrl: https://api.imf.org/external/sdmx/3.0 # SDMX 3.0 endpoint for annotations +dataExplorerUrl: https://data.imf.org/en/Data-Explorer # link surfaced to end users +providerDiscovery: dataflows # how providers/agencies are discovered ``` -Once the datasource is created, you can edit its details by clicking the "Edit" button in the data sources section. -This is typically required when datasource characteristics or connection parameters have changed. +### Editing a data source + +To change connection parameters later, open the row **⋯** menu and choose **Configure**. This reopens the YAML editor. + +![Configure Data Source](./content/admin-guide/ds-configure.png) -![Edit Data Source](./content/admin-guide/datasource-edit.png) +1. **Configuration editor** — edit the YAML in place. +2. **Save** — apply the changes. + +> See [Module 05 — Data Sources & Channel Configuration](../learning/administration/05-data-sources-and-channels.md) +> of the learning course for connector details and source-discovery options, and +> [SDMX Compatibility & Requirements](../architecture/sdmx-compatibility.md) for the technical requirements a source +> must meet. + +--- ## Datasets -To add a new dataset from an already configured datasource: +A dataset is an SDMX dataflow plus the StatGPT configuration that tells the agent how to query and index it. + +### Browsing datasets + +The **Datasets** screen lists every onboarded dataset and a single global availability **Status** (`Online`). +Per-channel indexing state lives on the [channel datasets page](#the-channel-datasets-page), not here. + +![Datasets list](./content/admin-guide/datasets-list.png) + +1. **+ Add** — start the *Add Dataset* wizard. +2. **Column filters** — filter by Name, Description, or Data Source. +3. **Row menu (⋯)** — **Edit dataset** or **Delete** it. + +![Dataset row menu](./content/admin-guide/datasets-row-menu.png) -1. Navigate to the "Datasets" tab and click on the "Add" button -2. Select the datasource from the table and click "Next": - ![Select Datasource](./content/admin-guide/dataset-datasource.png) -3. You will see a table of datasets available in the selected datasource. Select the desired dataset from the - table and click "Next": - ![Select Dataset](./content/admin-guide/dataset-select.png) -4. Fill in the required fields in the form and save the dataset: - ![Add Dataset](./content/admin-guide/dataset-config.png) +### Adding a dataset -Configuration for the IMF WEO dataset is as follows: +The *Add Dataset* wizard has four steps: **Data Source → Provider → Dataset → Configuration**. + +**Step 1 — Select the data source.** + +![Add Dataset — select source](./content/admin-guide/dataset-add-source.png) + +1. **Filter** the list to find the source. +2. **Select** the source (e.g. `IMF_SDMX21`). +3. **Next.** + +**Step 2 — Select the provider (agency).** + +![Add Dataset — select provider](./content/admin-guide/dataset-add-provider.png) + +1. **Select** the agency that publishes the dataflow (e.g. `IMF.RES` for the World Economic Outlook). +2. **Next.** + +**Step 3 — Select the dataflow.** + +![Add Dataset — select dataflow](./content/admin-guide/dataset-add-dataflow.png) + +1. **Select** the dataflow to onboard (e.g. `IMF.RES:WEO`). +2. **Next.** + +**Step 4 — Configuration.** StatGPT pre-fills a configuration template detected from the dataflow's structure. Review +and adjust it, then click **Finish**. + +![Add Dataset — configuration](./content/admin-guide/dataset-add-config.png) + +1. **`dimensions` map** — classify each dimension (see below). This is the most important part of the configuration. +2. **Finish** — create the dataset. + +A configuration for the IMF WEO dataset looks like this: ```yaml -urn: IMF.RES:WEO(6.0.0) # SDMX urn of the dataset -citation: # citation for the dataset - url: https://data.imf.org/en/datasets/IMF.RES:WEO # URL to the dataset web page - provider: IMF.RES # data provider - description: > # detailed description of the dataset +urn: # SDMX URN of the source dataflow + version: latest + agencyId: IMF.RES + resourceId: WEO +indexer: + indicator: + unpack: true # WEO ships packed indicators -> unpack them for search + annotations: null + superPrimary: false + useCodeListDescription: true + description: > # dataset summary added to the agent context The World Economic Outlook (WEO) database contains selected macroeconomic data series from the statistical appendix of the World Economic Outlook - report, which presents the IMF staff's analysis and projections of economic - developments at the global level, in major country groups and in many - individual countries. The WEO is released in April and September/October - each year. Use this database to find data on national accounts, gross - domestic product (GDP), inflation, unemployment rates, balance of payments, - fiscal indicators, trade for countries and country groups (aggregates), and - commodity prices whose data are reported by the IMF. Data are available from - 1980 to the present, and projections are given for the next two years. - Additionally, medium-term projections are available for selected indicators. - For some countries, data are incomplete or unavailable for certain years. -isOfficial: false # whether the dataset is from an official source (usually country-level sources are considered official and prioritized in answers for specific country queries) -pinnedColumns: # list of columns to pin in the dataset table + report... (abbreviated) +citation: # provenance shown to end users + url: https://data.imf.org/en/datasets/IMF.RES:WEO + provider: IMF.RES + description: > + The World Economic Outlook (WEO) database... (abbreviated) +updatedAt: # where to read the dataset's "last updated" date from + - field: lastUpdatedAt + source: annotation + formats: null + - field: last_updated + source: citation + formats: null +dimensions: # per-dimension roles and behaviour + COUNTRY: + dimensionType: NON_INDICATOR + subtype: REGION # marks this as the country/region dimension + alias: Country/Reference area + isRequired: false + allValues: # wildcard value used for "all countries" queries + id: ALL_COUNTRIES + name: All countries - must be selected when query explicitly asks for all countries + description: Special value to query all countries + defaultQueries: null + FREQUENCY: + dimensionType: NON_INDICATOR + subtype: FREQUENCY + isRequired: false + alias: null + allValues: null + defaultQueries: null + INDICATOR: + dimensionType: INDICATOR # the indicator dimension(s) the agent searches over + isRequired: true # a query cannot run without an indicator selected + alias: null + allValues: null + defaultQueries: null + TIME_PERIOD: + dimensionType: TIME_PERIOD + isRequired: false + defaultQueries: # default time window when the user does not specify one + - values: ["-5y", "+2y"] + operator: between +isOfficial: false # official (national) sources are prioritised for country queries +pinnedColumns: # columns pinned in the result table - FREQUENCY_Name - COUNTRY_Name - INDICATOR_Name -useTitleFromSrc: true # whether to use the title from the source Dataflow -countryDimension: COUNTRY # dimension representing the country or region -includeAttributes: # list of attributes to include in the AI Agent context +useTitleFromSrc: false # use the configured title instead of the source dataflow title +includeAttributes: # SDMX attributes added to the agent context - SCALE - UNIT - PUBLISHER - SOURCE -indicatorDimensions: # list of the dimensions representing the dataset indicators - - INDICATOR -updatedAtAnnotation: lastUpdatedAt # SDMX annotation to use for the "last updated at" information -countryDimensionAlias: Country/Reference area # alias for the country dimension (dimensions can be named differently in different datasets, therefore alias helps to unify the naming for indexing and querying) -dimensionDefaultQueries: - TIME_PERIOD: - - values: - - "2020" - - "2025" - operator: between -indicatorDimensionsRequiredForQuery: # indicator dimension that required to be filled for querying the dataset - - INDICATOR - -indexer: # indexer configuration for the dataset - indicator: - unpack: true # whether to unpack the indicator dimensions (e.g. in WEO packed indicators are used, therefore unpack=true) ``` -Once the dataset is created, you can edit its details by clicking the "Edit" button in the datasets section. This is -typically required when the dataset structure has changed or indexing parameters need to be updated. +**Understanding the `dimensions` map.** Every dimension is given a `dimensionType`: + +- `INDICATOR` — what is measured (GDP, inflation, …). Mark the indicator(s) the agent searches over; set + `isRequired: true` for dimensions a query cannot omit. +- `NON_INDICATOR` — context dimensions such as country, counterpart, currency, or frequency. Use `subtype` + (`REGION`, `FREQUENCY`, …), `alias` (a stable name used for indexing across datasets), and `allValues` (a wildcard + member, e.g. "all countries"). +- `TIME_PERIOD` — the time dimension, optionally with `defaultQueries` (relative offsets such as `-5y`/`+2y`). + +> Dimension classification, packed vs. unpacked indicators, and the full field reference are covered in the learning +> course: [Module 03a — Dimension Types & Named Entities](../learning/administration/03a-dimension-types.md), +> [Module 03b — Indicator Configuration](../learning/administration/03b-indicator-configuration.md), and +> [Module 04 — Configuring a Dataset](../learning/administration/04-dataset-configuration.md). + +### Editing a dataset + +Choose **Edit dataset** from the row menu to reopen the configuration. This is typically required when the dataflow +structure changes or indexing parameters need tuning. -![Edit Dataset](./content/admin-guide/dataset-edit.png) +![Edit Dataset](./content/admin-guide/dataset-edit-config.png) + +1. **Configuration editor** — the dataset YAML. +2. **Save** — saving a configuration change creates a new index version in every channel that uses the dataset. + +--- ## Channels -To add a new channel: +A channel is a self-contained StatGPT application: its own agent configuration, set of datasets, glossary, and index. + +### Browsing channels + +![Channels list](./content/admin-guide/channels-list.png) + +1. **Import** — create a channel from an exported zip archive (see [Import / Export & Jobs](#import--export--jobs)). +2. **+ Add** — create a new, empty channel. +3. **Row menu (⋯)** — per-channel actions: + +![Channel context menu](./content/admin-guide/channel-menu.png) + +The channel **⋯** menu offers **Configure**, **Glossary**, **Jobs**, **Delete**, and **Export**. -1. Navigate to the "Channels" tab and click on the "Add" button -2. Fill in the required fields in the form and save the channel: - ![Add Channel](./content/admin-guide/channel-create.png) -3. Add channel configurations as described in the next section: - ![Channel Configurations](./content/admin-guide/channel-config.png) -4. Once the channel is created you will be redirected to the channel details page. Here you can add datasets to the - channel: - ![Add Dataset to Channel](./content/admin-guide/channel-add-dataset.png) +### Configuring a channel -Configuration for the Global Data channel is as follows: +Choose **Configure** to edit the channel's YAML configuration (agent behaviour, conversation starters, tools, etc.). + +![Channel configuration](./content/admin-guide/channel-configure.png) + +1. **Configuration editor** — the channel YAML. +2. **Save** — apply the configuration. + +A configuration for the **StatGPT Sample** channel looks like this (long tool descriptions are abbreviated): ```yaml -conversationStarters: # predefined conversation starters to show in the chat interface - introText: What would you like to learn about? +locale: en +conversationStarters: # prompts shown on the empty chat screen + introText: Search {indicators_total} official indicators from the IMF + title: What would you like to learn about? + inputPlaceholder: Ask about GDP, inflation or any other IMF indicator buttons: - title: Check available data text: What datasets are available? - - title: Annual US investment in Mexico - text: What was the annual total investment of the US economy in the Mexican - economy? - - title: What is inflation? - text: What is inflation and how it's used in economy and statistics? -namedEntityTypes: # list of named entity types to recognize in user queries. Used for non-indicator dimensions recognition. Set of named entity types depends on the datasets composition in the channel. + - title: WEO US GDP projection + text: What is the IMF WEO projection for US GDP for next 2 years? + - title: What is CPI? + text: What is CPI, how is it related to inflation, and how is it used in economy and statistics? +onboarding: null +namedEntityTypes: # non-indicator dimensions recognised in user queries - Time frequency - Counterpart area/country - Currency/Unit of measure -countryNamedEntityType: Country/Reference area # named entity type representing countries/regions -supremeAgent: # main agent configuration - name: StatGPT # agent name to be used in the agent context - domain: Statistics, economics and SDMX. # agent domain to be used in the agent context - terminologyDomain: economics, statistics and SDMX # domain for the terminology to be used in the agent context - languageInstructions: # additional language instructions for the agent +countryNamedEntityType: Country/Reference area +supremeAgent: # main orchestrating agent + name: StatGPT + domain: Statistics, economics and SDMX. + terminologyDomain: economics, statistics and SDMX + languageInstructions: - Use more formal/business tone, but be friendly, polite and helpful. - Avoid using exclamation marks, emojis, or slang. - maxAgentIterations: 5 # maximum number of agent iterations (tool calls) per user query + - When asked about list of available datasets, use markdown table to show them. llmModelConfig: - apiVersion: 2024-08-01-preview # LLM API version - deployment: gpt-4.1-2025-04-14 # LLM deployment name - temperature: 0 # LLM temperature - seed: 820288 # LLM seed for reproducibility -outOfScope: # configurations of the out-of-scope filter of the agent - domain: Statistics, economics and SDMX. # domain to be used in the out-of-scope filter context - customInstructions: null # additional custom instructions for the out-of-scope filter -tokenUsage: # token usage tracking configuration - debugOnly: false # whether to track token usage only in debug mode - stageName: Token Usage # name of the DIAL stage to log token usage -availableDatasets: # configurations for the Available_Datasets tool - type: AVAILABLE_DATASETS - name: Available_Datasets - description: >- # Extensive description of the tool and its usage instructions, will be provided to the agent's context - Provides a list of all available datasets onboarded to the `Query_Data` tool - with metadata and some details about them. Details include the name and - description of the dataset, the provider (agency), and the last update date. + deployment: gpt-4.1-2025-04-14 +outOfScope: # guardrail that declines off-topic questions + domain: Statistics, economics and SDMX. + useGeneralTopicsBlacklist: true + llmModelConfig: + deployment: gpt-4.1-2025-04-14 +tokenUsage: # token-usage reporting stage + debugOnly: false + stageName: Token Usage - This tool does not accept any arguments. +# ---- Tools available to the agent ---- - For questions about the availability of indicators you should refer to the - `Query_Data` tool. +availableDatasets: # lists onboarded datasets with metadata + type: AVAILABLE_DATASETS + name: Available_Datasets + description: >- + Provides a list of all available datasets onboarded to the `Query_Data` tool... (abbreviated) details: - fakeCall: # fake tool call to put the tool call along with tool response in the agent's context by default + fakeCall: toolCallId: call_EBJJeaOMKeCzm8h378ubURQN - args: "{}" - stagesConfig: - debugOnly: true - rules: [ ] version: full -dataQuery: # Data Query tool configuration + includeIndicatorCount: true + +datasetStructure: # describes the structure (dimensions/attributes) of one dataset + type: DATASET_STRUCTURE + name: Dataset_Structure + description: >- + Provides the structure of a specific dataset, including its dimensions, attributes, + their types and sample values... (abbreviated) + details: + stagesConfig: + toolCallName: "Looking into structure of dataset '{dataset_id}'" + toolResultName: "Structure of dataset '{dataset_id}'" + +dataQuery: # builds and executes SDMX queries from natural language type: DATA_QUERY name: Query_Data description: >- - Executing sdmx query on available datasets. Some datasets include forecasts - for next years. + Executing sdmx query on available datasets. Some datasets include forecasts for next years... (abbreviated) + details: + allowAutoUpdate: true # allow scheduled auto-update jobs to refresh the index + indexerVersion: hybrid + indicatorSelectionVersion: hybrid + hybridSearchConfig: # hybrid (keyword + semantic + LLM) search tuning + namedEntitiesToRemove: + - Country/Reference area + - Counterpart area/country + prompts: + relevancyPrompts: { systemMessage: "...", userMessage: "..." } # (abbreviated) + llmModels: # per-stage LLM configuration (all gpt-4.1 here) + datasetsSelectionModelConfig: { deployment: gpt-4.1-2025-04-14, temperature: 0.0 } + dimensionsSelectionModelConfig: { deployment: gpt-4.1-2025-04-14, temperature: 0.0 } + indicatorsSelectionModelConfig: { deployment: gpt-4.1-2025-04-14, temperature: 0.0 } + # ...incompleteQueries, groupExpander, namedEntities, queryNormalization, timePeriod + attachments: # artifacts returned alongside the answer + customTable: { enabledStr: "True", name: "Data: {dataset_source_id}" } + plotlyGrid: { enabledStr: "$env:{DIAL_SHOW_PLOTLY_GRID|False}", name: "Plotly Grid: {dataset_source_id}" } + csvFile: { enabledStr: "False" } + plotlyGraphs: { enabledStr: "False" } + jsonQuery: { enabledStr: "True", name: "Query (JSON): {dataset_source_id}" } + pythonCode: { enabledStr: "True", name: "Python Code: {dataset_source_id}" } + mergedPythonCode: { enabledStr: "True", name: "Python Code" } # single consolidated code block + stagesConfig: + debugOnly: true + toolCallName: "Searching for data: {query}" + toolResultName: "Data search result: {query}" + rules: # which stages are visible to end users + - { key: constructing_data_query, debugOnly: false } + - { key: extracting_named_entities, debugOnly: false } + - { key: executing_data_query, debugOnly: false } + - { key: normalizing_query, debugOnly: false } + - { key: selecting_indicators, debugOnly: false } + messages: + noData: No data was found for the provided query. Try to change the query. + # ...noDataForCountry, dataQueryExecutedAgentOnly, multipleDatasetsAgentOnly - Constructed query is used to fetch indicators from one of the datasets. +availableTerms: # lists glossary terms + type: AVAILABLE_TERMS + name: Available_Terms + description: >- + Use this tool to retrieve a comprehensive list of all terms currently available in the glossary... (abbreviated) + details: + fakeCall: + toolCallId: call_EBJJeaOMKeCzm8h378ubU003 - Instructions: +termDefinitions: # returns definitions for requested glossary terms + type: TERM_DEFINITIONS + name: Term_Definitions + description: >- + Use this tool to retrieve definitions for up to 10 requested terms that appear in the glossary... (abbreviated) + details: + stagesConfig: + toolCallName: "Searching in Glossary of Terms: {terms}" + toolResultName: "Glossary search result: {terms}" + limit: 10 +``` - * Don't try to expand country groups or regions, it's done by tool itself +> The agent design and the role of each tool are described in [architecture/agent.md](../architecture/agent.md) and +> [architecture/tools.md](../architecture/tools.md). For a configuration walkthrough, see +> [Module 05 — Data Sources & Channel Configuration](../learning/administration/05-data-sources-and-channels.md). - * Summarize but DON'T REPHRASE time filter: "from now to 2030" must remain - "from now to 2030" +### The channel datasets page - * Tool works best for single indicator query (e.g. GDP, inflation) +Double-click a channel row (or open it) to reach the **channel datasets page**. This is where datasets are attached to +the channel and where all indexing happens. - * Tool supports star-queries for countries, e.g. "Give GDP for all - countries" +![Channel datasets page](./content/admin-guide/channel-details.png) - * Tool may ask clarifications if query is unclear. If query is modified - accordingly, tool will provide - requested data. - * Good query example: "Please give me wage information of USA" +The toolbar provides channel-level operations: - * Bad query example: "What are the recent economy indicators for Baltic - countries?" - Reason: ambiguous query, specific indicators should be mentioned, e.g. GDP, unemployment rate +1. **Deduplicate statistics** — show how many duplicate embeddings exist across the channel's datasets. +2. **Export** — export the channel (configuration + index) as a zip. +3. **Recalculate all indexes ▾** — reindex every dataset in the channel. +4. **+ Add** — attach an existing dataset to this channel. - Keep in mind: tool works best when detailed and concise query is provided - details: - stagesConfig: # configurations of the different stages of the tool execution to be shown to user - debugOnly: true - rules: - - pattern: Constructing Data Queries - debugOnly: false - - pattern: Extracting Named Entities - debugOnly: false - - pattern: Executing Data Queries - debugOnly: false - - pattern: Normalizing Query - debugOnly: false - - pattern: Selecting Indicators - debugOnly: false - version: v2 # version of the tool - indexerVersion: hybrid # version of the indexer used for the tool - indicatorSelectionVersion: hybrid # version of the indicator selection algorithm used for the tool - llmModels: # Configurations of the LLM models used in different stages of the tool execution - datasetsSelectionModelConfig: - apiVersion: 2024-08-01-preview - deployment: gpt-4.1-2025-04-14 - temperature: 0 - seed: 820288 - dimensionsSelectionModelConfig: - apiVersion: 2024-08-01-preview - deployment: gpt-4.1-2025-04-14 - temperature: 0 - seed: 820288 - indicatorsSelectionModelConfig: - apiVersion: 2024-08-01-preview - deployment: gpt-4.1-2025-04-14 - temperature: 0 - seed: 820288 - incompleteQueriesModelConfig: - apiVersion: 2024-08-01-preview - deployment: gpt-4.1-2025-04-14 - temperature: 0 - seed: 820288 - groupExpanderModelConfig: - apiVersion: 2024-08-01-preview - deployment: gpt-4.1-2025-04-14 - temperature: 0 - seed: 820288 - namedEntitiesModelConfig: - apiVersion: 2024-08-01-preview - deployment: gpt-4.1-2025-04-14 - temperature: 0 - seed: 820288 - timePeriodModelConfig: - apiVersion: 2024-08-01-preview - deployment: gpt-4.1-2025-04-14 - temperature: 0 - seed: 820288 - queryNormalizationModelConfig: - apiVersion: 2024-08-01-preview - deployment: gpt-4.1-2025-04-14 - temperature: 0 - seed: 820288 - messages: # predefined messages to be used in different situations during the tool execution - noDataForCountry: No data was found for {country_details}. Try to change the query. - noData: No data was found for the provided query. Try to change the query. - dataQueryExecutedAgentOnly: >- - If the executed query is only remotely related to the user query, you - must mention that fact to the user, - - to not mislead them. It is recommended to search in other sources using - tools available. - - Result of the executed query is shown to the user in the table - attachment. - multipleDatasetsAgentOnly: >- - If the executed query is only remotely related to the user query, it is - recommended to mention that fact to - - the user, instead of suggesting user to choose one of the datasets. - Other tools might be used to search for - - the data. - attachments: # configurations of the attachments to be provided to user along with the tool response - customTable: - enabledStr: "True" - name: "Data: {dataset_source_id}" - plotlyGrid: - enabledStr: $env:{DIAL_SHOW_PLOTLY_GRID|False} - name: "Plotly Grid: {dataset_source_id}" - csvFile: - enabledStr: "True" - name: "Data (CSV): {dataset_source_id}.csv" - plotlyGraphs: - enabledStr: "True" - name: "Graph: {figure_title}" - jsonQuery: - enabledStr: "True" - name: "Query (JSON): {dataset_source_id}" - pythonCode: - enabledStr: "True" - name: "Python Code: {dataset_source_id}" -availableTerms: # configurations for the Available_Terms tool - type: AVAILABLE_TERMS - name: Available_Terms - description: >- # Extensive description of the tool and its usage instructions, will be provided to the agent's context - Use this tool to: +The table shows, per dataset: **Data Source**, the global **Dataset Status** (`Online`), the **Completed Version** +and **Completed At** (the version currently served), the **Latest Version** / **Latest Updated** / **Latest Status** +of the most recent indexing attempt, and **Last check** (the result of the most recent auto-update check). - * Retrieve a comprehensive list of all terms currently available in the - glossary. +> To attach a dataset, click **+ Add** and pick from the datasets already onboarded under **Datasets**. (A dataset +> must exist before it can be added to a channel.) - * Confirm whether a specific term exists in the glossary. +#### Per-dataset actions +Each dataset row has a **⋯** menu with its own set of actions: - Detailed Guidance: +![Per-dataset actions](./content/admin-guide/channel-dataset-menu.png) - * The list of available glossary terms provided by this tool is complete; - there are no additional terms beyond - what is returned. - * Whenever referring to or explaining to user any glossary terms you must - obtain the definitions of any listed - terms using the "Term_Definitions" tool. - details: - fakeCall: # fake tool call to put the tool call along with tool response in the agent's context by default - toolCallId: call_EBJJeaOMKeCzm8h378ubU003 - args: "{}" - includeDomain: false - includeSource: false -termDefinitions: # configurations for the Term_Definitions tool - type: TERM_DEFINITIONS - name: Term_Definitions - description: >- # Extensive description of the tool and its usage instructions, will be provided to the agent's context - Use this tool to: +- **Edit dataset** — edit the dataset configuration. +- **Auto update jobs** — view the scheduled source-check history (below). +- **Versions** — view the dataset's index versions in this channel (below). +- **Recalculate indexes** — rebuild this dataset's index for this channel. +- **Delete** — remove the dataset from this channel. - * Retrieve definitions for up to 10 requested terms that appear in the - glossary. +#### Versions - * Consult the "Available_Terms" tool if you are unsure which terms are in - the glossary. +The **Versions** page lists every index version built for the dataset in this channel. +![Dataset versions](./content/admin-guide/dataset-versions.png) - Detailed Guidance: +1. **Status** — `COMPLETED` versions are eligible to be served; failed builds are kept for diagnostics. +2. **Creation Reason** — why the version was created: + - *Applied dataset config change* — the dataset configuration was edited. + - *Manually initiated reindex* — an admin triggered **Recalculate indexes**. + - *Imported from zip* — the version arrived with a channel import. - * Confirm availability of terms using the "Available_Terms" tool first. - details: - stagesConfig: - toolCallName: Glossary search result - debugOnly: true - rules: [ ] - limit: 10 -``` +Other columns record the **Created At** / **Updated At** timestamps and any **Harmonization** / **Normalization** +errors encountered while indexing. + +#### Auto update jobs + +When `dataQuery.details.allowAutoUpdate` is enabled, StatGPT runs a scheduled job that checks the source for new data. +The **Auto update jobs** page is the history of those checks. + +![Auto update jobs](./content/admin-guide/dataset-auto-update-jobs.png) + +1. **Result** — `NO_CHANGES` means the source was unchanged and no new version was built; a change would trigger a new + index version automatically. +2. **Updated At** — the daily cadence of the checks. -Channel has the following context menu options: +Each row also records the **Base Version ID** that was checked, any **Created Version ID** (when a new version was +built), the job **Status**, and the URN that was inspected under **Details**. -![Channel Menu](./content/admin-guide/channel-menu.png) +#### Indexing operations -Once the channel is created, you can edit its details by clicking the "Edit" button in the channels section. This -is typically required when configurations need to be updated. +StatGPT relies on a hybrid index (keyword + semantic + LLM reasoning) over dataset metadata. You can rebuild indexes at +two levels. -![Edit Channel](./content/admin-guide/channel-edit.png) +**One dataset** — choose **Recalculate indexes** from the dataset's **⋯** menu. + +**All datasets in the channel** — use the **Recalculate all indexes ▾** button and pick a mode: + +![Recalculate all indexes](./content/admin-guide/channel-recalc-dropdown.png) + +1. **Recalculate all indexes** opens the mode menu. +2. Choose **Sequential recalculation** (one dataset at a time, lighter on resources) or **Parallel recalculation** + (faster, more resource-intensive). + +**Deduplicate statistics** reports duplicate embeddings across the channel's datasets — useful before or after large +reindexing operations. In the sample channel everything is already deduplicated, so all counts are `0`. + +![Deduplicate statistics](./content/admin-guide/channel-deduplicate.png) + +> Indexing, deduplication, auto-update, and cost considerations are covered in detail in +> [Module 06 — Indexing, Deduplication & Operations](../learning/administration/06-indexing-and-operations.md). ### Glossary of Terms -Once a channel is created, you can create and edit a glossary for that channel. The glossary is a set of term-definition -pairs -used by glossary tools to explain various terms and their definitions to users. +Each channel has its own glossary — a set of term/definition pairs the glossary tools use to explain terminology to +users. Open it from the channel **⋯** menu → **Glossary**. + +![Glossary page](./content/admin-guide/glossary-page.png) -By selecting "Glossary" in the Channel context menu, you will be redirected to the glossary page: +1. **+ Add Term** — add a new term. +2. Each term carries a **Source** and a **Domain** (e.g. `Macro`, `Statistics`) in addition to its definition. -![Glossary Page](./content/admin-guide/glossary-page.png) +Use **+ Add Term** (or **Edit** from a row's menu) to open the term form: -There you can add and edit terms: +![Add term](./content/admin-guide/glossary-add-term.png) -![Add Term](./content/admin-guide/glossary-add-term.png) +1. **Term** — the term itself. +2. **Definition** — the explanation surfaced to users. +3. **Source** — where the definition comes from. +4. **Domain** — a category used to group terms. -Delete terms: +Each row's **⋯** menu offers **Edit** and **Delete**: -![Delete Term](./content/admin-guide/glossary-delete-term.png) +![Term row menu](./content/admin-guide/glossary-term-menu.png) -### Reindexing +> **Note:** deleting a glossary term takes effect immediately — there is no confirmation dialog. Delete with care. -StatGPT relies on different search strategies that require indexing dataset metadata. You can see each dataset's -indexing status in the "Status" column. The following status options exist: +### Import / Export & Jobs -1. `Queued` - Dataset is waiting in the queue for the indexing job to start. -2. `In progress` - Indexing job is currently running. -3. `Finished` - Dataset was successfully indexed. -4. `Failed` - Dataset indexing job failed and requires either reconfiguration or a simple retry. +Import/Export lets you move a channel — its configuration and index — between environments as a zip archive. Each +import or export runs as a background **job** linked to the channel. -You can run reindexing for a specific dataset in the channel: +**Export** a channel from its **⋯** menu (or with the **Export** button on the channel datasets page). -![Reindex Dataset](./content/admin-guide/reindex-dataset.png) +**Import** a channel with the **Import** button on the channels list: -You can also run reindexing for all datasets in the channel: +![Import channel](./content/admin-guide/channel-import.png) -![Reindex All Datasets](./content/admin-guide/reindex-all-datasets.png) +1. **File** — drop or browse for the exported zip. +2. **Toggles:** + - `Remove channel with the same id` — delete any existing channel with the same ID before importing. If left off, + the import fails when a channel with that ID already exists. + - `Update data sets` — update datasets to the versions in the archive. + - `Update data sources` — update data sources to the versions in the archive. +3. **Import** — start the import job. -### Import/Export & Jobs +**Jobs.** Open the channel **⋯** menu → **Jobs** to see the import/export history for the channel, review job status, +and download artifacts. -Import/Export functionality allows you to download and upload channel dumps with their configurations and -indexes. When exporting or importing a channel, a dedicated job will be created. Jobs linked to the channel can be -reviewed on the Jobs page, where related artifacts can also be downloaded. +![Channel jobs](./content/admin-guide/channel-jobs-download.png) -You can export a specific channel by clicking the "Export" button in the channel context menu. +1. **Type** — `EXPORT` or `IMPORT`, each with a `Status` (e.g. `COMPLETED`). +2. **Export (download)** — for a completed export, the row **⋯** menu lets you download the resulting artifact. -You can import a channel by clicking the "Import" button on the channels list page: +--- -![Import Channel](./content/admin-guide/channel-import.png) +## Audit Logs -The import menu has the following toggles: +The **Audit Logs** screen is a chronological record of configuration changes across data sources, channels, and +datasets. Use the column filters to scope the view (here it is filtered to the sample entities). -1. `Remove channel with the same id` - If enabled, any existing channel with the same ID will be removed before - importing - the new one. If disabled, the import will fail if a channel with the same ID already exists. -2. `Update datasets` - If enabled, datasets will be updated to the version in the import file. If disabled, existing - datasets will not be updated. -3. `Update data sources` - If enabled, data sources will be updated to the version in the import file. If disabled, - existing data sources will not be updated. +![Audit Logs](./content/admin-guide/audit-logs.png) -![Import Channel Menu](./content/admin-guide/channel-import-menu.png) +1. **Action** — what happened (`create`, `update`, `delete`). +2. **Entity name** — which object was affected, alongside its **Entity type** (`data_source`, `channel`, `dataset`) + and **Entity ID**. -After starting either import or export, a new job will appear in the corresponding channel's jobs list, which can be -accessed by clicking the "Jobs" button in the channel menu: +Each entry also records who initiated the change (the **Initiated** column, redacted in this screenshot), an +**Activity ID**, and the **Time**. -![Channel Jobs](./content/admin-guide/channel-jobs.png) +--- -The Jobs page allows you to review jobs related to the channel, their status, and download related artifacts: +## Related resources -![Jobs Page](./content/admin-guide/channel-jobs-download.png) +- [Admin Learning Course](../learning/administration/README.md) — end-to-end dataset onboarding methodology. +- [Architecture Overview](../architecture/overview.md) and [Agent design](../architecture/agent.md). +- [SDMX Compatibility & Requirements](../architecture/sdmx-compatibility.md). diff --git a/guides/content/admin-guide/audit-logs.png b/guides/content/admin-guide/audit-logs.png new file mode 100644 index 0000000..cee2238 Binary files /dev/null and b/guides/content/admin-guide/audit-logs.png differ diff --git a/guides/content/admin-guide/channel-configure.png b/guides/content/admin-guide/channel-configure.png new file mode 100644 index 0000000..c741d5f Binary files /dev/null and b/guides/content/admin-guide/channel-configure.png differ diff --git a/guides/content/admin-guide/channel-dataset-menu.png b/guides/content/admin-guide/channel-dataset-menu.png new file mode 100644 index 0000000..4381cf6 Binary files /dev/null and b/guides/content/admin-guide/channel-dataset-menu.png differ diff --git a/guides/content/admin-guide/channel-deduplicate.png b/guides/content/admin-guide/channel-deduplicate.png new file mode 100644 index 0000000..f01ad4a Binary files /dev/null and b/guides/content/admin-guide/channel-deduplicate.png differ diff --git a/guides/content/admin-guide/channel-details.png b/guides/content/admin-guide/channel-details.png new file mode 100644 index 0000000..c3ad9d5 Binary files /dev/null and b/guides/content/admin-guide/channel-details.png differ diff --git a/guides/content/admin-guide/channel-import.png b/guides/content/admin-guide/channel-import.png new file mode 100644 index 0000000..c36cdb0 Binary files /dev/null and b/guides/content/admin-guide/channel-import.png differ diff --git a/guides/content/admin-guide/channel-jobs-download.png b/guides/content/admin-guide/channel-jobs-download.png index 3e9bea2..05d35cb 100644 Binary files a/guides/content/admin-guide/channel-jobs-download.png and b/guides/content/admin-guide/channel-jobs-download.png differ diff --git a/guides/content/admin-guide/channel-menu.png b/guides/content/admin-guide/channel-menu.png index c58b922..95cdf24 100644 Binary files a/guides/content/admin-guide/channel-menu.png and b/guides/content/admin-guide/channel-menu.png differ diff --git a/guides/content/admin-guide/channel-recalc-dropdown.png b/guides/content/admin-guide/channel-recalc-dropdown.png new file mode 100644 index 0000000..c31e6b4 Binary files /dev/null and b/guides/content/admin-guide/channel-recalc-dropdown.png differ diff --git a/guides/content/admin-guide/channels-list.png b/guides/content/admin-guide/channels-list.png new file mode 100644 index 0000000..1b37efc Binary files /dev/null and b/guides/content/admin-guide/channels-list.png differ diff --git a/guides/content/admin-guide/dataset-add-config.png b/guides/content/admin-guide/dataset-add-config.png new file mode 100644 index 0000000..9b4d924 Binary files /dev/null and b/guides/content/admin-guide/dataset-add-config.png differ diff --git a/guides/content/admin-guide/dataset-add-dataflow.png b/guides/content/admin-guide/dataset-add-dataflow.png new file mode 100644 index 0000000..d966cc9 Binary files /dev/null and b/guides/content/admin-guide/dataset-add-dataflow.png differ diff --git a/guides/content/admin-guide/dataset-add-provider.png b/guides/content/admin-guide/dataset-add-provider.png new file mode 100644 index 0000000..a6b6b2c Binary files /dev/null and b/guides/content/admin-guide/dataset-add-provider.png differ diff --git a/guides/content/admin-guide/dataset-add-source.png b/guides/content/admin-guide/dataset-add-source.png new file mode 100644 index 0000000..ce8d827 Binary files /dev/null and b/guides/content/admin-guide/dataset-add-source.png differ diff --git a/guides/content/admin-guide/dataset-auto-update-jobs.png b/guides/content/admin-guide/dataset-auto-update-jobs.png new file mode 100644 index 0000000..602e2b5 Binary files /dev/null and b/guides/content/admin-guide/dataset-auto-update-jobs.png differ diff --git a/guides/content/admin-guide/dataset-edit-config.png b/guides/content/admin-guide/dataset-edit-config.png new file mode 100644 index 0000000..f8135e4 Binary files /dev/null and b/guides/content/admin-guide/dataset-edit-config.png differ diff --git a/guides/content/admin-guide/dataset-versions.png b/guides/content/admin-guide/dataset-versions.png new file mode 100644 index 0000000..77a8321 Binary files /dev/null and b/guides/content/admin-guide/dataset-versions.png differ diff --git a/guides/content/admin-guide/datasets-list.png b/guides/content/admin-guide/datasets-list.png new file mode 100644 index 0000000..c46518f Binary files /dev/null and b/guides/content/admin-guide/datasets-list.png differ diff --git a/guides/content/admin-guide/datasets-row-menu.png b/guides/content/admin-guide/datasets-row-menu.png new file mode 100644 index 0000000..9a421fb Binary files /dev/null and b/guides/content/admin-guide/datasets-row-menu.png differ diff --git a/guides/content/admin-guide/ds-add-config.png b/guides/content/admin-guide/ds-add-config.png new file mode 100644 index 0000000..805bab4 Binary files /dev/null and b/guides/content/admin-guide/ds-add-config.png differ diff --git a/guides/content/admin-guide/ds-add-properties.png b/guides/content/admin-guide/ds-add-properties.png new file mode 100644 index 0000000..9c3e395 Binary files /dev/null and b/guides/content/admin-guide/ds-add-properties.png differ diff --git a/guides/content/admin-guide/ds-configure.png b/guides/content/admin-guide/ds-configure.png new file mode 100644 index 0000000..caf9029 Binary files /dev/null and b/guides/content/admin-guide/ds-configure.png differ diff --git a/guides/content/admin-guide/ds-list.png b/guides/content/admin-guide/ds-list.png new file mode 100644 index 0000000..4fbc9e0 Binary files /dev/null and b/guides/content/admin-guide/ds-list.png differ diff --git a/guides/content/admin-guide/glossary-add-term.png b/guides/content/admin-guide/glossary-add-term.png index e427f2a..46ff932 100644 Binary files a/guides/content/admin-guide/glossary-add-term.png and b/guides/content/admin-guide/glossary-add-term.png differ diff --git a/guides/content/admin-guide/glossary-page.png b/guides/content/admin-guide/glossary-page.png index 3f15b04..9dc43b5 100644 Binary files a/guides/content/admin-guide/glossary-page.png and b/guides/content/admin-guide/glossary-page.png differ diff --git a/guides/content/admin-guide/glossary-term-menu.png b/guides/content/admin-guide/glossary-term-menu.png new file mode 100644 index 0000000..5d922cd Binary files /dev/null and b/guides/content/admin-guide/glossary-term-menu.png differ