From d8205e9d7d5066e7c046595639c3b3b7f3d87fcb Mon Sep 17 00:00:00 2001 From: Daniil Yarmalkevich Date: Thu, 2 Apr 2026 14:58:40 +0100 Subject: [PATCH] feat: added admin learning course --- learning/administration/01-core-concepts.md | 309 +++++++ .../administration/02-dataset-assessment.md | 458 ++++++++++ .../administration/03a-dimension-types.md | 472 ++++++++++ .../03b-indicator-configuration.md | 628 ++++++++++++++ .../04-dataset-configuration.md | 819 ++++++++++++++++++ .../05-data-sources-and-channels.md | 472 ++++++++++ .../06-indexing-and-operations.md | 367 ++++++++ .../07-testing-and-validation.md | 446 ++++++++++ .../08-end-to-end-walkthrough.md | 361 ++++++++ learning/administration/README.md | 46 + learning/administration/quick-reference.md | 183 ++++ 11 files changed, 4561 insertions(+) create mode 100644 learning/administration/01-core-concepts.md create mode 100644 learning/administration/02-dataset-assessment.md create mode 100644 learning/administration/03a-dimension-types.md create mode 100644 learning/administration/03b-indicator-configuration.md create mode 100644 learning/administration/04-dataset-configuration.md create mode 100644 learning/administration/05-data-sources-and-channels.md create mode 100644 learning/administration/06-indexing-and-operations.md create mode 100644 learning/administration/07-testing-and-validation.md create mode 100644 learning/administration/08-end-to-end-walkthrough.md create mode 100644 learning/administration/README.md create mode 100644 learning/administration/quick-reference.md diff --git a/learning/administration/01-core-concepts.md b/learning/administration/01-core-concepts.md new file mode 100644 index 0000000..1e36a0d --- /dev/null +++ b/learning/administration/01-core-concepts.md @@ -0,0 +1,309 @@ +# Module 01: Core Concepts & Entity Relationships + +## What You'll Learn + +- What StatGPT is and how it works at a high level +- What users see when they interact with StatGPT +- The three core entities: Data Source, Dataset, and Channel +- How these entities relate to each other +- How the Admin UI maps to these entities +- What SDMX is and why it matters for StatGPT +- Key SDMX terms every admin needs to know + +--- + +## What is StatGPT? + +StatGPT is an AI-driven Talk-To-Your-Data platform that enables users to interact with official statistics data using +natural language. Instead of writing database queries or navigating complex data portals, users simply ask questions +like *"What was the GDP of Germany in 2023?"* and StatGPT retrieves the relevant data from statistical databases, +presents it in tables, and generates visualizations. + +StatGPT combines large language models (LLMs) with structured SDMX metadata to translate natural language questions into +precise data queries. All responses are grounded in actual data — the system is designed to prevent hallucination by +citing sources and using exact values from query results. + +### What Users Experience + +StatGPT presents a **conversational chat interface** — not a search box or a data portal. Users type natural language +questions and receive structured responses that include: + +- **Data tables** with columns for dimensions, values, and attributes +- **Interactive charts** showed along with the data tables for visual insights +- **Citations** that reference the exact dataset, data source, and time period +- **Exact values** from query results — never approximated or hallucinated + +Users can start conversations using **conversation starters** — predefined example questions that appear in the chat +interface. From there, they can ask follow-up questions in **multi-turn conversations**, refining their queries or +exploring related data. + +As an admin, you configure which datasets feed these answers, what columns appear in tables, what conversation starters +users see, and how the agent behaves. Every configuration choice you make directly affects what users experience. + +## The Three Core Entities + +StatGPT's configuration revolves around three core entities that you'll work with as an administrator: + +### Data Source + +A **Data Source** is a connection to an external statistical data provider that exposes data via the SDMX protocol. +Examples include: + +- **IMF** — International Monetary Fund (SDMX 2.1 API) +- **Eurostat** — Statistical office of the European Union +- **World Bank** — World Development Indicators +- **ECB** — European Central Bank +- **BIS** — Bank for International Settlements + +A Data Source defines *how* StatGPT connects to the provider: the API URL, authentication settings, supported SDMX +features, and request headers. + +### Dataset + +A **Dataset** is a direct representation of an SDMX dataflow within a Data Source, combined with StatGPT-specific +configuration. For example, the IMF Data Source contains many datasets: + +- **IMF.RES:WEO** — World Economic Outlook (macroeconomic projections) +- **IMF.STA:CPI** — Consumer Price Index +- **IMF.STA:BOP** — Balance of Payments +- **IMF.STA:EER** — Effective Exchange Rates + +Each dataset has its own configuration that tells StatGPT how to interpret the dataset's dimensions, what to index, how +to cite the data, and more. This is where the bulk of onboarding work happens. + +### Channel + +A **Channel** is a deployment of the StatGPT application for end users. Each channel has: + +- Its own set of linked datasets +- Agent configuration (LLM model, behavior instructions, domain) +- Named Entity types for dimension recognition +- Conversation starters for the chat interface +- A glossary of terms + +Having multiple channels allows experimentation with different configurations, or serving different user groups with +tailored dataset selections. Each channel is exposed as a separate application in the DIAL platform. + +### The Admin UI + +> **StatGPT Admin** is a web application — you do not edit configuration files directly. The Admin UI has three main +> tabs that correspond to the three core entities: **Data Sources**, **Datasets**, and **Channels**. + +The typical workflow in the Admin UI follows this order: + +1. **Create a Data Source** — configure the connection to an SDMX provider +2. **Add Datasets** — select dataflows from the Data Source and configure them +3. **Create a Channel** — set up the user-facing deployment +4. **Link Datasets to the Channel** — choose which datasets this channel can query +5. **Index the Channel** — build the search index so the agent can find data + +The YAML-like configurations shown in later modules (Modules 04-05) correspond to form fields in the Admin UI, not files +you edit manually. When you see a configuration example, it maps to a specific section of the web interface. + +For detailed walkthroughs, see [Module 04 — Dataset Configuration](04-dataset-configuration.md) +and [Module 05 — Data Sources & Channels](05-data-sources-and-channels.md). + +## Entity Relationships + +```mermaid +erDiagram + DATA_SOURCE ||--o{ DATASET: "provides" + CHANNEL ||--o{ DATASET: "includes" + CHANNEL ||--|| GLOSSARY: "has" + CHANNEL ||--o{ TOOL: "configures" + + DATA_SOURCE { + string title + string type + string url + boolean authEnabled + } + DATASET { + string urn + string title + json dimensions + json citation + json indexer + } + CHANNEL { + string deployment_id + string title + json supreme_agent + json named_entity_types + } +``` + +The key relationships: + +- A **Data Source** can have many **Datasets** (one-to-many) +- A **Channel** can include many **Datasets** (many-to-many) +- A **Dataset** belongs to exactly one **Data Source** but can appear in multiple **Channels** +- Each **Channel** has its own agent configuration, glossary, and tool settings + +### Tools + +The diagram shows that a Channel configures **Tools**. These are the capabilities the agent can invoke when answering +user questions: + +| Tool | Purpose | +|------------------------|-----------------------------------------------------| +| **Query Data** | Build and execute SDMX queries to fetch actual data | +| **Available Datasets** | List which datasets are available in the channel | +| **Available Terms** | List glossary terms the agent can reference | +| **Term Definitions** | Retrieve definitions for specific glossary terms | + +Admins configure tool behavior per channel — for example, adjusting how the data query tool selects indicators or how +many results it returns. Tool configuration is covered in detail in [Module 05](05-data-sources-and-channels.md). + +### Practical Example: An IMF Channel + +A channel configured for IMF data would connect to an IMF SDMX data source and include datasets like WEO, BOP, CPI, EER, +and others. The channel's agent would be configured as "StatGPT" with a domain of "Statistics, economics and SDMX" and +use an LLM like GPT-4.1 as the underlying model. + +## Introduction to SDMX + +### What is SDMX? + +**SDMX** (Statistical Data and Metadata eXchange) is an international standard for exchanging statistical data and +metadata. It provides a common vocabulary and structure that statistical organizations worldwide use to publish their +data. + +Why SDMX matters for your work as an admin: + +1. **It's your configuration language** — every dataset configuration you write maps directly to SDMX concepts ( + dimensions, code lists, attributes). Understanding SDMX means understanding what you're configuring. +2. **It determines what's searchable** — code lists (the lists of allowed values for dimensions) become the search index + that the agent uses to find relevant data. If a code list is poorly labeled, users won't find what they need. +3. **It controls data accuracy** — the Data Structure Definition (DSD) defines what dimensions and values exist. StatGPT + uses this to construct valid queries and verify data availability. +4. **It structures the responses** — dimensions and attributes from the SDMX metadata determine what columns appear in + data tables and what citation information is available. + +You don't need to become an SDMX expert, but you need the core concepts below to configure datasets effectively. + +### Key SDMX Terms for Admins + +| Term | Definition | Why It Matters | What You Configure | +|-------------------------------------|----------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------| +| **Dataflow** | A published dataset available for querying. Identified by agency, ID, and version (e.g., `IMF.STA:CPI(5.0.0)`) | Each StatGPT dataset corresponds to one SDMX dataflow | You select which dataflow when adding a dataset in the Admin UI | +| **Data Structure Definition (DSD)** | Defines the structure of a dataflow — its dimensions, attributes, and measures | StatGPT uses the DSD to understand what dimensions exist and what values they can take | You specify which dimensions are indicators vs. non-indicators — see [Module 03a](03a-dimension-types.md) | +| **Dimension** | A structural component that identifies data points (e.g., COUNTRY, INDICATOR, TIME_PERIOD) | Dimension types (INDICATOR, NON_INDICATOR, TIME_PERIOD) are configured per dataset — see [Module 03a](03a-dimension-types.md) | You classify dimensions as INDICATOR, NON_INDICATOR, or TIME_PERIOD — see [Module 03a](03a-dimension-types.md) | +| **Code List** | An enumerated list of allowed values for a dimension (e.g., country codes, indicator codes) | Code list quality directly affects search accuracy — see [Module 02](02-dataset-assessment.md) | StatGPT indexes these; you configure how indicators are indexed — see [Module 03b](03b-indicator-configuration.md) | +| **Attribute** | Additional information attached to data (e.g., UNIT, SCALE, SOURCE) | Some attributes are included in the agent context via `includeAttributes` | You select which attributes appear in query results — see [Module 04](04-dataset-configuration.md) | +| **URN** | Unique Resource Name — the unique identifier for a dataflow (e.g., `IMF.RES:WEO(9.0.0)`) | Used in dataset configuration to link to the correct SDMX dataflow | Pre-filled when you select a dataflow in the dataset wizard | +| **Concept Scheme** | A collection of related concepts used across datasets | Helps understand what dimensions represent | You don't configure this directly — it's used internally by StatGPT | + +> **Metadata Quality Warning** +> +> Not all SDMX datasets are ready to onboard. Watch for these signs: +> +> **Red flags** (will cause problems): +> - Duplicate or generic code list names (e.g., multiple dimensions with labels like "Code" or "Value") +> - Missing English labels on code list items +> - Slow or unreliable API responses +> - Vague dimension IDs that don't indicate their purpose +> +> **Green flags** (ready to onboard): +> - Unique, descriptive names for all code list items +> - English localization present across dimensions and attributes +> - Fast, reliable API with consistent response times +> +> For the full assessment methodology, see [Module 02 — Assessing Datasets for Onboarding](02-dataset-assessment.md). + +### How StatGPT Uses SDMX + +When a user asks a question, StatGPT's Data Query tool follows this pipeline: + +1. **Query Normalization** — Processes the natural language input + - *Your role:* Ensure dataset metadata uses clear, standard terminology so the LLM can interpret queries correctly +2. **Named Entity Recognition** — Extracts countries, time periods, and other known entities + - *Your role:* Configure Named Entity types in the channel configuration — + see [Module 05](05-data-sources-and-channels.md) +3. **Indicator Selection** — Uses semantic and keyword search over indexed code list items to find relevant indicators + - *Your role:* Configure indexer settings and dimension types so the right indicators are indexed — + see [Modules 02](02-dataset-assessment.md)-[03a](03a-dimension-types.md) +4. **Dataset Selection** — Identifies which dataset(s) contain the requested data + - *Your role:* Write clear dataset descriptions and link appropriate datasets to channels — + see [Module 04](04-dataset-configuration.md) +5. **Availability Queries** — Verifies that data exists for the requested combination of dimensions + - *Your role:* Assess data completeness during onboarding — see [Module 02](02-dataset-assessment.md) +6. **Query Execution** — Fetches and formats the SDMX data + - *Your role:* Configure which columns and attributes appear in query results — + see [Module 04](04-dataset-configuration.md) + +The accuracy of each step depends on the quality of the underlying SDMX metadata and how well the dataset is configured +in StatGPT. + +## Check Your Understanding + +Test your grasp of the core concepts before moving on. + +
+1. A colleague asks you to add a new World Bank dataset to StatGPT. What three entities do you need to set up, and in what order? + +**Answer:** Data Source (World Bank SDMX API), Dataset (the specific dataflow), then link it to a Channel. The Data +Source defines the connection, the Dataset configures how StatGPT interprets the data, and the Channel makes it +available to users. + +
+ +
+2. A user reports that StatGPT can't find data about "inflation in France." Which SDMX concept is most likely involved in finding the right indicator? + +**Answer:** Code List — the indicator code list must contain an item that semantically matches "inflation" (e.g., a CPI +indicator). The indexer searches code list items to map natural language terms to specific indicators. + +
+ +
+3. You're told the IMF has released a new version of the CPI dataset (version 6.0.0, up from 5.0.0). Which StatGPT entity needs updating? + +**Answer:** It depends on how the Dataset is configured. If the dataset uses `version: "latest"` (recommended), no +manual update is needed — the system automatically tracks the current published version, and +[auto-update](06-indexing-and-operations.md#auto-update) detects the new version and reindexes if needed. If a pinned +version is used (e.g., `"5.0.0"`), you need to update the URN in the Dataset configuration to `"6.0.0"`. + +
+ +
+4. What is the difference between a Data Source and a Dataset? + +**Answer:** A Data Source is the connection to a provider's API (e.g., the IMF SDMX endpoint). A Dataset is a specific +dataflow within that source (e.g., CPI) plus StatGPT configuration for how to interpret and index it. One Data Source +has many Datasets. + +
+ +
+5. You want to test a new LLM model for answering queries. Do you need to create new Datasets, a new Channel, or both? + +**Answer:** A new Channel. Channels have their own agent/LLM configuration. You can link the same Datasets to the new +Channel without reconfiguring them. + +
+ +
+6. Why does StatGPT require SDMX-format data sources rather than generic CSV files or databases? + +**Answer:** SDMX provides structured metadata (dimensions, code lists, attributes) that StatGPT uses to understand +dataset structure, build search indexes, and construct valid queries. Without this structure, the system couldn't +automatically map natural language to the right data. + +
+ +## Key Takeaways + +- StatGPT presents a **conversational chat interface** where users get data tables, charts, and cited responses +- Three core entities: **Data Source** (connection), **Dataset** (configured dataflow), and **Channel** (user-facing + deployment) +- You manage all three entities through the **Admin UI** web application — not config files +- A Data Source can have many Datasets; a Channel includes many Datasets +- Understanding basic SDMX concepts (dataflow, DSD, dimension, code list) is essential — they map directly to what you + configure +- The quality of SDMX metadata directly impacts how well StatGPT answers user queries +- Not all datasets are ready to onboard — assess metadata quality before investing configuration effort + +--- + +**Next:** [Module 02 — Assessing Datasets for Onboarding](02-dataset-assessment.md) diff --git a/learning/administration/02-dataset-assessment.md b/learning/administration/02-dataset-assessment.md new file mode 100644 index 0000000..91cd127 --- /dev/null +++ b/learning/administration/02-dataset-assessment.md @@ -0,0 +1,458 @@ +# Module 02: Assessing Datasets for Onboarding + +## What You'll Learn + +- Why assessment before configuration prevents costly rework +- How to inspect dataset metadata in the Admin UI +- What users experience when assessment is skipped +- SDMX metadata quality criteria that affect StatGPT performance +- API performance requirements and how to test them +- A go/no-go decision framework for onboarding +- How to evaluate business value for onboarding decisions + +--- + +## Why Assessment Matters + +Configuring a dataset in StatGPT involves significant effort — dimension classification, indexer settings, testing, and +validation. Assessing a dataset *before* starting configuration helps you: + +- Identify metadata quality issues that would degrade search accuracy +- Understand the dataset structure to make correct dimension type decisions +- Determine whether the dataset's structure is compatible with StatGPT +- Evaluate whether the dataset provides enough value to justify onboarding + +Skipping assessment often leads to rework when issues surface during testing. + +## What Users Experience When Assessment Fails + +In [Module 01](01-core-concepts.md), you saw what users experience when things work well — data tables, charts, and +cited responses. When assessment is skipped or done poorly, users see the consequences directly: + +| Assessment Failure | User-Visible Consequence | +|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------| +| Generic or numeric code list names | "No data found" for valid queries — the search index can't match natural language to meaningless codes | +| Duplicate names in code lists | Wrong indicator selected silently, or confusing clarification prompts asking users to choose between identically-named items | +| Missing English localization | Raw indicator codes (e.g., `IND_042`) shown instead of readable text in tables and responses | +| Slow API responses | Long waits (10+ seconds), possible timeouts, and incomplete responses | +| Sparse data (many empty observations) | Tables full of empty cells that confuse users and undermine trust in the system | + +> **As you evaluate each criterion below, ask yourself: if this fails, what will the user see?** + +## SDMX Metadata Quality Criteria + +StatGPT relies heavily on SDMX metadata for indicator search and query construction. Poor metadata quality leads to poor +search results. Evaluate these criteria before onboarding: + +### How to Inspect Metadata + +Before assessing quality criteria, you need to know *where* to look. Three methods, in order of preference: + +1. **Admin UI Dataset Wizard** — When adding a dataset (**Datasets > Add**), step 3 shows the dataset structure after + you select a dataflow: dimensions, code lists, and sample values. You can cancel the wizard after reviewing — no need + to finish adding the dataset. This is the primary inspection method for all admins. + +2. **Provider's Data Explorer or similar tool** - Many SDMX providers have a web-based data explorer that shows metadata + in a user-friendly way. This can be a good secondary method if the Admin UI doesn't show enough detail or if you want + to see how the provider presents the data to users. + +3. **SDMX API directly** — Advanced fallback: query the provider's REST API to browse full code lists and dataflow + structures. Not normally needed, but useful for systematic testing or when the Admin UI doesn't show enough detail. + +### 1. Meaningful, Complete Names and Descriptions + +**What to check:** + +- Do dataflow names and descriptions clearly indicate what data is available? +- Do code list items have human-readable names (not just codes)? +- Are descriptions provided for dimensions and concepts? + +**Why it matters:** StatGPT uses names and descriptions for both semantic search (embedding similarity) and keyword +search (exact term matching). Vague or missing names mean users can't find the data. + +**Good example:** + +``` +Code: LP +Name: "Population, Persons for countries / Index for country groups" +``` + +**Bad example:** + +``` +Code: IND_042 +Name: "Indicator 42" +``` + +### 2. Meaningful Hierarchies in Code Lists + +**What to check:** + +- If code lists are hierarchical, do the hierarchies follow a logical structure? +- Are parent-child relationships consistent and non-contradictory? + +**Why it matters:** StatGPT can use hierarchical structure to understand relationships between indicators (e.g., "Food +and non-alcoholic beverages" is a category under CPI). + +### 3. Non-Duplicated Item Names in Code Lists + +**What to check:** + +- Are there code list items with identical names but different IDs? +- Are names unique enough to distinguish items when searching? + +**Why it matters:** Duplicate names create ambiguity. When a user asks for "GDP" and there are three code list items all +named "GDP" with different IDs, the system cannot reliably select the correct one. + +**Critical:** This is listed as a Critical-priority requirement in +the [SDMX Compatibility Guide](../../architecture/sdmx-compatibility.md). + +### 4. English Localization + +**What to check:** + +- Are all structural metadata elements available in English? +- Are localizations consistent across structures (code lists, concepts, dataflows)? + +**Why it matters:** StatGPT requires English localization as a minimum. Additional languages must be consistent across +all structures. + +### 5. API Performance + +**What to check:** + +StatGPT makes multiple API calls during a single user query (structure lookups, availability checks, data fetches). Slow +responses at any step compound into poor user experience. + +**Performance thresholds** (from +the [SDMX Compatibility Guide](../../architecture/sdmx-compatibility.md)): + +| Endpoint Type | Request Scenario | Max Latency | +|--------------------------|---------------------------------------|-------------| +| **Structure** | Single structure (no reference stubs) | < 1 sec | +| **Data** | 20 series | < 1 sec | +| **Available Constraint** | 300 series | < 1 sec | + +**How to test:** + +- **Quick gauge:** Use the Admin UI Dataset Wizard — if step 3 (structure loading) takes noticeably long, investigate + further. +- **Systematic testing:** Query the provider's SDMX REST API directly for each endpoint type and measure response times. + +**Interpretation:** + +| Response Time | Assessment | +|----------------------|--------------------------------------------------------------------| +| < 1 sec | Pass | +| 1–2 sec | Warning — document, proceed with caution | +| 2–5 sec | Significant concern — test thoroughly, may degrade user experience | +| Consistently > 5 sec | **Blocker** — do not proceed until resolved with the provider | + +## Packed vs. Unpacked Indicators + +During assessment, you'll notice that some datasets have codelist values with comma-separated multi-concept strings ( +e.g., `"GDP, constant prices, Percent change"`) while others have single-concept values (e.g., +`"Consumer price index (CPI)"`). This distinction — packed vs. unpacked — is critical for configuration but requires +understanding indicator dimensions first. + +**This topic is covered +in [Module 03b — Indicator Configuration](03b-indicator-configuration.md#packed-vs-unpacked-indicators)**, where +you'll learn the decision algorithm after understanding dimension classification. + +For now during assessment, simply note whether you see multi-concept comma-separated values in the codelist — you'll +need this observation in Module 03. + +## Annotations and Attributes + +### Annotations + +Some SDMX providers use annotations to add metadata beyond the standard fields: + +- **Last updated date** — useful for showing data freshness (configured via `updatedAt`) +- **Additional context** — descriptions, notes, methodology references + +Check if the data source supports annotations and whether they provide useful information. + +### Attributes + +SDMX attributes attach additional information to data observations or series: + +- **UNIT** — Unit of measure (e.g., "US Dollars", "Percent") +- **SCALE** — Scale of the values (e.g., "Millions", "Billions") +- **SOURCE** — Data source reference +- **PUBLISHER** — Publishing organization + +Relevant attributes should be included via the `includeAttributes` configuration field so the AI agent has context about +the data it presents. + +## Empty Observations + +Check whether the dataset has many empty (null) observations for certain dimension combinations. + +**How to detect:** During assessment, try querying common dimension combinations — for example, request GDP data for a +few major countries over recent years. If many cells come back empty, the dataset may be sparse. + +**User impact:** Tables full of empty cells confuse users and undermine trust. A user asking "What is the GDP of +Germany?" expects a clean table, not rows of missing values. + +**Severity:** This is a Warning-tier issue, not a Blocker. Sparse data doesn't prevent onboarding but should be +documented and tested thoroughly during [Module 07 — Testing and Validation](07-testing-and-validation.md). + +## Assessment Decision Framework + +The criteria above tell you *what* to check. This framework tells you *what to do* with the results. + +### Blockers (do NOT proceed) + +These issues will cause fundamental failures in the user experience. If any Blocker is present, **stop and escalate to +the data provider** before investing configuration effort. + +- Generic or numeric code list names with no meaningful labels (e.g., "Indicator 42", "Code_001") +- Missing English localization on code lists — indicator codes will appear as raw IDs +- API consistently exceeding 5x performance thresholds (> 5 sec responses) +- Available Constraint endpoint doesn't work or returns errors + +### Warnings (proceed with caution, document) + +These issues degrade quality but don't prevent onboarding. Document them and plan extra testing. + +- Some duplicate names (< 10% of code list items) +- Borderline API performance (1–2 sec responses) +- Sparse data in some dimension combinations +- Hierarchies partially inconsistent + +### Nice-to-Have (proceed, note for future improvement) + +These are desirable but not required. Note them for future enhancement. + +- Code list descriptions would help search quality but are missing +- No last-updated annotations (data freshness won't be shown) +- Some non-essential attributes missing (e.g., SOURCE, PUBLISHER) + +### Decision Logic + +``` +All Blockers pass + business value confirmed → PROCEED to configuration +Any Blocker fails → STOP, document the issue, escalate to the provider +Warnings present → PROCEED, document warnings, plan extra testing in Module 07 +``` + +## Business Value Assessment + +Before investing effort in onboarding, consider: + +### Main Purpose + +- What questions can users answer with this dataset? +- Does it fill a gap not covered by existing datasets? + +### Target Users + +- Who will query this data? Economists? Journalists? General public? +- How technical is the expected user base? + +### Potential Frequently Asked Questions + +- What are the most likely queries users will make? +- Are the key indicators well-represented in the code lists? +- Can users find what they expect using common terms (e.g., "inflation" should map to CPI)? + +### Overlap with Existing Datasets + +- Does this dataset overlap with already-onboarded datasets? +- If so, which should take priority for overlapping queries? +- **Overlapping datasets can both be onboarded** if they serve different use cases. For example, the IMF channel has + both ANEA (annual national accounts) and QNEA (quarterly national accounts) — same underlying data at different + frequencies. ANEA serves annual comparison queries while QNEA serves recent-trend queries. +- Use the `isOfficial` flag to mark national-level or authoritative sources. Official datasets are prioritized in + dataset selection when queries match multiple datasets. + +## Assessment Checklist + +Before proceeding to configuration, verify the following items organized by severity: + +### Blockers (must ALL pass) + +- [ ] Code list items have meaningful, descriptive names (not generic codes) +- [ ] English localization is complete and consistent across structures +- [ ] Structure endpoint responds in < 1 sec (single structure) +- [ ] Data endpoint responds in < 1 sec (20 series) +- [ ] Available Constraint endpoint responds in < 1 sec (300 series) +- [ ] Available Constraint endpoint is functional (returns valid results) + +### Warnings (document if any fail) + +- [ ] No duplicate names within code lists (or minimal duplicates, < 10%) +- [ ] Hierarchies (if any) are logically structured and consistent +- [ ] Data is reasonably dense (not mostly empty observations) +- [ ] API performance is consistently within thresholds (no intermittent slowdowns) + +### Required Understanding (must determine before configuration) + +- [ ] You've noted whether codelist values contain comma-separated multi-concept strings ( + see [Module 03b](03b-indicator-configuration.md#packed-vs-unpacked-indicators)) +- [ ] You've reviewed the dataset's dimensions and codelist values (classification is covered + in [Module 03a](03a-dimension-types.md)) +- [ ] Relevant attributes are available (UNIT, SCALE, etc.) +- [ ] The dataset provides clear business value for the target audience +- [ ] You've checked for overlap with existing datasets + +## Key Takeaways + +- Always assess metadata quality before starting configuration — it prevents rework +- **Always inspect metadata through the Admin UI wizard before assessing** — don't guess at quality, look at actual code + list values +- Good code list names are critical — StatGPT cannot find data that isn't described clearly in the metadata +- **Use the three-tier decision framework** (Blockers / Warnings / Nice-to-Have) to make clear go/no-go decisions +- Consider business value and user needs to prioritize onboarding efforts +- Note codelist value patterns during assessment — you'll use them for dimension classification and indexer settings + in [Module 03a](03a-dimension-types.md) + +## Check Your Understanding + +Test your grasp of the assessment concepts before moving on. + +
+1. You need to assess a new World Bank dataset. Where do you start inspecting the metadata? + +**Answer:** Start with the **Admin UI Dataset Wizard**. Go to Datasets > Add, select the World Bank data source, browse +available dataflows, and select the one you want to assess. Step 3 of the wizard shows the dataset structure — +dimensions, code lists, and sample values. You can cancel the wizard after reviewing without actually adding the +dataset. + +
+ +
+2. During assessment, you find that 15% of code list items share duplicate names. Should you proceed with onboarding? + +**Answer:** This is in the **Warning tier** (> 10% duplicates is concerning). Proceed with caution — document the +duplicates, plan extra testing in [Module 07](07-testing-and-validation.md) to verify that search accuracy is +acceptable, and consider whether the provider can improve the metadata. + +
+ +
+3. Users report that "inflation in France" returns no results. You inspect the dataset and see codelist values like IND_042 (Name: "Indicator 42"), IND_043 (Name: "Indicator 43"). What went wrong? + +**Answer:** **Code list quality** (Criterion 1 — Meaningful, Complete Names). The indicator codelist has generic +labels (`"Indicator 42"`) instead of descriptive ones (`"Consumer Price Index"` or +`"Inflation, average consumer prices"`). The search index can't match "inflation" to `"Indicator 42"` — there's no +semantic or keyword overlap. This is a **Blocker** — do not proceed until the provider adds meaningful names. + +
+ +
+4. The Available Constraint endpoint takes 3 seconds for 300 series. What's your assessment? + +**Answer:** This exceeds the < 1 sec threshold. It falls in the **Warning tier** (1–5 sec range) — document it and +proceed, but monitor for degradation. If it consistently exceeds 5 seconds, it becomes a **Blocker** that should be +escalated to the data provider. + +
+ +
+5. The IMF has both ANEA (annual) and QNEA (quarterly) national accounts datasets with overlapping data. Should you onboard both? + +**Answer:** **Yes** — they serve different use cases. ANEA is better for long-term annual comparisons (e.g., "GDP growth +over the last 10 years"), while QNEA serves recent-trend queries (e.g., "quarterly GDP change in 2024"). Document the +overlap in both dataset descriptions so the agent can select the right one based on the user's query. + +
+ +## Practical Exercises + +Apply the assessment checklist to real dataset metadata. These exercises test your ability to evaluate datasets for +onboarding — quality, performance, and business value. + +### Exercise 1: Assess a Dataset with Quality Issues + +You open the Admin UI Dataset Wizard and inspect a dataset from a new provider. Here's what you see: + +**Dataflow:** "National Accounts Data" (no further description) + +**Dimensions:** + +| Dimension ID | Concept Name | Sample Codelist Values | +|---------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `IND` | Indicator | "GDP_C" (Name: "Gross Domestic Product, current prices"), "GDP_K" (Name: "GDP constant"), "IND_003" (Name: "Indicator 3"), "IND_004" (Name: "Trade balance") | +| `REF_AREA` | Reference area | "US" (Name: "United States"), "DE" (Name: "Germany"), "JP" (Name: "Japan") | +| `FREQ` | Frequency | "A" (Name: "Annual"), "Q" (Name: "Quarterly") | +| `TIME_PERIOD` | Time period | "2020", "2021", "2022", "2023" | + +**Performance:** Structure endpoint: ~0.8 sec. Data endpoint (20 series): ~1.5 sec. Available Constraint (300 series): ~ +0.9 sec. + +**Your task:** Apply the decision framework. Is this a go, no-go, or conditional proceed? + +
+Solution + +**Blockers identified:** + +- **Mixed code list quality:** "IND_003" with name "Indicator 3" is a generic label — a Blocker for that item. However, + most items have meaningful names ("Gross Domestic Product, current prices", "Trade balance"). +- **Verdict:** This is a **borderline Blocker**. If only a few items have generic names, you might proceed but those + items won't be findable by users. If the generic items represent important indicators, this is a hard Blocker. + +**Warnings identified:** + +- **Data endpoint at 1.5 sec** exceeds the < 1 sec threshold — Warning tier. Document this and test whether it degrades + user experience under load. +- **Abbreviated code list name:** "GDP constant" is incomplete — should specify "constant prices" in which currency/base + year. This could lead to user confusion. + +**Nice-to-Have:** + +- Dataflow description is vague ("National Accounts Data") — better description would help dataset selection. + +**Recommendation:** Escalate to the provider about the generic code names ("IND_003", "Indicator 3") and the incomplete +name "GDP constant". If the provider can fix these, proceed with caution on the borderline performance. Document the 1.5 +sec data endpoint response time. + +
+ +### Exercise 2: Assess Overlapping Datasets for Business Value + +Your channel already has **IMF WEO** (World Economic Outlook) onboarded. A stakeholder requests onboarding **IMF ANEA +** (Annual National Accounts Estimates). You inspect ANEA and find: + +**IMF WEO (already onboarded):** + +- 45 macroeconomic indicators (GDP, inflation, trade, fiscal, etc.) +- 194 countries, annual frequency +- Forecasts included (up to 5 years ahead) + +**IMF ANEA (candidate):** + +- ~100 national accounts indicators (GDP components, expenditure, income, savings) +- 194 countries, annual frequency +- Historical data only (no forecasts) +- All assessment criteria pass (no Blockers, no Warnings) + +**Your task:** Should you onboard ANEA? Both? Neither? Justify using the business value framework. + +
+Solution + +**Onboard both** — they serve different use cases despite overlap: + +- **WEO** covers broad macroeconomic indicators with forecasts — best for questions like "What is the GDP forecast for + Brazil?" or "Compare inflation across G7 countries" +- **ANEA** provides detailed national accounts breakdowns — best for questions like "What is the share of government + consumption in GDP for Germany?" or "How has gross capital formation changed over the last decade?" + +**Key differences:** + +- ANEA has more granular GDP components that WEO doesn't cover +- WEO has forecasts that ANEA doesn't +- Overlap exists for headline GDP figures, but ANEA provides more detailed decomposition + +**Recommendation:** Onboard ANEA. Document the overlap in both dataset descriptions so the agent can select the right +dataset based on query specificity. Use clear `indexer.description` values: WEO for "broad macroeconomic indicators and +forecasts", ANEA for "detailed national accounts components and breakdowns". + +
+ +--- + +**Previous:** [Module 01 — Core Concepts & Entity Relationships](01-core-concepts.md) | **Next:** [Module 03a — Dimension Types & Named Entities](03a-dimension-types.md) diff --git a/learning/administration/03a-dimension-types.md b/learning/administration/03a-dimension-types.md new file mode 100644 index 0000000..d452058 --- /dev/null +++ b/learning/administration/03a-dimension-types.md @@ -0,0 +1,472 @@ +# Module 03a: Dimension Types & Named Entities + +> **This is a key module.** Dimension type classification is the most common source of configuration errors. Take time +> to understand the decision framework before configuring datasets. + +## What You'll Learn + +- The three dimension categories: INDICATOR, NON_INDICATOR, and TIME_PERIOD +- A decision framework for classifying dimensions (the part admins struggle with most) +- How dimension types map to dataset configuration fields in the Admin UI +- What Named Entity types are and how they relate to NON_INDICATOR dimensions +- How to handle special dimensions with large hierarchical code lists +- Concrete examples from IMF, Eurostat, ECB, BIS, and other agencies + +--- + +## The Three Dimension Categories + +Every dimension in a StatGPT dataset must be classified into one of three types: + +### INDICATOR + +Dimensions that describe **what is being measured** — the concept or metric. + +Examples of indicator values: + +- GDP, CPI, unemployment rate, population +- Balance of payments accounting entry (credit/debit) +- Price type (current prices, constant prices) +- Seasonal adjustment method +- Type of transformation (index, percent change) + +**Key characteristic:** These are domain-specific concepts that typically require subject-matter knowledge to +understand. + +> **Why it matters for search:** Values from INDICATOR dimensions are indexed in the search engine (embeddings + +> fulltext). When a user asks *"What was inflation?"*, the system searches the INDICATOR index to find matching concepts +> like "Consumer Price Index (CPI)". If a dimension is misclassified as NON_INDICATOR, its values won't be searchable — +> users looking for those concepts will get no results. + +### NON_INDICATOR + +Dimensions that describe **general, universally understood concepts** independent of the statistical domain. + +Examples: + +- Country / Reference area +- Frequency (annual, quarterly, monthly) +- Counterpart country +- Currency +- Unit of measure (in some contexts) + +**Key characteristic:** An average person without domain expertise would understand what these concepts mean. + +> **How StatGPT handles these:** NON_INDICATOR values are matched via Named Entity Recognition (NER), not search. +> The LLM extracts entities like "Germany" or "quarterly" directly from the user query and maps them to dimension +> values. This is why NON_INDICATOR dimensions must map to Named Entity types in the channel config — without that +> mapping, the system won't know to look for them. + +### TIME_PERIOD + +The temporal dimension. There is exactly one TIME_PERIOD dimension per dataset. It is typically configured explicitly +to set `defaultQueries` (default time ranges), though the system can identify it automatically from the SDMX structure. + +--- + +## Consequences of Misclassification + +Getting dimension types wrong has concrete, observable consequences: + +| Misclassification | What Goes Wrong | +|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **INDICATOR → NON_INDICATOR** | Values not indexed for search. Users searching for those concepts get no results. The system tries NER instead, which is unreliable for domain-specific terms like "Chain linked volumes" or "Gross fixed capital formation". | +| **NON_INDICATOR → INDICATOR** | Values indexed unnecessarily, adding noise to search results. Named entities not recognized — "Germany" gets searched in the indicator index instead of being recognized as a country. | + +**Concrete example:** If you classify `COUNTRY` as INDICATOR in WEO, a query *"GDP of Germany"* searches for "Germany" +in the indicator index alongside "GDP". Germany won't match any indicator, causing the query to fail or return +unexpected results. Meanwhile, the NER step doesn't look for country entities in that dimension, so "Germany" is never +matched to the correct country code. + +--- + +## How Dimension Types Map to the Admin UI Config + +In the Admin UI, each dimension is configured explicitly with a `dimensionType` field in the `dimensions` map: + +| Dimension Type | How You Configure It | +|-----------------------------|---------------------------------------------------------------------------------------------| +| **INDICATOR** | `dimensionType: "INDICATOR"` on the dimension | +| **INDICATOR (required)** | Also set `isRequired: true` on the dimension | +| **SPECIAL** | `dimensionType: "SPECIAL"` with `processorId` on the dimension | +| **NON_INDICATOR (country)** | `dimensionType: "NON_INDICATOR"` with `subtype: "REGION"` and `alias` | +| **NON_INDICATOR (frequency)** | `dimensionType: "NON_INDICATOR"` with `subtype: "FREQUENCY"` | +| **NON_INDICATOR (other)** | `dimensionType: "NON_INDICATOR"` (no subtype needed) | +| **TIME_PERIOD** | `dimensionType: "TIME_PERIOD"` with optional `defaultQueries` | + +**Example — IMF WEO:** + +```yaml +dimensions: + INDICATOR: + dimensionType: "INDICATOR" + isRequired: true + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["-5y", "+2y"] + operator: "between" +``` + +This tells StatGPT: + +- `INDICATOR` is an indicator dimension (and it's required) +- `COUNTRY` is the country/region dimension (NON_INDICATOR with `subtype: "REGION"`) +- `FREQUENCY` is explicitly NON_INDICATOR with `subtype: "FREQUENCY"` +- `TIME_PERIOD` is the time dimension with a default range + +--- + +## Decision Framework + +This is where administrators struggle the most. For each dimension in the dataset, you typically have three pieces of +information to work with: + +- **Dimension ID** — the technical identifier (e.g., `SECTOR`, `REF_AREA`, `COICOP_1999`) +- **Concept name** — the human-readable label (e.g., "Sector", "Reference area", "Classification of Individual + Consumption by Purpose") +- **Codelist values** — the actual values the dimension can take (e.g., "Households", "Germany", "01 - Food and + non-alcoholic beverages") + +The dimension ID and concept name give you a starting hint, but **always check the codelist values** — they are the most +reliable input for classification. The same dimension name (e.g., `UNIT`) can require different classification depending +on what values it actually contains. + +### The Core Question + +> **"Would an average person understand this concept without domain knowledge?"** + +- **Yes** → NON_INDICATOR (country, frequency, currency, counterpart area) +- **No** → INDICATOR (GDP component, price type, adjustment method, sector classification) + +### Step-by-Step Checklist + +For each dimension in the dataset: + +1. **Is it the time dimension?** + - If yes → `dimensionType: "TIME_PERIOD"`. Configure `defaultQueries` if needed. Done. + +2. **Is it the country/region dimension?** + - If yes → `dimensionType: "NON_INDICATOR"` with `subtype: "REGION"`. Add an `alias` if the dimension ID is not + self-explanatory. Done. + +3. **Is it a frequency dimension?** + - If yes → `dimensionType: "NON_INDICATOR"` with `subtype: "FREQUENCY"`. Done. + +4. **Check the Named Entity types for the channel.** + - Does this dimension map to an existing Named Entity type (e.g., "Counterpart area/country", "Currency/Unit of + measure")? + - If yes → NON_INDICATOR. Done. + +5. **Apply the core question — using all three inputs.** + - Start with the concept name for a first impression, but **always check the codelist values**. + - Would a non-expert understand what these codelist values represent? + - If the values describe *what is being measured* or *how it's measured* → INDICATOR (`dimensionType: "INDICATOR"`) + - If the values describe a general concept (age groups, gender, geographic regions) → NON_INDICATOR (`dimensionType: "NON_INDICATOR"`) + +6. **When in doubt, lean toward INDICATOR.** + - It's better to classify a borderline dimension as INDICATOR than NON_INDICATOR. Incorrect NON_INDICATOR + classification can cause the system to misinterpret user queries. + +### Grey Areas and How to Resolve Them + +| Dimension ID | Concept name | Codelist values (sample) | Classification | Reasoning | +|--------------------------|-------------------------------|--------------------------------------------------------------------------------|----------------|---------------------------------------------------------------| +| `UNIT` | Unit of measure (IMF BOP) | `USD` — US Dollars, `PC_GDP` — Percent of GDP | NON_INDICATOR | Currency/unit concepts are universally understood | +| `SECTOR` | Sector (IMF FSIC) | `DT` — Core FSI: Deposit Takers, `OFC` — Additional FSI: Other Financial Corps | INDICATOR | Financial sector classifications require domain knowledge | +| `COUNTERPART_COUNTRY` | Counterpart country (IMF DIP) | `US` — United States, `DE` — Germany, `JP` — Japan | NON_INDICATOR | Countries are universally understood | +| `TYPE_OF_TRANSFORMATION` | Transformation (IMF CPI) | `IX` — Index, `PC_CP_A_PT` — Percentage change, prev. year | INDICATOR | Describes how the indicator is computed/presented | +| `ADJUSTMENT` | Adjustment (ECB BSI) | `Y` — Working day and seasonally adjusted, `N` — Neither adjusted | INDICATOR | Statistical adjustment methods require expertise | +| `ACCOUNTING_ENTRY` | Accounting entry (BIS) | `F` — Net flows, `S` — Stocks | NON_INDICATOR | General accounting concepts; understandable without expertise | + +### Common Mistakes + +1. **Classifying UNIT as INDICATOR in all cases.** UNIT is often NON_INDICATOR — the key is whether the values are + universally understood (USD, EUR, Percent → NON_INDICATOR) vs. domain-specific. + +2. **Classifying sector/industry dimensions as NON_INDICATOR.** Dimensions like SECTOR, NACE, or COICOP describe + domain-specific classifications and should generally be INDICATOR. + +3. **Forgetting to check Named Entity types.** Before classifying any dimension as NON_INDICATOR, verify it maps to a + Named Entity type in the channel config. + +--- + +## Named Entity Types + +### What Are Named Entity Types? + +Named Entity types are categories of real-world entities that StatGPT can recognize in user queries. They are configured +per channel and used during the Named Entity Recognition step of query processing. + +When a user asks *"What was GDP of Germany in 2023?"*, the system recognizes: + +- "Germany" as a **Country/Reference area** entity +- "2023" as a time period + +This recognition relies on the channel's Named Entity type configuration. + +### How NER Works in Practice + +During query processing, the LLM receives the list of Named Entity types configured for the channel. For each user +query, it extracts entities and categorizes them: + +- User: *"quarterly GDP for Germany in euros"* +- NER extracts: + - "Germany" → Country/Reference area + - "quarterly" → Time frequency + - "euros" → Currency/Unit of measure +- These extracted entities are then matched to specific code list values in each dataset + +**If a Named Entity type is missing from the channel config**, the NER step won't know to look for those entities. The +dimension may still work through default queries or LLM reasoning, but recognition is less reliable and may fail for +unusual values. + +### Standard Named Entity Types + +Most channels use these standard types: + +| Named Entity Type | Description | Example Dimension IDs | +|----------------------------|---------------------------------------------------------------------------|---------------------------------------------------| +| `Country/Reference area` | Countries and regions (configured as `countryNamedEntityType` in channel) | COUNTRY, REF_AREA, geo | +| `Time frequency` | Data frequency (annual, quarterly, monthly) | FREQUENCY, FREQ, freq | +| `Counterpart area/country` | The "other" country in bilateral data | COUNTERPART_COUNTRY, COUNTERPART_AREA, COUNT_AREA | +| `Currency/Unit of measure` | Currency and measurement units | UNIT, CURRENCY_TRANS, UNIT_MEASURE | + +### How NON_INDICATOR Dimensions Map to Named Entity Types + +Every NON_INDICATOR dimension (except country and frequency, which are handled specially) should map to a Named Entity +type in the channel configuration. + +**Process:** + +1. Classify a dimension as NON_INDICATOR +2. Check if an existing Named Entity type in the channel covers it +3. If not, **add a new Named Entity type** to the channel config + +It's perfectly normal to add new Named Entity types when onboarding datasets with dimensions that don't fit existing +types. + +### When to Add New Named Entity Types + +If you classify a dimension as NON_INDICATOR but no existing Named Entity type matches it, add one. For example: + +- The BIS Debt Securities dataset has `CONSOLIDATION` and `EXPENDITURE` as NON_INDICATOR dimensions +- If no existing Named Entity type covers these, you might add types like "Consolidation basis" or "Expenditure type" + +**Example channel Named Entity types for an IMF-focused channel:** + +```yaml +namedEntityTypes: + - Time frequency + - Counterpart area/country + - Currency/Unit of measure +countryNamedEntityType: Country/Reference area +``` + +### The allValues Pattern + +Some datasets support star-queries where users ask *"for all countries"* or *"global GDP"*. This is configured via +`allValues` on the country dimension — a synthetic value that represents "all countries" in the query: + +```yaml +COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + allValues: + id: "ALL_COUNTRIES" + name: "All countries - must be selected when query explicitly asks for all countries" + description: "Special value to query all countries" +``` + +When a user asks *"What is global GDP?"*, the system selects the `ALL_COUNTRIES` value instead of listing individual +countries. This is covered in detail in [Module 04](04-dataset-configuration.md) — mentioned here because it affects +how the country NON_INDICATOR dimension operates. + +--- + +## Special Dimensions + +### What Are Special Dimensions? + +Some dimensions don't fit cleanly into the INDICATOR or NON_INDICATOR processing paths. **Special dimensions** are an +extensibility mechanism — they allow pluggable processors to handle edge cases without reworking the core dimension type +system. + +Currently, one subtype of special dimension exists: **Large Hierarchical Code Lists (LHCL)**. The architecture is +designed so that new processor types can be added in the future to handle other edge cases, keeping the system agile +without requiring changes to the fundamental INDICATOR / NON_INDICATOR framework. + +### The LHCL Use Case: Why NACE/ISIC/KVED Need Special Treatment + +The primary use case today involves economic activity classification systems: + +- **NACE** — the EU standard for classifying economic activities +- **ISIC** — the UN equivalent (International Standard Industrial Classification) +- **KVED** — Ukraine's national adaptation of NACE + +These dimensions sit at an awkward intersection. The underlying concept — economic activity — isn't truly +domain-specific. A layperson understands "manufacturing" or "agriculture". But the *classifications themselves* create +problems for both standard processing paths: + +- **Too large for standard indicator indexing.** These code lists contain hundreds to thousands of items (NACE Rev. 2 + has ~600+ codes across 4 levels). When indexed alongside regular indicators, semantically similar items crowd the + embedding space — "Manufacture of food products", "Manufacture of beverages", "Manufacture of tobacco products" all + cluster together, making it hard for vector search to pick the right one. +- **Too hierarchical for flat search.** Standard indicator search treats all values equally. But NACE codes have a + parent-child structure (e.g., "C - Manufacturing" → "C10 - Manufacture of food products" → "C10.1 - Processing and + preserving of meat"). The right level of specificity depends on the user's query, and flat search loses this + structure. +- **Too numerous for NER.** Standard Named Entity Recognition can handle dimensions with tens of values (countries, + frequencies), but not thousands of nuanced classification codes. + +Result: these dimensions need a specialized processor that combines search with LLM reasoning. + +### How the LHCL Processor Works + +The LHCL processor uses a **two-phase approach** — vector search to narrow candidates, then LLM selection to pick the +right ones: + +1. **Vector search** retrieves the top ~50 candidates (configurable via `top_k`) from the full code list using semantic + similarity. This narrows thousands of codes to a manageable candidate set. +2. **LLM selection** receives the candidate list and the user's query, then selects the most relevant codes. The LLM + picks the right level of specificity — for example, choosing "C - Manufacturing" for a broad query vs. + "C10 - Manufacture of food products" for a specific one. +3. **Grounding** ensures no hallucination — the LLM can only select from the retrieved candidates. Any IDs not present + in the candidate set are automatically discarded. + +The processor's prompt is configured per channel (by the system administrator) and guides the LLM's domain-specific +selection logic — for example, instructing it to prefer more specific codes when multiple overlapping categories match. + +### Example: How a Query Flows Through the LHCL Processor + +**User query:** *"What is the GDP contribution of manufacturing in Ukraine?"* + +1. The system identifies that the dataset has a NACE dimension configured with a special processor +2. **Vector search** retrieves ~50 KVED codes semantically related to "manufacturing" from the code list +3. **LLM selection** reviews the candidates and selects "C - Manufacturing" (or a more specific sub-category like + "C10 - Manufacture of food products" if the query warrants it) +4. The selected code is added to the SDMX query alongside other dimension filters (indicator, country, time period) + +### Admin Configuration + +In the `dimensions` map, declare the special dimension with `dimensionType: "SPECIAL"` and a `processorId`: + +```yaml +dimensions: + NACE: + dimensionType: "SPECIAL" + processorId: "KVED" +``` + +**Key points:** + +- Setting `dimensionType: "SPECIAL"` tells the system to route this dimension through a specialized processor instead + of the standard indicator search pipeline. +- The `processorId` must match a processor configured in the channel's tool configuration by the system administrator. + If it doesn't match, special dimension search fails silently — no error, just no results for that dimension. +- Most datasets don't need SPECIAL dimensions — omit them entirely if no dimensions have large hierarchical code lists. + +See [Module 04](04-dataset-configuration.md#special-dimensions--large-hierarchical-classifications) for field-by-field +details and common mistakes. + +--- + +## Key Takeaways + +- Every dimension must be one of three core types (**INDICATOR**, **NON_INDICATOR**, **TIME_PERIOD**) or the special extensibility type **SPECIAL** +- The core question: *"Would an average person understand this concept?"* — Yes means NON_INDICATOR, No means INDICATOR +- **Misclassification has concrete consequences:** INDICATOR as NON_INDICATOR = values not searchable; NON_INDICATOR as + INDICATOR = entities not recognized by NER +- In the Admin UI, each dimension is explicitly configured with `dimensionType` in the `dimensions` map, along with + settings like `isRequired`, `subtype`, `alias`, and `defaultQueries` +- Every NON_INDICATOR dimension must map to a Named Entity type — add new types when needed +- **Special dimensions** (e.g., NACE, ISIC, KVED) use a dedicated LHCL processor with vector search + LLM selection — + set `dimensionType: "SPECIAL"` with `processorId` +- The same dimension name can be classified differently across datasets depending on its code list values +- When in doubt, classify as INDICATOR +- Different agencies use different dimension IDs for the same concept — use `alias` on the country dimension to unify them + +--- + +## Check Your Understanding + +Test your grasp of dimension classification before moving on. Each question presents what you'd actually see in the Admin UI — dimension ID, concept name, and sample codelist values. + +
+1. Classify this dimension: CURRENCY_TRANS (Concept: "Transaction currency") with codelist values EUR — Euro, USD — US Dollar, GBP — Pound Sterling. INDICATOR or NON_INDICATOR? + +**Answer:** NON_INDICATOR. Apply the core question: *"Would an average person understand this concept?"* — yes, everyone +knows what currencies are. Set `dimensionType: "NON_INDICATOR"` and map it to the "Currency/Unit of measure" Named Entity +type so that NER can recognize currency mentions in user queries like "trade flows in euros." + +
+ +
+2. Classify this dimension: ADJUSTMENT (Concept: "Seasonal adjustment") with codelist values Y — Working day and seasonally adjusted, N — Neither seasonally nor working day adjusted, S — Seasonally adjusted. INDICATOR or NON_INDICATOR? + +**Answer:** INDICATOR. Seasonal adjustment is a statistical method — most users don't know the difference between +"seasonally adjusted" and "working day adjusted," or why it matters. These values describe *how the data was processed*, +which requires domain expertise to interpret. Set `dimensionType: "INDICATOR"`. If you mistakenly set this to +NON_INDICATOR, the adjustment values would never appear in indicator search results, and users searching for +"seasonally adjusted GDP" wouldn't find the right series. + +
+ +
+3. This one is tricky. Classify ACCOUNTING_ENTRY (Concept: "Accounting entry", BIS dataset) with codelist values F — Net flows, S — Stocks. Check the grey areas table in this module if you're unsure. + +**Answer:** NON_INDICATOR. Despite sounding financial, "net flows" and "stocks" are general accounting concepts that an +average person can understand without specialized training — flows are changes over a period, stocks are totals at a point +in time. This is one of the grey area cases where the dimension ID sounds domain-specific but the actual codelist values +are accessible. See the grey areas table above for more examples of cases where intuition can mislead you. + +
+ +
+4. A Eurostat dataset has a dimension ACTIVITY (Concept: "Economic activity") with 615 codelist values organized in a deep hierarchy: A — Agriculture, forestry and fishing → A01 — Crop and animal production → A011 — Growing of non-perennial crops → A0111 — Growing of cereals. What dimension type should this be? + +**Answer:** SPECIAL with a `processorId`. With 600+ values in a deep hierarchical classification (NACE Rev. 2), standard +INDICATOR indexing would create an enormous search space where keyword and semantic search struggle to navigate the +hierarchy. Instead, set `dimensionType: "SPECIAL"` and assign the `processorId` of the LHCL (Large Hierarchical Code List) +processor configured in the channel's tool settings. The LHCL processor uses vector search combined with LLM selection to +navigate the hierarchy effectively. If you classified this as INDICATOR, users searching for "manufacturing output" would +get poor results because the flat indicator index can't capture the parent-child relationships in the NACE tree. + +
+ +
+5. You classify COUNTERPART_AREA (Concept: "Counterpart country") as NON_INDICATOR — the values are country codes like US, DE, JP. The channel currently has these Named Entity types: Country/Reference area, Time frequency, Currency/Unit of measure. What must you do next? + +**Answer:** Add a new Named Entity type to the channel configuration — something like "Counterpart area/country." Every +NON_INDICATOR dimension must map to a Named Entity type so that NER can recognize mentions in user queries. The existing +"Country/Reference area" type is already used by the main `REF_AREA` or `COUNTRY` dimension, so `COUNTERPART_AREA` needs +its own type. If you skip this step, NER won't extract counterpart country mentions from queries like "bilateral trade +with Japan," and the dimension filter won't be applied. + +
+ +
+6. A colleague configured COUNTRY (values: US, DE, FR, JP) as INDICATOR instead of NON_INDICATOR. Users now report that "GDP of Germany" returns no results or wrong results. What went wrong? + +**Answer:** Two things broke simultaneously. First, because COUNTRY is marked as INDICATOR, the NER system doesn't look +for country entities in that dimension — so "Germany" in the user's query is never recognized as a country filter. +Instead, "Germany" gets treated as a search term in the indicator index, where it matches nothing (indicator values are +things like "Gross domestic product" or "Consumer prices," not country names). Second, the actual country codes (US, DE, +FR) are now polluting the indicator search index with entries that aren't real indicators. The fix: change COUNTRY to +`dimensionType: "NON_INDICATOR"` with `subtype: "REGION"`, map it to the "Country/Reference area" Named Entity type, and +reindex the dataset. + +
+ +--- + +**Previous:** [Module 02 — Assessing Datasets for Onboarding](02-dataset-assessment.md) | **Next:** [Module 03b — Indicator Configuration](03b-indicator-configuration.md) diff --git a/learning/administration/03b-indicator-configuration.md b/learning/administration/03b-indicator-configuration.md new file mode 100644 index 0000000..583a9c4 --- /dev/null +++ b/learning/administration/03b-indicator-configuration.md @@ -0,0 +1,628 @@ +# Module 03b: Indicator Configuration + +> **Prerequisite:** This module requires [Module 03a — Dimension Types & Named Entities](03a-dimension-types.md). You should already understand how to classify dimensions as INDICATOR, NON_INDICATOR, or TIME_PERIOD before proceeding. + +## What You'll Learn + +- The concept of required vs. optional indicator dimensions +- The difference between packed and unpacked indicators and the `unpack` indexer setting +- The `useCodeListDescription` indexer setting +- Concrete examples from IMF, Eurostat, ECB, BIS, and other agencies +- How to apply these concepts in practice through exercises + +--- + +## Required vs. Optional Indicator Dimensions + +Once you've identified all INDICATOR dimensions, decide which are **required**: + +- **Required** (`isRequired: true`) — The user's query must specify a filter for at least one + required indicator dimension. Queries without any required indicator dimension filter are rejected. +- **Optional** (no `isRequired`, or `isRequired: false`) — The dimension is optional; + queries can proceed without filtering on it. + +### The Decision Question + +> *"If the user doesn't specify this dimension, can the system still return a meaningful answer?"* +> +> **Yes** → optional. **No** → required. + +### Required — Without It, the Query Is Meaningless + +- WEO `INDICATOR` — *"What is [something] for Germany?"* has no meaning without specifying what indicator. Required. +- CPI `INDEX_TYPE` + `COICOP_1999` — without specifying CPI vs. HICP **and** a product category, *"What is inflation?"* + is too vague. Both required. + +### Optional — Without It, the System Returns Useful (but Less Specific) Data + +- CPI `TYPE_OF_TRANSFORMATION` — if not specified, the system returns "Index" by default. The answer is still + meaningful, just with a default transformation. Optional. +- ECB BSI `ADJUSTMENT` — seasonal adjustment is a refinement, not essential for returning meaningful data. Optional. + +### Rules + +1. **Every dataset must have at least one indicator dimension with `isRequired: true`.** +2. The "main" indicator dimension is almost always required. +3. Supporting dimensions (like TYPE_OF_TRANSFORMATION) are often optional — the system can apply sensible defaults. + +--- + +## Packed vs. Unpacked Indicators + +Now that you understand dimension classification and required vs. optional, there's one more critical structural distinction: whether a dataset's indicator values are **packed** or **unpacked**. This determines the `unpack` indexer setting, which directly affects search quality. + +### What Are Packed Indicators? + +A **packed indicator** combines multiple concepts into a single value, typically separated by commas. + +**Example — IMF WEO (packed):** +``` +INDICATOR dimension values: +- "Gross domestic product, constant prices, Percent change" +- "Gross domestic product, current prices, U.S. dollars" +- "Inflation, average consumer prices, Percent change" +- "Volume of imports of goods and services, Percent change" +``` + +Each value in the INDICATOR dimension packs together: +- **What** is being measured (GDP, Inflation, Volume of imports) +- **How** it's measured (constant prices, current prices) +- **What unit** (Percent change, U.S. dollars) + +### What Are Unpacked Indicators? + +An **unpacked indicator** separates these concepts into individual dimensions. + +**Example — IMF CPI (unpacked):** +``` +INDEX_TYPE dimension values: +- "Consumer price index (CPI)" +- "Harmonised index of consumer prices (HICP)" + +COICOP_1999 dimension values: +- "All items" +- "Food and non-alcoholic beverages" +- "Clothing and footwear" +- "Housing, water, electricity, gas and other fuels" + +TYPE_OF_TRANSFORMATION dimension values: +- "Index" +- "Percentage change, previous period" +- "Weight" +``` + +Here, what is being measured is split across three separate indicator dimensions. No single dimension value contains comma-separated multi-concept strings. + +### How to Identify Packed Indicators + +Now that you know how to classify dimensions, identifying packed vs. unpacked is straightforward: + +1. **Look at the indicator dimensions you classified** — focus on the "important" ones (those describing *what* is being measured, not *how*) +2. **Check their codelist values for comma-separated multi-concept strings:** + - `"GDP, current prices, annual, USD"` — **Packed** (multiple concepts in one value) + - `"Consumer price index (CPI)"` — **Unpacked** (single concept) +3. **Cross-check with the number of indicator dimensions:** + - Packed datasets typically have **fewer indicator dimensions** (often just one) with many values each + - Unpacked datasets typically have **multiple indicator dimensions**, each with fewer values + +### Impact on Configuration + +| Aspect | Packed | Unpacked | +|--------|--------|----------| +| Indexer `unpack` setting | `true` | `false` | +| Number of indicator dimensions | Usually 1-2 | Usually 2-4+ | +| Code list item length | Long, multi-concept strings | Short, single-concept strings | +| Example datasets | IMF WEO, IMF BOP, IMF ANEA | IMF CPI, Eurostat NAMA_10_GDP, ECB BSI | + +**Setting `unpack: true`** tells the indexer to decompose packed indicator names into individual concepts during indexing, improving search accuracy. Setting it incorrectly degrades search quality. + +### Real-World Examples + +The following table shows actual datasets and their `unpack` settings. Study these to develop pattern recognition: + +| Dataset | Provider | `unpack` | Key Evidence | +|---------|----------|----------|-------------| +| WEO | IMF | `true` | Values like "GDP, constant prices, Percent change" — multi-concept packed strings | +| BOP | IMF | `true` | Both INDICATOR and BOP_ACCOUNTING_ENTRY have multi-concept values combining what + how | +| ANEA | IMF | `true` | INDICATOR values are packed multi-concept strings | +| WDI | World Bank | `true` | Values like "GDP (current US$)" pack indicator + unit into one string | +| CPI | IMF | `false` | Each indicator dim has single-concept values — "Consumer price index (CPI)", "All items", "Index" | +| EER | IMF | `true` | Values like "Real effective exchange rate (REER), Index (2010=100) Adjusted by relative consumer prices" — packs type, index spec, and adjustment method | +| ER | IMF | `false` | Despite commas in values, they are natural descriptions (see [Grey Area](#grey-area-semi-packed-indicators) below) | +| NAMA_10_GDP | Eurostat | `false` | Single-concept values in each dimension | +| BSI | ECB | `false` | 7 indicator dims but each has single-concept values like "Loans", "Deposits" | +| TiVA | OECD | `false` | Single-concept values — trade measures and activity sectors | +| Debt Securities | BIS | `false` | 7 indicator dims with single-concept values | + +### The Decision Algorithm + +Follow this algorithm during manual configuration: + +1. **Start with your indicator dimensions** — the dimensions you classified as INDICATOR in the steps above +2. **Among them, find the "important" ones** — those describing *what* is being measured, not *how* (e.g., INDICATOR is important; TYPE_OF_TRANSFORMATION is less important) +3. **Look at their codelist values** — inspect sample values +4. **If any important dim has comma-separated multi-concept values** → set `unpack: true` + - Example: `"GDP, current prices, annual, USD"` — multiple concepts packed together +5. **If all values are single concepts** → set `unpack: false` + - Example: `"Consumer price index (CPI)"` — one concept, even though it has parentheses + +### Grey Area: Semi-Packed Indicators + +Some datasets fall between the extremes. The rule of thumb: + +> Look at the **important** indicator dimensions — the ones describing *what* is being measured (not *how*). If at least one important indicator dimension has values with multiple concepts packed together (often comma-separated), set `unpack: true`. + +**Teaching case — IMF ER (Exchange Rates):** + +The ER dataset's INDICATOR dimension contains values like: +- `"US dollar exchange rate, period average"` +- `"US dollar exchange rate, end of period"` +- `"SDR exchange rate, period average"` + +These contain commas, which might suggest packing. But look more carefully: `"US dollar exchange rate, period average"` is a *natural description of a single concept* — the period-average exchange rate against the US dollar. The comma separates a noun phrase from a temporal clarifier, not two independent concepts. + +Compare with WEO's `"Gross domestic product, constant prices, Percent change"` — here the commas separate three genuinely independent concepts (what, price basis, unit). + +**Result:** ER uses `unpack: false`. The config confirms this with the comment `# No need to unpack`. + +**Rule of thumb:** If the comma is part of natural English phrasing describing one thing, it's not packed. If the comma separates independent concepts that could each be a separate dimension, it's packed. + +--- + +## The `useCodeListDescription` Setting + +You'll see `useCodeListDescription` in indexer configurations alongside `unpack`: + +- **What it does:** When set to `true`, the indexer includes code list item *descriptions* (not just names) in the search index. This gives the semantic and keyword search more text to match against. +- **When to use it:** Currently set to `true` in all IMF dataset configurations. Follow this pattern for new datasets when the provider includes meaningful descriptions in their code lists. +- **Distinct from `indexer.description`:** The `indexer.description` field describes the *entire dataset* (used for dataset selection). `useCodeListDescription` controls whether *individual code list item descriptions* are included in the indicator search index. + +```yaml +indexer: + description: "Dataset-level description used for dataset selection" # indexer.description + indicator: + unpack: true + useCodeListDescription: true # includes code list item descriptions in the search index +``` + +--- + +## Concrete Examples + +### IMF WEO — Simple Structure + +| Dimension | Type | Required? | Reasoning | +|-------------|-------------------------|-----------|---------------------------------------------| +| INDICATOR | INDICATOR | Yes | Main (and only) indicator — always required | +| COUNTRY | NON_INDICATOR (country) | — | Countries are universally understood | +| FREQUENCY | NON_INDICATOR | — | Frequency is universally understood | +| TIME_PERIOD | TIME_PERIOD | — | Time dimension | + +```yaml +dimensions: + INDICATOR: + dimensionType: "INDICATOR" + isRequired: true + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" +``` + +WEO uses `unpack: true` — the INDICATOR values are packed multi-concept strings like "GDP, constant prices, Percent change". + +### IMF BOP — Multiple Required Indicators + NON_INDICATOR UNIT + +| Dimension | Type | Required? | Reasoning | +|----------------------|-------------------------|-----------|---------------------------------------------------------------------| +| INDICATOR | INDICATOR | Yes | Main indicator | +| BOP_ACCOUNTING_ENTRY | INDICATOR | Yes | Credit vs. Debit is essential context | +| COUNTRY | NON_INDICATOR (country) | — | Countries | +| FREQUENCY | NON_INDICATOR | — | Frequency | +| UNIT | NON_INDICATOR | — | Values like "US Dollars", "Percent of GDP" — universally understood | +| TIME_PERIOD | TIME_PERIOD | — | Time dimension | + +```yaml +dimensions: + INDICATOR: + dimensionType: "INDICATOR" + isRequired: true + BOP_ACCOUNTING_ENTRY: + dimensionType: "INDICATOR" + isRequired: true + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + UNIT: + dimensionType: "NON_INDICATOR" +``` + +BOP uses `unpack: true` — both INDICATOR and BOP_ACCOUNTING_ENTRY have multi-concept packed values like "Goods, debit" and "Credit, debit and balance". + +### IMF CPI — Complex with Optional Indicators + +| Dimension | Type | Required? | Reasoning | +|------------------------|-------------------------|-----------|---------------------------------------------| +| INDEX_TYPE | INDICATOR | Yes | CPI vs HICP — essential | +| COICOP_1999 | INDICATOR | Yes | Product category — essential | +| TYPE_OF_TRANSFORMATION | INDICATOR | No | Index vs. Percent change — optional context | +| COUNTRY | NON_INDICATOR (country) | — | Countries | +| FREQUENCY | NON_INDICATOR | — | Frequency | +| TIME_PERIOD | TIME_PERIOD | — | Time dimension | + +```yaml +dimensions: + INDEX_TYPE: + dimensionType: "INDICATOR" + isRequired: true + COICOP_1999: + dimensionType: "INDICATOR" + isRequired: true + TYPE_OF_TRANSFORMATION: + dimensionType: "INDICATOR" # Not required — optional refinement + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" +``` + +Note that TYPE_OF_TRANSFORMATION has `dimensionType: "INDICATOR"` but no `isRequired: true` — +making it optional. + +CPI uses `unpack: false` — each indicator dimension has single-concept values like "Consumer price index (CPI)", "All items", "Index". + +### Eurostat NAMA_10_GDP — Different Naming Conventions + +| Dimension | Type | Required? | Reasoning | +|-------------|-------------------------|-----------|--------------------------------------------------------------------------| +| na_item | INDICATOR | Yes | National accounts item — main indicator | +| unit | INDICATOR | No | Measurement methodology (e.g., "Chain linked volumes") — domain-specific | +| geo | NON_INDICATOR (country) | — | Geographic area | +| freq | NON_INDICATOR | — | Frequency | +| TIME_PERIOD | TIME_PERIOD | — | Time dimension | + +```yaml +dimensions: + na_item: + dimensionType: "INDICATOR" + isRequired: true + unit: + dimensionType: "INDICATOR" # Not required — optional refinement + geo: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + freq: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" +``` + +**Contrast with IMF BOP:** In BOP, `UNIT` is NON_INDICATOR because its values are currencies and percentages. In +Eurostat NAMA_10_GDP, `unit` is INDICATOR because its values are economics measurement concepts like "Chain linked +volumes" and "Current prices". **The same concept can be classified differently across datasets depending on its actual +values.** + +### ECB BSI — Many Dimensions + +| Dimension | Type | Required? | Reasoning | +|-----------------|-------------------------|-----------|--------------------------------| +| BS_ITEM | INDICATOR | Yes | Balance sheet item | +| BS_REP_SECTOR | INDICATOR | Yes | Reporting sector | +| DATA_TYPE | INDICATOR | Yes | Data type | +| BS_COUNT_SECTOR | INDICATOR | Yes | Counterpart sector (financial) | +| ADJUSTMENT | INDICATOR | No | Seasonal adjustment | +| MATURITY_ORIG | INDICATOR | No | Maturity | +| BS_SUFFIX | INDICATOR | No | Balance sheet suffix | +| REF_AREA | NON_INDICATOR (country) | — | Reference area | +| FREQ | NON_INDICATOR | — | Frequency | +| COUNT_AREA | NON_INDICATOR | — | Counterpart area | +| CURRENCY_TRANS | NON_INDICATOR | — | Currency | +| TIME_PERIOD | TIME_PERIOD | — | Time dimension | + +This dataset has 7 indicator dimensions (4 required, 3 optional) and 4 non-indicator dimensions. Despite the many indicator dimensions, BSI uses `unpack: false` — each dimension has single-concept values like "Loans", "Deposits", "Outstanding amounts". + +### BIS Debt Securities — COUNTERPART dimensions as NON_INDICATOR + +| Dimension | Type | Required? | Reasoning | +|--------------------|-------------------------|-----------|-----------------------| +| STO | INDICATOR | Yes | Stock/flow concept | +| REF_SECTOR | INDICATOR | Yes | Reference sector | +| INSTR_ASSET | INDICATOR | Yes | Instrument/asset type | +| MATURITY | INDICATOR | No | Maturity | +| CURRENCY_DENOM | INDICATOR | No | Currency denomination | +| VALUATION | INDICATOR | No | Valuation method | +| CUST_BREAKDOWN | INDICATOR | No | Customer breakdown | +| REF_AREA | NON_INDICATOR (country) | — | Reference area | +| FREQ | NON_INDICATOR | — | Frequency | +| COUNTERPART_AREA | NON_INDICATOR | — | Counterpart area | +| COUNTERPART_SECTOR | NON_INDICATOR | — | Counterpart sector | +| ADJUSTMENT | NON_INDICATOR | — | Adjustment type | +| CONSOLIDATION | NON_INDICATOR | — | Consolidation basis | +| ACCOUNTING_ENTRY | NON_INDICATOR | — | Accounting entry | +| EXPENDITURE | NON_INDICATOR | — | Expenditure type | +| UNIT_MEASURE | NON_INDICATOR | — | Unit of measure | +| PRICES | NON_INDICATOR | — | Price type | +| TRANSFORMATION | NON_INDICATOR | — | Transformation type | +| TIME_PERIOD | TIME_PERIOD | — | Time dimension | + +Note: BIS classifies ADJUSTMENT as NON_INDICATOR while ECB BSI classifies it as INDICATOR — the classification depends +on the specific code list values in each dataset. + +### Cross-Agency Dimension Naming Comparison + +The same concept often has different dimension IDs across agencies: + +| Concept | IMF | Eurostat | World Bank | OECD | ECB | BIS | +|----------------|-----------------------|-----------|------------|--------------------|--------------|--------------------| +| Country | `COUNTRY` | `geo` | `REF_AREA` | `REF_AREA` | `REF_AREA` | `REF_AREA` | +| Frequency | `FREQUENCY` | `freq` | `FREQ` | `FREQ` | `FREQ` | `FREQ` | +| Main indicator | `INDICATOR` | `na_item` | `SERIES` | `MEASURE` | `BS_ITEM` | `STO` | +| Counterpart | `COUNTERPART_COUNTRY` | — | — | `COUNTERPART_AREA` | `COUNT_AREA` | `COUNTERPART_AREA` | + +This is why the `alias` field on the country dimension exists — it unifies different dimension names across datasets within a +channel (e.g., both `COUNTRY` and `geo` can be aliased to "Country/Reference area"). + +--- + +## Check Your Understanding + +Classify each dimension using the information an admin actually sees — dimension ID, concept name, and codelist values. + +**1. You see this dimension in a new dataset. INDICATOR or NON_INDICATOR?** + +| | | +|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------| +| **Dimension ID** | `SECTOR` | +| **Concept name** | Institutional sector | +| **Codelist values (sample)** | `S1` — Total economy, `S11` — Non-financial corporations, `S121` — Central bank, `S128` — Insurance corporations, `S1311` — Central government | + +
+Answer + +INDICATOR. Don't be misled by the simple dimension ID `SECTOR`. The concept name "Institutional sector" hints at domain +specificity, but the codelist values confirm it — codes like `S121`, `S128`, `S1311` with labels like "Non-financial +corporations" and "Insurance corporations" require knowledge of the System of National Accounts (SNA) sector +classification. Set `dimensionType: "INDICATOR"` in the `dimensions` map. +
+ +**2. Users report searching for "unemployment" returns no results, but the dataset has an unemployment indicator. You +check the config and see:** + +```yaml +dimensions: + FREQ: + dimensionType: "INDICATOR" + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" +``` + +The dataset dimensions are: `INDICATOR` (concept: "Subject"), `COUNTRY`, `FREQ`, `TIME_PERIOD`. What went wrong? + +
+Answer + +`INDICATOR` (the subject dimension with values like "Unemployment rate") is not in the `dimensions` map with `dimensionType: "INDICATOR"` — so its values +are never indexed for search. Meanwhile, `FREQ` is incorrectly configured as an indicator dimension. Fix: set `INDICATOR` to +`dimensionType: "INDICATOR"` with `isRequired: true`, change `FREQ` to `dimensionType: "NON_INDICATOR"` with `subtype: "FREQUENCY"`, and reindex the dataset. +
+ +**3. Classify this dimension:** + +| | | +|------------------------------|----------------------------------------------------------------------------| +| **Dimension ID** | `UNIT_MEASURE` | +| **Concept name** | Unit of measure | +| **Codelist values (sample)** | `USD` — US Dollar, `EUR` — Euro, `PC` — Percent, `PC_GDP` — Percent of GDP | + +
+Answer + +NON_INDICATOR. The concept name "Unit of measure" could go either way, but the codelist values are decisive — +currencies (USD, EUR) and basic units (Percent, Percent of GDP) are universally understood. Map it to the "Currency/Unit +of measure" Named Entity type. +
+ +**4. A CPI dataset has three indicator dimensions. An admin marked all three as required. You review:** + +| Dimension ID | Concept name | Codelist values (sample) | Required? | +|--------------------------|----------------|-------------------------------------------------------------------------------------------|-----------| +| `INDEX_TYPE` | Index type | `CPI` — Consumer Price Index, `HICP` — Harmonized Index of Consumer Prices | Yes | +| `COICOP_1999` | COICOP 1999 | `011` — Food, `0451` — Electricity, `0722` — Passenger transport by air | Yes | +| `TYPE_OF_TRANSFORMATION` | Transformation | `IX` — Index, `PC_CP_A_PT` — Percentage change over corresponding period of previous year | Yes | + +Should all three be required? + +
+Answer + +No — `TYPE_OF_TRANSFORMATION` should be optional. `INDEX_TYPE` is essential (CPI vs. HICP changes what data you get), +and `COICOP_1999` is essential (which product category?). But `TYPE_OF_TRANSFORMATION` is a refinement — the system can +default to "Index" (`IX`) and still return meaningful data. Apply the decision question: *"Can the system still return a +meaningful answer without it?"* — yes, so remove `isRequired: true` from TYPE_OF_TRANSFORMATION. +
+ +**5. You're onboarding a new dataset. Classify this dimension and decide what to do next:** + +| | | +|------------------------------|-----------------------------------------------------------------------------------------------------------------------| +| **Dimension ID** | `COUNTERPART_SECTOR` | +| **Concept name** | Counterpart institutional sector | +| **Codelist values (sample)** | `S1` — Total economy, `S11` — Non-financial corporations, `S12K` — Non-MMF investment funds, `S2` — Rest of the world | + +The channel has these Named Entity types: `Country/Reference area`, `Time frequency`, `Currency/Unit of measure`, +`Counterpart area/country`. + +
+Answer + +INDICATOR. The codelist values use SNA sector codes (`S11`, `S12K`) that require domain knowledge — this is the same +classification system as question 1. Set `dimensionType: "INDICATOR"`. No Named Entity type mapping needed since it's not +NON_INDICATOR. If you mistakenly classified it as NON_INDICATOR, you'd also need to add a new Named Entity type like " +Counterpart sector" to the channel — but the codelist values clearly point to INDICATOR. +
+ +**6. Two datasets both have a dimension called `UNIT`. Classify each:** + +**Dataset A:** + +| | | +|-------------------------|----------------------------------------------------------| +| **Dimension ID** | `UNIT` | +| **Concept name** | Currency | +| **Codelist values** | `USD` — US Dollar, `EUR` — Euro, `GBP` — Pound Sterling | + +**Dataset B:** + +| | | +|------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------| +| **Dimension ID** | `UNIT` | +| **Concept name** | Unit of measure | +| **Codelist values (sample)** | `CLV15_MEUR` — Chain linked volumes (2015), million euro, `CP_MEUR` — Current prices, million euro, `PD15_EUR` — Price deflator (2015), euro | + +
+Answer + +Different classification despite the same dimension ID. Dataset A: NON_INDICATOR — concept name "Currency" confirms it, +and codelist values are universally understood currency codes. Dataset B: INDICATOR — "Chain linked volumes", "Price +deflator" are economics measurement concepts that require domain knowledge. This is why you must always check codelist +values, not just the dimension ID. +
+ +**7. An INDICATOR dimension has values like "Exports of goods and services, Percent of GDP". Packed or unpacked?** + +
+Answer + +**Packed.** The value combines what is being measured ("Exports of goods and services") with a unit/transformation ("Percent of GDP") — two independent concepts separated by a comma. Set `unpack: true`. +
+ +**8. An INDICATOR dimension has values like "US dollar exchange rate, period average". Packed or unpacked?** + +
+Answer + +**Unpacked** despite the comma. "US dollar exchange rate, period average" is a natural English description of a single concept — the period-average USD exchange rate. The comma separates a noun phrase from a temporal clarifier, not two independent concepts. Set `unpack: false`. +
+ +**9. A dataset has 7 indicator dimensions, each with single-concept values like "Loans", "Deposits", "Outstanding amounts". Is this packed because there are so many dimensions?** + +
+Answer + +**No — this is unpacked.** Many dimensions with single-concept values ≠ packing. Packed vs. unpacked is about the *structure of values*, not the *number of dimensions*. Each value describes one concept, so set `unpack: false`. This is exactly how ECB BSI works — 7 indicator dimensions, all unpacked. +
+ +--- + +## Key Takeaways + +- **Required indicator dimensions** (`isRequired: true`) are those without which a query is meaningless; optional ones are refinements with + sensible defaults +- At least one indicator dimension must have `isRequired: true` +- **Packed vs. unpacked** is the most important structural distinction for indicator dimensions — it determines the `unpack` indexer setting +- Packed indicators combine multiple concepts in comma-separated values → set `unpack: true` +- Unpacked indicators have single-concept values across multiple dimensions → set `unpack: false` +- Commas in natural English descriptions (e.g., "exchange rate, period average") do not indicate packing +- Use `useCodeListDescription: true` when the provider includes meaningful code list descriptions + +--- + +## Practical Exercises + +Apply dimension classification and packed/unpacked analysis to real dataset metadata. Each exercise shows what you'd see when inspecting a dataset — dimension IDs, concept names, and sample codelist values. + +### Exercise 1: ECB BSI (Balance Sheet Items) + +You open the Admin UI Dataset Wizard and select the ECB BSI dataflow. The structure step shows: + +**Dimensions:** + +| Dimension ID | Concept Name | Sample Codelist Values | +|---|---|---| +| `BS_ITEM` | Balance sheet item | "Loans", "Debt securities held", "Deposits", "Total assets/liabilities" | +| `ADJUSTMENT` | Adjustment indicator | "Neither seasonally nor working day adjusted", "Working day adjusted" | +| `BS_REP_SECTOR` | Balance sheet reporting sector | "Domestic (home or reference area)", "Monetary financial institutions" | +| `MATURITY_ORIG` | Original maturity | "Total", "Up to 1 year", "Over 1 year and up to 2 years" | +| `DATA_TYPE` | Data type | "Outstanding amounts", "New business", "Transactions" | +| `BS_COUNT_SECTOR` | Balance sheet counterpart sector | "Domestic (home or reference area)", "Non-financial corporations" | +| `BS_SUFFIX` | Series variation - Loss and impairment suffix | "Original maturity", "Residual maturity" | +| `REF_AREA` | Reference area | "AT", "BE", "DE", "FR", ... | +| `FREQ` | Frequency | "A", "M", "Q" | +| `CURRENCY_TRANS` | Currency of transaction | "EUR", "USD", "GBP" | +| `TIME_PERIOD` | Time period | "2020", "2021-Q1", "2023-06" | + +**Your task:** Classify each dimension, determine packed/unpacked, and decide the `unpack` setting. + +
+Solution + +**Dimension classification:** +- **Indicator dims:** BS_ITEM, ADJUSTMENT, BS_REP_SECTOR, MATURITY_ORIG, DATA_TYPE, BS_COUNT_SECTOR, BS_SUFFIX — all describe aspects of *what* is being measured or *how* the measurement is characterized +- **Non-indicator dims:** REF_AREA (country/region), FREQ (frequency), CURRENCY_TRANS (currency context) +- **Time dim:** TIME_PERIOD + +**Packed/Unpacked:** **Unpacked** (`unpack: false`). Look at the codelist values: "Loans", "Deposits", "Outstanding amounts", "Non-financial corporations" — each is a single concept. No comma-separated multi-concept strings. The 7 indicator dimensions may seem like a lot, but many dimensions with single-concept values ≠ packing. It means the dataset has a complex but well-structured dimensionality. + +**Configuration notes:** +- 7 indicator dimensions means many possible combinations — plan thorough testing in [Module 07](07-testing-and-validation.md) +- All codelist values are descriptive English names (no generic codes) — no assessment Blockers + +
+ +### Exercise 2: IMF BOP (Balance of Payments) + +You open the Admin UI Dataset Wizard and select the IMF BOP dataflow. The structure step shows: + +**Dimensions:** + +| Dimension ID | Concept Name | Sample Codelist Values | +|---|---|---| +| `INDICATOR` | Indicator | "Goods, debit", "Current Account, Total, Net", "Financial Account, Direct Investment, Assets, Net Incurrence of Liabilities", "Services, Credit" | +| `BOP_ACCOUNTING_ENTRY` | BOP accounting entry | "Credit, debit and balance", "Supplementary items", "As a ratio to GDP" | +| `COUNTRY` | Reference area | "US", "DE", "JP", "FR", ... | +| `FREQUENCY` | Frequency | "A", "Q", "M" | +| `UNIT` | Unit | "US Dollars", "National Currency" | +| `TIME_PERIOD` | Time period | "2020", "2023-Q1", "2024-06" | + +**Your task:** Classify each dimension, determine packed/unpacked, and decide the `unpack` setting. + +
+Solution + +**Dimension classification:** +- **Indicator dims:** INDICATOR, BOP_ACCOUNTING_ENTRY — both describe *what* is being measured +- **Non-indicator dims:** COUNTRY (region), FREQUENCY, UNIT (values like "US Dollars" are universally understood) +- **Time dim:** TIME_PERIOD + +**Packed/Unpacked:** **Packed** (`unpack: true`). Look at the INDICATOR values: +- `"Goods, debit"` — packs the concept (Goods) with the accounting direction (debit) +- `"Financial Account, Direct Investment, Assets, Net Incurrence of Liabilities"` — packs multiple hierarchy levels and accounting concepts into one comma-separated string + +BOP_ACCOUNTING_ENTRY also has multi-concept values like `"Credit, debit and balance"`. + +Compare with the ECB BSI exercise: BSI's values like "Loans" and "Outstanding amounts" are each one concept. BOP's values like "Current Account, Total, Net" pack multiple concepts together. + +**Configuration:** +```yaml +indexer: + indicator: + unpack: true + useCodeListDescription: true +``` + +
+ +--- + +**Previous:** [Module 03a — Dimension Types & Named Entities](03a-dimension-types.md) | **Next:** [Module 04 — Configuring a Dataset](04-dataset-configuration.md) diff --git a/learning/administration/04-dataset-configuration.md b/learning/administration/04-dataset-configuration.md new file mode 100644 index 0000000..4b44940 --- /dev/null +++ b/learning/administration/04-dataset-configuration.md @@ -0,0 +1,819 @@ +# Module 04: Configuring a Dataset + +## What You'll Learn + +- The Admin UI workflow for adding a dataset +- The complete dataset configuration YAML — field by field +- How to fill each field with correct values +- Annotated examples: IMF WEO, IMF BOP, Eurostat NAMA_10_GDP +- Virtual dimensions for single-country datasets +- Common mistakes and how to avoid them + +--- + +> **Prerequisite:** This module assumes a Data Source already exists. To set up a new Data Source, see [Module 05 — Data Sources](05-data-sources-and-channels.md). + +## Admin UI Workflow + +Adding a dataset in the StatGPT Admin UI follows a wizard: + +1. **Navigate** to the "Datasets" tab and click "Add" +2. **Select the Data Source** from the table (e.g., the IMF SDMX 2.1 data source) and click "Next" +3. **Select the Dataflow** from the list of available dataflows in that data source — this selects the URN +4. **Fill in the configuration YAML** and save + +The URN is selected in step 3 (you don't type it manually), but it appears in the configuration YAML for reference. The rest of the configuration is what you need to reason about and fill in. + +--- + +## Configuration YAML — Field by Field + +Below is the complete set of fields you'll fill in the Admin UI YAML editor. We'll walk through each one. + +### `urn` — Dataset Identifier + +The SDMX URN of the dataflow. This is pre-filled from your selection in the wizard. + +```yaml +urn: + agency_id: "IMF.RES" + resource_id: "WEO" + version: "latest" +``` + +Format: an object with `agency_id`, `resource_id`, and `version` fields. + +The `version` field defaults to `"latest"`, which tells StatGPT to always track the current published version of the +dataflow. The system resolves `"latest"` to a concrete version (e.g., `"9.0.0"`) at indexing time and stores it +internally. Pinned versions like `"9.0.0"` are supported but not recommended — they require manual updates when the +provider releases a new version. Using `"latest"` also enables +[auto-update](06-indexing-and-operations.md#auto-update) to automatically detect upstream changes. + +### `citation` — Data Source Attribution + +Tells users where the data comes from: + +```yaml +citation: + provider: IMF.RES # Data provider name or ID + url: https://data.imf.org/en/datasets/IMF.RES:WEO # Link to the dataset's web page + description: &weo_description > # Description of the dataset + The World Economic Outlook (WEO) database contains selected macroeconomic + data series from the statistical appendix of the World Economic Outlook + report... +``` + +**`description` field decision:** + +``` +Source dataflow has a meaningful description? +├── Yes → citation.description: null (fetched from source on each access) +│ indexer.description: copy the source description verbatim +└── No / Missing / Vague → + Write a custom description + citation.description: &anchor > "Your description..." + indexer.description: *anchor (reuse via YAML anchor) +``` + +**Key rules:** +- Citation and indexer description decisions are **independent** — citation controls user-facing attribution, indexer controls what gets searched +- Both paths must result in a **non-empty `indexer.description`** +- When `citation.description: null`, write `indexer.description` independently — do NOT use a YAML anchor pointing to the null citation description (the anchor resolves to `null`, making indexer description empty) +- When you write a custom citation description, use a YAML anchor (e.g., `&weo_description`) so you can reuse the same text in `indexer.description` + +> **Multi-agency datasets:** For datasets that aggregate data from multiple providers, `citation` supports additional +> fields: `providerTemplate` (a template string with `{n_agencies}` and `{agencies_sample}` placeholders) and +> `providerAgencies` (a list of `{id, count, name}` objects). These are advanced — most single-agency datasets only +> need `provider`, `url`, and `description`. + +### `isOfficial` + +```yaml +isOfficial: false +``` + +Set to `true` for official national-level data sources (e.g., a national statistics office). Official datasets are prioritized in answers for country-specific queries. Most international organization datasets (IMF, World Bank, Eurostat) are `false`. + +### `useTitleFromSrc` + +```yaml +useTitleFromSrc: true +``` + +- `true` — Use the dataflow title from the SDMX source metadata +- `false` — Use a custom title (you provide it when creating the dataset in the UI) + +Set to `true` when the source dataflow has a good, descriptive title. Set to `false` when you need to override with something clearer. + +### `dimensions` — Unified Dimension Configuration + +The `dimensions` field is a map where each key is a dimension ID and each value configures that dimension's type and behavior. This is where you declare which dimensions are indicators, which is the country dimension, which is the time dimension, etc. + +```yaml +dimensions: + INDICATOR: + dimensionType: "INDICATOR" + isRequired: true + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["-5y", "+2y"] + operator: "between" +``` + +#### Per-Dimension Fields + +| Field | Description | +|-------|-------------| +| `dimensionType` | **Required.** One of: `"INDICATOR"`, `"NON_INDICATOR"`, `"TIME_PERIOD"`, `"SPECIAL"`. See [Module 03a](03a-dimension-types.md). | +| `isRequired` | For INDICATOR dimensions only. Set to `true` if the user's query must filter on this dimension. At least one INDICATOR dimension must be required. | +| `subtype` | For NON_INDICATOR dimensions. `"REGION"` for the country/region dimension, `"FREQUENCY"` for the frequency dimension. | +| `alias` | For the country dimension (`subtype: "REGION"`). A human-readable alias that unifies country dimension names across datasets in a channel (e.g., `"Country/Reference area"`). | +| `allValues` | For the country dimension. A synthetic value representing "all countries" for star-queries. Object with `id`, `name`, and `description`. | +| `defaultQueries` | Default query filters applied when the user doesn't specify a value. Can be set on any dimension. Array of objects with `values` and `operator`. | +| `processorId` | For SPECIAL dimensions only. References a processor configured by the system administrator at the channel/tool level (e.g., `"KVED"` for NACE/ISIC/KVED hierarchies). | +| `virtual` | For virtual dimensions (see [Virtual Dimensions](#virtual-dimensions)). Object with `name`, `description`, and `value` (containing `id`, `name`, `description`). | + +#### `dimensionType` Values + +| Value | Usage | +|-------|-------| +| `"INDICATOR"` | Dimensions describing what is being measured. Values are indexed for search. | +| `"NON_INDICATOR"` | General, universally understood dimensions (country, frequency, currency). Matched via NER. | +| `"TIME_PERIOD"` | The temporal dimension. Exactly one per dataset. | +| `"SPECIAL"` | Dimensions with large hierarchical code lists (NACE, ISIC, KVED) that need a specialized processor. Must also have `processorId`. | + +#### `defaultQueries` Operators + +| Operator | Use Case | Example | +|----------|----------|---------| +| `between` | Continuous range (typically TIME_PERIOD) | `values: ["2020", "2025"]` | +| `in` | Non-contiguous value list | `values: ["A", "Q"]` | +| `equals` | Single default value | `values: ["A"]` | + +**Common patterns:** +- **TIME_PERIOD range:** `values: ["2020", "2025"], operator: between` — last ~5 years +- **TIME_PERIOD with forecasts** (e.g., WEO): include future years in the range +- **Relative expressions:** `values: ["-5y", "now"]` — dynamic range +- **Frequency filter:** `values: ["A", "Q"], operator: in` — restrict to annual and quarterly by default +- **Single value:** `values: ["A"], operator: equals` — default to annual frequency + +#### Special Dimensions — Large Hierarchical Classifications + +Some indicator dimensions use very large hierarchical code lists (hundreds or thousands of items) — for example, NACE economic activity codes, ISIC industry codes, or KVED codes. Standard search indexing can't handle these effectively. Setting `dimensionType: "SPECIAL"` with a `processorId` configures a specialized LLM-powered processor to navigate these hierarchies. + +```yaml +dimensions: + NACE: + dimensionType: "SPECIAL" + processorId: "KVED" +``` + +**When to use:** When a dimension uses an industry/activity classification system like NACE, ISIC, or KVED with a deep hierarchy. + +**Important notes:** +- The `processorId` must reference a processor that exists in the channel's tool configuration. If it doesn't match, special dimension search fails silently. Verify with your system administrator. +- Most datasets don't need SPECIAL dimensions — omit them entirely if no dimensions have large hierarchical code lists. + +See [Module 03a — Special Dimensions](03a-dimension-types.md#special-dimensions) for the conceptual explanation. + +### `includeAttributes` — SDMX Attributes in Agent Context + +Lists SDMX attributes to include alongside data when presenting results to the AI agent: + +```yaml +includeAttributes: + - SCALE # e.g., "Millions", "Billions" + - UNIT # e.g., "US Dollars", "Percent" + - PUBLISHER # Publishing organization + - SOURCE # Data source reference +``` + +These give the agent context about the data values (units, scale, source) so it can describe results accurately. Set to `null` if no relevant attributes exist. + +### `pinnedColumns` — Column Display Order + +Defines which columns appear in the data table and their order, from **least important to most important** (left to right): + +```yaml +pinnedColumns: + - FREQUENCY_Name + - COUNTRY_Name + - INDICATOR_Name +``` + +**Rules:** +- List ALL dataset dimensions except TIME_PERIOD +- Append `_Name` to each dimension ID (e.g., `INDICATOR` → `INDICATOR_Name`) +- Case must match the actual dimension ID (e.g., `freq_Name` for Eurostat, `FREQUENCY_Name` for IMF) +- Order from least to most important — the main indicator dimension typically goes last + +### `updatedAt` — Last Updated Timestamp + +Optional. Configures how to determine when the dataset was last updated. Supports multiple sources checked in order: + +```yaml +updatedAt: + - source: "attribute" + field: "UPDATED" + formats: ["%d.%m.%Y", "%Y-%m-%d"] + - source: "annotation" + field: "lastUpdatedAt" +``` + +| Field | Description | +|-------|-------------| +| `source` | Where to look: `"annotation"` (SDMX annotation), `"attribute"` (SDMX attribute), or `"citation"` (citation metadata) | +| `field` | The annotation ID, attribute ID, or citation field to read | +| `formats` | Optional. Date format strings for parsing (Python `strftime` format) | + +Multiple entries are checked in order — the first one that returns a value is used. + +### `indexer` — Search Index Configuration + +Controls how the dataset is indexed for semantic and keyword search: + +```yaml +indexer: + indicator: + unpack: true # true for packed indicators, false for unpacked + useCodeListDescription: false # Use code list descriptions during indexing + description: *weo_description # Dataset description for indexing (must be non-empty) +``` + +| Field | Description | +|-------|-------------| +| `indicator.unpack` | `true` for packed indicators (comma-separated multi-concept values like "GDP, current prices, USD"). `false` for unpacked. See [Module 03b](03b-indicator-configuration.md#packed-vs-unpacked-indicators). | +| `indicator.useCodeListDescription` | `true` to include code list item descriptions in the index. Improves search when descriptions are meaningful. | +| `indicator.superPrimary` | Advanced. Default `false`. When `true` (only applies when `unpack: false`), the primary indicator label is concatenated from the first 3 indicator dimensions instead of just the first one. | +| `indicator.annotations` | Advanced. Optional object with a `description` field specifying an SDMX annotation name to use as the indicator description in the index. | +| `description` | **Must be a non-empty string.** If citation description is `null`, copy the dataflow description from the source metadata here. If you wrote a custom citation description, reference it with a YAML anchor (`*weo_description`). | + +> **Indexing-relevant vs. display-only fields:** Some fields affect the search index and require reindexing when +> changed: `dimensionType`, `alias`, `virtual`, `processorId`, `subtype`, and all `indexer.*` fields. Others are +> display-only and can be changed without reindexing: `isRequired`, `defaultQueries`, `allValues`, `isOfficial`, +> `citation`, `pinnedColumns`, `includeAttributes`. The system detects this automatically — see +> [Module 06](06-indexing-and-operations.md#when-to-reindex). + +--- + +## Complete Annotated Example: IMF WEO + +```yaml +urn: + agency_id: "IMF.RES" + resource_id: "WEO" + version: "latest" # Always track the current published version +citation: + url: https://data.imf.org/en/datasets/IMF.RES:WEO + provider: IMF.RES + description: &weo_description > + The World Economic Outlook (WEO) database contains selected macroeconomic + data series from the statistical appendix of the World Economic Outlook + report, which presents the IMF staff's analysis and projections of economic + developments at the global level, in major country groups and in many + individual countries. The WEO is released in April and September/October + each year. Use this database to find data on national accounts, gross + domestic product (GDP), inflation, unemployment rates, balance of payments, + fiscal indicators, trade for countries and country groups (aggregates), and + commodity prices whose data are reported by the IMF. Data are available from + 1980 to the present, and projections are given for the next two years. + Additionally, medium-term projections are available for selected indicators. + For some countries, data are incomplete or unavailable for certain years. +isOfficial: false # IMF is international, not a national agency +useTitleFromSrc: true # Source dataflow title is good +updatedAt: # How to determine when data was last updated + - source: "attribute" + field: "UPDATE_DATE" + formats: ["%Y-%m-%dT%H:%M:%S.%fZ", "%Y-%m-%dT%H:%M:%SZ"] + - source: "annotation" + field: "lastUpdatedAt" + - source: "citation" + field: "last_updated" +dimensions: + INDICATOR: + dimensionType: "INDICATOR" + isRequired: true # Required — queries need an indicator + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + allValues: + id: "ALL_COUNTRIES" + name: "All countries - must be selected when query explicitly asks for all countries" + description: "Special value to query all countries" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["-5y", "+2y"] + operator: "between" +includeAttributes: + - SCALE + - UNIT + - PUBLISHER + - SOURCE +pinnedColumns: + - FREQUENCY_Name # Least important (usually "Annual") + - COUNTRY_Name # Country context + - INDICATOR_Name # Most important — what's being measured +indexer: + indicator: + unpack: true # Packed indicators: "GDP, current prices, Percent change" + useCodeListDescription: true + description: *weo_description # Reuse citation description via YAML anchor +``` + +**Key decisions:** +- `version: "latest"` — always tracks the current published WEO version +- `unpack: true` — WEO indicator values pack multiple concepts into one string +- Single indicator dimension, marked as required +- `isOfficial: false` — IMF is an international organization +- Description uses a YAML anchor so it can be shared between `citation` and `indexer` +- `allValues` on COUNTRY enables star-queries like "global GDP" +- `updatedAt` checks three sources in order for the last-updated date + +--- + +## Second Example: IMF BOP (Multiple Indicator Dimensions) + +```yaml +urn: + agency_id: "IMF.STA" + resource_id: "BOP" + version: "latest" # Always track the current published version +citation: + url: https://data.imf.org/en/datasets/IMF.STA:BOP + provider: IMF.STA + description: &bop_description > + The Balance of Payments (BOP) is a statistical statement that summarizes + transactions between residents and nonresidents during a period. It consists + of the goods and services account, the primary income account, the secondary + income account, the capital account, and the financial account. +isOfficial: false +useTitleFromSrc: true +updatedAt: + - source: "attribute" + field: "UPDATE_DATE" + formats: ["%Y-%m-%dT%H:%M:%S.%fZ", "%Y-%m-%dT%H:%M:%SZ"] + - source: "annotation" + field: "lastUpdatedAt" + - source: "citation" + field: "last_updated" +dimensions: + INDICATOR: + dimensionType: "INDICATOR" + isRequired: true # Main indicator — required + BOP_ACCOUNTING_ENTRY: + dimensionType: "INDICATOR" + isRequired: true # Credit vs. Debit — also required + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + allValues: + id: "ALL_COUNTRIES" + name: "All countries - must be selected when query explicitly asks for all countries" + description: "Special value to query all countries" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["-5y", "now"] + operator: "between" + UNIT: + dimensionType: "NON_INDICATOR" # Values like "US Dollars" are universally understood +includeAttributes: + - SCALE + - UNIT + - PUBLISHER + - SOURCE +pinnedColumns: + - FREQUENCY_Name + - COUNTRY_Name + - BOP_ACCOUNTING_ENTRY_Name + - INDICATOR_Name # Main indicator last (most important) +indexer: + indicator: + unpack: true + useCodeListDescription: true + description: *bop_description +``` + +**Key differences from WEO:** +- Two required indicator dimensions — both INDICATOR and BOP_ACCOUNTING_ENTRY are essential +- UNIT is explicitly `dimensionType: "NON_INDICATOR"` — its values (currencies, percentages) are universally understood +- No forecast data, so default time range ends at `"now"` instead of future years + +--- + +## Third Example: Eurostat NAMA_10_GDP (Different Agency) + +```yaml +urn: + agency_id: "ESTAT" + resource_id: "NAMA_10_GDP" + version: "latest" # Always track the current published version +citation: + provider: Eurostat (ESTAT) + url: https://ec.europa.eu/eurostat/cache/metadata/en/nama10_esms.htm + description: null # Source metadata has a meaningful description +isOfficial: false +useTitleFromSrc: false # Custom title: "GDP and main components" +dimensions: + na_item: + dimensionType: "INDICATOR" + isRequired: true # National accounts item — main indicator, required + unit: + dimensionType: "INDICATOR" # Measurement methodology — domain-specific + geo: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + freq: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["2014-01-01", "2024-12-31"] + operator: "between" +includeAttributes: null # No relevant attributes +pinnedColumns: + - freq_Name # Lowercase to match dimension IDs + - geo_Name + - na_item_Name + - unit_Name +indexer: + indicator: + unpack: false # Unpacked: each value is a single concept + useCodeListDescription: false + description: > # Custom indexer description (since citation desc is null) + National accounts indicator (ESA 2010) - a coherent and consistent set of + macroeconomic indicators, which provide an overall picture of the economic + situation and are widely used for economic analysis and forecasting, policy + design and policy making +``` + +**Key contrasts with IMF datasets:** +- **Lowercase dimension IDs** (`na_item`, `unit`, `geo`, `freq`) — Eurostat convention +- **`unit` is INDICATOR** with `dimensionType: "INDICATOR"` — its values describe measurement methodology (e.g., "Chain linked volumes"), not simple currencies +- **`citation.description: null`** — the source metadata is good, so StatGPT fetches it directly. But `indexer.description` must still be non-empty, so a custom description is provided there +- **`unpack: false`** — each indicator value is a single concept +- **`useTitleFromSrc: false`** — a custom title is used for clarity +- **Fixed date range** in default queries instead of relative expressions +- **`pinnedColumns`** use lowercase to match actual dimension IDs + +--- + +## Virtual Dimensions + +Some datasets only cover a single country and have no country dimension in their SDMX structure. A **virtual dimension** adds a synthetic dimension with a fixed value, making the dataset discoverable for country-based queries. + +In the `dimensions` map, configure the virtual dimension inline using the `virtual` field: + +```yaml +dimensions: + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + virtual: + name: "Country" + description: "Country" + value: + id: "USA" + name: "United States" + description: "United States" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + # ... other dimensions ... +``` + +| Virtual Field | Description | +|---------------|-------------| +| `name` | Display name for the virtual dimension | +| `description` | Description of the virtual dimension | +| `value.id` | The fixed code value (e.g., `"USA"`) | +| `value.name` | Human-readable name for the fixed value | +| `value.description` | Description of the fixed value | + +**Example: FRB FOR (US Federal Reserve — Household Debt Service Ratios)** + +This dataset only contains US data. Without a virtual COUNTRY dimension with a fixed value of "USA", a query like *"What is the household debt service ratio for the US?"* wouldn't match this dataset — there's no country dimension to match against. + +The virtual dimension is configured directly in the `dimensions` map alongside all other dimension settings. + +--- + +## Common Mistakes and How to Avoid Them + +| Mistake | Impact | Prevention | +|---------|--------|-----------| +| Wrong `unpack` setting | Search misses relevant indicators or returns noise | Check code list values for comma-separated multi-concept strings (see [Module 03b](03b-indicator-configuration.md#packed-vs-unpacked-indicators)) | +| All indicator dimensions marked `isRequired: true` | Queries get rejected unnecessarily | Only mark dimensions as required if omitting them makes queries meaningless | +| No required indicator dimensions | Validation fails | At least one INDICATOR dimension must have `isRequired: true` | +| Wrong pinned column order | Data table is hard to read | Order from least to most important; main indicator should be last | +| Incorrect `_Name` suffix casing in pinnedColumns | Column not shown in output | Match exact dimension ID + `_Name` (case-sensitive: `freq_Name` not `FREQ_Name`) | +| Empty `indexer.description` | Indexing fails | Must be a non-empty string — copy from source or write custom | +| `citation.description: null` but also `indexer.description` referencing it | Empty indexer description | When citation is null, write the indexer description independently | +| Missing `dimensionType` on a dimension | Dimension not processed correctly | Every dimension in the `dimensions` map must have a `dimensionType` | +| Forgetting a dimension | Queries miss relevant data or fail | Ensure all dimensions from the DSD are accounted for in the `dimensions` map | +| Country dimension `alias` doesn't match channel's `countryNamedEntityType` | Country recognition fails | Use the same string as the channel's country named entity type | +| `processorId` on SPECIAL dimension doesn't match channel config | Special dimension search fails silently — no error, just no results | Verify the `processorId` exists in the channel's tool configuration before saving | +--- + +## Key Takeaways + +- Dataset configuration is entered as a single YAML in the Admin UI per dataset +- The URN is selected from a table; everything else you fill in manually +- The `dimensions` map is the core of the config — each dimension gets a `dimensionType` and optional settings like `isRequired`, `subtype`, `alias`, `defaultQueries` +- Key decisions per dataset: which dimensions are INDICATOR, which are required, packed vs. unpacked, description source +- Use the `citation.description` / `indexer.description` anchor pattern to avoid duplicating text +- Different agencies use different naming conventions — `alias` on the country dimension and consistent values unify them +- Virtual dimensions enable single-country datasets to participate in country-based queries — configured inline via the `virtual` field +- Always validate: at least one required indicator dimension (`isRequired: true`), non-empty indexer description, correct column casing +- SPECIAL dimensions enable LLM-powered search for large hierarchical classifications (NACE, ISIC, KVED) — set `dimensionType: "SPECIAL"` with `processorId` +- Default queries (`defaultQueries`) are set directly on each dimension, not in a separate top-level field + +--- + +## Check Your Understanding + +**1. A dataset's source metadata has a meaningful description. What do you set for `citation.description` and `indexer.description`?** + +
+Answer + +`citation.description: null` — StatGPT fetches it from the source on each access. `indexer.description:` copy the source description text verbatim (write it independently, don't use a YAML anchor pointing to the null citation). +
+ +**2. You configure `pinnedColumns` as `["INDICATOR_Name", "COUNTRY_Name", "FREQUENCY_Name"]`. What's wrong?** + +
+Answer + +Order is least→most important (left to right), with the main indicator last. Should be `["FREQUENCY_Name", "COUNTRY_Name", "INDICATOR_Name"]` — frequency is least important, the main indicator goes last as the most important column. +
+ +**3. A dataset has a single indicator dimension with values like "Exports of goods and services, Percent of GDP". What `unpack` setting do you use?** + +
+Answer + +`unpack: true` — the values are comma-separated multi-concept strings packing what is measured ("Exports of goods and services") with a unit ("Percent of GDP"). +
+ +**4. You set `citation.description: null` and `indexer.description: *some_anchor`. What happens?** + +
+Answer + +The YAML anchor resolves to `null`, making `indexer.description` empty. Indexing will fail because `indexer.description` must be a non-empty string. When citation is null, write the indexer description independently — don't use an anchor pointing to the null value. +
+ +**5. A Eurostat dataset has dimensions `geo`, `freq`, `na_item`, `unit`, `TIME_PERIOD`. You write `pinnedColumns: ["FREQ_Name", "GEO_Name", "NA_ITEM_Name", "UNIT_Name"]`. What's wrong?** + +
+Answer + +Case mismatch. Eurostat uses lowercase dimension IDs, so the `_Name` suffix must follow the exact casing: `freq_Name`, `geo_Name`, `na_item_Name`, `unit_Name`. Using uppercase (`FREQ_Name`, `GEO_Name`) won't match the actual dimension IDs, and those columns won't appear in the output. +
+ +**6. An FRB dataset only covers the US and has no country dimension in its SDMX structure. A user asks "What is the household debt ratio for the US?" and gets no results. What's missing?** + +
+Answer + +A virtual country dimension. In the `dimensions` map, add a COUNTRY dimension with `dimensionType: "NON_INDICATOR"`, `subtype: "REGION"`, and a `virtual` field specifying a fixed value of "USA": + +```yaml +COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + virtual: + name: "Country" + description: "Country" + value: + id: "USA" + name: "United States" + description: "United States" +``` + +Without this, there's no country dimension to match "US" against, so the dataset is invisible to country-based queries. +
+ +--- + +## Practical Exercises + +### Exercise 1: Configure an OECD Employment Dataset from Scratch + +You're onboarding a new dataset. The Admin UI wizard shows: + +**Dataflow:** `OECD:EMP(1.0.0)` — Employment by economic activity + +**Dimensions:** + +| Dimension ID | Concept Name | Sample Codelist Values | +|---|---|---| +| `MEASURE` | Measure | "Employment", "Unemployment rate", "Labour force participation rate" | +| `ACTIVITY` | Economic activity | "Agriculture, forestry and fishing", "Manufacturing", "Services", "Total" | +| `REF_AREA` | Reference area | "USA", "DEU", "JPN", "FRA", ... | +| `FREQ` | Frequency | "A", "Q" | +| `TIME_PERIOD` | Time period | "2015", "2020", "2023-Q1" | + +**Additional info:** +- Source description is meaningful: "Employment indicators by economic activity, covering employment levels, unemployment rates, and labour force participation across OECD countries" +- Provider attributes available: `UNIT_MEASURE`, `DECIMALS` + +**Your task:** Write the complete dataset configuration YAML. + +
+Solution + +**Dimension classification:** +- `MEASURE` → INDICATOR (required) — describes what is being measured +- `ACTIVITY` → INDICATOR (optional) — economic activity classifications require domain knowledge ("Agriculture, forestry and fishing" is a classification, not a universally understood concept). Optional because queries like "What is the unemployment rate?" are still meaningful without specifying an activity +- `REF_AREA` → NON_INDICATOR (country, `subtype: "REGION"`) +- `FREQ` → NON_INDICATOR (`subtype: "FREQUENCY"`) +- `TIME_PERIOD` → TIME_PERIOD + +**Packed/Unpacked:** `unpack: false` — each value is a single concept ("Employment", "Manufacturing"). No comma-separated multi-concept strings. + +**Citation description:** `null` — the source description is meaningful, so StatGPT fetches it directly. + +**Indexer description:** Copy the source description independently (can't anchor to null citation). + +```yaml +urn: + agency_id: "OECD" + resource_id: "EMP" + version: "latest" +citation: + provider: OECD + url: https://data-explorer.oecd.org/ + description: null # Source has a meaningful description +isOfficial: false # OECD is international, not a national agency +useTitleFromSrc: true # Source title is descriptive +dimensions: + MEASURE: + dimensionType: "INDICATOR" + isRequired: true # Required — "unemployment rate" vs. "employment" matters + ACTIVITY: + dimensionType: "INDICATOR" # Economic activity — domain-specific classification + # Not required — queries work without specifying an activity + REF_AREA: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + FREQ: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["2018", "2025"] + operator: "between" +includeAttributes: + - UNIT_MEASURE +pinnedColumns: + - FREQ_Name # Least important + - REF_AREA_Name # Country context + - ACTIVITY_Name # Economic activity + - MEASURE_Name # Most important — what's being measured +indexer: + indicator: + unpack: false # Single-concept values + useCodeListDescription: false + description: > # Independent copy (citation is null) + Employment indicators by economic activity, covering employment levels, + unemployment rates, and labour force participation across OECD countries +``` + +**Key decisions explained:** +- `ACTIVITY` is INDICATOR, not NON_INDICATOR — "Agriculture, forestry and fishing" is a classification code, not a universally understood concept like a country name +- `ACTIVITY` is optional (no `isRequired: true`) — a query for "unemployment rate in Germany" is meaningful even without specifying an activity sector +- `DECIMALS` omitted from `includeAttributes` — decimal precision is metadata for display, not useful context for the AI agent +- `indexer.description` written independently since `citation.description` is null + +
+ +### Exercise 2: Diagnose a Broken CPI Configuration + +A colleague configured a Eurostat CPI dataset but users report problems. Find and fix all errors: + +```yaml +urn: + agency_id: "ESTAT" + resource_id: "PRC_HICP_MIDX" + version: "latest" +citation: + provider: Eurostat (ESTAT) + url: https://ec.europa.eu/eurostat/cache/metadata/en/prc_hicp_esms.htm + description: null +isOfficial: false +useTitleFromSrc: true +dimensions: + coicop: + dimensionType: "INDICATOR" + isRequired: true + unit: + dimensionType: "INDICATOR" + isRequired: true + index_type: + isRequired: true # Missing dimensionType! + geo: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["2019-01-01", "2024-12-31"] + operator: "between" +includeAttributes: null +pinnedColumns: + - FREQ_Name + - GEO_Name + - COICOP_Name + - UNIT_Name +indexer: + indicator: + unpack: false + useCodeListDescription: false + description: *cpi_desc +``` + +**Your task:** Find all configuration errors, explain their impact, and provide the fix. + +
+Solution + +**Error 1: `index_type` is missing `dimensionType`** + +The `index_type` dimension has `isRequired: true` but no `dimensionType` field. Every dimension in the `dimensions` map must have a `dimensionType`. Since index type values (CPI vs. HICP) are domain-specific, it should be `"INDICATOR"`. + +Fix: +```yaml + index_type: + dimensionType: "INDICATOR" + isRequired: true +``` + +**Error 2: Missing `freq` dimension** + +The `pinnedColumns` reference `FREQ_Name`, but there's no `freq` dimension in the `dimensions` map. Eurostat datasets have a frequency dimension that needs to be configured. + +Fix — add to the `dimensions` map: +```yaml + freq: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" +``` + +**Error 3: `pinnedColumns` use wrong casing** + +Eurostat uses lowercase dimension IDs (`geo`, `freq`, `coicop`, `unit`), but `pinnedColumns` uses uppercase (`FREQ_Name`, `GEO_Name`, `COICOP_Name`, `UNIT_Name`). Column names are case-sensitive — these won't match. + +Fix: +```yaml +pinnedColumns: + - freq_Name + - geo_Name + - coicop_Name + - unit_Name + - index_type_Name +``` +(Also add `index_type_Name` since it's an indicator dimension.) + +**Error 4: `indexer.description: *cpi_desc` resolves to `null`** + +`citation.description` is `null`, and `*cpi_desc` is presumably a YAML anchor pointing to it. The anchor resolves to `null`, making `indexer.description` empty. Indexing will fail. + +Fix: Write the indexer description independently: +```yaml +indexer: + description: > + Harmonised Index of Consumer Prices (HICP) - monthly data providing + comparable measures of inflation across EU member states +``` + +
+ +--- + +**Previous:** [Module 03b — Indicator Configuration](03b-indicator-configuration.md) | **Next:** [Module 05 — Data Sources & Channel Configuration](05-data-sources-and-channels.md) diff --git a/learning/administration/05-data-sources-and-channels.md b/learning/administration/05-data-sources-and-channels.md new file mode 100644 index 0000000..733ef44 --- /dev/null +++ b/learning/administration/05-data-sources-and-channels.md @@ -0,0 +1,472 @@ +# Module 05: Data Sources & Channel Configuration + +## What You'll Learn + +- How to add a Data Source via the Admin UI +- How to create and configure a Channel +- Supreme Agent configuration (name, domain, LLM model, language instructions) +- Named Entity types setup +- Out-of-scope filter, conversation starters, and token tracking +- Linking datasets to channels +- Tool configuration overview +- Glossary management + +--- + +## Adding a Data Source + +A Data Source defines the connection to an external SDMX data provider. You must create a Data Source before adding datasets from it. + +### Steps in the Admin UI + +1. Navigate to **Data Sources** tab and click **Add** +2. Select the data source type (e.g., SDMX 2.1) +3. Fill in the configuration YAML +4. Save + +### Data Source Configuration + +Here is the configuration YAML for connecting to the IMF SDMX 2.1 API: + +```yaml +sdmxConfig: + id: IMF_SDMX21 # Unique identifier + name: Public IMF SDMX 21 Registry + url: https://api.imf.org/external/sdmx/2.1 # SDMX REST API endpoint + headers: # Only override non-default headers + data: + accept: Text/JSON # JSON for data requests (default is XML) +authEnabled: false # No authentication needed for IMF public API +annotationsUrl: https://api.imf.org/external/sdmx/3.0 # Optional: separate endpoint for annotations +``` + +Most fields have sensible defaults — you only need to override what differs from the standard. Structure endpoints +default to `application/xml`, and all SDMX features default to `true`, so you typically only need to specify `headers` +for the data endpoint. If a provider doesn't support a particular SDMX feature, disable it in the `supports` block: + +```yaml +sdmxConfig: + supports: + availableconstraint: false # Provider doesn't support availability queries + preview: false # Provider doesn't support data preview + # All other features default to true +``` + +### Key Fields + +| Field | Description | +|-------|-------------| +| `sdmxConfig.id` | Unique identifier for the data source (e.g., `IMF_SDMX21`) | +| `sdmxConfig.name` | Human-readable name for the data source | +| `sdmxConfig.url` | The SDMX REST API base URL | +| `sdmxConfig.headers` | HTTP headers per request type. Only override non-defaults (e.g., `data: {accept: Text/JSON}`) | +| `authEnabled` | Whether the data source requires authentication | +| `apiKey` | API key for authenticated sources. Use `$env:{VAR_NAME}` for sensitive values | +| `annotationsUrl` | Optional URL for fetching SDMX annotations (some providers serve annotations from a different endpoint) | +| `attributesUrl` | Optional URL for fetching SDMX attributes from a different endpoint | +| `dataExplorerUrl` | Optional URL to the provider's web-based data explorer (shown in citations) | + +### Authentication + +When `authEnabled: true`, provide credentials: +- **API Key** — set in the configuration +- **Environment variable references** — use `$env:{VAR_NAME}` for sensitive values to avoid storing secrets in config + +### What Data Sources Exist For + +Each data provider you want to query needs its own Data Source. For example: +- An IMF data source for IMF datasets (WEO, CPI, BOP, etc.) +- A Eurostat data source for Eurostat datasets (NAMA_10_GDP, etc.) +- A World Bank data source for WDI datasets + +A single channel can include datasets from multiple data sources. + +--- + +## Creating a Channel + +A Channel is the user-facing deployment of StatGPT. Each channel has its own configuration, datasets, index, and glossary. + +### Steps in the Admin UI + +1. Navigate to **Channels** tab and click **Add** +2. Fill in basic fields (deployment ID, title, description) +3. Fill in the configuration YAML +4. Save +5. Add datasets to the channel from the channel detail page + +### Channel Configuration + +The channel configuration YAML is entered in the Admin UI. Here is an example for a channel serving IMF datasets: + +```yaml +locale: EN # Channel language: EN (English) or UK (Ukrainian) +conversationStarters: # Pre-defined prompts shown to users + introText: What would you like to learn about? + buttons: + - title: Check available data + text: What datasets are available? + - title: US GDP projection + text: What is the IMF WEO projection for US GDP for next 2 years? + - title: What is inflation? + text: What is inflation and how it's used in economy and statistics? +namedEntityTypes: # Named entity types for query recognition + - Time frequency + - Counterpart area/country + - Currency/Unit of measure +countryNamedEntityType: Country/Reference area +supremeAgent: # Main AI agent configuration + name: StatGPT + domain: Statistics, economics and SDMX. + terminologyDomain: economics, statistics and SDMX + languageInstructions: + - Use more formal/business tone, but be friendly, polite and helpful. + - Avoid using exclamation marks, emojis, or slang. + additionalContext: > # Optional: extra instructions for the agent + When listing available datasets, use a markdown table format. + maxAgentIterations: 5 + llmModelConfig: + deployment: gpt-4.1-2025-04-14 + temperature: 0 +outOfScope: + domain: Statistics, economics and SDMX. + llmModelConfig: # Optional: separate LLM for scope filtering + deployment: gpt-4.1-2025-04-14 + temperature: 0 + customBlacklist: null # Optional: list of topic descriptions to block + useGeneralTopicsBlacklist: true # Built-in general topic filter (default: true) +tokenUsage: + debugOnly: false # Default is true (debug only); set false for production + stageName: Token Usage +``` + +Below we'll walk through each section. + +### Supreme Agent + +The supreme agent is the main AI orchestrator: + +```yaml +supremeAgent: + name: StatGPT # Agent name shown in context + domain: Statistics, economics and SDMX. # Knowledge domain + terminologyDomain: economics, statistics and SDMX + languageInstructions: # Behavior guidelines added to system prompt + - Use more formal/business tone, but be friendly, polite and helpful. + - Avoid using exclamation marks, emojis, or slang. + additionalContext: > # Optional: extra context added to the system prompt + When listing available datasets, use a markdown table format. + Prefer NSDP dataset for national-level data. + maxAgentIterations: 5 # Max tool calls per user query + llmModelConfig: + deployment: gpt-4.1-2025-04-14 # LLM model (must exist in your DIAL instance) + temperature: 0 # 0 for deterministic responses +``` + +**Key decisions:** +- `domain` — Defines what the agent knows about. Used for scope filtering +- `languageInstructions` — Customize tone and formatting preferences (e.g., formal tone, no emojis) +- `additionalContext` — Optional free-form text appended to the agent's system prompt. Use it for dataset-specific instructions (e.g., dataset preferences, formatting rules) that don't fit as single-line `languageInstructions` +- `maxAgentIterations` — Limits cost; 5 is typical. Increase if complex queries need more tool calls +- `deployment` — Choose the LLM model. Must match a deployment available in your DIAL instance + +### Named Entity Types + +Configure based on the NON_INDICATOR dimensions across all datasets in the channel (see [Module 03a](03a-dimension-types.md)): + +```yaml +namedEntityTypes: + - Time frequency # For FREQUENCY/FREQ dimensions + - Counterpart area/country # For COUNTERPART_COUNTRY, COUNTERPART_AREA, etc. + - Currency/Unit of measure # For UNIT, CURRENCY_TRANS, etc. +countryNamedEntityType: Country/Reference area # Special: the country entity type +``` + +**Important:** +- `countryNamedEntityType` must match the `alias` on the country dimension in your datasets +- When adding datasets with new NON_INDICATOR dimensions, add corresponding Named Entity types here +- The set of Named Entity types depends on which datasets are in the channel + +### Out-of-Scope Filter + +Prevents the agent from answering questions outside its domain: + +```yaml +outOfScope: + domain: Statistics, economics and SDMX. # Domain description for scope filtering + llmModelConfig: # Optional: separate LLM for scope filtering + deployment: gpt-4.1-2025-04-14 + temperature: 0 + customBlacklist: # Optional: list of topic descriptions to block + - "Requests for data manipulation or misrepresentation..." + - "Questions about internal organizational operations..." + useGeneralTopicsBlacklist: true # Built-in general topic filter (default: true) +``` + +| Field | Description | +|-------|-------------| +| `domain` | Domain description — same as or similar to `supremeAgent.domain` | +| `llmModelConfig` | Optional separate LLM for scope filtering (can differ from the main agent model) | +| `customBlacklist` | Optional list of topic description strings to block. Each item describes a category of queries to reject | +| `useGeneralTopicsBlacklist` | When `true` (default), applies a built-in filter for general off-topic queries | +| `startNewConversationMessagesThreshold` | Number of consecutive out-of-scope messages before suggesting the user start a new conversation (default: 3) | +| `startNewConversationMessage` | Custom message shown when the threshold is reached | + +### Conversation Starters + +Pre-defined prompts shown as buttons in the chat interface: + +```yaml +conversationStarters: + introText: What would you like to learn about? + buttons: + - title: Check available data # Button label (short) + text: What datasets are available? # Full query sent when clicked + - title: US GDP projection + text: What is the IMF WEO projection for US GDP for next 2 years? +``` + +Choose starters that demonstrate the channel's capabilities and cover different query types (listing datasets, specific data queries, terminology questions). + +### Token Usage Tracking + +```yaml +tokenUsage: + debugOnly: false # false = track in production + stageName: Token Usage # DIAL stage name for reporting +``` + +--- + +## Linking Datasets to a Channel + +After creating a channel: + +1. Go to the channel detail page +2. Click **Add Dataset** +3. Select datasets from the available list +4. Save + +A channel can include datasets from different data sources. For example, a "Global Data" channel might include IMF, Eurostat, World Bank, and OECD datasets. + +--- + +## Tool Configuration + +Tools are configured in the channel YAML. Each tool type serves a specific purpose: + +### Available Datasets Tool + +Lists all datasets in the channel with metadata. Pre-populates the agent's context at conversation start: + +```yaml +availableDatasets: + type: AVAILABLE_DATASETS + name: Available_Datasets + description: >- + Provides a list of all available datasets onboarded to the Query_Data tool + with metadata and some details about them... + details: + fakeCall: # Pre-loads tool output into agent context + toolCallId: call_EBJJeaOMKeCzm8h378ubURQN + args: "{}" + version: full + includeIndicatorCount: true # Show per-dataset indexed indicator counts + statsHeaderFormat: agencies # Summary format: "totals" or "agencies" +``` + +The `fakeCall` is important — it makes the dataset list available to the agent from the start of every conversation without needing the user to ask. + +| Field | Description | +|-------|-------------| +| `fakeCall` | Pre-loads tool output into the agent's context. The `toolCallId` can be any unique string | +| `version` | Output detail level: `"full"` (all metadata) or `"short"` (compact) | +| `includeIndicatorCount` | When `true`, shows the number of indexed indicators per dataset and total. Useful for [validating indexing](06-indexing-and-operations.md#validating-indexing-results) (default: `false`) | +| `statsHeaderFormat` | Summary format: `"totals"` shows counts, `"agencies"` shows provider agencies | + +### Data Query Tool + +The main tool that executes SDMX queries from natural language: + +```yaml +dataQuery: + type: DATA_QUERY + name: Query_Data + description: >- + Executing sdmx query on available datasets... + details: + indexerVersion: hybrid # "hybrid" for semantic + fulltext search + indicatorSelectionVersion: hybrid + allowAutoUpdate: true # Enable auto-update for this channel's datasets + llmModels: # LLM models for pipeline stages + datasetsSelectionModelConfig: + deployment: gpt-4.1-2025-04-14 + temperature: 0 + indicatorsSelectionModelConfig: + deployment: gpt-4.1-2025-04-14 + temperature: 0 + # ... other stage-specific configs (named entities, time period, etc.) + attachments: # What outputs to show to users + customTable: + enabledStr: "True" + csvFile: + enabledStr: "True" + plotlyGraphs: + enabledStr: "True" + jsonQuery: + enabledStr: "True" + pythonCode: + enabledStr: "True" +``` + +Key settings: +- `indexerVersion: hybrid` — Uses both semantic and fulltext search for indexing (recommended) +- `indicatorSelectionVersion: hybrid` — Uses both semantic and fulltext search for indicator selection at query time (recommended). Related to but distinct from `indexerVersion` +- `allowAutoUpdate` — When `true`, enables [auto-update](06-indexing-and-operations.md#auto-update) for datasets in this channel (default: `false`). Only useful when datasets use `version: "latest"` in their URN +- Each pipeline stage can use a different LLM model/temperature +- Attachments control what's shown to users (tables, CSV files, charts, query JSON, Python code). `enabledStr` uses a string (`"True"`/`"False"`) rather than a boolean to support `$env:{VAR_NAME}` environment variable references + +### Dataset Structure Tool + +Returns the full structure of a specific dataset — dimensions, attributes, types, sample values, and optionally provider +agencies: + +```yaml +datasetStructure: + type: DATASET_STRUCTURE + name: Dataset_Structure + description: >- + Returns dimensions, attributes, types, and sample values for a dataset... + details: + includeProviderAgencies: true # Include list of provider agencies in output +``` + +### Datasets Metadata Tool + +Provides metadata about datasets and the channel. Used by external integrations: + +```yaml +datasetsMetadata: + type: DATASETS_METADATA + name: Datasets_Metadata + description: >- + Provides metadata about datasets in the channel... +``` + +### Glossary Tools + +Two tools for glossary access: + +```yaml +availableTerms: + type: AVAILABLE_TERMS + name: Available_Terms + description: >- + Retrieve a comprehensive list of all terms currently available in the glossary... + details: + fakeCall: + toolCallId: call_EBJJeaOMKeCzm8h378ubU003 + args: "{}" + +termDefinitions: + type: TERM_DEFINITIONS + name: Term_Definitions + description: >- + Retrieve definitions for up to 10 requested terms... + details: + limit: 10 +``` + +--- + +## Glossary Management + +Each channel can have a glossary of term-definition pairs. + +### Adding Terms via Admin UI + +1. Select **Glossary** from the channel context menu +2. Click **Add Term** +3. Fill in: + - **Term** — The term or phrase (e.g., "GDP") + - **Definition** — Official definition + - **Source** — Reference URL or document (e.g., "IMF", "World Bank") + - **Domain** — Context area (e.g., "economics", "statistics") +4. Save + +You can also edit and delete existing terms from the same page. + +### Best Practices + +- Include key statistical terms users are likely to ask about +- Use official definitions from the data provider when available +- Cover terms that appear in dataset indicator names (GDP, CPI, BOP, etc.) +- Keep definitions concise but complete + +--- + +## Key Takeaways + +- Data Sources define connections to SDMX providers — each provider needs its own Data Source +- Channels are user-facing deployments with agent, datasets, tools, and glossary +- The Supreme Agent configuration controls the LLM model, domain, and behavior instructions +- Named Entity types must cover all NON_INDICATOR dimensions across the channel's datasets — `countryNamedEntityType` must match the country dimension's `alias` in datasets +- Tools are configured per channel — Data Query, Available Datasets, Dataset Structure, and Glossary are the main ones +- `allowAutoUpdate` on the Data Query tool enables automatic dataset updates — see [Module 06](06-indexing-and-operations.md#auto-update) +- `includeIndicatorCount` on Available Datasets helps validate indexing results +- The glossary provides consistent terminology definitions for the AI agent + +--- + +## Check Your Understanding + +Test your grasp of data source and channel configuration before moving on. + +
+1. You onboard a new dataset that has a COUNTERPART_COUNTRY dimension classified as NON_INDICATOR. The channel's current Named Entity types are: Time frequency, Currency/Unit of measure. What happens if you don't update the channel config? + +**Answer:** The NER step won't know to look for counterpart country entities in user queries. The dimension may still +work through defaults or LLM reasoning, but recognition will be unreliable. Fix: add `"Counterpart area/country"` to +`namedEntityTypes` in the channel configuration. + +
+ +
+2. Your datasets use alias: "Country/Reference area" on the country dimension, but the channel has countryNamedEntityType: "Country". What breaks? + +**Answer:** Country recognition fails — the system can't match the country Named Entity type to the dimension alias. +`countryNamedEntityType` and the country dimension's `alias` must be identical strings. In this case, set +`countryNamedEntityType: "Country/Reference area"` to match the dataset. + +
+ +
+3. You want to enable auto-update for a channel. Where in the YAML does allowAutoUpdate go? + +**Answer:** On the Data Query tool details: `dataQuery.details.allowAutoUpdate: true`. It's a channel-level setting on +the tool, not a per-dataset setting. This enables [auto-update](06-indexing-and-operations.md#auto-update) for all +datasets in the channel that use `version: "latest"` in their URN. + +
+ +
+4. You want to block queries about organizational governance while keeping the general topic filter active. Which fields do you configure? + +**Answer:** Set `outOfScope.customBlacklist` with a description like `"Questions about how organizations operate, +internal governance, and administrative processes..."`, and keep `outOfScope.useGeneralTopicsBlacklist: true` (the +default). Optionally configure a separate `outOfScope.llmModelConfig` for the scope filter. + +
+ +
+5. After onboarding, you want to verify indexing results from the chat interface. Which tool and field do you configure? + +**Answer:** Enable `includeIndicatorCount: true` on the Available Datasets tool details +(`availableDatasets.details.includeIndicatorCount: true`). After triggering Available Datasets in chat, it will show +per-dataset indicator counts — letting you verify that the expected number of indicators were indexed. + +
+ +--- + +**Previous:** [Module 04 — Configuring a Dataset](04-dataset-configuration.md) | **Next:** [Module 06 — Indexing, Deduplication & Operations](06-indexing-and-operations.md) diff --git a/learning/administration/06-indexing-and-operations.md b/learning/administration/06-indexing-and-operations.md new file mode 100644 index 0000000..30ceb60 --- /dev/null +++ b/learning/administration/06-indexing-and-operations.md @@ -0,0 +1,367 @@ +# Module 06: Indexing, Deduplication & Operations + +## What You'll Learn + +- What indexing does and why it's needed +- When to reindex datasets — and how the system detects this automatically +- How to run and monitor indexing via the Admin UI +- How to validate indexing results using indicator counts +- What deduplication is and when to use it +- Auto-update — automatic detection and reindexing of upstream changes +- Cost awareness — indexing uses LLM tokens +- Channel import/export and job monitoring + +--- + +## What Indexing Does + +StatGPT uses multiple search strategies to find relevant indicators when a user asks a question: + +- **Semantic search** — Finds indicators by meaning similarity (e.g., "economic growth" matches "GDP") +- **Keyword (fulltext) search** — Finds indicators by exact term matching (e.g., "CPI" matches "Consumer Price Index (CPI)") + +Both strategies require a **search index** — a pre-computed representation of all code list items across the dataset's indicator dimensions. Indexing is the process of creating this representation. + +### What Gets Indexed + +For each dataset, the indexer processes: +- **Code list items** from all indicator dimensions — their IDs, names, and (optionally) descriptions +- **Dataset description** — from the `indexer.description` field in the dataset config + +The indexer creates embeddings (vector representations) for semantic search and text entries for fulltext search. The `unpack` setting controls whether packed indicator names are decomposed into individual concepts before indexing (see [Module 03b](03b-indicator-configuration.md#packed-vs-unpacked-indicators)). + +### Why Indexing Is Needed + +Without indexing, StatGPT would have no way to match user queries to specific indicators. The search index is what enables a question like *"What was inflation in Germany?"* to find the indicator *"Consumer price index (CPI), All items"* in the CPI dataset. + +--- + +## When to Reindex + +StatGPT uses an **indexing hash** to automatically determine whether a configuration change requires reindexing. +When you save a dataset configuration, the system compares the hash of indexing-relevant fields against the last +indexed version. If the hash changed, the dataset status shows **NEEDS_REINDEX** — you must trigger reindexing +manually. If the hash is unchanged, the change is applied silently without reindexing. + +### Changes That Require Reindexing + +These fields are **indexing-relevant** — changing them alters the search index: + +| Field Changed | Example | System Behavior | +|---------------|---------|-----------------| +| `dimensionType` on any dimension | Reclassifying INDICATOR ↔ NON_INDICATOR | Hash changes → `NEEDS_REINDEX` | +| `alias` on a dimension | Changing the country alias string | Hash changes → `NEEDS_REINDEX` | +| `virtual` dimension config | Adding or modifying a virtual dimension | Hash changes → `NEEDS_REINDEX` | +| `processorId` on SPECIAL dimension | Changing the LHCL processor reference | Hash changes → `NEEDS_REINDEX` | +| `subtype` on NON_INDICATOR | Changing REGION ↔ FREQUENCY | Hash changes → `NEEDS_REINDEX` | +| `indexer.description` | Updating the dataset description for search | Hash changes → `NEEDS_REINDEX` | +| `indexer.indicator.unpack` | Switching packed ↔ unpacked | Hash changes → `NEEDS_REINDEX` | +| `indexer.indicator.useCodeListDescription` | Toggling code list descriptions (reserved — not yet implemented) | Hash changes → `NEEDS_REINDEX` | +| `indexer.indicator.superPrimary` | Switching primary concatenation from 1 to 3 dimensions | Hash changes → `NEEDS_REINDEX` | +| `indexer.indicator.annotations` | Adding or modifying indicator annotation config | Hash changes → `NEEDS_REINDEX` | +| Upstream code list items changed | Provider added/renamed/removed indicators | Detected by [auto-update](#auto-update) | +| Upstream SDMX structure changed | New dataflow version with different dimensions | Detected by [auto-update](#auto-update) | + +### Changes That Do NOT Require Reindexing + +These fields are **display-only** — changing them is applied immediately without reindexing: + +| Field Changed | Example | +|---------------|---------| +| `isOfficial` | Toggling official status | +| `citation` (provider, url, description) | Updating attribution text | +| `pinnedColumns` | Reordering table columns | +| `isRequired` on a dimension | Changing required ↔ optional | +| `defaultQueries` on a dimension | Adjusting default time range | +| `allValues` on a dimension | Adding/changing the "all countries" value | +| `includeAttributes` | Adding/removing SDMX attributes | +| `useTitleFromSrc` | Switching title source | + +### Changes That Never Need Reindexing + +- Only observation values changed (data updates without structural changes) +- Channel configuration changed (agent settings, conversation starters, etc.) +- Glossary terms were added or modified + +--- + +## Indexing via the Admin UI + +### Reindexing a Single Dataset + +1. Navigate to the channel detail page +2. Find the dataset in the list +3. Click the **Reindex** button for that dataset + +### Reindexing All Datasets in a Channel + +1. Navigate to the channel detail page +2. Click the **Reindex All** button at the top of the datasets list + +This queues reindexing jobs for every dataset in the channel. + +### Monitoring Indexing Status + +Each dataset shows its indexing status in the "Status" column: + +| Status | Meaning | +|--------|---------| +| **Not Started** | Dataset has never been indexed | +| **Queued** | Waiting in the queue for the indexing job to start | +| **In Progress** | Indexing is currently running | +| **Completed** | Successfully indexed — the dataset is searchable | +| **Failed** | Indexing failed — check the configuration or retry | + +**If indexing fails:** +1. Check the dataset configuration for errors (missing `indexer.description`, invalid dimension settings) +2. Verify the data source is accessible +3. Try reindexing again — transient errors (network timeouts, API rate limits) may resolve on retry +4. If the problem persists, review the dataset's SDMX metadata for issues (see [Module 02](02-dataset-assessment.md)) + +--- + +## Deduplication + +### What It Is + +During indexing, some code list items may produce duplicate or near-duplicate entries in the search index. This can happen when: +- Multiple indicator dimensions have overlapping values +- Code list hierarchies produce parent and child items with similar embeddings +- Packed indicators generate similar unpacked components + +Deduplication identifies and removes these redundant entries, keeping the index clean and improving search precision. + +### When to Run It + +Run deduplication: +- **After initial indexing** of a dataset +- **After reindexing** if you suspect duplicate entries are affecting search quality +- **When search results show noise** — extra, irrelevant indicators appearing in results + +Deduplication is available as a separate operation in the Admin UI / CLI. + +--- + +## Validating Indexing Results + +After indexing completes with status **Completed**, verify that the right data was indexed. + +### Per-Dataset Indicator Counts + +The most reliable validation is checking per-dataset indicator counts via the **Available Datasets** tool in the chat +interface. This requires `includeIndicatorCount: true` in the Available Datasets tool configuration +(see [Module 05](05-data-sources-and-channels.md#available-datasets-tool)). + +When enabled, the tool output includes: +- Per dataset: `* Number of indicators: {count}` +- Summary: `Total number of indicators: {sum}` + +**What to check:** +- A dataset with **0 indicators** after indexing means something went wrong — check the dataset configuration + (missing `indexer.description`, incorrect `dimensionType` on indicator dimensions, etc.) +- Compare the indicator count against the expected number of code list items in the indicator dimensions +- A significant drop in count after reindexing may indicate a configuration regression + +### Channel-Wide Totals via Admin API + +The admin API provides channel-wide aggregates: + +``` +GET /channels/{channel_id}/index-status?scope=latest_completed_versions +``` + +This returns total `indicator_dimensions_size` (indexed indicators) and `indicator_dimensions_duplicate_count` +(duplicates before deduplication) across all datasets in the channel. + +--- + +## Cost Awareness + +Indexing uses LLM tokens for embedding generation: + +- Each code list item is sent to the embedding model to generate a vector representation +- Datasets with many indicator values (thousands of code list items) consume more tokens +- The embedding model is configured per channel (e.g., `text-embedding-3-large`) + +**Cost factors:** +- Number of code list items across all indicator dimensions +- Whether `useCodeListDescription: true` (includes longer text per item) +- Whether `unpack: true` (may increase the number of indexed items for packed indicators) +- Reindexing the entire channel multiplies the cost by the number of datasets + +**Best practices:** +- Reindex only the datasets that changed, not the entire channel +- Verify dataset configuration before indexing to avoid failed jobs that waste tokens +- Be aware that a full channel reindex can be expensive for channels with many large datasets + +--- + +## Import/Export + +### Exporting a Channel + +Export creates a dump of a channel's configuration, datasets, and indexes: + +1. Navigate to the channel context menu +2. Click **Export** +3. A job is created — monitor progress on the Jobs page +4. Download the export artifact when the job completes + +Exports are useful for: +- Backing up a channel before making changes +- Moving a channel configuration between environments (dev → staging → production) +- Sharing configurations with other administrators + +### Importing a Channel + +Import loads a channel dump, creating or updating the channel, its datasets, and data sources: + +1. Navigate to the Channels list page +2. Click **Import** +3. Upload the channel dump file +4. Configure import options: + - **Remove channel with the same ID** — If enabled, deletes the existing channel before importing. If disabled, import fails if the channel already exists + - **Update datasets** — If enabled, updates datasets to the version in the import file + - **Update data sources** — If enabled, updates data sources to the version in the import file +5. Start the import + +### Jobs Page + +Both import and export operations create jobs: + +1. Access the Jobs page from the channel context menu → **Jobs** +2. Review job status (Queued, In Progress, Completed, Failed) +3. Download artifacts (export dumps, import logs) + +--- + +## Auto-Update + +Auto-update automatically checks upstream SDMX data sources for changes and reindexes datasets when needed. This +eliminates the need to manually monitor provider releases and trigger reindexing. + +### How It Works + +The auto-update system runs as a batch job that: + +1. **Discovers** all channels with auto-update enabled +2. **Resolves** each dataset's URN against the upstream SDMX registry (e.g., resolves `"latest"` to the current + concrete version) +3. **Compares** the upstream structure and data against the last indexed version +4. **Acts** based on what changed — reindex, update config, or do nothing +5. **Deduplicates** any channels that had reindexed datasets (batch workflow only — individual API-triggered auto-updates require manual deduplication) + +### Enabling Auto-Update + +Auto-update is a **channel-level** opt-in. Set `allowAutoUpdate: true` on the Data Query tool configuration +(see [Module 05](05-data-sources-and-channels.md#data-query-tool)): + +```yaml +dataQuery: + type: DATA_QUERY + details: + allowAutoUpdate: true +``` + +**Important:** Auto-update only makes sense when datasets use `version: "latest"` in their URN. With pinned versions, +the resolved config never changes, and every auto-update run results in `NO_CHANGES`. + +### Auto-Update Outcomes + +Each dataset in an auto-update run produces one of these outcomes: + +| Outcome | Meaning | Action Taken | +|---------|---------|--------------| +| `NO_CHANGES` | Upstream data and structure are identical to the last indexed version | Nothing | +| `CONFIG_UPDATED` | The resolved version pointer changed but the actual data is identical | Lightweight config-only version created (no reindex) | +| `REINDEX_TRIGGERED` | Upstream structure or data changed | Full reindex triggered, followed by deduplication | +| `CONFIG_INCOMPATIBLE` | The upstream version's config fails validation | Nothing — investigate the new version manually | +| `NO_COMPLETED_VERSION` | The dataset has never been successfully indexed | Nothing — index it manually first | + +### Scheduling + +The auto-update script does not have a built-in scheduler. It is invoked externally: + +- **Production:** A Kubernetes CronJob (or similar scheduler) runs the admin container with `ADMIN_MODE=AUTO_UPDATE` +- **Manual trigger (per-dataset):** Admin API `POST /{channel_id}/datasets/{dataset_id}/versions/auto-update-jobs` +- **Development:** `make statgpt_auto_update` + +The scheduling frequency depends on how often upstream providers publish new versions — nightly or weekly is typical. + +--- + +## Operational Workflow Summary + +Typical workflow when onboarding a new dataset: + +1. **Assess** the dataset metadata (Module 02) +2. **Configure** the dataset in the Admin UI (Module 04) +3. **Add** the dataset to a channel (Module 05) +4. **Update channel** Named Entity types if new NON_INDICATOR dimensions were added (Module 03/05) +5. **Index** the dataset +6. **Monitor** indexing status until "Completed" +7. **Validate** — check indicator counts via Available Datasets tool or admin API +8. **Deduplicate** if needed +9. **Test** the dataset with representative queries (Module 07) +10. **Iterate** — fix configuration issues and reindex if test results are unsatisfactory + +**Ongoing maintenance:** If the channel has `allowAutoUpdate: true` and datasets use `version: "latest"`, auto-update +handles upstream changes automatically. Otherwise, monitor provider releases and reindex manually when needed. + +--- + +## Key Takeaways + +- Indexing creates the search index that enables indicator discovery — without it, the dataset is not searchable +- The **indexing hash** automatically detects which config changes require reindexing — you don't have to guess +- Indexing-relevant fields (`dimensionType`, `alias`, `indexer.*`, etc.) trigger `NEEDS_REINDEX`; display-only fields (`citation`, `pinnedColumns`, `isRequired`) are applied silently +- **Validate** indexing results using indicator counts (`includeIndicatorCount: true`) — a count of 0 means something went wrong +- Deduplication cleans up redundant index entries — run it after indexing +- **Auto-update** (`allowAutoUpdate: true`) automatically detects upstream changes and reindexes when needed — pair it with `version: "latest"` in dataset URNs +- Indexing costs LLM tokens — reindex selectively, not the entire channel +- Import/Export enables channel portability between environments + +--- + +## Check Your Understanding + +Test your grasp of indexing and operations before moving on. + +
+1. You changed the pinnedColumns order in a dataset config and saved. Do you need to reindex? + +**Answer:** No. `pinnedColumns` is a display-only field — the indexing hash doesn't change, so the system applies it silently without reindexing. See the "Changes That Do NOT Require Reindexing" table above. + +
+ +
+2. You reclassified a dimension from NON_INDICATOR to INDICATOR. What happens when you save? + +**Answer:** The indexing hash changes because `dimensionType` is an indexing-relevant field. The dataset status shows **NEEDS_REINDEX**. You must trigger reindexing manually so the dimension's values get added to the search index. + +
+ +
+3. A channel has allowAutoUpdate: true, but one dataset uses version: "21.0.0" instead of "latest". The provider releases version 22.0.0. Will auto-update detect this? + +**Answer:** No. With a pinned version, `resolve_config()` always returns `"21.0.0"` — the same as last time. The auto-update result will be `NO_CHANGES`. Change the URN to `version: "latest"` for auto-update to work. + +
+ +
+4. Indexing completed with status "Completed" for a new dataset, but users can't find any indicators from it. How do you diagnose this? + +**Answer:** Check the indicator count. Enable `includeIndicatorCount: true` on the Available Datasets tool, then trigger it in chat. If the dataset shows 0 indicators, the indexing technically succeeded but produced no index entries — likely due to missing or incorrect `dimensionType: "INDICATOR"` on indicator dimensions. + +
+ +
+5. An auto-update job for a WEO dataset returns CONFIG_UPDATED. What happened and what was the result? + +**Answer:** The resolved URN version changed (e.g., the provider published a new version of the dataflow), but the actual data structure and content are identical to the last indexed version. The system created a lightweight config-only version pointing to the existing index data — no reindex was triggered, no tokens were consumed. + +
+ +--- + +**Previous:** [Module 05 — Data Sources & Channel Configuration](05-data-sources-and-channels.md) | **Next:** [Module 07 — Testing & Validation](07-testing-and-validation.md) diff --git a/learning/administration/07-testing-and-validation.md b/learning/administration/07-testing-and-validation.md new file mode 100644 index 0000000..4d60c79 --- /dev/null +++ b/learning/administration/07-testing-and-validation.md @@ -0,0 +1,446 @@ +# Module 07: Testing & Validation + +> **This is a key module.** Testing is essential for quality assurance. LLM-powered systems are non-deterministic — systematic testing is the only way to ensure consistent quality. + +## What You'll Learn + +- Why LLM non-determinism makes testing essential (not optional) +- The concept of ground truth for data queries +- How to design test cases that cover a dataset effectively +- Evaluation concepts: precision, recall, and how StatGPT trades off between them +- A manual verification workflow for validating results +- Example test cases from IMF datasets + +--- + +## LLM Non-Determinism + +### Why the Same Query Can Produce Different Results + +StatGPT uses large language models at multiple stages of the query pipeline (indicator selection, dataset selection, dimension matching, etc.). LLMs are inherently non-deterministic: + +- **Same prompt, different results** — Even with `temperature: 0` and a fixed `seed`, LLMs may produce slightly different outputs across runs due to infrastructure-level variability +- **Context sensitivity** — Conversation history affects results. The same question asked as a first message vs. after a series of related questions may yield different indicator selections +- **Model updates** — When the underlying LLM provider updates or patches models, behavior may change even with identical configuration + +### What This Means for Admins + +- A query that worked correctly yesterday might produce slightly different results today +- You cannot rely on a single test run to validate a dataset +- Systematic testing with defined ground truth is the only way to measure quality over time +- Occasional deviations from expected results are normal — evaluate patterns, not individual runs + +--- + +## Ground Truth + +### What Is Ground Truth? + +In the context of StatGPT data queries, **ground truth** is the expected set of results for a given natural language query: + +- **Expected dataset** — Which dataset should be queried +- **Expected indicators** — Which indicator dimension values should be selected +- **Expected dimensions** — Which non-indicator dimension values should match (country, etc.) + +Ground truth defines what "correct" means for a specific query, enabling objective evaluation. + +### Defining "Correct" When Results Can Vary + +Because of LLM non-determinism, "correct" is not binary. Instead: +- A result that matches all ground truth items is **ideal** +- A result that matches most ground truth items with a few extras is **acceptable** (high recall, slightly lower precision) +- A result that misses key ground truth items is **problematic** (low recall) +- A result from a completely wrong dataset is a **failure** + +### Test Case Structure + +Test cases are maintained in YAML format. Here is a simplified structure: + +```yaml +id: c48d7624-d376-48ca-b2d8-386999befb45 +name: population_numbers_for_mexico +conversation: + - role: user + content: Could you give me the population numbers for Mexico? + target: + indicator_selection: + - dataset_id: IMF.RES:WEO + dimensions: + - dimension_name: INDICATOR + values: + - id: LP + name: Population, Persons for countries / Index for country groups + - dimension_name: COUNTRY + values: + - id: MEX + name: Mexico +``` + +Each test case specifies: +- **Query** — The natural language question +- **Expected dataset** — Which dataset should be matched (`dataset_id`) +- **Expected dimensions** — For each relevant dimension, the expected values (both ID and name) + +> **Note:** This is a simplified view. The full test case schema includes additional fields (`id`, `name`, `tags`, `comments`) and supports multi-turn conversations. For the complete schema and automated evaluation methodology, see [Data Query Evaluation](../../evaluation/data_query.md). + +--- + +## Test Case Categories + +Design test cases across these categories to ensure thorough coverage: + +### 1. Single Indicator Queries + +The simplest and most common query type. One specific indicator for one or more countries. + +> *"What was GDP of Germany in 2023?"* +> Expected: WEO dataset, INDICATOR = GDP-related item, COUNTRY = DEU + +### 2. Multi-Indicator Queries + +Queries asking for multiple related indicators simultaneously. + +> *"GDP and unemployment in France over the last 5 years"* +> Expected: WEO dataset, INDICATOR = GDP + unemployment items, COUNTRY = FRA + +### 3. Indicator Group Queries + +Queries asking for a category of indicators rather than specific ones. + +> *"Show me labor market indicators for Japan"* +> Expected: Relevant dataset, multiple labor-related INDICATOR items, COUNTRY = JPN + +### 4. Synonym Queries + +Queries using common terms that don't match indicator names exactly. + +> *"How did inflation change in Brazil?"* +> Expected: CPI dataset (inflation → Consumer Price Index) + +### 5. Complex/Ambiguous Queries + +Queries that could match multiple datasets or require clarification. + +> *"What are the economic indicators for Kenya?"* +> Expected: Could match WEO, BOP, or other datasets — system should ask for clarification or return the most relevant + +--- + +## How to Think About Dataset Coverage + +When creating test cases for a newly onboarded dataset, aim to cover: + +### Indicator Dimensions +- At least one test case per indicator dimension +- For required indicator dimensions, test cases that exercise different values +- For optional indicator dimensions, test cases both with and without filtering on them + +### Country/Region Dimension +- Test with specific countries (e.g., "USA", "Germany") +- Test with country groups or regions if applicable (e.g., "ASEAN countries", "Euro area") +- Test with "all countries" queries if the dataset supports star-queries + +### Time Period Variations +- Explicit dates: *"GDP in 2023"* +- Relative periods: *"GDP over the last 5 years"* +- Ranges: *"GDP from 2015 to 2020"* +- Future (for forecast datasets): *"GDP projection for 2026"* + +### Synonym Coverage +- Identify common synonyms for key indicators (inflation → CPI, economic growth → GDP growth) +- Create test cases using those synonyms instead of official indicator names + +### Cross-Dataset Scenarios +- If the channel has multiple datasets, test queries that could match more than one +- Verify the system selects the most appropriate dataset + +--- + +## Evaluation Concepts + +### Precision and Recall + +Two standard metrics measure the quality of indicator selection: + +**Precision** — Of all indicators the system selected, how many were correct? + +$$\text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$$ + +- High precision = few irrelevant results (no noise) +- Low precision = many extra, unwanted indicators in the results + +**Recall** — Of all correct indicators, how many did the system find? + +$$\text{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$$ + +- High recall = all expected indicators found (nothing missed) +- Low recall = some expected indicators are missing from results + +### StatGPT's Trade-Off + +StatGPT is designed to **trade off slightly in favor of recall** over precision. This means: +- The system prefers to find all relevant indicators (even at the cost of some extra results) +- It's better to show the user a few extra indicators than to miss the ones they asked for +- Users can easily ignore irrelevant results, but missing data is a worse experience + +### How to Interpret Results + +| Scenario | Precision | Recall | Meaning | +|----------|-----------|--------|---------| +| Ideal | High | High | System found exactly the right indicators | +| Acceptable | Medium | High | System found all expected indicators plus some extras | +| Problematic | High | Low | System is precise but missed important indicators | +| Poor | Low | Low | System missed indicators and returned irrelevant ones | + +### Practical Example + +Target (ground truth) INDICATOR values: GDP, GDPPC + +System selected: GDP, GDPPC, GDP_CONST + +- **True Positives:** GDP, GDPPC (2 items — correctly selected) +- **False Positives:** GDP_CONST (1 item — selected but not in target) +- **False Negatives:** none (0 items — nothing missed) +- **Precision:** 2 / (2 + 1) = 0.67 +- **Recall:** 2 / (2 + 0) = 1.0 + +This is an acceptable result — recall is perfect (nothing missed), precision is slightly lower because one extra indicator was included. + +--- + +## Manual Verification Workflow + +When testing a newly onboarded dataset, follow this workflow: + +### Step 0: Verify Indexing + +Before running test queries, verify the dataset was indexed correctly. Use the Available Datasets tool in the chat +(with `includeIndicatorCount: true` — see [Module 06](06-indexing-and-operations.md#validating-indexing-results)) to +check that the dataset appears and has a non-zero indicator count. A count of 0 means the dataset is not searchable — +fix the configuration and reindex before proceeding with testing. + +### Step 1: Run the Query + +Open StatGPT (the chat interface for the channel) and type the test query. + +### Step 2: Check the Dataset + +Verify the system queried the expected dataset. The response should cite the data source. + +### Step 3: Compare Indicators Against Ground Truth + +Look at the indicators returned in the data table: +- **Are all expected indicators present?** If not, note which are missing (recall issue) +- **Are there unexpected extra indicators?** If so, note which (precision issue) + +### Step 4: Check Dimension Values + +Verify country, time period, and other dimension values match expectations. + +### Step 5: Note Discrepancies + +Record any differences between expected and actual results: +- Missing indicators → **Recall issue** → May need to adjust indexer config or check code list quality +- Extra indicators → **Precision issue** → Usually acceptable; may indicate indexer tuning needed +- Wrong dataset selected → **Dataset selection issue** → Check dataset descriptions and indicator overlap + +### Step 6: Re-Run for Variance + +Run the same query 2-3 times to understand the variance: +- If results are consistent across runs → Reliable +- If results vary significantly → Note the range of variation + +### Step 7: Iterate + +If test results are unsatisfactory: +1. Check the dataset configuration (dimension types, `unpack` setting, indexer description) +2. Review the code list metadata quality (see [Module 02](02-dataset-assessment.md)) +3. Reindex the dataset after making changes +4. Re-test + +--- + +## Creating Good Test Cases + +### Start from the Dataset Structure + +1. Look at the dataset's indicator code list items +2. For each major indicator category, write a query a real user might ask +3. Don't write test cases from the statistician's perspective — write them from the user's + +### Think Like a User + +- Users don't know indicator IDs or exact names +- Users use informal language: "economic growth" not "Gross domestic product, constant prices, Percent change" +- Users may use abbreviations: "GDP", "CPI", "BOP" +- Users ask in context: "How did China's economy perform?" not "Select INDICATOR=GDP WHERE COUNTRY=CHN" + +### Include Edge Cases + +- **Ambiguous terms** — "investment" could mean foreign direct investment, portfolio investment, or capital formation +- **Multiple possible datasets** — "trade data" could match IMTS, BOP, or TiVA +- **Synonyms** — "inflation" → CPI, "economic output" → GDP, "joblessness" → unemployment rate + +### Include Synonym Coverage + +For key indicators in the dataset, identify common synonyms and create test cases using them: + +| Official Name | Common Synonyms | +|---------------|-----------------| +| Consumer Price Index (CPI) | inflation, cost of living, price level | +| Gross Domestic Product (GDP) | economic output, economic growth, national income | +| Unemployment rate | jobless rate, joblessness | +| Balance of Payments | trade balance, current account | + +--- + +## Example Test Cases (IMF Datasets) + +### WEO — Simple Single Indicator + +```yaml +- role: user + content: Could you give me the population numbers for Mexico? + target: + indicator_selection: + - dataset_id: IMF.RES:WEO + dimensions: + - dimension_name: INDICATOR + values: + - id: LP + name: Population, Persons for countries / Index for country groups + - dimension_name: COUNTRY + values: + - id: MEX + name: Mexico +``` + +### CPI — Synonym Query with Multiple Indicator Dimensions + +```yaml +- role: user + content: Can you give me the retail price index data for edible products + and non-alcoholic drinkables in Ecuador? + target: + indicator_selection: + - dataset_id: IMF.STA:CPI + dimensions: + - dimension_name: INDEX_TYPE + values: + - id: CPI + name: Consumer price index (CPI) + - dimension_name: COICOP_1999 + values: + - id: CP01 + name: Food and non-alcoholic beverages + - dimension_name: COUNTRY + values: + - id: ECU + name: Ecuador +``` + +Note how the query uses "retail price index" (→ CPI), "edible products and non-alcoholic drinkables" (→ Food and non-alcoholic beverages), and "Ecuador" (→ ECU). Testing synonym handling is critical. + +--- + +## Key Takeaways + +- LLM non-determinism means the same query can produce different results — systematic testing is essential, not optional +- Ground truth defines expected results (dataset, indicators, dimensions) for each test query +- Design test cases across categories: single indicator, multi-indicator, synonyms, ambiguous, cross-dataset +- **Precision** measures noise (extra results), **recall** measures completeness (missing results) +- StatGPT favors recall — it's better to find all relevant data with some extras than to miss important indicators +- Run queries multiple times to understand variance +- Write test cases from the user's perspective using natural language, not statistician terminology +- Iterate: test → analyze discrepancies → fix config → reindex → retest + +--- + +## Check Your Understanding + +Test your grasp of testing and validation before moving on. + +
+1. A test case for "What is GDP of France?" expects dataset IMF.RES:WEO, INDICATOR values including "Gross domestic product, constant prices", COUNTRY=FRA. The system returns WEO with GDP, GDPPC, and GDP_CONST for FRA. Is this acceptable? + +**Answer:** Yes — this is acceptable. Recall is high because the expected GDP items were found. Precision is slightly lower because GDPPC and GDP_CONST are extras that weren't in the target. Since StatGPT favors recall, returning extra related indicators is expected behavior — it's better to surface all GDP-related indicators than to miss one the user might need. + +
+ +
+2. Your ground truth target indicators are GDP and CPI. The system returned only GDP. Which metric is low — precision or recall? + +**Answer:** Recall is low — CPI was missed (false negative). Precision is 1.0 because everything the system returned (GDP) was correct. This is a problematic result because StatGPT is designed to favor recall. A missed indicator means the user gets an incomplete answer. Investigate why CPI wasn't found — likely a synonym mapping or indicator classification issue. + +
+ +
+3. You run the same query 3 times and get slightly different indicator sets each time. Is this a bug? + +**Answer:** No — this is expected LLM non-determinism. Even with `temperature: 0`, results can vary due to infrastructure-level variability (floating-point non-determinism, batching differences). This is exactly why the module recommends running queries multiple times. Evaluate the pattern across runs — if the core indicators appear consistently and only peripheral ones vary, the system is working as intended. If core indicators appear and disappear unpredictably, that signals a real problem. + +
+ +
+4. You just onboarded a new CPI dataset. Name three categories of test cases you should write. + +**Answer:** (1) **Single indicator queries** — e.g., "What is CPI for Germany?" Tests basic indicator retrieval. (2) **Synonym queries** — e.g., "What is inflation in Brazil?" Tests whether "inflation" maps correctly to CPI indicators. (3) **Cross-dataset queries** — e.g., "What is GDP?" Tests that a GDP query does *not* match the CPI dataset and instead routes to the correct dataset (like WEO). Cross-dataset cases catch misclassification and overly broad indexer descriptions. + +
+ +--- + +## Practical Exercises + +### Exercise 1: Write a Ground Truth Test Case + +Your channel has the IMF BOP (Balance of Payments) dataset with these indicator dimensions: + +| Dimension | Required | Sample Codelist Values | +|-----------|----------|----------------------| +| `INDICATOR` | Yes | "Goods, credit", "Goods, debit", "Services, credit", "Services, debit", "Current Account, Total, Net" | +| `BOP_ACCOUNTING_ENTRY` | Yes | "Credit", "Debit", "Balance" | + +A user asks: *"What is the trade balance for Japan?"* + +**Your task:** Write the ground truth test case YAML following the format shown earlier in this module (the WEO and CPI examples). + +
+Solution + +```yaml +- role: user + content: What is the trade balance for Japan? + target: + indicator_selection: + - dataset_id: IMF.STA:BOP + dimensions: + - dimension_name: INDICATOR + values: + - name: Goods, credit + - name: Goods, debit + - name: Services, credit + - name: Services, debit + - dimension_name: BOP_ACCOUNTING_ENTRY + values: + - name: Balance + - dimension_name: COUNTRY + values: + - id: JP + name: Japan +``` + +**Key decisions:** + +- **"Trade balance" maps to goods and services indicators.** Trade balance = exports minus imports of goods and services. That means four INDICATOR values (goods credit/debit, services credit/debit), not just one. +- **Accounting entry should be "Balance" (net).** The user asked for the *balance*, not the raw credit/debit flows. +- **Multiple INDICATOR values are expected.** A reader might be tempted to pick a single indicator, but trade balance is an aggregate concept spanning multiple BOP line items. +- **"Current Account, Total, Net" is also acceptable.** "Trade balance" is ambiguous — it could refer to the narrower goods-and-services balance or the broader current account. If your test case includes this value, that's a valid interpretation. This is a good example of where ground truth should document acceptable alternatives. + +> **Note:** Real test cases include both `id` and `name` for each value (e.g., `id: LP`, `name: Population...`). This exercise omits IDs because you don't have access to the full BOP codelist — in practice, you'd look up the exact IDs from the dataset structure in the Admin UI. + +
+ +--- + +**Previous:** [Module 06 — Indexing, Deduplication & Operations](06-indexing-and-operations.md) | **Next:** [Module 08 — End-to-End Walkthrough](08-end-to-end-walkthrough.md) diff --git a/learning/administration/08-end-to-end-walkthrough.md b/learning/administration/08-end-to-end-walkthrough.md new file mode 100644 index 0000000..d0900d0 --- /dev/null +++ b/learning/administration/08-end-to-end-walkthrough.md @@ -0,0 +1,361 @@ +# Module 08: End-to-End Walkthrough + +## What You'll Learn + +- How to apply all modules in sequence to onboard a real dataset end-to-end +- The complete onboarding lifecycle: assessment → classification → configuration → indexing → testing +- How to make and document key configuration decisions at each step + +This capstone exercise walks through onboarding a single dataset from scratch, applying every module in sequence. Use it to validate your understanding before onboarding real datasets independently. + +--- + +## The Dataset: IMF EER (Effective Exchange Rates) + +We'll onboard the IMF Effective Exchange Rates dataset — a real dataset that combines several interesting characteristics: + +- A single indicator dimension with packed multi-concept values +- Packed structure that looks superficially similar to IMF ER's unpacked natural descriptions — a good test of Module 03b's decision algorithm +- Country dimension with standard naming +- Moderate complexity (not too simple, not overwhelming) + +**Dataflow:** `IMF.STA:EER(3.0.0)` — Effective Exchange Rates + +**What it provides:** Real and nominal effective exchange rates for countries, measuring the value of a country's currency relative to its trading partners. + +--- + +## Step 1: Assess the Dataset (Module 02) + +Before configuring anything, inspect the metadata through the Admin UI Dataset Wizard. + +### Metadata Inspection + +**Dimensions found:** + +| Dimension ID | Concept Name | Sample Codelist Values | +|---|---|---| +| `INDICATOR` | Indicator | "Real effective exchange rate (REER), Index (2010=100) Adjusted by relative consumer prices", "Nominal effective exchange rate (NEER), Index (2010=100) Advanced economy trade partner weighted", "Nominal effective exchange rate (NEER), Index (2010=100) Weighted index" | +| `COUNTRY` | Reference area | "US" — United States, "DE" — Germany, "JP" — Japan, "GB" — United Kingdom | +| `FREQUENCY` | Frequency | "A" — Annual, "M" — Monthly | +| `TIME_PERIOD` | Time period | "2020", "2023-01", "2024-06" | + +### Assessment Checklist + +**Blockers:** +- [x] Code list items have meaningful, descriptive names — "Real effective exchange rate (REER), Index (2010=100) Adjusted by relative consumer prices" is clear +- [x] English localization is complete +- [x] API performance within thresholds (IMF SDMX 2.1 API — already validated for other IMF datasets) +- [x] Available Constraint endpoint functional + +**Warnings:** +- None identified + +**Business value:** +- Fills a gap: exchange rate data not covered by WEO or BOP +- Target users: economists, forex analysts, trade policy researchers +- Common queries: "What is the exchange rate of the euro?", "How has the yen depreciated?" + +**Decision: PROCEED** + +--- + +## Step 2: Classify Dimensions (Module 03a) + +Apply the decision framework to each dimension: + +| Dimension | Core Question | Classification | Reasoning | +|---|---|---|---| +| `INDICATOR` | Would an average person understand "Real effective exchange rate (REER), Index (2010=100) Adjusted by relative consumer prices"? **No** — requires knowledge of trade-weighted baskets and price deflation methods | **INDICATOR** | Domain-specific measurement concepts | +| `COUNTRY` | Would an average person understand "Germany"? **Yes** | **NON_INDICATOR** | Countries are universally understood | +| `FREQUENCY` | Would an average person understand "Annual"? **Yes** | **NON_INDICATOR** | Frequency is universally understood | +| `TIME_PERIOD` | Time dimension | **TIME_PERIOD** | Auto-detected | + +**Named Entity types check:** +- `COUNTRY` → maps to existing "Country/Reference area" ✓ +- `FREQUENCY` → maps to existing "Time frequency" ✓ +- No new Named Entity types needed + +--- + +## Step 3: Determine Packed/Unpacked + Required/Optional (Module 03b) + +### Packed vs. Unpacked + +Look at the INDICATOR codelist values: +- `"Real effective exchange rate (REER), Index (2010=100) Adjusted by relative consumer prices"` +- `"Nominal effective exchange rate (NEER), Index (2010=100) Advanced economy trade partner weighted"` +- `"Nominal effective exchange rate (NEER), Index (2010=100) Weighted index"` + +Apply the decision algorithm from [Module 03b](03b-indicator-configuration.md#how-to-identify-packed-indicators). Each value packs together multiple independent concepts: +- **What** is being measured (REER vs. NEER) +- **Index specification** (Index, 2010=100) +- **Adjustment/weighting method** (relative consumer prices, advanced economy trade partner weighted) + +**Don't confuse EER with ER.** Module 03b's [grey area section](03b-indicator-configuration.md#grey-area-semi-packed-indicators) discusses IMF **ER** (bilateral Exchange Rates), where values like `"US dollar exchange rate, period average"` are natural descriptions of single concepts → `unpack: false`. EER's values are structurally different — they combine genuinely separable concepts, similar to WEO's `"Gross domestic product, constant prices, Percent change"`. + +**Decision: `unpack: true`** — these are packed multi-concept indicator values. + +### Required vs. Optional + +Only one indicator dimension (`INDICATOR`), so the decision is simple: + +> *"Can the system return a meaningful answer without specifying an indicator?"* +> **No** — "What is [something] for Germany?" is meaningless without knowing which exchange rate. + +**Decision: `INDICATOR` is required.** + +--- + +## Step 4: Write the Dataset YAML (Module 04) + +```yaml +urn: + agency_id: "IMF.STA" + resource_id: "EER" + version: "latest" # Always track the current published version +citation: + provider: "IMF Statistics Department (STA)" + url: https://data.imf.org/en/datasets/IMF.STA:EER + description: &eer_description > + The Effective Exchange Rate (EER) dataset includes annual, quarterly and + monthly nominal and real effective exchange rates by economy. Nominal + effective exchange rates (NEERs) measure the value of a country's currency + in relation to a weighted average of the currency values of their major + trading partners. Real effective exchange rates (REERs) adjust the NEER + to account for a country's inflation rate in relation to the weighted + inflation rate of their major trading partners. +isOfficial: false +useTitleFromSrc: true +updatedAt: # How to determine when data was last updated + - source: "attribute" + field: "UPDATE_DATE" + formats: ["%Y-%m-%dT%H:%M:%S.%fZ", "%Y-%m-%dT%H:%M:%SZ"] + - source: "annotation" + field: "lastUpdatedAt" + - source: "citation" + field: "last_updated" +dimensions: + INDICATOR: + dimensionType: "INDICATOR" + isRequired: true + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + allValues: + id: "ALL_COUNTRIES" + name: "All countries - must be selected when query explicitly asks for all countries" + description: "Special value to query all countries" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["-5y", "now"] + operator: "between" +includeAttributes: + - SCALE + - UNIT + - PUBLISHER + - SOURCE +pinnedColumns: + - FREQUENCY_Name + - COUNTRY_Name + - INDICATOR_Name +indexer: + indicator: + unpack: true + useCodeListDescription: true + description: *eer_description +``` + +**Key decisions documented:** +- `version: "latest"` — always tracks the current published EER version +- `unpack: true` — packed multi-concept values combining exchange rate type, index specification, and adjustment method +- `useCodeListDescription: true` — following IMF dataset pattern +- `isOfficial: false` — IMF is international, not national +- `allValues` on COUNTRY — enables star-queries like "exchange rates for all countries" +- `updatedAt` — checks three sources in order for the last-updated date +- `pinnedColumns` ordered: FREQUENCY (least important) → COUNTRY → INDICATOR (most important) +- `indexer.description` reuses citation description via YAML anchor +- Each dimension explicitly configured with `dimensionType` in the `dimensions` map + +--- + +## Step 5: Verify Data Source and Channel (Module 05) + +### Data Source + +The IMF SDMX 2.1 Data Source should already exist (used by WEO, BOP, CPI, etc.). No new data source needed. + +### Channel Configuration + +Check the channel's Named Entity types: +```yaml +namedEntityTypes: + - Time frequency + - Counterpart area/country + - Currency/Unit of measure +countryNamedEntityType: Country/Reference area +``` + +- `countryNamedEntityType: Country/Reference area` matches our country dimension's `alias` ✓ +- No new NON_INDICATOR dimensions requiring new Named Entity types ✓ + +**Link the dataset to the channel** via the channel detail page. + +--- + +## Step 6: Index and Deduplicate (Module 06) + +1. Navigate to the channel detail page +2. Find the EER dataset in the list +3. Click **Reindex** +4. Monitor status until **Finished** +5. Run **Deduplication** to clean up any near-duplicate entries + +**Cost awareness:** EER has a relatively small indicator code list, so indexing cost is low compared to datasets like WEO or CPI. + +**Ongoing maintenance:** If the channel has `allowAutoUpdate: true` and the dataset uses `version: "latest"`, the +auto-update system will automatically check for upstream changes to the EER dataset and reindex when needed. See +[Module 06 — Auto-Update](06-indexing-and-operations.md#auto-update). + +--- + +## Step 7: Design Test Cases (Module 07) + +The test case YAML shown below is simplified for readability. The full ground truth schema includes additional evaluation fields (`is_out_of_scope`, `tool_calls`, `query_normalization`, etc.) — see [Module 07](07-testing-and-validation.md) for the complete format. + +### Test Case 1: Simple REER Query + +```yaml +- role: user + content: Give me REER adjusted by relative consumer prices for Armenia. + target: + indicator_selection: + - dataset_id: IMF.STA:EER + dimensions: + - dimension_name: INDICATOR + values: + - id: REER_IX_RY2010_ACW_RCPI + name: Real effective exchange rate (REER), Index (2010=100) Adjusted by relative consumer prices + - dimension_name: COUNTRY + values: + - id: ARM + name: Armenia, Republic of +``` + +**What to verify:** System selects the EER dataset and matches the REER CPI-based indicator by abbreviation. + +### Test Case 2: Synonym Query + +```yaml +- role: user + content: How has the Japanese yen depreciated over the last 5 years? + target: + indicator_selection: + - dataset_id: IMF.STA:EER + dimensions: + - dimension_name: INDICATOR + values: + - id: REER_IX_RY2010_ACW_RCPI + name: Real effective exchange rate (REER), Index (2010=100) Adjusted by relative consumer prices + - dimension_name: COUNTRY + values: + - id: JPN + name: Japan +``` + +**What to verify:** "depreciated" maps to exchange rate concepts. The system might select nominal or real — both are acceptable. + +### Test Case 3: Cross-Dataset Ambiguity + +```yaml +- role: user + content: What is the GDP of Germany? + target: + indicator_selection: + - dataset_id: IMF.RES:WEO + dimensions: + - dimension_name: INDICATOR + values: + - name: Gross domestic product, current prices, U.S. dollars +``` + +**What to verify:** GDP query does NOT match the EER dataset — it should go to WEO. + +### Test Case 4: NEER with Specific Weighting + +```yaml +- role: user + content: What is the value of the advanced economy trade partner weighted NEER of Norway? + target: + indicator_selection: + - dataset_id: IMF.STA:EER + dimensions: + - dimension_name: INDICATOR + values: + - id: NEER_IX_RY2010_AEW + name: Nominal effective exchange rate (NEER), Index (2010=100) Advanced economy trade partner weighted + - dimension_name: COUNTRY + values: + - id: NOR + name: Norway +``` + +**What to verify:** "advanced economy trade partner weighted NEER" precisely matches the specific NEER variant. + +--- + +## Step 8: Run Tests and Iterate + +### Execution + +Run each test case 2-3 times in the chat interface: + +| Test | Run 1 | Run 2 | Run 3 | Assessment | +|---|---|---|---|---| +| Simple REER | ✓ EER selected | ✓ EER selected | ✓ EER selected | Consistent | +| Synonym ("depreciated") | ✓ REER CPI | ✓ NEER weighted | ✓ REER CPI | Acceptable variance | +| Cross-dataset (GDP) | ✓ WEO selected | ✓ WEO selected | ✓ WEO selected | No false match | +| NEER specific weighting | ✓ Exact match | ✓ Exact match | ✓ Exact match | Ideal | + +### Iteration Example + +If test 2 consistently returns NEER instead of REER for depreciation queries, consider: +- Is the `indexer.description` clear enough about what the dataset contains? +- Would adding more detail to the description help disambiguation between REER and NEER? +- Is `useCodeListDescription: true` providing enough search context? +- Would the `unpack: true` setting improve or degrade search for these multi-concept indicator names? + +Adjust configuration → reindex → retest until results are satisfactory. + +--- + +## Key Takeaways + +This walkthrough covered the complete onboarding lifecycle: + +| Step | Module | Key Decision | +|---|---|---| +| Assessment | [02](02-dataset-assessment.md) | All blockers clear, business value confirmed | +| Dimension classification | [03a](03a-dimension-types.md) | INDICATOR (1), NON_INDICATOR (2), TIME_PERIOD (1) | +| Indicator configuration | [03b](03b-indicator-configuration.md) | `unpack: true`, INDICATOR required | +| Dataset YAML | [04](04-dataset-configuration.md) | Complete configuration with YAML anchor pattern | +| Data Source & Channel | [05](05-data-sources-and-channels.md) | Existing data source, no new Named Entity types | +| Indexing | [06](06-indexing-and-operations.md) | Index + deduplicate | +| Testing | [07](07-testing-and-validation.md) | 4 test cases across categories | + +- The onboarding lifecycle follows a consistent pattern regardless of the dataset +- Assessment (Module 02) prevents costly rework — always check metadata quality first +- Dimension classification and packed/unpacked decisions are the highest-impact configuration choices — EER and ER look similar but have different `unpack` settings +- Testing with multiple runs reveals LLM non-determinism — evaluate patterns, not individual results +- With `version: "latest"` and `allowAutoUpdate: true`, ongoing maintenance is automated + +Use the [Quick-Reference Card](quick-reference.md) for a condensed checklist during future onboarding. + +--- + +**Previous:** [Module 07 — Testing & Validation](07-testing-and-validation.md) | **Back to:** [Module Index](README.md) diff --git a/learning/administration/README.md b/learning/administration/README.md new file mode 100644 index 0000000..c708e69 --- /dev/null +++ b/learning/administration/README.md @@ -0,0 +1,46 @@ +# StatGPT Admin Learning Materials: Dataset Onboarding + +> **Disclaimer:** StatGPT is actively evolving. We do our best to keep these materials up-to-date, but some details (field names, UI workflows, default values) may lag behind the latest release. When in doubt, refer to the Admin UI itself as the source of truth. If you spot an inconsistency, please open an issue or submit a correction. + +## Purpose + +These learning materials guide StatGPT administrators through the process of onboarding new datasets. After completing all modules, you will be able to independently assess, configure, index, and validate new SDMX datasets in StatGPT. + +## Audience + +- StatGPT administrators responsible for adding and managing datasets +- Technical staff involved in channel configuration and maintenance +- Data source providers preparing datasets for StatGPT integration + +## Prerequisites + +- Access to the StatGPT Admin UI +- Basic understanding of statistics and economics terminology +- Familiarity with YAML configuration format +- (Optional) Familiarity with SDMX concepts — covered in Module 01 + +## Modules + +| Module | Title | Description | +|--------|-------|-------------| +| [01](01-core-concepts.md) | Core Concepts & Entity Relationships | StatGPT's three core entities (Data Source, Dataset, Channel), how they relate, and an introduction to SDMX | +| [02](02-dataset-assessment.md) | Assessing Datasets for Onboarding | Evaluating SDMX metadata quality, identifying packed vs. unpacked indicator patterns (explained fully in Module 03b), and assessing business value | +| [03a](03a-dimension-types.md) | Dimension Types & Named Entities | **Key module** — Classifying dimensions as INDICATOR, NON_INDICATOR, or TIME_PERIOD, and configuring Named Entity types | +| [03b](03b-indicator-configuration.md) | Indicator Configuration | Required vs. optional indicators, packed vs. unpacked, `useCodeListDescription`, and concrete multi-agency examples | +| [04](04-dataset-configuration.md) | Configuring a Dataset | Field-by-field walkthrough of the dataset configuration YAML, with annotated IMF and multi-agency examples | +| [05](05-data-sources-and-channels.md) | Data Sources & Channel Configuration | Adding data sources, creating channels, configuring the supreme agent, tools, and glossary | +| [06](06-indexing-and-operations.md) | Indexing, Deduplication & Operations | Running and monitoring indexes, deduplication, auto-update, import/export, validating indexing results, and cost awareness | +| [07](07-testing-and-validation.md) | Testing & Validation | **Key module** — LLM non-determinism, ground truth, test case design, precision/recall evaluation, and manual verification | +| [08](08-end-to-end-walkthrough.md) | End-to-End Walkthrough | Capstone exercise — onboard a dataset from assessment through testing, applying all modules in sequence | + +## Primary Examples + +These materials use **IMF datasets** as the primary examples (WEO, BOP, CPI, and others from the IMF SDMX 2.1 API). Secondary examples cover **Eurostat**, **World Bank**, **OECD**, **ECB**, **BIS**, and **FRB** to illustrate multi-agency patterns and data modeling differences. + +## Additional Resources + +- [Quick-Reference Card](quick-reference.md) — One-page decision trees, YAML template, and onboarding checklist +- [StatGPT Public Admin Guide](../../guides/admin-guide.md) — UI screenshots and step-by-step instructions +- [SDMX Compatibility & Requirements](../../architecture/sdmx-compatibility.md) — technical requirements for data sources +- [StatGPT Architecture Overview](../../architecture/overview.md) — system-level architecture +- [Data Query Evaluation Methodology](../../evaluation/data_query.md) — detailed evaluation metrics diff --git a/learning/administration/quick-reference.md b/learning/administration/quick-reference.md new file mode 100644 index 0000000..fbaf58b --- /dev/null +++ b/learning/administration/quick-reference.md @@ -0,0 +1,183 @@ +# Quick-Reference Card: Dataset Onboarding + +A one-page reference for common decisions during dataset onboarding. For full explanations, see the linked modules. + +--- + +## Dimension Classification Decision Tree + +For each dimension in the dataset, follow this flowchart: + +``` +Is it the time dimension? +├── Yes → TIME_PERIOD (auto-detected; configure default queries) +└── No ↓ + +Is it the country/region dimension? +├── Yes → NON_INDICATOR (dimensionType: "NON_INDICATOR", subtype: "REGION", + alias) +└── No ↓ + +Is it a frequency dimension? +├── Yes → NON_INDICATOR (dimensionType: "NON_INDICATOR", subtype: "FREQUENCY") +└── No ↓ + +Does it map to an existing Named Entity type? +├── Yes → NON_INDICATOR +└── No ↓ + +Would an average person understand the codelist values? +├── Yes → NON_INDICATOR (add a new Named Entity type if needed) +└── No → INDICATOR (dimensionType: "INDICATOR") +``` + +When in doubt, classify as **INDICATOR**. See [Module 03a](03a-dimension-types.md). + +--- + +## Packed vs. Unpacked Decision Rule + +Look at the **important** indicator dimensions (the ones describing *what* is being measured): + +| Codelist Values | Setting | +|---|---| +| Comma-separated multi-concept strings (e.g., `"GDP, constant prices, Percent change"`) | `unpack: true` | +| Single-concept values (e.g., `"Consumer price index (CPI)"`) | `unpack: false` | +| Commas in natural English (e.g., `"exchange rate, period average"`) | `unpack: false` | + +**Quick test:** Does the comma separate independent concepts that could each be a separate dimension? If yes → packed. If it's natural English phrasing → not packed. + +See [Module 03b](03b-indicator-configuration.md#packed-vs-unpacked-indicators). + +--- + +## Required vs. Optional Indicator Dimensions + +> *"If the user doesn't specify this dimension, can the system still return a meaningful answer?"* + +| Answer | Classification | +|---|---| +| **No** — the query is meaningless without it | Required (`isRequired: true` on the dimension) | +| **Yes** — system can apply a sensible default | Optional (no `isRequired`, or `isRequired: false`) | + +**Rule:** Every dataset must have at least one required indicator dimension. + +See [Module 03b](03b-indicator-configuration.md#required-vs-optional-indicator-dimensions). + +--- + +## Dataset YAML Template + +```yaml +# --- Identity --- +urn: # Pre-filled from wizard + agency_id: "AGENCY.DEPT" + resource_id: "DATAFLOW" + version: "latest" # Recommended: always track current version +citation: + provider: AGENCY.DEPT # Data provider + url: https://... # Link to dataset page + description: &ds_description > # Use null if source has good description + Dataset description text... + +# --- Flags --- +isOfficial: false # true only for national statistics offices +useTitleFromSrc: true # false if you need a custom title + +# --- Dimensions --- +dimensions: + INDICATOR: + dimensionType: "INDICATOR" + isRequired: true # At least one indicator must be required + COUNTRY: + dimensionType: "NON_INDICATOR" + subtype: "REGION" + alias: "Country/Reference area" + FREQUENCY: + dimensionType: "NON_INDICATOR" + subtype: "FREQUENCY" + TIME_PERIOD: + dimensionType: "TIME_PERIOD" + defaultQueries: + - values: ["2020", "2025"] + operator: "between" + +# --- Timestamps --- +updatedAt: # How to determine last updated date + - source: "attribute" + field: "UPDATE_DATE" + formats: ["%Y-%m-%dT%H:%M:%S.%fZ"] + - source: "annotation" + field: "lastUpdatedAt" + - source: "citation" + field: "last_updated" + +# --- Display --- +pinnedColumns: # Least → most important + - FREQUENCY_Name + - COUNTRY_Name + - INDICATOR_Name # Main indicator last +includeAttributes: # SDMX attributes for agent context + - SCALE + - UNIT + +# --- Indexing --- +indexer: + indicator: + unpack: false # true for packed indicators + useCodeListDescription: false # true if code list descriptions are meaningful + description: *ds_description # Must be non-empty +``` + +See [Module 04](04-dataset-configuration.md) for field-by-field details. + +--- + +## Onboarding Checklist + +### Phase 1: Assessment ([Module 02](02-dataset-assessment.md)) + +- [ ] **Blockers clear:** Meaningful code list names, English localization, API < 1 sec, Available Constraint works +- [ ] **Warnings documented:** Duplicates < 10%, borderline performance, sparse data +- [ ] **Business value confirmed:** Fills a gap, target users identified, FAQ coverage + +### Phase 2: Dimension Classification ([Module 03a](03a-dimension-types.md)) + +- [ ] Every dimension in the `dimensions` map has a `dimensionType` +- [ ] Named Entity types added for all NON_INDICATOR dimensions +- [ ] Country dimension `alias` matches channel's `countryNamedEntityType` + +### Phase 3: Indicator Configuration ([Module 03b](03b-indicator-configuration.md)) + +- [ ] At least one indicator dimension has `isRequired: true` +- [ ] Packed/unpacked determined; `unpack` set correctly +- [ ] `useCodeListDescription` set based on code list quality + +### Phase 4: Dataset Configuration ([Module 04](04-dataset-configuration.md)) + +- [ ] All YAML fields filled (citation, dimensions, pinnedColumns, indexer) +- [ ] `indexer.description` is non-empty +- [ ] `pinnedColumns` ordered least → most important, correct `_Name` casing + +### Phase 5: Data Source & Channel ([Module 05](05-data-sources-and-channels.md)) + +- [ ] Data Source exists for the provider +- [ ] Dataset linked to channel +- [ ] Channel Named Entity types updated if new NON_INDICATOR dimensions added + +### Phase 6: Indexing ([Module 06](06-indexing-and-operations.md)) + +- [ ] Dataset indexed successfully (status: Finished) +- [ ] Deduplication run if needed + +### Phase 7: Testing ([Module 07](07-testing-and-validation.md)) + +- [ ] Verify indicator counts via Available Datasets tool (non-zero for the new dataset) +- [ ] Test cases cover: single indicator, multi-indicator, synonyms, edge cases +- [ ] Ground truth defined for each test case +- [ ] Queries run 2-3 times to check variance +- [ ] Recall acceptable (all expected indicators found) +- [ ] Iterate if needed: fix config → reindex → retest + +### Ongoing: Auto-Update ([Module 06](06-indexing-and-operations.md#auto-update)) + +- [ ] If channel has `allowAutoUpdate: true` and dataset uses `version: "latest"`, auto-update handles ongoing maintenance