diff --git a/guides/admin-guide.md b/guides/admin-guide.md index 190814c..53a1df5 100644 --- a/guides/admin-guide.md +++ b/guides/admin-guide.md @@ -108,7 +108,7 @@ urn: IMF.RES:WEO(6.0.0) # SDMX urn of the dataset citation: # citation for the dataset url: https://data.imf.org/en/datasets/IMF.RES:WEO # URL to the dataset web page provider: IMF.RES # data provider - description: &weo_description > # detailed description of the dataset + description: > # detailed description of the dataset The World Economic Outlook (WEO) database contains selected macroeconomic data series from the statistical appendix of the World Economic Outlook report, which presents the IMF staff's analysis and projections of economic @@ -149,8 +149,6 @@ indicatorDimensionsRequiredForQuery: # indicator dimension that required to be f indexer: # indexer configuration for the dataset indicator: unpack: true # whether to unpack the indicator dimensions (e.g. in WEO packed indicators are used, therefore unpack=true) - use_code_list_description: false # whether to use code list description for indexing - description: *weo_description # description to use for indexing ``` Once the dataset is created, you can edit its details by clicking the "Edit" button in the datasets section. This is diff --git a/learning/administration/02-dataset-assessment.md b/learning/administration/02-dataset-assessment.md index 91cd127..a92e3c3 100644 --- a/learning/administration/02-dataset-assessment.md +++ b/learning/administration/02-dataset-assessment.md @@ -447,9 +447,10 @@ Your channel already has **IMF WEO** (World Economic Outlook) onboarded. A stake - WEO has forecasts that ANEA doesn't - Overlap exists for headline GDP figures, but ANEA provides more detailed decomposition -**Recommendation:** Onboard ANEA. Document the overlap in both dataset descriptions so the agent can select the right -dataset based on query specificity. Use clear `indexer.description` values: WEO for "broad macroeconomic indicators and -forecasts", ANEA for "detailed national accounts components and breakdowns". +**Recommendation:** Onboard ANEA. Document the overlap in both dataset descriptions +so the agent can select the right dataset based on query specificity. For example, +WEO: "provides broad macroeconomic indicators and forecasts", +ANEA: "provides detailed national accounts components and breakdowns". diff --git a/learning/administration/03b-indicator-configuration.md b/learning/administration/03b-indicator-configuration.md index 583a9c4..1ebec21 100644 --- a/learning/administration/03b-indicator-configuration.md +++ b/learning/administration/03b-indicator-configuration.md @@ -6,7 +6,6 @@ - The concept of required vs. optional indicator dimensions - The difference between packed and unpacked indicators and the `unpack` indexer setting -- The `useCodeListDescription` indexer setting - Concrete examples from IMF, Eurostat, ECB, BIS, and other agencies - How to apply these concepts in practice through exercises @@ -16,27 +15,30 @@ Once you've identified all INDICATOR dimensions, decide which are **required**: -- **Required** (`isRequired: true`) — The user's query must specify a filter for at least one - required indicator dimension. Queries without any required indicator dimension filter are rejected. -- **Optional** (no `isRequired`, or `isRequired: false`) — The dimension is optional; - queries can proceed without filtering on it. +- **Required** (`isRequired: true`) — Filtering by this dimension alone is enough to trigger + a query against this dataset. +- **Optional** (no `isRequired`, or `isRequired: false`) — The dimension provides supplementary + context, not sufficient on its own to trigger a query. + +A dataset query is **only executed** when the user's query filters on at least one of the dataset's +required dimensions. Queries that don't match any required dimension are skipped. ### The Decision Question -> *"If the user doesn't specify this dimension, can the system still return a meaningful answer?"* +> *"If the user's query specifies a filter for this dimension, should it be enough to trigger a query against this dataset?"* > -> **Yes** → optional. **No** → required. +> **Yes** → required. **No** → optional. -### Required — Without It, the Query Is Meaningless +### Required — Filtering by This Dimension Produces a Sensible Query -- WEO `INDICATOR` — *"What is [something] for Germany?"* has no meaning without specifying what indicator. Required. -- CPI `INDEX_TYPE` + `COICOP_1999` — without specifying CPI vs. HICP **and** a product category, *"What is inflation?"* - is too vague. Both required. +- WEO: `INDICATOR` — *"What is [something] for Germany?"* filters on `INDICATOR`, and that alone is enough to query WEO. Required. +- CPI: `INDEX_TYPE` + `COICOP_1999` — filtering by either CPI vs. HICP or a product category is enough + to trigger a query against this dataset. Both required. -### Optional — Without It, the System Returns Useful (but Less Specific) Data +### Optional — This Dimension Is Supplementary, Not Sufficient on Its Own -- CPI `TYPE_OF_TRANSFORMATION` — if not specified, the system returns "Index" by default. The answer is still - meaningful, just with a default transformation. Optional. +- CPI `TYPE_OF_TRANSFORMATION` — filtering only by transformation type + should not be enough to trigger a query against this dataset. Optional. - ECB BSI `ADJUSTMENT` — seasonal adjustment is a refinement, not essential for returning meaningful data. Optional. ### Rules @@ -169,24 +171,6 @@ Compare with WEO's `"Gross domestic product, constant prices, Percent change"` --- -## The `useCodeListDescription` Setting - -You'll see `useCodeListDescription` in indexer configurations alongside `unpack`: - -- **What it does:** When set to `true`, the indexer includes code list item *descriptions* (not just names) in the search index. This gives the semantic and keyword search more text to match against. -- **When to use it:** Currently set to `true` in all IMF dataset configurations. Follow this pattern for new datasets when the provider includes meaningful descriptions in their code lists. -- **Distinct from `indexer.description`:** The `indexer.description` field describes the *entire dataset* (used for dataset selection). `useCodeListDescription` controls whether *individual code list item descriptions* are included in the indicator search index. - -```yaml -indexer: - description: "Dataset-level description used for dataset selection" # indexer.description - indicator: - unpack: true - useCodeListDescription: true # includes code list item descriptions in the search index -``` - ---- - ## Concrete Examples ### IMF WEO — Simple Structure @@ -534,7 +518,6 @@ values, not just the dimension ID. - Packed indicators combine multiple concepts in comma-separated values → set `unpack: true` - Unpacked indicators have single-concept values across multiple dimensions → set `unpack: false` - Commas in natural English descriptions (e.g., "exchange rate, period average") do not indicate packing -- Use `useCodeListDescription: true` when the provider includes meaningful code list descriptions --- @@ -618,7 +601,6 @@ Compare with the ECB BSI exercise: BSI's values like "Loans" and "Outstanding am indexer: indicator: unpack: true - useCodeListDescription: true ``` diff --git a/learning/administration/04-dataset-configuration.md b/learning/administration/04-dataset-configuration.md index 4b44940..bb8e1ca 100644 --- a/learning/administration/04-dataset-configuration.md +++ b/learning/administration/04-dataset-configuration.md @@ -57,7 +57,7 @@ Tells users where the data comes from: citation: provider: IMF.RES # Data provider name or ID url: https://data.imf.org/en/datasets/IMF.RES:WEO # Link to the dataset's web page - description: &weo_description > # Description of the dataset + description: > # Description of the dataset The World Economic Outlook (WEO) database contains selected macroeconomic data series from the statistical appendix of the World Economic Outlook report... @@ -67,20 +67,10 @@ citation: ``` Source dataflow has a meaningful description? -├── Yes → citation.description: null (fetched from source on each access) -│ indexer.description: copy the source description verbatim -└── No / Missing / Vague → - Write a custom description - citation.description: &anchor > "Your description..." - indexer.description: *anchor (reuse via YAML anchor) +├── Yes → set `citation.description: null` (this will fetch the description from the source on each access) +└── No / Missing / Vague → write a custom description as a string ``` -**Key rules:** -- Citation and indexer description decisions are **independent** — citation controls user-facing attribution, indexer controls what gets searched -- Both paths must result in a **non-empty `indexer.description`** -- When `citation.description: null`, write `indexer.description` independently — do NOT use a YAML anchor pointing to the null citation description (the anchor resolves to `null`, making indexer description empty) -- When you write a custom citation description, use a YAML anchor (e.g., `&weo_description`) so you can reuse the same text in `indexer.description` - > **Multi-agency datasets:** For datasets that aggregate data from multiple providers, `citation` supports additional > fields: `providerTemplate` (a template string with `{n_agencies}` and `{agencies_sample}` placeholders) and > `providerAgencies` (a list of `{id, count, name}` objects). These are advanced — most single-agency datasets only @@ -244,17 +234,13 @@ Controls how the dataset is indexed for semantic and keyword search: indexer: indicator: unpack: true # true for packed indicators, false for unpacked - useCodeListDescription: false # Use code list descriptions during indexing - description: *weo_description # Dataset description for indexing (must be non-empty) ``` | Field | Description | |-------|-------------| | `indicator.unpack` | `true` for packed indicators (comma-separated multi-concept values like "GDP, current prices, USD"). `false` for unpacked. See [Module 03b](03b-indicator-configuration.md#packed-vs-unpacked-indicators). | -| `indicator.useCodeListDescription` | `true` to include code list item descriptions in the index. Improves search when descriptions are meaningful. | | `indicator.superPrimary` | Advanced. Default `false`. When `true` (only applies when `unpack: false`), the primary indicator label is concatenated from the first 3 indicator dimensions instead of just the first one. | | `indicator.annotations` | Advanced. Optional object with a `description` field specifying an SDMX annotation name to use as the indicator description in the index. | -| `description` | **Must be a non-empty string.** If citation description is `null`, copy the dataflow description from the source metadata here. If you wrote a custom citation description, reference it with a YAML anchor (`*weo_description`). | > **Indexing-relevant vs. display-only fields:** Some fields affect the search index and require reindexing when > changed: `dimensionType`, `alias`, `virtual`, `processorId`, `subtype`, and all `indexer.*` fields. Others are @@ -274,7 +260,7 @@ urn: citation: url: https://data.imf.org/en/datasets/IMF.RES:WEO provider: IMF.RES - description: &weo_description > + description: > The World Economic Outlook (WEO) database contains selected macroeconomic data series from the statistical appendix of the World Economic Outlook report, which presents the IMF staff's analysis and projections of economic @@ -329,8 +315,6 @@ pinnedColumns: indexer: indicator: unpack: true # Packed indicators: "GDP, current prices, Percent change" - useCodeListDescription: true - description: *weo_description # Reuse citation description via YAML anchor ``` **Key decisions:** @@ -338,7 +322,6 @@ indexer: - `unpack: true` — WEO indicator values pack multiple concepts into one string - Single indicator dimension, marked as required - `isOfficial: false` — IMF is an international organization -- Description uses a YAML anchor so it can be shared between `citation` and `indexer` - `allValues` on COUNTRY enables star-queries like "global GDP" - `updatedAt` checks three sources in order for the last-updated date @@ -354,7 +337,7 @@ urn: citation: url: https://data.imf.org/en/datasets/IMF.STA:BOP provider: IMF.STA - description: &bop_description > + description: > The Balance of Payments (BOP) is a statistical statement that summarizes transactions between residents and nonresidents during a period. It consists of the goods and services account, the primary income account, the secondary @@ -407,8 +390,6 @@ pinnedColumns: indexer: indicator: unpack: true - useCodeListDescription: true - description: *bop_description ``` **Key differences from WEO:** @@ -458,18 +439,12 @@ pinnedColumns: indexer: indicator: unpack: false # Unpacked: each value is a single concept - useCodeListDescription: false - description: > # Custom indexer description (since citation desc is null) - National accounts indicator (ESA 2010) - a coherent and consistent set of - macroeconomic indicators, which provide an overall picture of the economic - situation and are widely used for economic analysis and forecasting, policy - design and policy making ``` **Key contrasts with IMF datasets:** - **Lowercase dimension IDs** (`na_item`, `unit`, `geo`, `freq`) — Eurostat convention - **`unit` is INDICATOR** with `dimensionType: "INDICATOR"` — its values describe measurement methodology (e.g., "Chain linked volumes"), not simple currencies -- **`citation.description: null`** — the source metadata is good, so StatGPT fetches it directly. But `indexer.description` must still be non-empty, so a custom description is provided there +- **`citation.description: null`** — the source metadata is good, so StatGPT fetches it directly on each access - **`unpack: false`** — each indicator value is a single concept - **`useTitleFromSrc: false`** — a custom title is used for clarity - **Fixed date range** in default queries instead of relative expressions @@ -527,8 +502,6 @@ The virtual dimension is configured directly in the `dimensions` map alongside a | No required indicator dimensions | Validation fails | At least one INDICATOR dimension must have `isRequired: true` | | Wrong pinned column order | Data table is hard to read | Order from least to most important; main indicator should be last | | Incorrect `_Name` suffix casing in pinnedColumns | Column not shown in output | Match exact dimension ID + `_Name` (case-sensitive: `freq_Name` not `FREQ_Name`) | -| Empty `indexer.description` | Indexing fails | Must be a non-empty string — copy from source or write custom | -| `citation.description: null` but also `indexer.description` referencing it | Empty indexer description | When citation is null, write the indexer description independently | | Missing `dimensionType` on a dimension | Dimension not processed correctly | Every dimension in the `dimensions` map must have a `dimensionType` | | Forgetting a dimension | Queries miss relevant data or fail | Ensure all dimensions from the DSD are accounted for in the `dimensions` map | | Country dimension `alias` doesn't match channel's `countryNamedEntityType` | Country recognition fails | Use the same string as the channel's country named entity type | @@ -541,10 +514,9 @@ The virtual dimension is configured directly in the `dimensions` map alongside a - The URN is selected from a table; everything else you fill in manually - The `dimensions` map is the core of the config — each dimension gets a `dimensionType` and optional settings like `isRequired`, `subtype`, `alias`, `defaultQueries` - Key decisions per dataset: which dimensions are INDICATOR, which are required, packed vs. unpacked, description source -- Use the `citation.description` / `indexer.description` anchor pattern to avoid duplicating text - Different agencies use different naming conventions — `alias` on the country dimension and consistent values unify them - Virtual dimensions enable single-country datasets to participate in country-based queries — configured inline via the `virtual` field -- Always validate: at least one required indicator dimension (`isRequired: true`), non-empty indexer description, correct column casing +- Always validate: at least one required indicator dimension (`isRequired: true`), correct column casing - SPECIAL dimensions enable LLM-powered search for large hierarchical classifications (NACE, ISIC, KVED) — set `dimensionType: "SPECIAL"` with `processorId` - Default queries (`defaultQueries`) are set directly on each dimension, not in a separate top-level field @@ -552,12 +524,12 @@ The virtual dimension is configured directly in the `dimensions` map alongside a ## Check Your Understanding -**1. A dataset's source metadata has a meaningful description. What do you set for `citation.description` and `indexer.description`?** +**1. A dataset's source metadata has a meaningful description. What do you set for `citation.description`?**
Answer -`citation.description: null` — StatGPT fetches it from the source on each access. `indexer.description:` copy the source description text verbatim (write it independently, don't use a YAML anchor pointing to the null citation). +`citation.description: null` — StatGPT fetches it from the source on each access.
**2. You configure `pinnedColumns` as `["INDICATOR_Name", "COUNTRY_Name", "FREQUENCY_Name"]`. What's wrong?** @@ -576,15 +548,7 @@ Order is least→most important (left to right), with the main indicator last. S `unpack: true` — the values are comma-separated multi-concept strings packing what is measured ("Exports of goods and services") with a unit ("Percent of GDP"). -**4. You set `citation.description: null` and `indexer.description: *some_anchor`. What happens?** - -
-Answer - -The YAML anchor resolves to `null`, making `indexer.description` empty. Indexing will fail because `indexer.description` must be a non-empty string. When citation is null, write the indexer description independently — don't use an anchor pointing to the null value. -
- -**5. A Eurostat dataset has dimensions `geo`, `freq`, `na_item`, `unit`, `TIME_PERIOD`. You write `pinnedColumns: ["FREQ_Name", "GEO_Name", "NA_ITEM_Name", "UNIT_Name"]`. What's wrong?** +**4. A Eurostat dataset has dimensions `geo`, `freq`, `na_item`, `unit`, `TIME_PERIOD`. You write `pinnedColumns: ["FREQ_Name", "GEO_Name", "NA_ITEM_Name", "UNIT_Name"]`. What's wrong?**
Answer @@ -592,7 +556,7 @@ The YAML anchor resolves to `null`, making `indexer.description` empty. Indexing Case mismatch. Eurostat uses lowercase dimension IDs, so the `_Name` suffix must follow the exact casing: `freq_Name`, `geo_Name`, `na_item_Name`, `unit_Name`. Using uppercase (`FREQ_Name`, `GEO_Name`) won't match the actual dimension IDs, and those columns won't appear in the output.
-**6. An FRB dataset only covers the US and has no country dimension in its SDMX structure. A user asks "What is the household debt ratio for the US?" and gets no results. What's missing?** +**5. An FRB dataset only covers the US and has no country dimension in its SDMX structure. A user asks "What is the household debt ratio for the US?" and gets no results. What's missing?**
Answer @@ -656,8 +620,6 @@ You're onboarding a new dataset. The Admin UI wizard shows: **Citation description:** `null` — the source description is meaningful, so StatGPT fetches it directly. -**Indexer description:** Copy the source description independently (can't anchor to null citation). - ```yaml urn: agency_id: "OECD" @@ -698,17 +660,12 @@ pinnedColumns: indexer: indicator: unpack: false # Single-concept values - useCodeListDescription: false - description: > # Independent copy (citation is null) - Employment indicators by economic activity, covering employment levels, - unemployment rates, and labour force participation across OECD countries ``` **Key decisions explained:** - `ACTIVITY` is INDICATOR, not NON_INDICATOR — "Agriculture, forestry and fishing" is a classification code, not a universally understood concept like a country name - `ACTIVITY` is optional (no `isRequired: true`) — a query for "unemployment rate in Germany" is meaningful even without specifying an activity sector - `DECIMALS` omitted from `includeAttributes` — decimal precision is metadata for display, not useful context for the AI agent -- `indexer.description` written independently since `citation.description` is null
@@ -754,8 +711,6 @@ pinnedColumns: indexer: indicator: unpack: false - useCodeListDescription: false - description: *cpi_desc ``` **Your task:** Find all configuration errors, explain their impact, and provide the fix. @@ -800,18 +755,6 @@ pinnedColumns: ``` (Also add `index_type_Name` since it's an indicator dimension.) -**Error 4: `indexer.description: *cpi_desc` resolves to `null`** - -`citation.description` is `null`, and `*cpi_desc` is presumably a YAML anchor pointing to it. The anchor resolves to `null`, making `indexer.description` empty. Indexing will fail. - -Fix: Write the indexer description independently: -```yaml -indexer: - description: > - Harmonised Index of Consumer Prices (HICP) - monthly data providing - comparable measures of inflation across EU member states -``` - --- diff --git a/learning/administration/06-indexing-and-operations.md b/learning/administration/06-indexing-and-operations.md index 30ceb60..2596830 100644 --- a/learning/administration/06-indexing-and-operations.md +++ b/learning/administration/06-indexing-and-operations.md @@ -26,7 +26,6 @@ Both strategies require a **search index** — a pre-computed representation of For each dataset, the indexer processes: - **Code list items** from all indicator dimensions — their IDs, names, and (optionally) descriptions -- **Dataset description** — from the `indexer.description` field in the dataset config The indexer creates embeddings (vector representations) for semantic search and text entries for fulltext search. The `unpack` setting controls whether packed indicator names are decomposed into individual concepts before indexing (see [Module 03b](03b-indicator-configuration.md#packed-vs-unpacked-indicators)). @@ -54,9 +53,7 @@ These fields are **indexing-relevant** — changing them alters the search index | `virtual` dimension config | Adding or modifying a virtual dimension | Hash changes → `NEEDS_REINDEX` | | `processorId` on SPECIAL dimension | Changing the LHCL processor reference | Hash changes → `NEEDS_REINDEX` | | `subtype` on NON_INDICATOR | Changing REGION ↔ FREQUENCY | Hash changes → `NEEDS_REINDEX` | -| `indexer.description` | Updating the dataset description for search | Hash changes → `NEEDS_REINDEX` | | `indexer.indicator.unpack` | Switching packed ↔ unpacked | Hash changes → `NEEDS_REINDEX` | -| `indexer.indicator.useCodeListDescription` | Toggling code list descriptions (reserved — not yet implemented) | Hash changes → `NEEDS_REINDEX` | | `indexer.indicator.superPrimary` | Switching primary concatenation from 1 to 3 dimensions | Hash changes → `NEEDS_REINDEX` | | `indexer.indicator.annotations` | Adding or modifying indicator annotation config | Hash changes → `NEEDS_REINDEX` | | Upstream code list items changed | Provider added/renamed/removed indicators | Detected by [auto-update](#auto-update) | @@ -113,7 +110,7 @@ Each dataset shows its indexing status in the "Status" column: | **Failed** | Indexing failed — check the configuration or retry | **If indexing fails:** -1. Check the dataset configuration for errors (missing `indexer.description`, invalid dimension settings) +1. Check the dataset configuration for errors (invalid dimension settings, missing required fields) 2. Verify the data source is accessible 3. Try reindexing again — transient errors (network timeouts, API rate limits) may resolve on retry 4. If the problem persists, review the dataset's SDMX metadata for issues (see [Module 02](02-dataset-assessment.md)) @@ -158,7 +155,7 @@ When enabled, the tool output includes: **What to check:** - A dataset with **0 indicators** after indexing means something went wrong — check the dataset configuration - (missing `indexer.description`, incorrect `dimensionType` on indicator dimensions, etc.) + (incorrect `dimensionType` on indicator dimensions, etc.) - Compare the indicator count against the expected number of code list items in the indicator dimensions - A significant drop in count after reindexing may indicate a configuration regression @@ -185,7 +182,6 @@ Indexing uses LLM tokens for embedding generation: **Cost factors:** - Number of code list items across all indicator dimensions -- Whether `useCodeListDescription: true` (includes longer text per item) - Whether `unpack: true` (may increase the number of indexed items for packed indicators) - Reindexing the entire channel multiplies the cost by the number of datasets diff --git a/learning/administration/07-testing-and-validation.md b/learning/administration/07-testing-and-validation.md index 4d60c79..3f5f566 100644 --- a/learning/administration/07-testing-and-validation.md +++ b/learning/administration/07-testing-and-validation.md @@ -253,7 +253,7 @@ Run the same query 2-3 times to understand the variance: ### Step 7: Iterate If test results are unsatisfactory: -1. Check the dataset configuration (dimension types, `unpack` setting, indexer description) +1. Check the dataset configuration (dimension types, `unpack` setting, citation description) 2. Review the code list metadata quality (see [Module 02](02-dataset-assessment.md)) 3. Reindex the dataset after making changes 4. Re-test @@ -384,7 +384,7 @@ Test your grasp of testing and validation before moving on.
4. You just onboarded a new CPI dataset. Name three categories of test cases you should write. -**Answer:** (1) **Single indicator queries** — e.g., "What is CPI for Germany?" Tests basic indicator retrieval. (2) **Synonym queries** — e.g., "What is inflation in Brazil?" Tests whether "inflation" maps correctly to CPI indicators. (3) **Cross-dataset queries** — e.g., "What is GDP?" Tests that a GDP query does *not* match the CPI dataset and instead routes to the correct dataset (like WEO). Cross-dataset cases catch misclassification and overly broad indexer descriptions. +**Answer:** (1) **Single indicator queries** — e.g., "What is CPI for Germany?" Tests basic indicator retrieval. (2) **Synonym queries** — e.g., "What is inflation in Brazil?" Tests whether "inflation" maps correctly to CPI indicators. (3) **Cross-dataset queries** — e.g., "What is GDP?" Tests that a GDP query does *not* match the CPI dataset and instead routes to the correct dataset (like WEO). Cross-dataset cases catch misclassification and overly broad dataset descriptions.
diff --git a/learning/administration/08-end-to-end-walkthrough.md b/learning/administration/08-end-to-end-walkthrough.md index d0900d0..b5dd385 100644 --- a/learning/administration/08-end-to-end-walkthrough.md +++ b/learning/administration/08-end-to-end-walkthrough.md @@ -117,7 +117,7 @@ urn: citation: provider: "IMF Statistics Department (STA)" url: https://data.imf.org/en/datasets/IMF.STA:EER - description: &eer_description > + description: > The Effective Exchange Rate (EER) dataset includes annual, quarterly and monthly nominal and real effective exchange rates by economy. Nominal effective exchange rates (NEERs) measure the value of a country's currency @@ -167,19 +167,15 @@ pinnedColumns: indexer: indicator: unpack: true - useCodeListDescription: true - description: *eer_description ``` **Key decisions documented:** - `version: "latest"` — always tracks the current published EER version - `unpack: true` — packed multi-concept values combining exchange rate type, index specification, and adjustment method -- `useCodeListDescription: true` — following IMF dataset pattern - `isOfficial: false` — IMF is international, not national - `allValues` on COUNTRY — enables star-queries like "exchange rates for all countries" - `updatedAt` — checks three sources in order for the last-updated date - `pinnedColumns` ordered: FREQUENCY (least important) → COUNTRY → INDICATOR (most important) -- `indexer.description` reuses citation description via YAML anchor - Each dimension explicitly configured with `dimensionType` in the `dimensions` map --- @@ -325,10 +321,8 @@ Run each test case 2-3 times in the chat interface: ### Iteration Example If test 2 consistently returns NEER instead of REER for depreciation queries, consider: -- Is the `indexer.description` clear enough about what the dataset contains? -- Would adding more detail to the description help disambiguation between REER and NEER? -- Is `useCodeListDescription: true` providing enough search context? - Would the `unpack: true` setting improve or degrade search for these multi-concept indicator names? +- Are the indicator code list names descriptive enough to disambiguate REER from NEER? Adjust configuration → reindex → retest until results are satisfactory. @@ -343,7 +337,7 @@ This walkthrough covered the complete onboarding lifecycle: | Assessment | [02](02-dataset-assessment.md) | All blockers clear, business value confirmed | | Dimension classification | [03a](03a-dimension-types.md) | INDICATOR (1), NON_INDICATOR (2), TIME_PERIOD (1) | | Indicator configuration | [03b](03b-indicator-configuration.md) | `unpack: true`, INDICATOR required | -| Dataset YAML | [04](04-dataset-configuration.md) | Complete configuration with YAML anchor pattern | +| Dataset YAML | [04](04-dataset-configuration.md) | Complete configuration with all required fields | | Data Source & Channel | [05](05-data-sources-and-channels.md) | Existing data source, no new Named Entity types | | Indexing | [06](06-indexing-and-operations.md) | Index + deduplicate | | Testing | [07](07-testing-and-validation.md) | 4 test cases across categories | diff --git a/learning/administration/README.md b/learning/administration/README.md index c708e69..49bd87e 100644 --- a/learning/administration/README.md +++ b/learning/administration/README.md @@ -26,7 +26,7 @@ These learning materials guide StatGPT administrators through the process of onb | [01](01-core-concepts.md) | Core Concepts & Entity Relationships | StatGPT's three core entities (Data Source, Dataset, Channel), how they relate, and an introduction to SDMX | | [02](02-dataset-assessment.md) | Assessing Datasets for Onboarding | Evaluating SDMX metadata quality, identifying packed vs. unpacked indicator patterns (explained fully in Module 03b), and assessing business value | | [03a](03a-dimension-types.md) | Dimension Types & Named Entities | **Key module** — Classifying dimensions as INDICATOR, NON_INDICATOR, or TIME_PERIOD, and configuring Named Entity types | -| [03b](03b-indicator-configuration.md) | Indicator Configuration | Required vs. optional indicators, packed vs. unpacked, `useCodeListDescription`, and concrete multi-agency examples | +| [03b](03b-indicator-configuration.md) | Indicator Configuration | Required vs. optional indicators, packed vs. unpacked, and concrete multi-agency examples | | [04](04-dataset-configuration.md) | Configuring a Dataset | Field-by-field walkthrough of the dataset configuration YAML, with annotated IMF and multi-agency examples | | [05](05-data-sources-and-channels.md) | Data Sources & Channel Configuration | Adding data sources, creating channels, configuring the supreme agent, tools, and glossary | | [06](06-indexing-and-operations.md) | Indexing, Deduplication & Operations | Running and monitoring indexes, deduplication, auto-update, import/export, validating indexing results, and cost awareness | diff --git a/learning/administration/quick-reference.md b/learning/administration/quick-reference.md index fbaf58b..ae0c7c2 100644 --- a/learning/administration/quick-reference.md +++ b/learning/administration/quick-reference.md @@ -52,14 +52,16 @@ See [Module 03b](03b-indicator-configuration.md#packed-vs-unpacked-indicators). ## Required vs. Optional Indicator Dimensions -> *"If the user doesn't specify this dimension, can the system still return a meaningful answer?"* +> *"If the user's query specifies a filter for this dimension, should it be enough to trigger a query against this dataset?"* | Answer | Classification | |---|---| -| **No** — the query is meaningless without it | Required (`isRequired: true` on the dimension) | -| **Yes** — system can apply a sensible default | Optional (no `isRequired`, or `isRequired: false`) | +| **Yes** — filtering by this dimension alone produces a sensible query | Required (`isRequired: true`) | +| **No** — this dimension is supplementary, not sufficient on its own | Optional (`isRequired: false` or omitted) | -**Rule:** Every dataset must have at least one required indicator dimension. +A dataset query is **only executed** when the user's query filters on at least one of the dataset's required dimensions. Queries that don't match any required dimension are skipped. + +**Rules:** Only INDICATOR dimensions can be marked required. Every dataset must have at least one required indicator dimension. See [Module 03b](03b-indicator-configuration.md#required-vs-optional-indicator-dimensions). @@ -76,7 +78,7 @@ urn: # Pre-filled from wizard citation: provider: AGENCY.DEPT # Data provider url: https://... # Link to dataset page - description: &ds_description > # Use null if source has good description + description: > # Use null if source has good description Dataset description text... # --- Flags --- @@ -124,8 +126,6 @@ includeAttributes: # SDMX attributes for agent context indexer: indicator: unpack: false # true for packed indicators - useCodeListDescription: false # true if code list descriptions are meaningful - description: *ds_description # Must be non-empty ``` See [Module 04](04-dataset-configuration.md) for field-by-field details. @@ -150,12 +150,9 @@ See [Module 04](04-dataset-configuration.md) for field-by-field details. - [ ] At least one indicator dimension has `isRequired: true` - [ ] Packed/unpacked determined; `unpack` set correctly -- [ ] `useCodeListDescription` set based on code list quality - ### Phase 4: Dataset Configuration ([Module 04](04-dataset-configuration.md)) - [ ] All YAML fields filled (citation, dimensions, pinnedColumns, indexer) -- [ ] `indexer.description` is non-empty - [ ] `pinnedColumns` ordered least → most important, correct `_Name` casing ### Phase 5: Data Source & Channel ([Module 05](05-data-sources-and-channels.md))