From 4b46eced3afd8ed204c2bfbd4ed959a72f49986e Mon Sep 17 00:00:00 2001 From: Daniil Yarmalkevich Date: Thu, 4 Jun 2026 20:34:28 +0300 Subject: [PATCH 1/2] docs: add Data Query and hybrid indicator search architecture docs Add a conceptual overview and an engineering deep dive describing how the Data Query tool turns natural-language questions into grounded SDMX data, and how hybrid indicator search (keyword + semantic + LLM relevance) works. Index both pages in the architecture README. --- architecture/README.md | 4 + architecture/data-query-hybrid-search.md | 269 ++++++++++++++ architecture/data-query-internals.md | 433 +++++++++++++++++++++++ 3 files changed, 706 insertions(+) create mode 100644 architecture/data-query-hybrid-search.md create mode 100644 architecture/data-query-internals.md diff --git a/architecture/README.md b/architecture/README.md index bc5f31f..82321c7 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -12,6 +12,7 @@ design, services, tools, and integration requirements. | **[πŸ“‹ Overview](./overview.md)** | Complete platform overview with requirements and features | β€’ Natural language querying
β€’ Data accuracy & reliability
β€’ Security & governance
β€’ Performance & scalability | | **[🏭 Services](./services.md)** | Core services and dependencies architecture | β€’ Chat Backend (DIAL app)
β€’ Admin Backend & Frontend
β€’ Portal Frontend
β€’ Third-party integrations | | **[πŸ”§ Tools](./tools.md)** | Agent tools and capabilities documentation | β€’ Data query tools
β€’ Publications RAG
β€’ Glossary management
β€’ Web search integration | +| **[πŸ”Ž Data Query & Hybrid Search](./data-query-hybrid-search.md)** | How NL queries become grounded SDMX data, and how hybrid indicator search works (conceptual) | β€’ Composite indicators
β€’ Keyword + semantic + LLM
β€’ Two-phase indexing
β€’ Availability grounding | | **[πŸ”Œ Application MCP](./mcp.md)** | Agentic access via Model Context Protocol | β€’ Channel tools surfaced to AI agents
β€’ DIAL Application registration
β€’ MCP-spec discovery + IDP OAuth flow | | **[πŸ“Š SDMX Compatibility](./sdmx-compatibility.md)** | SDMX standards and requirements guide | β€’ Version support (2.1/3.0)
β€’ Metadata requirements
β€’ Performance standards
β€’ Quality checklist | @@ -20,6 +21,7 @@ design, services, tools, and integration requirements. | Document | Description | Key Topics | |-----------------------------------|-----------------------------------------------|-----------------------------------------------------------------------------------| | **[πŸ€– Agent Design](./agent.md)** | StatGPT agent architecture and implementation | β€’ Tool-calling approach
β€’ Dynamic history management
β€’ Contextual grounding | +| **[πŸ› οΈ Data Query Internals](./data-query-internals.md)** | Engineering deep dive into the Data Query pipeline and hybrid search | β€’ Pipeline orchestration
β€’ Runtime hybrid engine
β€’ Offline indexer
β€’ Component & code references | ## πŸ—ΊοΈ Quick Navigation Guide @@ -84,6 +86,8 @@ Stateless services design enables horizontal scaling to handle varying loads eff 1. **Understanding the System** β†’ Read documents in this order: - [Overview](./overview.md) β†’ [Services](./services.md) β†’ [Agent Design](./agent.md) β†’ [Tools](./tools.md) + - Then, for data querying: [Data Query & Hybrid Search](./data-query-hybrid-search.md) (concepts) β†’ + [Data Query Internals](./data-query-internals.md) (implementation) 2. **Integration Planning** β†’ Focus on: - [SDMX Compatibility](./sdmx-compatibility.md) for data requirements diff --git a/architecture/data-query-hybrid-search.md b/architecture/data-query-hybrid-search.md new file mode 100644 index 0000000..85969be --- /dev/null +++ b/architecture/data-query-hybrid-search.md @@ -0,0 +1,269 @@ +# πŸ”Ž Data Query & Hybrid Indicator Search + +This document explains, at a conceptual level, how StatGPT's **Data Query** tool turns a natural-language +request into a grounded SDMX data result β€” and how its **hybrid indicator search** finds the right statistical +indicators across many datasets by combining keyword search, semantic search, and LLM reasoning. + +It is the conceptual companion to the engineering deep dive in +[**Data Query Internals**](./data-query-internals.md), which covers the same machinery at the component and +code level. If you are new to the platform, start here; if you are extending the pipeline, read this first and +then the internals doc. + +> **Related reading:** [Agent Design](./agent.md) Β· [Tools](./tools.md) Β· +> [SDMX Compatibility](./sdmx-compatibility.md) Β· Admin learning track: +> [Indicator Configuration](../learning/administration/03b-indicator-configuration.md), +> [Indexing & Operations](../learning/administration/06-indexing-and-operations.md). + +--- + +## 1. Overview & Scope + +### What the Data Query tool does + +Data Query is one of the tools the StatGPT [agent](./agent.md) can call. It accepts a single natural-language +query (e.g. *"unemployment rate in Spain since 2015"*) and returns: + +- an **agent-facing data summary** β€” the actual values, fed back into the agent's context so its answer is + grounded in real data rather than hallucinated; and +- **user-facing attachments** β€” tables, charts, a downloadable CSV, the underlying SDMX query, and reusable + Python code. + +When a request cannot be answered cleanly, the tool returns a structured outcome instead β€” *no data*, *multiple +candidate datasets* (asking the agent to disambiguate), *missing required information*, or *time period out of +range* β€” so the agent can respond helpfully rather than guess. + +### Why "hybrid" + +The hardest part of answering a statistics question is finding **which indicator, in which dataset** the user +actually means. Three search strategies each cover a different failure mode, and StatGPT combines all three: + +| Strategy | Strengths | Where it fails alone | +|----------|-----------|----------------------| +| **Keyword (lexical) search** | Exact terms, acronyms, codes, units | Misses paraphrases and synonyms ("jobless rate" vs "unemployment") | +| **Semantic (vector) search** | Synonyms, paraphrase, intent | Can drift toward "topically near but wrong"; weak on rare exact tokens | +| **LLM relevance reasoning** | Judges true relevance, handles composite indicators, prefers general vs specific appropriately | Too expensive to run over thousands of candidates directly | + +Hybrid search uses keyword + semantic retrieval to produce a small, high-recall candidate set, then has an LLM +**judge** that set for relevance. This is more accurate than any single method and keeps LLM cost bounded. + +### Audience & prerequisites + +This document assumes basic familiarity with [SDMX](./sdmx-compatibility.md) (datasets, dimensions, code lists) +and the StatGPT [agent model](./agent.md). No code knowledge is required. + +--- + +## 2. Key Concepts + +A handful of terms recur throughout. Understanding them up front makes the rest of the document straightforward. + +| Term | Meaning | +|------|---------| +| **Indicator (composite / virtual)** | There is **no single SDMX "indicator" dimension**. An administrator marks one or more code-list dimensions as `INDICATOR`. The *combination* of their values is what we call an indicator. So an "indicator" is a composite concept assembled from several SDMX dimensions, not a column you can point at. | +| **Dimension classes** | Every dataset dimension is classified as one of four types: **`INDICATOR`** (forms the composite indicator), **`NON_INDICATOR`** (ordinary filters such as *reference area / country* and *frequency*), **`TIME_PERIOD`**, or **`SPECIAL`** (large hierarchical code lists handled by a dedicated processor). | +| **Matching index vs Indicators index** | Two per-channel keyword (Elasticsearch) indices. The **matching index** holds the normalized name of every indicator. The **indicators index** (called the *"harmonized"* index in code) adds a canonical **primary** name on top. At query time, keyword search runs against the *indicators* index. | +| **Vector store** | A PostgreSQL + `pgvector` collection holding embeddings of indicator names. It powers the **semantic** half of hybrid search. | +| **Normalization vs Harmonization** | Two offline cleanup steps. **Normalization** rewrites an indicator name (expands acronyms, standardizes percent/currency wording, lowercases). **Harmonization** derives the canonical **primary** concept used for keyword matching. | +| **Fusion** | How keyword and semantic results are combined into one ranked list β€” a weighted blend of the two normalized scores, leaning toward the semantic side. | +| **Two score systems** | A **similarity score** (numeric, from fusion) decides *which* candidates reach the LLM and in what order. A separate **LLM relevance score** (an integer rating) decides which candidates are actually *kept*. Conflating these two is the most common source of confusion. | +| **Availability query** | An SDMX request that, given a partial selection, both narrows the valid values of chosen dimensions and lists available values for the others. StatGPT repeatedly intersects candidate selections with availability so the query it shows always matches the data it returns. | + +--- + +## 3. End-to-End Flow + +A Data Query call moves through five conceptual stages. The agent sees the data after **every** call; the user +sees attachments once, at the end of the turn. + +```mermaid +flowchart TD + A[Agent emits Data Query tool call
natural-language query] --> B[Search Preparation
normalize Β· pick datasets Β· entities Β· time] + B --> C[Non-indicator dimension search
countries, frequency, …] + C -->|country expected but none found| Z1[No data for that area] + C --> D{Indicator + Special-dimension search
run in parallel} + D --> E[Hybrid indicator search
keyword + semantic + LLM] + D --> F[Special dimensions
large hierarchical code lists] + E --> G[Merge selections Β· re-check availability] + F --> G + G --> H[Construct and complete dataset queries
apply time period] + H --> R{Outcome routing} + R -->|one or more valid queries| EX[Execute SDMX queries
all valid datasets] + R -->|multiple valid datasets
and clarification enabled| Z3[Ask which dataset] + R -->|no queries built| Z2[No data] + R -->|required info missing| Z4[Ask for clarification] + R -->|requested period unavailable| Z5[Explain available range] + EX --> I[Ground the agent
data summary into context] + EX --> J[Display attachments
table Β· chart Β· CSV Β· query Β· code] +``` + +| Stage | What happens | +|-------|--------------| +| **1. Search preparation** | The query is normalized (acronyms expanded), explicit dataset references are detected and stripped, named entities (countries, etc.) are extracted, and any time period is parsed. | +| **2. Non-indicator search** | Countries, frequency, and similar filters are resolved first, seeding the working selection and establishing which datasets are viable. | +| **3. Indicator + special-dimension search** | Hybrid indicator search and special-dimension (large code list) selection run in parallel; their results are merged into the working selection. | +| **4. Construction & routing** | Remaining dimensions are filled from defaults and availability, the time period is applied, and the result is routed to one of the outcomes below. | +| **5. Execution & output** | Valid queries are executed against the SDMX source; the data is summarized back to the agent and rendered as attachments for the user. | + +### Outcome branches + +| Outcome | When | What the user gets | +|---------|------|--------------------| +| βœ… **Execute** | At least one valid, complete query. **All** valid datasets are executed and returned together. | Data summary + attachments | +| ❓ **Multiple datasets** *(optional)* | More than one valid dataset **and** dataset clarification is enabled (a configurable channel option). | The agent is prompted to pick one dataset or ask the user. **When clarification is disabled, this step is skipped and every matching dataset is returned via Execute** β€” no forced choice. | +| ⚠️ **Incomplete** | No valid query could be built because a required dimension is unresolved | A clarification prompt with tables of available values | +| πŸ“… **Invalid time period** | No valid query, and a built query's requested period is outside the available range | The available range is explained | +| ❌ **No data** | No query could be built at all | A "no relevant data" message | + +> Routing is ordered and the branches are mutually exclusive. The optional **Multiple datasets** clarification +> takes precedence when enabled; otherwise **Execute** handles one *or many* valid datasets. **Incomplete** and +> **Invalid time period** are only reached when there are no valid queries at all. + +--- + +## 4. How Hybrid Indicator Search Works + +Hybrid search has **two halves**: an offline **indexing** side that makes indicators searchable, and a runtime +**retrieval-and-selection** side that answers each query. + +### 4.1 Indexing side (offline build) + +When a dataset is indexed for a hybrid channel, StatGPT enumerates every available combination of the dataset's +`INDICATOR` dimensions β€” each combination becomes one indexable indicator β€” and runs a two-phase pipeline: + +```mermaid +flowchart LR + S[Indicator combinations
per dataset] --> N + + subgraph P1[Phase 1 Β· Normalize] + N[Clean and lowercase name
expand acronyms, units] + end + + N --> ES1[(Keyword:
matching index)] + N --> VS[(Semantic:
vector store)] + + ES1 --> H + + subgraph P2[Phase 2 Β· Harmonize] + H[Derive canonical 'primary' name] + end + + H --> ES2[(Keyword:
indicators index)] +``` + +- **Phase 1 β€” Normalize.** Each indicator's name is cleaned by an LLM (acronyms expanded, percent/currency + wording standardized) and lowercased. The normalized record is written to **both** the keyword *matching + index* and the *vector store*. This is what makes every indicator simultaneously findable by exact wording + and by meaning. +- **Phase 2 β€” Harmonize.** A canonical **primary** name is derived for each indicator β€” either by an LLM that + inspects similar indicators across the channel, or structurally from the dimension names β€” and written to the + keyword *indicators index*. The primary is the field keyword search matches against at runtime. + +How the primary is derived is controlled per dataset by two administrator flags, `unpack` and `super_primary` +(see [Β§6](#6-configuration-at-a-glance) and the +[Indicator Configuration guide](../learning/administration/03b-indicator-configuration.md)). + +> Indexing happens only for channels configured as **hybrid**. A *semantic* channel builds embeddings only (no +> keyword indices, no harmonization). Re-indexing is triggered automatically when an +> indexing-relevant part of the configuration changes; cosmetic changes do not trigger it. See +> [Indexing & Operations](../learning/administration/06-indexing-and-operations.md). + +### 4.2 Runtime side (retrieval, fusion, selection) + +For each query, the searcher first cleans the query text and splits it into independent **subqueries** (so +*"unemployment and inflation"* is handled as two indicator searches). Each subquery then runs this pipeline: + +```mermaid +flowchart TD + Q[Subquery] --> L[Keyword search
indicators index] + Q --> S[Semantic search
vector store] + L --> AV[Filter by availability
drop indicators with no data] + S --> AV + AV --> F[Fuse scores
semantic-weighted blend] + F --> D[Diversify
spread across datasets and concepts] + D --> J[LLM relevance rating
0–3 per candidate] + J --> K[Keep best-rated per dataset] + K --> M[Selected indicators
β†’ dataset queries] +``` + +1. **Retrieve** the top candidates by keyword (against the indicators index) and by semantics (against the + vector store), in parallel. +2. **Filter by availability** *before* combining β€” an indicator is dropped if its dataset or its specific + value combination has no data for the already-chosen filters. Hybrid selection is therefore always gated by + real data availability. +3. **Fuse** the two result sets into one ranked list using a weighted blend that leans toward the semantic + side, so semantics drive recall while keyword scores refine the ordering. +4. **Diversify** the ranked list so no single dataset or concept monopolizes the candidate budget, with a + safety net that re-includes the strongest candidates if diversification dropped them. +5. **Rate with an LLM.** The candidate list is shown to an LLM that scores each one for relevance on a small + integer scale (irrelevant β†’ ideal). General questions are steered toward general indicators rather than + overly specific ones. +6. **Select.** Only the best-rated indicators per dataset are kept (and a dataset whose best candidate is only + weakly relevant is dropped entirely). The survivors become the indicator portion of the dataset query. + +> **The two score systems again:** the numeric fusion score from step 3 only governs *which* candidates reach +> the LLM and their order. The integer **LLM relevance rating** from step 5 governs what is actually *kept*. +> They are independent. + +### 4.3 Why this design + +- **Semantic-anchored, keyword-refined** β€” semantics provide recall (catching paraphrase); keyword scores + sharpen ranking (rewarding exact terms and units). +- **LLM as judge, not retriever** β€” the LLM never scans the whole index; it only rates a small, pre-filtered, + diversified set, which keeps it both accurate and affordable. +- **Availability-gated** β€” because candidates are filtered against real availability before selection, the + pipeline rarely proposes an indicator that turns out to have no data. + +--- + +## 5. Dimensions & Availability + +Selecting an indicator is only part of building a query. A complete SDMX query also pins the other dimensions. + +- **Indicator dimensions** are resolved by hybrid search (above). +- **Non-indicator dimensions** β€” most importantly *reference area / country* and *frequency* β€” are resolved + from the named entities found in the query, validated by an LLM, and (for countries) used to drop datasets + that have no data for the requested area. +- **Special dimensions** are large hierarchical code lists (e.g. industry classifications) that are too big to + index as indicators and too structured for entity extraction. A dedicated processor retrieves candidate codes + and has an LLM pick the right ones. +- **Virtual dimensions** and **"all values"** let configuration inject a fixed value (e.g. a country for a + national-agency dataset) or explicitly request every value of a dimension. + +All of these streams produce the same currency β€” a per-dataset set of dimension selections β€” which is +repeatedly **intersected with availability** so the final query matches the data that actually exists. Any +dimensions still unset are filled from configured defaults (and, where the option set is small enough, by +auto-selecting all available values). If a *required* dimension cannot be resolved, the query is reported as +incomplete rather than executed. + +See [SDMX Compatibility](./sdmx-compatibility.md) for how these selections map onto SDMX REST requests. + +--- + +## 6. Configuration at a Glance + +Hybrid search behavior is governed by configuration at three layers. Exact field names and current defaults +live in the code and the admin guides; this is the conceptual map. + +| Layer | Controls | Examples | +|-------|----------|----------| +| **Channel** | Which search strategy is used, and how it is tuned | *index version* (semantic vs hybrid), *indicator selection version* (hybrid vs the LLM-only variants), fusion weights, candidate limits, relevance thresholds, special-dimension processors | +| **Dataset** | Which dimensions form the indicator and how names are harmonized | which dimensions are `INDICATOR` / `NON_INDICATOR` / `SPECIAL` / `TIME_PERIOD`, the country and frequency subtypes, required dimensions, defaults, and the `unpack` / `super_primary` harmonization flags | +| **Infrastructure** | The search backends | Elasticsearch connection (keyword indices), PostgreSQL + `pgvector` (vector store), the embedding model | + +For administrator-facing guidance β€” how to choose `INDICATOR` dimensions, set `unpack`/`super_primary`, and run +indexing and deduplication β€” see the learning track: + +- [Core Concepts](../learning/administration/01-core-concepts.md) +- [Dimension Types](../learning/administration/03a-dimension-types.md) +- [Indicator Configuration](../learning/administration/03b-indicator-configuration.md) +- [Indexing & Operations](../learning/administration/06-indexing-and-operations.md) + +--- + +## 7. Where to Go Next + +- **Implementation detail** β€” every component, control-flow path, and data structure described here is covered + at the code level in [**Data Query Internals**](./data-query-internals.md). +- **Agent context** β€” how the Data Query tool is called and how its results are grounded into the conversation: + [Agent Design](./agent.md) and [Tools](./tools.md). +- **Data layer** β€” what the platform expects from SDMX sources: [SDMX Compatibility](./sdmx-compatibility.md). diff --git a/architecture/data-query-internals.md b/architecture/data-query-internals.md new file mode 100644 index 0000000..86ef7d6 --- /dev/null +++ b/architecture/data-query-internals.md @@ -0,0 +1,433 @@ +# πŸ› οΈ Data Query Internals (Engineering Deep Dive) + +This is the engineer-facing companion to [**Data Query & Hybrid Indicator Search**](./data-query-hybrid-search.md). +It maps the pipeline to concrete components, control flow, and data structures. Read the overview first for the +*why*; this document is the *how*. + +It refers to the implementation by **component, class, and method names** rather than by line numbers, so it +stays useful as the code evolves. The navigation table below points at the relevant packages in the +[`statgpt-backend`](https://github.com/epam/statgpt-backend) repository (paths relative to the repo root); use +your editor's symbol search to jump to a named component. + +> **Map:** [Agent Design](./agent.md) Β· [Tools](./tools.md) Β· [SDMX Compatibility](./sdmx-compatibility.md) + +--- + +## 1. Orientation + +### Where things live + +| Concern | Package | +|---------|---------| +| Tool entry & agent integration | `statgpt/app/chains/data_query/`, `statgpt/app/chains/tools.py`, `statgpt/app/chains/supreme_agent.py` | +| Pipeline orchestration | `statgpt/app/chains/data_query/query_builder/` | +| Search preparation | `…/query_builder/misc/` | +| Dimension search & merge | `…/query_builder/dimensions/`, `…/query_builder/special_dimensions_selection/` | +| Indicator selection | `…/query_builder/indicator_selection/` | +| Runtime hybrid search engine | `statgpt/app/services/hybrid_searcher.py` | +| Offline hybrid indexer | `statgpt/common/hybrid_indexer/` | +| Vector store (pgvector) | `statgpt/common/vectorstore/` | +| Keyword index (Elasticsearch) | `statgpt/common/utils/elastic.py`, `statgpt/common/settings/elastic.py` | +| Finalize / construct / execute | `…/query_builder/query/`, `…/data_query/query_constructor/` | +| SDMX data layer | `statgpt/common/data/sdmx/`, `statgpt/common/data/base/query.py` | +| Config & data model | `statgpt/common/data/base/config.py`, `statgpt/common/schemas/data_query_tool.py`, `statgpt/common/models/models.py` | + +### Key data structures + +| Type | Role | +|------|------| +| `ChainState` | The dict-state threaded through every pipeline stage. | +| `DataSetAvailabilityQuery` / `DimensionQuery` / `Query` | Per-dataset `{dim_id β†’ Query(values, operator)}`. The common currency of the whole pipeline; the operator is `IN` / `ALL` / a time operator. | +| `DataSetQuery` | An executable query carrying `is_valid` / `invalidity_reason`. | +| `ComplexIndicator` / `CodeIndicator` | The composite indicator β€” an ordered list of `(dimension, code)` values. | +| `MatchingIndex` / `IndicatorIndex` | Keyword index documents. `IndicatorIndex` adds `primary` / `primary_normalized` on top of `MatchingIndex`. | +| `DataQueryArtifact` | What the tool returns: `data_responses` + `state` + `eval_attachment`. | + +### The threaded state + +The pipeline is a LangChain (LCEL) composition over a single mutated dict, abstracted as `ChainState`. Stages +read keys written by earlier stages (`normalized_query`, `named_entities`, `strong_queries`, +`strong_availability`, …) and write their own. The most important working keys: + +- `strong_queries` β€” the progressively built per-dataset selection. +- `strong_availability` β€” cached availability per dataset. +- `strong_queries_best_nonempty_attempt` β€” last non-empty snapshot, used for fallback messaging. + +--- + +## 2. Tool Entry & Agent Integration + +### Schema exposed to the LLM + +`DataQueryArgs` defines a single LLM-visible field, `query: str`, with a description steering the model to send +one concise query per indicator while allowing multiple values for countries and other dimensions. The tool +**name and description** themselves are *not* hardcoded β€” they come from channel config via +`StatGptTool.from_config`; only the argument shape is fixed. + +The cross-cutting execution context (`auth_context`, `choice`, `history`, `configuration`, target stage, and the +query) is passed as an injected `inputs` argument marked `InjectedToolArg`, so it is set in code by +`ToolCaller.call_tool` and stripped from the public schema β€” the LLM never sees it. `SupremeAgent` binds tools +with `model.bind_tools(..., strict=True)` so the provider enforces the argument schema. + +### Execution + +`DataQueryTool._arun` builds a `QueryBuilderFactory` from the tool config, sets the query input, invokes the +chain, then returns `(response_str, DataQueryArtifact)`. Because `StatGptTool` sets +`response_format='content_and_artifact'`, the returned artifact rides on the tool message's `artifact`. + +### Two output channels + +`DataQueryArtifactDisplayer` produces two distinct outputs from the merged `DataResponse`s: + +- **Agent-facing** β€” `get_system_message_content` builds a TSV `` block that is injected as a system + message after **every** data-query round so the agent reasons over exact values. Columns are prefixed + `DIMENSION:` / `ATTRIBUTE:`, coded values are split into `_ID`/`_Name`, and the block is capped (see + `tool_response_max_cells` on `DataQueryDetails`); oversized or empty results emit a short message instead of + the table. +- **User-facing** β€” `display` is called only on the **final** agent turn (no further tool calls) and uploads + attachments: custom TTYD table, CSV, Plotly grid + per-indicator graphs, JSON query, and Python code. + Per-type toggles live in `DataQueryAttachments`. + +The loop itself (`SupremeAgentExecutor`) gathers tool calls concurrently, collects `DataQueryArtifact`s by tool +call id, injects the grounded system message after data-query rounds, and calls `display` once at the end. + +```mermaid +flowchart LR + LLM[LLM tool call] --> TC[ToolCaller
inject inputs] + TC --> AR[DataQueryTool._arun] + AR --> QB[QueryBuilder chain] + QB --> ART[DataQueryArtifact
on tool message artifact] + ART --> SM[get_system_message_content
every round to agent] + ART --> DISP[display
final turn to user] +``` + +--- + +## 3. Pipeline Orchestration + +`QueryBuilderFactory` composes three sub-chains as an LCEL pipe: + +``` +search_preparation | dimensions_search | finalize_query +``` + +| Sub-chain | Factory | Role | +|-----------|---------|------| +| Search preparation | `SearchPreparationChainFactory` | Query understanding (Β§4) | +| Dimension search | `DimensionSearchChainFactory` | Resolve all dimensions (Β§5) | +| Finalize | `FinalizeQueryChainFactory` | Construct, time-filter, route, execute (Β§9) | + +`set_tool_state` serializes the `QueryBuilderAgentState` and the debug eval attachment near the end of +finalization. + +--- + +## 4. Search Preparation (Query Understanding) + +`SearchPreparationChainFactory` sequences: + +1. **Get available datasets** β€” the versioned dataset dict for the channel. +2. **Normalization** β€” `NormalizationChain` expands commonly-known acronyms (GDP, CPI) without expanding country + groups into members, writing `normalized_query`. +3. **Dataset selection** β€” `DataSetsSelectionChain` detects explicit dataset references, maps the LLM's 1-based + indexes to dataset UUIDs, and **overwrites** `normalized_query` with a dataset-reference-stripped rewrite. The + pre-strip version is preserved as `normalized_query_raw`. Hallucination guards are double-layered: the LLM + post-processor drops unknown indexes, and the apply step re-checks each id against the dataset dict. +4. **NER + datetime in parallel** β€” `NamedEntitiesChain` runs NER restricted to channel-configured entity types; + `DateTimeDimensionChain` extracts temporal intent and start/end bounds. The datetime LLM must **not** infer + missing bounds β€” post-processing fills them in code only for historical (end = today) / forecast + (start = today) intents and clamps to the current period where flagged. +5. **Country entities** β€” the country-typed named entities (selected with a `startswith` heuristic flagged as a + temporary workaround) are extracted for downstream country filtering. + +`DateTimeQueryResponse.to_query` maps the parsed period to a `TIME_PERIOD` `DimensionQuery` (`BETWEEN`/`GTE`/ +`LTE`), applied depending on `time_period_strategy` (Β§9). + +> `GroupExpanderChain` for reference-area group expansion is **dormant**: its `create_chain` raises +> `NotImplementedError` and its call site is commented out. + +--- + +## 5. Dimension Search & Merge + +`DimensionSearchChainFactory` orchestrates dimension resolution. Shared helpers β€” concurrent availability fetch, +availability-based filtering, and best-attempt bookkeeping β€” live in `DimensionSearchChainFactoryBase`. + +### Order of operations + +1. **Non-indicator search runs first** to seed `strong_queries` and establish viable datasets. + `NonIndicatorsSearchChainFactory`: + - per non-`dataset` named entity, vector-searches non-indicator dimension values, scoped to the selected + version ids; + - augments with synthetic "All values" candidates; + - validates with an LLM (`CandidatesSelectionSimpleChainFactory` β†’ `SelectedCandidates`) and propagates the + decision to de-duplicated candidates; + - applies country filtering (GitHub issue #75): if any dataset got a country value, datasets lacking one are + dropped; gated by `DataQueryDetails.filter_by_country_entities`. + +2. **Routing** β€” if a country entity exists but no `strong_queries` survive, short-circuit to a + "no data for {country}" message. + +3. **Indicator + special dimensions in parallel** (`RunnableParallel`): + - `IndicatorsSearchChainFactory` dispatches through `IndicatorSelectionFactory` (Β§6/Β§7), **overwrites** + `strong_queries` with the selected indicator queries, then filters by required dimensions and by + availability. + - `SpecialDimensionsSearchChainFactory` runs the configured special-dimension processors β€” currently only + `LHCLChainFactory` (Large Hierarchical Code Lists): vector retrieval of code candidates β†’ grounded LLM + selection (de-hallucinated) β†’ per-dataset `DimensionQuery`. + +4. **Merge & re-filter** β€” special-dimension `IN`-queries are folded into `strong_queries` (unioning values; an + operator other than `IN` is rejected), then availability is recomputed and intersected again so displayed and + executed queries stay consistent. + +### Dimension model + +`DimensionType` has exactly four values: `INDICATOR`, `NON_INDICATOR`, `TIME_PERIOD`, `SPECIAL`. Config +validators enforce β‰₯1 indicator dimension, exactly one time and one frequency dimension, at most one region +(country) dimension, and unique special processor ids. + +- **The indicator is composite.** `Sdmx21DataSet` enumerates available combinations of `INDICATOR` dimensions + into `ComplexIndicator`s. Selecting one yields per-real-dimension value queries. +- **Virtual dimensions** inject a fixed value and are **skipped** when building the SDMX key. **"All values"** + maps a selected synthetic id to `QueryOperator.ALL`, which appends no filter for that dimension. +- **Availability is the narrowing core.** `DataSetAvailabilityQuery.filter` intersects only `IN`-operator + dimensions; `ALL`/time operators pass through. Datasets whose availability is empty are dropped. + +--- + +## 6. Indicator Selection Factory + +`IndicatorSelectionFactory` dispatches on `DataQueryDetails.indicator_selection_version`. This document covers +the **hybrid** path (`IndicatorSelectionVersion.hybrid`); the semantic LLM-selection variants are out of scope +here. + +The hybrid factory method resolves the two Elasticsearch indices via `ElasticSearchFactory` (the matching and +indicators index names are per-channel, derived on the `Channel` model), builds a `HybridSearcher` with the +channel's `HybridSearchConfig`, and wraps it in `IndicatorsSelectionHybrid`. + +`IndicatorsSelectionHybrid` calls `HybridSearcher.search`, exposes four debug retrieval stages (lexical / +semantic / llm_scored / final), and in its finalization step deep-copies each per-dataset entry in +`strong_queries` and adds the selected indicator `DimensionQuery`s. + +--- + +## 7. Runtime Hybrid Search Engine + +`HybridSearcher` (in `statgpt/app/services/hybrid_searcher.py`) is the core engine; its inner `HybridMatch` +class runs the per-subquery pipeline. The three LLM prompts (`normalizationPrompt`, `separateSubjectsPrompt`, +`relevancyPrompt`) live in the hybrid-search prompt asset. + +### Entry: `HybridSearcher.search` + +1. Compute `version_ids` from the selected datasets and an availability map (`dataset_id β†’ dim_id β†’ + set(values)`) from the strong-availability queries. +2. **Pre-match** β€” a lexical pre-match planner runs Elasticsearch highlighting plus a `primary` terms + aggregation to find multi-token "good candidates" vs single-token ones, deduped via ES token analysis. These + become protected "forbidden" phrases. +3. **Normalize input** β€” strips removable named entities (per `named_entities_to_remove`) and time text while + protecting the forbidden phrases, and lowercases. +4. **Separate subjects** β€” split into independent indicator subqueries (so "unemployment and inflation" becomes + two searches). +5. Run each subquery's `HybridMatch.search` concurrently (each with its own buffered DIAL stage to avoid + interleaved output). + +### Per-subquery: `HybridMatch.search` + +| Step | Method | Notes | +|------|--------|-------| +| Query planner | `_query_planner` | Picks search parameters (fusion weight + candidate counts): fall back to near-pure semantic when there are no good lexical candidates; a more balanced blend when there are many. | +| Lexical | `_lexical` | Boolean query against the **indicators** index: *must* match `primary_normalized`, *should* match `name_normalized` (down-weighted), filtered by version. Min-max normalized against the result-set max. | +| Semantic | `_semantic_raw` / `_semantic_result` | `VectorStore.search_with_similarity_score` over pgvector, scoped by version; score = `1 βˆ’ distance`. Min-max normalized. | +| Availability filter | `_filer_candidates_by_availability` | Applied to **both** sets **before** fusion: drop any candidate whose dataset is absent from availability, or whose series contains a `(dimension, value)` pair not in the availability set. | +| Fusion | `_hybrid_combination` / `_convex_combination` | **Not reciprocal rank fusion.** `score = alphaΒ·semantic + (1βˆ’alpha)Β·lexical`; `alpha` weights the **semantic** side. The fused set is **anchored on semantic results** β€” it iterates the availability-filtered semantic docs, so a purely lexical hit with no semantic neighbor in the top-k never enters fusion (recall depends on the semantic candidate count). | +| Diversify | round-robin helpers | Round-robin across datasets and `primary_normalized` groups so no dataset/concept monopolizes the candidate budget; a safety net force-includes the global top-N if diversification dropped them. | +| LLM relevance | `_relevance_candidates` | Renumber, batch, render as a primary-grouped markdown tree (with a synthetic best-so-far item for cross-batch context), and call the relevancy chain. The rubric scores **0–3** (3 = ideal … 0 = irrelevant; general queries should prefer general indicators). | +| Select | `_filter_candidates` | The keep/drop decision uses the **integer LLM score**, not the fusion float. A candidate is kept iff its score equals its dataset's max (or the global max when `use_only_best_score`), and a dataset is admitted only if its best score meets the configured threshold. | + +The survivors are collapsed into `dataset_id β†’ dim_id β†’ set(codes)`. Across subqueries, the per-dataset/dimension +sets are unioned into `DimensionQuery(operator=IN)`. + +### The two score systems + +This is the most important thing to internalize: + +```mermaid +flowchart LR + LEX[lexical score
min-max, min 0] --> FUSE + SEM[semantic score
min-max, min -1] --> FUSE + FUSE[fusion: alpha-weighted
FLOAT score] -->|decides which reach the LLM
and their order| LLM + LLM[LLM relevance
INTEGER 0 to 3] -->|decides what is kept| OUT[selected indicators] +``` + +- **Float fusion score** β€” governs candidate *admission and ordering* into the LLM step only. +- **Integer LLM relevance score** β€” governs *final keep/drop*. + +They are computed independently and serve different purposes. All tunables (the `alpha` variants, candidate +limits, batch size, score thresholds, `use_only_best_score`) are fields of `HybridSearchConfig`. + +--- + +## 8. Offline Hybrid Indexer + +`Indexer` (in `statgpt/common/hybrid_indexer/`) builds the searchable representation. It runs only for +**hybrid** channels (gated by `ChannelService.is_channel_hybrid`) and is driven as two background phases by the +admin dataset service. A *semantic* channel instead runs a plain embedding indexer (no Elasticsearch, no +harmonization). + +### Series + +The indexer builds an internal `_Series` β€” one `DimensionQuery` per `CodeIndicator`. The human-readable +indicator name joins component term names, **skipping** `ignored_term_ids` (default the SDMX "Not applicable" +code) so they don't pollute names β€” though those values are still recorded in the series' `where`. The series id +is deterministic, and the harmonized `IndicatorIndex` reuses it, giving a 1:1 mapping between the matching and +indicators documents. Series serialization persists only the first value of each `DimensionQuery`. + +### Phase 1 β€” Normalize + +An LLM cleans the name (expand acronyms, standardize percent/currency) and lowercases it into `name_normalized`. +A safe-invoke wrapper makes a single LLM failure fall back to a lowercased input rather than aborting the batch. +The `MatchingIndex` document is written to **both** the Elasticsearch matching index (upsert by id) and the +pgvector vector store (with `name_normalized` as the embedded content). + +### Phase 2 β€” Harmonize + +Derives a canonical `primary` and writes an `IndicatorIndex` (matching document + `primary` + +`primary_normalized`) to **only** the Elasticsearch indicators index. Two strategies, per the dataset's indexer +indicator config: + +- `unpack=True` β†’ hybrid-search similar normalized indicators across sibling versions (lexical Elasticsearch + + semantic vector store, fused with the indexing-time `indexer_alpha`), round-robin diversified across datasets, + then have the harmonize LLM extract the primary. +- `unpack=False` β†’ derive the primary structurally from dimension names β€” the first dimension, or + (`super_primary=True`) the first three concatenated. + +> Harmonization uses **cross-dataset context**: the searched version set includes other datasets' latest +> completed versions so the unpack search can find similar indicators channel-wide. + +### Index topology + +```mermaid +flowchart TD + series[Indicator series] --> N[Phase 1: normalize] + N --> MI[(ES matching index
name_normalized)] + N --> VS[(pgvector store
embeddings of name_normalized)] + MI --> H[Phase 2: harmonize] + H --> II[(ES indicators index
plus primary_normalized)] +``` + +Two Elasticsearch indices per channel plus the vector collection. At **runtime**, lexical search hits only the +*indicators* index; the *matching* index is used solely for token analysis; semantic search hits pgvector. + +### Reindex lifecycle + +An indexing hash (`compute_indexing_hash`) covers only the configuration fields marked as indexing-relevant +(dimension type, alias, virtual flag, special processor id, non-indicator subtype, and the `unpack`/ +`super_primary` indexer flags). A change to one of those flips the dataset's status to needs-reindex; +display-only fields (`is_required`, `default_queries`, `all_values`, citation, …) apply silently. The +`ChannelDatasetVersion` model stores the hashes and indexing stats. Operationally this is driven from the CLI: +`channel reindex`, `channel deduplicate`, and `channel status`. + +--- + +## 9. Availability, Construction & Execution + +`FinalizeQueryChainFactory` turns the merged selections into executable queries and routes the outcome. + +### Construction + +Empty `strong_queries` are dropped; for each remaining dataset, `QueryConstructorFactory` returns a +`CompositeQueryConstructor` wrapping `[Simple, Iterative]`: + +- `SimpleQueryConstructor` fills missing dimensions from default queries / available values in one pass. +- `IterativeQueryConstructor` re-runs availability after each default/time assignment (because narrowing one + dimension changes what's available for others). +- `CompositeQueryConstructor` returns the first valid query with non-empty availability; it falls through from + Simple to Iterative on a valid-but-empty-availability result. + +When a dimension without defaults has a small enough option set it is auto-selected (operator `ALL`); above the +threshold the query is left incomplete. These thresholds are constructor constants. + +### Time period + +`time_period_strategy` (`BEFORE` vs `AFTER`) decides when time is applied. Under `AFTER`, a post-construction +filter re-runs availability and either applies the selected period (marking queries `INVALID_TIME_PERIOD` when +outside the available range, recording the offending field/value and the available bounds) or the dataset +default. A time-range expander snaps bounds to frequency-aligned period start/end. + +### Routing + +Routing is **ordered and mutually exclusive**: + +| Order | Branch | Class | Condition | +|-------|--------|-------|-----------| +| 1 | No data | `NoDataChain` | No dataset queries at all. | +| 2 | Multiple datasets *(optional)* | `MultipleDatasetsChain` | `clarify_if_multiple_datasets` is enabled **and** more than one valid query exists. The agent is asked to pick one dataset or ask the user. | +| 3 | Execute | `ExecuteQueryChain` | One **or more** valid queries. When clarification is disabled, this branch executes **all** valid datasets together β€” there is no forced dataset choice. | +| 4 | Invalid time period | `InvalidSelectedTimePeriodChain` | No valid query, but at least one carries an `invalidity_reason`. | +| 5 | Incomplete | `IncompleteQueriesChain` | No valid query, due to unresolved required dimensions; emits a clarification with tables of available values. | + +Because the Execute branch catches "one or more valid queries", branches 4 and 5 are only reached when there are +**zero** valid queries. The Multiple-datasets clarification is the *only* thing that diverts multiple valid +datasets away from a single combined execution, and it is opt-in via `clarify_if_multiple_datasets`. + +```mermaid +flowchart TD + R{Routing} --> Q1{any dataset
queries?} + Q1 -->|no| ND[No data] + Q1 -->|yes| Q2{clarify enabled
and multiple valid?} + Q2 -->|yes| MD[Ask which dataset] + Q2 -->|no| Q3{one or more
valid queries?} + Q3 -->|yes| EX[Execute all valid datasets] + Q3 -->|no, has invalidity reason| IT[Invalid time period] + Q3 -->|no, missing required dims| IN[Incomplete: clarify] +``` + +### Execution + +`ExecuteQueryChain` runs the data query per dataset concurrently. `Sdmx21DataSet.query` builds the SDMX REST key +(DSD dimension order, values joined by `+`, dimensions by `.`; time via `startPeriod`/`endPeriod`), fetches the +data message, parses to a pandas dataframe, and returns an `Sdmx21DataResponse` with request/parsing status +(failures are captured, not raised). `availability_query` issues the SDMX `availableconstraint` call +(rate-limited) and parses the result back into a `DataSetAvailabilityQuery`; the SDMX 3.0 proxy reuses the same +parsing path. `SummarizeQueriesChain` adds per-query summaries before the response text is finalized. + +--- + +## 10. Outputs & Artifacts + +Covered structurally in Β§2. The data path, end to end: + +- The tool returns `DataQueryArtifact{data_responses, state, eval_attachment}` on the tool message artifact. +- Responses are merged per dataset via `DataResponse.merge`; the status merge combines request statuses (both + SUCCESS β†’ SUCCESS, both FAILED β†’ FAILED, otherwise PARTIALLY_FAILED), and a partial parse prepends a disclaimer + in the agent TSV. +- User attachments derive from `DataResponse`: a visual dataframe (time unstacked to columns), a CSV dataframe, a + custom TTYD table, a Plotly grid plus per-indicator graphs, the JSON query, and generated Python code + (per-dataset, or merged into one file when configured). +- The debug eval attachment (the retrieval stages) is surfaced as a downloadable JSON only when debug + attachments are enabled. + +--- + +## 11. Configuration Reference + +Knobs by layer, named by their config class so you can look up current fields and defaults in code. + +| Layer | Class | Selected fields | +|-------|-------|-----------------| +| **Channel β€” tool** | `DataQueryDetails` | `indexer_version`, `indicator_selection_version`, `time_period_strategy`, `clarify_if_multiple_datasets`, `filter_by_country_entities`, `candidates_per_entity`, `tool_response_max_cells`, `attachments`, `messages`, `prompts`, `llm_models` | +| **Channel β€” hybrid** | `HybridSearchConfig` | fusion weights (`*_alpha`), candidate limits (lexical / semantic / pre-match / `max_candidates`), `batch_size`, score thresholds, `use_only_best_score`, `named_entities_to_remove`, `indexer_alpha`, `ignored_term_ids`, normalize/harmonize model configs | +| **Channel β€” special dims** | `DataQueryDetails.special_dimensions_processors` | per-processor `id`, `type` (LHCL), `top_k`, `prompt`, `llm_model_config` | +| **Dataset β€” dimensions** | `DataSetConfig` / `BaseDimensionConfig` | `dimension_type`, `subtype` (REGION / FREQUENCY), `alias`, `is_required`, `virtual`, `all_values`, `default_queries`, `processor_id` | +| **Dataset β€” indexer** | `IndexerConfig` / `IndexerIndicatorConfig` | `unpack`, `super_primary`, `annotations` | +| **Infrastructure** | `ElasticSearchSettings`, `PostgresSettings`, `LangChainSettings` | Elasticsearch (keyword indices + analyzers), PostgreSQL + pgvector (vector store), the embedding model | +| **Runtime flags** | `StateVarsConfig` | `SHOW_DEBUG_STAGES`, `CMD_SKIP_TOOLS_EXECUTION`, `CMD_SKIP_DATA_QUERY_SUMMARIZATION` | + +--- + +## See Also + +- [Data Query & Hybrid Indicator Search](./data-query-hybrid-search.md) β€” the conceptual overview. +- [Agent Design](./agent.md) Β· [Tools](./tools.md) Β· [SDMX Compatibility](./sdmx-compatibility.md) +- Admin learning track: [Indicator Configuration](../learning/administration/03b-indicator-configuration.md), + [Indexing & Operations](../learning/administration/06-indexing-and-operations.md). From 12275ad3b1f683c66f6761fc8bdb5f799268eddd Mon Sep 17 00:00:00 2001 From: Daniil Yarmalkevich Date: Fri, 5 Jun 2026 11:33:09 +0300 Subject: [PATCH 2/2] mention relevance threshold --- architecture/data-query-hybrid-search.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/architecture/data-query-hybrid-search.md b/architecture/data-query-hybrid-search.md index 85969be..d1244da 100644 --- a/architecture/data-query-hybrid-search.md +++ b/architecture/data-query-hybrid-search.md @@ -194,11 +194,15 @@ flowchart TD side, so semantics drive recall while keyword scores refine the ordering. 4. **Diversify** the ranked list so no single dataset or concept monopolizes the candidate budget, with a safety net that re-includes the strongest candidates if diversification dropped them. -5. **Rate with an LLM.** The candidate list is shown to an LLM that scores each one for relevance on a small - integer scale (irrelevant β†’ ideal). General questions are steered toward general indicators rather than +5. **Rate with an LLM.** The candidate list is shown to an LLM that scores each one for relevance on a **0–3** + scale (0 = irrelevant, 3 = ideal). General questions are steered toward general indicators rather than overly specific ones. -6. **Select.** Only the best-rated indicators per dataset are kept (and a dataset whose best candidate is only - weakly relevant is dropped entirely). The survivors become the indicator portion of the dataset query. +6. **Select against a relevance threshold.** Only the best-rated indicators per dataset are kept, and a dataset + is retained only if its best candidate reaches a configurable **minimum relevance score** β€” + `single_dataset_score_threshold` / `multi_dataset_score_threshold` (default **2** on the 0–3 scale). A dataset + whose best candidate scores below the threshold is dropped entirely. If **no** candidate in any dataset clears + the bar, indicator search returns nothing β€” which on its own leads to a *no-data* outcome. The survivors + become the indicator portion of the dataset query. > **The two score systems again:** the numeric fusion score from step 3 only governs *which* candidates reach > the LLM and their order. The integer **LLM relevance rating** from step 5 governs what is actually *kept*.