From 08a04bc3de11df91f14cd5b15ad159ff6ed63314 Mon Sep 17 00:00:00 2001 From: Daniil Yarmalkevich Date: Fri, 29 May 2026 11:13:56 +0300 Subject: [PATCH] feat: init claude config for the repo --- .../statgpt-admin-fe-guide-macos/SKILL.md | 154 ++++++++++++++++++ .../references/annotation.md | 89 ++++++++++ .../references/capture-pipeline.md | 85 ++++++++++ .../references/config-and-structure.md | 101 ++++++++++++ .../scripts/annotate.py | 119 ++++++++++++++ .../scripts/cap.sh | 35 ++++ CLAUDE.md | 54 ++++++ 7 files changed, 637 insertions(+) create mode 100644 .claude/skills/statgpt-admin-fe-guide-macos/SKILL.md create mode 100644 .claude/skills/statgpt-admin-fe-guide-macos/references/annotation.md create mode 100644 .claude/skills/statgpt-admin-fe-guide-macos/references/capture-pipeline.md create mode 100644 .claude/skills/statgpt-admin-fe-guide-macos/references/config-and-structure.md create mode 100644 .claude/skills/statgpt-admin-fe-guide-macos/scripts/annotate.py create mode 100755 .claude/skills/statgpt-admin-fe-guide-macos/scripts/cap.sh create mode 100644 CLAUDE.md diff --git a/.claude/skills/statgpt-admin-fe-guide-macos/SKILL.md b/.claude/skills/statgpt-admin-fe-guide-macos/SKILL.md new file mode 100644 index 0000000..c29da0e --- /dev/null +++ b/.claude/skills/statgpt-admin-fe-guide-macos/SKILL.md @@ -0,0 +1,154 @@ +--- +name: statgpt-admin-fe-guide-macos +description: >- + Refresh or rewrite the StatGPT Administrator Guide (guides/admin-guide.md and its annotated + screenshots under guides/content/admin-guide/) to match the StatGPT Admin frontend. Use this + whenever the user asks to update, refresh, re-screenshot, or fix the admin guide; to document + new admin-UI features (data sources, datasets, channels, index versions, auto-update jobs, + indexing/deduplication, glossary, import/export, audit logs); or to regenerate its images — + even when phrased loosely ("the admin guide is outdated", "add screenshots to the admin guide", + "re-do the admin docs") or by just pasting the admin frontend URL. It captures sample-only, + annotated screenshots (numbered badges + highlight boxes), sources config YAML from the + statgpt-backend sample configs in admin (camelCase) format, and links the admin learning course. +compatibility: >- + macOS (uses `screencapture`); the claude-in-chrome MCP with a logged-in admin session; + Screen Recording permission granted; a python with Pillow installed. +--- + +# StatGPT Admin Frontend Guide Updater + +This skill regenerates the **StatGPT Administrator Guide** — `guides/admin-guide.md` plus its +annotated screenshots in `guides/content/admin-guide/` — so the prose, the configuration YAML, and +the images all match the current Admin frontend. It exists because doing this by hand is fiddly: +screenshots leak non-sample content, the Chrome debugging banner corrupts captures, badge +placement is easy to get wrong, and the config format in the guide must match what the admin +actually stores (not the snake_case seed files). + +The output is a rewritten guide with numbered-step prose whose numbers line up with numbered badges +on each screenshot, plus a set of clean, sample-only annotated PNGs. + +## Step 0 — Gather the three inputs + +This skill is **driven by three inputs**. Ask for any that aren't already given, then confirm them +back before doing work: + +1. **Scope** — what to refresh. Either a *full refresh* (every section + new features) or a + *targeted* update (named sections, or "just the new X feature"). Scope decides which screens you + capture and which parts of the guide you rewrite. Don't silently expand scope. +2. **Environment URL** — the admin frontend base URL to capture from + (e.g. `https://statgpt-admin-frontend.aks.dev.dial.parts`). Everything is captured live from + here through the claude-in-chrome MCP. +3. **Limitations** — hard constraints for this run. The defaults below are non-negotiable unless the + user explicitly relaxes one; the user may add more (e.g. "don't touch the channel X"). + +## Hard constraints (the defaults — honor unless the user overrides) + +These protect a **public** docs repo and a **shared** environment. Re-read them before each capture. + +- **Sample content only.** Every screenshot must show *only* sample entities. Filter every list to + the sample data source (e.g. `IMF_SDMX21`), the sample channel (e.g. **StatGPT Sample**), and the + "**- Sample**" datasets. Never let other orgs' content appear (e.g. QH_UAT_SWRE, STATGPT_SDMX30_PROXY, + SwissRe, GTDC, Global Data). If a screen can't be filtered to sample-only (e.g. an "Add datasets to + channel" picker that lists everything and has no source filter), **don't screenshot it** — describe + it in prose instead. Confirm the exact sample identifiers for the target environment up front. +- **Redact PII.** Audit logs and similar screens show real user emails. Mask those columns with a + redaction bar before the image leaves your hands (see `references/annotation.md`). +- **No deletions of any content.** To document a delete action, open the confirm dialog and Cancel. + Beware: some delete actions have **no confirmation dialog** (e.g. glossary terms delete + immediately). If you trigger one by accident, **restore the item immediately** from the seed + configs and tell the user. +- **You may modify / reindex / deduplicate / add SAMPLE content only** — never other channels/sources. + Don't create duplicate sources/datasets/channels when illustrating "Add"; fill the form for the + screenshot, then Cancel. +- **Documents page is out of scope** unless the user says otherwise. +- **No git commits or pushes** unless the user asks. When they do, branch first, stage *only* the + guide + its images (not stray local files), and follow the repo's PR conventions. + +## Workflow + +Work in a scratch dir (e.g. `/tmp/statgpt-admin-guide/{raw,annotated,specs}`) and only copy +finished images into the repo at the end. Track progress with a task list. + +### Phase 1 — Explore and map changes + +Drive the admin UI with the claude-in-chrome MCP (load the deferred `mcp__claude-in-chrome__*` +tools via ToolSearch first; call `tabs_context_mcp` to get the tab id). Walk the target scope's +screens and compare against the current `guides/admin-guide.md`: note new nav items, renamed/added +buttons, changed wizards, new columns, and new pages. Produce a short map of what changed so the +rewrite and the screenshot list are grounded in the live UI, not assumptions. + +### Phase 2 — Capture sample-only screenshots + +This is the part with the sharp edges. **Read `references/capture-pipeline.md` before starting** — +it covers window calibration, the debugging-banner problem, and the two coordinate spaces. The loop +per screenshot: + +1. Drive the UI to the target state via MCP (`browser_batch` of clicks/types/navigate). **Filter + every list to sample content** before capturing. +2. Make the **final MCP action a `hover` to a neutral, empty spot** so the cursor (which + `screencapture` includes) isn't covering content. +3. Capture with the bundled banner-aware script — **no CDP command between the hover and this**: + ``` + CAP_REGION= CAP_PYBIN= \ + scripts/cap.sh /tmp/statgpt-admin-guide/raw/.png + ``` +4. **View the PNG** (Read it) and confirm it's banner-free and shows only sample content. + +Capture the set of screens your scope needs (list pages, add wizards, edit/config modals, context +menus, per-dataset Versions / Auto update jobs pages, indexing dialogs, glossary, import modal, +jobs, audit logs). `references/config-and-structure.md` has the canonical screen inventory and +suggested filenames. + +### Phase 3 — Annotate (numbered badges + highlight boxes) + +Style is **numbered badges + rounded highlight boxes** (numbers match the numbered prose steps), +with **redaction bars** for PII. Use the bundled annotator: + +``` + scripts/annotate.py spec.json +``` + +The spec is JSON with `src`, `dst`, optional `crop`, and `boxes` / `badges` / `redactions` arrays. +**Coordinates are measured from the actual captured PNG**, and getting this right is the single +biggest source of rework — **read `references/annotation.md`**, especially the part about *not* +trusting downscaled previews (measure from clean, full-resolution crops). Verify every rendered +annotation by viewing it; box columns to their *full* visible height; keep badges off the content +they label. + +### Phase 4 — Rewrite the guide + +Rewrite the in-scope parts of `guides/admin-guide.md`: + +- Match the new UI: structure, nav, wizards, new pages and columns. +- Numbered prose steps that **correspond to the badges** in each referenced screenshot. +- **Configuration YAML in admin (camelCase) format**, sourced from the `statgpt-backend` sample + configs — *do not invent values*. The seed files are snake_case; the admin stores camelCase. See + `references/config-and-structure.md` for the field mapping (data source, the dataset `dimensions` + map, channel + tools). When in doubt, the live "Configure"/"Edit" editor is the source of truth — + read it via Monaco `getValue()`. +- When covering dataset configuration, **link the admin learning course** + (`../learning/administration/README.md`, esp. Modules 03a/03b/04/05/06). +- Add a table of contents; fix any broken image links; keep every cross-link resolvable. + +### Phase 5 — Review and (if asked) ship + +Verify all referenced images exist; flag now-orphaned old images (don't delete files you didn't +create without asking). If the user asks to ship: branch, stage only the guide + images, commit +with a Conventional-Commits title, push, and open a PR with the repo's template. + +## Key paths + +- Guide: `guides/admin-guide.md`; images: `guides/content/admin-guide/`. +- Sample configs (source of truth for YAML): `statgpt-backend/configurations/clients/sample/` + (`data_sources.yaml`, `datasets/*.yaml`, `channels.yaml`, `tools.yaml`, `glossaries/*.csv`). +- Learning course: `learning/administration/` (README + numbered modules). +- Bundled `scripts/cap.sh` (capture) and `scripts/annotate.py` (annotate). + +## References (read when you reach the relevant phase) + +- `references/capture-pipeline.md` — window calibration, the Chrome debugging banner, coordinate + spaces, and the capture protocol. **Read before Phase 2.** +- `references/annotation.md` — the annotator spec, coordinate measurement (and the downscaled-preview + trap), badge/box/redaction conventions. **Read before Phase 3.** +- `references/config-and-structure.md` — seed→admin config field mapping, the guide's section + structure, the screenshot inventory, and learning-course links. **Read before Phase 4.** diff --git a/.claude/skills/statgpt-admin-fe-guide-macos/references/annotation.md b/.claude/skills/statgpt-admin-fe-guide-macos/references/annotation.md new file mode 100644 index 0000000..4ea38c3 --- /dev/null +++ b/.claude/skills/statgpt-admin-fe-guide-macos/references/annotation.md @@ -0,0 +1,89 @@ +# Annotation — numbered badges, highlight boxes, redaction bars + +Read this before Phase 3. The annotator itself is simple; getting coordinates right is the work. + +## The annotator (`scripts/annotate.py`) + +Run: ` scripts/annotate.py spec.json` + +Spec JSON (all coords in the captured PNG's pixel space; use `"scale": 1`): + +```json +{ + "src": "/tmp/.../raw/X.png", + "dst": "/tmp/.../annotated/X.png", + "scale": 1, + "crop": [x, y, w, h], + "redactions": [ {"xywh": [x, y, w, h]} ], + "boxes": [ {"xywh": [x, y, w, h], "n": 1, "color": "red"} ], + "badges": [ {"xy": [x, y], "n": 2, "color": "red"} ], + "arrows": [ {"from": [x, y], "to": [x, y], "color": "red"} ] +} +``` + +- **boxes** — rounded highlight rectangle; if `"n"` is set, a numbered badge is drawn at its + top-left corner. This is the workhorse. +- **badges** — a standalone numbered badge centered at `xy` (use sparingly; a detached badge reads as + disconnected — prefer attaching the number to a box). +- **redactions** — solid filled bar (default gray) drawn *first*, under everything. Use for PII. +- `crop` is applied before drawing; if you crop, the box coords are relative to the cropped image. +- Colors: `red` (default), `blue`, `green`, `orange`, `purple`. Fonts come from + `/System/Library/Fonts/Supplemental/Arial*.ttf`. + +## ⚠️ Measuring coordinates — the #1 source of rework + +The numbers in your spec must land on real UI elements. The trap: when you Read a captured PNG, the +viewer often shows it at **half scale** (or otherwise downscaled), and at that size misalignments are +invisible — you'll think a box is on "Result" when it's actually a column off, or that a redaction +covers emails when it covers the *next* column. This burned multiple iterations in the original run. + +**Rules that prevent it:** + +1. **Measure from clean, full-resolution crops — not from the downscaled full image.** Crop the raw + PNG to the region of interest with PIL and read pixel positions there. The most reliable form is a + crop where displayed-x maps directly to file-x (i.e. `file_x = crop_x + x_offset`, no scaling): + ```python + from PIL import Image + Image.open('raw/X.png').crop((x0, y0, x1, y1)).save('/tmp/_measure.png') # NO resize + ``` + Then `file_x = displayed_x + x0`. Avoid scaled crops for measurement — the scale math compounds + reading errors. +2. **A ruler crop is great for nailing columns.** Draw labeled vertical/horizontal lines at known + file coordinates onto a crop, then read where content falls between them: + ```python + from PIL import Image, ImageDraw + c = Image.open('raw/X.png').crop((x0,y0,x1,y1)).convert('RGB'); d = ImageDraw.Draw(c) + for fx in range(((x0//50)+1)*50, x1, 50): + cx = fx-x0; d.line([(cx,0),(cx,c.height)], fill=(255,60,60)); d.text((cx+1,1), str(fx), fill=(255,255,0)) + c.save('/tmp/_ruler.png') + ``` +3. **Always verify the rendered annotation by viewing it** — and if it's a table/columns shot, verify + with a *clean crop* of the box region, not just the full downscaled image. If a box looks even + slightly off, re-measure from a clean crop and re-render. Trust the clean crop over the thumbnail. + +## Conventions that look right + +- **Tables/columns:** box a column to its **full visible height** (header through the last row). A box + that stops partway down looks broken. If rows fill to the viewport bottom, extend the box there. +- **Action buttons (Save/Finish/Next):** wrap the button tightly; the badge sits at the top-left + corner. Make sure the box doesn't drift onto the adjacent button (e.g. Cancel) or label the wrong + thing. +- **Context menus:** box the whole menu *including the left-edge icons* — the menu background starts a + bit left of the text. Verify icons are inside the box. +- **Don't cover content with badges.** A number sitting on a term name, a value, or the wrong column + header reads as an error. Put badges on empty corners, headers, or gaps — not on the data they + point to. +- **Match badge numbers to prose steps.** The numbers exist to map to the numbered steps in the guide + text near that image. Keep them consistent (1,2,3… in the order the prose introduces them). + +## Redacting PII (audit logs, etc.) + +Audit-log screens include a column of real user emails. Measure that column precisely (ruler crop), +then add a `redactions` entry covering the *data cells* (leave the column header visible so it's clear +what was hidden). Verify by zooming into the rendered Initiated/email column — confirm no text +survives the bar. Non-PII identifiers (activity UUIDs, entity IDs) don't need redaction. + +## Bolder annotations on hi-res images + +The annotator defaults to a ~4px border and ~16px badge radius, which read well on ~1600px-wide +captures. If your viewport is much larger or smaller, adjust `bw`/`br` in `annotate.py`'s `main()`. diff --git a/.claude/skills/statgpt-admin-fe-guide-macos/references/capture-pipeline.md b/.claude/skills/statgpt-admin-fe-guide-macos/references/capture-pipeline.md new file mode 100644 index 0000000..f9a4576 --- /dev/null +++ b/.claude/skills/statgpt-admin-fe-guide-macos/references/capture-pipeline.md @@ -0,0 +1,85 @@ +# Capture pipeline — calibration, the banner, and coordinate spaces + +Read this before Phase 2. These are the non-obvious things that otherwise cost a lot of rework. + +## Why macOS `screencapture`, not the MCP screenshot + +The claude-in-chrome MCP screenshot is returned inline and is fine for *driving* the UI, but it is +not a clean file you can annotate, and `save_to_disk` does not reliably write a local file you can +read. The pixel-perfect, file-on-disk method is macOS `screencapture -R`, which the bundled +`cap.sh` wraps. This needs **Screen Recording permission** granted to the terminal/app; if +`screencapture` errors with "could not create image from display", that permission is missing — +ask the user to grant it (System Settings → Privacy & Security → Screen Recording) and retry. + +## ⚠️ The Chrome debugging banner (the main gotcha) + +After *any* CDP command (any MCP `computer`/`navigate`/`javascript_tool` call — **including a +`window.innerHeight` check**), Chrome shows a **"Claude started debugging this browser [Cancel] ✕"** +infobar. It: + +- appears a fraction of a second after the CDP command, +- **persists** and only auto-dismisses after an idle period with no CDP commands (observed: still + there at ~7s, gone by ~15–30s), +- while present, **shifts page content down ~46px**, which breaks any fixed capture region and looks + unprofessional. + +Consequences: + +- **Never run a CDP command immediately before capturing.** `cap.sh` polls with *pure* screencapture + (no CDP) until the banner clears, so it never re-triggers it. Make your last MCP action a `hover`, + then call `cap.sh`. +- **Do NOT click the banner's Cancel/✕** — that can detach the debugger and disconnect the MCP. Just + let it time out (cap.sh waits for you). +- If the MCP disconnects, the user re-runs `/chrome` and you reload the tools via ToolSearch. + +How `cap.sh` detects "clear": it screenshots only the top 28px strip and measures mean brightness. +The banner is light (bright); the dark "StatGPT ADMIN" app bar is dark. When mean brightness < 90 it +captures the full region. It prints `OK ...` on a clean grab, or `WARN ...` if the banner persisted +past ~45s (then investigate / wait longer / inspect the image). + +## Window calibration (re-derive every session — the window moves) + +The capture region is a fixed rectangle `X,Y,W,H` passed to `screencapture -R`. It must frame the +viewport from the top of the "StatGPT ADMIN" app bar down through the content, with **no browser +chrome** and **no banner**. The values are environment- and session-specific (monitor layout, +window position). To derive them: + +1. Get window geometry via `javascript_tool`: + `({sx:screenX, sy:screenY, ow:outerWidth, oh:outerHeight, iw:innerWidth, ih:innerHeight, dpr:devicePixelRatio})` +2. The content top is roughly `screenY + (outerHeight - innerHeight) - ~48`. (Empirically the app bar + sat ~48px above the naive `screenY + chromeHeight`; in one session `-922` worked where the naive + value was `-873`. Negative coords are fine — the window was on a secondary monitor.) +3. Width ≈ innerWidth; height ≈ innerHeight minus a few px so you don't catch the window's bottom edge. +4. **Calibrate by capturing and viewing**: grab the candidate region, view it, and adjust until the + top edge sits exactly at the app bar and nothing is cut. Then lock it and **don't move/resize the + window** for the rest of the session. + +Pass the locked region to cap.sh via `CAP_REGION="X,Y,W,H"`. (The default baked into cap.sh is just +an example from one session — always recalibrate.) + +Set `CAP_PYBIN` to a python that has Pillow (e.g. a backend `.venv/bin/python`); cap.sh uses it for +the brightness probe, and you'll use the same interpreter for `annotate.py`. + +## Two coordinate spaces — don't mix them + +- **MCP clicks/inline screenshots** use the inline-screenshot space (e.g. ~1546×795). Use these + coords to *drive* the UI (clicks, hovers, types). +- **macOS captures** are the real viewport size (e.g. 1600×812). **Measure annotation coordinates + from the actual captured PNG**, never from MCP coords — the two spaces have slightly different + aspect ratios, so scaling between them (especially vertically) is unreliable. + +## Capture protocol (per screenshot) + +1. `browser_batch`: navigate/click/type to the target state. Filter lists to sample content. +2. Final MCP action: `hover` to a neutral empty spot (e.g. lower sidebar, or an empty modal area) so + the cursor isn't over content. Context menus stay open across the hover + cap.sh wait. +3. `CAP_REGION=... CAP_PYBIN=... scripts/cap.sh /tmp/.../raw/.png` (no CDP in between). +4. Read the PNG. Confirm: banner-free, full app bar visible, only sample content, cursor parked. + +## Extracting authoritative config while a modal is open + +The admin stores camelCase config; the seed files are snake_case. To capture the *exact* admin +format, open a "Configure"/"Edit" modal and read the Monaco editor value via `javascript_tool`: +`monaco.editor.getModels().map(m=>m.getValue())` — pick the model containing the config. Long values +get truncated in tool output; slice the string (`t.slice(0,3700)`, then from `t.indexOf('\ndimensions:')`, +etc.) and reassemble. This is how you confirm new fields without guessing. diff --git a/.claude/skills/statgpt-admin-fe-guide-macos/references/config-and-structure.md b/.claude/skills/statgpt-admin-fe-guide-macos/references/config-and-structure.md new file mode 100644 index 0000000..8e7d891 --- /dev/null +++ b/.claude/skills/statgpt-admin-fe-guide-macos/references/config-and-structure.md @@ -0,0 +1,101 @@ +# Config sourcing, guide structure, and screenshot inventory + +Read this before Phase 4 (rewriting the guide). + +## Sourcing configuration YAML + +The guide's YAML examples must be **real**, not invented, and in the **admin (camelCase) format** the +admin UI actually stores — which differs from the seed files. + +- **Seed files** live in `statgpt-backend/configurations/clients/sample/` and are **snake_case** + (`data_sources.yaml`, `datasets/*.yaml`, `channels.yaml`, `tools.yaml`, `glossaries/*.csv`). Use + them for the *content* (ids, urls, descriptions, dimension lists, tool descriptions, glossary terms). +- **The admin stores camelCase.** Translate seed snake_case → admin camelCase. When unsure of the exact + shape or a new field, the **live "Configure"/"Edit" editor is the source of truth** — read it with + Monaco `getValue()` (see capture-pipeline.md). Abbreviate very long descriptions in the guide with + a `# ...` comment; keep the structure faithful. + +### Data source config (the `details` content shown in the Configuration editor) + +Keys seen: `apiKey`, `locale`, `authConfig`, `rateLimits`, `sdmxConfig` (`id`, `url`, `name`, +`headers` per resource type, `supports`, `versions`, `dataContentType`), `authEnabled`, `sdmx1Source`, +`apiKeyHeader`, `attributesUrl`, `annotationsUrl`, `dataExplorerUrl`, `providerDiscovery`. The +connector (`SDMX21` / `QH_SDMX21` / `PROXY_SDMX30`) is chosen in the wizard's Properties step. + +### Dataset config — the **`dimensions` map** is the headline schema + +Old flat fields (`indicatorDimensions`, `countryDimension`, `countryDimensionAlias`, +`dimensionDefaultQueries`, `indicatorDimensionsRequiredForQuery`) are replaced by a **`dimensions:` +map**, one entry per dimension: + +```yaml +dimensions: + COUNTRY: { dimensionType: NON_INDICATOR, subtype: REGION, alias: Country/Reference area, + isRequired: false, allValues: {id: ALL_COUNTRIES, name: "...", description: "..."} } + FREQUENCY: { dimensionType: NON_INDICATOR, subtype: FREQUENCY, isRequired: false } + INDICATOR: { dimensionType: INDICATOR, isRequired: true } + TIME_PERIOD: { dimensionType: TIME_PERIOD, isRequired: false, + defaultQueries: [ {values: ["-5y","+2y"], operator: between} ] } +``` + +`dimensionType` is `INDICATOR` / `NON_INDICATOR` / `TIME_PERIOD`. Plus: `urn` ({version, agencyId, +resourceId}), `indexer.indicator` ({unpack, annotations, superPrimary, useCodeListDescription}), +`indexer.description`, `citation` ({url, provider, description}), `updatedAt` (list of {field, source, +formats}), `isOfficial`, `pinnedColumns`, `useTitleFromSrc`, `includeAttributes`. Reference learning +Modules **03a** (dimension types), **03b** (indicators), **04** (dataset config). + +### Channel config — new fields and the tool set + +`locale`, `conversationStarters` ({introText, title, inputPlaceholder, buttons:[{title,text}]}), +`onboarding`, `namedEntityTypes`, `countryNamedEntityType`, `supremeAgent` +({name, domain, terminologyDomain, languageInstructions, llmModelConfig}), +`outOfScope` ({domain, useGeneralTopicsBlacklist, llmModelConfig}), `tokenUsage` +({debugOnly, stageName}). Tools: `availableDatasets`, **`datasetStructure` (`DATASET_STRUCTURE`, the +newer tool)**, `dataQuery`, `availableTerms`, `termDefinitions`. `dataQuery.details` now has +`allowAutoUpdate: true`, `hybridSearchConfig` ({namedEntitiesToRemove, prompts.relevancyPrompts}), +`indexerVersion`/`indicatorSelectionVersion: hybrid`, `llmModels` (per-stage), and a `mergedPythonCode` +attachment. Build this from `channels.yaml` + `tools.yaml`. Reference learning Module **05**. + +## Guide structure (`guides/admin-guide.md`) + +1. **Administrator App** + intro to the left nav. +2. **Concepts** — Data Source, Dataset, Channel, plus *index version* and *auto-update*. +3. **Data Sources** — list → add (Properties → Configuration) → edit/Configure. Data-source YAML. +4. **Datasets** — list + row menu → 4-step add wizard (Source → Provider → Dataflow → Configuration) → + edit. The `dimensions` schema; link the learning course. +5. **Channels** — list (+ Import) + context menu → Configure (YAML) → **channel datasets page** + (toolbar: Deduplicate statistics / Export / Recalculate all indexes ▾ / + Add) → per-dataset menu → + **Versions** → **Auto update jobs** → indexing (Recalculate / Sequential vs Parallel / Deduplicate) → + Glossary → Import/Export & Jobs. +6. **Audit Logs**. +7. Related resources / learning-course link. + +Add a Table of Contents after the intro. Fix broken image links. Keep every cross-link resolvable +(this repo has `architecture/` and `learning/administration/` but **no** `deployment/` dir). + +## Screenshot inventory (filenames → what the badges point to) + +Adjust to scope. Each list/menu screen filtered to sample content. + +- `ds-list` (Add, filter, row ⋯) · `ds-add-properties` (Name, connector, Next) · + `ds-add-config` (editor, Finish) · `ds-configure` (editor, Save) +- `datasets-list` (Add, filter, row ⋯) · `datasets-row-menu` (Edit/Delete) · + `dataset-add-source` / `dataset-add-provider` / `dataset-add-dataflow` (select + Next) · + `dataset-add-config` (the `dimensions` block, Finish) · `dataset-edit-config` (editor, Save) +- `channels-list` (Import, Add, row ⋯) · `channel-menu` (Configure/Glossary/Jobs/Delete/Export) · + `channel-configure` (editor, Save) · `channel-details` (toolbar: Deduplicate/Export/Recalculate/Add) · + `channel-dataset-menu` (Edit/Auto update jobs/Versions/Recalculate/Delete) +- `dataset-versions` (Status, Creation Reason) · `dataset-auto-update-jobs` (Result, Updated At) · + `channel-recalc-dropdown` (Sequential/Parallel) · `channel-deduplicate` (stats dialog) +- `glossary-page` (Add Term, Source/Domain) · `glossary-add-term` (Term/Definition/Source/Domain) · + `glossary-term-menu` (Edit/Delete — note: delete is immediate) +- `channel-import` (drop zone, the 3 toggles, Import) · `channel-jobs-download` (Type column + + the Export/download row action — one shot covers the jobs list and the download) +- `audit-logs` (Action, Entity name; **redact the Initiated/email column**) + +## Shipping (only if asked) + +Branch (e.g. `feat/updated-admin-guide`); stage only `guides/admin-guide.md` + `guides/content/admin-guide/*.png` +(not stray local files); commit with a Conventional-Commits title (e.g. `docs: ...`); push; open a PR +against the repo's default branch using `.github/pull_request_template.md`. The repo enforces a +Conventional-Commits PR title. Don't delete now-orphaned old images without asking. diff --git a/.claude/skills/statgpt-admin-fe-guide-macos/scripts/annotate.py b/.claude/skills/statgpt-admin-fe-guide-macos/scripts/annotate.py new file mode 100644 index 0000000..a0d6677 --- /dev/null +++ b/.claude/skills/statgpt-admin-fe-guide-macos/scripts/annotate.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +"""Annotate StatGPT admin screenshots with numbered badges + highlight boxes. + +Usage: + python annotate.py spec.json + +Spec JSON schema: +{ + "src": "/abs/path/raw.png", + "dst": "/abs/path/out.png", + "scale": 2.0, # multiply all coords below by this (CSS px -> image px) + "crop": [x, y, w, h], # optional, in IMAGE px (applied before scale-based draw); omit for none + "boxes": [ + {"xywh": [x, y, w, h], "n": 1, "color": "red"} # highlight rect; n -> badge at top-left corner + ], + "badges": [ + {"xy": [x, y], "n": 2, "color": "red"} # standalone numbered badge centered at xy + ], + "arrows": [ + {"from": [x, y], "to": [x, y], "color": "red"} + ] +} +All coords (xywh / xy / from / to) are in CSS/logical px and get multiplied by "scale". +If you pass coords already in image px, set "scale": 1. +""" +import json, sys, math +from PIL import Image, ImageDraw, ImageFont + +PALETTE = { + "red": (244, 63, 71), + "blue": (56, 132, 255), + "green": (52, 199, 89), + "orange": (255, 149, 0), + "purple": (175, 82, 222), +} +DEFAULT = PALETTE["red"] + +FONT_PATHS = [ + "/System/Library/Fonts/Supplemental/Arial Bold.ttf", + "/System/Library/Fonts/Supplemental/Arial.ttf", +] + +def load_font(size): + for p in FONT_PATHS: + try: + return ImageFont.truetype(p, size) + except Exception: + continue + return ImageFont.load_default() + +def col(name): + if isinstance(name, (list, tuple)): + return tuple(name) + return PALETTE.get(name, DEFAULT) + +def rounded_rect(draw, box, radius, outline, width): + draw.rounded_rectangle(box, radius=radius, outline=outline, width=width) + +def draw_badge(img, cx, cy, n, color, r): + """Filled circle with white number, centered at (cx,cy).""" + d = ImageDraw.Draw(img) + # subtle dark outline for contrast on light areas + d.ellipse([cx-r-2, cy-r-2, cx+r+2, cy+r+2], fill=(255, 255, 255)) + d.ellipse([cx-r, cy-r, cx+r, cy+r], fill=color) + font = load_font(int(r*1.25)) + txt = str(n) + bb = d.textbbox((0, 0), txt, font=font) + tw, th = bb[2]-bb[0], bb[3]-bb[1] + d.text((cx-tw/2-bb[0], cy-th/2-bb[1]), txt, fill=(255, 255, 255), font=font) + +def draw_arrow(draw, p0, p1, color, width): + draw.line([p0, p1], fill=color, width=width) + # arrowhead + ang = math.atan2(p1[1]-p0[1], p1[0]-p0[0]) + L = 18 + for da in (math.radians(28), -math.radians(28)): + ex = p1[0] - L*math.cos(ang+da) + ey = p1[1] - L*math.sin(ang+da) + draw.line([(p1[0], p1[1]), (ex, ey)], fill=color, width=width) + +def main(): + spec = json.load(open(sys.argv[1])) + img = Image.open(spec["src"]).convert("RGB") + if spec.get("crop"): + x, y, w, h = spec["crop"] + img = img.crop((x, y, x+w, y+h)) + s = float(spec.get("scale", 1)) + draw = ImageDraw.Draw(img) + bw = max(4, int(round(4*s))) # box border width + br = max(16, int(round(16*s))) # badge radius + + # Redactions: solid filled bars to mask sensitive content (e.g. emails). + for rd in spec.get("redactions", []): + x, y, w, h = [v*s for v in rd["xywh"]] + fill = col(rd["color"]) if rd.get("color") else (70, 76, 88) + draw.rectangle([x, y, x+w, y+h], fill=fill) + + for b in spec.get("boxes", []): + x, y, w, h = [v*s for v in b["xywh"]] + c = col(b.get("color")) + rounded_rect(draw, [x, y, x+w, y+h], radius=int(8*s), outline=c, width=bw) + if "n" in b: + draw_badge(img, x, y, b["n"], c, br) + draw = ImageDraw.Draw(img) + + for a in spec.get("arrows", []): + draw_arrow(draw, tuple(v*s for v in a["from"]), tuple(v*s for v in a["to"]), + col(a.get("color")), bw) + + for bd in spec.get("badges", []): + x, y = [v*s for v in bd["xy"]] + draw_badge(img, x, y, bd["n"], col(bd.get("color")), br) + draw = ImageDraw.Draw(img) + + img.save(spec["dst"]) + print("wrote", spec["dst"], img.size) + +if __name__ == "__main__": + main() diff --git a/.claude/skills/statgpt-admin-fe-guide-macos/scripts/cap.sh b/.claude/skills/statgpt-admin-fe-guide-macos/scripts/cap.sh new file mode 100755 index 0000000..8d02688 --- /dev/null +++ b/.claude/skills/statgpt-admin-fe-guide-macos/scripts/cap.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# cap.sh — capture the StatGPT Admin viewport with macOS `screencapture`, +# waiting out the Chrome "Claude started debugging this browser" infobar +# (which shifts page content down and ruins a fixed-region capture). +# +# It polls the top strip of the viewport with PURE screencapture (no CDP, so +# it never re-triggers the banner) until that strip goes dark — i.e. the dark +# "StatGPT ADMIN" app bar is showing rather than the bright banner — then grabs +# the frame. See references/capture-pipeline.md for why this matters. +# +# Usage: cap.sh /abs/out.png +# Env (override per session — the defaults are example values, NOT universal): +# CAP_REGION : screencapture -R value "X,Y,W,H" for the viewport region. +# Re-derive this every session; the window moves. See the +# calibration steps in references/capture-pipeline.md. +# CAP_PYBIN : a python interpreter with Pillow installed (brightness probe). + +OUT="$1" +REGION="${CAP_REGION:--219,-922,1600,812}" +PYBIN="${CAP_PYBIN:-python3}" + +IFS=',' read -r RX RY RW _RH <<< "$REGION" +PROBE="$(dirname "$OUT")/_probe_top.png" + +for i in $(seq 1 45); do + screencapture -x -t png -R"${RX},${RY},${RW},28" "$PROBE" + bright=$("$PYBIN" -c "from PIL import Image,ImageStat;print(int(ImageStat.Stat(Image.open('$PROBE').convert('L')).mean[0]))") + if [ "$bright" -lt 90 ]; then + screencapture -x -t png -R"$REGION" "$OUT" + echo "OK $OUT (clear, bright=$bright, ~${i}s)"; exit 0 + fi + sleep 1 +done +screencapture -x -t png -R"$REGION" "$OUT" +echo "WARN $OUT (banner may still be present, bright=$bright) — inspect before using" diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..59daa84 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,54 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this repository is + +This is the **StatGPT documentation repository** — Markdown, images, and draw.io diagrams only. There is **no application code, build step, or test suite here.** Docs render directly on GitHub; nothing is compiled or served by a static-site generator. + +The product itself lives in sibling repositories. The backend is mounted as a **second working directory** at `../statgpt-backend` (Python / FastAPI / LangChain — the admin and chat backends, SDMX handling, the agent). When documenting a feature, treat that repo as the implementation source of truth. The other components (admin & portal frontends, SDMX proxy, Helm chart) are external GitHub repos linked from `README.md`. + +StatGPT is an AI "talk-to-your-data" platform for official SDMX statistics, built on the AI DIAL platform. Its agent uses **tool-calling** (not code generation) and **grounds** every answer in real query results to prevent hallucination. + +## Validation + +There is no build or test. The only CI check is formatting hygiene, run by `.github/workflows/pr.yml` on PRs to `main`: + +```bash +pip install pre-commit && pre-commit install # one-time setup +pre-commit run --all-files # check / auto-fix everything +``` + +Hooks (`.pre-commit-config.yaml`): `check-yaml`, `end-of-file-fixer`, `trailing-whitespace`, `mixed-line-ending`. A red CI is almost always a missing final newline or trailing whitespace — run the command above and commit the result. + +## Contributing + +- **PRs target `main`** and are squash-merged. PR **titles must follow [Conventional Commits](https://www.conventionalcommits.org/)** — enforced by the `pr-title-check` workflow. +- Reference an issue in the PR (`fixes #123`); contributions are made under the **MIT license** — this is a public repo. +- `CONTRIBUTING.md` covers the broader multi-repo branching/release process and the coding principles the whole project follows. + +## Documentation layout + +Each area has its own `README.md` index — read it first. + +| Area | Purpose | +|------|---------| +| `architecture/` | Technical design. Intended read order: `overview.md` → `services.md` → `agent.md` → `tools.md`. `sdmx-compatibility.md` and `mcp.md` are specialized topics. | +| `guides/` | Task-oriented guides: `admin-guide.md`, `admin-azure-auth-guide.md`, `gtdc-portal-guide.md`. | +| `learning/administration/` | Structured 8-module course on **dataset onboarding** for admins (IMF datasets as the primary examples) plus `quick-reference.md`. The deepest treatment of dimension / indicator / dataset configuration. | +| `evaluation/` | `data_query.md` — methodology for evaluating SDMX data-query quality (ground truth, precision/recall). | + +These areas overlap intentionally and **cross-link with relative paths** (e.g. the learning course points into `guides/` and `architecture/`). When you move or rename a doc, fix the inbound links yourself — there is no link-checker in CI. + +## Conventions + +- **Images** live under `/content//` (e.g. `guides/content/admin-guide/`); reference them with relative paths. +- **Architecture diagrams** are SVGs exported from the single editable source `architecture/content/diagrams.drawio`. Edit the `.drawio` in draw.io and re-export — do not hand-edit the SVGs. +- **Public-repo content rules:** screenshots must show **sample content only** (the sample data source / channel / "- Sample" datasets) and **PII must be redacted**. Never commit other orgs' data or real user emails. +- Architecture docs use emoji section headers and Markdown tables; match the style of the file you are editing. + +## Domain model (keep terminology consistent across docs) + +- **Core entities:** **Data Source → Dataset → Channel.** A channel bundles datasets + tools + glossary and is what an end user talks to. +- **Components:** Chat Backend (DIAL app), Admin Backend (FastAPI + Alembic), Admin & Portal Frontends (React/Next.js), SDMX Proxy (Spring Boot — unified SDMX 3.0 facade over upstream registries), all deployed via the Helm chart. Built on **AI DIAL** for auth, rate limits, model access, and file storage. +- **Agent tools:** Available Datasets, Data Query (natural language → SDMX), Available Publications + Publications RAG, Glossary Terms/Definitions, Web Search. Channel tools are also exposed to external AI agents over **MCP**.