diff --git a/README.md b/README.md index b903cec3..148f8235 100644 --- a/README.md +++ b/README.md @@ -507,6 +507,7 @@ For detailed deployment patterns (Docker, Kubernetes, NATS workers, etc.), see: - `scripts/README_CLIMRETRIEVE.md` – ClimRetrieve benchmark and generic CSV evaluation - `report_analyst_jobs/README.md` – Job processing and worker patterns - `docs/CI.md` – Continuous integration and testing +- `docs/wireframes/` – Sketch screens of the Report Analyst UI ([GitHub-rendered gallery](docs/wireframes/README.md)) --- diff --git a/docs/wireframes/README.md b/docs/wireframes/README.md new file mode 100644 index 00000000..55e3332d --- /dev/null +++ b/docs/wireframes/README.md @@ -0,0 +1,45 @@ +# Report Analyst wireframes + +Sketch screens of the Report Analyst UI (no CT Platform chrome). GitHub renders the PNGs below. + +Syntax that these pages use (`{#id}`, `::: callout`, PDF overlays, selected config cards) lives in the [devolute/wiremd](https://github.com/devolute/wiremd) fork, branch [`feat/callouts`](https://github.com/devolute/wiremd/tree/feat/callouts). + +```bash +npm install github:devolute/wiremd#feat/callouts +``` + +## Analysis + +![Analysis](renders/analysis.png) + +[Source](pages/analysis.md) · [HTML](out/analysis.html) + +## View all results + +![View all results](renders/view-all-results.png) + +[Source](pages/view-all-results.md) · [HTML](out/view-all-results.html) + +## PDF viewer + +![PDF viewer](renders/report-viewer.png) + +[Source](pages/report-viewer.md) · [HTML](out/report-viewer.html) + +## Benchmarking + +![Benchmarking](renders/benchmarking.png) + +[Source](pages/benchmarking.md) · [HTML](out/benchmarking.html) + +## Rebuild + +Edit a page under `pages/`, then from a checkout of [devolute/wiremd@feat/callouts](https://github.com/devolute/wiremd/tree/feat/callouts): + +```bash +npm run build +node bin/wiremd.js /path/to/report-analyst/docs/wireframes/pages/analysis.md \ + -s sketch -o /path/to/report-analyst/docs/wireframes/out/analysis.html +``` + +Repeat for `view-all-results`, `report-viewer`, and `benchmarking`. Recapture PNGs into `renders/` if the figure should update on GitHub. diff --git a/docs/wireframes/out/analysis.html b/docs/wireframes/out/analysis.html new file mode 100644 index 00000000..6c4addd7 --- /dev/null +++ b/docs/wireframes/out/analysis.html @@ -0,0 +1,958 @@ + + + + + + wiremd Mockup + + + +
+

Report Analyst

+

Analysis parameters tailored to your configurations.

+
+
+

Report

+ +

Uploaded, 12.03.2025

+
+
+

Analysis configuration

+
+ Question set + +
+

Climate-related financial disclosures.

+

Processing steps

+

Chunk — Embed — Map — Answer

+

Stored Chunking +Stored Embedding +New Question mapping +New Question answering

+
+ Top K + +
+
+ Chunk size + +
+
+ Overlap + +
+
+ LLM Model + +
+

Scoring

+
    + + +
+
+
+

Select questions

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SelectQIDQuestion
[x]TCFD-1Describe the board’s oversight of climate-related risks and opportunities.
[x]TCFD-2Describe management’s role in assessing and managing climate-related risks.
[ ]TCFD-3Describe the climate-related risks and opportunities identified over the short, medium, and long term.
[x]TCFD-4Describe the impact of climate-related risks on the organization’s businesses, strategy, and financial planning.
+
+ + +
+
+

Analysis in progress… 3 of 3 questions

+
+

Analysis results

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
QIDScoreAnalysisKey evidence
TCFD-18.2Board reviews climate risk quarterly via the Sustainability Committee…p. 14 — “Board oversight”
TCFD-27.4Management owns the climate risk register; CRO reports monthly…p. 18 — “Management role”
TCFD-46.1Strategy impacts described qualitatively; limited financial quantification…p. 22 — “Financial planning”
+
+ + + +
+ + + + +
+ + + \ No newline at end of file diff --git a/docs/wireframes/out/benchmarking.html b/docs/wireframes/out/benchmarking.html new file mode 100644 index 00000000..c3033e50 --- /dev/null +++ b/docs/wireframes/out/benchmarking.html @@ -0,0 +1,885 @@ + + + + + + wiremd Mockup + + + +
+

Benchmarking

+

Evaluate retrieval and extraction against a gold set.

+
+
+ + + + +
+
+
+

Evaluation type

+ +
+
+
+

Ground truth / Benchmark

+ +

Expert-annotated relevant chunks.

+
+
+

Chunks downloaded via CSV

+ +

Export from Analysis → Download chunks data.

+
+
+
+ Evaluation name + +
+
+

Ranking configuration

+
+ Top K + +
+
+ K values for metrics + +
+ +
+ + + + + + +
+ + + \ No newline at end of file diff --git a/docs/wireframes/out/report-viewer.html b/docs/wireframes/out/report-viewer.html new file mode 100644 index 00000000..128dd1ee --- /dev/null +++ b/docs/wireframes/out/report-viewer.html @@ -0,0 +1,876 @@ + + + + + + wiremd Mockup + + + +
+

PDF viewer

+
+
+

Chunks

+
+ Question + +
+ +
+

EvidencePage 12

+

“…risk management framework is reviewed quarterly by the board…”

+

Sim: 0.850 · LLM: 0.920

+
+
+ Page 14 +

“…Performance metrics for transition plans are disclosed in section 4…”

+

Sim: 0.710

+
+
+
+

PDF

+
+

Page 12 of 84

+

3. Governance

+
+

The Board Sustainability Committee meets quarterly to review climate risk and opportunities.

+
+
+

Evidence +“…risk management framework is reviewed quarterly by the board…”

+
+
+

The committee reports to the full board after each sitting.

+
+
+

Management’s role is described in section 3.2. The CRO owns the climate risk register.

+
+
+
+
+ + + + + +
+ + + \ No newline at end of file diff --git a/docs/wireframes/out/view-all-results.html b/docs/wireframes/out/view-all-results.html new file mode 100644 index 00000000..060ef795 --- /dev/null +++ b/docs/wireframes/out/view-all-results.html @@ -0,0 +1,945 @@ + + + + + + wiremd Mockup + + + +
+

View all results

+

Consolidated results across analyzed reports.

+
+
+

Question set

+ +

Climate-related financial disclosures.

+
+
+

Report

+ +

Uploaded, 12.03.2025

+
+
+

Configuration

+
+
+

GPT-4o Mini

+

Chunk: 500 · Overlap: 20 · Top-K: 5

+
+
+

GPT-4o

+

Chunk: 800 · Overlap: 40 · Top-K: 8

+
+
+

Gemini

+

Chunk: 500 · Overlap: 20 · Top-K: 5

+
+
+

Similarity search

+
+ Question + +
+

TCFD-1: Describe the board’s oversight of climate-related risks…

+
+ Or custom question + +
+

Document chunks

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RankSimilarityTextPageEmbedding
10.912The Board Sustainability Committee meets quarterly to review climate risk…14[x]
20.847The CRO reports climate metrics to the audit committee…18[x]
30.731Scenario analysis is presented annually to the full board…22[x]
+

Consolidated analysis

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
QIDScoreAnalysisKey evidenceGaps
TCFD-18.2Board reviews climate risk quarterly via the Sustainability Committee.p. 14No independent board climate expert named.
TCFD-27.4Management owns the climate risk register; CRO reports monthly.p. 18Escalation path not documented.
TCFD-46.1Strategy impacts described qualitatively; limited financial quantification.p. 22Capex / opex figures missing.
+ + + + + + +
+ + + \ No newline at end of file diff --git a/docs/wireframes/pages/analysis.md b/docs/wireframes/pages/analysis.md new file mode 100644 index 00000000..d4620d50 --- /dev/null +++ b/docs/wireframes/pages/analysis.md @@ -0,0 +1,91 @@ +# Report Analyst +Analysis parameters tailored to your configurations. + +::: grid-2 {.start} + +### Report + +[Microsoft Sustainability Report 2024.pdf___________v]{#report} +- Microsoft Sustainability Report 2024.pdf +- Northwind TCFD 2023.pdf + +Uploaded, 12.03.2025 + +### Analysis configuration + +Question set +[TCFD___________v] +- TCFD +- GRI +- ESRS + +Climate-related financial disclosures. + +#### Processing steps {#steps} +Chunk — Embed — Map — **Answer** + +|Stored| Chunking +|Stored| Embedding +|New|{.primary} Question mapping +|New|{.primary} Question answering + +Top K +[5___] + +Chunk size +[500___] + +Overlap +[20___] + +LLM Model +[gpt-4o-mini___________v] +- gpt-4o-mini +- gpt-4o +- gemini-2.5-flash + +Scoring + +- [x] LLM Scoring +- [x] Batch Scoring + +::: + +::: callout {for:report side:left} +report select +::: + +::: callout {for:steps side:left} +step cache +::: + +## Select questions + +[Select all]* + +| Select | QID | Question | +|--------|-----|----------| +| [x] | TCFD-1 | Describe the board’s oversight of climate-related risks and opportunities. | +| [x] | TCFD-2 | Describe management’s role in assessing and managing climate-related risks. | +| [ ] | TCFD-3 | Describe the climate-related risks and opportunities identified over the short, medium, and long term. | +| [x] | TCFD-4 | Describe the impact of climate-related risks on the organization’s businesses, strategy, and financial planning. | + +[Analyze selected questions]*{#run} [Reanalyze] + +::: callout {for:run side:left} +run +::: + +::: alert +Analysis in progress… 3 of 3 questions +::: + +## Analysis results + +| QID | Score | Analysis | Key evidence | +|-----|-------|----------|--------------| +| TCFD-1 | 8.2 | Board reviews climate risk quarterly via the Sustainability Committee… | p. 14 — “Board oversight” | +| TCFD-2 | 7.4 | Management owns the climate risk register; CRO reports monthly… | p. 18 — “Management role” | +| TCFD-4 | 6.1 | Strategy impacts described qualitatively; limited financial quantification… | p. 22 — “Financial planning” | + +[Download analysis results] [Download chunks data] [Open PDF viewer] diff --git a/docs/wireframes/pages/benchmarking.md b/docs/wireframes/pages/benchmarking.md new file mode 100644 index 00000000..bb6ede5e --- /dev/null +++ b/docs/wireframes/pages/benchmarking.md @@ -0,0 +1,70 @@ +# Benchmarking +Evaluate retrieval and extraction against a gold set. + +::: card {#tabs} +[Datasets] [Evaluate]* [Results] [Annotate] +::: + +::: callout {for:tabs side:left} +tabs +::: + +::: card {#mode} +Evaluation type + +- (x) Ranking (retrieval) +- ( ) Classification +- ( ) Both + +::: + +::: callout {for:mode side:left} +mode +::: + +::: grid-2 {.start} + +### Ground truth / Benchmark {#gt} +[ClimRetrieve___________v] +- ClimRetrieve +- ClimRetrieve report-level + +Expert-annotated relevant chunks. + +### Chunks downloaded via CSV {#chunks} +[chunks_Microsoft_2024.csv___________v] +- chunks_Microsoft_2024.csv +- chunks_data.csv + +Export from Analysis → Download chunks data. + +::: + +::: callout {for:gt side:left} +gold labels +(ClimRetrieve) +::: + +::: callout {for:chunks side:right} +your retrieval +export +::: + +Evaluation name +[eval-20250312-1430___________] + +::: card {#metrics} +#### Ranking configuration + +Top K +[10___] + +K values for metrics +[1,3,5,10___________] + +[Run ranking evaluation]* +::: + +::: callout {for:metrics side:left} +metrics +::: diff --git a/docs/wireframes/pages/report-viewer.md b/docs/wireframes/pages/report-viewer.md new file mode 100644 index 00000000..6517264c --- /dev/null +++ b/docs/wireframes/pages/report-viewer.md @@ -0,0 +1,80 @@ +# PDF viewer + +::: grid-2 {.start} + +### Chunks + +Question +[Q1 How does org…___________v]{#question} +- All Questions +- Q1 How does org… +- Q2 What targets… + +[x] Show evidence only{#evidence} + +::: card {#chunk-3} +|Evidence|{.success} |Page 12| + +“…risk management framework is reviewed quarterly by the board…” + +Sim: 0.850 · LLM: 0.920 +::: + +::: card +|Page 14| + +“…Performance metrics for transition plans are disclosed in section 4…” + +Sim: 0.710 +::: + +### PDF + +::: card {.document #pdf} +[Previous] Page 12 of 84 [Next] + +**3. Governance** + +::: card {.retrieved} +The Board Sustainability Committee meets quarterly to review climate risk and opportunities. +::: + +::: card {.evidence #cite} +|Evidence|{.success} +“…risk management framework is reviewed quarterly by the board…” +::: + +::: card {.retrieved} +The committee reports to the full board after each sitting. +::: + +::: card {.retrieved} +Management’s role is described in section 3.2. The CRO owns the climate risk register. +::: +::: + +::: + +::: callout {for:question side:left} +restrict chunks +and overlays +to one question +::: + +::: callout {for:evidence side:left} +hide retrieved +boxes that were +not cited +::: + +::: callout {for:chunk-3 side:left} +click a chunk: +PDF jumps to +that page +::: + +::: callout {for:cite side:right} +hover a box: +similarity, +LLM score, order +::: diff --git a/docs/wireframes/pages/view-all-results.md b/docs/wireframes/pages/view-all-results.md new file mode 100644 index 00000000..98aa3554 --- /dev/null +++ b/docs/wireframes/pages/view-all-results.md @@ -0,0 +1,87 @@ +# View all results +Consolidated results across analyzed reports. + +::: grid-2 {.start} + +### Question set + +[TCFD___________v]{#qset} +- TCFD +- GRI +- ESRS + +Climate-related financial disclosures. + +### Report + +[Microsoft Sustainability Report 2024.pdf___________v]{#report} +- Microsoft Sustainability Report 2024.pdf +- Northwind TCFD 2023.pdf + +Uploaded, 12.03.2025 + +::: + +::: callout {for:qset side:left} +question set +::: + +::: callout {for:report side:right} +report +::: + +## Configuration + +::: grid-3 card + +### GPT-4o Mini {#config .primary} +Chunk: 500 · Overlap: 20 · Top-K: 5 + +### GPT-4o +Chunk: 800 · Overlap: 40 · Top-K: 8 + +### Gemini +Chunk: 500 · Overlap: 20 · Top-K: 5 + +::: + +::: callout {for:config side:left} +click a config: +switch cached +results +::: + +## Similarity search + +Question +[TCFD-1___________v]{#rank} +- None +- TCFD-1 +- TCFD-2 + +TCFD-1: Describe the board’s oversight of climate-related risks… + +Or custom question +[How does the board oversee climate risk?___________] + +::: callout {for:rank side:left} +chunk rank +::: + +## Document chunks + +| Rank | Similarity | Text | Page | Embedding | +|------|------------|------|------|-----------| +| 1 | 0.912 | The Board Sustainability Committee meets quarterly to review climate risk… | 14 | [x] | +| 2 | 0.847 | The CRO reports climate metrics to the audit committee… | 18 | [x] | +| 3 | 0.731 | Scenario analysis is presented annually to the full board… | 22 | [x] | + +## Consolidated analysis + +| QID | Score | Analysis | Key evidence | Gaps | +|-----|-------|----------|--------------|------| +| TCFD-1 | 8.2 | Board reviews climate risk quarterly via the Sustainability Committee. | p. 14 | No independent board climate expert named. | +| TCFD-2 | 7.4 | Management owns the climate risk register; CRO reports monthly. | p. 18 | Escalation path not documented. | +| TCFD-4 | 6.1 | Strategy impacts described qualitatively; limited financial quantification. | p. 22 | Capex / opex figures missing. | + +[Export CSV] diff --git a/docs/wireframes/renders/analysis.png b/docs/wireframes/renders/analysis.png new file mode 100644 index 00000000..1637fcce Binary files /dev/null and b/docs/wireframes/renders/analysis.png differ diff --git a/docs/wireframes/renders/benchmarking.png b/docs/wireframes/renders/benchmarking.png new file mode 100644 index 00000000..ee9e650f Binary files /dev/null and b/docs/wireframes/renders/benchmarking.png differ diff --git a/docs/wireframes/renders/report-viewer.png b/docs/wireframes/renders/report-viewer.png new file mode 100644 index 00000000..55fbfacd Binary files /dev/null and b/docs/wireframes/renders/report-viewer.png differ diff --git a/docs/wireframes/renders/view-all-results.png b/docs/wireframes/renders/view-all-results.png new file mode 100644 index 00000000..d1f8a294 Binary files /dev/null and b/docs/wireframes/renders/view-all-results.png differ