GenAR PADER Analyzer

AI-assisted, evidence-grounded PADER-style safety analysis prototype for the supplied synthetic Bisoprolol ICSR dataset (GenAR AI Engineering Challenge).

Status: Engineering prototype — not a regulatory submission, not medical advice, not a real-world pharmacovigilance assessment.

The central architecture principle:

DATASET → VALIDATION → DETERMINISTIC ANALYSIS → VERIFIED EVIDENCE / CONTEXT
       → CONTROLLED AI NARRATIVE GENERATION → VALIDATION → HUMAN REVIEW
       → FINAL PADER-STYLE REPORT

Deterministic code calculates the truth. AI explains the truth. Humans decide what the truth means clinically or regulatorily.

The LLM never receives the raw CSV. It receives only verified context packets built from deterministic calculations, and its output is automatically validated before the report can be published.

1. What the project does

Given the supplied ICSR line-listing for Bisoprolol, the project:

Ingests and schema-validates the dataset (CSV or XLSX).
Deduplicates at case level (safetyreportid, highest safetyreportversion), so 1,068 rows are never treated as 1,068 cases (the dataset contains 1,024 unique cases).
Computes every statistic deterministically: seriousness, Alert/expedited status, age buckets, sex, countries (occurcountry), reactions (case counts vs. reaction-record counts), outcomes, reporters, and monthly trends.
Registers traceable evidence for every important claim (claim, value, source fields, calculation, case IDs).
Assembles section-specific verified context packets and generates four narrative sections via an LLM (or a deterministic mock when no API key is set).
Validates both the deterministic numbers and the AI output — numbers must exist in the context, and no invented claims, case IDs, expectedness, SOC, NDA numbers, causality or safety signals are allowed.
Requires human review (approve / flag / note / edit / regenerate) before the report is considered final.
Emits outputs/report_output.md, .html, .pdf and .docx, a searchable case index, evidence JSON, sentence-level traceability, and validation/evaluation reports.
Serves a professional React dashboard (FastAPI backend) with 10 pages: Dashboard, Dataset, Case Analysis, Reaction Analysis, Serious Cases, Trend Analysis, AI Report, Evidence, Case Index and Settings.

2. Why this architecture was chosen

Deterministic-first. Statistics are the ground truth of a safety report. An LLM asked to "compute" counts from raw rows will silently miscount, double-count versions, or invent cases. All numbers are computed by tested pandas code; the LLM can only explain them.
Evidence tracing. Every claim in the report can be traced back to the code that computed it and the case IDs behind it. The Evidence page maps each narrative sentence to the claims, calculations, source fields and case IDs that ground it.
Minimized AI surface. The LLM receives small, section-specific packets of verified numbers — never the whole CSV — so there is little room for hallucination and each call is cheap and auditable.
A publication gate. No report is shown as final unless 22 automated checks pass (totals, percentages, period, case-ID membership, reaction membership, AI-number grounding, and a vocabulary scan for unsupported claims).
Works without an API key. A deterministic mock narrative generator keeps the whole system runnable and testable; real LLM calls are opt-in via LLM_API_KEY.

The system is deliberately not a multi-agent framework or a RAG pipeline: there is no retrieval problem (the "corpus" is one dataset), and the evidence layer is simpler and stronger than vector search for grounding.

3. Folder structure

Prathyusha_GenAR_Challenge/
├── app/
│   ├── backend/            FastAPI: cached pipeline state, REST endpoints,
│   │                       review controls, report exports, serves dashboard
│   └── frontend/           React 18 + Vite + Recharts dashboard (10 pages)
├── src/                    Core library
│   ├── data/               loader · schema · dates · case dedup
│   ├── analysis/           overview · seriousness · alerts · demographics ·
│   │                       countries · reactions · outcomes · reporters · trends
│   ├── evidence/           EvidenceStore + unsupported-claim vocabulary
│   ├── ai/                 context builder · LLM client · deterministic
│   │                       MockLLM · section generators
│   ├── validation/         publication gate (D1-D7, A1-A8, R1-R3)
│   ├── reporting/          case index · report assembly · HTML · exporters
│   │                       (PDF/DOCX) · sentence traceability · evaluation ·
│   │                       review state
│   └── pipeline.py         end-to-end orchestrator
├── prompts/                system_prompt.md + 4 section prompts + context_schema.json
├── tests/                  70 pytest tests (real dataset + fixtures)
├── scripts/generate_report.py
├── outputs/                report_output.md/.html/.pdf/.docx · case_index.csv ·
│                           evidence.json · sentence_evidence.json ·
│                           validation.json · evaluation.json · analysis_summary.json
├── version1/README.md      config-driven PADER/PSUR/PBRER/DSUR/CSR engine design
├── docs/data_profile.md    verified dataset profile + documented assumptions
├── architecture.md + architecture.svg
└── README.md · requirements.txt · .env.example · .gitignore

4. Install

Requires Python 3.11+ and Node 18+ (only for the dashboard).

# 1) Backend + pipeline
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# 2) Optional LLM key (copy .env.example to .env and fill in)
cp .env.example .env               # set LLM_API_KEY=... to use a real model

# 3) Dashboard (optional)
cd app/frontend
npm install
npm run build                      # bundles into app/frontend/dist
cd ../..

Dataset placement: put Bisoprolol_icsr_sample_1068rows.csv (or the XLSX variant) into data/. The loader auto-detects .csv / .xlsx; a DATASET_PATH env var overrides the location. The dataset is git-ignored and excluded from the submission (see Data Usage Notice).

5. How to run

One command to regenerate the report:

python scripts/generate_report.py          # mock narrative mode (no API key needed)
python scripts/generate_report.py --no-ai  # force deterministic mock
python scripts/generate_report.py --section trends_analysis   # regenerate one section
python scripts/generate_report.py --dataset /path/to/file.csv

The CLI runs the pipeline, prints the validation checks, and writes:

outputs/report_output.md, outputs/report_output.html, outputs/report_output.pdf, outputs/report_output.docx, outputs/case_index.csv, outputs/evidence.json, outputs/sentence_evidence.json, outputs/validation.json, outputs/evaluation.json, outputs/analysis_summary.json, outputs/ai_sections.json.

Exit code is 0 when the report passes validation and is published, 1 otherwise (a draft is written to outputs/generated/draft_report_output.md with the failed checks).

Run the tests:

pytest tests/ -q          # 70 tests (real-dataset tests skip if data/ is absent)

Run the backend + dashboard:

uvicorn app.backend.main:app --port 8000   # API + built dashboard at http://127.0.0.1:8000

Development mode for the frontend (hot reload, proxies /api to port 8000):

cd app/frontend && npm run dev             # http://127.0.0.1:5173

6. The API

Endpoint	Purpose
`GET /api/health`	API status, narrative mode, published flag
`GET /api/overview`	KPI cards, reporting period, dedup facts
`GET /api/dataset`	Schema profile, dedup stats, missing values
`GET /api/case-analysis`	Monthly, seriousness, age, sex, countries, outcomes, reporters
`GET /api/reaction-analysis`	Top reactions, by-age/by-sex, monthly trends
`GET /api/alert-analysis`	Alert metrics, criteria, outcomes, example cases
`GET /api/serious`	Seriousness criteria, top serious reactions, alert overlap
`GET /api/trends`	Monthly series, top reactions per month, observations
`GET /api/case-index`	Searchable / filterable / sortable case records (paginated)
`GET /api/case-index/export`	Download the current filter as CSV
`GET /api/evidence`	Claim register
`GET /api/evidence/sentences`	Sentence → evidence grounding map
`GET /api/report`	Report markdown/html, sections, validation, review state
`POST /api/report/review`	Approve / flag (with comment + reviewer name)
`POST /api/report/notes`	Add a review note
`POST /api/report/regenerate`	Regenerate one AI section
`POST /api/report/edit-section`	Replace a section with a human revision (re-validated)
`GET /api/export/{md,html,pdf,docx}`	Download the current report
`GET /api/validation`	Full check results
`GET /api/settings`	Masked runtime configuration

Interactive docs: http://127.0.0.1:8000/docs.

7. How the dashboard is organized

Page	Contents
Dashboard	Reporting-period banner, animated KPI cards with sparklines and trend indicators, validation status, monthly/severity/age/alert charts
Dataset	Input rows vs. unique cases, dedup facts, schema validation, missing values, reporting period
Case Analysis	Monthly volume, serious vs non-serious, age, sex, countries, outcomes, reporters
Reaction Analysis	Top reactions (cases/records/serious/alert), by-age, by-sex, monthly trends
Serious Cases	Criteria breakdown, alert overlap, top serious reactions → jump to a filtered Case Index
Trend Analysis	Monthly volume/serious/alert/fatal, deterministic observations, top reactions per month
AI Report	Collapsible sections, markdown rendering, copy + MD/HTML/PDF/DOCX export, review workflow, 22 validation checks
Evidence	Sentence → evidence → calculation → source fields → case IDs, plus the full claim register
Case Index	Search, filters, sortable columns, pagination, column selector, CSV export, details drawer
Settings	LLM provider (masked), paths, review workflow, data usage notice

Every chart has a title, subtitle, axis labels, counts and a Download PNG button. The top-right badge is API Connected (green) or Backend Offline (red) — the UI never implies the data is mocked.

8. Where deterministic processing happens

All of src/ except src/ai/:

Module	Responsibility
`src/data/loader.py`	CSV/XLSX ingestion, schema inspection
`src/data/dates.py`	E2B `YYYYMMDD` date parsing, reporting period
`src/data/dedup.py`	Case-level dedup (highest `safetyreportversion`), reaction explosion
`src/analysis/`	overview, seriousness, alerts, demographics, countries, reactions, outcomes, reporters, trends
`src/evidence/store.py`	traceable evidence objects
`src/reporting/`	case index, report assembly, HTML, PDF/DOCX exporters, sentence traceability, evaluation
`src/validation/validate.py`	publication gate checks

9. Where AI is used — and only here

src/ai/ — narrative prose for four report sections (Executive Summary, Reaction Analysis, Alert Analysis, Trends Analysis). The LLM:

receives only a verified JSON context packet (no raw rows),
is told to transform the numbers into professional narrative,
never computes statistics (the system prompt forbids it),
is validated after generation (see §12).

Provider is configurable via .env (LLM_BASE_URL, LLM_MODEL, LLM_API_KEY) using any OpenAI-compatible /chat/completions endpoint. With no key, the deterministic mock renders the same contexts (clearly labelled in the report header as the narrative source).

10. Why AI is not used for basic calculations

Accuracy. Case counts, seriousness and deduplication must be exact; generative models are not calculators.
Auditability. Every number must point to source fields and case IDs — deterministic code can prove it; prose cannot.
Stability. Statistics must not change between runs or models.
Cost/latency. Computing 20 metrics from 1,068 rows is milliseconds in pandas vs. seconds and tokens on an LLM.
The regulatory spirit. The report's numbers are the ground truth a human reviewer signs off on; AI only adds narrative interpretation.

11. How context is assembled (prompt engineering)

All prompts are files in prompts/ — never hard-coded in Python:

prompts/
├── system_prompt.md          # shared pharmacovigilance constraints
├── narrative_summary.md      # Executive / Narrative Summary section
├── reaction_analysis.md      # Reaction / Adverse Event Analysis section
├── alert_analysis.md         # 15-Day Alert Analysis section
├── trends_analysis.md        # Trends and Important Observations section
└── context_schema.json       # JSON schema every context packet must satisfy

src/ai/context.py builds one section-specific packet per narrative section. Each packet is validated against the JSON schema before the LLM call, and the context hash is stored with the generated section for reproducibility. Section-specific prompts are used instead of one giant prompt because each section needs different verified values: smaller contexts reduce the chance of hallucination, keep prompts auditable, and make regeneration of one section cheap (the analysis is cached; only that section is re-generated).

12. How evidence tracing works

Two layers:

Claim register — every important claim registers an Evidence object:

{
  "claim": "Unique cases reporting 'Acute kidney injury'",
  "value": 80,
  "unit": "cases",
  "source_fields": ["patient_reaction_reactionmeddrapt", "safetyreportid"],
  "calculation": "unique safetyreportid reporting the PT (case count)",
  "n_case_ids": 80,
  "case_ids": [24780680, 24795755, ...],
  "basis": "deterministic",
  "section": "reactions"
}

Sentence grounding (src/reporting/traceability.py) — every narrative sentence is linked to the claims whose verified values and keywords it repeats. The Evidence page shows sentence → evidence chips → calculation → source fields → case IDs, and outputs/sentence_evidence.json persists the map. This is the same containment rule the validation gate enforces, applied at sentence granularity.

The case index lets you drill from any aggregate down to individual cases, and the API exposes the full chain (/api/evidence, /api/evidence/sentences, /api/case-index).

13. How hallucination prevention / validation works

Two families of checks plus a completeness check (all deterministic, in src/validation/validate.py):

D1–D7 (data consistency): unique-case total == safetyreportid nunique; reaction counts ≤ reaction records; serious count is case-level (any flag) and matches the serious field; alert count from fulfillexpeditecriteria; percentages/bucket sums correct; reporting period == MIN/MAX receivedate; every case ID and reaction in the report exists in the dataset.
A1–A8 (AI grounding): no expectedness claims; no safety-action claims; no SOC claims; every number the LLM wrote must exist in its context packet; no fabricated NDA/application identifiers; no invented patient details or case IDs; no causality/signal/diagnosis/label claims; all required sections non-empty.
R1–R3 (report completeness): all 13 sections, the required disclaimer, and the mandated statements ("SOC-level analysis was not performed…", "Expectedness assessment was not performed…", "No safety-action information was provided…", "Not supplied in dataset", "PADER-style engineering report").

If any hard check fails, the report is not published — a draft with the failed checks is written instead and the API surfaces the errors.

14. How human review works

The report is born with status Human Review Required. In the dashboard (AI Report page) a reviewer can:

Approve the report, or Flag it — each with an optional comment and reviewer name (recorded with a timestamp),
Add notes without changing status,
Regenerate a single narrative section (re-run → re-validate → re-assemble),
Edit a section directly — human revisions go through the same grounding validation, so an ungrounded number still blocks publication.

Review state is persisted in outputs/review_state.json and shown alongside three badges: AI Generated, Deterministically Verified, Human Review Required / Approved / Flagged. The system never implies AI output is a final clinical or regulatory conclusion.

15. Architecture

See architecture.md and architecture.svg.

CSV → Schema & Data Validation → Deterministic Analysis (pandas)
    → Analysis Result + Evidence Store → Context Builder (per section)
    → LLM Narrative (or deterministic mock) → Output Validator (22 checks)
    → Human Review → Report Generator → report_output.{md,html,pdf,docx}
    + Case Index and Dashboard (FastAPI + React) for traceability

16. Actual prompts used

See prompts/ — the system prompt (prompts/system_prompt.md) is shared by every section; each section has its own prompt file (.md); the packet shape is declared in prompts/context_schema.json. The mock generator (src/ai/mock.py) documents the exact rendering contract the real LLM is asked to follow. The prompt_files field of every generated section (and outputs/ai_sections.json) records exactly which prompt files produced it.

17. Models used

Default provider: OpenAI-compatible chat completions (default model gpt-4o-mini, configurable via LLM_MODEL / LLM_BASE_URL).
Fallback: deterministic template mock — no external calls, used whenever LLM_API_KEY is unset. All shipped outputs in this repository were generated in mock mode; the narrative source is always disclosed in the report.

18. Tests

tests/ — 70 tests. Highlights:

test_dedup.py proves 1068 rows != 1024 cases and that the highest version per case is selected.
test_analysis.py verifies seriousness (1023/1024), criteria counts, alert/serious identity, age buckets (75+ = 408, Unknown = 83), sex (493/503/28), countries (occurcountry; top = eu 342), outcomes, reporters, monthly peaks.
test_reactions.py proves case counts ≠ record counts (AKI = 80 cases).
test_validation.py shows fabricated numbers / NDA / signal claims fail, and grounded text passes.
test_pipeline.py proves reproducibility, single-section regeneration, that partial runs never clobber published artifacts, and that human edits still pass the grounding gate.
test_exporters.py verifies PDF/DOCX output and sentence-level traceability.
Fixture-based tests run even without the dataset; real-dataset tests skip cleanly when data/ is absent.

19. Evaluation approach

src/reporting/evaluation.py scores every report 0–100 on five deterministic criteria: metric consistency, AI-number grounding, evidence coverage, unsupported-claim absence, and required-section presence. Results are written to outputs/evaluation.json.

Scaling to 1,000 reports: all checks are cheap regex/set operations, so a batch runner can ProcessPoolExecutor the pipeline per dataset and aggregate scorecards; LLM-as-judge is only needed later for semantic quality (clarity, tone), which is documented in version1/README.md.

20. Known limitations

Mock vs. real model: shipped outputs use the deterministic mock; with a real LLM the vocabulary scan is best-effort (regex) — a sophisticated rewrite could still pass, which is why human review remains mandatory.
Number-grounding is containment, not semantics: the validator ensures every AI number exists in context, but cannot yet prove the number is used in the right place (LLM-as-judge planned for V1).
Outcome alignment: 6 rows have longer outcome lists than reaction lists; the surplus is truncated (documented policy, docs/data_profile.md).
Age units: rows with garbage/missing unit codes and missing ages → Unknown bucket; ages with missing unit but a value are assumed years (configurable).
Country values are heterogeneous (eu, ISO-2 codes, full names) and are reported as supplied; only unambiguous ISO codes are normalized for display.
No SOC / expectedness / safety actions / NDA — the dataset does not provide them and the report says so explicitly.
The narrative field contains only placeholder text (CASE EVENT DATE: …) and is not used for analysis.

21. Version 1 design

See version1/README.md: the same engine parameterized as a config-driven reporting system — PADER, PSUR, PBRER, DSUR and CSR become configurations of one evidence-grounded engine.

22. Data Usage Notice

The Bisoprolol ICSR dataset and accompanying reference materials are provided solely for the GenAR hiring/technical evaluation exercise. The dataset is synthetic and/or derived from publicly available adverse-event reporting sources and does not represent confidential, proprietary, or real patient records.

The materials must not be used commercially, redistributed, published, or reused outside the scope of the assignment. Local copies should be deleted once the evaluation is complete.

This project is an engineering prototype, not a complete regulatory PADER authoring system.
Generated outputs require qualified human review.
It must not be used for clinical or regulatory decision-making.

Git safety: the dataset (data/), the supplied reference materials (supplied/, docs/_pdf_text/), .env, .venv/, node_modules/, outputs/generated/ and all *.csv files are git-ignored. The final submission ZIP must not contain the dataset CSV.

Navigation Menu

Repository files navigation

GenAR PADER Analyzer

1. What the project does

2. Why this architecture was chosen

3. Folder structure

4. Install

5. How to run

6. The API

7. How the dashboard is organized

8. Where deterministic processing happens

9. Where AI is used — and only here

10. Why AI is not used for basic calculations

11. How context is assembled (prompt engineering)

12. How evidence tracing works

13. How hallucination prevention / validation works

14. How human review works

15. Architecture

16. Actual prompts used

17. Models used

18. Tests

19. Evaluation approach

20. Known limitations

21. Version 1 design

22. Data Usage Notice

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
app		app
docs		docs
outputs		outputs
prompts		prompts
scripts		scripts
src		src
tests		tests
version1		version1
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
architecture.md		architecture.md
architecture.svg		architecture.svg
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation

GenAR PADER Analyzer

1. What the project does

2. Why this architecture was chosen

3. Folder structure

4. Install

5. How to run

6. The API

7. How the dashboard is organized

8. Where deterministic processing happens

9. Where AI is used — and only here

10. Why AI is not used for basic calculations

11. How context is assembled (prompt engineering)

12. How evidence tracing works

13. How hallucination prevention / validation works

14. How human review works

15. Architecture

16. Actual prompts used

17. Models used

18. Tests

19. Evaluation approach

20. Known limitations

21. Version 1 design

22. Data Usage Notice

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages