Clinical reasoning across imaging, pathology and genomics
A clinical agent harness connecting language-model reasoning with imaging, pathology, genomic information, and medical evidence.
Project overview from the existing repository. See the publication for the study figures, methods, and accompanying credits.
- 📄 Publication: Nature Cancer · 2025
Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology
Ferber et al. · Nature Cancer 6, 1337–1349 (2025). - 📊 Published results
- 🧩 Agent harness
- 🗂️ Code map
- ⚙️ Getting started
- 📚 Citation / BibTeX
The study investigates how a GPT-4-based agent can select tools, chain their outputs, and combine multimodal findings with clinical knowledge to answer precision-oncology questions. The harness brings together pathology models, medical-image analysis, genomic interpretation, literature search, and evidence-grounded response generation.
Four medical reviewers evaluated 20 constructed patient scenarios combining real imaging/pathology data with simulated patient histories and genomic information. The following are the results reported for the study system.
| Evaluation axis | Agent result | Definition or comparator |
|---|---|---|
| Completeness | 87.2% (95/109) | Expected clinical decisions covered; 30.3% (33/109) with GPT-4 alone |
| Correctness | 91.0% (223/245) | Answer paragraphs judged factually correct |
| Helpfulness | 94.0% (63/67) | User questions or instructions effectively addressed |
| Required tool use | 87.5% (56/64) | Required tool invocations successfully completed |
The GPT-4-alone comparison above applies to completeness. Reviewers assessed the other axes separately; see Figure 4 and the supplementary material for scoring definitions and the full evaluation.
The workflow combines three stages:
- Reason and select tools: interpret the question and patient context, then identify useful analyses.
- Execute and integrate: coordinate tool calls and use their outputs in subsequent reasoning steps.
- Ground and synthesize: combine the resulting evidence with clinical documents and return a referenced answer.
The research system integrates medical-image segmentation with MedSAM, pathology-model inference, OncoKB, PubMed and web search, calculation, and document retrieval. Tool interfaces and orchestration are organized separately from evidence indexing and response evaluation.
Implementation files are under RAGent/DSPY/.
| Area | Entry points |
|---|---|
| Agent orchestration | med_agent.py |
| Clinical tool interfaces | agent_tools.py |
| Experiment walkthrough | run_experiment.ipynb |
| Retrieval and evidence synthesis | rag.py, chroma_db_retriever.py |
| Citation handling and prompts | citations_utils.py, signatures.py |
| Indexing and preprocessing | embed.py, filter_data_sources.py, deduplicate_data.py, preprocess_sources.py |
| Configuration | rag_config.py, rag_utils.py |
The original experiments used Python 3.11.6. Install the dependencies in an isolated environment:
git clone https://github.com/Dyke-F/LLM_RAG_Agent.git
cd LLM_RAG_Agent
python3.11 -m venv medvenv
source medvenv/bin/activate
python -m pip install -r requirements.txtTool-specific computation may use a CUDA-capable GPU. Model and external-service access should be configured for the components you intend to use.
Keep API credentials in a local .env file at the repository root. The existing configuration uses:
OPENAI_API_KEY=your_openai_api_key
COHERE_API_KEY=your_cohere_api_key
GOOGLE_API_KEY=your_google_api_key
GOOGLE_SEARCH_ENGINE=your_search_engine_idFor OncoKB-enabled workflows, arrange the appropriate access through OncoKB and configure the tool for your authorized environment. Do not commit credentials.
rag_config.py defines document locations, the Chroma collection and storage path, chunking, and model settings. rag_utils.py defines the document metadata used during indexing. Model identifiers in the repository describe the original experimental setup; check service availability before a run and record any changes as part of your experimental configuration.
Use clinical documents that you are authorized to process. One source used in the project is the Meditron guidelines collection; review the source documents' terms before use.
The preprocessing utilities support topic filtering, duplicate removal, and document IDs:
- scrape_meditron.py: obtain the source collection.
- filter_data_sources.py: select relevant documents and topics.
- deduplicate_data.py: remove duplicate records.
- preprocess_sources.py: prepare IDs and metadata for indexing.
Set the input and output directories for your local corpus. The embedding stage expects JSONL documents with the text field clean_text and the metadata configured for that source.
From RAGent/DSPY/, configure RAGConfig and start Chroma using the storage path selected in default_client_path. In another terminal, run embed.py with --to_embed set to the intended input files. Keep the collection settings consistent between indexing and retrieval.
Open run_experiment.ipynb from RAGent/DSPY/. It provides an experiment walkthrough for the agent's tool interface and evidence-synthesis workflow. Review the notebook's selected tool configuration, case selection, service settings, and local paths before execution.
For the published evaluation protocol and reported results, use the paper and its supplementary material.
This repository supports research on clinical agent systems. Keep any patient-level inputs, images, generated outputs, and credentials within the environment authorized for those data. Dataset, model, and API access conditions apply independently of the source-code license. Only use external services under the applicable data-use permissions.
The original setup uses caching in DSPy and the retrieval stack. Review the selected cache configuration when comparing experimental runs, and record model, prompt, corpus, and tool settings alongside results.
@article{ferber2025oncologyagent,
title = {Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology},
author = {Ferber, Dyke and El Nahhas, Omar S. M. and W{\"o}lflein, Georg and
Wiest, Isabella C. and Clusmann, Jan and Le{\ss}mann, Marie-Elisabeth and
Foersch, Sebastian and Lammert, Jacqueline and Tschochohei, Maximilian and
J{\"a}ger, Dirk and Salto-Tellez, Manuel and Schultz, Nikolaus and
Truhn, Daniel and Kather, Jakob Nikolas},
journal = {Nature Cancer},
volume = {6},
pages = {1337--1349},
year = {2025},
doi = {10.1038/s43018-025-00991-6},
url = {https://doi.org/10.1038/s43018-025-00991-6}
}Repository code is covered by the existing MIT license. The article is published under Creative Commons Attribution 4.0, subject to its third-party credit lines. The study summary above is newly written; the existing repository overview image is retained without modification. External data, model weights, software, and services retain their own terms.