Skip to content
ebenbarukPublic

About

Multi-agent engine that turns a research goal into ranked, defensible scientific hypotheses — grounded in open-access literature, claim-level verified, and red-teamed. Runs on any OpenAI-compatible model (default: NVIDIA Nemotron).

Topics

Resources

Code of conduct

Contributing

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

██╗  ██╗███████╗██╗   ██╗██████╗ ███████╗██╗  ██╗ █████╗
██║  ██║██╔════╝██║   ██║██╔══██╗██╔════╝██║ ██╔╝██╔══██╗
███████║█████╗  ██║   ██║██████╔╝█████╗  █████╔╝ ███████║
██╔══██║██╔══╝  ██║   ██║██╔══██╗██╔══╝  ██╔═██╗ ██╔══██║
██║  ██║███████╗╚██████╔╝██║  ██║███████╗██║  ██╗██║  ██║
╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝
        generate · verify · falsify · rank · evolve

Heureka

A multi-agent engine that turns a research goal into ranked, defensible scientific hypotheses.

CI License: Apache 2.0 Python 3.12+ Lint: ruff PRs welcome

Given a research goal in plain language, Heureka runs a team of specialised agents that mirror the scientific method — generate → critique → verify → falsify → rank → evolve. It grounds its reasoning in real open-access literature, decomposes each idea into individually-checkable claims, red-teams the survivors, and converges on a small set of high-quality, testable hypotheses with proposed experiments, per-passage citations, and calibrated confidence.

Heureka is open source, model-agnostic, and ships on NVIDIA Nemotron through a single LLM client. Every run is fully reproducible: it produces a self-contained directory with the configuration, every hypothesis, every review and claim graph, every tournament match, a quality-vs-compute trajectory, and a readable report.

⚠️ Research aid, not an oracle. Hypotheses are AI-generated and must be reviewed by domain experts and validated experimentally before any use.

How it works

Heureka is budget-driven, not round-driven. A Supervisor runs an asynchronous priority queue of typed tasks over a bounded worker pool until a compute budget — counted in LLM calls — is spent. Compute flows to promising, well-tested lineages; fresh ideas keep being injected so the search never collapses. Measured quality rises with compute up to a sweet spot (~40 calls on the bundled model) rather than unboundedly.

flowchart TD
    GOAL["Research goal<br/>(natural language)"] --> SUP

    subgraph SUP["Supervisor — async task queue · budget = LLM calls"]
      direction LR
      GEN["Generation"] --> REV["Reflection"]
      REV --> DEEP["Deepen<br/>verify · falsify · novelty · safety"]
      DEEP --> MATCH["Ranking<br/>Elo tournament"]
      MATCH --> EVO["Evolution"]
      EVO -. "evolve leaders" .-> GEN
      MATCH -. "meta-review feedback" .-> GEN
    end

    SUP --> RANK["rank_final<br/>LCB Elo + verified quality + novelty − falsification"]
    RANK --> OUT["report.md · trajectory.csv · resumable state.json"]
Loading
Agent Role
Generation Proposes new hypotheses via a simulated debate between experts, grounded in literature.
Reflection Reviews each hypothesis (novelty, plausibility, testability, safety, correctness); on survivors, runs deep verification — decomposing the idea into a claim graph and checking each premise against passages — plus quantitative novelty.
Falsification A red team that tries to kill each survivor: a killer experiment, contradicting evidence, and hidden assumptions, with a severity score.
Safety A guardrail that flags dual-use, biosafety, or otherwise clearly dangerous directions for human review.
Ranking Runs pairwise debate-style comparisons and converts outcomes into Elo ratings (optional swap-check for positional bias).
Evolution Improves and recombines the leaders using diverse strategies (simplify, combine, analogy, challenge-assumption, feasibility, deepen-mechanism).
Meta-review Distils recurring critiques into feedback that steers later generation/evolution, and writes a closing research overview.

Defensibility pipeline

A hypothesis is only as strong as its weakest premise, so Heureka verifies the claims, not just the prose — a survival funnel where a refuted idea or a contradicted premise is gated down, never averaged up:

flowchart TD
    HYP["Candidate hypothesis"] --> DEC["Decompose → claim graph<br/>premise · novel_link · entailment"]
    DEC --> VER["Verify each premise vs literature<br/>supported / unverified / contradicted<br/>(+ verbatim cited snippet)"]
    VER --> NOV["Measure novelty of the novel link<br/>distance to nearest prior work"]
    NOV --> FAL["Red-team falsification<br/>killer experiment + contradicting evidence"]
    FAL --> SEL{"Survives the attack?"}
    SEL -- "yes" --> TOP["Ranked, defensible hypothesis<br/>+ calibrated confidence"]
    SEL -- "refuted / premise contradicted" --> GATE["Score gated down<br/>not the top recommendation"]
Loading
  • Claim graphs with per-passage provenance. Each survivor is decomposed into atomic claims (premises, the novel link, entailments). Premises are checked against literature and marked supported / unverified / contradicted, each with a verbatim snippet and the paper it came from.
  • Deep verification + quantitative novelty. Novelty is measured, not asserted: a distance from the nearest prior passage plus a prior-art count for the novel link.
  • Falsification / red team. Survivors face a killer experiment and a search for contradicting evidence; the outcome seeds the tournament (survivors enter higher, killed ones lower).
  • Gated scoring. Quality is a conjunction, not an average — one contradicted premise collapses the score instead of being diluted by good ratings elsewhere.
  • Calibrated uncertainty + surprise. After the run, each hypothesis gets a plain-language confidence band (speculative → promising but untested → well-supported) and a deterministic surprise score estimating how much would change if it were true.
  • Lower-confidence-bound ranking. The headline ranking uses an LCB Elo, so thinly-tested hypotheses are penalised rather than trusted.

Install

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Configure

cp .env.example .env
# then set NVIDIA_API_KEY in .env

A free API key is available at https://build.nvidia.com.

Run

# Start a run. --budget is the compute budget in LLM calls: more budget -> better
# hypotheses (quality scales with test-time compute).
heureka run --goal "Identify novel epigenetic targets for liver fibrosis" --budget 60

# Add preferences/constraints, or disable literature grounding
heureka run -g "Repurpose approved drugs for acute myeloid leukemia" \
  --preferences "prioritise drugs with established safety profiles" \
  --constraints "in vitro validatable in standard cell lines"

# Expert-in-the-loop: pause at the budget midpoint to veto weak leaders and add a
# steering note, then resume against the full budget.
heureka run -g "Mechanisms of resistance to senolytics in aged tissue" --review

# Resume an interrupted run (continues toward the budget)
heureka resume runs/20260620-143000-identify-novel-epigenetic-targets

Each run writes to runs/<timestamp>-<slug>/:

runs/<run>/
  state.json        full, resumable run state (config, hypotheses, claim graphs, matches)
  rounds/           periodic provenance snapshots
  trajectory.csv    the quality-vs-compute curve (best/LCB Elo per LLM call)
  report.md         ranked hypotheses with claim graphs, falsification, novelty, safety, and proposals

End-to-end example

# 1. Configure once
cp .env.example .env && $EDITOR .env        # set NVIDIA_API_KEY

# 2. Run with a modest budget
heureka run \
  -g "Identify host metabolic pathways that limit influenza A replication" \
  --constraints "testable in human lung epithelial organoids" \
  --budget 80

# 3. Inspect the results
#    - the terminal prints the top hypotheses by Elo
#    - open the report and plot the scaling curve
open runs/*/report.md
column -s, -t runs/*/trajectory.csv | less    # best Elo rising with each LLM call

The report opens with the quality-vs-compute summary and research overview, then lists the ranked hypotheses — each with its claim graph (✓/?/✗ premises and cited snippets), measured novelty, falsification verdict, safety flag, and confidence/surprise — followed by ready-to-run research proposals for the leaders.

Literature grounding

Heureka grounds hypotheses in legal, open-access sources only — OpenAlex, Europe PMC, and arXiv, with Unpaywall resolving open-access full text. Grounding is semantic: it over-fetches candidates and reranks them by embedding similarity to the claim being checked (with a lexical fallback when embeddings are unavailable). You can pin grounding to literature published on or before a given year, which enables recapitulation — asking whether Heureka would have proposed a known discovery from only the literature that preceded it.

Adding a source is as simple as implementing the LiteratureProvider protocol (src/heureka/literature/base.py) and registering it with the Aggregator. Heureka deliberately ships no tooling for circumventing paywalls.

Configuration

All settings are read from the environment with the HEUREKA_ prefix (see .env.example): the compute budget, initial breadth, meta-review interval, deep-verify depth, falsification seeding, semantic-rerank options, evolution strategies, LCB constant, concurrency, and max_generations (cap evolution depth; 0 = uncapped).

Providers & model routing

Heureka is provider-agnostic — it speaks the OpenAI-compatible Chat Completions + Embeddings API, which is what Nemotron is served through. To switch providers, set three things in .env:

Variable NVIDIA Nemotron (default) Plain OpenAI
Key HEUREKA_API_KEY (or NVIDIA_API_KEY) nvapi-… sk-…
Endpoint HEUREKA_BASE_URL https://integrate.api.nvidia.com/v1 https://api.openai.com/v1
Model HEUREKA_MODEL nvidia/nemotron-3-super-120b-a12b gpt-5.4

Two provider-specific request extras are configurable so non-Nemotron endpoints don't choke on Nemotron-only fields: HEUREKA_ENABLE_THINKING (set false for plain OpenAI; or supply a full HEUREKA_CHAT_EXTRA_BODY JSON override) and HEUREKA_EMBEDDING_INPUT_TYPE (set empty for OpenAI-style embeddings).

Per-role model routing is optional: HEUREKA_MODELS is a JSON map of agent role → model, so you can send heavy work to a strong model and cheap work to a fast one. Empty (the default) uses HEUREKA_MODEL for every role.

# e.g. a strong model for idea generation, a cheaper one for the tournament
HEUREKA_MODELS='{"generation": "strong-model", "ranking": "cheap-model"}'

Roles: generation, reflection, falsification, ranking, evolution, meta_review, safety, default.

Evaluation

Heureka ships a reproducible harness that tests its core claim — does the multi-agent scaffold beat a single one-shot call to the same model? For each benchmark goal it runs Heureka, generates a one-shot baseline through the same client, and has a blind, majority-vote judge compare the two (side assignment randomised). Because both sides share the model, the comparison isolates the value of the scaffold rather than the model's raw ability.

python -m heureka.eval                      # all goals, budget 40/goal, 5-vote judge
python -m heureka.eval --budget 40 --goals 5 --votes 5
python -m heureka.eval --cutoff             # recapitulation mode (date-restricted grounding)

What we observe on the bundled model (NVIDIA Nemotron, free tier) is a quality sweet spot around a budget of ~40 calls, where the scaffold wins a clear majority of blind comparisons; very small budgets under-cook the hypotheses, and at large budgets the evolution step tends to over-elaborate (set HEUREKA_MAX_GENERATIONS to cap this). These numbers are indicative, not definitive: the benchmark is small and a single-model LLM judge is noisy, so treat the win-rate as a relative signal. A stronger model and a larger judged benchmark are expected to sharpen — and likely raise — the result.

Develop

pytest                 # unit + integration tests (no network — LLM and HTTP are mocked)
ruff check src tests

Project layout

src/heureka/
  cli.py            CLI (run / resume, with --budget and --review)
  config.py         settings (pydantic-settings)
  models.py         data models (pydantic v2)
  llm.py            single-model client (async, retries, reasoning traces)
  elo.py            tournament rating maths
  scoring.py        deterministic confidence band + surprise
  vectors.py        cosine similarity for semantic grounding
  memory.py         persistent, resumable run store + trajectory writer
  orchestrator.py   the budget-driven Supervisor
  report.py         markdown report rendering
  agents/           the specialised agents
  literature/       pluggable open-access providers + semantic aggregator
  eval/             benchmark goals, baseline, blind judge, harness

Ethics & limitations

Heureka is a research aid: its hypotheses are AI-generated and must be reviewed by domain experts and validated experimentally before any use. A safety agent flags dual-use and biosafety concerns for human review, but it is not a substitute for institutional oversight. Heureka relies on open-access literature, so it can miss paywalled prior art; it inherits the factuality limits and possible hallucinations of its underlying model; its novelty and grounding signals are heuristic; and its rankings are relative, not ground truth. Treat its output as a starting point for expert inquiry.

Output quality is bounded by the base model. Heureka is model-agnostic — point HEUREKA_MODEL / HEUREKA_BASE_URL at any OpenAI-compatible endpoint — and recombines and reasons over existing knowledge rather than running experiments, so its ceiling rises with the strength of the underlying LLM and the breadth of literature it can reach. The bundled results use a small, free-tier model; a frontier model and a higher API quota are expected to produce markedly stronger hypotheses.

Contributing

Heureka is open source and contributions are welcome — bug reports, documentation, new literature providers, new agents, evaluation goals, and core improvements. See CONTRIBUTING.md for setup, the test/lint workflow, and the project conventions, and the Code of Conduct. The full test suite is offline (no API key needed):

pip install -e ".[dev]"
pytest -q && ruff check src tests

CI runs the same checks on every pull request. Small, focused PRs land fastest — when in doubt, open an issue first to discuss.

License

Apache-2.0 — see LICENSE. By contributing, you agree your contributions are licensed under the same terms.

About

Multi-agent engine that turns a research goal into ranked, defensible scientific hypotheses — grounded in open-access literature, claim-level verified, and red-teamed. Runs on any OpenAI-compatible model (default: NVIDIA Nemotron).

Topics

Resources

Code of conduct

Contributing

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages