An open-source, agent-agnostic extraction and fetch layer that returns typed data with a transparent confidence score on every result.
Turn any URL — or any question about the web — into verified, typed data, callable by any AI agent through MCP, REST, or a native SDK.
Quickstart · Tools · Write a recipe · Vision · Security
Evident is not another scraper. Best-in-class open-source scraping/rendering engines already exist (Crawl4AI, Playwright). Evident orchestrates them behind a resilience ladder, scores every result's trustworthiness, and lets you extract structured data from any site — not just ones someone hand-wrote a parser for — via a versioned, community-contributable recipe system.
Full vision, architecture, and roadmap:
docs/VISION.md.
Most extraction tools give you clean text and let you figure out whether to trust it. Evident's whole design centers on one missing piece: every result carries a confidence score and a method explaining how it was produced, so an autonomous agent — not a human — can decide whether to act on it.
What
confidenceactually measures today: which code path produced a result (recipe match vs. LLM extraction vs. raw fetch, official API vs. reverse-engineered, fully-rendered vs. partial content) — a provenance/method signal, not a correctness signal. It is deterministic and internally consistent (a recipe match always outscores raw fetch, for example), but it is not yet calibrated against any ground truth — nothing in the pipeline compares an extracted value to what's actually correct. Treat a 0.9 as "produced by a method that's usually reliable," not as "90% likely to be factually right." Calibration against a held-out benchmark is tracked as future work indocs/VISION.md§9.
git clone https://github.com/Kaushalendra-Marcus/evident
cd evident
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"
# Run the MCP server (stdio) — works with Claude Desktop, Claude Code, Cursor,
# or any other MCP-compatible client
evident-mcpAdd to your MCP client config (example for Claude Desktop):
{
"mcpServers": {
"evident": {
"command": "/absolute/path/to/.venv/bin/evident-mcp"
}
}
}Not using an MCP client? Same engine, plain Python:
import asyncio
from evident.core import ladder
async def main():
result = await ladder.run("https://example.com")
record = ladder.to_record(result)
print(record.confidence, record.method)
print(record.data.get("markdown", "")[:500])
asyncio.run(main())| Tool | What it does |
|---|---|
fetch(url, mode) |
Universal fetch, escalates the resilience ladder automatically |
extract(url, json_schema) |
Structured extraction against any caller-supplied schema — works on any site |
list_recipes() |
Discover built-in, verified extraction recipes |
use_recipe(recipe_id, slug, entity_name) |
Invoke a deterministic, high-confidence recipe (e.g. ats_greenhouse) |
health_check(target) |
Proactively check whether a recipe or URL is still working |
Evident's core (Tier 1 static fetch) has minimal dependencies on purpose. Heavier capabilities are opt-in:
pip install "evident[render]" # Tier 2: JS-rendered pages via Crawl4AI/Playwright
pip install "evident[llm]" # extract(): LLM-based schema extraction (bring your own ANTHROPIC_API_KEY)
pip install "evident[api]" # REST API interface
pip install "evident[all]" # everything, plus dev/test toolingIf render isn't installed and Tier 1 fails, fetch() reports failure_reason: dependency_missing instead of crashing — Tier-1-only installs stay fully usable for the large share of the web that's server-rendered.
Contributions are welcome. The single most valuable — and lowest-friction — contribution is a recipe: one YAML metadata file plus one small async fetcher function, no need to understand the resilience ladder or confidence engine. See docs/RECIPE_GUIDE.md for the recipe walkthrough, and CONTRIBUTING.md for the development setup, the two project rules every change must satisfy (a test that would have caught the bug; no unverified "it works" claims), and how to run the checks CI runs.
pip install -e ".[dev]"
pytestTests use recorded/mocked HTTP responses (respx) so they run deterministically without live network access — this was the single biggest gap in earlier hand-rolled scraping projects this one grew out of, and it's non-negotiable here.
Early / pre-1.0. Tier 1 (static fetch) and the recipe registry (Greenhouse, Lever, Ashby) are implemented and unit-tested against mocked fixtures. Tier 2 (rendered fetch via Crawl4AI) is implemented and has been smoke-tested against a live page. LLM-based extract() is implemented but requires your own ANTHROPIC_API_KEY and hasn't been live-tested end-to-end yet — see docs/VISION.md roadmap for what's next.
Evident's whole job is server-side fetching of caller-supplied URLs — treat it accordingly. The shipped code protects part of that surface and deliberately leaves the rest to your own deployment.
What the code protects. Every fetch path — fetch(), extract(), health_check(), the resilience ladder (Tier 1 and Tier 2), and the recipe fetchers — enforces a built-in SSRF guard. A URL whose host resolves to a non-public address (loopback, RFC1918 private ranges, link-local including the cloud metadata endpoint 169.254.169.254, and multicast/reserved/unspecified ranges) is refused before any connection is opened, as is any non-http(s) scheme. Hostnames are resolved and the resulting IPs are checked — not string-matched — so a DNS-rebinding name that points at an internal address is still blocked. A refused URL returns a diagnosable ssrf_blocked result rather than failing silently. See src/evident/core/ssrf.py.
What the code does not protect (by design, for now). The REST API and Docker image ship with no authentication and no rate limiting (docker-compose.yml publishes port 8000 directly). This is a deliberate scope decision for a self-hosted, single-operator tool, not an oversight — so:
- Do not expose the REST API directly to the public internet. Run it behind your own reverse proxy with auth, or keep it on localhost / a private network, if you use
evident.api.restor the Docker image. - The MCP server (stdio, single local user) has no such network exposure and is the lowest-risk way to run Evident today.
Full security posture and how to report a vulnerability: SECURITY.md. Please report security issues through GitHub's private vulnerability reporting — not a public issue or PR with exploit details.
Apache-2.0 — see LICENSE. Deliberately not AGPL, to stay commercial-use-friendly.
Evident — how sure should an agent be that a result is correct?