Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Available specs:
- `specs/initial.md` - WebFetch tool specification (types, behavior, conversions, error handling)
- `specs/fetchers.md` - Pluggable fetcher system for URL-specific handling
- `specs/maintenance.md` - Periodic maintenance checklist (deps, docs, spec-code alignment)
- `specs/threat-model.md` - Security threat model (SSRF, network, input validation, DoS)

Specification format: Abstract and Requirements sections.

Expand Down
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ AI-friendly web content fetching tool designed for LLM consumption. Rust library
- **Binary detection** - Returns metadata only for images, PDFs, etc.
- **Timeout handling** - 1s first-byte, 30s body with partial content on timeout
- **URL filtering** - Allow/block lists for controlled access
- **SSRF protection** - Resolve-then-check blocks private IPs by default
- **MCP server** - Model Context Protocol support for AI tool integration

## Installation
Expand Down Expand Up @@ -171,6 +172,26 @@ Errors are returned in the `error` field:
- `ContentError` - Failed to read body
- `BinaryContent` - Binary content not supported

## Security

FetchKit blocks connections to private/reserved IP ranges by default, preventing SSRF attacks when used in server-side or AI agent contexts.

**Blocked by default:** loopback, private networks (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x including cloud metadata), IPv6 equivalents, multicast, and other reserved ranges.

```rust
// Default: private IPs blocked (safe for production)
let tool = Tool::default();

// Explicit opt-out for local development only
let tool = Tool::builder()
.block_private_ips(false)
.build();
```

DNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canonicalized before validation.

See [`specs/threat-model.md`](specs/threat-model.md) for the full threat model.

## Configuration

### Timeouts
Expand Down
5 changes: 5 additions & 0 deletions crates/fetchkit/src/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
//! This module provides the main entry points for fetching URLs.
//! The actual fetch logic is implemented by fetchers in the [`fetchers`](crate::fetchers) module.

use crate::dns::DnsPolicy;
use crate::error::FetchError;
use crate::fetchers::FetcherRegistry;
use crate::types::{FetchRequest, FetchResponse};
Expand All @@ -20,6 +21,8 @@ pub struct FetchOptions {
pub enable_markdown: bool,
/// Enable as_text option
pub enable_text: bool,
/// DNS resolution policy for SSRF prevention
pub dns_policy: DnsPolicy,
}

/// Fetch a URL and return the response
Expand Down Expand Up @@ -92,5 +95,7 @@ mod tests {
assert!(options.block_prefixes.is_empty());
assert!(!options.enable_markdown);
assert!(!options.enable_text);
// Safe by default: private IPs blocked
assert!(options.dns_policy.block_private);
}
}
Loading