Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@ client = Wreq::Client.new(
tcp_nodelay: true, # disable Nagle algorithm (default: true)
tcp_keepalive: 15, # SO_KEEPALIVE interval in seconds (default: 15)
local_address: "1.2.3.4", # bind outgoing connections to this source IP
resolve: { # override DNS resolution per domain
"example.com" => "1.2.3.4",
"api.example.com" => ["1.2.3.4:8443", "[::1]:8443"],
},
tls_sni: true, # send SNI in TLS handshake (default: true)
min_tls_version: "tls1.2", # minimum TLS version: tls1.0, tls1.1, tls1.2, tls1.3
max_tls_version: "tls1.3", # maximum TLS version
Expand All @@ -100,6 +104,62 @@ resp = client.get("https://api.example.com/data")
resp = client.post("https://api.example.com/data", json: { key: "value" })
```

## Overriding DNS

Pass `resolve` to point specific domains at IP addresses of your choice,
bypassing DNS for them (similar to curl's `--resolve`, except that an entry
covers every port of the domain). Useful for hitting a staging host by IP while
keeping the real `Host` header and TLS SNI, or for pinning a domain to one node
of a load balancer.

```ruby
client = Wreq::Client.new(
resolve: {
"example.com" => "127.0.0.1", # one address
"api.example.com" => ["10.0.0.1", "10.0.0.2"], # tried in order
}
)

resp = client.get("https://example.com/") # connects to 127.0.0.1:443
```

Keys are bare domain names, without a scheme, port or path. They match the
URL's host regardless of case or a trailing dot, so `https://EXAMPLE.com./`
also connects to `127.0.0.1`.

Addresses are IPv4 or IPv6, with an optional port:

| Value | Connects to |
|-------|-------------|
| `"1.2.3.4"` | `1.2.3.4` on the URL's port, or the scheme default |
| `"1.2.3.4:8443"` | `1.2.3.4:8443` |
| `"::1"` / `"[::1]"` | `[::1]` on the URL's port, or the scheme default |
| `"[::1]:8443"` | `[::1]:8443` |

An IPv6 address with a port **must** be bracketed: `"::1:8080"` is itself a
valid IPv6 address, so it is parsed as one (with no port) rather than as `::1`
on port 8080.

An explicit port in the request URL always wins over a port in the override.
Domains without an entry resolve normally, including the target of a redirect
(when `redirect` is enabled).

### Proxies

`resolve` only takes effect when the client looks the domain up itself. An HTTP
or HTTPS proxy, or a `socks4a://` or `socks5h://` proxy, receives the domain
name and does its own lookup, so the entry is ignored. This applies to proxies
from the `HTTP_PROXY`, `HTTPS_PROXY` and `ALL_PROXY` environment variables too,
which the client uses unless you pass `no_proxy: true`. `socks4://` and
`socks5://` proxies get the entry's first address, on the URL's port or the
scheme default.

> **Pinning domains after a security check:** if you resolve and vet a domain
> yourself (for example to block SSRF or DNS rebinding) and then pin it with
> `resolve`, also pass `no_proxy: true` and leave redirects off (the default).
> Otherwise a proxy can do its own lookup, or a redirect can take the request
> to a domain you never checked.

## HTTP Methods

All methods are available on both `Wreq` (module-level) and `Wreq::Client` (instance-level):
Expand Down
141 changes: 121 additions & 20 deletions ext/wreq_rb/src/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ use std::ffi::c_void;
use std::panic::{self, AssertUnwindSafe};
use std::ptr;
use std::any::Any;
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;

Expand All @@ -13,7 +14,8 @@ use tokio::runtime::Runtime;
use tokio::sync::Semaphore;
use tokio::task::JoinSet;
use tokio_util::sync::CancellationToken;
use std::net::IpAddr;
use std::net::{IpAddr, SocketAddr};
use wreq::dns::{Addrs, GaiResolver, Name, Resolve, Resolving};
use wreq::header::{HeaderMap, HeaderName, HeaderValue, OrigHeaderMap};
use wreq::tls::TlsVersion;
use wreq_util::{Emulation as BrowserEmulation, Platform as EmulationPlatform, Profile as BrowserProfile};
Expand Down Expand Up @@ -393,6 +395,14 @@ impl Client {
builder = builder.local_address(addr);
}

if let Some(resolve_hash) = hash_get_hash(&opts, "resolve")? {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the client uses a proxy, it ignores resolve: and gives no error. The PR description says this option is for use after a DNS security check, so this matters.

I tested this with a local HTTP proxy. In all 3 cases (3 of 3), the proxy received GET http://wreq-resolve.invalid/ HTTP/1.1. The proxy got the host name, not the pinned IP, so the proxy does its own DNS lookup:

  • client option proxy: "http://127.0.0.1:<port>"
  • per-request proxy: (client.rs:695)
  • HTTP_PROXY set in the environment, and no no_proxy: true. wreq uses the system proxy by default (vendor/wreq/src/client.rs:286 and :484).

SOCKS5h has the same behavior (vendor/wreq/src/conn/proxy/socks.rs:190). The tests do not find this because every new test sets no_proxy: true (test/client_test.rb:160).

Please add this to the README. If you use resolve: for security, you must also set no_proxy: true. Another option is to raise an error when you get resolve: and proxy: together.

let overrides = hash_to_dns_overrides(&resolve_hash)?;
builder = builder.dns_resolver(OverrideResolver {
overrides,
fallback: GaiResolver::new(),
});
}

if let Some(v) = hash_get_bool(&opts, "tls_sni")? {
builder = builder.tls_sni(v);
}
Expand Down Expand Up @@ -802,16 +812,20 @@ fn hash_get_array(hash: &RHash, key: &str) -> Result<Option<RArray>, magnus::Err
}
}

/// Convert a String or Symbol (a hash key or header name) into a String.
fn key_to_string(val: Value) -> Result<String, magnus::Error> {
let ruby = unsafe { Ruby::get_unchecked() };
if val.is_kind_of(ruby.class_symbol()) {
val.funcall("to_s", ())
} else {
TryConvert::try_convert(val)
}
}

fn array_to_orig_header_map(ary: RArray) -> Result<OrigHeaderMap, magnus::Error> {
let mut orig = OrigHeaderMap::with_capacity(ary.len());
for elem in ary.into_iter() {
let ruby = unsafe { Ruby::get_unchecked() };
let name_str: String = if elem.is_kind_of(ruby.class_symbol()) {
elem.funcall("to_s", ())?
} else {
TryConvert::try_convert(elem)?
};
orig.insert(name_str);
orig.insert(key_to_string(elem)?);
}
Ok(orig)
}
Expand All @@ -822,12 +836,7 @@ fn hash_to_header_map(hash: &RHash) -> Result<HeaderMap, magnus::Error> {
if v.is_nil() {
return Ok(magnus::r_hash::ForEach::Continue);
}
let ruby = unsafe { Ruby::get_unchecked() };
let ks: String = if k.is_kind_of(ruby.class_symbol()) {
k.funcall("to_s", ())?
} else {
TryConvert::try_convert(k)?
};
let ks = key_to_string(k)?;
let vs: String = v.funcall("to_s", ())?;
let name =
HeaderName::from_bytes(ks.as_bytes()).map_err(|e| generic_error(e))?;
Expand All @@ -838,15 +847,107 @@ fn hash_to_header_map(hash: &RHash) -> Result<HeaderMap, magnus::Error> {
Ok(hmap)
}

/// DNS resolver behind the `resolve:` option. wreq's own overrides
/// (`resolve_to_addrs`) compare the URL host byte for byte, so "EXAMPLE.com"
/// or "example.com." in a URL would miss an "example.com" entry and silently
/// fall back to a real DNS lookup. Here both sides are normalized first.
struct OverrideResolver {
overrides: HashMap<String, Vec<SocketAddr>>,
/// wreq's default resolver for the features this crate enables.
fallback: GaiResolver,
}

impl Resolve for OverrideResolver {
fn resolve(&self, name: Name) -> Resolving {
match self.overrides.get(&normalize_host(name.as_str())) {
Some(addrs) => {
let addrs: Addrs = Box::new(addrs.clone().into_iter());
Box::pin(std::future::ready(Ok(addrs)))
}
None => self.fallback.resolve(name),
}
}
}

/// Host names are case-insensitive, and a trailing dot ("example.com.") names
/// the same host.
fn normalize_host(host: &str) -> String {
host.strip_suffix('.').unwrap_or(host).to_ascii_lowercase()
}

/// Validate and normalize a `resolve:` key. Keys that can never equal a URL
/// host, such as curl-style "example.com:443" or an IP address (which is never
/// looked up), are rejected instead of being silently ignored.
fn parse_resolve_host(key: &str) -> Result<String, magnus::Error> {
let host = normalize_host(key);
let is_domain = !host.is_empty()
&& host.bytes().all(|b| b.is_ascii_alphanumeric() || matches!(b, b'-' | b'.' | b'_'))
&& host.parse::<IpAddr>().is_err();
if !is_domain {
return Err(generic_error(format!(
"invalid resolve host: '{}'. Use a bare domain name such as 'example.com' \
(ASCII, without a scheme, port or path)", key
)));
}
Ok(host)
}

/// Parse a DNS override target into a SocketAddr. Accepts a bare IP
/// ("1.2.3.4", "::1", "[::1]") or an IP with a port ("1.2.3.4:8080",
/// "[::1]:8080"). A missing port is stored as 0, which makes wreq fall back to
/// the port in the request URL (or the scheme's default port).
///
/// An IPv6 address with a port must be bracketed: "::1:8080" is a valid IPv6
/// address in its own right and parses as one, not as "::1" on port 8080.
fn parse_resolve_address(host: &str, s: &str) -> Result<SocketAddr, magnus::Error> {
if let Ok(addr) = s.parse::<SocketAddr>() {
return Ok(addr);
}
let bare = s.strip_prefix('[').and_then(|s| s.strip_suffix(']')).unwrap_or(s);
match bare.parse::<IpAddr>() {
Ok(ip) => Ok(SocketAddr::new(ip, 0)),
Err(_) => Err(generic_error(format!(
"invalid resolve address for '{}': '{}'. Use an IP ('1.2.3.4', '::1') \
or an IP with a port ('1.2.3.4:8080', '[::1]:8080')", host, s
))),
}
}

/// Convert a `resolve:` hash of host => address(es) into DNS overrides keyed by
/// normalized host. Each value is a single address string or an array of them.
fn hash_to_dns_overrides(hash: &RHash) -> Result<HashMap<String, Vec<SocketAddr>>, magnus::Error> {
let mut overrides: HashMap<String, Vec<SocketAddr>> = HashMap::new();
hash.foreach(|k: Value, v: Value| {
let key = key_to_string(k)?;
let host = parse_resolve_host(&key)?;
let addr_strs: Vec<String> = match RArray::from_value(v) {
Some(ary) => ary.to_vec()?,
None => vec![TryConvert::try_convert(v)?],
};
if addr_strs.is_empty() {
return Err(generic_error(format!(
"resolve entry for '{}' has no addresses", key
)));
}
let addrs = addr_strs
.iter()
.map(|s| parse_resolve_address(&key, s))
.collect::<Result<Vec<_>, _>>()?;
if overrides.insert(host, addrs).is_some() {
return Err(generic_error(format!(
"duplicate resolve host: '{}' (hosts match regardless of case \
and a trailing dot)", key
)));
}
Ok(magnus::r_hash::ForEach::Continue)
})?;
Ok(overrides)
}

fn hash_to_pairs(hash: &RHash) -> Result<Vec<(String, String)>, magnus::Error> {
let mut pairs: Vec<(String, String)> = Vec::new();
hash.foreach(|k: Value, v: Value| {
let ruby = unsafe { Ruby::get_unchecked() };
let ks: String = if k.is_kind_of(ruby.class_symbol()) {
k.funcall("to_s", ())?
} else {
TryConvert::try_convert(k)?
};
let ks = key_to_string(k)?;
let vs: String = v.funcall("to_s", ())?;
pairs.push((ks, vs));
Ok(magnus::r_hash::ForEach::Continue)
Expand Down
Loading
Loading