Skip to content

requests.md

Latest commit

 

History

History
169 lines (123 loc) · 6.6 KB

requests.md

File metadata and controls

169 lines (123 loc) · 6.6 KB
Raw

requests

The de-facto HTTP client for Python — a clean, high-level API for talking to web servers and REST APIs during testing and automation.

Overview

requests wraps the messy urllib stack behind a simple, readable interface for GET/POST/PUT/DELETE, sessions, cookies, headers, and TLS. It is the workhorse of most web-security tooling: fuzzers, API clients, login scripts, and reconnaissance helpers are almost always built on top of it. For any authorized web-application assessment, it lets you script requests exactly the way a browser or client would.

Installation

pip install requests

Basic Usage

import requests

response = requests.get("https://httpbin.org/get", timeout=10)

print(response.status_code)     # 200
print(response.headers["Content-Type"])
print(response.json())          # parsed JSON body
print(response.text[:40])       # raw body as str

Every call takes a timeout. Without one, a request can block forever.

Important APIs

API Purpose
requests.get/post/put/delete(url, ...) One-off requests
requests.Session() Connection pooling, persistent cookies and headers
response.status_code HTTP status as an int
response.text / .content Body as str / bytes
response.json() Parsed JSON; raises on invalid JSON
response.headers Case-insensitive header mapping
response.raise_for_status() Raise HTTPError on 4xx/5xx
response.history Redirect chain that led here
timeout=, allow_redirects=, verify=, proxies= Per-request controls
requests.RequestException Base class for every error this library raises

Example

Simple GET with custom headers and a timeout:

import requests

resp = requests.get(
    "https://httpbin.org/get",
    headers={"User-Agent": "recon-scanner/1.0"},
    timeout=10,
)

print(resp.status_code)
print(resp.headers.get("Server"))
print(resp.json()["headers"]["User-Agent"])

Output

200
gunicorn/19.9.0
recon-scanner/1.0

Persisting cookies and auth across requests with a Session:

import requests

session = requests.Session()
session.headers.update({"User-Agent": "authorized-test/1.0"})

# Authenticate once; the session keeps the cookie for later calls.
session.post("https://httpbin.org/cookies/set/token/abc123")
resp = session.get("https://httpbin.org/cookies")

print(resp.json())

Output

{'cookies': {'token': 'abc123'}}

Probing a list of paths on an authorized target and reporting live endpoints:

import requests

BASE = "https://httpbin.org"
paths = ["/get", "/status/404", "/status/500", "/headers"]

for path in paths:
    try:
        r = requests.get(BASE + path, timeout=5)
        print(f"{r.status_code}  {path}")
    except requests.RequestException as exc:
        print(f"ERR   {path}  ({exc})")

Output

200  /get
404  /status/404
500  /status/500
200  /headers

Security Use Cases

  • Web-application testing — script logins, replay requests, and manipulate parameters/headers to test for IDOR, auth bypass, and injection on systems you are authorized to assess.
  • API assessment — build lightweight clients that send crafted JSON bodies and inspect responses, status codes, and rate-limit headers.
  • Directory / endpoint discovery — iterate a wordlist of paths and flag 200/302/403 responses to map an application's surface.
  • OSINT & data collection — pull public pages, JSON feeds, and certificate-transparency or WHOIS APIs for passive recon.
  • Health / regression monitoring — verify that hardening changes (security headers, redirects to HTTPS) actually took effect.

Warning

Only send automated traffic to hosts you own or have written authorization to test. High request rates against third-party services can constitute abuse.

Common Mistakes

  • Omitting timeout= — the single most common mistake; a request without one can hang a scanner indefinitely.
  • Setting verify=False to silence a certificate error, which disables the protection TLS provides.
  • Catching bare Exception instead of requests.RequestException.
  • Assuming response.json() succeeds — an HTML error page raises json.JSONDecodeError.
  • Not checking the status code — requests does not raise on 404 or 500 unless you call raise_for_status().
  • Creating a new Session per request, losing connection pooling entirely.
  • Retrying a 401 or 400 — those will never succeed.
  • Logging the Authorization header, leaking the token.

Security Considerations

[!warning] Authorized use only Only send automated traffic to hosts you own or have written authorization to test. High request rates against third-party services can constitute abuse.

  • Never disable certificate verification. verify=False makes credentials and data readable to anyone who can intercept the connection. If a lab uses a private CA, pass its bundle via verify="/path/to/ca.pem".
  • Redirects can leak credentials. By default requests follows redirects; a redirect to another host can forward headers. Set allow_redirects=False when sending secrets, and inspect response.history.
  • Treat every response as untrusted input. Never eval() a body or pass it unescaped into a shell, SQL query, or HTML template.
  • Keep requests and urllib3 current — both ship security fixes.
  • Set an identifying User-Agent during an authorized test so the target's operators can attribute the traffic.
  • Rate-limit deliberately. Concurrency multiplies load; an unthrottled loop is indistinguishable from an attack.

Best Practices

  • Always set an explicit timeout= — a missing timeout can hang a scanner indefinitely.
  • Reuse a Session for many requests to the same host to benefit from connection pooling and shared cookies.
  • Catch requests.RequestException (the base class) rather than individual errors.
  • Never disable TLS verification (verify=False) against real targets; if a lab requires it, scope it narrowly and expect the InsecureRequestWarning.
  • Read resp.raise_for_status() when you want non-2xx responses to become exceptions.

References

Related Topics

  • [[BeautifulSoup]] — parse the HTML that requests fetches
  • [[Selenium]] — when a target needs a full JavaScript-rendering browser
  • [[Readme|Python for Security Professionals]] — course home