Skip to content

BeautifulSoup.md

Latest commit

 

History

History
165 lines (119 loc) · 6.53 KB

BeautifulSoup.md

File metadata and controls

165 lines (119 loc) · 6.53 KB
Raw

BeautifulSoup

A forgiving HTML/XML parser (bs4) that turns messy markup into a navigable tree — the standard tool for extracting data from web pages during OSINT and scraping.

Overview

BeautifulSoup sits on top of a parser (html.parser, lxml, or html5lib) and gives you an easy API to search a document by tag, attribute, CSS selector, or text. Paired with requests it forms the classic scraping stack: fetch with requests, extract with BeautifulSoup. In security work it is used to harvest links, emails, form fields, comments, and metadata from authorized targets and public sources.

Installation

pip install beautifulsoup4 lxml

Basic Usage

from bs4 import BeautifulSoup

html = "<html><body><a href='/admin'>Admin</a><a href='/docs'>Docs</a></body></html>"
soup = BeautifulSoup(html, "html.parser")

print(soup.find("a")["href"])                       # /admin
print([a["href"] for a in soup.find_all("a")])      # ['/admin', '/docs']
print(soup.get_text())                              # AdminDocs

The second argument selects the parser. "html.parser" is built in; "lxml" is faster but needs an extra package.

Important APIs

API Purpose
BeautifulSoup(markup, parser) Build the parse tree
soup.find(name, attrs) First matching element, or None
soup.find_all(name, attrs, limit=) All matching elements
soup.select(css_selector) CSS-selector search
soup.select_one(css_selector) First CSS-selector match
tag["attr"] / tag.get("attr") Attribute access — .get() is safe
tag.text / tag.get_text() Text content, descendants included
tag.attrs All attributes as a dict
soup.prettify() Re-indented markup, useful for debugging

Parsers: "html.parser" (built in), "lxml" (fast, needs lxml), "html5lib" (most lenient, slowest).

Example

Extract all hyperlinks from a page:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com", timeout=10).text
soup = BeautifulSoup(html, "lxml")

for a in soup.find_all("a", href=True):
    print(a["href"], "->", a.get_text(strip=True))

Output

https://www.iana.org/domains/example -> More information...

Enumerate every form and its input fields (useful for mapping attack surface):

from bs4 import BeautifulSoup

html = """
<form action="/login" method="post">
  <input name="username" type="text">
  <input name="password" type="password">
  <input name="csrf" type="hidden" value="tok123">
</form>
"""
soup = BeautifulSoup(html, "lxml")

for form in soup.find_all("form"):
    print("action:", form.get("action"), "method:", form.get("method"))
    for field in form.find_all("input"):
        print("  ", field.get("name"), "=", field.get("type"))

Output

action: /login method: post
   username = text
   password = password
   csrf = hidden

Harvest email addresses and HTML comments (often leak developer notes):

import re
from bs4 import BeautifulSoup, Comment

html = '<p>Contact admin@example.com</p><!-- TODO: remove debug endpoint /api/v0 -->'
soup = BeautifulSoup(html, "lxml")

emails = re.findall(r"[\w.+-]+@[\w-]+\.[\w.-]+", soup.get_text())
comments = soup.find_all(string=lambda t: isinstance(t, Comment))

print("emails:", emails)
print("comments:", [c.strip() for c in comments])

Output

emails: ['admin@example.com']
comments: ['TODO: remove debug endpoint /api/v0']

Security Use Cases

  • OSINT scraping — collect emails, usernames, subdomains, and social links from public web pages for footprinting.
  • Attack-surface mapping — enumerate forms, hidden fields, CSRF tokens, and input names to seed a fuzzer or login script.
  • Information leakage checks — pull HTML comments and metadata that expose internal paths, versions, or developer notes.
  • Link crawling — extract href/src targets to feed a controlled spider over an authorized application.
  • Content diffing — compare scraped snapshots to detect defacement or unexpected changes.

Common Mistakes

  • Parsing HTML with regular expressions instead — HTML is not a regular language, and this fails on any real page.
  • Indexing an attribute that may be absent — tag["href"] raises KeyError; use tag.get("href").
  • Forgetting find() returns None — chaining onto it raises AttributeError.
  • Omitting the parser argument, producing a warning and inconsistent behaviour across machines.
  • Assuming lxml is installed — it is a separate package.
  • Expecting JavaScript-rendered content — BeautifulSoup parses the HTML it is given, and runs no scripts. Use [[Selenium]] for that.
  • Building absolute URLs by string concatenation rather than urllib.parse.urljoin().

Security Considerations

[!warning] Authorized use only Parse only content you retrieved lawfully from systems you own or are authorized to test.

  • Parsed content is untrusted input. Extracted text and attributes can contain injection payloads. Never pass them unescaped into a shell command, SQL query, or another HTML page.
  • Do not follow extracted links automatically without a scope check — that turns a parser into an unbounded crawler hitting third parties.
  • html5lib and lxml parse pathological markup differently. A deeply nested or malformed document can consume significant CPU and memory; bound the size of what you parse.
  • Extracted data may be personal data. Scraped names, emails, and addresses carry legal obligations regardless of how you obtained them.
  • Respect robots.txt and rate limits on whatever fetched the HTML in the first place — see [[requests]].

Best Practices

  • Prefer the lxml parser for speed; fall back to html5lib for badly broken markup.
  • Use CSS selectors (soup.select("form input[type=hidden]")) for concise, readable queries.
  • Always pass a parser name explicitly to avoid version-dependent defaults.
  • Respect robots.txt and rate limits when scraping third-party sites; scrape only what you are authorized to.
  • Combine with requests.Session to reuse cookies across authenticated pages.

References

Related Topics

  • [[requests]] — fetch the pages you feed to BeautifulSoup
  • [[Selenium]] — for JavaScript-rendered content BeautifulSoup can't see
  • [[Readme|Python for Security Professionals]] — course home