reconspread

reconspread is a command-line tool and Python library designed to extract URLs and links from websites.
It follows redirects, crawls internal pages, and scans HTML content to identify all URLs from websites.

---

Features

• Follow HTTP redirects to get the final page URL
• Crawl internal pages to find more URLs
• Extract URLs from text content and HTML links
• Categorize URLs as internal (same domain) or external
• URL deduplication to avoid duplicate entries
• Provide a simple CLI for quick usage
• Python module for integration in your own projects
• Configurable crawling parameters (max pages, timeout, delay)
• JSON output support for structured data

---

Installation

Using virtualenv (recommended)

1. Install virtualenv if necessary:

pip install virtualenv

2. Create and activate a new virtual environment:

virtualenv reconspread-env
source reconspread-env/bin/activate

3. Install reconspread locally (run this command in the project root where setup.py is located):

pip install -r requirements.txt

---

Usage

Command-line Interface (CLI)

After installation, use the reconspread/cli.py command:

python3 reconspread/cli.py <url>

Example:

python3 reconspread/cli.py https://example.com

Sample output:

Fetching content from: https://example.com
Final URL after redirects: https://www.example.com
Extracting urls...

Found internal items:

-> https://example.com/about
-> https://example.com/contact
-> https://example.com/services

Found external items:

-> https://github.com/example
-> https://twitter.com/example

Summary: 5 urls found across 1 pages

For help and additional options:

python3 reconspread/cli.py -h

Available options:

• --max-pages: Maximum number of pages to crawl (default: 50)
• --timeout: Request timeout in seconds (default: 30)
• --delay: Delay between requests in seconds (default: 1.0)
• --verbose: Print every page being searched
• --recursive: Follow every internal link (default: only crawl the final page after redirects)
• --all-domains: Extract URLs from all domains, not just same domain (default: same domain only)
• -o, --output: Output filename to save results as JSON

Save Results to File

You can save the extracted URLs to a JSON file:

python3 reconspread/cli.py https://example.com -o results.json

This will create a JSON file with the following structure:

{
  "base_url": "https://example.com",
  "final_url": "https://www.example.com", 
  "timestamp": "2024-01-15T10:30:00.123456",
  "internal_links": [
    "https://example.com/about",
    "https://example.com/contact"
  ],
  "external_links": [
    "https://github.com/example",
    "https://twitter.com/example"
  ]
}

---

Python Module

You can also use reconspread programmatically:

Add to your requirements.txt:

# requirements.txt
git+https://github.com/reconurge/reconspread.git

Or install using python:

pip install git+https://github.com/reconurge/reconspread.git

from reconspread import Crawler

# Create a crawler instance
crawler = Crawler(
    url="https://example.com",
    max_pages=50,
    timeout=30,
    delay=1.0,
    verbose=False,
    recursive=False,
    same_domain_only=True
)

# Fetch the URL and follow redirects
crawler.fetch()
print(f"Final URL: {crawler.final_url}")

# Extract URLs
crawler.extract_urls()

# Get results
results = crawler.get_results()

# Process results
print(f"Found {len(results.internal)} internal URLs:")
for item in results.internal:
    print(f"  -> {item.url}")

print(f"Found {len(results.external)} external URLs:")
for item in results.external:
    print(f"  -> {item.url}")

# Save to file
crawler.save_to_file("my_results.json")

Or use the UrlItem dataclass directly:

from reconspread import UrlItem

# Create URL items
url_item = UrlItem(url="https://example.com/contact")
print(f"URL: {url_item.url}")

Advanced Usage

# Crawl with custom settings
crawler = Crawler(
    url="https://example.com",
    max_pages=100,           # Crawl up to 100 pages
    recursive=True,          # Follow internal links
    same_domain_only=False,  # Include external URLs
    verbose=True,            # Print progress
    delay=2.0               # 2 second delay between requests
)

crawler.fetch()
crawler.extract_urls()

# Access raw results
results = crawler.get_results()
all_urls = results.internal + results.external
print(f"Total URLs found: {len(all_urls)}")

Real-time URL Processing with Callbacks

You can process URLs as they're found using a custom callback function:

def my_url_handler(url, is_external=False):
    """Custom callback to handle URLs as they're discovered."""
    scope = "EXTERNAL" if is_external else "INTERNAL"
    print(f"[{scope}] Found: {url}")
    
    # Example: Save to database, send to API, etc.
    if is_external:
        # Handle external URLs differently
        print(f"  -> External link to: {url}")
    else:
        # Handle internal URLs
        print(f"  -> Internal page: {url}")

# Create crawler with custom callback
crawler = Crawler(
    url="https://example.com",
    recursive=True,
    verbose=False,  # Disable default verbose output
    _on_result_callback=my_url_handler  # Use our custom handler
)

crawler.fetch()
crawler.extract_urls()

# URLs are processed in real-time via the callback
# but results are still available normally
results = crawler.get_results()
print(f"\nTotal processing complete: {len(results.internal + results.external)} URLs")

This is particularly useful for:

• Real-time processing: Handle URLs immediately as they're found
• Custom filtering: Apply your own logic to each URL
• Integration: Send URLs to databases, APIs, or other systems
• Progress tracking: Monitor crawling progress in real-time

---

Dependencies

• Python 3.6+
• requests
• beautifulsoup4
• lxml

These dependencies are automatically installed via pip when you install the package.

---

Development

• Core code is in the reconspread/ package
• CLI script is in reconspread/cli.py
• Tests can be run with: python -m unittest discover tests -v

---

License

MIT License.