Network Surfacing

A modular Python-based network diagnostics, monitoring, and analysis toolkit designed to demonstrate networking, systems programming, cybersecurity fundamentals, data persistence, concurrency, and software engineering principles.

This project started as a simple collection of networking scripts and evolved into a lightweight network observability platform capable of collecting, storing, analyzing, and visualizing network data.

---

Installation

Clone the Repository

git clone https://github.com/shahbajsingh/network-surfacing.git

cd network-surfacing

---

Create a Virtual Environment

macOS / Linux

python3 -m venv venv

source venv/bin/activate

Windows

python -m venv venv

venv\Scripts\activate

---

Install Dependencies

pip install -r requirements.txt

or install the package directly:

pip install -e .

---

CLI Installation

The toolkit can be installed as a system command.

After running:

pip install -e .

you can execute commands directly using:

netsurf

instead of:

python main.py

---

Verify Installation

netsurf --help

Expected output:

Usage: netsurf [OPTIONS] COMMAND [ARGS]...

Commands:
  ping
  dns
  scan
  discover
  http
  api
  ssl
  banner
  capture
  pcap
  log
  compare
  graph
  graph-ports
  report

---

Quick Start

Perform a ping:

netsurf ping google.com

DNS lookup:

netsurf dns google.com

Port scan:

netsurf scan scanme.nmap.org

Analyze a website:

netsurf http https://google.com

Inspect an API:

netsurf api https://api.github.com

Generate a latency graph:

netsurf graph google.com

Generate a port history graph:

netsurf graph-ports scanme.nmap.org

Generate an HTML report:

netsurf report

---

Package Configuration

This project uses a Python entry point defined in:

pyproject.toml

Example:

[project]
name = "network-surfacing"
version = "1.0.0"

[project.scripts]
netsurf = "cli.commands:app"

This creates the executable command:

netsurf

which launches the Typer CLI application.

---

Example Workflow

Collect data:

netsurf ping google.com

netsurf dns google.com

netsurf scan scanme.nmap.org

netsurf http https://google.com

netsurf api https://api.github.com

netsurf ssl google.com

Generate visualizations:

netsurf graph google.com

netsurf graph-ports scanme.nmap.org

Generate reports:

netsurf report

Inspect logs:

tail -f data/logs/toolkit.log

Inspect database:

sqlite3 data/toolkit.db

---

Features

Network Diagnostics

Ping Hosts

Measure host reachability and collect latency metrics.

netsurf ping google.com

Capabilities:

• ICMP reachability testing
• Historical latency tracking
• Performance trend analysis
• Automatic logging
• Error tracking

---

DNS Lookup

Retrieve common DNS record types.

netsurf dns google.com

Supported Records:

• A
• AAAA
• MX
• TXT
• NS

Capabilities:

• DNS enumeration
• DNS response timing
• Historical DNS query metrics
• Error tracking

---

Port Scanning

Perform concurrent TCP port scanning.

netsurf scan scanme.nmap.org

Capabilities:

• Multi-threaded scanning
• Common service detection
• Historical port tracking
• Open/closed port monitoring
• Scan persistence

Common Ports:

Port	Service
21	FTP
22	SSH
25	SMTP
53	DNS
80	HTTP
110	POP3
143	IMAP
443	HTTPS
3306	MySQL
5432	PostgreSQL

---

Network Discovery

Perform subnet sweeps and discover active hosts.

netsurf discover 192.168.1.0/24

Capabilities:

• CIDR expansion
• Host discovery
• Concurrent execution
• Active host reporting
• Historical scan storage

---

Web Analysis

HTTP Analyzer

Analyze website responses and performance.

netsurf http https://google.com

Collected Information:

• HTTP Status Code
• Response Time
• Content Type
• Server Header
• Historical Latency Metrics

---

API Analyzer

Inspect REST APIs.

netsurf api https://api.github.com

Collected Information:

• Status Code
• Response Headers
• Content Length
• JSON Structure
• Response Timing

---

Security Features

SSL Certificate Inspection

Inspect remote TLS certificates.

netsurf ssl google.com

Collected Information:

• Certificate Expiration
• Certificate Issuer
• TLS Connectivity
• SSL Timing Metrics

---

Banner Grabbing

Retrieve service banners from listening services.

netsurf banner scanme.nmap.org 22

Common Targets:

• SSH
• SMTP
• FTP
• Telnet
• Custom TCP Services

---

Packet Capture

Capture live network traffic and save PCAP files.

Requires administrative privileges.

sudo netsurf capture 30

Example:

sudo netsurf capture 60

Output:

data/pcaps/capture_60.pcap

Capabilities:

• Live packet capture
• PCAP generation
• Traffic collection for analysis

---

PCAP Analysis

Analyze previously captured PCAP files.

netsurf pcap data/pcaps/capture_60.pcap

Capabilities:

• Packet counting
• Top talker identification
• Protocol analysis
• Traffic statistics

---

Log Analysis

Analyze server log files.

netsurf log access.log

Capabilities:

• Top IP identification
• Endpoint analysis
• Request frequency analysis

Supported Formats:

• Apache Access Logs
• Nginx Access Logs
• Similar structured logs

---

Historical Tracking

All scan results are stored in SQLite.

Database:

data/toolkit.db

Tracked Information:

• Scan Results
• Latency Measurements
• Open Port History
• Errors
• Discovery Results

---

Graphing

Latency Trends

Generate latency graphs from historical measurements.

netsurf graph google.com

Output:

data/reports/google.com_latency.png

Displays:

• Historical latency measurements
• Performance trends over time

---

Port Trends

Generate graphs of observed open ports.

netsurf graph-ports scanme.nmap.org

Output:

data/reports/scanme.nmap.org_ports.png

Displays:

• Port frequency
• Service availability trends

---

Reporting

Generate HTML reports.

netsurf report

Output:

data/reports/dashboard.html

Contains:

• Recent scans
• Error summaries
• Historical metrics
• Generated visualizations

---

Logging

All modules write to a centralized log file.

Location:

data/logs/toolkit.log

View logs:

tail -f data/logs/toolkit.log

Logged Events:

• Scan execution
• DNS lookups
• SSL checks
• HTTP requests
• Errors
• Performance metrics

---

Error Tracking

Errors are automatically stored in the database.

Examples:

• DNS failures
• Connection failures
• Timeouts
• SSL issues
• Invalid hosts

Example failure:

netsurf dns any-fake-domain-1234.com

---

Scan Comparison

Compare historical scan results.

netsurf compare scanme.nmap.org

Displays:

• Newly opened ports
• Newly closed ports
• Service changes

---

Database Inspection

Open SQLite database:

sqlite3 data/toolkit.db

List tables:

.tables

Expected Tables:

scans
errors
latency_measurements
port_history

View recent scans:

SELECT *
FROM scans
ORDER BY timestamp DESC
LIMIT 20;

View errors:

SELECT *
FROM errors;

View latency history:

SELECT *
FROM latency_measurements;

---

Project Structure

network-surfacing/
│
├── analyzers/
│   ├── api_analyzer.py
│   ├── http_analyzer.py
│   ├── log_parser.py
│   └── pcap_analyzer.py
│
├── cli/
│   └── commands.py
│
├── database/
│   ├── db.py
│   ├── models.py
│   └── schema.py
│
├── reports/
│   ├── compare.py
│   ├── graphs.py
│   ├── html_report.py
│   └── network_graph.py
│
├── scanners/
│   ├── banner_grabber.py
│   ├── dns_lookup.py
│   ├── network_discovery.py
│   ├── packet_capture.py
│   ├── ping.py
│   ├── port_scanner.py
│   └── ssl_checker.py
│
├── utils/
│   ├── formatting.py
│   ├── logger.py
│   ├── networking.py
│   └── validators.py
│
├── data/
│   ├── logs/
│   ├── pcaps/
│   └── reports/
│
├── main.py
├── requirements.txt
└── README.md

---

Technologies Used

• Python
• SQLite
• Typer
• Rich
• Requests
• dnspython
• Scapy
• Matplotlib
• NetworkX
• ThreadPoolExecutor
• Socket Programming
• TLS/SSL
• REST APIs

---

Sample Demo Workflow

Populate historical data:

netsurf ping google.com
netsurf ping google.com

netsurf dns google.com

netsurf scan scanme.nmap.org

netsurf http https://google.com

netsurf api https://api.github.com

netsurf ssl google.com

netsurf banner scanme.nmap.org 22

netsurf graph google.com

netsurf report

Review outputs:

data/logs/toolkit.log
data/toolkit.db
data/reports/google.com_latency.png
data/reports/dashboard.html