Skip to content
janholtzhausenPublic

About

High-performance kernel-assisted reverse proxy — XDP/eBPF, io_uring, kTLS, HTTP/3

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

Vortex

A high-performance, kernel-assisted reverse proxy and TLS terminator written in C.

Vortex is built for extreme throughput on Linux using modern kernel interfaces: XDP/eBPF for sub-microsecond packet filtering, io_uring for async I/O, kTLS for kernel-offloaded TLS crypto, and HTTP/3 via QUIC. It targets bare-metal deployments where latency and throughput matter more than operational simplicity.


Feature Summary

Core Proxy

  • io_uring async I/O — multishot accept, zero-syscall-per-packet data path; re-arms automatically on termination
  • Per-CPU worker threads — one io_uring ring per thread, auto-scales to CPU count (max 64)
  • SNI-based routing — TLS Server Name Indication routes connections to named backends before the full handshake completes
  • Wildcard hostname matching — planned but not yet implemented; current matching is exact (case-insensitive)
  • Load balancing — round-robin, weighted round-robin (honors per-backend weight), least-connections, IP-hash
  • HTTP/1.1 keep-alive — backends with pool_size > 0 are kept alive and returned to the shared origin pool after each complete response; non-pooled backends are forced closed
  • HTTPS backend TLS offload — origin SSL_connect runs in the shared TLS pool instead of blocking the io_uring worker thread
  • HTTPS backend session reuse — repeated TLS-origin connections reuse cached client TLS sessions per worker/backend when the origin supports resumption
  • HTTPS backend pooling — pooled https:// origins reuse both the TCP socket and live SSL* session across client connections when the origin keeps the connection alive
  • Optional compression offload — gzip/brotli response compression can run in a dedicated thread pool instead of blocking the io_uring worker thread
  • WebSocket passthrough — detects Upgrade: websocket and 101 Switching Protocols; switches to paired io_uring recv/send chains for full-duplex streaming
  • TCP tunnel mode — TLS-terminated raw TCP passthrough; after SNI routing, bytes are forwarded verbatim to the backend with no HTTP parsing. Used for SSH-over-TLS (protocol: tcp in route config)
  • HTTP/2 and HTTP/3 — HTTP/3 via ngtcp2 + nghttp3 (conditional compile); Alt-Svc: h3=":443" injected into HTTP/1.x responses to advertise QUIC

TLS (picotls + minicrypto)

  • TLS 1.2 and 1.3, configurable minimum/maximum versions and cipher suites
  • kTLS offload — after handshake, kernel handles symmetric crypto; io_uring operates on plaintext via the socket directly
  • SNI-multiplexed certificates — one listener, unlimited virtual hosts
  • Session tickets with configurable timeout and rotation interval
  • Per-route certificate providers: static files, ACME HTTP-01, ACME DNS-01

Certificate Management (ACME)

  • Automatic certificate issuance and renewal via Let's Encrypt (or any ACME v2 CA)
  • HTTP-01 challenge — built-in challenge server on port 80
  • DNS-01 challenge — Cloudflare DNS provider supported; pluggable dns_provider interface for others
  • Renewal background thread checks every hour; renews N days before expiry (default 30)
  • Hot-reload of static certs on SIGHUP without restart

XDP/eBPF Acceleration

  • XDP program attached in native mode (falls back to SKB/generic mode automatically)
  • IP blocklist — kernel-space drop before packet reaches userspace; loaded from file on startup and SIGHUP reload
  • Per-IP token-bucket rate limiting — applied to new connection SYNs; configurable RPS and burst; LRU map auto-evicts stale entries
  • Port-scoped protection — conntrack, rate limiting, and tarpit apply only to configured protected_ports (default: 80, 443). All other TCP traffic and all non-TCP traffic (SSH, ICMPv6, UDP, DNS, etc.) passes through XDP untouched, so server-initiated connections and management access are never affected.
  • Stateful L4 TCP connection tracking — fully stateful state machine in XDP (protected ports only):
    • SYN → creates CT_SYN_SENT entry via BPF_NOEXIST (SYN flood cannot overwrite ESTABLISHED entries)
    • ACK (client completing handshake) → advances to CT_ESTABLISHED
    • FIN → CT_FIN_WAIT, then CT_CLOSING
    • RST → deletes CT entry, always passes
    • Non-SYN with no CT entry → XDP_DROP (drops ACK scans, spoofed packets, mid-stream injection attempts)
    • Idle timeouts: 30 s (SYN), 120 s (ESTABLISHED), 30 s (FIN)
    • Map capacity: 512 K connections (LRU_HASH, auto-evicts)
    • TCP doff field validated before conntrack; invalid offsets passed to kernel without CT processing
  • BPF maps pinned under /sys/fs/bpf/vortex for external inspection
  • Per-CPU metrics: rx_packets, rx_bytes, passed, dropped_ratelimit, dropped_blocklist, dropped_invalid, dropped_conntrack

Tarpit (Honeypot)

  • Connections with unrecognised SNI are tarpitted instead of rejected
  • TCP receive window clamped to 1 byte — scanner stalls attempting to send
  • 256 bytes of /dev/urandom noise sent immediately — confuses protocol fingerprinting
  • Periodic drip of additional noise (every 1 s) keeps the connection alive
  • When the tarpit FIFO is full (configurable), the oldest IP is automatically added to the XDP blocklist for WORKER_BLOCK_TTL_SECS (60 minutes), then removed when the TTL expires
  • All tarpitted IPs logged to /var/log/vortex/tarpit.log with timestamps

Response Caching

  • Per-worker in-memory cache (RAM slab, optionally disk-backed via file-backed mmap)
  • 64-byte cache index entries, one per cache line; hugepage-backed index for TLB efficiency
  • LRU eviction; per-URL TTL selection based on path/extension patterns
  • ETag support — body fingerprint stored as ETag; default xxhash64, optional SHA-256-derived 64-bit value via cache.etag_sha256; If-None-Match → 304 Not Modified
  • Optional CRC32C cache integrity verification via cache.verify_crc; corrupted RAM/disk-slab entries are invalidated and treated as misses
  • X-Cache: HIT header injected on cache hits
  • URL-pattern-driven Cache-Control rewrite:
    • Static assets (images, fonts, bundles): public, max-age=N, immutable
    • Dynamic content: public, max-age=60 injected if backend sent nothing
    • API endpoints (/api/*): Cache-Control left untouched
    • Pragma: no-cache stripped from responses for static assets
  • Full one-buffer responses are cached immediately; chunked responses use a bounded reassembly path and are cached only after full decode
  • RAM slab defaults to 30% of system RAM (floor 64 MB, cap 4 GB); optional disk slab

HTTP Header Rewrites

  • Inbound (client → backend):
    • X-Real-IP and X-Forwarded-For injected from actual client socket address
    • X-Api-Key injected per route (if configured)
    • Authorization header handling controlled per-route by backend_auth (default: strip before forwarding; see Configuration Reference)
    • Custom per-header rules (strip or inject) for any backend-bound header via backend_headers
    • Connection rewritten to close for HTTP, passed through for WebSocket upgrades
  • Outbound (backend → client):
    • Server header replaced globally (configurable; per-route server_header overrides global; "none" = pass through)
    • Strict-Transport-Security injected if absent (max-age=31536000; includeSubDomains)
    • Connection rewritten to keep-alive
    • Alt-Svc: h3=":443"; ma=86400 injected when HTTP/3 is compiled in
    • Custom per-header rules (strip or inject/replace any response header) via response_headers

HTTP Basic Auth

  • Per-route hashed verifier list, either inline under users: or loaded from auth.file
  • Verifier format: username:$scrypt$ln=...,r=...,p=...$salt_b64$hash_b64
  • Startup fails if auth.enabled: true is set without at least one verifier
  • 401 response with WWW-Authenticate: Basic realm="vortex"
  • Authorization header stripped before forwarding to backend

Generate verifiers with:

python3 tools/vortex-passwd.py admin

Observability

  • Prometheus metrics on a separate HTTP server (default 127.0.0.1:9090/metrics)
    • Per-worker: accepted connections, completed, errors, TLS version counts, kTLS count, bytes in/out
    • Cache: hits, misses, stores, evictions
    • XDP: all drop/pass counters aggregated from per-CPU BPF array
    • TLS cert expiry timestamps per hostname (vortex_cert_expiry_seconds)
  • Live dashboard on a separate HTTP server (default 127.0.0.1:9091) with a WebSocket-fed status view for routes, cache, TLS, tarpit, and XDP state
  • Structured JSON logging (or plain text via config) — all log events are key=value pairs
  • Log levels: debug, info, warn, error
  • XDP metrics logged every second at debug level in the main loop

Operations

  • Foreground by default; systemd should supervise the process directly
  • -d enables background daemonisation for non-systemd use
  • PID file written only in daemon mode to /run/vortex.pid (configurable); created with O_EXCL|O_NOFOLLOW to prevent symlink attacks
  • SIGHUP — hot-reload config, refresh XDP blocklist and rate-limit config, re-read static certs from disk; route/backend topology changes are rejected and require a restart
  • SIGTERM / SIGINT — graceful shutdown: stops workers, closes connections, detaches XDP, removes PID file
  • -t — test config and exit
  • -v — force debug logging
  • -X — disable XDP (run without BPF acceleration)
  • -T — disable TLS (plain HTTP only)
  • -b <path> — path to BPF object file
  • Environment variable expansion in config values: ${VAR} and $VAR

Architecture

                              ┌──────────────────────────────────┐
NIC ──► XDP program ──────►  │  blocklist drop                  │
        (kernel space)        │  conntrack state machine         │
                              │  SYN rate limiting               │
                              └────────────┬─────────────────────┘
                                           │ XDP_PASS
                                           ▼
                              ┌──────────────────────────────────┐
                              │  Listening socket (SO_REUSEPORT) │
                              └─────┬──────────────────────┬─────┘
                                    │                      │
                          ┌─────────▼──────┐    ┌──────────▼─────┐
                          │  Worker 0      │    │  Worker N      │   (one per CPU)
                          │  io_uring ring │    │  io_uring ring │
                          │  ┌───────────┐ │    │                │
                          │  │multishot  │ │    │                │
                          │  │accept     │ │    │                │
                          │  └─────┬─────┘ │    │                │
                          │        │        │    │                │
                          │  ┌─────▼─────┐ │    │                │
                          │  │TLS accept │ │    │                │
                          │  │(SNI peek) │ │    │                │
                          │  └─────┬─────┘ │    │                │
                          │        │ tarpit?│    │                │
                          │  ┌─────▼─────┐ │    │                │
                          │  │route/LB   │ │    │                │
                          │  └─────┬─────┘ │    │                │
                          │        │ HTTP or TCP tunnel           │
                          │  ┌─────▼──────────────────────────┐ │
                          │  │ io_uring data path             │ │
                          │  │  RECV_CLIENT → SEND_BACKEND    │ │
                          │  │  RECV_BACKEND → SEND_CLIENT    │ │
                          │  │  cache lookup / store          │ │
                          │  │  header rewrite                │ │
                          │  └────────────────────────────────┘ │
                          └────────────────────────────────────┘

Separate threads:
  - Metrics HTTP server (Prometheus scrape endpoint)
  - ACME renewal thread (checks every hour)
  - Shared TLS handshake pool
  - Shared compression offload pool
  - QUIC server thread (HTTP/3, when compiled)

Connection State Machine (XDP)

Applies to protected_ports only (default: 80, 443). All other ports pass through untracked.

               SYN (rate-limit OK, BPF_NOEXIST)
[no entry] ─────────────────────► CT_SYN_SENT
                                       │
                              ACK from client
                                       │
                                       ▼
                                CT_ESTABLISHED ◄──── data packets (ACK)
                                       │
                                  FIN seen
                                       │
                                       ▼
                                 CT_FIN_WAIT
                                       │
                               second FIN seen
                                       │
                                       ▼
                                  CT_CLOSING
                                  (LRU evicts)

RST at any state → delete entry, XDP_PASS
Non-SYN with no entry → XDP_DROP (dropped_conntrack++)
Idle timeout exceeded → delete entry, XDP_DROP (dropped_conntrack++)
SYN on existing ESTABLISHED entry → BPF_NOEXIST fails; existing state preserved

io_uring Connection State Machine (userspace)

ACCEPT ──► [TLS handshake] ──► RECV_CLIENT
                                    │
                           (TCP tunnel?) ──► RECV_CLIENT_WS ⇄ RECV_BACKEND_WS (raw relay)
                                    │
                           (cache hit?) ──► SEND_CLIENT ──► RECV_CLIENT
                                    │
                                    ▼
                              SEND_BACKEND
                                    │
                                    ▼
                              RECV_BACKEND ──► SEND_CLIENT
                                    │                │
                              (streaming?)          done
                                    │
                              RECV_BACKEND (next chunk)

Build

Prerequisites

Dependency Package Notes
CMake 3.20+ cmake
clang clang BPF compiler; also needed for -DVORTEX_FUZZ=ON
libbpf libbpf-dev eBPF/XDP
liburing liburing-dev io_uring
libyaml libyaml-dev config parsing
bpftool linux-tools-common vmlinux.h generation
picotls build from source TLS termination (optional but recommended)
micro-ecc bundled in picotls ECDH/ECDSA primitives
ngtcp2 1.16 build from source HTTP/3 (optional)
nghttp3 1.8 build from source HTTP/3 (optional)

Pass paths via CMake variables; none are assumed to be at a fixed system location:

cmake -B build \
  -DPICOTLS_SRC_DIR=/path/to/picotls \
  -DPICOTLS_BUILD_DIR=/path/to/picotls/build \
  -DNGTCP2_BUILD_DIR=/path/to/ngtcp2-picotls-build \
  -DNGHTTP3_BUILD_DIR=/path/to/nghttp3/build \
  -DNGTCP2_SRC_DIR=/path/to/ngtcp2-1.16.0 \
  -DNGHTTP3_SRC_DIR=/path/to/nghttp3-1.8.0

Debug build

cmake -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build -j$(nproc)

Debug builds enable AddressSanitizer and UBSan.

Release build

cmake -B build-release -DCMAKE_BUILD_TYPE=Release
cmake --build build-release -j$(nproc)

Release builds enable -fstack-protector-strong and -D_FORTIFY_SOURCE=2. -Werror=format-security is enforced across all build types.

Run tests

ctest --test-dir build --output-on-failure --parallel "$(nproc)"

Code quality

.clang-format is checked into the repo (LLVM style, indent=4, column=100). Enforce before committing:

clang-format --dry-run --Werror $(git ls-files '*.c' '*.h' | grep -v 'bpf/vmlinux.h')

Auto-fix in place:

clang-format -i $(git ls-files '*.c' '*.h' | grep -v 'bpf/vmlinux.h')

Use ccache for faster iterative builds:

cmake -B build -DCMAKE_C_COMPILER_LAUNCHER=ccache

Fuzzer (clang only)

cmake -B build-fuzz -DVORTEX_FUZZ=ON -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_C_COMPILER=clang
cmake --build build-fuzz --target fuzz_http_parser
./build-fuzz/fuzz/fuzz_http_parser fuzz/corpus_seeds.txt -max_len=16384

Covers the HTTP/1.1 request-line parser under libFuzzer with ASan and UBSan enabled.

Build outputs

File Description
build/vortex Debug executable
build-release/vortex Release executable
build-release/vortex_xdp.bpf.o Compiled BPF/XDP object (load with -b)

Deployment

Building a .deb package

tools/build_deb.sh produces a self-contained Debian package. picotls is statically linked into the binary — no TLS shared libraries are bundled or required on the target. If no version is passed, the script picks the next patch version after the highest version it can find from VERSION, git tags, and local .deb artifacts, and refuses version regressions.

bash tools/build_deb.sh 1.0.0
# produces vortex_1.0.0_amd64.deb

Installing on a fresh machine

# Install runtime dependencies (all available in standard Debian/Ubuntu repos)
sudo apt-get install -y libbpf1 liburing2 libyaml-0-2

# Install the package
sudo dpkg -i vortex_1.0.0_amd64.deb

The postinst script will:

  • Create /var/log/vortex and /etc/vortex
  • Copy vortex.example.yaml to /etc/vortex/vortex.yaml if no config exists yet
  • Run ldconfig and systemctl daemon-reload

Edit /etc/vortex/vortex.yaml, then:

sudo systemctl enable --now vortex

Upgrading

sudo dpkg -i vortex_<new-version>_amd64.deb
sudo systemctl restart vortex

dpkg -i over an existing install is safe — it will not overwrite /etc/vortex/vortex.yaml if it already exists.


Running

# Foreground, debug logging, custom config and BPF object
sudo ./build-release/vortex -v \
  -c /etc/vortex/vortex.yaml \
  -b ./build-release/vortex_xdp.bpf.o

# Legacy daemon mode outside systemd
sudo ./build-release/vortex -d \
  -c /etc/vortex/vortex.yaml \
  -b ./build-release/vortex_xdp.bpf.o

# Test config validity
./build-release/vortex -t -c /etc/vortex/vortex.yaml

# Reload config (blocklist, rate limits, static certs)
sudo systemctl reload vortex

# Graceful shutdown
sudo systemctl stop vortex

Root is required at startup to attach XDP programs, pin BPF maps, and bind to privileged ports. Set run_as_user in config to drop privileges immediately after initialisation. Running without -b or without a network interface configured skips XDP silently. -X disables XDP explicitly. -T disables TLS.


Configuration Reference

global:
  workers: auto             # Number of worker threads (default: auto = nproc)
  compress_pool_threads: 0  # 0 = synchronous compression in worker; 2-4 = offload pool
  ipv4_only: true           # false = AF_INET6 dual-stack with IPV6_V6ONLY=0
  bind_address: "0.0.0.0"
  bind_port: 443
  http_port: 80             # Used by ACME HTTP-01 challenge server
  interface: "eth0"         # Network interface for XDP attachment
  log_level: "info"         # debug | info | warn | error
  log_format: "json"        # json | text
  pid_file: "/run/vortex.pid"
  run_as_user: ""           # Drop privileges to this user after socket/BPF init.
                            # Empty = run as current user (warn if root).
                            # Example: "vortex"
  sqpoll: false             # Enable io_uring SQPOLL: kernel thread polls SQ, eliminates
                            # submit syscall on the data path. Requires CAP_SYS_ADMIN.
  hugepages: false          # Use 2 MB huge pages for connection buffer slabs.
                            # Requires vm.nr_hugepages to be set on the host.
  server_header: ""         # Global Server header sent to clients. Default value is
                            # a spoofed Apache/OpenVMS string. Set "none" to pass
                            # backend Server header through.
  congestion_control: ""    # TCP congestion control for backend connections.
                            # Empty = kernel default (cubic). e.g. "bbr".
                            # Check /proc/sys/net/ipv4/tcp_allowed_congestion_control.
  max_request_header_bytes: 32768  # Reject HTTP/1.1 requests whose header block
                                   # exceeds this size (default 32 KB).
  max_request_body_mb: 8    # Reject buffered request bodies larger than this (MB).
                            # Also configurable as max_request_body_bytes for byte precision.

tls:
  min_version: "1.2"        # "1.2" or "1.3"
  max_version: "1.3"
  ciphersuites: "TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384"
  session_timeout: 3600           # TLS session cache lifetime (seconds)
  session_ticket_rotation: 3600   # Rotate session ticket keys every N seconds
  ktls: true                # Enable kernel TLS offload (requires kernel 4.13+)

xdp:
  mode: "auto"              # auto | native | skb
  blocklist_file: "/etc/vortex/blocklist.txt"  # One IPv4 or IPv6 literal per line
  protected_ports:          # TCP ports XDP applies conntrack/rate-limit/tarpit to.
    - 80                    # Default: 80 and 443. All other ports pass through
    - 443                   # untracked — required so server-initiated connections
                            # (SSH, apt, backend health checks) are not blocked.
  rate_limit:
    enabled: true
    requests_per_second: 1000   # New connections per second per source IP
    burst: 2000                 # Initial token bucket fill (allows initial burst)

cache:
  enabled: true
  etag_sha256: false        # Use SHA-256-derived 64-bit ETags instead of xxhash64
  verify_crc: false         # Verify cached headers+body with CRC32C on fetch
  index_entries: 16384      # Number of URL slots (power of 2 recommended)
  slab_size_mb: 64          # RAM slab size; default = 30% of system RAM
  default_ttl: 300          # Default TTL in seconds
  use_hugepages: true       # Use 2 MB hugepages for the index
  disk_cache_path: ""       # Path for disk-backed slab (empty = RAM only)
  disk_slab_size_mb: 0      # 0 = auto (50% of free space)

acme:
  enabled: false
  email: "admin@example.com"
  directory_url: "https://acme-v02.api.letsencrypt.org/directory"
  account_key_path: "/etc/vortex/acme-account.key"
  storage_path: "/etc/vortex/certs/"
  renewal_days_before_expiry: 30
  preferred_challenge: "http-01"   # http-01 | dns-01
  dns_provider: "cloudflare"       # Required for dns-01
  dns_provider_config:
    api_token: "${CF_API_TOKEN}"   # Env-var expansion; zeroed from memory after init

metrics:
  enabled: true
  bind_address: "127.0.0.1"
  port: 9090
  path: "/metrics"

dashboard:
  enabled: false
  bind_address: "127.0.0.1"
  port: 9091                # WebSocket-fed live status view at http://<bind>:<port>/

routes:
  # HTTP reverse proxy route
  - hostname: "api.example.com"
    load_balancing: "round_robin"  # round_robin | weighted_round_robin | least_conn | ip_hash
    cert_provider: "static_file"   # static_file | acme_http01 | acme_dns01
    cert_path: "/etc/ssl/api.crt"
    key_path:  "/etc/ssl/api.key"
    x_api_key: ""                  # Injected as X-Api-Key to backend if set
    backend_timeout_ms: 30000      # Max ms to wait for first byte from backend
    backend_auth: "block"          # block (default) | passthrough | rewrite
                                   # block: strip Authorization before forwarding
                                   # passthrough: forward Authorization unchanged
                                   # rewrite: replace with backend_credentials
    backend_credentials: ""        # "user:pass" injected as Authorization: Basic <b64>
                                   # (implies backend_auth: rewrite when set)
    backend_headers:               # Per-header rules for backend-bound requests
      - name: "X-Internal-Token"   # BLOCK: strip this header before forwarding
        action: "block"
      - name: "X-Api-Version"      # SET: inject/replace header with fixed value
        action: "set"
        value: "v2"
    server_header: ""              # Per-route Server header sent to clients.
                                   # Empty = use global server_header.
                                   # "none" = pass backend's Server header through.
    response_headers:              # Per-header rules for responses sent to clients.
                                   # Applied after all built-in rewrites.
      - name: "X-Powered-By"       # BLOCK: strip this header from every response
        action: "block"
      - name: "X-Frame-Options"    # SET: inject/replace with fixed value
        action: "set"
        value: "DENY"
    congestion_control: ""         # TCP congestion control for backend connections.
                                   # Empty = kernel default (cubic). Overrides global.
                                   # e.g. "bbr" (check /proc/sys/net/ipv4/tcp_allowed_congestion_control)
    health:
      fail_threshold: 3            # Consecutive connect failures to open circuit breaker (0 = default 3)
      open_ms: 10000               # ms to keep circuit open before retry probe (0 = default 10000)
    backends:
      - address: "10.0.0.1:8080"
        weight: 3
        pool_size: 0
      - address: "https://10.0.0.3:8443"   # HTTPS backend — TLS terminated at origin
        weight: 1
        verify_peer: true                   # Verify origin cert chain and hostname (default: true for https://)
        insecure_skip_verify: false         # Set true to skip cert verification (not recommended)
    cache:
      enabled: true
      ttl: 60
    auth:
      enabled: false
      file: "/etc/vortex/auth/api-users.auth"
      users:
        - "alice:$scrypt$ln=15,r=8,p=1$AQIDBAUGBwgJCgsMDQ4PEA==$1b98GT+mgzwTa+tOSyi5AORw9EiZ0XG3ILt27KXWTzc="
    rate_limit:
      enabled: false
      rps: 100
      burst: 200

  # TCP tunnel route — TLS terminated at proxy, raw bytes forwarded to backend.
  # Use for SSH-over-TLS or any non-HTTP protocol that should travel over port 443.
  - hostname: "shell.example.com"
    protocol: "tcp"
    cert_provider: "static_file"
    cert_path: "/etc/ssl/shell.crt"
    key_path:  "/etc/ssl/shell.key"
    backends:
      - address: "192.168.1.10:22"

Connect via SSH over TLS:

ssh -o 'ProxyCommand=openssl s_client -connect shell.example.com:443 -quiet' user@shell.example.com

For dual-stack ingress, set:

global:
  ipv4_only: false
  bind_address: "::"

Environment variable expansion

Config values support ${VAR} and $VAR substitution at load time:

  api_token: "${CF_API_TOKEN}"
  cert_path: "${CERT_DIR}/fullchain.pem"

Prometheus Metrics

All metrics are exposed at http://127.0.0.1:9090/metrics (configurable).

Metric Description
vortex_connections_accepted_total Total accepted connections per worker
vortex_connections_completed_total Total completed connections per worker
vortex_connections_errors_total Total errored connections per worker
vortex_bytes_in_total Bytes received from clients
vortex_bytes_out_total Bytes sent to clients
vortex_tls12_total TLS 1.2 handshakes
vortex_tls13_total TLS 1.3 handshakes
vortex_ktls_total Connections using kTLS offload
vortex_cache_hits_total Cache hits
vortex_cache_misses_total Cache misses
vortex_cache_stores_total Cache entries written
vortex_cache_evictions_total Cache entries evicted (LRU)
vortex_tls_pool_queue_depth Pending TLS handshakes in the shared handshake queue
vortex_tls_pool_active_handshakes TLS handshakes currently executing
vortex_tls_pool_submitted_total Handshakes submitted to the TLS pool
vortex_tls_pool_completed_total Successful TLS handshakes completed by the pool
vortex_tls_pool_failed_total TLS handshakes that failed in the pool
vortex_tls_pool_dropped_total Handshakes dropped because the TLS pool queue was full
vortex_compress_pool_queue_depth Pending compression jobs in the shared compression queue
vortex_compress_pool_active_jobs Compression jobs currently executing
vortex_compress_pool_submitted_total Compression jobs submitted to the compression pool
vortex_compress_pool_completed_total Compression jobs completed successfully
vortex_compress_pool_failed_total Compression jobs that failed or were not beneficial
vortex_compress_pool_dropped_total Compression jobs dropped because the pool queue was full
vortex_xdp_rx_packets_total Packets seen by XDP
vortex_xdp_rx_bytes_total Bytes seen by XDP
vortex_xdp_passed_total Packets passed to kernel stack
vortex_xdp_dropped_ratelimit_total SYNs dropped by rate limiter
vortex_xdp_dropped_blocklist_total Packets dropped by IP blocklist
vortex_xdp_dropped_conntrack_total Packets dropped by conntrack (invalid state)
vortex_xdp_dropped_invalid_total Malformed/unparseable frames
vortex_cert_expiry_seconds TLS cert not-after timestamp (Unix) per hostname

Security

XDP Layer (kernel space)

  • IP blocklist — drop packets from known-bad IPs before they consume any CPU
  • Connection-rate limiting — token bucket per source IP, applied to SYN packets only
  • Stateful TCP conntrack — non-SYN packets with no matching state are dropped; prevents ACK scans, spoofed mid-stream injection, and half-open attacks
  • Conntrack timeouts — stale entries expire (30 s SYN, 120 s idle, 30 s FIN)
  • SYN flood protection — BPF_NOEXIST on CT insert prevents a SYN flood from evicting ESTABLISHED entries from the LRU map
  • Port-scoped enforcement — conntrack, rate limiting, and tarpit apply only to protected_ports (default 80, 443); all other TCP and all non-TCP traffic passes through unmodified
  • TCP doff validation — packets with a data-offset that points outside the packet are passed to the kernel without CT tracking rather than processed

Application Layer

  • Tarpit — unrecognised SNI connections are held open with a 1-byte TCP window; /dev/urandom noise confuses scanners; persistent offenders escalate to the XDP blocklist automatically
  • Blocklist TTL — IPs added by tarpit escalation expire after 60 minutes; removed cleanly on shutdown
  • Request header limit — HTTP/1.1 requests rejected before routing if the header block reaches max_request_header_bytes (default 32 KB) without \r\n\r\n
  • URI length enforcement — request lines with a URI exceeding the parser's output buffer are rejected with 400 (not silently truncated)
  • Ambiguous framing rejection — requests with both Transfer-Encoding and Content-Length are rejected (HTTP request smuggling defence)
  • Authorization header control — per-route backend_auth (block/passthrough/rewrite) controls whether the incoming Authorization header is stripped, forwarded unchanged, or replaced with backend-specific credentials; default is strip
  • Server header masquerade — backend identity hidden behind CSWS/2.4.62 OpenVMS/V9.2-2 (Alpha) to mislead automated vulnerability scanners
  • HTTP Basic Auth — per-route credential enforcement at the proxy level
  • SNI pre-check — TLS ClientHello is peeked before the full handshake; unrecognised SNI → tarpit without handshake cost

Privilege Model

  • Vortex starts as root to attach XDP and bind to privileged ports
  • Set run_as_user to drop to an unprivileged user immediately after initialisation; setgroups/setgid/setuid are called in that order
  • PR_SET_NO_NEW_PRIVS is set after the drop to prevent re-escalation via exec
  • A warning is logged at startup if running as root with no run_as_user configured

Secrets Handling

  • ACME/DNS API tokens are zeroed with explicit_bzero from the config struct immediately after being passed to the DNS provider's init function
  • X-Api-Key values are stored in the config struct and never written to logs
  • PID file is created with O_EXCL|O_NOFOLLOW — pre-existing symlinks are rejected; a stale regular file owned by root is removed and retried

Build Hardening

  • Release builds: -fstack-protector-strong, -D_FORTIFY_SOURCE=2, -O3 -flto
  • All build types: -Werror=format-security (format-string bugs are hard compile errors)
  • Linker: -pie -Wl,-z,relro,-z,now — PIE executable, full RELRO (read-only GOT at startup)
  • Debug builds: AddressSanitizer + UBSan
  • libFuzzer target for HTTP/1.1 parser available via -DVORTEX_FUZZ=ON

Implementation Notes

BPF maps

All maps pinned under /sys/fs/bpf/vortex/:

Map Type Key Value Purpose
rate_limit_map LRU_HASH (64K) src IPv4 token bucket entry SYN rate limiting
rate_limit_map_v6 LRU_HASH (64K) src IPv6 token bucket entry SYN rate limiting
blocklist_map HASH (10K) src IPv4 u8 flag IP blocklist
blocklist_map_v6 HASH (10K) src IPv6 u8 flag IP blocklist
metrics_map PERCPU_ARRAY (1) 0 vortex_metrics per-CPU counters
rate_config_map ARRAY (1) 0 rate_config global rate limit config
conn_track_map LRU_HASH (512K) conn_tuple (5-tuple) conn_state TCP state machine
conn_track_map_v6 LRU_HASH (512K) conn_tuple_v6 (5-tuple) conn_state TCP state machine

kTLS fast path

When picotls completes the handshake and negotiates a kTLS-compatible cipher (AES-128-GCM or AES-256-GCM):

  1. Symmetric session keys are extracted from the picotls context and installed into the kernel via setsockopt(SOL_TLS, TLS_TX/RX)
  2. The ptls_t * context is freed; io_uring operates directly on the fd
  3. Normal send/recv on the socket trigger kernel-side AES-GCM — no userspace crypto on the data path
  4. CONN_FLAG_KTLS_TX and CONN_FLAG_KTLS_RX flags track which directions are offloaded
  5. EIO on a kTLS recv is treated as close_notify / TLS alert (normal close path)
  6. begin_splice and send_zc are gated on !CONN_FLAG_KTLS_TX — splice and zero-copy send are incompatible with kTLS TX on kernel 6.8

When kTLS is not available (negotiated cipher not supported, older kernel), the ptls_t * context is kept and ptls_send/ptls_receive handle TLS record encryption in userspace.

Cache URL patterns

cache_ttl_for_url() selects TTL by URL path/extension:

  • /api/*, /auth/*, /login* → TTL 0 (do not cache)
  • *.html, *.json → TTL 60 s
  • *.css, *.js, *.woff2, *.png, *.jpg, *.svg, *.ico → TTL 86400 s (1 day), immutable
  • Everything else → default TTL from config (default 300 s)

Known Limitations

  • Load balancer — least_conn uses process-wide active backend counters and is approximate across workers/threads rather than globally serialized
  • Backend TLS (https:// origins) offloads SSL_connect to the shared TLS pool, but SSL_write/SSL_read still run synchronously in the worker thread after the handshake completes. A slow or unresponsive HTTPS backend during request or response I/O can stall all connections on that worker until the operation completes or times out. Set backend_timeout_ms to a low value (e.g. 5000) for TLS backend routes to limit the blast radius.
  • Backend TLS verification is enabled by default for https:// origins. Per backend, set verify_peer: false or insecure_skip_verify: true to disable certificate-chain and hostname checks for that origin.
  • Single-segment caching — full one-buffer responses are cached immediately; chunked responses use a bounded reassembly path and are cached only after full decode
  • WebSocket relay uses one in-flight io_uring recv/send chain per direction and relies on socket backpressure rather than deep proxy-side queues
  • Config reload rejects route/backend topology changes on SIGHUP; changing route order, backend order, or backend counts still requires a restart
  • Wildcard hostname matching — not yet implemented; routing is exact and case-insensitive

Development

Code layout

src/        Core proxy engine
  main.c            Entry point, signal handling, lifecycle, privilege drop
  worker.c          Worker init/teardown and io_uring event loop
  worker_accept.c   Accept path, SNI pre-check, tarpit
  worker_backend.c  Backend connect and HTTPS origin handling
  worker_cache.c    Cache hit/store helpers
  worker_compress.c Response compression helpers
  worker_h2.c       HTTP/2 frontend path
  worker_proxy.c    HTTP/1.x proxy state machine, header rewrites, TCP tunnel relay
  conn.c            Connection pool (hot/cold split, cache-line aligned)
  router.c          SNI route lookup, backend selection (LB algorithms)
  tls.c             picotls TLS context, accept, kTLS install, cert rotation
  tls_pool.c        Shared TLS handshake offload pool
  compress_pool.c   Shared compression offload pool
  pool.c            Shared backend keep-alive pool
  cache.c           Response cache (RAM+disk slab, LRU index, ETag/CRC support)
  auth.c            HTTP Basic Auth
  metrics.c         Prometheus HTTP server
  dashboard.c       Live dashboard HTTP/WebSocket server
  log.c             Structured JSON/text logger
  config.c          YAML config parser with env-var expansion
  bpf_loader.c      libbpf wrapper: load, attach, map accessors
  uring.c           io_uring helpers (init, submit, timeout)
  quic.c            HTTP/3 server (ngtcp2 + nghttp3, conditional)
  simd.c            SIMD helpers and CPU-feature probes

include/
  util.h            vortex_memcpy, vortex_strncpy, explicit_bzero, misc helpers

bpf/
  vortex_xdp.bpf.c  XDP kernel program
  maps.h             Shared struct definitions (BPF + userspace)
  vortex_xdp.h       Shared constants (ports, pin paths, map names)
  vmlinux.h          BTF type definitions (generated from running kernel)

cert/
  cert_provider.h    Provider interface
  static_file.c      Load cert+key from PEM files
  acme_client.c      ACME v2 client (JWK, JWS, account/order lifecycle)
  acme_http01.c      HTTP-01 challenge server
  acme_dns01.c       DNS-01 challenge orchestration
  dns_cloudflare.c   Cloudflare DNS API provider

fuzz/
  fuzz_http_parser.c  libFuzzer target for HTTP/1.1 request-line parser
  corpus_seeds.txt    Initial seed corpus

tests/
  test_config.c
  test_cache.c
  test_log.c
  test_router.c
  test_auth.c

tools/
  gen_vmlinux.sh     Regenerate bpf/vmlinux.h from running kernel
  build_deb.sh       Build a self-contained .deb package
  bench.sh           wrk/hey benchmark runner
  vortex-passwd.py   Generate scrypt password verifiers for Basic Auth
  test_phase6.sh     Integration tests for Phase 6 features
  test_phase7.sh     Integration tests for Phase 7 (HTTP/3)
  test_xdp.sh        XDP map inspection and drop counter validation

Adding a new cert provider

Implement struct cert_provider_ops from cert/cert_provider.h and register it in main.c alongside the existing static/ACME providers.

Adding a new DNS provider for DNS-01

Implement struct dns_provider_ops from cert/dns_provider.h and add a selection branch in cert_manager_init() in main.c.


Acknowledgements

Vortex is original software but is built on the shoulders of several excellent open-source libraries. Their authors deserve full credit.

Libraries used

Project Version License Role in Vortex
libbpf system LGPL-2.1 / BSD-2-Clause Loads and manages the XDP/eBPF kernel programs; provides BPF map accessors and the skeleton API
liburing system LGPL-2.1 All async I/O: multishot accept, send/recv, async backend connect, timeouts
libyaml system MIT Parses vortex.yaml configuration files
picotls HEAD MIT TLS 1.2/1.3 handshake, kTLS kernel-offload key installation
micro-ecc bundled BSD-2-Clause ECDH/ECDSA primitives (P-256, P-384) used by picotls minicrypto
ngtcp2 1.16.0 MIT QUIC transport protocol (HTTP/3)
nghttp3 1.8.0 MIT HTTP/3 application layer over QUIC

License compatibility

Vortex itself is released under the GNU General Public License v3.0 (GPLv3). All dependencies are compatible:

  • MIT, BSD-2-Clause — permissive, compatible with GPLv3.
  • LGPL-2.1 — compatible with GPLv3 when linked dynamically (libbpf, liburing, libyaml are system shared libraries).
  • MIT (picotls, micro-ecc) — permissive, compatible with GPLv3.
  • GPL-2.0 (kernel BPF program, vmlinux.h) — the XDP program is a separate kernel-space component; it is explicitly GPL-2.0 as required by the kernel's module ABI.

Commercial entities wishing to use Vortex without complying with the GPLv3 (for example, in a proprietary product) must obtain a separate commercial licence from the author.

Kernel interfaces

  • XDP (eXpress Data Path) — in-kernel packet processing framework, part of the Linux kernel (GPL-2.0).
    The BPF program at bpf/vortex_xdp.bpf.c runs inside the kernel and is licensed GPL-2.0 as required.
  • io_uring — Linux kernel async I/O subsystem (GPL-2.0). Vortex uses the userspace liburing wrapper.
  • kTLS — Kernel TLS offload (Linux 4.13+, GPL-2.0). After TLS handshake, the kernel handles symmetric crypto.
  • bpf/vmlinux.h — auto-generated from the running kernel's BTF type information via bpftool btf dump. Reflects the kernel's own type definitions and is covered by the kernel's GPL-2.0 license.

About

High-performance kernel-assisted reverse proxy — XDP/eBPF, io_uring, kTLS, HTTP/3

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages