chaos-engineering-toolkit

Chaos experiments (pod kill, network latency, network partition) against a sample app on Kubernetes using Chaos Mesh, with runbooks and scheduled experiment support. Demonstrates a production-grade chaos-engineering workflow: define → run → observe → clean up → schedule.

Architecture

chaos-engineering-toolkit/
├── experiments/
│   ├── pod-kill/          # PodChaos – kills one API pod randomly
│   │   ├── experiment.yaml
│   │   └── schedule.yaml  # runs every 10 min
│   ├── network-latency/   # NetworkChaos – 200 ms ± 50 ms on API egress
│   │   ├── experiment.yaml
│   │   └── schedule.yaml  # runs every 6 h
│   └── network-partition/ # NetworkChaos – severs API ↔ DB for 30 s
│       ├── experiment.yaml
│       └── schedule.yaml  # runs weekly (Mon 02:00)
├── sample-app/
│   ├── manifests/         # Namespace, Deployment (3 replicas), Service, HPA
│   └── src/               # Flask API with /health and /ready probes + Dockerfile
├── runbooks/              # Step-by-step guides for each chaos scenario
├── scripts/               # setup.sh, run-experiment.sh, check-steady-state.sh
└── tests/                 # pytest: validates YAML schemas for experiments + K8s manifests

Prerequisites

Tool	Version
kubectl	≥ 1.28
helm	≥ 3.14
kind or k3d (local cluster)	any recent
Python	≥ 3.11 (for tests/linting)

Quick Start

1. Spin up a local cluster (kind example)

kind create cluster --name chaos-demo

2. Install Chaos Mesh + deploy the sample app

make setup

This runs scripts/setup.sh, which:

• Adds the Chaos Mesh Helm chart
• Installs Chaos Mesh in the chaos-testing namespace
• Applies the sample-app Kubernetes manifests
• Waits for the demo-api deployment to roll out

3. Verify steady state

make check-steady-state

4. Run an experiment

make experiment EXP=pod-kill
# or
make experiment EXP=network-latency
# or
make experiment EXP=network-partition

The script applies the experiment, polls status, and cleans up on expiry.

5. Enable scheduled experiments

kubectl apply -f experiments/pod-kill/schedule.yaml
kubectl apply -f experiments/network-latency/schedule.yaml
kubectl apply -f experiments/network-partition/schedule.yaml

Experiments

Pod Kill (pod-kill)

What: Kills one randomly chosen demo-api pod. Duration: 60 s (Kubernetes reschedules within ~15 s). SLO impact: Available replicas must stay ≥ 2; success rate ≥ 99 %.

Network Latency (network-latency)

What: Adds 200 ms ± 50 ms latency to all outbound traffic from demo-api. Duration: 120 s. SLO impact: p99 latency rises ~200 ms; circuit breakers must not open.

Network Partition (network-partition)

What: Completely severs bidirectional traffic between demo-api and demo-db. Duration: 30 s. SLO impact: DB-dependent endpoints return 503; no data corruption.

See the runbooks/ directory for detailed step-by-step guides.

Running Tests & Linting Locally

make install   # pip install -r requirements-dev.txt
make validate  # yamllint + pytest

Tests validate:

• All experiment YAML files parse correctly and have required Chaos Mesh fields
• All Kubernetes manifests have required fields, probes, and resource limits

Tech Stack

• Chaos Mesh 2.6 — Kubernetes-native chaos engineering platform
• Kubernetes — Deployment, Service, HPA for the sample app
• Flask 3 — Minimal demo API with health/readiness probes
• pytest + PyYAML — YAML schema tests
• yamllint — YAML linting
• GitHub Actions — CI: lint + test on every push