ViZDoom Local 4-Player Team Deathmatch

A fully local (no Docker / Kubernetes / gRPC / cloud) multi-agent ViZDoom simulation: 1 human player + 3 bots (rule-based Offensive, rule-based Defensive, PyTorch DQN RL agent) playing Team Deathmatch over the loopback interface (127.0.0.1), each participant running in its own OS process.

1. Install

Requires Python 3.9+ and a C/C++ toolchain (needed by the vizdoom wheel on some platforms).

cd vizdoom_tdm
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

vizdoom ships its bundled scenario WADs (including cig.wad, used here) inside the package itself — no separate IWAD purchase/download is required for this demo.

If you have an NVIDIA GPU + CUDA-enabled PyTorch installed, agents/rl_agent.py will automatically use it (torch.cuda.is_available()); otherwise it falls back to CPU.

2. Directory layout

vizdoom_tdm/
├── config/tdm_config.cfg     # shared ViZDoom engine/game settings
├── agents/
│   ├── base_agent.py         # shared engine lifecycle, reward, recording
│   ├── offensive_bot.py      # rule-based aggressive bot
│   ├── defensive_bot.py      # rule-based evasive/survival bot
│   └── rl_agent.py           # PyTorch CNN DQN agent
├── utils/
│   ├── video_recorder.py     # async OpenCV MP4 writer
│   └── logger.py             # per-agent CSV/console episode logger
├── checkpoints/               # rl_ep_<N>.pth saved every 200 episodes
├── recordings/                 # ep_<N>.mp4 saved every 200 episodes
├── practice.py                # interactive CLI match launcher (8 modes)
└── train_rl.py                 # self-play DQN training pipeline

3. Play a practice match

python practice.py

You'll be shown a menu to pick one of 8 modes:

 1) 1v1   Human            vs Offensive
 2) 1v1   Human            vs Defensive
 3) 1v1   Human            vs RL
 4) 2v1   Human + RL       vs Offensive + Defensive
 5) 3v1   Human            vs Offensive + Defensive + RL
 6) 1v2   RL               vs Offensive + Defensive
 7) 1v1   RL               vs Offensive
 8) 1v1   RL               vs Defensive

When a human is present, they always host the match and get an interactive GUI window (mouse/keyboard control via ViZDoom's SPECTATOR mode). Every bot runs headlessly in its own background process and joins the host over 127.0.0.1. In modes 6-8 (no human), the RL agent hosts headlessly instead.

Modes 3-8 automatically load the latest checkpoint from checkpoints/ (if one exists) for the RL agent; otherwise it plays with random weights.

4. Train the RL agent

python train_rl.py

This launches a 4-player self-play training match, entirely local:

• Learner (host) — the DQN being trained: epsilon-greedy exploration, a 50k-transition replay buffer, Huber loss, Adam optimizer, and a target network synced every 20 episodes.
• Self-play opponent (joiner) — a frozen copy of the policy that reloads the learner's latest checkpoint every 10 episodes, so the learner keeps facing a recent version of itself.
• Offensive / Defensive bots (joiners) — rule-based curriculum opponents for stable, non-degenerate training signal.

Every 200 episodes:

• Model weights are saved to checkpoints/rl_ep_<N>.pth.
• That episode's gameplay is asynchronously encoded to recordings/ep_<N>.mp4 (video encoding happens on a background thread, so it never blocks or slows down environment stepping).

Per-episode metrics (reward, steps, kills, deaths) are appended to logs/<agent_name>_episodes.csv by each process independently.

5. Reward function

Every agent (rule-based or RL) is scored with the same shaping formula, computed purely from ViZDoom GameVariable deltas each step:

R_t = +100 * Kills + 0.1 * DmgDealt - 0.1 * DmgTaken - 50 * FriendlyFire - 100 * Deaths

FriendlyFire is a best-effort heuristic (see compute_reward in agents/base_agent.py): ViZDoom's bundled multiplayer scenario has no native team-kill flag, so it is approximated from kills that don't produce a matching frag increment. This is documented in code as an approximation, not a ground-truth signal.

6. Notes & limitations

• All networking is strictly loopback (127.0.0.1); nothing here binds to a non-local interface or requires internet access.
• "Team" behavior (color/name convention, friendly-fire heuristic) is layered on top of ViZDoom's inherently free-for-all deathmatch engine in Python (agents/base_agent.py::TeamAffiliation), since the stock multiplayer WAD has no built-in team concept.
• episode_timeout / timelimit in config/tdm_config.cfg bound how long a single match/episode can run; adjust for shorter/longer showcase runs.