Geometry Dash Reinforcement Learning Agent

This project is an exploration into teaching an AI agent how to play Geometry Dash using screen capture, Computer Vision, and Deep Reinforcement Learning.

The game presents a unique challenge for Reinforcement Learning because the physics and required actions completely change depending on the current game mode (e.g., the Cube jumps, while the Ship flies). To solve this, the project utilizes a combination of Semi-Supervised Learning (FixMatch) to visually classify the game mode and a Multi-Task Deep Q-Network (Multi-Head DQN) to make decisions.

---

System Architecture

The AI pipeline is divided into two major components: the Vision System and the Decision System (Agent).

graph TD
    %% Define Styles
    classDef environment fill:#2d3436,stroke:#74b9ff,stroke-width:2px,color:#fff;
    classDef vision fill:#0984e3,stroke:#74b9ff,stroke-width:2px,color:#fff;
    classDef rl fill:#6c5ce7,stroke:#a29bfe,stroke-width:2px,color:#fff;
    
    subgraph Environment [Game Environment]
        Screen[Screen Capture<br/>64x128 Grayscale]:::environment
        Death[Death Detector<br/>Template Matching]:::environment
        Reward[Reward Calculation<br/>Survival Time]:::environment
    end

    subgraph Vision [Vision System - CNN Backbone]
        Mask[Image Masking]:::vision
        CNN[FixMatch Trained CNN Backbone]:::vision
        PredMode[Predict Mode<br/>Cube / Ship]:::vision
    end

    subgraph Agent [RL Agent - Multi-Head DQN]
        Stack[Frame Stacking<br/>4 Consecutive Frames]:::rl
        SharedFC[Shared Fully Connected Layer]:::rl
        CubeHead[Cube Head Q-Values]:::rl
        ShipHead[Ship Head Q-Values]:::rl
        Action[Epsilon-Greedy Policy<br/>Jump / Do Nothing]:::rl
    end

    %% Flow
    Screen --> Mask
    Screen --> Stack
    Screen --> Death
    Death --> Reward
    
    Mask --> CNN
    CNN --> PredMode
    
    Stack --> SharedFC
    PredMode --> CubeHead
    PredMode --> ShipHead
    
    SharedFC --> CubeHead
    SharedFC --> ShipHead
    
    CubeHead --> Action
    ShipHead --> Action
    
    Action -.-> |Keyboard Input| Screen

Loading

---

Implementation Details

1. Vision System (Semi-Supervised Learning)

Before the agent can decide when to jump, it needs to know how it jumps.

• Data Collection: The system captures 64x128 grayscale frames of the gameplay.
• FixMatch: Labeling thousands of frames manually is tedious. I used FixMatch, a semi-supervised learning technique, to train a Convolutional Neural Network (CNN) classifier. By feeding the network a small dataset of labeled images and a massive dataset of unlabeled images, the model learned to confidently predict pseudo-labels for the unlabeled data and train itself.
• Masking: Horizontal bounds of the image are masked to force the model to focus strictly on the player's character.

2. Reinforcement Learning Agent

The decision-making process is modeled as a Markov Decision Process (MDP) handled by a Deep Q-Network.

• State Space & Frame Stacking: A single static frame provides no information about momentum. I stack 4 consecutive frames so the agent can perceive velocity (e.g., distinguishing whether the ship is moving up or falling).
• Multi-Head DQN Architecture:
  • I used Transfer Learning to extract the convolutional backbone from the FixMatch vision model.
  • The DQN splits into two separate output "heads": one for the Cube and one for the Ship. Based on the vision system's prediction, the network activates the correct head. This isolates the distinct physics of the two game modes.
• Experience Replay: Instead of learning from sequential frames (which are highly correlated and cause instability), the agent stores experiences (State, Action, Reward, Next_State, Done) into a memory buffer and samples random mini-batches to train via the Bellman Equation.
• Target Network: A delayed copy of the main network is used to generate the Q-value targets, preventing the agent from chasing a wildly moving target during backpropagation.
• Epsilon-Greedy: The agent balances exploring the level randomly and exploiting its learned network using a decaying $\epsilon$ value.

---

Challenges & Hardware Bottlenecks

While the theoretical machine learning architecture is sound, the agent struggled to beat the game in practice. This project highlighted a critical lesson in Real-Time Reinforcement Learning: Latency.

• The Latency Disconnect: RL relies on the Markov property—the assumption that an action directly results in the next state. Because my hardware relied on CPU processing for screen capturing, CNN inference, and DQN backpropagation, severe latency was introduced.
• Desync: By the time the agent processed frame $t$ and decided to jump, the actual game was already at frame $t+5$. The jump happened too late, resulting in a death that the agent incorrectly blamed on the state it saw in frame $t$.
• Frame Skipping: To alleviate CPU load, I implemented a frame skip (running the agent every 4 frames). While this is standard practice (e.g., in Atari DQN environments), when combined with the baseline hardware latency, it made the agent effectively "blind" to fast-approaching obstacles.

Conclusion: DRL in real-time environments without pausing requires highly optimized, low-level screen hooking or GPU-accelerated tensor operations to keep action-latency in the milliseconds range.

---

How to Run

1. Install Dependencies:

   pip install -r requirements.txt

2. Start the Agent: Open Geometry Dash (ensure it is windowed in the correct capture location) and run:

   python main.py

3. Training Data / FixMatch: If you want to view the FixMatch training process, check out the Jupyter Notebook inside the /notebooks directory.

---

Created as an exploratory research project in Reinforcement Learning and Computer Vision.