Extend Gridmind

.github/workflows/ci.yml

@@ -0,0 +1,38 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    name: clojure tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '21'
+
+      - name: Set up Clojure CLI
+        uses: DeLaGuardo/setup-clojure@12
+        with:
+          cli: '1.11.1'
+
+      - name: Cache Clojure dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.m2/repository
+            .cpcache
+          key: ${{ runner.os }}-deps-${{ hashFiles('deps.edn') }}
+          restore-keys: |
+            ${{ runner.os }}-deps-
+
+      - name: Run tests
+        run: clojure -M:test

CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+
+All notable changes appear in this file.
+The format follows Keep a Changelog conventions.
+
+## 0.1.0 - unreleased
+
+### Added
+
+- Tabular Q-learning and SARSA agents
+- Epsilon-greedy exploration with per-episode decay
+- Seeded training for reproducible results
+- Deterministic test suite for agents and worlds
+- Demo runner for the built-in worlds
+- CI workflow for the test suite
+- README, roadmap, and license

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Gridmind contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,158 @@
+# Gridmind
+
+![CI](https://github.com/DanielCuevas1208/gridmind/actions/workflows/ci.yml/badge.svg)
+
+Gridmind is a Clojure library that learns to solve grid worlds.
+It trains agents with tabular Q-learning and SARSA.
+The only dependency is the Clojure language itself.
+
+## Highlights
+
+- Deterministic grid world model
+- Tabular Q-learning and SARSA
+- Seeded, reproducible training runs
+- Built-in worlds: small, windy, and cliff
+- No runtime dependencies beyond Clojure
+
+## Quick start
+
+Add Gridmind to your deps.edn as a git dependency.
+
+```clojure
+{:deps {gridmind/gridmind {:git/url "https://github.com/DanielCuevas1208/gridmind"
+                           :sha "<latest-commit>"}}}
+```
+
+Train an agent and read its policy.
+
+```clojure
+(require '[gridmind.worlds :as worlds]
+         '[gridmind.agent :as agent])
+
+(let [{:keys [q-table rewards]}
+      (agent/train worlds/small {:episodes 200 :seed 0})]
+  (agent/evaluate worlds/small q-table))
+;; => [0 3]
+```
+
+The agent learns to reach the goal.
+
+## Architecture
+
+Gridmind has three source namespaces.
+
+- `gridmind.world` defines the deterministic grid world model.
+- `gridmind.worlds` loads the built-in worlds from EDN resources.
+- `gridmind.agent` implements the learning algorithms.
+
+The world model returns the next state, the reward, and a done flag.
+The agent stores one value per state-action pair in a Q-table.
+A high value means the action moves the agent toward the goal.
+
+## Learning algorithms
+
+### Q-learning
+
+Q-learning is an off-policy algorithm.
+The agent updates the value of the action it took.
+It uses the best action at the next state for the target.
+Set `:algorithm :q-learning` to use it. This is the default.
+
+### SARSA
+
+SARSA is an on-policy algorithm.
+The agent updates with the action it actually takes next.
+Set `:algorithm :sarsa` to use it.
+
+### Epsilon-greedy exploration
+
+The agent picks random actions with probability `:epsilon`.
+This explores the world during early training.
+Set `:epsilon-decay` below 1.0 to reduce randomness over time.
+
+## API
+
+| Function | Purpose |
+| --- | --- |
+| `gridmind.agent/q-table` | Create an empty Q-table. |
+| `gridmind.agent/q-value` | Read a state-action value. |
+| `gridmind.agent/update-q!` | Update one state-action value. |
+| `gridmind.agent/greedy-action` | Pick the best action for a state. |
+| `gridmind.agent/epsilon-greedy` | Pick an action with exploration. |
+| `gridmind.agent/run-episode` | Run one learning episode. |
+| `gridmind.agent/train` | Train across many episodes. |
+| `gridmind.agent/greedy-policy` | Build a policy from a Q-table. |
+| `gridmind.agent/evaluate` | Walk the greedy policy to a terminal state. |
+
+### Training options
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `:episodes` | 100 | Number of episodes to train. |
+| `:alpha` | 0.5 | Learning rate. |
+| `:gamma` | 1.0 | Discount factor. |
+| `:epsilon` | 0.1 | Exploration probability. |
+| `:epsilon-decay` | 1.0 | Multiplier for epsilon per episode. |
+| `:seed` | 0 | Seed for the random source. |
+| `:algorithm` | `:q-learning` | `:q-learning` or `:sarsa`. |
+
+## Built-in worlds
+
+| World | Description |
+| --- | --- |
+| `small` | A 3 by 4 grid with a goal and a hazard. |
+| `windy` | The classic windy grid world. |
+| `cliff` | The cliff walking world. |
+
+Load any built-in world by name with `gridmind.worlds/load`.
+
+## Sample output
+
+Run the demo to see training curves and learned paths.
+
+```
+== small ==
+  reward, first episode: -1.090
+  reward, last episode:  0.980
+  greedy path: [[0 0] [0 1] [0 2] [0 3]] -> goal
+== windy ==
+  reward, first episode: -1000.000
+  reward, last episode:  -15.000
+  greedy path: [[3 0] [3 1] [3 2] [2 3] [1 4] [0 5] [0 6] [0 7]
+                [0 8] [0 9] [1 9] [2 9] [3 9] [4 9] [5 9] [6 9]
+                [5 8] [3 7]] -> goal
+== cliff ==
+  reward, first episode: -112.000
+  reward, last episode:  -11.000
+  greedy path: [[3 0] [2 0] [2 1] [2 2] [2 3] [2 4] [2 5] [2 6]
+                [2 7] [2 8] [2 9] [2 10] [2 11] [3 11]] -> goal
+```
+
+Each world ends at the goal after training.
+
+## Limitations
+
+- Tabular methods only suit small state spaces.
+- Transitions are deterministic.
+- The library does not scale to large grids.
+- Training reward depends on the chosen seed.
+
+## Test status
+
+The suite has 24 tests and 49 assertions.
+Every test is deterministic and needs no network.
+Run it with the command below.
+
+```
+clojure -M:test
+```
+
+The CI workflow runs the same suite on every push.
+
+## Roadmap
+
+See ROADMAP.md for what is done and what remains.
+
+## License
+
+Gridmind is released under the MIT License.

ROADMAP.md

@@ -0,0 +1,32 @@
+# Roadmap
+
+This file tracks the direction of Gridmind.
+It shows what is complete and what remains.
+
+## Complete
+
+- Deterministic grid world model
+- EDN world definitions and validation
+- Built-in worlds: small, windy, and cliff
+- Tabular Q-learning
+- SARSA
+- Epsilon-greedy exploration with decay
+- Seeded, reproducible training
+- Demo and test suite
+- CI workflow and project documentation
+
+## In progress
+
+Nothing is in progress right now.
+
+## Next
+
+- Value iteration and policy iteration
+- Function approximation for large grids
+- Stochastic transitions
+- Export Q-tables to EDN
+- Publish to Clojars
+
+## Completed in detail
+
+- 0.1.0: learning agents. See CHANGELOG.md.

dev/gridmind/demo.clj

@@ -0,0 +1,40 @@
+(ns gridmind.demo
+  "Train an agent on each built-in world.
+  Prints the reward curve and the final greedy path."
+
+  (:require [gridmind.agent :as agent]
+            [gridmind.world :as world]
+            [gridmind.worlds :as worlds]))
+
+(defn- greedy-path
+  "Return the positions visited by the greedy policy, up to a terminal state."
+  [world q]
+  (let [policy (agent/greedy-policy q)]
+    (loop [pos (:start world)
+           seen []]
+      (if (or (world/terminal? world pos) (some #{pos} seen))
+        (conj seen pos)
+        (recur (:state (world/step world pos (policy pos)))
+               (conj seen pos))))))
+
+(defn- train-and-report
+  "Train on `world` and print one line per metric."
+  [world options]
+  (let [{:keys [q-table rewards]} (agent/train world options)
+        path (greedy-path world q-table)
+        end (peek path)
+        end-label (cond (= end :exhausted) "budget exhausted"
+                        (world/goal? world end) "goal"
+                        (world/hazard? world end) "hazard"
+                        :else "cycle")]
+    (println (str "== " (:name world) " =="))
+    (println (str "  reward, first episode: " (format "%.3f" (first rewards))))
+    (println (str "  reward, last episode:  " (format "%.3f" (last rewards))))
+    (println (str "  greedy path: " (pr-str path) " -> " end-label))))
+
+(defn -main
+  "Train each built-in world and print the results."
+  [& _]
+  (doseq [world worlds/all]
+    (train-and-report world {:episodes 200 :seed 0 :epsilon 0.15 :epsilon-decay 0.995}))
+  (shutdown-agents))