feat(model): implement LLaMA-style SwiGLU feed-forward network

CHANGELOG.md

@@ -11,8 +11,19 @@ and this project aims to adhere to [Semantic Versioning](https://semver.org/).
 
 ### Added
 
-- Odyssey Specification **v1.0.0** under `spec/` (architecture, shapes, weights, GGUF mapping, tokenizer, runtime contract)
-- Cross-links from README / math / docs to the normative Spec
+- (pending)
+
+---
+
+## [0.6.0] — 2026-07-29
+
+### Added
+
+- LLaMA-style SwiGLU FFN (`model.OdysseySwiGLU`, `activations`, `parameter_counter`)
+- `FeedForwardConfig` + `feed_forward:` in `configs/model.yaml` / `default.yaml`
+- Cross-implementation validator `scripts/validate_swiglu.py` (vs Phalanx)
+- Benchmarks, papers (`swiglu`, `llama_ffn`), `math/swiglu.md`, experiment `ODY-0006`
+- Shared suite entry `../validation/test_swiglu.py`
 
 ---
 

EXPERIMENTS.md

@@ -117,3 +117,21 @@ Details: [experiments/ODY-0004/README.md](experiments/ODY-0004/README.md)
 | Lessons | Float32 sum-of-squares + identical ε/γ keep train/serve aligned; shared suite under `../validation/` |
 
 Details: [experiments/ODY-0005/README.md](experiments/ODY-0005/README.md)
+
+
+---
+
+## ODY-0006 — SwiGLU Feed-Forward
+
+| Field | Value |
+| --- | --- |
+| ID | ODY-0006 |
+| Date | 2026-07-29 |
+| Phase | 6 |
+| Purpose | LLaMA-style SwiGLU + Phalanx numerical parity |
+| Config | `configs/model.yaml` / experiment `config.yaml` |
+| Result | **Successful** |
+| Validation | See `swiglu_validation.json` (PASS @ 1e-3 GEMM tol) |
+| Lessons | Float64 GEMM accum + documented SwiGLU abs tol keep train/serve aligned |
+
+Details: [experiments/ODY-0006/README.md](experiments/ODY-0006/README.md)

PAPERS.md

@@ -61,11 +61,19 @@ Math companions: [math/rmsnorm.md](math/rmsnorm.md), [math/residuals.md](math/re
 
 ---
 
+## Phase 6 — SwiGLU *(complete)*
+
+| Paper / Study | Summary |
+| --- | --- |
+| GLU Variants (Shazeer) | [papers/swiglu.md](papers/swiglu.md) |
+| LLaMA FFN notes | [papers/llama_ffn.md](papers/llama_ffn.md) |
+
+---
+
 ## Planned Reading (later phases)
 
 | Topic | Canonical paper / resource | Phase |
 | --- | --- | --- |
 | Attention | *Attention Is All You Need* (Vaswani et al., 2017) | 6+ |
-| SwiGLU | *GLU Variants Improve Transformer* (Shazeer) | 7 |
 | GPT-style LMs | GPT / Llama technical reports | 9–10 |
 | DPO | *Direct Preference Optimization* (Rafailov et al.) | 14 |

README.md

@@ -6,7 +6,7 @@
 
 ---
 
-**Status:** Research Project — Phase 5 complete · **Spec v1.0.0** frozen  
+**Status:** Research Project — Phase 6 complete · **Spec v1.0.0** frozen  
 **Language:** Python 3.12+  
 **Framework:** PyTorch  
 **Target Runtime:** [Phalanx Runtime](https://github.com/404khai/phalanx)  
@@ -36,6 +36,7 @@ Odyssey is a research repository for building a small, carefully engineered deco
 | 3 | **Token embedding layer** (`OdysseyEmbedding`) |
 | 4 | **RoPE** (`OdysseyRoPE`) + Phalanx numerical validation |
 | 5 | **RMSNorm** + pre-norm residuals + Phalanx validation |
+| 6 | **SwiGLU** (`OdysseySwiGLU`) + Phalanx validation |
 
 ```mermaid
 flowchart TD
@@ -46,11 +47,30 @@ flowchart TD
     EmbeddingLookup --> Vectors[Embedding Vectors]
     Vectors --> RoPE[RoPE]
     RoPE --> RMSNorm[RMSNorm]
-    RMSNorm --> TransformerBlock[Transformer Block]
+    RMSNorm --> SwiGLU[SwiGLU]
+    SwiGLU --> TransformerBlock[Transformer Block]
 ```
 
 ---
 
+## SwiGLU Feed-Forward (Phase 6)
+
+```python
+from model import OdysseySwiGLU, load_feed_forward_config
+ffn = OdysseySwiGLU(load_feed_forward_config())
+y = ffn(x)  # (B, S, D) → (B, S, D)
+```
+
+Cross-check against Phalanx Runtime:
+
+```bash
+python scripts/validate_swiglu.py
+```
+
+Docs: [`docs/architecture/swiglu.md`](docs/architecture/swiglu.md) · Spec: [`spec/feedforward.md`](spec/feedforward.md)
+
+---
+
 ## RMSNorm & Residuals (Phase 5)
 
 ```python
@@ -226,7 +246,8 @@ MYPYPATH=tokenizer mypy --explicit-package-bases -p odyssey_tokenizer
 | 3 | Embedding layer | **Complete** |
 | 4 | RoPE (+ Phalanx validation) | **Complete** |
 | 5 | RMSNorm + residuals (+ Phalanx validation) | **Complete** |
-| 6–20 | Attention → Odyssey v1 | Planned |
+| 6 | SwiGLU FFN (+ Phalanx validation) | **Complete** |
+| 7–20 | Attention → Odyssey v1 | Planned |
 
 ---
 
@@ -240,6 +261,7 @@ MYPYPATH=tokenizer mypy --explicit-package-bases -p odyssey_tokenizer
 | ODY-0003 | Token embedding layer | Successful |
 | ODY-0004 | RoPE + Phalanx parity | Successful |
 | ODY-0005 | RMSNorm + Phalanx parity | Successful |
+| ODY-0006 | SwiGLU + Phalanx parity | Successful |
 
 ---
 

ROADMAP.md

@@ -61,17 +61,19 @@ Development proceeds **one phase at a time**. Complete, document, commit, then w
 
 ---
 
-## Phase 6 — Multi-Head Attention
+## Phase 6 — SwiGLU Feed Forward *(complete)*
 
-- Causal masking
-- Scaled dot-product attention
+- LLaMA-style SwiGLU (`OdysseySwiGLU`)
+- Manual SiLU, configurable intermediate size
+- Cross-implementation validation vs Phalanx (`scripts/validate_swiglu.py`)
+- ODY-0006 baseline
 
 ---
 
-## Phase 7 — SwiGLU Feed Forward
+## Phase 7 — Multi-Head Attention
 
-- Activation research
-- FFN implementation
+- Causal masking
+- Scaled dot-product attention
 
 ---
 

assets/swiglu/README.md

@@ -0,0 +1 @@
+Odyssey SwiGLU assets (Phase 6)

configs/default.yaml

@@ -2,8 +2,8 @@
 # All hyperparameters should eventually flow from config files.
 
 experiment:
-  id: ODY-0005
-  name: odyssey-rmsnorm-baseline
+  id: ODY-0006
+  name: odyssey-swiglu-baseline
   seed: 42
 
 model:
@@ -33,6 +33,13 @@ model:
     epsilon: 0.000001
     device: cpu
     dtype: float32
+  feed_forward:
+    type: swiglu
+    hidden_size: 768
+    intermediate_size: 2048
+    activation: silu
+    device: cpu
+    dtype: float32
 
 tokenizer:
   path: assets/tokenizer/bpe/odyssey.model

configs/model.yaml

@@ -1,4 +1,4 @@
-# Odyssey model configuration (Phase 5+)
+# Odyssey model configuration (Phase 6+)
 
 model:
   name: odyssey-tiny
@@ -20,11 +20,9 @@ model:
   # RoPE — must stay aligned with Phalanx layers::Rope / Odyssey Spec v1
   rope:
     theta: 10000.0
-    # head_dim defaults to hidden_size / num_heads (= 64 for Tiny)
-    # Experiment ODY-0004 also exercises rotary_dim=128 with head_dim=128
     rotary_dim: 64
     max_position_embeddings: 2048
-    scaling: none          # none | linear  (NTK / YaRN deferred)
+    scaling: none
     scaling_factor: 1.0
     device: cpu
     dtype: float32
@@ -35,3 +33,12 @@ model:
     epsilon: 0.000001
     device: cpu
     dtype: float32
+
+  # SwiGLU FFN — must stay aligned with Phalanx layers::SwiGlu
+  feed_forward:
+    type: swiglu
+    hidden_size: 768
+    intermediate_size: 2048
+    activation: silu
+    device: cpu
+    dtype: float32

docs/architecture/README.md

@@ -10,6 +10,7 @@ Notes for Odyssey's decoder-only transformer.
 | [rope.md](rope.md) | 4 | Complete |
 | [rmsnorm.md](rmsnorm.md) | 5 | Complete |
 | [residuals.md](residuals.md) | 5 | Complete |
-| Attention | 6 | Planned |
+| [swiglu.md](swiglu.md) / [feedforward.md](feedforward.md) | 6 | Complete |
+| Attention | 7 | Planned |
 
 Equation-level pedagogy: [`math/`](../../math/README.md) (non-normative).

docs/architecture/feedforward.md

@@ -0,0 +1,11 @@
+# Feed-Forward Network
+
+Odyssey's position-wise MLP is **SwiGLU only** (Spec v1). See [swiglu.md](swiglu.md) for the full derivation.
+
+```text
+x → gate_proj → SiLU ─┐
+                       ⊙ → down_proj → y
+x → up_proj ──────────┘
+```
+
+Complexity: \(O(B S D I)\) time; peak activation memory \(O(B S I)\).

docs/architecture/swiglu.md

@@ -0,0 +1,48 @@
+# SwiGLU Feed-Forward Network
+
+## Motivation
+
+Attention mixes tokens; the FFN lets each position apply a nonlinear transform independently, adding most of the model's capacity. LLaMA-style models replace GeLU MLPs with **SwiGLU**.
+
+## Mathematics
+
+See Spec [`spec/feedforward.md`](../../spec/feedforward.md).
+
+\[
+\mathrm{SiLU}(z)=z\cdot\sigma(z)
+\qquad
+\mathrm{FFN}(x)=\bigl(\mathrm{SiLU}(x W_1^\top)\odot(x W_3^\top)\bigr)W_2^\top
+\]
+
+| Weight | Role | Shape |
+| --- | --- | --- |
+| \(W_1\) (`gate_proj`) | Gate | `(I, D)` |
+| \(W_3\) (`up_proj`) | Up | `(I, D)` |
+| \(W_2\) (`down_proj`) | Down | `(D, I)` |
+
+No biases. Intermediate size \(I\) is independent of \(D\) (Tiny: 2048 vs 768) — larger than the classic \(4D\) GeLU FFN after the SwiGLU \(2/3\) parameter adjustment.
+
+## Implementation
+
+| Module | Role |
+| --- | --- |
+| `model/activations.py` | Manual SiLU |
+| `model/swiglu.py` | `OdysseySwiGLU` |
+| `model/feedforward.py` | Public FFN factory |
+| `configs/model.yaml` | `feed_forward:` |
+
+```python
+from model import OdysseySwiGLU, load_feed_forward_config
+ffn = OdysseySwiGLU(load_feed_forward_config())
+y = ffn(x)  # (B,S,D) → (B,S,D)
+```
+
+## Phalanx Compatibility
+
+```bash
+python scripts/validate_swiglu.py
+# or: python ../validation/test_swiglu.py
+```
+
+Tolerance default **`1e-3`** (GEMM accumulation; mean error typically ≪ `1e-6`).
+Report: `experiments/ODY-0006/swiglu_validation.json`.

experiments/ODY-0006/README.md

@@ -0,0 +1,37 @@
+# ODY-0006 — SwiGLU Feed-Forward Network
+
+| Field | Value |
+| --- | --- |
+| Phase | 6 |
+| Date | 2026-07-29 |
+| Purpose | Implement LLaMA-style SwiGLU + cross-validate vs Phalanx |
+| Result | **Successful** |
+
+## Configuration
+
+| Knob | Value |
+| --- | --- |
+| type | swiglu |
+| hidden_size | 768 (Tiny) / 64 (validation default) |
+| intermediate_size | 2048 (Tiny) / 128 (validation default) |
+| activation | silu |
+
+## Cross-Implementation Validation
+
+```bash
+python scripts/validate_swiglu.py
+# or: python ../validation/test_swiglu.py
+```
+
+| Metric | Value |
+| --- | --- |
+| Max abs error | ≈ 1.22e-04 |
+| Mean abs error | ≈ 4.75e-07 |
+| Tolerance | 1e-3 (GEMM accum; documented) |
+| Status | **PASS** |
+
+## Artifacts
+
+- Metrics: `metrics.json`
+- Validation: `swiglu_validation.json`
+- Config snapshot: `config.yaml`

experiments/ODY-0006/config.yaml

@@ -0,0 +1,13 @@
+# ODY-0006 experiment snapshot
+
+experiment:
+  id: ODY-0006
+  name: odyssey-swiglu-baseline
+  seed: 42
+
+feed_forward:
+  type: swiglu
+  hidden_size: 768
+  intermediate_size: 2048
+  activation: silu
+  dtype: float32

experiments/ODY-0006/metrics.json

@@ -0,0 +1,36 @@
+{
+  "hidden_size": 768,
+  "intermediate_size": 2048,
+  "dtype": "float32",
+  "device": "cpu",
+  "shape": [
+    2,
+    64,
+    768
+  ],
+  "parameter_count": 4718592,
+  "memory_bytes": 18874368,
+  "forward_mean_seconds": 0.00580928,
+  "tokens_per_second": 22033.71,
+  "inspect": {
+    "type": "swiglu",
+    "activation": "silu",
+    "hidden_size": 768,
+    "intermediate_size": 2048,
+    "expansion_ratio": 2.6666666666666665,
+    "parameter_count": 4718592,
+    "memory_bytes": 18874368,
+    "projections": {
+      "gate_proj (w1)": 1572864,
+      "up_proj (w3)": 1572864,
+      "down_proj (w2)": 1572864
+    },
+    "shapes": {
+      "input": "(..., 768)",
+      "gate/up": "(..., 2048)",
+      "output": "(..., 768)"
+    },
+    "device": "cpu",
+    "dtype": "torch.float32"
+  }
+}