Itinerant-Net

A neural sequence model that computes by traveling through a sequence of metastable states — the way cortex does — trained end-to-end by gradient descent.

Most sequence models compute with attention over a window (Transformers) or a linear recurrence in the hidden state (S4/Mamba). Itinerant-Net does something different: its core is a nonlinear competitive dynamical system that settles transiently into one activity pattern, then makes a fast, input-triggered transition to the next, producing a reliable ordered trajectory through learned states. The computation is that trajectory — an ordered path — not a single settled point.

To our knowledge this is the first learned, gradient-trained, generative heteroclinic core benchmarked as a compositional engine. Heteroclinic/winnerless-competition dynamics have been studied for decades but almost always hand-designed; here the dynamics are learned end-to-end.

Status: research prototype. It works — it learns 120-state permutation composition (S₅) and its internal dynamics are verifiably metastable. It does not currently beat a parameter-matched GRU on compositional generalization (see Results, which reports that plainly). The contribution here is a working new mechanism, not a state-of-the-art result.

---

What it is / how it works

The core is a rate network of K units obeying generalized Lotka–Volterra (competitive) dynamics — the mathematical image of winnerless competition (Rabinovich et al. 2001):

dxᵢ/dt = xᵢ · (λᵢ + Σⱼ Aᵢⱼ xⱼ  + driveᵢ)      xᵢ ≥ 0

• A — the competition matrix. Asymmetric inhibition is the whole mechanism: if unit i suppresses j more than j suppresses i, activity flows i → j → …, chaining saddle points into a stable heteroclinic channel — a reliable ordered sequence of metastable states.
• λ — self-drive. Sets where each metastable state sits.
• drive — the input. Input selects and redirects the trajectory (input-triggered transitions); this is how the network tracks state.
• A, λ, and the input coupling B are all LEARNED by backprop-through-time. Only the initial A is seeded from the Rabinovich winnerless-competition construction so training starts on a valid channel.

Four engineering choices make this trainable on a laptop (details in itinerant_net/core.py):

1. Log-space integration (u = log x). Positivity is then exact and differentiable everywhere; the naive clamp(x, 0) zeroes ~44% of gradients.
2. A soft activity floor restores the noise level σ as the dwell-time control, dwell ≈ |log σ| / (λ_u · dt).
3. Input-gated competition A_eff(g) = A + A_mod[g]. A shared additive drive provably cannot represent a group action (a linear-feasibility check in scripts shows the additive form is infeasible at every width for S₃); letting the input select the effective competition matrix makes it representable. A_mod is learned, initialized to zero.
4. A discrete, sequence-level readout. The output is decoded from the identity of the winning unit at each step (straight-through), not from continuous activity — so the output is a function of the visited-state trajectory, not a point in representation space.
5. Gradient flossing (Engelken 2023): a regularizer that pushes the dynamics' Lyapunov exponents toward zero — the edge of stability, where metastability lives. Without it, gradient descent drifts the core toward chaos.

input symbols → encoder → drive u(t)
                              ↓
        K-unit metastable core  (learned A, λ, input-gated competition)
                              ↓
        trajectory = ordered sequence of visited metastable states
                              ↓
        sequence-level decoder  (reads the winner identity per step)
                              ↓
                       output sequence

The task

Permutation composition over S₅ (the symmetric group on 5 elements, 120 states): read a sequence of primitive permutations, output the running composition at every step. This is a canonical nonlinear state-tracking problem — provably outside the reach of fixed-depth diagonal state-space models (Merrill, Petty & Sabharwal 2024), because S₅ is non-solvable. The benchmark has held-out orderings (primitive pairs never seen adjacent) and held-out lengths (test sequences longer than any in training).

Install & run

pip install torch numpy          # that's all it needs
python demo.py                   # watch the core itinerate through its states
python train.py                  # train on S_5 (uses Apple MPS if available; ~a few minutes)
python train.py --K 96 --steps 1500

demo.py prints the ordered path the core travels and its dwell times and stability. train.py trains the full model and reports in-distribution accuracy plus held-out ordering/length generalization and the live regime diagnostics.

Results (honest)

Reference run: K=128 (≈115k params), gradient flossing on, 3000 steps, seed 0, one M1 laptop.

metric	Itinerant-Net	GRU (param-matched, 115k)
in-distribution (per-step)	0.86	0.98
in-distribution (exact sequence)	0.60	0.92
held-out ordering generalization	0.14	0.61
held-out length generalization	0.11	0.44

Read this straight: a parameter-matched GRU learns the task faster, higher, and generalizes substantially better. Itinerant-Net learns the task (well above the ~0.006–0.011 degenerate floors, so it is genuinely tracking state, not memorizing) and its substrate is confirmed metastable throughout training (≈100 of 120 distinct metastable states in use; Lyapunov exponent held near zero by flossing). But on the compositional-generalization axis this model was built to test, it does not currently win.

What is established and reproducible here:

• Gradient descent can learn and maintain a multi-state heteroclinic channel without collapsing it to a fixed point — the central open risk for this class of model.
• Gradient flossing measurably controls the dynamical regime (holds the Lyapunov exponent near zero) and helps every metric.
• The mechanism scales from a toy (S₃) to the full 120-state task.

Known open questions (why this is a prototype, not a paper): whether the itinerancy is causally necessary for the accuracy it does reach (an ablation, not yet run), and whether higher capacity / different training closes the generalization gap. See the parent research log for the full falsification history.

Citations

Load-bearing references, verified against source. See CITATIONS.md for the full list with notes on what each does and does not support.

• Rabinovich et al., Phys. Rev. Lett. 87:068102 (2001) — winnerless competition, heteroclinic channels.
• Afraimovich, Zhigulin & Rabinovich, Chaos 14:1123 (2004) — existence conditions for reproducible heteroclinic sequences.
• Merrill, Petty & Sabharwal, "The Illusion of State in State-Space Models," ICML 2024 (arXiv:2404.08819) — why fixed-depth SSMs can't do S₅ state-tracking.
• Engelken, "Gradient Flossing," NeurIPS 2023 (arXiv:2312.17306) — Lyapunov-exponent regularization.
• Recanatesi et al., Neuron 110:139 (2022); Mazzucato et al., Nat. Neurosci. 22:787 (2019) — metastable sequential dynamics in cortex.

License & author

Author: Muhammad Rakibul Islam · Contact: imrakibul@gmail.com

Licensed under the Apache License, Version 2.0 — see LICENSE and NOTICE. Copyright 2026 Muhammad Rakibul Islam.

Built and reported under a kill-first / no-unmeasured-numbers discipline; every number above comes from a committed run log. Questions, corrections, or collaboration — email the address above.