Reservoir Computing Language Model (RC-LLM)

A complete implementation of a language model using Echo State Networks (ESN) and linear readouts - no deep learning, no backpropagation.

Overview

This implementation provides a production-ready RC-LLM with:

Multiple reservoir support (multi-scale temporal dynamics)
Efficient sparse matrix operations
Ridge regression training (closed-form solution)
Comprehensive evaluation tools
Interactive generation interface
Model checkpointing and resumption

Architecture

Input: Text → Tokenization → Embedding vectors
Core: Fixed random recurrent reservoir (Echo State Network)
Output: Linear readout layer trained with ridge regression
Generation: Autoregressive sampling with temperature/top-k/top-p

Installation

# Install dependencies
pip install -r requirements.txt

# Or install manually
pip install numpy scipy matplotlib

Quick Start

1. Train a Model

Train on sample data (for testing):

python train_rc_llm.py \
    --data-size tiny \
    --vocab-size 500 \
    --reservoir-size 1024 \
    --input-dim 64 \
    --seq-length 64 \
    --save-dir ./my_model

Train on your own text file:

python train_rc_llm.py \
    --text-file path/to/your/text.txt \
    --max-chars 1000000 \
    --vocab-size 8000 \
    --reservoir-size 4096 \
    --input-dim 128 \
    --seq-length 128 \
    --num-reservoirs 3 \
    --save-dir ./my_large_model

2. Generate Text (Interactive)

python inference_rc_llm.py \
    --model-dir ./my_model \
    --mode interactive \
    --temperature 0.8 \
    --max-length 50

Commands in interactive mode:

/temp 1.0 - Change temperature
/len 100 - Change max length
/topk 20 - Change top-k value
quit or exit - Exit

3. Generate Text (Batch)

Create a file prompts.txt with one prompt per line, then:

python inference_rc_llm.py \
    --model-dir ./my_model \
    --mode batch \
    --prompts-file prompts.txt \
    --max-length 100

3b. Question-Answering Mode

Ask questions and get answers:

# Interactive Q&A
python qa_interface.py --model-dir ./my_model

# Single question
python qa_interface.py \
    --model-dir ./my_model \
    --mode single \
    --question "What is machine learning?"

# Batch questions from file
python qa_interface.py \
    --model-dir ./my_model \
    --mode batch \
    --questions-file questions.txt

See QA_GUIDE.md for detailed Q&A documentation.

4. Evaluate Model

python evaluate_rc_llm.py \
    --model-dir ./my_model \
    --test-file path/to/test.txt \
    --num-samples 200 \
    --analyze-reservoir \
    --generate-samples \
    --output-dir ./eval_results

Configuration Options

Model Hyperparameters

# Core settings
--vocab-size        # Vocabulary size (500-50000)
--reservoir-size    # Neurons per reservoir (1024-50000)
--input-dim         # Embedding dimension (64-512)
--num-reservoirs    # Number of reservoirs (1-5)

# Reservoir dynamics
--spectral-radius   # Controls memory (0.7-1.2, default 0.9)
--leak-rate         # Time scale (0.05-0.5, default 0.1)

# Training
--ridge-lambda      # Regularization (1e-4 to 1.0, default 1e-3)
--seq-length        # Training sequence length (64-512)

# Tokenization
--tokenizer-level   # 'char' or 'word'

Recommended Configurations

Tiny Model (Fast experimentation):

--vocab-size 500 --reservoir-size 512 --input-dim 64

Small Model (Basic language modeling):

--vocab-size 2000 --reservoir-size 2048 --input-dim 128 --num-reservoirs 2

Medium Model (Better quality):

--vocab-size 8000 --reservoir-size 8192 --input-dim 256 --num-reservoirs 3

Large Model (Research-grade):

--vocab-size 16000 --reservoir-size 20480 --input-dim 512 --num-reservoirs 4

File Structure

rc_llm.py            # Core model implementation
training_utils.py    # Data loading, tokenization, evaluation
train_rc_llm.py      # Training script
inference_rc_llm.py  # Interactive and batch generation
qa_interface.py      # Question-answering interface (NEW!)
evaluate_rc_llm.py   # Comprehensive evaluation
requirements.txt     # Dependencies

Usage Examples

Example 1: Character-Level Model

# Train character-level model
python train_rc_llm.py \
    --text-file data.txt \
    --tokenizer-level char \
    --vocab-size 100 \
    --reservoir-size 2048 \
    --seq-length 256 \
    --save-dir ./char_model

# Generate
python inference_rc_llm.py \
    --model-dir ./char_model \
    --mode interactive

Example 2: Multi-Scale Reservoir

# Train with 3 reservoirs at different time scales
python train_rc_llm.py \
    --text-file data.txt \
    --num-reservoirs 3 \
    --reservoir-size 4096 \
    --leak-rate 0.05 \
    --save-dir ./multiscale_model

# The script will create reservoirs with leak rates:
# - Reservoir 1: 0.05 (slow, long-term memory)
# - Reservoir 2: 0.10 (medium)
# - Reservoir 3: 0.20 (fast, short-term)

Example 3: Programmatic Usage

from rc_llm import RCLLM, ModelConfig, ReservoirConfig
from training_utils import Tokenizer, DataLoader, train_model

# Create configuration
config = ModelConfig(
    vocab_size=5000,
    input_dim=128,
    reservoirs=[
        ReservoirConfig(size=4096, leak_rate=0.1),
        ReservoirConfig(size=4096, leak_rate=0.2),
    ],
    ridge_lambda=1e-3
)

# Initialize model
model = RCLLM(config)

# Prepare data
texts = ["your", "training", "texts"]
tokenizer = Tokenizer(vocab_size=5000, level='word')
tokenizer.train(texts)

loader = DataLoader(texts, tokenizer, sequence_length=128)
sequences = loader.get_sequences()

# Train
train_model(model, sequences)

# Generate
prompt_tokens = tokenizer.encode("Hello", add_special_tokens=False)
generated = model.generate(prompt_tokens, max_length=50, temperature=0.8)
generated_text = tokenizer.decode(generated)
print(generated_text)

# Save
model.save("./my_model")

Technical Details

Memory Requirements

Approximate memory usage:

Model parameters: vocab_size × reservoir_size × 4 bytes
Training states: reservoir_size × num_samples × 4 bytes
Reservoir weights: Sparse, ~2% of reservoir_size²

Example: 8K vocab, 4K reservoir, 100K training samples:

Model: 128 MB
Training states: 1.6 GB
Reservoir: ~13 MB (sparse)

Training Time

On modern CPU (approximate):

Configuration	Training Time
Tiny (512)	1-2 minutes
Small (2K)	5-10 minutes
Medium (8K)	20-40 minutes
Large (20K)	1-3 hours

Why No Deep Learning?

This implementation demonstrates that:

Fixed random features can encode temporal patterns
Linear readouts with proper regularization are powerful
Closed-form solutions enable fast training
Reservoir computing provides interpretable dynamics

Trade-offs:

✓ Fast training (no gradient descent)
✓ Mathematically interpretable
✓ No local minima issues
✗ Lower quality than transformers
✗ Limited compositional reasoning
✗ Requires careful hyperparameter tuning

Troubleshooting

Issue: Out of Memory During Training

Solution: Reduce reservoir size or use chunked state collection:

--reservoir-size 2048  # Instead of 4096

Or modify the code to process data in chunks (already implemented in collect_states).

Issue: Poor Generation Quality

Solutions:

Increase reservoir size:
```
--reservoir-size 8192
```
Add more reservoirs:
```
--num-reservoirs 3
```
Tune spectral radius:
```
--spectral-radius 0.95  # More memory
```

Adjust regularization:

--ridge-lambda 1e-4  # Less regularization

Issue: Model Too Large to Save

Solution: Model files are already compressed. For very large models, consider:

Reducing vocabulary with subword tokenization
Using hierarchical softmax
Quantizing readout weights to float16

Issue: Singular Matrix Error

Solution: Increase ridge regularization:

--ridge-lambda 1e-2  # Or higher

Advanced Features

Custom Reservoir Configuration

Edit the model creation in train_rc_llm.py:

reservoirs = [
    ReservoirConfig(
        size=4096,
        spectral_radius=0.95,
        sparsity=0.01,  # More sparse
        input_scaling=0.8,
        leak_rate=0.05,
        activation='tanh',  # or 'sin', 'relu'
        bias_scaling=0.1
    )
]

State Analysis

The evaluation script provides:

Singular value spectrum (effective dimensionality)
Activation statistics (saturation analysis)
Autocorrelation (temporal dependencies)
Reservoir condition number

Online Learning (Future)

The codebase is structured to support online readout adaptation using RLS (Recursive Least Squares). This can be added for personalization without retraining.

Citation

If you use this implementation in research, please cite the original Echo State Network papers:

Jaeger, H. (2001). The "echo state" approach to analysing and training 
recurrent neural networks. GMD Report 148, German National Research 
Center for Information Technology.

License

MIT License - feel free to use and modify.

Future Improvements

Planned enhancements:

Subword tokenization (BPE/SentencePiece integration)
Hierarchical softmax for large vocabularies
Key-value cache for improved coherence
FORCE-style online adaptation
Multi-threaded state collection
GPU acceleration for inference

Name		Name	Last commit message	Last commit date
FILE_STRUCTURE.md		FILE_STRUCTURE.md
FINAL_SUMMARY.md		FINAL_SUMMARY.md
IMPLEMENTATION_SUMMARY.md		IMPLEMENTATION_SUMMARY.md
QA_GUIDE.md		QA_GUIDE.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
all_topics.txt		all_topics.txt
clean_wikipedia.py		clean_wikipedia.py
config.json		config.json
demo_qa.py		demo_qa.py
download_wikipedia.py		download_wikipedia.py
evaluate_rc_llm.py		evaluate_rc_llm.py
example_shakespeare.py		example_shakespeare.py
format_conversational.py		format_conversational.py
inference_rc_llm.py		inference_rc_llm.py
qa_interface.py		qa_interface.py
rc_llm.py		rc_llm.py
requirements.txt		requirements.txt
reservoir_0.pkl		reservoir_0.pkl
reservoir_1.pkl		reservoir_1.pkl
test_rc_llm.py		test_rc_llm.py
tokenizer.json		tokenizer.json
train_rc_llm.py		train_rc_llm.py
train_ultimate.py		train_ultimate.py
training_utils.py		training_utils.py
wikipedia_clean.txt		wikipedia_clean.txt
wikipedia_conversational.txt		wikipedia_conversational.txt
wikipedia_mega.txt		wikipedia_mega.txt
wikipedia_training.txt		wikipedia_training.txt

Latest commit

History

Folders and files

Repository files navigation