Symbolic Arithmetic: Language Models are Symbolic Learners in Arithmetic

This repository contains the official implementation of "Language Models are Symbolic Learners in Arithmetic", a framework for understanding how language models learn arithmetic through subgroup induction.

Overview

This project investigates a fundamental question in AI: Do language models truly learn to compute, or do they simply master sophisticated pattern matching? We present compelling evidence that LMs act as greedy symbolic learners, prioritizing the simplest possible "shortcuts" to solve arithmetic tasks.

Key Contributions

• Subgroup Induction Framework: A practical adaptation of Solomonoff Induction for analyzing arithmetic learning in LMs
• Subgroup Quality & Entropy: Novel metrics to quantify the viability of symbolic shortcuts
• U-Shaped Learning Pattern: Discovery that LMs master edge digits first while struggling with middle positions
• CoT Analysis: Extension to multi-step reasoning, showing that lower entropy paths achieve better performance

Installation

Prerequisites

• Python 3.9+
• CUDA-compatible GPU (recommended for training)
• 16GB+ GPU memory (for model training)

Quick Setup

# Clone the repository
git clone https://github.com/yourusername/Symbolic-Arithmetic.git
cd Symbolic-Arithmetic

# Create virtual environment (optional but recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install the package
pip install -e .

# For GPU-accelerated data generation (optional)
pip install taichi

# For development
pip install -e ".[dev]"

Installation from Requirements

Alternatively, install dependencies directly:

pip install -r requirements.txt

Quick Start

1. Generate Data

Generate arithmetic datasets for experiments:

python experiments/generate_data.py \
    --operation multiplication \
    --digit-length 3 \
    --num-train 32400 \
    --num-dev 4050 \
    --num-test 4050 \
    --output-dir data

Options:

• --operation: addition or multiplication
• --digit-length: Number of digits (e.g., 3 for 3×3)
• --use-gpu: Enable GPU acceleration (requires Taichi)
• --seed: Random seed for reproducibility

2. Analyze Subgroup Patterns

Analyze the U-shaped accuracy pattern and subgroup quality:

python experiments/analyze_subgroups.py \
    --data-dir data \
    --operation multiplication \
    --digit-length 3 \
    --token-budgets 2 4 6 8 \
    --output-dir results/subgroups \
    --plot

This will generate:

• CSV files with subgroup quality and entropy metrics
• Visualization plots showing the U-shaped pattern

3. Train a Model

Train a language model on arithmetic tasks:

python experiments/train_model.py \
    --config configs/default.yaml \
    --model-name gemma-2-2b \
    --data-dir data

Supported Models:

• gpt2-xl: GPT-2 XL (1.5B parameters)
• gemma-2-2b: Google Gemma 2 (2B parameters)
• llama-3.1-8b: Meta Llama 3.1 (8B parameters)

Training Configuration:

Edit configs/default.yaml or create a custom config file:

model:
  model_name: "gemma-2-2b"
  use_lora: true
  learning_rate: 3.0e-4
  num_epochs: 12
  batch_size: 16

data:
  operation: "multiplication"
  digit_length: 3
  num_train: 32400

4. Evaluate a Trained Model

Evaluate model performance on test data:

python experiments/evaluate_model.py \
    --checkpoint checkpoints/final \
    --data-dir data \
    --operation multiplication \
    --digit-length 3 \
    --split test \
    --output-file results/predictions.csv

This will output:

• Exact match accuracy
• Digit-level accuracy
• Position-wise accuracy (revealing the U-shape)
• Error analysis

Usage Examples

Python API

from symbolic_arithmetic.data import create_generator
from symbolic_arithmetic.metrics import SubgroupAnalyzer

# Generate data
generator = create_generator(
    operation="multiplication",
    digit_length=3,
    add_spaces=True,
    seed=42
)
dataset = generator.generate_dataset(num_samples=1000)

# Analyze subgroup quality
analyzer = SubgroupAnalyzer(operation="multiplication", digit_length=3)

# Compute quality for a specific subgroup
# ([1], [1], 0) means: A2 × B2 -> C1 (units digit)
quality = analyzer.compute_subgroup_quality(dataset, ([1], [1], 0))
print(f"Quality: {quality:.4f}")

# Analyze U-shaped pattern
u_shape_df = analyzer.analyze_u_shape(dataset, token_budgets=[2, 4, 6, 8])
print(u_shape_df)

# Compute entropy for all output positions
entropies = analyzer.compute_all_entropies(dataset)
for pos, entropy in sorted(entropies.items()):
    print(f"Position {pos}: {entropy:.4f} bits")

Training with Custom Configuration

from symbolic_arithmetic.models import ArithmeticTrainer
from symbolic_arithmetic.data import load_and_prepare_data
from symbolic_arithmetic.utils.config import setup_seed

setup_seed(42)

# Initialize trainer
trainer = ArithmeticTrainer(
    model_name="gemma-2-2b",
    use_lora=True,
    use_unsloth=True,
    lora_config={"lora_r": 8, "lora_alpha": 16}
)

# Load data
datasets = load_and_prepare_data(
    data_dir="data",
    tokenizer=trainer.tokenizer,
    operation="multiplication",
    digit_length=3
)

# Train
trainer.train(
    train_dataset=datasets["train"],
    eval_dataset=datasets["validation"],
    training_config={
        "output_dir": "checkpoints",
        "learning_rate": 3e-4,
        "num_epochs": 12,
        "batch_size": 16,
    }
)

# Save
trainer.save("checkpoints/final")

Reproducing Paper Results

To reproduce the main experiments from the paper:

Experiment 1: U-Shaped Accuracy Pattern (Section 3.2)

# Generate 3-digit multiplication data (multiple sizes)
for size in 6480 12960 32400 64800; do
    python experiments/generate_data.py \
        --operation multiplication \
        --digit-length 3 \
        --num-train $size \
        --output-dir data/3digit_${size}
done

# Train Gemma-2-2B on different dataset sizes
for size in 6480 12960 32400 64800; do
    python experiments/train_model.py \
        --model-name gemma-2-2b \
        --data-dir data/3digit_${size} \
        --output-dir checkpoints/gemma_3digit_${size}
done

# Evaluate and analyze position-level accuracy
for size in 6480 12960 32400 64800; do
    python experiments/evaluate_model.py \
        --checkpoint checkpoints/gemma_3digit_${size}/final \
        --data-dir data/3digit_${size} \
        --output-file results/gemma_3digit_${size}_predictions.csv
done

Experiment 2: Subgroup Quality Analysis (Section 3.3)

# Analyze subgroup quality for 2-5 digit multiplication
for digits in 2 3 4 5; do
    python experiments/analyze_subgroups.py \
        --data-dir data/${digits}digit \
        --operation multiplication \
        --digit-length $digits \
        --token-budgets 2 4 6 8 \
        --output-dir results/subgroups/${digits}digit \
        --plot
done

Experiment 3: Subgroup Entropy and CoT (Section 4.2)

See notebooks/cot_analysis.ipynb for detailed Chain-of-Thought experiments.

Project Structure

Symbolic-Arithmetic/
├── symbolic_arithmetic/          # Main package
│   ├── data/                     # Data generation and loading
│   │   ├── generators.py         # Arithmetic data generators
│   │   └── loaders.py            # Dataset loaders
│   ├── models/                   # Model training and evaluation
│   │   ├── trainer.py            # Training utilities
│   │   └── evaluator.py          # Evaluation utilities
│   ├── metrics/                  # Evaluation metrics
│   │   ├── subgroup.py           # Subgroup quality & entropy
│   │   └── accuracy.py           # Accuracy metrics
│   └── utils/                    # Utilities
│       └── config.py             # Configuration classes
├── experiments/                  # Experiment scripts
│   ├── generate_data.py          # Data generation script
│   ├── train_model.py            # Training script
│   ├── evaluate_model.py         # Evaluation script
│   └── analyze_subgroups.py      # Subgroup analysis script
├── configs/                      # Configuration files
│   └── default.yaml              # Default configuration
├── notebooks/                    # Jupyter notebooks
│   └── quickstart.ipynb          # Quick start guide
├── tests/                        # Unit tests
├── requirements.txt              # Dependencies
├── setup.py                      # Package setup
└── README.md                     # This file

Key Concepts

Subgroup Induction

A subgroup s = ((A, B), C) represents a minimal mapping from input digit positions A and B to an output digit position C.

Example: For 32 × 61 = 1952:

• Subgroup ([1], [1], 0) = (A₂, B₂, C₁) = (2, 1, 2)
• This subgroup checks if the units digit can be predicted from just the units digits of the inputs

Subgroup Quality Q(s)

Measures how accurately a subgroup predicts its output digit across the dataset:

Q(s) = E[(f(φ(a,A), φ(b,B)) = φ(c,C))]

where φ is a masking function that keeps only specified positions.

Subgroup Entropy H(s)

Measures the uncertainty in predicting an output digit:

H(s) = -Σ P(C=c) log₂ P(C=c)

Lower entropy indicates easier prediction.

U-Shaped Pattern

LMs show high accuracy on edge digits (first and last) but struggle with middle digits, forming a U-shape. This pattern mirrors subgroup quality: edge positions have high-quality, low-token shortcuts, while middle positions require more input tokens.

Citation

If you use this code or find our work helpful, please cite:

@article{symbolic2024,
  title={Language Models are Symbolic Learners in Arithmetic},
  author={Anonymous},
  journal={Under review at TMLR},
  year={2024}
}

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development

Running Tests

pytest tests/

Code Formatting

black symbolic_arithmetic/ experiments/
flake8 symbolic_arithmetic/ experiments/

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Built with HuggingFace Transformers
• Uses Unsloth for efficient training
• Inspired by Solomonoff Induction and Occam's Razor principles

Contact

For questions or issues, please open a GitHub issue or contact the authors.

---

Keywords: Language Models, Arithmetic Reasoning, Solomonoff Induction, Symbolic Learning, Pattern Matching, Transformer Analysis