RINGS integration for existing GNN pipelines

.github/workflows/ci.yml

@@ -0,0 +1,64 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Check package version matches __init__
+        run: |
+          python - <<'PY'
+          from pathlib import Path
+          import re
+          import sys
+          import tomllib
+
+          pyproject_version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
+          init_text = Path("rings/__init__.py").read_text()
+          init_version = re.search(r'^__version__ = "([^"]+)"$', init_text, re.MULTILINE)
+
+          if init_version is None:
+              sys.exit("Could not find __version__ in rings/__init__.py")
+
+          if pyproject_version != init_version.group(1):
+              sys.exit(
+                  f"Version mismatch: pyproject.toml has {pyproject_version}, "
+                  f"rings/__init__.py has {init_version.group(1)}"
+              )
+          PY
+
+
+      - name: Install dependencies
+        run: uv sync --dev --python ${{ matrix.python-version }}
+
+      - name: Run Python tests
+        run: uv run pytest -v
+
+      - name: Run Ruff lint checks
+        if: matrix.python-version == '3.13'
+        run: uv run ruff check rings examples tests
+
+      - name: Run Ruff formatting checks
+        if: matrix.python-version == '3.13'
+        run: uv run ruff format --check rings examples tests

.github/workflows/pages.yml

@@ -4,6 +4,9 @@ on:
   push:
     branches:
       - main
+  pull_request:
+    branches:
+      - main
 
 # security: restrict permissions for CI jobs.
 permissions:
@@ -15,13 +18,22 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
         with:
           python-version: "3.13"
 
-      # Install dependencies and documentation tools
-      - run: python -m pip install -r requirements.txt
-      - run: sphinx-build -M html docs/source docs/build
+      - name: Install dependencies
+        run: uv sync --dev --group docs --python 3.13
+
+      - name: Build documentation
+        run: uv run sphinx-build -M html docs/source docs/build
 
       - uses: actions/upload-pages-artifact@v3
         with:
@@ -30,6 +42,7 @@ jobs:
   # Deploy the artifact to GitHub pages.
   # This is a separate job so that only actions/deploy-pages has the necessary permissions.
   deploy:
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
     needs: build
     runs-on: ubuntu-latest
     permissions:

.gitignore

@@ -1,3 +1,8 @@
+# Dataset files
+data/
+checkpoints/
+
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
@@ -7,6 +12,7 @@ __pycache__/
 *.so
 
 # Distribution / packaging
+uv.lock
 .Python
 build/
 develop-eggs/

README.md

@@ -1,181 +1,113 @@
 # RINGS
 
-This is the official repository for our ICML paper:
+**[No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets](https://arxiv.org/abs/2502.02379)** — ICML 2025.
 
-**[No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets](https://arxiv.org/abs/2502.02379)**
-
-which introduces **RINGS**: a perturbation framework for attributed graphs, designed to enable more principled evaluations of graph-learning benchmarks from first principles.
-
----
-
-## 🚧 Repository Status
-
-This repository is **under active development**.  
-We’re making it public early to invite feedback, discussion, and **transparency** as we move from research prototypes to a stable, user-friendly package.
-
-In the coming weeks, we’ll release updates, architectural notes, and implementation details via a series of pull requests. You're welcome to follow along, open issues, or suggest improvements!
-
----
-
-## 🚀 Current MVP Release
-
-This initial Minimum Viable Product (MVP) includes:
-
-- A set of **graph perturbation transformations** for manipulating node features and graph structure
-- The **SeparabilityFunctor**, which enables statistical comparisons between distributions
-- The **ComplementarityFunctor**, which computes mode complementarity between node features and graph structure
-- Example scripts demonstrating usage with PyTorch Geometric datasets and toy performance distributions
+RINGS is a perturbation framework for attributed graphs that lets you evaluate graph-learning datasets and models from first principles: apply structured perturbations, train as usual, and compare performance distributions with statistically rigorous tests.
 
 ---
 
-## 💍 Framework Overview
+## Install
 
-We are developing a community-friendly implementation of the **RINGS** framework introduced in the paper. Our goal is to make it easy for the graph-learning community to:
-
-- Apply dataset perturbations tailored to graph-learning datasets
-- Conduct more rigorous and insightful evaluations of both datasets and models
-- Promote better dataset practices and evaluation hygiene across the field
-
-If you have feedback on the paper or suggestions for how this package could better integrate with popular frameworks, please feel free to reach out to the authors.
-
----
-
-## 📦 Installation
-
-RINGS uses [uv](https://github.com/astral-sh/uv) as the package manager, which provides faster dependency resolution and installation.
-
-### Prerequisites
-
-Install `uv` if you don’t have it already:
+Install from [PyPI](https://pypi.org/project/rings-evaluation/):
 
 ```bash
-pip install uv
+pip install rings-evaluation
 ```
 
-### Installing RINGS
+Requires Python 3.11+.
 
-Clone the repository and install dependencies using `uv`:
+### From source
 
-```bash
-# Clone the repository
-git clone https://github.com/aidos-lab/rings.git
-cd rings
+To contribute or run the examples in this repo:
 
-# Install dependencies
-uv sync
-
-# Activate environment
-source .venv/bin/activate
+```bash
+pip install uv
+git clone https://github.com/aidos-lab/rings.git && cd rings
+uv sync && source .venv/bin/activate
 ```
 
 ---
 
-## 🔑 Key Components
-
-### Mode Perturbations
-
-RINGS provides several perturbation transforms that can be applied to graph datasets:
-
-**Node Feature Perturbations:**
+## Quickstart 
 
-- `Original`: Keeps node features unchanged (baseline)
-- `EmptyFeatures`: Replaces node features with zero vectors
-- `RandomFeatures`: Replaces node features with random values
-- `CompleteFeatures`: Assigns unique one-hot vectors to nodes
+> Drop RINGS evaluations into your GNN pipeline.
 
-**Graph Structure Perturbations:**
+Keep your training loop. Wrap it with `SeparabilityStudy` to iterate perturbation × seed, record one scalar per run from *your* evaluator, and get a pairwise separability table back.
 
-- `EmptyGraph`: Removes all edges from the graph
-- `CompleteGraph`: Connects every pair of nodes
-- `RandomGraph`: Generates a random graph structure
-- `RandomConnectedGraph`: Generates a random graph that is guaranteed to be connected
+```python
+from rings import Original, EmptyGraph, RandomFeatures, CompleteFeatures
+from rings.integrations import SeparabilityStudy
 
-### SeparabilityFunctor
+study = SeparabilityStudy(
+    perturbations={
+        "Original":         Original(),
+        "EmptyGraph":       EmptyGraph(),
+        "RandomFeatures":   RandomFeatures(shuffle=True),
+        "CompleteFeatures": CompleteFeatures(max_nodes=max_nodes),
+    },
+    num_seeds=5,
+    comparator="ks",   # or "wilcoxon"
+    alpha=0.05,
+)
 
-The `SeparabilityFunctor` computes statistically rigorous comparisons between multiple distributions to determine if they differ significantly. This is useful for:
+for name, transform, seed in study.runs():
+    perturbed = study.apply(base_dataset, transform)
+    score = train_and_eval(perturbed, seed=seed)   # your code
+    study.record(name, score)
 
-- Evaluating whether different graph perturbations produce statistically distinct model performances
-- Identifying which perturbations most impact model behavior
-- Making rigorous, statistically valid claims about distribution separability
-
-It employs statistical tests with permutation testing and built-in correction for multiple hypotheses (Bonferroni correction).
-
-Available comparators include:
-
-- `KSComparator`: Kolmogorov–Smirnov test for comparing distributions
-- `WilcoxonComparator`: Wilcoxon signed-rank test for paired comparisons
-
-### ComplementarityFunctor
-
-The `ComplementarityFunctor` measures the alignment between node features and graph structure by comparing their induced metric spaces. It can help you understand:
-
-- Whether node features and graph structure contain complementary information
-- How different perturbations affect this complementarity
-- The distribution of information content across modalities in graph datasets
-
----
-
-## 🔍 Example Usage
-
-The repository includes example scripts that demonstrate how to use RINGS to analyze graph datasets.
-
-### Performance Separability
-
-Measure distances between performance distributions to test whether the original dataset statistically outperforms perturbed versions.
-
-```bash
-# Run a basic separability analysis with the default KS comparator
-python -m examples.separability --comparator ks --alpha 0.05
+results = study.evaluate(n_permutations=1000)
+# DataFrame: mode1, mode2, score, pvalue_adjusted, significant
 ```
 
-```bash
-# Use the Wilcoxon test comparator
-python -m examples.separability --comparator wilcoxon --alpha 0.01
-```
+**PyTorch Lightning** — attach `SeparabilityCallback` to your `Trainer` and it records the logged `test_acc` automatically:
 
-```bash
-# Get help and see all available options
-python -m examples.separability --help
-```
+```python
+import pytorch_lightning as pl
+from rings.integrations import SeparabilityStudy, SeparabilityCallback
 
-The script analyzes and compares distributions from synthetic data, showing how to determine if differences are statistically significant.
+for name, transform, seed in study.runs():
+    pl.seed_everything(seed, workers=True)
+    dm = make_datamodule(study.apply(base_dataset, transform), seed=seed)
+    trainer = pl.Trainer(
+        max_epochs=20,
+        callbacks=[SeparabilityCallback(study, perturbation_name=name)],
+    )
+    trainer.fit(model, datamodule=dm)
+    trainer.test(model, datamodule=dm)
 
----
+results = study.evaluate()
+```
 
-### Mode Complementarity
+**Custom evaluator** (GraphBench, OGB, anything that returns a scalar): just pass the number to `study.record(name, score)`.
 
-Assess the complementarity of geometric information in the metric spaces of node features and graph structure.
+### Runnable examples
 
 ```bash
-# Run the example on the MUTAG dataset with original (unperturbed) graphs
-python -m examples.complementarity --dataset MUTAG --perturbation original
+uv run -m examples.integrations.pyg
+uv run --with lightning -m examples.integrations.lightning
+uv run --with graphbench-lib -m examples.integrations.graphbench
 ```
 
-```bash
-# Try different perturbations
-python -m examples.complementarity --dataset MUTAG --perturbation random-features
-```
+---
 
-```bash
-# Get help and see all available options
-python -m examples.complementarity --help
-```
+## Learn more
 
-The script outputs complementarity statistics that measure how well node features align with graph structure in the dataset.
+- **[Perturbations](https://aidos-lab.github.io/rings/perturbations.html)** — node-feature and graph-structure transforms (`Original`, `EmptyGraph`, `RandomFeatures`, `CompleteFeatures`, `RandomConnectedGraph`, …)
+- **[SeparabilityFunctor](https://aidos-lab.github.io/rings/separability/functor.html)** — pairwise distribution tests (KS, Wilcoxon) with permutation p-values and Bonferroni correction
+- **[ComplementarityFunctor](https://aidos-lab.github.io/rings/complementarity/functor.html)** — metric-space alignment between node features and graph structure
+- **[Integrations API](https://aidos-lab.github.io/rings/integrations.html)** — full reference for `SeparabilityStudy` and `SeparabilityCallback`
+- **`examples/`** — end-to-end scripts for separability, complementarity, and the three integration recipes
 
-## 📚 Citation
+---
 
-If you use RINGS in your research, please cite our paper:
+## Citation
 
 ```bibtex
-@inproceedings{
-coupette2025metric,
-title={No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets},
-author={Corinna Coupette and Jeremy Wayland and Emily Simons and Bastian Rieck},
-booktitle={Forty-second International Conference on Machine Learning},
-year={2025},
-url={https://openreview.net/forum?id=XbmBNwrfG5}
+@inproceedings{coupette2025metric,
+  title     = {No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets},
+  author    = {Corinna Coupette and Jeremy Wayland and Emily Simons and Bastian Rieck},
+  booktitle = {Forty-second International Conference on Machine Learning},
+  year      = {2025},
+  url       = {https://openreview.net/forum?id=XbmBNwrfG5}
 }
 ```
-
-Stay tuned for more examples and documentation!