Efficient attention modules for xLLM.
xattn provides efficient attention modules for the xLLM framework, with Python APIs for use in other PyTorch projects.
| Feature | Capabilities |
|---|---|
| Attention precision | Five precision correction levels, with unchanged output and gradient dtypes. |
| Flexible segment-aware attention masking | Isolates packed sequences using segment IDs or BOS masks. |
Select an attention type for API examples, tensor layouts, and standalone forward/backward usage.
| Attention |
|---|
| Causal Flash Attention |
| Sliding Window Attention |
| Sliding Chunk Attention |
| SoftDelta Attention |
| Sliding Window SoftDelta Attention |
| Sliding Chunk SoftDelta Attention |
All native attention paths accept FP16/BF16 inputs and support MHA, GQA, and MQA.
Source builds require Linux, Python 3.10+, PyTorch 2.10+ with CUDA support, CUDA Toolkit 12.8+
(including nvcc), and a compatible host compiler. Set CUDA_HOME to the
selected toolkit. Its CUDA major version must match torch.version.cuda;
using the same major/minor version is recommended. PyTorch wheels do not
include the complete build toolkit. Install a PyTorch CUDA wheel matching your
toolkit before the commands below. For CUDA 12.8, for example:
python -m pip install "torch>=2.10" --index-url https://download.pytorch.org/whl/cu128git clone --recurse-submodules https://github.com/ifm-ai/xattn.git
cd xattn
python -m pip install -r requirements-build.txt
python -m pip install --no-build-isolation .python -m pip install "setuptools>=64" wheel ninja "torch>=2.10"
python -m pip install --no-build-isolation "xattn @ git+ssh://git@github.com/ifm-ai/xattn.git"Build configuration
For an existing checkout, run git submodule update --init --recursive before
building.
By default, the build targets visible GPUs; with no visible GPU it targets SM80.
Set XATTN_TARGET_SM=90a for a headless SM90 build, or
XATTN_TARGET_SM=80,86,89,90a for all supported targets. SM90 builds on an SM90
system include the SM90 kernels by default. SM100 and newer architectures
are currently unsupported.
Set MAX_JOBS to limit concurrent compilation jobs and XATTN_NVCC_THREADS
to set nvcc threads per job (default: 4).
Some Conda toolkits place CUDA headers under
$CUDA_HOME/targets/x86_64-linux/include. If $CUDA_HOME/include does not
contain cuda_runtime.h, set CUDA_INC_PATH to that target include directory
when building.
The build selects the default C++ standard from the installed PyTorch version:
| PyTorch version | Default C++ standard |
|---|---|
| 2.10 / 2.11 | C++17 |
| 2.12+ | C++20 |
Set XATTN_CXX_STANDARD=17 or 20 to override the default, for example:
XATTN_CXX_STANDARD=20 python -m pip install --no-build-isolation .The build rejects an explicit C++17 selection with PyTorch 2.14+. Both host and CUDA compilation use the selected standard.
Rebuild xattn after changing PyTorch, the toolkit, or the C++ standard; installed binaries are not promised to work across PyTorch versions.
xattn is licensed under the Apache License 2.0. Third-party components retain their licenses; see third-party notices.