CommInsight

CommInsight is an observability and semantic-analysis toolkit for collective communication in distributed AI workloads. It captures communication events inside the training process, preserves raw evidence outside the hot path, and recovers higher-level training semantics such as Steps and communication roles.

The first source release contains four independently usable components:

Component	Purpose
NCCL Profiler	Captures Collective and P2P lifecycle, timing, topology, communicator metadata, Stream IDs, and timeout evidence through the NCCL Profiler ABI.
Agent	Consumes Profiler SHM and FIFO data, exposes runtime APIs and Metrics, runs Hang diagnostics, and archives raw and derived data.
Semantic Engine	Recovers Step boundaries and summaries from communication sequences with template-driven FSMs and resumable checkpoints.
Inject	Passes PyTorch process-group descriptions to NCCL commName without requiring a patched PyTorch build.

The design keeps training safety as the primary boundary: observability must fail open, the Profiler never waits for the Agent, and a failed consumer must not stop the training process or prevent raw inspector_logs from being preserved.

Whitepaper

• CommInsight 产品白皮书 v2.0
• CommInsight Product Whitepaper v2.0

Architecture

Click the image to open the editable draw.io source.

The Server is shown for architectural completeness but is not part of the first source release.

Data flow

1. CommInsight Inject is installed before torchrun starts and propagates <group_desc>:<group_name> into NCCL commName when supported.
2. NCCL loads libnccl-profiler-comminsight.so and reports communication lifecycle events through its Profiler ABI.
3. The Profiler writes bounded real-time data to SHM and durable raw records to segmented inspector_logs. It can start the local Agent automatically.
4. The Agent consumes SHM/FIFO data, generates Metrics and Hang evidence, runs the Semantic Engine, and archives sealed files through a Provider.
5. The Semantic Engine writes per-GPU Step summaries and checkpoints, allowing parsing to resume after process or Pod restarts.

Requirements

• Linux
• Go 1.23 or newer
• Python 3.8 or newer
• A C++11 compiler
• CUDA Toolkit headers under /usr/local/cuda, or CUDA_HOME set explicitly
• NCCL with Profiler Plugin ABI v4 or v5 support

CommInsight depends on CUDA Runtime headers at build time and the CUDA Runtime at run time, both of which must be provided by the user's environment. The CUDA Toolkit is not distributed with this repository. Users who enable CUDA functionality must install it separately and comply with the NVIDIA CUDA Toolkit license terms.

Build

Build the Agent, NCCL Profiler, and Python Inject wheel:

make

Artifacts are written to:

dist/comminsight-agent
dist/comminsight_inject-*.whl
profilers/nccl/libnccl-profiler-comminsight.so

Individual components can be built with:

make build-agent
make build-profiler
make build-inject

Install

The default installation prefix is /opt/comminsight:

sudo make install

Use a different prefix when needed:

sudo make install INSTALL_PREFIX=/path/to/comminsight

make install installs the Agent binary, NCCL Profiler shared library, comminsight.conf, and the Inject wheel into the active Python environment. Make sure this is the same Python environment used to launch the workload.

Run

For the default standalone deployment, enable the NCCL Profiler before starting the workload:

export NCCL_PROFILER_PLUGIN=/opt/comminsight/lib/libnccl-profiler-comminsight.so
torchrun ...

The installed Inject wheel is loaded through a Python .pth file. PyTorch is imported lazily, so the hook does not add an eager torch import during Python startup.

By default, the local-rank-0 Profiler starts comminsight-agent, and the Agent listens on 0.0.0.0:19102. Runtime configuration is loaded from /opt/comminsight/etc/comminsight.conf; exported environment variables take precedence.

Useful runtime paths under the default prefix are:

/opt/comminsight/inspector_logs     raw NCCL records
/opt/comminsight/shms               Profiler-Agent shared memory
/opt/comminsight/fifos              Hang-event transport
/opt/comminsight/semantics/step     Step summaries and checkpoints
/opt/comminsight/metadata           Agent and communicator metadata
/opt/comminsight/archives           default 3FS archive mount
/opt/comminsight/logs               component logs

The default settings are suitable for a single-node demonstration. Multi-node Agent mode and external archive mounts require the corresponding COMMINSIGHT_AGENT_* and COMMINSIGHT_COMMON_ARCHIVE_* settings in config/comminsight.conf.

Verify

make verify-generated
make test
make test-race
make -C profilers/nccl test-mem test-json test-inflight

Protocol-derived files under api/gen and generated NCCL SHM definitions are maintained by tools/codegen/generate.rb. After changing a protocol, regenerate and verify them with:

make generate
make verify-generated

Interfaces and protocols

• Architecture and component interfaces
• Protocol registry
• NCCL SHM ABI v15
• NCCL Inspector Log v4.8
• Semantic Engine ABI v1
• Step Summary v5
• Agent Master API
• Agent Master-Slave API

Repository layout

agent/                 runtime Agent and Profiler adapters
inject/                Python import hook
profilers/nccl/        NCCL Profiler Plugin
semantic-engine/step/  Step semantic recovery
protocols/             versioned ABI, Schema, Protobuf, and OpenAPI sources
api/gen/               generated protocol bindings
contract-tests/        cross-language protocol fixtures
config/                runtime configuration
tools/codegen/         protocol code generation
draw.io/               architecture source

License

CommInsight is licensed under the Apache License 2.0. NCCL-derived files retain their original copyright and license notices. See LICENSE.txt and ThirdPartyNotices.txt for details.