Comprehensive Benchmark for PyTorch backend

[refraction-ray]

Description

With the PyTorch backend now supporting jit (see issue #4), it is crucial to establish a comprehensive benchmark suite. This suite will be used to evaluate and compare the performance of the PyTorch backend agaisnt TensorFlow, and JAX backends across different hardware environments.

The goal is to measure and analyze both compilation and execution times for a variety of representative workloads, as these can vary significantly depending on the problem size, ML backend, and hardware (CPU vs. GPU).

Tasks

1. Benchmark Suite Development

• Develop a suite of benchmarks for the following components:
  • Circuit
  • DMCircuit
  • MPScircuit
  • AD
  • VMAP
  • all with jit
• Ensure benchmarks are sufficiently large to yield meaningful and stable performance measurements, cotengra contractor is often required for consistent performance.
• Parameterize benchmarks to allow for scaling of problem size.

2. Backend and Hardware Testing Matrix

• Configure the benchmark runner to execute across the following configurations:
  • Backends:
    • PyTorch
    • TensorFlow
    • JAX
  • Hardware:
    • CPU
    • GPU

3. Reporting and Analysis

• Log and report both compilation time and execution time for each test case.
• Analyze the results to identify performance bottlenecks and areas for optimization for each backend.

Acceptance Criteria

• A set of benchmark scripts are created and added potentially to the repository.
• The benchmarks can be executed against all three backends (PyTorch, TensorFlow, JAX) and both devices.
• A clear and reproducible report of the results is generated, comparing the performance of the different backends.

Additional Context

This benchmark suite will be a critical tool for ongoing performance monitoring and will help guide future development and optimization efforts. It will also provide valuable information to users, helping them choose the best backend for their specific use cases and hardware.

[Aditya-132]

hi can i work on this issue

[ninjaduck7]

Hi, I'd like to contribute to this issue.

I have experience with CUDA, Triton, TileLang, and GPU performance profiling, so I'm particularly interested in the GPU/backend benchmarking part.

I noticed that someone previously expressed interest in this issue. Is this still available, or is there a particular subset you would prefer me to take?

I could start with a small PyTorch GPU benchmark PR first and expand from there.

[refraction-ray]

@ninjaduck7 , This benchmark is somehow previously added in https://github.com/tensorcircuit/tensorcircuit-ng/tree/master/benchmarks/scripts_v2, if you are interested in tensorcircuit-ng, you can explore other issues especially reproduced paper series, actually you can bring any paper you are interested with TCNG implementation, and it will be shown at: https://tensorcircuit-ng.readthedocs.io/en/latest/reproduce/index.html

[ninjaduck7]

@ninjaduck7 , This benchmark is somehow previously added in https://github.com/tensorcircuit/tensorcircuit-ng/tree/master/benchmarks/scripts_v2, if you are interested in tensorcircuit-ng, you can explore other issues especially reproduced paper series, actually you can bring any paper you are interested with TCNG implementation, and it will be shown at: https://tensorcircuit-ng.readthedocs.io/en/latest/reproduce/index.html

Cool, I'll give it a try! Actually, I've been wanting to revisit some topics from my quantum information course and experiment with them using the PyTorch techniques I've learned recently. TCNG seems like a great place to do that.

[refraction-ray]

great, please feel free to go ahead. and a suggestion for reproducing papers using pytorch backend, you may want to compare it with jax backend, where the latter in general has better speed