Run on a Cluster#

The benchmark can submit inference and training experiments to a SLURM cluster through BenchOpt’s Submitit backend. This page uses Jean Zay as a concrete example. Adapt module, account, QoS, and GPU names to your own cluster.

The login and compute nodes must be able to access the cloned repository and the Python environment used to install it.

Load the Environment#

Start by loading the cluster-provided Python and GPU software stack. The checked-in parallel configuration currently uses this Jean Zay module:

module purge
module load pytorch-gpu/py3/2.7.0

This module provides Python and a GPU-enabled PyTorch installation. Module names change over time, so use the version recommended by the cluster when the example above is no longer available.

Create a Virtual Environment#

A virtual environment is optional, but it keeps benchmark dependencies isolated from other projects. Using --system-site-packages makes the PyTorch stack provided by the loaded module visible inside the environment:

python -m venv --system-site-packages benchmark_env
source benchmark_env/bin/activate

Create the environment on a shared filesystem. On later sessions, load the same module before activating it:

module purge
module load pytorch-gpu/py3/2.7.0
source /path/to/benchmark_env/bin/activate

If you use the module environment directly, omit the virtual-environment commands and install into an appropriate writable Python environment according to your cluster policy.

Clone and Install the Benchmark#

Clone the repository, enter it, and install the project with pip:

git clone https://github.com/numpex/benchmark_invprob_largescale.git
cd benchmark_invprob_largescale
python -m pip install .

For the optional radio interferometry dependencies, install the corresponding extra instead:

python -m pip install '.[radio]'

Check that the active environment can discover both benchmarks:

benchopt info benchmark_inference/.
benchopt info benchmark_training/.

Configure SLURM#

Edit the parallel configuration for the benchmark you want to run:

  • benchmark_inference/configs/config_parallel.yml;

  • benchmark_training/configs/config_parallel.yml; or

  • benchmark_training/configs/config_parallel_nsys.yml for an Nsight Systems profiling run.

At minimum, review account, qos, constraint, slurm_time, and slurm_setup. The setup commands execute inside every submitted job. They must load the same software stack used during installation and, when applicable, activate the shared virtual environment:

backend: submitit
slurm_time: 1800
slurm_python: python
slurm_additional_parameters:
  cpus-per-task: 10
  qos: qos_gpu-dev
  account: your_account
  constraint: v100-32g
slurm_setup:
  - module purge
  - module load pytorch-gpu/py3/2.7.0
  - source /path/to/benchmark_env/bin/activate
  - export NCCL_DEBUG=INFO

If you did not create a virtual environment, remove its source line. slurm_python must resolve to an interpreter that can import benchopt, toolsbench, DeepInv, and the remaining project dependencies on the compute nodes.

The experiment YAML separately specifies resources for each solver run through slurm_gres, slurm_ntasks_per_node, and slurm_nodes. See Configuration Guide for the full configuration model.

Submit an Experiment#

From the repository root, submit the multi-frame super-resolution example with:

benchopt run benchmark_inference/. \
    --parallel-config benchmark_inference/configs/config_parallel.yml \
    --config benchmark_inference/configs/examples/multiframe_superres.yml \

benchopt run benchmark_training/. \
    --parallel-config benchmark_training/configs/config_parallel.yml \
    --config benchmark_training/configs/synthetic_unrolled.yml \

BenchOpt expands the configured dataset and solver grids, Submitit creates the corresponding SLURM jobs, and each distributed solver starts the requested processes across its assigned GPUs and nodes. Submission files and logs are written below the benchmark’s benchopt_run/ directory.

Collect Results#

Completed benchmark tables and reports are stored in the selected benchmark’s outputs/ directory. Filenames are timestamped by default, for example:

benchmark_inference/outputs/benchopt_run_<timestamp>.parquet
benchmark_inference/outputs/benchmark_inference_benchopt_run_<timestamp>.html

The HTML report contains convergence and quality curves together with the timing and resource metrics returned by the benchmark. Solver profiling may also create per-rank metric files in outputs/.