Troubleshooting

This page collects the failures most often seen when running the benchmark on a cluster, with the checks that usually resolve them. It assumes the setup in Run on a Cluster.

The Job Hangs

The submission is accepted but nothing progresses: no reconstruction metrics appear, the log stops after the banner, and waiting several minutes changes nothing. The common causes are missing preparation and a job that is not actually running.

Datasets were not prepared. BenchOpt calls each dataset’s prepare() before the run, which is where reusable inputs are downloaded, simulated, and cached. Compute nodes may not have internet access, so a first-time download launched from inside the job blocks indefinitely. Prepare the data from a login node, which does have network access, before submitting:

benchopt prepare benchmark_inference/. --config <experiment.yml>

Preparation is cached by BenchOpt, so this is a one-time cost per dataset. See Use Cases for per-dataset preparation details.

Denoiser weights were not cached. Pretrained networks weights are downloaded from the internet on first use (pretrained="download"), which blocks on an offline compute node the same way an unprepared dataset does. benchopt prepare caches these weights into the shared torch hub cache (~/.cache/torch/hub/checkpoints/) alongside the dataset inputs, so running it from a login node covers both.

The job is not actually running. A “hang” is often a job still sitting in the queue or already failed at submission. Check its state:

squeue --me

If the job is pending (PD) with a reason like (Resources) or (Priority), the requested resources are simply not free yet. If it never appears at all, the submission was rejected; if it stays PD with a QoS reason such as (QOSMaxJobsPerUserLimit) or (QOSMaxWallDurationPerJobLimit) in the reason column, it was accepted but capped by the QoS. Each QoS caps the number of simultaneous or queued jobs and the maximum wall time, and a development QoS is far more restrictive than a production one. Requesting more jobs than the QoS allows, or a slurm_time longer than the QoS permits, is refused. Review account, qos, and slurm_time in the parallel configuration against your cluster’s QoS table, and inspect the Submitit log files under the benchmark’s benchopt_run/ directory for the rejection message.

The Job Runs but Stops Early

The run starts and produces some results, then ends before completing every configured solver or all max-runs iterations. Two independent time limits can cause this.

BenchOpt per-solver timeout. BenchOpt stops a solver once a single run exceeds its --timeout, which defaults to 100 seconds. A large image, a slow forward operator, or many iterations can cross this limit and the solver is recorded as timed out rather than failed. Raise or remove the limit explicitly:

benchopt run benchmark_inference/. --config <experiment.yml> --timeout 30m
# or, to disable it entirely:
benchopt run benchmark_inference/. --config <experiment.yml> --no-timeout

SLURM wall time. The job itself is killed when it exceeds slurm_time in the parallel configuration (the checked-in examples use 1800 seconds). Increase it for long experiments, keeping it within what the selected QoS allows.

In both cases the cause is stated in the log: check the Submitit output under benchopt_run/ for a timeout notice or a SLURM TIMEOUT / CANCELLED line.