Communication under the hood: transfer vs wait

Splitting a computation across many GPUs means the GPUs cannot work in complete isolation. At some point they need to exchange information — a gradient, an updated tensor, a piece of an image — before they can continue. That exchange is what we call communication, and this page looks inside it: not to ask how much communication there is, but what that communication is actually made of. That distinction is what lets you tell why adding more GPUs to a given section keeps paying off, slows down, or stops helping altogether — and, when it stops, whether the fix is rebalancing work across GPUs or something inherent to how much data has to move, rather than just knowing “communication is expensive” without knowing which problem you actually have.

What do we mean by “communication,” concretely? Every GPU-to-GPU exchange in this benchmark happens through a collective operation — specifically an all_reduce: every participating rank contributes its own tensor, the collective combines the contributions (by summing them), and every rank ends up with the same combined result. None of the actual workload — the neural network, the physics operator — happens inside that operation; it exists purely to keep every GPU’s copy of the data consistent. So the time spent on one section of one iteration splits cleanly in two:

section time = compute + communication

compute is the useful work — the forward pass, the gradient evaluation — and it is the part we are trying to speed up by adding more GPUs. communication is everything else: the price paid for keeping those GPUs coordinated.

Communication itself is not one thing. It splits again, into transfer and wait:

communication = transfer + wait

• transfer is the real data movement — the actual bytes of a tensor travelling over NVLink or the network from one GPU to another. This part is genuinely necessary: the collective cannot produce a correct result without moving the data, and it would still happen even in a perfectly balanced, perfectly synchronized run.
• wait is idle time — a GPU sitting inside the collective operation doing nothing, because the operation cannot finish until every participating GPU has arrived at it. wait has up to three distinct causes, worth telling apart:
  1. Uneven work. If the computation is split unevenly across GPUs, the GPU given less work finishes its compute earlier, then simply sits idle at the collective until its slower neighbor catches up — for instance, one GPU getting a different number of denoiser tiles, or a different number of physics operators, than another. More on this in “Are the GPUs actually staying balanced?” below.
  2. Same work, different finishing time. Even when every GPU is handed an identical job, they do not necessarily finish at exactly the same wall-clock moment. Small hardware and system effects — scheduling jitter, memory contention, clock variation — mean any one GPU can randomly lag behind on a given iteration, with no actual imbalance in what it was asked to do.
  3. Different starting point. A GPU that enters the section already behind — because it was still catching up at the end of the previous section — carries that lag into this collective too, even if its own work here is perfectly balanced. Not a factor in the runs measured on this page: every iteration is closed out with a barrier, so every GPU starts each section from the same aligned point regardless of how the previous one went.

These causes all produce the same symptom — a GPU parked inside a collective, waiting — but they point to different fixes: the first is a workload assignment problem, the second is closer to irreducible system noise, and the third is a synchronization question that this benchmark’s barrier already rules out. What follows can tell you how much wait there is, separately from transfer, which is already far more than the profiler’s raw, fused number gives you.

Put together, one section’s time now has three additive parts:

section time = compute + transfer + wait

This split matters because reading the fused transfer + wait number as “communication cost” can be badly wrong. In one of the runs measured below, 3D denoising looked like 0.87s of communication at 2 GPUs — but the data actually being moved cost only 0.05s. The other 0.82s was one GPU waiting for the other, not moving a single byte.

This page reruns the measurement for both inference and training with the collective time properly split into compute / transfer / wait, to answer what that growing share is actually made of.

1. How does communication cost grow as GPU count increases, and what’s actually driving that growth — pure data movement, or one GPU waiting on another?
2. Are the GPUs actually staying balanced, or is idle time quietly eating into what looks like healthy scaling?
3. How much of the critical path is communication, and does the measured transfer time correspond to real, physical bandwidth — the sanity check that this methodology measures data movement and not leftover skew?

Experimental setting

What is being distributed. Every run on this page is the same kind of distributed architecture: a plug-and-play (PnP) solver that alternates a physics step — a gradient step through the acquisition model — with a denoising step, a DRUNet acting as the learned image prior. The two are parallelized in completely different ways. The acquisition operators are distributed whole across the workers, one or more per GPU, so a section’s critical path is set by the busiest GPU’s operator count. The image handed to the denoiser is instead split into overlapping tiles, one group of tiles per GPU; the overlap gives the network the spatial context it needs at tile boundaries, at the cost of extra computation and extra data movement. Between the two steps, the workers reconcile their copies with an all_reduce — that collective is the communication this page takes apart. The inference runs execute that alternation as a plain loop with fixed denoiser weights. The training runs unroll it instead: a fixed number of iterations — five in 2D, one in 3D — is treated as a single differentiable network and backpropagated through, which is why training’s sections are forward and backward, each containing both physics and denoiser work, rather than one section per step.

Experiments. Four result sets: inference’s denoiser and gradient sections, and training’s forward and backward sections, each run on a 2D problem (8192² multiframe super-resolution for inference, 4096² synthetic for training) and a 3D (512³ volume) problem. All profiled every iteration with torch.profiler, on Tesla V100 GPUs, 4 per node, up to 64 GPUs.

Configurations. Four benchmark configurations produce everything below — one per result set, two under benchmark_inference/configs/experiments/ and two under benchmark_training/configs/experiments/. Each figure on this page names the ones it came from:

Two details to read off that table before the figures. The operator counts differ by configuration — 8 and 2 for inference, 4 for both training runs — and that single number turns out to explain most of the wait behaviour below, so it is worth keeping in view. And 3D training starts at 4 GPUs, not 1: its baseline in the scaled charts is its own 4-GPU run, so its bars are not directly comparable in absolute terms to the three configurations that start from a single GPU.

Metrics, per the POP methodology:

• transfer = the minimum kernel time across ranks — the last rank to arrive waits for nobody, so its time is pure transfer.
• wait = the average gap between that minimum and each rank’s own raw time.
• compute = the mean per-rank section time minus its own collective time.
• compute + transfer + wait are additive — they sum exactly to the section’s CUDA time.
• Per-section load balance: LB = compute / compute_max, where compute_max is the critical-path (slowest) rank.
• Combined load balance, summed across every section instead of one at a time: LB = sum(compute) / sum(compute_max).
• Communication efficiency, summed the same way: CommE = sum(compute_max) / total_sec, where total_sec is cuda_sec summed across every section.
• LB x CommE is the overall parallel efficiency — the fraction of each GPU-second spent on useful, balanced work.

How does communication cost grow, and what’s driving it?

Each bar below is scaled relative to the smallest GPU count tested, in GPU-seconds. The baseline bar is always 1.0 by construction — perfect linear scaling would keep every bar at 1.0 regardless of GPU count. A bar taller than 1.0 means that configuration spent more total GPU-seconds than ideal scaling would have, and the color split shows exactly how much of that excess is compute versus communication. The charts further down break that excess down again, into compute, transfer, and wait individually, in absolute seconds.

The pattern is consistent across both charts: 3D scales better than 2D, and training scales better than inference. By 64 GPUs, the combined cost sits at 1.08× ideal for training 3D, 1.12× for inference 3D, 1.91× for training 2D, and 2.22× for inference 2D — the gap between 3D and 2D (roughly 2×) dwarfs the gap between training and inference (a few percent either way).

The reason is a race between compute and transfer. The fair comparison here is compute_max against transfer alone, not against transfer+wait: compute_max is the critical-path rank’s own compute, and that rank (by construction) has no wait of its own, so this comparison isn’t muddied by idling — it’s asking whether the best-case rank still has more real work to do than the unavoidable data it has to move. As GPU count grows, each GPU’s share of compute shrinks, but transfer does not shrink at the same rate, so the two get closer together. 2D inference comes right to the edge of that crossover by 64 GPUs — combined critical-path compute (0.33s) is only 4% above combined transfer (0.32s) — without quite crossing it in the range tested. Training 2D keeps more distance (1.85s vs 1.13s, compute 64% above transfer). Both 3D cases stay solidly compute-bound (inference 3D: 2.79s vs 0.21s, 13x; training 3D: 10.13s vs 0.66s, 15x) — the 3D problem is simply much larger, leaving far more compute to outrun transfer with. This is why 2D inference is the closest of the four to becoming transfer-bound: its compute shrinks fastest of all, bringing it nearest to the point where transfer alone would exceed it.

Splitting the total between sections, the denoiser is the main contributor: it makes up 98.5% of the 2D baseline and effectively all of the 3D baseline (99.99%), and its own cost only grows modestly by 64 GPUs (1.54× in 2D, 1.07× in 3D). The physics gradient starts from a tiny baseline share (1.5% in 2D, 0.01% in 3D) but grows far faster in relative terms — 47× its own baseline in 2D, 367× in 3D — making it the most communication-exposed section even though its absolute contribution stays small. Training’s forward and backward sit between these two extremes (forward grows 2.49× in 2D vs 1.20× in 3D; backward grows 1.72× in 2D vs 1.04× in 3D).

To see exactly where that growth comes from — pure data transfer, or GPUs waiting on each other — the next figure breaks compute, transfer, and wait apart individually, in absolute seconds per section, 2D (solid) and 3D (dashed) overlaid on the same axes.

• Compute — denoiser vs. gradient. The denoiser’s compute keeps roughly halving every time GPU count doubles, all the way to 64 GPUs (2D: 18.3s → 0.29s; 3D: 172s → 2.8s). The physics gradient does not: its compute plateaus — 2D flattens at ~0.038s from 8 GPUs onward, 3D flattens at ~0.013s from 2 GPUs onward, and stays there no matter how many more GPUs are added. This isn’t noise: the inference configs distribute a fixed number of acquisition operators — 8 for 2D, 2 for 3D (comm_inference_2D.yml / comm_inference_3D.yml) — each assigned whole to one GPU. Once GPU count exceeds the operator count, the busiest GPU still holds exactly one operator, so more GPUs beyond 8 (2D) / 2 (3D) cannot shrink the critical-path compute any further.
• Compute — training’s forward/backward. Training distributes the same fixed operator count too (4, for both 2D and 3D), so the same plateau exists in principle — but forward and backward bundle physics and the denoiser’s model computation into one section, and the denoiser dominates that total. Its larger, smoothly-scaling compute masks the smaller physics-operator plateau underneath it, so neither section shows a visible flattening here — both keep decreasing all the way to 64 GPUs.
• Transfer. Grows with GPU count, in every section and dimensionality — then levels off at the largest sizes. One exception on the way up: 4 GPUs is consistently lower than 2 GPUs. We come back to both in What sets the transfer time? below.
• Wait. Generally a small percentage of compute — except for the physics gradient, for the same reason as above: once GPU count exceeds the operator count, some GPUs hold zero physics operators, finish instantly, and sit idle in the collective until the others catch up.

Are the GPUs actually staying balanced?

The denoiser stays under 4% wait at every GPU count, in both dimensionalities. That means each GPU genuinely gets the same amount of tile work — the residual few percent is just timing noise between otherwise identical GPUs, not real imbalance. Worth keeping in mind, though, that this balance is a property of the tiling, not something guaranteed: the denoiser’s load balance depends on how many tiles the image is split into relative to the number of GPUs. These runs are tiled favourably; a poorly chosen tiling — too few tiles to spread evenly across the ranks — would leave some GPUs with more tile work than others and degrade performance in exactly the same way the physics gradient does below.

The physics gradient tells a completely different story, and it tracks the operator count exactly. In 2D (8 operators), wait stays under 3% through 8 GPUs — one operator per GPU, perfectly balanced — then jumps to 83% at 16 GPUs and keeps climbing to 302% at 64, because every GPU added past 8 has zero operators and nothing to do but wait. In 3D (2 operators), the same thing happens two steps earlier: wait is low at 2 GPUs (one operator each), then jumps to 54% by 4 GPUs, for the identical reason.

This is normal and expected behaviour rather than a defect in the implementation, and with the current design it is essentially unavoidable. The PnP iteration is serialized — physics, then denoising, then physics again — so every GPU has to be present at the end of the physics step before the denoising step can start, whether or not it was given an operator to work on. With a fixed number of acquisition operators, once GPU count passes that number the surplus GPUs have nothing to do but wait at the collective. How far this can be avoided — overlapping the physics step with denoising, or splitting the operators themselves — is an open question, not something these measurements settle.

Training’s forward and backward are a mix of both stories, because — as established earlier — they bundle physics and denoiser computation into one section. When the denoiser dominates (3D, at any GPU count tested; 2D, at low GPU counts), wait stays small, tracking the denoiser’s behavior. Once the physics portion becomes a larger share of the work (2D, at higher GPU counts), wait grows and lands between the pure-gradient and pure-denoiser curves — forward reaches 38% and backward 17% at 64 GPUs in 2D, well below gradient’s 302% but well above denoiser’s 3%.

Per-section wait tells you where imbalance shows up; load balance (LB) below restates it as an efficiency, and puts each section’s own LB (solid) next to the value summed across every section (dashed) on the same axis — so the gap between the worst section and the number you would quote for the run as a whole is directly visible.

By 64 GPUs, combined load balance stays relatively high everywhere (86–97%) — even 2D inference, the worst performer overall, keeps 89%. That might look surprising given the physics gradient’s own load balance collapses to 27% (2D) / 37% (3D) at 64 GPUs, the lowest solid line on the chart — but combined LB is a weighted average (sum(compute) / sum(compute_max)), and by 64 GPUs the gradient is still only 11% of the combined compute_max in 2D, and just 0.5% in 3D. Even at that weight, an 11% share pulls the near-perfect denoiser figure (97%) down to 89% — visible, but nowhere near as severe as the gradient’s own 27% would suggest in isolation.

That weighting is the whole argument, and it rests on a property of this setup: the acquisition models used here are not computationally heavy compared with the denoiser, which is why the gradient can collapse to 27% load balance and still barely move the combined figure. A heavier physics operator would carry a much larger share of compute_max, and the same 27% would then drag combined load balance down with it. We have no such example in this benchmark yet, but the conclusion above should not be read as holding for any acquisition model.

How much of the critical path is communication — and does it check out?

Communication efficiency asks a different question from load balance above. Load balance is about imbalance between GPUs — mainly a wait, mainly- gradient story, as just shown. CommE instead asks: even on a perfectly balanced system, how much of the critical path would still be spent moving data rather than computing? This isolates the transfer-driven cost of scaling, which is a separate mechanism from imbalance.

Communication efficiency drops much further than load balance does: it falls to 51% for 2D inference and 60% for 2D training, versus 93–94% for both 3D cases. Split by section, the same ordering as load balance holds — at 64 GPUs the denoiser keeps 66% in 2D and 96% in 3D, while the gradient is down to 18% and 10% — but note the denoiser’s own CommE falls a long way in 2D even though its 2D load balance never left 97%: this is a different problem from imbalance, and the next paragraph is about why. By 64 GPUs, 2D inference’s critical-path compute (0.33s) is nearly matched by transfer alone (0.32s); 2D training still has more room (1.85s vs 1.13s). Neither 3D case comes close: 2.79s compute vs 0.21s transfer for inference, 10.13s vs 0.66s for training — compute stays far larger because the 3D problem itself is much bigger.

This is a genuinely different mechanism from the gradient’s load-balance problem, not a restatement of it: summed across every section, transfer — not wait — is the majority of the non-compute time (89% for 2D inference, 76% for 2D training; even 3D, where CommE barely drops, is 64–69% transfer). The CommE drop is a data-movement story, concentrated in the dominant denoiser/backward sections, which stay well balanced but still have to move more data relative to their shrinking compute as GPU count grows. The gradient’s load-balance collapse, from the wait chart earlier, is a separate problem in a separate (small) section — both are real, but they are not the same thing, and fixing one would not fix the other.

What sets the transfer time?

Transfer time is bytes divided by a rate, so it is worth knowing what each of those is. It turns out both follow simple rules, and together they explain the shape of every transfer measurement on this page — including why transfer time stops growing once there are enough GPUs.

Each all_reduce shares one image across all the GPUs. That image is the payload: 8192 x 8192 x 3 channels x 4 bytes ≈ 805 MB for 2D inference, and 512³ x 4 bytes ≈ 537 MB for 3D. But no GPU ever sends the whole 805 MB, and how much it does send depends on which route the GPUs take.

Two ways to add up numbers spread across many GPUs

A ring. The GPUs are arranged in a logical circle. The image is divided into one slice per GPU, and the slices travel around the ring twice: once accumulating the partial sums, once propagating the completed result. Each GPU therefore sends just under two payloads. The transfers are numerous and small, and every link is active simultaneously, which is what makes the ring efficient for large messages.

A tree. Each node first reduces across its own 4 GPUs over the fast intra-node links. One node then acts as the root: the others send it their partial sums, it completes the reduction, and the result is propagated back down. Far less data crosses the slower inter-node network, and the saving is largest when few nodes participate.

The choice is not made by the application. NCCL constructs both topologies at startup and selects one per call, from the message size and the cluster topology. Seven algorithms are implemented in total, but only these two are available here — the remainder require NVSwitch hardware or a network plugin this cluster does not provide. Running with NCCL_DEBUG_SUBSYS=INIT,TUNING reports the algorithm selected:

The 8-GPU row is the one that matters for the byte accounting, because the two algorithms do not move the same volume:

ring:  bytes = 2 x payload x (gpus  - 1) / gpus      -> 1.4 GB at 8 GPUs
tree:  bytes = 2 x payload x (nodes - 1) / nodes     -> 805 MB at 2 nodes

Attributing the ring’s 1.4 GB to a run that in fact used the tree overestimates the implied bandwidth by a factor 1.75, so every point below is computed with the formula for the algorithm NCCL reported. The distinction only affects the 2-node case: from 16 GPUs onward the two byte counts agree to within 5%.

Both expressions are bounded above. (gpus - 1) / gpus and (nodes - 1) / nodes increase towards 1 as GPUs are added, so under either algorithm the volume moved per GPU converges to two payloads and never exceeds it, however many GPUs participate. This bound is what produces the flat tail of the chart below.

The byte counts are exact for inference, where denoiser and gradient each issue exactly one all_reduce per iteration. Training’s sections carry additional messages the formula does not account for, so the chart is restricted to inference.

The rate works out at about 40 GB/s at 2 GPUs, 120 GB/s at 4, 17 GB/s at 8 and 10 GB/s from 16 GPUs on. Those numbers are measured, not looked up: each is the bytes above divided by the transfer time beside it. What makes them useful is that they depend only on the route the data takes, not on how many GPUs are involved:

NVLink, 2 GPUs      40 GB/s
NVLink, 4 GPUs     119 GB/s
network, tree       17 GB/s    (2 nodes, the only size NCCL uses a tree)
network, ring     10.5 GB/s    <- the same at 2, 4, 8 and 16 nodes

With a byte rule and a rate, the chart can be read from left to right.

2 and 4 GPUs — inside one node. This is the oddity flagged earlier: 4 GPUs transfers faster than 2. With 2 GPUs a pair is joined by only 2 of the V100’s 6 NVLink cables; with 4 GPUs all 6 are in play, so the rate triples. Bytes rise too, but only by half (1.0 → 1.5 payloads, as (gpus - 1) / gpus goes from ½ to ¾). Three times the rate against 1.5× the bytes halves the time, and the measurement obliges: 20.1 ms down to 10.1 ms. That the measured ratio is 2.97 to 3.07 across all four series — following the cable count that exactly — is also the sharpest sign these rates really are cables and not bookkeeping.

8 GPUs — the data leaves the node. A node holds only 4 GPUs, so beyond that every byte has to cross the network, which is far slower than the cables inside a box. Transfer time jumps accordingly. But this is the one size where NCCL picks a tree, and the dip in the middle of the chart is exactly that: one payload across the network instead of nearly two, at a higher rate, so a little over a third of the time a ring would have needed.

16 to 64 GPUs — a ring over the network. Here the rate is a single number, 10.5 GB/s, which is worth testing since one constant covering four node counts is a real claim rather than a restatement. It can be tested without circularity: force a ring onto the 2-node job — a configuration that appears nowhere in the chart — and it gives 10.5 GB/s. Feeding only that number back through the byte formula predicts every ring row, in both 2D and 3D (denoiser shown here):

gradient is not shown but is predicted by the same constant, a little less tightly: twelve predictions in all, mostly inside 5% and never worse than 13% (gradient 2D, the noisiest series). Skew leaking through the min-reduction could not do that — it has no reason to scale with a byte count that changes when the image size does. transfer really is transfer.

Which brings the flat tail of the chart, and the point of all this. The bytes stall at two payloads; the rate does not change. So transfer time stops growing — roughly 150 ms per iteration from 16 GPUs on for a 2D image, no matter how many more GPUs are added. Compute per GPU keeps falling as the job grows. Transfer does not. That gap is the whole mechanism behind the communication-efficiency drop earlier on this page, and it is arithmetic rather than a quirk of the ring: a tree’s byte count converges to the same two payloads, so it flattens out too, just at a lower level.

Conclusions

“Communication” is not one thing, and this study is an illustration of why that distinction matters. Splitting it into transfer and wait exposes two different failure modes, each with its own remedy — and this study finds a clear example of each, playing out in different sections.

On the wait side, the lesson is that scaling holds up exactly where work is divided evenly across GPUs, and breaks down exactly where it isn’t. The physics gradient distributes a fixed number of acquisition operators, one whole operator to a GPU — 8 in 2D, 2 in 3D — so once GPU count passes that number, every extra GPU gets zero operators and nothing to do but idle: by 64 GPUs its wait time is three times its compute time, in both 2D and 3D, and its own load balance collapses to 27% (2D) / 37% (3D). That ceiling is a property of this physics problem having only a handful of operators to divide, not a general communication limit — a physics operator that decomposes more finely (tomography’s per-angle or per-slice split, for instance) would not hit the same wall at these GPU counts. The denoiser is the counterexample inside this same study: its patch-based split can cut the image into as many pieces as there are GPUs, keeping every GPU’s share essentially equal, which is exactly why its own load balance never drops below 96.6% and wait never exceeds 4% of its compute time at every GPU count tested here.

On the transfer side, though, being well balanced isn’t enough. More GPUs means less compute per GPU, but the data that has to move doesn’t shrink at the same rate — so transfer eats a growing share of the critical path regardless of balance. Summed across every section, transfer — not wait — is already the majority of the non-compute time by 64 GPUs (89% for 2D inference, 76% for 2D training; 64–69% even in 3D). That’s why communication efficiency drops much further than load balance does in 2D, down to 51% for inference and 60% for training — a separate, data-movement mechanism from the gradient’s wait problem above, one rebalancing can’t fix. What does help is more compute per GPU to hide the same transfer behind — exactly why the bigger 3D problem stays compute-dominated at every GPU count tested while 2D does not.

Put together, these two mechanisms are why 2D pays roughly double the ideal GPU-seconds cost that perfect linear scaling would by 64 GPUs (2.22× for inference, 1.91× for training), while 3D stays close to ideal (1.12× and 1.08×) — and in every one of these four cases, the compute portion itself barely moves (0.97–1.01×). Communication, in one form or the other, is responsible for essentially all of the gap between measured and ideal scaling.