How GPUs Talk to Each Other During LLM Inference
From NCCL to NVLink Multicast: A Look at Multi-GPU Communication in SGLang
Draft. 6 measured numbers are still placeholders (highlighted in the text). Figure 9 is drawn from simulated data.
Tensor parallelism was introduced in the Megatron-LM paper back in 2019. And if we look at how inference engines run a large model like GLM-5.3 across multiple GPUs today, the basic recipe is still the same.
We cut every weight matrix into slices, give each GPU one slice, and let each GPU compute a partial result. Then, we have to add those partial results back together, which means that the GPUs have to talk to each other. (For GLM-5.3, which has 78 layers, this happens 156 times in every forward pass.)
The GPUs have gotten a lot faster since 2019, and the models have gotten a lot bigger. But what about the step where the partial results are added back together? Is that still done the same way?
As it turns out, a lot has changed here. If we read through the source code of SGLang and FlashInfer (the inference engine and the kernel library that I will use as the reference throughout this article), we find that this one step is handled in several quite different ways. Which one runs can change from one call to the next, depending on how many bytes have to be sent, and some setups avoid the step altogether.
In this article, I want to walk through what the GPUs actually send to each other during a forward pass, and why there is more than one way to do it. I won't compare inference engines or focus on benchmark numbers here. Instead, we will start with the general-purpose approach and then work our way towards the more specialized tricks.
(To keep the scope reasonable, I will only cover inference on NVIDIA GPUs from the Hopper generation onwards, since several of the tricks below rely on hardware features that older GPUs don't have. I will also mostly stick to a single machine where the GPUs are connected via NVLink.)
One quick note before we start: everything I describe about how the mechanisms work comes from reading the source code, and I link to the relevant files along the way. Whenever I need concrete numbers, I will use GLM-5.3 as the running example, which is a Mixture-of-Experts model with 78 transformer layers. The few measurements are from one machine with eight B300 GPUs.
1. Why GPUs Need to Talk at All
Before we get to the interesting tricks, let's briefly go over why the GPUs need to communicate in the first place. If you have worked with tensor parallelism before, most of this section will look familiar. I still recommend skimming it, though, because the three ideas in it (partial sums, the all-reduce, and the message size) come up again in every later section.
1.1 Tensor Parallelism
Tensor parallelism (TP) is the most common way to run a model that is too big for one GPU. The idea is to split each weight matrix across the GPUs so that each GPU only has to store, and multiply by, a fraction of it.
There is a catch, though. Depending on the direction in which we cut a matrix, a GPU ends up with either a piece of the result or an incomplete version of the whole result.
To make this concrete, let's look at the feed-forward module in one of the dense layers of GLM-5.3, running on 8 GPUs. This module consists of two matrices: the first one expands the 6,144-dimensional hidden state to 12,288 dimensions, and the second one projects it back down to 6,144.
We cut the first matrix along its output dimension, which is called a column-parallel layer. This means that each GPU gets 1,536 of the 12,288 outputs, sees the full input, and computes its own 1,536-wide piece of the intermediate result. Since these pieces don't depend on each other, no communication is needed here.
We cut the second matrix along its input dimension instead, which is called a row-parallel layer. This is convenient because each GPU already has exactly the 1,536-wide piece it needs as input. However, each GPU now produces a full 6,144-wide output that is only a partial sum. To get the actual result, we have to add up the outputs of all 8 GPUs, as illustrated in Figure 1 below.
As shown in Figure 1, the same thing happens in the attention module: the query, key, and value projections are column-parallel, and the output projection is row-parallel. So, each transformer layer has two places where the GPUs have to add up their partial sums, one after the attention module and one after the feed-forward (or Mixture-of-Experts) module. This is where the 156 from the introduction comes from (2 × 78 layers).
To illustrate what this looks like in code, below is a simplified excerpt of the row-parallel layer in SGLang:
# Simplified from RowParallelLinear.forward()
def forward(self, x):
partial = self.quant_method.apply(self, x) # this GPU's partial sum
if self.reduce_results and self.tp_size > 1:
return tp_group.all_reduce(partial) # add the partial sums now
return partial # or leave the sum for later
Note the last line of the excerpt. In recent versions of SGLang, these layers often don't finish the sum themselves but hand the partial sum to whatever comes next. We will come back to this in section 5, since it is what makes the fused kernel possible.
1.2 All-Reduce and Other Collectives
The operation that adds up the partial sums is called an all-reduce: every GPU contributes a tensor of the same shape, and afterwards, every GPU holds the element-wise sum of all of them.
The all-reduce has three siblings that we will run into later in this article, so let's briefly introduce them here as well. To tell them apart, it helps to cut the tensor of each GPU into chunks. With 4 GPUs, let's say that each tensor consists of the chunks A, B, C, and D, and let's write A2 for chunk A as GPU 2 has it. Figure 2 below shows what happens to these chunks.
Let's go through Figure 2 from top to bottom. In a reduce-scatter, the chunks are summed across the GPUs, but each GPU only gets one of the summed chunks back: GPU 0 ends up with ΣA, GPU 1 with ΣB, and so on. In an all-gather, nothing is summed. Instead, every GPU hands out the chunk it has, so that every GPU ends up with all of them.
If we do both, one after the other, every GPU holds all four summed chunks, which is exactly the result of an all-reduce. In other words, an all-reduce is the same as a reduce-scatter followed by an all-gather. (This is going to be useful in section 4.1.)
The fourth one, the all-to-all, is shown at the bottom of Figure 2. Here, every GPU sends a different chunk to each GPU (chunk A goes to GPU 0, chunk B to GPU 1, and so on), and nothing is summed.
These operations are called collectives because all GPUs in the group have to participate, which also means that a collective is only finished once the slowest GPU is done.
1.3 Message Sizes in Decode and Prefill
Now, how much data does an all-reduce actually have to send? The tensor we are summing has one row per token in the batch, so its size is the number of tokens times the hidden size times the size of one element.
For example, GLM-5.3 has a hidden size of 6,144, and the activations are 16-bit floats. This means that each token costs 12,288 bytes, or 12 KB. (I am using KB and MB for powers of 1,024 throughout this article.)
During decoding, each request contributes one token per step. With 8 requests in a batch, the message is only 96 KB, and even with 64 requests, it is still below 1 MB.
During prefill, on the other hand, the entire prompt is processed at once. If the batch contains 8,192 prompt tokens, the message is 96 MB, which is a thousand times larger than in the decode example above. Figure 3 below shows both cases on the same axis.
The reason I am emphasizing this is that small and large messages are slow for very different reasons. For a 96 MB message, what matters is how many bytes per second we can push through NVLink. For a 96 KB message, moving the bytes takes almost no time, and what matters are the fixed costs, such as launching a kernel and waiting for the other GPUs.
As we will see, most of the mechanisms in this article exist because it's hard to be good at both.
(You may wonder why one setup should have to deal with both ends of this axis in the first place. That's a fair question, and we will come back to it in section 8.2.)
1.4 Summary
To summarize, tensor parallelism leaves us with two partial sums per transformer layer, and an all-reduce is the operation that adds them up across the GPUs.
The other thing to keep in mind is the message size: about 100 KB during decoding and about 100 MB during prefill for a model like GLM-5.3. We will come back to the axis from Figure 3 at the very end of this article.
2. NCCL
The standard way to run an all-reduce on NVIDIA GPUs is NCCL, NVIDIA's collective communication library. It is what PyTorch uses under the hood, and inference engines typically use it as their default.
The classic all-reduce algorithm in NCCL arranges the GPUs in a ring. (NCCL also has other algorithms and chooses between them at runtime, but the ring is the easiest one to picture.) The message is cut into as many chunks as there are GPUs, and then the chunks are passed around the ring in two rounds, as shown in Figure 4 below.
In the first round, each GPU receives a chunk from its neighbor, adds its own data to it, and passes it on. After N-1 steps, each GPU holds one chunk that has been fully summed. (In the terms of section 1.2, this first round is a reduce-scatter, and the second one is an all-gather.) In the second round, these finished chunks travel around the ring once more so that every GPU ends up with all of them.
The nice thing about the ring is that it is very economical with bandwidth. In total, each GPU only receives 2(N-1)/N times the message size, which is 1.75 times for 8 GPUs. (Keep this number in mind, since we will compare the algorithms in the next two sections against it.)
However, the downside is that the ring needs 2(N-1) steps, one after the other. For 8 GPUs, that's 14 steps, and each step has to wait for the previous one.
For the 96 MB prefill message from section 1.3, this doesn't matter, since the time is dominated by moving the bytes anyway. For the 96 KB decode message, though, the bytes take almost no time, so almost all of the time goes into this step-by-step coordination.
So, NCCL is a solid default for large messages, which is also why SGLang keeps it as the fallback for everything the faster mechanism below doesn't handle. (Whether it is the best we can do for large messages is a different question, and I will come back to it at the very end.)
3. MNNVL One-Shot
The ring makes sense if each GPU can only talk to its neighbors. But inside a machine with NVLink, that's not the situation we are in, since every GPU can reach every other GPU directly. The all-reduce that we will look at in this section and the next one takes advantage of this.
It is part of FlashInfer, the kernel library that SGLang builds on, and it is called MNNVL, which is short for multi-node NVLink. (The code was originally ported from NVIDIA's TensorRT-LLM.)
Don't let the name confuse you, though. MNNVL was designed for racks where one NVLink network spans several machines, but it works just the same on a single machine with 8 GPUs and regular NVLink, which is the setup I will assume here. We will come back to the multi-node case in section 8.3.
MNNVL has two strategies, one-shot and two-shot. Let's start with one-shot, which is the one for small messages.
3.1 Symmetric Memory and Multicast
Before we get to the algorithm itself, we need two things that NVLink gives us.
The first one is that a kernel running on one GPU can directly read from and write to the memory of another GPU, as long as that memory has been mapped beforehand. MNNVL sets this up once at startup: every GPU allocates a buffer of the same size, and each of these buffers is mapped into the address space of all the other GPUs. As a result, each GPU has a small table with N pointers (one per GPU), and a kernel on GPU 0 can store to GPU 5's buffer with a regular store instruction. This arrangement is often called symmetric memory.
The second one is multicast, a hardware feature that is available from Hopper onwards. Here, the buffers of all GPUs are additionally mapped to one special address. If a kernel stores a value to this address, the value is written to the buffers of all GPUs at once, as illustrated in Figure 5 below.
As far as I understand, the copies on the right side of Figure 5 are made by the NVLink switch, so the sending GPU only has to send its data once. (To tell the two kinds of pointers apart, a regular pointer to one particular GPU's buffer is called a unicast pointer.)
3.2 Writing to All GPUs at Once
Now, if every GPU can write into every other GPU's buffer, we don't need a ring for small messages. In the one-shot strategy, every GPU simply writes its entire message to the multicast address, all at the same time.
Each GPU's buffer has one slot per sender. So, after this single round of writes, every GPU has all N messages sitting in its own buffer and can add them up locally. Figure 6 below compares this to the ring from the previous section.
As we can see in Figure 6, one-shot needs a single step where the ring needs 14. The price is that much more data arrives at each GPU: each GPU now receives 7 full messages instead of 1.75. On top of that, each GPU needs a buffer that is large enough to hold all N messages.
For small messages, this is a great trade. For example, 7 copies of a 96 KB message are still less than 1 MB, which is a small amount of data for NVLink. Saving 13 sequential steps matters much more.
3.3 Knowing When the Data Has Arrived
There is one problem we have to solve, though. How does a GPU know that the messages from all the other GPUs have arrived? If it starts adding too early, it reads whatever happened to be in the buffer.
The usual answer would be a handshake, where the GPUs signal each other via flags. However, a handshake costs an extra round trip between the GPUs, and this is exactly the kind of fixed cost that one-shot is supposed to avoid.
This is where MNNVL uses a clever trick. It doesn't send a separate signal at all. Instead, the receiving GPU can tell from the data whether it has arrived. Before a buffer is used, it is filled with one special value, negative zero (-0.0), and a GPU never writes that value. If its data happens to contain a -0.0, it replaces it with a regular +0.0 before sending, which makes no difference in a sum.
The receiving GPU then simply keeps re-reading the slots of the other GPUs until none of them contains a -0.0 anymore:
slot before: [ -0.0 -0.0 -0.0 -0.0 ] nothing has arrived yet
slot after: [ 1.5 +0.0 2.0 0.3 ] no -0.0 left, so everything is here
Below is a simplified version of the one-shot kernel (the full version is here). Note that each thread handles 16 bytes of one token:
// Simplified from oneshotAllreduceFusionKernel
val = load(input, token, idx);
replace_neg_zero_with_pos_zero(val);
store(val, slot(multicast_buffer, token, my_rank)); // lands on every GPU
do {
for (rank in other_ranks)
remote[rank] = load(slot(my_buffer, token, rank));
} while (any_is_neg_zero(remote)); // poll until all have arrived
sum = add_in_rank_order(val, remote); // in 32-bit floats
store(sum, output, token, idx);
There are a few things worth noticing in this excerpt. The store in the third line goes to the multicast address, whereas the loads in the loop go to the GPU's own buffer. Also, the GPU's own value never takes the detour through the buffer, since it is still in the registers. And the values are always added in the same order (by GPU index), so that all GPUs end up with exactly the same result.
(By the way, the check for -0.0 has to compare the raw bits. A comment in the source warns that a regular floating-point comparison can mistake very small negative numbers for -0.0, in which case the kernel would wait forever.)
3.4 Three Rotating Buffers
After a round, the slots contain data, so they have to be filled with -0.0 again before they can be reused. MNNVL doesn't do this right away. Instead, it keeps three buffers and rotates through them: each call writes into the current buffer and, while it is at it, refills the buffer of the previous call with -0.0. Figure 7 below shows the rotation.
This way, the cleanup doesn't need a kernel of its own, and there is always one full call between the moment a buffer is cleared and the moment it is used again.
Why does MNNVL need three buffers when two would seem to be enough? The source code doesn't say. My guess is that with only two, a fast GPU could already write its next message into a buffer that a slower GPU is still busy clearing.
3.5 One-Shot Summary
To summarize, one-shot replaces the 14 sequential steps of the ring with a single round of multicast writes, and it replaces the handshake with a marker value in the data itself. Since a GPU doesn't have to wait for anything except the data, this is the fastest option for small messages.
The downside is the amount of data. Each GPU receives N-1 full messages and needs room for all N of them, so the larger the message gets, the longer this takes.
4. MNNVL Two-Shot
For larger messages, MNNVL switches to its second strategy, two-shot, which moves much less data in exchange for one extra step.
4.1 Scatter and Broadcast
The idea is to make each GPU responsible for a share of the tokens. With N GPUs, token 0 belongs to GPU 0, token 1 belongs to GPU 1, and so on, and after N tokens, the assignment starts over at GPU 0. The all-reduce then happens in two stages:
- Scatter: every GPU sends each of its tokens only to the GPU that is responsible for it. (This uses the regular unicast pointers from section 3.1.)
- Broadcast: each GPU adds up the N copies of the tokens it is responsible for and then writes the finished tokens to the multicast address so that they end up on all GPUs.
If this sounds familiar, it's the "reduce-scatter followed by an all-gather" decomposition from section 1.2. Both stages use the same -0.0 trick as one-shot to detect when the data has arrived. Figure 8 below shows the difference between the two strategies.
Looking at the right side of Figure 8, each GPU now receives only (N-1)/N of a message during the scatter stage and another (N-1)/N during the broadcast stage. That's 2(N-1)/N in total, which is exactly the number we had for the ring. In other words, two-shot is as economical as the ring, but it needs only 2 steps instead of 2(N-1), because every GPU can reach every other GPU directly.
For example, with 8 GPUs, each GPU receives 7 messages' worth of data in one-shot but only 1.75 in two-shot, which is 4 times less. In general, the savings factor is N/2, so the more GPUs we have, the bigger the advantage of two-shot becomes. (The buffers shrink as well: instead of room for N messages, each GPU only needs room for 2.)
4.2 Choosing Between One-Shot and Two-Shot
So, how does MNNVL decide which of the two strategies to use? The answer is surprisingly simple. Below is the function that makes the decision (slightly shortened):
# Empirical result calculated from num_tokens * hidden_dim * tp_size * elem_size
MNNVL_ONE_SHOT_THRESHOLD = 64 * 1024 * 8 * 2
def select_strategy(tp_size, num_tokens, hidden_dim, dtype):
elem_size = torch.tensor([], dtype=dtype).element_size()
if num_tokens * hidden_dim * tp_size * elem_size <= MNNVL_ONE_SHOT_THRESHOLD:
return MNNVLAllreduceFusionStrategy.ONESHOT
else:
return MNNVLAllreduceFusionStrategy.TWOSHOT
In other words, MNNVL multiplies the message size by the number of GPUs, which gives the size of the one-shot buffer (remember that it has to hold N messages). As long as the result stays below 1 MB, it uses one-shot. According to the comment, this threshold is an empirical result, that is, it comes from measurements and not from a formula.
For GLM-5.3 on 8 GPUs, this works out to messages of up to 128 KB, which corresponds to a decode batch of 10 requests. And since the number of GPUs is part of the formula, the limit drops as we add GPUs: one-shot covers a batch of 42 requests with 2 GPUs, 21 with 4, 10 with 8, and only 5 with 16. This is consistent with what we discussed in section 4.1: the amount of data in one-shot grows with the number of GPUs, whereas in two-shot, it stays roughly constant.
It also means that with 8 GPUs, anything beyond a handful of concurrent requests already uses two-shot.
So, how much faster is all of this compared to NCCL in practice? On a machine with 8 B300 GPUs, a 128 KB all-reduce takes nccl_us_128k_tp8 microseconds with NCCL versus mnnvl_us_128k_tp8 microseconds with MNNVL, which is a speedup_128k_tp8x difference. For a 16 MB message, it's nccl_ms_16m_tp8 versus mnnvl_ms_16m_tp8 milliseconds (speedup_16m_tp8x). The full sweep is shown in Figure 9 below.
Note that Figure 9 only measures the all-reduce by itself. It doesn't include the fusion that we will discuss in the next section.
4.3 Two-Shot Summary
To summarize, two-shot splits the all-reduce into a scatter stage and a broadcast stage. Each GPU then only receives 2(N-1)/N message sizes instead of N-1, so two-shot is the faster one as soon as the message is large enough. And the more GPUs we have, the smaller the message at which this happens.
MNNVL chooses between the two strategies purely based on the message size and the number of GPUs, using a single threshold of 1 MB.
5. Fused All-Reduce, Residual Add, and RMSNorm
Everything so far makes the all-reduce itself faster. But MNNVL goes one step further and stops treating the all-reduce as a separate operation. (In fact, you may have noticed the word "Fusion" in the name of the kernel in section 3.3.)
Let's think about what happens right after the all-reduce in a transformer layer. First, the summed result is added to the residual stream. Then, the updated residual stream goes through the RMSNorm of the next module.
Normally, these are three separate kernels (all-reduce, add, norm), and each of them reads and writes the same tensor. This is a bit wasteful, especially during decoding, since the tensor is so small there that the overhead of launching each kernel becomes noticeable.
The MNNVL kernel can do all three steps in one go, as shown in Figure 10 below.
The reason this works so well is that, after the sum, each thread already has the summed values at hand. This means it can add the residual and apply the normalization right away, before writing anything back to memory. In terms of the simplified kernel from section 3.3, the fused version just keeps going after the sum:
sum += load(residual, token, idx);
store(sum, residual_out, token, idx); // the updated residual stream
sum = sum * rsqrt(mean_of_squares(token) + eps) * gamma;
store(sum, output, token, idx); // the normalized result
There is one detail here that I find quite elegant. RMSNorm needs the mean of the squares over a whole token, so the threads that work on the same token have to combine their partial results. The kernel is laid out so that this is cheap: each token gets its own small group of thread blocks, and that group computes the mean together.
For two-shot, the fusion looks slightly different. Here, a second kernel takes care of the residual add and the RMSNorm, and it reads the summed tokens straight out of the broadcast buffer, so the result is never copied into a separate tensor first.
This is also the reason for the "leave the sum for later" line that we saw in the code excerpt in section 1.1. The attention (or MoE) module returns its partial sum as is, and the code at the layer boundary then decides whether the fused kernel can finish the job.
Note that SGLang doesn't use the fused kernel everywhere. It is limited to batches of up to 2,048 tokens, and on Blackwell GPUs, it is enabled automatically for a list of supported model architectures when tensor parallelism is used. It is also turned off when DP attention is used (section 6) or when deterministic inference is requested.
(FlashInfer also has variants of this kernel that additionally quantize the normalized result to FP8 or NVFP4 for the next layer. I will leave those for another time.)
By the way, this kind of fusion is not specific to SGLang. As mentioned in section 3, the kernel was originally written for TensorRT-LLM, and vLLM uses the same FlashInfer kernel through a compilation pass that looks for an all-reduce followed by an RMSNorm. So, merging communication into the computation around it seems to be a general trend among inference engines. We will see an even more aggressive example of this in section 7.
6. DP Attention
So far, we have looked at ways to make the two sums per layer faster. The remaining two mechanisms take a different route: they change how the model is split across the GPUs so that the sums are not needed in the first place. Let's start with DP attention.
6.1 Data-Parallel Attention
The motivation for DP attention comes from models with Multi-Head Latent Attention (MLA). This includes the DeepSeek models and also GLM-5.3, whose attention module follows the design of DeepSeek V3.2. The SGLang documentation explains the problem as follows:
However, TP might not be the most efficient strategy for certain models. For example, DeepSeek models use MLA and only have one KV head.
Why is a single KV head a problem? With tensor parallelism, the attention heads are distributed across the GPUs. But if there is only one KV head, there is nothing to distribute, so every GPU ends up storing the full KV cache for every request. With 8 GPUs, that's 8 copies of the same cache.
DP attention avoids this by not splitting the attention module at all. Instead, each GPU runs the complete attention module, but only for its own share of the requests, and it only keeps the KV cache for those requests. (The "DP" stands for data parallelism.) The feed-forward or MoE module, on the other hand, is still split via tensor parallelism, as shown in Figure 11 below.
As we can see on the right side of Figure 11, there is no all-reduce after the attention module anymore, simply because attention is no longer split. (The gather and reduce-scatter steps that appear in its place are the topic of the next section.) In SGLang, DP attention is enabled via the --attn-dp-size flag.
6.2 Gather and Reduce-Scatter
Of course, this doesn't come for free, because the MoE module is still tensor-parallel and needs every GPU to work on every token. This is where the siblings of the all-reduce from section 1.2 come in.
After the attention module, each GPU only has the hidden states for its own requests. That's why a gather step collects the rows from all GPUs before the MoE module, so that every GPU has all of them. After the MoE module, every GPU holds a partial sum over all rows, and a reduce-scatter adds up the partial sums and gives each GPU back only its own rows.
The gather step is implemented in two different ways, and I find the second one quite neat. If all GPUs have a similar number of tokens, each GPU pads its rows to the length of the longest one, and a regular all-gather concatenates them.
However, if the numbers are very different, padding becomes wasteful. This happens, for example, when one GPU is prefilling a 4,000-token prompt while the other 7 GPUs are decoding 10 tokens each: we would send 8 × 4,000 rows to transport 4,070.
In this case, SGLang uses an all-reduce instead. Each GPU writes its rows at its own offset into a zero-filled buffer that is as long as all rows combined, and then the buffers are summed. Since adding zeros doesn't change anything, the sum is effectively a concatenation:
GPU 0 writes: [ a a a 0 0 0 0 0 ]
GPU 1 writes: [ 0 0 0 b b 0 0 0 ]
GPU 2 writes: [ 0 0 0 0 0 c c c ]
all-reduce: [ a a a b b c c c ]
SGLang picks whichever of the two moves less data. (In the code, a row in an all-reduce counts as twice as expensive as a row in an all-gather. My guess is that this reflects that an all-reduce is a reduce-scatter plus an all-gather, as we saw in section 1.2.)
7. Expert Parallelism
DP attention takes care of the attention half of the layer. Expert parallelism (EP) does something similar for the other half, the MoE module.
As a quick recap, with tensor parallelism, every GPU holds a slice of every expert. This way, every GPU does a bit of the work for every token, and the results are summed via an all-reduce at the end.
With expert parallelism, each GPU instead holds a subset of the experts, but holds them in full. For example, GLM-5.3 has 256 experts per MoE module, so with 8 GPUs, each GPU holds 32 of them. The consequence is that a token is usually not on the GPU where its experts are, which means that the tokens now have to travel.
To see how this works, let's follow a single token. The router picks 8 of the 256 experts for it, and these 8 experts are typically spread over several GPUs. In the first step, which is called dispatch, the token's hidden state is sent to every GPU that holds one of these experts. Each GPU then ends up with a pile of tokens from all the other GPUs, grouped by expert, and runs its experts on them.
In the second step, which is called combine, the results travel the other way. Each expert output is weighted by the score that the router gave to this expert and sent back to the GPU that the token came from. There, the 8 outputs for the token are added up, which gives the output of the MoE module. Both steps are shown in Figure 12 below.
Both dispatch and combine are all-to-all operations (the fourth collective in section 1.2), since every GPU sends something different to every other GPU. And in contrast to an all-reduce, the amount of data that travels between two GPUs is not fixed. It depends on which experts the router picked, so a GPU that happens to hold popular experts receives more tokens than the others.
SGLang doesn't implement dispatch and combine itself but delegates them to a backend that is selected via --moe-a2a-backend. The most popular one is DeepEP, the communication library that DeepSeek open-sourced alongside its models. (In SGLang, the current generation of DeepEP is selected with --moe-a2a-backend deepep_v2.)
DeepEP was rewritten from the ground up for its second version, and the latest release at the time of writing is V2.5. So, if you have read about DeepEP before, a few things are different now:
- Earlier versions had two separate sets of kernels, one for throughput (prefill) and one for low latency (decoding). Now, there is a single interface for both, and the caller only passes different options.
- The dependency on NVSHMEM is gone. DeepEP now builds on NCCL instead (more precisely, on a lightweight device-side API in NCCL), and it can reuse existing NCCL communicators.
- The number of SMs that the kernels use is calculated analytically, so the auto-tuning step of earlier versions is no longer needed.
What hasn't changed is the transport: DeepEP still uses NVLink within a machine and RDMA between machines. It requires Hopper or newer GPUs, which fits the scope of this article.
Interestingly, the split between small and large messages from section 1.3 is still visible here, even though it's no longer a separate mode. For example, during decoding, SGLang asks DeepEP to dispatch without synchronizing with the CPU so that the call can be recorded in a CUDA graph, whereas during prefill, it uses the synchronized variant.
(I would have liked to include some numbers here. However, the current DeepEP README doesn't contain any benchmarks, and the numbers that are often quoted are from the first version, which has been removed from the repository.)
DeepSeek has since taken this one step further. Their kernel library DeepGEMM contains a kernel called Mega MoE, which, in the words of its README, "fuses and overlaps EP dispatch, linear 1 and linear 2 [...], SwiGLU, and EP combine into a single mega-kernel, overlapping NVLink communication and tensor core computation".
In other words, there is no separate dispatch kernel, expert kernel, and combine kernel anymore. If we look at the source code, the threads of this one kernel are divided into groups with different jobs. One group pulls the tokens for the local experts directly out of the memory of the other GPUs. A second group runs the experts on a block of tokens as soon as this block has arrived, which means that the computation already starts while the rest of the tokens are still on their way. And a third group writes each result straight into a buffer on the GPU that the token came from, where the results for each token are summed at the end.
This is the same idea as the fused all-reduce in section 5, just applied to the MoE module: communication is no longer a separate step that the computation has to wait for. (In SGLang, Mega MoE is available as another choice for --moe-a2a-backend, called megamoe. Like MNNVL, it relies on direct memory access via NVLink.)
In practice, DP attention and expert parallelism are usually combined. With both enabled, neither of the two all-reduces from section 1.1 is left: the attention module is not split anymore, and the MoE module communicates via dispatch and combine instead.
8. Choosing a Layout
At this point, we have seen all the mechanisms. What we haven't talked about yet is when to use which, since tensor parallelism, DP attention, and expert parallelism each fit a different situation. In this section, let's go over the three considerations that decide most setups in practice.
8.1 Low Latency or High Throughput
Let's start with the attention module. As a rule of thumb, tensor-parallel attention is the choice for low latency, and DP attention is the choice for high throughput.
The reason is that with tensor parallelism, all GPUs work on the same request at the same time, so that each decoding step finishes as fast as possible. With DP attention, a request only gets a single GPU for its attention module, which is slower for this one request. However, there is no all-reduce after the attention module anymore, and the KV cache is no longer duplicated, which leaves room for much larger batches.
This rule of thumb also shows up in the SGLang cookbook. The low-latency recipe for GLM-5.3 uses plain tensor parallelism on 8 GPUs, whereas the high-throughput recipe adds DP attention and expert parallelism on top.
8.2 Aggregated and Disaggregated Serving
The second consideration is whether prefill and decoding run on the same server.
In an aggregated deployment (the SGLang documentation calls this a unified engine), a single server handles both phases, which means that we have to pick one layout for both. If we pick tensor parallelism because we care about decode latency, prefill runs through the same tensor-parallel layers. And this is where the 96 MB all-reduces from section 1.3 come from.
This happens in practice. For example, the aggregated recipe for GLM-5.2 on 8 B200 GPUs in the open InferenceX benchmark uses plain tensor parallelism and feeds the prompts through in chunks of 8,192 tokens, which is exactly the 96 MB message. (GLM-5.2 is the predecessor of GLM-5.3 and has the same architecture. At the time of writing, InferenceX only has SGLang recipes on NVIDIA GPUs for the former.)
In a disaggregated deployment, on the other hand, prefill and decoding run on separate servers, and the KV cache is sent from one to the other. Now, each server can get the layout that fits its job. In the disaggregated recipes for the same model and GPUs, the prefill server uses DP attention together with expert parallelism, since it should process as many prompt tokens per second as possible. The decode side, in contrast, consists of several workers with plain tensor parallelism on 4 GPUs each, since each of them should respond as fast as possible. Figure 13 below compares the two deployments.
As we can see in Figure 13, the large all-reduce only exists on the left. On the right, the server that sees the large batches doesn't split the attention module at all, and the workers that split it only ever see small batches.
(These layouts are not the only possible ones, of course. When the same recipes are run with many concurrent requests, the decode server switches to DP attention and expert parallelism as well, which is the throughput side of the rule from section 8.1. And on H200 GPUs, the prefill server uses context parallelism instead, which is one of the things I am leaving out of this article.)
So, coming back to the question from section 1.3: one setup has to deal with both ends of the message-size axis only if we ask a single server to do both jobs. In other words, large all-reduces mainly come up in aggregated deployments.
8.3 Beyond a Single Machine
The third consideration is what to do when the model needs more GPUs than a single machine has. (For example, the unquantized GLM-5.3 weights take up about 1.5 TB, which is more than 8 H200 or B200 GPUs can hold.)
The set of GPUs that can reach each other via NVLink is called an NVLink domain. In a machine with 8 GPUs, such as the B300 machine I used for the measurements, the domain is the machine itself. As soon as we use two of these machines, the data that travels between them has to go through a regular network such as InfiniBand.
This matters a lot for tensor parallelism. All the tricks from sections 3 to 5 (writing directly into the memory of another GPU, multicast, the fused kernel) only work inside an NVLink domain. If a tensor-parallel group spans two machines, each of the 156 sums in a forward pass has to cross InfiniBand instead, and none of these tricks apply there. (The SGLang documentation makes a similar remark, noting that tensor parallelism "frequently encounters communication bottlenecks during multi-node deployments".)
That's why I would suggest a simple rule here: a tensor-parallel group should never be wider than one NVLink domain. On B300 machines, this means at most 8 GPUs.
So, how do we use 16 GPUs then? This is where expert parallelism comes in handy, since DeepEP is designed to send tokens between machines via RDMA. The expert-parallel group can span all 16 GPUs, while the attention module stays inside each machine. Ideally, we use DP attention for it, which needs no all-reduce at all. Alternatively, we can use tensor-parallel attention groups of at most 8 GPUs. Figure 14 below illustrates the idea.
The important thing to notice in Figure 14 is that the tensor-parallel group is no longer "all the GPUs". It is a subgroup of a larger world, and only expert parallelism spans that world.
(A quick note on the flags, since they can be confusing here: in SGLang, --tp sets the total number of GPUs, and --attn-dp-size divides them into attention groups. For example, --tp 16 --attn-dp-size 2 on two machines results in two tensor-parallel attention groups with 8 GPUs each, one per machine. The 16-GPU decode server in the InferenceX recipes uses 16 attention groups together with expert parallelism over all 16 GPUs, which leaves no tensor-parallel attention group at all.)
This layout is not specific to SGLang either. The InferenceX recipes for Kimi K3 on vLLM do the same on four H200 machines: tensor-parallel groups of 8 GPUs, and expert parallelism over all 32 GPUs. (To be fair, the latency-oriented recipe in the same directory uses tensor-parallel groups of 16, which span two machines. So, the rule above is a default, and it does get broken when latency is the main concern.)
The picture changes when the NVLink domain itself is larger than one machine, and this is where MNNVL got its name from. In Blackwell rack systems such as the GB300, a single NVLink domain spans up to 72 GPUs in multiple machines, so a tensor-parallel group can span several machines as well. From the perspective of the kernels, nothing changes in this setup: a GPU in another machine is just one more pointer in the table from section 3.1, and the multicast address covers the buffers in all machines.
The only difference is in the setup. On a single machine, the GPUs share their buffers by passing file descriptors between the processes. Across machines, that's not possible, so they exchange so-called fabric handles instead. Once the buffers are mapped, one-shot and two-shot run exactly as described above.
The opposite case is a machine without any NVLink. There is no multicast on such a machine, so MNNVL can't be used there, and we are back to NCCL.
9. What I Left Out
This article doesn't cover all the communication that happens in SGLang. To keep it at a manageable length, I skipped the following:
- Pipeline parallelism, where consecutive groups of layers live on different GPUs, and the activations are handed from one stage to the next.
- Context parallelism, where a long prompt is split across GPUs along the sequence dimension.
- The KV cache transfer in a disaggregated deployment, where the prefill server hands its result over to the decode server.
Each of these would deserve its own article, so I will leave them for another time.
10. Summary
To wrap up, let's put the mechanisms back onto the message-size axis from section 1.3. Figure 15 below shows which one handles which message size for GLM-5.3 on 8 GPUs.
Reading Figure 15 from left to right, messages up to 128 KB go through one-shot, and larger messages go through two-shot. In both cases, the all-reduce is fused with the residual add and the RMSNorm as long as the batch has at most 2,048 tokens (24 MB). For the very large messages of a long prefill, there is still NCCL.
This last part is the one that is most likely to change. As we saw in section 8.2, aggregated deployments still send prefill-sized messages through tensor parallelism, and faster alternatives to NCCL's default algorithms are still being explored for them. One candidate is NCCL itself. Since version 2.27, NCCL supports symmetric memory, where the tensors are allocated at the same virtual address on every GPU so that NCCL can run low-latency kernels on them. (In SGLang, this is available via the --enable-symm-mem flag.) NVIDIA reports the largest gains for small messages, though, so how much this helps at prefill sizes still has to be measured.
DP attention and expert parallelism don't appear on this axis, since they change which collectives are needed in the first place.
So, coming back to the question from the beginning: on paper, adding the partial results back together is the same step it was in 2019, but a lot has changed underneath.
For the small messages that dominate decoding, direct NVLink access between all GPUs made it possible to replace 14 sequential steps with one or two, to drop the handshake, and to merge the sum into the operations around it. With Mega MoE, the same kind of merging has reached expert parallelism. For large messages, on the other hand, NCCL is still in use, and faster alternatives are still being explored.
The hardware keeps changing as well. Today, an NVLink domain has at most 72 GPUs. According to NVIDIA's roadmap, the Rubin Ultra generation will bring domains of 144 GPUs in a single rack and 576 GPUs across eight racks, and 1,152 GPUs are planned for the generation after that. So, the rule from section 8.3 will soon allow much wider groups than it does today, and it will be interesting to see which of the algorithms in this article still make sense at that scale. (For example, with the threshold from section 4.2, a group of 72 GPUs would use one-shot only for a batch of exactly 1 token.)