Two RTX 5090s, One Qwen Model, and a Thunderbolt Flow-Control Bug

Distributed inference failures rarely belong to one layer. A model can load correctly, NCCL can form a process group, the API can become reachable, and the system can still be slower—or less reliable—than a single GPU because the transport underneath it is silently retransmitting data.

This post is the complete story of one such experiment. We split a Qwen 27B NVFP4 model across two RTX 5090 workstations using tensor parallelism, enabled the model's native NEXTN speculative decoder, pushed context length close to 200K tokens, and then traced a large performance gap to Thunderbolt networking. A small, default-off Linux thunderbolt_net change eliminated receiver corruption on our controller pair and raised measured single-stream generation from 48.4 to 77.1 output tokens/s.

The ending matters as much as the speedup: we restored the stock driver and the original single-GPU production service. The patch is promising evidence for an opt-in mechanism, not proof that transmit-side end-to-end flow control is safe on every USB4 controller.

The short version

Our test system looked like this:

                       one Qwen 27B request
                                |
                    SGLang TP=2, PP=1, NEXTN
                         /              \
                RTX 5090                RTX 5090
                 host A                  host B
                    \                      /
                     NCCL over ordinary TCP/IP
                         Thunderbolt/USB4
                         private 20 Gb/s link

Each tensor-parallel rank loaded about 14.48 GiB of target-model weights and 2.65 GiB of native MTP draft weights. NEXTN used three speculative steps, top-k 1, and up to four proposed tokens per verification round.

After fixing an SGLang target/draft autotune-cache bug, the stock Thunderbolt driver could serve real TP2+NEXTN requests, including a near-200K-token input. But a six-request sample added receiver errors and thousands of TCP retransmissions. With a temporary tx_e2e=1 driver candidate, the same model topology produced no receiver CRC, missed, length, overrun, or total errors during the measured window and ran materially faster.

These are controlled observations from one hardware pair, not a statistically complete benchmark. The near-200K requests also reused a large cached prefix, so they prove capacity and correctness—not cold 200K prefill performance.

Why try tensor parallelism over Thunderbolt?

The two machines already served independent single-GPU Qwen replicas. That is the simplest and safest arrangement for throughput: each request stays local to one GPU, while a router distributes requests across replicas.

Tensor parallelism asks a different question. Instead of two independent models, can two consumer GPUs behave like one larger accelerator?

With TP2, every large matrix operation is split across the two GPUs. This can unlock a larger memory envelope and different performance points, but it also creates communication on nearly every layer. A transport problem that is almost invisible to independent replicas becomes part of the token-generation critical path.

Thunderbolt was attractive because the workstations already had a direct 20 Gb/s private link. It was not NVLink, InfiniBand, or PCIe peer-to-peer. NCCL used normal sockets over a ThunderboltIP network interface. That distinction became central later.

What NEXTN adds

NEXTN is the model's native speculative-decoding path. A smaller draft component predicts several next tokens, and the target model validates those candidates in a batch. If several guesses are accepted, the system advances multiple tokens per expensive target-model step.

Conceptually:

draft model:   proposes [t1, t2, t3, t4]
                         |
target model:  verifies the proposed block
                         |
result:        accepts a prefix, then repeats

Our configuration used 3/1/4: three speculative steps, top-k 1, and up to four proposed tokens. During the successful TX-E2E single-stream run, the observed acceptance length was 3.83–4.00. That shows the speculative path was active. It does not isolate NEXTN's speedup, because the final passing experiment did not have a matched TP2-without-NEXTN control. The experiment proves compatibility and absolute performance of the combined topology.

The experiment discipline

Changing a model service, a distributed runtime, and a kernel transport at the same time would make every result ambiguous. We therefore used a staged process:

1. Verify both hosts, GPUs, the direct link, storage, and the original model services.
2. Verify a real alternate inference route before temporarily borrowing both GPUs.
3. Run a TP2 baseline before adding speculation when possible.
4. Change one suspected layer at a time.
5. Measure real API requests, not just process startup or GPU allocation.
6. Record application metrics and transport counters in the same bounded window.
7. Remove the test jobs and temporary routing.
8. Restore the distribution driver and verify the original services with real requests.

This sounds conservative, but it prevented several false conclusions.

Failure 1: the fallback existed only on paper

The first attempt stopped before measuring TP at all. The gateway configuration contained a context-window fallback from the small role to a larger model, but not a general failure fallback. A direct request to the alternate backend succeeded, while a request through the normal small-model role failed when the Qwen workers were paused.

The lesson was simple: a healthy fallback target does not prove the client-facing route will use it. We added the supported, test-local general fallback, exercised the exact normal route during the outage, and removed it after each bounded experiment.

This distinction kept an infrastructure mistake from being misreported as a model or Thunderbolt result.

Failure 2: NCCL success was not inference success

In the next runs, both ranks formed the cross-host NCCL group and loaded the FP8 weights. That looked encouraging, but the rank on one host exited during CUDA initialization before serving a request.

The crash moved as we changed graph settings:

• With the default graph configuration, it appeared during prefill graph capture in a GDN Triton path.
• Disabling prefill CUDA graphs moved it to decode graph capture near cuModuleLoadData.
• Disabling both prefill and decode graphs moved it again, this time to the first multimodal warmup in a CUDA GELU kernel.

That progression falsified “prefill CUDA graph is the root cause.” Disabling a component had moved the symptom without removing the underlying failure.

We also observed a service container whose parent exited cleanly after a child scheduler crashed. A green container exit or a formed NCCL group therefore could not count as acceptance. Our acceptance condition became a real generated response, followed by repeated, concurrent, and long-context requests.

Failure 3: target and draft shared the wrong autotune-cache lifecycle

The decisive software failure occurred after the target model had initialized but while the native draft model entered FlashInfer autotuning and CUDA graph setup. TP weight loading and NCCL establishment had already completed.

The root cause was the lifecycle of FlashInfer autotune caches across speculative workers: target and draft initialization did not merge and preserve the cache state correctly. The upstream SGLang fix, commit fc5a979f, explicitly merges FlashInfer autotune caches across speculative workers, alongside related TP shard handling.

With that merged fix, both ranks completed:

• target and draft FlashInfer autotuning;
• target prefill and verification graph capture;
• draft prefill, decode, and extend graph capture;
• API startup and real inference without a restart.

This was the first trustworthy TP2+NEXTN result on the pair.

First passing result: functional, but transport-limited

With the stock distribution thunderbolt_net leaf, the combined system produced:

• 512 output tokens: HTTP 200, 159 ms TTFT, 10.73 s total, about 48.4 output tok/s;
• six concurrent requests: 1,536 output tokens in 8.89 s, or 172.9 aggregate tok/s;
• repeated 120K prefix: all 120,000 prompt tokens reported cached, 1.35 s total;
• long request: 199,860 input tokens plus 64 output tokens in 96.78 s, with 120,000 cached input tokens.

The acceptance length ranged from about 2 to 3 tokens during the sampled requests, confirming that NEXTN was doing useful work.

But the network counters were alarming. One repeated six-request sample moved roughly 5.58 GiB in each direction to produce 1,536 output tokens. During that window, one receiver added 109 link errors, while TCP retransmissions increased by 239 on one host and 1,787 on the other.

This gave us a useful separation:

• TP2, NEXTN, NCCL, long context, and prefix caching were functionally working.
• The transport was not maintaining integrity efficiently under the communication pattern.

Following the evidence down the stack

We then treated Thunderbolt as its own system rather than changing model flags.

Several tempting explanations did not survive controlled tests:

• “Just disable E2E.” Turning off receive-side E2E did not eliminate the problem and could make the path worse.
• “It is only too much concurrency.” One, two, and four TCP streams changed the shape of the result but did not remove receiver errors.
• “It is a checksum or GRO artifact.” Offload toggles did not explain the hardware receive-counter deltas.
• “It is simply a bandwidth ceiling.” USB4 stream tests could move byte-exact traffic near link rate when spread across independent stream paths.
• “NCCL is broken.” Ordinary network-only traffic could reproduce asymmetric retransmissions and receiver errors without the model runtime.

The useful clue was asymmetry: the faster sender could overrun a receive path and trigger CRC and missed-frame accounting, followed by TCP recovery. This made end-to-end credit flow control a plausible mechanism, not just a tuning knob.

Why Linux does not enable TX-E2E by default

Linux had already tried enabling end-to-end flow control for transmit rings—and reverted it. Upstream commit 1881f2ef documents controllers that accept the configuration but never return transmit credits. On those systems, enabling TX-E2E can stop all traffic.

That history rules out a global default. A safe experiment needed three properties:

1. preserve the upstream default;
2. require an explicit operator opt-in;
3. enable TX-E2E only when receive-side E2E is enabled and the peer advertises the capability.

The resulting candidate, commit 203ce9e7, adds a read-only tx_e2e module parameter that defaults to false. It does not infer safety from a device ID, because two controllers with the same nominal ID can still differ by firmware or implementation behavior.

The patch was built against the exact running kernel headers, passed strict checkpatch and diff --check, and was loaded only as a temporary leaf module. It was never installed under /lib/modules, added to initramfs, or made persistent.

Network-only A/B: integrity before model speed

Before returning to the model, we tested the transport directly.

A representative stock-direction control delivered 15.249 Gbit/s but reported 25,717 TCP retransmissions and 4,346 receiver errors. With the opt-in candidate, matched four-flow runs produced:

One hundred low-rate probes in each direction had zero loss and mean RTTs around 0.28–0.29 ms. The concurrent load-and-ping sample also had zero loss.

The important result was not “E2E always increases raw throughput.” In the stock control, the sender could sustain a high apparent useful rate while the receiver recorded corruption and TCP repaired the stream. TX-E2E introduced backpressure and preserved integrity on this pair.

A separate USB4-stream experiment also showed why a future high-performance design may need multiple independently credited paths. ThunderboltIP v1 negotiates one transmit path and stores one TX/RX ring pair; a local multiqueue change alone cannot create new peer-visible HopIDs. Multiple USB4 stream services can, and four such streams reached about 18.1 Gbit/s byte-exact in our tests. That is a protocol-design direction, not part of the small opt-in patch.

Repeating the real TP2+NEXTN workload

We then changed only the transport leaf and repeated the same model topology and request shapes.

Both ranks again formed TP2, loaded the target and draft weights, captured their graph sets, and served without restart. The results were:

During the TX-E2E model window, the private link moved about 283 GiB in each direction while remaining trained at 20 Gb/s. Both receivers retained zero total, CRC, missed, length, and overrun errors. TCP still recorded retransmissions—529 on one host and 318 on the other—so this was not a zero-retransmission application run. It was, however, a large improvement over the stock model sample's asymmetric 1,787 retransmissions plus receiver errors.

The single-stream gain was about 59%, and aggregate six-request throughput improved about 15%. The much larger single-stream improvement is consistent with reduced stalls in a latency-sensitive all-reduce path, while concurrent work can overlap some communication and computation. That interpretation is plausible, but a fuller profiler trace would be required to assign every millisecond.

What the experiment proves—and what it does not

It proves

• Qwen 27B NVFP4 can run as two-host TP2 across these two RTX 5090 systems.
• Native NEXTN can initialize and serve real, concurrent, cached-prefix, and near-200K requests after the SGLang cache-lifecycle fix.
• Thunderbolt receiver errors and TCP recovery materially affected this workload.
• Opt-in TX-E2E preserved receive-path integrity and improved measured inference performance on this controller pair.
• The full system could be restored to its original driver, routing, and single-GPU service state.

It does not prove

• that TX-E2E is safe as a global Linux default;
• that the candidate works on ASMedia, AMD, or every Intel USB4 controller;
• that TP2 is a better production architecture than two independent replicas;
• that NEXTN alone caused the measured speedup;
• that 200K cold prefill is fast—the long request reused roughly 119K cached tokens;
• that aggregate 199.6 tok/s means each request ran at 199.6 tok/s;
• that ThunderboltIP is GPUDirect or zero-copy GPU-to-NHI DMA.

That last boundary deserves emphasis. A separate GPUDirect investigation successfully exported an NVIDIA dma-buf only after a process-local test patch, but attaching it to the Thunderbolt NHI failed with ENOTSUPP: the GPU and NHI did not share a supported downstream PCIe topology. The TP2 tests here used ordinary NCCL sockets through the kernel network stack.

Why we rolled back a faster result

The candidate improved our measured system, but production defaults need broader evidence than one good pair.

The upstream revert exists because some controllers never return TX credits. A default-on change could turn a performance fix into a complete connectivity failure. We also lacked a controller/firmware compatibility matrix, long-duration soak testing, suspend/resume coverage, hotplug coverage, and independent review of the patch.

So the experiment ended by:

• deleting the bounded model jobs and temporary routing;
• reloading the distribution thunderbolt_net module on both hosts;
• verifying the candidate-only parameter and temporary copies were gone;
• confirming the link retrained at 20 Gb/s;
• checking both hosts, GPUs, and cluster membership;
• sending real requests through the restored direct backend, router, and public model roles.

“Rollback” here means restoring runtime state. No Git history was rewritten, and the candidate remains available for review.

The mistakes that were most educational

1. Testing the backend instead of the user route. A direct healthy fallback was irrelevant until the exact client-facing role failed over successfully.
2. Treating initialization as acceptance. NCCL connection, loaded weights, allocated VRAM, or a Ready container did not prove the scheduler could return one token.
3. Changing flags until a crash moved. Moving from prefill capture to decode capture to warmup was evidence against the first hypothesis, not progress toward validating it.
4. Combining layers too early. We had to separate model-runtime failures from network corruption before either could be fixed confidently.
5. Mixing throughput definitions. Single-request output rate, aggregate concurrent throughput, prefill time, and cached-prefix latency answer different questions.
6. Calling a cached long prompt a cold-context benchmark. The near-200K case demonstrated addressable context and correctness, not cold prefill speed.
7. Ignoring negative upstream history. A patch that works locally can still reproduce a known no-progress failure elsewhere; that is why tx_e2e defaults off.
8. Stopping after the exciting number. The restoration test was part of the experiment, not cleanup after it.

Broader lessons for distributed inference

The largest lesson is that distributed inference is a whole-stack workload. Per-token latency can depend on model architecture, speculative acceptance, CUDA graph behavior, collective scheduling, TCP recovery, driver ring semantics, firmware credit behavior, and physical topology at the same time.

A useful debugging order is:

real API correctness
  -> runtime and graph lifecycle
    -> NCCL collective behavior
      -> TCP and interface counters
        -> driver rings and protocol credits
          -> PCIe / USB4 physical topology

Measure at every boundary. Keep a known-good control. Change one layer at a time. Treat negative experiments as results. And never retain a kernel behavior merely because it produced the fastest benchmark once.

Where this goes next

The immediate kernel path is review of the default-off TX-E2E opt-in, together with a broader controller and firmware matrix. The performance path is a backward-compatible way for ThunderboltIP to negotiate multiple independently credited paths rather than pretending local netdev queues alone can create them.

For inference, the next useful measurements are repeated cold-prefix and warm-prefix trials, per-collective NCCL traces, longer concurrency sweeps, and a direct comparison with two independently routed single-GPU replicas. For true GPU-direct transport, the hardware topology must first provide a supported GPU-to-I/O peer path; software cannot patch around a missing PCIe relationship.

The most valuable result was not 77.1 tokens/s. It was a causal chain we could defend:

shared target/draft autotune state was broken
  -> upstream SGLang fix made TP2+NEXTN serve real requests
    -> stock Thunderbolt showed receiver corruption and retransmission
      -> opt-in TX-E2E removed receiver errors on this pair
        -> the same model workload became materially faster
          -> stock production state was restored because safety evidence is not universal

That is the standard we should apply to systems experiments: not merely “it ran,” but a reproducible explanation of why it failed, why the fix helped, what remains uncertain, and how the original service was recovered.

References

• SGLang fix: merge FlashInfer autotune caches across speculative workers
• Linux revert explaining controllers that fail to return TX-E2E credits
• Experimental default-off TX-E2E opt-in candidate
• SGLang server arguments