The Split Personality of AI Inference

 How LLM-D Parallel Runs Are Rewriting the Rules of Model Inference

Copyright: Sanjay Basu

---

When One Brain Isn’t Enough

What if the secret to making AI faster wasn’t building bigger machines, but teaching it to think with two minds at once?

For anyone who’s ever typed a prompt into ChatGPT and watched those little dots dance across the screen, there’s an invisible orchestra playing behind the curtain. Large language models don’t just materialize answers from thin air. They’re running a two-act play every single time: first, they digest your question (prefill), and then they generate your answer, token by token (decode). Traditionally, these two acts happened on the same stage, using the same resources. And like any double-booked theater, chaos ensued.

Enter LLM-D, the distributed inference framework that said, “What if we gave each act its own theater?”

The result? A system that can serve AI models faster, cheaper, and more reliably by splitting the inference process into specialized workloads that run in parallel across multiple computing resources. It’s disaggregated serving, and it’s changing how we think about deploying AI at scale.

The Two-Phase Tango

Let’s start with something concrete. Every time you prompt an LLM, you’re triggering a two-phase process that’s as predictable as it is computationally expensive.

Prefill, the first phase, is compute-bound. It takes your entire prompt and processes every token in parallel, building what’s called a KV cache — essentially a memory of what you asked. This phase is brutally computational. If your prompt is 2,000 tokens long, the model has to do heavy mathematical lifting across all of them simultaneously. It’s like reading an entire book in one glance, then writing detailed margin notes.

Decode, the second phase, is memory-bound. It generates your response one token at a time, reusing that KV cache it built during prefill. This phase is lighter on computation but hungry for memory bandwidth. It’s the difference between lifting a heavy weight once (prefill) and doing a thousand lighter reps (decode).

Here’s the problem: traditional inference systems lump both phases together on the same GPU. They’re forcing your hardware to be a weightlifter and a marathon runner simultaneously. And as anyone who’s tried to specialize in everything knows, you end up mediocre at both.

The Interference Problem

Picture this scenario, courtesy of the engineering teams at Red Hat, Google, and IBM who built LLM-D: You’re serving a chatbot application. Ten users are having conversations (decode operations, lots of quick token generation). Suddenly, a new user arrives with a 2,600-token document and asks for a summary (massive prefill operation).

In a traditional system, that prefill request body-slams all ten ongoing conversations. Their smooth token generation stutters. Some requests timeout. Service level agreements get violated. Your users get that spinning wheel of doom.

Standard LLM deployments perform the prefill and decode phases of inference within a single replica, and given that these phases have different resource requirements, co-locating them leads to inefficient resource use, especially for long sequences.

Researchers call this “prefill-decode interference,” and it’s not just annoying. It’s expensive. When your P95 latency spikes by 3.7x because prefill operations are preempting ongoing generation, you’re not just providing a bad user experience. You’re burning money on overprovisioned hardware trying to muscle through the problem.

Disaggregation is The Radical Split

So what did the LLM-D team do? Something beautifully simple and deceptively complex: they split prefill and decode into separate workloads running on separate pods in a Kubernetes cluster.

The key innovation behind LLM-D is how it distributes inference by splitting the inference process into two distinct phases, prefill and decode, and running each in separate workloads. The project calls this approach “disaggregated serving.”

Think of it as specialization at the infrastructure level. Your prefill pods become expert readers, optimized for parallel computation. Your decode pods become expert writers, optimized for memory bandwidth and consistent token generation. Each can scale independently based on workload patterns.

But here’s where it gets interesting: disaggregation comes with an obvious cost. When prefill finishes processing your prompt, it needs to hand off that KV cache to a decode worker. That’s data transfer between GPUs or even between nodes. At first glance, shipping potentially gigabytes of cached data between workers sounds like a performance killer.

The researchers discovered something surprising. With proper placement, KV cache transfer overhead can be effectively minimized to be as low as less than the time of a decoding step, thanks to today’s high-speed networks such as NVLink and PCIe 5.0. For an OPT-175B model running on A100 GPUs, the transfer takes about 30–50ms, less than a single decode step.

The math checks out. If you’re running on 8-channel PCIe 5.0 (64GB/s per link) or NVIDIA NVLink (600GB/s), even large KV caches transfer faster than you can blink.

The Intelligent Scheduler

Disaggregation alone isn’t enough. You need something smart routing traffic. That’s where LLM-D’s inference scheduler comes in, leveraging the Kubernetes Gateway API’s Inference Extension.

Traditional load balancers use round-robin routing. Server 1, Server 2, Server 3, repeat. It’s democratic, predictable, and completely ignorant of what’s actually happening inside your model servers. Traditional load balancers treat LLM workers like identical black boxes, missing cache reuse opportunities, causing load imbalances, and increasing latency when conversations get routed to the wrong replicas.

LLM-D’s inference scheduler implements filtering and scoring algorithms necessary to make “smart” scheduling decisions around disaggregated serving, prefix-cache-awareness, and load-awareness.

The scheduler looks at runtime telemetry from vLLM instances. How full is the KV cache? What’s the work queue depth? Does this worker already have cached prefixes that match the incoming request? It scores each potential destination and routes intelligently.

Here’s a concrete example from the Solo.io team’s deep dive. A request comes in with a 5-token prompt asking for code review. The scheduler checks the prompt length against a configurable threshold (let’s say 100 tokens). Since 5 < 100, disaggregated inference isn’t needed. The overhead wouldn’t be worth it. The scheduler eliminates prefill-only workers from consideration and routes to a combined “both” worker or a decode worker with low load.

But when a request with a 2,600-token document summary hits? Different story. The scheduler triggers disaggregation, sends the prefill to a specialized prefill worker, then orchestrates the KV cache transfer to a decode worker with good cache hit probability.

The Competition Heats Up

LLM-D isn’t operating in a vacuum. The race to optimize distributed inference is crowded with brilliant minds and competing architectures.

vLLM, the inference engine that LLM-D builds upon, pioneered PagedAttention and continuous batching. vLLM supports distributed tensor-parallel and pipeline-parallel inference using Megatron-LM’s tensor parallel algorithm, with Ray as the default distributed runtime for multi-node inference. It’s fast, it’s widely adopted, and crucially, it now supports disaggregated serving as a pluggable feature through its KV Connector API.

NVIDIA’s TensorRT-LLM brings hardware-software co-design to the table. TensorRT-LLM delivers breakthrough performance on the latest NVIDIA GPUs with optimizations including FP8 and NVFP4 quantization, disaggregated serving, wide expert parallelism, and advanced speculative decoding techniques. When you’re running DeepSeek-R1 on Blackwell GPUs, TensorRT-LLM achieves world-record inference performance. NVIDIA also released Dynamo, a distributed inference framework that enables seamless scaling of inference workloads across GPU nodes and dynamic GPU worker allocation to efficiently respond to fluctuating user demand.

SGLang takes a different optimization angle. SGLang introduced RadixAttention, which reuses shared prompt prefixes across multiple requests, achieving up to 5x higher throughput compared to existing systems. It’s deployed at massive scale, over 300,000 GPUs worldwide, powering xAI, LinkedIn, and major cloud providers. SGLang’s zero-overhead batch scheduler keeps GPUs continuously engaged by running one batch ahead, eliminating CPU bottlenecks that traditionally cause idle GPU time.

Microsoft’s DeepSpeed-FastGen went a different direction with its Dynamic SplitFuse technique. Rather than fully disaggregating phases, DeepSpeed-FastGen splits lengthy prefills into smaller chunks and combines them with decoding tasks in the same batch — a process called piggybacking. The result: up to 2.3x higher throughput and 3.7x lower P95 latency compared to vLLM in their benchmarks.

And there’s Ray Serve, which takes a more general-purpose approach. Ray Serve is framework-agnostic, supporting everything from PyTorch and TensorFlow models to arbitrary Python business logic, with features for batching, streaming responses, and multi-model composition. Recently, Ray added support for custom request routing with PrefixCacheAffinityRouter, achieving 60% reduction in time-to-first-token and 40% improvement in throughput by keeping related requests together.

Then there’s the research that started it all. DistServe. DistServe’s disaggregation approach demonstrated that you can serve 7.4x more requests or achieve 12.6x tighter SLO compared to state-of-the-art systems while staying within latency constraints for over 90% of requests. The techniques pioneered in DistServe’s academic research have since been implemented in production systems like vLLM, validating the real-world impact of disaggregated serving.

The Performance Numbers Don’t Lie

Let’s talk brass tacks. Does disaggregation actually work in production?

LLM-D demonstrated up to 2.2k output tokens per second per GPU on DeepSeek with Expert Parallel serving on H200 GPUs, and provides up to 3x better P90 latency on long prefill with predicted latency balancing.

The Google Cloud team, one of LLM-D’s founding contributors, integrated these optimizations into their infrastructure running at “billion-user scale.” When you’re serving AI to that many people, every percentage point of efficiency translates to millions in infrastructure savings.

But here’s the nuance. Disaggregation isn’t always the answer. If your workload is too small or your GPU setup isn’t tuned for disaggregation, performance can drop by 20–30%. For shorter prompts or when the decode engine has a high prefix cache hit rate, running prefill locally on the decode worker is often faster and simpler.

That’s why sophisticated schedulers matter. They need to know when to disaggregate and when not to. It’s not about following a rigid rule; it’s about adapting to the workload in real-time.

Open Source, Open Minds

One of the most refreshing aspects of this entire space is how aggressively open-source it is. LLM-D is hosted under the LLM-D community with contributions from Red Hat, Google, IBM, CoreWeave, NVIDIA, AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI. SGLang is hosted under the non-profit LMSYS organization. vLLM, TensorRT-LLM, DeepSpeed, and Ray are all open-source with thriving communities.

This isn’t just academic altruism. The reality is that no single company can optimize for every model, every accelerator, every use case. The combinatorial explosion of configurations demands a community approach.

LLM-D supports NVIDIA GPUs, AMD GPUs, Google TPUs, and Intel XPUs with tested configurations and common operational patterns to improve production reliability. It’s not about vendor lock-in. It’s about interoperability.

The code is on GitHub. The papers are on arXiv. The benchmarks are reproducible. You can spin up your own disaggregated serving cluster this afternoon if you’re so inclined. That transparency breeds trust and accelerates innovation.

The Broader Philosophical Question

Strip away the technical jargon, and disaggregated serving raises an interesting question about specialization versus generalization.

For decades, the computing industry chased generalization. One server that does everything. One GPU that runs any workload. The promise of flexibility and simplicity.

But as workloads become more complex and scale becomes more demanding, we’re rediscovering the power of specialization. Not every problem needs the same hammer. Sometimes you need two different tools for two different jobs, even if that means more moving parts.

It’s the biological equivalent of specialized organs. Your stomach doesn’t also try to be your brain. Each does one thing exceptionally well, and the organism as a whole benefits from that division of labor.

Computer scientist John Hennessy, co-creator of the RISC architecture, once said, “The key to performance is elegance, not battalions of special cases.” Disaggregated serving is elegant precisely because it embraces specialization within a clean architectural pattern.

The efficiency gains aren’t coming from clever hacks or magic optimizations. They’re coming from fundamental alignment, matching computational characteristics (compute-bound vs. memory-bound) to hardware capabilities (lots of compute vs. lots of bandwidth).

The Future is Distributed and Disaggregated

Where does this go from here?

First, expect disaggregation to become the default, not the exception. As LLMs grow larger and context windows extend to millions of tokens (Claude 3 and Gemini 1.5 are already there), the resource mismatch between prefill and decode will only intensify. Systems that don’t disaggregate will be leaving enormous performance on the table.

Second, watch for more sophisticated routing algorithms. Today’s schedulers are smart about cache hits and load balancing. Tomorrow’s will likely incorporate real-time latency predictions, multi-objective optimization (cost vs. latency vs. throughput), and maybe even learned routing policies that adapt to specific workload patterns.

Third, hardware will co-evolve. NVIDIA’s Blackwell architecture, Intel’s Gaudi accelerators, AMD’s Instinct series. They’re all being designed with distributed inference in mind. Expect tighter integration between software orchestration layers like LLM-D and hardware capabilities like faster interconnects and more efficient KV cache management.

Finally, this isn’t just about chatbots. Disaggregated serving applies to any system with distinct computational phases. Batch processing pipelines, video transcoding, real-time analytics. The patterns discovered in LLM inference will ripple outward to other domains.

A Practical Call to Action

If you’re running LLMs in production, or thinking about it, here’s what you should do:

Start with your workload profile. Are your prompts highly variable? Do you have bursts of long-context requests mixed with chatbot-style short interactions? That variability is where disaggregation shines.

Benchmark your current system. Measure your P50, P95, and P99 latencies for both TTFT and TPOT. Know your baselines.

Try disaggregation on a pilot workload. LLM-D and vLLM both offer straightforward paths to experiment. You don’t need a massive cluster. Even a few GPUs can demonstrate the benefits.

Monitor the impact on goodput, not just throughput. Requests per second looks great until you realize half of them are violating SLOs. Goodput completed requests that meet latency requirements, and that is what actually matters for user experience.

And most importantly, stay curious. This field is moving fast. The system you deploy today will be outdated in six months. That’s not a bug; it’s a feature. We’re in the golden age of infrastructure innovation for AI.

The Orchestra Playing in the Shadows

So the next time you prompt an LLM and watch those tokens stream back, remember. There’s an invisible orchestra performing a carefully choreographed dance. Prefill workers digesting your question in parallel. KV caches flying across high-speed networks. Decode workers generating responses token by token. Intelligent schedulers routing traffic based on real-time telemetry.

It’s all happening in milliseconds. And it’s all disaggregated.

The future of AI isn’t just bigger models. It’s smarter infrastructure. It’s knowing when to split the work and when to combine it. It’s specialization without fragility. It’s performance without waste.

As computer scientist Butler Lampson observed, “All problems in computer science can be solved by another level of indirection.” Disaggregated serving adds exactly that, a level of indirection that transforms interference into orchestration, bottlenecks into specialization.

And honestly? That’s pretty elegant.

After all, if AI can teach itself to write poetry and solve math problems, surely we can teach our infrastructure to think with two minds at once.

---

Further Reading:

1. LLM-D GitHub Repository: https://github.com/llm-d/llm-d
2. DistServe Research Paper: “Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving”
3. vLLM Documentation: https://docs.vllm.ai/
4. NVIDIA TensorRT-LLM: https://developer.nvidia.com/tensorrt-llm
5. SGLang Project: https://github.com/sgl-project/sglang

---

Appendix

H200 Benchmarks & Code Examples

The Numbers That Matter. H200 Performance in the Real World

Let’s cut through the marketing and look at what actually happens when you run disaggregated inference on NVIDIA’s H200 GPUs.

The H200 isn’t just an incremental upgrade. It’s a memory beast with 141GB of HBM3e at 4.8 TB/s bandwidth — 76% more memory and 43% faster bandwidth than the H100’s 80GB at 3.35 TB/s. Same compute FLOPs, but fundamentally different memory profile. For LLM inference, which is overwhelmingly memory-bound during decode, that matters.

TensorRT-LLM on H200. The Baseline

NVIDIA’s own TensorRT-LLM benchmarks show what the hardware can do when optimized to the metal:

Llama2–13B Single GPU:

1. H200: 11,819 tokens/second
2. H100: ~6,200 tokens/second

1.9x speedup

Llama2–70B (TP8 — Full HGX):

1. Offline summarization (ISL/OSL: 2048/128): 1.9x more performant on H200
2. Online chat (ISL/OSL: 80/200): 1.6x more performant on H200

For GPT-3 175B at maximum throughput, H200 delivers 1.6x improvement over H100 on a full 8-GPU node. Not revolutionary, but substantial when you’re burning thousands of dollars per hour on inference.

vLLM: The Open-Source Champion

The vLLM community ran comparative benchmarks on H200 vs H100 using Llama 3.1 8B across multiple workload types:

Table 1

More importantly, the H200 showed dramatically better KV cache efficiency:

Cache hit rate: 98–99% across workloads (vs. 89–94% on H100)

Waiting requests: Near zero on H200, 5–15 queued on H100 under load

This isn’t just about raw speed. It’s about consistency under pressure.

Multi-GPU Scaling: Where Things Get Interesting

Running the vLLM throughput benchmark on Llama 3.1 8B with data parallelism (independent instances on each GPU), researchers measured scaling efficiency:

NVIDIA H200:

Table 2

NVIDIA H100:

Table 3

The H200 maintains a consistent 9–10% performance advantage across all scaling levels. That 141GB of memory keeps more of the model and KV cache in fast storage, reducing bottlenecks.

SGLang on H200: DeepSeek-V3 at Scale

DataCrunch ran extensive benchmarks on DeepSeek-V3 (671B parameters, 37B active) using SGLang v0.4.1 on H200:

Single Node (8xH200) vs (8xH100):

Throughput improvement: 25–30% higher on H200 for long-context workloads

Memory bandwidth utilization: 87% on H200 vs. 76% on H100

TTFT (Time to First Token): Similar (compute-bound phase)

TPS (Tokens Per Second): 1.35x higher on H200 (memory-bound decode)

The key insight: H200’s extra memory allows BF16 precision on Llama 405B without multi-node setup, eliminating expensive cross-node communication overhead. That’s a deployment simplification worth real money.

The Disaggregation Advantage: LLM-D Results

Google Cloud’s early tests with llm-d on H200 showed the disaggregation benefit clearly:

Code Completion Workload (Short prompt, quick response):

Standard vLLM (H100): 1,247 req/s, 14ms TTFT

llm-d disaggregated (H200): 2,489 req/s, 7ms TTFT

2x improvement in both metrics

Document Summarization (Long prompt, medium response):

Standard vLLM (H100): 342 req/s, 87ms TTFT

llm-d disaggregated (H200): 623 req/s, 41ms TTFT

1.8x throughput, 2.1x faster TTFT

The disaggregation scheduling overhead? Negligible. Transfer times for KV caches on PCIe 5.0 or NVLink averaged 28–35ms — less than a single decode step.

Code Examples

Now let’s make this real

Setting Up vLLM on H200 (Basic)

The simplest path to H200 inference:

# Install vLLM (supports H200 out of the box)
pip install vllm

# Serve Llama 3.1 70B on 2xH200 with tensor parallelism
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --port 8000 \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

For H200’s larger memory, you can push GPU memory utilization higher:
# Utilize H200's 141GB more aggressively
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --block-size 32 \
  --max-num-batched-tokens 1024 \
  --max-num-seqs 512 \
  --gpu-memory-utilization 0.95
Benchmarking Your Setup
Run the official vLLM benchmark to see real performance: 
# Clone vLLM repo
git clone https://github.com/vllm-project/vllm.git
cd vllm/benchmarks

# Run throughput benchmark
python benchmark_throughput.py \
  --backend vllm \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --num-prompts 1000 \
  --dtype bfloat16
Expected results on 1xH200:
Throughput: 52.3 requests/s, 9,847 total tokens/s
Mean TTFT: 18.2ms
Mean TPOT: 2.1ms
Mean ITL: 2.3ms
Deploying LLM-D with Disaggregation on Kubernetes
This is where it gets powerful. LLM-D orchestrates prefill and decode as separate workloads:
# llm-d-deployment.yaml
apiVersion: inference.x-k8s.io/v1alpha1
kind: InferencePool
metadata:
  name: llama-70b-disaggregated
spec:
  model: meta-llama/Meta-Llama-3.1-70B-Instruct
  
  # Prefill workers: compute-optimized
  prefillConfig:
    replicas: 3
    resources:
      nvidia.com/gpu: "2"  # 2xH200 per prefill worker
    vllmArgs:
      - "--dtype=bfloat16"
      - "--tensor-parallel-size=2"
      - "--max-model-len=8192"
      - "--disable-log-requests"
  
  # Decode workers: memory bandwidth-optimized
  decodeConfig:
    replicas: 6
    resources:
      nvidia.com/gpu: "1"  # 1xH200 per decode worker
    vllmArgs:
      - "--dtype=bfloat16"
      - "--max-model-len=8192"
      - "--max-num-seqs=256"
  
  # Intelligent scheduling configuration
  scheduler:
    enableCacheAwareRouting: true
    enableLoadBalancing: true
    disaggregationThreshold: 100  # tokens
Deploy it:
# Install llm-d using Helm
helm repo add llm-d https://llm-d.github.io/helm-charts
helm repo update

# Install with H200-optimized settings
helm install llama-serve llm-d/llm-d \
  --set accelerator=nvidia-h200 \
  --set model=meta-llama/Meta-Llama-3.1-70B-Instruct \
  --set disaggregation.enabled=true \
  --set scheduler.cacheAware=true

# Verify deployment
kubectl get inferencepool
kubectl get pods -l app=llm-d
SGLang Setup for Maximum H200 Performance
SGLang requires a bit more configuration but delivers impressive throughput:
# Install SGLang
pip install "sglang[all]" --find-links https://flashinfer.ai/whl/cu124/torch2.4

# Start SGLang server with disaggregation
python -m sglang.launch_server \
  --model-path meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tp 2 \
  --port 30000 \
  --mem-fraction-static 0.85 \
  --context-length 8192 \
  --enable-mixed-chunk \
  --chunked-prefill-size 8192 \
  --disable-cuda-graph
For multi-node disaggregation on H200:
# SGLang disaggregated serving config
import sglang as sgl

# Launch prefill workers
sgl.Runtime(
    model_path="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tp_size=4,  # 4xH200 for prefill
    is_prefill_node=True,
    kv_cache_dtype="fp8",
    mem_fraction_static=0.90,
    port=30000
)

# Launch decode workers (different machines)
sgl.Runtime(
    model_path="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tp_size=2,  # 2xH200 per decode worker
    is_decode_node=True,
    connect_to_prefill="192.168.1.10:30000",
    kv_cache_dtype="fp8",
    mem_fraction_static=0.92,
    port=30001
)
TensorRT-LLM: Maximum Performance
For absolute maximum performance on H200, TensorRT-LLM with FP8 quantization:
# Build optimized engine for H200
trtllm-build \
  --checkpoint_dir ./llama-70b-hf \
  --output_dir ./llama-70b-trt-h200 \
  --gemm_plugin bfloat16 \
  --max_batch_size 256 \
  --max_input_len 2048 \
  --max_output_len 512 \
  --use_fp8 \
  --strongly_typed

# Run inference server
mpirun -n 2 --allow-run-as-root \
  python3 ../run.py \
  --engine_dir ./llama-70b-trt-h200 \
  --tokenizer_dir ./llama-70b-hf \
  --max_output_len 512 \
  --input_text "Explain quantum computing"
Expected performance on 2xH200:
Input tokens/s: 8,400
Output tokens/s: 2,100
TTFT: 12ms
Generation latency: 2.8ms/token
Cost-Performance Analysis: Does H200 Justify the Premium?
Let’s do the math. RunPod pricing (as of this benchmark):
H100: $2.69/hour
H200: $4.54/hour (69% premium)
On Llama 3.1 8B single GPU:
H100: 23,243 tok/s = 8,642 tok/s per dollar/hour
H200: 24,876 tok/s = 5,480 tok/s per dollar/hour
Wait, that looks worse for H200! But here’s the nuance:
For batch/throughput workloads where latency doesn’t matter: H100 wins on cost efficiency.
For latency-sensitive workloads (chat, code completion, real-time agents):
H200’s lower TTFT (7ms vs 14ms) enables 2x more concurrent users within SLO
H200’s better cache utilization reduces queue waiting times by 80%
Effective user capacity: H200 serves 1.8x more users per GPU
Adjusted cost-per-active-user:
H100: $2.69 / 180 users = $0.0149/user/hour
H200: $4.54 / 325 users = $0.0140/user/hour
For latency-constrained production workloads, H200 is actually cheaper per served user.
The Verdict
When to Use What
Choose H200 when
• Running models >100B parameters (memory capacity matters)
• Serving latency-sensitive applications (chat, code completion)
• Processing long contexts (>8K tokens)
• Using disaggregated architectures (better bandwidth utilization)
• Running mixture-of-experts models (DeepSeek, Qwen)
Stick with H100 when:
• Running smaller models (❤0B parameters)
• Doing batch/offline inference (latency-tolerant)
• Budget is the primary constraint
• Models fit comfortably in 80GB
Use B200 when available:
• You need 180GB memory for massive MoE models
• You’re running cutting-edge research models
• Your workload justifies the premium (benchmark first!)
The Production Reality
Here’s what the benchmarks don’t tell you: infrastructure complexity matters more than raw performance.
A perfectly tuned H200 setup with disaggregated serving, intelligent routing, and prefix caching might deliver 2.3x better goodput than a basic H100 deployment. But if your team spends three weeks debugging Kubernetes networking, RDMA configuration, and vLLM parameter tuning, you’ve burned more than the hardware savings.
Start simple. Deploy vLLM on H200 with standard settings. Measure your actual workload. Then incrementally add complexity:
Week 1: Baseline vLLM deployment, measure TTFT and throughput
Week 2: Enable prefix caching, tune batch sizes
Week 3: Deploy LLM-D for cache-aware routing
Week 4: Experiment with disaggregation on subset of traffic
Week 5: Full disaggregated deployment with load testing
Performance optimization is a marathon, not a sprint. The H200’s hardware advantages give you headroom to grow into as your workload scales and your team’s expertise deepens.
And honestly? That extra headroom might be the best feature of all.
The End