How to Set Up vLLM for Multi-GPU Inference

Large language models grow faster than single GPUs do. A 70B parameter model in full precision won't fit on an 80GB H100, and even when a model technically fits on one card, serving it to more than a handful of concurrent users on a single GPU quickly becomes a bottleneck. This is where vLLM's multi-GPU support comes in.

vLLM is one of the most widely used inference engines for self-hosted LLMs, built around PagedAttention and continuous batching for high-throughput serving. Its multi-GPU capability — tensor parallelism — lets you split a single model's weights across several GPUs so it can run at all, or run faster than one card allows. This guide covers setting it up from scratch on a multi-GPU dedicated server.

Who This Guide Is For

This tutorial is for anyone deploying a self-hosted LLM on a server with two or more NVIDIA GPUs — whether that's a pair of RTX 4090s, a 4x H100 node, or anything in between — and wants an OpenAI-compatible inference endpoint running locally instead of relying on a hosted API.

Prerequisites

• A Linux server (Ubuntu 24.04/26.04) with 2 or more NVIDIA GPUs

• NVIDIA drivers installed and working (nvidia-smi runs cleanly)

• Python 3.9 or later

• CUDA 11.8 or later

• A Hugging Face account and access token if you're pulling gated models (e.g. Llama models)

If you haven't set up GPU-accelerated Docker yet, see our guide on installing the NVIDIA Container Toolkit with Docker first — vLLM can run in a container using the same GPU passthrough.

Step 1: Understand Tensor Parallelism vs Pipeline Parallelism

Before touching any commands, it helps to know what you're actually configuring:

• Tensor parallelism splits individual layers of the model across GPUs. This is the default and preferred approach for GPUs within a single node, especially when they're connected via NVLink, since it needs frequent, low-latency communication between cards.

• Pipeline parallelism splits different layers of the model across GPUs or nodes, which tolerates higher latency between devices. This matters more for multi-node setups or GPUs without a fast interconnect (PCIe-only setups, for example).

For a single server with multiple GPUs, you'll almost always want tensor parallelism set to your GPU count, with pipeline parallelism left at its default.

Step 2: Check Your VRAM Math Before You Start

Before picking a tensor-parallel size, work out whether the model actually needs multiple GPUs, or whether you just want the throughput boost:

• FP16: each parameter takes 2 bytes, so a 70B model needs roughly 140GB — too large for a single 80GB card.

• FP8: each parameter takes roughly 1 byte, so a 70B model needs around 70-80GB — this can just about fit on a single H100 at FP8, but two cards give you headroom for KV cache and more concurrent requests.

• A100 caveat: A100s lack native FP8 Tensor Cores, so requesting FP8 on that hardware will either error or silently fall back to FP16. Plan your GPU count around FP16 math if you're on A100s.

Rule of thumb: if the model doesn't fit on one card at your target precision, multi-GPU isn't optional — it's the only way to load it at all. If it does fit, multi-GPU is still worth it purely for throughput once you have concurrent users.

Step 3: Install vLLM

Set up a clean Python environment and install vLLM:

python3 -m venv vllm-env
source vllm-env/bin/activate
pip install vllm

Verify the install and confirm your GPUs are visible to PyTorch:

python3 -c "import torch; print(torch.cuda.device_count())"

This should print the number of GPUs on the server. If it prints 0, the driver or CUDA install needs fixing before vLLM will work — go back and confirm nvidia-smi runs correctly first.

Step 4: Set Your Hugging Face Token (Gated Models Only)

If you're pulling a gated model like Llama 3.1, export your token before launching:

export HF_TOKEN=your_token_here

Skip this step for open-access models that don't require authentication.

Step 5: Launch vLLM with Tensor Parallelism

This is the core step. The --tensor-parallel-size flag is the main change from a single-GPU launch — vLLM handles sharding the model and coordinating GPU communication automatically via NCCL under the hood.

For a 2-GPU server:

vllm serve meta-llama/Llama-3.1-70B \
  --tensor-parallel-size 2 \
  --dtype float16 \
  --max-model-len 8192

For a 4-GPU server:

vllm serve meta-llama/Llama-3.1-70B \
  --tensor-parallel-size 4 \
  --quantization fp8 \
  --max-model-len 8192

A few flags worth understanding before you run this:

• --tensor-parallel-size: must match, or evenly divide into, your available GPU count.

• --max-model-len: set this to your application's actual maximum context requirement, not the model's theoretical max — over-allocating this needlessly eats into KV cache space and reduces how many concurrent requests you can serve.

• --quantization fp8: roughly halves VRAM usage versus FP16 with a small, usually acceptable quality tradeoff, but only works on GPUs with native FP8 support (H100, RTX PRO 6000, and similar — not A100).

• --gpu-memory-utilization: defaults to 0.9 (90% of VRAM). Lower this if you're running other processes on the same GPUs.

This starts an OpenAI-compatible API server, by default on port 8000.

Step 6: Verify Multi-GPU Usage

Once the server is running, confirm all GPUs are actually being used, not just visible:

watch -n 1 nvidia-smi

Send a test request while this is running, and watch the utilization column move on every GPU, not just one:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-70B",
    "prompt": "The capital of France is",
    "max_tokens": 20
  }'

A response with generated text confirms the server, tensor parallelism, and GPU communication are all working together correctly.

Step 7: Using vLLM as a Python Library Instead of a Server

If you don't need an HTTP API and just want to run batch inference from a script, vLLM's Python interface takes the same parallelism setting directly:

from vllm import LLM

llm = LLM("facebook/opt-13b", tensor_parallel_size=4)
output = llm.generate("San Francisco is a")
print(output)

This is useful for offline batch jobs — evaluation runs, dataset labeling, or bulk generation — where you don't need a persistent server.

Choosing Between Tensor Parallelism and Pipeline Parallelism at Scale

For a single node, tensor parallelism is almost always the right default. But two situations change that:

• No NVLink between GPUs (common with L40S or PCIe-only builds): pipeline parallelism often gives better throughput here, since it needs less frequent cross-GPU communication.

• Multi-node deployments: keep tensor parallelism scoped to GPUs within a single node, and use pipeline parallelism to bridge across nodes. Mixing both is standard for very large models — an 8-GPU-per-node, 2-node setup might use --tensor-parallel-size 8 --pipeline-parallel-size 2.

Common Errors and How to Fix Them

• CUDA out of memory on startup: The model doesn't fit at your chosen precision and tensor-parallel size. Either increase --tensor-parallel-size, switch to FP8 (on supported hardware), or reduce --max-model-len and --gpu-memory-utilization.

• Server starts but only one GPU shows activity: Check that --tensor-parallel-size actually matches the number of GPUs you intended to use, and that CUDA_VISIBLE_DEVICES isn't restricting vLLM to fewer GPUs than expected.

• NCCL errors during startup: These usually point to a GPU interconnect or driver mismatch issue. Confirm all GPUs are on the same driver version, and check nvidia-smi topo -m to see the actual interconnect topology between cards.

• Slow first request, fast afterward: This is expected — the first request includes model loading and CUDA graph warmup. Subsequent requests use the warmed-up state and will be noticeably faster.

Next Steps

Once multi-GPU serving is running, a logical next step is putting a reverse proxy and load balancer in front of it for production traffic, or pairing it with a self-hosted UI — see our guide on self-hosting an AI search engine with Open WebUI and SearXNG for a ready-made front end that can point at your vLLM endpoint.

This guide was tested with vLLM on Ubuntu with NVIDIA H100 and RTX 4090 multi-GPU configurations. Exact flags and VRAM figures will vary by model and vLLM version — check vllm serve --help for the flags available in your installed version. Last updated August 2026.