Running multiple self-hosted Large Language Models (LLMs) can become difficult when each inference engine exposes a different API endpoint, configuration and authentication method. For developers building AI applications, managing these separate connections can increase application complexity and make it harder to switch between models.
LiteLLM Proxy solves this problem by providing a central, OpenAI-compatible API gateway for different LLM providers and self-hosted inference engines. Instead of connecting applications directly to individual model servers, you can route requests through LiteLLM and use a consistent API interface.
However, exposing an AI inference server without proper network restrictions and authentication can create security risks. Unauthorised users may be able to access model endpoints, consume GPU resources or submit requests using your infrastructure.
In this guide, we explain how to install and configure LiteLLM Proxy on Ubuntu Linux, connect it to Ollama and vLLM, protect API credentials, restrict internal service access and configure HTTPS using Nginx. We also cover model routing, OpenAI SDK integration and common security and connection problems.
1. What Is LiteLLM Proxy and Why Use It with Ollama and vLLM?
LiteLLM is an open-source LLM integration library that provides a unified interface for communicating with different AI model providers. Its Proxy Server acts as a self-hosted AI gateway between applications and model inference services.
For example, an application may connect directly to Ollama through port 11434, while a vLLM server exposes an OpenAI-compatible API through port 8000.
Without a gateway, developers need to manage these backend connections separately. LiteLLM provides one central endpoint that routes requests to the correct model based on the model name supplied by the client.
How the architecture works
The following diagram shows the secure single-server architecture used in this tutorial:
AI APPLICATION
(Python / OpenAI SDK)
|
|
HTTPS :443
|
v
+-------------+
| Nginx |
| TLS / HTTPS |
+-------------+
|
| Local connection
v
+----------------------+
| LiteLLM Proxy |
| 127.0.0.1:4000 |
| |
| API Authentication |
| Model Routing |
+----------------------+
|
Internal requests
/ \
/ \
v v
+------------+ +------------+
| Ollama | | vLLM |
| 127.0.0.1 | | 127.0.0.1 |
| :11434 | | :8000 |
+------------+ +------------+
| |
v v
Local LLM GPU Inference
Models Models
In this architecture:
Nginx handles incoming HTTPS connections.
LiteLLM authenticates application requests and routes them to the configured model.
Ollama and vLLM handle model loading and inference.
Internal inference endpoints remain restricted to localhost.
Applications do not connect directly to Ollama or vLLM.
Why use LiteLLM as an AI gateway?
Unified API: Applications can use the OpenAI SDK without maintaining separate client integrations for each backend.
Centralised model routing: Route requests to Ollama or vLLM using model aliases.
Simpler model management: Add or change backend configurations without rewriting application code.
Gateway authentication: Protect application access using a LiteLLM master key or database-backed virtual keys.
Usage management: With PostgreSQL configured, LiteLLM supports database-backed key management and usage tracking.
Flexible infrastructure: Run inference engines on separate GPU servers while maintaining one API endpoint for applications.
Important: LiteLLM does not replace Ollama or vLLM. These services still load and execute the AI models. LiteLLM manages the API connection and request routing between your applications and those inference engines.
2. Prerequisites for Setting Up LiteLLM on Linux
Before installing LiteLLM Proxy, prepare the Linux environment and confirm that the inference backends can run on your server.
This tutorial uses Ubuntu 22.04 or Ubuntu 24.04 as the primary operating system.
Example environment
| Component | Example configuration |
|---|---|
| Operating system | Ubuntu 22.04 / 24.04 |
| Python | 3.10 or newer |
| LiteLLM Proxy | 127.0.0.1:4000 |
| Ollama API | 127.0.0.1:11434 |
| vLLM API | 127.0.0.1:8000 |
| Nginx HTTPS | Public port 443 |
| Configuration | YAML |
These are example settings for the single-server architecture in this guide. They are not mandatory ports or addresses for every deployment.
Required software and services
Linux server: You need a Linux server with sufficient CPU, RAM and storage for the gateway and supporting services.
Ollama: Install Ollama if you want to run local models through its inference engine. Official documentation:
https://ollama.com/vLLM: Install vLLM if you want to serve supported language models through an inference engine designed for GPU-accelerated workloads. Official documentation:
https://docs.vllm.ai/LiteLLM Proxy: LiteLLM can be installed through its Python package and proxy dependencies. Official documentation:
https://docs.litellm.ai/NVIDIA GPU environment: If you use NVIDIA GPU acceleration with vLLM, install a compatible NVIDIA driver and CUDA environment. Verify that the GPU is accessible before starting inference services.
Run:
nvidia-smi
Network connectivity: Make sure the services can communicate with one another. In this tutorial, all three services run on the same server and use localhost addresses.
GPU memory planning
If Ollama and vLLM use the same NVIDIA GPU, check available VRAM before loading models.
vLLM uses GPU memory for model weights, KV cache and inference workloads. Ollama may also allocate GPU memory when a model is loaded.
If both services compete for the same GPU resources, you may experience CUDA out-of-memory errors or model loading failures.
For better resource management:
Use separate GPUs for Ollama and vLLM where possible.
Select models that fit the available GPU memory.
Adjust vLLM memory settings when sharing a GPU.
Monitor GPU usage before starting additional inference workloads.
For related configuration guidance, see our tutorials on NVIDIA driver installation, GPU monitoring and CUDA out-of-memory troubleshooting.
3. Prepare Secure API Credentials
Before starting the inference services, generate separate credentials for the gateway and vLLM.
This tutorial uses three credentials:
| Credential | Purpose |
|---|---|
| LITELLM_MASTER_KEY | Authenticates requests to the LiteLLM gateway and administrative API |
| VLLM_API_KEY | Authenticates supported vLLM API endpoints |
| LITELLM_SALT_KEY | Encrypts sensitive credentials stored in the LiteLLM database when database features are used |
The master key and upstream API key should not be placed directly inside the YAML configuration or application source code.
Step 1: Create a protected configuration directory
sudo install -d -o root -g root -m 750 /etc/litellm
Step 2: Generate the credentials
Create the environment file:
sudo bash -c '
umask 077
printf "LITELLM_MASTER_KEY=sk-%s\n" "$(openssl rand -hex 32)" > /etc/litellm/litellm.env
printf "VLLM_API_KEY=%s\n" "$(openssl rand -hex 32)" >> /etc/litellm/litellm.env
printf "LITELLM_SALT_KEY=sk-%s\n" "$(openssl rand -hex 32)" >> /etc/litellm/litellm.env
'
Step 3: Verify the file permissions
sudo chmod 600 /etc/litellm/litellm.env
sudo chown root:root /etc/litellm/litellm.env
Check the permissions:
sudo stat -c '%a %U:%G %n' /etc/litellm/litellm.env
Expected output:
600 root:root /etc/litellm/litellm.env
The file is now readable only by root. systemd can read this file when starting the services and pass the required environment variables to their processes.
Important: Keep the LITELLM_MASTER_KEY and VLLM_API_KEY private. The LITELLM_SALT_KEY must also be backed up securely. Once database credentials have been encrypted using this salt key, changing it can make those stored credentials unreadable. Do not commit this environment file to Git or include it in public backups.
4. Install and Configure Ollama on Linux
Start by preparing the Ollama inference backend. If Ollama is already installed and working, you can skip the installation steps and move directly to API verification.
Step 1: Install Ollama
On a supported Linux distribution, use the official installation script:
curl -fsSL https://ollama.com/install.sh | sh
After installation, check the service:
sudo systemctl status ollama
If the service is not running, start it:
sudo systemctl enable --now ollama
Step 2: Download an AI model
For this example, we use Llama 3.2:
ollama pull llama3.2
Check the locally available models:
ollama list
The model size and resource requirements depend on the selected model variant.
Step 3: Verify the Ollama API
Ollama exposes its local API on port 11434 by default. Check whether the service responds:
curl http://127.0.0.1:11434/api/tags
You can also test its native chat endpoint:
curl http://127.0.0.1:11434/api/chat \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.2",
"messages": [
{
"role": "user",
"content": "Explain what an AI language model is."
}
],
"stream": false
}'
If the request returns a generated response, Ollama is running and the model is available.
Step 4: Keep Ollama restricted to localhost
The local Ollama API does not require authentication by default. This means anyone who can reach an exposed Ollama endpoint may be able to submit inference requests.
For the single-server setup, keep the service listening on: 127.0.0.1:11434
Do not configure Ollama to listen on
0.0.0.0for this deployment.Do not create a public firewall rule for port 11434.
If Ollama is already configured to listen on another interface, review its service configuration and change the listening address before continuing.
Step 5: Understand Ollama's OpenAI-compatible API
Ollama also provides an OpenAI-compatible API at: http://127.0.0.1:11434/v1/
This allows compatible applications to send Chat Completions requests using an OpenAI-style interface. However, Ollama supports only a subset of the OpenAI API. Not every OpenAI feature or parameter is necessarily supported.
5. Install and Configure vLLM on Linux
Next, prepare vLLM as the second inference backend. vLLM is designed for LLM serving workloads and provides features such as continuous batching and efficient GPU memory management. It is commonly used to serve language models on dedicated GPU infrastructure.
Step 1: Prepare the Python environment
Create a virtual environment:
python3 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
Upgrade pip:
python -m pip install --upgrade pip
Step 2: Install vLLM
Install vLLM using its official installation instructions:
pip install vllm
Important: vLLM installation depends on the Python version, operating system, GPU architecture, NVIDIA driver and compatible CUDA environment. Use the official installation instructions if a prebuilt package is not compatible with your server.
Step 3: Load the vLLM API key
Retrieve the API key from the protected environment file:
export VLLM_API_KEY="$(sudo sed -n 's/^VLLM_API_KEY=//p' /etc/litellm/litellm.env)"
Check that the variable has been set without printing its value:
test -n "$VLLM_API_KEY" && echo "vLLM API key loaded"
Step 4: Start the vLLM OpenAI-compatible server
For this example, we use a Hugging Face model identifier:
vllm serve Qwen/Qwen2.5-7B-Instruct \
--host 127.0.0.1 \
--port 8000 \
--api-key "$VLLM_API_KEY" \
--served-model-name qwen-local
The model identifier is an example. Select a model that is compatible with your GPU memory, vLLM version and available hardware.
This configuration:
Loads the selected model into vLLM.
Makes the HTTP server listen only on localhost.
Uses an API key for supported authenticated endpoints.
Exposes the model using the name
qwen-local.
Why use 127.0.0.1 instead of 0.0.0.0? The localhost binding prevents other machines from directly connecting to the vLLM service through the server's network interfaces. This is important because vLLM's API key does not authenticate every HTTP endpoint. Some other endpoints, including /invocations, may remain accessible without the API key. Do not rely on the API key alone as the complete security boundary. Restrict network access as well.
Step 5: Verify the vLLM API
Open another terminal and load the key if necessary:
export VLLM_API_KEY="$(sudo sed -n 's/^VLLM_API_KEY=//p' /etc/litellm/litellm.env)"
Check the available models:
curl http://127.0.0.1:8000/v1/models \
-H "Authorization: Bearer $VLLM_API_KEY"
A successful response should include the served model name:
{
"data": [
{
"id": "qwen-local",
"object": "model"
}
]
}
The response may contain additional fields depending on the vLLM version.
You can also test a chat completion directly:
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Authorization: Bearer $VLLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen-local",
"messages": [
{
"role": "user",
"content": "Explain how GPU inference works."
}
],
"max_tokens": 150
}'
If the request returns a generated response, the vLLM API is ready to be connected to LiteLLM. For a production deployment, run vLLM under its own service manager or container and configure it to start automatically. The terminal command above is intended for initial setup and testing.
6. Install LiteLLM Proxy on Linux
Now that both inference backends are available, install the LiteLLM Proxy. For this example, we use a dedicated Python virtual environment and a separate Linux service account. This avoids running the gateway as root and keeps LiteLLM dependencies separate from system Python packages.
Step 1: Install the required packages
sudo apt update
sudo apt install -y \
python3-venv \
python3-pip \
openssl \
curl \
nginx
Step 2: Create a dedicated LiteLLM user
sudo useradd \
--system \
--create-home \
--home-dir /var/lib/litellm \
--shell /usr/sbin/nologin \
litellm
This creates a system account named litellm without normal interactive login access.
Step 3: Create the virtual environment
sudo python3 -m venv /opt/litellm/venv
Upgrade pip:
sudo /opt/litellm/venv/bin/pip install --upgrade pip
Step 4: Install LiteLLM Proxy
Install LiteLLM with its proxy dependencies:
sudo /opt/litellm/venv/bin/pip install 'litellm[proxy]'
For production, pin a tested LiteLLM release and update it through a controlled maintenance process rather than automatically installing an untested latest version.
Step 5: Verify the installation
sudo /opt/litellm/venv/bin/litellm --version
The command should return the installed LiteLLM version.
7. Configure LiteLLM Model Routing
LiteLLM uses a YAML configuration file to define model aliases, upstream providers and gateway settings. In this section, we configure Ollama and vLLM in the same file.
Step 1: Create the configuration file
sudo nano /etc/litellm/config.yaml
Add:
model_list:
# Ollama backend
- model_name: llama-local
litellm_params:
model: ollama_chat/llama3.2
api_base: http://127.0.0.1:11434
# vLLM backend
- model_name: qwen-local
litellm_params:
model: openai/qwen-local
api_base: http://127.0.0.1:8000/v1
api_key: os.environ/VLLM_API_KEY
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
trusted_proxy_ranges:
- "127.0.0.1/32"
Save the file. The trusted_proxy_ranges setting identifies the local Nginx reverse proxy as a trusted proxy when it forwards client information to LiteLLM. If you change the reverse proxy architecture or its address, update this setting to match the actual trusted proxy address. Do not trust arbitrary client-supplied forwarding headers.
Step 2: Set configuration permissions
The LiteLLM service account needs permission to read the YAML configuration. Set the correct ownership and permissions:
sudo chown root:litellm /etc/litellm/config.yaml
sudo chmod 640 /etc/litellm/config.yaml
This allows root to modify the configuration and the LiteLLM service group to read it.
Step 3: Understand the model aliases
Ollama configuration:
- model_name: llama-local
litellm_params:
model: ollama_chat/llama3.2
api_base: http://127.0.0.1:11434
Here, llama-local is the public model alias used by applications. The ollama_chat/ prefix tells LiteLLM to use its Ollama chat integration, while llama3.2 identifies the actual model downloaded through Ollama.
vLLM configuration:
- model_name: qwen-local
litellm_params:
model: openai/qwen-local
api_base: http://127.0.0.1:8000/v1
api_key: os.environ/VLLM_API_KEY
The openai/ prefix tells LiteLLM to communicate with vLLM through its OpenAI-compatible API. The model name must match the --served-model-name value configured when starting vLLM. The /v1 suffix is required because vLLM exposes its OpenAI-compatible endpoints under that path.
Step 4: Understand the environment variables
The configuration uses master_key: os.environ/LITELLM_MASTER_KEY and api_key: os.environ/VLLM_API_KEY. LiteLLM retrieves these values from the environment when the service starts. The systemd service will load the credentials from /etc/litellm/litellm.env. This keeps the actual secret values outside the YAML configuration.
Separate-server deployments
The configuration above assumes that Ollama, vLLM and LiteLLM run on the same server. If you deploy them on separate machines, replace the localhost addresses with reachable private network addresses. For example:
model_list:
- model_name: llama-local
litellm_params:
model: ollama_chat/llama3.2
api_base: http://10.0.0.20:11434
- model_name: qwen-local
litellm_params:
model: openai/qwen-local
api_base: http://10.0.0.30:8000/v1
api_key: os.environ/VLLM_API_KEY
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
The IP addresses above are illustrative private network addresses. Replace them with the actual addresses of your inference servers.
For separate-server deployments:
Bind inference services to the required private network interface.
Restrict incoming connections using firewall rules.
Allow only the LiteLLM server to access the inference endpoints.
Use encrypted connections or a private trusted network between servers.
Do not expose Ollama or vLLM directly to the public internet.
8. Run LiteLLM Proxy as a systemd Service
Running LiteLLM manually from a terminal is useful for testing. For a Linux server, systemd provides a better way to manage the gateway process. It allows LiteLLM to start automatically during boot and restart when the process exits unexpectedly.
Step 1: Create the systemd service file
sudo nano /etc/systemd/system/litellm.service
Add:
[Unit]
Description=LiteLLM Proxy
Wants=network-online.target
After=network-online.target ollama.service
[Service]
Type=simple
User=litellm
Group=litellm
EnvironmentFile=/etc/litellm/litellm.env
ExecStart=/opt/litellm/venv/bin/litellm --config /etc/litellm/config.yaml --host 127.0.0.1 --port 4000
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
[Install]
WantedBy=multi-user.target
The service uses:
User=litellmto run the proxy under a dedicated Linux account.EnvironmentFileto load the gateway and upstream credentials.ExecStartto launch LiteLLM with the YAML configuration.127.0.0.1to restrict the gateway listener to the local machine.Restart=on-failureto restart the service when it exits unexpectedly.Additional systemd restrictions to reduce unnecessary process privileges.
Note: The service does not automatically start vLLM. Run vLLM under its own systemd service or container and configure it to start independently. LiteLLM may start before vLLM is ready, but requests to that backend will fail until the inference endpoint is available.
Step 2: Reload systemd
sudo systemctl daemon-reload
Step 3: Enable and start LiteLLM
sudo systemctl enable --now litellm
Step 4: Check the service status
sudo systemctl status litellm
The service should display an active running status.
Step 5: Check service logs
If LiteLLM fails to start, inspect its logs:
sudo journalctl -u litellm -n 100 --no-pager
Look for configuration errors, missing environment variables, invalid model definitions or connection problems.
9. Test the LiteLLM Gateway
After starting the proxy, verify that it can communicate with both inference backends.
Step 1: Load the master key for testing
Load the key into your current terminal session:
export LITELLM_MASTER_KEY="$(sudo sed -n 's/^LITELLM_MASTER_KEY=//p' /etc/litellm/litellm.env)"
Check that it has been loaded without printing its value:
test -n "$LITELLM_MASTER_KEY" && echo "LiteLLM master key loaded"
Step 2: Check gateway health
Run:
curl http://127.0.0.1:4000/health/liveliness
LiteLLM provides this endpoint to check whether the proxy process is alive.
Step 3: List the configured models
curl http://127.0.0.1:4000/v1/models \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
The response should include the configured model aliases: llama-local and qwen-local. The exact JSON response may contain additional metadata.
Step 4: Send a request to Ollama
curl http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "llama-local",
"messages": [
{
"role": "user",
"content": "What is a self-hosted AI model?"
}
],
"max_tokens": 150
}'
LiteLLM should identify llama-local and forward the request to Ollama.
Step 5: Send a request to vLLM
curl http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen-local",
"messages": [
{
"role": "user",
"content": "Explain how an AI inference server works."
}
],
"max_tokens": 150
}'
LiteLLM should forward the request to vLLM and return the generated response. At this stage, both inference backends should be accessible through the same local gateway endpoint.
10. Connect Applications Using the OpenAI Python SDK
One of the main advantages of LiteLLM Proxy is that applications can use the standard OpenAI Python SDK. You do not need to create separate API clients for Ollama and vLLM.
Step 1: Install the OpenAI Python package
Create a Python virtual environment if your application does not already have one and install the SDK:
python3 -m venv ~/ai-client-env
source ~/ai-client-env/bin/activate
pip install openai
Step 2: Configure the gateway connection
For local testing, load the master key:
export OPENAI_API_KEY="$LITELLM_MASTER_KEY"
export OPENAI_BASE_URL="http://127.0.0.1:4000/v1"
In production, use an individual virtual key instead of sharing the administrative master key.
Step 3: Send a request to Ollama
Create a Python file (nano test_ollama.py) and add:
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["OPENAI_API_KEY"],
base_url=os.environ["OPENAI_BASE_URL"]
)
response = client.chat.completions.create(
model="llama-local",
messages=[
{
"role": "user",
"content": "Explain the benefits of self-hosted AI."
}
]
)
print(response.choices[0].message.content)
Run:
python test_ollama.py
The request should reach LiteLLM and be routed to Ollama.
Step 4: Send a request to vLLM
Create another file (nano test_vllm.py) and add:
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["OPENAI_API_KEY"],
base_url=os.environ["OPENAI_BASE_URL"]
)
response = client.chat.completions.create(
model="qwen-local",
messages=[
{
"role": "user",
"content": "Explain how continuous batching improves LLM serving."
}
]
)
print(response.choices[0].message.content)
Run:
python test_vllm.py
The same OpenAI client can communicate with either inference backend by changing the model alias.
11. Secure Remote Access with Nginx and HTTPS
The LiteLLM gateway currently listens on 127.0.0.1:4000. This is suitable for local applications. If remote applications need to access the gateway, use Nginx as a reverse proxy and configure HTTPS. The internal LiteLLM port should remain private.
Step 1: Configure your domain
Create a DNS A record pointing your chosen subdomain to the public IP address of your server. For this example, we use llm.example.com. Replace this with your actual domain.
Step 2: Create the Nginx configuration
Create sudo nano /etc/nginx/sites-available/litellm and add:
server {
listen 80;
server_name llm.example.com;
location / {
proxy_pass http://127.0.0.1:4000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
}
Replace llm.example.com with your actual domain. The configuration forwards incoming requests to the local LiteLLM gateway. The proxy_buffering off setting helps streaming responses reach clients without unnecessary response buffering.
Step 3: Enable the Nginx site
sudo ln -s /etc/nginx/sites-available/litellm \
/etc/nginx/sites-enabled/litellm
sudo nginx -t
sudo systemctl reload nginx
Step 4: Install Certbot
sudo apt update
sudo apt install -y certbot python3-certbot-nginx
Step 5: Request an HTTPS certificate
sudo certbot --nginx -d llm.example.com
Follow the prompts to complete certificate installation. Once HTTPS is configured, applications can access the gateway using: https://llm.example.com/v1
Step 6: Configure LiteLLM's public URL
Add PROXY_BASE_URL=https://llm.example.com to /etc/litellm/litellm.env, keep its permissions restricted, and restart LiteLLM:
sudo chmod 600 /etc/litellm/litellm.env
sudo chown root:root /etc/litellm/litellm.env
sudo systemctl restart litellm
This informs LiteLLM of the public HTTPS origin when it operates behind TLS termination at Nginx.
Step 7: Configure the firewall
For a server using UFW, allow SSH and HTTPS before enabling the firewall:
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose
Do not add public firewall rules for these internal service ports: 4000 (LiteLLM), 8000 (vLLM), 11434 (Ollama), and 5432 (PostgreSQL, if installed). These ports should remain inaccessible from untrusted external networks.
If your hosting provider has a separate cloud firewall or security group, apply the same restrictions there as well.
Important: HTTPS protects data in transit between remote applications and Nginx. It does not replace API authentication, firewall restrictions or secure upstream configuration.
12. Configure Database-Backed Virtual API Keys
The basic configuration in this guide uses a master key to authenticate gateway requests. This is useful for initial testing and a simple single-application environment. However, sharing the master key with multiple applications is not a good production access-control strategy.
For larger deployments, LiteLLM supports virtual API keys backed by PostgreSQL. Virtual keys can provide:
Individual credentials for different applications.
Model access restrictions.
Requests-per-minute limits.
Key expiration.
Usage tracking.
Database-backed spend management.
PostgreSQL is required for these database-backed virtual key features. The basic YAML configuration in this tutorial does not automatically provide virtual key management. For production deployments, follow the official LiteLLM PostgreSQL and virtual key documentation.
When enabling database-backed features:
Use a dedicated PostgreSQL database and restricted database account.
Keep the database listener private.
Store
DATABASE_URLin a protected environment file.Set
LITELLM_SALT_KEYbefore adding credentials to the database.Back up the database and encryption key securely.
Issue separate virtual keys for individual applications.
Apply model restrictions and appropriate rate limits.
Do not expose PostgreSQL port 5432 to the public internet.
For self-hosted models, configure appropriate pricing information if you want LiteLLM's spend budgets to represent meaningful usage costs. Request limits and model restrictions remain useful even when model pricing is not configured.
13. Troubleshoot Common LiteLLM, Ollama and vLLM Errors
Even when the individual services are running, LiteLLM may fail to route requests because of incorrect model names, network settings, authentication or incompatible API configurations. The following checks help identify common problems.
Problem 1: Connection refused when accessing Ollama
Possible cause: Ollama is not running or the configured API endpoint is incorrect.
sudo systemctl status ollama
curl http://127.0.0.1:11434/api/tags
sudo systemctl start ollama
Problem 2: Connection refused when accessing vLLM
Possible cause: The vLLM server has stopped or is listening on a different port.
sudo ss -lntp | grep 8000
curl http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"
Problem 3: Model not found
Possible cause: The requested model alias does not match the model_name in the LiteLLM configuration. Ensure your application requests "model": "qwen-local" matching your YAML config and --served-model-name.
Problem 4: Authentication failed
Possible cause: The client is using an incorrect LiteLLM key, or LiteLLM is sending an incorrect upstream API key to vLLM.
test -n "$LITELLM_MASTER_KEY" && echo "Master key loaded"
test -n "$VLLM_API_KEY" && echo "vLLM key loaded"
Problem 5: 404 Not Found from vLLM
Possible cause: The vLLM API base URL is missing the /v1 path. Ensure it is set to api_base: http://127.0.0.1:8000/v1 and restart LiteLLM.
Problem 6: LiteLLM cannot reach a backend running in Docker
Possible cause: The configured localhost address points to the LiteLLM container rather than the host or another container. Use a shared Docker network and the backend service name where possible, or configure an appropriate host gateway address.
Problem 7: GPU out-of-memory errors
Possible cause: The selected model requires more GPU memory than is available, or multiple inference processes are competing for GPU resources. Check nvidia-smi, reduce model size, or adjust vLLM memory utilization settings.
Problem 8: LiteLLM systemd service fails to start
Possible cause: Missing configuration permissions, incorrect environment variables or a Python installation problem.
sudo systemctl status litellm
sudo journalctl -u litellm -n 100 --no-pager
sudo -u litellm test -r /etc/litellm/config.yaml && echo "Configuration readable"
ls -l /opt/litellm/venv/bin/litellm
Problem 9: Streaming responses arrive all at once
Possible cause: A reverse proxy is buffering the response. Check that the Nginx configuration includes proxy_buffering off;.
Problem 10: External users can access an internal inference port
Possible cause: The inference service is listening on a public interface or a firewall rule allows external connections.
sudo ss -lntp | grep -E '4000|8000|11434'
Expected listeners are 127.0.0.1:4000, 127.0.0.1:8000, and 127.0.0.1:11434. If an internal service is listening on 0.0.0.0, restrict it to localhost and verify UFW and cloud firewall rules.
14. Maintain and Monitor Your Self-Hosted AI Gateway
Once LiteLLM is running, regular maintenance helps keep the gateway and its inference backends reliable. For production Linux infrastructure, consider the following practices:
Monitor API performance: Track request latency, token generation speed, response times and backend availability.
Monitor GPU resources: Check GPU memory, temperature, utilisation and hardware errors to identify resource bottlenecks.
Review authentication activity: Monitor failed authentication attempts, key usage and unusual request volume.
Maintain model compatibility: Check that configured model identifiers and inference parameters remain supported after updating Ollama, vLLM or LiteLLM.
Back up configuration and database data: Keep the LiteLLM YAML file, protected environment file and PostgreSQL backups in secure storage.
Update software carefully: Test LiteLLM, Python dependencies and inference engine updates before applying them to production systems.
Use least-privilege access: Keep administrative credentials restricted and use individual virtual keys for applications when database-backed key management is configured.
Review firewall rules: Periodically verify that internal inference ports are not publicly accessible.
For larger deployments, Prometheus, Grafana and NVIDIA DCGM Exporter can help monitor server and GPU metrics.
Frequently Asked Questions
1. What is LiteLLM Proxy used for?
LiteLLM Proxy provides a central OpenAI-compatible API gateway for multiple LLM providers and self-hosted inference engines. It allows applications to communicate with different models through one API endpoint.
2. Can LiteLLM connect to both Ollama and vLLM?
Yes. LiteLLM supports both inference engines. You can define separate model entries in one YAML configuration file and route requests using model aliases.
3. Does LiteLLM replace Ollama or vLLM?
No. Ollama and vLLM continue to execute the AI models. LiteLLM acts as the API gateway between applications and those inference services.
4. Does LiteLLM require an NVIDIA GPU?
No. LiteLLM is an API gateway and does not require an NVIDIA GPU to perform model inference. GPU requirements depend on the inference engines and models you run.
5. Can Ollama and vLLM run on separate GPU servers?
Yes. They can run on different Linux servers. LiteLLM can connect to their private API endpoints, provided the required network connectivity, firewall rules and authentication settings are configured.
6. Does LiteLLM require PostgreSQL?
No. A basic LiteLLM deployment can run with a YAML configuration and master key without PostgreSQL. However, database-backed virtual keys, spend tracking and related management features require a database.
7. Can I use the OpenAI Python SDK with LiteLLM?
Yes. LiteLLM exposes OpenAI-compatible API endpoints. Applications can use the OpenAI Python SDK by changing the base URL to the LiteLLM gateway and supplying an authorised API key.
8. Is the vLLM API key enough to secure the inference server?
No. vLLM's API key protects supported API endpoints, but not every HTTP endpoint exposed by the server. Restrict network access using localhost binding, private networking and firewall rules.
9. Can LiteLLM be exposed directly to the internet?
It is better to place LiteLLM behind a reverse proxy or load balancer with HTTPS, keep internal inference services private and enforce API authentication. Do not expose Ollama, vLLM or PostgreSQL ports publicly.
10. Is LiteLLM suitable for production AI applications?
Yes. LiteLLM can be used in production environments with appropriate authentication, secret management, database configuration where required, HTTPS, network restrictions, monitoring and resource management. Production readiness also depends on the security of the surrounding infrastructure.
Conclusion
LiteLLM Proxy provides a practical way to manage multiple self-hosted LLM inference engines through a single OpenAI-compatible API gateway. By connecting Ollama and vLLM, developers can use different models without maintaining separate API integrations for every application.
For Linux server administrators and AI infrastructure teams, this architecture simplifies model routing and centralises API access. However, a reliable deployment requires more than simply starting the proxy. Internal inference endpoints should remain private, API credentials must be protected, and remote access should use HTTPS with appropriate firewall restrictions.
With the right model configuration, GPU resources, authentication and monitoring, LiteLLM can provide a useful foundation for managing self-hosted AI inference workloads.
Explore GTZHost for infrastructure solutions designed to support demanding AI, GPU computing and dedicated server workloads.