llama-server

Audio version — Estimated duration: 6 min 12 sec

llama-server — HTTP API & Web UI

llama-server is the HTTP server executable that comes with llama.cpp. It exposes an OpenAI-compatible API and includes a built-in web chat interface. This is the most convenient way to use llama.cpp — you don’t need to craft prompts manually or keep typing in a terminal.

What You Get

• Web Chat UI — A ChatGPT-like interface in your browser.
• OpenAI-compatible API — Works with any app that supports the OpenAI API format.
• Multi-user support — Multiple people can chat simultaneously.
• Persistent sessions — Conversations stay alive between requests.

Basic Usage

After building llama.cpp, you’ll find llama-server.exe in build/bin/Release/.

./bin/Release/llama-server.exe -m "C:\AI\models\llama-3.1-8b.gguf" -ngl 99

Open your browser to http://localhost:8080 — you’ll see the chat interface.

Essential Flags

Flag	Purpose
-m	Path to your .gguf model file
-ngl N	Offload N layers to GPU (use -ngl 99 for all layers)
-c N	Context size in tokens (default: 2048, use -c 8192 for long conversations)
-host 0.0.0.0	Listen on all network interfaces (access from other devices)
-port N	Port number (default: 8080)
-t N	Number of CPU threads
--embeddings	Enable embedding mode (for RAG / vector search apps)

Examples

Minimal Chat Server

./bin/Release/llama-server.exe -m "C:\AI\models\llama-3.1-8b.gguf"
# → Open http://localhost:8080

Long Context + GPU Offload

./bin/Release/llama-server.exe -m "C:\AI\models\llama-3.1-8b.gguf" -ngl 99 -c 16384

Accessible from Other Devices (LAN)

./bin/Release/llama-server.exe -m "C:\AI\models\llama-3.1-8b.gguf" -host 0.0.0.0 --port 8080

Then connect from another computer using http://YOUR_IP:8080.

OpenAI-Compatible API

llama-server provides a /v1/chat/completions endpoint that works with the OpenAI SDK. This means you can use tools like Open WebUI, LangChain, or any OpenAI-compatible client.

Example: Using with cURL

curl http://localhost:8080/v1/chat/completions `
  -H "Content-Type: application/json" `
  -d '{\"model\": \"llama\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]}'

Example: Using Python

from openai import OpenAI
 
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="llama",
    messages=[{"role": "user", "content": "Write a haiku about AI"}]
)
print(response.choices[0].message.content)

API Endpoints

Endpoint	Description
GET /	Web chat UI
POST /v1/chat/completions	Chat completions (OpenAI-compatible)
POST /v1/completions	Text completions
POST /v1/embeddings	Generate embeddings
GET /health	Health check
GET /slots	Active slots/connections

Stopping the Server

Press Ctrl+C in the terminal where llama-server is running.

Pro Tips

• Save time with a batch file: Create start-server.bat with your server command so you don’t type it every time.
• Run in background: Open a dedicated terminal window for the server and keep it running.
• Performance: The first request is slower (model loading). Subsequent requests are fast.
• Multiple models: You can only run one model per server instance. Use multiple terminals for different models.

---

Last Updated: 2026-07-19