Audio version — Estimated duration: 9 min 35 sec

CLI Usage

Once you have built llama.cpp and downloaded a model, you are ready to start interacting with it. The primary tool for this is llama-cli.exe.

🧠 Key Concepts for Beginners

What is Inference?

Inference is a fancy word for “running the model.” It is the process where you give the AI a prompt (a question or instruction) and it uses its trained knowledge to generate a response.

---

🚀 Basic Syntax

The simplest way to run a model is to provide the path to the model file and a text prompt.

./bin/Release/llama-cli.exe -m "C:\AI\models\your-model-q4_k_m.gguf" -p "The capital of France is"

🛠️ Essential Flags

To get the most out of your experience, you will frequently use these flags:

Flag	Long Name	Description
-m	--model	Required. The full path to your .gguf model file.
-p	--prompt	The text prompt you want the AI to respond to.
-n	--n-predict	The maximum number of tokens the AI should generate (e.g., -n 512).
-t	--threads	The number of CPU threads to use. Usually set to the number of physical cores on your CPU.
-ngl	--n-gpu-layers	The Magic Button for GPU users. This tells the computer to move parts of the model from your slow RAM to your fast GPU memory (VRAM).
-cnv	--conversation	Enables a chat-like interactive mode.
-temp	--temp	Sets the “temperature” (creativity). Higher = more creative/random, Lower = more focused/deterministic.

🔍 How to Read the Output

When you run llama-cli.exe, your terminal will show a lot of text. Don’t be intimidated! Here is what to look for:

1. The Setup Phase: You’ll see lines about “loading model,” “system info,” and “graph computation.” This is just the computer getting ready.
2. The GPU Check: If you used -ngl, look for lines mentioning “CUDA” or “HIP”. If you see them, your GPU is working!
3. The Response: After a short pause, the AI will start typing its answer. This is the part you actually care about.
4. The Stats: Once the AI is done, it will show some statistics like “tokens per second” (how fast it was) and “time to first token” (how long it took to start talking).

🚀 Advanced Usage Examples

1. The “Standard” CPU Run

Best for users without a dedicated GPU.

./bin/Release/llama-cli.exe -m "C:\AI\models\llama3-8b.gguf" -p "Write a poem about coding." -n 256 -t 8

2. The “Fast” GPU Run (NVIDIA/AMD)

This is how you get high speed. By offloading layers to your GPU, you move the heavy lifting away from your CPU.

If your GPU has enough VRAM, set this to a high number (like 33, 50, or even 99 to ensure all layers are offloaded).

./bin/Release/llama-cli.exe -m "C:\AI\models\llama3-8b.gguf" -p "Explain quantum physics." -ngl 33

Note: If you see “failed to allocate” or your computer freezes, your GPU doesn’t have enough VRAM for that many layers. Try a smaller number (e.g., -ngl 10) or a smaller/more quantized model.

3. Interactive Chat Mode

Instead of a single prompt and exit, this mode lets you have a back-and-forth conversation.

./bin/Release/llama-cli.exe -m "C:\AI\models\llama3-8b.gguf" -cnv -p "You are a helpful assistant."

---

📋 Complete Flag Reference

Model & Input

Flag	Description
-m, --model	Path to GGUF model file
-p, --prompt	Input prompt text
-f	Read prompt from a file
--in-prefix	Prefix for user input (e.g., “User: “)
--in-suffix	Suffix for user input (e.g., “\nAssistant: “)

Generation Control

Flag	Description
-n, --n-predict	Max tokens to generate (default: -1 = infinite in chat mode)
-c, --ctx-size	Context size in tokens (default: 2048, 8192 for modern models)
-temp, --temp	Temperature (0.0 = deterministic, 1.0 = creative)
--top-p	Nucleus sampling threshold (default: 0.9)
--top-k	Top-K sampling (default: 40)
--repeat-penalty	Repeat penalty (default: 1.1)
--seed	Random seed (-1 = random)
-b, --batch-size	Batch size for prompt processing (default: 2048)

Hardware

Flag	Description
-t, --threads	Number of CPU threads
-ngl, --n-gpu-layers	GPU layers to offload (-1 = all)
-ts, --tensor-split	Tensor split ratios for multi-GPU (e.g., 0.6,0.4)
-mg, --main-gpu	Main GPU index (for multi-GPU)
--no-mmap	Disable memory-mapped model loading (reduces RAM usage at speed cost)

Chat & Interaction

Flag	Description
-cnv, --conversation	Interactive chat mode
--template	Prompt template name (e.g., llama-3, alpaca, chatml)
--chat-template	Jinja2 chat template string
-i, --interactive	Interactive mode (single prompt, then wait for input)
--color	Colorize output for better readability

Server Mode

Flag	Description
--embedding	Enable embedding mode
--draft-model	Path to draft model for speculative decoding
--draft-n	Number of tokens to speculate
--log-file	Write log output to a file
--verbose	Detailed logging

🔧 Choosing the Right Flags

For a Chatbot (Interactive)

./bin/Release/llama-cli.exe -m "C:\AI\models\llama-3.1-8b.gguf" -cnv -ngl 99 -c 8192 -temp 0.7

For Fast Single Prompts (Scripting)

./bin/Release/llama-cli.exe -m "C:\AI\models\llama-3.2-3b.gguf" -p "Summarize this text" -n 256 -t 8

For Maximum Quality

./bin/Release/llama-cli.exe -m "C:\AI\models\llama-3.1-8b.gguf" -p "Write a story" -n 2048 -temp 0.8 --top-p 0.95 --repeat-penalty 1.15

For Maximum Speed

./bin/Release/llama-cli.exe -m "C:\AI\models\llama-3.1-8b.gguf" -p "Hello" -ngl 99 -b 2048 -t 8

---

Last Updated: 2026-07-19