Advanced Inference

Audio version — Estimated duration: 7 min 23 sec

Advanced Inference — Flags & Techniques

Beyond the basic flags described in CLI Usage, llama-cli and llama-server support many advanced options for fine-tuning behavior, performance, and output quality.

Generation Control

-temp (Temperature)

Controls randomness. Higher = more creative/unpredictable, lower = more focused/deterministic.

Value	Behavior
0.0	Deterministic (always picks the most likely token)
0.5	Balanced — good for general use
0.8	Creative — good for writing/storytelling
1.0	Very random — may produce incoherent output
>1.0	Chaotic — rarely useful

./bin/Release/llama-cli.exe -m model.gguf -p "Write a story" -temp 0.8

--top-p (Nucleus Sampling)

Only considers tokens whose cumulative probability reaches top-p. Lower values = more focused.

./bin/Release/llama-cli.exe -m model.gguf -p "Explain gravity" --top-p 0.9

--repeat-penalty

Penalizes tokens that have already appeared, reducing repetition. Default is 1.1. Increase to 1.15-1.2 for models that tend to loop.

./bin/Release/llama-cli.exe -m model.gguf -p "Tell me about space" --repeat-penalty 1.15

-n (Tokens to Generate)

Maximum number of tokens (roughly words) to generate.

./bin/Release/llama-cli.exe -m model.gguf -p "Explain quantum physics" -n 1024

Memory & Context

-c (Context Size)

The number of tokens the model can “remember” at once. Longer context = more RAM/VRAM usage.

Model	Recommended -c
Small (1-3B)	4096
Medium (7-8B)	8192
Large (13B+)	4096 (or lower if RAM-limited)

./bin/Release/llama-cli.exe -m model.gguf -c 8192 -p "Long conversation..."

-b (Batch Size)

Number of tokens processed in parallel during prompt ingestion. Higher = faster prompt processing, more RAM.

./bin/Release/llama-cli.exe -m model.gguf -b 512 -p "Process this prompt fast"

-ub (Ubatch Size)

Micro-batch size for prompt processing. Can reduce VRAM spikes at the cost of slightly slower prompt ingestion.

./bin/Release/llama-cli.exe -m model.gguf -ub 256

GPU & Hardware

-ngl (GPU Layers)

Already covered in Backend Comparison. Set to the number of layers your GPU VRAM can hold, or -ngl 99 to offload everything possible.

-ts (Tensor Split)

For multi-GPU setups — distributes layers across multiple GPUs by proportion.

# Two GPUs: 60% on GPU 0, 40% on GPU 1
./bin/Release/llama-cli.exe -m model.gguf -ngl 99 -ts 0.6,0.4

-mg (Main GPU)

Which GPU index to use as the primary device (for multi-GPU). Default is 0.

./bin/Release/llama-cli.exe -m model.gguf -ngl 99 -mg 1

Output & Logging

--verbose

Shows detailed logging — device detection, model loading, timing information.

./bin/Release/llama-cli.exe -m model.gguf -p "Hello" --verbose

--log-file

Write logs to a file instead of the terminal.

./bin/Release/llama-cli.exe -m model.gguf -p "Hello" --log-file inference.log

--no-display-prompt

Only show the model’s response, not your prompt (useful for scripting).

Performance Tuning Cheat Sheet

Goal	Flag	Value
Fastest prompt processing	-b	2048 (or highest without OOM)
Fastest generation	-ngl	99 (full GPU offload)
Lower VRAM usage	-ub	128-256
Best quality	-temp	0.6-0.8, --top-p 0.9
Prevent repetition	--repeat-penalty	1.1-1.2
Multi-GPU	-ts	Split by VRAM ratio

Combining Multiple Flags

./bin/Release/llama-cli.exe ^
  -m "C:\AI\models\llama-3.1-8b.gguf" ^
  -ngl 99 ^
  -c 8192 ^
  -temp 0.7 ^
  --top-p 0.9 ^
  --repeat-penalty 1.1 ^
  -n 2048 ^
  -t 8 ^
  -p "Write a detailed explanation of how neural networks work."

---

Last Updated: 2026-07-19