Audio version — Estimated duration: 4 min 56 sec

Vulkan Installation (Cross-Vendor GPU)

Vulkan is a low-level graphics and compute API that works across GPU vendors. In llama.cpp, the Vulkan backend provides GPU acceleration on any GPU that supports Vulkan — including NVIDIA, AMD, Intel, and even some Qualcomm/ARM GPUs.

This is a great fallback when:

• You have an unsupported GPU (e.g., very old NVIDIA without recent CUDA, or a niche GPU).
• You want one build that works on multiple machines with different GPUs.
• You don’t want to install vendor-specific SDKs (CUDA Toolkit, HIP SDK, oneAPI).

How It Compares

Backend	Performance	Setup Complexity	GPU Support
CUDA	★★★★★	Medium (CUDA Toolkit)	NVIDIA only
HIP (AMD)	★★★★☆	Medium (HIP SDK)	AMD only
SYCL (Intel)	★★★★☆	Medium (oneAPI)	Intel only
Vulkan	★★★☆☆	Low (Vulkan SDK)	NVIDIA + AMD + Intel

Vulkan is slightly slower than vendor-native backends (CUDA/HIP/SYCL) because it’s a universal layer, but it’s the most portable option.

Prerequisites

In addition to the core tools in the Prerequisites Guide, you need:

1. Vulkan-capable GPU + driver — Most modern GPUs support Vulkan. Verify at vulkan.gpuinfo.org.
2. Vulkan SDK — Provides the glslc compiler and Vulkan-1.dll loader.
   • Download: Vulkan SDK
   • Installation: Run the installer (default options are fine).
   • Verification:

     glslc --version

The Vulkan loader is included in your GPU driver. The SDK only provides development tools — you don’t need it at runtime.

Step-by-Step Installation

1. Clone the Repository

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

2. Create a Build Directory

mkdir build
cd build

3. Configure with Vulkan Support

cmake .. -DGGML_VULKAN=ON

4. Compile

cmake --build . --config Release

Verification

Run the help command:

./bin/Release/llama-cli.exe --help

To confirm Vulkan is working, run a small model with verbose logging:

./bin/Release/llama-cli.exe -m "C:\AI\models\llama-3.2-3b.gguf" -ngl 99 -p "Hello!" --verbose

Look for lines like:

ggml_vulkan: Found 1 Vulkan devices:
  Device 0: NVIDIA GeForce RTX 4090 (Vulkan 1.3)

Running with Vulkan

Usage is identical to the GPU backends — use -ngl to offload layers:

./bin/Release/llama-cli.exe -m "C:\AI\models\llama-3.2-3b.gguf" -ngl 99 -p "Write a poem."

Selecting a Specific GPU (Multi-GPU)

If you have multiple GPUs, set the device index:

$env:GGML_VULKAN_DEVICE="0"

Run with --verbose to see the list of available Vulkan devices and their indices.

Troubleshooting

• “Vulkan not found” during CMake — The Vulkan SDK isn’t in your PATH. Re-run the SDK installer and ensure “Add Vulkan SDK to PATH” is checked, or restart your terminal.
• “Failed to create Vulkan instance” at runtime — Your GPU driver doesn’t support Vulkan or is outdated. Update your GPU driver.
• Lower performance than expected — Vulkan has some overhead compared to native backends. If your GPU supports CUDA/HIP/SYCL, use those instead for ~10-20% more performance.
• “Out of device memory” — Reduce -ngl or use a smaller quantization.

---

Last Updated: 2026-07-19