Audio version — Estimated duration: 5 min 18 sec
Model Conversion — Hugging Face → GGUF
Most models on Hugging Face are distributed in the raw PyTorch format (a folder of .safetensors or .bin files). llama.cpp cannot use these directly — they must first be converted to GGUF.
In most cases, you don’t need to do this. Pre-converted GGUF files are available from community members like Bartowski, MaziyarPanahi, and others. Only follow this guide if:
- The model you want does not have a GGUF version available.
- You want to convert your own fine-tuned model.
- You want a custom quantization not available pre-built.
How It Works
Raw model (safetensors) → convert.py → FP16 GGUF → quantize.exe → Q4_K_M GGUF
Step-by-Step
1. Install Python Dependencies
pip install torch transformers sentencepiece2. Download the Raw Model from Hugging Face
git lfs install
git clone https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3GitHub LFS is required for large model files. Install from git-lfs.com.
3. Convert to FP16 GGUF
From the llama.cpp directory:
python convert.py "C:\AI\raw-models\Mistral-7B-Instruct-v0.3" ^
--outfile "C:\AI\models\Mistral-7B-Instruct-v0.3.fp16.gguf" ^
--outtype f164. Quantize to a Smaller Format
quantize.exe "C:\AI\models\Mistral-7B-Instruct-v0.3.fp16.gguf" ^
"C:\AI\models\Mistral-7B-Instruct-v0.3.Q4_K_M.gguf" ^
Q4_K_M5. Verify
./bin/Release/llama-cli.exe -m "C:\AI\models\Mistral-7B-Instruct-v0.3.Q4_K_M.gguf" -p "Hello!"Supported Quantization Types
Pass the quant type as the last argument to quantize:
| Type | Size vs FP16 | Quality |
|---|---|---|
Q2_K | ~17% | Poor |
Q3_K_M | ~25% | Low |
Q4_0 | ~27% | Medium |
Q4_K_M | ~29% | Good (sweet spot) |
Q5_K_M | ~34% | Very good |
Q6_K | ~40% | Excellent |
Q8_0 | ~50% | Almost lossless |
Converting Without a GPU
The convert.py and quantize steps are CPU-only — they don’t need a GPU. The FP16 file will be large (2× the final size), so ensure you have enough disk space.
One-Line Conversion (convert-hf-to-gguf)
Newer versions provide a convenience script:
python convert-hf-to-gguf.py "C:\AI\raw-models\Mistral-7B" ^
--outfile "C:\AI\models\model.gguf" ^
--outtype q4_k_mLast Updated: 2026-07-19