GPU Optimisation

Audio version — Estimated duration: 7 min 48 sec

VRAM tuning

The two most impactful VRAM knobs are batch size and chunk size.

Batch size

Controls how many text chunks are processed in a single GPU call. Larger batches increase throughput (better GPU utilisation) but consume more VRAM.

GPU VRAM	Base --batch_size	Custom --batch-size
4 GB	1	1
6 GB	1–2	1
8 GB	2–4	1–2
12 GB	4–8	2–4
16 GB+	8–16	4–8

The CustomVoice script defaults to --batch-size 1 and doesn’t have a sequential fallback. If you hit OOM with batch-size > 1, you must restart. Start conservative and work up.

Chunk size

Smaller chunks = less VRAM per chunk = faster per chunk (fewer audio tokens generated). But chunks that cut mid-sentence produce unnatural prosody.

max_chars	VRAM per chunk	Prosody quality
100	Lowest	Often choppy
150 (Custom default)	Low	Acceptable
200 (Base default)	Moderate	Good
300	High	Better
400	Highest	Best

Recommendation: Find the largest chunk size that doesn’t OOM with your desired batch size, then dial it back by 10% for headroom.

dtype selection

dtype	Speed vs float32	VRAM vs float32	Requirements	Notes
bfloat16	~2× faster	Half	Ampere+ (RTX 30xx, A100, H100)	Recommended — same exponent range as float32, no underflow
float16	~1.8× faster	Half	Any CUDA GPU	May underflow on small values (gradients, denormals)
float32	1×	Full	Any CUDA GPU	Compatible everywhere, but 2× slower and 2× more VRAM

Why bfloat16 > float16 on Ampere:

Both use 16 bits, but bfloat16 keeps the same 8-bit exponent as float32 (only the mantissa is truncated). float16 uses 5 exponent bits, which means it can’t represent numbers below ~6×10⁻⁸ — a common cause of silent underflow in transformer softmax layers. Since RTX 30xx has native bfloat16 hardware, there’s no reason to use float16.

The Base script is hardcoded to float16. The CustomVoice script defaults to bfloat16 and accepts --dtype to change it.

Attention backends

The CustomVoice script tries attention implementations in order:

1. sdpa — PyTorch’s scaled dot-product attention (CUDA kernel fused). Fastest on Ampere. Supports memory-efficient flash attention variants automatically when available.
2. eager — Naive PyTorch attention. No kernel fusion, uses more VRAM, ~2–3× slower.

The fallback is automatic — the script prints which backend was used so you can confirm.

To force a specific backend (e.g., if sdpa crashes on your GPU driver), edit md_to_audio_custom.py line 82:

for attn_impl in ("sdpa", "eager"):  # → ("eager",)

Why Windows is slower

The flash-attn library is Linux-only. It provides a further ~1.5–2× speedup over sdpa by using fused attention kernels that avoid materialising the full attention matrix in VRAM.

On Windows native:

• flash-attn is unavailable — the best you get is sdpa
• On some Windows configurations even sdpa may fail, forcing the eager fallback (another 2–3× penalty)
• Total: Windows native can be 2–6× slower than Linux with sdpa or flash-attn

Recommendation: Run on Linux natively, or on Windows via WSL2 which has full CUDA support and access to sdpa/flash-attn. See the Windows performance note.

Token budget optimisation

The adaptive token estimator (estimate_max_tokens()) in the CustomVoice script is the single biggest speed win on short texts:

Document length	Fixed 1024 tokens	Adaptive budget	Speedup
300 chars (2 chunks)	~20s	~8s	~60%
1000 chars (7 chunks)	~70s	~42s	~40%
5000 chars (33 chunks)	~330s	~300s	~9%

The effect diminishes on long documents because most chunks saturate at 1024 tokens anyway.

If you find that chunks are consistently hitting the 1024 ceiling (visible in estimate_max_tokens output), you can increase max_new_tokens in the function — but the model’s architecture may not benefit beyond 1024.

TF32 matmul

Both scripts enable TF32 for matrix multiplications and cuDNN operations:

torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

TF32 is a precision mode on Ampere GPUs that uses 10-bit mantissa (vs float32’s 23-bit) while keeping the full 8-bit exponent. This gives ~8× throughput on matrix math with negligible quality loss for inference. If you’re seeing numerical instability (unlikely for TTS inference), you can disable these flags.

cuDNN benchmark (CustomVoice only)

torch.backends.cudnn.benchmark = True

This tells cuDNN to run a short benchmark on the first convolution it sees and select the fastest kernel for your specific GPU and input sizes. The first generation call will be slightly slower (benchmarking overhead), but all subsequent calls benefit. If you process many documents in sequence, this is a net win. Leave it on.

Interactive demo

The qwen-tts package includes a Gradio web UI for quick experimentation:

qwen-tts-demo ./Qwen3-TTS-12Hz-0.6B-CustomVoice \
  --port 8000 \
  --ip 127.0.0.1 \
  --device cuda:0 \
  --dtype bfloat16 \
  --no-flash-attn

This launches a browser interface where you can type text, select speakers, and hear results instantly — useful for testing speakers and instructions before running batch script synthesis.

---

See Troubleshooting if you run into issues.