On September 22, 2026, Hugging Face announced on its official blog that transformers now supports loading GGUF quantized models directly. Users only need to pass a gguf_file argument to from_pretrained to pull a GGUF checkpoint from the Hub into the familiar transformers API and run it on a local machine. Thanks to reusing ggml's low-level kernels, inference performance is now close to llama.cpp, the company says.
One argument, and GGUF is inside transformers
Usage is simpler than you might expect:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained( "unsloth/Qwen3.5-4B-GGUF", gguf_file="Qwen3.5-4B-Q4_K_M.gguf",
)Everything after that is the standard transformers workflow: tokenizer, generate, custom logits processors — all work as usual. The same checkpoint can also be exposed as an OpenAI-compatible API via transformers serve, turning clients like Jan or Pi into chat front ends.
Performance is the most convincing part of this update. Hugging Face benchmarked three checkpoint types on a MacBook Pro M2 Max (32 GB unified memory) — a small dense model, a larger dense model, and a mixture-of-experts model — and transformers' generation throughput essentially matched llama.cpp's llama-bench scores. The secret is the reuse of ggml's Metal kernels: quantized matrix math, fused normalization, flash attention and other key operators call ggml implementations directly, with Python only orchestrating.
Who this is for: researchers, not pure speed seekers
The official blog is explicit about the target audience. The integration serves four main needs: experimenting with GGUF in Python/PyTorch, evaluating quantized checkpoints with existing evaluation workflows, validating that GGUF conversions are correct, and dequantizing from GGUF to continue fine-tuning. What it solves is "developer convenience," not "faster" — the blog states plainly that llama.cpp remains the recommended engine for efficient local inference.
The boundaries are worth noting too: initial support is Apple Silicon only, starting with the Qwen3.5 architecture; you need transformers' main branch plus the kernels package; and without a compatible quantization kernel the loader falls back to dequantized loading, which uses noticeably more memory. We previously covered llama.cpp's role as a local inference base (Who is llama.cpp for? A lightweight local-model runtime, not a chat product (/zh/article/1856-llama-cpp-shi-he-shui-ben-di-pao-mo-xing-de-qing-liang-di-zuo-bu-shi-liao-tian-c)); this update can be read as: keep using llama.cpp for inference, but when researching or prototyping you can now tinker with the same GGUF file directly in transformers — no more shuttling weights between two toolchains.
There is also a practical angle for Chinese developers: GGUF distributions of popular models like Qwen3.5 (from Unsloth, bartowski and others) are downloaded enormously on the Hub. Using them inside the PyTorch ecosystem previously meant writing your own loading logic; the official path is now paved.
