On September 22, 2026, Hugging Face announced on its official blog that transformers can now load GGUF quantized models directly. For anyone who runs models on a laptop, this means a familiar from_pretrained call is all it takes — no more shuttling between the transformers and llama.cpp toolchains.
Why it used to be a hassle
GGUF is the quantized-model format of the llama.cpp ecosystem, and the vast majority of downloadable quantized weights in the community are GGUF files. Transformers users who wanted those weights either had to convert formats themselves or spin up a separate llama.cpp inference flow. Neither option was elegant: conversion carries precision and compatibility risks, while a second toolchain means two sets of dependencies and two APIs.
How this release fixes it
The key point is that Hugging Face did not reinvent the wheel. The stated goal is performance "close to llama.cpp", achieved by reusing llama.cpp's own ggml kernels (via the kernels library) and cutting overhead in the generate step. The blog also notes that transformers serve can now expose a GGUF model through an OpenAI-compatible API, so desktop inference clients like Jan or Pi can connect directly. The official example is Qwen3.5-4B: an 8.42 GB file at BF16 precision shrinks to 2.74 GB at Q4_K_M — the entire reason GGUF exists: turning a model that won't fit in laptop memory into one that will.
Who can use it now, and who should wait
Initial optimization focuses on Apple Silicon, with throughput tested on ggml's Metal kernels; the test machine was a MacBook Pro M2 Max with 32 GB of unified memory. Mac users are the first to benefit. Hugging Face has not published equally detailed numbers for CUDA or CPU, so anyone chasing maximum speed may want to wait. One more caveat: quantization itself has a cost. 4-bit quantization like Q4_K_M loses some precision — a 2.74 GB file is not the same thing as the 8.42 GB original. Don't judge a model by its file size alone.
