GGUF - Model File Format

Official Introduction

• GGUF is a file format for storing models for inference with GGML and executors based on GGML.

• GGUF is a binary format that is designed for fast loading and saving of models, and for ease of reading. Models are traditionally developed using PyTorch or another framework, and then converted to GGUF for use in GGML.

• It is a successor file format to GGML, GGMF and GGJT, and is designed to be unambiguous by containing all the information needed to load a model.

• It is also designed to be extensible, so that new information can be added to models without breaking compatibility.

Understanding

• GGUF is a binary file format used for storing machine learning models, optimized for fast loading and saving, making it highly efficient for inference scenarios.

• It saves the model's Tensors (weight and other tensor data) and Metadata (model architecture, Tokenizer, parameters, etc.) in a single file, and is designed for fast loading and saving of model data. Taking Qwen3.8 as an example, comparing the Qwen3.8-27B Qwen3.8-27B standard model repository with the Qwen3.8-27B-GGUF Qwen3.8-27B-GGUF GGUF repository:

  • The file structure of the standard model repository is as follows, mainly serving inference frameworks like Transformers, vLLM, SGLang, TokenSpeed, etc. When Transformers loads a model, it needs to read files like config.json to learn about the model architecture, the number of layers, hidden_size, etc., which is part of the HF / Transformers model directory protocol;

Qwen3.8-27B/
├── config.json
├── generation_config.json
├── tokenizer.json
├── tokenizer_config.json
├── preprocessor_config.json
├── model-00001-of-xxxxx.safetensors
├── model-00002-of-xxxxx.safetensors
└── ...

• The core design goal of GGUF is: a single-file format. Besides tensor weights, it can also contain configuration properties, Tokenizer vocabulary, and other model properties. It is mostly used for inference with llama.cpp (a popular C/C++ LLM inference framework), as shown below, where files like config.json and tokenizer.json do not appear;

unsloth/Qwen3.8-27B-GGUF/
├── Qwen3.8-27B-Uncensored-Q5_K_M.gguf
├── Qwen3.8-27B-Uncensored-draft-Q4_0.gguf
├── Qwen3.8-27B-Uncensored-draft-Q8_0.gguf
└── mmproj-Qwen3.8-27B-Uncensored-F16.gguf
└── ...

• Rough understanding of the file structure:

Transformers Model
─────────────────────────────
config.json
tokenizer.json
tokenizer_config.json
generation_config.json
*.safetensors
...
 
        ↓ Convert to GGUF
 
GGUF
─────────────────────────────
Qwen3.8-27B-Uncensored-Q5_K_M.gguf
 
Internal:
├── metadata
│   ├── architecture
│   ├── context length
│   ├── embedding size
│   ├── attention heads
│   ├── RoPE parameters
│   └── ...
│
├── tokenizer
│   ├── vocabulary
│   ├── token types
│   ├── special tokens
│   └── ...
│
└── tensors
    ├── embedding
    ├── attention weights
    ├── FFN weights
    └── ...

• GGUF is specifically designed to work with GGML and other execution engines. It was developed by ggerganov, who is also the developer of llama.cpp.

• GGML is a tensor library for machine learning that emphasizes C/C++, cross-platform support, low dependencies, and highly efficient local inference. It supports multiple backends like CPU and GPU, as well as quantization types from 2 to 8-bit. Projects like llama.cpp commonly use the GGML ecosystem.

• It often appears alongside quantization, but GGUF itself is not a quantization method. However, it is highly suitable for saving and distributing quantized models. It is common to see quantized models like Q4_K_M, Q5_K_M, and Q8_0 provided as .gguf files.

• It has become a very common model format in the local LLM inference ecosystem and can be directly used by tools like llama.cpp, Ollama, and LM Studio;

GGML/llama.cpp ecosystem → GGUF file format → Tensor + Metadata → Commonly used for quantized models → .gguf → Filenames usually directly reflect model size, fine-tuning type, quantization Encoding, and shards.

About the Full Name

The official documentation does not actually provide a full name expansion for GGUF. GGML Unified Format is currently the most reasonable and widely accepted expansion by the community; however, it is not a formal full name explicitly defined in the official GGML specification.

The source code gguf.h also simply refers to it as: "GGUF" files, the binary file format used by ggml. It does not explain what each of the four letters stands for.

In the original materials from when GGUF was created, the Issue #220 proposing the new format in 2023 was simply titled: GGJTv4/GGUF.

The author stated that he wanted to propose a new universal format to solve the fragmentation issues of the previous GGML/GGMF/GGJT file formats, while retaining single-file deployment.

The currently common expansions are as follows:

There is also a Reddit post ranting about this matter, which you can check out if you are interested.

GGUF Naming Convention

The greatest value of the Naming Convention is that you can obtain the core information of the model just from the filename without opening the model card. For example: Qwen3-8B-Instruct-Q4_K_M.gguf, quickly leads to the following conclusion:

Qwen3 series → 8 Billion parameter scale → Instruction fine-tuned model → Q4_K_M quantization/encoding → GGUF format.

Rule: [<Sidecar>]<BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf, each part is usually separated by -:

[Sidecar]-BaseName-SizeLabel-FineTune-Version-Encoding-Type-Shard.gguf

This convention describes the standard writing of each field in the GGUF filename, designed for human readability. It does not guarantee that all real-world GGUF filenames can be strictly parsed according to this rule.

In practice, many files released by the community follow "conventions" rather than strictly copying the official specification, and variant naming styles are frequently seen.

Qwen3-8B-Instruct-Q4_K_M.gguf
│     │  │        │       └─ GGUF file format
│     │  │        └───────── Q4_K_M encoding/quantization
│     │  └────────────────── Instruct fine-tuning
│     └───────────────────── 8B parameter size
└─────────────────────────── Model name

Encoding Explanation

Assume a model has 8B = 8 Billion parameters, and each parameter weight uses FP16, then:

Q = Quantization.

High Precision / Large Size
        ↑
FP16
Q8
Q6
Q5
Q4      ← Very common
Q3
Q2
        ↓
Low Precision / Small Size

Generally:

Higher bit

→ Larger file size
→ Higher RAM/VRAM usage
→ Usually preserves more model quality

Lower bit

→ Smaller file size
→ Lower RAM/VRAM usage
→ Easier to run large models
→ Usually increases quantization loss

Q4 is approximately the 4-bit quantization level;

Q4_K_M uses the GGML K-quant series scheme, primarily based on 4-bit, and adopts the tensor-type mixture corresponding to M.

K represents a K-quant (K-quants) quantization system later introduced by GGML/llama.cpp. K-quants use more complex block, scale, and other quantization representations, and allow different tensors in the model to use different quantization types.

S/M/L reflects: what combination of quantization types (quantization mix) is used for different Tensors in the model.

Q4_K_M
│  │ │
│  │ └── M: A medium/mixed quantization configuration
│  │
│  └──── K: K-quant quantization system
│
└─────── Q4: Primarily quantized at the 4-bit level

              Model Quality
                 ↑
 
Q8_0         Higher
Q6_K
Q5_K_M
Q5_K_S
Q4_K_M       ← Common balance point
Q4_K_S
Q3_K_M
Q3_K_S
Q2_K
 
                 ↓
          File Size / Memory Requirement Decreases

Related Links

gguf.md

GGUF - Wikipedia

Hugging Face GGUF Documentation

Transformer GGUF Documentation