Layer 4: How Are Models Saved and Published?

Core question: What files do training results ultimately become?

Key concepts: Checkpoint, Serialization, Weights, Safetensors, Shard, Index, Config, Tokenizer, Processor, Chat Template, Model Card, Model Repository, Revision

Training produces parameters and training state in memory; to pause training, distribute models, and run inference, these must be serialized into files and published together with configuration, Tokenizer, documentation, and other resources.

During training, model parameters exist as Tensors in memory or VRAM. A linear layer may contain: Weight Tensor of shape [4096, 4096], Bias Tensor of shape [4096].

The whole model is made of large numbers of such Tensors:

Model Parameters
├── model.embed_tokens.weight
├── model.layers.0.self_attn.q_proj.weight
├── model.layers.0.self_attn.k_proj.weight
├── model.layers.0.mlp.up_proj.weight
├── model.layers.1...
└── lm_head.weight

The core work of saving a model is: serialize in-memory Tensors—their names, shapes, data types, and values—into disk files.

Serialization

Convert in-memory data structures into a file format that can be stored or transmitted.

Loading a model does the reverse: disk file → deserialization → Tensor → place into memory or VRAM.

A weight file is not the model's source code; it is the serialized result of the large set of numeric values obtained after training.

Checkpoint: Save Points During Training

A Checkpoint is the model state saved at a point in time during training, used to resume training, compare different stages, or choose the final model.

A complete training Checkpoint may include:

Training Checkpoint
│
├── Model Weights
├── Optimizer State
├── Learning Rate Scheduler State
├── Training Step / Epoch
├── Gradient Scaler State
└── Random Number Generator State

Checkpoint is also used loosely to mean a complete model file that can be loaded directly.

Weights: Data That Stores Model Capability

Weight files store the numeric values of each parameter after training. Conceptually they represent:

• Tensor shape
• Tensor data type
• Tensor concrete values

Safetensors: A Format for Saving Weight Files

A container format specialized for holding large numbers of model-parameter Tensors. Early PyTorch models often used: pytorch_model.bin.

Such files are usually based on Python Pickle. Pickle can theoretically execute code carried in the payload during deserialization, so loading untrusted files has security risk.

One design focus of Safetensors is: save only Tensor data, without deserializing arbitrary Python objects via Pickle, reducing the risk of executing malicious code at the format level.

Shard: Why Does One Model Have Many Weight Files?

Weights of large models may be tens of GB, hundreds of GB, or even larger. Putting everything in one file causes:

• Single file too large;
• Awkward upload and download;
• Possible filesystem or storage-service limits;
• Harder on-demand loading;
• Inconvenient distributed handling.

Therefore model weights are usually split into multiple files. This process is called Sharding. Each file is called a Shard.

model-00001-of-00018.safetensors
model-00002-of-00018.safetensors
model-00003-of-00018.safetensors
...
model-00018-of-00018.safetensors
 
model-00001-of-00018
      │         │
      │         └── 18 shards in total
      └── This is shard 1

Weight Index: How Do You Know Which Shard a Parameter Is In?

After weights are sharded, the inference framework needs to know which file each parameter is stored in. Therefore repositories usually include: model.safetensors.index.json. Think of it as a directory or index table for weight shards. For example:

{
  "metadata": {
    "total_size": 55562855904.0
  },
  "weight_map": {
    "lm_head.weight": "model-00018-of-00018.safetensors",
    "model.language_model.embed_tokens.weight": "model-00003-of-00018.safetensors",
    "model.language_model.layers.0.input_layernorm.weight": "model-00001-of-00018.safetensors",
    "model.language_model.layers.0.linear_attn.A_log": "model-00001-of-00018.safetensors",
    "model.language_model.layers.0.linear_attn.conv1d.weight": "model-00001-of-00018.safetensors",
    ...
  }
}

Config: Telling the Program What Model Structure to Create

The most common configuration file is config.json, containing:

• Model type;
• Architecture class name;
• Number of layers;
• Hidden Size;
• Number of Attention Heads;
• Vocabulary Size;
• Position Encoding configuration;
• Maximum context length;
• Data type hints;
• Special Token IDs;
• Multimodal structure configuration.

Generation Config: Controlling Default Generation Behavior

generation_config.json stores default generation parameters, for example:

• Temperature;
• Top-p;
• Top-k;
• Max Length;
• Repetition Penalty;
• EOS Token ID;
• PAD Token ID.

Tokenizer Files: Saving Text Splitting Rules

A Tokenizer is not simply a vocabulary table. A complete Tokenizer may include:

Tokenizer
│
├── Vocabulary
├── Tokenization Algorithm
├── Merge Rules
├── Normalization Rules
├── Special Tokens
└── Tokenizer Config

• Model weights are trained on a specific Token ID system.
• Tokenizer and model weights are usually paired.

Processor: Input Processing Rules for Multimodal Models

For image, audio, and video models, the model repository may also include:

preprocessor_config.json
processor_config.json
video_preprocessor_config.json

These may specify:

• Image scaling method;
• Image cropping method;
• Pixel normalization rules;
• Video sampling method;
• Frame count or frame-rate handling;
• Audio sample rate;
• How different modalities are combined.

Multimodal model input resources can be summarized as:

• Text → Tokenizer
• Image → Image Processor
• Video → Video Processor
• Audio → Audio Processor

Processing rules must stay consistent with those used during model training.

Model Repository: The Complete Carrier After a Model Is Published

A model repository is a versioned store for saving, managing, and publishing the resources needed for a model version.

Model Repository
│
├── Model Identity
│   ├── Repository Name
│   ├── Model Family
│   └── Model Variant
│
├── Model Structure
│   └── config.json
│
├── Model Weights
│   ├── *.safetensors
│   └── *.index.json
│
├── Input Processing
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   ├── vocab.json
│   ├── merges.txt
│   └── preprocessor_config.json
│
├── Inference Behavior
│   ├── generation_config.json
│   └── chat_template.jinja
│
└── Documentation
    ├── README.md / Model Card
    └── LICENSE

LoRA and Quantized Model Repositories

Not every model repository contains a full set of weights.

A LoRA or Adapter repository usually contains only a small number of new or modified parameters relative to the base model.

A quantized model repository contains conversions from the original model such as:

• INT8;
• INT4;
• FP8;
• GGUF;
• GPTQ;
• AWQ.

Summary:

Parameters obtained from training are first saved as Checkpoints. For release, model weights are extracted and serialized into formats such as Safetensors; when too large they are split into multiple Shards, with an index file recording each parameter's location. Weights together with Config, Tokenizer, Processor, Chat Template, Model Card, License, and other resources form a model repository for others to download, load, and run.

• Weight files store the large Tensor values obtained from training;
• Config describes how to create the model structure; weights fill in the concrete parameters;
• Shards are only file splits of the same set of weights, not multiple independent models;
• In engineering, models are usually published as model repositories, not as a single weight file alone.