Model · Layer 2: How Is a Model Trained?

Core question: How do a model's capabilities form? How do a set of initially nearly incapable parameters gradually develop language understanding and generation abilities through data, objective functions, and repeated adjustment?

Key concepts: Dataset, Training, Objective, Loss, Backpropagation, Optimizer, Pretraining, Post-training, Fine-tuning, Checkpoint

Definition

One-sentence definition: Model training is the process of repeatedly feeding the model training data, measuring the error between its outputs and the targets, and continuously adjusting parameters so the model gradually learns to perform a class of tasks.

Intuitive understanding: A model starts with a fixed architecture and a large set of initial parameters. The training program repeatedly feeds it data, computes error, and adjusts parameters, eventually producing a set of weights that can generate useful outputs.

Training does not store materials verbatim inside the model; instead, it continuously adjusts parameters over large numbers of samples so the model learns statistical patterns and regularities in the data.

Five Elements of Training:

Training
│
├── Dataset
├── Model Architecture
├── Training Objective
├── Loss Function
└── Optimizer

1. DataSet

What data the model learns from. A Dataset is the collection of data used during training.

Large language model data may include: web text, books, code, papers, Q&A data, conversation data, human-annotated data, synthetic data;
Multimodal models may also include: images with text descriptions, video with text descriptions, audio with transcripts, and correspondences among images, audio, and video;

Data typically goes through: raw data -> collection -> cleaning -> deduplication -> filtering -> annotation and construction -> filtering -> splitting -> Tokenize -> training data.

Concept	Meaning
Training Set	Data used to update model parameters
Validation Set	Data used to monitor training progress and tune configuration
Test Set	Data used for final evaluation of model capability
Sample	A single training example
Label / Target	The target the model should predict
Synthetic Data	Training data generated by a model or program

Data determines what the model is exposed to; data quality, quantity, and distribution all affect the capabilities and biases the model ultimately develops.

2. Training Objective

What the model is asked to learn. For autoregressive large language models, the most typical training objective is: given previous Tokens, predict the next Token.

For example:

Input: The weather today is
Target: nice

During training the model does not only predict the last Token; it usually computes predictions for multiple positions in the sequence at once:

The          → weather
The weather  → today
The weather today → is
The weather today is → nice

Through next-Token prediction on massive amounts of text, the model gradually learns:

Language structure;
Word relationships;
Grammar;
Common knowledge;
Styles of expression;
Some reasoning and task patterns.

3. Loss

How wrong the model is. Loss measures the gap between the model's current output and the training target.

The closer the prediction to the target → the smaller the Loss
The farther the prediction from the target → the larger the Loss

The overall goal of training is usually: find a set of parameters that makes Loss on the training data as low as possible.

Loss Function: the method for computing error;
Training Loss: loss on training data;
Validation Loss: loss on validation data.

A decreasing Loss does not necessarily mean the model will be better in real use; evaluation on validation sets, test sets, and actual tasks is still needed.

How Does One Parameter Update Happen?

Forward Pass

The model takes an input and, with current parameters, computes an output: input + current parameters → model computation → prediction

Loss Calculation

Compare the prediction with the correct target to obtain Loss.

Backpropagation

Backpropagation computes: how much each parameter contributed to this error.

Optimizer: update parameters

The Optimizer adjusts parameters based on gradients and learning rate: new parameters = old parameters - learning rate × gradient.

The full training loop is: prepare data → forward pass → compute loss → backpropagation → update parameters → repeat, until Loss reaches the target or training ends.

Batching of training data

Basic units: Dataset -> Epoch -> Batch -> Sample

Concept	Meaning
Sample	A single training example
Batch	A group of samples fed to the model at once
Batch Size	Number of samples in one Batch
Step	Usually refers to one parameter update
Epoch	One full pass over the training set
Iteration	One training loop; in context often close to Step
Learning Rate	Step size for each parameter update

Training Stages a Model Goes Through

Pretraining
↓
Post-training
↓
Optional domain or task adaptation

1. Pretraining

Train the model on large-scale general data so it acquires basic language ability, knowledge patterns, and general representation capacity.

Also commonly called:

Pretrained Model
Base Model
Foundation Model

For example, a Base Model may be good at writing but not good at:

Following instructions;
Multi-turn conversation;
Refusing unsafe requests;
Answering in a specified format;

Pretraining solves: first give the model general foundational capabilities.

2. Post-training

Post-training is a series of training and alignment processes after pretraining that make the model better suited for interacting with people and completing concrete tasks, including:

Supervised Fine-tuning (SFT)
Instruction Tuning
Preference Tuning
Alignment / Safety Tuning

Supervised Fine-tuning (SFT)

Continue training the model on high-quality "input—target output" data. For example: user question -> ideal answer.

This teaches the model:

How to follow instructions;
How to organize answers;
How to conduct multi-turn conversation;
How to output in a specified format.

Instruction Tuning

Instruction tuning can be seen as a common form of SFT; the focus is using large amounts of instruction data across different tasks so the model learns to understand and execute natural-language instructions.

Preference Tuning

Give the model multiple answers plus preference signals from humans or models, so it learns: which answers are more helpful, more aligned with expectations, or safer.

Common terms include:

RLHF;
Reward Model;
PPO;
DPO;
RLAIF.

Alignment

"Alignment" is a broader goal: make the model's behavior better match human intent, value requirements, and safety norms.

3. Fine-tuning

A broad concept: on top of an already trained model, continue training with new data so its capabilities or behavior adapt to a specific goal.

Fine-tuning may happen on:

General instruction data;
Domain-specific data;
Task-specific data;
Style-specific data.

Type	Meaning
Full Fine-tuning	Update most or all of the model's parameters
PEFT	Train only a small number of new or selected parameters
LoRA	A common parameter-efficient fine-tuning method
Domain Fine-tuning	Adapt to domains such as medical, legal, or finance
Task Fine-tuning	Adapt to tasks such as classification, extraction, or code generation

Post-training can include fine-tuning and preference optimization; fine-tuning can also be used for later domain or task adaptation.

Checkpoint

A Checkpoint is the model state saved at a point in time during training. It includes:

Checkpoint
│
├── Model Weights
├── Optimizer State
├── Learning Rate Scheduler State
├── Current Step / Epoch
└── Random State or other training state

Two main uses:

Resume training. Training may last days or even months. If interrupted, you can continue from the latest Checkpoint without starting over.
Save different training stages. Different Checkpoints may perform differently; when releasing, the team may choose the one with the best validation results, not necessarily the last saved Checkpoint.

Concept	Main use	Contents
Training Checkpoint	Resume or continue training	Weights and optimizer and other training state
Model Weights	Load the model for inference	Usually mainly parameter values
Released Model	Distribution and use	Weights, config, Tokenizer, docs, etc.

Model capability does not come from a single factor, but from several factors working together:

Model Capability
│
├── Architecture (how the model can compute)
├── Data (what the model has seen)
├── Objective (what the model is asked to learn)
├── Training Scale (how much it was trained)
├── Optimization (whether parameters were trained well)
└── Post-training (how the model follows human intent)

Architecture determines the computation framework the model has; data provides learning material; the training objective sets the learning direction; optimization writes these patterns into parameters; post-training further shapes the model's interaction style and behavioral boundaries.

Concept Map

The essence of model training is using data and training objectives to produce error signals, then repeatedly adjusting parameters via backpropagation and an optimizer. Pretraining gives the model general foundational capabilities; post-training and fine-tuning further shape instruction following, interaction style, and specialized abilities; the parameters formed during training are saved via Checkpoints and final weights.

Model Training
│
├── Data
│   ├── Dataset
│   ├── Sample
│   ├── Training Set
│   ├── Validation Set
│   └── Test Set
│
├── Training Mechanism
│   ├── Training Objective
│   ├── Forward Pass
│   ├── Loss
│   ├── Backpropagation
│   ├── Gradient
│   └── Optimizer
│
├── Training Units
│   ├── Token
│   ├── Batch
│   ├── Step
│   └── Epoch
│
├── Training Stages
│   ├── Pretraining
│   ├── Post-training
│   │   ├── SFT / Instruction Tuning
│   │   ├── Preference Tuning
│   │   └── Alignment
│   └── Domain / Task Fine-tuning
│
└── Training Output
    ├── Parameters / Weights
    └── Checkpoint