Layer 5: How Is a Model Loaded and Run?

Core question: After downloading a model, how does the computer actually run it?

Key concepts: Inference, Inference Engine, Runtime, CPU, RAM, GPU, VRAM, Precision, Quantization, Device, Offload, KV Cache, Batch, Prefill, Decode, Latency, Throughput.

A model on disk is only weights, configuration, and input-processing files; the inference engine must create the computation structure from the config, load weights into RAM or VRAM, and call the CPU and GPU to run Tensor computations again and again before the model truly "runs."

Two processes:

1. Load the model: read files, create structure, allocate memory, place weights;
2. Run inference: process input, run model computation, generate Tokens one by one.

Pipeline

Inference

Inference means using already-trained model parameters to process new inputs and compute output results.

Large language model:

User input
→ Model computation
→ Predict next Token
→ Gradually generate an answer

Image classification model:

Image
→ Model computation
→ Classification probabilities

Image or video generation model:

Prompt / image / video
→ Multiple model computations
→ Gradually generate image or video

Both inference and training run the model's Forward Pass, but with different goals:

For the same model, inference usually needs less VRAM and compute than training.

Inference Engine

A weight file only stores Tensor values; what loads and runs the model is the Inference Engine. It typically handles:

Inference Engine
│
├── Read model configuration
├── Create model structure
├── Load weights
├── Manage CPU / GPU devices
├── Manage RAM / VRAM
├── Execute Tensor operations
├── Manage KV Cache
├── Organize Batch
├── Run sampling
└── Return or stream results

Common inference tools and engines include:

What Does the Inference Engine Do When Loading a Model?

1. Read Config

The inference engine first reads: config.json. From it it obtains:

• Model type;
• Number of layers;
• Hidden Size;
• Number of Attention Heads;
• Vocabulary Size;
• Maximum context;
• Data type;
• Multimodal component configuration.

2. Create Model Structure

Based on Config and the corresponding model implementation code, create the computation structure:

Embedding
→ Transformer Block 1
→ Transformer Block 2
→ ...
→ LM Head

At this point you can think of it as:

The inference framework first builds an "empty model" composed of computation layers.

3. Locate Weight Files

If weights are sharded, the inference engine reads:

model.safetensors.index.json

to determine which Shard each parameter is in.

4. Deserialize Weights

The inference engine reads disk data into Tensors:

Safetensors file
→ Tensor name
→ Tensor shape
→ Tensor data type
→ Tensor values

5. Match Structure and Weights

Parameters in the model structure must correspond to Tensors in the weight files.

6. Place Weights on Devices

Weights may be placed on:

• RAM associated with the CPU;
• VRAM associated with the GPU;
• Multiple GPUs;
• A combination of RAM and VRAM;
• In a few cases, disk-mapped regions.

What Do CPU and GPU Each Handle?

CPU, Central Processing Unit, is a general-purpose compute processor good at:

• Operating-system and program control;
• File reading;
• Input preprocessing;
• Tokenizer;
• Task scheduling;
• Branching and complex control logic;
• Memory management;
• Some model computation.

A model can run entirely on CPU, but large-model computation is usually very slow.

GPU, Graphics Processing Unit, is a massively parallel compute processor originally aimed at graphics, but very well suited to:

• Matrix multiplication;
• Vector computation;
• Parallel execution of large numbers of identical operations;
• Tensor operations in neural networks.

Model inference involves large amounts of matrix computation; GPUs can run many similar computations at once, so they are usually better suited than CPUs for large neural-network inference.

The GPU does not take over the entire application. The CPU usually still handles program control, data preparation, and task scheduling; the GPU mainly carries parallelizable model computation.

What Are RAM and VRAM?

RAM, Random Access Memory, system memory, often just called "memory." Managed by the CPU and OS, it mainly stores:

• Running programs;
• Model files read from disk;
• Weight Tensors on CPU;
• Input and output data;
• Intermediate computation results;
• File caches;
• Model layers that have been Offloaded.

VRAM, Video Random Access Memory, often called "video memory" or GPU memory. It sits on the graphics card and mainly stores data needed for GPU computation:

• Model weights;
• Input Tensors;
• Activations;
• KV Cache;
• Temporary compute buffers;
• CUDA Kernel workspace;
• Intermediate results in image or video generation.

What Is in VRAM When a Model Runs?

Many people equate VRAM usage directly with weight size, but actual VRAM usually includes several parts:

VRAM Usage
│
├── Model Weights
├── KV Cache
├── Activations
├── Input / Output Tensors
├── Temporary Buffers
├── Framework Overhead
└── CUDA Context

Therefore: VRAM needed to run > model weight file size

An 8GB weight file does not mean an 8GB GPU can necessarily run it, because space is still needed for other data. Different inference frameworks, model architectures, input lengths, Batch Sizes, and compute precisions all affect extra overhead.

Precision: What Numeric Format Is Used?

Precision means what numeric format model parameters and computation use.

Common formats include:

The coarsest weight-size relationship is:

FP32
→ FP16 / BF16: about half
→ FP8 / INT8: about one quarter
→ INT4: about one eighth

Quantization: Why Quantize a Model?

Quantization means: represent model weights or computation data with lower bit-widths to reduce storage, memory use, and compute cost. For example, original BF16 weights -> quantize -> INT8 or INT4 weights.

Main goals of quantization include:

• Reduce model file size;
• Reduce RAM and VRAM use;
• Lower memory-bandwidth pressure;
• Speed up inference on suitable hardware;
• Let large models run on devices with fewer resources.

Trade-offs may include:

• Loss of numeric precision;
• Drop in model quality;
• Some tasks more sensitive to quantization;
• Need for specialized operators and inference engines;
• Different speedups on different hardware.

Device: Which Device Does the Model Actually Run On?

Device means where Tensors and computation live.

Common forms include:

CPU
CUDA GPU
Apple Metal GPU
Other AI accelerators

In PyTorch you may see:

cpu
cuda:0
cuda:1
mps

Where:

cuda:0 → first NVIDIA GPU
cuda:1 → second NVIDIA GPU
mps    → Apple Silicon GPU backend

Model weights and input Tensors involved in computation usually must live on compatible devices.

Offload: What If VRAM Is Not Enough?

Offload in model inference usually means: do not keep all model components in GPU VRAM at all times; instead keep some in RAM, or even on disk, and move them to the GPU when needed.

Common approaches include:

1. CPU Offload

Keep some weights in RAM
→ Move into VRAM before computation
→ GPU finishes computation
→ Swap in other weights when needed

Advantages:

• Lower VRAM demand;
• Let smaller-VRAM GPUs run larger models.

Costs:

• Data must move between CPU and GPU;
• Limited by PCIe bandwidth;
• Speed may drop significantly;
• Higher RAM use.

2. Disk Offload

Keep some weights on disk
→ Read into RAM
→ Then move into VRAM

This can further reduce RAM demand, but disk is far slower than memory and usually causes larger performance loss.

3. Sequential Offload

Some image or video generation workflows contain multiple model components:

Text Encoder
→ Diffusion Model
→ VAE

The inference engine can load them in stages:

Load Text Encoder
→ Finish text encoding
→ Unload Text Encoder
 
Load Diffusion Model
→ Run generation
→ Unload Diffusion Model
 
Load VAE
→ Run decoding

This approach especially suits devices with limited VRAM, but increases time spent swapping models in and out.

Offload is not the same as treating RAM as VRAM

A program can keep some data in RAM, but: RAM does not truly become the GPU's local VRAM. Data the GPU needs for computation still must travel over the CPU–GPU path.

An analogy:

VRAM: the GPU's workbench at hand
RAM: shelves in the room
SSD: the warehouse

When the workbench is full, materials can be parked on shelves or in the warehouse, but each use still requires carrying them over.

Multiple GPUs: Can You Just Add VRAM Across Cards?

Multiple GPUs can run a model together, but it is not simply: 12GB + 12GB = one 24GB GPU

The inference framework must explicitly support model splitting and cross-card communication. Common approaches include:

The first three are mainly for:

Multiple GPUs
→ Jointly complete one model instance or one inference

Data Parallelism is:

GPU 1 → request A
GPU 2 → request B
GPU 3 → request C

So in batch generation scenarios, first confirm: is the goal for multiple cards to jointly run one large model, or for each card to run a task independently?

The latter usually has a simpler architecture and more easily yields linear throughput gains.

KV Cache: Why Does Generation Keep Consuming More VRAM?

Large language models generate Tokens one by one autoregressively. If every new Token recomputed Attention intermediates for all previous Tokens, that would produce huge redundant computation. Therefore inference engines usually save Key and Value produced by past Tokens in Attention; this cache is called KV Cache (Key-Value Cache).

• Without KV Cache, every generation step reprocesses the full history.
• With KV Cache, save some intermediate results for historical Tokens and only compute new results for the new Token.

Effects:

• Reduce redundant computation;
• Speed up Token-by-Token generation;
• Support efficient autoregressive inference.

But it also uses RAM or VRAM, and usually grows with:

• Context length;
• Batch Size;
• Number of concurrent requests;
• Number of model layers;
• Number of KV Heads;
• KV Cache precision.

Therefore: being able to fit model weights in VRAM does not mean you can necessarily handle very long context or many concurrent requests.

Because remaining VRAM must also hold the KV Cache.

Latency and Throughput: How Do You Measure Inference Speed?

Latency means how long an operation takes to complete.

Throughput means how much work the system can complete per unit time. Common forms include:

• Tokens per Second (TPS): Tokens generated per second;
• Requests per Second (RPS): requests handled per second;
• Images per Minute (IPM): images generated per minute;
• Videos per Hour (VPH): videos generated per hour.

Differences for Generative Image and Video Models

Image and video generation models may run differently. For example, diffusion-style models:

Prompt
→ Text Encoder
→ Initial noise
→ Diffusion Model multi-step denoising
→ VAE Decode
→ Image or video

Runtime VRAM may include:

Generative Model VRAM
│
├── Text Encoder Weights
├── Diffusion Model Weights
├── VAE Weights
├── Latent
├── Attention intermediates
├── Temporary compute buffers
└── Image / video Tensors

Concepts

Model Loading & Inference
│
├── Inference Software
│   ├── Framework
│   ├── Inference Engine
│   ├── Runtime
│   └── Backend
│
├── Hardware
│   ├── CPU
│   ├── RAM
│   ├── GPU
│   ├── VRAM
│   └── SSD
│
├── Model Loading
│   ├── Config
│   ├── Architecture Implementation
│   ├── Deserialization
│   ├── Weight Loading
│   └── Device Map
│
├── Numerical Representation
│   ├── Precision
│   ├── FP32 / FP16 / BF16
│   ├── FP8 / INT8 / INT4
│   └── Quantization
│
├── Memory Management
│   ├── Model Weights
│   ├── Activations
│   ├── KV Cache
│   ├── Temporary Buffers
│   └── Offload
│
├── Execution
│   ├── Prefill
│   ├── Decode
│   ├── Forward Pass
│   └── Sampling
│
├── Request Scheduling
│   ├── Batch
│   ├── Dynamic Batching
│   ├── Concurrency
│   └── Queue
│
└── Performance
    ├── Latency
    ├── TTFT
    ├── Tokens per Second
    ├── Throughput
    └── OOM

Summary

After a model is downloaded to disk, it still cannot run directly. The inference engine first reads Config to create the model structure, then deserializes weights from Safetensors and similar files into Tensors, and places them into RAM or VRAM according to device and memory strategy. After input is encoded, the CPU handles program organization and scheduling while the GPU runs the main parallel Tensor computation. A large language model first Prefills to process input and build the KV Cache, then enters Decode to generate Tokens one by one. Precision, Quantization, Offload, context length, and Batch Size together determine how much memory the model needs, how fast it runs, and how many requests it can handle at once.

1. Downloading a model only saves files to disk; loading a model places weights into RAM or VRAM;
2. Weight files themselves cannot run; they must be executed jointly by an inference engine, compute framework, and hardware;
3. VRAM stores more than model weights; it must also hold KV Cache, intermediates, and temporary buffers;
4. Quantization can lower memory demand, but results depend on the quantization method, inference engine, and hardware support;
5. Offload lets small-VRAM devices run large models, but usually slows down due to data movement.