Layer 3: How Does a Model Turn Input into Output?

Core question: After a user enters a sentence, what happens inside the model?

Key concepts: Prompt, Chat Template, Tokenizer, Token, Embedding, Position, Attention, Forward Pass, Logits, Sampling, Context Window, Autoregressive Generation.

Pipeline

Taking the user input "Introduce LLMs" as an example, the full model processing flow is as follows:

Two stages:

1. Input processing: convert human input into numbers the model can compute on;
2. Autoregressive generation: predict one Token at a time, gradually forming the final output.

1. Prompt

A prompt is the input the user provides to the model. It is the content handed to the model for processing, but in chat models what the model actually receives is usually more than just the user's single sentence. For example:

System: You are a professional technical assistant.
User: Introduce LLMs.
Assistant:

Chat Template: messages stored in a chat application may be structured data:

[
  {
    "role": "system",
    "content": "You are a professional technical assistant."
  },
  {
    "role": "user",
    "content": "Introduce LLMs"
  }
]

2. Chat Template

A Chat Template concatenates them into an input format the model saw during training. Conceptually it may look like:

<|im_start|>system
You are a professional technical assistant.
<|im_end|>
<|im_start|>user
Introduce LLMs
<|im_end|>
<|im_start|>assistant

• A Chat Template does not understand the question or generate the answer; it organizes system, user, assistant, and other messages as the model requires.

• Different models may use different special Tokens and message formats. If the format is incorrect, the model may still run, but answer quality, role recognition, and tool-calling ability may suffer.

3. Tokenizer: Converting Text into Tokens

A model cannot process text directly; it can only process numbers. The Tokenizer's job is: following rules predefined for the model, split text into Tokens and convert each Token into an integer ID. For example:

Introduce LLMs
 
May be split into:
 
["Introduce", " LLMs"]
 
Then converted to:
 
[1045, 6821, 2379, 9134]

Token

The basic unit of text processing for the model.

A Token may be:

• A Chinese character;
• A word;
• Part of a word;
• Punctuation;
• A space;
• A newline;
• A special control marker.

English example: unbelievable

May be split into: ["un", "believ", "able"]

Different Tokenizers may produce different splits. A Token is not equal to a Chinese character, nor to a word; it is a text unit defined by the current model's Tokenizer rules.

Token ID

What the model actually receives is not the string "Introduce", but an integer ID in the vocabulary: Token → Token ID.

A Tokenizer usually depends on a Vocabulary:

Vocabulary
├── Token A → ID 0
├── Token B → ID 1
├── Token C → ID 2
└── ...
 
Taking Qwen3.8 vocab.json as an example, you can see:
 
"@": 31,
"A": 32,
"B": 33,
"C": 34,
"D": 35,
"E": 36,
"F": 37,
"G": 38,
"H": 39,
"I": 40,
"J": 41,
"K": 42,
"L": 43,
"M": 44,
"N": 45,
"O": 46,
"P": 47,
"Q": 48,

Looking at these files in the Qwen repository makes their roles clear:

• tokenizer.json: the complete Tokenizer definition;
• vocab.json: mapping between Tokens and IDs;
• merges.txt: BPE Token merge rules;
• tokenizer_config.json: Tokenizer type, special Tokens, and other configuration.

4. Embedding: Converting Token IDs into Vectors

A Token ID is only a number; it has no computable semantic relationship by itself. For example: LLM -> 9134, Introduce -> 2841.

You would not conclude that LLM is somehow "greater than" Introduce just because 9134 > 2841.

The model uses an Embedding Table to map each Token ID to a set of floating-point numbers:

Token ID
→ Look up Embedding Table
→ Embedding Vector
 
"LLM"
→ Token ID: 9134
→ [0.18, -0.42, 0.07, 0.91, ...]

This set of numbers is called:

• Embedding;
• Embedding Vector;
• Token Embedding;
• Embedding vector.

They are not hand-written; they are learned through training.

The Tokenizer converts text into IDs; Embedding then converts IDs into vectors the model can perform math on.

Position

Token Embedding alone is not enough, because the following two sentences contain similar Tokens but mean different things:

The cat chased the dog
The dog chased the cat

The model also needs to know each Token's position in the sequence, so it adds position information.

Vector representation of a Token
=
Token Embedding
+
Position Information

Modern Transformers may use:

• Positional Encoding;
• Positional Embedding;
• RoPE (Rotary Position Embedding).

Embedding represents "what Token this is"; position information represents "where it appears."

5. Transformer Computes with Context

After Embedding, the input has become a set of vectors. These vectors then pass through multiple Transformer Blocks in sequence.

Embedding
→ Transformer Block 1
→ Transformer Block 2
→ ...
→ Transformer Block N
→ Final hidden state

Each Transformer Block typically contains:

Transformer Block
├── Attention
├── Feed-Forward Network / MLP
├── Normalization
└── Residual Connection

Attention: Using Context to Judge What Matters

Attention helps the model, when processing the current position, refer to other Tokens in the context. For example: Xiaoming handed the book to Xiaoli because he had finished reading it.

When the model processes "he," it needs earlier context to judge who "he" might refer to.

• Attention assigns different degrees of focus to different Tokens in the context based on what the current computation needs.

• Attention is a computation mechanism that combines vector information from different Tokens with weighted sums.

Forward Pass: One Complete Forward Computation

The process of Token vectors flowing from the first layer of the model to the last is called: Forward Pass. During inference:

Input Tokens
→ Embedding
→ Multiple Transformer layers
→ Output computation results

Both training and inference run a Forward Pass, but what happens afterward differs:

6. Logits: Candidate Scores from the Model

After Transformer computation, the model does not output text directly; instead it computes a score for every Token in the vocabulary. These raw scores are called: Logits.

Suppose the vocabulary has only a few candidate Tokens; the model might get:

Higher Logit means that Token is more likely to be the next Token given the current context. Softmax then converts Logits into a probability distribution:

Logits → Softmax → probability distribution over the next Token

Logits are neither final text nor probabilities; they are raw scores the model computes for all candidate Tokens.

7. Sampling: Choosing the Next Token from Candidates

After obtaining a probability distribution, you still need to decide which Token to pick. This process is usually called Decoding or Sampling.

Intuitive understanding of Temperature

• Lower Temperature:

Probabilities are more concentrated;
Output is usually more stable;
More likely to pick high-probability Tokens.

• Higher Temperature:

Probabilities are flatter;
The candidate range is wider;
Output may be more diverse, and also less stable.

This also explains why you often see settings like "Temperature: 0.7" during inference. The usual guidance is: if you need diversity, set a higher Temperature; if you need stability, set a lower Temperature.

8. Autoregressive Generation: Predicting One Token at a Time

Suppose the input is: The capital of France is

First model computation: The capital of France is → next Token: "Paris"

Append it to the context: The capital of France is Paris

Model computes again: The capital of France is Paris → next Token: "."

Continue appending: The capital of France is Paris.

Until it stops.

Generating subsequent content step by step from what already exists is called: Autoregressive Generation.

Therefore, large language model generation can be summarized as:

Given existing Tokens
→ Predict the next Token
→ Add the new Token back to the input
→ Predict the next Token again
→ Repeat

This is also why model output can usually be streamed Token by Token.

9. When Does Generation Stop?

The autoregressive loop does not run forever. Common stop conditions include:

• An end Token is generated, such as EOS;
• The maximum number of generated Tokens is reached;
• A preset stop string is encountered;
• The application terminates generation;
• A tool call or structured output is complete.

Where:

EOS stands for End of Sequence, meaning the sequence has ended.

After the model selects EOS, the inference program usually stops generating further.

10. Decode: Converting Tokens Back to Text

What the model generates is still a series of Token IDs: [1384, 6752, 1773, 9]

The Tokenizer finally runs Decode: Token ID → Token → concatenate and restore → human-readable text

The Tokenizer actually handles conversion in both directions:

• Encode: text → Token ID
• Decode: Token ID → text

11. Context Window: How Much Context the Model Can Handle at Once

Context Window means the range of Tokens the model can process in one generation process.

Context usually includes more than the current user question; it may include:

Context
├── System Prompt
├── Conversation History
├── Current user input
├── RAG retrieved content
├── Tool return results
└── Tokens the model has already generated

Input Tokens + Output Tokens ≤ available context range

A model that supports 128K Context means it can handle about 128K Tokens in one context, not 128K Chinese characters or words.

When context exceeds the limit, applications usually need to:

• Truncate earlier content;
• Compress or summarize history messages;
• Reduce retrieved content;
• Limit maximum output length.

12. Prefill and Decode: Two Stages of Inference

The model first processes existing input in one go:

System Prompt
+ Conversation history
+ Current question
→ Build context representation
→ Predict the first output Token

This is usually called Prefill.

Then the model generates new Tokens one by one:

Generate Token 1
→ Generate Token 2
→ Generate Token 3
→ ...

This is usually called Decode.

13. How Do Multimodal Models Handle Images and Video?

Text is converted via a Tokenizer, while images, audio, and video are usually handled by the corresponding Processor or Encoder:

Text
→ Tokenizer
→ Text Tokens
 
Image
→ Image Processor / Vision Encoder
→ Visual Tokens
 
Audio
→ Audio Processor / Audio Encoder
→ Audio Tokens or features
 
Video
→ Video Processor / Vision Encoder
→ Video Tokens

Representations from different modalities are fed into the model for joint computation.

For example, in the Qwen repository:

• preprocessor_config.json;
• video_preprocessor_config.json;
• tokenizer.json;

each serve preprocessing for different input types.

Multimodal inputs do not necessarily all use the same Tokenizer as text; different models use different Encoder, Processor, and feature-representation schemes.

Concepts

Input to Output
│
├── Input Construction
│   ├── User Input
│   ├── System Prompt
│   ├── Conversation History
│   └── Chat Template
│
├── Input Encoding
│   ├── Tokenizer
│   ├── Token
│   ├── Token ID
│   ├── Embedding
│   └── Position Information
│
├── Model Computation
│   ├── Forward Pass
│   ├── Transformer Block
│   ├── Attention
│   └── Hidden State
│
├── Next-token Prediction
│   ├── Logits
│   ├── Softmax
│   ├── Probability Distribution
│   └── Sampling / Decoding
│
├── Autoregressive Generation
│   ├── Append Token
│   ├── Repeat
│   ├── Stop Condition
│   └── Tokenizer Decode
│
└── Input Limits
    ├── Context Window
    ├── Input Tokens
    ├── Output Tokens
    └── Max New Tokens

A large language model cannot read text directly. Input is first organized by a Chat Template, then converted by a Tokenizer into Token IDs, and then by Embedding into vectors. After multilayer Transformer and Attention computation, the model produces a set of Logits for the next Token; the inference program then picks a Token according to the sampling strategy, appends it to the context, and repeats, until the Tokenizer finally decodes human-readable text.

• What the model processes is not text, but numbers and vectors corresponding to Tokens;
• The model's core task is to predict the next Token given context;
• A complete answer is generated step by step through the "predict—append—predict again" loop.