Lesson 1. Inside an LLM: tokens, attention and sampling

Level: Foundations · Time: 3 to 4 hours · Output: notebook l01_probe.ipynb

Objectives

Explain how an LLM produces text, from a string of characters to a probability distribution over its vocabulary.
Count tokens precisely and predict the cost of a Vietnamese passage versus an English one.
Control temperature and top-p, and understand why the same prompt gives different results.
Recognise three structural limits: the context window, knowledge frozen at training time, and hallucination.

Core mechanics

Tokens. A model does not read letters; it reads pieces (tokens) produced by a tokeniser such as BPE. Common English words are usually a single token, while accented Vietnamese tends to be split more finely, so the same idea can cost 1.5 to 3 times as many tokens. In practice, price, latency and context limits are all counted in tokens, not words.

Next-token prediction. At each step the Transformer takes every token so far and produces a score (logit) for every token in the vocabulary. Softmax turns these into probabilities, one token is chosen and appended, and the loop repeats. All of the model's apparent intelligence lies in having learnt this distribution from an enormous amount of text.

Attention. In every layer each token produces three vectors: query, key and value. The current token compares its query with the keys of all earlier tokens to decide where to "look", then blends their values by those weights. The cost of attention grows with the square of sequence length, which is why long contexts are both expensive and slow. The KV cache stores keys and values already computed so they need not be recomputed for each new token.

Sampling. Temperature divides the logits before softmax: below 1 sharpens the distribution (stable, repetitive), above 1 flattens it (varied, prone to drift). Top-p keeps the smallest set of tokens whose probabilities sum to p and drops the tail. Temperature 0 almost always picks the most likely token, yet results can still differ slightly because of parallel arithmetic on GPUs.

Three training stages. Pre-training on raw text yields a "text completion" model. Instruction tuning teaches it to answer in conversational form. Learning from feedback (RLHF, or variants such as training against a written set of principles) shapes helpfulness and safety.

Trade-offs and common failures

Symptom	Real cause	Remedy
Miscounts the letters in a word	The model sees tokens, not letters	Ask it to spell the word out first, or use code
Invents book titles or figures	It optimises for plausible text, not true text	Supply documents (Lesson 4), require citations, allow "I don't know"
Forgets instructions at the top of a long prompt	Material in the middle of long contexts is easily missed	Put key instructions first and repeat them at the end; trim surplus context
Different output every time	Random sampling	Lower temperature, constrain output with a schema, run repeatedly and measure

Lab: look inside a small model

Use a small open model (for example Qwen/Qwen2.5-0.5B or gpt2) on Colab to see the probabilities directly.

Install: pip install transformers torch.
Count tokens for three sentences with the same meaning: Vietnamese, English, and Vietnamese without diacritics. Record the token-to-word ratio.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
name = "Qwen/Qwen2.5-0.5B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
# samples.txt: one line each for the Vietnamese sentence, its English equivalent,
# and the same Vietnamese sentence with diacritics removed
for s in open("samples.txt", encoding="utf-8").read().splitlines():
    ids = tok(s)["input_ids"]
    print(len(ids), [tok.decode([i]) for i in ids])

Print the ten most likely next tokens after "The capital of Vietnam is":

ids = tok("The capital of Vietnam is", return_tensors="pt")
with torch.no_grad():
    logits = model(**ids).logits[0, -1]
for T in (0.5, 1.0, 1.5):
    p = torch.softmax(logits / T, dim=-1)
    top = torch.topk(p, 10)
    print(T, [(tok.decode([i]), round(v.item(), 3)) for v, i in zip(top.values, top.indices)])

Generate 20 times at each temperature (0.2, 0.8, 1.4) for the prompt "Name one Vietnamese dish:" and count the distinct answers.
Finish the notebook with a note: when you want a low temperature and when a high one.

Challenges

Easy (10 points): token cost calculator. Write estimate_cost(text, price_in, price_out, out_ratio) using a real tokeniser. Acceptance: run it on five passages of your choice, print a table of tokens and cost, with zero discrepancy against a direct token count.

Medium (20 points): a certainty map. For 20 knowledge questions (10 with definite answers, 10 about obscure or invented events), compute the entropy of the distribution over the first answer token. Acceptance: a chart comparing the two groups and a one-sentence conclusion backed by numbers; name at least one case where entropy was low yet the model was wrong.

Boss (40 points): the Vietnamese token tax. Take 200 Vietnamese-English sentence pairs (your own, or from an openly licensed dataset). Compare token counts across at least three tokenisers. Acceptance: a table of mean and 90th-percentile ratios, an explanation of why one tokeniser is "cheaper" for Vietnamese, and a prompt-writing rule that cuts tokens by at least 15% without changing meaning, with before-and-after measurements.

Self-check quiz

Why are APIs priced per token rather than per word?
Does temperature 0 guarantee identical output every time?
What does the KV cache save?
How would a pre-trained-only model respond to "List three fruits"?
Why do key instructions at the end of a very long prompt often work better than in the middle?

Answers

The model processes and generates one token at a time; compute scales with tokens, and "word" has no common definition across languages.
Not entirely. It nearly always picks the top token, but floating-point error and GPU batching can cause small differences.
Recomputing keys and values for tokens already processed, at the cost of GPU memory that grows with context length.
It might continue the text instead of answering, for instance by adding another similar instruction, because it has only learnt to complete text.
Material near the end sits right next to where the answer is generated; research shows models often miss information in the middle of long contexts.