Large Language Models (LLMs) like GPT-4, Gemini, and Claude are built on the transformer architecture, introduced in the landmark 2017 paper "Attention Is All You Need."
Key Concepts
Tokenization: Text is split into tokens (subwords), not characters. "unhappiness" → ["un", "happiness"]. Token count directly affects cost and context window usage.Large Language Models (LLMs) like GPT-4, Gemini, and Claude are built on the transformer architecture, introduced in the landmark 2017 paper "Attention Is All You Need."
Key Concepts
Tokenization: Text is split into tokens (subwords), not characters. "unhappiness" → ["un", "happiness"]. Token count directly affects cost and context window usage.Attention Mechanism: The model weighs how much each token should "attend to" every other token. This is what gives LLMs their ability to understand context across long passages.Autoregressive Generation: LLMs generate text one token at a time, each token conditioned on all previous tokens. They predict "what comes next."Parameters: The "knowledge" is stored in billions of numerical weights learned during training.Frameworks
Pre-training → Fine-tuning → RLHF Pipeline: Raw text → supervised examples → human preference alignmentScaling Laws: Performance improves predictably with more data, more parameters, and more compute (Chinchilla scaling)Why This Matters for Prompting
| Concept | Prompting Implication |
|---|
| Token-based processing | Be concise — every token costs money and context space |
| Autoregressive generation | The model cannot "go back" — front-load important context |
| Attention mechanism | Structure matters — well-organized prompts get better attention |
| Training data cutoff | The model does not know recent events unless you provide them |
Common Misconceptions
LLMs do NOT "understand" — they predict statistically likely continuationsThey do NOT have memory across conversations (unless context is re-injected)Larger models are NOT always better for every taskTemperature controls randomness, NOT creativity