Large Language Models (LLMs) like GPT-4, Gemini, and Claude are built on the transformer architecture, introduced in the landmark 2017 paper "Attention Is All You Need."
Key Concepts
- Tokenization: Text is split into tokens (subwords), not characters. "unhappiness" → ["un", "happiness"]. Token count directly affects cost and context window usage.
- Attention Mechanism: The model weighs how much each token should "attend to" every other token. This is what gives LLMs their ability to understand context across long passages.
- Autoregressive Generation: LLMs generate text one token at a time, each token conditioned on all previous tokens. They predict "what comes next."
- Parameters: The "knowledge" is stored in billions of numerical weights learned during training.
Frameworks
- Pre-training → Fine-tuning → RLHF Pipeline: Raw text → supervised examples → human preference alignment
- Scaling Laws: Performance improves predictably with more data, more parameters, and more compute (Chinchilla scaling)
Why This Matters for Prompting
| Concept | Prompting Implication |
|---|---|
| Token-based processing | Be concise — every token costs money and context space |
| Autoregressive generation | The model cannot "go back" — front-load important context |
| Attention mechanism | Structure matters — well-organized prompts get better attention |
| Training data cutoff | The model does not know recent events unless you provide them |
Common Misconceptions
- LLMs do NOT "understand" — they predict statistically likely continuations
- They do NOT have memory across conversations (unless context is re-injected)
- Larger models are NOT always better for every task
- Temperature controls randomness, NOT creativity