Every interaction with an LLM is measured in tokens — the fundamental unit of processing. Understanding tokens is essential for cost management and effective prompting.
Key Concepts
- Token: A subword unit, roughly ¾ of a word in English. "ChatGPT is great" ≈ 4 tokens.
- Context Window: The maximum number of tokens the model can process in a single request (input + output combined). GPT-4 Turbo: 128K tokens. Claude 3: 200K tokens.
- Input vs Output Tokens: Input tokens (your prompt) are cheaper than output tokens (model response). Optimize accordingly.
- Token Budgeting: Plan your prompt + expected response to fit within the context window.
Frameworks
- Cost Estimation Formula: Total Cost = (Input Tokens × Input Price) + (Output Tokens × Output Price)
- Context Window Management: System prompt + conversation history + user query + expected response ≤ context window
Practical Guidelines
- Use a tokenizer tool (tiktoken for OpenAI, etc.) to estimate token counts before sending expensive prompts
- For long documents, summarize or chunk rather than stuffing the entire context window
- Output tokens are 2-4x more expensive than input tokens — ask for concise responses when possible
- Different models have different tokenizers — the same text may use different token counts
Common Mistakes
- Ignoring token limits and getting truncated responses
- Paying for unnecessarily verbose outputs
- Not accounting for system prompts in the token budget
- Assuming all models tokenize text the same way