The same prompt can produce wildly different outputs depending on the generation parameters you set. Understanding these controls is fundamental to consistent prompting.
Key Concepts
- Temperature: Controls randomness. 0 = deterministic (always picks most likely token). 1 = balanced. 2 = very creative/random.
- Top-p (Nucleus Sampling): Instead of picking from all tokens, only consider tokens whose cumulative probability reaches p (e.g., 0.9 = top 90% probability mass).
- Top-k: Only consider the k most likely next tokens (e.g., top-k=50).
- Max Tokens: Hard limit on output length. The model stops generating when reached.
- Frequency/Presence Penalty: Discourages repetition by penalizing tokens that have already appeared.
Frameworks
- Task-Temperature Mapping: Factual tasks → temp 0-0.3 | Creative tasks → temp 0.7-1.0 | Code generation → temp 0-0.2
- Parameter Interaction Rule: Adjust temperature OR top-p, rarely both simultaneously
Practical Guidelines
| Task Type | Temperature | Top-p | Notes |
|---|---|---|---|
| Factual Q&A | 0 | 1.0 | Deterministic, reproducible |
| Code generation | 0–0.2 | 0.95 | Precise, minimal variation |
| Creative writing | 0.7–1.0 | 0.9 | Varied, interesting output |
| Brainstorming | 1.0–1.5 | 0.95 | Maximum diversity |
Common Mistakes
- Using high temperature for factual tasks (causes hallucination)
- Setting both temperature and top-p to extreme values simultaneously
- Forgetting to set max_tokens and getting unexpectedly long/costly responses
- Not testing different parameter combinations for your specific use case