The same prompt can produce wildly different outputs depending on the generation parameters you set. Understanding these controls is fundamental to consistent prompting.
Key Concepts
Temperature: Controls randomness. 0 = deterministic (always picks most likely token). 1 = balanced. 2 = very creative/random.Top-p (Nucleus Sampling): Instead of picking from all tokens, only consider tokens whose cumulative probability reaches p (e.g., 0.9 = top 90% probability mass).Top-k: Only consider the k most likely next tokens (e.g., top-k=50).Max Tokens: Hard limit on output length. The model stops generating when reached.Frequency/Presence Penalty: Discourages repetition by penalizing tokens that have already appeared.Frameworks
Task-Temperature Mapping: Factual tasks → temp 0-0.3 | Creative tasks → temp 0.7-1.0 | Code generation → temp 0-0.2Parameter Interaction Rule: Adjust temperature OR top-p, rarely both simultaneouslyPractical Guidelines
| Task Type | Temperature | Top-p | Notes |
|---|
| Factual Q&A | 0 | 1.0 | Deterministic, reproducible |
| Code generation | 0–0.2 | 0.95 | Precise, minimal variation |
| Creative writing | 0.7–1.0 | 0.9 | Varied, interesting output |
| Brainstorming | 1.0–1.5 | 0.95 | Maximum diversity |
Common Mistakes
Using high temperature for factual tasks (causes hallucination)Setting both temperature and top-p to extreme values simultaneouslyForgetting to set max_tokens and getting unexpectedly long/costly responsesNot testing different parameter combinations for your specific use case