A prompt is not just "a question you ask the AI." It is the entire input context that shapes the probability distribution over the model's next-token predictions.
Key Concepts
- System Prompt: Sets the model's persona, constraints, and behavioral guidelines. Processed first, highest priority.
- User Prompt: The actual task or question. This is what most people think of as "the prompt."
- Assistant Prefix: Pre-filling part of the response to steer the model (e.g., starting with "Here is the JSON:")
- Instruction Following: Modern models are fine-tuned to follow instructions — but the quality of the instruction determines the quality of the output.
Frameworks
- Prompt = Context + Instruction + Format: Every effective prompt has these three components
- Attention Budget: Tokens at the beginning and end of the context window get more attention than those in the middle ("lost in the middle" problem)
The "Lost in the Middle" Problem
Research shows that LLMs pay less attention to information placed in the middle of long contexts. Place critical information at the beginning or end of your prompt.
How Different Prompt Positions Affect Output
| Position | Attention Level | Use For |
|---|---|---|
| Start of prompt | Highest | System instructions, critical context |
| Middle of prompt | Lowest | Supporting details, examples |
| End of prompt | High | The specific question/task |
| Assistant prefix | Very High | Format control, response steering |
Common Mistakes
- Burying the actual question deep inside a long prompt
- Not using system prompts when available
- Ignoring the "lost in the middle" effect in long-context tasks
- Assuming the model reads prompts like a human (linearly with equal attention)