You cannot improve what you cannot measure. Prompt engineering requires systematic evaluation of model outputs — not just "does this look right?"
Key Concepts
- Accuracy: Is the output factually correct? (Critical for knowledge tasks)
- Relevance: Does the output address the actual question asked?
- Completeness: Are all parts of the prompt addressed?
- Consistency: Does the model give the same answer for the same prompt? (Affected by temperature)
- Toxicity/Safety: Does the output contain harmful, biased, or inappropriate content?
Frameworks
- BLEU/ROUGE Scores: Automated metrics comparing generated text to reference text (mostly for NLP research)
- LLM-as-Judge: Using a stronger model to evaluate a weaker model's outputs (increasingly popular)
- Human Evaluation Rubrics: Structured scoring by human evaluators on defined criteria
Practical Evaluation Checklist
- ✅ Does it answer the question asked (not a different question)?
- ✅ Are all facts verifiable?
- ✅ Is the format correct (JSON, markdown, table, etc.)?
- ✅ Is it the right length (not too verbose, not too terse)?
- ✅ Would you trust this output in a production system?
Common Mistakes
- Evaluating only one output (run the same prompt 3-5 times)
- Using subjective "vibes" instead of defined criteria
- Not establishing a baseline before iterating on prompts
- Ignoring edge cases and only testing the happy path