Module 1 • Lesson 6

Evaluating AI Outputs: Metrics & Judgment

Learn how to objectively measure and evaluate the quality of LLM-generated outputs.

Evaluating AI Outputs

You cannot improve what you cannot measure. Prompt engineering requires systematic evaluation of model outputs — not just "does this look right?"

Accuracy: Is the output factually correct? (Critical for knowledge tasks)

You cannot improve what you cannot measure. Prompt engineering requires systematic evaluation of model outputs — not just "does this look right?"

Accuracy: Is the output factually correct? (Critical for knowledge tasks)

Relevance: Does the output address the actual question asked?

Completeness: Are all parts of the prompt addressed?

Consistency: Does the model give the same answer for the same prompt? (Affected by temperature)

Toxicity/Safety: Does the output contain harmful, biased, or inappropriate content?

BLEU/ROUGE Scores: Automated metrics comparing generated text to reference text (mostly for NLP research)

LLM-as-Judge: Using a stronger model to evaluate a weaker model's outputs (increasingly popular)

Human Evaluation Rubrics: Structured scoring by human evaluators on defined criteria

✅ Does it answer the question asked (not a different question)?

✅ Are all facts verifiable?

✅ Is the format correct (JSON, markdown, table, etc.)?

✅ Is it the right length (not too verbose, not too terse)?

✅ Would you trust this output in a production system?

Evaluating only one output (run the same prompt 3-5 times)

Using subjective "vibes" instead of defined criteria

Not establishing a baseline before iterating on prompts

Ignoring edge cases and only testing the happy path

Read to the end of the lesson to mark it as read.