The Five Principles of Prompting

Reading time: 5 min

1. HOOK

Google launched Gemini in December 2023 claiming it beat ChatGPT on the MMLU benchmark. But when both models were shown the same number of examples, ChatGPT won. The difference wasn't the model — it was how the prompt was engineered.

2. WHY FIRST

Prompt engineering exists because foundation models are open-ended. Unlike fraud detection where you can check against ground truth, a chatbot can generate millions of valid responses. When everyone uses the same model, your prompts become your only competitive advantage.

3. FEYNMAN LAYER 1

A prompt is just giving instructions to a model — like telling a super-smart intern exactly what output you need, what role to play, and showing them a few examples so they nail it the first time.

4. FEYNMAN LAYER 2

Think of a prompt as the entire brief you'd give a contractor before they build your house. A prompt typically contains three things: a task description (what you want done, what role the model should play, and the output format), examples of what good looks like (called "shots"), and the actual task itself.

The number of examples matters. Zero-shot means no examples — just the instruction. Few-shot means you include a handful of examples in the prompt. For GPT-3, few-shot dramatically improved performance over zero-shot. But for GPT-4 and newer models, the gap has narrowed significantly because stronger models are better at following instructions without examples.

Here's the critical insight from the source: as models become more capable, they also become more robust. Robustness means if you change "5" to "five" or add a newline, the output doesn't swing wildly. This is why working with stronger models saves you hours of fiddling with prompt formatting.

One more thing — placement matters. Most models perform better when the task description comes first. But Llama 3 seems to prefer it at the end. There's no universal rule; you have to experiment.

5. REAL CASE STUDY

When Google launched Gemini in December 2023, their technical report claimed Gemini Ultra scored 90.04% on the MMLU benchmark, beating GPT-4. But there was a catch. Google used a technique called CoT@32 — they showed Gemini 32 examples in the prompt. GPT-4 was shown only 5 examples. When both were evaluated with 5-shot prompts, ChatGPT performed better. This wasn't a model quality story — it was a prompt engineering story. Google's evaluation used a prompt construction technique that gave their model an unfair advantage in the benchmark. The lesson for PMs: when comparing models or evaluating vendor claims, always check the prompt configuration. The same model can look like a champion or an also-ran depending on how many examples you pack into the prompt. Your evaluation methodology is part of your product strategy.

6. MIND PALACE ANCHOR

Picture a chef who can cook anything but has no pantry of their own — every ingredient, every recipe, every plating instruction has to be handed to them on a single sheet of paper. That sheet is your prompt. A crumpled, vague note gets you a random meal. A precise, well-structured brief gets you a Michelin plate.

7. CONTRAST SECTION

Beginners think prompt engineering is about writing clever one-liners. Experts know it's about systematically structuring task descriptions, calibrating the number of examples based on model capability, and controlling output through format specifications.

8. DOMAIN BRIDGE

This is exactly like writing product requirement documents for engineering teams — the difference is your engineer has no prior context about your product, so every assumption must be made explicit in the brief itself.

9. KEY TAKEAWAYS

Start every prompt by defining the role and output format before stating the task — this alone can shift your output quality dramatically. Test your prompts with small perturbations like changing numbers to words or adding line breaks to measure how robust your application really is. When evaluating models for your product, standardize the number of examples in your prompts — otherwise you're measuring prompt engineering skill, not model capability.