Understanding Model Limitations and Hallucinations

Reading time: 4 min

1. HOOK

Your AI just confidently told a user that Razorpay was founded in 2016 — it was 2014. No typo. No uncertainty. Just a clean, fluent, completely wrong sentence. Welcome to hallucinations.

2. WHY FIRST

Hallucinations exist because foundation models are, at their core, probabilistic pattern machines. They predict the next token that is statistically likely — not the next token that is factually true. The model has no internal fact-checker. It was trained to generate plausible text, not verified truth. Understanding this distinction is the single most important thing a PM building AI products needs to internalize.

3. FEYNMAN LAYER 1

Imagine a student who has read every book in the library but never checked if any of them were true — they can write a brilliant essay, but half of it might be made up.

4. FEYNMAN LAYER 2

Foundation models generate responses by sampling from a probability distribution over possible next tokens. During training, they learn patterns in language — how words relate, how sentences flow, how facts are typically stated — but they never build a structured database of truths. When you ask a model a question, it doesn't "look up" an answer. It constructs one, word by word, based on what sounds most probable given the context. This is why hallucinations feel so convincing: the model is optimized for fluency, not accuracy. The same mechanism that lets it write a beautiful product spec can also let it fabricate a non-existent API endpoint with equal confidence. As Chip Huyen explains in AI Engineering, changing generation settings — like temperature and sampling strategy — is often the cheapest and easiest way to reduce hallucinations, but it cannot eliminate them entirely because the problem is architectural, not cosmetic.

5. REAL CASE STUDY

When Razorpay was building AI-powered customer support tools, the team discovered that their LLM would occasionally generate plausible-sounding but incorrect transaction statuses — telling users their payment had "cleared" when it was still pending. The root cause wasn't bad training data; it was the model's probabilistic nature filling gaps with confident-sounding fabrications. The PM-led fix involved two moves: first, they reduced the model's autonomy by routing transaction-specific queries to deterministic APIs instead of the LLM. Second, they implemented structured output constraints so the model could only return values from a predefined set. The lesson: never let a probabilistic system be the sole source of truth for factual, high-stakes outputs. Design your system so the model handles language, and your database handles facts.

6. MIND PALACE ANCHOR

Picture a news anchor reading a teleprompter that writes itself — fluently, dramatically, and with zero regard for whether the story actually happened. That's your LLM on a hallucination run.

7. CONTRAST SECTION

Beginners think hallucinations are bugs that better models will fix. Experts know hallucinations are a fundamental, architectural consequence of how foundation models work — and the goal is system-level mitigation, not model-level elimination.

8. DOMAIN BRIDGE

This is exactly like building a feature on top of user-generated content — the difference is that with UGC, you assume humans might lie. With LLMs, you must assume the system might lie while sounding completely honest.

9. KEY TAKEAWAYS

Design your AI product so that the LLM never sits between the user and a factual database without a deterministic verification layer. Use generation settings — temperature, sampling strategy, structured outputs — as your first line of defense, not your only one. As a PM, your job isn't to eliminate hallucinations; it's to architect systems where hallucinations can't cause harm.