Understanding LLM Architecture and Tokenization

Reading time: 5 min

1. HOOK

GPT-4 needs 72 tokens to express in Burmese what English says in 7. If you've ever wondered why your AI feature works great in English but feels broken for half your Indian users, the answer starts with how LLMs chop up language — and it's not nearly as fair as you'd hope.

2. WHY FIRST

Tokenization exists because neural networks can't process raw text — they process numbers. Every LLM needs a system to convert human language into discrete numerical units it can compute over. The way you design that system determines which languages the model handles efficiently, how much it costs to serve users, and where hidden quality gaps silently creep in.

3. FEYNMAN LAYER 1

Imagine you're packing a Swiggy order — if the container sizes are designed for small dosa orders, you'll need ten containers to pack a single biryani, wasting time, money, and space. Tokenization is the container system for language, and it was built mostly for English.

4. FEYNMAN LAYER 2

When an LLM receives your prompt, the first thing it does is break your text into tokens — chunks of characters that the model has a predefined vocabulary for. English is efficient: one word is often one or two tokens. But for languages like Hindi or Burmese, the same meaning can require 5–10x more tokens because those languages were severely under-represented in the training data (Common Crawl, the dominant pre-training dataset, is overwhelmingly English). This isn't just a linguistic curiosity — it has direct product consequences. More tokens means higher inference cost, higher latency, and often worse model quality because the model has fewer "thinking units" left within its context window to actually reason about the request. Think of it like a Razorpay payment flow where the transaction fee structure was designed for UPI but suddenly breaks when applied to international cards — the underlying unit economics are misaligned with the user's reality.

5. REAL CASE STUDY

When benchmarking GPT-4 on the MASSIVE dataset (one million short texts across 52 languages), researcher Yennie Jun found that the median token length in English was 7, while Hindi required 32 tokens and Burmese required 72 — ten times longer — to convey identical meaning. This directly impacts any Indian product team building multilingual AI features. A PhonePe or CRED team deploying an LLM-based assistant for Hindi-speaking users isn't just facing a translation problem — they're facing a token economics problem. Each Hindi query costs significantly more to process, responds slower, and consumes context budget faster, which means fewer conversation turns before the model "forgets" the conversation history. The PM lesson: when scoping AI features for non-English markets, tokenization efficiency must be a first-class evaluation criterion in model selection, not an afterthought.

6. MIND PALACE ANCHOR

Picture a conveyor belt at a Zomato cloud kitchen where English ingredients arrive pre-chopped into perfect cubes, but Hindi ingredients arrive as whole raw vegetables that need ten times more cutting — the kitchen (model) gets overwhelmed, orders slow down, and the dish quality drops.

7. CONTRAST SECTION

Beginners think tokenization is just "splitting text into words." Experts know it's a design decision baked into the model's training data that creates systematic cost and quality inequalities across languages — and it directly shapes which markets your AI product can serve profitably.

8. DOMAIN BRIDGE

This is exactly like how UPI's success in India was built on optimizing for low-cost, high-volume micro-transactions — the difference is that tokenization was optimized for English, so every other language pays a hidden premium that most PMs never see on the invoice.

9. KEY TAKEAWAYS

When evaluating an LLM for a multilingual product, always benchmark token efficiency across your target languages — it will change your unit economics calculation more than model accuracy scores will. If your context window is 8,192 tokens and each Hindi query eats 5x more budget than English, you've silently capped your conversation depth for your largest user base. Tokenization isn't a solved problem — it's a product constraint that belongs in your PRD alongside latency and cost targets.