Generative AI Explained: Tokens, Embeddings, and How GPT Actually Works

Most explanations of Generative AI are either hand-wavy ("it's like a brain!") or drop you straight into matrix multiplication. Neither helps you build anything.

This post takes the middle path. After 4+ years of full-stack development — the last stretch of it shipping LLM features into production — the mental model below is the one I actually use when debugging why a prompt behaves strangely. You'll finish knowing what happens to your text at each stage, and why each stage exists.

No maths degree required. We'll use a mobile shop in Bengaluru the whole way through.

The Real-Life Analogy: The Shop Assistant

You run a mobile shop. A customer walks in:

"Anna, ₹15,000 budget alli best camera phone idya?" (Bro, do you have a good camera phone in the ₹15,000 range?)

You don't answer instantly. In about a second, you:

1. Break the sentence into pieces — ₹15,000 / budget / best / camera / phone
2. Attach meaning to each piece — "camera" is about photo quality, not a separate product
3. Note the order — "under 15,000" means something different from "over 15,000"
4. Decide what matters most — the budget is a hard constraint; "best" is vague
5. Produce an answer, one word at a time

Those five steps are, in order, tokenization, embeddings, positional encoding, self-attention, and generation. That's the whole architecture. The rest of this post is just detail on each.

The generative AI pipeline — prompt to tokens to embeddings through attention to output, with generated tokens fed back in

Step 1: Tokens — The Unit the Model Actually Sees

Models do not read characters or words. They read tokens — chunks of text somewhere between a character and a word.

You split the customer's sentence into meaningful pieces automatically. The model does the same with a fixed vocabulary learned during training.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

print(tokenizer.tokenize("Best camera phone under 15000"))
# ['best', 'camera', 'phone', 'under', '15', '##00', '##0']

Look at what happened to 15000. It became three tokens, because that exact number wasn't common enough in training to earn its own vocabulary entry. The ## prefix marks a continuation of the previous token.

This isn't trivia — it explains three things you'll hit in practice:

• Billing. APIs charge per token, not per word. A rough English guide is ~4 characters per token, but code, JSON, and non-English text are far denser.
• Context limits. A "128k context window" means 128k tokens.
• Why models miscount letters. Ask a model how many r's are in "strawberry" and it may fail — it never saw the letters, only two or three tokens.

Step 2: Embeddings — Turning Tokens Into Meaning

A token ID is just an index — camera might be token 4,382. The number carries no meaning; 4,382 isn't "near" 4,383 in any useful sense.

So each token is mapped to an embedding: a long list of numbers positioned so that related meanings sit close together.

In the shop, you know "battery backup", "long-lasting", and "charge holds well" are the same request. Embeddings are how software gets that.

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")
vectors = model.encode(["camera quality", "photo clarity", "battery life"])

sim = util.cos_sim(vectors, vectors)
print(f"camera vs photo:   {sim[0][1]:.2f}")   # ~0.74 — closely related
print(f"camera vs battery: {sim[0][2]:.2f}")   # ~0.19 — unrelated

The model was never told "camera relates to photo". It learned that from seeing them used in similar contexts across billions of sentences.

This same mechanism powers semantic search and is the foundation of RAG.

Step 3: Positional Encoding — Order Has to Be Injected

Here's a genuinely surprising fact: the core of a transformer has no built-in sense of order. It sees a set of embeddings, not a sequence.

Without help, these would be identical to the model:

• "phone under 15,000"
• "15,000 under phone"

You'd never confuse them. The model would, so position information is added into the embeddings before processing.

Step 4: Self-Attention — Deciding What Matters

This is the mechanism that made modern AI work, and it's more intuitive than its reputation suggests.

The customer says: "Phone beku under 15k, good battery, 5G, best camera."

To answer, you weigh the parts. Budget is a hard filter. "Good" attaches to "battery", not to "phone". You do this without thinking.

Self-attention is the model doing exactly that, numerically. Every token looks at every other token and produces a weight for it — how much that other token should influence its own interpretation.

Self-attention — each token weighs every other token, showing why "good" binds to "battery"

When processing battery, the model assigns high weight to good (it's the modifier) and low weight to under (irrelevant here). The weights sum to 1, so attention is a budget being allocated.

This is why LLMs handle long-range dependencies that older models couldn't. In "The phone I bought last year from the shop near the station still has great battery", battery connects to phone across fourteen words — attention makes that link directly, in one step.

Generation is a loop, not a single answer. Step through it:

Step 5: The Transformer — Everything Stacked

A transformer is these pieces stacked in layers — attention, then a feed-forward network, repeated dozens of times.

Early layers capture grammar. Deeper layers capture meaning and intent. It's the difference between a new hire who knows the words and a veteran assistant who knows what the customer actually wants.

What GPT Stands For

Training vs Inference

Two completely different activities, constantly confused:

Common Mistakes to Avoid

Mistake 1: Expecting Reliable Arithmetic

The model predicts plausible tokens; it does not calculate. It often gets sums right because it saw similar sums during training, which makes the failures unpredictable and easy to miss.

# ❌ Wrong — trusting the model to compute
"What is 18.5% of 47,320?"

# ✅ Correct — let the model write code, then execute it
"Write a Python expression for 18.5% of 47320. Return only the expression."

For anything numeric, use tool calling and run the actual computation.

Mistake 2: Thinking Bigger Context Means Better Answers

Filling a 128k window with everything you have usually makes answers worse. Attention is a finite budget spread across all tokens, and relevant instructions get diluted. Send the least context that contains the answer.

Mistake 3: Treating Temperature as a Quality Setting

Temperature controls randomness in token selection, not accuracy. Low temperature makes output more deterministic — repeatable, not more correct. A confidently wrong answer at temperature 0 is still wrong.

Mistake 4: Assuming Tokens Map to Words

Estimating cost or context usage by word count will underestimate, badly, for code, JSON, and non-English text. Count tokens with the real tokenizer:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
print(len(tok.tokenize(your_prompt)))

Hands-On Assignment — Inspect the Machinery

Three short experiments that make the abstract concrete.

pip install transformers sentence-transformers

Task 1: Find the Tokenizer's Breaking Point

Tokenize each string and count the tokens.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")

for text in [
    "hello world",
    "antidisestablishmentarianism",
    "15000",
    "ಮೊಬೈಲ್ ಫೋನ್",
    '{"price": 15000, "brand": "Samsung"}',
]:
    tokens = tok.tokenize(text)
    print(f"{len(tokens):3d} tokens | {text[:36]:38s} | {tokens[:6]}")

Which costs more tokens — the long English word, or the two Kannada words? Note the answer; it directly affects what you'd charge users.

Task 2: Map Meaning by Distance

Embed six phrases and find which pairs the model considers related.

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")
phrases = [
    "good camera phone", "great photo quality", "long battery life",
    "charge lasts all day", "fast processor", "budget under 15000",
]
vectors = model.encode(phrases)
sim = util.cos_sim(vectors, vectors)

for i in range(len(phrases)):
    for j in range(i + 1, len(phrases)):
        if sim[i][j] > 0.5:
            print(f"{sim[i][j]:.2f}  {phrases[i]}  <->  {phrases[j]}")

The pairs that surface share almost no keywords. That's meaning, not matching.

Task 3: Prove Position Matters

Send the same instruction to any LLM twice — once at the top of a long block of text, once at the bottom. Compare how faithfully each is followed. This is attention dilution, and you can measure it in five minutes.

Key Takeaways

What's Next?

• RAG Explained: How to Make an LLM Answer From Your Own Data — the practical way to give a model your private data
• Understanding Variables and Data Types in JavaScript — foundations, if you're building the app around the model