Six ideas explain most of what you will observe in production:

1. Tokens determine what the model actually reads and writes.

2. Context and attention determine how much it can use at once, and what that costs.

3. Sampling determines how each next token gets picked.

4. Statelessness explains why your application, not the model, owns memory.

5. Three training stages explain how a text predictor became an assistant.

6. Hallucination follows from the gap between sounding plausible and being true.

The shape of a single request:

Application builds the prompt
        ↓
Tokenizer turns text into token IDs
        ↓
Model processes the tokens with attention
        ↓
Model outputs next-token probabilities
        ↓
Sampling picks one token  ──┐
        ↓                   │ repeat until done
Application validates  ←────┘
and stores the answer

Notice how little of that the model does. It handles the middle. Everything else retrieval, tool calls, storage, validation, and deciding what the user finally sees is your architecture. That division is the single most useful thing to internalize.

1. Tokens and tokenization

The model never sees your text

Before anything reaches the neural network, a tokenizer splits the text into units called tokens and maps each one to a number, a token ID. Those numbers are the model's input. The original characters are gone.

A token is not a word. It can be a whole word, a fragment of a word, a space plus the following word, a piece of punctuation, part of a number, part of an emoji, or a raw byte sequence used as a last resort.

Text:   The engineer redesigned the tokenizer.
Tokens: ["The", " engineer", " redesigned", " the", " token", "izer", "."]

Three things to notice there. "engineer" happened to be a single token. "tokenizer" needed two. And the spaces are attached to the front of the following token, not sitting on their own.

How the tokenizer picks the pieces

A tokenizer has a fixed vocabulary of a few tens of thousands of reusable text fragments. When it encounters text, it tries to cover that text using the largest available pieces. Sequences that appeared constantly in its training data earned their own single entry; rare sequences have to be assembled from smaller parts.

unhappiness  →  ["un", "happi", "ness"]

The common families are Byte Pair Encoding (BPE), WordPiece, and Unigram, usually with byte-level fallback so no input can ever be unrepresentable. You will almost never implement one. What matters is the consequence: common words are cheap, rare words are expensive, and a technical term like electroencephalography might cost four tokens where the costs one.

For a given tokenizer and a given string, the output is deterministic. Across model families it is not. The same paragraph can have meaningfully different token counts on two different providers.

Tokens are the unit of billing

Almost every API charges per token, not per word or character, split into two buckets:

• Input tokens system instructions, conversation history, retrieved documents, tool results, and the current message.

• Output tokens everything generated in response.

total cost = input tokens  × input rate
           + output tokens × output rate

Output usually costs more per token, because it is produced one token at a time and cannot be batched the way input can. Many providers add further categories: cached input, uncached input, reasoning tokens, audio, images.

A single realistic request:

The interesting number is 3,000. The user typed a hundred-token question, and 97% of the input charge came from re-sending things that were already sent before. This is the central economic fact of chat applications, and it gets worse linearly with conversation length.

Tokens are also the unit of latency

Input tokens are processed in one parallel pass, a stage called prefill. Output tokens are then produced strictly one after another, a stage called decoding.

That asymmetry is why a 4,000-token answer can take twenty times longer than a 200-token answer while a 4,000-token prompt does not take twenty times longer than a 200-token prompt. If your product feels slow, the output length is usually the first thing to look at, and capping max_tokens is usually the cheapest fix available.

Why models miscount letters

Ask a person how many rs are in "strawberry" and they scan the characters. The model does not start with characters. It starts with something like:

To answer, it has to reconstruct a character-level view from token-level pieces. That is an indirect operation, and indirect operations fail. Models can often get there by spelling the word out deliberately or by writing code, but their default representation simply is not an array of letters.

The engineering answer is not a better prompt:

"strawberry".count("r")   # 3

Use the model for language. Use deterministic code for exact counting, arithmetic, parsing, and validation. This principle will reappear in section 6, because it is also the main defense against hallucination.

Tokens are not equally fair across languages

Token efficiency depends on how much text in a language existed in the tokenizer's training data, which sequences therefore became single vocabulary entries, the writing system, whether words are space-separated, the language's morphology, and the vocabulary size.

So a tokenizer may compress an English sentence into four tokens and split an equally simple sentence in another language into ten. This is not a statement about the languages. It is a statement about the tokenizer's coverage. But the practical consequences are real and land entirely on your users:

• higher cost for the same information

• higher latency and more generation steps

• less usable content fitting inside the same context window

Modern multilingual tokenizers narrow the gap. They do not close it. If your product ships in several languages, measure each one rather than extrapolating from English.

A few more things that quietly cost tokens

The rule of thumb that one token is about four English characters, or about three-quarters of a word, is fine for capacity planning and wrong often enough that you should never bill, truncate, or gate on it. When the number matters, run the real tokenizer for the exact model.

2. Context, attention, and the two kinds of caching

The context window is a size limit, not a place

The context window is the maximum amount of tokenized information the model can work with in a single request. What you actually send is the context; the window is how much the container holds.

It helps to think of a suitcase. What you pack is the contents; the suitcase's capacity is the window. In casual speech people say "my context window is full" the way they say "my suitcase is full," and that is fine, as long as you keep two facts straight:

The output shares the budget. If the window is 200k tokens and you send 195k, you have about 5k left for the answer. Some APIs also impose a separate output cap on top of that.

Everything competes for the same space. Instructions, history, retrieved documents, tool results, images, and the reply are all drawing on one pool. A giant pasted document does not just cost money, it crowds out the answer.

When a request will not fit, your options are to shorten history, summarize older turns, retrieve fewer passages, reduce the planned output, or move to a model with a larger window. A larger window buys capacity. It does not buy comprehension, which is a distinction section on lost-in-the-middle will make painfully concrete.

Attention: how tokens consult each other

Attention is the mechanism that lets one position in the sequence look at other positions and decide which of them matter. Consider:

The animal did not cross the road because it was tired.

Resolving it requires connecting it back to animal rather than road. Attention is what makes that connection available.

Inside an attention layer, every token produces three vectors:

• Query (Q) what am I looking for?

• Key (K) what kind of information do I contain?

• Value (V) what can I actually contribute?

Each token's Query is compared against every Key. The resulting weights decide how much of each Value flows into the next representation. Real models do this across many parallel attention heads and repeat it through dozens of layers, which is how the simple comparison above turns into something that tracks syntax, reference, and meaning.

Why attention gets expensive: the handshake problem

In standard full self-attention, every token can interact with every other token. Picture a room where everyone must shake hands with everyone else, including themselves. Each token is a person; each handshake is one relationship the model evaluates.

    4 people  →      4 ×     4 =         16 handshakes
1,000 people  →  1,000 × 1,000 =  1,000,000 handshakes
2,000 people  →  2,000 × 2,000 =  4,000,000 handshakes

Double the room and the handshakes quadruple, because each new arrival must greet everyone already there. That is quadratic scaling, written n². It is the reason long contexts are structurally hard rather than just "bigger."

Optimizations soften this. FlashAttention performs the same mathematics with far less memory traffic. Sparse attention and sliding-window attention let each token consult only a subset of the others instead of all of them. Architectural changes sidestep full attention entirely in some models. So real-world cost is not cleanly proportional to n². The direction of the problem is unchanged: longer contexts need more compute and more memory.

A clarification worth making: you are not billed for handshakes

It is easy to slide from "long prompts do more work" into "you pay for that work." You do not. Your invoice counts input and output tokens. Attention never appears on it.

Attention still reaches you, in three indirect ways:

1. It sets the price per token. The provider pays for the hardware doing all that internal work and prices tokens to cover it on average. A restaurant does not bill you for the kitchen's gas; the menu price already includes it.

2. Some providers charge a premium on very long prompts. Because long inputs are disproportionately expensive to serve, a higher per-token rate above a length threshold is a real pricing pattern. This is where the quadratic problem shows up directly on the bill.

3. It costs you time and ceiling. Even at a flat price, a very long prompt delays the first token. And memory growth is a major reason the context window has a maximum at all.

So: you pay for tokens, and attention is a large part of why tokens cost what they do.

Beyond the invoice, large prompts raise time-to-first-token, GPU memory use, attention computation, and most insidiously the amount of irrelevant material competing for the model's attention.

Lost in the middle

Models tend to use information near the beginning and the end of a long context more reliably than information buried in the middle.

 Beginning            Middle             End
 ██████████        ███░░░░░░░       ██████████
usually used      easy to miss      usually used

This means a fact can be comfortably inside the official token limit and still fail to affect the answer. Position effects, weak relevance signals, several similar-looking passages competing, and patterns from training all contribute.

What actually helps:

• put stable, critical instructions near the top

• keep the current question clearly at the bottom

• retrieve fewer, more relevant passages instead of more

• use headings and a consistent document structure

• split very large tasks into stages

• test retrieval with the target fact planted at the start, the middle, and the end

That last one is not optional. It is the only way you will find out that your context is technically correct and functionally invisible.

KV caching: skipping work inside one answer

The model generates one token at a time, and before each new token it must look back at everything so far. A naive implementation would rebuild the internal representation of every earlier token at every single step rebuilding 499 tokens to produce token 500, then rebuilding 500 to produce token 501.

Here is the part that makes the fix obvious, and that most explanations skip. The model was never able to "look back at tokens" directly. A stored token is an ID number, say word #8342. That number tells attention nothing on its own. To be usable, it has to be pushed through the model's layers and matrix multiplications to produce its Key and Value vectors, and that conversion is the expensive part. Looking back was always looking at K and V. The wasteful version just rebuilt them from scratch every time.

And the crucial property: a past token's K and V never change. Token 7 contains the same thing whether you are generating token 8 or token 800. Only the new token brings a fresh Query.

So the model computes each token's K and V once and stores them in the KV cache:

Without cache, to generate token 500:
    rebuild K,V for 499 tokens  →  compare  →  499 expensive conversions

With cache, to generate token 500:
    499 sets of K,V already sitting there
    convert only the new token   →  compare  →  1 expensive conversion

It is margin notes. The first time you read a page you write your notes; afterwards you glance at the notes instead of re-reading the book. What you skip is the making, not the looking.

Which is exactly why the cache does not make long sequences free:

• The cache consumes memory roughly in proportion to sequence length.

• Every new token still has to attend over all cached positions, so each token gets slower as the sequence grows.

• Long generations remain expensive.

• The cache normally lives only for the active request, unless the serving system supports reuse.

KV caching removes repeated work, not the work of looking back. And it is a serving optimization it is not memory, and it does not mean the model learned anything.

Prefix caching: skipping work across requests

Now take the same idea and stretch it across separate requests.

Imagine a support assistant. Every request looks like this:

[same system instructions] + [same product docs] + [different question]

The first two blocks never change, and their K and V are identical every single time. So a provider can compute that shared beginning once, store it, and reuse it leaving only your new question to be processed fresh. Depending on the vendor this is called prompt caching, prefix caching, or cross-request KV-cache reuse. Cached input tokens are typically billed at a lower rate.

The ordering rule matters more than people expect. Matching runs from the very first token forward and stops at the first difference. It behaves like a contact list: typing a few letters jumps deep into the list, but only because it matches from the left.

This shares thousands of tokens:

Request 1: [Instructions][Docs][Question A]
Request 2: [Instructions][Docs][Question B]

This shares almost nothing:

Request 1: [Question A][Instructions][Docs]
Request 2: [Question B][Instructions][Docs]
             ↑ differs at token 1, match ends immediately

Identical content. Identical token count. One layout saves most of the cost and the other throws it away.

So put stable content first and dynamic content last:

1. System instructions
2. Tool definitions
3. Stable reference material
4. Conversation history
5. Current user message

And watch for the things that silently break the match, because anything that changes inside the stable region invalidates everything after it: a timestamp at the top of the system prompt (the classic mistake), random request IDs, JSON keys serialized in a different order, changing metadata, and inconsistent whitespace where it shifts token boundaries. Cache lifetime, minimum prefix length, and billing details are all provider-specific.

The two caches, side by side

Both are efficiency. Neither is memory, and neither puts a fact inside the model.

3. Sampling: temperature and top-p

From logits to one token

For each output position the model produces a raw score a logit for every token in the vocabulary. A softmax turns those scores into a probability distribution. Then a decoding method picks one.

For the fragment The sky is, an illustrative distribution:

blue      65%
clear     15%
dark      10%
bright     6%
green      3%
falling    1%

The chosen token is appended to the sequence, and the whole distribution is recomputed for the next position. This is why one different choice early can send the rest of the answer somewhere completely different, and why the same prompt can produce two answers that diverge after the fourth word.

Greedy decoding

Greedy decoding always takes the top-scoring token. Here it picks blue.

It is maximally consistent and worth being clear-eyed about: the highest-probability continuation is the one that best fits the model's learned patterns. That is not the same thing as the true one.

Temperature

Temperature reshapes the distribution before sampling:

probability = softmax(logit / temperature)

You never compute this by hand. The behavior is the point.

Low temperature sharpens. The gaps get exaggerated:

before          after (low temp)
blue    65%     blue    91%
clear   15%     clear    5%
dark    10%     dark     3%
other   10%     other    1%

Temperature near 1 leaves the distribution roughly as the model produced it. This is the reference point.

High temperature flattens. The gaps shrink:

before          after (high temp)
blue    65%     blue    37%
clear   15%     clear   23%
dark    10%     dark    20%
other   10%     other   20%

Think of weighted dice. Low temperature loads them heavily onto one face; high temperature evens the faces out. Crucially, temperature adds no knowledge and no reasoning. It only changes how conservatively the model chooses among options it already had.

What temperature 0 actually is

You cannot divide by zero, so temperature: 0 is a special mode in most APIs meaning "take the highest-scoring token every time." It is effectively greedy decoding.

Pin it to 0 when repeatability beats variety:

• classification

• field extraction from invoices and documents

• format conversion and schema-constrained JSON

• generating database filters

• known code transformations

• reproducible evaluations and test cases

And what temperature 0 is not

This distinction is worth over-emphasizing, because it is the most common misunderstanding in the entire subject:

Deterministic ≠ correct
Consistent    ≠ factual

If the model confidently prefers a wrong answer, temperature 0 will hand you that wrong answer reliably, every single time, with a straight face. It does not verify facts, fill knowledge gaps, prevent hallucination, guarantee valid JSON, or make arithmetic trustworthy. For those you need schema validation, retrieval, and real tools.

It is not even perfectly deterministic. Two candidates can be nearly tied, and tiny floating-point differences or the order in which parallel GPU operations complete can flip the winner. Add backend routing, hardware differences, model version updates, changed hidden system instructions, and non-deterministic tools or external data. A seed, where offered, improves reproducibility but a seed will not survive a model or infrastructure change.

Top-p (nucleus sampling)

Top-p restricts sampling to the smallest set of top tokens whose probabilities add up to at least p.

blue       45%   cumulative  45%   ← in
clear      25%   cumulative  70%   ← in
dark       15%   cumulative  85%   ← in
bright      8%   cumulative  93%   ← in (crosses 90% here)
green       4%   cumulative  97%   ← cut
falling     3%   cumulative 100%   ← cut

With top_p = 0.90 the nucleus is blue, clear, dark, bright. The rest are discarded and the survivors are renormalized.

The useful property is that top-p is adaptive. When the model is confident, the nucleus might hold two tokens. When it is genuinely uncertain, reaching the same threshold might take thirty. Temperature applies the same adjustment regardless of how confident the model was; top-p responds to it.

Using them together

The usual conceptual order:

logits → temperature → probabilities → top-p cutoff → sample

Implementations vary. Practically: tune one randomness control and leave the other near default. Moving both aggressively makes behavior almost impossible to diagnose when something goes wrong at 2am.

Starting points, not laws:

Note that some reasoning-oriented models manage sampling internally or expose limited controls, so check before assuming these knobs exist.

Sampling cannot make things true

Lower randomness removes unnecessary variation. It cannot check a fact. If the model's top answer is wrong, greedy decoding returns that wrong answer with perfect consistency.

For hard reasoning problems, generating several candidates and comparing them can beat one greedy pass the idea behind self-consistency. But it multiplies token cost, and it still needs an independent way to decide which candidate is right.

4. Statelessness and application-managed memory

The model does not remember anything

An inference request is stateless. When it ends, the conversation does not become part of the model. If a later request needs earlier information, your application has to supply it again.

User → App:   "My preferred language is Python"
App  → Model: "My preferred language is Python"
Model → App:  "Understood"
App:          saves the preference to a database
              ── later ──
User → App:   "What language do I prefer?"
App:          loads the saved preference
App  → Model: "Preferred language: Python
               What language do I prefer?"
Model → App:  "Python"

The application remembered. The model was simply told, again.

The desk

The clearest mental image is a desk with limited space, wiped clean after every request.

Chat history is one pile of paper on that desk. Beside it sit the system instructions, retrieved documents, tool outputs, and the current question. All of it has to fit at once. Then the desk is cleared.

This is why "is the context window just the chat history?" has a subtle answer. In a plain chat app, yes the desk usually happens to contain the entire conversation, because that is what the app chose to put there. But that is a choice, not a definition:

• The window has a fixed size; the history does not. History grows until it collides with the ceiling, and then something must be dropped or summarized. If they were the same thing, that collision could not happen.

• The window can be full with no history at all. A one-shot API call with a system prompt and a document has a full context and zero conversation. A RAG app might send instructions, three retrieved chunks, and your question while skipping most older messages entirely.

Once a conversation gets long, or the app starts selecting what to include, the two come apart completely. That selection logic is one of the most important things you will build.

Four different things called "memory"

Confusing these causes architecture mistakes that are expensive to undo.

1. Model weights. Billions of numeric parameters learned during training, holding distributed language and knowledge patterns sometimes called parametric memory. Not a searchable database of training documents, and not updated by ordinary conversation. Telling the model your name does not rewrite its weights.

2. Context window. Temporary working space for one inference, available only because it was supplied or reconstructed for that request. Gone afterwards.

3. Application memory. A real store that you own: relational database, document store, key-value store, vector database, user profile, conversation table. This is the only one that genuinely persists, and its job is to fetch the right things and place them on the desk.

4. KV cache. Stored attention calculations for fast generation. Computational state, not semantic memory. It does not mean a fact was learned, stored permanently, or will be recalled in an unrelated future request.

When a product claims it "remembers" you across sessions, that is almost always number 3 quietly loading things into number 2.

Why chat apps feel stateful

Because they do the work:

history = database.load(conversation_id)
history.append({"role": "user", "content": new_message})

response = model.generate(messages=history)

history.append({"role": "assistant", "content": response})
database.save(conversation_id, history)

Some services expose conversation IDs, threads, or stored responses and manage this for you. That moves the bookkeeping, not the underlying reality: the relevant information still has to be present in the context for that inference, even if prefix caching spares the provider from recomputing the unchanged part.

Why resending everything stops working

As the conversation grows, including every message causes token cost to rise, prefill time to lengthen, attention work to increase, KV-cache memory to grow, irrelevant or contradictory material to accumulate, and eventually the window to overflow. Note the second-to-last item it is a quality failure, not just a cost one. A long history full of superseded decisions actively degrades answers.

The standard strategies:

Full history. Send everything. Simple and accurate; fine for short conversations; does not scale.

Sliding window. Send only the most recent N messages. Bounded size, but silently drops the important decision made on turn 3.

Summary memory. Compress older turns into a shorter summary. Saves tokens; loses exact details and can bake in a summarization error that then propagates forever.

Structured state. Store the facts that matter explicitly:

{
  "preferred_language": "Python",
  "database": "PostgreSQL",
  "currency": "INR",
  "use_decimal_for_money": true
}

Far safer than hoping a natural-language summary preserved them. If a fact would be a bug when lost, it belongs in a field, not a paragraph.

Retrieval-based memory. Search stored fragments and inject only what is relevant essentially RAG applied to conversation history.

Real systems combine these:

new user message
        ↓
   orchestrator
   ├── user profile (structured)
   ├── recent messages (window)
   ├── older summary
   └── retrieved relevant memories
        ↓
 build current context
        ↓
    model inference

External systems are the source of truth

If the model used a tool to create a ticket, send an email, or read a balance, the external system holds the real result. On a later request, query that system again rather than trusting a sentence the model generated earlier. A model's claim that an action succeeded is text, not evidence.

One more distinction: statelessness does not mean nobody stores your data. Your application, the provider, logs, analytics, and tool vendors may all retain information according to their own policies. Model memory and data retention are entirely separate questions, and conflating them will get you in trouble in a security review.

5. From base model to assistant: three training stages

All three stages adjust the weights. They differ in what signal they use and what behavior they teach.

How training changes weights at all

Every training step, at a high level:

1. Show the model an example.

2. Get its prediction.

3. Measure the error with a loss function.

4. Use back propagation to work out how each weight contributed to that error.

5. Use an optimizer to nudge the weights slightly.

Repeat across an enormous number of examples.

Cross-entropy loss, briefly

Since it comes up constantly, it is worth understanding directly. Cross entropy is the score for how wrong the model's probabilities were.

The model outputs probabilities over the whole vocabulary. You then look at what the correct next token actually was and ask: how much probability did you give that?

gave it 90%  →  small loss
gave it 50%  →  bigger loss
gave it  1%  →  huge loss

The loss for one token is -log(p_correct). The logarithm is what makes confident mistakes hurt disproportionately. Being 50/50 and wrong is cheap; insisting on blue when the answer was dark is expensive. So cross entropy specifically punishes confident wrongness harder than uncertainty.

Training averages this over billions of tokens and nudges the weights so that correct tokens receive more probability next time. Falling loss means the model is assigning more probability to what actually comes next.

And notice the connection: this is the very same distribution that temperature reshapes at generation time. Cross entropy is how that distribution got its shape in the first place.

Pre training: learning what text looks like

A causal language model learns to predict the next token across enormous amounts of text.

The capital of France is  →  Paris
def add(a, b):            →  return

Every sequence yields many examples:

AI                      → systems
AI systems              → can
AI systems can          → process
AI systems can process  → language

The targets come from the data itself, which is why this is called self-supervised learning.

To get good at this narrow task, the model has to absorb grammar, vocabulary, style, code structure, relationships between concepts, a great many facts, and some reasoning patterns. The result is a base model or foundation model.

A base model is a text continuer, and that is all it is. Given a question it may well continue with more questions, because that is a plausible continuation of a document containing a question. Pre training creates capability. It says nothing about how to behave.

Instruction tuning: learning to respond

Instruction tuning, usually via Supervised Fine-Tuning (SFT), trains on prompt-and-response demonstrations:

Instruction:  Summarize this paragraph in one sentence.
Response:     [a good one-sentence summary]

Thousands of these teach the model to answer rather than continue, follow system and user instructions, handle summarization and classification and translation and transformation, produce requested formats, use conversational roles, and follow some safety and refusal patterns.

The examples may be human-written, model-generated, expert-reviewed, or some mix. Instruction tuning can add specialized knowledge, but its primary job is behavioral: teaching a capable text predictor how to apply itself to a request.

Preference tuning: learning which response is better

Many answers can be valid, correct, and still not equally good. Preference tuning uses comparisons to teach which behavior is preferred.

one prompt
   ├── Response A ──┐
   └── Response B ──┤
                    ↓
          human or AI evaluator
                    ↓
        chosen  /  rejected
                    ↓
        preference optimization

Evaluators weigh helpfulness, correctness, relevance, safety, clarity, honesty, tone, and instruction-following. A rejected response is often not wrong merely worse.

RLHF (Reinforcement Learning from Human Feedback). Three steps: people compare pairs of responses; those comparisons train a reward model whose only job is to score a response with a number; then reinforcement learning nudges the assistant toward higher-scoring behavior. PPO is the algorithm historically used for that last step.

The reward model is the clever part. It is a scalable stand-in for human taste. You cannot have people rate millions of responses, but you can train something that imitates their judgment and let it rate them all day.

RLAIF (Reinforcement Learning from AI Feedback). Another model provides some or all of the preference judgments, typically applying principles that people wrote or selected. Humans set the standard; the AI applies it at volume.

DPO (Direct Preference Optimization). Skips the separate reward model entirely. Feed it chosen/rejected pairs and it adjusts directly:

chosen response   → make more likely
rejected response → make less likely

Simpler pipeline, same destination.

All three share one goal: make preferred responses more likely than less-preferred ones. And note the careful wording preference tuning shifts the odds across behaviors the model could already produce. It is not teaching new facts.

The limits of all three stages

Pretraining:         What text is likely?
Instruction tuning:  How should I respond to a request?
Preference tuning:   Which possible response is preferred?

Read that list again and notice what is missing. None of them asks whether the answer is true.

Worse, preference data can actively pull against truth. It can favor persuasive answers over accurate ones. It can produce sycophancy, where the model agrees with an incorrect user claim because agreement was rated pleasant. It can produce reward hacking, where the model satisfies the measured proxy without satisfying the actual goal.

Real pipelines include more than three stages: continued pre training on specialized text, tool-use training, safety training, synthetic data, reasoning-oriented reinforcement learning, and distillation from a larger model. The three-stage picture stays useful because it separates capability, instruction-following, and preferred behavior three things that fail independently.

6. Why language models hallucinate

What it means

A hallucination is generated information that is false, unsupported, fabricated, or inconsistent with the evidence provided. In practice:

• an invented person, date, product, or event

• a nonexistent paper, quotation, DOI, or URL

• a statement contradicting a document you supplied

• an API method, parameter, or config option that does not exist

• a claim that a tool action succeeded when it never ran

This is not lying. Lying requires intent to deceive. The model is producing likely token sequences, and a confident false sentence is frequently the most likely one.

The root cause is an objective mismatch

Fluency is not evidence of accuracy. A high token probability means a continuation matches learned patterns well. It is emphatically not a calibrated probability that the statement is true. Everything in section 5 optimized for the left side of that diagram. The dotted line is your job.

The mechanisms, which stack

Knowledge is distributed, not stored as records. Weights compress statistical patterns. That is what enables paraphrase and generalization, and it is also why exact retrieval of rare names, dates, long numbers, quotations, and citations is unreliable. The model often recovers the correct shape of an answer and fills an exact detail wrongly which is precisely why fabricated citations look so convincing.

Training data is imperfect. It contains material that is wrong, outdated, contradictory, fictional, biased, or simply low quality. Next-token training does not fact-check sources or reliably identify the authoritative one.

Required context may be absent. If a company rule, an earlier agreement, or a current event is not in the prompt and not reachable by a tool, the model may still generate the most natural-looking answer. Statelessness makes this sharp: details from an earlier request are not available just because they once existed.

The question may be ambiguous. Mercury is a planet, an element, a deity, a company, a product, and a person. Pick the wrong one and you get a polished, fluent, entirely irrelevant answer. When ambiguity would change the result materially, asking is safer than guessing.

Attention can miss evidence you supplied. In long or noisy contexts the right passage gets overlooked especially in the middle, especially when several similar passages compete. The model then falls back on pretrained patterns instead of your document. This is lost-in-the-middle turning into a factual error.

Errors compound auto regressively. Every generated token becomes input for the next. One wrong early detail causes the model to generate supporting details around it, then an explanation for those, producing something internally consistent and completely false.

Sampling can choose a bad path. Non-zero temperature lets a less likely token through, which can steer the whole answer wrong. Lowering temperature reduces variation and, as established, greedy decoding will still pick a wrong top answer with total confidence.

Preference pressure can discourage honesty. Users tend to prefer complete, confident answers. If feedback rewards confidence more than justified uncertainty, the model learns to answer when it should abstain. Good preference training tries to reward honest uncertainty; the tension does not fully go away.

Knowledge goes stale. Weights do not update when prices, laws, software versions, schedules, or leadership change. Current questions need current sources.

Retrieval and tools fail in their own ways. RAG can fetch the wrong passage, miss the right one, return outdated material, or drown the model in noise. Tools can get bad arguments, fail silently, or return data the model misreads. Grounding lowers risk only when the whole pipeline works.

Types worth distinguishing

Separating these matters because they need different defenses and different metrics:

Reducing hallucination through system design

No prompt setting removes hallucination. Reliable systems layer defenses:

Provide authoritative context. Give the model the exact policy, record, or specification it should use. Tell it not to go beyond that evidence, and explicitly permit it to say the answer is not available.

Retrieve a small, relevant set. Trusted and current documents, real access controls, useful metadata, deduplication, relevance ranking. More context is not better it is more competition for attention and more room to get lost in the middle.

Require evidence links. Have the model attach document IDs, sections, or evidence spans to important claims. Then verify programmatically that the reference exists and actually supports the claim. A citation the model invented and you never checked is worse than no citation, because it manufactures false confidence.

Use deterministic tools. Calculator for arithmetic, database for balances, search for current facts, code execution for data processing, compiler and tests for generated code. The model should orchestrate exact operations, not imitate them in prose. This is the strawberry principle applied at system scale.

Validate structured output. JSON Schema, type checks, required fields, allowed values, range checks, business rules. Necessary and note that structural validity proves nothing about factual correctness. A perfectly well-formed invoice total can be wrong.

Verify high-risk claims independently. Extract the claims that matter and check them against authoritative data. A second model can help only if it receives independent evidence; two models given the same bad context will agree confidently on the same mistake.

Design for abstention. "I don't have enough evidence for that" must be an acceptable product outcome, with UI that supports it. Then test whether the model actually abstains when information is missing, rather than assuming an instruction was obeyed.

Evaluate specific failure modes. Build cases for known answers, unanswerable questions, ambiguous questions, conflicting sources, very long contexts, stale facts, adversarial instructions, and tool failures. Measure factual accuracy, grounded ness, citation correctness, abstention quality, tool-call correctness, and output validity as separate numbers. Averaged into one score, they hide exactly the problems you need to see.

Putting the six together

A production assistant answering a question about company policy:

1. The application receives the question and tokenizes the instructions, evidence, and history which is also where it learns what this request will cost.

2. It keeps the request inside the context window, retrieves only relevant passages, orders them so stable content sits first for prefix reuse, and positions the critical material where attention will actually find it.

3. The model emits next-token probabilities, and conservative sampling builds the answer token by token.

4. Because the model is stateless, the application stores the conversation and stable user preferences outside it the important facts in structured fields, not in a summary paragraph.

5. Pre training supplied the language ability, instruction tuning made it answer the request, preference tuning made it helpful and safe.

6. Because none of those stages checks truth, the application limits hallucination by grounding every claim in approved policy text and verifying the cited sections before the user sees anything.

stored conversation and user state
        ↓
retrieve relevant evidence
        ↓
build token-limited context
        ↓
instruction + preference-tuned model
        ↓
sample answer tokens
        ↓
validate evidence and output
        ↓
return answer, store new state

Which leads to the one lesson underneath all six:

A language model is a probabilistic generation component, not an information system. Reliability lives in the architecture around it context management, external state, retrieval, tools, validation, and evaluation.

Shipping checklist

Before an LLM feature goes live:

• Which tokenizer does the model use, and what are the real token counts in every language you support?

• How much context is genuinely necessary, and where are the critical instructions and evidence placed within it?

• Can the stable part of the prompt be reused or cached, and is anything volatile leaking into the prefix?

• Which sampling settings match this task's need for consistency versus creativity?

• What information must persist outside the model?

• Which facts belong in structured state rather than a conversation summary?

• Is the model you selected a base model, an instruction model, or a preference-tuned assistant?

• Which claims require retrieval, tools, or independent verification?

• Can the system abstain safely when evidence is missing, and have you tested that it does?

• Which evaluations measure factuality, groundedness, tool correctness, and failure behavior separately?

Clear answers to those ten questions get you most of the way to a system that is predictable, affordable, and trustworthy.

What's in this article

Part 1 The central idea

1.1 One job: predict the next word

Imagine you find a short movie script. A person is talking to an AI assistant, but the assistant's reply is missing.

Now imagine a machine that can read any text and predict the next word. You feed it the script. It predicts one word. You append that word to the script and ask again. You repeat until the reply is complete.

That is a modern chatbot. Nothing more exotic is going on.

1.2 Probabilities, not certainty

A large language model (LLM) is a very large mathematical function. It reads text and predicts what comes next but it is rarely sure about a single word. Instead it assigns a probability to every possible next word in its vocabulary.

💡 Keep this in mind The model never outputs a word. It outputs a distribution over all words. Something else picks from it.

Part 2 From prediction to conversation

2.1 The generation loop

To build a chatbot, the software first sets up text that looks like a conversation between a user and an assistant. Your message is inserted into it. Then the model writes the assistant's turn, one word at a time:

1. The model reads all the text so far and produces probabilities for the next word.

2. The software picks one word from those probabilities.

3. That word is appended to the text.

4. The whole thing runs again until the answer is finished.

2.2 Why the same question gets different answers

If the software always chose the single most likely word, replies would come out stiff and repetitive. So it sometimes picks a less likely word at random. This is called sampling.

Here's the subtle part: the model itself can be deterministic the same input produces the same list of probabilities every time. The randomness is added afterwards, at the picking step. That's why asking the same question twice can give you two different replies.

Part 3 How the model learns

3.1 Parameters: billions of dials

The model learns by reading an enormous amount of text, mostly collected from the internet. For GPT-3 alone, a person reading nonstop day and night would need more than 2,600 years to get through it. Newer models read far more.

Inside the model sit billions of numbers called parameters or weights. Think of them as dials on a giant machine. Turning a dial changes the probabilities the model assigns to the next word.

That's part of what "large" means in large language model: hundreds of billions of dials.

Nobody sets them by hand. They start random, so a fresh model produces pure nonsense. Training slowly turns them until predictions get good.

3.2 Pre-training and back propagation

Pre-training is the first big learning phase, and the task is deliberately simple: predict the next word.

A training example might be a few words or a few thousand. The model sees everything except the final word, guesses it, and the training system compares that guess to the real word.

Back propagation is how the model learns from being wrong:

1. The model makes a prediction say "dog" instead of "mat".

2. The size of the mistake is measured as a number called the loss.

3. Back propagation works backward through the network and works out how much each weight contributed to that error.

4. Every weight is nudged slightly so the correct word becomes a little more likely and the wrong ones a little less. That nudging is gradient descent.

Repeat across trillions of examples, and those tiny nudges accumulate into a model that predicts text remarkably well.

⚠️ A common misunderstanding Weights do not belong to individual words. There is no single weight for "mat" or "dog". All the weights work together on every prediction, so one update affects how the model handles many different words and sentences. That sharing is exactly why the model learns general patterns instead of memorising a lookup table.

3.3 How can it possibly get through all that text?

It really does train on almost all of it. A few tricks make that feasible:

• Batches the model processes huge chunks of text together, averages the errors, and updates weights once per batch rather than once per word.

• Every position at once in "The cat sat on the mat" it predicts "cat" from "The", "sat" from "The cat", and so on, all in a single pass. One sentence yields many training examples.

• Thousands of GPUs working simultaneously.

• One pass is usually enough the model typically reads its giant dataset only about once.

3.4 Generalization

The model doesn't only get better at predicting its training text. It also makes sensible predictions on text it has never seen before. That's called generalization, and it's the whole reason the thing is useful outside the lab.

Part 4 Training vs. using the model

It's tempting to assume the model is learning while you chat with it. It isn't.

Back propagation happens only during training. When you ask a question, the model is in inference mode: the weights are frozen. Your text runs forward through the network and words come out. Nothing gets updated.

It also almost certainly hasn't seen your exact question before. Training shaped its weights to capture general patterns grammar, facts, how explanations are structured, how code works. When you ask something new, it recombines those patterns into an answer it may never have seen word for word.

🎓 Think of a student A student practises thousands of maths problems with a teacher correcting them that's training. In the exam they get a new problem that's inference. They don't learn during the exam, and they never saw that exact problem, but they solve it with the methods they absorbed.

This also explains why LLMs get things wrong. If the learned patterns don't fit your question well, the model can produce an answer that sounds confident and is simply incorrect.

Part 5 The scale of computation

Inside the model, training is just simple arithmetic: adding and multiplying numbers. But it happens an absurd number of times, because the dataset is huge and every one of the billions of weights participates in every single prediction.

So here's a thought experiment. Imagine you're a superhuman calculator doing one billion calculations every second, never stopping. How long to complete all the math used to train the largest language models?

Real training finishes in weeks or months only because thousands of specialised chips do this math simultaneously. Even then, it costs millions of dollars.

Part 6 From text predictor to assistant

Everything so far is pre-training. But being excellent at continuing random internet text is not the same as being a helpful assistant. Ask a purely pre-trained model "How do I bake a cake?" and it might reply with more questions because that's what a forum page looks like.

So chatbots get a second round of training: RLHF, or reinforcement learning with human feedback.

1. The model produces answers.

2. Human reviewers flag answers that are unhelpful or problematic, and indicate which answers are better.

3. That feedback is used to adjust the weights again.

4. The model becomes more likely to produce answers people actually prefer.

🎓 The student, again Pre-training is a student reading an entire library and absorbing enormous knowledge. RLHF is a teacher then showing that student how to answer a question clearly, helpfully, and politely.

For assistant-like behaviour, this stage matters roughly as much as pre-training does.

Part 7 Why GPUs and transformers mattered

7.1 GPUs: thousands of workers at once

A normal CPU is like one very capable worker handling tasks one after another. A GPU (graphics processing unit) is like thousands of simpler workers each handling a small task at the same moment. GPUs were built for video games, where millions of pixels must be computed at once. Training an LLM also needs millions of small calculations at once a near-perfect match.

7.2 The architecture has to allow parallel work too

Thousands of workers are useless if the job can only be done one step at a time. Before 2017, most language models read text one word at a time, in order. To process word 5 they had to finish words 1–4 first so most of the GPU sat idle.

In 2017, a team of Google researchers introduced the transformer. It doesn't have to read start to finish. It ingests all the words at the same time, which keeps every GPU worker busy and makes training on massive text practical.

📖 Picture it Older models read a book one word at a time, left to right. A transformer looks at the whole page at once.

7.3 Turning words into numbers

Computers can't do arithmetic on the word "cat". And training needs smooth numbers that can be nudged slightly up or down. So the first step inside a transformer is converting each word into a long list of numbers called a vector.

cat  →  [  0.2, −1.3,  0.8, … ]
dog  →  [  0.3, −1.1,  0.7, … ]
car  →  [ −0.9,  0.5,  2.1, … ]

(These numbers are invented for illustration real vectors have hundreds or thousands of dimensions.)

Those numbers are learned during training, and a vector can carry aspects of a word's meaning. Words with similar meanings end up with similar vectors: cat and dog land close together, while car sits far away.

Part 8 Inside the transformer

8.1 Attention: words talk to each other

Initially, a word's vector is identical in every sentence. But plenty of words change meaning with context. The signature operation inside a transformer attention lets the vectors for different words communicate. Each vector looks at the words around it and updates itself. And because the transformer sees all words at once, this happens in parallel.

Take the word bank:

• "I deposited money in the bank." → a financial institution.

• "We sat on the bank of the river." → the edge of a river.

In the second sentence, attention notices "river" nearby and adjusts the vector for "bank" so it leans toward the riverbank meaning. The financial sense fades out.

🗣️ Picture a group discussion Everyone listens to everyone else simultaneously, and each person updates their understanding based on what the others said.

8.2 Feed-forward networks: stored patterns

A transformer also contains feed-forward neural networks. Where attention mixes information between words, a feed-forward network operates on each word's vector independently, applying language patterns the model absorbed during training. They give the model room to store what it knows.

8.3 The journey through a transformer

This doesn't happen once. The vectors pass through many layers, each adding more useful context like revising a draft over and over.

1. Encode turn each input word into a long vector of numbers.

2. Attend let vectors share context with each other, in parallel.

3. Transform apply feed-forward networks that layer in learned patterns.

4. Repeat pass through many layers, making every vector richer.

5. Predict use the final vector to produce a probability for every possible next word.

At the end, only the last vector in the sequence is used for the prediction. By then it has absorbed information from the entire input plus everything the model learned in training. A final function converts it into probabilities: "mat" 40%, "floor" 25%, and so on.

Then the loop closes: the transformer turns context into next-word probabilities, the chatbot picks a word, appends it, and runs the whole thing again.

Part 9 Designed framework, emergent behaviour

Researchers design the structure attention, feed-forward networks, the training procedure. But they don't write rules like "here's how to translate French" or "answer cooking questions this way".

Skills such as grammar, translation, summarising, and coding appear on their own once training tunes hundreds of billions of parameters across huge amounts of data. That's an emergent phenomenon.

🌱 Think of a farmer A farmer supplies soil, water, and sunlight, but doesn't design each leaf and branch. The tree grows its own shape. Researchers set up the conditions; the skills grow out of training.

❗ An LLM is not a database of answers Nobody wrote a list of questions and answers into it. It has weights shaped by training, and it constructs each answer fresh, one word at a time.

Why it's hard to explain

Researchers know precisely which mathematical operations run inside. But with billions of weights acting together, it's extraordinarily hard to say why the model chose one particular word over another. It's a little like the brain: we understand how neurons signal, yet we can't point to why you had one specific thought.

Simple goal, powerful result

The objective sounds almost too plain: predict the next word. But combine it with enormous models, enormous datasets, GPUs, transformers, and human-feedback training and that plain objective produces fluent, useful, occasionally surprising behaviour.

A simple prediction goal can create powerful behaviour once the model, the data, and the computation all become very large.

Part 10 The whole idea at a glance

Based on 3Blue1Brown, Large Language Models explained briefly (YouTube, 7:57), with added explanations. Figures are presented as stated in the video; the vector values and probabilities shown are illustrative.

1. The Router Pattern – simple, rule-based, and predictable.

2. The Autonomous Pattern – powerful, self-directed, and flexible.

But as we’ll see, each has its pros and cons. Eventually, we’ll learn how LangGraph helps us combine the best of both worlds — flexibility and control.

1. The Big Picture

When you design an AI agent, you’re basically building a flow — a system that takes input from the user, reasons about it, maybe calls a few tools, and then returns a response.

There are two common ways to design this flow:

The Router Pattern

A controlled flow.
Your application (not the LLM) decides where the request should go — which tool, chain, or action should handle it. The LLM is used more like a helper, not the decision-maker.

The Autonomous Pattern

A self-directed flow.
Here, the agent (the LLM) takes the lead. It reasons, plans, and acts on its own — usually using the ReAct (Reasoning + Acting) framework.
It can think: “What should I do next?” → call a tool → observe the result → think again — repeating until it decides the job is done.

However, there’s a problem:
In a fully autonomous system, we start to lose control.
The agent can:

• Loop endlessly,

• Retry too often, or

• Skip important validation steps.

This makes the system unpredictable and hard to manage at scale.

That’s where LangGraph comes in.
It gives us a way to design a custom workflow (custom graph) where we decide the steps, the order, and the rules — without losing the reasoning power of an LLM.

2. The Router Pattern — Simple and Controlled

Let’s start with the Router Pattern, the simpler one.

Imagine your application as a traffic police officer standing in the middle of a busy road.
Every user request is like a vehicle approaching an intersection.
The router decides which direction each request should go.

For example:

• A user says, “Search for top AI frameworks” → route to the Search Tool

• A user says, “Add meeting to my calendar” → route to the Calendar Tool

• A user says, “Summarize this article” → route to the Summarizer Chain

You can implement these routing rules using:

• Simple if-else conditions, or

• A lightweight LLM classifier that picks the right branch.

Once the decision is made, that specific tool or chain runs, gives a result, and you return it to the user.

Pros of the Router Pattern

• Predictable: You always know what path the query will take.

• Easy to test: Each route can be tested in isolation.

• Production-friendly: Ideal when consistency and reliability matter most.

Cons of the Router Pattern

• Limited flexibility: If a task needs multiple tools or steps, the router can’t easily handle it.

• Manual maintenance: You have to design and update all the routing rules yourself.

In short:

Router pattern = “You control every move.”
Great for structured tasks, not for open-ended ones.

3. The Autonomous Pattern — Flexible but Risky

The Autonomous Pattern is where the agent truly feels “intelligent”.
Here, the LLM becomes the planner and decision-maker.

This is the pattern used in the ReAct (Reasoning + Acting) framework — one of the most common agent architectures today.

Let’s break it down.

How ReAct Works

The ReAct agent follows a simple loop:

1. Think – The LLM reasons: “What do I need to do next?”

2. Act – It performs that action, such as calling a tool.

3. Observe – It reads the tool’s output.

4. Repeat – Based on the observation, it decides the next step.

The agent keeps looping through this “Think → Act → Observe → Repeat” process until it believes the goal is achieved.

Example

If you ask the agent:

“Schedule a meeting with John for tomorrow and also find the latest AI trends.”

It might:

1. Call the calendar tool to schedule the meeting.

2. Then, call the search tool to find AI trends.

3. Finally, summarize both results and respond.

All of this happens autonomously — without you writing specific routing rules.

Pros

• Highly flexible: Works for complex, multi-step, or unstructured tasks.

• Dynamic reasoning: The agent can adapt based on context.

Cons

• Less developer control: You can’t easily predict what the agent will do next.

• Risk of infinite loops: The agent may keep retrying the same steps endlessly.

• Unstable results: Each run might behave slightly differently.

To avoid this, developers usually set a maximum loop limit, such as 25 iterations.
But even then, the control is coarse, not fine-grained.

In short:

Autonomous pattern = “The agent decides everything.”
Great for exploration, risky for production.

4. Why Build a Custom Graph with LangGraph?

Now that we’ve seen both extremes — one too rigid (Router), and one too free (Autonomous) — we need a middle ground.

That’s exactly what LangGraph provides.

LangGraph allows you to design your agent as a custom graph, where you decide:

• The sequence of steps (like a router),

• The intelligence inside each step (like an autonomous agent), and

• The rules and conditions that control the flow.

You can define exactly what happens after each step let say for example:

1. The LLM generates a response →

2. You send that response for validation →

3. If it’s invalid, you can regenerate or adjust it →

4. If it’s valid, you send it to the user.

You can also:

• Limit retries to avoid loops,

• Add timeouts for slow tools, and

• Log or monitor every step in the process.

So, with a custom LangGraph, you get the reasoning power of an autonomous agent, but with the safety and structure of a router.

In short:

LangGraph = Controlled Intelligence.
You keep the agent’s brain, but you decide its behavior.

5) LangGraph Basics (Explained with the Veg Biryani Example)

To really understand LangGraph, let’s imagine we are cooking Veg Biryani.
Cooking biryani is a sequence of clear steps that happen one after another — some steps depend on others, and sometimes we even repeat a step if something doesn’t taste right.

This cooking process is a perfect analogy to how LangGraph works.

Step 1: Understanding Nodes — the Building Blocks

In LangGraph, each node represents one action or step in your workflow.
You can think of a node like a function in code or a task in real life — it does one specific thing and passes its result forward.

For example, when you cook Veg Biryani, you might follow these steps:

1. Cut the vegetables

2. Boil the rice

3. Add salt

4. Taste the biryani

Each of these is a node in our cooking graph.
Every node performs one job, and when it’s done, it sends its result (or “state”) to the next node in the sequence.

So in LangGraph, if you create a node called cut_vegetables, it might run a function that handles all cutting actions. Then the output of this node goes to the next node — boil_rice.

Step 2: Understanding Edges — Connecting the Steps

Now, these nodes don’t exist alone. They are connected by edges.
An edge in LangGraph represents the flow of control — it shows which step comes next.

When you finish cutting the vegetables, what do you do next? You boil the rice.
So, you draw a connection (or arrow) from Cut Vegetables ➜ Boil Rice.

Visually, it’s like drawing arrows between boxes:

Cut Vegetables → Boil Rice → Taste Biryani

This flow tells LangGraph how to move from one task to another.
Edges are like the recipe instructions that say “after this, do that.”

Step 3: Conditional Edges — Making Decisions

Cooking isn’t always a straight line.
Sometimes, we make decisions based on what happens during the process.

For example, after you taste the biryani, you might think:

• “Hmm, it needs more salt.”

• or, “Perfect! It’s ready to serve.”

In LangGraph, these “if/else” kinds of decisions are represented using conditional edges.
A conditional edge checks some condition and then decides which node to go to next.

So in our example:

• If the biryani needs salt, go to the add_salt node.

• If the biryani tastes fine, go to the End node.

Once you add salt, you come back and taste again — this creates a small loop:

Taste Biryani → Add Salt → Taste Biryani → (Repeat until OK)

This loop continues until the condition changes — in this case, until the taste is perfect.

So, conditional edges make your graph smart.
They give your agent the ability to make decisions just like you do when cooking.

Step 4: State — The Shared Shelf of Ingredients

Now, when you’re cooking, you have a kitchen shelf or counter where all your ingredients sit — rice, salt, vegetables, spices, pots, etc.
You keep using and updating them as you cook.

In LangGraph, that shelf is called the state.

A state is the shared data that flows through the graph.
Each node can:

• Read from the state (e.g., “How much salt do we have left?”)

• Write to the state (e.g., “We added 1 tsp of salt.”)

This makes every step aware of what happened before.
When the “Taste Biryani” node runs, it can look at the state and decide:
“Did we already add salt? How much? Should we add more?”

That’s why LangGraph is called a StateGraph — it doesn’t just run steps; it also carries and updates shared memory across them.

Step 5: Start and End — The Built-In Boundaries

In every workflow, there’s always a beginning and an end.
In LangGraph, you don’t need to manually create them — they are built-in as Start and End nodes.

Your graph will automatically begin from the Start node and finish at the End node once there are no more steps to execute.

Step 6: Visualizing It All — The Cooking Flow

If we draw this entire cooking process as a graph, it might look like this:

Here’s how to read it:

• The solid arrows represent the normal flow of steps.

• The dotted arrows (or labeled arrows) represent conditional edges — they depend on the situation.

• The state (ingredients, salt level, etc.) moves through every node.

So, this simple biryani recipe actually behaves like a mini LangGraph workflow!

6) Detailed code example:

graph.js

/**
 * Custom LangGraph agent with:
 * - LLM node
 * - Tool execution node
 * - Conditional routing (LLM -> tools -> LLM ... -> END)
 * - Persistent memory via checkpointer
 */

import readline from 'node:readline/promises';
import { tool } from '@langchain/core/tools';
import { ChatGroq } from '@langchain/groq';
import { END, MemorySaver, MessagesAnnotation, StateGraph } from '@langchain/langgraph';
import { ToolNode } from '@langchain/langgraph/prebuilt';
import { TavilySearch } from '@langchain/tavily';
import z from 'zod';
import { printGraph } from './utils.js';

/** -------------------------------
 * MEMORY / CHECKPOINTER
 * -------------------------------
 * - "state" = data passed inside one graph run (between nodes).
 * - "checkpointer" = saves/loads state snapshots across multiple runs (turns) using thread_id.
 *   This gives you conversation memory across user messages.
 */
const checkpointer = new MemorySaver();

/** -------------------------------
 * TOOLS
 * -------------------------------
 * Tools are functions the LLM may call during reasoning.
 * We wrap custom functions with `tool(...)` to give them a name + schema.
 */

// Internet search tool (prebuilt)
const search = new TavilySearch({
  maxResults: 3,
  topic: 'general',
});

// Dummy calendar tool (replace with real Google Calendar)
const calendarEvents = tool(
  async ({ query }) => {
    // do calendar lookup here (query is validated by zod schema below)
    return JSON.stringify([
      {
        title: 'Meeting with Sujoy',
        date: '9 Aug 2025',
        time: '2 PM',
        location: 'Google Meet',
      },
    ]);
  },
  {
    name: 'get-calendar-events',
    description: 'Get calendar events by query.',
    schema: z.object({
      // zod schema tells the LLM what input shape is allowed
      query: z.string().describe('Query used to search calendar events.'),
    }),
  }
);

// The ToolNode is a LangGraph node that executes any tool calls requested by the LLM.
// We pass all available tools so it knows what it can run.
const tools = [search, calendarEvents];
const toolNode = new ToolNode(tools);

/** -------------------------------
 * LLM
 * -------------------------------
 * - We bind tools so the LLM can emit structured `tool_calls`.
 * - temperature: 0 for more deterministic outputs.
 */
const llm = new ChatGroq({
  model: 'openai/gpt-oss-120b',
  temperature: 0,
}).bindTools(tools);

/** -------------------------------
 * LLM NODE
 * -------------------------------
 * Simple node that:
 *  - reads chat history from state.messages
 *  - calls the LLM
 *  - appends the assistant message (possibly with tool_calls) back into state
 */
async function callModel(state /* MessagesAnnotation state */) {
  const response = await llm.invoke(state.messages);
  return { messages: [response] };
}

/** -------------------------------
 * ROUTER / BRANCH FUNCTION
 * -------------------------------
 * After the LLM responds, decide where to go next:
 *  - If LLM asked to call a tool => go to 'tools' node.
 *  - Otherwise => finish (END).
 * This is a "conditional edge predicate" (router/policy).
 */
function shouldContinue(state) {
  const last = state.messages[state.messages.length - 1];
  return (last.tool_calls?.length) ? 'tools' : '__end__';
}

/** -------------------------------
 * BUILD THE GRAPH
 * -------------------------------
 * - StateGraph(MessagesAnnotation): we use a chat-style state (has `messages`).
 * - Nodes:
 *    'llm'   => runs callModel (think/respond)
 *    'tools' => runs toolNode  (act: execute tool calls)
 * - Edges:
 *    __start__ -> llm                  (begin by asking LLM what to do)
 *    tools -> llm                      (after tools, return to LLM to observe/continue)
 * - Conditional edges from 'llm':
 *    if shouldContinue() === 'tools'   => go to tools
 *    if shouldContinue() === '__end__' => END (finish run)
 */
const graph = new StateGraph(MessagesAnnotation)
  .addNode('llm', callModel)
  .addNode('tools', toolNode)
  .addEdge('__start__', 'llm')
  .addEdge('tools', 'llm')
  .addConditionalEdges('llm', shouldContinue, {
    __end__: END,
    tools: 'tools',
  });

/** -------------------------------
 * COMPILE
 * -------------------------------
 * - compile() turns the graph into an executable app.
 * - we pass the checkpointer so the app can persist conversation by thread_id.
 */
const app = graph.compile({ checkpointer });

/** -------------------------------
 * CLI MAIN
 * -------------------------------
 * - thread_id identifies the conversation for the checkpointer.
 * - printGraph renders a PNG of the graph for docs/diagnostics.
 * - loop: read user input -> run app.invoke -> print final assistant message.
 */
async function main() {
  const config = { configurable: { thread_id: '1' } };

  // optional: export a diagram of the compiled graph
  await printGraph(app, './customGraph.png');

  const rl = readline.createInterface({ input: process.stdin, output: process.stdout });

  while (true) {
    const userInput = await rl.question('You: ');
    if (userInput === '/bye') break;

    // Minimal state input: start with the new user message.
    // The checkpointer loads previous context for this thread_id under the hood.
    const result = await app.invoke(
      { messages: [{ role: 'user', content: userInput }] },
      config
    );

    const final = result.messages[result.messages.length - 1];
    console.log('AI:', typeof final.content === 'string' ? final.content : JSON.stringify(final.content));
  }

  rl.close();
}

main().catch(console.error);

utils.js

// utils.js
import { writeFileSync } from 'node:fs';

/**
 * printGraph:
 * - gets a drawable graph from the compiled app
 * - renders Mermaid PNG
 * - writes it to disk
 */
export async function printGraph(agent, graphPath) {
  const drawable = await agent.getGraph();
  const png = await drawable.drawMermaidPng();
  const buf = Buffer.from(await png.arrayBuffer());
  writeFileSync(graphPath, buf);
}

1. Setting the Stage: Why Agents Need More Than Just a Brain

Large Language Models (LLMs) are like brilliant thinkers — they can reason, plan, and write, but they don’t do anything on their own.
They need a way to interact with the world — to fetch information, remember past conversations, or call APIs.

That’s where agentic patterns come in.

An agent is an AI system that can:

• Think (reason about what to do),

• Act (use tools or APIs),

• Observe (see what happened),

• and Learn or adjust based on those observations.

Among the many agent patterns, one of the most powerful and influential is the ReAct pattern.

2. The ReAct Pattern — Reasoning + Acting

ReAct (short for Reason + Act) is a framework that allows an AI model to think through its steps while using external tools when needed.
It was introduced by Yao et al. (2023) in the paper “ReAct: Synergizing Reasoning and Acting in Language Models.”

At its core, ReAct blends two abilities:

• Chain-of-Thought Reasoning (CoT): letting the LLM “think out loud.”

• Action-Taking: allowing the LLM to perform real operations (search, calculate, look up data).

Here’s how a ReAct loop typically works:

Thought → Action → Observation → Thought → ... → Final Answer

For example:

Thought: “I don’t know today’s weather.”
Action: “Search for current weather in London.”
Observation: “It’s 18°C and cloudy.”
Thought: “I now know the answer.”
Final Answer: “It’s 18°C and cloudy in London.”

This loop repeats until the model feels confident enough to finalize an answer — or hits a limit to prevent endless reasoning.

3. Why ReAct Matters

Before ReAct, most AI systems separated “thinking” and “doing.”
That meant:

• Models reasoned internally, but couldn’t access external data.

• Tools or APIs worked independently, without intelligent context.

ReAct merges both.
It turns a passive chatbot into an active problem solver.

Benefits include:

• Real-world grounding: Uses live data sources.

• Transparency: You can read the reasoning steps.

• Adaptability: Handles new, unpredictable tasks dynamically.

• Accuracy: Reduces hallucination by checking facts through tools.

4. Giving Agents Memory: Short-Term and Long-Term

Even the best reasoning loop isn’t enough if the agent forgets everything.

Agents, like humans, need memory.

🔹 Short-Term Memory

This is like your working memory — it stores the immediate context of an ongoing conversation or session.
In LangChain, short-term memory helps the agent recall recent messages, previous tool outputs, or system instructions.

A checkpointer (like MemorySaver in LangGraph) can maintain this conversation history or reasoning trace, so each new step is informed by what just happened.

🔹 Long-Term Memory

Long-term memory goes beyond a single session.
It stores structured knowledge — like past projects, user preferences, or important facts — in databases or vector stores.
This allows the agent to remember who you are, or refer back to previous work, even days later.

5. Let’s Build a Personal Assistant Agent

Now that we understand the ReAct pattern and memory, let’s bring it to life.

We’ll build a personal assistant that can:

• Search the web,

• Fetch calendar events,

• Maintain conversation history (short-term memory), and

• Draw a graph of its internal workflow.

We’ll use:

• LangChain LangGraph for ReAct agent creation,

• TavilySearch as a prebuilt search tool, and

• A custom calendar tool built with @langchain/core/tools.

6. Setting Up the Code

Here’s the full implementation:

import { ChatGroq } from "@langchain/groq";
import { createReactAgent } from "@langchain/langgraph/prebuilt";
import { TavilySearch } from "@langchain/tavily";
import { tool } from "@langchain/core/tools";
import { z } from "zod";
import { writeFileSync } from "node:fs";
import readLine from "node:readline/promises";
import { MemorySaver } from "@langchain/langgraph";

async function main() {
  // 1) Choose and configure the LLM that will do the reasoning.
  //    Here we use a Groq-hosted model and set temperature to 0 for deterministic outputs.
  const model = new ChatGroq({
    model: "openai/gpt-oss-120b",
    temperature: 0
  });

  // 2) Add a prebuilt tool: web search via Tavily.
  //    The agent can call this when it needs fresh information.
  const search = new TavilySearch({
    maxResults: 3,
    topic: "general"
  });

  // 3) Define a custom tool: fetch calendar events.
  //    We wrap an async function with `tool()`, and validate inputs with Zod.
  const calendarEvents = tool(
    async ({ query }) => {
      // In a real app, call your calendar API here and return structured data.
      return JSON.stringify([
        {
          title: "meeting with harshal",
          time: "2 PM",
          location: "Google Meet"
        }
      ]);
    },
    {
      name: "get-calendar-events",
      description: "Fetch calendar events for a given time/person query.",
      schema: z.object({
        query: z
          .string()
          .describe("Natural-language query with time/person to schedule or fetch a meeting")
      })
    }
  );

  // 4) Short-term memory (checkpointer).
  //    This persists intermediate state across steps within a thread.
  const checkpointer = new MemorySaver();

  // 5) Create the ReAct agent.
  //    We pass the LLM, all available tools, and the checkpointer (for short-term memory).
  const agent = createReactAgent({
    llm: model,
    tools: [search, calendarEvents],
    checkpointer
  });

  try {
    // 6) Simple CLI loop to chat with the agent.
    const rl = readLine.createInterface({
      input: process.stdin,
      output: process.stdout
    });

    while (true) {
      const userQuery = await rl.question("You: ");
      if (userQuery === "/bye") break;

      // 7) Invoke the agent with a system instruction and the user's message.
      //    `thread_id` tells the checkpointer which conversation to attach to.
      const result = await agent.invoke(
        {
          messages: [
            {
              role: "system",
              content: `You are a personal assistant. Use the provided tools if you lack information. Current date/time: ${new Date().toUTCString()}`
            },
            { role: "user", content: userQuery }
          ]
        },
        {
          configurable: { thread_id: "1" }
        }
      );

      // 8) Print the agent's final message content.
      console.log(
        "Assistant:",
        result.messages[result.messages.length - 1].content
      );
    }

    rl.close();

    // 9) Export a graph image of the workflow the agent can execute.
    //    Useful for visualizing tool-use and control flow.
    const drawableGraphState = await agent.getGraphAsync();
    const graphStateImage = await drawableGraphState.drawMermaidPng();
    const graphStateArrayBuffer = await graphStateImage.arrayBuffer();

    const filePath = "./graphState.png";
    writeFileSync(filePath, new Uint8Array(graphStateArrayBuffer));
  } catch (error) {
    console.error("An error occurred during agent invocation:", error);
  }
}

main();

What each part does (the “why” behind the code)

1) LLM selection

• ChatGroq is the model client. You’ve set temperature: 0 to make outputs deterministic and more task-focused—great for a personal assistant.

• The model: "openai/gpt-oss-120b" string is the model identifier used by Groq. Treat it like a model choice handle. If you swap models later, the agent logic stays the same.

2) Prebuilt tool (TavilySearch)

• ReAct agents shine when they can act (call tools) between reasoning steps.

• TavilySearch lets the agent fetch fresh information (news, facts, pages). You set maxResults: 3 to keep token usage down.

3) Custom tool (calendar)

• tool() wraps your function so the agent can discover and call it as an action.

• The Zod schema defines and validates inputs. This is important because the LLM will generate the tool arguments in structured form—validation keeps things robust.

• Right now it returns a JSON string. LangChain is fine with that, but returning an object/array is often nicer (the runtime will stringify as needed). Your current approach still works.

4) Memory (short-term)

• MemorySaver is a checkpointer that keeps state per thread_id.

• This gives you short-term memory: the agent can look back at previous steps and tool results in the same thread.

• For long-term memory, you’d typically add a vector store or database and retrieve relevant chunks into the context. (Not included here, but easy to extend later.)

5) The ReAct agent

• createReactAgent glues reasoning + acting + memory together.

• It uses ReAct-style prompting behind the scenes: Thought → Action → Observation loops until it’s confident enough to answer or hits internal limits.

6–8) The CLI loop

• Simple REPL: user types a prompt, agent responds.

• You set a system message to define role and policy (“use tools if needed”).

• thread_id: "1" makes all turns part of the same memory thread. Change it to start a new short-term memory context.

9) Graph export

• getGraphAsync() and drawMermaidPng() produce a Mermaid-based PNG of the workflow graph (the agent’s possible control-flow).

• This is great for explaining or debugging how the agent could move between nodes/tools.

A ReAct agent is an AI system that combines reasoning (thinking through problems step-by-step) with acting (using tools or taking actions).
The term ReAct stands for Reasoning + Acting.

This approach was first introduced by Yao et al. (2023) in the paper “ReAct: Synergizing Reasoning and Acting in Language Models.”

In short:

• The LLM (Large Language Model) serves as the “brain” of the agent.

• The agent uses the LLM’s reasoning ability to decide what to do, and then acts by using tools (like a search engine, API, or calculator).

• The process repeats — the agent thinks, acts, observes what happened, and thinks again — until it reaches a final answer.

This integration makes ReAct agents more powerful and flexible than older AI systems that separated thinking from doing.

How ReAct Agents Work

The ReAct framework mimics how humans solve problems:

1. We think (“What’s the next step?”)

2. We act (e.g., check something, calculate, or look it up)

3. We observe the result

4. We think again based on what we learned.

Example:

If you’re packing for a trip:

• Thought: “What’s the weather like?”

• Action: Check the forecast.

• Observation: It’ll be cold.

• Thought: “I need warm clothes.”

• Action: Look in your closet.

• Observation: Clothes are in storage.

• Thought: “I’ll layer lighter clothes instead.”

ReAct agents do exactly this — but with prompt engineering guiding them.

Key Components

• Thoughts (Reasoning): The LLM breaks the big problem into smaller parts using chain-of-thought (CoT) reasoning.

• Actions: The model performs tasks using tools — like calling an API, querying a database, or searching the web.

• Observations: The model reads results and decides what to do next.

The process repeats until the model finds a satisfactory answer or reaches a stopping condition (like a loop limit).

The ReAct Loop

The ReAct loop is the repeating cycle:

Thought → Action → Observation → (repeat or end)

The agent keeps looping through this cycle until:

• It’s confident in an answer, or

• It hits a max loop limit (to save tokens, time, or cost).

This feedback loop lets the agent reason dynamically, adjusting to new data or unexpected results in real time.

ReAct Prompting

ReAct prompting is a special technique to teach an LLM to follow the “think–act–observe” cycle.
The prompt tells the model:

• How to reason step by step (CoT reasoning)

• What actions/tools it can use

• How to make observations after each action

• When to loop or stop

• How to output the final answer

A simple ReAct prompt format looks like this:

Question: <user query>
Thought: <model reasoning>
Action: <chosen tool>
Action Input: <input for that tool>
Observation: <tool result>
... (repeats as needed)
Thought: I now know the final answer
Final Answer: <output>

Example tools (as in LangChain’s built-in ReAct module):

• Wikipedia: to look up facts

• DuckDuckGo Search: to check current events

• Calculator: to handle math problems

This design lets the agent reason transparently while using external tools intelligently.

Benefits of ReAct Agents

ReAct’s mix of reasoning + tool use was a key milestone that led to more advanced AI agents, like Reflexion and modern reasoning models.

ReAct Agents vs. Function Calling

In short:

• Function calling is efficient for fixed, well-defined APIs.

• ReAct is better when the task requires adaptive reasoning or multiple uncertain steps.

Building ReAct Agents

You can build ReAct agents:

• From scratch in Python or Javascript

• Using frameworks like LangChain (LangGraph), LlamaIndex, or BeeAI, which provide ready-made ReAct modules.

These frameworks let developers easily plug in tools (APIs, search engines, calculators, etc.) and control the agent’s reasoning loop.

Summary

ReAct agents are AI systems that merge reasoning (thinking) with acting (tool use) in a feedback loop.
They:

• Think → Act → Observe → Repeat

• Use chain-of-thought reasoning and external tools

• Are flexible, explainable, and accurate

• Form the foundation of modern autonomous AI agents

In short:

A ReAct agent is an intelligent AI framework that doesn’t just think — it does, learns from results, and adapts in real time.

So far, so good.

But what happens when you say, “Do I have any meetings today?” or “Schedule a call with John at 3 PM”?

You’ll probably get a polite but firm reply:

“I don’t have access to your calendar.”

That’s the moment we hit the limits of traditional AI assistants — they can talk, but they can’t act.
They don’t have access to your personal data, and they can’t take real-world actions on your behalf.

This is where the next evolution begins — Agentic AI.

What Exactly Is Agentic AI?

To understand Agentic AI, let’s first break down the word agent.
An agent is something that can observe, decide, and act.

Traditional Large Language Models (LLMs) — like ChatGPT, Bard, or Claude — are powerful, but they’re bound by a few walls:

1. Knowledge Cutoff:
   They only know what they were trained on — books, articles, code, and data up to a certain point (for example, till 2024).
   Anything after that? They’re in the dark.

2. No Real-World Access:
   They can’t look up today’s weather, check your calendar, or book you a flight — unless they have special tools.

3. No Action Capability:
   They can’t perform tasks — they just suggest what you should do.

Agentic AI changes all that.

When an LLM is given the ability to make decisions and use external tools or APIs to perform actions, it becomes Agentic — capable of reasoning, acting, and learning like a mini digital being.

A Simple Example

Let’s ask ChatGPT:

“What’s the weather in Ahmedabad?”

If the AI shows you real-time weather data — it just used a web search tool behind the scenes.
That tool call is what makes it agentic.

Here, the model didn’t just “answer.”
It decided it needed fresh data → called a tool → and gave you the result.
That’s decision-making in action — the core idea behind Agentic AI.

Behind the scenes, ChatGPT itself works as a smart AI agent, capable of understanding your query, deciding what to do next, and taking the right step — just like a digital assistant who knows when to Google, when to calculate, and when to just talk.

Why Do We Need Agentic Frameworks?

Okay, now we know what Agentic AI is.
But how do we build one?

We could, in theory, wire everything manually — writing endless tool calls, loops, and API logic.
But that’s messy, error-prone, and hard to maintain.

That’s why developers use Agentic Frameworks — ready-made, battle-tested structures designed for production-grade AI systems.

These frameworks handle the boring stuff — error recovery, workflow control, memory, and state management — so that you can focus on what your agent does, not how it does it.

Among many, two names dominate the space:

• LangChain

• LangGraph

Let’s see how they differ — and how they work together.

LangChain vs. LangGraph: The Dynamic Duo

LangChain — The Chain Builder

LangChain started as a utility library but has now evolved into a full company offering multiple AI tools and products — including LangGraph.

Think of LangChain as a workflow builder.
It helps you connect multiple steps (like Retrieve → Generate → Store → Respond) into a seamless chain.

Each step might involve:

• A vector database for retrieving context

• A Large Language Model for generating text

• A memory component for storing chat history

• A final response generator for the user

All these steps are connected like nodes in a flowchart — what we call a Directed Acyclic Graph (DAG).

A DAG means your workflow flows in one direction only — from start → process → end.
It can’t loop back to a previous step.

That’s perfect for simple tasks, but the real world isn’t always that linear.
Sometimes, we need to loop, retry, or rethink decisions dynamically.

That’s where LangGraph comes in.

LangGraph — The Brain with Loops

LangGraph takes the concept of LangChain further.
It still uses nodes and edges, but here’s the twist — loops are allowed.

Imagine a flowchart where the AI generates a draft answer, then checks:

“Is this good enough?”

If yes, the agent ends.
If no, it loops back and regenerates until it’s satisfied.

That’s LangGraph — dynamic, iterative, and much closer to how humans think.

In real-world use cases, this is a game changer.
Why? Because most intelligent systems need to reason, retry, and refine — not just follow a straight path.

LangGraph simplifies this complexity.
Each node can run an LLM, call tools, check conditions, or update memory — all while maintaining a shared state that every node can access.

Explore more on LangGraph with below article:

https://langchain-ai.github.io/langgraphjs/concepts/high_level/

Why do we need tool calling?

LLMs learn from a snapshot of data. This snapshot ends at a knowledge cutoff date. So if you ask, “When was iPhone 16 launched?”, a model that only knows up to earlier data might say, “There is no iPhone 16 yet,” because it simply doesn’t know the latest.

With tool calling, we give the model a web search tool. Now the model can decide: “Hmm, I might be out of date. Let me use the webSearch tool.” Your program performs that search, returns the live result, and the model answers with up-to-date information.

We can attach many tools, some of examples are:

• Web search → for news and real-time facts

• Database → for inventory, users, orders

• Calendar → to check availability or create meetings

• Internal APIs → to place orders, generate reports, fetch metrics

How it works?

The LLM does not execute code. It proposes a tool call in a structured format (like JSON): “Call tool webSearch with { query: "…" }.”
Your app (the tool runner) reads that proposal, runs the tool, collects the result, and feeds the result back to the LLM as a special “tool” message. The LLM then uses that result to finish the answer (or to ask for another tool call).

So the loop is:

1. You send the conversation + tool definitions to the LLM.

2. The LLM either replies with a normal message or emits one or more tool calls.

3. Your code detects tool calls, executes them, then attaches their outputs.

4. You send those tool outputs back to the LLM.

5. The LLM produces the final, natural-language answer (or asks for more tools).

Code example

import Groq from "groq-sdk";
import { tavily } from "@tavily/core";

const groq = new Groq({ apiKey: process.env.GROQ_API_KEY });
const tvly = tavily({ apiKey: process.env.TAVILY_API_KEY });

async function main() {
  const messages = [
    {
      role: "system",
      content: `You are a smart personal assistant who answers asked questions.
You have access to the following tools.

The current date and time is: ${new Date().toLocaleString("en-IN", {
    timeZone: "Asia/Kolkata",  
    dateStyle: "full",
    timeStyle: "long",
  })}

1. webSearch({query}: {query: string}) // Search the latest information and realtime data on the internet.`,
    },
    {
      role: "user",
      content: "Today's weather in Ahmedabad?",
    },
  ];

  while(true){
    const completion = await groq.chat.completions.create({
        model: "llama-3.3-70b-versatile",
        temperature: 0,
        messages,
        tools: [
          {
            type: "function",
            function: {
              name: "webSearch",
              description: "Search the latest information and realtime data on the internet.",
              parameters: {
                type: "object",
                properties: {
                  query: {
                    type: "string",
                    description: "The search query to perform search on.",
                  },
                },
                required: ["query"],
              },
            },
          },
        ],
        tool_choice: "auto",
      });

      messages.push(completion.choices[0].message)

      const toolCalls = completion.choices[0].message.tool_calls;

      if (!toolCalls) {
        console.log(completion.choices[0].message.content)
        break;
      }

      for (const tool of toolCalls) {
        const functionName = tool.function.name;
        const parameters = tool.function.arguments;

        if (functionName === "webSearch") {
          const toolResult = await webSearch(JSON.parse(parameters));

          messages.push({
            tool_call_id: tool.id,
            role: 'tool',
            name: functionName,
            content: toolResult
          })
        }
  }
  }
}

async function webSearch({ query }) {
  const response = await tvly.search(query);
  const finalResult = response.results.map((result) => result.content).join("\n\n");

  return finalResult;
}

main();

Let’s walkthrough code and understand it.

import Groq from "groq-sdk";
import { tavily } from "@tavily/core";

const groq = new Groq({ apiKey: process.env.GROQ_API_KEY });
const tvly = tavily({ apiKey: process.env.TAVILY_API_KEY });

• Set up SDKs. One for the LLM (Groq), one for the search tool (Tavily).

const messages = [
  {
    role: "system",
    content: `You are a smart personal assistant...
The current date and time is: ... 
1. webSearch({query}...) // Search the latest information...`,
  },
  { role: "user", content: "Today's wheather in ahemedabad?" },
];

• messages is the chat history.

• The system message explains the role and lists available tools.

• The user asks a question (with a few typos—no problem).

const completion = await groq.chat.completions.create({
  model: "llama-3.3-70b-versatile",
  temperature: 0,
  messages,
  tools: [
    {
      type: "function",
      function: {
        name: "webSearch",
        description: "Search the latest information...",
        parameters: {
          type: "object",
          properties: {
            query: { type: "string", description: "The search query..." },
          },
          required: ["query"],
        },
      },
    },
  ],
  tool_choice: "auto",
});

• You send the conversation to the model plus a tool schema that describes webSearch(query: string).

• tool_choice: "auto" tells the model:
  “You can choose to call a tool or not—up to you.”

messages.push(completion.choices[0].message)
const toolCalls = completion.choices[0].message.tool_calls;

• You add the model’s message to the history.

• If the model wants a tool, it will fill tool_calls with something like:
  { id, function: { name: "webSearch", arguments: "{...}" } }.

if (!toolCalls) {
  console.log(completion.choices[0].message.content)
  break;
}

• If there are no tool calls, the model already gave the final answer. Print it and stop.

for (const tool of toolCalls) {
  const functionName = tool.function.name;
  const parameters = tool.function.arguments;

  if (functionName === "webSearch") {
    const toolResult = await webSearch(JSON.parse(parameters));
    messages.push({
      tool_call_id: tool.id,
      role: 'tool',
      name: functionName,
      content: toolResult
    })
  }
}

• For each requested tool call:

  1. Parse the tool’s arguments.

  2. Run your real function (webSearch).

  3. Push the result back as a special tool message that references tool_call_id.

• After this push, your while(true) loop runs again. You call the model again with the updated messages, so it can read the tool output and produce the final answer (or ask for another tool).

async function webSearch({ query }) {
  const response = await tvly.search(query);
  const finalResult = response.results.map(r => r.content).join("\n\n");
  return finalResult;
}

• Your tool implementation. It calls Tavily, merges snippets, and returns a text block to the LLM.

What’s happening?

1. The model sees: “What’s today’s weather in Ahmedabad?”

2. It thinks: “I need live data → call webSearch with a query like weather Ahmedabad today.”

3. Your code runs Tavily, collects results, and returns them to the model.

4. The model reads those results and answers in simple language.

The meaning of tool_choice: "auto" (and the “none” option)

• tool_choice: "auto": The model decides when to call a tool. This is the most natural setting. It improves answers for fresh/unknown facts and avoids calling tools when not needed.

• tool_choice: "none" (you wrote “note”; I’m explaining the likely intent): Force no tool calls. Useful when you want a purely “brain-only” answer (drafting text, explaining concepts) and you know the model already has everything it needs.

• Force a specific tool (provider-specific syntax): Sometimes you can instruct: “Use exactly this tool now.” This is helpful for strict workflows (e.g., always check the calendar first).

How we “pass tools” and “attach tool responses”

• Pass tools: You include a list of tool definitions (name, description, JSON schema) in the API call. You’re not passing function pointers; you’re passing contracts that explain what arguments a tool expects.

• Attach tool responses: After the model asks for a tool, you run it, then add a role: "tool" message that includes tool_call_id, name, and the content (the result). The model then treats that content like evidence and continues reasoning.

Understanding the Roles in messages

Every conversation has a role, which tells the model who is speaking and what context to follow:

• system → the rules of the game. Sets instructions, personality, or constraints.
  Example: “You are a polite weather assistant who always answers in simple English.”

• user → the person asking the question.
  Example: “What’s the weather today in Ahmedabad?”

• assistant → the AI’s replies. The model generates this role automatically.
  Example: “It is sunny and 27°C in Ahmedabad.”

• tool → responses from external functions you called.
  Example: “Tool result: Weather API says 27°C, humidity 40%.”

Think of this as actors in a play: system = director, user = audience, assistant = actor, tools = props.

Key Parameters in .create()

Now let’s explain those famous fields (temperature, top_p, etc.) in plain English.

temperature

Controls how creative or random the model is.

• Low (0–0.2) → safe, predictable, accurate. Best for coding, math, and tool calls.

• High (0.7–1.0) → more variety, storytelling, or creative text.

Example: Ask “Write a slogan for a coffee shop.”

• Temperature 0 → “Fresh coffee every morning.”

• Temperature 0.9 → “Awaken your senses with a cup of magic!”

top_p (nucleus sampling)

Controls how much of the probability space the model considers.

• top_p = 1.0 → consider all possible words.

• top_p = 0.9 → only keep the top 90% most likely words, ignore the rest.

Usually, you use either temperature or top_p, not both.

stop

Tells the model where to stop generating text.

Example:

stop: ["\nHuman:"]

This makes the model stop whenever it sees "\nHuman:". Useful if you want to cut output at certain markers.

max_completion_tokens

The maximum number of tokens (words + pieces of words) the model can generate in one reply.

• Small value → shorter answers.

• Large value → longer essays or detailed outputs.

presence_penalty

Encourages the model to talk about new topics instead of repeating what’s already in the conversation.

• High presence penalty = more diverse, less repetitive.

• Low (or 0) = model can repeat itself freely.

frequency_penalty

Reduces the chance of repeating the same words/phrases over and over.

• Example: Without penalty, it might say:
  “Yes, yes, yes, yes.”

• With penalty, it tries to vary:
  “Yes, of course. Absolutely.”

So together:

• temperature = creativity dial

• top_p = filter for word pool

• stop = cutoff point

• max_completion_tokens = response length limit

• presence_penalty = discourages repeating ideas

• frequency_penalty = discourages repeating words

Structured Output with response_format

Sometimes, you don’t just want plain text. You want the LLM to always return JSON that your code can parse safely. That’s where response_format comes in.

1. response_format: { type: "json_object" }

This forces the model to return a valid JSON object.

Example:

{
  "weather": "Sunny",
  "temperature": "27°C"
}

👉 Good when you just want any JSON, but don’t care about exact shape.

2. response_format: { type: "json_schema" }

This is more strict. You give the model a schema (like a blueprint) of what fields are required, their types, and constraints. The model must follow it.

Example schema:

response_format: {
  type: "json_schema",
  schema: {
    type: "object",
    properties: {
      city: { type: "string" },
      temperature: { type: "number" },
      condition: { type: "string" }
    },
    required: ["city", "temperature"]
  }
}

Output (LLM must match this):

{
  "city": "Ahmedabad",
  "temperature": 27,
  "condition": "Sunny"
}

This is super important when your app depends on predictable structure (like saving to a database or making API calls).

One-minute mental model

• LLM = brain

• Tools = hands (but only when you move them)

• Tool calling = the conversation where the brain politely asks you to use certain hands, you do it, and then the brain continues with better knowledge.

That’s the whole magic. Simple, powerful, and safe when you keep control.

Let’s break them down one by one.

1. Tokens – The Smallest Building Blocks

Think of a token as the LEGO block of language.

LLMs don’t directly understand human text like “What is ChatGPT?”. Instead, they break it down into tokens. A token can be:

• A whole word (like “what”)

• A part of a word (like “chat” + “gpt”)

• Or sometimes just punctuation (“?”)

This process is called tokenization, and it’s done by a special component called a tokenizer. Each token is then mapped to a number (an integer), because LLMs understand numbers, not raw text.

Why tokens matter:

• Pricing – When you use an AI API, pricing is usually based on tokens (e.g., $1 per 1M tokens). Both input and output tokens count separately.

• Efficiency – The fewer tokens your text uses, the cheaper and faster the processing.

So, next time you hear “token limit”, just remember it’s about how many LEGO blocks of language the model can handle.

2. Context – The Model’s Understanding of Your Input

Imagine you’re explaining something to a friend. The way they respond depends on the context—what you’ve said before, what instructions you give, and any extra details.

For an LLM, context = everything the model uses to understand your request. This usually includes:

• Your input / question → e.g., “Summarize this document.”

• Instructions → how you want the answer (bullet points, formal, casual).

• Additional information → company data, documents, or references you provide.

• Message history → previous conversation turns, so the AI “remembers” the flow.

Important: LLMs are stateless. They don’t truly remember past chats unless you provide that history again as context. That’s why chatbots pass message history in every request.

3. Context Window – The Memory Span of a Model

The context window is the maximum number of tokens a model can “see” and use at once while generating an answer.

• A small context window → The model forgets older parts of the conversation.

• A large context window → The model can keep track of long documents, history, and instructions without losing track.

For example:

• GPT-3 had a context window of ~2,000 tokens.

• GPT-4 can handle up to 128,000 tokens (almost a full book!).

Choosing the right model often depends on your context needs. If your input is too large for the context window, the model may “forget” earlier details or start skipping parts.

4. Inference – The Act of Thinking and Responding

Finally, we come to inference.

Inference is the process where the LLM takes your input (in tokens), processes it through its trained neural network, and generates an output.

You can think of inference as:

• The “reasoning process” of the model.

• The speed at which it responds. (Bigger models with more parameters may be slower, but smarter.)

Every time you interact with ChatGPT, Bard, or Claude—you’re essentially running an inference request.

Wrapping It Up

These four concepts may sound technical at first, but they’re actually quite intuitive:

• Tokens → The words/parts of words broken into numbers.

• Context → The information the model uses to understand your request.

• Context Window → How much information the model can “remember” at once.

• Inference → The actual process of generating output.

Once you get these basics, you’ll not only understand how LLMs work under the hood but also use them more effectively—saving cost, improving accuracy, and picking the right model for your needs.

When people first step into the world of AI, they often make a common mistake: they treat all AI models as the same. For example, while building an agentic system, many simply pick whichever model they have access to and plug it in. But here’s the truth—not every model is designed for every task.

Just like you wouldn’t use a sports car to plow a field, you shouldn’t use the wrong AI model for the wrong problem.

Why So Many Models?

At the core, AI models aren’t “magical” in themselves. They’re just neural networks trained on data for specific purposes. Depending on how they’re trained, some models are better at reasoning, while others are optimized for speed and efficiency.

That’s why the market has so many different AI models—each tuned for a unique use case.

Two Broad Categories of Models

1. Reasoning Models

Think of these as the “deep thinkers” of AI.

• They don’t just spit out an answer instantly.

• They go through a process of step-by-step reasoning or “chain of thought” before responding.

• This makes them ideal for complex, critical tasks where accuracy and logical planning matter.

Example: If you ask such a model to solve a multi-step math problem or plan a project timeline, it will carefully think it through instead of rushing.

Downside? They take a bit longer to generate results.

2. GPT-like Models (Fast Responders)

These are the “sprinters” of AI.

• They prioritize speed over deep reasoning.

• They generate output quickly without long internal deliberation.

• Perfect for tasks where fast responses are more important than multi-step thinking.

Example: Drafting emails, summarizing text, or quickly answering straightforward queries.

Downside? For critical decision-making tasks, they may give shallow or incorrect answers.

Important Note: These categories aren’t “official” standards. They’re a way to understand how different models behave. In reality, companies design models with different trade-offs depending on what they want to optimize—speed, accuracy, reasoning depth, or cost.

Big vs Small Models

Another dimension is model size, often measured in parameters (the internal weights that the model learns during training).

• Small Models (e.g., 8 billion parameters) → Faster, cheaper, but limited in intelligence. Great for lightweight tasks if paired with good prompting techniques.

• Large Models (hundreds of billions or even trillions of parameters) → More intelligent and context-aware, but heavier, slower, and more expensive to run.

With clever prompting and fine-tuning, even smaller models can often deliver surprisingly good results at a fraction of the cost.

Which Model Should You Use?

There’s no universal “best” model—it depends entirely on your use case:

• Critical, reasoning-heavy tasks → Go with reasoning models.

• Quick, repetitive tasks → GPT-like fast models are enough.

• Budget-sensitive use cases → Try smaller models with optimized prompting.

• Enterprise-grade applications → Large, advanced models might be worth the cost.

The Bottom Line

AI models are like tools in a toolbox. Some are hammers, some are screwdrivers. If you try to use one tool for every job, you’ll only make a mess.

• Reasoning Models → Accuracy & depth.

• GPT-like Models → Speed & efficiency.

• Small Models → Cost-effective with smart prompting.

• Large Models → High power, but resource-intensive.

The real skill lies not in just knowing AI, but in choosing the right model for the right task.

From Simple Predictions to True Understanding

Long before ChatGPT or Bard, there were models that could “generate” text. But they were nothing like today’s LLMs. Early systems were mostly statistical models, which predicted the next word based only on the previous word.

For example, if you typed “I like to drink …”, the model might predict “water” or “coffee” simply because those words often followed that phrase in the data it had seen.

But there were problems:

• Context was shallow → Models could only “look” at a few words behind.

• Accuracy was low → Predictions often felt random or repetitive.

• Memory was limited → They couldn’t hold on to long sentences or paragraphs.

Enter NLP and Recurrent Models

To improve text generation, researchers started using NLP (Natural Language Processing) with techniques like Recurrent Neural Networks (RNNs). These allowed models to handle sequences better—like remembering what was said a few words earlier.

But even RNNs had big limitations:

• They struggled with long-range dependencies (couldn’t remember words far back in the text).

• Training was slow and complex.

So while they were better, they still weren’t enough.

2017: The Breakthrough Moment

Everything changed in 2017, when Google published a groundbreaking paper: “Attention Is All You Need.”

This introduced a new architecture called the Transformer. Unlike earlier models, which read text word by word in sequence, Transformers use self-attention.

In simple terms: instead of just looking at the last word, a Transformer can look at all words in a sentence at the same time and figure out which ones are most important for predicting the next word.

This was a revolution because:

• It captured longer context → a paragraph, even pages, not just a few words.

• It was faster and more efficient to train.

• It scaled beautifully → the bigger the model, the smarter it got.

The Birth of LLMs

Based on this Transformer architecture, researchers built models that were trained on massive datasets: books, websites, research papers, Wikipedia, and more.

The result? Large Language Models (LLMs).

• These models have billions or even trillions of parameters (weights in their neural network).

• Parameters are like the “memory” of the model—it’s what the model learns during training.

• The more parameters, the more patterns it can capture, and the more powerful it becomes.

Examples of Today’s LLMs

Some of the most well-known LLMs include:

• OpenAI’s GPT-4 (the brain behind ChatGPT)

• Anthropic’s Claude

• Google’s Gemini

• Mistral and other open-source models

Some are private, some are open-source, but all share the same Transformer DNA.

So, What Does an LLM Actually Do?

At its core, the job of an LLM is simple yet powerful:

1. You provide input (a prompt) → e.g., “Who is the President of India?”

2. The model processes it using its neural network and context understanding.

3. It generates an output → “Droupadi Murmu is the President of India.”

That’s it. But when scaled to billions of parameters and trained on vast knowledge, this simple mechanism becomes the foundation for everything we call Generative AI today.

Why It Matters

Without LLMs, there would be no ChatGPT writing stories, no AI assistants summarizing reports, and no educational bots answering your doubts.

LLMs are the engine that powers modern AI applications. They are not just statistical text predictors anymore—they are context-aware, knowledge-rich, and creative collaborators.

In short:

• LLMs = Large Language Models = neural networks trained on text to generate text.

• Built on the Transformer architecture introduced in 2017.

• They are the foundation of Generative AI, powering tools like ChatGPT, Bard, and Gemini.

Imagine walking into an art gallery where every painting, sculpture, and piece of music was not created by humans but by machines. Sounds futuristic, right? Well, welcome to the world of Generative AI—a field of artificial intelligence that doesn’t just analyze data, it creates.

AI: More Than Just Classifiers

Traditionally, when we thought of AI, we pictured systems that could classify or detect patterns. For example:

• Spam detection in emails – Is this message spam or safe?

• Image classification – Is this a cat or a dog?

This kind of AI is powerful, but limited. It looks at data and makes a decision. That’s where Generative AI steps in and changes the game.

The Rise of Generative AI

Generative AI, often called Gen AI, is a special class of AI that can generate new content. Instead of only classifying existing information, it creates fresh material text, images, audio, even video.

The magic lies in its training process. These models are trained on massive datasets, and then they learn patterns so well that they can produce content that feels similar to what they’ve seen yet still new.

Examples We See Every Day

• ChatGPT → You type: “Write me a story about a time-traveling chef”. Within seconds, you get a creative, human-like story.

• DALL·E → You write: “Draw a cat in astronaut gear floating on Mars”. The AI paints the image for you.

• Sora (by OpenAI) → From just text, it generates full-blown videos.

• Text-to-Speech & Voice Cloning → AI that doesn’t just read aloud but can mimic your favorite actor’s voice.

• Music Generation Models → Compose background scores, songs, or beats, all from simple text prompts.

Why “Generative”?

The name says it all because this AI generates. Whether it’s words, visuals, sounds, or motion, generative AI goes beyond recognition. It creates something new, something we can interact with, something that feels alive.

The Bigger Picture

Generative AI is not just a cool party trick. It’s already transforming industries:

• Education – Personalized learning material.

• Healthcare – Simulating proteins for drug discovery.

• Entertainment – Movies, music, and gaming experiences tailored on demand.

• Business – Automating reports, designs, and even marketing content.

We are standing at the edge of a creative revolution where machines are not just calculators but collaborators.