How Do Chatbots Work?

The five stages of a single request

Almost every bad chatbot answer is one specific stage failing rather than the whole system being wrong. Learning to tell which stage broke is the most useful thing on this page.

1. Input processing

The message arrives from a widget, a Slack event or a channel API. Before anything intelligent happens it is normalised: markup stripped, language detected, and the text tokenised. Tokens are the units a language model actually reads. They are common character sequences rather than words, so a short common word is usually one token and an unusual one is split into several, and everything downstream is measured and priced in them.

2. Intent understanding

The system decides what kind of request this is, because different kinds need different handling. Asking about delivery times is a knowledge question. Asking for a human is an escalation. Typing an email address into a form prompt is neither. Classification is done either by a small dedicated model, which is fast, cheap and returns a confidence score you can threshold on, or by asking the language model itself to route. Dedicated classifiers stay in the stack because routing benefits from being predictable and measurable, which is why matram.ai runs one in front of generation. Routing matters more than it sounds: a question that should have reached a person but got answered generically is a worse failure than a wrong fact, because nobody notices it happened.

3. Retrieval

If the message is a knowledge question, the system searches an indexed copy of your content for the passages most likely to contain the answer, and it searches on meaning rather than shared keywords. Question and stored passages all exist as vectors, and the passages sitting closest to the question are pulled back, usually several rather than one, because answers often span two documents. This stage decides quality. A model handed the right three paragraphs almost always writes a good answer, and no amount of prompt engineering rescues a retrieval failure.

4. Generation

The retrieved passages, the system instructions, the recent history and the current question are assembled into one prompt and sent to a large language model, which produces the reply one token at a time. Nothing about that answer existed until this moment. The instructions carry most of the weight: answer only from the supplied passages, say plainly when they do not cover the question, cite the source of each claim. Strict grounding is a configuration choice, not a property of the model.

SYSTEM
Answer only from CONTEXT. If CONTEXT does not cover the question, say so
and offer to pass the visitor to a person. Cite the source URL you used.

CONTEXT
[1] /delivery   "Orders confirmed before 16:00 are dispatched the same
                 working day. Ireland and the EU are served from the
                 Rotterdam warehouse."
[2] /returns    "Unworn items may be returned within 14 days."

HISTORY
user:      do you ship to ireland
assistant: Yes, Ireland is covered by our EU delivery zone. [/delivery]

USER
and how fast is it

That is roughly what the model receives. Read it and the behaviour of every AI chatbot you have used becomes predictable: if the answer is not in CONTEXT, the model is being asked to write about something it cannot see.

5. Delivery and logging

Tokens are streamed back as they are produced, which is why replies appear word by word, formatted for wherever they are going, since a website widget accepts different markup from Slack or Messenger. Then the conversation is logged. Buyers underrate this step. The list of questions the bot could not answer is the raw material for improving the system, and in a retrieval-based chatbot every one of them points at a gap in your documentation, which means the fix is content rather than engineering.

The technology stack, term by term

These words get used interchangeably in marketing copy and they are not interchangeable. Each names a distinct layer, and knowing which layer a vendor is describing tells you what they have actually built.

NLP (natural language processing)

NLP is the broad field concerned with getting computers to process human language, covering everything from splitting text into sentences to translating it.

It is the umbrella, not a component you can buy. A product described as powered by NLP has told you almost nothing, because tokenisation, spam filtering, translation and text generation are all NLP. The useful question is which specific tasks the system performs, and with what.

Examples: Tokenisation, Language detection, Named entity recognition, Sentiment scoring

NLU (natural language understanding)

NLU is the subset of NLP that extracts structured meaning from a sentence, principally the user's intent and any entities mentioned.

"Cancel my order 4471" becomes intent: cancel_order, entity: order_id = 4471, and that structured output is what lets a system route a request or call an API. In pre-2022 platforms NLU was the whole product, and building one meant supplying dozens of example phrasings per intent. Language models do this implicitly, which is why modern bots need no training phrases.

Examples: Intent classification, Slot filling, Coreference resolution

Large language model (LLM)

An LLM is a neural network trained on very large amounts of text to predict the next token, which turns out to be enough to write coherent, context-appropriate prose.

The model holds a general command of language and a broad but imprecise memory of its training data. It has no knowledge of your business, no access to your database and no awareness of today's date unless the prompt supplies them. Everything specific it says correctly about you got there because your system put it in front of the model at request time.

Examples: Claude Opus 4.8, Claude Sonnet 4.6, GPT-5.6-terra, Gemini 2.5 Pro

Embeddings

An embedding is a list of numbers representing a piece of text, positioned so that text with similar meaning ends up close together in that numerical space.

An embedding model turns a passage into a vector of several hundred or a few thousand numbers. The individual numbers mean nothing readable; what matters is relative position. "What is your refund window" and "how long do I have to send something back" share almost no words, yet their vectors sit near each other, because the model was trained so that similar meanings occupy similar regions.

Closeness is usually measured by cosine similarity, which compares the direction two vectors point in rather than their length, so a one-line answer and a long paragraph on the same subject still count as similar. This is the entire reason a retrieval-based chatbot handles phrasings nobody anticipated. It is not matching your words, it is matching your meaning against stored meaning.

Examples: Semantic search, Deduplication, Clustering support tickets by topic

Vector database

A vector database stores embeddings alongside their original text and metadata, and finds the nearest vectors to a query fast enough to sit inside a live request.

Comparing a query against every stored passage does not scale, so these systems use approximate nearest neighbour indexes that trade a little recall for a large speed gain. Each vector is kept with its passage text, source URL, title and timestamp, which is what makes citation possible later. Good implementations run keyword search alongside vector search, because exact strings such as product codes and error numbers are a weakness of semantic matching.

Examples: pgvector, Pinecone, Qdrant, Weaviate, Elasticsearch hybrid search

RAG (retrieval-augmented generation)

RAG is the pattern of retrieving relevant source passages first and giving them to a language model to answer from, instead of relying on what the model absorbed during training.

RAG is what makes a general model useful for a specific business. Answers stay current, because updating one means editing a document rather than retraining anything. They are attributable, because the system knows which passage produced which claim. And invention narrows, because the model is summarising supplied text rather than recalling facts. Nearly every credible business chatbot sold today is a RAG system with a widget on the front.

Examples: matram.ai, Intercom Fin, Zendesk AI agents, Most internal help-desk bots

NLG (natural language generation)

NLG is the production of human-readable text by a machine, which in current systems means the language model writing the reply token by token.

The term predates language models. It used to describe template filling, where a program slotted values into pre-written sentences, as weather and financial summaries were produced for years. In a modern chatbot NLG and the LLM are the same step, which is why the term is rarely used on its own now.

When this pipeline is the wrong thing to build

Everything above is machinery, and machinery is only worth assembling at certain points. Here are four, in the order most businesses meet them.

Stage one: a page of text beats a pipeline

If your entire body of knowledge is one FAQ page and a delivery table, retrieval has nothing to do. Embedding six paragraphs so that a model can quote one of them back to a visitor is a search box with an inference bill attached. Write the FAQ properly and link it from the header. The architecture starts paying for its own complexity when there is more content than one person can hold in their head, and not before that.

Stage two: the questions repeat, and your content contradicts itself

Repetition is the signal people wait for and it is not sufficient on its own. Retrieval inherits your documentation exactly as it stands. If the delivery cut-off appears on three pages with two different times, all of them go into the index, and which one comes back depends on which chunk happened to score higher for that particular phrasing. You have not built an answering machine. You have built a fluent, well-cited way of exposing an inconsistency you already had. Fix the pages first. That work is not wasted, because it is the same work either way.

Stage three: the manual version becomes the expensive one

At some point the arithmetic flips. Questions arrive faster than anyone can read them, they arrive at hours nobody is staffed for, and the answers they need sit in documents that are current and agree with each other. Now the pipeline does something a person genuinely cannot: it reads every page in the index for every question, in about a second, at three in the morning, in whatever language the question was typed in. That is the condition this architecture was designed for. It is narrower than the marketing suggests.

Stage four: the question whose answer is not in any document

And then there is the request that defeats the design regardless of volume. Where is my order. Is that jumper in stock in the Leeds shop right now. Neither answer lives in a page that can be crawled, chunked and embedded, because it changes by the hour and belongs to a system rather than to your website. A retrieval chatbot pointed at your content will answer from the closest matching page, which is the wrong shape of thing entirely and will be confidently out of date. Those need an integration to whatever holds the live state, or a handover, and treating them as a content problem is how a good pipeline gets blamed for a design mistake.

How a chatbot actually learns your content

This is the stage most explainers get wrong. Training a chatbot on your website does not modify any model. Nothing about the language model changes. What you are building is a searchable index that gets consulted at request time.

The confusion is fair, because every product in the category uses the word train in its interface. But fine-tuning, which genuinely does adjust model weights, is a different and far more expensive process, it teaches style and format rather than facts, and it goes stale the moment your prices change. Retrieval is what business chatbots use, and it is the right choice: your content changes weekly, model weights should not.

Ingestion

Content is collected from wherever it lives: a crawled URL, a sitemap, uploaded PDFs and Word files, or connectors into Notion, Google Drive, Confluence, SharePoint and existing help centres. The work here is extraction rather than transfer. HTML has to be stripped of navigation and footers or every chunk carries the same menu text, and PDFs have to be converted into a sensible reading order, which is harder than it sounds for multi-column layouts. Bad extraction quietly ruins retrieval and stays invisible until you inspect what was stored.

Chunking, with overlap

Whole documents are the wrong unit. A twelve-page policy retrieved in full spends most of the model's attention on irrelevant sections, and a single sentence retrieved alone lacks the context to be understood. So documents are split into passages of a few hundred tokens, ideally along heading boundaries so each chunk is about one thing.

Chunks are given an overlap, meaning the end of one repeats at the start of the next. Without it a definition split across a boundary is destroyed: the term sits at the bottom of chunk 7 and its explanation at the top of chunk 8, and neither answers the question alone. Overlap costs a little storage and prevents a whole class of silent failure.

/returns-policy   (1,850 words)

chunk 07  tokens    0 –  420   "Faulty items…"        overlap → 60 tokens
chunk 08  tokens  360 –  780   "…collected free of charge within the UK."
chunk 09  tokens  720 – 1140   "Exchanges are processed once…"

stored per chunk:
  { text, embedding: float[], url, title, heading, updated_at }

Embedding, indexing and retrieval

Each chunk passes through the embedding model and is stored as a vector with its text and metadata. That metadata is not decoration: the URL is what lets an answer cite its page, and the timestamp is what lets you find content that has drifted. When a question arrives it goes through the same embedding model, so question and chunks share one space and compare directly. The index returns the closest handful, and better systems rerank those candidates with a slower, more accurate model, then discard anything below a similarity threshold.

Discarding weak matches is what produces an honest "I do not have that information" rather than a confident guess. It is a configuration decision, and it is worth asking any vendor how theirs behaves when nothing relevant is found.

Grounded generation and citation

The surviving chunks go into the prompt with instructions to answer from them and nothing else. Because each arrived with its source URL attached, the reply can point at the page it used, and that citation is the practical difference between a chatbot you can audit and one you have to trust. When an answer is wrong you open the cited page, fix the sentence that misled it, and the bot corrects itself on the next crawl.

matram.ai works this way end to end: crawl a URL or import a sitemap, upload PDFs and DOCX files up to 10MB, or connect Notion, Google Drive, Confluence and Zendesk, then answers are generated by Claude from the retrieved passages with the source page cited, and strict mode refuses anything your approved content does not cover. For a shop that means the product and delivery pages you already maintain become the knowledge base, with no script to write. The full ingestion list is on the features page.

How a chatbot follows a conversation

Language models have no memory between calls. Every appearance of continuity is manufactured by the surrounding system, and knowing how explains both why long conversations get expensive and why bots lose the thread.

On each turn the application resends the conversation. The prompt is rebuilt from scratch: system instructions, freshly retrieved passages, recent message history, then the new question. The model reads the whole thing as if for the first time, because for the model it is. All of it has to fit inside the context window, the maximum number of tokens a model can consider at once, and that window is a shared budget: long instructions, many retrieved chunks and a long history compete for the same space, and the reply needs room too. Since history is resent every turn, a conversation gets more expensive as it runs, which is why caching repeated prefixes is standard.

When the window fills

• Truncation. The oldest turns are dropped, which is why a bot can forget something you said early in a long chat.
• Rolling summary. Older turns are condensed into a short recap that stays in the prompt while the detail is discarded.
• Retrieval over the transcript. The conversation is itself indexed, and earlier turns come back only when relevant.

The follow-up problem

Here is a failure worth knowing, because it explains a lot of frustrating bot behaviour. A visitor asks "do you ship to Ireland", gets an answer, then asks "and how fast is it". Embedded on its own, that second message is about nothing, and searching an index for it returns noise. The fix is query rewriting: before retrieval runs, the system uses the recent history to rewrite the message into a standalone question, in this case something like "how fast is delivery to Ireland", and embeds that instead. Systems that skip this step handle first questions well and follow-ups badly, which you can spot in a trial within about five messages.

Between sessions

What survives after the visitor closes the tab is a product decision, not a model capability. Whether a returning visitor is recognised depends on identity: an anonymous browser can be matched by cookie at best, whereas a logged-in user or a Slack account gives a stable identifier. Any chatbot that claims to remember a customer is describing its own database, not the model.

The models doing the generating

The generation step calls a model from one of a small number of providers, and the choice affects quality, cost and speed. These are the published list prices on 20 July 2026, per million tokens of input and output.

Read that table rather than sorting it. These are list rates before batching and prompt caching, both of which move the real figure a long way, and a token is not a standard unit across vendors: models tokenise differently and are differently verbose, with Anthropic's own documentation noting that its newer models emit around 30% more tokens for the same text. Comparing per-token rates across vendors will mislead you. The only reliable comparison is running your own traffic through both and reading the bill.

Almost none of this reaches you as a buyer of a hosted chatbot, because the vendor absorbs inference cost inside a subscription. It matters when you are considering building your own, where every conversation is a variable cost. The chatbot ROI calculator and the development cost breakdown work through that arithmetic, and the Intercom comparison shows how the same pipeline gets priced at the other end of the market. matram.ai generates with Claude and prices flat, so conversation volume does not change your bill.

What chatbots genuinely cannot do

Every limitation below follows from the architecture described above. None is fixed by a better model, and a vendor who says otherwise is selling.

Answer from content that does not exist

This is the dominant cause of disappointing deployments and it is rarely diagnosed correctly. If your refund window is not written down anywhere, retrieval finds nothing and no model rescues it. Teams read that as "the AI is not smart enough" when the real finding is that a policy lives in a manager's head. Read the unanswered-question log in the first fortnight and the gaps name themselves.

Notice that a source is out of date

Retrieval is a similarity search, not a truth check. A stale pricing page in the index will be quoted confidently and cited properly, and the citation makes the wrong answer look more credible. The index inherits your content exactly, contradictions included: two pages disagreeing about a delivery cut-off produce a bot that disagrees with itself depending on which chunk scored higher.

Calculate reliably

A language model predicts plausible text, and arithmetic is not a text-plausibility problem. Anything numerically consequential, such as tax, shipping bands or pro-rated refunds, should be computed by ordinary code or fetched from your systems through an API integration and handed to the model to relay, not worked out by the model.

Guarantee wording

Generated text varies between runs. For most support questions that is fine and even desirable. For a regulated disclosure, a medical instruction or a binding term it is unacceptable, and the correct design is fixed text delivered by a rule, with the model routing to it rather than paraphrasing it.

Act on its own initiative

A chatbot answers a message and stops. Anthropic draws the useful line: agents are "systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks", while workflows are "systems where LLMs and tools are orchestrated through predefined code paths". Retrieval chatbots, matram.ai included, are the second thing, following a fixed path of retrieve then generate. Vendors blur that line constantly, so the difference between an AI agent and a chatbot is worth reading before you buy against an agent pitch.

Judge when a person is needed

Not reliably. Distress, legal threat and a customer on their fourth attempt at the same problem are all poorly served by a technically correct answer. Escalation triggers are a business decision to configure deliberately, and the systems that feel best to use are usually the ones that hand over earliest.

What getting this wrong costs, long after launch

The subscription is the visible cost and it is rarely the one that decides whether this was a good idea. A few things go wrong quietly, and none of them reach an invoice.

The first is the customer who acted on an answer that was wrong. Retrieval checks similarity, not truth, so a paragraph that stopped being accurate when the policy changed gets quoted back with a link to itself attached. That link is the problem. A cited answer reads as verified, so nobody double-checks it the way they would double-check a stranger on live chat. They book the slot. They order the part. The cost is not the refund, it is that your own system produced the error in writing and a person now has to explain that to somebody who is entitled to be annoyed.

The second is trust, and you get to spend it once. A visitor who is told something wrong rarely writes in to correct it. They close the window and stop asking, and every question they might have asked afterwards simply never arrives. Your logs record that conversation as ended without escalation, which in most dashboards is indistinguishable from a success. This is the strongest argument for reading real transcripts by hand in the early weeks instead of watching a resolution rate, because the failure you most need to see is the one that generated no complaint.

The third is content debt, and it is structural rather than accidental. Every unanswered question the bot logs is an instruction to write something, and the index is only ever as current as the last crawl of a site that keeps changing underneath it. So somebody has to own the loop: read the gaps, write the page, confirm the answer changed. Nobody is assigned that at launch, because at launch the thing works. Months later you have a confident, well-spoken, properly cited description of a business you no longer run. Which is the question worth settling before you buy: who owns the content this thing reads, and what happens in the weeks when they are busy?

Frequently asked questions

Sources

• Weizenbaum, J. "ELIZA", Communications of the ACM, 1966 - ELIZA's date, authorship and pattern-matching design.
• Anthropic, "Building effective agents" (19 Dec 2024) - The quoted definitions of agents and workflows used in the limitations section.
• OpenAI API pricing - List prices for gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna and gpt-5.4-nano, 20 July 2026.
• Anthropic pricing - List prices for Claude Opus 4.8, Sonnet 4.6 and Haiku 4.5, and Anthropic's note that its newer models emit around 30% more tokens for the same text.
• Google Gemini API pricing - List prices for Gemini 2.5 Pro, Gemini 3.5 Flash and Gemini 2.5 Flash-Lite, 20 July 2026.