Every chatbot proposal we are asked to review costs the same way: someone finds the per-million-token price, multiplies by an estimated message volume, and writes down a number. Then the thing ships, and the actual bill arrives with four line items nobody planned for.
This is not an argument against building one. It is an argument for costing it honestly, because a chatbot that gets switched off in month three because it surprised the finance team is worse than one that was never built.
The five things you are actually paying for
| Cost line | What drives it |
|---|---|
| Inference | Tokens in and out. Input is usually larger than teams expect, because retrieved context and system prompts are resent on every turn. |
| Retrieval and storage | A vector database, priced on stored dimensions and queries. Small at launch, grows with your knowledge base, not your traffic. |
| Logging and traces | Every conversation stored for debugging and audit. This is what makes the system fixable, and it is billed per GB ingested. |
| Human review | Someone reading a sample of conversations weekly. This is a real salaried cost and the one most often omitted entirely. |
| Idle infrastructure | Hosting, queues, monitoring. Runs whether anyone chats or not. |
Why the inference estimate is always low
The usual mistake is costing a single message. In practice a chatbot turn resends the system prompt, the retrieved documents, and some conversation history. A 200-token question can become a 4,000-token request before the model sees it.
Multiply that across a conversation and the ratio of input to output tokens is frequently ten to one or worse. If you are budgeting from output length, you are budgeting for a tenth of the request.
- Count the whole request, not the user's message.
- Assume conversations are multi-turn. Single-shot questions are the exception.
- Cache aggressively: identical system prompts and retrieved chunks are the cheapest thing to stop resending.
- Route by difficulty. Most support questions do not need your most expensive model.
The line item nobody budgets
Human review is the one that gets cut from the estimate and then reappears as an unplanned demand on someone's week. A production chatbot that nobody reads is a system with no feedback loop, and it will quietly degrade as your product, pricing and policies change underneath it.
A chatbot is not a project you finish. It is a system you operate, and operating it has a staffing cost.
The realistic version is a couple of hours a week: sample the conversations, tag the failures, fix the retrieval or the prompt, and note what the bot should have escalated. That is not overhead. That is the mechanism by which the thing gets better.
How to bring the number down
- Scope the knowledge base to what people actually ask. Indexing every document you own inflates retrieval cost and reduces answer quality at the same time.
- Set a token ceiling per conversation and handle it deliberately, rather than discovering it as a bill.
- Escalate early. A bot that hands off cleanly at the edge of its competence costs less and damages trust less than one that improvises.
- Sample your logs rather than retaining everything at full fidelity forever. Keep enough to debug, not enough to fund a data lake.
What to ask before you commit
If you are being quoted for a chatbot, ask what the monthly running cost looks like at ten times launch volume, and ask which line items scale with traffic versus with content. If the answer only covers model calls, the estimate is incomplete.
We scope this before anything gets built, because the operating cost frequently changes what is worth building. That conversation is free, and it is better had early.