Halfway through building MoveAI, our AI dispatcher for a moving company, we realized we'd architected ourselves into a corner. The bot could hold a conversation, collect job details, and hand off to a human dispatcher — but it couldn't remember anything between sessions. Every time a customer returned to change their move date, the bot started from scratch. The dispatcher still had to dig through Telegram history manually. We'd automated the easy part and left the painful part untouched.

That's the most common failure mode in AI chatbot development: solving the demo problem instead of the operational problem. This guide is about the decisions that separate a chatbot that impresses in a pitch from one that actually reduces workload.

The First Decision Is Not "Which LLM"

Before you pick a model, you need to know what kind of conversational task you're actually solving. We've found that most chatbots fall into one of three categories, and each has a different architecture profile:

Structured intake — the bot needs to collect specific fields (name, date, address, inventory) and hand them off to a downstream system. MoveAI is this. The failure mode is freeform conversation that never converges on the data you need.

Stateful service — the bot needs to remember a user's history, preferences, or status across sessions. TER.A Coffee, our loyalty bot for a specialty coffee shop, is this. Users ask how many stamps they have, whether their reward is ready, what they got last time. The failure mode is treating each message as a fresh conversation.

Open-domain assistant — the bot needs to handle wide, unpredictable queries. This is actually the hardest and usually the least appropriate starting point for a real product. If you're building something for a specific business, you almost certainly don't need this.

Why does this matter before choosing a model? Because structured intake bots benefit heavily from function calling and strict output schemas, while stateful service bots live or die on how you design your persistence layer. Picking GPT-4o first and figuring out architecture second means you'll retrofit the plumbing after the walls are already up.

State Management Is Where Most Bots Break

For TER.A Coffee, we built persistence on top of Supabase with a straightforward schema: one row per user, storing their stamp count, last visit timestamp, and reward tier. Reads happen on every message; writes happen on confirmed purchases. Simple, and it worked at 325+ active users with zero infrastructure cost because Supabase's free tier handled the read volume without complaint.

For MoveAI, the state problem was more complex. A moving job has a lifecycle — inquiry, quote, confirmed, completed — and the bot needed to know where in that lifecycle each customer was. We used Redis to store session state (what the bot asked last, what fields were still missing) and Supabase for durable job records. Redis gave us sub-millisecond context retrieval during active conversations. Supabase gave us queryable history that the dispatch team could actually use.

The mistake we made early was storing conversation state in the LLM's context window. We passed the last 10 messages with every API call, assuming that was "memory." It worked until conversations got long — token costs climbed, and more importantly, the model started getting confused by earlier contradictory information in the thread. A customer who said "I'm moving on June 15th" in message 3 and then "actually, June 22nd" in message 11 would sometimes get a confirmation for June 15th. The model wasn't ignoring the correction; it was averaging across conflicting signals.

The fix was extracting structured state explicitly and injecting only the current state into the system prompt, not the raw conversation history. The system prompt would say: Move date: June 22nd (updated). Origin: confirmed. Destination: pending. This was more reliable than full transcript injection, and it cost roughly 60-70% fewer input tokens per call (our estimate from eyeballing usage dashboards — we didn't run a formal benchmark).

Tool Calls vs. Prompt Engineering: When Each Makes Sense

LangChain was our first instinct for MoveAI because it offered ready-made chains and agent abstractions. We ended up stripping most of it out.

The problem wasn't that LangChain is bad — it's that its abstractions made it hard to control exactly what was being sent to the model. When a dispatcher reported that the bot confirmed a job with an address it had never actually verified, we spent two hours tracing through LangChain's internals to find where the prompt was being constructed. After that, we moved most of the orchestration logic to plain Python functions with explicit prompt templates. We still used LangChain's document loaders for a retrieval component (a FAQ the bot could reference), but the core conversation loop was ours.

OpenAI's function calling — now called tool use — is the right abstraction for structured intake. Instead of parsing freeform model output with a regex or hoping the model formats JSON correctly, you define the schema of what you want and the model fills it in. For MoveAI, we defined tools like update_move_date, confirm_origin_address, and flag_for_human_review. The model would call these tools as it gathered information. When flag_for_human_review was called, the conversation immediately escalated to a human dispatcher. This single mechanism is what got us to a 70% reduction in dispatcher handling time — not the conversation itself, but the reliable handoff logic.

Prompt engineering alone isn't sufficient for structured intake. You will get hallucinated confirmations. You will get fields silently skipped. Function calling enforces the contract at the API level.

Deployment Constraints Shape Architecture More Than You'd Expect

TER.A Coffee runs on Render's free tier. One service, one process, polling the Telegram Bot API with python-telegram-bot. No webhooks, no queue, no separate worker. For a loyalty bot handling a few dozen messages per hour across a small café's customer base, this is not a compromise — it's the right call. Webhooks would have required a public HTTPS endpoint and more operational overhead for zero functional benefit at that scale.

MoveAI runs on a paid Render instance with a webhook endpoint and a simple task queue using Celery and Redis. Why the difference? Moving jobs have SLAs. If a customer submits a move request at 9am and the bot doesn't respond until 9:45 because Render spun down the free instance, that's a business problem. The polling model is fine for low-stakes interactions. For anything where response latency affects trust, pay for an always-on process.

The other deployment decision that bit us: we initially stored uploaded photos of apartments (for estimating move complexity) in the Telegram servers via file_id. Telegram's file_id links expire and aren't guaranteed to be stable across bot restarts. When we tried to retrieve them for the dispatch team days later, some were gone. We moved to downloading files immediately on receipt and storing them in Supabase Storage. Obvious in hindsight, but not something any tutorial warned us about.

Evaluation Before You Scale

The hardest part of AI chatbot development isn't building the bot — it's knowing whether it's working. We didn't have a formal eval setup for TER.A Coffee because the stakes were low and the logic was simple enough to verify manually. For MoveAI, we needed something more rigorous.

We built a small eval suite using a set of 40 synthetic conversations representing common scenarios: customers who changed their minds mid-conversation, customers who gave ambiguous addresses, customers who asked about pricing (which the bot wasn't supposed to answer). We ran these against every prompt change before deploying. The suite didn't use a framework — it was a Python script that called the bot logic directly, compared the extracted job fields against expected values, and flagged regressions.

This is not sophisticated. But it caught three separate regressions during prompt iteration that would have caused real dispatch errors. If you're shipping a chatbot into a workflow that affects real operations, some form of regression testing on representative conversations is not optional.

The question worth asking before you start building: what does a silent failure look like for your bot, and how would you know it happened? For MoveAI, a silent failure was a job confirmed with a wrong date. We found out about it from a dispatcher complaint, not from our monitoring. We added explicit logging of every field extraction after that. Define your failure modes before deployment, not after.

What we haven't fully solved is evaluating tone and recovery behavior — how the bot handles confused or frustrated users. That's still largely manual review of flagged conversations. If you have a systematic approach to evaluating conversational quality at scale without human review, we'd genuinely like to know what you're using.