Building a Multi-Tenant AI Phone Agent SaaS with Python FastAPI, Laravel, Twilio, Deepgram & Gemini

Most businesses lose customers on the phone. Calls go unanswered after hours, front desks get swamped, and callers who reach voicemail rarely call back. I built AI Callerto fix that: a multi-tenant SaaS where any company can point a phone number at the platform, upload their knowledge base as PDFs, and get an AI receptionist that answers calls in a natural voice — and only says things it can back up with the company's own documents. It's live in production at ai-caller.hassanrao.com.

This post walks through the architecture, the real-time audio pipeline, the retrieval-augmented generation (RAG) layer, and the anti-hallucination design that makes the whole thing safe to put in front of real customers.

What the platform does

A caller dials a business's number. Within a couple of seconds an AI agent greets them by the company's name, answers questions about services, prices, coverage and policies, tracks requests, captures leads, and hands off to a human when it should. Every word of the conversation is transcribed and stored, so the business can review calls in an admin dashboard.

Crucially, it's multi-tenant: one deployment serves many companies, each with its own phone numbers, documents, agent personality, voice, and even its own API keys if they want billing on their own accounts.

The architecture: two services, clear responsibilities

The system is split into two services that talk over an internal token-authenticated API:

• Laravel (PHP) — the brain of record. Multi-tenant admin built with Inertia and Vue: companies, plans, industries, phone numbers, agent configuration, document management, call logs and transcripts. It also owns the Twilio webhook: when a call comes in, Laravel resolves the tenant, validates the request signature, creates the call record, and returns TwiML.
• Python (FastAPI) — the real-time voice service. A WebSocket bridge between Twilio Media Streams and Deepgram's Voice Agent API, plus the RAG pipeline: PDF ingestion, chunking, Gemini embeddings, and pgvector similarity search.

PHP is excellent for the CRUD-and-billing side but not for holding thousands of long-lived WebSocket connections pumping binary audio frames. Python's asyncio handles that naturally. Splitting along that line means each service does what its runtime is best at.

The life of a phone call

1. Twilio receives the call and hits Laravel's voice webhook. Laravel looks up the called number, verifies the X-Twilio-Signature, checks that the company is active and its agent is enabled, and creates a call record.
2. Laravel responds with TwiML: <Connect><Stream> pointing at the Python service's WebSocket endpoint. From this moment, raw call audio (mulaw, 8kHz) flows to Python as base64 frames.
3. On the stream's start message, Python takes the callSidand asks Laravel's internal API for the full call context: the tenant's system prompt, LLM model, voice, temperature, greeting, and the functions the agent may call. Tenant resolution is server-to-server — the caller can't influence it.
4. Python opens a second WebSocket to Deepgram's Voice Agent API and sends a settings message assembled from that context. Deepgram handles the whole speech loop in one connection: Nova-3 speech-to-text, a "think" LLM (Gemini 2.5 Flash by default), and Aura-2 text-to-speech.
5. The bridge then pumps audio both ways and reacts to events: transcripts are posted back to Laravel as they happen, and when the caller starts speaking over the agent, the bridge sends Twilio a clearmessage to drop queued agent audio — that's what makes interruptions ("barge-in") feel natural instead of the agent talking over you.
6. When the agent needs facts, Deepgram emits a function call request. Python executes it — usually a knowledge base search — and returns the result for the LLM to speak from.

RAG: grounding every answer in the tenant's documents

Each company uploads PDFs — price lists, service descriptions, policies. The ingestion pipeline extracts the text, chunks it with a paragraph-aware sliding window (~1,500 characters with 200 overlap), embeds each chunk with Gemini's embedding model, and stores the vectors in PostgreSQL with the pgvector extension. No separate vector database to run — cosine similarity search lives right next to the relational data, with every query scoped by company_idso tenants can never see each other's content.

One detail that measurably improves retrieval: Gemini lets you embed documents and queries with different task types (RETRIEVAL_DOCUMENT vs RETRIEVAL_QUERY), optimizing the two sides of the search separately. Results below a minimum similarity score are discarded rather than passed to the model — a weak match is worse than no match.

The anti-hallucination design

An AI that invents prices over the phone is a liability, not a product. The platform defends against that in three layers:

1. A mandatory search function

The agent is given a search_knowledge_base function and the system prompt requires calling it before answering anythingfactual — services, prices, delivery times, policies. The function's results are the only permitted source for factual claims. If retrieval returns nothing, the function replies NO_RELEVANT_INFORMATION_FOUNDand the agent must say "I don't have that information on hand" and offer a human — never guess.

2. Prompt-injection defense

Retrieved chunks are wrapped in tags and explicitly labelled as reference data, never instructions. If someone uploads a PDF containing "ignore your previous instructions", the agent has been told — at the system prompt level, which wins — to treat it as content, not commands.

3. Conservative generation

Low temperature, short spoken answers (one to three sentences — it's a phone call, not an essay), and industry-specific guardrails: healthcare agents never give diagnoses, service companies never quote binding prices that need an estimate.

The admin panel includes a test console — a text chat that runs the exact same system prompt, retrieval, and grounding rules as the phone agent. Businesses can interrogate their agent before a single customer calls it.

Multi-tenancy: industries as AI-generated templates

Every company belongs to an industry, and each industry has a base prompt template with rules tuned to what its callers actually ask. The interesting part: when a company signs up in an industry that doesn't exist yet, the platform generates the industry template with an LLM— instructed to keep every safety rule from a reference template, adapt the scope rules, and add two or three industry-specific guardrails. The output is validated programmatically (the grounding function and required placeholders must survive) before it's accepted.

Tenants can also bring their own keys: a company's own Gemini key runs embeddings and generation on their quota, and BYO Twilio numbers are validated against the company's own auth token instead of the platform's.

Lessons from real-time voice

• Silence kills connections. When a caller goes on hold or mute, no audio flows and Deepgram times out the session. An 8-second keepalive ping keeps the agent alive through the gaps.
• Barge-in is a product feature, not a nicety. Without clearing Twilio's audio buffer the moment the caller speaks, the agent finishes its sentence over them and the conversation immediately feels robotic.
• Fail loud, and specifically. Free-tier LLM quota exhaustion surfacing as a generic 500 gets misdiagnosed as "the service is down." Mapping a 429 to an actionable message ("enable billing on the key or wait for reset") saved real debugging time.
• Every call must end in a known state. Whatever happens — Deepgram drops, Twilio disconnects, an exception mid-call — a finallyblock reports the call as completed, transferred, or failed, so no call is ever stuck "in progress" in the dashboard.

Deployment

The whole platform — nginx, Laravel, the Python voice service, and PostgreSQL with pgvector — runs on a single AWS EC2 instance behind Let's Encrypt TLS, provisioned by shell scripts. A t3.smallcomfortably handles launch traffic; the heavy lifting (speech models, LLM inference) is on Deepgram's and Google's side, so the server mostly shuffles WebSocket frames. Scaling later means resizing the instance or splitting the voice service out — the internal API boundary already makes that a config change, not a rewrite.

The stack, summarized

Building AI Caller reinforced something I keep seeing in production AI work: the model is the easy part. The product is the plumbing around it — tenant isolation, grounding guarantees, graceful failure, and an admin experience that lets a non-technical business trust what their agent will say.