- Live inbound calls answered
- 24/7Live inbound calls answered
- Self-serve signup and plans
- Multi-tenantSelf-serve signup and plans
- Telephony through to billing
- Solo buildTelephony through to billing
The challenge
Businesses lose customers on the phone — calls go unanswered after hours, front desks get swamped, and callers who reach voicemail rarely call back.
A generic voice bot makes it worse. Confidently invented answers about pricing, coverage or policy cost more trust than a missed call ever would, and real-time voice leaves no room for a slow retrieval step.
The approach
I engineered the real-time voice pipeline as a bridge from Twilio Media Streams to Deepgram's agent API (STT → LLM → TTS) over WebSockets, with barge-in interruption handling and LLM function calling for shipment tracking, quote capture and human transfer.
The grounding layer chunks and embeds PDF knowledge bases with Gemini into pgvector using HNSW cosine indexes. Retrieval is tenant-scoped with relevance thresholds, so the agent gives an honest "I don't know" instead of a fluent guess, with prompt-injection defenses around retrieved content.
Onboarding is itself LLM-automated: when a company signs up in a new industry, Gemini researches the domain and writes that industry's agent rulebook — scope fences and compliance rules — automatically.
Around all of that sits a full SaaS product: self-serve signup with plans, per-tenant agent configuration, bring-your-own-Twilio with API-verified credentials and automatic webhook setup, and complete call transcripts with captured leads.
The result
A live product on AWS EC2 behind nginx and systemd, deployed through GitHub Actions CI/CD.
Real inbound calls answered around the clock in a natural voice, with every answer traceable to a document the business uploaded.
Businesses onboard themselves — including their own Twilio credentials — without me touching a configuration file.
Engineering notes
Relevance thresholds are a product feature
Forcing an honest "I don't know" when retrieval comes back weak is what makes the agent safe to put in front of paying customers. Grounding is a product decision before it is a technical one.
Barge-in is what makes it feel human
Callers interrupt. Handling interruption in the audio pipeline, rather than waiting for a turn to finish, is the difference between a demo and something people will actually talk to.
Retrieved content is untrusted input
Anything pulled from a tenant's uploaded PDFs can carry instructions. Prompt-injection defenses around retrieval are not optional once the corpus is user-supplied.
Built with
- Python (FastAPI)
- Laravel
- Vue.js
- PostgreSQL + pgvector
- Deepgram Voice Agent
- Twilio
- Google Gemini
- AWS