how colyap scales to a million users

Direct answer up top: you scale a phone-native AI agent the way you scale anything. Stateless services behind load balancers, queues for anything slow, cache the hot path, shard the data. Except you do it under a constraint most web apps never feel: if any of it takes longer than ~800 milliseconds mid-call, the human on the other end thinks you hung up.

I've been sketching this architecture the way I like to sketch any system: start naive, add pieces only when a number forces you to. So let's do the numbers for Colyap.

the back of the envelope

A million users sounds scary until you write down what they actually do. People don't call an assistant the way they scroll a feed. Say the average user makes one 5-minute call a day plus a handful of texts. That's 5 million call-minutes a day. A day has 1,440 minutes, calls cluster into maybe 6 peak hours, so:

5,000,000 call-minutes / day
peak window ≈ 6 hours = 360 minutes
concurrent calls at peak ≈ 5,000,000 × 0.4 / 360 ≈ 5,500
realistically (not everyone calls daily)   ≈ 1,000-3,000

A few thousand concurrent calls. That's the real number you provision for, not one million. This is the single most useful habit in capacity planning: your users are not your load. Your concurrency is your load, and concurrency for voice is mercifully small because talking is slow. Humans max out around 150 words a minute; nobody scrolls a phone call.

speech to speech, not a relay race

Here's where Colyap diverges from most voice agents you've met. The traditional stack is a relay race of three separate models: speech-to-text transcribes you, a text LLM thinks about the transcript, and text-to-speech reads the reply out loud. Every baton pass costs time, and the times are serialized. You wait for enough transcript to be useful, then you wait for tokens, then you wait for audio synthesis to spin up. Getting that chain under a second at p50 is hard; at p99 under load it's a nightmare, because three models means three tail latencies multiplied together.

Colyap runs a speech to speech pipeline instead: one model that consumes audio and emits audio. No intermediate transcript on the hot path, no third model warming up while you wait. The benefits stack up fast:

• One tail, not three. Latency budgeting against a single model is a solvable problem. Budgeting against three chained models under burst traffic is whack-a-mole.
• The audio survives. Transcription flattens everything that isn't words: tone, hesitation, sarcasm, the difference between "fine." and "fine!". A speech-native model hears how you said it, not just what a transcript claims you said, and its own voice carries prosody back instead of reading a string in customer-service monotone.
• No error cascade. In a relay pipeline, a transcription mistake becomes the LLM's ground truth. "Maya" heard as "my ma" poisons everything downstream. Skip the transcript and the class of bug disappears.
• Interruptions work. Real conversation is full of barge-ins and "wait, actually." A speech-native loop can react mid-utterance instead of finishing a queued TTS clip that no longer applies.

Scaling-wise this simplifies the topology too. Audio streams terminate on gateways near the telco interconnects, the speech model runs co-located so audio never crosses an ocean twice, and the hot path gets nothing extra: no synchronous database writes, no third-party API calls, no "quick" analytics event mid-turn. Everything that isn't producing the next spoken sound gets deferred onto a queue.

the hot path produces the next sound. everything else is someone else's problem, specifically the queue's.

stateless everything, or you can't scale sideways

The classic move: no server holds anything you'd miss. The call orchestrator that's managing your conversation keeps its state in Redis, not in process memory, so when we need more capacity we add machines behind the load balancer and when one dies mid-call the session fails over and the caller hears, at worst, a half-second hiccup. Web tier, iMessage tier, task workers: same rule. State lives in the data layer; compute is cattle.

This is boring advice and that's the point. Horizontal scaling only works if any node can serve any user, and that only works if the nodes are interchangeable.

memory is a cache invalidation problem

I've written before that most latency problems in AI eventually cosplay as cache invalidation problems, and memory is the clearest case. Colyap's long-term memory, the multi-layered semantic store built from post-call analysis, lives in Postgres, sharded by user ID. Shard by user, always by user: no query in a live call ever needs two users' data, so user-sharding gives you embarrassingly clean horizontal cuts.

But you cannot go to Postgres, run four cosine-similarity searches, and re-rank with recency decay while a human is mid-sentence. So the moment a call connects, before you finish saying hello, a warmer job pre-loads that user's hot context into Redis: recent conversations, durable facts, open tasks. During the call, retrieval hits memory in microseconds. Cold, deep retrieval (that thing you mentioned in March) still happens against the shard, on the model's discretion, and it's allowed to say "let me check" like a human would. Postgres aggregates a million rows a second per core when the index is right; the trick is just never doing it synchronously when you don't have to.

queues are what make hanging up work

Colyap's whole promise is that the task outlives the call. You say "find me flights, text me options," hang up, and go live your life. Architecturally that sentence is a message on a durable queue. Long-running work like research, comparisons, waiting on a slow website, or your "remind me in three weeks" becomes jobs consumed by a fleet of stateless workers that scale on queue depth, completely decoupled from call traffic.

• Producers: the call orchestrator and the iMessage handler enqueue and immediately return to the conversation.
• Consumers: task workers with retries, idempotency keys, and exponential backoff; if a worker dies mid-task, the job re-runs and nobody notices.
• Scheduling: future-dated jobs sit in the same system with a delivery time. A reminder is just a very patient message.

Queues also absorb the burst problem. Voice traffic spikes (everyone calls at 6pm); model capacity is expensive to overprovision. The queue is the shock absorber between spiky demand and smooth-ish supply.

the rest of the checklist

Rounding out the list, briefly, because each of these is its own post:

• Rate limiting at the edge: phone numbers are a public API. Per-number and per-IP token buckets keep one enthusiastic script from eating the inference budget.
• Read replicas before sharding: reads (memory retrieval) dwarf writes (post-call analysis), so replicas buy a long runway before shard counts grow.
• Multi-AZ, multi-region by geography of the phone number: call latency is physics; put the stack where the users are.
• Observability per conversation: one trace ID from ring to hangup to the follow-up text three hours later. When something feels slow, you need to know which hop lied.
• Graceful degradation: if the deep retrieval times out, answer from hot cache and note the gap. A slightly forgetful answer beats dead air every time.

the honest part

Do we run all of this at full million-user glory today? No, and neither did anyone else at our stage. You build the version the current numbers demand and make sure nothing you ship today blocks the next order of magnitude. Stateless services, user-sharded data, queue-shaped tasks, and a speech-native pipeline are cheap decisions now and brutal migrations later, so those we took early. The rest scales when the envelope math says so.

A million users of a voice agent is a few thousand simultaneous conversations, each demanding an answer in under a second, forever. Frame it that way and it stops being scary and starts being a very fun engineering problem. Love to hear what I've missed here. Call and yap at me about it.