Designing a Real-Time Voice AI Agent Platform

When I design a real-time voice AI agent, the hardest constraint is not throughput or storage. It is a feeling. A person on a call expects the other side to answer about as fast as a human would, to stop talking when interrupted, and never to leave them in dead air wondering whether the line dropped. Conversation research has found that the gap between turns in human conversation is usually a few hundred milliseconds. A text chatbot that takes two seconds to start answering feels fine. A voice agent that takes two seconds feels broken.

That one constraint reshapes everything. Audio cannot wait behind a TCP retransmission. Speech recognition has to run while the person is still speaking. Speech synthesis has to start before the language model finishes its answer. And the system has to know what the user actually heard, because people interrupt, and an agent that believes it said things it never got to say steers the conversation wrong.

In this post I walk through how I would build a platform that runs live spoken conversations between people and AI agents: support lines, phone booking, AI interviewers, tutors. I cover scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives:

• Media transport over WebRTC and phone lines
• Streaming speech-to-text, voice activity detection and turn-taking
• The sub-second response pipeline from LLM to TTS
• Barge-in, tool calls and conversation state

Scoping the Problem and Clarifying Assumptions

I am building a multi-tenant platform. Businesses configure an agent (instructions, voice, tools, phone numbers), and the platform runs the calls. Each call is one person talking to one agent, in a browser over WebRTC or on a phone over the public telephone network.

What I'm including:

• Browser and app calls over WebRTC, and inbound and outbound phone calls over SIP trunks to the PSTN
• A cascaded pipeline per call: streaming speech-to-text (STT), a large language model (LLM), streaming text-to-speech (TTS)
• Turn-taking, interruption (barge-in) handling and tool calls mid-conversation
• Transcripts, recordings, per-turn latency metrics and per-call cost accounting

What I'm NOT building here:

• Training STT, TTS or language models. I consume them as services, some self-hosted on GPUs, some from external providers.
• The business systems behind the tools (calendars, CRMs, order databases)
• Video understanding and multi-party meetings

Key assumptions:

• Tens of thousands of concurrent calls. I plan for 50,000 at peak.
• A typical call lasts about 6 minutes, with the user taking a turn every 8 seconds or so
• Agents use a mid-size, fast LLM by default. Spoken turns are short, so time to first token matters more than deep reasoning.

How this differs from a text chatbot: a chatbot is request/response. The user submits a complete message and the system streams an answer. A voice agent is a full-duplex stream. Audio flows both ways all the time, the system must decide on its own when the user has finished, and the user can cut in at any moment. There is no submit button. The hardest problems live between the components: when to start answering, how to start speaking before the answer is complete, and what to do when the user talks over the agent.

I also considered a single speech-to-speech model (audio in, audio out). These models can reduce latency and I would keep the pipeline open to them. For a general platform I start with the cascaded design because each tenant can choose STT, LLM and TTS independently, every step produces text for auditing and tool use, and I can swap one stage when a better model ships.

Functional and Non-Functional Requirements

Functional Requirements

1. Connect: accept calls from browsers (WebRTC) and phones (SIP), inbound and outbound.
2. Converse: transcribe speech in real time, generate a reply with the configured LLM, and speak it with the configured voice.
3. Take turns: detect when the user has finished, and let the user interrupt the agent at any time.
4. Act: call tools (look up an order, book a slot, transfer to a human) mid-conversation.
5. Record: store transcripts, recordings (with consent), tool calls and per-turn timings.
6. Account: measure each call's cost across telephony, STT, LLM and TTS.

Non-Functional Requirements

Voice-to-voice latency is the number I organise the design around. I measure it from when the platform hears the user stop speaking to the first audio frame it sends back. The user also hears network delay on both legs, so their real gap is somewhat larger.

Back-of-the-Envelope Capacity Estimation

These are planning assumptions, not measurements. The method is what matters: estimate each stage's concurrency, then size each pool by what one unit sustains in a load test.

Calls:

• 50,000 concurrent calls at peak, about 20,000 on average
• 6-minute calls: 20,000 × (1,440 / 6) ≈ 4.8 million calls per day
• New calls at peak: 50,000 / 360 s ≈ 140 call setups per second

Media:

• Opus voice at about 32 kbps in 20 ms packets (50 per second). IP, UDP and RTP headers add about 40 bytes per packet: 50 × 40 × 8 = 16 kbps.
• About 50 kbps per direction per call, so 50,000 × 50 kbps ≈ 2.5 Gbps each way across the edge fleet. Modest in bytes, but it is 2.5 million small packets per second each way, and packet rate is what limits a media server.

STT: every call keeps an STT stream open while the user might speak: up to 50,000 concurrent streams at peak.

LLM:

• One user turn every 8 seconds: 50,000 / 8 ≈ 6,250 LLM requests per second at peak
• About 4,000 tokens of context per turn (instructions, tool definitions, history), almost all a repeated prefix, and about 50 output tokens, because spoken replies are short
• Output: about 310,000 tokens per second. Input: 25 million tokens per second, which is only affordable because prefix caching serves most of it.

TTS: if the agent speaks about 40% of the time, about 20,000 concurrent TTS streams, each faster than real time.

Storage: stereo Opus recordings at about 64 kbps are 8 KB per second, about 2.9 MB per call, so 4.8M × 2.9 MB ≈ 14 TB per day before retention policies. Transcripts are a few kilobytes per call.

My takeaway: media is a packet-rate problem, STT and TTS are concurrency problems, and the LLM is a time-to-first-token problem. Each is sized differently.

High-Level Architecture and Core Components

Component Walkthrough

Media edge (SFU). Media servers that terminate WebRTC: ICE, DTLS-SRTP, Opus, jitter buffers. A two-party call does not need a full selective forwarding unit, but SFU software already handles WebRTC well and lets a human supervisor join a live call later without redesign.

SIP gateway. Connects to carriers over SIP trunks, maps numbers to agents, handles DTMF and transfers, and transcodes telephone audio (G.711) to the internal format.

Agent worker. The orchestrator for one call. It receives audio frames, runs voice activity detection, streams audio to STT, decides when the user's turn is over, calls the LLM, splits its output into speakable chunks, streams them to TTS and sends audio back. It owns the call's conversation state. One async process runs many calls, and each call stays pinned to one worker.

STT, LLM and TTS pools. GPU inference pools, or external providers behind the same interface. Stateless per request, very sensitive to queueing.

Tool gateway. Runs tool calls against tenant systems with per-tenant credentials, timeouts and audit logs. The agent worker never holds tenant secrets.

Call events and storage. Each turn emits events (text, timings, tool calls, interruptions) for dashboards, transcripts and billing. Recordings go to object storage.

Control plane. Agent configs, phone numbers, call placement and tenant quotas. Never on the per-frame path.

Request Flow

1. A caller dials a tenant's number and the carrier sends a SIP INVITE to the SIP gateway.
2. Call placement picks a region, a media edge and an agent worker with capacity near the GPU pools, and loads the agent config.
3. Audio flows carrier to gateway (G.711), gateway to edge (Opus), edge to worker as 20 ms PCM frames.
4. The agent greets the caller. Fixed phrases like the greeting are pre-synthesised per agent, so they play instantly.
5. The user speaks. STT returns partial transcripts, and the turn detector decides the user has finished.
6. The worker sends the turn to the LLM, streams the first clause of the reply to TTS, and audio starts within the budget.
7. If the user interrupts, the worker stops playout, records what was actually heard, and listens.
8. On hang-up, the worker writes the transcript, closes the recording and emits a cost record.

Scaling It Out

• Co-location is the main scaling rule. The hops between worker, STT, LLM and TTS are paid several times per turn. A 70 ms cross-region hop paid three or four times can eat a third of the budget, so a call's worker and its inference pools share a region, ideally a zone.
• Place calls, do not just balance them. A call is a stateful session that lasts minutes. Placement picks a region near the user (WebRTC) or the carrier's interconnect (phone) that also has GPU headroom, then a worker with spare slots.
• Scale on concurrency. Workers scale on active calls, STT and TTS on concurrent streams, the LLM pool on queue depth and time to first token. GPU pools start slowly, so I scale ahead of the daily curve, which for call centres is very regular.
• Per-tenant admission. Each tenant has a concurrent call limit. Near capacity, new calls hear a polite pre-recorded "all lines are busy" message instead of getting a degraded live call.

Deep-Dive: Media Transport: WebRTC, SFUs, and Telephony

Why WebRTC Instead of WebSockets for Audio

The simplest way to get browser audio to a server is a WebSocket carrying audio chunks. It works in a demo and fails in production because WebSockets run over TCP. When a packet is lost, TCP holds back every later byte until the lost one is retransmitted. On a lossy mobile network that means stalls of hundreds of milliseconds followed by a burst of stale audio. For a voice call, late audio is worse than lost audio.

WebRTC is built for this:

• UDP with SRTP. Lost packets stay lost and the receiver conceals the gap.
• Opus, designed for interactive speech, with in-band forward error correction (a low-bitrate copy of the previous frame in the next packet) and packet loss concealment.
• Adaptive jitter buffers that hold just enough audio to smooth uneven arrival, growing on bad networks and shrinking on good ones.
• Congestion control, so the sender lowers bitrate on a weak link instead of flooding it.
• Browser audio processing. getUserMedia provides echo cancellation, noise suppression and gain control.

I still use the WebRTC data channel or a WebSocket for control messages (captions, UI events, agent state), because those are small and must not be lost.

NAT Traversal and TURN

ICE first tries a direct UDP path to the media edge, which has a public address, so most calls need nothing more. Corporate networks often block UDP, so I run TURN on TCP and TLS over port 443, which passes almost any firewall. Relayed calls get TCP head-of-line blocking back on that leg, but they connect. I track the TURN share per tenant, because a high share usually means a customer's restrictive network.

Media Server Placement

Network delay counts twice: once for the user's audio to reach me and once for the reply to come back. Media edges are cheap and can live in many regions. GPUs are expensive and live in fewer. My rule: put the media edge near the user and the agent worker next to the GPUs. The edge-to-worker leg runs on the cloud backbone, with far less jitter and loss than the public internet, so the user gets a short clean first hop even when inference is further away.

Telephony: SIP Trunks and the PSTN

Carriers send SIP signalling with RTP audio, usually G.711 (8 kHz, 64 kbps, narrowband). The SIP gateway terminates trunks and authenticates carriers, transcodes audio (and passes 8 kHz audio straight to STT models trained on telephone speech, which often beats upsampling), turns DTMF keypresses into events, and handles transfers to human agents (SIP REFER or a bridged second leg) and outbound dialling.

Phone calls add latency I cannot control in the carrier network and handset. I compensate by keeping my own processing tighter on phone calls, and by placing them in a region close to the carrier's interconnect rather than the caller.

Echo Cancellation and the Agent Hearing Itself

On a speakerphone, the agent's voice comes back through the microphone. Without echo cancellation, STT transcribes the agent's own words and it interrupts or answers itself. Browsers cancel most echo. Phone lines usually do, but not always well. So the worker adds a second defence: while the agent speaks, it raises the barge-in threshold and compares incoming transcripts with the text being spoken. A close match is echo, not an interruption. A server-side echo canceller using the outgoing TTS audio as its reference helps further, since the worker knows exactly what it played.

First vs later: first, managed TURN, an open-source WebRTC media server, a SIP trunking provider, and one format at the worker boundary (16 kHz mono PCM, 20 ms frames). Later, direct carrier interconnects in key countries, server-side echo cancellation and supervisor listen-in.

Deep-Dive: Streaming Speech-to-Text, VAD, and Turn-Taking

Streaming ASR with Partial Results

Batch recognition adds the whole utterance length to latency. I need streaming recognition: 20 ms frames go in, and the model returns partial hypotheses that keep changing ("I want to", "I want to book a table") and then final results that do not. Partials drive early turn detection, speculative LLM starts and live captions.

Whisper-class models are accurate but designed around chunks of audio, so streaming deployments of them use sliding windows and trade latency for accuracy. Purpose-built streaming models usually fit the live path better. A pattern I would adopt: a fast streaming model live, and a larger, more accurate model re-transcribing the recording afterwards for the stored transcript.

Domain vocabulary matters: product names, people's names, order numbers. Most STT services accept phrase hints, so tenants configure them per agent, and the worker adds context from the conversation, such as the time slots the agent just offered.

Voice Activity Detection

A VAD model (Silero VAD is a common open-source choice) scores each short window as speech or not. It is small enough to run on CPU in the worker for every call. I use it to gate STT (silence need not go to an expensive model, though I keep a short pre-roll buffer so the first syllable is not clipped), to detect barge-in, and as one input to endpointing. Raw VAD flickers, so I smooth it: speech must last a minimum time (say 150 to 250 ms) to count as a start. A cough or a door slam is not a turn.

Endpointing: Finished or Just Pausing?

This is the hardest decision in the system. People pause mid-sentence ("my order number is... 4 7 2... 9 1"). A fixed silence threshold forces a bad trade: around 300 ms is responsive but cuts people off; around 1,000 ms rarely interrupts but every reply feels sluggish. So I combine three signals:

1. Acoustic silence from the VAD
2. Semantic completeness from a small turn-detection model that reads the transcript and scores whether it sounds finished. "I'd like to book a table for" is not. "A table for two at seven" probably is. Voice agent frameworks such as LiveKit Agents and Pipecat ship open-source models for this, and a small fine-tuned classifier is also practical.
3. Dialogue context: if the agent just asked for a card number, the user will pause between digit groups, so I wait longer.

def end_of_turn_delay(turn_score: float, expecting: str | None) -> float:
    """Seconds of silence to wait before committing the user's turn."""
    if expecting in ("digits", "email", "address"):
        return 1.5                      # people pause while reading these out
    if turn_score > 0.85:
        return 0.25                     # clearly complete: answer fast
    if turn_score > 0.5:
        return 0.6
    return 1.2                          # sounds unfinished: give them room

# Turn manager, on every VAD silence tick:
silence = now() - last_speech_at
if silence >= end_of_turn_delay(detector.score(partial_text), state.expecting):
    commit_turn(partial_text)

These thresholds are starting points. I tune them per language and agent from recorded calls by tracking two error rates: false cut-offs (the user immediately talks over the agent's reply) and excess waits (long silences before replies).

Speculative Starts

The endpointing wait is mostly dead time, so I use it. When silence begins and the turn score is high, I start the LLM speculatively on the current transcript but play nothing. If the turn commits, the first tokens (sometimes the first audio) are ready, and most of the LLM's time to first token is hidden. If the user keeps talking, I cancel. This costs extra LLM calls, so I only speculate on confident scores and track the waste ratio.

Background Noise

Real calls happen in cars, cafes and call centres. I run noise suppression before VAD and STT (the browser does some; phone audio gets a server-side model). Barge-in requires words, not just sound: an interruption is confirmed when STT returns a word or two, or VAD stays confident for a sustained period, so a television in the background does not stop the agent every few seconds. When STT confidence is low, the prompt says so, and the agent asks the user to repeat rather than guess an order number.

First vs later: first, VAD plus a semantic turn detector with tuned thresholds. Later, speculative starts, per-agent tuning from recordings, and second-pass transcription.

Deep-Dive: The Sub-Second Response Pipeline: LLM and TTS

The Latency Budget

I start with a budget from end of user speech to the first agent audio frame leaving my edge. These are design targets, not measured values, and each line is measured and alerted on separately.

Two things follow. Endpointing is often the largest line, which is why the previous deep-dive matters. And nothing in this table works if any stage waits for the previous one to finish. Every stage streams.

Streaming LLM Tokens into Streaming TTS

The naive pipeline waits for the full reply, synthesises it, then plays it. Instead, LLM tokens flow into a sentence chunker that emits a chunk at each natural break, and each chunk goes to TTS at once. Chunking is a quality trade-off: TTS intonation sounds best on whole sentences, and tiny chunks sound choppy. So the first chunk is cut at the first clause boundary to start speaking fast, and later chunks are whole sentences, synthesised while earlier audio plays.

async def speak_reply(llm_stream, tts, playout):
    buf, first = "", True
    async for token in llm_stream:
        buf += token
        cut = find_break(buf, min_words=4 if first else 12)   # , . ? !
        if cut:
            chunk, buf = buf[:cut], buf[cut:]
            await playout.enqueue(tts.stream(normalise(chunk)))
            first = False
    if buf.strip():
        await playout.enqueue(tts.stream(normalise(buf)))

normalise turns "$42.50" into "forty-two dollars and fifty cents" and strips markdown, which must never reach TTS. I also instruct the LLM to write for speech (short sentences, no lists or symbols), but I do not rely on instructions alone.

Fillers and Backchannels

Some turns are slower anyway: a tool call, a long context, a busy GPU. Dead air is what users notice most, so I mask it. If first audio is not ready within a threshold (say one second), the worker plays a short filler like "Let me check that", pre-synthesised per voice and cached, never the same one twice in a row. The filler goes into the history as something the agent said, or the LLM may repeat it. Backchannels ("mm-hm" during a long user explanation) can make an agent feel attentive, but a badly timed one sounds like an interruption, so I add them only once turn detection is solid.

Model Choice and Co-Location

For the live turn I choose the fastest model that handles the agent's task reliably, not the largest. A larger model still helps off the hot path: post-call summaries, data extraction, or a rare complex step exposed as a tool.

Prompt layout matters for latency. Instructions and tool definitions form a stable prefix with the conversation appended, so the inference server's prefix cache covers almost all of the 4,000-token context every turn. Routing a call's turns to the same replica is what makes that cache hit; without it, full prefill runs every turn and adds directly to time to first token.

The worker, STT, LLM and TTS share a zone and talk over persistent streaming connections (gRPC or WebSocket) opened at call start, not per turn, so no turn pays for a TLS handshake. When a tenant uses an external LLM or TTS provider, I place the call in the region closest to that provider and measure its time to first token continuously, because its latency becomes mine.

Measuring Voice-to-Voice Latency

Every turn emits one timing event, with all timestamps from the same clock in the worker:

{
  "call_id": "c_8f2a",
  "turn": 7,
  "user_speech_end_ms": 182340,
  "turn_committed_ms": 182640,
  "llm_first_token_ms": 182890,
  "tts_first_audio_ms": 183050,
  "first_frame_sent_ms": 183070,
  "voice_to_voice_ms": 730,
  "speculative_hit": true,
  "filler_played": false
}

Dashboards show p50 and p95 per stage, tenant, model and region, so a regression points straight at the endpointer, the LLM pool or TTS. Internal timestamps cannot see the user's network, so I also run synthetic calls with recorded audio from several regions around the clock to measure the full path.

Deep-Dive: Barge-In, Tool Calls, and Conversation State

The Conversation State Machine

Each call runs an explicit state machine in its worker. Implicit state spread across callbacks is where most voice agent bugs come from.

LISTENING  --(turn committed)--------> THINKING
THINKING   --(first audio ready)-----> SPEAKING
THINKING   --(tool call)-------------> TOOL_WAIT
TOOL_WAIT  --(result or timeout)-----> THINKING
SPEAKING   --(playout done)----------> LISTENING
SPEAKING   --(barge-in confirmed)----> LISTENING  (cancel LLM + TTS, truncate)
THINKING   --(user speaks again)-----> LISTENING  (cancel, merge into same turn)
any        --(transfer)--------------> TRANSFERRING
any        --(hang-up / max length)--> ENDED

The state decides what each event means. User speech while LISTENING is a turn. While SPEAKING it is a possible interruption. While THINKING it usually means the user was not finished, so I cancel the in-flight reply and merge the new words into the same turn.

Barge-In: Stop Fast, Then Fix the History

When the user talks over the agent, three things happen in order:

1. Stop the audio. Clear the playout queue and stop sending frames. Audio already in flight sits in jitter buffers, so the user may hear a little more. That is why I stop the moment barge-in is confirmed, not at the end of the sentence.
2. Cancel upstream work. Cancel TTS and LLM generation to stop paying for output nobody will hear.
3. Truncate the history to what was heard. This is the step teams miss. The LLM generated a full reply, but the user heard part of it. If the history keeps the full text, the agent later refers to things it never said ("as I mentioned, the fee is waived").

The playout tracker knows which frames were sent, and TTS engines that return word timestamps map that back to text. Without them, I estimate from the share of the chunk's audio that played.

def on_barge_in(call):
    played_ms = call.playout.stop()              # frames actually sent
    call.tts.cancel()
    call.llm.cancel()
    heard = call.playout.text_heard(played_ms)   # via word timestamps
    call.history.replace_last_assistant(heard + " [interrupted by user]")
    call.state = "LISTENING"

The marker helps the model respond naturally ("Sorry, go ahead") instead of resuming mid-sentence. Short acknowledgements like "uh-huh" or "okay" during a long explanation are listening signals, not interruptions, so I ignore them (the list is per language).

Tool Calls Mid-Conversation

Tools make the agent useful and are latency spikes I do not control. My rules:

• Announce, then act. Many models can emit a short text preface and a tool call in one response. The preface ("Let me look that up") plays while the tool runs, which masks most tool latency.
• Strict timeouts, a few seconds per tool, with a spoken fallback ("I can't reach the booking system right now; I can have someone call you back").
• Prefetch. At call start I load what the agent will likely need (the caller's account from their number, open orders), so many tool calls become cache reads.
• Confirm before side effects. For bookings, cancellations and refunds the agent reads the details back and waits for a clear yes, since a misheard date must not become a wrong booking. Write tools take an idempotency key per call and turn, so a retry after a timeout cannot book twice.
• Speech during a tool call. I let the tool finish (it may already have a side effect) and give its result to the LLM together with the new user turn.

Transcripts and Recording

Every committed user turn, every heard agent turn, every tool call and interruption goes to the call events stream in order, keyed by call id. At call end, a worker assembles the final transcript, optionally re-transcribes the recording with a more accurate model, and runs post-call jobs (summary, outcome, quality scoring) on a larger LLM where latency does not matter. Recordings are stereo, user and agent on separate channels, which makes analysis and redaction easier. Recording consent rules vary by jurisdiction, so recording is a per-tenant, per-region policy with a spoken disclosure when required. Card numbers spoken aloud are redacted from transcripts and muted in the recording.

Failover Mid-Call

Calls last minutes, so dependencies will sometimes fail during one:

• STT, LLM or TTS replica. The worker reconnects to another replica within the turn, retries an LLM request once, and plays a filler if the retry would blow the budget.
• Agent worker crash. The expensive case. After every turn the worker checkpoints history, state, pending tool calls and config version to a fast key-value store. When the edge sees the worker vanish, placement assigns a new one, which loads the checkpoint and says "Sorry, I lost you for a second. You were asking about your booking." A short gap instead of a dropped call.
• Media edge. WebRTC clients do an ICE restart to another edge. Phone calls are anchored at the SIP gateway, run as a redundant pair per trunk, so an edge failure behind it is recoverable.
• Provider outage. Each agent config lists fallback providers. A TTS switch changes the voice audibly, so it is a last resort mid-call.

Per-Call Cost

cost per call = telephony minutes x carrier rate             (phone calls only)
              + call minutes x media edge cost per minute
              + speech minutes sent to STT x STT rate
              + sum over turns (fresh input tokens x rate_in
                                + cached tokens x rate_cached
                                + output tokens x rate_out)
              + characters synthesised x TTS rate
              + tool calls and storage

The levers follow from the formula. VAD gating cuts STT minutes. Prefix caching makes the large repeated context cheap. Cancelling on barge-in stops paying for unheard tokens and audio. Short spoken replies cut output tokens and TTS characters, and help latency too. I track cost per call minute per tenant and agent, because two agents with the same traffic can differ widely in how much they talk and how many tools they call.

First vs later: first, the explicit state machine, barge-in with truncation, tool timeouts with fillers, and per-turn checkpoints. Later, backchannels, seamless worker failover and automatic tuning of interruption sensitivity per agent.

Conclusion: Bringing It All Together

A real-time voice agent looks like a chatbot with a microphone, but it behaves more like a telephone exchange wired to a pipeline of streaming models. The defining constraint is conversational latency, and the big decisions all follow from it:

• Audio travels over UDP (WebRTC and RTP), because late audio is worse than lost audio.
• Recognition, generation and synthesis all stream, and each stage starts before the previous one ends.
• Turn-taking uses both silence and meaning, because a fixed silence threshold is always too slow or too rude.
• The history records what the user actually heard, not what the model generated.

If I had to summarise the design in four decisions:

1. Put the media edge near the user and the agent worker next to the GPUs, with the hot path in one zone over persistent streaming connections.
2. Endpoint with VAD plus a semantic turn detector, and start the LLM speculatively. Endpointing is the largest line in the latency budget.
3. Stream LLM tokens into TTS by clause, keep the prompt prefix cacheable, and mask slow turns with fillers, measuring voice-to-voice latency per stage on every turn.
4. Run each call as an explicit state machine that truncates history on barge-in and checkpoints every turn, which makes interruptions, tool calls and failover predictable.

Related reading: Designing an LLM Inference Platform, REST, GraphQL, gRPC, WebSockets, SSE, and WebRTC Compared, AI Agents and Agentic Workflows, and Designing Discord.