WebRTC vs WebSocket for Voice AI Agents
Jaseem Abbas
- Published

WebRTC and WebSocket are two ways to move live audio between a caller's device and a voice AI agent. WebRTC is a browser standard built for real-time audio and video, and it usually runs over UDP. WebSocket is a general two-way channel that runs over TCP. A third path exists for phone calls: the carrier's phone network, which uses SIP to set up the call and RTP to carry the audio. This guide explains how the three differ, which one suits a browser widget, a mobile app and a phone call, and what to test before you commit.
Here is a made-up example. Imran builds a voice helper for a coaching institute's website in Hyderabad. On his office Wi-Fi it sounds perfect. A student in a train near Vijayawada opens the same page. Her signal drops for a second, and then the agent's answer arrives late, followed by a burst of words played too fast. Nothing was lost. Everything just waited. The cause is in the transport Imran chose. Imran, the institute and the student are made up for this guide.
What is the difference between WebRTC and WebSocket?
WebRTC carries live audio and video, peer to peer or through a media server, while WebSocket carries any messages in both directions over one TCP connection. Both let a browser and a server talk in real time, but they were made for different jobs.
The WebSocket protocol is defined in RFC 6455. It "enables two-way communication" between a client and a remote host, and consists of an opening handshake followed by basic message framing, layered over TCP. It is designed to work over the ordinary web ports, 80 and 443, and to fit existing web infrastructure (RFC 6455).
WebRTC is a set of protocols and browser APIs. RFC 8825 gives the overview: building blocks that let browsers use interactive audio and video and communicate directly across the Internet (RFC 8825). The browser API is defined by the W3C as WebRTC: Real-Time Communication in Browsers, a W3C Recommendation dated 13 March 2025 (W3C WebRTC).
Aspect: Built for
- WebRTC
- Live audio and video between browsers or apps
- WebSocket
- Two-way messages over one connection
- Phone line (SIP and RTP)
- Phone calls between networks
Aspect: Runs over
- WebRTC
- UDP mostly, with TCP fallbacks
- WebSocket
- TCP
- Phone line (SIP and RTP)
- SIP for setup, RTP for the audio
Aspect: Audio codec
- WebRTC
- Opus, with G.711 also required of endpoints
- WebSocket
- Whatever you choose to send, such as raw PCM
- Phone line (SIP and RTP)
- Usually G.711 at 8 kHz on the carrier leg
Aspect: Lost packets
- WebRTC
- A short gap, then audio carries on
- WebSocket
- Held up until the packet is resent
- Phone line (SIP and RTP)
- RTP carries the audio and gives no delivery guarantee
Aspect: Echo and noise handling
- WebRTC
- Built into the browser stack
- WebSocket
- Not provided, you add it
- Phone line (SIP and RTP)
- Depends on the phone and the carrier
Aspect: Getting through firewalls
- WebRTC
- ICE and TURN, including over TCP and TLS
- WebSocket
- Uses web ports 80 and 443
- Phone line (SIP and RTP)
- Between carriers. You connect to your carrier's endpoint
Aspect: Typical home in a voice agent
- WebRTC
- Browser widget, mobile app
- WebSocket
- Server to server, carrier media streams
- Phone line (SIP and RTP)
- Phone calls through a carrier or SIP trunk
Why is WebSocket a poor fit for live audio over weak networks?
Because WebSocket sits on TCP, and TCP never skips a packet. If one packet is lost, TCP resends it and holds back everything that arrived after it, so the receiver waits. For a file download that is exactly right. For a voice call it is not, because the late packet is no longer useful and everything behind it is delayed too. This is called head-of-line blocking.
Pipecat's documentation, written for developers building voice agents, makes this its main argument. It says that for any client-to-server voice application, WebRTC is the right choice, and that TCP "retransmits lost packets and holds up everything behind them", so a dropped packet is better discarded than waited for. A brief gap sounds better than a stutter (Pipecat: choosing a transport). The IETF's WebRTC transport document agrees on the general risk: it notes that TCP between a relay server and a peer can cause performance problems compared with UDP, for example from head-of-line blocking (RFC 8835).
That story is Imran's story. His student's brief signal dip lost a few packets. On TCP they were resent, and the later speech waited behind them. The agent was not slow. The transport made it late.
Pipecat's docs list four more reasons, each of which WebRTC handles and a raw WebSocket leaves to you:
- Opus loss handling. Opus has forward error correction and packet loss concealment, which the docs say are designed for UDP delivery and do not help much on TCP.
- Timing. WebRTC handles RTP timestamps and jitter buffering automatically. WebSocket leaves that to you.
- Echo and noise. Browser echo cancellation and noise suppression are built into the WebRTC stack and are not available to arbitrary WebSocket streams.
- Reconnects. WebRTC handles ICE restarts and network changes. A WebSocket on a phone that sleeps, wakes or switches from Wi-Fi to mobile data needs reconnection logic you write.
The same page says WebSocket is appropriate for server-to-server links, where both sides sit on stable, controlled networks, and for text-only bots. That is the honest limit of the argument: WebSocket is not bad, it is the wrong tool for audio over the open internet.
What does WebRTC add for voice?
It adds a speech codec, resilience to bad networks, a way through firewalls and echo control, all in one standard stack. Taking the parts in turn:
- A speech codec. Opus is defined in RFC 6716 as an interactive speech and audio codec for uses such as voice over IP, scaling from 6 kbit/s narrowband speech up to 510 kbit/s stereo music (RFC 6716). RFC 7874 says WebRTC endpoints are required to implement Opus, and also the two G.711 variants (PCMA and PCMU), plus comfort noise and DTMF events (RFC 7874). RFC 7874 says this minimum set exists to give a baseline of interoperability, and to avoid transcoding between codecs where possible.
- Timed media. Audio travels as RTP, which RFC 3550 defines for end-to-end delivery of real-time data such as audio and video. RFC 3550 is clear on the limit: RTP does not guarantee quality of service, and it adds a control protocol (RTCP) to monitor delivery (RFC 3550).
- A way through firewalls. RFC 8835 describes the transport: UDP is the protocol assumed by most elements, and TCP is used for HTTP and WebSockets, and for TURN over TCP and TLS. It says the TURN modes over TCP and TLS must be supported, so a call can still connect when a firewall blocks all UDP, and that ICE-TCP candidates must be supported too (RFC 8835).
None of this removes the need to test. WebRTC reduces a class of problems. It does not promise a good call on a poor line.
How do phone calls carry audio to an AI agent?
Over the carrier's network. A phone call is set up with SIP, which RFC 3261 defines as an application-layer signalling protocol for creating, modifying and ending sessions such as Internet telephone calls (RFC 3261). The audio itself rides on RTP, usually with the G.711 codec at 8,000 samples a second. Our guide to voice agent latency covers what that leg adds, and the glossary entries on VoIP and SIP trunk define the terms. For the line itself, see PRI vs SIP trunk and our guide to SIP trunking for AI calls in India.
Your agent can attach to a phone call in two common ways:
- A SIP connection. The carrier or your SIP trunk connects the call to a server that speaks SIP and RTP. Pipecat's docs describe SIP as the route for full call control, such as transfers and multi-party calls.
- A media stream over WebSocket. The carrier sends the call's audio to your server over a WebSocket, and you send audio back. Twilio's Media Streams documentation says it streams raw audio from live phone calls over WebSockets, and with a bidirectional stream your application can send audio back into the call (Twilio Media Streams). Twilio's message reference shows the audio as
audio/x-mulawat a rate of 8000 (Twilio WebSocket messages). Pipecat's telephony guide lists Twilio, Telnyx, Plivo and Exotel for WebSocket calls, and describes the control as basic and managed by the carrier (Pipecat telephony overview).
So WebSocket does appear in phone agents, and that does not contradict the section above. In our view the reason is where it sits: the WebSocket runs between the carrier's servers and yours, which is the stable, server-to-server kind of link where WebSocket works well. The caller's own phone line is not carried by your WebSocket at all.
Which transport should you use for a browser, an app or a phone call?
Use WebRTC for browsers and apps, and let your carrier's offer decide for phone calls. Here is the same advice by case.
Where the caller is: A browser widget on your site
- Use
- WebRTC
- Why
- Callers are on uncontrolled networks, and the browser already provides echo and noise handling
- Watch out for
- Microphone permission, and a fallback message when the network blocks media
Where the caller is: A mobile app
- Use
- WebRTC
- Why
- Phones switch between Wi-Fi and mobile data, and WebRTC handles network changes
- Watch out for
- Background and screen-lock behaviour on each phone
Where the caller is: A phone call, simple setup
- Use
- Carrier media stream over WebSocket
- Why
- The carrier handles the phone leg and gives you a stream, which is quick to connect
- Watch out for
- Basic call control as the carrier defines it. 8 kHz audio
Where the caller is: A phone call, full call control
- Use
- SIP and RTP
- Why
- Transfers, forwarding and multi-party calls are done at the SIP level
- Watch out for
- More to set up and operate
Where the caller is: Your server to a speech or model API
- Use
- Whatever the API documents
- Why
- Providers choose. OpenAI recommends WebRTC for its voice audio connections, and Google's Live API connects over WebSockets
- Watch out for
- Keep keys on the server, and use short-lived tokens for any browser
Where the caller is: A text-only bot
- Use
- WebSocket
- Why
- No audio, so no media problems
- Watch out for
- None for transport
The row about model APIs deserves a note. OpenAI's WebSocket guide says it recommends WebRTC for its voice audio connections, including server-to-server integrations, and also supports WebSockets (OpenAI voice WebSockets guide). Google's Live API guide describes connecting over WebSockets, from your server or from the client with ephemeral tokens (Gemini Live API overview). Our guide to speech-to-speech vs cascaded voice agents explains why the choice of model changes the choice of transport.
Servers can also hide the choice. LiveKit's documentation describes an open-source WebRTC server in which every client, agent and phone call joins a room as a participant, and in which telephony bridges phone calls into rooms as SIP participants (LiveKit documentation overview, and its Agents framework). That is one way to use WebRTC inside your system while phone calls arrive by SIP. Our guide to Pipecat for voice agents shows how one framework offers all of these transports.
How does the transport affect latency and call quality?
It adds a few parts of the wait, and the biggest effect is on bad networks, not good ones. A jitter buffer smooths uneven packet arrival and costs delay. A codec costs encode and decode time. A relay adds a hop. None of these is large on its own, and we do not give a figure, because we have no verified numbers for Indian networks and we will not invent one.
What you can say is how to look. Time the agent from the caller's side: the last word the caller says to the first sound the agent makes. Then repeat on a weak signal. A transport that looks good on office Wi-Fi and falls apart on a train is the one that will hurt real callers. Our latency guide shows how to measure and where the time goes.
How do you test a transport choice?
Test on the worst network your callers have, with the same script each time, and record both sides. This is a method, not a measurement, so run it yourself.
- Pick three networks. Office Wi-Fi, a weak mobile signal such as a lift or a moving train, and a congested shared network. A phone hotspot in a weak-signal spot works.
- Use one script. Read the same five lines, including a number read in groups of three or four digits, and one interruption.
- Record the caller's side and the agent's side. Listen for gaps, bursts of fast speech, echo and late replies.
- Time the reply. From the caller's last word to the agent's first sound, for each network.
- Test the reconnect. Turn the signal off for five seconds, then back on. Does the call recover, or end?
- For phone calls, call from a mobile and a landline. Carrier legs differ. Check the 8 kHz audio with your speech service. Our guide to testing a voice agent has a longer checklist.
How Bananaflow fits
Bananaflow is an AI voice agent platform for calls in India. You can put a voice and chat web widget on your site, as a floating button, inline, or headless, and there are a REST API and Python and TypeScript SDKs. You can also take inbound and outbound phone calls on your own numbers and carrier. See telephony. Before any real call, test the agent by voice or text in your browser. Each call gives you a transcript and a recording to check for the problems above. Our conversational AI platform page is the overview.
What should you remember about WebRTC vs WebSocket?
- WebRTC is built for live media over UDP, with a speech codec, jitter handling and a way through firewalls. WebSocket is a general channel over TCP.
- On TCP, one lost packet delays everything behind it. For voice that means late bursts instead of a short gap.
- Use WebRTC for browser widgets and mobile apps. Use WebSocket for server-to-server links and text bots.
- Phone calls arrive by SIP and RTP, or as a carrier media stream over WebSocket. The audio is usually G.711 at 8 kHz.
- Model APIs differ: OpenAI recommends WebRTC for its voice audio, and Google's Live API connects over WebSockets. Read each one's current guide.
- Test on a weak network and time the reply from the caller's ear.
FAQ
Frequently asked questions
What is the difference between WebRTC and WebSocket?
WebRTC is a browser standard for live audio and video that runs mostly over UDP and includes speech codecs and a way through firewalls. WebSocket is a general two-way channel layered over TCP, defined in RFC 6455. For voice agents, WebRTC suits callers on unpredictable networks, and WebSocket suits server-to-server links and text bots.
Is WebRTC better than WebSocket for voice AI?
For a browser or mobile app on an open network, yes. Pipecat's documentation recommends WebRTC for client-to-server voice because TCP holds up everything behind a lost packet, while WebRTC lets audio carry on after a short gap. WebSocket is a fine choice between two servers on stable networks, which is how many carriers stream phone-call audio.
Why does WebSocket audio stutter on a weak network?
WebSocket runs over TCP, which resends lost packets and holds back every packet that arrived after them. The listener hears a pause and then a burst of late audio. This is called head-of-line blocking. WebRTC mostly uses UDP, so a lost packet becomes a short gap and the audio moves on.
Do phone calls to an AI agent use WebRTC or WebSocket?
Usually neither on the caller's leg. A phone call is set up with SIP and carried as RTP audio, often G.711 at 8 kHz. Your agent then attaches by SIP, or by a carrier media stream over WebSocket. Twilio's Media Streams, for example, sends raw call audio to your server over WebSockets and shows it as mu-law at 8000.
Which transport should a browser voice widget use?
WebRTC. Browsers already provide echo cancellation and noise suppression in the WebRTC stack, and WebRTC handles network changes and reconnects. A raw WebSocket leaves timing, echo handling and reconnection to you. Test the widget on a weak mobile signal as well as on Wi-Fi before you decide.
Do OpenAI and Google's live voice APIs use WebRTC or WebSocket?
OpenAI's voice WebSocket guide says it recommends WebRTC for its audio connections and also supports WebSockets, and it documents a SIP connection for phone calls. Google's Live API guide describes connecting over WebSockets, either from your server or from the client with ephemeral tokens. Read each provider's current guide, because both change.




