Speech to Speech AI: Why ChatGPT and Gemini Now Sound Human

Ask ChatGPT's voice mode or Gemini Live a question and it answers almost before you finish your breath. It can be interrupted mid-sentence, it can sound amused, and it can hear when you sound unsure. Two years ago, talking to an AI felt like leaving a voicemail and waiting for a reply.

The change has a name: speech to speech AI. Instead of turning your voice into text, thinking in text, and reading the answer out loud, a single model listens to sound and replies in sound. OpenAI says its GPT-4o model can answer spoken input in as little as 232 milliseconds.

By the end of this piece you will know how the old voice assistants worked, what changed under the hood, why that change makes voices sound more human, and where these systems still trip up.

What is speech to speech AI?

Speech to speech AI is a model that takes audio in and produces audio out, without a written transcript sitting in the middle as the main step. A "model" here just means the trained AI system that does the work, the same kind of thing that powers a chatbot.

The key idea is that the model works with sound directly. It hears pitch, pace, pauses and background noise, not just the words. When it answers, it generates the voice itself rather than handing a script to a separate reading program.

You have probably used one already. ChatGPT's Advanced Voice Mode runs on this approach, and Google's Gemini Live uses what Google calls "native audio", meaning the model was built to handle sound as an input and output in its own right.

Do not confuse it with text to speech, which reads written words aloud in a synthetic voice. Text to speech starts with text. Speech to speech starts with your voice and answers with a voice.

How did voice assistants work before?

Older voice assistants were a relay race between three separate programs. OpenAI describes its own pre-2024 Voice Mode this way in the GPT-4o announcement: one simple model transcribed your audio into text, GPT-3.5 or GPT-4 read that text and wrote a text reply, and a third simple model turned the reply back into audio.

Engineers call this a "cascade" or "pipeline": each stage waits for the one before it to finish. Every handoff adds delay. OpenAI reports that this setup took 2.8 seconds on average with GPT-3.5 and 5.4 seconds with GPT-4. In a conversation, five seconds of silence feels like the line went dead.

Delay was only half the problem. Once your voice became plain text, everything except the words was gone. OpenAI's own words: the main model "can't directly observe tone, multiple speakers, or background noises". It also could not laugh, sing or express emotion in its reply, because the final voice came from a separate program reading a script.

The researchers behind Kyutai's open model Moshi make the same point in their paper. They write that these pipelines add "a latency of several seconds", lose information such as emotion, and force the conversation into strict turns that ignore interruptions and people talking over each other.

How does speech to speech AI work?

The short version: one model does the listening, the thinking and the talking. In May 2024 OpenAI said it "trained a single new model end-to-end across text, vision, and audio" for GPT-4o. "End-to-end" means one network is trained on the whole job from input to output, instead of three programs glued together.

To do that, the model needs a way to treat sound like language. Here is the plain-English version of the trick most of these systems use.

Sound becomes tokens

Chatbots read and write "tokens", small chunks of text a bit shorter than words. Speech models do something similar with audio. A tool called a neural audio codec (a compressor trained by AI) chops your voice into short slices and turns each slice into a compact code the model can read, much like a word token.

The Moshi paper gives a concrete example. Its codec, called Mimi, produces audio tokens at 12.5 frames per second, so each slice covers 80 milliseconds of sound. Those codes keep information about how something was said, not only what was said.

The model answers in tokens, then those become a voice

The model predicts the next audio tokens the same way a chatbot predicts the next word. The codec then turns those tokens back into a waveform you hear. Because the reply is generated as sound from the start, the model can choose to sound warm, hesitant or excited.

Moshi adds one more helpful step that its authors call "Inner Monologue": the model writes a time-aligned text version of what it is about to say just before it says it. The paper reports this improves how factual and well-formed the spoken answers are. So text has not vanished entirely; it rides along as a guide rather than acting as the bottleneck.

Listening and talking at the same time

Old assistants worked like a walkie-talkie: you talk, then it talks. Moshi is "full-duplex", the term for a line where both sides can speak at once, like a phone call. It models your voice and its own voice as two parallel streams, so it can handle interruptions and quick "mm-hm" replies. The paper reports a theoretical latency of 160 milliseconds and 200 milliseconds in practice.

Why does ChatGPT voice mode sound more human now?

Mostly because of speed and what the model can hear.

On speed, OpenAI says GPT-4o answers audio in 232 milliseconds at best and 320 milliseconds on average, which it compares to human response time in conversation. That is roughly a tenth of the 2.8 to 5.4 seconds the old Voice Mode needed. Latency, the gap between you stopping and the AI starting, is the single biggest thing that makes a call feel natural or awkward.

On hearing, a model that receives your actual audio can notice that you sound rushed, or that two people are talking, or that a dog is barking. Google's developer docs for its Gemini Live API say the model "adapts response style and tone to match the user's input expression" and that users can interrupt it at any time.

There has also been steady progress since the first launch. OpenAI's August 2025 model, gpt-realtime, scored 82.8% on Big Bench Audio, a reasoning test with spoken questions, up from 65.6% for its December 2024 model. In December 2025 Google upgraded Gemini 2.5 Flash Native Audio and said it now follows developer instructions 90% of the time, up from 84%.

These figures come from each company's own tests, run in different ways, so treat the table as a sense of scale rather than a head-to-head race.

What can you actually do with it, and where does it fall short?

For everyday users, the clearest win is conversation that does not feel like a phone menu. You can cut the assistant off when it rambles, change topic mid-sentence, or ask it to slow down.

Translation is another. In the same December 2025 announcement, Google introduced live speech-to-speech translation in the Google Translate app, covering over 70 languages and 2,000 language pairs. Google says it preserves the speaker's intonation, pacing and pitch, so the translated voice sounds less flat.

For businesses, the draw is customer service agents that can answer the phone. When OpenAI opened its Realtime API to developers in October 2024, it said they "no longer have to stitch together multiple models", and that the old chain "often resulted in loss of emotion, emphasis and accents, plus noticeable latency". An API is simply the connection developers use to plug a model into their own apps. The launch pricing worked out to roughly $0.06 per minute of audio in and $0.24 per minute out; gpt-realtime later cut prices by 20%. If you follow AI costs, our look at how LLM API prices are falling covers the wider trend.

The trade-offs are real, though:

• Less control. With a pipeline, a company can inspect and edit the text at each step. A single audio model is harder to audit, which matters in regulated fields like banking or health.
• Benchmarks are company-run. The scores above come from OpenAI and Google themselves. Independent tests are still catching up.
• It is still a chatbot. A natural voice does not make the answer more accurate. The model can be wrong in a very convincing tone, so check anything important.

Open research matters here too. Kyutai released Moshi publicly, and the paper says its language backbone, called Helium, has 7 billion parameters (the adjustable settings a model learns during training). If you are curious why openly released models matter, our explainer on open-weight models walks through it.

What to watch next

Three things are worth tracking. First, whether full-duplex behaviour, the "both talk at once" style Moshi demonstrated, becomes standard in the big consumer assistants. Second, whether independent benchmarks for spoken reasoning and interruption handling emerge, so we are not relying on vendors' own scorecards. Third, live translation: Google says it plans to bring live speech translation to more products, including the Gemini API, in 2026.

The short takeaway: voice assistants sound human now because they stopped turning your voice into a text message first. That one design change cut the wait from seconds to a fraction of a second and let the AI hear how you say things, not just what you say.