On the surface, building a great AI voice agent for sales calls seems simple.
All you have to do is ask yourself “What makes a good sales rep?”, identify those qualities, and encode them into your agent.
The first two steps are easy. We’ve all been on the receiving end of good and bad sales calls, and know the difference. Bad calls feel detached and scripted, and never build enough trust for honest conversations. Good calls feel warm and relevant. The rep sounds like they genuinely care about you, they get your details right, and they respond to what you actually say, even if it’s unexpected.
But the third step, taking those qualities and encoding them into your agent, is the tricky part. You quickly realize that capturing the human touch is harder than it looks, and filled with many little variables you might never have considered that make a truly great sales rep succeed.
We know this challenge first-hand. We’ve built AI voice agents that are deployed across many leading companies, and we’ve learned a lot along the way about what really matters.
In this article, we will walk you through one of those hidden variables we discovered that makes a meaningful difference in how a voice agent performs: timing. We’ll tell you…
- The timing that humans expect on a good sales call
- The technical reasons an agent might not match human timing
- How to get agents to sound natural while matching timing
- The right architecture to build an agent that matches human timing
The timing that humans expect on a good sales call
When we first considered timing as a variable in our AI voice agent’s performance, we set out to find a benchmark for how quickly people respond to each other in normal human-to-human speech.
We found that usually, in a human-to-human conversation, when one person stops talking the other person usually starts within about 200 milliseconds, sometimes even faster. Linguists have seen this pattern in ten languages across five continents, which suggests this might be a near-universal habit that’s hardwired into how we communicate with each other.
Yet that speed surprised us. We had been measuring the speed of speech, and knew that from deciding what to say, to making the sound, takes 600 milliseconds to say a single word, and 900 milliseconds to give a three-word reply. To respond within 200 milliseconds, you can’t wait for the other person to finish before you start thinking of your response. While they are talking you need to guess where they are heading and prepare what you are going to say, so you have your answer ready the moment they stop.
Once we saw that the benchmark for a natural-sounding response was somewhere around 200 milliseconds, we then looked at our own agents and found the problem. Our agents were taking almost 950 milliseconds to respond to prospects, nearly 5x slower than the natural pace of human speech.
This not only made our agents sound robotic, it also changed the tone of their responses. People naturally interpret a slow response as expressing hesitance. If someone asks you if Tuesday works for a meeting and you quickly say “sure”, it sounds like you’re happy to take that call. But if you first make a long pause and then say “sure”, it will sound like you’re uncertain about it. Because our agents were taking so long to respond, our prospects were naturally interpreting everything they said poorly.
And there’s research to back this up. One study on English phone calls found that nearly a third of rejections and a quarter of hesitant acceptances came after 700ms, and only 2% of genuine “yes” responses took that long. While this study was small, it gave us a good guideline, and helped us set concrete thresholds for how quickly our voice agents needed to sound to come across as human.
We decided our agents needed to respond to prospects between 300 milliseconds (so they didn’t give a completely instant response that sounded canned) and 700ms (so the prospect never had to sit in a long silence). Great reps follow this naturally, and we needed to find a way to make our agents match it.
The technical reasons an agent might not match human timing
After we defined this acceptable response time of 300 – 700 milliseconds, and saw that our agents were responding outside of it, in 950 milliseconds, we dug in to see how to fix this problem.
The truth is, before this exercise we did not know our agents were actually taking 950 milliseconds to respond to prospects. The model we had developed, and the speed figures we had been tracking, told us that our agent was responding much faster than that. Here’s why there was such a big gap.
We had been tracking how long our agents took from the moment it decided the prospect was done speaking to when it sent the first bit of audio. But we were not taking into account some technical items like the time it took our agents to decide if the prospect had stopped talking, network delays, buffering, and how long the phone needed to create and send the audio to the prospect.
Once we took those into account, we saw the problem. Our internal measurements were accurate, but they didn’t tell the whole story. There was extra silence created by technical latency that meant what prospects actually experienced was different from what our back-end benchmarks were saying they did.
This helped us see that our timing gap was primarily a technical problem, and we dug in to see how we could get our agents to respond faster (without needing to build a faster model).
The answer was hiding in our agent’s technical details. Each of our agent’s replies streamed to a text-to-speech provider over a WebSocket, but the provider closed the socket after every reply. This meant that every reply had to open a new connection, and that setup handshake sat right between the prospect’s last word and the first thing they heard back. Opening those connections should be quick, but in our case, some mix of distance and vendor setup added up to create a long wait.
The fix was simple: we began to keep a spare connection open ahead of time. The handshake still took the same amount of time, but it began to happen while the prospect was still talking, so they didn’t have to wait for it before they could receive a response. This alone dramatically increased the speed of our agent’s responses, and made them feel substantially warmer, more natural, and more human.
Going through this process also changed how we measure the speed of our agents, and the targets we are aiming for. Instead of just setting internal benchmarks that can be misleading or hide latency on the prospect’s end, we are moving towards goals that better reflect the prospect’s experience on a call.
For example, we now measure our performance against things like the percentage of replies that land within a timing window that carries the right meaning to a prospect, for example executing 95% of our replies under 700 milliseconds. We learned the hard way that certain benchmarks and averages can be misleading, and we seek to measure ourselves against, and report on, figures that mean something.
How to get agents to sound natural while matching timing
After we solved the baseline technical issue that was creating hidden latency for our agents, we began to tackle some of the less measurable aspects of good timing for a sales call. Just like responding within that 300 – 700 millisecond window, each of these was something we saw great sales rep do naturally, that we needed to train our agents to do as well. These included:
- Starting before they were certain. As noted above, people naturally begin to form their answers while the other person is still talking, and agents need to do the same. We began to have our agents begin to draft a reply based on what a prospect had said so far, and then toss it if the prospect kept going. The agent might waste a few drafts, but they eliminated a lot of lag time to give a response. To make this safe, we ensured our agents never committed to something irreversible, like booking a meeting, in a speculative draft.
- Saying something before they knew the answer. When a great rep needs to look something up, they’ll say “Let me check that for you” and then go check. That one line reassures the prospect they’ve been heard and resets expectations. But if a rep, or an agent, just goes silent for two seconds, it makes the prospect feel like nobody is there, which is even worse than being slow. The tricky part is knowing when to say a line like this, and when to give the follow-up. If an agent says “Let me check on that for you” then instantly gives a full answer, that sounds odd, but so does saying they will check, then going silent for a long stretch.
- Making the silence sound alive. A phone line with no background noise sounds broken or unnatural, but fake typing or shuffling paper feels too artificial and makes prospects distrust the agent. We’ve found that adding a low bed of room tone (e.g. the faint hum of a real office) under the whole call, as its own audio track separate from the agent’s voice, solves this problem.
- Never interrupting the prospect. Great reps never interrupt, and they never talk over a prospect. Doing both is worse than being slightly slow, and we’ve found it’s better to ensure an agent accurately identifies when a prospect stops talking, even if it slows them down a bit, than it is to optimize only for speed and risk them interrupting or speaking over the prospect.
The right architecture to build an agent that matches human timing
Everything we found, fixed, and suggest in this article can only happen if an agent is built properly. Even though every voice agent has the same basic jobs of understanding what a prospect says, deciding what to say back, and speaking it out loud, how those jobs are wired together varies meaningfully from agent to agent, and determines if that agent feels warm, smart, and fast. Getting this right is just as important as anything we have covered so far in this article. Here’s how.
Most voice agents follow one of three common setups:
Each row is one setup. The first is a cascade. One system turns the prospect’s speech into text, a language model reads the text and writes a reply, and a third system turns the reply back into speech. This is the standard arrangement, and you can swap out each piece for a better one, but in this setup the model never really hears the prospect. It just reads a transcript that does not communicate things like hesitation and tone that a great rep would pick up on, and adapt to.
The second is a single speech-to-speech model. Audio goes in, and audio comes out with no text in between. The agent hears how something was said, it can answer in kind, and it can respond faster, but you tend to lose control over how the agent is configured. One model steers everything, so you are locked-in to certain things like the short list of voices the agent’s provider offers, how names are pronounced, and whatever level of reasoning that model has.
The third setup is what we run, and we call it split-voice, and it takes the good parts of a cascade and speech-to-speech model, without their downsides. In this setup the main model hears the audio itself and picks up on things like hesitation and tone, and properly interprets what the prospect means, not only what they say. The voice model executes in a separate step, allowing us (and our customers) to pick things like how the agent sounds, how it pronounces names, and overall what voice it speaks with.
In addition, we run co-drivers, which are fast models that listen to the same audio in parallel and influence the main model with unspoken messages. They help micromanage the conversation and what/how the main model should speak.
.webp)
Some co-drivers run on every call, and others in specific scenarios. For example, one collects email addresses, another gets pronunciation right, and a third enforces safety guardrails. A fourth listens for background noise, tells the agent if there is too much, and has the agent tell the prospect it’s having trouble hearing, so the main model does not answer something the prospect never said. We also build more co-drivers for each use case, such as a co-driver that keeps the agent from giving tax advice.
Overall, we’ve found this split-voice setup, along with use-case specific co-drivers, offers the best way to build agents that match human timing, sound warm and intelligent, and adapt well to the shifting demands of unpredictable sales conversations.
Getting timing right is only one part of a great voice agent
We’ve reached a point where our agents consistently match a great rep’s timing, and we’re now moving on to getting our agents to better mirror the prospect. This means matching the prospect’s pace, like waiting out a slow thinker or keeping up with a fast talker, as we’ve learned that what feels like normal timing can change meaningfully from person-to-person.
Additional challenges have come up, like choosing the right voice for a specific prospect, or reasoning through tough objections. We’ll tackle those the same way we have tackled timing, by starting from what great reps already do, and finding ways to get an agent to do it as well. While a great rep does all of these things without thinking, an agent only does it if those habits are documented and built in, but we’ve found that encoding these nuances is what make our AI voice agents great.
To learn more about our AI voice agents, or to hear them in action, book a demo or test it out live on our site.
