What's the conversation in San Francisco about how to "Build Smarter Voice Agents"? Our CTO, Adam Schuld, recently joined a live panel at AssemblyAI's San Francisco office to discuss this topic, joined by leaders at Retell and AssemblyAI. The conversation covered what it actually takes to move a voice agent from an impressive demo to a system that holds up in production, day after day, call after call.
Here are six of the biggest takeaways from Adam's side of the conversation.
Asked to name the single biggest factor behind a successful voice AI deployment, Adam didn't hesitate: context. A demo earns its "wow" from raw capability. A production system earns trust by feeling personalized — by remembering who the customer is, recalling prior conversations, and using that history to stay aligned with what the caller actually needs. That sense of continuity is what makes a system feel intuitive rather than scripted, and it's a component teams tend to underinvest in.
Good engineering fundamentals — resilient, flexible systems — pay off long before launch, and measuring performance before shipping matters. But Adam was candid that no amount of pre-launch testing fully prepares you for real users. Different geographies, different age groups, different edge cases: production is where those surface. The teams that win aren't the ones that avoid testing in production; they're the ones that treat what they learn there as a moat, feeding those hard-won lessons back into the product to build real differentiation.
Adam shared an anecdote about a feature the team has had mixed reviews on: adding a typing sound effect to smooth over processing lag to improve the UX of latency. Some people liked it. But many customers hated it. A little add-on could create a genuinely polarizing moment — for those that felt it deceptive, like the system was trying to trick them into believing they were talking to a human, it would backfire. The lesson: conversations have real mechanics, and trust must be earned over the conversation quality. Faking being human may be worse than clearly-AI but highly functional.
Adam introduced a concept still new to much of the audience: loop engineering — a step beyond manually tweaking prompts, toward building systems that can cycle through changes against a defined, measurable metric and know automatically whether a change actually improved things. He noted that a large share of his own engineering time now goes into building the evaluation scaffolding that makes this possible for voice agents. His wish for the near future: tools that can safely automate more of that iteration loop, so teams spend less time manually re-testing every change.
One of the more striking data points Adam shared: in their own testing, callers who had to repeat information reported roughly 95% negative sentiment. If a caller already gave their name and address three calls ago, having to repeat it is close to a guarantee of a bad interaction. His takeaway is that this needs to be a first-class design priority — systems need a mechanism to carry forward what matters across calls. Super's answer is something they call a "scratchpad": a structured store that runs as a parallel process during the conversation, capturing relevant profile details as they come up so later interactions can draw on that history instead of asking again.
The last takeaway was about scope. Going vertical — building for a specific use case like apartment leasing, rather than a general-purpose assistant — means the goals are narrow and well defined: check availability, schedule a tour, book it. That narrowness lets teams fine-tune deeply around a small, discrete set of tasks instead of facing what Adam called "the Pandora's box" of every possible thing a general system could be asked to do. The constraint isn't a limitation; it's the advantage.
Watch the full panel on "Build Smarter Voice Agents"
Ready to see how voice AI is increasing leverage for property management companies? Book a demo with Super today.
