Realtime voice agent

# Realtime voice agent

A voice agent with median voice-to-voice latency under 800 ms (end of user speech to first agent audio), barge-in that stops speech within 200 ms, at least 90% task completion on 20 scripted conversations, reachable from a browser and a phone number.

## Recommended stack

- **LiveKit Agents 1.8 (Python 3.10-3.14; 2.0 is in beta)** - AgentSession, turn detector, tools, built-in test framework with judges. Apache-2.0, streams audio through STT-LLM-TTS or realtime models, and ships unit-test helpers.
- **LiveKit server (LiveKit Cloud or self-hosted)** - WebRTC transport and SIP telephony. One agent serves browser rooms and phone calls.
- **STT, LLM, and TTS via LiveKit plugins or LiveKit Inference** - Speech and reasoning models. Swap providers per stage to hit latency and cost targets without code changes.
- **pytest** - Turn-level tests. The framework asserts on messages, tool calls, and handoffs, with LLM judges for intent.

## Build steps

1. Write the scope: one goal (e.g. book an appointment), 3-5 tools, things the agent must never do, and a persona under 150 words written for speech (short sentences, no lists).
2. Install the LiveKit CLI (2.18.8+), run `lk cloud auth`, then scaffold with `lk agent init my-agent --template agent-starter-python` and `uv sync`.
3. Configure AgentSession with streaming STT, LLM, and TTS plus the turn detector (turn_handling with inference.TurnDetector()); confirm interruptions stop TTS.
4. Add tools as function tools; for calls over 1 s, have the agent say a short filler first and keep tool timeouts at 5 s.
5. Run `lk agent console` to talk to it in the terminal, then `lk agent dev` and connect a web front end (agent-starter-react).
6. Write pytest tests in text mode for 20 scripted conversations, asserting tool calls and using `.judge(llm, intent=...)` for open answers.
7. Log per-stage latency from session metrics (end of utterance, first LLM token, first TTS audio) and swap providers until the median is under 800 ms.
8. Add a SIP trunk and dispatch rule, test inbound calls on real phones, and run production with `uv run src/agent.py start`.

## Watch out for

- Long, list-shaped answers are painful to hear. Limit replies to 1-2 sentences and cap max tokens.
- Silence-only endpointing cuts people off mid-thought. Use the turn detector model.
- Markdown, URLs, and raw numbers read badly aloud. Normalize text for speech before TTS.
- Phone audio is 8 kHz and noisier than browser audio. Test STT accuracy on real calls, not only the browser.

## Definition of done

- A console-mode conversation completes the task end to end.
- The pytest suite passes at least 18 of 20 scripted conversations.
- The latency log shows median voice-to-voice under 800 ms over 20 calls.
- Calling the phone number completes the task, and talking over the agent stops its speech.
- A recorded barge-in test shows agent audio stops within 200 ms of user speech.
- The agent-starter-react front end connects to the agent and completes the task in a browser.