Wire an agent to a number

An OpenPhonex agent is the phone-side configuration that decides how a call is heard, thought about, and spoken. There are three modes:

• external_audio — your own audio agent, connected over a webhook + audio stream URL. Use connect_external_agent for the common case.
• external_text — your own text agent; OpenPhonex handles STT/TTS around it.
• hosted — OpenPhonex runs the conversation for you. The default engine is Standard speech-to-text, a model, then text-to-speech. You can instead choose GPT-Live 1, which listens and speaks at once.

1. Create the agent

you › connect my audio agent at wss://my-agent.example/stream
→ connect_external_agent(...)        # agents:write

create_agent is the full form (mode, first message, system prompt, voice stack). connect_external_agent is the shortcut for an external_audio agent with webhook and audio-stream URLs.

2. Choose a voice stack

For hosted or external-text agents, pick a conversation engine first. Standard uses STT / LLM / TTS providers from the catalog. GPT-Live 1 uses a GPT-Live voice and a backend model instead. get_agent_catalog lists selectable providers plus conversation_engines; filter Standard rows by component and language (for example, language=et requests the current Estonian options).

→ get_agent_catalog(component="tts", language="et", metered_only=true)   # agents:read or coverage:read
→ set_agent_voice_stack(agent_id="ag_...", ...)        # agents:write

Set metered_only=true only when you want component-metered choices; omit it to see every runnable adapter.

set_agent_voice_stack validates the effective selection against the declared runtime and its credential and billing contracts. The catalog exposes the runtime-declared combinations; a provider voice's advertised language count alone does not establish that its complete call stack is runnable. The tool maps to PATCH /v1/agents/{agent_id}/voice-stack; its language fields match the generated API reference.

Keep a fixed language per call, or enable Auto multilingual and choose all supported languages or your own subset. The underlying Follow the caller setting is optional: publish it only with a listening configuration that supports the accepted languages together. The AI agent may then switch among the call's admitted profiles on a clear request or stable language evidence, without changing its shared tools or business instructions. A voice's multilingual label alone is not proof of simultaneous listening support. See Multilingual calls for the API setting, boundaries, and test flow.

From MCP, use list_agents for a bounded newest-first page, get_agent for a single current projection, update_agent for ordinary mutable fields, and get_agent_voice_stack to read a published stack back. The key's organization is authoritative: omit the redundant organization_id, or repeat only the same workspace. A different workspace, project, or agent is rejected before a configuration write or preview render; treat that MCP error as a tenant validation failure, not a retry prompt for another workspace identifier.

preview_voice is the MCP Listen action. It has agents:write because it can contact a provider or validate a BYOK credential and is rate-limited per workspace; it is not a read-only preview. Browser-session create and settle remain browser/REST operations because they require a browser-owned LiveKit audio participant and later client settlement.

Optional outgoing ambience

An agent may also publish one optional audio_environment selection with its voice stack. It is strict and version-pinned: { preset_id, asset_version, volume }. Select it from GET /v1/audio-environments or MCP list_audio_environments; Off is an explicit selection with volume zero. An enabled preset needs a positive whole-number volume. Unknown fields, unknown presets, a missing asset version, and a mismatched version are refused rather than guessed or replaced.

This is an outgoing background track, shared across the agent's language profiles. It is never caller/STT input and does not change the chosen voice, language, or call's frozen voice-stack configuration. If you audition it with preview_voice, supply non-blank text; the exact text and selected language are preserved rather than substituted or translated. A preview is a bounded local audition, not proof of a receiver-side final mix.

Choose a voice, not a vendor. The Voice step shows voices available for your AI agent's language: play a sample, and picking a voice sets the provider, the model, and the exact voice id together. Voices in the sidebar is the same gallery unfiltered, for browsing across every provider and language before or without editing an AI agent.

Browsing and listening are open in every language a provider advertises, which is wider than the set an AI agent can take calls in — a call also needs transcription and a qualified stack. A voice you can hear in a language but not yet deploy in says so on its card rather than disappearing from it.

Each card shows only what the provider supplied — structured gender, its own tags, its own description — plus the source of each of those fields. Nothing is inferred from the audio: no guessed age, accent, personality, or quality score. Cost and latency each carry the source of that number, which may be a measurement from real calls, a figure the vendor publishes, or a catalog estimate that says it is one.

Galleries are bounded, ordered selections rather than an unrestricted provider dump, and they say when they are showing a slice. Filtering and search on the Voices page are answered by the server, so a voice on a later page is still found. An engine that cannot be billed yet, or that this catalog does not offer for calls in the language you are filtering by, stays visible and disabled with that reason rather than disappearing, and if an existing AI agent uses a voice outside the current selection, its exact stored choice stays in that agent's Voice step instead of being silently replaced.

The full Voices page has four horizontal, searchable multi-select filters: Language, Accent, Character, and Provider. Several values in one filter are an "any of" selection; filters together narrow the result. Character is only the provider's own style or persona metadata, never a gender proxy. The page derives its values from the cached provider inventory, so a new provider voice or declared language does not need a frontend list or manual promotion.

Use Hear it as your AI agent to listen before saving. It speaks any sentence you type — your AI agent's first message by default — in any language the provider advertises. The audio is never stored, and the request is bounded and rate-limited per workspace.

The gallery is backed by the provider's scoped cached inventory rather than a frontend-maintained voice list. New provider voices therefore appear after a cache refresh without a frontend release or human promotion step. A failed refresh serves the last good inventory, and a bounded partial inventory never proves that an unobserved voice was removed.

On the full library page, a cached audio sample is a fast path for a selected target language only when its catalog provenance proves that match. At present, an OpenPhonex ingestion-synthesized sample proves English; a provider sample with no spoken-language assertion is not assumed to be English. With no Language filter or chosen preview language, an existing recording can play as published and the row labels unverified spoken language rather than inferring one. A row with no suitable sample still offers Play, which makes the bounded, metered, rate-limited live preview request. If it cannot run, the page shows a local safe error rather than a provider credential, trace, or raw provider response. Several matching selected languages require an explicit choice in the row's searchable preview-language control; the library never silently chooses the first one. Entering your own text deliberately uses that live request and speaks the exact wording you enter without translation. With no typed text, the library uses a localized sample sentence for the chosen language when it has one; otherwise it asks you for text instead of substituting English.

Cartesia Sonic 3.6 follows that same contract. Its managed gallery contains only the provider's public, non-owned voices. If you use your own Cartesia credential, its owned or cloned voice IDs remain private to that credential's workspace and project; test the exact ID with your own text before saving it. The preview does not substitute a managed credential or a fallback voice.

On OpenPhonex credentials, a new selection must belong to the managed scoped inventory. With your own provider credential, validation and preview stay scoped to that exact credential; cloned and private voice IDs can work without becoming visible to another workspace. A preview started with your credential never falls back to an OpenPhonex one.

OpenPhonex does not automatically choose a default or recommended language or voice. Select both explicitly. If a provider later withdraws an existing stored voice, the editor keeps that exact choice visible with a warning; it is never silently replaced by a fallback voice. An exact provider response that the voice no longer exists blocks future call admission until you choose a new voice or a successful explicit validation clears the marker.

Several languages, one hosted agent

Choose the agent's supported languages, an opening language, and one shared voice. You do not need separate copies of its identity, instructions, tools, knowledge, and business policy. Optional language overrides can change a voice, transcriber, or exact greeting without duplicating the rest of the agent. Unspecified values inherit the shared settings; clearing an override restores inheritance. An exact greeting is optional, not a publish checklist for every language.

Saving the voice stack publishes the effective profiles. A browser test or outbound call selects a published language before admission, and the resolved configuration is frozen for that call. A later agent edit does not change an in-progress call or its quoted component identities. This is fixed-language calling, not automatic mid-call language switching.

See Multilingual outbound calls for the request fields and application boundary. A Voice Library language filter controls an audition; it does not grant the agent permission to call in that language.

Interruptions and barge-in

Every hosted phone call, in every language, requires 500 ms of continuous caller speech before it interrupts the agent. This is a floor, not a default: min_speech_ms tunes above it, and a phone call given a lower value is raised back to 500 ms. Shorter bursts of background noise were cutting agents off mid-word, which is why the floor exists. Browser and web calls do not apply this setting: it is still stored on the agent and returned to you, but those calls are not phone calls and keep the browser stack's own threshold. This deliberately reduces false interruptions from very brief background audio and backchannels while keeping a sustained request such as “stop” interruptible. Sustained background speech can still be treated as caller speech.

After the caller stops speaking, the agent waits for a fixed endpointing hold before it treats the turn as finished. An agent whose language has a trained turn detector holds for 200 ms and lets the detector decide; every other language holds for 800 ms, so that a mid-sentence pause is less likely to end the turn.

What turn_taking in set_agent_voice_stack adds is per-agent control over that threshold, the endpointing hold, and whether an interruption must also carry recognized words. min_speech_ms and silence_hold_ms are explicit numbers and apply in every language. interruption_sensitivity is a preset, and a preset is a name for a measurement taken on English audio, so it applies to a hosted English agent only — setting it on an agent in another language is accepted and returned, but that agent keeps the 500 ms floor and the platform hold. Set min_speech_ms or silence_hold_ms explicitly there instead. min_interruption_words has its own condition and is the one setting here with a non-zero default; both are described under the table.

The controls are applied when a call starts; saving a voice stack never changes an in-progress call.

If the runtime classifies a brief interruption as false, it resumes the agent’s speech.

Requiring words as well as duration

min_interruption_words addresses a different problem from the duration threshold. Raising min_speech_ms rejects short bursts, but a long stretch of non-speech audio — a television, a passing vehicle, a cough — can clear any duration threshold you would still be willing to interrupt on. Requiring a recognized word rejects that class outright: with min_interruption_words: 1, the agent yields the floor only when the caller has both held it for the duration threshold and actually said something.

Both conditions must hold. The setting never lowers the duration threshold and never makes an interruption happen sooner.

The default

A hosted English agent that has not set min_interruption_words runs with 1: one recognized word as well as the duration threshold. Agents in every other language default to 0 and run on the duration threshold alone, because the requirement is only enforceable where the transcriber can report a word in time (see below).

To turn the requirement off on an English agent, set min_interruption_words to 0 explicitly. An explicit 0 is stored and returned; leaving the setting out is what selects the default.

It is accepted and returned for any agent in any language, but it is effective only where the transcriber emits words fast enough to land inside the interruption decision — today that is the Deepgram stacks, which serve English and Estonian. On every other stack the setting is stored, returned unchanged by get_agent_voice_stack, and ignored at call time: those calls run on the duration threshold alone, exactly as they do with the setting unset. The distinction is the transcriber's timing, not the language: a transcriber that reports a caller's first word a second or two after they start speaking would refuse ordinary interruptions rather than only the wordless ones, which is worse than not having the setting.

Sounds that do not count as words

On hosted English calls, short listening noises from the caller do not count toward min_interruption_words. "mm-hm", "uh-huh", "yes", "yeah", "okay", "right", "sure", and hesitation sounds such as "uh" and "um" are what a caller says to show they are following, not to take the floor, so a transcript containing nothing but one or two of them will not interrupt the agent however long the caller holds it.

This is English-only, and deliberately so. Estonian is the other language whose transcriber is fast enough for min_interruption_words to take effect, and an Estonian agent that sets it runs the word requirement in full — but every word counts there, including "jah" and "okei", because the equivalent list has not been measured for that language. Applying an English list to it would refuse nothing while this page claimed otherwise.

The caller is still heard. What they said is delivered to the agent when the reply ends, so a caller who says "okay" is answered afterwards rather than ignored — they simply do not cut the agent off mid-sentence to do it.

Anything else counts, including short words that sound similar in isolation. "no" is a word; so is "wait", "stop", and "huh". So is any phrase that mixes the two: "yes but" carries a word the agent will stop for.

Expect a small added delay when the setting is on. On the Deepgram stacks transcription results arrive about once a second, so roughly half the time the caller's first word is already recognized when the duration threshold opens and the interruption is unchanged; the rest of the time it is delayed until the next result, by up to about half a second. A word recognized before the threshold opens costs nothing — the interruption still happens the moment the caller has spoken for long enough.

1 is the English default for this reason. Values above 1 require most of a phrase before the agent will stop, which callers experience as an agent that does not listen.

Four protections are not settings and are not English-only. On every hosted phone call, in any language:

• The runtime holds what the caller is transcribed as saying while the agent is preparing or delivering a reply, and delivers it once that reply ends. Background speech picked up by transcription therefore cannot cut the agent off mid-sentence.
• The agent's opening line cannot be cancelled before its first audio frame reaches the caller, so a "hello?" spoken over the greeting — or mis-transcribed background noise — can no longer open a call in silence. Once the greeting is audible the ordinary interruption rules apply to it. If the opening line cannot be synthesised at all, a short line in the call's own language is played in its place; that substitute exists for English, Estonian, Sinhala and Tamil, and a call in another language, or one whose substitute also fails to synthesise, still opens in silence.
• When an interruption does qualify, the agent plays on for a bounded moment instead of stopping at the exact instant it is cut, so the sound it is part way through finishes rather than being clipped. This is a time budget of up to 300 ms, and it decides when the agent stops rather than where in a word: an interruption early in a long word still lands inside it, one syllable later. What it buys is a less abrupt stop and a shorter gap when a brief interruption turns out to be false.
• When the agent does stop, the last fraction of a second it had already committed to the call is played out under a quick fade rather than dropped, so the final word ends instead of being clipped part way through. On agents whose voice provider reports word timings — today ElevenLabs and Cartesia, which covers English and Estonian — the stop is moved to the end of the word being spoken whenever that end is close enough to reach; otherwise, and on every other voice, the fade is used. Either way the agent is silent within about half a second of the interruption qualifying, and it never plays anything the caller was not already about to hear.

None of these needs configuration, and browser tests are unaffected.

3. Point a number at it

Number rules decide what happens when someone calls a number and which agents can use it to place calls. Update them with update_number_rules (numbers:write); read them with get_number_rules (numbers:read).

To take a number back off an agent, POST /v1/phone-numbers/{number_id}/detach-agent (numbers:write). It is a pure control-plane unbind: the number stays imported and owned by the workspace, its rules become disabled, and no carrier work happens. Inbound calls are declined until you attach an agent again or change the rules. Detaching a number that already answers without an agent replays as a no-op with idempotent: true. The same action is available in the workspace on the agent's Phone section and on the number's page.

If you decided to detach from a view of the number that may since have moved on — a page a teammate reassigned under you — name the binding you expect. Send expected_agent_id, or expected_agent_endpoint_id when the rule answers through an endpoint you cannot resolve to an agent. A number that by then answers with a different binding is refused with 409 and nothing changes, so you cannot take a colleague's fresh assignment offline by confirming a stale screen. Naming an identity the rule no longer carries is refused the same way: an expectation that cannot be confirmed is not treated as confirmed. A binding that is gone entirely is not a conflict — the state you wanted already holds, so it replays as the idempotent: true no-op. Send neither field to detach whichever agent answers.

Check incoming-call configuration

Open Phone numbers and select the number. The existing Activation checks distinguish purchase and carrier-order completion from incoming-call configuration. A purchased, active number can still need attention before its AI agent can answer.

A warning in the number list points to the explanation in the selected number's checks. Follow the suggested action, such as assigning an AI agent or contacting support when pricing is unavailable. Supported call-in/call-out capabilities describe the number's features; they do not prove that a call can connect right now.

These checks describe saved configuration. Available funds, current capacity, call policy and carrier reachability are still checked when a call arrives. Outbound support depends on the selected destination and is checked when the call is requested. Refresh the page after correcting a prerequisite.

Bring your own agent

Two of the three modes let your own application do the thinking. Both are created in the workspace from Agents → Bring your own agent, or over the API with POST /v1/agents.

Webhook agent (external_text)

OpenPhonex runs speech-to-text and text-to-speech around your app, so the app only deals in text.

• What OpenPhonex sends. A POST to your webhook URL for each call lifecycle event and each transcript turn, carrying X-OpenPhonex-Event-Id, X-OpenPhonex-Call-Id, and X-OpenPhonex-Signature when a webhook secret is set.
• What your app returns. The words to say next, as a text or NDJSON response. OpenPhonex speaks them to the caller.
• Where the URL goes. The Webhook URL field on the create form, or webhook_url on POST /v1/agents with "mode": "external_text".

Realtime audio agent (external_audio)

Your app hears and speaks the call itself; OpenPhonex runs no speech stack for it.

• What OpenPhonex sends. Call lifecycle events to your webhook URL, and bidirectional call audio to your realtime audio URL — the gateway opens that wss:// connection while the route is active.
• What your app returns. Audio, on the same connection.
• Where the URLs go. The Webhook URL and Realtime audio URL fields on the create form, or webhook_url plus audio_stream_url on POST /v1/agents with "mode": "external_audio". connect_external_agent is the shortcut for the same pair.

Webhook signature verification, idempotency and backpressure handling stay in your app. GET /v1/agents/{agent_id}/handoff returns the generated setup contract for either mode: the endpoints it will call, the events to expect, and the headers to verify.

To send a phone number's calls to a realtime audio agent, open Phone numbers, select the number, and pick Send live audio to my app under "When someone calls".