# Replace OpenAI Realtime or Gemini Live with VoxEvolv

> Machine-readable migration guide for moving an existing browser voice application from a vendor-specific realtime API to the provider-independent VoxEvolv session API. The VoxEvolv SDKs are recommended convenience layers, not mandatory transports.

Use protocol version `1`. The API base for this deployment is `https://voxevolv.com/v1`.

Read these sources before generating integration code:

1. VoxEvolv integration rules: `https://voxevolv.com/llms.txt`
2. This migration guide: `https://voxevolv.com/realtime-migration.txt`
3. Live VoxEvolv OpenAPI contract: `https://voxevolv.com/v1/openapi.json`

The OpenAPI document is authoritative for HTTP paths and schemas. This guide is authoritative for migration intent. Do not invent compatibility endpoints or forward OpenAI/Gemini wire messages to VoxEvolv.

## What changes

VoxEvolv is an architectural replacement, not a wire-compatible proxy for OpenAI Realtime or Gemini Live.

Before:

```text
browser -> vendor WebRTC or WebSocket -> OpenAI/Gemini model
```

After:

```text
trusted backend -> POST /v1/sessions/ -> VoxEvolv control plane
browser -> @voxevolv/client or protocol-v1 LiveKit client -> VoxEvolv worker -> selected providers
```

Provider choice, model configuration, reusable tools, and capabilities move into an immutable published VoxEvolv profile. Profiles may also permit inline client tools at session creation. Application code uses the same VoxEvolv HTTP and realtime protocol surface whether that profile uses OpenAI Realtime, Gemini Live, or a cascaded STT -> LLM -> TTS pipeline.

## SDKs are optional

Neither the server SDKs nor `@voxevolv/client` are required. A backend can call
the HTTP session API directly. A browser can use the self-contained client
bundle from plain HTML without npm, or implement protocol version `1` directly
with a LiveKit-compatible client.

The direct browser path must join with `connection.url` and `connection.token`,
publish and play audio, send `client.hello`, validate and order realtime
envelopes, suppress duplicates, replay sequence gaps, implement context
commands and browser-tool RPC, renew room tokens, and end the scoped session.
The SDK packages exist to provide those behaviors consistently; choosing not to
use them transfers that responsibility to the application.

No browser path may receive the organization API key. Session creation remains
backend-owned even when all browser transport code is written directly.

## Required setup

1. Create a VoxEvolv account and organization.
2. Connect the provider credentials required by the target profile.
3. Create an agent profile. Choose an OpenAI realtime engine, Gemini Live engine, or a cascaded pipeline.
4. Register reusable browser tools and all webhook tools in the Tools console and attach them to the profile. To pass client tool definitions at session creation, enable both Client RPC tools and Inline session tools in the profile's Capabilities tab.
5. Publish the profile after every configuration or tool-binding change.
6. Create an organization API key with `sessions:create`, `sessions:read`, and `sessions:end` scopes.
7. Store the organization API key only in a trusted backend or secret manager.

Do not expose OpenAI, Gemini, or VoxEvolv organization API keys to the browser. Inline session tools must use `type: "client"` and require both `inline_session_tools` and `client_rpc_tools` in the published profile. Inline webhook tools are rejected.

## Shared replacement implementation

### 1. Replace vendor token or call creation with a backend session endpoint

The application backend calls VoxEvolv. Generate one idempotency key per intended voice session and reuse it only when retrying that same creation request.

```bash
VOXEVOLV_API_BASE="https://voxevolv.com/v1"
VOXEVOLV_API_KEY="YOUR_BACKEND_ONLY_API_KEY"
VOXEVOLV_PROFILE="YOUR_PUBLISHED_PROFILE_NAME"
```

```ts
// Trusted application backend. Never run this code in the browser.
import { VoxEvolv } from "@voxevolv/server";

const voxevolv = new VoxEvolv({
  apiKey: process.env.VOXEVOLV_API_KEY!,
  baseUrl: process.env.VOXEVOLV_API_BASE,
});

export async function createVoiceSession({ intentId, userId, displayName }) {
  const session = await voxevolv.sessions.create({
    intentId,
    profile: process.env.VOXEVOLV_PROFILE!,
    instructions: "Help the participant clearly and concisely.",
    greeting: "Hello. How can I help?",
    participant: { identity: userId, displayName },
    media: { input: ["audio"], output: ["audio", "text"] },
    require: { transcriptFinal: true, bargeIn: true },
    metadata: { environment: process.env.NODE_ENV ?? "development" },
  });

  return session.browserCredentials();
}
```

The server SDK derives one deterministic idempotency key from `intentId`, preserves it across bounded 429/5xx retries, returns typed errors, and projects only `id`, `connection`, `access_token`, and `access_token_expires_at` to the authorized browser. A Python server SDK exposes the same lifecycle through `VoxEvolv.sessions`.

### 2. Replace vendor WebRTC or WebSocket code

Recommended: use `RealtimeSession`. It can be imported by a bundled application
or from a versioned, self-hosted browser bundle in plain HTML; npm is not a
runtime requirement.

```html
<button id="start-voice" type="button">Start voice</button>
<script type="module">
  import { RealtimeSession } from "/vendor/voxevolv/index.js";

  document.querySelector("#start-voice").addEventListener("click", async () => {
    const session = await RealtimeSession.create({
      endpoint: "/api/voice-session",
      apiBase: "https://voxevolv.com/v1",
      autoRejoin: true,
    });
    await session.connect({ microphone: true });
  });
</script>
```

The browser bundle is `sdk/js/dist/browser/index.js` in a source checkout. Copy
and serve a versioned build from application-owned static assets. For an
absolute VoxEvolv API base, browser lifecycle CORS is enabled by the default
`VOICE_SESSION_BROWSER_ORIGINS=*` policy. Configure exact page origins for
optional pinning, or leave the setting empty only when using a same-origin proxy.

UPGRADE WARNING: Earlier releases treated an unset
`VOICE_SESSION_BROWSER_ORIGINS` as disabled. The new default is `*`. Existing
self-hosted deployments that must remain closed need to set an explicit empty
value before upgrading; deployments that require origin pinning should set
their exact HTTPS origins.

Bundled JavaScript/TypeScript example:

```ts
import { RealtimeSession } from "@voxevolv/client";

const session = await RealtimeSession.create({
  endpoint: "/api/voice-session",
  apiBase: "https://voxevolv.com/v1",
  autoRejoin: true,
});

session.onStateChange((state) => {
  console.log("voice session state", state);
});
session.onStartupAttempt((event) => {
  console.log("voice session startup", event);
});
session.onRecovery((event) => {
  console.log("voice recovery", event.status, event.trigger, event.contextKey);
});
session.onConnectionQuality((quality) => console.log("call quality", quality));

session.onUserTranscript(({ text, final }) => renderUser(text, final));
session.onAgentTranscript(({ text, final }) => renderAgent(text, final));
session.onError((event) => console.error(event.payload));
session.onEnded((event) => console.log("session ended", event.payload.reason));

// Call from a user action so the browser may request microphone permission
// and play remote audio.
await session.connect({
  microphone: true,
  readyTimeoutMs: 15_000,
  apiBase: "https://voxevolv.com/v1",
});
```

The SDK joins the LiveKit room, publishes the microphone, plays subscribed agent audio, waits for `agent.ready`, validates realtime envelopes, suppresses duplicates, and requests durable replay after sequence gaps.

Pass tools and an application-owned audio track to `create()` before connection
when the application already owns or mixes its input track:

```ts
const session = await RealtimeSession.create({
  endpoint: "/api/voice-session",
  apiBase: "https://voxevolv.com/v1",
  microphone: false,
  audioTrack: mixedStream.getAudioTracks()[0],
  tools: {
    present_attachment: (arguments_) => presentAttachment(arguments_),
  },
});

session.onRemoteAudioTrack((track, stream) => {
  recordingMixer.addRemoteAudio(stream);
});

session.onRemoteAudioTrackUnsubscribed((track, stream) => {
  recordingMixer.removeRemoteAudio(stream);
});

await session.connect();
```

The SDK publishes a clone of `audioTrack`; disconnect and rejoin never stop the
application-owned source. `rejoin()` publishes a fresh clone of the same source
track. `microphone: true` and `audioTrack` are mutually exclusive.

### 3. Replace live context messages

Use `updateContext` for application state, instructions, document contents, or image snapshots that change while the session is active.

```ts
await session.updateContext({
  key: "application.current_task",
  mode: "replace",
  label: "Current task",
  text: "Help the user complete checkout.",
  respond: false,
});
```

Use `mode: "append"` for a new fact or event. Use `mode: "replace"` for state with one current value. Set `respond: true` only when the context update should immediately produce an agent response.

Do not use `updateContext` to change the model, provider, voice pipeline, tool manifest, or other published profile configuration.

### 4. Replace vendor function calling

Define reusable tools in the VoxEvolv Tools console, attach them to the profile, and publish the profile. Alternatively, pass client tool definitions in the backend session request's `tools` array after enabling and publishing both `inline_session_tools` and `client_rpc_tools`. Each inline definition uses `type: "client"`, `name`, `description`, and JSON Schema `parameters`; see `/llms.txt` for an example. Saving the draft alone is insufficient and results in HTTP `422` if the published revision lacks either capability. Inline tools supplement profile tools, must have unique names, and count toward the limit of 32 initially active tools. For either kind of browser-owned tool, register a matching handler before `connect()`:

```ts
session.registerTool("show_order_status", async (arguments_, { callId }) => {
  const orderId = String(arguments_.order_id ?? "");
  const response = await fetch(`/api/orders/${encodeURIComponent(orderId)}`, {
    headers: { "X-Tool-Call-ID": callId },
  });
  if (!response.ok) throw new Error("Order lookup failed");
  return response.json();
});
```

The SDK returns the handler result to the model. The browser SDK's `tools` option registers handlers; it does not declare tools in the backend session request. For server-owned actions, use a registered webhook tool attached to the published profile; inline webhook definitions are rejected. Verify `X-VoxEvolv-Signature`, reject stale `X-VoxEvolv-Timestamp` values, and deduplicate webhook execution by `Idempotency-Key`.

### 5. Replace vendor reconnect and shutdown handling

SDK 0.3.16 rollout: deploy the control API accepting the new diagnostic fields and the worker supporting presentation ownership before upgrading browser integrations. This release prevents repeated attachment displays for identical retries in the same turn, preserves browser transport diagnostics, and fixes recovery after a failed startup attempt. Keep the existing present_attachment handler; the SDK handles claims and receipts automatically. Other browser tools retain their existing behavior.

SDK 0.3.15 startup rollout: deploy the backend supporting 180,000 ms startup registration first, then upgrade the browser SDK bundle. The SDK registers up to 175 seconds, preserving five seconds within its 180-second limit. Backend redispatch continues within the fixed deadline rather than stopping after three replacements. Display waiting progress in the candidate UI and clear its timers when connect() settles or the user cancels. Existing unregistered clients retain their legacy fallback window.

Upgrade browser integrations to `@voxevolv/client@0.3.16` (update the lockfile), or
pin `https://cdn.jsdelivr.net/npm/@voxevolv/client@0.3.16/dist/browser/index.js`.
For vendored integrations, replace the browser bundle and update its versioned import.
Startup retries are enabled by default; omit `startupRetry` or pass `true`.
Pass `startupRetry: false` for one startup attempt using your existing `readyTimeoutMs`.
Keep the existing `RealtimeSession` instance and `autoRejoin: true`, and avoid a
second application reconnect loop. Startup retry allows up to three attempts
with 15/20/140-second readiness budgets, one- and three-second waits, fresh room
tokens for the same session ID, and a 180-second
overall deadline. Use `onStartupAttempt()` to render retry progress. Terminal
authorization, ended-session, microphone/audio, and non-retryable `session.error`
failures stop immediately. Version 0.3.2 detects agent departure/replacement and
context timeouts even while the room remains connected. Automatic rejoin has a
180-second deadline, including offline wait, configurable with
`autoRejoin: { timeoutMs: 180_000 }`; an explicit `maxAttempts` can stop earlier.

Recovery uses the same session ID and waits for hello/tool handlers, readiness,
remote audio (unless disabled), and context restoration before reporting connected.
The updated server protects worker startup with a three-minute participant grace,
restores up to 200 recent committed messages within 100,000 UTF-8 bytes, and skips
the initial greeting on resume. It does not replay raw audio, uncommitted speech,
provider-internal state, or tool executions. Timers and session maximum duration
continue; an application must implement any time compensation separately.

Send screens/timers with stable keys and `mode: "replace"`. The SDK keeps the
newest pending value per key during recovery; superseded promises reject with
`Context update superseded by a newer value`. Handle each promise. Restoration
retains up to 64 keys / 16 MiB and sends accepted snapshots with `respond: false`.
Append operations are not replayed automatically. Old SDKs remain API-compatible
but require this upgrade for browser-side agent-loss and context recovery.

Version 0.3.4 adds typed recovery triggers and content-free diagnostics. A
same-origin lifecycle proxy must also forward `POST /sessions/{id}/diagnostics`
with the browser's session token. Reports are bounded, throttled, and best
effort; a rate-limited trigger is retained for a trailing report, and each
diagnostics request stops waiting after five seconds.

With `autoRejoin: true`, the SDK handles temporary LiveKit reconnects, full
room-token rejoin, and durable event gaps. Render `reconnecting` while LiveKit
restores transport and `resuming` while VoxEvolv restores agent readiness. If
an application needs an explicit manual retry:

```ts
await session.rejoin();
```

`apiBase` is required when using `RealtimeSession.create()`. For a customer
domain, pass an absolute VoxEvolv `/v1` base under the default wildcard CORS
policy, configure that exact origin for optional pinning, or pass a same-origin
proxy base. The proxy needs only the scoped room-token, event-replay, and
session-end routes and must forward the browser's `vsa_...` authorization
unchanged. Lifecycle requests use `credentials: "same-origin"` by default,
preserving cookie-authenticated same-origin proxies without sending cookies to
a cross-origin VoxEvolv API.

The browser-safe `vsa_...` token can end only its own durable session. End and
disconnect in one operation:

```ts
await session.end();
```

Applications that require their own completion authorization may route ending
through a backend endpoint:

```ts
await session.end({ endpoint: "/api/voice-session/end" });
```

The organization API key can still end organization-owned sessions from a
trusted backend. Use `disconnect()` only to leave LiveKit without ending the
durable session.

## OpenAI Realtime replacement map

Use this section when removing direct OpenAI Realtime WebRTC, WebSocket, or `/v1/realtime/calls` integration code.

| OpenAI Realtime concept | VoxEvolv replacement |
|---|---|
| OpenAI API key or client secret | VoxEvolv organization key on the backend; short-lived `connection.token` and `vsa_...` session access token in the browser |
| `POST /v1/realtime/calls` and SDP exchange | Server SDK `sessions.create(...)`, then browser `RealtimeSession.create(...)` and `session.connect()` |
| Browser `RTCPeerConnection` | Managed by `@voxevolv/client` and LiveKit |
| Microphone track and remote audio element | Use `session.connect({ microphone: true })`, or pass an existing/mixed track with `connect({ microphone: false, audioTrack })`; agent audio is played automatically and exposed through `onRemoteAudioTrack` for recording |
| Realtime data-channel event loop | Typed helpers such as `onUserTranscript`, `onAgentTranscript`, `onError`, and `onEnded`; `onEvent` remains the normalized low-level stream |
| `session.update` model, audio, voice, VAD, and tool configuration | Configure and publish a VoxEvolv profile; per-session `voice`, `language`, `instructions`, and `greeting` are allowed only where the profile supports them |
| `session.update` application guidance | `session.updateContext(...)` for mutable application context; republish the profile for durable agent configuration |
| `input_audio_buffer.append`, commit, and clear | Removed; the SDK publishes the browser microphone and the worker owns turn detection |
| `response.create` after live application context changes | `session.updateContext({ ..., respond: true })`; ordinary spoken turns respond through the configured turn detection |
| `conversation.item.create` for application facts | `session.updateContext` with `append` or keyed `replace` semantics |
| Transcript server events | `transcript.user.interim`, `transcript.user.final`, and `transcript.agent.final` VoxEvolv events |
| Function definitions in `session.tools` | Register and publish profile tools, or translate client definitions to VoxEvolv's session `tools` array when both inline/client capabilities are published; inline webhooks are unsupported |
| Function-call output event | Return a JSON-serializable value from a registered browser handler, or return the signed webhook response |
| OpenAI reconnect logic | SDK reconnect, `session.rejoin()`, durable replay, and worker recovery |
| Close peer connection only | `session.end()` ends only the scoped durable session and disconnects media |

Delete direct calls to `api.openai.com/v1/realtime/*`, client-secret minting code, application-owned SDP exchange, OpenAI data-channel parsing, and browser exposure of OpenAI credentials after the VoxEvolv path is verified.

## Gemini Live replacement map

Use this section when removing direct Gemini Live `BidiGenerateContent` WebSocket or Google GenAI Live SDK integration code.

| Gemini Live concept | VoxEvolv replacement |
|---|---|
| Gemini API key in WebSocket URL or ephemeral token | VoxEvolv organization key on the backend; short-lived `connection.token` and `vsa_...` session access token in the browser |
| `ai.live.connect(...)` or `BidiGenerateContent` WebSocket | Server SDK `sessions.create(...)`, then browser `RealtimeSession.create(...)` and `session.connect()` |
| Initial `setup` message | Published VoxEvolv profile plus fields in the session creation request |
| `systemInstruction`, model, voice, response modalities, VAD, and tools | Configure and publish the VoxEvolv profile; select allowed per-session fields in `POST /v1/sessions/` |
| `realtimeInput.audio` and PCM resampling/chunking | Removed; the SDK publishes the browser microphone or an application-owned/mixed `audioTrack` through LiveKit |
| `clientContent` or text `realtimeInput` used as application context | `session.updateContext({ mode: "append" | "replace", ... })` |
| `serverContent.inputTranscription` | `transcript.user.interim` and `transcript.user.final` events |
| `serverContent.outputTranscription` | `transcript.agent.final` events |
| Audio inside `serverContent.modelTurn.parts` | Subscribed agent audio is played automatically by the SDK |
| `toolCall` and `sendToolResponse` | Profile or permitted inline client tool plus a browser handler return value; registered profile webhook tool plus a signed webhook response |
| `sessionResumption` handles and `GoAway` | SDK reconnect, `session.rejoin()`, durable event replay, and worker recovery; do not retain Gemini resumption handles in application code |
| Closing only the Gemini WebSocket | `session.end()` ends only the scoped durable session and disconnects media |

Delete direct `generativelanguage.googleapis.com/...BidiGenerateContent` WebSocket code, Google Live session callbacks, PCM chunking, Gemini message parsing, resumption-handle storage, and browser exposure of Gemini credentials after the VoxEvolv path is verified.

## Important non-equivalences

Do not hide these gaps when generating a migration:

- VoxEvolv does not accept OpenAI Realtime or Gemini Live wire events. Rewrite the integration around the VoxEvolv API and SDK.
- The browser SDK currently has no first-class typed conversational-turn method equivalent to arbitrary OpenAI `conversation.item.create` or Gemini `sendClientContent`. `updateContext` is for application context; do not silently treat every typed chat message as equivalent.
- Sustained Gemini-style video-frame input has no first-class browser SDK method. Gemini itself accepts JPEG/PNG frames at up to 1 fps rather than a WebRTC video track; VoxEvolv profiles with `image_input: true` currently receive keyed JPEG, PNG, or WebP snapshots through `updateContext`.
- Provider, model, pipeline, reusable tool definitions, and most realtime behavior are profile configuration. Permitted inline client definitions can supplement tools at session creation. Neither can be switched by forwarding a vendor `session.update` message.
- Browser handlers must be registered before `connect()` and match profile or accepted inline client tools. Inline tools require both published capabilities, and cannot introduce webhook URLs or replace existing tool names.
- When an application supplies `audioTrack`, it retains ownership and must stop the source track during its own media teardown. VoxEvolv publishes and cleans up only SDK-owned clones.
- VoxEvolv events are normalized product events, not raw vendor events. Preserve application behavior, not vendor event names.
- VoxEvolv reconnect and recovery are not vendor session-resumption handles. Durable events and accepted application context recover, but application code must not assume that raw vendor audio buffers or hidden model state survive a replacement worker.
- Server-side audio recording is not currently an integration contract. Do not generate recording-download code unless the live OpenAPI document exposes it.

If the existing application depends on one of these missing behaviors, preserve the old vendor path for that behavior or stop and ask for a product decision instead of inventing an API.

## Migration acceptance checklist

- The browser contains no OpenAI, Gemini, or VoxEvolv organization API key.
- The backend creates sessions with a unique and retry-safe `Idempotency-Key`.
- The selected VoxEvolv profile is published and its revision is saved with the application record.
- Browser tool handlers are registered before `connect()` and match the session's tools, including any permitted inline client definitions. Both inline/client capabilities are published when inline tools are supplied.
- The UI handles `connecting`, `connected_not_ready`, `connected`, `reconnecting`, `resuming`, `disconnected`, and `failed` states.
- Final user and agent transcripts arrive through normalized VoxEvolv events.
- Barge-in, greeting, selected voice, language, provider credentials, and required tools are tested with the published profile.
- Temporary network interruption recovers without creating a duplicate session.
- The backend can read and end only organization-owned sessions, and a browser session token can end only its own session.
- The old vendor transport, token, event parsing, and secret-handling code is removed only after the VoxEvolv path passes acceptance.

## Instruction block for a coding agent

```text
Migrate this application from its direct OpenAI Realtime or Gemini Live integration to VoxEvolv.

Before editing code, read:
1. https://voxevolv.com/llms.txt
2. https://voxevolv.com/realtime-migration.txt
3. https://voxevolv.com/v1/openapi.json

Use direct HTTP or an optional VoxEvolv server SDK from the application backend with a backend-only organization key and a stable intent ID. Return only browser-safe session credentials to the authorized browser. Prefer @voxevolv/client RealtimeSession.create(), including its self-contained bundle for plain HTML, unless the application deliberately implements the documented protocol-v1 LiveKit responsibilities itself. Move provider/model/reusable-tool configuration into a published VoxEvolv profile. Inline client definitions may be supplied in the backend session request only when the published profile enables inline_session_tools and client_rpc_tools; inline webhook tools are unsupported. Register matching browser handlers before connect(). Use typed transcript and lifecycle helpers or the normalized low-level event stream. End only the scoped session and disconnect media. Do not invent wire compatibility, typed-turn support, sustained video-frame support, or recording endpoints. Keep the old path for any required capability that the migration guide marks as unavailable, and report that gap explicitly.
```

## Vendor terminology references

These vendor pages explain the source concepts being replaced. They are not VoxEvolv contracts:

- OpenAI Realtime API reference: `https://platform.openai.com/docs/api-reference/realtime`
- OpenAI Realtime client events: `https://platform.openai.com/docs/api-reference/realtime-client-events`
- Gemini Live WebSocket guide: `https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket`
- Gemini Live session management: `https://ai.google.dev/gemini-api/docs/live-api/session-management`
- Gemini Live tool use: `https://ai.google.dev/gemini-api/docs/live-api/tools`