# VoxEvolv

> VoxEvolv is a provider-independent realtime voice-agent API. A backend creates a scoped session, and an authorized browser connects to its LiveKit room. The `@voxevolv/client` SDK is recommended but not required.

Use protocol version `1`. The API base for this deployment is `https://voxevolv.com/v1`.

Browser SDK release: `0.3.16`. Install `@voxevolv/client@0.3.16` for browser integrations.
The aligned backend packages are `@voxevolv/server@0.3.16` (npm) and `voxevolv==0.3.16`
(PyPI); startup retry is implemented by the browser SDK.
For script-tag/ES module integrations, pin
`https://cdn.jsdelivr.net/npm/@voxevolv/client@0.3.16/dist/browser/index.js`.
Read `https://voxevolv.com/realtime-migration.txt` for upgrade and recovery guidance.

SDK 0.3.16 rollout: deploy the control API accepting the new diagnostic fields and the worker supporting presentation ownership before upgrading browser integrations. This release prevents repeated attachment displays for identical retries in the same turn, preserves browser transport diagnostics, and fixes recovery after a failed startup attempt. Keep the existing present_attachment handler; the SDK handles claims and receipts automatically. Other browser tools retain their existing behavior.

SDK 0.3.15 startup rollout: deploy the backend supporting 180,000 ms startup registration first, then upgrade the browser SDK bundle. The SDK registers up to 175 seconds, preserving five seconds within its 180-second limit. Backend redispatch continues within the fixed deadline rather than stopping after three replacements. Display waiting progress in the candidate UI and clear its timers when connect() settles or the user cancels. Existing unregistered clients retain their legacy fallback window.

Upgrading an existing browser integration to 0.3.16:

1. Update `@voxevolv/client` to `0.3.16` and its lockfile. For a vendored integration, copy the release's self-contained `dist/browser/index.js` to a versioned static asset such as `voxevolv-client-0.3.16.js`, update the browser import, and deploy that asset. Updating a backend package alone does not update the browser SDK.
2. Startup retries are enabled by default in the `RealtimeSession` constructor and `RealtimeSession.create()`. No new flag is required; explicit `startupRetry: true` and custom policies remain supported. Pass `startupRetry: false` only to opt out and use one startup attempt with the existing `readyTimeoutMs`. Retain `autoRejoin: true` for recovery after a successful connection. Keep the existing session ID, lifecycle `apiBase`, registered tools, audio tracks, recording integration, and assessment timer behavior.
3. Register `onStartupAttempt()` before calling `connect()`. Show retry progress when `status === "started"` and `attempt > 1`. An attempt event with `status === "failed"` is diagnostic progress, not necessarily terminal failure: the SDK may try again. A retryable startup `session.error` must not independently end the assessment or show terminal support UI. Handle final startup failure through the rejected `connect()` promise or terminal SDK state, with idempotent cleanup.
4. Allow the SDK's default 180-second overall startup deadline; remove any shorter application timeout that would cancel it prematurely. With startup retry enabled, the configured attempts' 15/20/140-second readiness budgets replace the single `connect({ readyTimeoutMs })` value. These budgets and delays fit within the overall deadline; slow connection or token renewal may leave time for fewer than three attempts. Terminal errors can still stop immediately.
5. Continue sending screen and timer context through `updateContext()` with stable keys such as `screen.current` and `assessment.timer`, `mode: "replace"`, and appropriate promise handling. Version 0.3.5 orders command publication across keys automatically. Updated VoxEvolv workers also accept delayed independent-key context commands while rejecting stale same-key updates. No application-side command queue or additional reconnect loop is required.
6. Verify delayed initial readiness, terminal microphone/authorization failures, cancellation during startup, simultaneous timer/screen updates, and recovery of the same session after a connection interruption. Confirm that the deployed browser actually loads SDK 0.3.16. The server JS and Python 0.3.16 packages only align versions; upgrading them is not required for these browser fixes.

An LLM or coding agent may build an integration after the user supplies a VoxEvolv organization API key and a published profile name. Access is self-serve, not anonymous: a user must create an account and organization, connect the selected model providers, publish a profile, and issue an API key before a real voice session can start.

Setup sequence:

1. Create an account at `https://voxevolv.com/signup`.
2. Connect the profile's STT, LLM, TTS, or realtime provider credentials at `https://voxevolv.com/app/providers`.
3. Define any reusable browser or webhook functions at `https://voxevolv.com/app/tools`, attach them to an agent profile, and choose `Always active` or an on-demand activation group.
4. Create and publish an agent profile at `https://voxevolv.com/app/profiles`. Tool attachment changes do not affect new sessions until the profile is published again.
5. Create a test or live API key at `https://voxevolv.com/app/keys` with the required `sessions:create`, `sessions:read`, and `sessions:end` scopes.
6. Store the `sk_test_...` or `sk_live_...` organization key only in a trusted backend or secret manager.

Security rules for agents generating integrations:

- Never place an organization API key in browser JavaScript, mobile bundles, URLs, logs, prompts, metadata, source control, or an LLM-visible client response.
- A trusted backend must call the session-creation API. Return only the session `id`, `connection`, and `access_token` to the authorized browser.
- Treat the returned `connection.token` as a short-lived, single-room LiveKit credential and the `vsa_...` access token as a short-lived, single-session browser credential.
- Use HTTPS and WSS outside local development. Do not disable TLS verification.
- Do not invent endpoints or fields. Read the live OpenAPI document before generating production code.
- Reusable client tools and all webhook tools must be registered, attached to a profile, and published. A session request may additionally supply inline `type: "client"` tools when the published profile enables both `inline_session_tools` and `client_rpc_tools`.
- Each session captures the profile's tool manifest plus any accepted inline client tools. On-demand discovery can activate groups inside that captured manifest but cannot introduce new tools after creation. Inline webhook tools are not supported.
- Treat webhook signing secrets like API keys. They are shown once, encrypted at rest, and must never be exposed to the browser.

Minimal backend session creation:

```bash
export VOXEVOLV_API_BASE="https://voxevolv.com/v1"
export VOXEVOLV_API_KEY="YOUR_BACKEND_ONLY_API_KEY"

curl --fail-with-body -X POST "$VOXEVOLV_API_BASE/sessions/" \
  -H "Authorization: Bearer $VOXEVOLV_API_KEY" \
  -H "Idempotency-Key: voice-session-UNIQUE_INTENT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "protocol_version": 1,
    "client_reference_id": "your-record-123",
    "profile": "PUBLISHED_PROFILE_NAME",
    "instructions": "Be concise and helpful.",
    "greeting": "Hello. How can I help?",
    "participant": {
      "identity": "user-123",
      "display_name": "User 123"
    },
    "media": {
      "input": ["audio"],
      "output": ["audio", "text"]
    },
    "metadata": {
      "environment": "development"
    }
  }'
```

Generate one `Idempotency-Key` per intended session. Reuse the same value only when retrying that same creation request. HTTP `201` means created; HTTP `200` may mean a safe idempotent replay.

The successful response contains:

- `id`: durable VoxEvolv session UUID; save it with `client_reference_id` in the backend.
- `profile_revision`: immutable profile revision used by this session.
- `connection.url`, `connection.token`, and `connection.room_name`: short-lived browser room credentials.
- `access_token`: browser-safe `vsa_...` credential for rejoin, durable event replay, and ending only this session.

Custom tools at session creation:

- In Agent profiles, open the profile's Capabilities tab, enable both `Client RPC tools` and `Inline session tools`, then click `Publish`. Saving a draft alone does not enable these capabilities for new sessions.
- Add a `tools` array to the backend `POST /v1/sessions/` request. Each item defines a client tool's name, description, and JSON Schema parameters. For example:

```json
{
  "tools": [
    {
      "type": "client",
      "name": "present_attachment",
      "description": "Show an attachment to the participant.",
      "parameters": {
        "type": "object",
        "properties": {
          "attachment_key": { "type": "string" }
        },
        "required": ["attachment_key"]
      }
    }
  ]
}
```

- This is an additional request field; also supply the required `client_reference_id`, `profile`, and `participant` fields shown above.
- Register a matching browser handler before `connect()`, for example `session.registerTool("present_attachment", async (args) => { await showAttachment(args.attachment_key); return { shown: true }; })`, where `showAttachment` is your application's UI function. The browser SDK's `tools` option registers handlers; it does not send tool definitions to the session creation API.
- Inline client tools supplement the profile's tools and are always active. Names must be unique across both sets. Requests accept at most 32 inline definitions, and the total initially active tool count, including profile tools, must not exceed 32.
- HTTP `422` may indicate a missing published capability, an inline webhook tool, an invalid schema, a duplicate tool name, or too many active tools. Inspect the response details. For webhook execution, register and attach the webhook tool to the profile, then publish.

SDK requirement:

- VoxEvolv SDKs are optional. A backend may call the documented HTTP API directly instead of using `@voxevolv/server` or the Python package.
- A browser may use `@voxevolv/client`, load its self-contained browser bundle without npm, or directly implement the versioned LiveKit and realtime protocol contract.
- The client SDK is the recommended path because it owns audio publication/playback, `client.hello`, event validation and ordering, duplicate suppression, gap replay, rejoin, browser tools, context commands, remote-audio callbacks, and scoped session ending.
- A raw browser integration must implement those responsibilities and remain on protocol version `1`. SDK-free does not mean backend-free: the organization API key and session creation must remain on a trusted server.

Plain HTML without npm:

Copy the versioned self-contained `dist/browser/index.js` bundle into the application's static assets, then import it from a module script. Do not point a production page at an unversioned development file.

```html
<button id="start-voice" type="button">Start voice</button>
<script type="module">
  import { RealtimeSession } from "/vendor/voxevolv/index.js";

  document.querySelector("#start-voice").addEventListener("click", async () => {
    const session = await RealtimeSession.create({
      endpoint: "/api/my-voice-session",
      // Prefer the public VoxEvolv lifecycle API; a same-origin proxy is optional.
      apiBase: "https://voxevolv.com/v1",
      autoRejoin: true,
    });
    session.onAgentTranscript(({ text, final }) => {
      if (final) console.log("agent", text);
    });
    session.onStartupAttempt((event) => console.log("voice startup", event));
    await session.connect({ microphone: true });
  });
</script>
```

Direct browser-to-VoxEvolv lifecycle requests are enabled by the default `VOICE_SESSION_BROWSER_ORIGINS=*` policy. Configure exact HTTPS page origins for optional origin pinning, or leave it empty only when using a same-origin proxy. The proxy forwards the browser's `vsa_...` credential only; it must never inject an organization API key.

Upgrade warning: earlier releases treated an unset `VOICE_SESSION_BROWSER_ORIGINS` as disabled. The new default is `*`; set an explicit empty value before upgrading to preserve closed behavior.

Minimal browser connection:

```ts
import { RealtimeSession } from "@voxevolv/client";

// This endpoint belongs to your application backend. It creates the VoxEvolv
// session without exposing the organization API key to the browser.
const session = await RealtimeSession.create({
  endpoint: "/api/my-voice-session",
  apiBase: "https://voxevolv.com/v1",
  autoRejoin: true,
  tools: {
    show_order_status: async (arguments_, { callId }) => {
      const orderId = String(arguments_.order_id ?? "");
      const response = await fetch(`/api/orders/${encodeURIComponent(orderId)}`, {
        headers: { "X-Tool-Call-ID": callId },
      });
      if (!response.ok) throw new Error("Order lookup failed");
      return response.json();
    },
  },
});

session.onStateChange((state) => console.log("voice state", state));
session.onStartupAttempt((event) => console.log("voice startup", event));
session.onRecovery((event) => console.log("voice recovery", event));
session.onConnectionQuality((quality) => console.log("call quality", quality));
session.onUserTranscript(({ text, final }) => console.log("user", text, final));
session.onAgentTranscript(({ text, final }) => console.log("agent", text, final));
session.onError((event) => console.error(event.payload));

// Call from a user action so the browser may request microphone permission.
await session.connect({ microphone: true });
```

Runtime operations:

- Lifecycle routing: `apiBase` is required. Browser lifecycle CORS defaults to `VOICE_SESSION_BROWSER_ORIGINS=*`; use an absolute VoxEvolv `/v1` base, configure exact origins for pinning, or use a same-origin application proxy for the scoped room-token, event-replay, diagnostics, and session-ending routes. Lifecycle fetches default to `credentials: "same-origin"`; customize `lifecycleCredentials` only for a proxy with a different credential policy.
- Startup recovery: enabled by default since SDK 0.3.6; SDK 0.3.15 uses up to three attempts, 15/20/140-second readiness budgets, one- and three-second waits, fresh room tokens for the same session ID, and a 180-second overall deadline. Pass `startupRetry: false` for one attempt. Use `onStartupAttempt()` for retry UI. Terminal authorization, ended-session, microphone/audio, and non-retryable `session.error` failures stop immediately; configured credential refresh gets one opportunity after a `401`.
- Recovery: set `autoRejoin: true`. `reconnecting` is LiveKit transport recovery; `resuming` is a full VoxEvolv session restoration. The SDK preserves managed-microphone mute state and stops on terminal disconnect reasons, explicit end/disconnect, and `409 Session has ended`.
- Manual rejoin: call `await session.rejoin()`; the SDK uses the configured base and `POST /sessions/{session_id}/room-tokens` with the session access token.
- Recovery in SDK 0.3.2: `autoRejoin: true` also detects agent departure, replacement-worker readiness, and context-command timeouts even if the room still looks connected. Automatic rejoin retries for up to 180 seconds (including offline wait); configure `autoRejoin: { timeoutMs: 180_000 }` and optionally `maxAttempts` for an earlier stop. Keep the same `RealtimeSession`; recovery renews its room token and never creates a new session ID.
- Recovery diagnostics in SDK 0.3.4: every `onRecovery` event includes a typed `trigger`. Context timeouts also expose safe `messageId`, `contextKey`, and `timeoutMs` fields. The SDK and worker retain only their latest 128 content-free entries in memory and emit throttled reports only for operations over two seconds, drops, rejections, failures, timeouts, or reconnects. Reports contain correlation IDs, sequence numbers, byte counts, connection state, retry counts, and stage timings; they never contain context text, screenshots, tokens, or raw URLs. Browser reports use the session-scoped `POST /v1/sessions/{session_id}/diagnostics` route. Both buffers retain a rate-limited trigger for one trailing report, each browser request stops waiting after five seconds, and reporting failure never affects the call.
- Console recovery visibility: the Sessions detail screen counts unique durable `client.connection.recovered` events. A successfully restored session shows an amber recovery warning, `Recovered N×` in the Connection metric, the total connection count including the initial connection, and the longest interruption. An unresolved loss remains an error. Diagnostic log reports are not used for these counts because buffered reports may repeat evidence.
- Recovery completion: show `resuming` until hello/tool-handler registration, agent readiness, audio (unless explicitly disabled), and latest replacement context succeed. The server restores recent committed conversation (up to 200 messages / 100,000 UTF-8 bytes) and skips the opening greeting. Raw audio, uncommitted speech, hidden provider state, and tool executions are not replayed.
- Screen/timer context: use stable keys with `mode: "replace"`. The SDK coalesces unsent values per key and pauses transmission during recovery. Handle rejected promises: superseded pending values reject with `Context update superseded by a newer value`. Up to 64 keys / 16 MiB are retained; previously accepted snapshots restore with `respond: false`. Append updates are not automatically replayed during recovery.
- Application upgrade: update the browser SDK dependency/lockfile, vendored bundle, or pinned CDN import to 0.3.16; startup retry is automatic unless `startupRetry: false` is supplied. Retain `autoRejoin: true`, the same session and registered tools, recovery status UI, and context promise handling. Do not add a competing reconnect loop or recreate the assessment. Existing clients remain compatible but do not gain these browser recovery fixes without upgrading. Application clocks and configured session maximum duration continue during the outage.
- Recovery observers: use `onStartupAttempt`, `onRecovery`, `hasConnectedBefore`, `waitForState`, and `onConnectionQuality` instead of reaching into `session.room` or implementing retry loops.
- Replay durable events: the SDK handles gaps automatically; manually call `await session.replay()` only for explicit reconciliation.
- Read session state from the backend: `GET /v1/sessions/{session_id}` with the organization API key.
- Read the complete bounded session summary: `GET /v1/sessions/{session_id}/data`; it includes up to 200 final conversation messages and 100,000 UTF-8 text bytes. Follow `links.transcript` with pagination for the complete transcript when `conversation_truncated` is true.
- An organization API key may read `/events`; a browser session token continues to receive audience-filtered replay.
- End from the browser: call `await session.end()`; the `vsa_...` credential can end only its own session, and the SDK then disconnects media. The organization backend may still end an organization-owned session with `DELETE /v1/sessions/{session_id}`.
- Application-owned audio: pass `connect({ microphone: false, audioTrack })`; the SDK publishes a clone, preserves the source track, and republishes a fresh clone during `rejoin()`. Use `onRemoteAudioTrack` and `onRemoteAudioTrackUnsubscribed` to feed agent audio into an application recording mixer.
- Update text or image context with the SDK's `updateContext` API only when the published profile advertises the required capability.
- Browser tools: call `session.registerTool(modelName, handler)` before `connect()`. The handler receives validated model arguments and `{ callId }`; its JSON-serializable return value goes back to the model.
- Webhook tools: VoxEvolv sends a JSON `POST` with `session_id`, stable registry `tool`, `call_id`, and `arguments`. Deduplicate with `Idempotency-Key`.
- Lifecycle webhooks: administrators define reusable endpoints, then select them in a profile's Integration tab and publish. There is no organization-wide fan-out. Endpoints may include encrypted custom headers such as Authorization or X-API-Key; their values are write-only, while VoxEvolv's signature, event ID, timestamp, content type, and user agent remain managed. Delivery runs from the durable control-jobs outbox and includes the same bounded conversation, audio duration, usage, interruption, tooling, health, and metadata object returned by `/sessions/{id}/data`, with a link to the complete paginated transcript.
- Verify webhook-tool requests with lowercase hex `HMAC-SHA256(secret, timestamp + "." + raw_request_body)`. Verify lifecycle webhooks with `HMAC-SHA256(secret, timestamp + "." + event_id + "." + raw_request_body)`. Compare the value after `v1=` in `X-VoxEvolv-Signature` in constant time, reject stale timestamps, and deduplicate lifecycle events by `X-VoxEvolv-Event-Id`.
- On-demand tools: the worker exposes `search_tools` when the published manifest contains inactive groups. A successful search persists the activation before updating the model's active tool list, so worker recovery restores it.
- End call: attach the platform tool `voxevolv.end_call` to profiles allowed to let the model end the conversation. The worker records `session.ending`, drains the farewell, and closes the room.

Failure handling:

- `400`: malformed request or invalid/missing idempotency key; fix the request rather than retrying unchanged.
- `401`: missing, invalid, revoked, or expired credential.
- `403`: credential lacks scope or the session participant does not match.
- `404`: the key cannot access the requested profile/session, or the published profile name is wrong.
- `409`: idempotency conflict, ended session, or unsupported capability.
- `422`: schema or tool validation failure, including disabled inline-tool capabilities; inspect the structured response details and confirm the updated profile was published.
- `429`: quota or rate limit; honor retry guidance.
- `5xx`: temporary failure; retry with bounded exponential backoff and preserve the original `Idempotency-Key`.
- A provider-credential startup error means the organization must connect or replace the named provider credential in Provider connections; never ask for that secret in the browser.

## API references

- [Interactive API reference](https://voxevolv.com/v1/docs): Human-readable live Swagger documentation for the deployed API.
- [OpenAPI 3.1 document](https://voxevolv.com/v1/openapi.json): Machine-readable source of truth for current paths, schemas, authentication, and responses.
- [OpenAI Realtime and Gemini Live migration guide](https://voxevolv.com/realtime-migration.txt): Machine-readable replacement map, implementation pattern, known non-equivalences, and coding-agent instructions.
- [Health endpoint](https://voxevolv.com/v1/health/live): Unauthenticated liveness check; a healthy control plane returns `{"status":"ok"}`.

## Console setup

- [Create an account](https://voxevolv.com/signup): Self-serve account and organization creation.
- [Provider connections](https://voxevolv.com/app/providers): Encrypt and test organization provider credentials.
- [Agent profiles](https://voxevolv.com/app/profiles): Configure, validate, test, and publish profiles.
- [Tools](https://voxevolv.com/app/tools): Define reusable client/webhook functions, attach platform or organization tools, and configure on-demand groups.
- [API keys](https://voxevolv.com/app/keys): Issue or revoke scoped organization API keys.
- [Webhooks](https://voxevolv.com/app/webhooks): Define signed session lifecycle endpoints; select the endpoints per profile before publishing.
- [Sessions](https://voxevolv.com/app/sessions): Inspect session state, transcripts, timings, reconnect counts and history, and safe failure diagnostics.

## Optional

- [VoxEvolv homepage](https://voxevolv.com/): Product overview for human readers.