AI Recipes / Voice-driven UI

Register an ElevenLabs Client Tool

Let an ElevenLabs voice agent invoke functions that run inside the visitor's browser — open cards, scroll to sections, play videos — by wiring client-side tool calls to the page's JavaScript.

Why client tools

A voice agent that only talks is a chatbot with audio. A voice agent that can change the page it's running on is a UI layer. Client tools are the bridge: the agent decides to do something, the platform passes the invocation to the browser, and the page executes it.

This is different from server-side tools. Server tools hit an API, process data, and return a result. Client tools manipulate the DOM, play media, or trigger navigation. Both are useful; this recipe is about the client side.

What you'll build

  • A client tool definition in the ElevenLabs API — a function schema the agent learns to call.
  • An updated system prompt — instructions that tell the agent when and how to invoke the tool.
  • A frontend handler — JavaScript that listens for client_tool_call events and runs the matching function.
  • A fallback watcher — a scan of transcription events to catch missed calls when the primary channel drops.

Prerequisites

  • An ElevenLabs Conversational AI agent already running.
  • A webpage where the agent is embedded (via the ElevenLabs SDK, the LiveAvatar SDK, or the Conversational AI widget).
  • Basic JavaScript and DOM manipulation knowledge.

Step 1 — Define the client tool

Create a new tool via the ElevenLabs API. The critical field is "type": "client" — this tells the platform the function runs in the browser, not on a server.

curl -X POST https://api.elevenlabs.io/v1/convai/tools \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "client",
    "name": "scroll_to_section",
    "description": "Scroll the page to a named section. Call this when the user asks to navigate to pricing, docs, or a specific feature.",
    "parameters": {
      "type": "object",
      "properties": {
        "section": {
          "type": "string",
          "description": "The section ID to scroll to. Must be 'pricing', 'docs', 'features', or 'contact'."
        }
      },
      "required": ["section"]
    }
  }'

Save the returned tool ID — you'll attach it to the agent in Step 2.

Description quality matters. The agent decides whether to call this tool based on the description. Write it like you're explaining the function to a junior developer, not to a machine. "Call this when the user asks to navigate to..." is better than "Scrolls page."

Step 2 — Attach to the agent and update the prompt

Patch the agent to include the tool and add trigger instructions to the system prompt:

curl -X PATCH https://api.elevenlabs.io/v1/convai/agents/{agent_id} \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "conversation_config": {
      "agent": {
        "prompt": {
          "system_prompt": "You are a helpful assistant embedded in a website. When the user asks to see pricing, documentation, features, or contact information, use the scroll_to_section client tool. Do not just describe the section — scroll to it."
        }
      },
      "client_tools": ["tool-id-from-step-1"]
    }
  }'

Step 3 — Handle the call in the frontend

The frontend listens for client_tool_call events from the SDK and executes the matching function. The exact event shape depends on which SDK you're using.

Using the direct ElevenLabs SDK

import { Conversation } from '@ elevenlabs/elevenlabs-js';

const conversation = new Conversation({
  agentId: 'your-agent-id',
  onMessage: (message) => {
    if (message.type === 'client_tool_call') {
      handleClientTool(message);
    }
  }
});

function handleClientTool(message) {
  const { tool_name, parameters } = message;

  switch (tool_name) {
    case 'scroll_to_section':
      document.getElementById(parameters.section)?.scrollIntoView({ behavior: 'smooth' });
      break;
    case 'open_modal':
      document.getElementById(parameters.modal_id)?.showModal();
      break;
    default:
      console.warn('Unknown client tool:', tool_name);
  }
}

Using the LiveAvatar SDK

import { LiveAvatarSession } from '@heygen/liveavatar-web-sdk';

const session = new LiveAvatarSession({
  token: '...',
  videoElement: document.getElementById('avatar'),
  onClientToolCall: (data) => {
    // LiveAvatar nests the tool call differently
    const toolCall = data.client_tool_call || data;
    handleClientTool(toolCall);
  }
});
Gotcha — tool-call nesting. The LiveAvatar SDK nests the tool call under data.client_tool_call. The direct ElevenLabs SDK puts it at the top level of the message. Your handler must check both shapes or it will silently miss calls.

Step 4 — Set expects_response correctly

For LiveAvatar integrations, set "expects_response": "false" on the tool definition. Client tools through LiveAvatar cannot return values to the agent (the pipeline is one-way), so telling the agent to wait for a response will cause a timeout.

curl -X PATCH https://api.elevenlabs.io/v1/convai/tools/{tool_id} \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expects_response": false
  }'

Server-side effects (database writes, API calls) should not happen in client tools. If you need a side effect, use a server-side tool instead, or have the client tool call an internal API endpoint.

Step 5 — Add a fallback watcher

The primary event channel can drop tool calls under load or on mobile networks. A fallback scan of transcription events catches missed invocations:

// Fallback: if the agent says it will scroll but the event never fired,
// parse the transcript and trigger manually.
conversation.onMessage((message) => {
  if (message.type === 'transcript' && message.role === 'agent') {
    const text = message.text.toLowerCase();
    if (text.includes('scroll') && text.includes('pricing')) {
      // Check if scroll_to_section already fired
      if (!document.getElementById('pricing').classList.contains('scrolled-to')) {
        document.getElementById('pricing').scrollIntoView({ behavior: 'smooth' });
      }
    }
  }
});

This is defensive, not primary. The real fix is reliable event delivery, but the fallback prevents the agent from claiming it did something the user never saw.

Step 6 — Verify end-to-end

  1. Open the agent in your page. Start a conversation.
  2. Say: "Show me the pricing."
  3. The agent should respond with something like "Scrolling to pricing now..." and the page should smoothly scroll to the pricing section.
  4. Check the browser console. You should see the client_tool_call event logged with tool_name: "scroll_to_section" and the correct section parameter.
  5. Test a miss — say something unrelated. The tool should not fire.

What this gets you

  • Voice-driven navigation. "Take me to the docs" becomes a scroll action, not a description.
  • Contextual UI. The agent can open modals, play videos, or highlight elements based on what the user just asked.
  • Progressive enhancement. The page works normally without the agent. The tools are additive.

When not to use this

Server-side effects belong in server tools, not client tools. If the agent needs to write to a database, send an email, or charge a card, use a server tool with a webhook. Client tools should only manipulate the DOM or trigger client-side behavior.

What's next

  • Rich responses. Return structured data from client tools (if the SDK supports it) so the agent can adapt its next utterance based on what happened on the page.
  • Tool composition. Chain client tools — scroll to a section, then open a modal inside it, then focus an input. The agent can orchestrate multi-step UI flows.
  • RAG + tools. Combine this with a knowledge base so the agent answers from docs and then scrolls to the relevant section automatically. See the RAG recipe.

Companion materials