Skip to main content
Agent Tools and Actions

Test Tools and Monitor Usage

Run a tool once before an agent relies on it, then watch how it performs on real calls: attempts, success rate, response time, failures and which agents use it.

Written By voiceaiwrapper

Last updated 26 days ago

Test a tool before an agent depends on it, and keep an eye on it once it's live. The Test tab sends one real request and shows exactly what the agent would get back. The Usage tab shows how the tool performs on real calls, so you can spot a broken API key or a slow endpoint before callers complain.

Applies to: tools in a VAW Native Pod (an engine or provider connection).

Before you start

Test a tool

  1. In your VAW Native Pod, open Tools and click the tool. You can also choose Test from the row's menu.
  2. Open the Test tab, or click Test at the top of the tool's page.
  3. Under Connection, choose which API key to send with. It defaults to the tool's saved connection. A different choice applies to this run only, and the tool keeps the one on Setup.
  4. Under Call data, type the values a real call would supply, such as the caller's number. This section appears only if the tool uses call data.
  5. Under Per-agent values, type the values an agent would set when attaching the tool. This appears only if the tool has per-agent values.
  6. Under What the agent supplies, fill in each field the way the agent would during a call. If the tool needs nothing from the agent, you'll see "Nothing: you have already decided every value in this request."
  7. Click Run test.

The test uses the tool as it is on screen, including changes you haven't saved yet. Nothing you type and nothing that comes back is stored.

A custom HTTP tool's page on the Test tab after a successful run. Visible: the lead note at the top, the 'Connection' section with a connection chosen, the 'What the agent supplies' fields filled in, the 'Run test' button, and the 'Result' block showing a Succeeded badge, 'HTTP 200', a time in ms, 'What the agent receives' with a short JSON result and its size line. Blur any API keys, customer names, emails or phone numbers in the result.

Tools that create or change data

For a tool marked Creates or changes data, you must confirm before the test runs:

  • Custom HTTP tools ask "Run a test that creates or changes data?" and name the method and host the request goes to.
  • Cal.com booking tools ask "Book a real appointment?". The booking appears on the calendar and the attendee is emailed.

Click Run the test to go ahead.

A test is a real request. Whatever a write test creates or changes is real and isn't undone automatically. Pick a sandbox connection for the run if you have one, and clean up afterwards (for example, cancel a test booking in Cal.com).

You can run up to 10 tests a minute. After that, you'll see "You have run 10 tests in the last minute. Wait a moment and try again."

Read the result

Part of the resultWhat it tells you
Status badgeThe outcome, using the same words as call history (see the table below).
HTTP code and msWhat the API answered and how long it took. "Nothing was sent" means the test was stopped before the request went out.
What the agent hearsThe sentence the agent is given when the tool doesn't succeed.
What the agent receivesThe data the agent gets back, with its size in bytes and roughly how many tokens.
Saved for later toolsValues a later tool on the same call could read. A test saves nothing.
The request that went outThe exact method, URL, headers, query and body. Your API key is hidden.

If the result is large, you'll see a warning that it's a lot for an agent to work from in a spoken answer. Map only the fields you need on the Response tab. See Shape Tool Responses and Error Messages.

What a test updates

  • The Last test column on the Tools list shows Passed or Failed and when. A test of a tool you haven't saved yet doesn't update it.
  • If the API refuses the key, the connection is marked Failed, and every tool using it stops working until you replace the key. Live calls never change a connection's status. Only tests do.
  • Tests aren't counted on the Usage tab.

Monitor usage on real calls

Open the tool and go to the Usage tab, or choose Usage from the row's menu on the Tools list. Pick a period: 24 h, 7 days (the default) or 30 days.

A tool's page on the Usage tab with the '7 days' range selected and real activity. Visible: the four tiles (Attempts, Succeeded, Time to answer, Needs a look), the 'By day' bar chart with succeeded, failed and uncertain bands, the 'Used by' table with at least two agents, and the 'By outcome' and 'Most recent' lists side by side. Blur caller names and phone numbers in 'Most recent'.

The four tiles

TileWhat it shows
AttemptsHow many times agents ran the tool, and how many agents did. Attempts still in progress are shown separately.
SucceededThe share of attempts that completed. The tile is highlighted when it drops below 95%.
Time to answerThe median time the tool took, the time the slowest 5% took, and the tool's timeout.
Needs a lookAttempts that didn't complete, how many are uncertain, and the last failure's reason and time.

The rest of the tab

  • By time of day / By day / By week: a bar for each slot, split into succeeded, failed or rejected, and uncertain. Daily and weekly slots are counted in UTC.
  • Used by: each agent that ran the tool, with its Function name, When they use it, Attempts, Succeeded, Median and Last used. Agents that no longer have the tool are marked No longer attached or Agent deleted. Click Open to go to an agent.
  • By outcome: every outcome in the period with its reason, as call history shows it.
  • Most recent: the last five attempts, with the agent, caller, outcome and time. Click Open call to see the full call.
  • Open in Call Logs takes you to the Pod's call logs.

The tab doesn't score whether the agent picked the tool at the right moments. The attempts alone can't tell you that. Read a few calls to judge it. Attempts the agent filled in wrongly, and attempts stopped because the agent ran three tool calls in a row without speaking, are counted under By outcome.

What each outcome means

OutcomeMeaningWhat to check
SucceededThe tool completed and the agent got the data.Nothing.
RejectedStopped before anything was sent. For example, the agent filled a field wrongly, call data was missing or the connection had failed.The reason under By outcome.
FailedThe API was reached but answered with an error, or refused the key.The API, the endpoint or the connection.
Timed outThe API didn't answer within the tool's timeout.The API's speed, or the timeout on a custom HTTP tool's Safety tab. Cal.com tools have fixed timeouts.
Unknown (Uncertain in some places)A tool that changes data sent its request but got no usable answer, usually because it timed out. Nobody can tell whether it worked.The other system, to see whether the change happened.

For timeouts and how write tools handle them, see Tool Safety, Caller Confirmation and Timeouts.

Read the Tools list

The Tools list gives you a health check on every tool at once:

ColumnWhat it shows
ToolName and description.
OwnerAgency, or the client's name if a client created the tool or it uses a client's connection.
TypeCustom HTTP with its method, or Cal.com. Tools that write also show Creates or changes data.
Used byHow many agents have it attached, or Not attached.
Last 7 daysAttempts and success rate, plus the median time and any unknown outcomes. Shows No attempts if none ran.
Last testPassed or Failed and when, or Never tested ("Run one before attaching it").
ConnectionThe connection's name and status. "Key failed Β· this tool cannot run" or "Key removed Β· this tool cannot run" means every call to this tool is stopped until you fix the key.

Use the Type and Data effect filters to narrow the list. Each row's menu has Open, Test, Usage and Delete.

If Last 7 days shows "VAW error today", the problem is on our side, not yours. Contact support with the tool's name.

Troubleshooting

The tool uses call data, such as the caller's number, and the field was empty. On a real call this comes from the call itself. For a test, type a value into the matching field under Call data, then run it again.

The API refused the key. Open Connections on the Tools page, click Replace key, paste a working key, then test again. Until it's fixed, the tool can't run: on a call, the agent tells the caller it couldn't complete the action and carries on.

Open Usage, look at By outcome for the reason, and click Open call on a recent attempt to see exactly what the agent sent. Common causes are the agent filling in a field differently from your test, or call data that's missing on some calls (for example, no caller number on a web call).

It shows "No attempts in the last 7 days" (or whichever period you picked). If no agent has the tool attached, attach it on an agent's Tools tab first. If it's attached, try a test call with the agent, or pick a longer period. Tests from the Test tab aren't counted here.