Pick the tool for a request

An agent with many tools can spend a planning step, or a call, on the wrong one. This example reads a request and says which of your tools fits it, or that none does. Use it before your agent plans a call, so you can give the planner only the tool that fits.

How it works

The context is JSON with the user’s request. The keys are yours to choose. One single decision, tool, has one outcome per tool, named after it. Each outcome’s description is the tool’s own description, which is how the model learns what each tool does. A none outcome covers requests no tool can do. One decision can list up to 255 outcomes, none included. Its policy has abstain_below at 0.5: if no outcome reaches a probability of 0.5, the action is abstain.

Try it

Open this recipe in the playground to run it without an account. The request there is a table for four at Dishoom King’s Cross this Friday at 7pm, under Priya. The tools are search, book, directions and send a message. To run it from your code, set SQWISH_API_KEY as in the quickstart, then send this request:

from sqwishai import OneHot

with OneHot() as client:  # reads SQWISH_API_KEY
    result = client.decide(
        model="kettei-core",
        context={
            "request": "Can you book us a table for four at Dishoom King's Cross this Friday at 7pm? Put it under Priya.",
        },
        decisions=[
            {
                "id": "tool",
                "kind": "single",
                "question": "Which tool should the agent call for this request?",
                "outcomes": {
                    "search_restaurants": "Find restaurants by area, cuisine, price or opening hours, and return their names and ratings.",
                    "book_table": "Reserve a table at a named restaurant for a party size, date and time, under a guest's name.",
                    "get_directions": "Give walking, driving or public transport directions between two places.",
                    "send_message": "Send a text or chat message to a contact.",
                    "none": "None of these tools does what the user asked.",
                },
                "policy": {
                    "abstain_below": 0.5,
                },
            },
        ],
    )

import { OneHot } from "sqwishai";

const client = new OneHot(); // reads SQWISH_API_KEY
const result = await client.decide({
  "model": "kettei-core",
  "context": {
    "request": "Can you book us a table for four at Dishoom King's Cross this Friday at 7pm? Put it under Priya."
  },
  "decisions": [
    {
      "id": "tool",
      "kind": "single",
      "question": "Which tool should the agent call for this request?",
      "outcomes": {
        "search_restaurants": "Find restaurants by area, cuisine, price or opening hours, and return their names and ratings.",
        "book_table": "Reserve a table at a named restaurant for a party size, date and time, under a guest's name.",
        "get_directions": "Give walking, driving or public transport directions between two places.",
        "send_message": "Send a text or chat message to a contact.",
        "none": "None of these tools does what the user asked."
      },
      "policy": {
        "abstain_below": 0.5
      }
    }
  ]
});

curl --fail-with-body --silent --show-error https://api.sqwish.ai/v1/decide \
  -H "Authorization: Bearer $SQWISH_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "model": "kettei-core",
  "context": {
    "request": "Can you book us a table for four at Dishoom King's Cross this Friday at 7pm? Put it under Priya."
  },
  "decisions": [
    {
      "id": "tool",
      "kind": "single",
      "question": "Which tool should the agent call for this request?",
      "outcomes": {
        "search_restaurants": "Find restaurants by area, cuisine, price or opening hours, and return their names and ratings.",
        "book_table": "Reserve a table at a named restaurant for a party size, date and time, under a guest's name.",
        "get_directions": "Give walking, driving or public transport directions between two places.",
        "send_message": "Send a text or chat message to a contact.",
        "none": "None of these tools does what the user asked."
      },
      "policy": {
        "abstain_below": 0.5
      }
    }
  ]
}
JSON

Act on the answer

action = result["decisions"]["tool"]["policy"]["action"]

if action == "abstain":
    ...  # no tool reached 0.5: give your planner every tool
elif action == "none":
    ...  # answer without a tool
else:
    ...  # give your planner only the tool named in `action`

const action = result.decisions.tool.policy?.action;

if (action === "abstain") {
  // no tool reached 0.5: give your planner every tool
} else if (action === "none") {
  // answer without a tool
} else {
  // give your planner only the tool named in `action`
}

Picking a tool doesn’t check its arguments, such as the date, the party size or the name your agent will pass. Nor does it check whether the call is allowed. Check the call before it runs.

Adapt it to your data

Your tools. Add one outcome per tool, with the tool’s name as the outcome and its description as the outcome’s description. Say what each tool does and where it ends, as you would for a person. The descriptions are all the model knows about your tools.
Keep none. Without it, the model has to name a tool even when none fits.
Earlier turns. If the request refers back to the conversation, send that under its own key, such as conversation, next to request.
Many requests. Send one request per user request.

What to watch

Requests that need two tools, such as “find a place and book it”. The decision picks one.
Tools whose descriptions overlap. Rewrite them so each one says where it ends.
Requests no tool fits. In testing, it often named a tool for these. If a wrong call is costly, check probabilities.none against a threshold of your own, tested on your own requests.

How well it works

Measured with kettei-core on 1,000 examples from a public dataset, it chose the right answer 78.9% of the time. Always picking the most common answer scores 41.4%. Your own data will differ, so measure it on examples of yours before you rely on it (improve a decision).

Full results

Dataset: BFCL live (multiple functions and irrelevance) (Apache-2.0).
Sample: 1,000 rows, 1,000 measured decisions.
Decisions measured: tool.
Wording: hand-written.

Measure	Value	Range
Accuracy (higher is better)	0.789	0.7278 to 0.847
Log loss (lower is better)	0.5282	0.4186 to 0.6354
Brier score (lower is better)	0.3142	0.2353 to 0.3981
Calibration error (lower is better)	0.0987	0.0541 to 0.1558

Always picking the most common answer has an accuracy of 0.414, and guessing by how common each answer is has a log loss of 2.9656.These are past results on one dataset, not a promise of accuracy on yours.

Next steps

Check a tool call before it runs: run, ask about or refuse the call your agent makes.
Improve a decision: measure it on your own requests and make it more accurate.
API reference: every field in the request and the answer.

Overview

Triage and routing

Agents

Retrieval and answer quality