Act on the answer

The answer gives you probabilities. Your application still has to act: route the ticket, block the comment, run the tool call or ask a person. There are three ways to turn the probabilities into an action.

Way	Use it when
A threshold in your code	You want full control, or the action depends on things the request doesn’t know
`abstain_below`	Unclear cases should go to a person
`action_costs`	Some mistakes cost far more than others

You can use abstain_below and action_costs together, on any decision.

A threshold in your code

Read the probability you need and compare it with your own threshold:

deadline = result["decisions"]["urgent"]["probabilities"]["yes"]
if deadline > 0.8:
    page_on_call_team(ticket)

This keeps every rule in your code, where you can change it without changing the request.

Abstain below a threshold

Give a decision a policy with abstain_below. When p_top is below it, the action is abstain, and you hand the case to a person. Otherwise the action is the likeliest outcome.

{"id": "team", "kind": "single", "question": "Which team should handle this ticket?",
 "outcomes": {"billing": "Invoices, payments and payment methods, refunds and the refund policy, and cancellation fees.",
              "orders": "Placing, changing, cancelling or tracking an order, delivery options and times, and shipping addresses.",
              "account": "Signing up, signing in, passwords, changing or deleting an account, and newsletter subscriptions.",
              "feedback": "A complaint, a review or a claim about the company or its service.",
              "contact": "The customer asks how to reach customer service or wants to talk to a person."},
 "policy": {"abstain_below": 0.6}}

The answer gets a policy block:

action = result["decisions"]["team"]["policy"]["action"]
queue = "manual_review" if action == "abstain" else action

This is the team question of Route a support ticket. Its 0.6 threshold is the one the playground uses. Measure it on your own tickets before you rely on it.

Weigh what mistakes cost

Give a decision action_costs: for each action, what it costs under each outcome. OneHot picks the action with the lowest expected cost and returns it as policy.action, with every action’s expected cost in policy.expected_costs. Here the question is whether a £1,200 refund is fraudulent. Approving a fraudulent one costs 1,200, and a review costs 15 either way:

"policy": {
  "action_costs": {
    "approve": {"yes": 1200, "no": 0},
    "review": {"yes": 15, "no": 15}
  },
  "abstain_below": 0.6
}

Give every action a cost for every outcome of the decision.
A cost that is the same for every outcome, as review here, is how you price handing a case to a person.
With abstain_below as well, the action is abstain whenever p_top is below it, whatever the costs.
The costs are yours: they say what mistakes cost your application, not how confident the model is.

Check a tool call before it runs prices three actions this way.

Choose a threshold

A threshold is a trade. Set it higher and fewer cases are handled wrongly, but more go to review.

Collect a few hundred real cases and label the right answer for each.
Score them once, without a policy, and keep the answers.
For a few thresholds, count the wrong actions and the cases sent to review.
Pick the threshold whose numbers your team can live with. Recheck it when you reword the decision or change the model.

# cases: your labelled tickets, such as {"text": "...", "team": "billing"}
# answers: the answer to each case, from client.decide with no policy
for threshold in (0.5, 0.6, 0.7, 0.8):
    wrong = review = 0
    for case, answer in zip(cases, answers):
        team = answer["decisions"]["team"]
        if team["p_top"] < threshold:
            review += 1
        elif team["top"] != case["team"]:
            wrong += 1
    print(f"{threshold}: {wrong} wrong, {review} to review, of {len(cases)}")

To compare models on the same cases instead, upload them as a dataset and compare models. Keep the meaning of each outcome stable, and change the policy rather than the outcomes when your needs change.

Correct a lean towards one outcome

A model can pick one outcome too often, such as sending too many tickets to billing. Before you train a model of your own, try weighting the outcomes. A weight below 1 makes an outcome less likely to come top.

Count the outcomes picked when they were wrong, on the same labelled cases.
Try a few weights for the outcome picked too often, and keep the one that gets most cases right.
Check that weight on labelled cases you didn’t use to choose it.

from collections import Counter

# cases and answers as in the loop above
wrong = Counter(answer["decisions"]["team"]["top"] for case, answer in zip(cases, answers)
                if answer["decisions"]["team"]["top"] != case["team"])
print(wrong.most_common())  # the outcomes picked when they were wrong, most often first

def weighted_top(probabilities, weights):
    return max(probabilities, key=lambda outcome: probabilities[outcome] * weights.get(outcome, 1))

for weight in (1, 0.8, 0.6, 0.4):
    right = sum(weighted_top(answer["decisions"]["team"]["probabilities"], {"billing": weight})
                == case["team"] for case, answer in zip(cases, answers))
    print(f"billing x {weight}: {right} of {len(cases)} right")

To use the weights with a threshold too, multiply each probability by its weight, then divide each by their sum, so they add up to 1 again. If no weight helps enough, tune the wording or fine-tune a model.

Get started

Make decisions

Improve accuracy

Use the API

Account and billing

Privacy and security

Integrations

Act on the answer

A threshold in your code

Abstain below a threshold

Weigh what mistakes cost

Choose a threshold

Correct a lean towards one outcome

​A threshold in your code

​Abstain below a threshold

​Weigh what mistakes cost

​Choose a threshold

​Correct a lean towards one outcome

A threshold in your code

Abstain below a threshold

Weigh what mistakes cost

Choose a threshold

Correct a lean towards one outcome