> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqwish.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Act on the answer

> Turn probabilities into actions with a threshold in your code, an abstain policy or the cost of each mistake.

The answer gives you probabilities. Your application still has to act: route the ticket, block the
comment, run the tool call or ask a person. There are three ways to turn the probabilities into an action.

| Way | Use it when |
| - | - |
| A threshold in your code | You want full control, or the action depends on things the request doesn't know |
| `abstain_below` | Unclear cases should go to a person |
| `action_costs` | Some mistakes cost far more than others |

You can use `abstain_below` and `action_costs` together, on any decision.

## A threshold in your code

Read the probability you need and compare it with your own threshold:

```python theme={null}
deadline = result["decisions"]["urgent"]["probabilities"]["yes"]
if deadline > 0.8:
    page_on_call_team(ticket)
```

This keeps every rule in your code, where you can change it without changing the request.

## Abstain below a threshold

Give a decision a `policy` with `abstain_below`. When `p_top` is below it, the action is `abstain`, and
you hand the case to a person. Otherwise the action is the likeliest outcome.

```json theme={null}
{"id": "team", "kind": "single", "question": "Which team should handle this ticket?",
 "outcomes": {"billing": "Invoices, payments and payment methods, refunds and the refund policy, and cancellation fees.",
              "orders": "Placing, changing, cancelling or tracking an order, delivery options and times, and shipping addresses.",
              "account": "Signing up, signing in, passwords, changing or deleting an account, and newsletter subscriptions.",
              "feedback": "A complaint, a review or a claim about the company or its service.",
              "contact": "The customer asks how to reach customer service or wants to talk to a person."},
 "policy": {"abstain_below": 0.6}}
```

The answer gets a `policy` block:

```python theme={null}
action = result["decisions"]["team"]["policy"]["action"]
queue = "manual_review" if action == "abstain" else action
```

This is the team question of [Route a support ticket](https://docs.sqwish.ai/examples/customer-support.md). Its 0.6 threshold is
the one the playground uses. Measure it on your own tickets before you rely on it.

## Weigh what mistakes cost

Give a decision `action_costs`: for each action, what it costs under each outcome. OneHot picks the
action with the lowest expected cost and returns it as `policy.action`, with every action's expected
cost in `policy.expected_costs`.

Here the question is whether a £1,200 refund is fraudulent. Approving a fraudulent one costs 1,200, and a
review costs 15 either way:

```json theme={null}
"policy": {
  "action_costs": {
    "approve": {"yes": 1200, "no": 0},
    "review": {"yes": 15, "no": 15}
  },
  "abstain_below": 0.6
}
```

* Give every action a cost for every outcome of the decision.
* A cost that is the same for every outcome, as `review` here, is how you price handing a case to a
  person.
* With `abstain_below` as well, the action is `abstain` whenever `p_top` is below it, whatever the costs.
* The costs are yours: they say what mistakes cost your application, not how confident the model is.

[Check a tool call before it runs](https://docs.sqwish.ai/examples/tool-risk.md) prices three actions this way.

## Choose a threshold

A threshold is a trade. Set it higher and fewer cases are handled wrongly, but more go to review.

1. Collect a few hundred real cases and label the right answer for each.
2. Score them once, without a policy, and keep the answers.
3. For a few thresholds, count the wrong actions and the cases sent to review.
4. Pick the threshold whose numbers your team can live with. Recheck it when you reword the decision or
   change the model.

```python theme={null}
# cases: your labelled tickets, such as {"text": "...", "team": "billing"}
# answers: the answer to each case, from client.decide with no policy
for threshold in (0.5, 0.6, 0.7, 0.8):
    wrong = review = 0
    for case, answer in zip(cases, answers):
        team = answer["decisions"]["team"]
        if team["p_top"] < threshold:
            review += 1
        elif team["top"] != case["team"]:
            wrong += 1
    print(f"{threshold}: {wrong} wrong, {review} to review, of {len(cases)}")
```

To compare models on the same cases instead, upload them as a dataset and
[compare models](https://docs.sqwish.ai/guides/evaluations.md).

Keep the meaning of each outcome stable, and change the policy rather than the outcomes when your needs
change.

## Correct a lean towards one outcome

A model can pick one outcome too often, such as sending too many tickets to `billing`. Before you train
a model of your own, try weighting the outcomes. A weight below 1 makes an outcome less likely to come
top.

1. Count the outcomes picked when they were wrong, on the same labelled cases.
2. Try a few weights for the outcome picked too often, and keep the one that gets most cases right.
3. Check that weight on labelled cases you didn't use to choose it.

```python theme={null}
from collections import Counter

# cases and answers as in the loop above
wrong = Counter(answer["decisions"]["team"]["top"] for case, answer in zip(cases, answers)
                if answer["decisions"]["team"]["top"] != case["team"])
print(wrong.most_common())  # the outcomes picked when they were wrong, most often first

def weighted_top(probabilities, weights):
    return max(probabilities, key=lambda outcome: probabilities[outcome] * weights.get(outcome, 1))

for weight in (1, 0.8, 0.6, 0.4):
    right = sum(weighted_top(answer["decisions"]["team"]["probabilities"], {"billing": weight})
                == case["team"] for case, answer in zip(cases, answers))
    print(f"billing x {weight}: {right} of {len(cases)} right")
```

To use the weights with a threshold too, multiply each probability by its weight, then divide each by
their sum, so they add up to 1 again. If no weight helps enough, [tune the wording](https://docs.sqwish.ai/guides/writing-decisions.md)
or [fine-tune a model](https://docs.sqwish.ai/guides/fine-tuning.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.