Route a support ticket

A new ticket needs the right team, and some need someone straight away. This example takes a ticket and says which team should take it, whether it is urgent and how badly the customer is affected. Use it when a ticket arrives, before anyone looks at it.

How it works

The context is the ticket’s text. The request asks three decisions about it, one of each kind:

Decision	Asks	Outcomes
`team`	Which team should handle this ticket?	`billing`, `orders`, `account`, `feedback` and `contact`, each with a description of what it covers
`urgent`	Does the ticket state a deadline within one day?	`yes` or `no`
`impact`	How badly is the customer affected?	`0`, a minor annoyance, to `2`, blocked or losing money

team has a policy with abstain_below at 0.6. If no team reaches a probability of 0.6, its action is abstain, and a person should pick the team.

Try it

Open this recipe in the playground to run it without an account. The ticket there is a card charged twice for order 4471, with the money needed before rent goes out tomorrow. To run it from your code, set SQWISH_API_KEY as in the quickstart, then send this request:

from sqwishai import OneHot

with OneHot() as client:  # reads SQWISH_API_KEY
    result = client.decide(
        model="kettei-core",
        context="Hi, you charged my card twice for order 4471, 84.50 GBP each time. Please refund the second payment. I need the money back before my rent goes out tomorrow.",
        decisions=[
            {
                "id": "team",
                "kind": "single",
                "question": "Which team should handle this ticket?",
                "outcomes": {
                    "billing": "Invoices, payments and payment methods, refunds and the refund policy, and cancellation fees.",
                    "orders": "Placing, changing, cancelling or tracking an order, delivery options and times, and shipping addresses.",
                    "account": "Signing up, signing in, passwords, changing or deleting an account, and newsletter subscriptions.",
                    "feedback": "A complaint, a review or a claim about the company or its service.",
                    "contact": "The customer asks how to reach customer service or wants to talk to a person.",
                },
                "policy": {
                    "abstain_below": 0.6,
                },
            },
            {
                "id": "urgent",
                "kind": "binary",
                "question": "Does the ticket state a deadline within one day?",
                "outcomes": {
                    "yes": "An explicit deadline is today or within the next 24 hours.",
                    "no": "The deadline is later, missing or unclear.",
                },
            },
            {
                "id": "impact",
                "kind": "ordinal",
                "question": "How badly is the customer affected?",
                "outcomes": [
                    "0",
                    "1",
                    "2",
                ],
                "rubric": [
                    "A minor annoyance. Nothing is blocked.",
                    "Something is harder or delayed, but they can manage.",
                    "They are blocked or losing money, and can't fix it themselves.",
                ],
            },
        ],
    )

import { OneHot } from "sqwishai";

const client = new OneHot(); // reads SQWISH_API_KEY
const result = await client.decide({
  "model": "kettei-core",
  "context": "Hi, you charged my card twice for order 4471, 84.50 GBP each time. Please refund the second payment. I need the money back before my rent goes out tomorrow.",
  "decisions": [
    {
      "id": "team",
      "kind": "single",
      "question": "Which team should handle this ticket?",
      "outcomes": {
        "billing": "Invoices, payments and payment methods, refunds and the refund policy, and cancellation fees.",
        "orders": "Placing, changing, cancelling or tracking an order, delivery options and times, and shipping addresses.",
        "account": "Signing up, signing in, passwords, changing or deleting an account, and newsletter subscriptions.",
        "feedback": "A complaint, a review or a claim about the company or its service.",
        "contact": "The customer asks how to reach customer service or wants to talk to a person."
      },
      "policy": {
        "abstain_below": 0.6
      }
    },
    {
      "id": "urgent",
      "kind": "binary",
      "question": "Does the ticket state a deadline within one day?",
      "outcomes": {
        "yes": "An explicit deadline is today or within the next 24 hours.",
        "no": "The deadline is later, missing or unclear."
      }
    },
    {
      "id": "impact",
      "kind": "ordinal",
      "question": "How badly is the customer affected?",
      "outcomes": [
        "0",
        "1",
        "2"
      ],
      "rubric": [
        "A minor annoyance. Nothing is blocked.",
        "Something is harder or delayed, but they can manage.",
        "They are blocked or losing money, and can't fix it themselves."
      ]
    }
  ]
});

curl --fail-with-body --silent --show-error https://api.sqwish.ai/v1/decide \
  -H "Authorization: Bearer $SQWISH_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "model": "kettei-core",
  "context": "Hi, you charged my card twice for order 4471, 84.50 GBP each time. Please refund the second payment. I need the money back before my rent goes out tomorrow.",
  "decisions": [
    {
      "id": "team",
      "kind": "single",
      "question": "Which team should handle this ticket?",
      "outcomes": {
        "billing": "Invoices, payments and payment methods, refunds and the refund policy, and cancellation fees.",
        "orders": "Placing, changing, cancelling or tracking an order, delivery options and times, and shipping addresses.",
        "account": "Signing up, signing in, passwords, changing or deleting an account, and newsletter subscriptions.",
        "feedback": "A complaint, a review or a claim about the company or its service.",
        "contact": "The customer asks how to reach customer service or wants to talk to a person."
      },
      "policy": {
        "abstain_below": 0.6
      }
    },
    {
      "id": "urgent",
      "kind": "binary",
      "question": "Does the ticket state a deadline within one day?",
      "outcomes": {
        "yes": "An explicit deadline is today or within the next 24 hours.",
        "no": "The deadline is later, missing or unclear."
      }
    },
    {
      "id": "impact",
      "kind": "ordinal",
      "question": "How badly is the customer affected?",
      "outcomes": [
        "0",
        "1",
        "2"
      ],
      "rubric": [
        "A minor annoyance. Nothing is blocked.",
        "Something is harder or delayed, but they can manage.",
        "They are blocked or losing money, and can't fix it themselves."
      ]
    }
  ]
}
JSON

Act on the answer

team = result["decisions"]["team"]
urgent = result["decisions"]["urgent"]
impact = result["decisions"]["impact"]

# "abstain" means no team reached 0.6, so a person picks one.
action = team["policy"]["action"]
queue = "manual_review" if action == "abstain" else action

page_someone = urgent["probabilities"]["yes"] >= 0.8  # a threshold you choose
priority = impact["expected_level"]                   # 0 to 2: sort the queue by it

const { team, urgent, impact } = result.decisions;

// "abstain" means no team reached 0.6, so a person picks one.
const action = team.policy?.action;
const queue = action === "abstain" ? "manual_review" : action;

const pageSomeone = urgent.probabilities.yes >= 0.8; // a threshold you choose
const priority = impact.expected_level;              // 0 to 2: sort the queue by it

urgent gives probabilities.yes, the probability that the ticket states a deadline within 24 hours. impact gives expected_level, the average level from 0 to 2, which sorts a queue more finely than top does.

Adapt it to your data

Your teams. Rename the teams and rewrite their descriptions to match your own desk. Descriptions that say where one team ends and the next begins do most of the work (describe the answers).
Your deadline and levels. Change “within one day” to the deadline you care about, and the impact rubric to the levels your desk uses.
Many tickets. Send one request per ticket, with its three decisions, and you pay for the ticket’s text once.
One decision alone. Decisions in the same request can affect each other’s answers. If one must not be influenced by the others, send it in a request of its own.

What to watch

Tickets sent to the wrong team. Look at a sample of them, and sharpen the descriptions of the teams they confuse.
Tickets that mix several problems, such as a refund and a late delivery. team can pick only one.
Only team was measured. Check the urgent and impact answers on your own tickets before you act on them.

How well it works

Measured with kettei-core on 1,000 examples from a public dataset, it chose the right answer 98.6% of the time. Always picking the most common answer scores 31.0%. Your own data will differ, so measure it on examples of yours before you rely on it (improve a decision).

Full results

Dataset: Bitext customer support (CDLA-Sharing-1.0).
Sample: 1,000 rows, 1,000 measured decisions.
Decisions measured: team.
Wording: hand-written.

Measure	Value	Range
Accuracy (higher is better)	0.986	0.978 to 0.993
Log loss (lower is better)	0.1111	0.0988 to 0.1243
Brier score (lower is better)	0.0292	0.0224 to 0.0371
Calibration error (lower is better)	0.0757	0.0728 to 0.083

Always picking the most common answer has an accuracy of 0.31, and guessing by how common each answer is has a log loss of 1.4547.These are past results on one dataset, not a promise of accuracy on yours.

The test tickets were short, clean and written from templates, so expect lower accuracy on real tickets.

Next steps

Decide when to hand a conversation on: pass a chat to a person only when the bot can’t finish it.
Improve a decision: measure it on your own tickets and make it more accurate.
API reference: every field in the request and the answer.

Overview

Triage and routing

Agents

Retrieval and answer quality