Sort a request by the kind of work it needs

Different kinds of work suit different models and prompts. This example reads a user’s message and says which kind of work it needs, so your router can send it to the right place. Use it when a session starts, and again when a turn looks like a different job.

How it works

The context is JSON with one key, message, holding the user’s message. The keys are yours to choose. One single decision, session, asks what kind of work the request needs, with five outcomes:

Outcome	The request asks for
`code`	Writing, fixing, explaining or reviewing code, scripts or queries
`question`	Facts from general knowledge
`text_task`	Work on text the user supplies, such as a summary, a rewrite or a list pulled out of it
`writing`	Something new, such as a story, an email, a poem or a list of ideas
`chat`	Conversation, or playing a character set up by a system prompt

Its policy has abstain_below at 0.4. If no outcome reaches a probability of 0.4, the action is abstain.

Try it

Open this recipe in the playground to run it without an account. The message there asks to make a paragraph shorter and more confident. To run it from your code, set SQWISH_API_KEY as in the quickstart, then send this request:

from sqwishai import OneHot

with OneHot() as client:  # reads SQWISH_API_KEY
    result = client.decide(
        model="kettei-core",
        context={
            "message": "Can you make this paragraph shorter and more confident? \"I think that I might be a good fit for the role, as I have worked on some projects that were somewhat similar, and I would hope to learn quickly.\"",
        },
        decisions=[
            {
                "id": "session",
                "kind": "single",
                "question": "What kind of work does this request need?",
                "outcomes": {
                    "code": "Writing, fixing, explaining or reviewing code, scripts or queries.",
                    "question": "Choose question for factual information from general knowledge. If the user requests new content or ideas, choose writing.",
                    "text_task": "Working on text the user supplies: summarise, rewrite, extract, classify or answer questions about it.",
                    "writing": "Writing something new, such as a story, email, poem or post, or listing ideas.",
                    "chat": "Keeping up a conversation or playing a character, often set up by a system prompt.",
                },
                "policy": {
                    "abstain_below": 0.4,
                },
            },
        ],
    )

import { OneHot } from "sqwishai";

const client = new OneHot(); // reads SQWISH_API_KEY
const result = await client.decide({
  "model": "kettei-core",
  "context": {
    "message": "Can you make this paragraph shorter and more confident? \"I think that I might be a good fit for the role, as I have worked on some projects that were somewhat similar, and I would hope to learn quickly.\""
  },
  "decisions": [
    {
      "id": "session",
      "kind": "single",
      "question": "What kind of work does this request need?",
      "outcomes": {
        "code": "Writing, fixing, explaining or reviewing code, scripts or queries.",
        "question": "Choose question for factual information from general knowledge. If the user requests new content or ideas, choose writing.",
        "text_task": "Working on text the user supplies: summarise, rewrite, extract, classify or answer questions about it.",
        "writing": "Writing something new, such as a story, email, poem or post, or listing ideas.",
        "chat": "Keeping up a conversation or playing a character, often set up by a system prompt."
      },
      "policy": {
        "abstain_below": 0.4
      }
    }
  ]
});

curl --fail-with-body --silent --show-error https://api.sqwish.ai/v1/decide \
  -H "Authorization: Bearer $SQWISH_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "model": "kettei-core",
  "context": {
    "message": "Can you make this paragraph shorter and more confident? \"I think that I might be a good fit for the role, as I have worked on some projects that were somewhat similar, and I would hope to learn quickly.\""
  },
  "decisions": [
    {
      "id": "session",
      "kind": "single",
      "question": "What kind of work does this request need?",
      "outcomes": {
        "code": "Writing, fixing, explaining or reviewing code, scripts or queries.",
        "question": "Choose question for factual information from general knowledge. If the user requests new content or ideas, choose writing.",
        "text_task": "Working on text the user supplies: summarise, rewrite, extract, classify or answer questions about it.",
        "writing": "Writing something new, such as a story, email, poem or post, or listing ideas.",
        "chat": "Keeping up a conversation or playing a character, often set up by a system prompt."
      },
      "policy": {
        "abstain_below": 0.4
      }
    }
  ]
}
JSON

Act on the answer

ROUTES = {"code": "code-model", "question": "answer-model", "text_task": "edit-model",
          "writing": "writing-model", "chat": "chat-model"}

action = result["decisions"]["session"]["policy"]["action"]
route = ROUTES.get(action, "default-model")  # "abstain" isn't in ROUTES

const routes: Record<string, string> = { code: "code-model", question: "answer-model",
  text_task: "edit-model", writing: "writing-model", chat: "chat-model" };

const action = result.decisions.session.policy?.action ?? "abstain";
const route = routes[action] ?? "default-model"; // "abstain" isn't in routes

If the first message is too short to tell, policy.action is abstain: send it to your default route. Either way, pass the user’s original message on. The decision only picks where it goes.

Adapt it to your data

Your routes. Merge outcomes your router treats alike, or add one of your own with a description that says what it covers and where it ends.
System prompts. When the session has one, send it under its own key, such as system_prompt, next to message. It is often what marks a session as chat.
Many messages. Send one request per message, so one user’s request can’t change the answer for another’s.

What to watch

Requests that mix two kinds of work, such as code pasted in for a summary. A request that contains code isn’t always a coding job.
Long conversations. It was measured on opening messages, not on later turns.
Outcomes you add or merge. Check them on your own traffic before you route by them.

How well it works

Measured with kettei-core on 492 examples from a public dataset, it chose the right answer 88.8% of the time. Always picking the most common answer scores 55.7%. Your own data will differ, so measure it on examples of yours before you rely on it (improve a decision).

Full results

Dataset: No Robots (CC BY-NC 4.0).
Sample: 492 rows, 492 measured decisions.
Decisions measured: session.
Wording: tuned.

Measure	Value	Range
Accuracy (higher is better)	0.8882	0.8598 to 0.9167
Log loss (lower is better)	0.3387	0.2833 to 0.3953
Brier score (lower is better)	0.1706	0.1354 to 0.2063
Calibration error (lower is better)	0.0429	0.0331 to 0.0714

Always picking the most common answer has an accuracy of 0.5569, and guessing by how common each answer is has a log loss of 1.2393.These are past results on one dataset, not a promise of accuracy on yours.

Next steps

Route a support ticket: route tickets to teams the same way.
Improve a decision: measure it on your own traffic and make it more accurate.
API reference: every field in the request and the answer.

Overview

Triage and routing

Agents

Retrieval and answer quality