Improve a decision

A base model does well when the question is clear. To do better on your own traffic, give it examples whose answers you know. This page takes the quickstart’s team question through each step with the Python client. The TypeScript client has the same calls in camelCase (SDKs), and each step’s guide shows the HTTP request.

1. Build a dataset

Each row is a decide request, its context and decisions, plus targets: the answer you expect, as a probability for every outcome. Write one row per line of a JSONL file:

tickets.jsonl

{"context": "Please refund the duplicate charge on order 4471.", "decisions": [{"id": "team", "kind": "single", "question": "Which team should handle this ticket?", "outcomes": {"billing": "Charges, invoices, and financial exports.", "technical": "Other application failures.", "sales": "New purchases."}}], "targets": {"team": {"billing": 1, "technical": 0, "sales": 0}}}

Word each decision exactly as your application asks it. Some contexts are held out to measure models, and never train them (the split).

from sqwishai import OneHot

# The decision every row asks, worded as in the file; the steps below use it again.
team = {"id": "team", "kind": "single", "question": "Which team should handle this ticket?",
        "outcomes": {"billing": "Charges, invoices, and financial exports.",
                     "technical": "Other application failures.",
                     "sales": "New purchases."}}

client = OneHot()
dataset = client.datasets.upload("tickets.jsonl", name="Support tickets",
                                 idempotency_key="support-tickets-v1")

No labels yet? Teacher models can label real queries for you to review (review teacher labels). Build a dataset covers the format, rewards and the split.

2. Measure the base model

import time

evaluation = client.evaluations.create(dataset["id"], models=["kettei-core"])["data"][0]
while evaluation["status"] in ("queued", "running"):
    time.sleep(5)
    evaluation = client.evaluations.get(evaluation["id"])
print(evaluation["metrics"])

metrics has accuracy, log loss and calibration error on the held-out rows. An evaluation is charged as decisions (compare models).

3. Try clearer wording

Prompt tuning rewrites the question and the outcome descriptions, then tests the rewrite once on rows it never saw. It needs enough labelled rows for each outcome (limits).

job = client.prompt_tuning.create(dataset["id"], decision="team",
                                  idempotency_key="support-team-wording-v1")
job = client.prompt_tuning.wait(job["id"])
if job["result"]["verdict"] == "improved":
    team = job["result"]["tuned"]  # use this wording in new requests

Use the new wording only when the verdict is improved (tune the wording).

4. Train a model of your own

A fine-tune trains a model on your rows and starts a family, a named line of models such as support-router. The new model joins only if it passes its gate, a check for regressions against the model it started from on the held-out rows. A family needs enough training rows, and enough held-out decisions for the gate to judge (what the start checks).

quote = client.fine_tuning.estimate(dataset["id"], family="support-router")
job = client.fine_tuning.create(dataset["id"], family="support-router",
                                max_price_usd=quote["price_usd"],
                                idempotency_key="support-router-v1")
job = client.fine_tuning.wait(job["id"], timeout=6 * 3600)

if job["outcome"] == "joined":
    client.families.wait("support-router", job["output_model"])
    result = client.decide(context="Where is my refund for order 4471?", decisions=[team],
                           model="support-router")

The estimate is the exact fee, and a fine-tune is charged only if it succeeds (pricing). Calls to support-router get the family’s latest model that passed. Fine-tune a family shows how to read the gate.

5. Keep improving

Add newly labelled rows to the dataset as its next version, then continue the family. The next model trains from the latest one and joins only if it passes its gate too.

client.datasets.upload_rows(dataset["id"], "more-tickets.jsonl", family="support-router",
                            idempotency_key="support-tickets-v2")
quote = client.fine_tuning.estimate_continue("support-router")
job = client.fine_tuning.continue_family("support-router",
                                         dataset_version=quote["dataset_version"],
                                         max_price_usd=quote["price_usd"],
                                         idempotency_key="support-router-v2")

To go back, roll the family back to an earlier model, which discards the newer ones: client.families.rollback("support-router", "support-router@1", ["support-router@2"]). Families, models and datasets covers switching, keeping a model warm and deleting.

Get started

Make decisions

Improve your decisions

Account and API

Improve a decision

1. Build a dataset

2. Measure the base model

3. Try clearer wording

4. Train a model of your own

5. Keep improving

​1. Build a dataset

​2. Measure the base model

​3. Try clearer wording

​4. Train a model of your own

​5. Keep improving

1. Build a dataset

2. Measure the base model

3. Try clearer wording

4. Train a model of your own

5. Keep improving