Label with teacher models

Teacher models can label real queries for you. You review their labels, then turn the rows you accept into a dataset. First check whether labelling is available, and which teachers are offered:

curl --fail-with-body https://api.sqwish.ai/v1/labeling/teachers \
  -H "Authorization: Bearer $SQWISH_API_KEY"

Plan the job with available, defaults.teacher_models and limits. Check available: a catalogue of teacher names doesn’t mean the service is enabled.

1. Upload unlabelled queries

Save UTF-8 JSONL, one context per line. You can put decisions on each row, or give shared decisions when you start the job.

from sqwishai import OneHot

client = OneHot()
source = client.labeling.sources.upload("queries.jsonl", name="Refund-review",
                                        idempotency_key="refund-source-v1")

import { readFile } from "node:fs/promises";
import { OneHot } from "sqwishai";

const client = new OneHot();
const source = await client.labeling.sources.upload(await readFile("queries.jsonl", "utf8"),
  "Refund-review", { idempotencyKey: "refund-source-v1" });

cat > queries.jsonl <<'JSONL'
{"context":"Please refund the duplicate charge on order 4471."}
{"context":"Where is the parcel for order 2280?"}
JSONL

curl --fail-with-body 'https://api.sqwish.ai/v1/labeling/sources?name=Refund-review' \
  -H "Authorization: Bearer $SQWISH_API_KEY" \
  -H 'Content-Type: application/x-ndjson' \
  -H 'Idempotency-Key: refund-source-v1' --data-binary @queries.jsonl

Keep the returned source id. A source holds up to 2,000 rows and 16 MiB, and each query asks up to 64 decisions. Its input receipt records the uploaded bytes and hash. Every row must define a decision ID the same way, whether it has its own decisions or uses the shared ones. A job whose rows differ is refused with conflicting_decision before it is created, and nothing is charged.

Teacher labelling sends the selected queries to the teacher model’s provider. Choose data that suits that.

2. Start a job

Set SQWISH_SOURCE_ID to the source’s ID, and SQWISH_TEACHER_ID to a teacher ID from /v1/labeling/teachers:

import os

refund = {"id": "refund", "kind": "binary", "question": "Is a refund requested?"}

job = client.labeling.jobs.create(os.environ["SQWISH_SOURCE_ID"],
                                  teacher_models=[os.environ["SQWISH_TEACHER_ID"]],
                                  decisions=[refund], idempotency_key="refund-labels-v1")
job = client.labeling.jobs.wait(job["id"])  # returns once the labels are ready for review

import type { Decision } from "sqwishai";

const refund: Decision = { id: "refund", kind: "binary", question: "Is a refund requested?" };

const started = await client.labeling.jobs.create(
  { source_id: process.env.SQWISH_SOURCE_ID!, teacher_models: [process.env.SQWISH_TEACHER_ID!],
    decisions: [refund] },
  { idempotencyKey: "refund-labels-v1" });
const job = await client.labeling.jobs.wait(started.id); // returns once the labels are ready for review

curl --fail-with-body https://api.sqwish.ai/v1/labeling/jobs \
  -H "Authorization: Bearer $SQWISH_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: refund-labels-v1' \
  -d "{\"source_id\":\"$SQWISH_SOURCE_ID\",\"teacher_models\":[\"$SQWISH_TEACHER_ID\"],\"decisions\":[{
    \"id\":\"refund\",\"kind\":\"binary\",\"question\":\"Is a refund requested?\"}]}"

teacher_models names one or two different teachers. Left out, it is defaults.teacher_models. A job costs each of its teachers’ price for every row of its source. pricing.teacher_row_usd in /v1/onehot gives the price for a row.

The job’s price_usd is held while it runs. It is charged once the labels are ready for review.
A job that fails or is cancelled before then costs nothing.
Send the price you were shown as max_price_usd (USD amounts). A job that would cost more gets 409 price_changed.

A teacher that gives no answer is asked again. After 15 minutes with no answer, the job tries again from the rows already labelled. On the third such try, it fails. A job with no valid row fails too. Poll /v1/labeling/jobs/{job_id}:

Status	Meaning
`queued`, `running`	Labelling is under way
`cancelling`	You asked to cancel it, and it is stopping
`review_ready`	Its labels are ready for review. That doesn’t mean they are correct
`failed`, `cancelled`	It ended before review
`finalized`	You made a dataset from the rows you accepted
`discarded`	You set its labels aside
`deleted`	You deleted its source

3. Review and finalise

Once the job is review_ready, read /v1/labeling/jobs/{job_id}/results page by page, up to 200 rows a page. Before that, and for a job that failed or was cancelled, it answers 409 review_required. Each result has its source index, context, decisions, teacher labels, valid and agreement. valid means the teachers returned labels in the expected shape. agreement means they chose the same labels. Neither replaces your review. For valid rows, targets averages the teachers’ one-hot votes. Inspect disagreements and systematic mistakes against your labelling rules. Invalid rows can’t be finalised. Set SQWISH_LABEL_JOB_ID to your job’s ID. After reviewing and accepting rows 0 and 1:

dataset = client.labeling.jobs.finalize(os.environ["SQWISH_LABEL_JOB_ID"], [0, 1],
                                        idempotency_key="refund-reviewed-v1")

const dataset = await client.labeling.jobs.finalize(process.env.SQWISH_LABEL_JOB_ID!, [0, 1],
  { idempotencyKey: "refund-reviewed-v1" });

curl --fail-with-body "https://api.sqwish.ai/v1/labeling/jobs/$SQWISH_LABEL_JOB_ID/finalize" \
  -H "Authorization: Bearer $SQWISH_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: refund-reviewed-v1' -d '{"row_indices":[0,1]}'

Use the original row indices, not positions on a page. Finalising returns the dataset and changes the job to finalized. A different selection later conflicts. When the reviewed selection must change, start another job.

Choose the evaluation rows

Finalising also takes evaluation_fraction, from 0.1 to 1. It is 0.2 by default. The console offers the same choice before you finalise.

A mixed dataset needs at least two selected rows, with distinct contexts for training and evaluation.
At 1 the dataset is evaluation only, and a single reviewed context is enough.
To choose each row’s side yourself, send training_indices and evaluation_indices in place of row_indices. They work like a dataset’s training and evaluation parts (choose the evaluation set).

The dataset keeps its teacher and source lineage. Inspect it like any other dataset before you tune or train.

Discard or delete

Request	Does
`POST /v1/labeling/jobs/{job_id}/discard`	Sets the labels aside without making a dataset. The job becomes `discarded`, and its answers stay. You can still finalise it later
`DELETE /v1/labeling/sources/{source_id}`	Deletes your queries and the teachers’ answers kept for its jobs, for good. It answers `{"id": "…", "deleted": true}`. The source’s reads then return `410 source_deleted`. While one of its jobs is running, it returns `409 in_use`

A dataset you finalised from a source stays when you delete the source. Delete that dataset separately.

Get started

Make decisions

Improve accuracy

Use the API

Account and billing

Privacy and security

Integrations

Label with teacher models

1. Upload unlabelled queries

2. Start a job

3. Review and finalise

Choose the evaluation rows

Discard or delete