Compare models

An evaluation scores one model on the evaluation rows of a dataset. Start one for each model you want to compare, then read their metrics side by side. With SQWISH_DATASET_ID set to a dataset’s ID, this request compares two base models:

import os

from sqwishai import OneHot

client = OneHot()
submitted = client.evaluations.create(os.environ["SQWISH_DATASET_ID"], ["kettei-core", "kettei-chub"])
print(submitted["queued"], submitted["reused"])

import { OneHot } from "sqwishai";

const client = new OneHot();
const submitted = await client.evaluations.create(process.env.SQWISH_DATASET_ID!, ["kettei-core", "kettei-chub"]);
console.log(submitted.queued, submitted.reused);

curl --fail-with-body https://api.sqwish.ai/v1/evaluations \
  -H "Authorization: Bearer $SQWISH_API_KEY" -H 'Content-Type: application/json' \
  -d "{\"dataset_id\":\"$SQWISH_DATASET_ID\",\"models\":[\"kettei-core\",\"kettei-chub\"]}"

The response lists an evaluation for each model in data. Its queued count says how many started, and reused how many were already there. Name each model the way a call names it. A family such as support-router, or a model by its number such as support-router@2, works too:

{
  "dataset_id": "ds_...",
  "models": ["kettei-core", "support-router@2"]
}

In the console, choose Evaluate on custom data from Models, or Evaluate models from a dataset. Then select the dataset and one or more models. Progress and results appear on the dataset, beside earlier evaluations and fine-tuning outputs.

Manage evaluations

Request	Does
`POST /v1/evaluations`	Starts the evaluations. The response reports `queued` and `reused` counts
`GET /v1/evaluations?dataset_id=ds_...`	Lists the dataset’s evaluations, page by page
`GET /v1/evaluations/{evaluation_id}`	Reads one evaluation
`POST /v1/evaluations/{evaluation_id}/cancel`	Stops its work
`POST /v1/evaluations/{evaluation_id}/retry`	Retries a failed or cancelled evaluation, keeping its saved predictions

Sending the same selection again never retries an evaluation that failed or was cancelled. Use /retry for that.

What is reused and charged

Each model must be one a call could name. It can be a base model, a family that is switched on, or a model by its number whose own switch is on.
A family is resolved to the model it serves before work is queued.
Existing results or work for that model and dataset are reused. This holds in a request that also names models with no results yet.
Only new predictions are charged, as inference at each model’s price.
An evaluation never substitutes a fallback model.
An evaluation scores the evaluation rows of one dataset version, shown as dataset_version. So after you add rows, comparing models again makes new evaluations.

A model that joins its family gets an evaluation on its dataset too (read the gate).

Read the metrics

Read each evaluation until its status is no longer queued, running or cancelling. With the Python or TypeScript client:

import os
import time

from sqwishai import OneHot

client = OneHot()
dataset_id = os.environ["SQWISH_DATASET_ID"]
evaluation = client.evaluations.create(dataset_id, models=["kettei-core"])["data"][0]
while evaluation["status"] in ("queued", "running", "cancelling"):
    time.sleep(5)
    evaluation = client.evaluations.get(evaluation["id"])
print(evaluation["metrics"])

import { OneHot } from "sqwishai";

const client = new OneHot();
const datasetId = process.env.SQWISH_DATASET_ID!;
let [evaluation] = (await client.evaluations.create(datasetId, ["kettei-core"])).data;
while (["queued", "running", "cancelling"].includes(evaluation.status)) {
  await new Promise((resolve) => setTimeout(resolve, 5000));
  evaluation = await client.evaluations.get(evaluation.id);
}
console.log(evaluation.metrics);

metrics is null until the evaluation succeeds. For a dataset of expected answers it has accuracy, log loss and calibration error. For a dataset of outcome scores it has expected reward.

Field	Means	Better when
`n`	The decisions scored
`accuracy`	The share of decisions whose most likely outcome is the expected one	Higher
`loss`	The log loss against the expected answers	Lower
`brier`	The Brier score: the squared distance between the probabilities and the expected answers	Lower
`ece`	The calibration error: how far the probabilities are from how often outcomes happen	Lower
`expected_reward`	For outcome scores, in place of the four above: the mean reward the probabilities expect	Higher
`latency_ms`	The median server time of the evaluation’s requests, in milliseconds	Lower

A result that a fine-tune takes from its own training sent no requests, so it has no latency_ms. Compare models on the same examples. Small datasets give uncertain estimates.

Get started

Make decisions

Improve accuracy

Use the API

Account and billing

Privacy and security

Integrations

Compare models

Manage evaluations

What is reused and charged

Read the metrics

​Manage evaluations

​What is reused and charged

​Read the metrics

Manage evaluations

What is reused and charged

Read the metrics