> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqwish.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Compare models

> Score base models and your fine-tuned models on the same dataset, then compare their metrics.

An evaluation scores one model on the evaluation rows of a [dataset](https://docs.sqwish.ai/guides/datasets.md). Start one
for each model you want to compare, then read their `metrics` side by side.

With `SQWISH_DATASET_ID` set to a dataset's ID, this request compares two base models:

```python Python theme={null}
import os

from sqwishai import OneHot

client = OneHot()
submitted = client.evaluations.create(os.environ["SQWISH_DATASET_ID"], ["kettei-core", "kettei-chub"])
print(submitted["queued"], submitted["reused"])
```

```typescript TypeScript theme={null}
import { OneHot } from "sqwishai";

const client = new OneHot();
const submitted = await client.evaluations.create(process.env.SQWISH_DATASET_ID!, ["kettei-core", "kettei-chub"]);
console.log(submitted.queued, submitted.reused);
```

```bash cURL theme={null}
curl --fail-with-body https://api.sqwish.ai/v1/evaluations \
  -H "Authorization: Bearer $SQWISH_API_KEY" -H 'Content-Type: application/json' \
  -d "{\"dataset_id\":\"$SQWISH_DATASET_ID\",\"models\":[\"kettei-core\",\"kettei-chub\"]}"
```

The response lists an evaluation for each model in `data`. Its `queued` count says how many
started, and `reused` how many were already there.

Name each model the way a call names it. A family such as `support-router`, or a model by its
number such as `support-router@2`, works too:

```json theme={null}
{
  "dataset_id": "ds_...",
  "models": ["kettei-core", "support-router@2"]
}
```

In the console, choose **Evaluate on custom data** from Models, or **Evaluate models** from a
dataset. Then select the dataset and one or more models. Progress and results appear on the
dataset, beside earlier evaluations and fine-tuning outputs.

## Manage evaluations

| Request | Does |
| - | - |
| `POST /v1/evaluations` | Starts the evaluations. The response reports `queued` and `reused` counts |
| `GET /v1/evaluations?dataset_id=ds_...` | Lists the dataset's evaluations, page by page |
| `GET /v1/evaluations/{evaluation_id}` | Reads one evaluation |
| `POST /v1/evaluations/{evaluation_id}/cancel` | Stops its work |
| `POST /v1/evaluations/{evaluation_id}/retry` | Retries a failed or cancelled evaluation, keeping its saved predictions |

Sending the same selection again never retries an evaluation that failed or was cancelled. Use
`/retry` for that.

## What is reused and charged

* Each model must be one a call could name. It can be a base model, a family that is switched on,
  or a model by its number whose own switch is on.
* A family is resolved to the model it serves before work is queued.
* Existing results or work for that model and dataset are reused. This holds in a request that
  also names models with no results yet.
* Only new predictions are charged, as inference at each model's price.
* An evaluation never substitutes a fallback model.
* An evaluation scores the evaluation rows of one dataset version, shown as `dataset_version`. So
  after you [add rows](https://docs.sqwish.ai/guides/datasets.md#add-rows), comparing models again makes new evaluations.

A model that joins its family gets an evaluation on its dataset too
([read the gate](https://docs.sqwish.ai/guides/fine-tuning.md#read-the-gate)).

## Read the metrics

Read each evaluation until its `status` is no longer `queued`, `running` or `cancelling`. With the Python or
TypeScript client:

```python Python theme={null}
import os
import time

from sqwishai import OneHot

client = OneHot()
dataset_id = os.environ["SQWISH_DATASET_ID"]
evaluation = client.evaluations.create(dataset_id, models=["kettei-core"])["data"][0]
while evaluation["status"] in ("queued", "running", "cancelling"):
    time.sleep(5)
    evaluation = client.evaluations.get(evaluation["id"])
print(evaluation["metrics"])
```

```typescript TypeScript theme={null}
import { OneHot } from "sqwishai";

const client = new OneHot();
const datasetId = process.env.SQWISH_DATASET_ID!;
let [evaluation] = (await client.evaluations.create(datasetId, ["kettei-core"])).data;
while (["queued", "running", "cancelling"].includes(evaluation.status)) {
  await new Promise((resolve) => setTimeout(resolve, 5000));
  evaluation = await client.evaluations.get(evaluation.id);
}
console.log(evaluation.metrics);
```

`metrics` is `null` until the evaluation succeeds. For a dataset of expected answers it has
accuracy, log loss and calibration error. For a dataset of outcome scores it has expected reward.

| Field | Means | Better when |
| - | - | - |
| `n` | The decisions scored | |
| `accuracy` | The share of decisions whose most likely outcome is the expected one | Higher |
| `loss` | The log loss against the expected answers | Lower |
| `brier` | The Brier score: the squared distance between the probabilities and the expected answers | Lower |
| `ece` | The calibration error: how far the probabilities are from how often outcomes happen | Lower |
| `expected_reward` | For outcome scores, in place of the four above: the mean reward the probabilities expect | Higher |
| `latency_ms` | The median server time of the evaluation's requests, in milliseconds | Lower |

A result that a fine-tune takes from its own training sent no requests, so it has no `latency_ms`.

Compare models on the same examples. Small datasets give uncertain estimates.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.