> ## Documentation Index > Fetch the complete documentation index at: https://docs.sqwish.ai/llms.txt > Use this file to discover all available pages before exploring further. # Evaluate models on a dataset > Evaluate models on the evaluation rows of a dataset's latest version. Each queued evaluation records the model it scores (`model_id`, `model_label`), for a family the model it served then, and the family or model by its number it was asked by (`named`). It scores that model to its end, even if the family moves or is switched off meanwhile. A model evaluated on that version already, or being evaluated, is reused as it stands; a failed or cancelled one runs again only through `POST /v1/evaluations/{evaluation_id}/retry`. ## OpenAPI ````yaml /openapi.json post /v1/evaluations openapi: 3.1.0 info: title: OneHot API description: >- Decisions your application can act on: send a context and the questions you want answered, and get a probability for every answer you allow. Authenticate with `Authorization: Bearer onehot_sk_...`. A short guide for coding agents is at /llms.txt. An API key reads its account but never changes it: keys, billing and settings change in the console. Money fields ending in _usd are exact decimal strings, never integer cents. Over a rate limit a request gets 429 with Retry-After, and one with more input tokens than your minute allows gets 422 too_many_tokens. version: 0.1.0 servers: - url: https://api.sqwish.ai security: [] tags: - name: Decisions description: Probabilities for questions with fixed outcomes. - name: Models description: Base models, and fine-tuned models by their numbers. - name: Families description: >- Lines of models trained on one dataset, and the switches that let them answer calls. - name: Datasets description: >- Validated training examples. A dataset grows in versions, and a version never changes. - name: Evaluations description: Reusable model results on a dataset's fixed evaluation rows. - name: Fine-tuning description: Training jobs, their events and held-out results. - name: Prompt tuning description: Better wording for a decision, tested on your rows. Words, not weights. - name: Labeling description: Teacher labels for real queries, reviewed by you. - name: Accounts description: Email OTP sign-in, revocable sessions, security and API keys. - name: Billing description: Exact USD balances, purchases, automatic top-ups and referrals. - name: Integrations description: Endpoints other tools call, such as Claude Code. - name: System description: Health and discovery. paths: /v1/evaluations: post: tags: - Evaluations summary: Evaluate models on a dataset description: >- Evaluate models on the evaluation rows of a dataset's latest version. Each queued evaluation records the model it scores (`model_id`, `model_label`), for a family the model it served then, and the family or model by its number it was asked by (`named`). It scores that model to its end, even if the family moves or is switched off meanwhile. A model evaluated on that version already, or being evaluated, is reused as it stands; a failed or cancelled one runs again only through `POST /v1/evaluations/{evaluation_id}/retry`. operationId: createEvaluations requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EvaluationCreate' responses: '200': description: Successful Response content: application/json: schema: properties: dataset_id: type: string title: Dataset Id data: items: $ref: '#/components/schemas/Evaluation' title: Data type: array queued: title: Queued type: integer reused: title: Reused type: integer required: - dataset_id - data - queued - reused title: EvaluationSubmission type: object 4XX: description: >- Client error. On a 422, `error.details` lists each field that failed. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' 5XX: description: Server error. A 503 names what is not running on this server. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' security: - BearerAuth: [] components: schemas: EvaluationCreate: properties: dataset_id: type: string pattern: ^ds_[a-f0-9]{24}$ title: Dataset Id models: items: type: string type: array maxItems: 16 minItems: 1 title: Models description: >- What to evaluate, each named as a call names it: a family, such as router, evaluated with the model it serves when the evaluation is queued; a model by its number, such as router@4; or a base model's id. Each must answer calls, so one switched off is refused. additionalProperties: false type: object required: - dataset_id - models title: EvaluationCreate Evaluation: properties: id: type: string title: Id dataset_id: type: string title: Dataset Id dataset_version: type: integer title: Dataset Version description: The dataset version whose evaluation rows it scores. model_id: type: string title: Model Id model_sha256: type: string title: Model Sha256 model_label: type: string title: Model Label model_name: anyOf: - type: string - type: 'null' title: Model Name version: anyOf: - type: integer - type: 'null' title: Version status: type: string enum: - queued - running - cancelling - succeeded - failed - cancelled title: Status created_at: type: number title: Created At updated_at: type: number title: Updated At generation: type: integer title: Generation training_job_id: anyOf: - type: string - type: 'null' title: Training Job Id dataset_sha256: type: string title: Dataset Sha256 evaluation_sha256: type: string title: Evaluation Sha256 method: type: string enum: - sft - reward title: Method evaluation_rows: type: integer title: Evaluation Rows decisions: type: integer title: Decisions named: type: string title: Named description: >- The family, or the model by its number, the request named, whose idle clock the run moves when it finishes. completed_rows: type: integer title: Completed Rows metrics: anyOf: - additionalProperties: true type: object - type: 'null' title: Metrics description: >- Null until the evaluation succeeds. Then `n`, the decisions scored, with `accuracy`, `loss`, `brier` and `ece` for expected answers or `expected_reward` for rewards, and `latency_ms`, the median server milliseconds per request, when it was measured. A result taken from a fine-tune's own held-out scoring sends no request, so it has none. error: anyOf: - additionalProperties: true type: object - type: 'null' title: Error type: object required: - id - dataset_id - dataset_version - model_id - model_sha256 - model_label - model_name - version - status - created_at - updated_at - generation - training_job_id - dataset_sha256 - evaluation_sha256 - method - evaluation_rows - decisions - completed_rows - metrics - error title: Evaluation ErrorResponse: description: The body of every error. properties: error: $ref: '#/components/schemas/ErrorBody' required: - error title: ErrorResponse type: object ErrorBody: properties: code: description: A stable code to branch on, such as invalid_request. title: Code type: string message: description: What went wrong and what to change, in plain words. title: Message type: string retryable: description: >- True when another attempt may succeed; it does not prove that no work was done. Respect Retry-After when present. Only operations that support Idempotency-Key replay a prior result, including /v1/decide for 24 hours. title: Retryable type: boolean request_id: description: Also sent as X-Request-ID. Quote it in bug reports. title: Request Id type: string details: anyOf: - items: $ref: '#/components/schemas/ErrorDetail' type: array - type: 'null' default: null description: On validation errors, one entry for each field that failed. title: Details required: - code - message - retryable - request_id title: ErrorBody type: object ErrorDetail: properties: path: description: Where the problem is, such as ["body", "decisions", 0, "outcomes"]. items: anyOf: - type: string - type: integer title: Path type: array message: title: Message type: string type: anyOf: - type: string - type: 'null' default: null title: Type required: - path - message title: ErrorDetail type: object securitySchemes: BearerAuth: type: http scheme: bearer bearerFormat: onehot_sk_... description: Create an API key in the console. ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.