Review answer quality

Use this after a model drafts a reply. Include the original request and the draft so the rating can judge whether the response actually answers the task. Open this recipe in the playground, or set D1_API_KEY as in the quickstart and run the same request:

curl --fail-with-body --silent --show-error https://console.sqwish.ai/v1/decide \
  -H "Authorization: Bearer $D1_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "model": "sqwish-d1-core",
  "context": {
    "conversation": [
      {
        "role": "user",
        "content": "Write a Python function that removes duplicates from a list but keeps the original order."
      }
    ],
    "draft": "Use a set:\n\ndef dedupe(items):\n    return list(set(items))\n\nA set can't hold the same value twice, so every item appears once."
  },
  "decisions": [
    {
      "id": "quality",
      "kind": "ordinal",
      "question": "Is this draft reply ready to send to the user?",
      "outcomes": [
        "0",
        "1",
        "2"
      ],
      "rubric": [
        "No. It is wrong, unsafe or unhelpful, or it ignores part of what the user asked.",
        "Nearly. It is on the right track, but has an error, a gap or a missing step to fix first.",
        "Yes. It is correct and complete, and does what the user asked."
      ]
    }
  ]
}
JSON

Use the result

Read decisions.quality.expected_level on the 0–2 scale, plus the full distribution. Treat a middle value as uncertainty between rubric levels, not as a new categorical label. Establish which levels lead to revision or human review using your reviewers’ own ratings. Evaluate drafts rated 2 that a person would send back. Make the rubric concrete about missing requirements, factual errors and unusable advice. Quality does not establish groundedness; when sources matter, pair this with the groundedness check.

What was evaluated

The historical accuracy is 0.557, below the majority-class baseline of 0.615. This recipe has not established useful classification performance on that benchmark. Use it as a format and rubric starting point, then improve wording and measure on your own reviewed examples before adopting it.

Historical evaluation on 2026-09-28, using sqwish-d1-core. These measurements describe that checkpoint and dataset, not current production performance or an accuracy guarantee.

Dataset: HelpSteer3 (feedback, validation split) (CC BY 4.0).
Split: feedback/validation, prompts whose hash falls below 0.55.
Sample: 1,000 rows, 1,000 measured decisions.
Measured decision IDs: quality.
Wording: hand-written.
Checkpoint SHA-256: dd420ca652cfaa10279eabb54d15afd50fe9d24d357498c4dbc512b1c3d73bb2.

Metric	Value	Recorded interval
accuracy	0.557	0.5258 to 0.589
log loss	0.9374	0.9088 to 0.9664
brier	0.5572	0.538 to 0.577
ece	0.0785	0.0544 to 0.1087

The majority-class baseline has accuracy 0.615; the class-prior baseline has log loss 0.9319. Only the decision IDs listed above were measured. Dataset labels, class balance and wording affect these results. The intervals do not measure distribution shift. Re-evaluate with your own cases, including ambiguous and out-of-scope inputs.

Improve it for your application

Keep the decision IDs and outcome order stable while evaluating changes. Review a dataset, tune the wording, and compare the result against your current policy before changing production. Check fallback so you know which model actually answered.

Build with D1

Evaluate and retrieve

Review answer quality

Use the result

What was evaluated

Improve it for your application

​Use the result

​What was evaluated

​Improve it for your application

Use the result

What was evaluated

Improve it for your application