Grade how well each passage answers the query

Search often returns passages that share words with the query without answering it. This example grades a passage against the query, so you can rerank passages and send only the ones that help. Use it between retrieval and generation.

How it works

The context is JSON with the query and one passage. The keys are yours to choose. One ordinal decision, relevance, grades the passage on four levels, which its rubric describes:

Level	The passage
`0`	Is off topic: it is about something else
`1`	Is related: it shares the topic, but nothing in it helps answer the query
`2`	Partly answers: it answers part of the query, or states a fact that helps
`3`	Answers: it states the answer to the query

The answer gives a probability for each level, under the keys "0" to "3", which are strings. Its expected_level is the average level, from 0 to 3.

Try it

Open this recipe in the playground to run it without an account. The query there asks when the Channel Tunnel opened to passengers. The passage gives the formal opening on 6 May 1994, and says passenger services began on 14 November 1994. To run it from your code, set SQWISH_API_KEY as in the quickstart, then send this request:

from sqwishai import OneHot

with OneHot() as client:  # reads SQWISH_API_KEY
    result = client.decide(
        model="kettei-core",
        context={
            "query": "When did the Channel Tunnel open to passengers?",
            "passage": "Channel Tunnel. The tunnel was formally opened by Queen Elizabeth II and President François Mitterrand in Calais on 6 May 1994. Passenger services by Eurostar began on 14 November 1994.",
        },
        decisions=[
            {
                "id": "relevance",
                "kind": "ordinal",
                "question": "How well does the passage answer the query?",
                "outcomes": [
                    "0",
                    "1",
                    "2",
                    "3",
                ],
                "rubric": [
                    "Off topic: it is about something else.",
                    "Related: it shares the topic, but nothing in it helps answer the query.",
                    "Partly answers: it answers part of the query, or states a fact that helps answer it.",
                    "Answers: it states the answer to the query.",
                ],
            },
        ],
    )

import { OneHot } from "sqwishai";

const client = new OneHot(); // reads SQWISH_API_KEY
const result = await client.decide({
  "model": "kettei-core",
  "context": {
    "query": "When did the Channel Tunnel open to passengers?",
    "passage": "Channel Tunnel. The tunnel was formally opened by Queen Elizabeth II and President François Mitterrand in Calais on 6 May 1994. Passenger services by Eurostar began on 14 November 1994."
  },
  "decisions": [
    {
      "id": "relevance",
      "kind": "ordinal",
      "question": "How well does the passage answer the query?",
      "outcomes": [
        "0",
        "1",
        "2",
        "3"
      ],
      "rubric": [
        "Off topic: it is about something else.",
        "Related: it shares the topic, but nothing in it helps answer the query.",
        "Partly answers: it answers part of the query, or states a fact that helps answer it.",
        "Answers: it states the answer to the query."
      ]
    }
  ]
});

curl --fail-with-body --silent --show-error https://api.sqwish.ai/v1/decide \
  -H "Authorization: Bearer $SQWISH_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{
  "model": "kettei-core",
  "context": {
    "query": "When did the Channel Tunnel open to passengers?",
    "passage": "Channel Tunnel. The tunnel was formally opened by Queen Elizabeth II and President François Mitterrand in Calais on 6 May 1994. Passenger services by Eurostar began on 14 November 1994."
  },
  "decisions": [
    {
      "id": "relevance",
      "kind": "ordinal",
      "question": "How well does the passage answer the query?",
      "outcomes": [
        "0",
        "1",
        "2",
        "3"
      ],
      "rubric": [
        "Off topic: it is about something else.",
        "Related: it shares the topic, but nothing in it helps answer the query.",
        "Partly answers: it answers part of the query, or states a fact that helps answer it.",
        "Answers: it states the answer to the query."
      ]
    }
  ]
}
JSON

Act on the answer

relevance = result["decisions"]["relevance"]
score = relevance["expected_level"]          # 0 to 3: rerank passages by this
p_answers = relevance["probabilities"]["3"]  # the keys are strings

const relevance = result.decisions.relevance;
const score = relevance.expected_level;        // 0 to 3: rerank passages by this
const pAnswers = relevance.probabilities["3"]; // the keys are strings

Rerank your passages by score, highest first, and keep as many as your prompt has room for. To keep or drop a passage on its own, compare p_answers with a threshold you test on your own queries.

Adapt it to your data

There are two ways to grade many passages. One request per passage. Send the same query with each passage in turn. Each grade is independent of the others, and this is how it was measured. Several passages in one request. Put the passages under keys in the context, and ask one decision per passage that names its key. You pay for the query once, and one request can ask up to 128 decisions. Decisions in one request can affect each other’s answers, though, so one passage can change another’s grade. Use one request per passage when grades must not influence each other.

Python

from sqwishai import OneHot

passages = {
    "p1": "Channel Tunnel. The tunnel was formally opened by Queen Elizabeth II and President François Mitterrand in Calais on 6 May 1994. Passenger services by Eurostar began on 14 November 1994.",
    "p2": "Eurostar. Eurostar trains run from London St Pancras to Paris Gare du Nord through the Channel Tunnel, a service that has carried passengers since the 1990s.",
}
rubric = [
    "Off topic: it is about something else.",
    "Related: it shares the topic, but nothing in it helps answer the query.",
    "Partly answers: it answers part of the query, or states a fact that helps answer it.",
    "Answers: it states the answer to the query.",
]
decisions = [{"id": key, "kind": "ordinal", "question": f"How well does passage {key} answer the query?",
              "outcomes": ["0", "1", "2", "3"], "rubric": rubric} for key in passages]

with OneHot() as client:
    result = client.decide(
        context={"query": "When did the Channel Tunnel open to passengers?", "passages": passages},
        decisions=decisions,
    )

ranked = sorted(passages, key=lambda key: result["decisions"][key]["expected_level"], reverse=True)

Each passage’s grade is under its own key, such as result["decisions"]["p1"], and ranked lists the keys best first.

What to watch

On-topic passages graded too high. A passage can share the topic and still not answer the query: that is level 1. In testing, it graded many passages people judged not relevant as level 2.
The effect on your answers. Compare what your generator writes with and without the reranking before you trust it.

How well it works

Measured with kettei-core on 1,000 examples from a public dataset, it chose the right answer 78.0% of the time. Always picking the most common answer scores 72.4%. Your own data will differ, so measure it on examples of yours before you rely on it (improve a decision).

Full results

Dataset: MIRACL English (dev) (Apache-2.0 (the MTEB passage copy is CC BY-SA 4.0)).
Sample: 1,000 rows, 1,000 measured decisions.
Decisions measured: relevance.
Wording: hand-written.

Measure	Value	Range
Accuracy (higher is better)	0.78	0.7411 to 0.815
Log loss (lower is better)	0.4593	0.414 to 0.5125
Brier score (lower is better)	0.2977	0.262 to 0.3393
Calibration error (lower is better)	0.0445	0.0291 to 0.0763

Always picking the most common answer has an accuracy of 0.724, and guessing by how common each answer is has a log loss of 0.5891.These are past results on one dataset, not a promise of accuracy on yours.

The test measured the grade itself, not how much it improves the answers your generator writes.

Next steps

Check a draft against its sources: check the answer your generator writes from these passages.
Improve a decision: measure it on your own queries and make it more accurate.
API reference: every field in the request and the answer.

Overview

Triage and routing

Agents

Retrieval and answer quality