How it works
The context is JSON with theconversation so far and the draft reply.
The keys are yours to choose. One ordinal decision, quality, asks
whether the draft is ready to send, on three levels that its rubric describes:
Try it
Open this recipe in the playground to run it without an account. The user there asks for a Python function that drops duplicates and keeps the original order. The draft uses a set, so the order is lost. To run it from your code, setSQWISH_API_KEY as in the quickstart, then send this request:
Act on the answer
expected_level gives the average score, from 0 to 2, if you would rather rank drafts than send or
stop each one.
Adapt it to your data
- Rewrite the rubric. Say what makes a reply wrong, nearly right or ready in your product. Keep three levels, lowest first.
- Measure it. Run it on drafts your own reviewers have rated, and compare its scores with theirs (improve a decision).
- Train if you need to. If better wording isn’t enough, fine-tune a model on your reviewers’ ratings.
What to watch
- Drafts scored 2 that a person would have sent back.
- Level 1. It is the hardest level to tell apart from the others.
- Sources. A good score doesn’t mean the draft is backed by your sources. For that, check the draft against its sources.
How well it works
Measured withkettei-core on 1,000 examples from a public dataset, it chose the right answer 55.7% of the time. Always picking the most common answer scores 61.5%. Your own data will differ, so measure it on examples of yours before you rely on it (improve a decision).
Full results
Full results
- Dataset: HelpSteer3 (feedback, validation split) (CC BY 4.0).
- Sample: 1,000 rows, 1,000 measured decisions.
- Decisions measured:
quality. - Wording: hand-written.
Always picking the most common answer has an accuracy of 0.615, and guessing by how common each answer is has a log loss of 0.9319.These are past results on one dataset, not a promise of accuracy on yours.
Next steps
- Check a draft against its sources: catch claims your sources don’t support.
- Improve a decision: collect rated drafts, measure and train a model of your own.
- API reference: every field in the request and the answer.
