Analyse every comment

A post can collect thousands of comments. Reading them all with a language model costs too much, so teams often read a sample. Here each comment is one small request with three questions: how it feels about the post, whether the commenter wants the product, and whether it is spam. You pay for the input tokens, the comment and your questions, and there is no output to pay for.

The decisions

DECISIONS = [
    {"id": "sentiment", "kind": "single",
     "question": "How does the comment feel about the post or its creator?",
     "outcomes": {"positive": "Praise, thanks, excitement or agreement.",
                  "neutral": "A question, a tag, a plain fact or no clear feeling.",
                  "negative": "Criticism, complaint, mockery or disagreement."}},
    {"id": "intent", "kind": "binary",
     "question": "Does the commenter say they want to buy, try or find the product?",
     "outcomes": {"yes": "They ask where to buy it, its price or a link, or say they want it.",
                  "no": "Anything else."}},
    {"id": "spam", "kind": "binary",
     "question": "Is the comment spam?",
     "outcomes": {"yes": "Advertising, promotion of another account, a scam or repeated text.",
                  "no": "A genuine reaction, question or reply."}},
]

Rewrite the descriptions for your own posts. Descriptions that say where one outcome ends and the next begins do most of the work (write good decisions). The context below is the comment alone, which keeps each request small. If your questions need the post, send it with the comment as JSON, such as {"post": caption, "comment": text}. Every comment then pays for the caption’s tokens too.

Price a post first

from decimal import Decimal
from sqwishai import OneHot

with OneHot() as client:
    quote = client.estimate(context="Where can I get this? Need it for my trip!", decisions=DECISIONS)

print(quote["input_tokens"], "tokens for this comment")
print("5,000 comments like it:", Decimal(quote["maximum_charge_usd"]) * 5_000)

The estimate runs no model and charges nothing. Longer comments cost more, so price a few typical ones.

Score every comment

Each comment is one request. This runs as many at once as your tier allows, and paces them to stay under its requests limit. Score many texts explains the pacing and your tier’s limits.

import asyncio
import time
from sqwishai import AsyncOneHot

RPM = 30           # your tier's requests a minute: 30 at tier 0
CONCURRENCY = 1    # your tier's operations at once: 1 at tier 0

async def score_all(comments):
    results = [None] * len(comments)
    todo = iter(range(len(comments)))
    async with AsyncOneHot() as client:
        async def worker():
            for i in todo:
                started = time.monotonic()
                try:
                    results[i] = await client.decide(
                        context=comments[i]["text"], decisions=DECISIONS,
                        idempotency_key=f"comment-{comments[i]['id']}-v1")
                except Exception as error:  # keep it, and send it again later
                    results[i] = error
                await asyncio.sleep(max(0, 60 * CONCURRENCY / RPM - (time.monotonic() - started)))
        await asyncio.gather(*(worker() for _ in range(CONCURRENCY)))
    return results

comments = [{"id": "c1", "post_id": "p1", "text": "Where can I get this? Need it for my trip!"},
            {"id": "c2", "post_id": "p1", "text": "Follow me for free followers!!!"}]
results = asyncio.run(score_all(comments))  # in a notebook: results = await score_all(comments)

Each comment’s idempotency key comes from its ID. Run the same comments again within 24 hours and the ones already answered come back without a second charge (retries and idempotency).

Add them up per post

from collections import Counter, defaultdict

posts = defaultdict(Counter)
for comment, result in zip(comments, results):
    if isinstance(result, Exception):
        continue  # send these again, as Score many texts explains
    answer = result["decisions"]
    totals = posts[comment["post_id"]]
    totals["comments"] += 1
    if answer["spam"]["probabilities"]["yes"] > 0.5:
        totals["spam"] += 1
        continue
    totals[answer["sentiment"]["top"]] += 1
    totals["expected buyers"] += answer["intent"]["probabilities"]["yes"]

for post_id, totals in posts.items():
    print(post_id, dict(totals))

Spam is set aside before the other counts, so bots don’t move them.
sentiment counts each comment once, under its likeliest feeling.
expected buyers adds up the probability of buying intent across comments. When the probabilities are well calibrated on your comments, that sum is the number of comments you can expect to show intent, with no threshold to choose. Check it on comments you have labelled.

Every threshold here, such as 0.5 for spam, is yours to set. Pick each one on comments whose answers you know (act on the answer).

What was measured

Nothing yet. These three decisions have no measured accuracy, unlike most recipes in this tab. Before you rely on them, label two or three hundred of your own comments, including spam, other languages and comments that only tag a friend. Then compare models on them. If the answers miss your audience’s way of writing, tune the wording or fine-tune a model on your labels.

Overview

Triage and routing

Agents

Retrieval and answer quality

Analyse every comment

The decisions

Price a post first

Score every comment

Add them up per post

What was measured

​The decisions

​Price a post first

​Score every comment

​Add them up per post

​What was measured

The decisions

Price a post first

Score every comment

Add them up per post

What was measured