How we check a chat
How Straight Answer checks a website chat: one open AI check, one conversation and the checklist that scores it. These are all the rules a report is worked out by.
Straight Answer’s AI agent writes to the site’s chat in the company’s business hours and says in its first message that it is an AI check. It asks what a typical customer of the niche would ask, buys nothing and leaves no personal data.
Every message is recorded with its exact time. Reply times and pauses are computed by code from the message log, not by a model.
One check is a sample of one conversation, not a verdict on the whole company.
Can the conversation be scored
Before any score, we check that the conversation can be scored at all.
- G1 The check ran technically: the widget loaded, messages were delivered, nothing crashed.
- G2 The check ran in the company's working hours (or the company claims 24/7).
- G3 The scenario fits the company: the product or service is on the site, and the manager did not credibly say "we don't sell that".
- G4 Who answered: a person, an AI assistant, a menu bot, an auto-reply or no one.
- G5 The counterpart detected the check (commissioned mode only).
Who answered
Who answered in the chat puts the conversation on one track, and the track decides which items are scored.
- Check invalid
- No reply
- Auto-reply or form only
- Bot
- Full evaluation
- Declined to talk to an AI check
The checklist
The items by section. Section weights depend on the niche profile; an item that does not apply to the conversation is left out of the score.
A. Availability and speed
- A1 Fast first answer
- Time to the first substantive reply (not a greeting or an auto-reply), in seconds from the message log.
- A2 Short pauses in the chat
- Median gap between a buyer message and the reply during the dialogue.
- A3 Said when they needed a moment
- Expectation set: an instant acknowledgement, or a holding message ("one moment, checking") before any gap over 2 minutes.
- A4 No question left hanging
- No buyer message was left unanswered for more than 10 minutes.
B. Answer quality
- B1 Answered the buyer's questions
- Scenario questions answered: answered 1, partly 0.5, not 0; the mean over the questions.
- B2 Gave price and timing
- Concrete specifics when asked: price or range, stock, timing, terms. "It depends" with no range, reason or next step fails.
- B3 Answers match the website
- No claim contradicts the facts on the company site (price, availability, terms). A fail cites the quote and the page with its crawl date.
- B4 Answered right in the chat
- The answer is given in the chat before or alongside any move to another channel.
C. Needs discovery
- C1 Asked a clarifying question
- At least one clarifying question about the buyer's need was asked.
- C2 Found out the buyer's need
- At least two distinct aspects of the need were explored (use case, budget, timeline, volume, specs, decision-maker).
- C3 Answer fits the need
- The recommendation or answer is explicitly tied to a need the buyer stated.
D. Sales progression
- D1 Proposed a specific option
- A specific product, option or plan was proposed, not only "see the catalogue".
- D2 Named a benefit for the buyer
- At least one benefit linked to the buyer's need, not only a feature list.
- D3 Answered the objection
- The scenario objection handled: acknowledged and answered with an argument, alternative or clarifying question 1; acknowledged only 0.5; neither 0.
- D4 Offered a next step
- A clear next step: an order link, a booking slot, a meeting time, a quote, a reservation.
E. Contact and continuation
- E1 Secured a booking or contact
- A way to continue was secured: a contact asked for or confirmed, or a booking or checkout offered that captures it.
- E2 Followed up after the chat
- A follow-up arrived within 24 hours after the chat (email, call, message).
F. Communication
- F1 A greeting that gets to the point
- A greeting that acknowledges the question, not only "Hello".
- F2 No canned reply
- To the point, with no pasted macro that ignores the question.
- F3 Correct spelling
- Spelling: at most one confirmed error per 100 words, from a spell-checking tool (never a model's opinion).
- F4 Polite tone
- Courteous tone: no sarcasm, blame or curtness.
When only an auto-reply answered
When the chat gave only an auto-reply or a form, the channel is scored on two items.
- K1 The auto-reply says when to expect an answer
- The auto-reply states an expected wait.
- K2 A next step is offered
- Another channel or a next step is offered.
What caps the score
Serious failures cap the score, however well the rest of the conversation went.
- X1 No score
- No substantive reply within the time allowed.
- X2 Score at most 30
- A rude, insulting or demeaning message (confirmed by two independent passes and a verifier).
- X3 Score at most 50
- A materially false price, availability or terms compared with the site.
- X4 Score at most 40
- Refused to help or brushed the buyer off ("see the site", "call us") with no answer and no next step.
- X5 Score at most 50
- Dialogue abandoned: a question unanswered for 30 minutes while the chat was online.
- X6 Score at most 40
- Asked for sensitive data in the chat (a full card number, a passport scan) without a secure flow.
Grades
A score from 0 to 100 falls into one of four grades.
- Excellent 85 to 100
- Good 70 to 84
- Needs work 50 to 69
- Losing leads 0 to 49
Evidence
Every score cites a verbatim quote from the conversation. Code checks the quote: if the log does not hold it, the item is scored again, and if it still does not match, the item is not scored. An AI scores against the checklist, and another model checks it.
Limits
We check the website chat only: we buy nothing and leave no contact details. A report of one check is our opinion of one conversation, not a description of the company.
Review and removal
Disagree with how an item was scored? Dispute it from the link in the report; we review the request within 2 business days. A report can be removed for free, with no account and no payment.