Logo

AI - Online Marketing Academy

  • Home
  • About us
  • Courses
  • Lab
  • The Desk
  • Resources
  • Pay your Bill
  • Login / Register

Please enter key search to display results.

The 10-answer trace test: check what your AI answers from your own documents

HomeThe DeskThe 10-answer trace test: check what your AI answers from your own documents

Categories

  • Lab
  • The Desk

Recent Posts

  • Lab 1: Can the seven AI search crawlers read your site? A 15-minute robots.txt check

    Oct 25 2026
  • Lab 3: The 12 questions your buyers ask AI about your category. Write yours in 20 minutes

    Nov 13 2026
  • Lab 2: Does your site show AI a challenge page instead of content? The three-request test

    Nov 12 2026
  • The 10-answer trace test: check what your AI answers from your own documents

    Nov 13 2026

Tags Cloud

AI Search Visibility

The 10-answer trace test: check what your AI answers from your own documents

  • November 13, 2026
  • 0 Comments
  • 0 Likes

By Elizabeta Kuzevska, Co-Founder, Revenue Experts. Published 12 November 2026. About a 7-minute read, 40 minutes to do.

The scene

Imagine two of your own documents state two different payment terms, and the AI tool your team uses picked one of them and said it plainly. Nobody looked further, because the answer sounded finished. This entry is a 40-minute way to check your AI answers against your own documents, before a client does, using 10 answers your team already used and a spreadsheet.

What check this is

A generative AI tool writes fluent text whether or not a document supports it. The US National Institute of Standards and Technology calls this risk confabulation: confidently stated but false content. The confidence is in the writing, not in the evidence. The trace test turns that from a worry into a number.

Step 1. Collect 10 answers people actually used (10 minutes)

Ask three or four colleagues for AI answers they pasted into a report, an email, a client reply or a decision in the last 30 days. Forwarded answers were already trusted, so they are the ones worth checking. For each, save the question, the full answer, the date, the tool and the person.

Step 2. Build the trace sheet (10 minutes)

Create one sheet with one claim per row and a result column that holds exactly four values: Traced, Traced wrong version, Not found, Worked out. The trace sheet at the end of this entry shows the columns.

Step 3. Split every answer into claims (15 minutes)

Give each checkable statement its own row: every number, date, name and contract term. Leave out opinions and advice; you trace the facts they rest on, not the advice itself. A single paragraph often holds several claims, so the sheet has more rows than answers.

Step 4. Trace each claim (about 3 minutes a claim)

Open the document the answer cites. If it cites nothing, search your shared drive for the exact figure or phrase. Then pick the result. Traced: the figure or words are in the current version. Traced wrong version: they are there, but in an older or replaced document. Not found: nothing says it, the AI filled a gap. Worked out: the AI calculated it, so redo the calculation yourself and, if your result differs, change the row to Not found.

Step 5. Score the answers (5 minutes)

Count the four results. Then judge each answer as a whole: it passes only when every claim is Traced, or Worked out with a sum you checked. One bad claim fails the whole answer, because that is how it fails in front of a client. Our suggested bar, an estimate rather than a standard: 10 of 10 answers pass before AI answers go to clients unchecked; 8 or 9 means fix the sources first.

What a good result looks like

Ten answers, every claim marked, a number you can write down and date, and a short list of the documents that disagree with each other. That list is the start of next week’s entry.

The honest limit

Ten answers show a pattern, not a rate. This measures your answers, on your documents, in your month, and it does not replace the person who owns each document.

How Revenue Experts helps

Everything above works with a spreadsheet and an hour. Where it gets heavy is doing it every month across thousands of documents. Context Studio, our joint project with Ollon, turns your documents into records your team reviews and answers your team can check: its Document Chat cites the document and section behind each statement, and its evaluations compare each run with answers your reviewers have checked. Said first, the limit: it does not block an unsupported answer on its own; a person decides what sticks. The way in is a demo, then a scoped pilot on one document type with an accuracy score measured on your own documents. Book 20 minutes with John Bush, our co-founder: https://meetings.hubspot.com/john2956

If this was useful, the next entry arrives on Thursday. Subscribe to The Revenue Signal

Frequently asked questions

If an AI answer already shows citations, do we still trace it? Yes, a sample. A citation tells you where to look; the claim still has to sit in that section of the current version.

How many answers are enough? Ten real, forwarded answers show the pattern. For the monthly check, a fixed list of about 20 questions lets you compare month to month.

Who should own the answer key? The person who owns each document. IT runs the tool; the business owns the facts.

The trace sheet

Column What goes in it
A Question the question that was asked
B Claim one checkable statement from the answer
C Answer the full answer it came from
D Document the document that should hold the fact
E Page or section where in that document
F Exact words the words on the page that match
G Result Traced, Traced wrong version, Not found, or Worked out
H Note what the current version says, or the calculation you redid

Table 1. The trace sheet. One claim per row.

Related

  • The Desk: turn your documents into answers you can trust
  • The Lab: is your site ready for AI search?
  • See answers that cite your own documents: book 20 minutes with John Bush

Sources

[1] NIST, Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1), on confabulation.

[2] Revenue Experts, field service software citation study, 3 October 2026 (12 questions, 108 answers, 1,493 citations).

Lab 2: Does your site show AI a challenge page instead of content? The three-request test

Lab 3: The 12 questions your buyers ask AI about your category. Write yours in 20 minutes

© 2025 OMA | Powered by Revenue Experts| Hosted by MSP | 

  • Term & Conditions
  • Privacy Policy