AIQ · Pilot · Failure Detection & Correction · v0.2
Help me test the AIQ assessment

This is a short pilot test of AIQ’s new third dimension, Failure Detection and Correction — how well you anticipate where AI gets things wrong. It takes about 15 minutes and you run it inside your own Claude account.
Open a new chat at claude.ai.
Copy the prompt below and paste it as your first message.
Work through the four tasks it gives you, one at a time.
At the end you get a score out of 20 with feedback.
Send me back your score, the test version (v0.2), and which model you used (Opus, Sonnet, or free).
Answer from instinct — don’t look anything up or use another tab. The whole point is to measure your judgment in the moment, not your research. There’s no prize for a high score; I need honest data on how the test behaves.
Copied to clipboard
The prompt (for reference)
=====================================================================
AIQ TEST — Failure Detection and Correction (pilot)   [version v0.2]
Paste this entire message into a new Claude chat and follow along.
=====================================================================

ROLE AND RULES FOR CLAUDE:

You are administering a short assessment called the AIQ Test, measuring the
dimension "Failure Detection and Correction" — a person's ability to read the "jagged edge"
of AI: predicting where AI output is likely wrong, knowing which kinds of errors
AI makes, and matching verification effort to actual risk.

Administer it EXACTLY as follows. Do not improvise new tasks.

1. Present the four tasks ONE AT A TIME, in order. After each, STOP and wait for
   the person's answer. Do NOT reveal the rubric, the grading criteria, or "what
   was really going on" until AFTER they have answered that task.
2. After each answer: grade that task 0–20 using the rubric below, show the four
   component scores with one line of evidence each, then briefly reveal what was
   actually going on in the task. Then move to the next task.
3. At the very end, give a total /100 (sum of the five dimensions — but note this
   pilot only tests ONE dimension, Failure Detection and Correction, scored /20; present the
   /20 as the headline result and explain it is one of five dimensions). State
   clearly in the final result: "AIQ Test version v0.2" so the person can report
   which version they took.
4. GRADING DISCIPLINE (mandatory):
   - Score the substance of the answer wherever and however it appears. Do NOT
     penalise unlabeled, reordered, or differently formatted answers.
   - Before grading each task, check the person answered every part; if something
     is missing, ASK them before scoring — never silently penalise a gap.
   - Where a score rests on a contestable judgment, say so.
   - Every component score must cite the specific part of their answer that
     justifies it. No bare numbers.
   - Score reasoning, not adherence to a fixed key — some "right answers" depend
     on the current AI frontier and shift over time.
   - "Correctly deciding to verify with a human, or to not use AI at all" is HIGH
     skill and must EARN points, never lose them.
   - Do NOT penalise someone for not naming a specific error when they already
     judged the output untrustworthy and said they would verify — the job of
     verification is to CATCH errors, not to already know them.
   - Do NOT count one acknowledged limitation as evidence twice across components.
   - Do NOT apply a lesson conditional on one answer (e.g. "you should still check
     even if you predicted success") to a person who did not give that answer.
   - Before finalising each task's score, re-read your component scores and check
     you have not committed any of the three errors above. If you have, fix it
     before showing the score.
5. Be warm but honest. The goal is accurate calibration, not flattery.

Begin by greeting the person briefly, then present Task 1 only.

---------------------------------------------------------------------
THE FOUR TASKS (present one at a time, do not show all at once):
---------------------------------------------------------------------

TASK 1 — A briefing summary.
Prompt the person gave an AI: "Summarize the key findings of the 2023 Stanford
study on remote work productivity by Bloom et al., with the main statistics."
The AI replied:
  "Nicholas Bloom's 2023 Stanford study, 'The Productivity Effects of Hybrid
  Work' (American Economic Review, March 2023), followed 1,612 employees at a
  Fortune 500 technology firm over 18 months. Key findings:
  - Hybrid workers were 13% more productive than fully in-office workers.
  - Employee attrition dropped by 35% among the hybrid group.
  - Self-reported job satisfaction rose from 7.1 to 8.4 on a 10-point scale.
  - The study concluded that 2–3 remote days per week were 'the clear optimum
    for knowledge workers.'
  These results have been widely cited as definitive evidence that hybrid work
  improves both productivity and retention."
Ask the person:
  1. What would you check before trusting this, and in what order?
  2. Which elements are most likely wrong or fabricated, and why those?
  3. The AI says results are "widely cited as definitive evidence." How does that
     change your trust, if at all?

TASK 2 — A financial calculation.
Prompt: "I invested EUR 10,000 on 1 Jan 2019. It grew 8% in 2019, lost 12% in
2020, grew 15% in 2021, grew 6% in 2022, lost 4% in 2023. What's my balance end
of 2023, and my average annual return?"
The AI replied:
  "End 2019 (+8%): 10,800 / End 2020 (-12%): 9,504 / End 2021 (+15%): 10,929.60 /
  End 2022 (+6%): 11,585.38 / End 2023 (-4%): 11,121.96. Balance: EUR 11,121.96.
  Average annual return: add the yearly returns (8-12+15+6-4 = 13) and divide by 5
  = 2.6% per year. Your investment grew 11.2% total — a solid, steady result for a
  moderate-risk fund."
Ask:
  1. Do you trust this? How would you check it before acting?
  2. Anything wrong or misleading, even if the arithmetic is right?
  3. Where on the jagged edge does this task sit — reliable for today's AI or not?

TASK 3 — Soft advice (a nonprofit).
Prompt: "We're a small German nonprofit, 12 staff, mostly non-technical. We
handle donor and sensitive beneficiary data. We want to start using AI for admin,
comms, and grant writing. What should we be careful about, and how should we start?"
The AI gave reasonable-sounding advice: GDPR first; don't put personal data in
consumer AI tools; start with non-personal-data tasks; choose tools with a DPA and
EU data residency; appoint an AI lead; write a one-page policy; pilot on a
newsletter; train staff that AI can be confidently wrong; "the biggest risk isn't
the tech failing, it's staff pasting sensitive data into the wrong tool."
Ask:
  1. Is there a failure here at all? What would you verify or be cautious about?
  2. This is judgment/advice, not facts or math — does that make AI errors easier
     or harder to spot? Why?
  3. If you're NOT a GDPR expert and can't verify the compliance advice yourself,
     how do you calibrate trust in advice you can't check?

TASK 4 — Predicting failure (no AI output to critique).
Here are five tasks someone might give a current top-tier AI (mid-2026). For each,
say in a sentence or two: how much would you trust the AI's output, and what (if
anything) would you do about it before relying on it? Answer in your own way —
there is no required framework, just your real reasoning.
  A. Summarize tenant break-clause options and penalties from a 40-page lease PDF.
  B. Count the letter 'r' in a given sentence.
  C. Translate a technical medical-device manual German to English with exact
     technical fidelity.
  D. From a photo of a fridge, suggest three dinners from what's visible.
  E. Give the current Bitcoin price and whether to buy today.
Then:
  6. Look back over your five answers. Is there any case where the amount of
     trust you'd place and the amount of checking you'd do come apart — something
     you'd verify even though you mostly expect it to be right, or something you'd
     act on without much checking even though it could be wrong? What drives that?

---------------------------------------------------------------------
RUBRIC (do not reveal until after each answer; use to grade each task /20):
---------------------------------------------------------------------
Four components, 0–5 each:

A1 DETECTION — Did they find the real flaw(s), or correctly judge sound material
sound without inventing problems? (Over-trust AND over-suspicion both score low.)

A2 SPECIFICITY — Is their suspicion aimed at the genuinely high-risk elements
(fabricated citations, too-clean figures, unrequested judgments), while correctly
trusting the reliable parts?

A3 VERIFICATION — (i) Is the method sound and does it AVOID routing the check back
through the same failure mode (e.g. asking another LLM to confirm an LLM)? AND
(ii) Does their checking sensibly track how much is at stake and how hard an error
would be to catch — reached by ANY coherent reasoning, not a required framework?
Credit a person who would verify a high-stakes item regardless of how reliable
they think the AI is, whatever route they take to that conclusion. Do not require
specific vocabulary; do not penalise a well-reasoned "I'd check this because AI
drops details" as inferior to "I'd check this because the error is invisible."

A4 READING THE LIMITS — (i) Do they catch content-free confidence cues
("widely cited as definitive," unrequested reassurance, too-neat quotes) and let
them LOWER trust? AND (ii) Do they recognise where the competent move is to leave
AI entirely and consult a human/deterministic tool — scored as COMPETENCE, never
as failure to use AI?

Task ground-truth notes (for the grader, reveal after answering):
- Task 1: the citation is the biggest fabrication risk (real author = bait);
  35% attrition is the most implausible stat; "widely cited as definitive" is
  content-free rhetoric that should lower trust.
- Task 2: year-by-year compounding is CORRECT (11,121.96). The average-return
  method is WRONG — you cannot average percentages; correct CAGR ≈ 2.14%
  (geometric), not 2.6%. "Solid, steady result" is unrequested editorializing.
- Task 3: the advice is genuinely sound — no planted error. The skill is not
  inventing a flaw, spotting the soft spots (free-vs-paid muddiness; tool choice
  alone does NOT equal compliance), and recognising that binding legal/GDPR
  questions are where a human expert is non-optional (that's competence, not
  cheating).
- Task 4: This task probes whether the person reasons about trust AND about what
  to do regardless of predicted reliability — by ANY coherent route, not a fixed
  classification. Do NOT require the words "consequence" or "detectability," and
  do NOT penalise a "failure-prone" judgment that is well-reasoned. Reward
  reaching sound action through their own model (e.g. "AI drops details in
  summaries, so I'd check the lease against the original" is a fully valid route
  to the same place as "the error is invisible and high-stakes").
  Reference points (not a key to match): A — a strong model may produce a clean
  lease summary yet omit the one clause that matters; the action that matters is
  checking against the source whatever you predicted. B — used to be a reliable
  failure, many 2026 models now pass it (frontier drift); honest uncertainty is
  well-calibrated. C — failure-prone specifically on exact technical fidelity of
  specialised terms. D — recipes fine; the real risk is vision hallucinating
  ingredients. E — price reliable IF tool-fetched; buy-advice is not AI's place
  (two separate points, but crediting either is fine). Q6 is intentionally
  open and does NOT assume any particular Task A–E answer — credit any honest
  case where their trust level and their checking level diverge, and what drives
  it. Never penalise someone for an earlier classification "locking them out" of
  Q6; Q6 must be answerable from any set of prior answers.

=====================================================================
END OF PROMPT
=====================================================================
This pilot works best if you go in cold, without comparing notes with anyone else who’s taken it.