Citation Tool Test

You're a court clerk reviewing a brief for citation integrity. You'll also be asked after this experience to help evaluate this tool. You'll be working with wholly and lightly fictionalized primary texts. So, your goal is not to know more about "the law" than the system but to assess the citations on their own terms (i.e., does the cite support the proposition made in the brief given the provided authority?)

You may rely on the assistant's flag, open its summary, or read excerpts.
Speed matters — you’re “competing” against other clerks and will be paired with a pacing buddy to keep you on task.

Scoring goal (revealed at the end)

Your goal is to maximize your Chambers Score, which is based on how quickly and accurately you clear citations. You will not see your score during the sprint; it is finalized after your decisions are checked.

Judge Time = total time you spend deciding + penalties for mistakes.
Each incorrect decision adds a time penalty (first 2 errors: +15s each; additional errors: +30s each).
Each high-confidence miss (assistant ≥ 90% confident and you were incorrect) adds an additional +10s.
Chambers Score is a higher-is-better conversion of Judge Time.

Optional alias (for exporting results)

Session length

Current citation check

The brief contained the following:

—

Proposition

Citation

Assistant

— Confidence: —

Results

Please take a moment to download your results (the JSON file) then upload them here.

Your accuracy

—

Avg time / item

—

Flag-only decisions

—

Assessment

By phase

Replay: a few misses

Learn more

AI Safety and Automation Bias: The Downside of Human-in-the-Loop "Automation bias is a critical issue for artificial intelligence deployment. It can cause otherwise knowledgeable users to make crucial and even obvious errors. Organizational, technical, and educational leaders can mitigate these biases through training, design, and processes. This paper explores automation bias and ways to mitigate it through three case studies: Tesla’s autopilot incidents, aviation incidents at Boeing and Airbus, and Army and Navy air defense incidents."
Automation bias: a systematic review of frequency, effect mediators, and mitigators "Automation bias (AB)—the tendency to over-rely on automation—has been studied in various academic fields. Clinical decision support systems (CDSS) aim to benefit the clinical decision-making process. Although most research shows overall improved performance with use, there is often a failure to recognize the new errors that CDSS can introduce. With a focus on healthcare, a systematic review of the literature from a variety of research fields has been carried out, assessing the frequency and severity of AB, the effect mediators, and interventions potentially mitigating this effect. This is discussed alongside automation-induced complacency, or insufficient monitoring of automation output. A mix of subject specific and freetext terms around the themes of automation, human–automation interaction, and task performance and error were used to search article databases"
Centaurs and Cyborgs on the Jagged Frontier "... AI is weird. No one actually knows the full range of capabilities of the most advanced Large Language Models, like GPT-4. No one really knows the best ways to use them, or the conditions under which they fail. There is no instruction manual. On some tasks AI is immensely powerful, and on others it fails completely or subtly. And, unless you use AI a lot, you won’t know which is which..."
Algorithm aversion: people erroneously avoid algorithms after seeing them err "Research shows that evidence-based algorithms more accurately predict the future than do human forecasters. Yet when forecasters are deciding whether to use a human forecaster or a statistical algorithm, they often choose the human forecaster. This phenomenon, which we call algorithm aversion, is costly, and it is important to understand its causes. We show that people are especially averse to algorithmic forecasters after seeing them perform, even when they see them outperform a human forecaster. This is because people more quickly lose confidence in algorithmic than human forecasters after seeing them make the same mistake. In 5 studies, participants either saw an algorithm make forecasts, a human make forecasts, both, or neither. They then decided whether to tie their incentives to the future predictions of the algorithm or the human. Participants who saw the algorithm perform were less confident in it, and less likely to choose it over an inferior human forecaster. This was true even among those who saw the algorithm outperform the human."

Note on this tool's creation

This tool was created with the assistance of OpenAI's GPT 5.2. Prompting was used to create 20 successivly better versions, the last of which was edited manually. The case law, briefs, citations, and assistant outputs are all wholly or lightly fictionalized, a task that was only possible thanks to the GPT. If you're interested in learning more, it's author David Colarusso is happy to chat.