Courtrooms are full of bad science

In television crime dramas, savvy lawyers overcome improbable odds to win their cases by presenting seemingly iron-clad scientific evidence. In real-world courtrooms, however, the quality of scientific testimony can vary wildly, making it difficult for judges and juries to distinguish between solid research and so-called junk science.

This is true for all scientific disciplines, including psychology, which plays an important role in assessing such critical pieces of testimony as eyewitness accounts, witness recall, and the psychological features of defendants and litigants.

A new, multiyear study published in Psychological Science in the Public Interest finds that only 40% of the psychological assessment tools used in courts have been favorably rated by experts. Even so, lawyers rarely challenge their conclusions, and when they do, only one third of those challenges are successful.

What the researchers say: “Although courts are required to screen out junk science, legal challenges related to psychological-assessment evidence are rare,” said the lead author.

“Although some psychological assessments used in court have strong scientific validity, many do not. Unfortunately, the courts do not appear to be calibrated to the strength of the psychological-assessment evidence,” she added.

The new report examines more than 360 psychological assessment tools that have been used in legal cases, along with 372 legal cases from across all state and federal courts in the United States during the calendar years 2016, 2017, and 2018.

Psychologists provide expert evidence in a variety of court proceedings, ranging from custody disputes to disability claims to criminal cases. In developing their expert evaluation of, for example, a defendant’s competence to stand trial or a parent’s fitness for child custody,they may use tools that measure personality, intelligence, mental health,social functioning, and other psychological features. A number of federal court decisions and rules give judges the latitude to gauge the admissibility of evidence, largely by evaluating its empirical validity and its acceptance within the scientific community.

For their review, the researchers gathered results from 22 surveys of psychologists who serve as forensic experts in legal cases. They reviewed the 364 psychological assessment tools that the respondents reported having used in providing expert evidence. They found that most of them have been subjected to scientific testing, but only about 67 percent are generally accepted by the psychological community. What’s more, only 40% of the tool shave generally favorable reviews in handbooks and other sources of information about psychological tests.

The scientists also found that legal challenges to the admission of assessment evidence are rare, occurring in only about 5% of cases they reviewed. And only a third of those challenges succeeded.

According to the report: “Attorneys rarely challenge psychological expert assessment evidence, and when they do, judges often fail to exercise the scrutiny required by law.”

So, what? I am pleased to say that I have only once been asked to give evidence in a court of law. It was an experience I have no wish to repeat. The level of scientific ignorance I found was disturbing—even on very basic junior-school stuff.

There have been many studies indicating that most judges admit evidence from methodologically flawed studies and others that show attorneys and jurors lack the scientific literacy necessary to scrutinize scientific evidence.

Perhaps opportunistic psychological “experts” are the worst offenders. They are prone confuse lawyers and juries using “psychological assessment tools” that have been shown to be unscientific (Myers Briggs is a famous example of outdated rubbish dressed up as “science”) and inappropriate to the situation.

‍