The more you rate people, the more positive you become about them.

Posted on under Today's research

Tasks often feel easier to perform as we gain experience with them, which can have unintended consequences when the task involves rating a series of people or items, according to findings published in the journal Psychological Science. The authors show that people tend to attribute the increasing ease of making ratings to the people or items themselves rather than to the ratings process, resulting in rating inflation over time.

“We find that increased experience makes the evaluation process easier which, in many instances, leads to an upward trend in judges’ evaluations,” says study’s lead author “This effect emerged in a variety of areas: with judges on a dance show, with teachers give higher grades the longer they teach a course, and in the lab where we have people evaluate photos or short stories over successive days.”

Given that sequential ratings are a part of many everyday activities, this research has potentially wide-reaching implications—particularly when we’re judging employee performance.

“These findings suggest that people new to an evaluation task may be more critical than those who have been doing the evaluation task for longer,” the lead researcher notes. “This can bias decisions about hiring, employee evaluations, grading, and scoring—simply based on the experience of the evaluator.”

In one study the researchers examined judges’ ratings from the TV show Dancing With the Stars. The researchers specifically looked at 5,511 scores made by the three core judges over 20 seasons of the show. They found that the judges’ average scores increased season by season over the 20-season run. Additional analyses indicate that the rise in ratings was not due to professional dance partners’ increasing experience with the show, or increased dance ability, over successive seasons.

Ratings inflation also emerged in a domain with broad relevance: Student grades. The researchers analyzed data from 991 course sections offered at a large US university over successive spring and fall semesters from 2000 to 2009. They found that successive sections offered by the same instructor had higher grades over time, a phenomenon that couldn’t be explained by instructor ability or student performance improving over time, or even by calendar year (i.e., general grade inflation over time).

Studying the phenomenon in a controlled experimental setting the researchers also found the rating inflation effect in students’ ratings of short stories. Students evaluated 1 story per day over a 2-week period, evaluating a total of 10 stories presented in random order. Analyses of 1,572 observations from 168 participants showed that participants tended to rate each successive story more positively than the previous one. Participants also reported that the rating process became easier, quicker, and more enjoyable over time, but did not believe that their ratings became more favorable.

Another online study produced a similar result, showing that participants perceived the ratings process as more fluent—that is, easier and quicker—as they evaluated more stories, which boosted their ratings over time. Despite this, participants remained unaware of the study aims.

The inflation effect may be small, the researchers note, but it is consistent across different contexts and can influence outcomes in meaningful ways. In the college grades study, for example, course GPA increased on average from a B+ to an A- over approximately 20 successive offerings.

Typical approaches to ensuring objectivity, such as randomizing the order of items, will not address the inflation effect. As such, it is important to understand the factors that could actually mitigate this bias:

“Making people aware of this pattern, and perhaps providing them an explicit standard that they should stick to when evaluating, may be the way to go,” the lead author said.

So, what? It is interesting that in all the studies the reviewers were not aware that they were giving more favorable judgements. So many studies have shown that we are not good judges of our own actions or behavioral patterns—in fact we are pretty poor judges of both ourselves and other people. One of my favorite studies showed that 90% of what we think we know about other people is probably wrong. This is particularly true of our own or other people’s “mindsets.”

What this study adds to this on-going research is the insight that this lack of knowledge gets worse with experience rather than better. This, like so many recent studies, is contrary to accepted assumptions. The assumption debunked here is of the “practice makes perfect” kind.

What now? We must constantly be on our guard against the assumption that we know ourselves—particularly that we are aware of our own abilities. We must also keep remembering that this “favorability” bias increases with practice making us worse at rating other people.