Political pros no better than public in predicting which messages persuade

Well, the election is over, the polls were wrong (such a surprise), most analysts were wrong (again par for the course).

What it all showed was that political campaigns spend big bucks hiring consultants to craft persuasive messaging which fails to persuade. A timely new study demonstrates that political professionals perform no better than laypeople in predicting which messages will sway voters.

In the study the researchers evaluated how well sample groups of political practitioners — professionals who work for political campaigns, polling firms, and advocacy organizations — and members of the public could predict the effectiveness of 172 campaign messages concerning 21 political issues, including legalizing marijuana, cancelling student debt, and increasing border security.

They found that both groups performed barely better than chance and that the practitioners were no more perceptive than laypeople in identifying messaging that resonates with people.

What the researchers say: “We found that neither political practitioners nor the mass public are particularly accurate in predicting which persuasive messages are more effective than others,” the led author said. “This suggests that political practitioners who craft language intended to persuade have fairly poor intuitions about which messages people will find persuasive.”

The study was published in the journal Proceedings of the National Academy of Sciences.

For the study, the researchers gathered 172 text-based political messages that political practitioners have used to support or oppose 21 distinct issues. They pulled the messages from sources such as voter guides published by various advocacy organizations or the social media accounts of prominent politicians.

An example is a message used by the Marijuana Policy Project to support the legalization of cannabis: “Polls show that a strong and growing majority of Americans agree it is time to end cannabis prohibition. Nationwide, a recent Gallup poll found that 66% support making marijuana use legal for adults.”

To measure the effectiveness of these messages, the researchers conducted a large-scale survey experiment, in which they randomly assigned 23,167 participants into either a treatment group or a control group. The treatment groups were presented with messages for three specific issues; the control group saw no messages. Then they questioned participants in both the treatment and control groups on their opinions of the issues, for a total of 67,215 observations from the participants. The researchers used this data to estimate the efficacy of each message.

Next, they asked 1,524 political practitioners with varied experience and expertise and 21,247 laypeople to predict the messages’ effectiveness. (Ninety-one percent of the practitioners reported being directly involved with developing messaging.) Both groups did little better predicting the messages’ persuasiveness than if they had guessed randomly.

The study showed that the members of the public believed that other people are more persuadable than the initial survey showed, or the practitioners expected. But after accounting for those inflated expectations, practitioners did not predict meaningfully better than laypeople, the study found.

Among the political practitioners, the study found that experience or issue expertise did not translate into a greater ability to identify effective messages.

The findings suggest that, rather than relying on their intuition, political practitioners should consider incorporating data-science techniques into their evaluations of potential messages, the researchers said.

“The main takeaway here is that political practitioners have tools available to help them identify effective messages without having to rely on their gut feeling,” the team concluded. “They could use survey experiments similar to what we did in this study. We suspect more will adopt such techniques moving forward.”

So, what? Amen to that last. Democrats please note.