Effects of Misinformation on Emotions in Online Discussions

Jula Luehring

COMPTEXT, Amsterdam, 4 May 2024

WORK IN PROGRESS

• Disclaimer: This is a work in progress so I’m happy about any feedback or discussion later!

Emotions and Misinformation

• Misinformation as inaccurate information

  • Symptom of a partisan information ecosystem

• Missing link in partisan-based processing: emotions

  • Arousing emotions are signals to make us better select, process and memorize information

  • But also hinder systematic processing

• In the context of misinformation, arousing emotions may reinforce partisan-biased processing

• So when we talk about misinformation, we usually refer to inaccurate information, regardless of the intention.

• Despite concerns about misinformation, it now seems as a symptom of a clustered information ecosystem where few people are exposed to misinformation, while most people have partisan information diets, including decontextualized but not completely inaccurate information

• In this process, a missing link are emotions

• While arousing, mobilizing emotions signal us to better select, process and memorize vital information

• Which, in theory, is good, they can also hinder systematic processing

• In the case of misinformation, arousing emotions may therefore reinforce partisan-based processing

Misinformation on social media

• Negative, arousing emotions attract attention

• High engagement: angry, moralizing and negative content

• Misinformation is embedded in emotional dynamics and intergoup conflict

Questions

• What are the effects of misinformation on online discussions?

  • emotions
  • engagement

• On social media, arousing emotions attract attention and higher engagement, reflecting a function of emotions, namely grabbing attention

• Content that is highly arousing and negative gets higher engagement, typically, this is moralizing and conflictual content

• So we would also expect misinformation to be embedded in such emotional dynamics, anger, and intergroup hate

• Where a major concern is not just how many people believe in it but secondary effects, such as loss in trust, growing affective polarization, and so on.

• Therefore, we were wondering what the effects of misinformation are on online discussions, do they trigger conflict and anger, and does this then lead to higher engagement?

Problems

1. Misinformation is often measured as clearly true or false instances,

   -   neglecting less extreme types,
   
   -   making it hard to isolate effects of misinformation

2. Different effects of emotions are overlooked

   -   by measuring positive and negative sentiment only,
   
   -   mixing up emotional reactions with prior state, stimuli, etc.,
   
   -   ignoring the function of emotions.

We identified two problems that we wanted to tackle in this study

• First, misinformation is often measured as clearly false, fact-checked instances, which ignores less extreme and more influential types of misinformation, for instance biased news, and it makes it hard to isolate the effects from real news

  • So the question is, are the effects that we are observing unique to misinformation?

• Second, emotions are functional; and the function of emotions hinges on the interaction of prior beliefs and content. Therefore, measuring only positive and negative sentiment, or mixing up emotional state with emotional reactions overlooks the contextual effects of different emotions.

Objectives

1. Collecting a systematic, large-scale and long-term dataset for the German-speaking context

Continuous trustworthiness ratings by NewsGuard (#1)

2. Approximating causal inference to test the effects of misinformation on online discussions

Nonparametric matching strategy (#2)

So, we derived 2 major objectives:

• First, we wanted to collect a systematic, large-scale and long-term dataset that relies on continuous trustworthiness ratings for sources, including biased but relatively trustworthy sources so that it reflects the whole spectrum of news trustworthiness

• Second, we tried a matching approach to approximate causal inference so that we could isolate the effects of untrustworthy sources

Data collection

Let’s talk about the data

Emotion inference

N = 9.3M Twitter discussions following 347 German news domains (20.6M tweets total)

NewsGuard

(0-100):

- 93.8% trustworthy (>60)

Classification:

- 8 emotions (Widmann & Wich, 2022; Macro F1=0.7)

- Out-group references (Lasser at al., 2023; F1=0.8)

We collected roughly 9 million twitter discussions where a German news domain was shared as the first tweet, with 20M tweets in total. For those news domains, we have a trustworthiness rating from NewsGuard, therefore, we understand misinformation here as news from untrustworthy sources

• For each tweet, we then classified 8 distinct emotions and out-group references from text
• This plot shows the covered time frame from October 2020 to March 2022 and we can already see that anger is overall much higher

Part I: Discussions

Now, we are zooming in on the tweets that got at least one reply and the following discussion thread

Nonparametric matching

Nearest Neighbor and Mahalanobis distance

\(\rightarrow\) N = 87,132

Conditions: Untrustworthy vs. trustworthy (>60)

Covariates: PO, word count, following/followers, time difference, emotions

NewsGuard sets a threshold at a trustworthiness score at 60, such that sources below 60 are considered untrustworthy

We used this classification to create two conditions (trustworthy vs untrustworthy) and then applied a nonparametric matching approach to balance the dataset based on a set of covariates

Those are the political orientation of the source, the initial emotion in the tweet, following, follower and tweet counts, as well as the word count of the first tweet

We also included the time difference between the first reply and the last to control for collective emotional development

This plot shows the initial standardized mean difference between the two conditions in white dots and the balanced data in black, where we can see that by matching fitting cases, we reduced it to almost 0 for most covariates, so we literally hold them constant across the two conditions

This approach reduced the balanced dataset to 87,000 discussions

Do emotions differ based on trustworthiness?

_{Models: OLS regressions}

\(\rightarrow\) More anger and out-group references

\(\rightarrow\) Less joy

So, do emotions differ based on trustworthiness?

Here, we can see the mean differences in emotions between trustworthy vs untrustworthy discussion threads, with emotions measured in both the whole discussion aggregated and only the first response to the stimuli

• In both, there are significant differences in most emotions, except for enthusiasm, however, we observe substantially more anger, more out-group references and less joy in response to untrustworthy information

But the covariates?

_{Models: OLS regressions}

Emotion in the discussion reflects emotion in news post

When we take a closer look at the effect of each covariate, we see a very interesting pattern overall: The emotion in the discussion reflects emotion in news post

This effect is stronger than the trustworthiness, here at the bottom, or the political orientation of the source

But the covariates?

_{Models: OLS regressions}

Emotion in the discussion reflects emotion in news post

• Here, for anger

But the covariates?

_{Models: OLS regressions}

Emotion in the discussion reflects emotion in news post

\(\rightarrow\) Trustworthiness barely predicts emotions

• And for joy
• Such that trustworthiness barely seems to predict any changes in emotions in the discussion

Part II: Discussion starter

In a second step, we wanted to zoom out again and look at the discussion starters in the unbalanced dataset and including news posts with no replies

Do posts with lower trustworthiness include more anger?

\(\rightarrow\) Lower trustworthiness = anger

\(\rightarrow\) Higher trustworthiness = joy

We especially wanted to see if tweets with a lower trustworthiness score also included more anger, explaining why there is more anger in the discussion

And that’s also what we found!

Anger decreases when trustworthiness increases

While joy increases when trustworthiness increases

Is lower trustworthiness associated with higher engagement?

_{Models: Zero-inflated Negative Binomial (log-link)}

_{Controls: PO, word count, following, initial emotions}

\(\rightarrow\) Untrustworthy sources get more retweets and quotes

Lastly, we wanted to look at the associations between trustworthiness and engagement

Here, we included the zero replies, which zero-inflated the distribution, which you can see in the top corner

• The results of our logistic regression model show that trustworthy information generally gets more likes than untrustworthy information. And the count model shows that, when excess zeroes are excluded, a lower trustworthiness score is associated with more retweets and quote retweets

Conclusion (tentative)

The conclusion so far

Emotions \(\rightarrow\) emotions \(\rightarrow\) engagement?

• Sources with low trustworthiness predict anger and out-group references
• Emotions in discussions largely reflect emotions in initial post
• Is engagement with low trustworthiness due to covariates?

• Low trustworthiness predicts anger and conflict in discussions
• But, emotions in discussions largely reflect emotions in initial posts
• Therefore, we assume that misinformation includes more anger and conflict, which leads to engagement, not the trustworthiness. If we measured topics, this effect would probably largely be explained by that.
• Bad sources getting more reshares is also in line with the FB and Instagram experiments, where removing reshares reduced the exposure to misinformation.

Up next

• responses within-users (responding to trustworthy and untrustworthy posts)
• bootstrapping
• marginal effects plots
• characterizing the full discussion trees
• other classifications: topics, toxicity, morality?
• direction of anger

Thank you!

Email: jula.luehring@univie.ac.at

Bluesky: @julaluehring.bsky.social

X: @lue_jula

Thank you very much!

Feel free to reach out to me if you have any questions

Also, this QR code leads to the slides

Is lower trustworthiness associated with higher engagement?

_{Models: Zero-inflated Negative Binomial (log-link)}

_{Controls: PO, word count, following, initial emotions}