class: center, middle, inverse, title-slide .title[ # Combining survey and Big Data ] .subtitle[ ## The case of emotion macroscopes ] .author[ ### David Garcia
Graz University of Technology and CSH Vienna
] .date[ ### SICSS 2022, slides available at: dgarcia.eu/SICSS-2022 ] --- layout: true <div class="my-footer"><span>David Garcia - Inaugural Lecture</span></div> --- background-image: url(figures/AboutUS.svg) background-size: 98% --- <center> <img src="figures/CSSinfo.png" width="900" /> ### Aim: understanding human behavior and socio-technical phenomena in the digital society --- # Outline </br> ## 1. Methodological Deficits in CSS ## 2. Social Media Macroscopes of Emotions ## 3. Validating a Weekly UK Macroscope ## 4. Validating a Daily Austria Macroscope --- ## *Computational* in Computational Social Science - **Digital**<br> Based on large datasets of human behavior, for example produced by the Web and social media - **Computerized**<br> The quantitative analysis of data in an automated, tractable, repeatable, and extensible fashion - **Generative**<br> Application of data and results to design of agent-based models that explain complex social phenomena and motivate interventions [Social Media Data in Affective Science. Max Pellert, Simon Schweighofer and David Garcia. Handbook of Computational Social Science, vol 1 (2022)](https://www.routledge.com/Handbook-of-Computational-Social-Science-Volume-1-Theory-Case-Studies/Engel-Quan-Haase-Liu-Lyberg/p/book/9780367456528) --- ## Strengths and weaknesses of digital trace data **Strengths:** - Complementary approach to traditional survey and experimental methods - Unprecedented scales and granularities - Ease of data access, replicability of results - Potentially high external validity, behavior in vivo **Weaknesses:** - Limits to internal validity: Lack of counterfactuals hinders causal analysis - Self-selection bias: Who talks? Normalization issues: Who is silent? - Data gatekeepers, Twitter as a model organism - Intractability of black-box predictions and data piñatas <div style="font-size:15pt"> Bit By Bit: Social Research in the Digital Age. M. Salganik (2017) </div> <div style="font-size:15pt"> Meaningful measures of human society in the twenty-first century. D. Lazer et al. (2021) </div> --- # Avoid making a data piñata <img src="figures/pinata.png" width="1050" /> --- ## The Hype Cycle of Computational Social Science <center> <img src="figures/Hype1.svg" width="900" /> --- ## The Hype Cycle of Computational Social Science <center> <img src="figures/Hype2.svg" width="900" /> --- ## The Hype Cycle of Computational Social Science <center> <img src="figures/Hype3.svg" width="900" /> --- background-image: url(figures/VennV2-1.svg) background-size: 97% --- background-image: url(figures/VennV2.svg) background-size: 97% --- # Social Media Macroscopes of Emotions </br> ## 1. Methodological Deficits in CSS ## *2. Social Media Macroscopes of Emotions* ## 3. Validating a Weekly UK Macroscope ## 4. Validating a Daily Austria Macroscope --- # Social Media Macroscopes <img src="figures/earth.svg" width="800" style="display: block; margin: auto;" /> --- layout: true <div class="my-footer"><span> <a href=https://arxiv.org/abs/2107.13236> Social media emotion macroscopes reflect emotional experiences in society at large. David Garcia, Max Pellert, Jana Lasser, Hannah Metzler. https://arxiv.org/abs/2107.13236 (2021)</a></span></div> --- # Social Media Macroscopes of Emotions .pull-left[ <img src="figures/Macy.jpg" width="1100" /> <font size="5"> <a href="https://science.sciencemag.org/content/333/6051/1878/"> Diurnal and seasonal mood vary with work, sleep, and daylength across diverse cultures. Golder & Macy, Science (2011) </a> </font> ] .pull-right[ <img src="figures/hedonometer.png" width="1100" /> <font size="5"> <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0026752"> Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter. Dodds et al. PLoS One (2011) </a> </font> ] --- ## Examples of social media emotion macroscopes .pull-left[ <img src="figures/Paris.png" width="500" /> <font size="5"> <a href="https://journals.sagepub.com/doi/full/10.1177/0956797619831964"> Collective Emotions and Social Resilience in the Digital Traces After a Terrorist Attack. Garcia & Rime, Psychological Science (2019) </a> </font> ] .pull-right[ <img src="figures/COVID.png" width="1100" /> <font size="5"> <a href="https://psyarxiv.com/qejxv"> Collective Emotions During the COVID-19 Outbreak. Metzler et al. Psyarxiv (2021) </a> </font> ] --- ## Limits of Social Media Data to Study Emotion <img src="figures/Jaidka.png" width="1100" /> [Estimating geographic subjective well-being from Twitter: A comparison of dictionary and data-driven language methods. Jaidka et al. PNAS (2020)](https://www.pnas.org/content/117/19/10165.short) --- ## When the micro and macro level might not match .pull-left[ <img src="figures/earth.svg" width="500" /> ] .pull-right[ Concerns about macroscopes: 1. Representation issues 2. Performative behavior 3. Measurement error and bias 4. Researcher degrees of freedom ] [Social Media Data in Affective Science. Max Pellert, Simon Schweighofer and David Garcia. Handbook of Computational Social Science, vol 1 (2022)](https://www.routledge.com/Handbook-of-Computational-Social-Science-Volume-1-Theory-Case-Studies/Engel-Quan-Haase-Liu-Lyberg/p/book/9780367456528) --- # Validating a Weekly UK Macroscope </br> ## 1. Methodological Deficits in CSS ## 2. Social Media Macroscopes of Emotions ## *3. Validating a Weekly UK Macroscope* ## 4. Validating a Daily Austria Macroscope --- # Validating a UK emotion macroscope <img src="figures/MacroTest2.svg" width="975" style="display: block; margin: auto;" /> --- # Data to test the macroscope <img src="figures/Data.svg" width="1100" style="display: block; margin: auto;" /> - Text analysis: dictionary-based (LIWC) and supervised (RoBERTa) - Gender-rescaled time series of emotional expression - Pre-registered hypotheses with prediction period from Nov 2020 --- # Sadness in Twitter and YouGov <img src="figures/Sadness.svg" width="1200" style="display: block; margin: auto;" /> - Similar results with dictionary-based and supervised methods (r~0.65) --- # Anxiety in Twitter and YouGov <img src="figures/Anxiety.svg" width="1200" style="display: block; margin: auto;" /> - Improvement thanks to gender information in tweets --- # Joy in Twitter and YouGov <img src="figures/Joy.svg" width="1200" style="display: block; margin: auto;" /> - Good correlation with supervised method but no correlation with dictionary-based method --- # Overview of results <img src="figures/Corrs.png" width="1200" style="display: block; margin: auto;" /> - Consistent results for both methods in sadness - Similar for anxiety except for the classifier in the prediction period - LIWC positive largely fails: lexicon is too general - Joy supervised classifier has similar correlation as negative classifiers --- # Exploring 12 emotional states .pull-left[ - Time series of number sentences like "I am [emotion]" on Twitter - Weak correlations happen for infrequent emotions in text - Comparison: US weekly pre-election polls correlate with 0.66 - Arxiv preprint at https://arxiv.org/abs/2107.13236 ] .pull-right[ <img src="figures/Figure2.svg" width="700" /> ] --- layout: true <div class="my-footer"><span> Validating daily social media macroscopes of emotions. Max Pellert, Hannah Metzler, Michael Matzenberger, David Garcia. Scientific Reports (Forthcoming)</span></div> --- # Validating a Daily Austria Macroscope </br> ## 1. Methodological Deficits in CSS ## 2. Social Media Macroscopes of Emotions ## 3. Validating a Weekly UK Macroscope ## *4. Validating a Daily Austria Macroscope* --- # Austrian macroscope in Der Standard .pull-left[ - 20-day emotion survey in derstandard.at (N=268,128) - Daily frequency, 3-day windows - Text from Der Standard forum (N=452,013) - Austrian tweets (N=515,187) filtered as UK macroscope - Compared dictionary-based (LIWC) and supervised model (GS) ] .pull-right[ <img src="figures/Umfrage.png" width="450" /> ] --- ## Survey emotions and Der Standard sentiment <img src="figures/DerStandardResult.png" width="1000" style="display: block; margin: auto;" /> --- ## Survey emotions and Twitter sentiment <img src="figures/DS2.svg" width="1000" style="display: block; margin: auto;" /> --- ## Testing various configurations <img src="figures/DerStandardCorrs.png" width="1000" style="display: block; margin: auto;" /> --- # Correlations with new COVID-19 cases .pull-left[ <img src="figures/DS31.svg" width="600" /> ] .pull-right[ <img src="figures/DS32.svg" width="600" /> ] - Do correlations attenuate due to additional social media measurement error? - Survey emotion correlation with new cases as strong as Twitter sentiment - Errors sources might be different: Need for conceptual validations --- ## Online Media for Social Sensing of Emotions? <img src="figures/socialsensing.svg" width="850" style="display: block; margin: auto;" /> --- ## Social media macroscopes: Take-home message <img src="figures/summary2.svg" width="950" style="display: block; margin: auto;" /> **Despite important concerns about Computational Social Science methods in terms of representativity and online behavior, social media macroscopes of emotion can substantially agree with established social science methods** --- # Summary - **Computational Social Science and methods validation** - **Questions about social media macroscopes of emotions** - **Validation in the UK with Twitter** - **Validation in Austria with Der Standard** <a href=https://arxiv.org/abs/2107.13236> Social media emotion macroscopes reflect emotional experiences in society at large. David Garcia, Max Pellert, Jana Lasser, Hannah Metzler. (2021) <a href=https://arxiv.org/abs/2108.07646> Validating daily social media macroscopes of emotions. Max Pellert, Hannah Metzler, Michael Matzenberger, David Garcia. Scientific Reports (Forthcoming) .center[**More at: [www.dgarcia.eu](https://dgarcia.eu) and [@dgarcia_eu](https://twitter.com/dgarcia_eu)** #Thanks for listening! ]