Publications

A selection of our most recent and relevant publications is listed below. For the full list of GATE publications, visit the Publications page on the main GATE website.

The Chilling: Global trends in online violence against women journalists. 

Download (PDF, 20.8MB)

5 December 2024

Hostility Detection in UK Politics: A Dataset on Online Abuse Targeting MPs 

Download (PDF, 2.1MB)

Numerous politicians use social media platforms, particularly X, to engage with their constituents. This interaction allows constituents to pose questions and offer feedback but also exposes politicians to a barrage of hostile responses, especially given the anonymity afforded by social media. They are typically targeted in relation to their governmental role, but the comments also tend to attack their personal identity. This can discredit politicians and reduce public trust in the government. It can also incite anger and disrespect, leading to offline harm and violence. While numerous models exist for detecting hostility in general, they lack the specificity required for political contexts. Furthermore, addressing hostility towards politicians demands tailored approaches due to the distinct language and issues inherent to each country (e.g., Brexit for the UK). To bridge this gap, we construct a dataset of 3,320 English tweets spanning a two-year period manually annotated for hostility towards UK MPs. Our dataset also captures the targeted identity characteristics (race, gender, religion, none) in hostile tweets. We perform linguistic and topical analyses to delve into the unique content of the UK political data. Finally, we evaluate the performance of pre-trained language models and large language models on binary hostility detection and multi-class targeted identity type classification tasks. Our study offers valuable data and insights for future research on the prevalence and nature of politics-related hostility specific to the UK. 

18 October 2024

Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media

Download (PDF, 525KB) (preprint)

Misinformation spreads rapidly on social media, confusing the truth and targeting potentially vulnerable people. To effectively mitigate the negative impact of misinformation, it must first be accurately detected before applying a mitigation strategy, such as X’s community notes, which is currently a manual process. This study takes a knowledge-based approach to misinformation detection, modelling the problem similarly to one of natural language inference. The EffiARA annotation framework is introduced, aiming to utilise interand intra-annotator agreement to understand the reliability of each annotator and influence the training of large language models for classification based on annotator reliability. In assessing the EffiARA annotation framework, the Russo-Ukrainian Conflict Knowledge-Based Misinformation Classification Dataset (RUCMCD) was developed and made publicly available. This study finds that sample weighting using annotator reliability performs the best, utilising both inter- and intra-annotator agreement and soft-label training. The highest classification performance achieved using Llama-3.2- 1B was a macro-F1 of 0.757 and 0.740 using TwHIN-BERT-large.

4 October 2024

Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research

Download (PDF, 1.27MB)

Research in natural language processing (NLP) for Computational Social Science (CSS) heavily relies on data from social media platforms. This data plays a crucial role in the development of models for analysing socio-linguistic phenomena within online communities. In this work, we conduct an in-depth examination of 20 datasets extensively used in NLP for CSS to comprehensively examine data quality. Our analysis reveals that social media datasets exhibit varying levels of data duplication. Consequently, this gives rise to challenges like label inconsistencies and data leakage, compromising the reliability of models. Our findings also suggest that data duplication has an impact on the current claims of state-of-the-art performance, potentially leading to an overestimation of model effectiveness in real-world scenarios. Finally, we propose new protocols and best practices for improving dataset development from social media data and its usage. 

8 August 2024

Abuse in the time of COVID-19: the effects of Brexit, gender and partisanship 

Download (PDF, 2.4MB)

Purpose: The authors investigate how COVID-19 has influenced the amount, type or topics of abuse that UK politicians receive when engaging with the public.

Design/methodology/approach: This work covers the first year of COVID-19 in the UK, from March 2020 to March 2021 and analyses Twitter abuse in replies to UK MPs. The authors collected and analysed 17.9 million reply tweets to the MPs. The authors present overall abuse levels during different key moments of the pandemic, analysing reactions to MPs by gender and the relationship between online abuse and topics such as Brexit, the government’s COVID-19 response and policies, and social issues.

Findings: The authors have found that abuse levels towards UK MPs were at an all-time high in December 2020. Women (particularly those from non-White backgrounds) receive unusual amounts of abuse, targeting their credibility and capacity to do their jobs. Similar to other large events like general elections and Brexit, COVID-19 has elevated abuse levels, at least temporarily.

Originality/value: Previous studies analysed abuse levels towards MPs in the run-up to the 2017 and 2019 UK General Elections and during the first four months of the COVID-19 pandemic in the UK. The authors compare previous findings with those of the first year of COVID-19, as the pandemic persisted, and Brexit was forthcoming. This research not only contributes to the longitudinal comparison of abuse trends against UK politicians but also presents new findings, corroborates, further clarifies and raises questions about the previous findings.

Peer review: The peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-07-2022-0392

1 June 2024

Predicting and analyzing the popularity of false rumors in Weibo 

Download (PDF, 3.4MB)

Malicious online rumors with high popularity, if left undetected, can spread very quickly with damaging societal implications. The development of reliable computational methods for early prediction of the popularity of false rumors is very much needed, as a complement to related work on automated rumor detection and fact-checking. Besides, detecting false rumors with higher popularity in the early stage allows social media platforms to timely deliver fact-checking information to end users. To this end, we (1) propose a new regression task to predict the future popularity of false rumors given both post and user-level information; (2) introduce a new publicly available dataset in Chinese that includes 19,256 false rumor cases from Weibo, the corresponding profile information of the original spreaders and a rumor popularity score as a function of the shares, replies and reports it has received; (3) develop a new open-source domain adapted pre-trained language model, i.e., BERT-Weibo-Rumor and evaluate its performance against several supervised classifiers using post and user-level information. Our best performing model (KG-Fusion) achieves the lowest RMSE score (1.54) and highest Pearson’s r (0.636), outperforming competitive baselines by leveraging textual information from both the post and the user profile. Our analysis unveils that popular rumors consist of more conjunctions and punctuation marks, while less popular rumors contain more words related to the social context and personal pronouns. Our dataset is publicly available: https://github.com/YIDAMU/Weibo_Rumor_Popularity. 

June 2024

Sheffieldveraai at semeval-2024 task 4: Prompting and fine-tuning a large vision-language model for binary classification of persuasion techniques in memes 

Download (PDF, 633KB)

This paper describes our approach for SemEval2024 Task 4: Multilingual Detection of Persuasion Techniques in Memes. Specifically, we concentrate on Subtask 2b, a binary classification challenge that entails categorizing memes as either “propagandistic” or “nonpropagandistic”. To address this task, we utilized the large multimodal pretrained model, LLaVa. We explored various prompting strategies and fine-tuning methods, and observed that the model, when not fine-tuned but provided with a few-shot learning examples, achieved the best performance. Additionally, we enhanced the model’s multilingual capabilities by integrating a machine translation model. Our system secured the 2nd place in the Arabic language category 

18 May 2024

Identifying and Aligning Medical Claims Made on Social Media with Medical Evidence 

Download (PDF, 498KB)

Evidence-based medicine is the practice of making medical decisions that adhere to the latest, and best known evidence at that time. Currently, the best evidence is often found in the form of documents, such as randomized control trials, meta-analyses and systematic reviews. This research focuses on aligning medical claims made on social media platforms with this medical evidence. By doing so, individuals without medical expertise can more effectively assess the veracity of such medical claims. We study three core tasks: identifying medical claims, extracting medical vocabulary from these claims, and retrieving evidence relevant to those identified medical claims. We propose a novel system that can generate synthetic medical claims to aid each of these core tasks. We additionally introduce a novel dataset produced by our synthetic generator that, when applied to these tasks, demonstrates not only a more flexible and holistic approach, but also an improvement in all comparable metrics. We make our dataset, the Expansive Medical Claim Corpus (EMCC), available at https://zenodo.org/records/8321460 

13 February 2024

Lying Blindly: Bypassing ChatGPT's Safeguards to Generate Hard-to-Detect Disinformation Claims 

Download (PDF, 598KB) (preprint)

As Large Language Models become more proficient, their misuse in coordinated disinformation campaigns is a growing concern. This study explores the capability of ChatGPT with GPT-3.5 to generate short-form disinformation claims about the war in Ukraine, both in general and on a specific event, which is beyond the GPT-3.5 knowledge cutoff. Unlike prior work, we do not provide the model with human-written disinformation narratives by including them in the prompt. Thus the generated short claims are hallucinations based on prior world knowledge and inference from the minimal prompt. With a straightforward prompting technique, we are able to bypass model safeguards and generate numerous short claims. We compare those against human-authored false claims on the war in Ukraine from ClaimReview, specifically with respect to differences in their linguistic properties. We also evaluate whether AI authorship can be differentiated by human readers or state-of-the-art authorship detection tools. Thus, we demonstrate that ChatGPT can produce realistic, target-specific disinformation claims, even on a specific post-cutoff event, and that they cannot be reliably distinguished by humans or existing automated tools. 

13 February 2024

Generative AI and disinformation: recent advances, challenges, and opportunities 

Download (PDF, 1.21MB)

Over the past three years, generative AI technology (eg DALL-E, ChatGPT) made the sudden leap from research papers and company labs to online services used by hundreds of millions of people, including school children. In the United States alone, 18% of adults had used ChatGPT according to Pew Research in July 2023 (Park & Gelles-Watnick, 2023).

As the fluency and affordability of generative AI continues to increase from one month to the next, so does its wide-ranging misuse for the creation of affordable, highly convincing largescale disinformation campaigns. Highly damaging examples of AI-generated disinformation abound, including highly lucrative Facebook ads1 seeking to influence voters through deepfake videos of Moldova’s pro-Western president (Gilbert, 2024). YouTube has also been found to host ads with political deepfake videos, which used voice imitation (RTL Lëtzebuerg, 2023). Beyond videos, AI-generated images have been used to spread disinformation about Gaza (France, 2023; Totth, 2023) and propagate divisive, anti-immigrant narratives (The Journal, 2023). Audio deepfakes have also been reported by fact-checkers, so far, these have mostly focused on fake conversations and statements by politicians (Demagog, 2023; Dobreva, 2023; Bossev, 2023). Russian disinformation campaigns have also weaponised generative AI (eg a deep fake video of the Ukrainian president calling for surrender ( Kinsella, 2023), an AI-generated conversation between the Ukrainian president and his wife (Demagog, 2023).

13 December 2023

UTDRM: unsupervised method for training debunked-narrative retrieval models 

Download (PDF, 2.1MB)

A key task in the fact-checking workflow is to establish whether the claim under investigation has already been debunked or fact-checked before. This is essentially a retrieval task where a misinformation claim is used as a query to retrieve from a corpus of debunks. Prior debunk retrieval methods have typically been trained on annotated pairs of misinformation claims and debunks. The novelty of this paper is an Unsupervised Method for Training Debunked-Narrative Retrieval Models (UTDRM) in a zero-shot setting, eliminating the need for human-annotated pairs. This approach leverages fact-checking articles for the generation of synthetic claims and employs a neural retrieval model for training. Our experiments show that UTDRM tends to match or exceed the performance of state-of-the-art methods on seven datasets, which demonstrates its effectiveness and broad applicability. The paper also analyses the impact of various factors on UTDRM’s performance, such as the quantity of fact-checking articles utilised, the number of synthetically generated claims employed, the proposed entity inoculation method, and the usage of large language models for retrieval. 

6 November 2023

Dimensions of Online Conflict: Towards Modeling Agonism

Download (PDF, 823KB)

Agonism plays a vital role in democratic dialogue by fostering diverse perspectives and robust discussions. Within the realm of online conflict there is another type: hateful antagonism, which undermines constructive dialogue. Detecting conflict online is central to platform moderation and monetization. It is also vital for democratic dialogue, but only when it takes the form of agonism. To model these two types of conflict, we collected Twitter conversations related to trending controversial topics. We introduce a comprehensive annotation schema for labelling different dimensions of conflict in the conversations, such as the source of conflict, the target, and the rhetorical strategies deployed. Using this schema, we annotated approximately 4,000 conversations with multiple labels. We then train both logistic regression and transformer-based models on the dataset, incorporating context from the conversation, including the number of participants and the structure of the interactions. Results show that contextual labels are helpful in identifying conflict and make the models robust to variations in topic. Our research contributes a conceptualization of different dimensions of conflict, a richly annotated dataset, and promising results that can contribute to content moderation. 

1 November 2023

Similarity-Aware Multimodal Prompt Learning for fake news detection

Download (PDF, 10.9MB) (preprint)

Online fake news explosion has posed significant challenges to academics and industries by overloading fact-checkers and social media. The standard paradigm for fake news detection relies on utilizing text information to model news’ truthfulness. However, the subtle nature of online fake news makes it challenging to use textual information alone to debunk it. Recent studies, focusing on multimodal fake news detection, have outperformed text-only methods. Deep learning approaches, primarily utilizing pre-trained models, to extract unimodal features, or finetuning the pre-trained model, has become a new paradigm for detecting fake news. Nevertheless, this paradigm may require a large number of training instances or updating the entire set of pretrained model parameters, making it impractical for real-world fake news detection. In addition, traditional multimodal methods directly fuse the cross-modal features without considering that the uncorrelated semantic representation may introduce noise into the multimodal features. To address these issues, this paper proposed the Similarity-Aware Multimodal Prompt Learning (SAMPLE) framework. Incorporating prompt learning into multimodal fake news detection, we used three prompt templates with a soft verbalizer to detect fake news. Additionally, we introduced the similarity-aware fusing method, which adaptively fuses the intensity of multimodal representation and mitigates noise injection via uncorrelated cross-modal features. Evaluation results show that SAMPLE outperformed previous works by achieving higher F1 and accuracy scores on two benchmark multimodal datasets, demonstrating its feasibility in realworld scenarios, regardless of data-rich or few-shot settings. 

21 October 2023

Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study

Download (PDF, 283KB)

This paper analyses two hitherto unstudied sites sharing state-backed disinformation, Reliable Recent News (rrn.world) and WarOnFakes (waronfakes.com), which publish content in Arabic, Chinese, English, French, German, and Spanish. We describe our content acquisition methodology and perform cross-site unsupervised topic clustering on the resulting multilingual dataset. We also perform linguistic and temporal analysis of the web page translations and topics over time, and investigate articles with false publication dates. We make publicly available this new dataset of 14,053 articles, annotated with each language version, and additional metadata such as links and images. The main contribution of this paper for the NLP community is in the novel dataset which enables studies of disinformation networks, and the training of NLP tools for disinformation detection. 

25 September 2023

Examining Temporal Bias in Abusive Language Detection

Download (PDF, 528KB) (preprint)

The use of abusive language online has become an increasingly pervasive problem that damages both individuals and society, with effects ranging from psychological harm right through to escalation to real-life violence and even death. Machine learning models have been developed to automatically detect abusive language, but these models can suffer from temporal bias, the phenomenon in which topics, language use or social norms change over time. This study aims to investigate the nature and impact of temporal bias in abusive language detection across various languages and explore mitigation methods. We evaluate the performance of models on abusive data sets from different time periods. Our results demonstrate that temporal bias is a significant challenge for abusive language detection, with models trained on historical data showing a significant drop in performance over time. We also present an extensive linguistic analysis of these abusive data sets from a diachronic perspective, aiming to explore the reasons for language evolution and performance decline. This study sheds light on the pervasive issue of temporal bias in abusive language detection across languages, offering crucial insights into language evolution and temporal bias mitigation. 

11 September 2023

Evaluating VLOP and VLOSE Implementation of the Strengthened EU Code of Practice on Disinformation in Bulgaria 

Download (PDF, 1.57MB)

In June of 2022, Google, Meta, Microsoft, TikTok, Twitter (rebranded as X) and a selection of advertising industry companies all signed up to the strengthened Code of Practice on Disinformation (European Commission, 2022). One of the goals of this strengthened version of the code was to empower the industry to adhere to self-regulatory standards in order to combat disinformation. The strengthened code also claims to set a more ambitious set of commitments and measures aimed at combating disinformation online. 

10 August 2023

Breaking Language Barriers with MMTweets: Advancing Cross-Lingual Debunked Narrative Retrieval for Fact-Checking

Download (PDF, 2.2MB)

Finding previously debunked narratives involves identifying claims that have already undergone fact-checking. The issue intensifies when similar false claims persist in multiple languages, despite the availability of debunks for several months in another language. Hence, automatically finding debunks (or fact-checks) in multiple languages is crucial to make the best use of scarce fact-checkers' resources. Mainly due to the lack of readily available data, this is an understudied problem, particularly when considering the cross-lingual scenario, i.e. the retrieval of debunks in a language different from the language of the online post being checked. This study introduces cross-lingual debunked narrative retrieval and addresses this research gap by: (i) creating Multilingual Misinformation Tweets (MMTweets): a dataset that stands out, featuring cross-lingual pairs, images, human annotations, and fine-grained labels, making it a comprehensive resource compared to its counterparts; (ii) conducting an extensive experiment to benchmark state-of-the-art cross-lingual retrieval models and introducing multistage retrieval methods tailored for the task; and (iii) comprehensively evaluating retrieval models for their cross-lingual and cross-dataset transfer capabilities within MMTweets, and conducting a retrieval latency analysis. We find that MMTweets presents challenges for cross-lingual debunked narrative retrieval, highlighting areas for improvement in retrieval models. Nonetheless, the study provides valuable insights for creating MMTweets datasets and optimising debunked narrative retrieval models to empower fact-checking endeavours. The dataset and annotation codebook are publicly available at this https URL. 

10 August 2023

Finding already debunked narratives via multistage retrieval: enabling cross-lingual, cross-dataset and zero-shot learning.

Download (PDF, 704KB) (preprint)

The task of retrieving already debunked narratives aims to detect stories that have already been fact-checked. The successful detection of claims that have already been debunked not only reduces the manual efforts of professional fact-checkers but can also contribute to slowing the spread of misinformation. Mainly due to the lack of readily available data, this is an understudied problem, particularly when considering the cross-lingual task, i.e. the retrieval of fact-checking articles in a language different from the language of the online post being checked. This paper fills this gap by (i) creating a novel dataset to enable research on cross-lingual retrieval of already debunked narratives, using tweets as queries to a database of fact-checking articles; (ii) presenting an extensive experiment to benchmark fine-tuned and off-the-shelf multilingual pre-trained Transformer models for this task; and (iii) proposing a novel multistage framework that divides this cross-lingual debunk retrieval task into refinement and re-ranking stages. Results show that the task of cross-lingual retrieval of already debunked narratives is challenging and off-the-shelf Transformer models fail to outperform a strong lexical-based baseline (BM25). Nevertheless, our multistage retrieval framework is robust, outperforming BM25 in most scenarios and enabling cross-domain and zero-shot learning, without significantly harming the model's performance. 

31 July 2023

Noisy Self-Training with Data Augmentations for Offensive and Hate Speech Detection Tasks

Download (PDF, 268KB)

Online social media is rife with offensive and hateful comments, prompting the need for their automatic detection given the sheer amount of posts created every second. Creating high-quality human-labelled datasets for this task is difficult and costly, especially because non-offensive posts are significantly more frequent than offensive ones. However, unlabelled data is abundant, easier, and cheaper to obtain. In this scenario, self-training methods, using weakly-labelled examples to increase the amount of training data, can be employed. Recent “noisy” self-training approaches incorporate data augmentation techniques to ensure prediction consistency and increase robustness against noisy data and adversarial attacks. In this paper, we experiment with default and noisy self-training using three different textual data augmentation techniques across five different pre-trained BERT architectures varying in size. We evaluate our experiments on two offensive/hate-speech datasets and demonstrate that (i) self-training consistently improves performance regardless of model size, resulting in up to +1.5% F1-macro on both datasets, and (ii) noisy self-training with textual data augmentations, despite being successfully applied in similar settings, decreases performance on offensive and hate-speech domains when compared to the default method, even with state-of-the-art augmentations such as backtranslation. 

July 2023

SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification

Download (PDF, 240KB)

This paper describes our approach for SemEval2023 Task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. For Subtask 1 (News Genre), we propose an ensemble of fully trained and adapter mBERT models which was ranked joint-first for German, and had the highest mean rank of multi-language teams. For Subtask 2 (Framing), we achieved first place in 3 languages, and the best average rank across all the languages, by using two separate ensembles: a monolingual RoBERTa-MUPPETLARGE and an ensemble of XLM-RoBERTaLARGE with adapters and task adaptive pretraining. For Subtask 3 (Persuasion Techniques), we trained a monolingual RoBERTa-Base model for English and a multilingual mBERT model for the remaining languages, which achieved top 10 for all languages, including 2nd for English. For each subtask, we compared monolingual and multilingual approaches, and considered class imbalance techniques. 

2 June 2023

VaxxHesitancy: A Dataset for Studying Hesitancy towards COVID-19 Vaccination on Twitter

Download (PDF, 1.3MB)

Vaccine hesitancy has been a common concern, probably since vaccines were created and, with the popularisation of social media, people started to express their concerns about vaccines online alongside those posting pro- and anti-vaccine content. Predictably, since the first mentions of a COVID-19 vaccine, social media users posted about their fears and concerns or about their support and belief into the effectiveness of these rapidly developing vaccines. Identifying and understanding the reasons behind public hesitancy towards COVID-19 vaccines is important for policy markers that need to develop actions to better inform the population with the aim of increasing vaccine take-up. In the case of COVID-19, where the fast development of the vaccines was mirrored closely by growth in anti-vaxx disinformation, automatic means of detecting citizen attitudes towards vaccination became necessary. This is an important computational social sciences task that requires data analysis in order to gain in-depth understanding of the phenomena at hand. Annotated data is also necessary for training data-driven models for more nuanced analysis of attitudes towards vaccination. To this end, we created a new collection of over 3,101 tweets annotated with users' attitudes towards COVID-19 vaccination (stance). Besides, we also develop a domain-specific language model (VaxxBERT) that achieves the best predictive performance (73.0 accuracy and 69.3 F1-score) as compared to a robust set of baselines. To the best of our knowledge, these are the first dataset and model that model vaccine hesitancy as a category distinct from pro- and anti-vaccine stance. 

10 April 2023

A large-scale comparative study of accurate COVID-19 information versus misinformation 

Download (PDF, 562KB)

The COVID-19 pandemic led to an infodemic where an overwhelming amount of COVID-19 related content was being disseminated at high velocity through social media. This made it challenging for citizens to differentiate between accurate and inaccurate information about COVID-19. This motivated us to carry out a comparative study of the characteristics of COVID-19 misinformation versus those of accurate COVID-19 information through a large-scale computational analysis of over 242 million tweets. The study makes comparisons alongside four key aspects: 1) the distribution of topics, 2) the live status of tweets, 3) language analysis and 4) the spreading power over time. An added contribution of this study is the creation of a COVID-19 misinformation classification dataset. Finally, we demonstrate that this new dataset helps improve misinformation classification by more than 9% based on average F1 measure. 

10 April 2023

Examining temporalities on stance detection towards covid-19 vaccination

Download (PDF, 182KB)

Previous studies have highlighted the importance of vaccination as an effective strategy to control the transmission of the COVID-19 virus. It is crucial for policymakers to have a comprehensive understanding of the public's stance towards vaccination on a large scale. However, attitudes towards COVID-19 vaccination, such as pro-vaccine or vaccine hesitancy, have evolved over time on social media. Thus, it is necessary to account for possible temporal shifts when analysing these stances. This study aims to examine the impact of temporal concept drift on stance detection towards COVID-19 vaccination on Twitter. To this end, we evaluate a range of transformer-based models using chronological (splitting the training, validation, and test sets in order of time) and random splits (randomly splitting these three sets) of social media data. Our findings reveal significant discrepancies in model performance between random and chronological splits in several existing COVID-19-related datasets; specifically, chronological splits significantly reduce the accuracy of stance classification. Therefore, real-world stance detection approaches need to be further refined to incorporate temporal factors as a key consideration. 

10 April 2023

A large-scale comparative study of accurate COVID-19 information versus misinformation

Download (PDF, 562KB)

The COVID-19 pandemic led to an infodemic where an overwhelming amount of COVID-19 related content was being disseminated at high velocity through social media. This made it challenging for citizens to differentiate between accurate and inaccurate information about COVID-19. This motivated us to carry out a comparative study of the characteristics of COVID-19 misinformation versus those of accurate COVID-19 information through a large-scale computational analysis of over 242 million tweets. The study makes comparisons alongside four key aspects: 1) the distribution of topics, 2) the live status of tweets, 3) language analysis and 4) the spreading power over time. An added contribution of this study is the creation of a COVID-19 misinformation classification dataset. Finally, we demonstrate that this new dataset helps improve misinformation classification by more than 9% based on average F1 measure. 

9 April 2023

Similarity-Aware Multimodal Prompt Learning for fake news detection

Download (PDF, 10.9MB)

The standard paradigm for fake news detection relies on utilizing text information to model the truthfulness of news. However, the subtle nature of online fake news makes it challenging to solely rely on textual information for debunking. Recent studies that focus on multimodal fake news detection have demonstrated superior performance compared with text-only methods, thereby establishing a new paradigm for detecting fake news. However, this paradigm may require a large number of training instances or updating the entire set of pre-trained model parameters. Furthermore, existing multimodal approaches typically integrate cross-modal features without considering the potential introduction of noise from unrelated semantic representations. To address these issues, this paper proposes the Similarity-Aware Multimodal Prompt Learning (SAMPLE) framework. Incorporating prompt learning into multimodal fake news detection, we used three prompt templates with a soft verbalizer to detect fake news. Moreover, we introduced a similarity-aware fusing method, which adaptively fuses the intensity of multimodal representation so as to mitigate noise injection from uncorrelated cross-modal features. Evaluation results show that SAMPLE outperformed previous work, achieving higher F1 and accuracy scores on two multimodal benchmark datasets, demonstrating its feasibility in real-world scenarios, regardless of data-rich or few-shot settings. 

18 July 2022

Classifying COVID-19 vaccine narratives

Download (PDF, 288KB)

Vaccine hesitancy is widespread, despite the government’s information campaigns and the efforts of the World Health Organisation (WHO). Categorising the topics within vaccine related narratives is crucial to understand the concerns expressed in discussions and identify the specific issues that contribute to vaccine hesitancy. This paper addresses the need for monitoring and analysing vaccine narratives online by introducing a novel vaccine narrative classification task, which categorises COVID-19 vaccine claims into one of seven categories. Following a data augmentation approach, we first construct a novel dataset for this new classification task, focusing on the minority classes. We also make use of fact-checker annotated data. The paper also presents a neural vaccine narrative classifier that achieves an accuracy of 84% under cross-validation. The classifier is publicly available for researchers and journalists. 

12 October 2022

Comparative analysis of engagement, themes, and causality of Ukraine-related debunks and disinformation 

Download (PDF, 2.9MB)

This paper compares quantitatively the spread of Ukraine-related disinformation and its corresponding debunks, first by considering re-tweets, replies, and favourites, which demonstrate that despite platform efforts Ukraine-related disinformation is still spreading wider than its debunks. Next, bidirectional post-hoc analysis is carried out using Granger causality tests, impulse response analysis and forecast error variance decomposition, which demonstrate that the spread of debunks has a positive impact on reducing Ukraine-related disinformation eventually, albeit not instantly. Lastly, the paper investigates the dominant themes in Ukraine-related disinformation and their spatiotemporal distribution. With respect to debunks, we also establish that around 18% of fact-checks are debunking claims which have already been fact-checked in another language. The latter finding highlights an opportunity for better collaboration between fact-checkers, so they can benefit from and amplify each other’s debunks through translation, citation, and early publication online. 

7 October 2021

Cross-lingual Rumour Stance Classification: a First Study with BERT and Machine Translation. 

Download (PDF, 935KB)

Social media tend to be rife with rumours, which often have such high velocity and volume that fact-checkers struggle with debunking them with traditional methods. Prior research on English rumours has demonstrated that one can analyse the reactions (i.e. stance) expressed by social media users towards rumours, which ultimately enables automated flagging to journalists highly disputed rumours. This paper presents the first study of cross-lingual rumour stance classification. Through experiments with zero- and few-shot learning and in three languages (German, Danish and Russian), we show that models trained on English data can be used successfully for predicting stance in other languages. In the few-shot case, we also show that only few data points in the target language are needed to achieve the best results. In a multilingual setting, results for English are also further improved. Our results highlight the potential of multilingual BERT and machine translation for rumour analysis in languages where annotated data is scarce or not readily available. 

Categorising fine-to-coarse grained misinformation: An empirical study of COVID-19 infodemic

Download (PDF, 881KB)

The spreading COVID-19 misinformation over social media already draws the attention of many researchers. According to Google Scholar, about 26,000 COVID-19 related misinformation studies have been published to date. Most of these studies focusing on detecting and/or analysing the characteristics of COVID-19 related misinformation. However, the study of the social behaviours related to misinformation is often neglected.

In this paper, we introduce a fine-grained annotated misinformation tweets dataset including social behaviours annotation (eg comment or question to the misinformation). The dataset not only allows social behaviours analysis but also suitable for both evidence-based or non-evidence-based misinformation classification task. In addition, we introduce leave claim out validation in our experiments and demonstrate the misinformation classification performance could be significantly different when applying to real-world unseen misinformation.

Infodemic: disinformation and media literacy in the context of COVID-19

Download (PDF, 4.4MB)

The World Health Organization (WHO) has described the disinformation swirling amidst the COVID-19 pandemic as a "massive infodemic" – a major driver of the pandemic itself. Disinformation long predates COVID-19. The fabrications that contaminate public health information today rely on the same dissemination tools traditionally used to distribute disinformation. What's novel are the themes and their very direct impacts. COVID-19 disinformation creates confusion about medical science with an immediate impact on every person on the planet, and upon whole societies. It is more toxic and more deadly than disinformation about other subjects. That is why this article coins the term disinfodemic.

The false COVID-19 narratives that keep being debunked: A spatiotemporal analysis

Download (PDF, 3.6MB)

The onset of the Coronavirus disease 2019 (COVID-19) pandemic instigated a global infodemic that has brought unprecedented challenges for society as a whole. During this time, a number of manual fact-checking initiatives have emerged to alleviate the spread of dis/mis-information. This study is about COVID-19 debunks published in multiple languages by different fact-checking organisations, sometimes as far as several months apart, despite the fact that the claim has already been fact-checked before.

The spatiotemporal analysis reveals that similar or nearly duplicate false COVID-19 narratives have been spreading in multifarious modalities on various social media platforms in different countries. We also find that misinformation involving general medical advice has spread across multiple countries and hence has the highest proportion of false COVID-19 narratives that keep being debunked.

Furthermore, as manual fact-checking is an onerous task in itself, therefore debunking similar claims recurrently is leading to a waste of resources. To this end, we propound the idea of the inclusion of multilingual debunk search in the fact-checking pipeline.

MP Twitter engagement and abuse post-first COVID-19 lockdown in the UK: White paper

Download (PDF, 571KB)

The UK has had a volatile political environment for some years now, with Brexit and leadership crises marking the past five years. With this work, we wanted to understand more about how the global health emergency, COVID-19, influences the amount, type or topics of abuse that UK politicians receive when engaging with the public. This work covers the period of June to December 2020 and analyses Twitter abuse in replies to UK MPs. This work is a follow-up from our analysis of online abuse during the first four months of the COVID-19 pandemic in the UK.

The paper examines overall abuse levels during this new seven month period, analyses reactions to members of different political parties and the UK government, and the relationship between online abuse and topics such as Brexit, government's COVID-19 response and policies, and social issues. In addition, we have also examined the presence of conspiracy theories posted in abusive replies to MPs during the period.

We have found that abuse levels toward UK MPs were at an all-time high in December 2020 (5.4% of all reply tweets sent to MPs). This is almost 1% higher than the two months preceding the general election. In a departure from the trend seen in the first four months of the pandemic, MPs from the Tory party received the highest percentage of abusive replies from July 2020 onward, which stays above 5% starting from September 2020 onward, as the COVID-19 crisis deepened and the Brexit negotiations with the EU started nearing completion.

Classification aware neural topic model for COVID-19 disinformation categorisation

Download (PDF, 1.5MB)

The explosion of disinformation accompanying the COVID-19 pandemic has overloaded fact-checkers and media worldwide, and brought a new major challenge to government responses worldwide. Not only is disinformation creating confusion about medical science amongst citizens, but it is also amplifying distrust in policy makers and governments. To help tackle this, we developed computational methods to categorise COVID-19 disinformation. The COVID-19 disinformation categories could be used for

• focusing fact-checking efforts on the most damaging kinds of COVID-19 disinformation

• guiding policy makers who are trying to deliver effective public health messages and counter effectively COVID-19 disinformation.

This paper presents:

1. A corpus containing what is currently the largest available set of manually annotated COVID-19 disinformation categories.

2. A classification-aware neural topic model (CANTM) designed for COVID-19 disinformation category classification and topic discovery

3. An extensive analysis of COVID-19 disinformation categories with respect to time, volume, false type, media type and origin source.

Which politicians receive abuse? Four factors illuminated in the UK general election 2019

Download (PDF, 3MB)

The 2019 UK general election took place against a background of rising online hostility levels toward politicians, and concerns about the impact of this on democracy, as a record number of politicians cited the abuse they had been receiving as a reason for not standing for re-election. We present a four-factor framework in understanding who receives online abuse and why. The four factors are prominence, events, online engagement and personal characteristics.

Research materials relating to the work

Toxic language detection in social media for Brazilian Portuguese: New dataset and multilingual analysis

Download (PDF, 355KB)

Hate speech and toxic comments are a common concern of social media platform users. Although these comments are, fortunately, the minority in these platforms, they are still capable of causing harm. Therefore, identifying these comments is an important task for studying and preventing the proliferation of toxicity in social media. Previous work in automatically detecting toxic comments focus mainly in English, with very few work in languages like Brazilian Portuguese.

In this paper, we propose a new large-scale dataset for Brazilian Portuguese with tweets annotated as either toxic or non-toxic or in different types of toxicity. We present our dataset collection and annotation process, where we aimed to select candidates covering multiple demographic groups. State-of-the-art BERT models were able to achieve 76% macro-F1 score using monolingual data in the binary case. We also show that large-scale monolingual data is still needed to create more accurate models, despite recent advances in multilingual approaches.

An error analysis and experiments with multi-label classification show the difficulty of classifying certain types of toxic comments that appear less frequently in our data and highlights the need to develop models that are aware of different categories of toxicity.

Revisiting rumour stance classification: Dealing with imbalanced data

Download (PDF, 544KB)

Correctly classifying stances of replies can be significantly helpful for the automatic detection and classification of online rumours. One major challenge is that there are considerably more non-relevant replies (comments) than informative ones (supports and denies), making the task highly imbalanced. In this paper we revisit the task of rumour stance classification, aiming to improve the performance over the informative minority classes. We experiment with traditional methods for imbalanced data treatment with feature-and BERT-based classifiers. Our models outperform all systems in RumourEval 2017 shared task and rank second in RumourEval 2019.

Measuring the impact of readability features in fake news detection

Download (PDF, 421KB)

The proliferation of fake news is a current issue that influences a number of important areas of society, such as politics, economy and health. In the natural language processing area, recent initiatives tried to detect fake news in different ways, ranging from language-based approaches to content-based verification. In such approaches, the choice of the features for the classification of fake and true news is one of the most important parts of the process.

This paper presents a study on the impact of readability features to detect fake news for the Brazilian Portuguese language. The results show that such features are relevant to the task (achieving, alone, up to 92% classification accuracy) and may improve previous classification results.

Vindication, virtue, and vitriol

Download (PDF, 1.7MB)

COVID-19 has given rise to a lot of malicious content online, including hate speech, online abuse, and misinformation. British MPs have also received abuse and hate on social media during this time. To understand and contextualise the level of abuse MPs receive, we consider how ministers use social media to communicate about the pandemic, and the citizen engagement that this generates.

The focus of the paper is on a large-scale, mixed-methods study of abusive and antagonistic responses to UK politicians on Twitter, during the pandemic from early February to late May 2020. We find that pressing subjects such as financial concerns attract high levels of engagement, but not necessarily abusive dialogue. Rather, criticising authorities appears to attract higher levels of abuse during this period of the pandemic. In addition, communicating about subjects like racism and inequality may result in accusations of virtue signalling or pandering by some users. This work contributes to the wider understanding of abusive language online, in particular that which is directed at public officials.

MP Twitter abuse in the age of COVID-19: White paper

Download (PDF, 1.1MB)

As COVID-19 sweeps the globe, outcomes depend on effective relationships between the public and decision-makers. In the UK there were uncivil tweets to MPs about perceived UK tardiness to go into lockdown. The pandemic has led to increased attention on ministers with a role in the crisis. However, generally this surge has been civil. Prime minister Boris Johnson's severe illness with COVID-19 resulted in an unusual peak of supportive responses on Twitter. Those who receive more COVID-19 mentions in their replies tend to receive less abuse (significant negative correlation).

Following Mr Johnson's recovery, with rising economic concerns and anger about lockdown violations by influential figures, abuse levels began to rise in May. 1,902 replies to MPs within the study period were found containing hashtags or terms that refute the existence of the virus (eg #coronahoax, #coronabollocks, 0.04% of a total 4.7 million replies, or 9% of the number of mentions of "stay home save lives" and variants). These have tended to be more abusive. Evidence of some members of the public believing in COVID-19 conspiracy theories was also found. Higher abuse levels were associated with hashtags blaming China for the pandemic.

Partisanship, propaganda and post-truth politics: Quantifying impact in online debate

Download (PDF, 1.8MB)

The recent past has highlighted the influential role of social networks and online media in shaping public debate on current affairs and political issues. This paper is focused on studying the role of politically-motivated actors and their strategies for influencing and manipulating public opinion online: partisan media, state-backed propaganda, and post-truth politics. In particular, we present quantitative research on the presence and impact of these three "Ps" in online Twitter debates in two contexts:

1. The run up to the UK EU membership referendum ("Brexit").

2. The information operations of Russia-backed online troll accounts.

We first compare the impact of highly partisan versus mainstream media during the Brexit referendum, specifically comparing tweets by half a million "leave" and "remain" supporters. Next, online propaganda strategies are examined, specifically left- and right-wing troll accounts. Lastly, we study the impact of misleading claims made by the political leaders of the leave and remain campaigns. This is then compared to the impact of the Russia-backed partisan media and propaganda accounts during the referendum.

In particular, just two of the many misleading claims made by politicians during the referendum were found to be cited in 4.6 times more tweets than the 7,103 tweets related to Russia Today and Sputnik and in 10.2 times more tweets than the 3,200 Brexit-related tweets by the Russian troll accounts.

WeVerify: Wider and enhanced verification for you – project overview and tools

Download (PDF, 2.3MB)

This paper presents an overview of the WeVerify H2020 EU project, which develops intelligent human-in-the-loop content verification and disinformation analysis methods, tools and services. Social media and web content are analysed and contextualised within the broader online ecosystem, in order to expose fabricated content, through cross-modal content verification, social network analysis, micro-targeted debunking, and a blockchain-based public database of known fakes.

Journalist-in-the-loop: Continuous learning as a service for rumour analysis

Download (PDF, 421KB)

Automatically identifying rumours in social media and assessing their veracity is an important task with downstream applications in journalism. A significant challenge is how to keep rumour analysis tools up-to-date as new information becomes available for particular rumours that spread in a social network.

This paper presents a novel open-source web-based rumour analysis tool that can continuous learn from journalists. The system features a rumour annotation service that allows journalists to easily provide feedback for a given social media post through a web-based interface. The feedback allows the system to improve an underlying state-of-the-art neural network-based rumour classification model. The system can be easily integrated as a service into existing tools and platforms used by journalists using a REST API.

Rumour verification through recurring information and an inner-attention mechanism

Download (PDF, 918KB)

Verification of online rumours is becoming an increasingly important task with the prevalence of event discussions on social media platforms. This paper proposes an inner-attention-based neural network model that uses frequent, recurring terms from past rumours to classify a newly emerging rumour as true, false or unverified. Unlike other methods proposed in related work, our model uses the source rumour alone without any additional information, such as user replies to the rumour or additional feature engineering.

Our method outperforms the current state-of-the-art methods on benchmark datasets (RumourEval2017) by 3% accuracy and 6% F-1 leading to 60.7% accuracy and 61.6% F-1. We also compare our attention-based method to two similar models which however do not make use of recurrent terms. The attention-based method guided by frequent recurring terms outperforms this baseline on the same dataset, indicating that the recurring terms injected by the attention mechanism have high positive impact on distinguishing between true and false rumours.

Furthermore, we perform out-of-domain evaluations and show that our model is indeed highly competitive compared to the baselines on a newly released RumourEval2019 dataset and also achieves the best performance on classifying fake and legitimate news headlines.

Automated tackling of disinformation: Major challenges ahead

Download (PDF, 2.4MB)

This study maps and analyses current and future threats from online misinformation, alongside currently adopted socio-technical and legal approaches. The challenges of evaluating their effectiveness and practical adoption are also discussed. Drawing on and complementing existing literature, the study summarises and analyses the findings of relevant journalist and scientific studies and policy reports in relation to detecting, containing and countering online disinformation and propaganda campaigns. It traces recent development and trends and identifies significant new or emerging challenges. It also addresses potential policy implications of current socio-technical solutions for the EU.

Predicting news source credibility

Download (PDF, 222KB)

Assessing the credibility of a source of information is important in combating with misinformation. In this work we tackle the source credibility assessment as regression task. For this purpose we release a dataset containing around 700 news sources along with detailed credibility and transparency scores. These scores are manually assigned to every news source. We merge these scores to have final credibility score for every news source. The merged scores are then used to train prediction models.

Our results show highly satisfactory performances in predicting the merged credibility scores. Along with the dataset we also plan to release our models to allow the use for a wider community.

Credibility and transparency of news sources: Data collection and feature analysis

Download (PDF, 224KB)

The ability to discern news sources based on their credibility and transparency is useful for users in making decisions about news consumption. In this paper, we release a dataset of 673 sources with credibility and transparency scores manually assigned. Upon acceptance we will make this dataset publicly available. Furthermore, we compared features which can be computed automatically and measured their correlation with credibility and transparency scores annotated by human experts. Our correlation analysis shows that there are indeed features which highly correlate with the manual judgments.

Quantifying media influence and partisan attention on Twitter during the UK EU Referendum

Download (PDF, 380KB)

User generated media, and their influence on the information individuals are exposed to, have the potential to affect political outcomes. This is increasingly a focus for attention and concern. The British EU membership referendum provided an opportunity for researchers to explore the nature and impact of the new infosphere in a politically charged situation. This work contributes by reviewing websites that were linked in a Brexit tweet dataset of 13.2 million tweets, by 1.8 million distinct users, collected in the run-up to the referendum.

Research materials relating to the work (ODS, 1.1MB)

Twits, twats and twaddle: Trends in online abuse towards UK politicians

Download (PDF, 186KB)

Concerns have reached the mainstream about how social media are affecting political outcomes. One trajectory for this is the exposure of politicians to online abuse. In this paper we use 1.4 million tweets from the months before the 2015 and 2017 UK general elections to explore the abuse directed at politicians. Results show that abuse increased substantially in 2017 compared with 2015.

Abusive tweets show a strong relationship with total tweets received, indicating for the most part impersonality, but a second pathway targets less prominent individuals, suggesting different kinds of abuse. Accounts that send abuse are more likely to be throwaway. Economy and immigration were major foci of abusive tweets in 2015, whereas terrorism came to the fore in 2017.

Gazetteer of abusive terms used in the work (TXT, 12KB)

What matters most to people around the world? Retrieving Better Life Index priorities on Twitter

Download (PDF, 591KB)

Better Life Index (BLI), the measure of wellbeing proposed by the OECD, contains many metrics, which enable it to include a detailed overview of the social, economic, and environmental performances of different countries. However, this also increases the difficulty in evaluating the big picture. In order to overcome this, many composite BLI procedures have been proposed, but none of them takes into account societal priorities in the aggregation. One of the reasons for this is that at the moment there is no representative survey about the relative priorities of the BLI topics for each country. Using these priorities could help to design Composite Indices that better reflect the needs of the people.

The largest collection of information about society is found in social media such as Twitter. This paper proposes a composite BLI based on the weighted average of the national performances in each dimension of the BLI, using the relative importance that the topics have on Twitter as weights. The idea is that the aggregate of millions of tweets may provide a representation of the priorities (the relative appreciations) among the eleven topics of the BLI, both at a general level and at a country-specific level. By combining topic performances and related Twitter trends, we produce new evidences about the relations between people's priorities and policy makers' activity in the BLI framework.

RumourEval 2019: Determining rumour veracity and support for rumours

Download (PDF, 128KB)

This is the proposal for RumourEval-2019, which will run in early 2019 as part of that year's SemEval event. Since the first RumourEval shared task in 2017, interest in automated claim validation has greatly increased, as the dangers of "fake news" have become a mainstream concern. Yet automated support for rumour checking remains in its infancy. For this reason, it is important that a shared task in this area continues to provide a focus for effort, which is likely to increase.

We therefore propose a continuation in which the veracity of further rumours is determined, and as previously, supportive of this goal, tweets discussing them are classified according to the stance they take regarding the rumour. Scope is extended compared with the first RumourEval, in that the dataset is substantially expanded to include Reddit as well as Twitter data, and additional languages are also included.

Can rumour stance alone predict veracity?

Download (PDF, 230KB)

Prior manual studies of rumours suggested that crowd stance can give insights into the actual rumour veracity. Even though numerous studies of automatic veracity classification of social media rumours have been carried out, none explored the effectiveness of leveraging crowd stance to determine veracity. We use stance as an additional feature to those commonly used in earlier studies. We also model the veracity of a rumour using variants of Hidden Markov Models (HMM) and the collective stance information.

This paper demonstrates that HMMs that use stance and tweets' times as the only features for modelling true and false rumours achieve F1 scores in the range of 80%, outperforming those approaches where stance is used jointly with content and user based features.

Detection and resolution of rumours in social media: A survey

Download (PDF, 505KB)

Despite the increasing use of social media platforms for information and news gathering, its unmoderated nature often leads to the emergence and spread of rumours, ie items of information that are unverified at the time of posting. At the same time, the openness of social media platforms provides opportunities to study how users share and discuss rumours, and to explore how to automatically assess their veracity, using natural language processing and data mining techniques.

In this article, we introduce and discuss two types of rumours that circulate on social media: long-standing rumours that circulate for long periods of time, and newly emerging rumours spawned during fast-paced events such as breaking news, where reports are released piecemeal and often with an unverified status in their early stages. We provide an overview of research into social media rumours with the ultimate goal of developing a rumour classification system that consists of four components: rumour detection, rumour tracking, rumour stance classification, and rumour veracity classification.

We delve into the approaches presented in the scientific literature for the development of each of these four components. We summarise the efforts and achievements so far toward the development of rumour classification systems and conclude with suggestions for avenues for future research in social media mining for the detection and resolution of rumours.

Discourse-aware rumour stance classification in social media using sequential classifiers

Download (PDF, 917KB)

Rumour stance classification, defined as classifying the stance of specific social media posts into one of supporting, denying, querying or commenting on an earlier post, is becoming of increasing interest to researchers. While most previous work has focused on using individual tweets as classifier inputs, here we report on the performance of sequential classifiers that exploit the discourse features inherent in social media interactions or "conversational threads".

Testing the effectiveness of four sequential classifiers – Hawkes Processes, Linear-Chain Conditional Random Fields (Linear CRF), Tree-Structured Conditional Random Fields (Tree CRF) and Long Short Term Memory networks (LSTM) – on eight datasets associated with breaking news stories, and looking at different types of local and contextual features, our work sheds new light on the development of accurate stance classifiers. We show that sequential classifiers that exploit the use of discourse properties in social media conversations while using only local features, outperform non-sequential classifiers. Furthermore, we show that LSTM using a reduced set of features can outperform the other sequential classifiers; this performance is consistent across datasets and across types of stances.

To conclude, our work also analyses the different features under study, identifying those that best help characterise and distinguish between stances, such as supporting tweets being more likely to be accompanied by evidence than denying tweets. We also set forth a number of directions for future research.

Stance classification in out-of-domain rumours: A case study around mental health disorders

Download (PDF, 572KB)

Social media being a prolific source of rumours, stance classification of individual posts towards rumours has gained attention in the past few years. Classification of stance in individual posts can then be useful to determine the veracity of a rumour. Research in this direction has looked at rumours in different domains, such as politics, natural disasters or terrorist attacks. However, work has been limited to in-domain experiments, ie training and testing data belong to the same domain. This presents the caveat that when one wants to deal with rumours in domains that are more obscure, training data tends to be scarce.

This is the case of mental health disorders, which we explore here. Having annotated collections of tweets around rumours emerged in the context of breaking news, we study the performance stability when switching to the new domain of mental health disorders. Our study confirms that performance drops when we apply our trained model on a new domain, emphasising the differences in rumours across domains. We overcome this issue by using a little portion of the target domain data for training, which leads to a substantial boost in performance. We also release the new dataset with mental health rumours annotated for stance.

Simple open stance classification for rumour analysis

Download (PDF, 156KB)

Stance classification determines the attitude, or stance, in a (typically short) text. The task has powerful applications, such as the detection of fake news or the automatic extraction of attitudes toward entities or events in the media.

This paper describes a surprisingly simple and efficient classification approach to open stance classification in Twitter, for rumour and veracity classification. The approach profits from a novel set of automatically identifiable problem-specific features, which significantly boost classifier accuracy and achieve above state-of-the-art results on recent benchmark datasets. This calls into question the value of using complex sophisticated models for stance classification without first doing informed feature extraction.

SemEval-2017 Task 8 – RumourEval: Determining rumour veracity and support for rumours

Download (PDF, 115KB)

Media is full of false claims. Even Oxford Dictionaries named "post-truth" as the word of 2016. This makes it more important than ever to build systems that can identify the veracity of a story, and the kind of discourse there is around it. RumourEval is a SemEval shared task that aims to identify and handle rumours and reactions to them, in text.

We present an annotation scheme, a large dataset covering multiple topics – each having their own families of claims and replies – and use these to pose two concrete challenges as well as the results achieved by participants on these challenges.

A framework for real-time semantic social media analysis

Download (PDF, 766KB)

This paper presents a framework for collecting and analysing large volume social media content. The real-time analytics framework comprises semantic annotation, Linked Open Data, semantic search, and dynamic result aggregation components. In addition, exploratory search and sense-making are supported through information visualisation interfaces, such as co-occurrence matrices, term clouds, tree maps, and choropleths. There is also an interactive semantic search interface (Prospector), where users can save, refine, and analyse the results of semantic search queries over time.