key: cord-0479486-se0i3db4
authors: Rafi, Abdul Muntakim; Rana, Shivang; Kaur, Rajwinder; Wu, Q. M. Jonathan; Zadeh, Pooya Moradian
title: Understanding Global Reaction to the Recent Outbreaks of COVID-19: Insights from Instagram Data Analysis
date: 2020-09-15
journal: nan
DOI: nan
sha: 246f5b6687cdb67dd9d8cf3b06084e6a81e7fd71
doc_id: 479486
cord_uid: se0i3db4

The coronavirus disease, also known as the COVID-19, is an ongoing pandemic of a severe acute respiratory syndrome. The pandemic has led to the cancellation of many religious, political, and cultural events around the world. A huge number of people have been stuck within their homes because of unprecedented lockdown measures taken globally. This paper examines the reaction of individuals to the virus outbreak-through the analytical lens of specific hashtags on the Instagram platform. The Instagram posts are analyzed in an attempt to surface commonalities in the way that individuals use visual social media when reacting to this crisis. After collecting the data, the posts containing the location data are selected. A portion of these data are chosen randomly and are categorized into five different categories. We perform several manual analyses to get insights into our collected dataset. Afterward, we use the ResNet-50 convolutional neural network for classifying the images associated with the posts, and attention-based LSTM networks for performing the caption classification. This paper discovers a range of emerging norms on social media in global crisis moments. The obtained results indicate that our proposed methodology can be used to automate the sentiment analysis of mass people using Instagram data.

Advent of social media presents a promising opportunity for emotion analysis of the mass people. Different approaches have successfully analyzed social media data to detect several conditions [1] . There is a vast wealth of psychological data encoded in visual social media data, such as photographs posted to Instagram. In this paper, we explore the possibility of analyzing visual data from Instagram to understand the global reaction to the recent outbreak of the coronavirus.

It is reasonable to prioritize research into Instagram analysis for this problem. Instagram members currently contribute almost 100 million new posts per day [2] , and Instagram's rate of new users joining has recently outpaced Twitter, YouTube, LinkedIn, and even Facebook [3] . Users give more importance to the visual content while sharing posts on Instagram. The posts are accompanied by captions as well. Instagram, among all social media, is more of personal space for the users. People tend to be less formal in their reactions. The data shared here can often have hidden complex meanings, which makes the Instagram posts most suitable for our *The first 3 authors had equal contribution to the paper. The dataset is available at www.kaggle.com/muntakimrafi/global-reaction-to-covid19instagram-data-analysis task. Instagram data has proved to be a useful source for emotion analysis of the mass people. Instagram tends to be a place where people share personal emotions [4] .

According to [5] , the origin of the outbreak has been identified as Wuhan. The virus rapidly spread all over the globe. As of May 2020, more than 4 million cases of COVID-19 is confirmed worldwide [6] , [7] . World Health Organisation (WHO) has declared a Global Health emergency, as the number of this novel virus keeps ascending globally [8] . Worldwide 212 countries or territories have confirmed cases of this novel virus [6] . A lot of everyday activities around the world have been restricted except for necessity and health circumstances. Acknowledging the rapid spread of the novel virus, there is a considerable impact of this epidemic on social networks and media as well.

Understanding public reactions to a crisis is the first step to solving the situation. The initial assessment includes how the crisis is affecting people's lives. A concern that disrupts the everyday lives of more people is a more severe problem. Therefore, successfully identifying human reactions to a situation can lead to a successful estimation of the seriousness of the problem. Also, the mental health of individuals may deteriorate during crisis moments. As Instagram is a platform where people share their emotions, it is one of the best places to analyze the mental health of mass groups of people. On average, 100 million photos are uploaded daily on Instagram [9] . Manual analysis of all the information shared on a social media platform is almost impossible. Recently, deep learning algorithms have performed magnificently at extracting features from visual data. Therefore, we try to explore the feasibility of deep learning algorithms in analyzing Instagram data in this paper.

The major contributions of our paper are as follows: 1) We collect a dataset consisting of 11973 Instagram posts with location. 2) We manually annotate 2534 posts for getting insight into our collected Instagram data. 3) We explore the performance of deep learning algorithms in performing the analysis without human intervention. The rest of the paper is organized as follows. Section II describes the related researches conducted on the literature. Section III explains the deep learning terminologies used throughout the paper. Sections IV explains our proposed methodology in details, and section V contains all of our experiments and and results. Finally, we present our conclusion at Section VI.

Researchers have found that mental health is strongly associated with social activities [10] . Photographs posted to Instagram contain different dimensions that might be analyzed for psychological insight. The photographs can be analyzed to determine different characteristics, such as the presence of people in the photographs or videos, the setting in nature or indoors, the time of the day, etc. Some research have been conducted in extracting statistical properties from images for performing various types of analysis [11] . Instagram metadata offers additional information like how many comments or likes the post received. Finally, platform activity measures of different areas can yield clues to Instagram user's mental state. These dynamics were briefly discussed in [12] . In particular, Lup et al. [13] and Andalibi et al. [14] proposed to use Instagram data for mental health analysis. Recently, some works in the literature started using multi-modal analysis. In [15] , the authors showed that using both textual and visual analysis allowed them to distinguish between different types of users and their interests.

Instagram posts have been used to find out the emerging norms in social network dynamics. In [4] , researchers have used specific hashtags to pull off data from Instagram and discovered the nature of the reaction of general people to birth and death. They have raised a crucial privacy concern by showing that people create the first digital footprint of their children even before they are born. They also showed that people share their emotions on Instagram. In [16] , the authors showed that Instagram plays a vital role in body dissatisfaction and women's drive for thinness. In their research, it has been shown how people's lives are affected by the posts they see on social media. Previously, the manual or statistical feature-based analysis was used to infer decisions about social media data. The advancement of deep learning has made it possible to learn from images and associated text in a self-supervised way. The idea originated from the works in [17] , where the researchers proposed that instead of predicting the ImageNet classes of the image, deep learning models can be used to infer the Word2Vec representations of their corresponding labels. Researchers used the idea to analyze social media data. In [18] , the authors proposed a method to mine data from different social media platforms and modalities related to New York City and examined the behavior of the citizens. In [19] , the authors used geo-located social media photographs to identify the most visited places by tourists in major European cities. In [20] , the authors used popular hashtags on Instagram to find out cultural differences between different cities and neighborhoods. In [21] , the authors proposed a joint embedding of pictures of food and recipes. They were able to achieve interesting results on an image-recipe retrieval task. In [22] , the authors analyzed Instagram posts to study how the different neighborhoods, events, and cultures of the city are presented on Instagram. Researchers detected race, age, and gender of people from New York City Instagram images in [23] . They have also analyzed the social diversity of different neighborhoods and compare them to census-based metrics.

CNNs are a class of deep learning networks that can infer intricate visual patterns and symmetries from a set of images. It learns filters of specific kernel size and understands these patterns after passing the images through the filters [24] . The CNN network that is used in this project is ResNet-50 [25] .

Mathematically, an image is convolved at every layer in a CNN and features are made more accessible for dense layer at the final classification task. Suppose x l m is the m th input feature in the l th layer then the n th output feature in that y l n is calculated as:

Here, where M l−1 is the number of input maps, * denotes convolution operation, and b l n is the bias of the n th output map in the l th layer.

RNNs are a class of deep learning networks that consists of a series of memory cells, which allows input from the previous memory cells and can infer the context from timeseries data, then subsequently generate output [26] .

For defining how RNN computes mathematically it's output, following equation is described:

where o t is the output of the RNN at time t, x t is the input to the RNN at time t, and h t is the state of the hidden layer(s) at time t.

Aspect based sentiment analysis requires a sort of mechanism which can have a weighted focus on variegated words in a particular set of sentences. This mechanism is called attention, which gives different weights to words in a sentence while considering various types of aspects of the sentence [27] .

Taking H be a matrix comprising of hidden vectors [h 1 , . . . , h N ] that the LSTM has yielded, where d is the dimensionality of hidden layers & N is the length of the particular input sentence. Let v a represents the embedding of aspect and e N be a identity vector. The attention mechanism will produce an attention weight vector α and a weighted hidden representation r. Thereby, it allows the attention mechanism to focus on a particular segment of the sentence while considering one specific type of aspect [27] .

The final sentence representation is given by:

where, h * , W p and W x are projection parameters to be learned during training. h * is defined as the feature representation of a sentence with given an input aspect. Then linear layer to convert sentence vector to e, followed by softmax layer to transform e into a conditional probability is implemented.

where W s and b s are the parameters for softmax layer. For aspect-based sentiment analysis, the input aspect embedding is appended with each word input vector.

Learning from a pre-trained neural network can be applied to a different application. Usually, the initial layers of networks are kept frozen, and the final layers are trained again on the new data. This method enables transferring the learning from one application to another similar type of application without requiring much data. 

Our proposed methodology can be defined by dividing the whole procedure into four tasks. We scrape data from Instagram, clean the data, annotate the data, and apply deep learning models on the collected data. We obtain results for understanding the sentiment from images and text captions both.

We use igramscrapper, a python API [28] to collect data from Instagram using hashtags #wuhan, #corona, and #coronavirus. We scrap the data from February 16 to March 20. We save the metadata related to each post in excel files and the corresponding media data (for videos we save the first frame). The metadata we store for each post are the post identifier, Instagram short code (i.e., B8o8MQHJbMc), created time, data type (image or video), source link to the post, low resolution image source link, high resolution image source link, post caption, owner id, number of likes received, number of comments received, and location name. All the posts we scrap from Instagram are public posts and visible to everyone.

After gathering the data, we perform data cleaning. The data cleaning process is divided into several sub-tasks as following:

• Data deduplication: In this step, we remove the data which are repeated. • Data completeness: The part of data that is captured incompletely during the scraping process, can either be deleted or imputed to fill in meaning. In our case, missing data imputation may provide us wrong impressions about sentiments. For this reason, we delete the incomplete data points. • Corrupted Data: The corrupted data points are also removed. 

We categorize our annotated data into five categories, as shown in Fig. 1 . Humor, funny, and memes posts are classified as the first class. News articles, awareness posts, and information about cloud-19 and lockdown are marked as the second class. Happy, supportive, appreciable, positive, and nostalgic posts are put together in class 3. Negative posts regarding China, xenophobic, racist behavior, blaming, and protest against isolation are regarded as category 4. The class distribution of the image and caption annotations are illustrated in Fig. 2 and Fig. 3 . As depicted in the graphs, it can be inferred that the occurrence of "Positive" type and "Neutral/News" type annotations are the maximum , while "Negative" type annotation is observed the least.

During the annotation process, we use Google Lens for translating the media content to help us with the annotation process. Google translate Chrome extension is used for translating the caption. Further discussion about which type of content is segregated in which kind of annotation is given below (see Fig. 4 ).

1) Memes/Humor Type: This category comprises of any sort of meme or humor content, which mostly included people sharing memes surrounding toilet papers, corona beer, along with hilarious alternatives to sanitizers, and social distancing.

2) News/Neutral Type: This class covers posts, which were shared just to spread awareness about healthy habits like washing hands, wearing masks, and social distancing to be practiced during the pandemic. It also covers news information shared by a news channel on Instagram, along with advertisement about sanitizers, closure announcement of stores, info about toilet papers, masks, sanitizers available in a particular store. It also includes news reports about lower pollution and emptier streets.

3) Positive Type: This category includes any Instagram post, which is targeted to spread not only physical but also mental positivity among the users. It covers posts regarding how people are spending their leisure time while quarantining themselves, which includes reading novels, dancing, exercising, playing video games, having movie watch parties, and singing. It also includes Instagrammers posting about the appreciation of the front-line workers, medical teams working during the pandemic, and wearing unique masks. This type of annotation also includes people posting their nostalgic past travel pictures and sharing pictures of how they are spending more time with their adorable pet dogs and cats. Medical doctors and nurses posting their selfies are also included in this category.

4) Negative Type: The negatively annotated data includes posts criticizing the political systems, xenophobic posts against China, anger posts against imposed lockdown, fearmongering users by sharing content with people dying, ranting about loss of freedom, conducting public protests against the government. People blaming government and media for hiding factual data and spreading conspiracy theories also fall into this class. 5) Random Type: This class is used to separate Instagram influencers' posts and advertisements, which targets the promotion of unessential products by using the hashtags #wuhan and #coronavirus. It shows that the hashtags were misused to promote irrelevant agendas.

We process the video data by taking the first frame of the video and save it as a single image which is stored along with the scraped image dataset. We extract audio from the video and run a subtitle generator on it. Thus, subtitles that are upto 2 minutes are generated for the video and merged on top of stored caption data for the related post. Pytesseract API is used to extract text written on the image and saved along with the caption data stored for that post. Before deploying the deep learning models, first pre-processing steps are applied to caption text data and is translated into English using python API and trimmed up to word length of 300 words. Our proposed algorithms look at a single frame (image or the first frame of the video) for each post and the corresponding caption data.

The text data is used for training attention-based LSTM network for textual sentiment analysis. The image data is used for image classification using CNNs. We choose ResNet-50 for the image classification task. The proposed algorithm is illustrated in Fig. 5 .

We perform several analyses to get insights into our collected dataset. We also explore the performance of deep learning algorithms for checking the feasibility of automatic analysis. We use the Tensorflow library for implementing deep learning algorithms. All of the experiments regarding training and implementation of the models are performed in a hardware environment, which includes Intel Xeon, 2.60 GHz CPU and Nvidia Quadro M4000 (8 GB Memory) GPU.

We collect longitude and latitude for the location of the posts to perform several location-based analyses on our collected data. It enables us to see which regions have been most using Tableau. The geographic representation of data shows the intensity of global discussion worldwide on the Instagram platform. We illustrate the Instagram activity based on the number of posts, the number of likes received, and the number of comments received in Fig. 6-8 , respectively. In Fig. 6 , we can see that Indonesia, China, United States, Turkey, United Kingdom, Germany, Malaysia, and Italy have been most active concerning the number of posts posted on Instagram. It is also visible that Russia was quite inactive, given its 46M users [9] . We see somewhat similar trends in Fig. 7 and Fig. 8 .

To understand the demographic viewpoint further, we extract the country name based on the location of the Instagram posts using the pycountry library. The top 15 countries with the maximum number of posts are illustrated with respect to to the count of the different class of annotations in Fig.  9 . Mostly, News/neutral, memes, and positive content are observed among the countries. The United States deviates from this trend as more negative posts are observed than positive posts.

For getting further insights into our collected dataset, we plot several graphs by taking combinations of the following parameters:

• class of images in the Instagram posts (denoted by Img_ANN), • class of captions of the posts (denoted by Cap_ANN), • number of likes (denoted by likes_count), • number of comments (denoted by comments_count).

The following Fig. 10 shows the class-overlap between the images and captions for the posts. It is visible that there are not many mismatches between the image annotation and the caption annotation for the Instagram posts.

To understand the pattern between the number of likes received and the number of comments received, a scatterpoint graph between these two parameters is plotted (see Fig. 11 ). From the chart, we can infer two things. Firstly, negative posts tend to receive more comments than likes, as people provide their own opinion on that post without liking it. Secondly, positive posts receive more likes than comments.

A scatter-plot is shown with the following variables: the number of likes received, class of the image and the class of the and caption for each post (see 12) . We can see that posts with negative images but positive captions receive a high number of likes. It indicates that positive caption can change the class of the overall post and make it positive, despite containing a negative image.

The results we obtained from our experiments on the collected data are shown in Table I . We perform sentiment analysis on images and captions separately. For textual sentiment analysis; we use attention-LSTM [27] network and train it on Stanford's Movie Review Dataset [29] . We achieved 87.7% accuracy. Later, we applied transfer learning Textual Sentiment (Attention LSTM) [27] Movie Review Dataset [29] 87.7% 2 Textual Sentiment (Attention LSTM) [27] Our dataset 73.6%

3 Image Sentiment Analysis (ResNet-50) [30] Reddit Memes Dataset [31] 82.1%

4 Image Sentiment Analysis (ResNet-50) [30] Our Dataset 79.2% For conducting sentiment analysis on images, we used ResNet-50 [25] as suggested in the following paper [30] . We used Reddit Memes Dataset [31] first to train the network and obtained 82.1 % accuracy. Afterwards, we fine-tuned the pre-trained Resnet-50 on our dataset and achieved 79.2 % accuracy. It should be noted that we exclude the 'random' class while implementing our algorithms, and the networks are trained for classifying four classes.

In this paper, we have studied the reaction of Instagram users, in light of the recent outbreak of the coronavirus. It is essential to evaluate the public emotion in times of crisis and social media data, particularly Instagram, can be a valuable source for conducting such analysis in real-time. Consequently, we have collected in total, around 100k Instagram posts. Among them, around 14% of the posts containing location data were used for our location-based analysis. We manually annotated around 20% of the posts with location to perform our reaction analysis to the virus outbreak. The collected images and videos on Instagram reveal a range of emotions, highlighting social awareness, and mass mayhem. We found that users from most countries share positive and awareness posts. We also used deep learning algorithms to conduct the classification. Results show that the large-scale sentiment analysis can be automated with the help of deep learning algorithms.

In our future work, we want to improve the accuracy of the deep learning models and perform the emotion analysis at a larger scale.

Associations between displayed alcohol references on facebook and problem drinking among college students: 125

A tale of four platforms: Motivations and uses of facebook, twitter, instagram, and snapchat among college students?

Global social media research summary 2016

Visualising the ends of identity: pre-birth and post-death on instagram

A pneumonia outbreak associated with a new coronavirus of probable bat origin

2019-ncov global cases study (by johns hopkins csse)

Coronavirus live updates: Third case of coronavirus confirmed in london, ont., bringing canadian cases to four

Statement on the second meeting of the international health regulations (2005) emergency committee regarding the outbreak of novel coronavirus (2019-ncov)

Diagnostic and statistical manual of mental disorders

A graphical vector autoregressive modelling approach to the analysis of electronic diary data

Instagram photos reveal predictive markers of depression

Instagram# instasad?: exploring associations among instagram use, depressive symptoms, negative social comparison, and strangers followed

Depression-related imagery on instagram

Learning from barcelona instagram data what locals and tourists post about its neighbourhoods

Sensis social media report 2017

Zero-shot learning by convex combination of semantic embeddings

Discovering the city by mining diverse and multimodal data streams

Identification of tourist hot spots based on social networks: A comparative analysis of european metropolises using photo-sharing services and gis

Instagram post data analysis

Learning cross-modal embeddings for cooking recipes and food images

Reassembling the city through instagram

Towards measuring fine-grained diversity using social media photographs

Recent advances in convolutional neural networks

Deep residual learning for image recognition

Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network

Attention-based lstm for aspect-level sentiment classification

Learning word vectors for sentiment analysis

Emotion detection and sentiment analysis of images

Reddit Memes Dataset