key: cord-0920189-3ueqqqav
authors: Kang, EunKyo; Song, Narae; Ju, HyoRim
title: Contents and sentiment analysis of newspaper articles and comments on telemedicine in Korea: Before and after of COVID-19 outbreak
date: 2022-02-25
journal: Health Informatics J
DOI: 10.1177/14604582221075549
sha: 647a9b336014af257e0034e277f40fc5567eeef5
doc_id: 920189
cord_uid: 3ueqqqav

Telemedicine is rapidly growing to meet the increased needs for high-quality health care during the COVID-19 pandemic. However, telemedicine is still a sensitive issue as it is related to medical privatization. The use of telemedicine after the COVID-19 outbreak might be influenced by public opinion, and this may be an important key in implementing telemedicine. In this study, we aimed to assess if telemedicine-related newspaper articles and comments changed positively during the COVID-19 pandemic. From January 1, 2019, to March 1, 2020 (before COVID-19), a total of 1073 telemedicine-related articles were found in the Korean news network. Although the post-COVID-19 article collection period (from March 2, 2020, to September 30, 2020) was about half that of the pre-COVID-19, about twice the number (1934) of telemedicine-related articles were collected. And telemedicine-related news articles had a more positive tone post-COVID-19 than pre-COVID-19 (52.9% after vs 40.4% before). In conclusion, this study presented the association between the COVID-19 outbreak and changes in the media’s perception of telemedicine in Korea. This study presented that, as telemedicine begins to be utilized due to COVID-19, news media and readers who embrace it are beginning to view telemedicine positively, suggesting that COVID-19 has a positive foundation for the spread of telemedicine.

Introduction permitting medical practices to use Skype and FaceTime on the iPhone. In Korea, telemedicine was allowed for temporary consultations and prescriptions without a patient's visit from February 24, 2020. However, pursuing law amendments to implement telemedicine is difficult because of the opposing economic and safety claims about telemedicine by the government, the medical community, and civic groups. In particular in Korea, telemedicine is recognized as a process of medical privatization. Before COVID- 19 , some concerns about allowing telemedicine will lead to the invigoration of the medical industry. 10 To allow telemedicine, the legalization of remote monitoring devices should be the basis, which could be the beginning for for-profit corporations to participate in the medical industry. 10 In this regard, there was a negative public opinion on the permissibility of telemedicine in Korea. Also, along with safety issues such as the possibility of misdiagnosis for telemedicine, 11 it was argued that this was a paving stone for the commercialization of university hospitals. In the case of Korea, there were no hurdles for admission to tertiary medical institutions, so there were concerns that it could fatal the medical delivery system by accelerating the concentration of patients at tertiary medical institutions. However, the use of telemedicine during the COVID-19 outbreak may have influenced such public opinion, and this may be an important key in implementing telemedicine.

It would be very important to analyze media topics and sentiments in consideration of the impact that the media could have on public opinion. 12, 13 The news framing effect is a major component in the framing effect theory, one of the most leading theories in media and communication science. 14 Based on both psychology and sociology, the framing effect theory explains the ability of news media to affect people's attitudes and behaviors through making slight changes. 15, 16 In other words, the framework of perception provided is called a "frame," and the tone of the voice or topic of the media influences the decision-making of the audience of this content. Considering these media-framing effects, media topics and sentiment analysis for telemedicine are expected to be important factors for public acceptance.

The purpose of this study is to determine how the topic or sentiment of articles about telemedicine in the media has changed before and after COVID-19 in consideration of the fact that telemedicine has been used widely after COVID-19. Comparing pre and post will have important implications for the spread of telemedicine in the future because the media affects the public's acceptance of telemedicine. The study examined the association between the COVID-19 outbreak and changes in media and public opinion about telemedicine. Specifically, we investigated the media coverage, comment content, and sentiment analysis of telemedicine from January 1, 2019 (before the first confirmed case of COVID-19 on January 19, 2020) until post-activation of telemedicine following the outbreak (September 30, 2020). We focused on before and after the COVID-19 outbreak.

To investigate the portrayal of telemedicine before and after the COVID-19 outbreak, three different types of media were analyzed for the period from January 1, 2019, to September 30, 2020: newspapers, business magazines, and television articles. Thirteen general newspapers and five business magazines were analyzed in order of high circulation, except for regional newspapers and sports magazines whose readers are likely to be limited to specific population groups. These newspapers and magazines were coded Chosun Ilbo, Dong-A Ilbo, Joongang Ilbo, Hankyoreh, Kyonghyang Sinmun, Munhwa Ilbo, Hankook Ilbo, Kukmin Ilbo, Seoul Shinmun, Maeil Shinmun, Kookje Daily News, Segye Ilbo, Naeil Shinmun, Maeil Business Newspaper, The Korea Economic Daily, Money Today, Seoul Daily, and Korea Herald Business. The television articles originated from three broadcasters (MBC, KBS, and SBS) ranked at the top of the list in the first half of 2018.

A total of eight search formulae were used to collect online news articles in a test search to identify telemedicine-related articles. Four formulae were combinations of words; (1) "tele" AND "medicine," (2) "tele" AND "health," (3) "non-face-to-face" AND "medicine," (4) "non-face-to-face" AND "health," and four were words only; (5) "telemedicine," (6) "telehealth," (7) "non-face-to-face medicine," and (8) "nonface-to-face health." For analysis, 10% of the articles from the test search were randomly sampled to check the contents. When a search formula was composed of a combination of words, several articles that did not correspond to the contents of actual telemedicine were searched, or some articles with the contents of other remote services were searched, and so a search formula composed of a combination of words was judged as not suitable for analysis. Therefore, "telemedicine," "telehealth," "non-face-to-face medicine," and "non-face-to-face health" were used to collect telemedicine-related news articles. Article titles and full text were extracted, as well as unrelated text, such as the reporter's name or newspaper company name, for hyperlinking to other articles or advertisements.

It was demanding to manually investigate the plethora of text and comments of news articles, so natural language processing (NLP) procedures, including (1) tokenization, (2) stop words, and (3) stemming, were used in this study with the assistance of the Korean natural language processing in the Python (KoNLPy) package, 17, 18 Korean natural language processing procedures were performed in a form that allows morphological analysis.

Since March 2, 2020, the Ministry of Health and Welfare in Korea has allowed telemedicine for a "limited time". 19 We investigated the media coverage, comment content, and sentiment analysis of telemedicine from January 1, 2019 (before the first confirmed case of COVID-19 on 19 January 2020) until post-activation of telemedicine following the outbreak (September 30, 2020). We have defined "before COVID-19" as the period from January 1, 2019, to March 1, 2020. We focused on before and after the COVID-19 outbreak.

Topic modeling is a statistical model in which unstructured data (e.g., text data) are structured according to latent themes to manage multimodal data. Of these, Latent Dirichlet allocation (LDA) is a topic modeling technique that maps a document to a list of topics by allocating each word in the document to a separate topic, supposing that the document consists of words that help determine a topic. 20 LDA ignores the order of word's occurrence and the syntactic information and handles the document as a group of words or a set of words. LDA presumes that each document is produced through a statistical generative procedure. That means, each document is a mixture of topics, and each topic is a mixture of words. In the procedure of generating this document, it is assumed that you first pick a topic from a document-topic distribution, and later choose a word from a polynomial topic-word distribution from the selected topic. 20,21 LDA begins with randomly allocating topics to each word and iteratively enhances the allocation of topics to words throughout Gibbs sampling.

Latent Dirichlet allocation has got three hyperparameters. The "α" hyperparameter regulates the expected number of topics in the document. A low value of "α" means that a fewer number of topics in the mix is expected and a higher value implies that one would be expecting the documents to have a higher number of topics in the mix. The "β" hyperparameter regulates the dissemination of words by topic. A low value of "β" indicates that the topic contains fewer words, and a higher value suggests the topic to include more words. The "K" hyperparameter indicates the expected number of topics in the set of documents. 22 The selection of K values is usually based on domain knowledge or training different LDA models with different numbers of values of K and calculating a "Coherence Score". Coherence score is a measure of how good a topic model is in producing coherent topics. 23 Based on the calculated coherence score, the K value with the highest coherence score is selected.

In this study, we defined each news article as a single document. The number of topics can be selected based on the ease of interpretation or the coherence score in topic modeling. We aligned the training hyperparameters such as the number of repetitions and the number of optimized intervals to 1000 and 100, separately. We conducted the model altering the number of topics and measuring the coherence score. The coherence score varied around the highest levels of 0.32-0.35 fluctuating at 10 topics and after 22 topics. We chose 10 topics because of the easiness of interpretation.

Sentiment analysis techniques built for English pose practical challenges for Korean because Korean is an agglutinative language 24, 25 where part-of-speech is often not distinguished by the unit of space, and parts of speech may be different even within a bunch of words. In addition, Korean is a language with a drastic change in vocabulary. 26 Therefore, in this paper, we have proposed thematic analysis, which is a method for categorizing articles by sentiment. Two authors (K.E. and S.N.) independently reviewed 20% of the news articles, randomly extracted from the entire pool of news articles. The coders were asked to rate the sentiments of the contents of newspapers about telemedicine on a 3-point scale: (À1) negative; (0) neutral and (+1) positive. If it was judged that the article had a mixed attitude, the ratio of positive sentences and negative sentences was compared, and the one with a higher ratio was coded as the attitude. The most well-known coefficients for assessing the reliability of agreement are Cohen's kappa, 27 weighted kappa, 28 Fleiss' kappa, 29 Krippendorff's alpha, 30 and Intra-class correlation coefficient (ICC). 31 Of these, Cohen's kappa assesses the coefficient of agreement between a pair of raters deliberately selected on a nominal scale. 27 We measured Cohen's kappa coefficients to measure the reliability of agreement in this study. This sentiment analysis was found to have satisfactory inter-coder reliability; the inter-rater correlation for articles was very high (>95%). Both authors continued to code an additional 20% of the news articles, and a second inter-rater correlation was calculated. It was also confirmed as >95%. K.E. reviewed the sentiment for the remaining 60% of the news articles.

Topic classification, keywords, and percentage allocation before and after COVID-19

From January 1, 2019, to March 1, 2020, a total of 1073 telemedicine-related articles were found. Table 1 presents the topic classification, keywords, and percentage allocation before COVID-19. The most popular topic was "the government's regulation and policy of telemedicine," roughly accounting for 32.7% of all news articles. The next most-covered topics covered in telemedicinerelated articles were "the use of telemedicine in health services" (19.6%) and 'the medical workers" associations opposition to telemedicine" (15.4%).

Although the post-COVID-19 article collection period (from March 2, 2020, to September 30, 2020) was about half that of the pre-COVID-19, about twice the number (1934) of telemedicinerelated articles were collected. This result is perhaps a reflection of the accelerated interest in telemedicine generated by the COVID-19 outbreak. Table 2 shows the topic classification, keywords, and percentage allocation after COVID-19. "The use of telemedicine in medical services" during the COVID-19 pandemic has been the most popular topic after COVID-19 (35.5%). The second most frequent topic was "the medical workers' associations opposition" (15.6%), which was the third most common topic in news articles before COVID-19 but showed a similar proportion. Next in the sequence was the topic "the Korean government's support and regulation, policy, and political position on telemedicine," which accounted for 13.0% of all news articles, followed closely by "growth of the telemedicine industry," at 12.1%.

Before the COVID-19 pandemic, news articles on telemedicine were generally written in a negative tone rather than a positive one (Table 3) . By topic, "the government's regulation and policy of ) , which was the second-largest subject, presented similar proportions of positive and negative articles. By contrast, about twice the articles covering "the medical workers' associations' opposition" were written in a negative tone than a positive tone (28.5% positive vs. 65.5% negative vs. 6.0% neutral). The topics "growth of the telemedicine industry" and "the trend of the international market of telemedicine" presented a higher proportion of articles with a relatively negative tone. Only topics related to "deregulation associated with telemedicine, such as pilot projects or regional deregulation," "investment in telemedicine," and "overseas telemedicine regulations and policies" tended to have a higher proportion of articles with a positive tone than a negative one. Table 4 shows that the telemedicine-related news articles had a more positive tone post-COVID-19 than pre-COVID-19 (52.9% after vs 40.4% before). This was particularly apparent for articles on "the use of telemedicine in medical services in the pandemic of COVID-19," the topic most frequently dealt with in telemedicine post-COVID-19 (54.3% positive vs. 40.7% negative vs 5.0% neutral). However, the pattern was opposite for news articles related to the second most discussed topic, "the medical workers' associations opposition" (14.0% positive vs. 86.0% negative vs 0.0% neutral). Except for this topic, all other topics received a higher proportion of articles written in a positive tone than a negative tone.

The analysis of news articles conducted in this study could be used to explain changes in the perception of telemedicine in the media before and after COVID-19. Telemedicine-related articles before COVID-19 mostly covered "the government's regulation and policy of telemedicine." Reasons for this include the disparities in telemedicine policy and regulation between Korea and other countries. 32, 33 Next ranked by topic was "the use of telemedicine in health services" (19.6%), followed by "the medical workers' associations' opposition" (15.4%). This large amount of content released on these topics is thought to be related to the low telemedicine use rate in Korea. As mentioned above, previous to the COVID-19 pandemic, Korea utilized telemedicine services at a much lower rate than other countries, including the EU and Japan. 9, 34 The keywords for "the use of telemedicine in health services" included "utilization" and "acceptance." Many news articles on this topic focused on the necessity to use telemedicine, again reflecting the low telemedicine use rate in Korea. Interestingly, a previous examination of a social platform focused on health information technology and informatics reported that the social media users not only perceived significant opportunities associated with telemedicine but also obstacles to overcome to achieve these opportunities. 35 Even though the study was conducted before COVID-19, there was great interest in the potential of the new technology, telemedicine, and concerns about quality and cost stand out, which is different from the results of this study. These results might be since the study investigated the perception of social media users, and the focus on telemedicine differs depending on the information platform. There has been a dramatic change in the leading subject of telemedicinerelated articles after COVID-19, as exemplified by the fact that "the use of telemedicine in medical services in the pandemic of COVID-19" became the most popular topic. Early on during the COVID-19 pandemic, telemedicine was used to evaluate respiratory symptoms that could be initial symptoms of COVID-19 infection. 36, 37 Later, telemedicine was used as a means of providing medical care to non-COVID-19-infected people. 38 Specifically, the use of telemedicine to manage the condition of chronically ill patients has increased in earnest. [39] [40] [41] This increased utilization of telemedicine had encouraged discussion around the integration of telemedicine into the education of healthcare providers, 42, 43 reforming clinical care models, [42] [43] [44] [45] and barriers 44, [46] [47] [48] [49] and concerns about the safety of telemedicine. 43, 46, 47, 49 About a third of all news articles were associated with the use of telemedicine after COVID-19. Audiences who read news articles might obtain information on how telemedicine is being utilized in the field, as well as the advantages and disadvantages, and necessities of telemedicine. Therefore, a frame that includes the contents and tone of a news article could be very important. [50] [51] [52] In this study, keywords, such as "policy" and "safety," were included Table 4 . Sentiment analysis of news articles after COVID-19 by topic. in the topic "the medical workers' associations' opposition," and "support," "service," and "investment" were among the keywords for the topic "before COVID-19's news articles." The relevance of these keywords to these topics is demonstrated in the discussion about telemedicine mentioned above. [42] [43] [44] [45] [46] [47] [48] [49] Before COVID-19, studies that conducted sentiment analysis on telemedicine were limited. Although we did not include Twitter data in our analysis, a previous study about sentiment analysis on tweets that mentioned the key term #Digital Health described most posts as having a positive tone. 53 By contrast, many studies on telemedicine awareness before COVID-19 were concerned about the safety of telemedicine or barriers to patient adoption and use of telemedicine. [54] [55] [56] The current study showed a high proportion of articles with a relatively negative tone before COVID-19 but a higher ratio of articles with a positive tone after COVID-19 than before COVID-19. It suggests that the media's perception had changed concerning the active utilization of telemedicine due to the COVID-19 pandemic, resulting in a more positive attitude. As the results of this study, the perception of telemedicine that has changed before and after COVID-19 could affect the acceptance of telemedicine. According to a study conducted in 2019, perceived usefulness, social influences, and attitude toward telemedicine could influence the acceptance of telemedicine. 57 It has been reported that the relationship between this perception and acceptance appears not only in medical professionals but in the general population; previous studies reported that medical staff and the public who have an insight into the net function of telemedicine or have a positive attitude are much more likely to accept telemedicine. 58, 59 As telemedicine was implemented in earnest due to COVID-19, stakeholders' concerns began to grow. [60] [61] [62] Despite the many articles related to "the medical workers' associations' opposition" that employed a negative tone, the proportion of articles displaying a positive tone has increased significantly after COVID-19. These positive changes in news articles could affect the reader's perception. 63 In other words, it could influence the public reading the article to have a positive perception of telemedicine. Also, a positive change in the tone of a news article could affect public preference for telemedicine. 64 Considering these points, it is possible to respect the possibility that public perception has changed based on the positive change in the tone of news articles as a result of this study.

Post-COVID-19, telemedicine began to be used more actively in the management of various diseases, yet many clinical, legal, and social barriers to implementing telemedicine remain. In particular, the clinical barrier to the uptake of telemedicine is not irrelevant to the social perception associated with its use. In the context of developing a patient-doctor relationship, which is a representative clinical barrier to implementing telemedicine, it may be much more difficult to form a rapport than face-to-face treatment. In other words, it may be difficult to obtain patient trust in the patient-doctor relationship formed in telemedicine. 65, 66 This lack of rapport has the potential to lead to a limited provision of information and inappropriate treatment. That is, to overcome the limitations of telemedicine, first, the reliability of telemedicine itself must be increased. If social trust in telemedicine itself is not established, improving the trust between doctors and patients could be a more difficult goal. The fact that telemedicine was used in earnest along with the COVID-19 outbreak could be an important turning point in the change in telemedicine's reliability as it has the potential to change the perception of telemedicine itself. As seen from the results of our study, the perception of telemedicine changed positively after the COVID-19 outbreak, suggesting the possibility that it had a positive effect on the change in the reliability of telemedicine.

Although this research was conducted according to the recommended methodology, it has some limitations. First, as mentioned above, Korean is an agglutinative language, and although an aspectlevel sentiment analysis might be possible, a highly reliable Korean sentiment dictionary has not yet been established. 67 Therefore, we performed the traditional sentiment analysis method: manual coding. Although manual coding has limitations in that it does not always achieve a high level of reliability and is time-consuming, 68 the inter-rater reliability in our study showed a very high kappa coefficient of 0.95 or more. 69 Since we manually analyzed sentiment, to derive meaningful results based on limited time and resources, this study was conducted by confining the sample to news from a few representative media. Considering that the function of the current English-based NLP model is advancing, 70, 71 if the Korean sentiment analysis dictionary is further developed, it is thought that sentiment analysis targeting a larger sample size will be possible in the future. Second, we only searched online news databases and did not actively search paper news articles. As a result, our search might not have been comprehensive. However, we thought that only analyzing online news articles would be an optimized approach given the time sensitivity and large volume of news publications. It must also be considered that many months have passed from the time of our search to the writing of the manuscript, during which articles on telemedicine were continuously published. Accordingly, further research might be needed to re-evaluate the perception of the media.

This study highlights the growing media's interest in telemedicine after COVID-19 compared to before COVID-19 and the positive scope change in the media. Previously, research reports on perceptions related to telemedicine were mainly published for first-hand experience patients or medical staff. With the COVID-19 pandemic, telemedicine had begun to be used in earnest, and the point that it is changing positively in the media as a whole means that it is prominent to a change in social perception. Telemedicine could have a major influence in advancing health care in the future. To fully utilize telemedicine in environments with limited medical resources, not only must classifications, limitations, protocols, monitoring, assessments, and data privacy protection be determined but also social consensus must be required. In other words, for the continuing development in telehealth use, understanding the change in social perception of telemedicine could be important. Within the designated scope of our review, the change in the media's perception of current telemedicine reflects the change in social perception and answers emerging questions.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) received no financial support for the research, authorship, and/or publication of this article.

The raw data are being kept in the custody of Seoul National University hospital and are available upon request.

EunKyo Kang  https://orcid.org/0000-0001-5844-5625 HyoRim Ju  https://orcid.org/0000-0002-5122-8932

Supplemental material for this article is available online.

Telemedicine-maintaining quality during times of transition

Current status and progress of telemedicine in korea and other countries

Health at a Glance: State of Health in the EU Cycle

US telehealth satisfaction study

More room for telemedicine after COVID-19: lessons for primary care

Bringing health care to the patient: An overview of the use of telemedicine in OECD countries

The promotion of telemedicine: What is the problem?

Searching for digital technologies in containment and mitigation strategies: experience from South Korea COVID-19

Telemedicine practice: review of the current ethical and legal challenges

How the news media persuades: framing effects and beyond

Framing: media frames

News framing effects: Theory and practice

The polarizing effect of news framing: comparing the mediating roles of motivated reasoning, self-stereotyping, and intergroup animus

Framing climate change: exploring the role of emotion in generating advocacy behavior

Korean natural language processing in Python

Natural language processing in health care and biomedicine. Biomedical informatics

Lessons from temporary telemedicine initiated owing to outbreak of COVID-19

Latent dirichlet allocation (LDA) and topic modeling: models, applications, a survey

Sequential latent dirichlet allocation

Scalable hyperparameter selection for latent dirichlet allocation

Full-text or abstract? examining topic coherence scores using latent dirichlet allocation

Sentiment digitization modeling for recommendation system

A hybrid approach for the sentiment analysis of Turkish Twitter data

Kappa coefficient: a popular measure of rater agreement

Assessing the inter-rater agreement for ordinal data through weighted indexes

Fleiss' kappa statistic without paradoxes

Computing Krippendorff's alpha-reliability

A guideline of selecting and reporting intraclass correlation coefficients for reliability research

Identifying obstacles and research gaps of telemedicine projects: approach for a state-of-the-art analysis

State of telehealth

European hospital survey: benchmarking deployment of e-Health services

Social media users' perception of telemedicine and mhealth in China: exploratory study

Onsite telemedicine strategy for coronavirus (COVID-19) screening to limit exposure in

A double triage and telemedicine protocol to optimize infection control in an emergency department in Taiwan during the COVID-19 pandemic: retrospective feasibility Study

The use of telemedicine to maintain breast cancer follow-up and surveillance during the COVID-19 pandemic

The Importance of telemedicine during COVID-19 pandemic: a focus on diabetic retinopathy

Heart failure management during the COVID-19 outbreak in Italy: a telemedicine experience from a heart failure university tertiary referral centre

Telehealth services for persons with chronic lower respiratory disease and their informal caregivers in the context of the COVID-19 pandemic

Telehealth services for persons with chronic lower respiratory disease and their informal caregivers in the context of the COVID-19 pandemic

Telemedicine and the COVID-19 pandemic, lessons for the future

Telemedicine for diabetes after the COVID-19 pandemic: we can't put the toothpaste back in the tube or turn back the clock

COVID-19 transforms health care through telemedicine: evidence from the field

Paying for telemedicine after the pandemic

Clinician Telemedicine Perceptions During the COVID-19 Pandemic

Telemedicine: medical, legal and ethical perspectives

Opportunities in telemedicine, lessons learned after COVID-19 and the way into the future

Seven models of framing: Implications for public relations

Framing as a theory of media effects

Identifying framing bias in online news

Digital health: exploring users' perspectives through social media analysis

Perceptions of risk as a barrier to the use of telemedicine

The individual and contextual determinants of the use of telemedicine: a descriptive study of the perceptions of Senegal's physicians and telemedicine projects managers

Primary care physicians' perceived barriers to optimal dementia care: a systematic review

Theories predicting end-user acceptance of telemedicine use: systematic review

Telemedicine in Germany during the COVID-19 pandemic: multiprofessional national survey

Factors influencing the adoption of telemedicine health services during COVID-19 pandemic crisis: an integrative research model

A qualitative study of primary care physicians' experiences with telemedicine during COVID-19

Surgery provider perceptions on telehealth visits during the covid-19 pandemic: room for improvement

Increase in video consultations during the COVID-19 pandemic: healthcare professionals' perceptions about their implementation and adequate management

Spatial sentiment and perception analysis of BBC news articles using twitter posts mining

Turning words into consumer preferences: How sentiment analysis is framed in research and the news media

Provider barriers to telemental health: obstacles overcome, obstacles remaining

Barriers and facilitators that influence telemedicinebased, real-time, online consultation at patients' homes: systematic literature review

Attention-based long short-term memory network using sentiment lexicon embedding for aspect-level sentiment analysis in Korean

Extracting latent moral information from text narratives: relevance, challenges, and solutions

Computing inter-rater reliability for observational data: an overview and tutorial

Sentiment analysis of big data: methods, applications, and open challenges

The validity of sentiment analysis: comparing manual annotation, crowd-coding, dictionary approaches, and machine learning algorithms