key: cord-0833245-n2315d6n authors: Hassen, Hailemariam Mamo; Behera, Manas Ranjan; Jena, Pratap Kumar; Dewey, Rebecca S; Disassa, Getachew Abeshu title: Effectiveness and Implementation Outcome Measures of Mental Health Curriculum Intervention Using Social Media to Improve the Mental Health Literacy of Adolescents date: 2022-05-03 journal: J Multidiscip Healthc DOI: 10.2147/jmdh.s361212 sha: f12c56b19cff0ae64543fecaf9c916526c4910f0 doc_id: 833245 cord_uid: n2315d6n PURPOSE: Globally, adolescent mental health problems reportedly manifest more severely in individuals with lower mental health literacy. Mental health curriculum interventions using social media have been associated with positive implementation outcomes. This study aimed to investigate whether such an intervention significantly improves adolescent mental health literacy and is associated with positive implementation outcome measures. METHODS: This study employed a quasi-experimental design, comparing the before-and-after difference between the intervention group (n(1)=77) and the control group (n(1)=76). A mental health literacy module was delivered using social media. Data were collected using validated questionnaires. Effect size and difference-in-differences model calculations (and associated 95% confidence intervals, CI) were conducted alongside other descriptive analyses. RESULTS: The estimated intervention effect size and difference-in-differences estimates were greater than zero. The effect was greater in the intervention than in the control group (p<0.05) despite sex and age differences. The effect size estimate provided by Cohen’s d and Hedges’ g values was medium-to-large (d/g=0.429–0.767, p<0.05). The difference-in-differences reflected a significant effect (DID=0.348, CI: 0.154–0.542, p<0.001) in increasing mental health literacy despite differences associated with sex, age, and school grade. The intervention program was acceptable, appropriate, feasible, and satisfactory, and more than 80% of participants said they agree to completely agree with these implementation outcomes. Resources and personal and family-related factors were among the perceived influencing factors determining the effectiveness and implementation outcome measures. CONCLUSION: The intervention program effectively improved the mental health literacy of adolescents. It has been highlighted that social media interventions for mental health promotion could be practical and scalable; however, there is a need to take into account socio-demographic differences and barriers to inclusion/compliance. Adolescent mental health problems tend to be more serious in individuals with lower mental health literacy. 1, 2 Mental health interventions using social media are thought to be effective and have shown positive implementation outcome measures for mental health promotion. 3, 4 Addressing mental health problems in children and adolescents has emerged as a public health priority. Globally, approximately 10-20% of adolescents live with mental health problems, 5, 6 and most do not have access to the necessary health services. Approximately 17-23% of Ethiopian adolescents have mental health problems, 7, 8 which is higher than the global prevalence. 3 Implementing mental health services in low-income countries like Ethiopia is highly opportunities for improving social connectedness, resilience, and well-being. 36, 40 Social media is established as a medium of communication for health promotion and intervention in developed countries. Interventions to promote mental health using social media are effective, acceptable, feasible, appropriate, and popular. 35, 36 However, in low-income countries, digital inequality and related factors negatively impact the successes of these interventions. There is little quantitative evidence supporting the effectiveness of social media-assisted mental health promotion in adolescents in low-income countries, particularly in African countries such as Ethiopia. 39, 41 Little is known about the implementation outcome measures of such efforts, or about the factors influencing them. 42 Consequently, the present study was conducted to assess whether intervention using a mental health curriculum through guided social media use effectively improves adolescent mental health literacy. A hypothesis stating that an intervention using social media significantly improves the level of mental health literacy in adolescents compared to that in a control cohort (p<0.05) was tested. Evidence from direct beneficiaries is needed to understand the effectiveness of an intervention for health promotion to strive for equitable and resilient health systems for universal health coverage. 43 Understanding implementation outcome measures and the factors influencing them is necessary to help develop comprehensive intervention programs. Notwithstanding the importance of implementation outcome measures, little is known about them in the context of conducting such work in a low-income country. Consequently, this study was based on the theory of change aligning with the integrated theory of health behaviour change. The integrated theory of health behaviour change (ITHBC) states that enhancing knowledge, attitudes, and beliefs around mental health issues (mental health literacy) results in behaviour change towards improved mental health. 44, 45 The theory of change (ToC) explains situations and phenomena from the standpoint of conceptual and empirical evidence by describing the mechanisms by which the intervention is intended to work. 46 These theoretical backgrounds provide opportunities for reflection around intervention design, implementation, and outcome analysis. The theory of change framework (Figure 1 ) reflects the main target of intervention and the secondary objectives in the form of implementation outcome measures. This body of work primarily aimed to investigate the effectiveness of a mental health curriculum intervention delivered over social media to improve adolescent mental health literacy using effect size and difference-indifferences. The secondary analysis focused on the four measures of intervention implementation outcomes: the acceptability of the implementation, its appropriateness, its feasibility, and the satisfaction of the beneficiaries. Further, we assessed the perceived factors affecting the effectiveness of the intervention and these outcome measures. This study was conducted in public and private schools in urban areas of Dire Dawa province, Ethiopia, from the last week of June to the first week of August 2020. Dire Dawa city is diverse in the ethnicities, cultures, languages, and sociodemographic factors it hosts. In the pretest period, the research team recruited adolescents aged between 15 and 19 years who had smartphone and social media access. Adolescents below 15 years of age were excluded because they were not expected to have smartphones or internet access, or the familiarity with these that is required to engage fully with the intervention activity. 47 This study employed a quasi-experimental design, comparing the before-and-after difference between the intervention (treatment) group and the control group (Figure 2 ). Conducting a randomized control trial was unlikely to be feasible because it was costly and had complex ethical issues. As such, the assignment of participants to treatment and control groups was based on the need to avoiding contamination and bias during the intervention and to ensure recruiting a reasonable number of intervention and control participants. Furthermore, it was difficult to control all known and unknown factors that may influence the outcome of the intervention. Participants were recruited into geographically distant intervention and control groups based on the locations of their school and area of residency to reduce contact between the two groups during the intervention period and as such minimize contamination between groups. The power analysis for this study was based on the primary aim of comparing the change in mental health literacy scores from pretest to post-test and comparing this between intervention and control groups. The minimum sample size was based on a systematic review of interventions for mental health literacy, using on the reported effect size, δ=0.823 with a standard deviation, σ 2 =1.55. 14 The calculation assumed a confidence of level of 95% (Za/2 =1.96), power of 90% (Z β =1.282), and type 1 error α=5%. Using a validated sample size calculation for a quasi-experimental study, 48 a minimum of 146 participants (73 for each arm) were needed. However, the aim was to exceed that sample size. Of the 731 potential participants who met the initial inclusion criteria, 156 were recruited and allocated to the intervention and (n 1 = 77) and control (n 2 = 76) groups and completed the study (Figure 2 ). This sample size represents approximately 105% of the calculated minimum sample size. The intervention took the form of a mental health curriculum aimed at adolescents to improve their mental health literacy. It was adapted from the school mental health curriculum guides already prepared for African, Australian, and Canadian adolescents that aimed to improve mental health literacy and mental health first aid skills. 40, [49] [50] [51] [52] [53] [54] The intervention material was prepared and presented for the present study by summarizing the manual's texts, figures, and case vignettes in plain English. The contents of the intervention focused on how to seek information, awareness of common mental disorders and mental health problems, awareness of risk and protective factors, causes of mental illness, awareness of methods for self-help or for helping others, and awareness of the professional or informal help available, attitudes related to promoting recognition of mental health problems and/or appropriate help-seeking behaviour, and recognizing mental illnesses and sources of help. English is the medium of instruction used in Ethiopian junior schools and teaching above grade levels and is taken as a school subject starting from kindergarten. Furthermore, science subjects are given in English regardless of the area's native language. Thus, English was used for the intervention. A pilot intervention was conducted prior to the main intervention study. 47 Facebook and Telegram were reportedly the most commonly used social media platforms among Ethiopian adolescents, and similarly, in Dire Dawa. Hence, participants were registered to Facebook and Telegram groups. 47 A 45-megabyte internet package was provided to each participant for completing each intervention session as an incentive and to improve recruitment, compliance, and retention of study participants. For the intervention group, the summarized texts, figures, and case vignettes were posted on both social media platforms every 72 hours (three days) on working days (Monday to Friday) in eight sessions across approximately six consecutive weeks. Discussion between participants was allowed on the platform and a summary was provided at the end of each week (every two sessions). The control group received information with a similar frequency and timescale, but instead, the content was that of "secrets of successful students". Post-test data collection was conducted from all study participants (ie, both intervention and control groups) at the end of week seven, one week after the conclusion of the social media information delivery. All reasonable efforts were made to minimize the impact of errors and bias. Participants engaged in orientation and closing discussions throughout the study period. All data entry was double-checked by a data clerk. A thorough descriptive and inferential analysis was conducted objectively using standardized models. The study design controlled for the anticipated effect modifiers and confounding factors through the implementation of strict eligibility criteria. The study variables included sociodemographic characteristics (sex, school grade, and age), whether they were allocated to the intervention group (G i =1) or control group (G i =0), time point as pretest (t=0) or post-test (t=1), mental health literacy, and intervention implementation outcome measures and influencing factors. Mental health literacy was measured using the mental health literacy questionnaire, MHLq, 55 a freely available and validated tool used in adolescent populations. 17, [55] [56] [57] [58] [59] [60] [61] [62] [63] We further tested its reliability in the present study setting (Cronbach's alpha=0.834). The version of the MHLq used in this study was comprised of 33 items, with each item eliciting a response using a five-point Likert scale (1=strongly disagree, 2=slightly disagree, 3=neither agree nor disagree, 4=slightly agree, 5=strongly agree). Items comprised statements covering recognition, awareness, attitudes, and beliefs towards mental health issues. [16] [17] [18] [19] [20] The intervention implementation outcome measures were evaluated using a validated, questionnaire, 64 with items eliciting a response using a five-point Likert scale representing the level of agreement (1=completely disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, and 5=completely agree). Outcome measures were used to assess the extent of acceptability, feasibility, appropriateness, and satisfaction of the intervention. Acceptability was measured using 4 items encompassing whether the intervention met the participant's acceptance, the extent to which they liked it, whether they found it appealing, and whether they found it welcoming. Feasibility was measured using 4 items eliciting to what extent the intervention approach was implementable, possible, doable, and easy to use. Appropriateness was measured using 4 items covering whether the intervention seemed to fit its purpose, was suitable, applicable, and a good match. Satisfaction was measured using 7 items consisting of the quality of the intervention, whether it was wanted, whether it met the participant's needs, whether they would recommend it to others, the extent it helped them to deal with their problems, and the extent to which they would like to repeat the experience. Perceived factors influencing the effectiveness of the intervention and implementation outcome measures were measured in a similar manner using a 5-point Likert scale where participants were asked to indicate their level of access to resources, and their experience of personal and family-related challenges, so that these factors could be controlled for in the analyses. The baseline analysis was performed using descriptive statistics of the sociodemographic characteristics. Subsequently, we analyzed the effectiveness of the intervention in terms of the effect size and the difference-in-differences estimate. Finally, we analyzed the implementation outcome measures and related influencing factors. All statistical analyses were conducted using SPSS version 25. 65 Mental health literacy scores were treated as intervals following the measurement framework used by Stevens. 66 Negatively keyed items of the mental health literacy questionnaire (Q7, Q12, Q15, Q17, Q24, and Q26) were reverse scored and individuals' scores were computed. The analysis was restricted to subsets of groups and characteristics to address confounding factors and to determine which group of adolescent participants most benefited from the intervention, based on their background characteristics. We computed the mean differences associated with the intervention (posttest-pretest score) within each group using a paired t-test and then estimated the effect size. The effectiveness of the intervention program was expressed as effect size and difference-in-differences estimate. Significance was considered at a threshold of p≤0.05. Effect size is typically assumed to be the most important outcome measure in practical intervention and impact evaluation studies as recommended by Jacob Cohen. 67 significance of a result in evidence-based and field-based studies of interventions. 69, 70 Effect size was estimated using a 95% confidence interval and expressed using Cohen's d and Hedges's g. These expressions use a pooled standard deviation in the denominator; however, the latter uses n, and the former uses n-1. These two different methods of calculating the standard deviation reflect the mean score difference between the intervention and control groups. Hence, the absolute effect size is the difference between the mental health literacy scores in these two groups. The meaning of this intervention effect was interpreted from both the conventional approach and the comparison approach based on a broad conventional point of reference and comparison with previous studies, respectively. 68, 71 Difference-in-Differences Analysis The effectiveness of the intervention at improving mental health literacy was also estimated using a difference-indifferences analysis (Figure 3) . The difference-in-differences (DID) analysis is a regression analysis exploring the time dimension of the data to define the actual and counterfactual effects of the intervention. It was estimated by comparing the difference in the outcomes between the intervention and control groups over the six weeks of the intervention period. The difference that existed before the intervention program was addressed by assessing the presence of unobserved heterogeneity. Such an analysis approach has the advantage of controlling for differences arising from unobservable characteristics. Hence, the intervention effect was obtained by assessing for a difference between the groups, as shown in Figure 3 , which displays the steps taken in the analysis. After computing the before-to-after difference for each group [E (YIai-YIbi | Di=1) and E(YCai -YCbi|Di=0)], we computed the average intervention effect [E(YIai-YIbi | Di=1)-E(YCai -YCbi|Di=0)] where the subscript a denotes "after" and b denotes "before" the intervention, and the superscripts I and C indicate the intervention and control groups, respectively. Dgt =1 if unit g is exposed to the intervention in period t, and Dgt =0 if unit g is exposed to the control condition in period t. Y(1)gt represents the outcome of interest for unit g during period t under the hypothetical scenario g received the treatment during t. Y(0)gt is the outcome of the same unit and time under the alternative scenario in which g received the control condition. The estimation of the difference-in-differences regression relies on certain assumptions. The unobserved heterogeneity is assumed to be time-invariant, addressed by comparing the before and after states. The common trend assumption states that in the absence of the intervention or treatment, both the control and intervention groups experience the same measurement trend of outcome variables over time. Therefore, any difference from the trend observed in the intervention group is the effect of the intervention. Other assumptions considered for this analysis were the linearity of all relationships, the independence of observations, the absence of perfect collinearity, and the non-zero variances of independent variables; error terms were assumed to be zero and normally distributed. It is assumed that, in the absence of any intervention, the difference between the "intervention" and the "control" groups is constant over time (the two groups would have evolved in the same way in the absence of any intervention). Therefore, the average outcome in the control group is β0 during the pre-treatment period and β0+β2 in the posttreatment period. Likewise, the average untreated outcome in the intervention group is β0+β1 during the pre-treatment period and β0 + β1 + β2 in the post-treatment period, where β1 is different irrespective of the intervention effect. Thus, the difference in pre-treatment is normal while the difference in post-treatment can be assumed to be a causal effect (difference-in-differences, β3). In the difference-in-differences analysis, these assumptions restrict the scope of the possible confounders that are not entirely dependent on random assignment. It assumes that confounders varying across the groups are time-invariant and that time-varying confounders are constant. Hence, the difference-in-difference is estimated using a model that accounts for target variable measures, group membership, and time periods. The coefficient of the interaction term (β3) is an estimate of the treatment effect. Importantly, the timing effect in the intervention and control groups is not equal. The two groups are different at the start, and the observational characteristics change over time. Descriptive analyses were used for each outcome measure, and response. The mean score and differences across sexes, age groups, and school grades were analyzed using independent t-tests. Similarly, descriptive analyses were used for each factor that was being assessed for its effect on the implementation outcome measures and the potential effectiveness of the intervention program. Participant socio-demographic characteristics are presented in Table 1 . The mean ± standard deviation age of participants was 17.34 ± 1.31 years for the intervention group participants and 17.21 ± 1.36 years for the control group, and there was no statistically significant difference in any demographic characteristic between groups (ie, all p>0.05). Mental Health Literacy Pre-and Post-Intervention Pretest and post-test mental health literacy scores of participants in the intervention and control groups are presented by age group and sex ( Figure 4 ). The mean pretest and post-test mental health literacy scores of girls were slightly higher than those of the boys. The mean pretest and post-test mental health literacy scores of the control group were lower than those of the intervention group. The paired-samples t-test showed a significant change between mean pretest and post-test scores. Effect Size Estimate of the Intervention The paired samples t-test showed a significant change in the mean mental health literacy score between the pretest and posttest time points (p<0.01) for both the intervention and control groups (Table 2) . However, effect size estimates in the intervention group were greater than in the control group. The effect size for mental health literacy was greatest among participants aged 15-17 years, and for girls participants. The independent samples t-test on effect size showed a significantly greater effect in the intervention group compared to the control group (p=0.028). This effect size was higher in girls aged 15-17 years (Cohen's d=0.2.042, Hedges' g=2.045, p<0.001) than in boys of the same age (Cohen's d=0.141, Hedges' g=0.141, p>0.05). The effect size was lower, and non-significant in participants aged 18-19 years (p>0.05) ( Table 2 ). The regression model for the difference-in-differences analysis was stratified by sex, age group, and education level ( Figure 5 and Table 3 ). These models were used to estimate results for the intervention and it was estimated that the The analysis aimed to assess the self-rated implementation outcome measures and potential factors that may influence these implementation outcome measures. The majority of respondents either agreed or completely agreed about the acceptability, feasibility, appropriateness, and satisfaction of the implementation of the intervention ( Figure 6 ). However, ratings differed significantly across age groups, with a higher proportion of older adolescents reporting complete agreement for acceptability, appropriateness, feasibility, and satisfaction. Ratings did not differ significantly with sex (Table 4 ). More than 60% of participants responded that they completely agreed with the acceptability of the implementation (range 55-72%), the extent to which they liked it, found it appealing, and found its features welcoming. Ratings of feasibility, consisting of to what extent it is implementable, possible, doable, and easy to use, varied between 35 and 50%. Appropriateness, namely the east of fit, suitability, applicability, and the degree to which the implementation was a good match varied between 57 and 63%. Finally, satisfaction, the perceived quality of the implementation, the degree to which it is wanted, the degree to which it met the participant's needs, how likely they would be to recommend it to others, the extent to which it helped deal with their problems, and whether they would like to repeat it, also varied between 45 and 71%. With a minimum score of 5 and a maximum of 20, the mean and standard deviation ratings given by participants are separated and analysed by sex, age group, and school grade, along with testing the significance of any differences (Table 4 ). There was no significant sex difference in the scores given for the implementation outcome measures (p≥0.05). There were significant age differences in the ratings of implementation acceptability (p=0.003), The factors likely to influence implementation outcome measures were a lack of resources and personal and familyrelated challenges (Figure 7 ). Analysis focused on the proportion of participants who responded either in agreement or complete agreement with each of the items. Lack of access to the technological resources such as smart phone or computer (reported by 71.5% of participants), a lack of internet access (77.9%), or poor internet coverage (72.7%) were commonly reported. Personal influencing factors including a lack of digital skills and difficulty using technology (66.2%), lack of time to engage with the intervention (58.5%), unfamiliarity with social media (62.3%), and lack of motivation (57.2%) were less common but still reported by over half the respondents. Using social media incorrectly or spending time on unnecessary websites (57.2%), lacking confidence and trust in the privacy or confidentiality of the intervention (44.2%), and carelessness, laziness, or having little interest in social media (55.9%) were other factors reported by respondents in the sample. The family restriction of not being allowed to use social media or the internet (53.2%) was also reported as an influencing factor. The present study primarily investigated the effectiveness of a mental health curriculum intervention delivered over social media to improve adolescent mental health literacy using effect size and difference-in-differences. The secondary analysis evaluated the four measures of intervention implementation outcomes viz; the acceptability of the implementation, its appropriateness, its feasibility, and the satisfaction of the beneficiaries. Further, the perceived factors affecting the effectiveness of the intervention and these outcome measures were assessed. For the notion that mental health literacy is modifiable and is related to individuals' knowledge, beliefs, and awareness of mental health issues; 16-20 these findings are vital for enhancing outcomes such as self-efficacy, help-seeking intention, and health behaviour as well as fostering positive mental health and wellbeing. 3, 21 These changes are variously immediate and intermediate, with outcomes expressed as either proximal or distal outcomes. 45 The overall findings from this study are that the intervention effect was greater than zero, and that the magnitude of the effect can be expressed using the standardized metrics of effect size and difference-in-differences estimate. Both the effect size and the difference-in-differences analysis showed a significant effect of the intervention (p<0.05) compared to a control condition. Nevertheless, the magnitudes of these metrics varied with sex and age. The implementation of the intervention was found to be acceptable, appropriate, feasible, and satisfactory despite several influencing factors that varied with the socio-demographic characteristics of the participant. Effect sizes for quantitative intervention analyses of outcomes can be interpreted using a variety of approaches, including convention and comparison, 68, 69, 71, 72 with much evidence available in the published literature advocating strongly in favor of using a convention approach for studies such as this one. According to the conventional approach, effect sizes expressed primarily using Cohen's d or Hedges' g values can be categorized into small (below 0.2), medium (0.2 to 0.5), or large (0.5 or higher). 67 The estimated intervention effect size estimate of this study was greater than zero; and the intervention program has potentially improved the mental health literacy of adolescents. The effect size observed in the present study would be conventionally classified as medium-to-high, varying by both sex and age. Despite there being little evidence advocating the comparison approach, the effect size seen in the present study (d=0.429-0.767, p<0.05) was consistent with previously published meta-analysis findings on universal and selective intervention studies to improve mental health literacy using a mental health curriculum (d=0.541-0.774, p<0.05). 14 It was also consistent with a result of another metal analysis of a digital interventions, social media included to improve positive mental health outcomes among young people involving supervision (Cohen's d=0.52) and without involving supervision (Cohen's d = 0.33). 73 Results from regression models that stratified data by sex and age group have reported the coefficient of the estimated difference-in-differences as being reflective of an intervention program having a higher and more significant impact (p<0.001). The level of mental health literacy dramatically increased with the intervention despite sex and age differences. Hence, the intervention implemented using social media was shown to improve adolescents' mental health literacy significantly, in support of the original hypothesis. Previous qualitative evidence has shown that health interventions using online technology have been able to overcome multiple physical and logistical barriers and challenges. 1 Findings from the present study were consistent with those of previous studies despite little quantitative evidence being available. 1, 74 Socio-demographic and socio-economic factors have been shown to influence the effectiveness and outcomes of health interventions. 75 A growing number of studies have revealed that gender dynamics affect outcomes aiming to introduce child and adolescent health-related-behavioural change. 76 As such, the regression models were stratified after multiple observable confounders and had been controlled for by assessing the significance of the variability in mental health literacy across these groups. For example, an earlier study conducted in the Ethiopian population showed that girls had slightly higher mental health literacy scores than did boys 42 consistent with other studies conducted in Portugal, Australia, and the USA. 55, 59, 77 Further, the effectiveness of the mental health literacy intervention conducted in this study varied across sex and age groups, independent of the methods used to measure it. The difference-in-difference estimate was higher among girls than boys, and the effect size was higher for participants aged 15-17 years (p<0.05). Age is known to be a factor associated with health intervention outcomes. 78 From the present study, effect of the intervention on the mean score of mental health literacy estimated with difference-in-differences (DID) was significantly higher for older adolescents than for younger adolescents. The probable explanation might be that older adolescents spend more time and participate with more attention than the younger adolescents. The younger adolescents may have shorter attention spans and lower focusing. The differences in the cognitive and behavioural skills of adolescents have also been shown to influence such intervention outcomes. 27 The findings of this study contribute greatly to the existing literature describing interventions to improve the mental health literacy of adolescents, as well as to the literature assessing implementation outcome measures. First, such quantitative evidence in the form of standardized effect sizes and difference-in-difference estimates provides an opportunity to demonstrate the practical significance and impact of digital and web-based interventions promoting adolescent mental health literacy instead of only reporting the statistical significance. Second, effect sizes and differencein-difference analyses allow researchers to draw meta-analytic conclusions across studies. Third, these effect size and difference-in-difference findings contribute to the design and planning of new studies, including the estimation of required sample sizes with the desired likelihood. 70 Participant ratings of the perceived outcome measures of the intervention implementation process were significantly different across age groups (p<0.05), with older adolescents reporting closer agreement with the implementation outcome measures. This effect of age may explain the difference between technology and internet access and challenges related to digital literacy. Older adolescents may find it easier to access the internet and may have better digital literacy. These findings were consistent with the findings of other similar studies. 37, 79 Significant variation in observable measures was seen with the demographic characteristics of the participants. Participant sex has been shown to be one of the main factors predicting internet use behaviours, 79 with girls spending more time on social media, and being more receptive to digital interventions to behaviour change. 79 However, in the present study there was no significant sex difference in implementation outcome measure ratings, which is inconsistent with previous reports. 37, 79, 80 Detailed qualitative explanations are needed to interpret such quantitative findings. This study assessed several potential factors that were suggested to influence the effectiveness of the intervention and the intervention implementation outcome measures. Unavailability or lack of access to resources (technology or internet access), personal factors (confidence, concerns about privacy, digital skills, frequency of interaction, motivation, using other unnecessary websites, and other personality traits) and family-related factors (being prohibited from accessing social media or the internet) were perceived factors affecting effectiveness and implementation outcome measures. This finding was consistent with the outcomes of related studies of digital or web-based interventions, [81] [82] [83] [84] [85] whereby behavioural factors, 81 digital or internet literacy, 82 lack of money or internet access, 83 adherence, 84 and participant engagement, 85 were reported to be associated with the effectiveness of an intervention. Participant interest and motivation, as well as having an understanding of their concerns, emerge as key factors for effective digital or web-based intervention programs. 86 The present study has both strengths and limitations. The study used a before-and-after design with the control mechanism utilizing both a control group of participants and a pretest/post-test comparison to assess the effect of the intervention. This study was conducted in a real-life setting and with a low implementation cost, avoiding many confounds to internal validity. 87 However, like any quasi-experimental study, these findings may be subject to biases. The conventional interpretation of the effect size has limitations due to the absence of reliable empirical evidence of the effects of such an intervention at the population level. Further, due to the absence of much existing published evidence, subtle differences in study designs, scales, and measurement instruments may limit the validity of a comparison approach. 68, 69, 71, 72 Reliability and scalability might be confounded by deviation from pragmatic conditions, offering incentives, increased handholding, selection bias, confined study environment, and lack of follow-up assessments. The present study shared factors related to the disparity of internet access, difference in interest and internet literacy, negative feeling and doubts about online information among anticipated limitations of studies for online web-based interventions. 88, 89 The study population represents few subsets of adolescent population who reported they had internet access limiting generalizability of findings to other populations. Finally, the study was conducted during the unprecedented COVID-19 pandemic. As such, participants were at home during the social media intervention, likely having an impact on the intervention outcomes measured, either positively or negatively. These findings contribute novel and innovative approaches to assessing the effectiveness of the implementation of an intervention, as well as directly assessing the effect of the intervention itself and factors associated with it. The intervention used social media as a channel of communication for mental health promotion to improve adolescent mental health literacy. The findings of the study support the assertion that an intervention using an evidence-based mental health curriculum delivered over social media can significantly improve the mental health literacy of adolescents. We found that the implementation of the intervention was considered acceptable, appropriate, feasible, and satisfactory. The programme itself has practical scalability and sustainability despite some factors limiting its effectiveness in some individuals. The effectiveness of the intervention as reflected in the effect size and in the difference-in-differences estimate differed with both sex and age. The factors found to limit the success of the intervention were the availability of resources, and personal and familyrelated behaviours indicating the need to consider these challenges in the future when scaling up such an intervention to utilize emerging digital and online platforms for the promotion of mental health literacy. Although these findings are preliminary, they highlight potential areas for improvement in the application of adolescent mental health promotion using digital and online technologies including, most importantly, social media. Future interventions should consider other factors contributing towards good mental health, as well as other socio-demographic and contextual factors. DID, difference-in-differences; ITHBC, the integrated theory of health behaviour change; ToC, theory of change. The study complied with the International Ethical Guidelines for Health-Related Research Involving Humans, prepared by the Council for International Organizations of Medical Sciences in collaboration with the World Health Organization. 90 Ethical approval was obtained from KIIT University Health research institutional ethics review committee (Reference:KIIT/KIMS/IEC/63/2019) and Haramaya University institutional health research review committee (Reference:00.H.M.S./10.0/3763/2020. The Haramaya University health research review committee was the only recognized health research review committee mandated to approve health research in Dire Dawa province and Eastern Ethiopia during the study period, and as such reviewed the study independently from the authors' affiliation to Haramaya University. All participants and their parents had the study design and objectives explained to them online and gave written informed consent. The principals of the participating schools also had the study explained to them in person and gave written informed consent. Help-seeking measures and their use in adolescents: a systematic review help-seeking measures and their use in adolescents: a systematic review A national child mental health literacy initiative is needed to reduce childhood mental health disorders How adolescents search for and appraise online health information: a systematic review Harnessing social media for health promotion and behavior change Klainin-Yobas P. Mental health literacy levels Adolescents: health risks and solutions Mental health in Ethiopia: an exploratory study of counseling alignment with culture Multidimensional child deprivation in Ethiopia Mental health literacy: focus on developing countries Help-seeking behaviour: a concept analysis School Based Mental Health Promotion in and adolescents with StresSOS using online or face-to-face interventions: study protocol for a randomized controlled trial within the ProHEAD Consortium Research priorities for eight areas of adolescent health in low-and middle-income countries Promoting mental health. Concepts, emerging evidence, practice. A summary report Universal and selective interventions to promote good mental health in young people: systematic review and meta-analysis What is good mental health? A scoping review Youth mental health first aid : a description of the program and an initial evaluation Improving mental health literacy as a strategy to facilitate early intervention for mental disorders Why we need the concept of "Mental health literacy Mental health literacy": a survey of the public's ability to recognise mental disorders and their beliefs about the effectiveness of treatment Research on mental health literacy: what we know and what we still need to know Adolescent health interventions: conclusions, evidence gaps, and research priorities San Francisco: National Adolescent Health Information Center (NAHIC) Effectiveness of information and communication technologies interventions to increase mental health literacy: a systematic review Adolescence and the next generation Prevalence and risk factors of psychiatric disorders in early adolescence: 2004 Pelotas (Brazil) birth cohort Social development in adolescence: brain and behavioural changes. PQDT -UK Irel Health literacy in childhood and youth: a systematic review of definitions and models Pathways to adolescent health: early intervention Health literacy and adolescents: a framework and agenda for future research What does health literacy mean to children? Contemp Nurs Sustained improvements in students' mental health literacy with use of a mental health curriculum in Canadian schools Effectiveness of a web-based screening and brief intervention with weekly text-message-initiated individualised prompts for reducing risky alcohol use among teenagers: study protocol of a randomised controlled trial within the ProHEAD consortium Mobile phone-based peer support in the prevention of postpartum depression among adolescent mothers: a pilot randomized controlled trial Efficacy and cost-effectiveness of two online interventions for children and adolescents at risk for depression (E.motion trial): study protocol for a randomized controlled trial within the ProHEAD consortium What motivates health information exchange in social media? The roles of the social cognitive theory and perceived interactivity Functional assessment of social media in child and adolescent psychiatry Feasibility and acceptability of "cuida tu Ánimo" (Take care of your mood): an internetbased program for prevention and early intervention of adolescent depression in Chile and Colombia Adolescent and young adult use of social media for health and its implications Perceived benefits and risks of social media: Ethiopian secondary school students' perspectives Mental health first aid: an international programme for early intervention The role of social media in preventing and managing non-communicable diseases in low-and-middle income countries: hope or Hype? Health Policy Technol Mental health literacy of adolescents and the effect of socio-demographic characteristics: a cross-sectional study What does universal health coverage mean? Testing the integrated theory of health behaviour change for postpartum weight management Integrated theory of health behavior change: background and intervention development Theory of change analysis: building robust theories of change A quasi-experimental and guided social media intervention to improve mental health literacy level of urban school adolescents in Ethiopia: a detailed study protocol Sample size calculation Mental health & high school curriculum guide (Guide v.3): understanding mental health and mental illness Mental health literacy in secondary schools. A Canadian approach Successful application of a Canadian mental health curriculum resource by usual classroom teachers in significantly and sustainably improving student mental health literacy Mental health & high school curriculum guide African school mental health curriculum: understanding mental health and mental illness Van Bortel T. WHO guidance on mental health training: a systematic review of the progress for non-specialist health workers Development and psychometric properties of a new questionnaire for assessing mental health literacy in young people Measuring mental health literacy-a review of scale-based measures A narrative literature review of child-focused mental health literacy attributes and scales Mental health literacy and mental health status in adolescents: a population-based survey Mental health literacy in young adults: adaptation and psychometric properties of the mental health literacy questionnaire Mental health literacy measures evaluating knowledge, attitudes and help-seeking: a scoping review Mental health promotion in school context -validation of the ES ' COOL scale for teachers Positive mental health literacy: development and validation of a measure among Norwegian adolescents Review of mental health promotion interventions in schools Psychometric assessment of three newly developed implementation outcome measures IBM SPSS Statistics for Windows, Version 25.0 How to analyze Likert and other rating scale data Statistical Power Analysis for the Behavioral Sciences The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases Interpreting effect sizes of education interventions Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs Standardized or simple effect size: what should be reported? Beyond small, medium, or large: points of consideration when interpreting effect sizes What works and what doesn't work? A systematic review of digital mental health interventions for depression and anxiety in young people Promoting Help-seeking using E-technology for ADolescents with mental health problems: study protocol for a randomized controlled trial within the ProHEAD Consortium Classifying the population by socioeconomic factors associated with support for policies to reduce social inequalities in health An evidence review of gender-integrated interventions in reproductive and maternal-child health Adolescent mental health literacy: young people's knowledge of depression and social anxiety disorder The independent effect of age groups on the effectiveness of lifestyle intervention Adolescent's preferred modes of delivery for mental health services The Brief Intervention for School Clinicians (BRISC): a mixed-methods evaluation of feasibility, acceptability, and contextual appropriateness Factors that influence young people' s mental health help-seeking behaviour: a study based on the Health Belief Model Engagement with online psychosocial interventions for psychosis: a review and synthesis of relevant factors Do we still have a digital divide in mental health? A five-year survey follow-up Review of predictors of, and reasons for, adherence to online psychological interventions Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis Acceptability study of "ascenso": an online program for monitoring and supporting patients with depression in chile Quasi-experimental study designs series-paper 4: uses and value Web 2.0 for health promotion: reviewing the current evidence Consumer perceptions of using social media for health purposes: benefits and drawbacks International Ethical Guidelines for Health-Related Research Involving Humans: prepared by the Council for International Organizations of Medical Sciences (CIOMS) in Collaboration with the World Health Organization (WHO) This includes studies and reviews conducted by multidisciplinary teams as well as research which evaluates the results or conduct of such teams or healthcare processes in general. The journal covers a very wide range of areas and welcomes submissions from practitioners at all levels, from all over the world. The manuscript management system is completely online and includes a very quick and fair peer-review system We would like to acknowledge Dire Dawa Education bureau, school administrators, principals, and teachers in Dire Dawa city in Ethiopia for their assistance during data collection. The authors report no conflicts of interest for this work and declare that they do not have any financial interest or benefit that has resulted from the direct applications of this research study.