key: cord-0023223-4j2yzqu4 authors: Doumas, Diana M; Midgett, Aida; Myers, Valerie; Buller, Mary Klein title: Usability of a Technology-Based Bystander Bullying Intervention for Middle School Students in Rural, Low-Income Communities: Mixed Methods Study date: 2021-10-26 journal: JMIR Form Res DOI: 10.2196/32382 sha: 8a3296d37684fcd57bae5770a64a0c2cff771527 doc_id: 23223 cord_uid: 4j2yzqu4 BACKGROUND: Students who are targets of bullying and who witness bullying are at high risk for negative mental health outcomes. Bystander training is essential to reduce bullying and the negative associated consequences for targets and bystanders. Resources necessary for program delivery, however, pose significant barriers for schools, particularly those in rural, low-income communities. Technology-based programs can reduce health disparities for students in these communities through cost-effective, easy-to-disseminate programming. OBJECTIVE: The aim of this study is to conduct usability testing of a bystander bullying web app prototype, STAC-T (technology-based STAC, which stands for the 4 bystander strategies Stealing the Show, Turning it Over, Accompanying Others, and Coaching Compassion) as an initial step in the development of a full-scale STAC-T intervention. Objectives include assessing usability and acceptability of the STAC-T prototype, understanding school needs and barriers to program implementation, and assessing differences in usability between school personnel and students. METHODS: A sample of 16 participants, including school personnel and students recruited from 3 middle schools in rural, low-income communities, completed usability testing followed by a qualitative interview. Descriptive statistics, 2-tailed independent sample t tests, and consensual qualitative research were used to assess usability and program satisfaction and to extract themes related to acceptability, feasibility, needs, barriers, and feedback for intervention development. RESULTS: Usability testing indicated that the app was easy to use, acceptable, and feasible. Both school personnel (mean rating 89.6, SD 5.1) and students (mean rating 91.8, SD 7.0) rated the app well above the standard cutoff score for above-average usability (ie, 68), and both school personnel (mean rating 5.83, SD 0.41) and students (mean rating 6.10, SD 0.57) gave the app high user-friendliness ratings (0-7 scale, with 7 as high user-friendliness). The overall ratings also suggested that school personnel and students were satisfied with the program. Of the 6 school personnel who said they would recommend the program, 1 (17%), 4 (66%), and 1 (17%) rated the program as 3, 4, and 5 stars, respectively; 80% (8/10) of students said they would recommend the program; and 60% (6/10) and 40% (4/10) rated the program as 4 stars and 5 stars, respectively. Qualitative data revealed that school personnel and students found the STAC-T app to be useful, user-friendly, and relevant, while providing feedback related to the importance of digital learning activities that engage the user. Data from school personnel also indicated positive perceptions regarding program feasibility and probability of program adoption, with the most significant barrier being cost, suggesting the importance of considering the financial resources available to schools in rural, low-income communities when setting the price point for the full-scale STAC-T intervention. CONCLUSIONS: This study provides support for the full-scale development of the STAC-T app and provides key information for revision to enhance used engagement. TRIAL REGISTRATION: ClinicalTrials.gov NCT04681495; https://clinicaltrials.gov/ct2/show/NCT04681495 Bullying is a national public health issue in the United States, with 20.2% of students aged 12 to 18 years reporting being bullied at school and 15.3% reporting being cyberbullied in the previous year [1]. Both bullying and cyberbullying peak in middle school, with 28% of students reporting being a target of school bullying, and 33% reporting being cyberbullied [2] . Among middle school students, bullying victimization is associated with a variety of mental health problems, including somatic symptoms [3] [4] [5] [6] [7] , anxiety [6] [7] [8] , social anxiety [9] [10] [11] , depression [5] [6] [7] [8] [9] , suicidal ideation, and suicide attempts [8] . Similarly, being a target of cyberbullying is associated with internalizing symptoms and suicidal ideation [12] [13] [14] . Thus, the development of effective interventions for middle school students is important for reducing bullying and its negative consequences. Youth in rural [15] [16] [17] and low-income [18] [19] [20] communities are particularly vulnerable to school bullying and cyberbullying. US national data indicate a higher prevalence of school bullying victimization among students in rural areas (23.8%) compared with students in urban areas (19.9%) [1] . Furthermore, students in rural areas are 3% to 5% more likely to report bullying their peers [17] . Students at the lowest income levels report the highest rates of bullying (21%-26.6%) compared with students at higher income levels (16.6%-19.8%) [1]. Students from low-income households also report the highest rates of physical bullying, bullying-related injury, and cyberbullying, yet they are the least likely to report bullying to an adult [1] . In addition, low-income students report the highest rates of consequences associated with bullying victimization, including negative effects on school work, relationships, feelings about oneself, and physical health [1]. Among middle school students attending schools in rural, low-income communities, bullying victimization is associated with poor school relationships, negative school experiences [19] , and depression and anxiety [19, 21] . These data reveal significant mental health disparities for youth in rural and low-income communities. Negative consequences are not only limited to targets of bullying but also extend to students who witness bullying [7, 22] and cyberbullying [23] [24] [25] [26] as bystanders. Students who witness school bullying are at increased risk for mental health problems, including somatic symptoms [7] , sadness [27] , helplessness [7, 27] , isolation, guilt [28] , depression, anxiety [7, 22] , and suicidal ideation [7] . Similarly, witnessing cyberbullying is associated with anxiety, depression [23, 25, 26] , and somatic symptoms [25] , even when controlling for the effects of witnessing school bullying [23, 25] . In addition, bystanders who intervene in bullying situations experience higher rates of anxiety and depressive symptoms than students who remain passive [29, 30] , possibly owing to using maladaptive behaviors to defend targets [30] . Research indicates 80% of students report witnessing bullying [7] and more than 50% witnessing cyberbullying [31] . Therefore, developing interventions to teach student bystanders how to appropriately intervene is important, as most students are bullying bystanders. Results from a recent meta-analysis indicate that training student bystanders to intervene as defenders is an important component of comprehensive school-based bullying programs [32] . Although up to 80% of students report witnessing bullying [7] , only 20% to 30% intervene [33] ; for cyberbullying, as few as 10% may intervene [34] . Researchers have demonstrated that when bystanders are trained to act, it not only reduces school bullying but also leads to improved mental health in bystanders [35] [36] [37] [38] [39] . Bystander disapproval of cyberbullying acts can also effectively limit its prevalence [40] . Furthermore, self-efficacy for defending is positively related to intervening in both bullying [41] and cyberbullying [42] ; however, few comprehensive, school-based programs incorporate bystander training. In addition, there are several barriers to implementing school-based bullying prevention. Available interventions require substantial resources, including demands on teachers, limited access to training, lack of funding, and few school mental health professionals [43] , reducing access and posing significant barriers to implementation. Rural, low-income communities face economic disparities, creating further obstacles to implementing these programs [44] . These barriers include a lower tax base for funding programs, training costs inflated by transportation needs related to bringing in expert trainers, frequent staff turnover with limited resources to reestablish expertise, school closures, staff overload and burnout, and lack of program advocates and local expertise in bullying prevention [16] . Therefore, brief programs that focus on bystander training and reduce barriers for implementation are needed to reduce bullying and its negative consequences among middle school students in rural and low-income communities. National data in the United States indicate that the number 1 barrier to implementing educational technology is that school districts do not have the funding [45] . Technology-based interventions may be one way to increase access and reduce implementation challenges specific to rural and low-income communities [16] . Computer access and internet connectivity in middle school is high; 86% of students have computers in school [46] , and only 6% of schools do not meet federal connectivity marks for broadband capacity [47] . Notably, low-income students have similar access to school computers compared with middle to high income peers [46] . Some rural areas have higher rates of poor internet connectivity; however, to assist in reducing this disparity, eligible rural and low-income schools receive discounts ranging from 20% to 90% for telecommunications services, including internet access and broadband services through the Telecommunications Act of 1996 [48] . Grants are also available through the US Department of Agriculture to build broadband infrastructure in rural areas [49] , and more recently, the US Department of Education for discounted internet service for low-income households [50] . Thus, most students in rural, low-income communities have access to the infrastructure needed for technology-based programs. STAC, which stands for the 4 bystander strategies Stealing the Show, Turning it Over, Accompanying Others, and Coaching Compassion, [51] is a brief, stand-alone bystander intervention developed to reduce barriers to program implementation for middle school students. STAC was originally designed as a 90-minute in-person intervention, including didactic and experiential role-play components, as well as 2 booster sessions. The didactic training includes education about bullying and cyberbullying, consequences of bullying, bystander roles, and a description of the 4 STAC strategies: (1) stealing the show-using humor or distraction to interrupt the bullying situation, removing the attention away from the target; (2) turning it over-informing an adult about the bullying and asking for help; (3) accompanying others-befriending or providing supporting the targeted student; and (4) coaching compassion-gently confronting the perpetrator to increase empathy for target. The experiential component includes role-plays in which students practice using the STAC strategies in hypothetical bullying situations. Students also participate in 2 biweekly 15-minute booster sessions to reinforce their skill acquisition. Research indicates that the STAC intervention is effective in reducing bullying [52, 53] and negative mental health outcomes for bystanders [35] [36] [37] [38] [39] . Researchers have also adapted the STAC intervention to be culturally appropriate for middle school students in rural, low-income communities [54] [55] [56] with similar positive effects on bullying reduction [54, 57] and improved mental health [54, 58] . Most bullying programs are comprehensive school-wide interventions that require significant resources, creating implementation barriers for schools. Although the STAC intervention reduces some of these barriers, in-person interventions pose implementation challenges, particularly for schools in rural, low-income communities. To increase access for these schools, we propose to translate the STAC intervention into a technology-based format. As a first step, we conducted a needs assessment with the goal of understanding needs and perceived program implementation challenges for schools in rural, low-income communities to provide information on how to best serve students in these schools [59] . Findings from interviews and focus groups with key school personnel (ie, administrators, teachers, and school counselors) from 3 middle schools in rural, low-income communities indicated a strong interest in a technology-based bullying intervention. Participants also described positive conditions for implementation, including support from the administration and technology readiness of their schools. Participants identified implementation challenges, such as time and financial resources. In addition, participants provided feedback related to translating the intervention into a technology-based format, including the importance of activities that require user input to increase user engagement. Overall, the findings supported the need for the proposed technology-based STAC (STAC-T) intervention and provided feedback on challenges that need to be addressed for successful adoption and sustained implementation in middle schools in rural, low-income communities. The STAC-T web-based app is intended to shift program delivery from an in-person intervention to a technology-delivered intervention, thereby increasing accessibility and eliminating implementation barriers. STAC-T will also allow large groups of students to be trained simultaneously, accessed from a computer, tablet, or smartphone. The STAC-T app is designed to be a modular program that can be customized to meet the needs of individual schools. The app will allow students to customize their experience by selecting avatars and bullying scenarios. Assessment and personalized feedback components are also infused through the program to individualize the user experience to promote behavior change [60, 61] . The innovative, user-centered design of the STAC-T app will be inherently sensitive to the cultural needs of students and identify personally appropriate strategies. The STAC-T app addresses both bullying and negative mental health outcomes for targets and bystanders through an evidence-based approach that is adapted for a broader audience and uses technology to effectively implement bullying prevention. The success of the STAC-T design can be evaluated through usability testing. Usability testing is an important step in the development of technology-based programs, providing information from end users on what works and gaps in how the program functions [62] , as well as the acceptability and relevance of program content. Usability ultimately affects the likelihood that the program will be adopted [63] and thus is a critical component of technology-based intervention development [64] . Assessing usability with school personnel and students is also important, as school personnel make decisions about program adoption [65] , and students, as end-users, need to understand and respond positively to the program to benefit from the content [62, 66] . This study aims to evaluate the usability and acceptability of the STAC-T prototype to inform full-scale development of the STAC-T intervention app. To achieve this aim, we implemented usability testing with key stakeholders (ie, school personnel and students) at 3 middle schools in rural, low-income communities (N=16). The study had the following objectives: (1) to assess the usability and acceptability of the STAC-T prototype, (2) to understand school needs for and barriers to program implementation, and (3) to assess the differences in usability between school personnel and students. We used a mixed-methods design to assess the usability and acceptability of the app prototype, as well as ways to improve the app, likelihood of program use, and potential implementation barriers. Participants were key school personnel (ie, administrators, teachers, and school counselors) and students were recruited from 3 middle schools in rural, low-income communities in the Northwest region of the United States. The schools were selected based on previous and ongoing research partnerships. The 3 schools were Title 1 schools, with 52.9% (339/641), 68.98% (725/1051), and 98.8% (663/671) of the student population at the 3 schools being below the poverty line. A total of 6 school personnel and 10 students were recruited. Among the school personnel, the age ranged from 28 to 59 years (mean 42.0 years, SD 10.8 years) and the majority were women (5/6, 83%) and White (6/6, 100%). For students, age ranged from 11 to 14 years (mean 12.2 years, SD 0.9 years), with 40% (4/10) in grade 6, 30% (3/10) in grade 7, and 30% (3/10) in grade 8, and the majority were girls (8/10, 80%) and White (7/10, 70%). We used a multitheoretical framework to guide prototype production. We translated the STAC in-person intervention into a technology framework guided by Persuasive System Design, a theoretical guide for translating clinical aims to health-related technology frameworks [67] [68] [69] . Instructional objectives and design of STAC-T were guided by the existing STAC intervention for rural and low-income students, and input from an expert advisory board, school personnel, and students. The STAC-T prototype was developed using AGILE programming, a collaborative and incremental programming methodology [67] [68] [69] . The prototype was functional on all web browsers that support HTML5 and was built on a full stack web application using HTML or JavaScript as the main interface. React.js was used as the front-end framework. The system was accessible on desktop computers, iOS, Android tablets, and smartphones. Design ideas were created in written form, combined with scripts, flowcharts, and storyboards, before creating actual images and authoring elements. Programmers produced the STAC-T prototype, alpha-and beta tested it in-house for stability and code errors, tested it for usability, and revised it following an iterative, agile production process. The initial prototype was created with clickable wireframes that showed the progression of content with still graphics ( Figure 1 ). In addition to Persuasive System Design, User-centered Design [70, 71] , and the ADDIE (Analyze, Design, Develop, Implement, and Evaluate) model [72] [73] [74] , we followed a user interface and instructional design approach to develop the STAC-T prototype. User input was solicited through iterative cycles, with adjustments made based on feedback [75] , resulting in enhanced user experience and more reliable and effective results [76, 77] . Design elements such as space (colors and visual space), components (characters and objects), and mechanics (actions) were determined for program features (ie, activities and games). Consistent with research on mobile health strategies for adolescents [78] [79] [80] , the program development approach emphasized gaming as a teaching strategy, and interactive components were the centerpiece of the STAC-T intervention. STAC strategy practice required students to select between 2 avatars, view a bullying event, select actions to operationalize the STAC strategy, view the avatar enacting the selected action, and receive feedback on its effectiveness. To reward learning and bolster adherence, badges (visual reward icons eg, Stealing theShow badge) were included for intermittent awards to encourage user engagement. The STAC prototype content comprised 3 overarching modules: (1) What is Bullying? Users were presented with background information on bullying, including bullying definitions (ie, physical, verbal, relationship, and cyberbullying), bullying facts and statistics, characteristics of students who bullied, and negative consequences of bullying; (2) What are Bystanders? Users were taught what a bystander is, and how bystanders affect bullying outcomes. This module explained the 4 bystander roles: (1) Assistants: those who intentionally help the bully; (2) Reinforcers: those who are not directly involved in hurting another student, but encourage the bully by standing around, laughing, or watching quietly; (3) Outsiders: those who do not take sides while witnessing bullying; and (4) Defenders: those who do something to stop the bullying situation or help the target in some way, and (3) STAC strategies: Users were introduced to 4 STAC strategies: (1) stealing the show, (2) turning it over, (3) accompanying others, and (4) coaching compassion. The module also included the STAC strategy practice using avatars selected by the user. Iterative focus groups (3 rounds) with middle school students (N=21; 14 girls and 7 boys) attending schools in rural, low-income communities informed prototype development. Groups ascertained whether the app concept was easily understood and engaging and identified essential features for a successful prototype. The first round (n=10) was conducted with students who had been trained in the in-person STAC intervention. The second (n=6) and third (n=5) rounds were conducted with students who were naive to the STAC program. Group 1 provided feedback on the translation of the in-person STAC program content to a technology-based format for prototype 1. Group 1 reported that the content was realistic, expressed interest in a web translation, and made suggestions for modifications for web translation that were incorporated into prototype 1. For group 2, prototype 1 was very well received; participants thought the content was useful, important, and understandable and were enthusiastic about using the program. Group 2 participants suggested more interactivity (eg, short clips, videos, thought bubbles, graphs, and cartoons) to increase engagement. Group 3 evaluated prototype 2, reporting that they liked the program content and ease of navigation. Participants liked the sketches and discussed changes to characters (eg, wear hoodies instead of clothes with buttons) and surroundings (eg, school bus looked too clean) to more closely match the middle school environment. Input from the focus groups informed the development of the final prototype used in this study. Participant recruitment and usability testing occurred during the spring of 2020. All research procedures were approved by the University Institutional Review Board and by the School District or Administration. The researchers provided the school counselor from each school with an email script describing the purpose and procedures of the study. School counselors were also provided with rubrics developed by the research team to identify key school personnel and students who demonstrated the following characteristics assessed by the rubric: school personnel: (1) caring for students, (2) desire to be a positive influence on school climate, (3) approachable to students, (4) caring about addressing the problem of bullying, and (5) leadership qualities; and students: (1) leadership, (2) maturity, (3) responsibility, (4) caring toward others, (5) influence, and (6) a desire to be a positive influence on peers. For each item, school personnel and students were assessed on a 3-point scale, which included the ratings of yes and somewhat to no for each item described above. School personnel and students who scored yes or somewhat on all inclusion criteria were eligible to participate. The school counselor used the rubric to identify and contact key school personnel and students and then used the script to invite them to participate in the study. Usability testing and interviews were conducted remotely. Researchers obtained informed consent for school personnel and parental consent and student assent for students and collected demographic information from participants immediately before usability testing. Participants interacted with the STAC prototype app by selecting an avatar and environment to tailor content and talked aloud while completing the tasks and identifying problems and solutions attempted. Researchers and users were on videoconference and shared screens. Researchers could see what participants were doing, and they were able to communicate with each other in real time. The researchers observed the users as they worked through the tasks and asked questions to gather more data. Participants were asked to complete a brief usability survey followed by a semistructured interview protocol. All participants were asked to provide information about their perceptions of (1) program utility, (2) relevance and appropriateness of program content, and (3) ways they would improve the program. School personnel were also asked about (1) their thoughts on implementation feasibility, (2) likelihood of school program adoption, and (3) barriers to program use. All individual interviews lasted for 1 hour and were audio-recorded. School personnel received a web-based US $50 Amazon gift card as an incentive for participation in the usability testing and individual interview. There were no incentives for student participants. Participants self-reported their age, ethnicity, race, and gender. Students also reported their grade level. Usability was assessed using the System Usability Scale (SUS) [81] . The SUS is a widely used 10-item validated tool used to measure the usability and acceptability of technology-based programs. Responses were measured on a 5-point Likert scale ranging from 0 (strongly disagree) to 4 (strongly agree). Items were summed, and the total was multiplied by 2.5, creating an overall SUS score ranging from 0 to 100. An SUS score of ≥68 was considered above average [82] . One item was used to assess the user-friendliness of the program. Participants were asked to rate the user-friendliness with the question: Overall, I would rate the user-friendliness of this program as... with a 7-point scale ranging from 0 (worst imaginable) to 7 (best imaginable). Two items were used to assess the program satisfaction. Participants were asked the question, "Would you tell your friends/colleagues to use the program?" with response choices yes, no, and don't know. Participants were also asked how many stars they would give the program (1 star being the lowest and 5 stars being the highest). Following the usability testing sessions, participants were asked a series of open-ended questions about the utility and relevance of the app prototype, as well as ways to improve the app, likelihood of program use, and potential implementation barriers. School personnel and students were asked the following: (1) Please talk about your perception of how useful this program could be to helping to address the problem of bullying at school, (2) Please share your thoughts on whether you think the content of this program is relevant and appropriate for students at your school and your community, and (3) Can you talk about ways that you would improve the program? School personnel were asked the following: (4) What are your thoughts on how practical or workable you think it would be to use this program at your school? (5) What do you believe is the likelihood that your school would use this intervention? and (6) What, if anything, would keep you from using this program? Quantitative data from the questionnaires were analyzed using the SPSS version 25.0. Before conducting statistical analyses, the data were examined for outliers and normality, and all variables were within the normal range for skewness and kurtosis. Descriptive statistics were used and presented separately for the school personnel and students. We examined the differences between school personnel and students using 2-tailed independent sample t tests for continuous variables and chi-square analyses for categorical variables. All analyses were considered significant at P<.05. Qualitative data from open-ended questions were analyzed separately for the school personnel and students. A team member who participated in conducting the usability tests transcribed the data verbatim. We used thematic analysis [83, 84] to identify, analyze, organize, describe, and report themes found within the qualitative data. NVivo was used to track quotes and organize themes. Two trained master's students with previous experience in qualitative data analysis and a faculty member with expertise in qualitative methodologies analyzed the data. Before analyzing the data, the analysts discussed their assumptions and expectations regarding possible findings. Themes for each question were determined based on consensus. The process began with each team member individually developing initial themes for each question for school personnel and students. Next, the team met 2 times in person and conducted email communications over a 4-week period to arrive at a consensus on themes and frequency categories supported by participant quotations. During the first meeting, the lead analyst trained the team on thematic coding procedures, and each analyst coded the data separately. Next, the team met again for a second time to share each of their themes, followed by team members commenting and voicing agreement or disagreement. The analysts relied on participants' quotes to resolve disagreements. Once the team reached a consensus, an external auditor reviewed the interview transcripts and themes. Overall, the auditor agreed with the team's findings but provided an alternative way to organize one of the findings. The analysts used email correspondence to discuss and incorporate the auditor's feedback and obtain a final consensus for themes. The external auditor agreed with the team's final themes. Interview data were deidentified to ensure anonymity, and quotes were identified by participant type (ie, school personnel or students). The usability scores of the SUS are presented in Table 1 . Overall, the scores for both school personnel and students suggested a very high level of usability, functionality, and acceptability. As presented in Table 1 , there were no differences in any of the individual items or the SUS total score between school personnel and students, with both participant groups scoring the STAC-T app at a very high level of usability. School personnel and students rated the program high regarding user-friendliness. For school personnel, scores on user-friendliness ranged from 5 to 6 (mean 5.83, SD 0.41). For students, scores on user-friendliness ranged from 5 to 7 (mean 6.10, SD 0.57). There were no differences in scores between school personnel and students in terms of user-friendliness (t 14 =−1.00; P=.33). The program satisfaction ratings are listed in Table 2 . Overall ratings suggested that school personnel and students were satisfied with the program. There were no differences in scores between school personnel and students on program recommendation (χ 2 1 =1.3; P=.24, or star ratings, χ 2 1 =2.3; P=.31). Qualitative feedback for the STAC-T prototype supported the quantitative findings and was very positive overall, with participants sharing the perception that the STAC app is useful, relevant, and appropriate, as well as ways to improve the program. In addition, school personnel shared positive thoughts about program feasibility, a high likelihood of program adoption, and implementation barriers. The results are presented below, organized by the following themes: (1) utility and user-friendliness, (2) relevance, (3) program feedback, (4) feasibility, and (5) program adoption. Regarding participants' perceptions on the program's utility, school personnel and students indicated that STAC-T is useful and delivers helpful educational content to intervene in bullying. For example, 1 teacher stated, "this program is definitely one that can give strategies to students that they can apply and they also have a variety of strategies that they can choose from." A student shared: I thought it was pretty good. It gave lots of information; gave different ways to deal with the problem or kind of bullying. Furthermore, students emphasized that the program allowed them to learn about bullying. One student reflected, "I think it'd be really really good cause there's some things I didn't know and I learned about it." School personnel also indicated that the program is user-friendly and straightforward, and that developing the program on the web is a strength. A school counselor indicated: Participants were also asked to share their thoughts regarding the relevance and appropriateness of the program content for students at their school and their community. School personnel indicated that the program is relevant and can help increase students' understanding of relationships. For example, a school counselor stated, "I definitively think that it's relevant, just like we talked about, especially the online portion." Another school counselor indicated that the program helps "...kids to continue to understand the differences..." (between people). Students also indicated that the program was age-appropriate. One student said: When asked for feedback regarding how to improve the program, school personnel suggested that it is important to follow through while making the program interactive. One teacher stated: School personnel were also asked to speak about the feasibility of the program implementation. Participants provided positive responses indicating that the program was user-friendly and would align with existing programs that address social and emotional development. One school counselor indicated: ...it's very user-friendly, even for me who had struggles getting going. It really gives you very clear, concise steps to be able to move forward and directions of what you need to do. A teacher stated, "So, for me I can easily put it into the current curriculum or the current places in which we are but I think that would be up to individual teachers." Finally, when asked about the likelihood of program adoption, the school personnel indicated that their school would be open to using this program. One teacher stated: "I know I would really enjoy it personally because I think it's the strategies that are important for kids to understand and be able to use and apply." A school counselor indicated, "I think our school has been pretty open to learning new or picking up new tools that would enrich out student population." The school personnel indicated that resource constraints were the major barriers to adopting the program. A school counselor shared: This study aims to examine the usability of a technology-based bystander bullying intervention designed specifically for middle schools in rural, low-income communities. We were particularly interested in perspectives from both middle school personnel and students, as these participant groups represent key stakeholders who are in the position of making decisions regarding bullying programming and are the end users of the program. We aimed to test the usability of the STAC-T prototype; assess program utility, user-friendliness, and relevance; and gain an in-depth understanding of the needs and challenges related to program adoption and feedback regarding program content and delivery. Quantitative analysis of the survey data demonstrated a very positive response to the program, which was supported by qualitative data from individual interviews. Overall, the results indicate that participants perceived the STAC-T app to be useful, user-friendly, and appropriate for students at their schools and reported high levels of satisfaction with the program. Findings from this study indicate that the STAC-T app is relevant and feasible for implementation in middle schools in rural, low-income communities. The findings of this study support the usability of the STAC-T app. For both school personnel (mean 89.58, SD 5.10) and students (mean 91.75, SD 6.98), scores on the SUS indicated a very high level of usability, well exceeding the standard cutoff score of 68 [82] . The user-friendliness of the program was also rated very high by both school personnel, with all participants rating the STAC app at ≥5 on a 0 to 7 point scale. We found no differences between school personnel and students in terms of SUS scores or user-friendliness ratings, suggesting that all participants found the program to be highly usable. Qualitative data supported these results, with both school personnel and students indicating that they perceived the program to be user-friendly, as well as age-appropriate and relevant for middle school students in rural, low-income communities. Furthermore, both school personnel and students reported high levels of satisfaction with the program, with most participants indicating that they would recommend the program to others. These findings are particularly important, as usability and acceptability are associated with both program adoption and implementation [63] . Regarding feasibility of implementation, school personnel indicated that they believed that their school would use the STAC-T app if the intervention was cost-effective. This finding is consistent with research indicating that administrators in rural middle schools would support a technology-based bullying program [59] and parallels research on bullying prevention in rural communities, identifying cost as a barrier to program implementation [16, 59] . These findings also echo research suggesting the number 1 barrier to implementing educational technology is the lack of school district funding [45] and that financial resources and program effectiveness are necessary conditions for program implementation and sustainability of delivery of school-based programs [65] . Both school personnel and students discussed STAC-T program strengths and considerations for increasing user engagement. Qualitative findings suggested that participants perceived the STAC-T app to be useful and helpful in addressing the problem of bullying. They also indicated that the content of the program was appropriate for their schools, confirming the need for bullying programming that teaches students skills to use to intervene in bullying situations. This finding is consistent with previous research in which school personnel in rural, low-income communities indicated a need for bystander training that teaches students strategies to intervene on behalf of targets of bullying and includes having students actively practice these strategies across different scenarios [59] . Regarding user engagement, participants suggested adding more color and more realistic pictures, as well as increasing interactivity. This feedback is consistent with research suggesting that the motivating elements of technology-based interventions, including program content, length, and interactivity, are important in promoting behavior change [85] . This study supports the usability, relevance, and feasibility of the STAC-T prototype, providing valuable information for the development of a full-scale STAC-T app. However, certain limitations of this study must be noted. Participants were recruited from 3 schools in rural, low-income areas from 1 state in the Northwestern region of the United States. Although participants were recruited from 3 different counties to increase generalizability, school personnel and students from different regions of the country may have a different perspective. Furthermore, most participants in this study were female, further limiting the generalizability of the study. Although variability was low in the quantitative data and we did not identify divergent responses among participant interview responses, interpretation of the results for males should be made cautiously as females comprised approximately 81% (13/16) of the sample. Further formative research conducted during the development of the full-scale STAC app should include purposeful sampling to ensure appropriate representation of male participants. Furthermore, although most participants were White (13/16, 81%), this parallels the ethnic and racial composition of rural communities in the United States [86] . It is also possible that social desirability influenced participants as they were aware that the goal of the study was to translate the in-person STAC intervention into a technology-based format. This study has important implications for the development of a technology-based bullying intervention that addresses the needs and challenges specific to middle schools in rural, low-income communities. First, participants provided very high usability ratings for the STAC-T prototype, with qualitative data supporting the usability, utility, and relevance of the program. In addition, participants indicated that program implementation is feasible as long as the program is cost-effective. These findings support the development of the full-scale STAC-T app, while keeping the price point as an important consideration. Program cost considerations are particularly important in rural schools, as they face significant financial challenges due to a lower tax base for funding programs [16] . Furthermore, to enhance program implementation and sustainability, sufficient resources must be available [65] . Thus, the STAC-T app needs to be designed to be low in cost to be successfully adopted by schools in rural, low-income communities. The findings from this study provide a strong scientific premise for moving forward with a full-scale production of the STAC-T app and have implications for program development. The full-scale STAC-T app will include 3 core modules, including strategy training, in which students select customized avatars as they move through role-plays in which they practice the STAC strategies. On the basis of participant feedback on the STAC-T prototype, all core modules will be built to include gaming and features requiring user input (eg, drag and drop, hover, click and reveal, and video). Real instances of game dynamics and mechanics will include several options, such as badges, leaderboards, levels, points, achievements, avatars, content unlocking, quests, social recognition, teams, and tokens. These items target the social cognitive theory components of modeling, outcome expectancies, self-efficacy, self-regulation, identification, and reciprocity. Consistent with research on mobile health strategies for adolescents [78] [79] [80] , this design emphasizes gaming as a teaching strategy. The outcomes of each game will be used to ensure information uptake and demonstrate comprehension (eg, users must achieve a targeted score to receive a reward), thus providing feedback to participants early and often. To reinforce learning and bolster adherence, badges (ie, visual icons), which are highly effective gaming tools used to encourage user engagement and build game loyalty [87, 88] , will be awarded intermittently. These components will be the centerpiece of the program and are designed to increase engagement and learning. Bullying is a significant public health issue for middle schools in rural, low-income communities. Schools in these communities face multiple and competing demands on their time and have limited access to training, funding, and mental health professionals to implement bullying programming. The findings of this study demonstrate the usability, relevance, and feasibility of a brief, technology-based bystander bullying intervention. This study provides support for the development of the full-scale development of the STAC-T app and provides information that can be used to enhance program usability while addressing the unique needs of schools in rural, low-income communities. 2017 school crime supplement to the national crime victimization survey. U.S. Department of Education -NCES 2019-054 Bullying and lower urinary tract symptoms: why the pediatric urologist should care about school bullying The relation between peer victimization and sleeping problems: a meta-analysis Bullying behavior and associations with psychosomatic complaints and depression in victims Impacts of traditional bullying and cyberbullying on the mental health of middle school and high school students Observing bullying at school: the mental health implications of witness status Childhood bullying, paranoid thinking and the misappraisal of social threat: trouble at school Bullying among young adolescents: the strong, the weak, and the troubled Peer victimization and social anxiety in adolescents: prospective and reciprocal relationships The relationship of peer victimization to social anxiety and loneliness in adolescent females Peer cybervictimization among adolescents and the associated internalizing and externalizing problems: a meta-analysis Bullying in the digital age: a critical review and meta-analysis of cyberbullying research among youth Is cyberbullying worse than traditional bullying? Examining the differential roles of medium, publicity, and anonymity for the perceived severity of bullying Student reports of peer bullying victimization in a rural school Bullying and victimization in rural schools: risks, reasons, and responses Bullying behaviors among US youth: prevalence and association with psychosocial adjustment School bullying: a systematic review of contextual-level risk factors in observational studies Cumulative bullying victimization: an investigation of the dose-response relationship between victimization and the associated mental health outcomes, social supports, and school experiences of rural adolescents Socioeconomic status and bullying: a meta-analysis Demographic, psychological, and school environment correlates of bullying victimization and school hassles in rural youth Witnessing bullying at school: the association between being a bystander and anxiety and depressive symptoms Witnessing cyberbullying and internalizing symptoms among middle school students The association between witnessing cyberbullying and depressive symptoms and social anxiety among elementary school students The relationship between witnessing cyberbullying and depressive symptoms and social anxiety among middle school students: is witnessing school bullying a moderator? The moderation of empathy in the longitudinal association between witnessing cyberbullying, depression, and anxiety Bystanders' reactions to witnessing repetitive abuse experiences Exploring the impact of bullying on young bystanders Does defending come with a cost? Examining the psychosocial correlates of defending behaviour among bystanders of bullying in a Canadian sample Defending behaviors, bullying roles, and their associations with mental health in junior high school students: a population-based study Deciding whether to look after them, to like it, or leave it: a multidimensional analysis of predictors of positive and negative bystander behavior in cyberbullying among adolescents Evaluating the effectiveness of school-bullying prevention programs: an updated meta-analytical review Connections between attitudes, group norms, and behaviour in bullying situations Teens, kindness and cruelty on social network sites. Pew Research Center The impact of a brief, bullying bystander intervention on internalizing symptoms: is gender a moderator of intervention effects? The impact of a brief bullying bystander intervention on depressive symptoms Evaluation of a brief, school-based bullying bystander intervention for elementary school students A randomized controlled study evaluating a brief, bystander bullying intervention with junior high school students Efficacy of a brief, school-based bystander bullying intervention on high school students' alcohol use Bystanders' support of cyberbullied schoolmates Bystander behaviour in peer victimisation: moral disengagement, defender self-efficacy and student-teacher relationship quality. Res Pap Educ The role of moral disengagement and self-efficacy in cyberbullying Supporting children's mental health in schools: teacher perceptions of needs, roles, and barriers NetSupport DNA -IT asset management for schools E-learning and digital education -Statistics and facts Survey questionnaires results: students' computer access and use. National Assessment of Educational Progress (NAEP) Is E-Rate doing enough to effectively expand broadband in rural schools? K-12 Dive Broadband internet access and the digital divide: federal assistance programs USDA launches new program to create high-speed internet e-connectivity in rural America. U.S. Department of Agriculture Department of education releases resource to help schools improve ventilation systems to prevent COVID-19. U.S. Department of Education A bystander bullying psychoeducation program with middle school students: a preliminary report Establishing school counselors as leaders in bullying curriculum delivery Training students who occasionally bully to be peer advocates: is a bystander intervention effective in reducing bullying behavior? J Child Adolesc Couns Acceptability and short-term outcomes of a brief, bystander intervention program to decrease bullying in an ethnically blended school in low-income community Cultural adaptation of a school-based bullying intervention A mixed method evaluation of a culturally adapted, brief, bullying bystander intervention for middle school students Evaluation of a brief, bystander bullying intervention (STAC) for ethnically blended middle schools in low-income communities Impact of a brief, bystander bullying prevention program on depressive symptoms and passive suicidal ideation: a program evaluation model for school personnel Technology-based bullying intervention for rural schools: perspectives on needs, challenges, and design. Rural Ment Health The role of feedback in the process of health behavior change Changing norms to change behavior Usability testing in 2000 and beyond User-centered redesign of evidence-based psychosocial interventions to enhance implementation-hospitable soil or better seeds? JAMA Psychiatry Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda Sustainability of teacher implementation of school-based mental health programs Internet interventions: in review, in use, and into the future Persuasive system design does matter: a systematic review of adherence to web-based interventions Persuasive systems design: key issues, process model, and system features A systematic framework for designing and evaluating persuasive systems User-centered design User Centered System Design: New Perspectives on Human-Computer Interaction What is instructional design? In search of the elusive ADDIE Model Bringing ADDIE to life: instructional design at its best Application of theory: minimalism and user centered design. In: Society for Technical Communication A user-centered theoretical framework for meaningful gamification User-centered design in games Design of an mHealth app for the self-management of adolescent type 1 diabetes: a pilot study How to systematically assess serious games applied to health care A tablet game for risk reduction and HIV prevention in adolescents SUS: A 'Quick and Dirty' usability scale Determining what individual SUS scores mean: adding an adjective rating scale Using thematic analysis in psychology Thematic analysis: striving to meet the trustworthiness criteria Persuasive features in web-based alcohol and smoking interventions: a systematic review of the literature Rural America at a glance. U.S. Department of Agriculture 7 things you should know about badges A game plan: gamification design principles in mHealth applications for chronic disease management Coaching Compassion STAC-T: technology-based STAC SUS: System Usability Scale Edited by G Eysenbach Usability of a Technology-Based Bystander Bullying Intervention for Middle School Students in Rural ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information This study was registered with ClinicalTrials.gov (identifier NCT04681495). This research was supported by the National Institute on Minorities and Health Disparities of the National Institutes of Health under award number R41MD014943. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. MKB is the owner of Klein Buendel, Inc, the organization that received the National Institutes of Health grant award for this project. As it is a small business technology transfer, a concerted effort to commercialize a final product that may result from this and subsequent National Institutes of Health-funded research will be made, and the organization may receive royalties or revenue from the commercialization. DMD and AM may also receive royalties or revenue from the commercialization.