key: cord-0974562-jbj1coqk authors: Shanley, Nicole; Martin, Florence; Hite, Nicole; Perez-Quinones, Manuel; Ahlgrim-Delzell, Lynn; Pugalee, David; Hart, Ellen title: Teaching Programming Online: Design, Facilitation and Assessment Strategies and Recommendations for High School Teachers date: 2022-04-12 journal: TechTrends DOI: 10.1007/s11528-022-00724-x sha: 416d34929d6ba02192399f39df77c7a96a12e351 doc_id: 974562 cord_uid: jbj1coqk Current research surrounding online computer science education emphasizes the need for high-quality professional development opportunities. However, there is a gap in research in the inclusion of online computer science educators to identify needs and strategies that make the online computer science courses effective. Through a Research-to-Practice Partnership (RPP), this paper examines the instructional strategies and recommendations from online Computer Science teachers. This study seeks to better understand (1) What design, facilitation, and assessment strategies do teachers use to teach programming online? and (2) What recommendations do teachers have for those interested in teaching programming online? The feedback teachers provided during the study assisted in identifying the current needs in online AP Computer Science. The participants suggested additional ways the RPP could support teachers in strengthening their practice, which has assisted in the production of high-quality professional development to support novice teachers entering the field of Computer Science. Many high school students currently take computer science courses through virtual schools due to the lack of offerings and unavailability of teacher expertise at their local schools. Goode et al. (2020) consider preparing thousands of teachers with high-quality, accessible professional development as a grand challenge. While most teachers enter the classroom with the general ability and skills necessary to teach, many of those teachers are not content experts nor have been trained to specifically teach online. When combining both a new content area and a teaching platform, many challenges arise. This highlights the issue that teaching computer science online requires the use and implementation of different instructional strategies. In this study, we discuss strategies and recommendations from teachers through a Research to Practice Partnership (RPP) with a State Virtual Public School (SVPS), through which we plan to design and offer online professional development for teachers across the state to teach AP Computer Science advanced courses. This paper discusses the findings from a needs assessment conducted via three focus groups with 14 teachers from SVPS and a collaborative seminar held in the summer of 2020 by the Research to Practice Partnership. It will also address the direct connection between the results and thoughts shared in the focus group sessions and the data collected during a workshop using Jamboard, a collaborative digital whiteboard. Using the framework proposed by Martin et al. (2019) , we focus on design, facilitation, and assessment strategies teachers use while teaching computer science online (Fig. 1 ). Design strategies include the various instructional strategies that teachers and designers use while designing a course. According to Veletsianos et al. (2016) , the National Science Foundation has been encouraging the computing education community to partner with education researchers to support the overall development of computer science education, curriculum, and course design. There are few sources in the literature surrounding online course design to speak to the intricacies of online instructional design specifically as it applies to computer science education. Zendler and Klaudt (2015) identified several instructional methods such as problem-based learning, learning tasks, discovery learning, computer simulation, project work and direct instruction that work well in computer science courses. They propose recommendations on how each of the computer science courses can be designed for each of these instructional methods. McGowan (2016) presented a four-component theorybased design framework that can be used in computer programming eLearning courses. Building on the framework proposed by Dabbagh (2005) , McGowan's framework included "pedagogical model, a body of exemplars, instructional strategies and learning technologies to facilitate meaningful learning of proper CP practices and knowledge building (p.11)". A study conducted by Proulx (2000) sought to assist instructional designers in streamlining their online courses by creating a framework to help guide course design. Their "goal is to help students focus on mastering reasoning and design skills before the language idiosyncrasies muddy the water. (p.80)". Similarly, subsequent studies have alluded to the fact that while the course work and content taught within computer science courses can be difficult for novice computer science students to pick up, there is room for improvement in how the course is designed. There is also research conducted to support the development of Massive Online Open Courses (MOOCs) for computer science foundational courses. An article published in 2015 highlighted the design of these MOOCs to include new and improved approaches to computer science course design; the research team implemented a more balanced pedagogical approach, one that worked to assist the overall course design to better address the "cognitive, interpersonal, and intrapersonal" (p.1) needs of students and take into students ``deeper learning" (Grover et al., 2015) . When considering design strategies within the context of online computer science courses, another design process frequently appears within the literature, design thinking. This iterative process has been recently explored within the world of K-12 education through the work of Crane et al. (2018) and Li and Fu (2020) . Both authors have used design thinking as a framework that guides course design within K-12 education, promotes and builds community within teachers who are acting as course designers, and works to support both physical and technical innovation into the design process. Martin et al. (2020) define facilitation as "how, what, when, and why an online faculty member makes decisions and takes actions to help students meet the learning outcomes (p.36)". A recent study published in Science Direct, The Effectiveness of Online Learning with Facilitation Method (Zulfikar et al., 2019) evaluates the level of student participation in online discussion forums and other tools useful in online course design. Specifically, the authors reviewed the effects of facilitation methods and teacher involvement in student participation in online discussion forums. Applying these generalizable studies to the field of online computer science could provide a new lens through which we view facilitation as applied to computer science. While there is little research that directly seeks to identify and understand the effectiveness of online facilitation strategies for computer science teachers, there are a few studies that review the effectiveness that online computer science courses have had, and ways that teachers in this learning environment have worked to support their students, virtually. Evidence of this can be found in Huan et al.'s article, Teaching Computer Science Courses in Distance Learning (2011) . Throughout this study, the research team highlights the influence of distance learning and its increasing popularity due to both flexibility and convenience of learning and as more recently notes, out of necessity because of the 2020 COVID-19 pandemic. Throughout this study, we see mention of online tools that increase learner engagement such as the inclusion of multiple learning modalities, PowerPoint presentations embedded into the course, PDF documents, and the ability of the course to work across multiple platforms, allowing for accessibility among mobile devices. One of the seminal texts that support the foundational understanding of online facilitation strategies for computer science students, comes from the work of Wilson et al. (1997) . In its inception, online facilitation for computer science students had the goal of supporting students in an asynchronous environment, increasing their engagement and overall understanding of the content being taught. During this time, the major question being addressed was "How can we best support such teaching and learning and what aspects of this process work well when compared to faceto-face teaching?" It has been 24 years since its publication, and this same question is being asked across computer science publications with educational researchers working to support the connection between online facilitation strategies and computer science. Computer science courses, especially those operating underneath the heading of "Advanced Placement," are heavily tested with careful consideration and alignment placed on the final Advanced Placement (AP) examination. However, computer science teachers have the autonomy to create and apply assessment strategies throughout the course, with respect to the overarching needs placed by their district or school administration. When considering assessments and the various types that can be utilized within a course, it is important to consider the value of both formative and summative assessment. According to Black and Wiliam (2009) , formative assessments are defined as "…evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of the evidence that was elicited" (p. 9). Similarly, Grover (2021) agrees with the work of Black and Wiliam (2009) that the major difference between summative and formative assessment lies within motivation behind the assessment and how teachers respond to the data collected. The goal of summative assessments is to grade students, using commonplace tests or quizzes, typically through several multiple-choice questions (Sorva & Sirkiä, 2015) . However, formative assessments are aimed at monitoring students learning and coaching them through the learning process (Grover, 2021) . With these differences in mind, granting teachers the autonomy to carefully select their assessment types and tools in a way that supports their students needs and allows them to continue to coach them through the learning process is vital. According to a systematic review conducted by Garcia et al. (2018) , many of the available e-tools, such as automated tools (or auto graders) assist instructors in grading large quantities of work and provide students with instant feedback. When instructors include auto graders, they can support students by offering hints or guidance on their assignments by flagging compiler, test case, solution, and style errors (Keuning et al., 2016) . As previously mentioned, there is a significant emphasis placed on the College Board Advanced Placement Computer Science examinations, as these are the assessments used to determine a student's content proficiency. Specifically for the AP Computer Science A (CSA) course, the exam consists of 40 multiple choice questions and four free-response questions that require students to demonstrate their understanding of basic skills related to computer science programing (The College Board, 2006) . Of the four Free Response Questions (FRQ), students will be asked to write code that displays mastery of the following skills: Methods and Control Structures, Classes, Array/ArrayList, and 2D Array. An assessment strategy commonly cited in computer science research is incorporating short free-response questions built into computer science courses, especially in an online learning environment. According to Klein et al. (2011) , standardized tests and commonplace multiple-choice questions provide a shallow understanding of students' actual ability. To truly engage students and understand the depth and complexity of their knowledge and evaluate application skills, instructors need to invest in meaningful forms of assessment such as the inclusion of open-ended or freeresponse questions. The study by Klein and colleagues tested the effectiveness of auto graders when used to grade freeresponse questions to assist instructors in providing students with more meaningful opportunities to demonstrate their learning. They support the overarching theme of this paper, which aims to support investment and integration of highquality auto graders into online computer science courses. Computer science courses are not offered by all school districts, and therefore some students enroll to complete computer science courses online through the SVPS. Teaching computer science online requires different instructional strategies, and both students and instructors experience challenges teaching and learning online. In this study, we examine the instructional strategies currently used by high school teachers who teach computer science online and analyze the current design, facilitation, and assessment strategies they use to engage with their students. The research question addressed in this study include, 1. What design, facilitation, and assessment strategies do teachers use to teach programming online? 2. What recommendations do teachers have for those interested in teaching programming online? This section describes the instructional context, research participants, data collection, and data analysis. The context of this study is based on a Research to Practice Partnership (RPP) currently held between a southeastern public university in the United States and a state Virtual Public High School. As part of a (Foundation) Grant -Computer Science for All, the research team collaborated to create and offer online professional development to teach the AP Computer Science advanced course to high school teachers. We established a RPP to guide the development of professional development for online computer science instruction. Establishing an intentional, long-standing, and collaborative partnership between computer science education researchers and computer science teachers at the State Virtual Public School is critical to addressing the professional development needs of a larger audience of online computer science teachers. Our RPP approach stresses the role of our lead teachers from the State Virtual Public School as key researchers in shaping the design of the professional development. Using participatory research approaches, the project team engaged the lead teachers during the project's first year to identify critical instructional strategies and resources vital to their success in teaching computer science online. Through the RPP, the lead teachers' roles as key stakeholders in the design process was reinforced. The leaders were reminded of their roles as experts in the partnership and the critical importance their input plays in the future design and implementation of the professional development program. Teachers were invited to participate in a focus group, followed by a one-week summer workshop, where the research team engaged the participants in online professional development. The teachers were put into the role of "content expert" working to identify best practices for online instruction. A primary goal of this focus group and workshop was to allow teachers to extend their thinking and consider approaches to formative assessment and methods for promoting equity in computer science instruction. The use of formative assessment within this context refers to the use of a "low stakes" assessment as an ongoing way to monitor student learning (Black & Wiliam, 2009 ). This participatory research approach allowed the project team to capture ideas and outcomes from the teachers that will guide the professional development design. An ongoing process of sharing and refining establishes a synergistic partnership that will continue to be the foundation of this RPP project. Purposive sampling was used to select participants for this focus group. The Instructional Director at the SVPS High School facilitated the recruitment of teachers who teach computer science from within that high school. The teachers were then sent invitations to participate in the study. Interested teachers completed the consent form to participate in a focus group. Three focus groups were scheduled with ten teachers. The focus groups included two groups of 3 and one group of 4 participants, facilitated by members of the research team. The teachers who participated in the focus groups varied in their background and experience but taught a computer science course for the SVPS. The research team then recruited the online Computer Science teachers from SVPS to participate in a summer workshop. These same teachers had identified themselves as interested in participating in a workshop created to identify large-scale needs and provide support to online Computer Science teachers. The participants engaged in a weeklong seminar where the research team presented topics such as Approach, Challenges, Solutions to Online APCSA, Online Teaching Strategies, Engagement within Online Learning, Auto graders, and Other Online Tools, and Culturally Relevant Computing and Social Impact. In addition to presentations, participants actively engaged in discussions surrounding these topics and connected their experiences teaching online Computer Science courses to the research presented. The researchers conducted three semi-structured focus groups using the breakout room functionality in Zoom. Each interview averaged about 26 min. The sessions were audio-recorded and then transcribed using Otter machine transcription, followed by human transcription. Two focus group questions were discussed and finalized by the research team. The focus group questions were directly aligned to the research questions of this study and were (1) What design, facilitation, and assessment strategies do teachers use to teach programming online? And (2) What recommendations do teachers have for those interested in teaching programming online? The responses from an additional four questions are not included in this study. Following the participation in the focus group sessions, a subset of participants volunteered to participate in the summer workshop held in 2021. This workshop provided the opportunity to discuss questions asked during the focus group interviews to support a deeper understanding of teachers' thoughts, experiences, and perceptions related to our research questions. The question for the online collaborative Jamboard activity was, "What design, facilitation, and assessment strategies are helpful to include in an AP Computer Science advanced course?" The researchers used an inductive coding process (Miles et al., 2013) to analyze the data. Two researchers analyzed the data from each research question using the same process. The transcribed interviews were initially coded using an open coding process. These were color-coded to form different categories and grouped to develop themes. Once the coding was completed, the larger research team met to discuss the codes and categories generated. Throughout the summer workshop, the participants were asked to engage with a Jamboard on an online collaborative activity responding to specific questions. Jamboard (Google Workspace, n.d.) is a digital interactive whiteboard developed by Google to work within the Google Workspace. This tool allows for collaboration by using a digital whiteboard, making it easy to create and share ideas in real-time, regardless of distance. The posts on the Jamboard were grouped to identify common themes. The results section presents the findings from the digital collaborative activity data collected during the summer workshop and the online focus groups. During the summer workshop, the participants were asked to engage with and reflect on the topics presented, and to share their experiences and expertise related to the research questions within this study. They were asked, "What design strategies are helpful to include in APCSA?" Fig. 2 includes a screenshot of the Design Jamboard. The following themes emerged: go to resources, examples, assessments, and making real-world connections. This included purposeful exposure to common misconceptions and resources to support these errors, a bank of high-quality resources (such as access to high-frequency vocabulary words related to the content area, and short videos created for students that align to the computer science curriculum). Additionally, multiple participants expressed the need for access to superiorly designed questions with answers that are not located using search engines (i.e., Google). However, it should also be noted that many teachers expressed that while courses can be adapted and additional resources may be included, there is hesitancy to make significant adjustments to the course shell as the school aims to provide continuity among its courses. There was a significant misconception surrounding the use of auto graders and other feedback tools. Many participants stated that they believed auto graders to be the automatic grading function in Canvas and were unaware of the potential impact of auto graders when applied to their course design. The following themes on computing and pedagogical tools and resources used emerged from the focus groups regarding design strategies. Several teachers proposed the theme, online resources, as an essential instructional strategy and included both computing/programming resources and pedagogy tools. For the purpose of this study, we are operating under the shared understanding that a pedagogical tool is that which enhances a student's understanding of the content or a support for a teacher who teaches in an online space. Furthermore, a computing tool or resource is a content specific application or software that supports students' understanding strictly as it relates to computer science. Some of the online resources used by the teachers are mentioned in Table 1 . Online facilitation is the ability of an instructor to promote learning in an online environment by fostering a positive learning experience and engaging with students in a way that supports personal growth. During the summer workshop, the teachers were presented with the question, "What facilitation strategies are helpful to include in APCSA?" Many of the summer workshop and focus group participants have various levels of experience teaching simultaneously online and in traditional face-to-face settings, requiring the instructor to be both effective facilitators virtually and in person. With limited time and resources, many teachers struggle when asked to transition between the two, as the digital divide has impacted both students and teachers. The following themes emerged from the participants' responses on the facilitation Jamboard: AP CSA specific Free Response Questions (FRQ) examples, video resources and feedback. This included the need for a bank of freeresponse questions (without published answers), purposeful video resources that isolate skills (designed for students), and a way to support students by providing more detailed and meaningful feedback. Figure 3 includes a screenshot of the Facilitation Jamboard. During the focus group interviews, many teachers mentioned that they struggled with online facilitation and described frustrations when they experienced a lack of student engagement or felt that their whole group communication was limited due to no synchronous learning sessions built into the course. These elements of course facilitation that teachers identified resulted in teachers having a limited understanding of students' abilities, or lack thereof, until it was too late in the semester to provide additional support. Many participants emphasized that teachers must take an active role in their virtual learning environment and demonstrate their engagement in the course by promptly responding to students and their questions, promptly providing meaningful feedback, and incorporating additional resources to support errors made on an individual basis. While some of the teachers who participated in the focus groups were involved in course design, several were only tasked with teaching and facilitating the course. From the focus groups, the following facilitation themes emerged. Weekly Announcements Some of the instructional strategies mentioned as part of the course facilitation included "weekly announcements" and "we can add materials to our announcements." One teacher commented, "we don't really have flexibility in designing the courses. They're structured for us, and the teachers get a Canvas shell, but we do have the flexibility to add supplemental material." Live synchronous sessions were also mentioned as part of the facilitation. A teacher added, "a few kids that would come in and ask questions, she would always record our sessions and make them available as archives so that students could then go back and view them." A teacher added that the live sessions might not have worked for all students, but they conducted a live session for each topic. More Practice Videos Teachers thought it was essential to include more practice videos as part of course facilitation. They noted that providing students with various videos for each standard or concept provided similar explanations in slightly different ways to allow students multiple opportunities for enhanced clarity. Providing additional practice videos was a course facilitation strategy the teachers implemented to assist students in an asynchronous online setting. In the summer workshop, when the participants were asked to respond to the question "What assessments are helpful to include in AP CSA?", the following themes emerged: the need for both a larger bank of programming questions, and access to shorter formative assessments. Additionally, teachers mentioned the need for supplemental assessments to be created, with emphasis placed on alignment to mastering specific computer science concepts. Figure 4 includes a screenshot of the Jamboard. Furthermore, in the responses to the need for summative assessments, participants identified the need to create assessments unique to the course each semester and situate the assignments within the "real world" context. Instructors also included the need to support students using computer science programs. Additionally, an overarching theme that was identified during the focus group interviews and reinforced The most popular and commonplace assessment strategies are quizzes, tests, state-administered standardized tests, and essays. While each of these relatively traditional forms of assessment has its place in a curriculum, it is becoming increasingly common within the field of education to be critical of these assessments as they limit students' ability to demonstrate their knowledge on a specific topic or within a content area. Similarly, many of the participants in this study agreed that assessment strategies need to be carefully selected before applying them to a course. Assessments need to be both meaningful and carefully aligned with the course objectives and content standards. From the focus groups, the following assessment strategies emerged. Teachers mentioned several instructional strategies exercised in their classrooms to align with the College Board examination. They used College Board materials, videos in the AP Classroom, and AP free-response style questions to prepare students for the AP Classroom. One teacher commented, "we've added things that have made it a much better course. We've added structure to it to make it seem more realistic, as far as testing is concerned with the AP exam." Teachers mentioned utilizing a variety of assessments in their online computer science course. Some of the teachers' assessments included checkpoints to ensure students are prepared, tests including multiple-choice questions, projects, and timed free-response questions. They also emphasized the importance of providing feedback. In addition, teachers mentioned the importance of including an evaluation in the end. Evaluation is used to collect student feedback on the course to make improvements before the following implementation. While the data collected were categorized by design, facilitation, and assessment, there were some strategies that included various aspects. The interviewed teachers discussed the collaborative aspect of both design and facilitation used by this virtual public school. The course was assigned a course lead and included a large team of teachers. A teacher commented, "…have a team of the content experts develop the course, lay out the outline, and actually develop the content for the course." While every teacher's opinion is considered, changes are made based on the consensus. Also, one teacher noted, "Typically, we don't take them away unless it's a group decision…." Student Engagement A few of the teachers discussed the importance of student engagement. While getting the content on time is essential, it is also crucial to embed engaging and collaborative activities. One teacher commented, "A major platform that we started using to facilitate our content, which allowed the students to be more engaging, more engaged in the course as well as access to those tools." Teachers discussed the importance of including short videos about 10 min in length to engage the students. Evidence-Based Teaching Practices A few teachers described using evidence-based practices such as modeling, guided practice, tutorials explaining how something is done, and scaffolding as instructional strategies in their online computer science course. The participants within this study provided valuable feedback and insight to support those interested in teaching computer science online. While several themes and pieces of advice emerged during the focus group discussion, the most prevalent theme was preparation. Among all three focus groups, roughly half of the responses spoke to the need for high-quality teacher preparation by attending a technical college specifically for computer science or attending ongoing professional development sessions. The professional development workshop offered by the technical college provides intensive support designed to prepare instructors of all levels, especially those who have not taught or studied computer science. The second most prevalent theme that resulted from this research question was the need to inform new teachers about the demands of the course, more specifically, the demand on the instructor's time. Most participants spoke to the commitment that the course requires, both in providing feedback to students (as described as a feedback loop due to its continuous nature and the revisions students need to make to improve), the communication required to maintain relationships with students in a virtual, asynchronous environment and the most considerable demand stated was the time spent grading assignments. It was noted that while the grading takes significant time and effort from the instructor, the result is worth the work as the student implements the changes and improves both their product and understanding. The final theme addressed during the focus group discussion was the need to adjust expectations and adapt to the students within the classroom, albeit virtual. Many instructors noted that students and their prior knowledge vary significantly from semester to semester. Adjusting instruction to meet that baseline understanding of computer science is necessary to set up students for success. However, it is also important to note that students will be expected to still take the College Board assessment at the end of their course regardless of course entry knowledge. With this in mind, a participant shared the following, ".... I've taught computer science face to face and teaching it online, and different methods to approach, you know, the same objective, same units and just be very flexible. You can't do it the same way." Based on the conversations highlighted within this study among the instructors at the SVPS, it is evident that adjusting teaching, course design, and facilitation are necessary to meet the learners' needs. However, when discussing assessment strategies, the teachers seem to agree that the presence of the AP examination weighs heavily on both students and instructors. Due to this pressure, there is less flexibility with assessment practices than with other strategies associated with course design. This has resulted in the necessity to have near-perfect alignment between practice problems, free response questions, and course assignments to mirror the possible questions students will be expected to answer on the AP exam providing opportunities for exposure. The phrase instructional strategies is being used here in an overarching way to encompass the three main strategies discussed within this study; design, facilitation, and assessment strategies. Each of these strategies has been identified as necessary, and through the assistance of our Research to Practice Partnership, several examples of each strategy have been provided. The facilitation strategies outlined in this paper also align with Berge's (1995) roles of the Online Facilitator as the instructor engages in a Pedagogical, Managerial, Social and Technical Role simultaneously. Based on the examples that were shared by our participants, we can see a clear alignment between the needs and experiences that these teachers identified, and the similarities shared with the current body of research. As noted by Grover et al. (2015) , design and facilitation strategies are essential to the overall success of an online course, such as incorporating active learning components into computer science courses. Similarly, by creating a course that challenges how our students encounter, engage, and reflect on their learning (Fink, 2013) , teachers work to create a course that fosters active engagement and increases both student engagement and the overall effectiveness of the course. As highlighted in our literature review, while there is a large body of research that supports the design and facilitation of online course creation, there is limited evidence to support which facilitation strategies best support students within computer science courses. The few content-specific studies that were included that evaluated facilitation and course design (Grover et al., 2015; Huan et al., 2011 & Proulx, 2000 were able to provide an insight into traditional course design strategies and their effectiveness within online computer science courses. Similarly, the studies included about design thinking, while relevant to the K-12 educational sector, were not specific to the world of computer science education. However, when viewing this study through the lens of design thinking and its phases, there is clear alignment between the processes in which we completed this study and the phases that the design thinking process follows. With the significant emphasis that design thinking places on identifying problems and creating possible solutions, the basis of this study, follows these general principles. Additionally, throughout the duration of this study, it became apparent that there is a significant need to further develop high-quality resources that are available to online computer science teachers. Teachers expressed the need to have access to well-designed question banks that aligned with the course standards and assisted in preparing students to take the College Board examination, directly tied to the completion of the online course. Furthermore, the teachers expressed the desire to have these question banks inaccessible to the students, as they have experienced the negative effects of students using search engines to simply identify the answer, which limits the instructor's understanding of their student's knowledge. Similar recommendations were made by Klein et al. (2011) as they supported the addition and integration of auto graders as a potential solution for this challenge. The findings of this study have implications for teachers who currently teach or wish to teach computer science online in the future. The various strategies used by these teachers will be beneficial when teaching computer science online. Teachers Computer Science teachers must use strong course design, facilitation and assessment strategies. Specifically, in their design teachers could use, "go-to resources' ', examples, assessments, and assignments based within real-world situation. During facilitation, teachers could integrate AP CSA specific Free Response (FRQ) examples as well as video resources for practice and feedback. In addition they should communicate by sending weekly announcements, and live synchronous sessions. During assessment, teachers could choose questions from a larger bank of programming questions and also include shorter formative assessments in the course to provide a variety of assessments. Teachers should provide regular feedback to the students on these various assessments. Additionally, teachers must integrate methods for assessment and evaluation, promote student engagement, through evidence-based teaching practices. We see specific examples of this in the Jamboard and focus group findings as teachers identified the importance of maintaining a positive online presence through providing continuous feedback, meeting the needs of students through various communication methods, aligning course assignments and assessments to meet the demand of the course while embedding the task within a realworld context. The findings also have implications for administrators and instructional designers who support teachers in designing and delivering online courses. Instructional designers could use all the instructionalstrategies discussed above for design, facilitation, and assessment in the design of the course. Administrators can also benefit from these research focused findings and support the instructional designers and teachers to use these strategies in the Computer Science courses. The participants in this study explained the process in which changes, or edits can be made to their online course frameworks. Alterations would not be granted without stakeholder approval, which is built into the course design process to ensure that all online courses within the same state high school are held to the same standard. Participating in a study structured similarly to ours, allowed for the participants to share their thoughts and experiences openly and honestly with the current framework in place, which allowed for direct communication between teachers and their administration without fear or repercussions. This open forum and exchanging of ideas have directly benefited the stakeholders involved because when high quality edits are made the AP CSA course shell, teachers are supplied with a more supportive foundation, students are equipped with more online supports that are directly related to their course standards, and school leaders view their teachers as both content experts and advocates for their students, which should ultimately result in more students successfully passing their AP CSA exam. Students Finally, the study has implications for online students who will benefit from various instructional strategies used in the courses. There are a few methodological limitations to this study. This study only included teachers from one virtual public school from one state, and data was collected in three online focus groups and three Jamboards during a collaborative activity. This data may not be generalizable to non-virtual school settings. Teachers may have responded differently to the online facilitation of the focus groups and online workshop through Zoom and Jamboard than they might with face-to-face focus groups or interviews. Accessing the meeting with a phone instead of a computer, or only some teachers turning on their video may have impacted how they participated in the focus group. Additionally, the subset of participants who engaged in the summer workshop met synchronously for more extended periods (approximately five hours per day for five days). During this time, the data was collected in a group-like setting, which may have resulted in conformity among participants. While there was a significant benefit to conducting this portion of the study in a collaborative seminar setting, the largest of which was the sharing and melting of ideas and past experiences, social pressure was a likely natural consequence. In this unavoidable limitation of social pressure, participants change their beliefs or behavior to fit in with others, creating the possibility for swayed responses leading that might have influenced data. While this study was conducted using interviews and from a digital whiteboard from online teachers at one virtual public school, this could be extended to teachers teaching online in various settings nationwide. Also, a large-scale survey will assist in collecting data on teacher perceptions regarding instructional strategies they use and teacher and student challenges. It is recommended that further research be conducted to directly identify and determine which course tools or programs, and student engagement techniques are best suited to support online computer science courses. In addition, interviewing administrators, parents and students will help us understand successful online teaching and learning strategies and challenges identified from various perspectives. Facilitating computer conferencing: Recommendations from the field Developing the theory of formative assessment Exploring best practices for implementing design thinking processes in K12 education by Pedagogical models for E-Learning: A theorybased design framework Creating significant learning experiences: An integrated approach to designing college courses Systematic literature review: Self-regulated learning strategies using e-learning tools for computer science Computer science for equity: Teacher education, agency, and statewide reform Google Jamboard: Interactive Business Whiteboard | Google Workspace Toward A framework for formative assessment of conceptual learning in K-12 computer science classrooms Designing for deeper learning in a blended computer science course for middle school students Teaching computer science courses in distance learning Towards a systematic review of automated feedback generation for programming exercises Automated assessment of short free-text responses in computer science using latent semantic analysis general terms Creativity initiative: Design thinking drives K12 education from a future thinking Awardwinning faculty online teaching practices: Course design, assessment and evaluation, and facilitation. The Internet and Higher Education Facilitation matters: Instructor perception of helpfulness of facilitation strategies in online courses Towards a theory-based design framework for an effective E-learning computer programming course. International Association for Development of the Information Society Qualitative data analysis: A methods sourcebook Programming patterns and design patterns in the introductory computer science course Embedded questions in ebooks on programming The College Board Design principles for thriving in our digital world Facilitation of on-line learning environments: What works when teaching distance learning computer science students Instructional methods to computer science education as investigated by computer science teachers The effectiveness of online learning with facilitation method Acknowledgements This project has been funded by the National Science Foundation under Grant No. 2031496. Conflict of Interests/Competing Interests This paper and the research conducted to support its conclusions was the result of a grant awarded by the (foundation) to three of the authors. The support provided by the grant allowed for the research to be conducted but had no influence on the data collected or the interpretation of the findings. There are no conflicts of interest to report.