CDIE667957 5..30 Evaluating the quality of interaction in asynchronous discussion forums in fully online courses Dip Nandi*, Margaret Hamilton, and James Harland School of Computer Science and Information Technology, RMIT University, Melbourne, Australia (Received 12 September 2011; final version received 14 December 2011) Fully online courses are becoming progressively more popular because of their “anytime anywhere” learning flexibility. One of the ways students interact with each other and with the instructors within fully online learning environments is via asynchronous discussion forums. However, student engagement in online discussion forums does not always take place automatically and there is a lack of clarity about the ideal role of the instructors in them. In this article, we report on our research on the quality of discussion in fully online courses through analysis of discussion forum activities. We have conducted our research on two large fully online subjects for computing students over two consecutive semes- ters and used a grounded theoretic approach for data analysis. Our results reveal what students and instructors consider as quality interaction in fully online courses. We also propose two frameworks based on our findings that can be used to ensure effective online interaction. Keywords: asynchronous discussion forums; fully online course; quality frame- work Introduction Encouraging interaction in online courses in tertiary education has long been an interesting research topic. The introduction of fully online tertiary courses, such as those offered by Open Universities Australia (http://www.open.edu.au) or the Open University (http://www.open.ac.uk), means that interaction between teachers and learners must be enabled purely via online environments. Online interactive activi- ties can assist learners to share and gain knowledge from each other. Although there is a great deal of focus on better uses of the technology to support online learning, the way online interaction and participation can be designed has yet to be ade- quately investigated (Nandi, Chang, & Balbo, 2009). A gap exists in the literature, where the dedication to analyze the content of forum participation has overshad- owed detail into how quality can be evaluated and how to increase productive stu- dent participation (Blignaut & Trollip, 2003). In particular, most of the research in this area has been conducted in blended learning environments, that is, those in which there is a mixture of traditional face-to-face classes and online activities. Our work deals with courses offered in a fully online environment, in which there are no face-to-face classes. *Corresponding author. Email: dip.nandi@rmit.edu.au Distance Education Vol. 33, No. 1, May 2012, 5–30 ISSN 0158-7919 print/ISSN 1475-0198 online � 2012 Open and Distance Learning Association of Australia, Inc. http://dx.doi.org/10.1080/01587919.2012.667957 http://www.tandfonline.com http://www.open.edu.au http://www.open.ac.uk This article draws on literature concerned with online learning, interactivity, dis- cussion forum participation, and quality of participation for students and instructors. Our key research question is, “How can we evaluate quality online interaction in a fully online course?” To accomplish this research objective, qualitative analysis was performed on discussion forums in several fully online courses in a tertiary education context, using a grounded theoretic approach to capture the inner meaning of the data (Lechner, 2001; Strauss & Corbin, 1998). Based on this analysis, specific guide- lines are proposed through which productive interaction in fully online courses can be ensured. Background Online learning and interactivity Online learning comprises digitally formatted content accessible via the World Wide Web and channels of communication for instructors and students to actively interact between and among themselves (Ng & Murphy, 2005). The importance of interactivity in online learning is highlighted by research on online learning systems (Maor & Volet, 2007; Persico, Pozzi, & Sarti, 2010). The quality of this interaction determines the success of learning and teaching online. To support ongoing online interaction, a range of asynchronous (e.g., newsgroups and wikis) and synchronous (e.g., Elluminate and Skype) technologies has been used (Bradshaw & Hinton, 2004; Sher, 2009). The way instructors intervene in online forums can help to set up the direction for discussion and define its boundaries. Such intervention includes the methods by which instructors teach, guide, assess, and support students’ learning and construct knowledge (Albion & Ertmer, 2004), and is one of the most critical factors in enhancing student satisfaction in an online course. Moore (1989) distinguished between three types of student interaction in online courses: (1) student–student interaction, (2) student–instructor interaction, and (3) student–content interaction. Hence, the learning process is no longer an individual endeavor; rather it can incorporate and leverage the many-to-many relations among learners and instructors (Piccoli, Ahmad, & Ives, 2001). Dennen and Wieland (2007) suggested that learners must interact with each other and the course material at deeper levels, which has the potential for negotiation and internalization of knowledge rather than just rote memorization of knowledge. Asynchronous discussion forums are widely used to facilitate this kind of inter- action. Discussion forums and participation Online asynchronous discussion forums are becoming a common feature in on- campus and online courses as they allow students and instructors to communicate with each other regardless of time and space. Broadly, there are three levels of participation in such discussion forums: (1) “Lurkers” (Salmon, 2003) who simply read the messages and do not partici- pate. They may learn by reading the posts and incorporating the ideas into their assignments (Guzdial & Carroll, 2002). 6 D. Nandi et al. (2) Students who treat the forum as a notice board, posting their own position and having limited interaction. (3) The participation is interactive and used to its full potential (Ho, 2002). Gerbic (2006) identified the main motivators for participation in online discus- sion forums as the need to exchange ideas and seek feedback from instructors, and the de-motivators for online participation as irrelevant discussion and arrogant con- tributors. Participation is not always welcome by students who fail to understand the benefits gained. Some demonstrate their uncertainty by not actively participating in the discussions because of the lack of standard guidelines (Armatas, Holt, & Rice, 2003; Dennen, Darabi, & Smith, 2007; Farmer, 2004; Ramsden, 2003). Design for quality student participation The quality of discussion in online forums has been investigated and measured by several researchers from different angles. These include tone (Grady, 2003), gram- mar (Edelstein & Edwards, 2002), number of words (Biesenbach-Lucas, 2003), rea- soning (Edelstein & Edwards, 2002), level of controversy (Burstall, 2000), and content (Edelstein & Edwards, 2002; Grady, 2003; Im & Lee, 2003). The results of these investigations suggest that posts are not always clearly understood by every- one (Love, 2002) and enhanced interaction often occurs when controversial issues are discussed, with participants constantly disagreeing and clarifying (Blignaut & Trollip, 2003; Burstall, 2000). A conceptual framework by Nandi et al. (2009) defined the main themes on which qualitative online interaction can be designed. This framework identified 12 criteria classified under the three main themes of content, interaction quality, and objective measures. Although the framework looks at the quality of interaction related to student participation in online discussion forums, it does not provide guidelines about how learner–instructor interaction should be designed to ensure enhanced student interac- tion. In addition, the criteria in the framework were derived through research into blended learning environments. Researchers suggest that Web-based learning presents a format in which instruc- tors must interact with each student (Wagner, 2001). Volery (2001) and Meyer (2002) also emphasized the critical role of the instructor in promoting a high-quality online learning experience, identifying the role of the instructor as that of a “learn- ing catalyst and knowledge navigator” (Volery, 2001, p. 77). A comprehensive framework with a set of criteria is essential for facilitating interactions in fully online courses to ensure effective and high-quality interaction. This would make explicit to students the expectations of their engagement in dis- cussions, and thereby shape that engagement (Jackson, 2010). For this to occur, we need to address how the issue of quality of interaction can be defined and what are the general criteria for quality online interaction for both students and instructors. Facilitation by instructors Facilitation by instructors refers to the methods by which they teach, guide, assess, and support students’ learning (Albion & Ertmer, 2004). Student–instructor interac- tion is one of the most critical factors in enhancing student satisfaction in an online Distance Education 7 course. Students and instructors must find ways to convey information, determine level of understanding, and create a workable feedback system (Sher, 2009). Thurmond and Wambach (2004) suggested that interactions between students and instructors help students clarify and obtain a correct understanding of the course content. Baran, Correia, and Thompson (2011) suggested that because online students are expected to take greater control of their learning process and be more active in stimulating their peers’ learning, facilitation of online learning plays an important role in guiding these student-centered approaches. Dennen (2005) suggested that all aspects of facilitation require different approaches in the context of asynchronous communication. The instructor’s role in asynchronous discussion forums can vary from being the “sage on the stage” to being the “guide on the side” or even “the ghost in the wings” (Mazzolini & Maddison, 2003). Discussion in the literature generally suggests that it is important that instruc- tors play an active, visible part in forum discussions (Salmon, 2000). Consequently, it is natural to ask what that ideal role should be for an instructor, and in particular, how can we define guidelines for participating in ongoing discus- sion in a fully online course? Davidson-Shivers (2009) conducted a study and reported what types of informa- tion are shared with students by the instructors in online courses. While this study provides some guidelines for the type of information the instructor needs to convey to the students, it does not provide any principles for instructor participation in an ongoing discussion forum about specific topics and how to inspire the students or extend such discussion toward quality interaction. Research conducted by Mazzolini and Maddison (2007) shows that, according to students, the role of an online instructor should incorporate: • asking follow-up questions while answering one; • Introducing new concepts or new ways of thinking about solutions; • answering questions as soon as possible; • providing feedback; • discussing the student solutions from different dimensions or angles. However, instructors classified most of their participation in: • answering students’ questions; • asking leading questions; • while responding to students’ questions, asking subsidiary ones in order to continue the discussion thread. One school of thought proposes that instructors are critical to the success of an online discussion (Blignaut & Trollip, 2003; Ferdig & Roehler, 2004; Greenlaw & DeLoach, 2003; Love, 2002; Roblyer & Wiencke, 2003). The instructor is there to raise the discussion to a higher level, and giving students the responsibility to deter- mine the direction of discussion is not a viable approach (Moller, 1998). The other school of thought claims that instructors should take a back seat and let students construct their own knowledge (Burstall, 2000; Li, 2003; Mazzolini & Maddison, 2003; Poole, 2000; Rourke & Anderson, 2002). It is reasonable to conclude then that how instructors should be facilitating a discussion forum that provides the backbone of a fully online course is neither fixed 8 D. Nandi et al. in a way, nor uniform. Blignaut and Trollip (2003) summarized this by saying that our understanding of how to design online discussion in an effective and meaningful way is still developing. Through this research, we intend to identify the ideal role of the instructor in facilitating discussion forums in fully online courses. Research questions The foregoing discussion suggests that there is a need to investigate the role of the students and instructors in online discussion forums. The two sub-questions that we set out to explore as part of this investigation are as follows: (1) How can we evaluate quality interaction between students in fully online courses? (2) How can we define the ideal role of the instructor while interacting with the students in fully online courses? Investigating the above sub-questions will enable us to answer our key research question: How can we evaluate quality online interaction in a fully online course? We investigated these research questions using a case study, which is described in the following sections. Methodology and data analysis According to Yin (1994), a case study should be employed to examine questions related to how and why, where the investigators have little or no possibility of con- trolling events and the study is on contemporary phenomena in a real-life context. The case study approach was used in the study in the analysis of discussion forum contents in two fully online introductory computing courses at a large metropolitan university in Australia. Using this method we sought to investigate “how” related questions where we had no possibility of controlling events; our context being the real-life online interaction among students and instructors. Introduction to Information Technology and Introduction to Programming are taught in almost all introductory computer science or information technology (IT) degrees and in some non-IT-related degrees. In such settings, all the interac- tions between students, instructor, and content take place online when the subject is conducted in a fully online mode. For this research, we investigated how these interactions are being conducted and identified if they can be better facili- tated. The courses The research reported in this article was carried out in two fully online courses offered by Open Universities Australia (OUA): Introduction to Programming and Introduction to Information Technology, both for first-year undergraduate and post- graduate students. The Introduction to IT course covers elementary IT concepts, for example, computer fundamentals, operating systems and applications, the Internet, and spreadsheets. This course had students from various degrees including Bachelor of Technology, Bachelor of Business IT, Bachelor of Indigenous Studies, and Bachelor of Accountancy. Distance Education 9 The Introduction to Programming course covers introductory concepts of programming through the use of two programming languages: Alice and Java. Students enrolled in this course were only from the Bachelor of Technology degree. Students enrolled in these courses were located in different parts of Australia and also different parts of the world, that is, New Zealand, Canada, China, and India. Throughout the two study periods, one (same) instructor happened to be in charge of the courses. Apart from the Programming course in the study period September–November 2009, the instructor was assisted by a number of tutors (see Table 1). Both courses were conducted in a fully online mode. Data were collected from the online discussion forums from these two courses. Table 1 presents a general overview of the courses, their duration, and participants. Both courses had online discussion forums where students are encouraged to participate and interact with each other through Blackboard, the learning management system. Several threads were created by the instructor to allow students to communicate via the online medium similar to those recommended by Davidson-Shivers (2009). The discussion threads were: (1) Welcome and Introduction: this was set up to allow students to introduce themselves, which would make them known to each other virtually and take a step toward forming a virtual community of learners. (2) General Discussion: this was set up to allow students to ask and respond to questions about the general management of the course. Discussion in this thread mainly consisted of which programming language they should use and any problems in installing it, issues regarding navigating though the online environment, issues regarding how assessment would be implemented in the courses. (3) Assignment and Exam Discussion: the courses had separate threads for sep- arate assignments. Students and the instructor used this thread to ask and respond to questions regarding general assignment issues, marking guide- lines, and clarification of questions. In the programming course, assignments 1 and 2 were based on Alice programming language whereas assignment 3 was based on Java. In the IT course, assignment 1 was based on using hypertext markup language (HTML) to develop a basic website, and assign- ment 2 was based on writing reports. Students submitted assignments via Blackboard and sat for a paper-based exam at the end of the course adminis- trated by OUA. (4) Feedback: this was set up to allow students to provide overall feedback about the management and content of the course. Table 1. Overview of the courses. Courses Study period Students enrolled Tutors (apart from the instructor) A Intro to IT September–November 2009 299 2 B Intro to IT March–May 2010 406 3 C Intro to Programming September–November 2009 346 0 D Intro to Programming March–May 2010 301 2 10 D. Nandi et al. (5) Group Discussion Forums: these forums were facilitated mainly by the tutors (except in the programming course in 2009) and students used them to discuss weekly study materials. Students were divided into groups equally and each group was assigned a separate tutor. Group discussion forums were divided into weeks according to study material so that discussions could take place in relevant weeks. Each week questions relating to the week’s study material were uploaded in the group discussion forums by the tutors. The students were advised to post the answers to those questions in the discussion board and discuss each other’s responses. Students were also encouraged to post their own questions online and respond to each other’s queries. The tutors also posted questions to encourage stu- dents to discuss course materials in the group discussion forums. All other threads were facilitated by the instructor. Apart from the asynchronous discussion each week, the instructor and the tutors conducted a synchronous chat session via Elluminate, which is a text-, audio-, and video-enabled interaction tool. Attendance in the chat sessions was very low primar- ily because of the location, time, and work constraints of the students and hence we did not consider the interaction via synchronous mode for analysis in this article. Participants Table 2 provides a general overview of the background of the students. It also shows the number of undergraduate (UG) and postgraduate (PG) students in the courses along with the number of students who achieved a high distinction (HD) result of 80% or higher, those who failed the course, and those who did not sit for the course until the end and dropped out (DNS). Table 2 shows that most of the students were undergraduate students within the age range of 20–30 and using the fully online learning environment for the first time. There was no student younger than 20 or older than 40 years in these courses. The instructor for these four courses had a number of years’ experience in facilitating fully online courses. The tutors received training on facilitating group discussion. Data sources The content of the discussion forum in Blackboard for these two courses provided the data for this study. On average, there were 60–70 posts from students and 20–25 posts from the instructor and tutors in each week’s group discussion forums and in each of the assignment threads in the IT course. This number was around 30–40 for the students and 10–15 for the instructors in the programming course. Participation in the group discussion forum was assessed by the instructor based on his experience at Table 2. Overview of the students and final results. Course Results (%) Study level (%) Age level (%) First time online (%) HD DNS Fail UG PG 20–30 30–40 Yes No A 32.12 39.46 6.02 95 5 60 40 70 30 B 36.21 25.12 17.98 97 3 50 50 80 20 C 37.86 34.68 17.34 95 5 70 30 80 20 D 32.56 32.89 14.62 98 2 60 40 80 20 Distance Education 11 the end of the course. Students were not given any criteria for assessment beforehand, but it was mentioned that assessment would be based on the quality of posts and not quantity. This data was collected from two discussion forums from each of the courses over two study periods; one was an assignment discussion forum and the other was a general discussion forum on weekly topics from the course material. We decided to choose the “Assignment 1” and “Week 6” forums from the IT courses and “Assign- ment 1” and “Week 2” from the programming courses as these forums had a high participation rate. In the IT courses, “Week 6” discussion was about learning net- works and the Internet and learning the initial concepts of programming using Alice programming language in the programming courses. The topics for assignment 1 in both the courses are mentioned above under “Assignment and Exam Discussion.” Data analysis method We attempted to uncover all the themes by analyzing the discussion forum posts through qualitative data analysis. These themes provided a clear representation regarding what the participants regard as quality and productive discussion between students and the instructors and how they actually participated in the forums. Small-scale quantitative analysis was also performed to calculate the percentage of times each theme appeared in discussion. Qualitative data was processed using a grounded theoretic approach (Strauss & Corbin, 1998), that is, open, axial, and selective coding (Neuman, 2006) so that information relevant to the research could be extracted. All the data from the dis- cussion forums were loaded into NVivo 8 software for investigation through open, axial, and selective coding. A similar three-stage data analysis technique was used by Vlachopoulos and Cowan (2010) to explore the different styles and practice of e-moderation; they reported that this method is useful for gaining deep understand- ing of a phenomenon or theme from raw data. Our purpose in employing open coding was to identify the themes emerging from the discussion forums. After analysis of the data at the end of the open coding phase, we identified approximately 45–50 themes. Each separate concept in the data was labeled and similar ideas were grouped and labeled. Following open coding, the next step was axial coding, where the aim was to assemble coding categories into larger conceptual groupings (Glaser & Strauss, 1967). The two major catego- ries that emerged were student participation and instructor contribution. Each cate- gory consisted of a number of themes and sub-themes. This process was repeated until no additional categories were identified and all the data had been analyzed. The third and final coding step was selective coding. Again, the data were re- examined and the prior coding and grouping were revisited and verified or changed as required. This set of themes is presented in the “Findings and Discussion” section. At the end of the data analysis, the “student participation” category had around 14 sub- categories and “instructor contribution” had around 12 subcategories. This categoriza- tion explains what types of posts are valued as quality participation by the students and instructors. An example of the coding procedure is provided in the Appendix. We used prolonged engagement, persistent observation, triangulation, and peer debriefing (Baran & Correia, 2009) to ensure the credibility of the findings. The instructor was not a part of the investigation team, which eliminated any chance of bias in this analysis. One researcher analyzed the data, categorized the themes, and presented the findings to co-researchers through peer debriefing. Inconsistencies and 12 D. Nandi et al. disagreement were discussed and managed through consensus reaching. A complete explanation of the courses and participants are presented in the above sections to ensure the transferability of the study. The data analysis enabled the extraction of key and relevant information to the research and, as a result, the research question was explored based on the results ascertained. Through this research, we intend to pro- pose design principles through which online participation of students and contribu- tion of instructors can be developed in fully online computer science or IT courses. Findings and discussion Student participation To carry out the analysis we chose one weekly discussion forum and one assign- ment discussion forum each from the two courses. We have combined the findings from the IT (A and B) and the programming courses (C and D) as the content was common in both study periods (the themes are presented in Table 3). On average 40–50 students were active in the selected discussion forums. Table 3 presents the themes that emerged from the analysis of the data of all of the courses put together. The percentages relate to the proportion of times the themes emerged during all the student interactions. The themes are discussed below along with the actual posts from the discussion forum. Table 3. Themes related to student participation appearing in the discussions. Criteria Intro to IT Intro to Programming Asking questions Administrative (+ for assignments) 1.15% 6.96% Leading questions 8.62% 0.87% Questions drawn from own experience and real-world situation 9.77% 0 Straight and in detail 10.34% 18.26% With lines of code 0 1.74% To instructors 1.72% 0 Answering questions Straight and in detail 14.37% 16.52% With tips 0 3.48% With real-world or own coding example 12.64% 3.48% Justification 5.75% 6.09% Acknowledgement for understanding 2.87% 0 Asking for feedback From students 1.72% 2.61% From instructors 1.15% 2.61% Clarification 9.20% 18.26% Critical discussion of contribution 1.72% 1.74% Ideas from interaction 1.72% 2.61% Opinions regarding the topic of discussion 6.90% 0 Providing feedback 0 1.74% Sharing own experience and knowledge 6.90% 6.96% Suggesting multiple solutions 0.57% 0.87% Distance Education 13 Asking questions Asking questions refers to queries regarding the course material or management of the subject. We found that the students were asking many questions in the discussion forums. This indicates they were active in their courses, trying to grasp the subject material, wanting to learn and hence asking questions. It also shows that students considered themselves within a virtual community of learners and felt free to submit their queries online. We classified the questions into the following categories. Administrative (+ for assignments). This type of question refers to queries where students were trying to understand the management of the course, that is, when tests were due, what software to use, and similar issues: Will you be placing an answers section to the tutorial questions, so we can mark our own progress? (Intro to IT_B) I was wondering if there was some way once the first assignment had been uploaded to “weblearn” whether it could be further modified or retrieved. (Intro to Prog_D) Relatively few such administrative questions were asked by the students. Leading questions. Leading questions refer to queries that came out of a post in the discussion board. It indicates that students have read the posts but do not fully understand the meaning prompting them to ask follow-up questions, such as the fol- lowing: So if there is, for all intents and purposes, only one backbone to/from Australia, what happens when/if there is a serious disruption or disabling to that line. (Intro to IT_A) Drawn from own experience and real-world situation. This type of question shows that students are trying to link the course material with real-world situations: When I access or send emails using a web-based account, such as yahoo or gmail, what protocols are at play? (Intro to IT_B) Understandably, a lot of such questions were asked during the discussion in the IT course, where most of the students are familiar with day-to-day use of IT. In comparison, very few such questions were asked in the programming course, which might be due to the different nature of the subject material of the courses. Straight and in detail. These are straightforward questions posted by the students regarding the topic of discussion. There were lots of such questions posted in the discussion board in both courses. With lines of code. This type of question was mainly observed in the introductory programming course. It actually serves two purposes. Firstly, it shows that the students have tried to solve the problem unsuccessfully and were confused; secondly, it makes it easier for others to answer the query by looking at the code: If (humvee distance to redbox+humvee distance to redbox)=redbox distance to bluebox) 14 D. Nandi et al. I’m having a major brain haemorrhage with this, I can only figure that it fails to be true because distance to is measured from the humvees edge rather than centre point. Any help appreciated much. Is there a better way that i am missing? (Intro to Prog_C) (Humvee, redbox and bluebox are objects from assignment 1 where students were required to compute the distances between each of these objects in the Alice pro- gramming environment, so the objects would not collide.) To instructors. These are straightforward questions directed toward instructors only, such as: Are you able to elaborate on Q1. part 2 below as I seem not understand the question clearly.1. Who (or what) is typically connected to each type of communication line? (Intro to IT_B) Sometimes during discussions, students appeared confused by the variety of information presented in the forum and asked direct questions to the instructors for clarification. Others asked to make sure they were on the right track before they went further along. Answering questions Answering questions refers to responding to the queries in the discussion forums. Our data analysis shows the students were freely answering questions in a variety of ways, which we have classified in the following categories. Straight and in detail. This refers to answering questions in a straightforward manner. These questions might be asked by the instructors or posted by other students in the forum. This practice assists in sharing and reinforcing knowledge. Students posted these types of answers to tutorial questions as well as questions from other students. With tips. Answers with tips do not directly provide a solution for the question or the problem, but provide some guidelines depending on which solutions can be worked through. These types of answers are especially important while discussing assignments where students and instructors are not supposed to give away the solution through their answers and instead provide clues: Look very carefully at all of the proximity functions and you’ll find one that is more suitable. (Intro to Prog_C) Understandably, these types of answers were seen in the introduction to the programming course as writing the code would give away the actual solution. With real-world or own coding example. Providing examples while answering questions allows students to link the theory with real-world practice. Many such answers were seen in the introductory IT course: Twitter has just removed XMPP for latency concerns. (Intro to IT_A) I have solved the problem with a few more IF statements. :) (Intro to Prog_D) Distance Education 15 These posts assisted students to apply theory to practice while answering ques- tions so other students could learn from them. Students also posted programming code. Justification. Justification refers to providing clear rationalization for the posts while participating in online forums. It acts as a source of validation for a specific comment. Students tried to justify their answers in different ways, for example (as mentioned above), by providing a link or the source for the information or by providing examples that show the application of the theory: This link explains it well and in detail of just what I said. http://www.ltg.ed.ac.uk/~ht/ WhatAreURIs/ (Intro to IT_B) This article outlines the terminology that I mentioned quite well—http://computerpro- gramming.suite101.com/article.cfm/procedure__subroutine_or_function_ (Intro to Prog_C) The above posts assisted others in the discussion board to verify the answer by visiting the links and acquire more knowledge. Students in both courses tried to provide a source or link while answering questions. Acknowledgement for understanding Acknowledging for understanding is an indication that the discussion is productive and may inspire other students to engage in effective discussion: After viewing some of the post’s i think I now know how this part of html code works :-) (Intro to IT_B) Asking for feedback Asking for feedback refers to posts where students comment or give solutions and ask other students or instructors for verification. From student: Am I on the right track here? (Intro to IT_A) From instructor: So. Mr. X, COULD YOU PLEASE SHED SOME LIGHT ON THIS (Intro to Prog_C) It shows that students were free to communicate and create a thoughtful interac- tion online. Students in both courses asked for feedback from other students and instructors. Clarification Clarification refers to explaining the posts clearly so the meaning can be easily understood. This is one of the most important criteria for participating in any dis- cussion forum. Technical courses have a lot of scope for misunderstanding and mis- interpretation: 16 D. Nandi et al. http://www.ltg.ed.ac.uk/~ht/WhatAreURIs/ http://www.ltg.ed.ac.uk/~ht/WhatAreURIs/ http://computerprogramming.suite101.com/article.cfm/procedure__subroutine_or_function http://computerprogramming.suite101.com/article.cfm/procedure__subroutine_or_function The relationship between International backbones are as follows: Shark eats Big fish which eats little fish which eats even littler fish :) i.e. Backbone > Regional Network > ISP’s > Me (Intro to IT_A) Objects in the Alice world will have both “properties” and “methods”—properties are like the attributes of an object for example its colour, the methods will be action the object is capable of performing—such as moving. (Intro to Prog_C) In all courses, students tried to clarify their posts in the above-mentioned ways to make logical sense to everyone in the discussion forum. Critical discussion of contribution Critical discussion of contribution covers agreement or disagreement with posts providing a logical explanation of the reasons. This assists students to learn the topic under discussion more clearly and to reflect on their own posts to consider different ways of answering an IT question or solving a programming problem: You say “the more expensive VoIP.” I don’t know what the costs are but isn’t the purpose of VoIP to supply cheap call rates using Internet technology. (Intro to IT_A) It appears your trying to overcomplicate something that is relatively simple … (Intro to Prog_D) Very few students critically discussed their peers’ posts, and this criterion should be encouraged by the instructors. Ideas from interaction Ideas from interaction refer to students learning a new concept from other students and using that knowledge to solve a problem or answer a question: I think there can be more than one backbone per area, because as Alison said, what if one gets disrupted, we would be completely cut-off. (Intro to IT_A) Thanks hips, i didn’t try “forward,” because worried that her upper torso would actually leaving her lower torso, I’ll try it now n c what happens. (Intro to Prog_D) This criterion shows that interaction was productive as students were learning from each other. Opinions regarding the topic of discussion Opinion-based posts may not be very helpful for other students. However, they can trigger discussion if other students reply and contribute new knowledge, creating an atmosphere for further more meaningful discussion: I would think instant messaging is not possible on a standard phone. (Intro to IT_B) Students posted their opinions about the course material in the IT course and interestingly, as noted previously, none in the programming course. Almost every time these types of opinions were posted, productive discussion was triggered. Distance Education 17 Providing feedback Providing feedback to each other is a criterion that shows students freely assist each other to develop knowledge: I completely agree with your statements. However I am sure that it is all workable with the left right things. (Intro to Prog_C) This criterion occurred only twice. Sharing own experience and knowledge This classification refers to students sharing their own experiences while interacting in the discussion forum. Sharing the experience of solving a problem can provide an assurance to others that certain IT or programming questions can be solved in that way: I’ve used Lotus Sametime at a company I worked for and it uses the same SIP as googletalk (Intro to IT_A) Here here!! I have found out by doing the prac that we were given, that the “objects” (skaters, cows, etc) move in relation to the way they are facing … (Intro to Prog_C) The above quotes indicate that students have tried or completed the task, and they are willing to share their experience and what they have learned from it. Suggesting multiple solutions Suggesting multiple solutions for a single problem shows that the student has done some research regarding that problem. Furthermore, it assists other students to con- sider different angles about certain problems or questions and in this way acquire more knowledge: The alternative would be to use the corresponding IP address which isn’t really the most user friendly approach (Intro to IT_B) These were the only two multiple solutions suggested in either discussion forum. Relevance and informal posts Relevance in participation refers to posts that are directly or indirectly related to the subject of discussion. Most of the posts in our data were relevant to the topic. A few informal posts were observed, however, and have been included here to show that they can assist in building a learning community. It is easier to post informal funny messages if the students feel they belong to the community: Sweet dreams and remember ice packs for the flying fingers. :-P (Intro to IT_B) The world is “large,” u didn’t get lost, just shift the world, u can find “yourself” again:-) (Intro to Prog_C) 18 D. Nandi et al. This type of informal posting made the discussion lighthearted and interesting. Instructor contribution Table 4 presents the themes that emerged from our data analysis. They provide an overview of how instructors (and tutors) facilitated the discussion forums. Table 4 also presents the number of times the themes appeared as a percentage of the total contributions in the forums. This highlights the different emphases of both courses. We discuss the themes further below and provide examples from the posts in the discussion forums. Administrative guidelines or technical assistance Administrative guidelines refer to the rules and regulations of the course and the strategies that should be followed by the students. The instructors and tutors pro- vided guidelines and technical assistance during the initial weeks of the course in the following ways: There are two group discussions boards, located away from the main Assignment discussion board. (Intro to IT_A) The a2w files must be opened from within alice, double clicking on them may open them in a zip program for extraction. (Intro to Prog_D) This guidance made it easier for online students to settle down and get into the subject materials. It also clarified which software to use or not to use and how to get the best out of a fully online course. As the students were fully online, this is essential, as it sets the tone for the study ahead. Clarification of questions Clarification refers to providing a clear explanation of the problem and surrounding issues. Examples of posts using this theme occurred only for the programming course, where instructors attempted to clarify the problem scenario in the assign- ment specification: Yes, you can use the existing Alice Functions and method parameters, but must NOT create new ones. (Intro to Prog_D) Table 4. Themes related to instructor participation as they appeared in the discussion. Criteria Intro to IT Intro to Programming Administrative guideline or technical assistance 2.65% 8.93% Clarification of questions 0 7.14% Declaration of expectation 15.93% 3.57% Periodic intervention to direct and extend discussion 17.70% 0 Promoting deep learning 4.24% 19.64% Providing direct answers 9.73% 50% Providing feedback with example 15.04% 0 Providing feedback 18.58% 10.71% Raising new questions 15.93% 0 Distance Education 19 Mostly evident in the assignment discussion, this theme allowed students to think in the appropriate way to solve a certain problem. Declaration of expectation It is important for the instructors to set expectations to assist students to understand what the instructor wants out of them and enable them to act accordingly. It specifies what the students should be doing to achieve the ultimate goal of learning in the course: Participate in tutorial discussion for one group for each week. (Intro to IT_A) It’s not something that would be examined; it’s a point of extension for those whom are curious. (Intro to IT_B) Particularly in a course like Introduction to Information Technology, which covers vast areas from the field of IT, it was important to clearly state expectations, in appropriate contexts like the above. Periodic intervention to direct and extend discussion This theme refers to continuing the discussion by broadening the focus while ensur- ing it does not halt after a certain point. Consistent intervention by the instructor keeps the discussion on track: For those who would like a little more to debate about, a valid point was raised between cable TV and cable Internet. (Intro to IT_A) Why do you think other countries would have Cable more popular? Do you think all cables are fibre optic? (Intro to IT_B) This intervention also assists in keeping the topic relevant, but was evident only in the IT course data. This verifies Nandi, Hamilton, Harland, and Warburton’s hypothesis that it might not be easy to extend a discussion in an introductory programming course (2011). The course content is narrow and often has only one solution to a problem. Providing feedback Periodic and summarized feedback is regarded as one of the major roles of instruc- tors (Mazzolini & Maddison, 2007). Feedback provides students with an overview regarding whether they are on the right track or not: X was on the right track and Y nailed it here. (Intro to IT_A) I am assuming you mean the distance between the cars, distance to is measured between the centre axis of the two objects, don’t worry too much about it looking right. (Intro to Prog_D) Periodic feedback from instructors and tutors was provided in both courses. Mostly evident in the IT course, this theme assisted students to clarify their way of thinking and determine a solution. 20 D. Nandi et al. Providing feedback with example Feedback with an example explains to the students in which way they should be con- centrating on solving the problems. This theme was evident only in the introductory IT course: Not really, since most of these are private commercial deals that us mere mortals are not allowed to know. (Intro to IT_B) Examples provided by the instructors, such as sample code for a problem or examples of how a network structure works, were probably considered as the most credible source of information by the students. Promoting deep learning Providing hints for discussion refers to not giving the answers directly but provok- ing the students’ thinking processes by providing certain helpful clues. It assisted the discussion to move ahead and helped students to find solutions: The simple idea of this question is to get you thinking about the differences between Cable and ADSL. (Intro to IT_A) You have just answered your own question; think about what you said in the question. (Intro to Prog_D) Providing hints for discussion to promote deep learning was a feature in both courses, especially in the programming course. Providing direct answers Providing direct answers assists students to learn what the solution to a problem is and verify their own research. Students should be provided enough time to engage with the problem before being given direct answers. There were a lot of direct answers provided by the instructors. Raising (new) questions Asking new questions provides the students with a chance to explore beyond the course material and learn more: Do you think the benefit would be great anyhow? (Intro to IT_B) Only evident in the IT course, this theme was used by the instructors lots of times during discussion. Lessons learned The key focus of this research was on the quality of interaction of the students and instructors in the discussion forum. By analyzing the discussion forum participation, we have uncovered several themes that can act as a base for designing online par- ticipation and several important features that affect the quality of participation. These are discussed in the following. Distance Education 21 Student participation On a general level, students are making the most of their online forum discussions to gain, share, deepen, and expand knowledge. A number of criteria in the area of cognitive skills, use of both formal and informal language (Gerbic, 2006), and frequency of participation were evident (Nandi et al., 2009). Themes such as justifi- cation of posts, clarification of ideas, critical discussion of contribution, and sugges- tions of multiple solutions were valued and exercised by the students regularly. Some of the criteria (Nandi et al., 2009) were not evident in the discussion, such as prioritization and interpretation; while some new and different themes emerged from data analysis, such as asking questions and providing feedback. The main tasks that students were performing in the discussion forum were ask- ing and answering questions. Research to date on analyzing themes for online par- ticipation (Nandi et al., 2009) largely focuses on how to answer questions and not on how the answers can be justified. Most of the themes provide guidelines on how quality responses should be posted in online forums. Our research verifies most of them as students used sources or links and examples while providing answers and wanted others to justify their responses. Different types of questions were being asked by the students in the two courses, which triggered discussions most of the times. We classified these ques- tions into six categories. They are administrative (+ for assignments), leading ques- tions, questions drawn from own experience and real-world situations, straight and in detail, with lines of code, and questions directed to instructors. Almost 50% of the posts in both courses were questions and answers from the students. The next highest number of posts were classified in the clarification category, which consisted of around 9% and 18% of the posts in the IT course and programming course, respectively. Researchers have argued that comments or answers posted online should be accurate and backed up by justification or clarification (Edelstein & Edwards, 2002). We have found that posting opinions can also be useful as it assists in trig- gering discussion. Blignaut and Trollip (2003) and Burstall (2000) suggested that controversial posts attract enhanced interaction. This theme was evident only in the introductory IT course, where students probably knew about the uses of IT in their day-to-day life and were trying to relate it with the topic of discussion. The content for the programming course is more prescriptive, algorithmic, and more narrowly focused; hence opportunities for direct discussion and asking ques- tions are limited. Often a single solution posted by a student in answer to a problem raised by another student or the instructor ended the discussion at that point. The same situation applied to assignments. Once the solution was obtained, there was little and no variety of solution provided. Conversely, the intro to IT course, which consists of basic topics from general IT covering a vast area from both hardware and software, often had a lot of discussions about these topics from different angles. During these, students pointed toward examples and real-world situations from past and current use of IT in their personal and work life, which broadened the discussion. This could explain why more questions (9.77%) and answers (12.64%) drawn from own experience or real-world situations were posted in the introductory IT course and very few in the programming course (3.48% only answers). On the con- trary, a lot more administrative questions were asked in the programming course 22 D. Nandi et al. (6.96%) than the IT course (1.15%), where students were trying to become acquainted with which software program to use and how to install it. It confirms Nandi et al.’s findings that the relative importance and the relevance of each theme depends very much on the instructors, the subject matter content, and the cohort and demography of the online students (2009). Based on the findings of our research reported in this article, we have modified the earlier framework (Nandi et al., 2009) and proposed this new framework in Table 5 for evaluating the quality of student interaction in fully online courses. This new framework consists of 11 criteria and assessment guidelines for each criterion. We did not include the criteria of “objective measures” (Nandi et al., 2009, p. 668) in this framework, and intend to investigate this further in the future. Instructor contribution Instructors played an active role in initiating and carrying the discussion forward. Data analysis indicates that periodic feedback from instructors is always valued highly by students and keeps the students on track, and hence this feedback is essential. Our results validate that handing students the responsibility to direct dis- cussion is not always the best option (Moller, 1998) and instructors should be in control of the discussion at all times through an active presence. There were many questions being asked by the students in both the courses that could explain why instructors had to provide so many direct answers. In the programming course, 50% of all posts by the instructors were direct answers to questions and none was related to extending or directing discussion. Conversely, only 9.73% of the posts in the IT course were direct answers and 17.70% were related to extending discussion, which highlights the impact of content on discussion. It falls to the instructor to draw a balance between these two criteria of answering direct questions and providing clues or hints while facilitating discussion. Investigation of the data reveals that it is important to provide administrative or technical guidance early in the course. Technical courses like IT and programming can sometimes be hard to study initially and the fully online environment of study adds to the problem. Students also need to know which software to install and guid- ance on how to install it. Around 7% of the posts by the students in the program- ming course were related to administrative issues and around 9% of the posts by the instructors were responses to those questions. Hence, clear and detailed guide- lines can assist the students to become accustomed to the fully online mode of learning early on. Instructors should declare early in the course their expectations of the students on how to participate and acquire the best out of the discussion forum. This decla- ration may consist of directions regarding how many and how often students should post in the discussion board, what should be the pattern of their contribution, how the students should approach the subject, and in general what is expected of them. The expectation might be different considering the difference in the content of the courses (Nandi, Hamilton, Harland, & Warburton, 2011). Hence, through specific subject-specific guidelines, students can follow the guidance and try to achieve the goal of learning accordingly. In both the courses, instructors set up threads named “Welcome and intro” reserved exclusively for students to introduce themselves. A lot of the posts in the first teaching weeks of the courses allowed students to introduce themselves and Distance Education 23 T ab le 5 . F ra m ew o rk fo r ev al u at in g in te ra ct io n q u al it y b et w ee n st u d en ts . C ri te ri a P o o r S at is fa ct o ry G o o d E x ce ll en t A sk in g q u es ti o n s A sk in g ir re le v an t q u es ti o n s Q u es ti o n s fr o m su b je ct m at te r o r to p ic o f d is cu ss io n Q u es ti o n s w it h cl ea r b ac k g ro u n d Q u es ti o n s in d ic at in g ab il it y an d ev id en ce to ca rr y o u t re se ar ch A n sw er in g q u es ti o n s P ro v in g w ro n g an sw er P o st in g co rr ec t an sw er s P ro v id in g d et ai le d an sw er s D et ai le d an sw er s w it h ex am p le s an d su g g es ti n g m u lt ip le so lu ti o n s if ap p li ca b le Ju st ifi ca ti o n N o ju st ifi ca ti o n o f p o in ts Ju st ifi ca ti o n b as ed o n p er so n al o p in io n Ju st ifi ca ti o n u si n g ex is ti n g ca se s, co n ce p ts , o r th eo ri es Ju st ifi ca ti o n u si n g ex is ti n g ca se s, co n ce p ts , o r th eo ri es an d p ro v id in g cl ea r d is cu ss io n o f im p li ca ti o n s C la ri fi ca ti o n R eg u rg it at io n o f in fo rm at io n A cl ea r ex p la n at io n o f av ai la b le in fo rm at io n E x p la in in g av ai la b le in fo rm at io n u si n g re le v an t ex am p le s A rt ic u la ti n g av ai la b le in fo rm at io n to ex p an d o n id ea s p re se n te d , in cl u d in g th e u se o f ex am p le s C ri ti ca l d is cu ss io n o f co n tr ib u ti o n s N o en g ag em en t w it h o th er le ar n er s’ co n tr ib u ti o n s S o m e b as ic d is cu ss io n ab o u t o th er le ar n er s’ co n tr ib u ti o n s C o n si st en t en g ag em en t w it h o th er le ar n er s’ co n tr ib u ti o n s an d ac k n o w le d g em en t o f o th er le ar n er s’ co m m en ts o n o w n co n tr ib u ti o n s C o n tr ib u ti n g to a co m m u n it y o f le ar n er s, w it h co n si st en t en g ag em en t an d ad v an ce m en t o f ea ch o th er ’s id ea s Id ea s (+ n ew ) fr o m in te ra ct io n s N o ev id en ce o f n ew id ea s an d th o u g h ts fr o m in te ra ct io n S o m e n ew id ea s d ev el o p ed as a re su lt o f in te ra ct io n S o m e so lu ti o n s an d n ew id ea s as a re su lt o f in te ra ct io n s C o ll ab o ra ti v e ap p ro ac h to so lu ti o n se ek in g an d n ew id ea s d ev el o p ed P o st in g o p in io n s O p in io n o n ir re le v an t to p ic O p in io n s o n re le v an t to p ic O p in io n s th at tr ig g er d is cu ss io n O p in io n s th at in d ic at e th e k n o w le d g e o f th e su b je ct m at te r an d p ro m p ts fe ed b ac k P ro v id in g fe ed b ac k W ro n g an d sh o rt fe ed b ac k A cc u ra te fe ed b ac k D et ai le d fe ed b ac k D et ai le d fe ed b ac k w it h ac k n o w le d g em en t fo r u n d er st an d in g if ap p li ca b le S h ar in g k n o w le d g e an d ex p er ie n ce N o sh ar in g o f o u ts id e k n o w le d g e S h ar in g g en er ic in fo rm at io n th at is ea si ly av ai la b le fr o m o u ts id e so u rc es S h ar in g re al -w o rl d ex am p le s th at m ay n o t b e im m ed ia te ly o b v io u s to o th er le ar n er s S h ar in g re al -l if e k n o w le d g e, p er so n al ex p er ie n ce , an d ex am p le s o f si m il ar p ro b le m s/ so lu ti o n s R el ev an ce (C o n ti n u ed ) 24 D. Nandi et al. N o ap p li ca ti o n o r d is cu ss io n o f re le v an ce to q u es ti o n s as k ed A p p li ca ti o n o f k n o w le d g e to q u es ti o n s as k ed A p p li ca ti o n o f k n o w le d g e in cl u d in g d is cu ss io n u si n g re le v an t ex am p le s K n o w le d g e is cr it ic al ly ap p li ed an d m ay in cl u d e d is cu ss io n o f li m it at io n s U si n g so ci al cu es to en g ag e o th er p ar ti ci p an ts N o en g ag em en t w it h o th er s in th e d is cu ss io n fo ru m A n sw er in g so m e b as ic q u es ti o n p o se d b y fa ci li ta to r o r o th er le ar n er s E n g ag in g w it h th e w o rk an d d is cu ss io n o f o th er le ar n er s E n g ag in g an d en co u ra g in g p ar ti ci p at io n w it h fe ll o w d is cu ss an ts in th e fo ru m Distance Education 25 meet each other virtually. On-campus students enjoy the benefits of easily establish- ing a group to study together. Therefore, a sense of virtual community is required so that students can feel free to interact with each other and share knowledge and ideas. Initiatives for building a virtual community need to be taken early in the course to allow students to communicate with each other to break the ice by introducing themselves. The above discussion clarifies that instructors and tutors fulfilled all the criteria mentioned by Mazzolini and Maddison (2007) as the ideal role of the online instructor. Baran et al. (2011) identified the ideal role of an online instructor. Our research has investigated how the instructor’s role might influence the quality of interaction in a fully online computing course. We decided to match our findings with the ideal role outlined by Baran et al. (2011). Based on our research presented in Table 4, we have identified the roles played by instructors in online discussion forums and matched these with the roles identified by Baran et al. (2011) for the ideal instruc- tor. We now propose a new framework to provide implementation guidelines for online instructors (see Table 6). This framework can provide guidelines for instructors on which roles to perform and how to execute them. It can also be beneficial for large classes where instruc- tors are assisted by tutors and individual roles can be clearly divided and defined. Conclusion and directions for further research We have investigated the quality of interaction with a view to evaluating quality in online discussions in fully online courses. Due to the exploratory nature of the research, the major focus has been to identify key themes that apply to online for- ums in fully online courses. We presented the key themes that emerged in Tables 3 and 4, identified from student and instructor contributions. A number of issues relat- ing to effective online participation and engagement were discovered through the analysis and discussed as lessons learned. Table 6. Ideal roles of an instructor and how to implement them. Ideal role of an instructor (from Baran et al., 2011) How to implement it (based on our research in Table 4) Managerial and instructional design • Providing administrative guideline • Declaring expectations Pedagogical • Clarifying questions and problems • Periodic intervention to direct and extend discussion • Promoting deep learning • Raising new questions Technical • Proving technical assistance Facilitator • Providing direct answers • Providing feedback (+ with examples) Social roles • Initiatives for community building 26 D. Nandi et al. In order to gain a better understanding of what it means by quality of participa- tion, the two major areas that were researched, analyzed, and extended were the type of participation by the students and the facilitation activities by the instructors. Our results and frameworks define a set of criteria for instructors to implement for quality participation for interactive learning. Results of the data analysis show that students were actively participating in the discussion, asking and answering questions. In response, instructors posted both direct answers and hints to promote deep learning of important course content. They also actively attempted to extend the discussion and raise new questions in the IT course and provided feedback with examples relevant to the course content. Our research shows that rather than designing a fully student-centered or instruc- tor-centered discussion, a combination of both approaches can be advantageous. This requires students and instructors to take responsibility to construct and share knowledge and ideas. Students can have guidance on what is expected of them through our framework in Table 5. Instructors can design their role and workload through the framework in Table 6. The themes and frameworks presented in this article provide clear guidelines that can be used as design principles for developing and supporting quality discussion forums in fully online courses. We plan to extend this research by applying the set of criteria and frameworks in online courses over multiple semesters to investigate patterns over time. Future research could extend our evaluation and include learner content interaction in the analysis. The implications of the frameworks can be tested in different higher edu- cation contexts with different online courses. This research can provide more insights into how students and instructors interact to learn and develop in online courses. The effects of the frameworks on design and structure of online activities and role distribution could benefit from future research. Notes on contributors Dip Nandi is a PhD candidate in the School of Computer Science and Information Technology at RMIT University. His research interests include human–computer interaction, online learning, computer science and IT education, the use of technology in education, and social communication. Margaret Hamilton is a senior lecturer in the School of Computer Science and Information Technology at RMIT University. Her research interests include human–computer interaction, computer science education, computer modeling for mobility, sustainability and transportation, usability testing, and ethnographic action research. James Harland is an associate professor in the School of Computer Science and Information Technology at RMIT University. His research interests include computing theory, artificial intelligence, proof theories, and educational technologies. References Albion, P. R., & Ertmer, P. A. (2004). Online courses: Models and strategies for increasing interaction. In Proceedings of AusWeb 04, Tenth Australian World Wide Web Conference. Retrieved from http://ausweb.scu.edu.au/aw04/papers/refereed/albion/index.html Armatas, C., Holt, D., & Rice, M. (2003). Impacts of an online-supported resources-based learning environment: Does one size fit all? Distance Education, 24, 141–158. doi: 10.1080/0158791032000127446 Baran, E., & Correia, A. (2009). Student-led facilitation strategies in online discussions. Distance Education, 30, 339–361. doi: 10.1080/01587910903236510 Distance Education 27 http://ausweb.scu.edu.au/aw04/papers/refereed/albion/index.html Baran, E., Correia, A., & Thompson, A. (2011). Transforming online teaching practice. Critical analysis of the literature on the roles and competencies of online teachers. Distance Education, 32, 421–439. doi: 101080/01587919.2011.610293 Biesenbach-Lucas, S. (2003). Asynchronous discussion groups in teacher training classes: Perceptions of native and non-native students. Journal of Asynchronous Learning Networks, 7(3). 24–46. Retrieved from http://sloanconsortium.org/publications/jaln_main Blignaut, S., & Trollip, S. R. (2003). Developing a taxonomy of faculty participation in asynchronous learning environments—An exploratory investigation. Computers & Education, 41, 149–171. doi: 10.1016/S0360-1315(03)00033-2 Bradshaw, J., & Hinton, L. (2004). Benefits of an online discussion list in a traditional distance education course. Turkish Online Journal of Distance Education, 5(3). Retrieved from http://tojde.anadolu.edu.tr/index.htm Burstall, J. (2000). Learning communities for social change in forums on the web. Austra- lian Journal of Adult Learning, 40(1), 33–52. Davidson-Shivers, G. V. (2009, Spring). Frequency and types of instructor interactions in online instruction. Journal of Interactive Online Learning, 8, 23–40. Retrieved from http://www.ncolr. org/jiol Dennen, V. P. (2005). From message posting to learning dialogues: Factors affecting learner participation in asynchronous discussion. Distance Education, 26, 127–148. doi: 10.1080/01587910500081376 Dennen, V. P., Darabi, A. A., & Smith, L. J. (2007). Instructor–learner interaction in online courses: The relative perceived importance of particular instructor actions on performance and satisfaction. Distance Education, 28, 65–79. doi: 10.1080/01587910701305319 Dennen, V. P., & Wieland, K. (2007). From interaction to intersubjectivity: Facilitating online group discourse processes. Distance Education, 28, 281–297. doi: 10.1080/ 01587910701611328 Edelstein, S., & Edwards, J. (2002). If you build it, they will come: Building learning com- munities through threaded discussions. The Online Journal of Distance Learning Admin- istration, 5(1). 21–34. Retrieved from http://www.westga.edu/~distance/ojdla/ Farmer, J. (2004). Communication dynamics: Discussion boards, weblogs and the develop- ment of communities of inquiry in online learning environments. In R. Atkinson, C. McBeath, D. Jonas-Dwyer, & R. Phillips (Eds. ), Beyond the comfort zone: Proceedings of the 23rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 274–283). Retrieved from http://www.ascilite.org.au/conferences/ perth04/procs/farmer.html Ferdig, R. E., & Roehler, L. R. (2004). Student uptake in electronic discussions: Examining online discourse in literacy pre-service classrooms. Journal of Research on Technology in Education, 36. 119–136. Retrieved from http://www.iste.org/learn/publications/jour- nals/jrte-old.aspx Gerbic, P. (2006). To post or not to post: Undergraduate student perceptions about participating in online discussions. In L. Markauskaite, P. Goodyear, & P. Reimann (Eds.), Proceedings of the 23rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 271–281). Sydney: Sydney University Press. Retrieved from http:// www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p124.pdf Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Chicago, IL: Aldine. Grady, D. B. (2003). Mapping online discussions with lexical scores. Journal of Interactive Learning Research, 14. 209–229. Retrieved from http://www.aace.org/pubs/jilr/ Greenlaw, S. A., & DeLoach, S. B. (2003). Teaching critical thinking with electronic discus- sion. Journal of Economic Education, 34(1). 36–53. Retrieved from http://www.tandf. co.uk/journals/titles/00220485.asp Guzdial, M., & Carroll, K. (2002). Exploring the lack of dialogue in computer-supported collaborative learning. In G. Stahl (Ed.), Proceedings of the Conference on Computer Support for Collaborative Learning: Foundations for a CSCL Community, CSCL ’02 (pp. 418–424). Boulder, CO: International Society of the Learning Sciences. Ho, S. (2002, February). Evaluating students’ participation in on-line discussions. Paper presented at the Australian World Wide Web Conference (AUSWEB), Sunshine Coast, Queensland, Australia. Retrieved from http://ausweb.scu.edu.au/aw02/papers/refereed/ho/ paper.html 28 D. Nandi et al. http://sloanconsortium.org/publications/jaln_main http://tojde.anadolu.edu.tr/index.htm http://www.ncolr.org/jiol http://www.westga.edu/~distance/ojdla/ http://www.ascilite.org.au/conferences/perth04/procs/farmer.html http://www.ascilite.org.au/conferences/perth04/procs/farmer.html http://www.iste.org/learn/publications/journals/jrte-old.aspx http://www.iste.org/learn/publications/journals/jrte-old.aspx http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p124. pdf http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p124. pdf http://www.aace.org/pubs/jilr/ http://www.tandf.co.uk/journals/titles/00220485.asp http://www.tandf.co.uk/journals/titles/00220485.asp http://ausweb.scu.edu.au/aw02/papers/refereed/ho/paper.html http://ausweb.scu.edu.au/aw02/papers/refereed/ho/paper.html Im, Y., & Lee, O. (2003). Pedagogical implications of online discussion for preservice teacher training. Journal of Research in Technology Education, 36. 155–170. Retrieved from http://www.iste.org/learn/publications/journals/jrte-old.aspx Jackson, K. (2010). What value assessment rubrics in shaping students’ engagement in asyn- chronous online discussions? In C. H. Steel, M. J. Keppel, P. Gerbic, & S. Housego (Eds.), Proceedings of the 27th Annual Conference of the Australasian Society for Com- puters in Learning in Tertiary Education (pp. 454–458). Sydney: Sydney University Press. Retrieved from http://ascilite.org.au/conferences/sydney10/procs/Jackson-concise. pdf Lechner, S. K. (2001). Evaluation of teaching and learning strategies. Medical Education Online, 6. 1–5. Retrieved from http://www.med-ed-online.org Li, Q. (2003). Would we teach without technology? A professor’s experience of teaching mathematics education incorporating the internet Educational Research, 45, 61–77. doi: 10.1080/0013188032000086127 Love, K. (2002). Mapping online discussion in senior English. Journal of Adolescent & Adult Literacy, 45. 382–396. Retrieved from http://www.reading.org/general/publications/ journals/jaal.aspx Maor, D., & Volet, S. (2007). Interactivity in professional learning: A review of research based studies. Australasian Journal of Educational Technology, 23. 227–247. Retrieved from http://www.ascilite.org.au/ajet/ajet.html Mazzolini, M., & Maddison, S. (2003). Sage, guide or ghost? The effect of instructor intervention on student participation in online discussion forums Computers & Education, 40, 237–253. doi: 10.1016/S0360-1315(02)00129-X Mazzolini, M., & Maddison, S. (2007). When to jump in: The role of the instructor in online discussion forums. Computers & Education, 49, 193–213. Meyer, K. (2002). Quality in distance education: Focus on online learning. In A. J. Kezar (Ed.), ASHE–ERIC Higher Education Report (pp. I–vii). San Francisco, CA: Jossey-Bass. Moller, L. (1998). Designing communities of learners for asynchronous distance education. Educational Technology Research and Development, 46. 115–122. Retrieved from http:// www.springer.com/education+language/learning+instruction/journal/11423 Moore, M. G. (1989). Three types of interaction. The American Journal of Distance Educa- tion, 3(2). 1–6. Retrieved from http://www.ajde.com/index.htm Nandi, D., Chang, S., & Balbo, S. (2009). A conceptual framework for assessing interaction quality in online discussion forum. In R. J. Atkinson & C. McBeath (Eds. ), Proceedings Ascilite Auckland 2009 (pp. 665–673). Retrieved from http://www.ascilite.org.au/confer- ences/auckland09/procs/Nandi.pdf Nandi, D., Hamilton, M., Harland, J., & Warburton, G. (2011). How active are students in online discussion forums? In J. Hamer & M. de Raadt (Eds.), Proceedings of the Aus- tralasian Computing Education Conference (ACE 2011) (pp. 125–134). Sydney: Austra- lian Computer Society. Retrieved from http://crpit.com/confpapers/CRPITV114Nandi.pdf Neuman, W. L. (2006). Social research methods: Qualitative and quantitative approaches (6th ed.). Boston, MA: Pearson. Ng, K. C., & Murphy, D. (2005). Evaluating interactivity and learning in computer confer- encing using content analysis techniques. Distance Education, 26, 89–109. doi: 10.1080/ 01587910500081327 Persico, D., Pozzi, F., & Sarti, L. (2010). Monitoring collaborative activities in computer supported collaborative learning. Distance Education, 31, 5–22. doi: 10.1080/ 01587911003724603 Piccoli, G., Ahmad, R., & Ives, B. (2001). Web-based virtual learning environments: A research framework and a preliminary assessment of effectiveness in basic IT skills train- ing (1). MIS Quarterly, 25. 401–426. Retrieved from http://www.jstor.org/pss/3250989 Poole, D. M. (2000). Student participation in a discussion-oriented online course: A case study. Journal of Research on Computing in Education, 33, 162–177. Retrieved January 26, 2012, from the EBSCO database. Ramsden, P. (2003). Learning to teach in higher education (2nd ed.). London: Kogan Page. Roblyer, M. D., & Wiencke, W. R. (2003). Design and use of a rubric to assess and encourage interactive qualities in distance courses. The American Journal of Distance Education, 17(2), 77–98. doi: 10.1207/S15389286AJDE1702_2 Distance Education 29 http://www.iste.org/learn/publications/journals/jrte-old.aspx http://ascilite.org.au/conferences/sydney10/procs/Jackson-concise.pdf http://ascilite.org.au/conferences/sydney10/procs/Jackson-concise.pdf http://www.med-ed-online.org http://www.reading.org/general/publications/journals/jaal.aspx http://www.reading.org/general/publications/journals/jaal.aspx http://www.ascilite.org.au/ajet/ajet.html http://www.springer.com/education+language/learning+instruction/journal/11423 http://www.springer.com/education+language/learning+instruction/journal/11423 http://www.ajde.com/index.htm http://www.ascilite.org.au/conferences/auckland09/procs/Nandi.pdf http://www.ascilite.org.au/conferences/auckland09/procs/Nandi.pdf http://crpit.com/confpapers/CRPITV114Nandi.pdf http://www.jstor.org/pss/3250989 Rourke, L., & Anderson, T. (2002). Using peer teams to lead online discussions. Journal of Inter- active Media in Education, 1. 1–21. Retrieved from http://jime.open.ac.uk/jime/index Salmon, G. (2000). E-moderating: The key to teaching and learning online. London: Kogan Page. Salmon, G. (2003). E-tivities: The key to active online learning. London: Kogan Page. Sher, A. (2009, Summer). Assessing the relationship of student–instructor and student– student interaction to student learning and satisfaction in web-based online learning envi- ronment. Journal of Interactive Online Learning, 8(2), 102–120. Retrieved from http:// www.ncolr.org/jiol/ Strauss, A. L., & Corbin, J. M. (1998). Basics of qualitative research: Techniques and pro- cedures for developing grounded theory (2nd ed.). Thousand Oaks, CA: Sage. Thurmond, V., & Wambach, K. (2004, January). Understanding interactions in distance education: A review of literature. International Journal of Instructional Technology & Distance Learning. Retrieved from http://itdl.org/index.htm Vlachopoulos, P., & Cowan, J. (2010). Reconceptualising moderation in asynchronous online discussions using grounded theory. Distance Education, 31, 23–36. doi: 10.1080/ 01587911003724611 Volery, T. (2001). Online education: An exploratory study into success factors. Journal of Educational Computing Research, 24(1). 77–92. Retrieved from http://jrnledcompre- search.com/index.php/jecr/index Wagner, J. G. (2001). Assessing online learning. Reston, VA: National Business Education Association. Retrieved January 26, 2012, from ERIC Document Reproduction Service No. ED449386. Yin, R. K. (1994). Case study research: Design and methods. Thousand Oaks, CA: Sage. Appendix. Example of the coding procedure Initial codes after open coding phase Administrative (+ for assignments) questions, Questions drawn from own experience and real-world situation, Straight and in detail questions, Administrative guidelines or technical assistance, Declaration of expectation, Periodic intervention to direct and extend discussion, Questions with lines of code, Clarification of questions, Declaration of expectation, Straight and in detail answers, Answers with real-world or own coding example, Justification. Grouped themes after the axial coding phase Student Participation → Straight and in detail answers, Answers with real-world or own coding example, Justification, Questions with lines of code, Administrative (+ for assign- ments) questions, Questions drawn from own experience and real-world situation, Straight and in detail questions. Instructor Contribution → Administrative guideline or technical assistance, Declaration of expectation, Periodic intervention to direct and extend discussion, Clarification of questions, Declaration of expectation. Grouped themes after the selective coding phase Student Participation → Asking Questions → Questions with lines of code, Administrative (+ for assignments) questions, Questions drawn from own experience and real-world situa- tion, Straight and in detail questions. Student Participation → Answering Questions → Straight and in detail answers, Answers with real-world or own coding example, Justification. Instructor Contribution → Administrative guideline or technical assistance, Declaration of expectation, Periodic intervention to direct and extend discussion, Clarification of questions, Declaration of expectation. 30 D. Nandi et al. http://jime.open.ac.uk/jime/index http://www.ncolr.org/jiol/ http://www.ncolr.org/jiol/ http://itdl.org/index.htm http://jrnledcompresearch.com/index.php/jecr/index http://jrnledcompresearch.com/index.php/jecr/index Copyright of Distance Education is the property of Routledge and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.