HAJD22(03).book(HAJD_A_322612.fm) The Amer. Jrnl. of Distance Education, 22: 159–176, 2008 Copyright © Taylor & Francis Group, LLC ISSN 0892-3647 print / 1538-9286 online DOI: 10.1080/08923640802224451 HAJD0892-36471538-9286The Amer. Jrnl. of Distance Education, Vol. 22, No. 3, June 2008: pp. 1–31The Amer. Jrnl. of Distance Education Improving the Motivation and Retention of Online Students Through the Use of ARCS-Based E-Mails IMPROVING MOTIVATION AND RETENTIONHUETT ET AL. Jason Bond Huett University of West Georgia Kevin E. Kalinowski University of North Texas Leslie Moller University School of Education Kimberly Cleaves Huett University of West Georgia Abstract: This study examined the use of ARCS-based, motivational mass e-mail messages designed to improve the motivation and retention of students enrolled in an online, entry-level, undergraduate computer applications course. Data from the Course Interest Survey, based on Keller’s ARCS model, were gathered and analyzed for two online groups (one treatment and one control) and compared with a face-to-face class for differences in attention, relevance, confidence, satisfaction, and overall motivation. Based on the analysis, this article argues that simple, cost-effective, and easy-to-design mass e-mail messages show potential for addressing some of the motivational needs and retention concerns of online students. Numerous scholars highlight the general lack of research concerning the moti- vational needs of online learners (Astleitner and Keller 1995; Gabrielle 2003; Huett et al. 2007; Means, Jonassen, and Dwyer 1997; Shellnut, Knowlton, and Savage 1999; J. Visser and Keller 1990). According to Song and Keller (2001), instructional designers often ignore motivational design concerns in Web or site-based, computer-assisted instruction (CAI) and incorrectly assume that the novelty effect of the technology is enough to stimulate learner motivation. Correspondence should be sent to Jason Bond Huett, University of West Georgia, 1601 Maple Street, 152 Ed. Annex, MIT Department, Carrollton, GA 30118. E-mail: jhuett@gmail.com 160 HUETT ET AL. Distance education environments clearly allow for new and unique chal- lenges for instructors and designers who wish to motivate online students and, as L. Visser, et al. (2002) noted, there is a disturbing trend in the publication of specialized distance education handbooks to pay little, if any, attention to the motivational concerns of learners. They examined eight years of the pro- ceedings of the World Conferences of the International Council for Distance Education and found that only 6 of 801 studies addressed motivational con- cerns of online learners. However, one could argue that findings such as these are due to the inherent difficulty of conducting research on the diverse and constantly changing nature of learner motivation and not a lack of interest by the research community. Regardless, the literature establishes the need for studies, such as this one exploring Keller’s ARCS model, that specifically address how to motivate this rapidly growing population of online learners. To stimulate and manage student motivation to learn, Keller (1987a, 1987b, 1987c) created the ARCS model of motivation. ARCS is short for (A)ttention, (R)elevance, (C)onfidence, and (S)atisfaction and serves as the overall framework for the motivational mass e-mail messages used in this study. The ARCS model is an attempt to synthesize behavioral, cognitive, and affective learning theories and demonstrates that learner motivation can be influenced through external conditions. The purpose of the ARCS model is to employ strategies that are used to improve the motivational appeal of instruction. This, in turn, should translate into improvement in learner moti- vation. Given the relationship between motivation and persistence, it was hypothesized that using ARCS strategies to improve the motivational appeal of instruction would also lead to increased learner persistence or retention. There are significant challenges when it comes to the retention of online learners. Although students succeed or fail for a variety of reasons, anecdotal evidence supports the contention that distance education initia- tives often have higher noncompletion rates than face-to-face courses (Dupin-Bryant 2004). The main intent of the ARCS-based messages used in this study was to improve learner motivation. However, although it might be impossible to account for all variables related to retention of online learners, a review of the literature led the researchers to theorize that the messages might also have an effect on the completion rates of students in the treat- ment group. A need exists for simpler approaches to motivating and retaining online learners that are appropriate for the audience, the delivery system, and the course. Such methods should be cost-effective, fit within the time restraints of the class, and be easily integrated into the instruction. One such approach that shows promise is the creation of simple, systematically designed mass e-mail messages based on established ARCS model principles (Gabrielle 2003; Huett et al. 2008; Keller and Suzuki 2004; L. Visser et al. 2002; L. Visser, Plomp, and Kuiper 1999). IMPROVING MOTIVATION AND RETENTION 161 LITERATURE REVIEW The ARCS Model Building on the earlier work of Tolman’s (1932) and Lewin’s (1938) expect- ancy value theory, the ARCS model, with its four categories and twelve subcategories (see Table 1), attempts to synthesize behavioral, cognitive, and affective learning theories and demonstrate that learner motivation can be influenced through external conditions (Moller 1993). The attention, rele- vance, confidence, and satisfaction categories serve as a framework for devel- oping instructional strategies for capturing and maintaining learner attention, establishing relevance of the material being taught, improving and sustaining learner confidence, and providing a sense of learner satisfaction through intrinsic and extrinsic rewards. According to Keller (1987a), each of the categories can be applied to a variety of instructional contexts, and “motivational interventions can be focused within a general category, or specific subcategory of the model” (6). However, there is some debate regarding claims that learner motivation can be isolated or compartmentalized into separate categories. Studies of ARCS- enhanced instructional materials have returned inconsistent results on the indi- vidual subsections as well as on the overall measure of learner motivation (Huett et al. 2008). Debates about the independent nature of the categories aside, among the limited research available, Keller’s ARCS model enjoys support in the litera- ture and researchers have applied its guidelines to different learning and design environments. For example, ARCS research can be found concerning the traditional classroom (Bickford 1989; Klein and Freitag 1992; Means, Jonassen, and Dwyer 1997; Moller 1993; Naime-Diefenbach 1991; Small and Gluck 1994; J. Visser and Keller 1990); computer-assisted instruction (Astleitner and Keller 1995; Bohlin and Milheim 1994; ChanLin 1994; Lee and Boling 1996; Shellnut, Knowlton, and Savage 1999; Song 1998; Song and Keller 1999; Suzuki and Keller 1996); blended learning environments (Gabrielle 2003); and online, distant, and Web-based classrooms (Chyung 2001; Huett 2006; Song 2000; L. Visser 1998). Table 1. Keller’s ARCS Model Summary (Keller 1987a, 1987b) Attention Relevance Confidence Satisfaction A1 Perceptual Arousal R1 Goal Orientation C1 Learning Requirements S1 Natural Consequences A2 Inquiry Arousal R2 Motive Matching C2 Success Opportunities S2 Positive Consequences A3 Variability R3 Familiarity C3 Personal Control S3 Equity 162 HUETT ET AL. Motivation in CAI and Distance Education Motivation has been defined on one extreme as the product of environmental conditioning—of deprivation and reinforcement schedules (Skinner 1953). On the other extreme, individuals like Carl Rogers (1951) and Abraham Maslow (1954) adopt a humanistic approach where motivation is almost entirely a byproduct of free will and an internal drive for self-actualization. Bandura’s (1969) social learning theory argued that learning and motiva- tion were a result not just of the environment or the individual’s free will but a combination of the two. Gagné (1985) advised that both internal and external conditions existed that influenced learning and motivation. Means, Jonassen, and Dwyer (1997) cite studies that show that motivation accounts for 16% to 38% of the variations in overall student achievement. Keller and Burkman (1993) acknowledge that motivation is often thought of as solely a product of learner personality and perceptions—much of which is assumed beyond the control of the instructional designer. However, they believe that providing for motivation is largely the responsibility of the designer. Additionally, they feel that motivation is a systematic process that must be considered during all stages of design. The concept of designing appealing instruction that allows for manipulations of learner motivation is at the heart of distance learning, and Keller and others have recently begun examining how the ARCS model can be applied to computer-based instruction and distance education (Keller 1999). Even today, many learners initially find Web-based environments like distance education and other CAI programs novel or fun. This often translates into a temporary increase in learner motivation. Unfortunately, if the CAI is poorly designed or lacks continuing motivational appeal beyond that of an ini- tial novelty level, learners will eventually lose interest, and motivation and confidence will wane (Keller and Suzuki 1988). It becomes the responsibility of the instructional designer to incorporate strategies that maintain this initial level of motivation through effectively designed distance courses. ARCS-Based Motivational Messages There is little research regarding the ARCS model and motivational messages in distance education as well as an absence of empirical research regarding motivation in technology-based learning environments generally (Gabrielle 2003; Huett 2006). Of the research available, J. Visser (1990) showed an increase in learner motivation through the use of motivational messages in a conventional classroom. In a study of adult students in Mozambique, J. Visser and Keller (1990) delivered ARCS-based motivational messages to students and showed improvement in motivation. They stressed the potential of motivational messages in distance education. IMPROVING MOTIVATION AND RETENTION 163 L. Visser, Plomp, and Kuiper (1999) and L. Visser et al. (2002) used the ARCS model as a guide for developing motivational communications with international distance education students. They found positive outcomes for learner motivation. They also found no statistically significant difference between the use of mass messages versus personalized messages and recom- mended using mass messages to increase “the chance of successful implemen- tation” (1999, 410). Additionally, L. Visser (1998) conducted a pilot study and a main study using the Motivational Messages Support System (MMSS), upon which the e-mails in this study were partly based. In both studies, she found that the messages increased the confidence levels of students. Also, she found no statistically significant difference in the use of mass messages versus personalized messages in terms of effectiveness. She recommends using mass messages to avoid problems such as the increased time associated with designing, developing, and preparing individualized messages. Gabrielle (2003) used Keller’s ARCS model and L. Visser’s MMSS as the basis for interventions and mass messages designed to improve learner motivation and performance in a study of undergraduate students in a public military school. She found statistically significant differences between the groups regarding motivation, academic performance, and self-directed learning and concluded that strategies based on Keller’s ARCS model were worthy instructional design considerations. ARCS and Retention There is no lack of research concerning student retention. In fact, it is arguably one of the most studied areas in education. Decades of research have tied fluc- tuations in face-to-face and online student retention to internal factors such as locus of control, motivation, confidence, and all manner of psychological traits. Other issues influencing student retention have been attributed to personal attributes like age and gender as well as other external and environ- mental concerns such as socioeconomic status, relationship with the educa- tional institution, educational background, familial commitments, grade point average, and any number of societal or cultural factors (Andreu 2002; Morris, Wu, and Finnegan 2005). Some researchers such as Spady (1970) and Tinto (1975) have developed conceptual or predictive models of retention in higher education with Tinto’s model being, by far, the most widely cited and studied (Dupin-Bryant 2004). Kember (1995) built on these earlier models to design the first retention model specifically targeted toward adults studying at a dis- tance. However, unlike the ARCS model, Kember’s model focuses less on psychological factors and more on certain personal aspects and social or demographic characteristics of students such as employment, family life, gender, and educational background (Woodley, de Lange, and Tanewski 2001; Yorke 2004). 164 HUETT ET AL. Building on the work of Rotter (locus of control), Bandura (self-efficacy), and others, Keller’s ARCS model more neatly parallels Bean and Eaton’s (2000) retention model, which emphasizes the key concepts of locus of con- trol, self-efficacy, and approach-avoidance theory as integral to the student’s commitment to persist. Additionally, established research into general reten- tion theory often makes a connection between learner persistence and core ARCS principles such as motivation, relevance, confidence, and satisfaction (Andreu 2002; Dille and Mezack 1991; Tinto 1975, 1993, 2007). Although the ARCS model was not specifically designed to address retention, Song (2000) insists three types of motivational categories should be taken into consideration when designing ARCS-related distant or Web-based instruction: motivation to initiate, motivation to persist, and motivation to con- tinue. Each of Song’s three categories is related to the concept of engagement— a core principle in retention theory: “Throughout these changes [in student retention research, theory, and practice] and the putting forth of alternative models, one fact has remained clear. Involvement, or what is increasingly being referred to as engagement, matters . . .” (Tinto 2007, 4). It seems that motivational theory informs retention theory and vice versa, with both acknowledging that there is a connection between persistence, motivation, and learner engagement. Specifically, the ARCS model allows for external strate- gies that lead to a perceived increase in the motivational appeal of instruction. This, in turn, helps create conditions that support learner engagement and persistence. Although exploring the complexity of student retention is beyond the scope of this study, there is some limited research directly applicable to ARCS-based motivational messages. For instance, Keller and Suzuki (2004) cited a 1998 report by L. Visser outlining a 70–80% improvement in retention rates of distance learners when motivational messages based on the ARCS model were used. Additionally, with their ARCS-based motivational commu- nications, L. Visser, Plomp, and Kuiper (1999) and L. Visser et al. (2002) found that motivational messages “considerably increased the completion rates of students” (2002, 410). Given the available literature, it seemed feasi- ble that well-designed motivational mass e-mail messages could impact the retention rates of the treatment group. RESEARCH QUESTIONS The purpose of the current study was to determine what effect motivational e-mails had on student motivation and retention in distance education. To this end, two specific research questions were proposed: 1. Will the motivational mass e-mail messages used in this study produce statistically significant differences (a) between the control group and the IMPROVING MOTIVATION AND RETENTION 165 treatment group and (b) between the treatment group and a face-to-face classroom, both in terms of learner motivation as measured by the overall summative score of the Course Interest Survey (CIS)? 2. Will the motivational mass e-mail messages used in this study produce statistically significant differences between the control group and the treat- ment group in terms of learner retention based on completion rates of stu- dents for the course? METHOD Participants In order to answer the research questions, we examined 153 students in three sections of a required entry-level computer applications course at a Texas university rated Carnegie Doctoral/Research Universities—Extensive. An online section was used as a control, and another online section was treated with several simple, mass-mailed motivational e-mails throughout the semes- ter. For comparison purposes, a face-to-face classroom section of the same course was included in the study. See Table 2 for the composition of study participants. Of the 153 original participants, 124 students successfully com- pleted the course. Of these students, 119 (52 males and 67 females) completed the CIS (see Table 3). In addition to gender, other demographic information such as age, race, and enrollment status was consistent with university-reported demographics concerning the campus undergraduate population as a whole. Table 2. Composition of Study Participants Control Treatment Face-to-Face Total Passed course Completed CIS 37 54 28 119 Did not complete CIS 1 2 2 5 Failed course 11 4 2 17 Withdrew from course 9 3 0 12 Total 58 63 32 153 Table 3. Gender Composition of Participants Completing CIS Control Treatment Face-to-Face Total Male 16 25 11 52 Female 21 29 17 67 Total 37 54 28 119 166 HUETT ET AL. The students preferring to enroll in the online version of the course natu- rally formed two groups depending on the section in which they were enrolled. Students were automatically assigned to the online sections by the university registrar. Those students preferring the face-to-face version enrolled themselves in that section of the course. The face-to-face section covered the same material as the online sections except the face-to-face students completed the work in the campus lab at a specific time, whereas the online students completed the work on their own machines and according to their own schedules. Due dates for the completed work were the same regardless of the delivery format. There was no required contact between the online students during the semester, so for this study it will be presumed that participants were independent and randomly assigned between the two online groups. To minimize the possible impact of instructional differences, the same doctoral candidate taught all three sections involved in this study. Other than the motivational e-mails given to the treatment group, there were no other per- ceivable differences between the online groups. The only difference between the online section and the face-to-face section was direct face-to-face contact with the instructor. Procedures Keller (1999) noted that an excessive number of motivational tactics in computer-based instruction might prove distracting to already motivated stu- dents. Keller (1987c) also recommended that motivational strategies not be overly time-consuming, work within the delivery system, and be presented in an acceptable way to participants. With this in mind, the motivational mass e-mail messages in this study were delivered at roughly two-week intervals during the class to the students in the treatment group. The two-week interval was a discretionary decision based on Keller’s recommendations, past research, the structure of the class assignments, and the desire to show a con- cerned motivational presence—not an overpowering one. At the end of the semester, 119 students passing the course took the CIS through a Web-based survey tool to measure the four factors (attention, rele- vance, confidence, and satisfaction) of the ARCS model as well as an overall motivation score. The survey was voluntary and had no impact on course grades. Measures were taken to ensure only students enrolled in the three sec- tions took the survey, and no student took the survey more than once. Instruments For each student, the four latent factors of the ARCS model were computed from the thirty-four CIS questions as instructed by Keller and Subhiyah IMPROVING MOTIVATION AND RETENTION 167 (1993). This survey uses a Likert-type scale with five possible choices: (1) Not true, (2) Slightly true, (3) Moderately true, (4) Mostly true, and (5) Very true. Nine questions were reverse-worded by design and had to be recoded prior to the construction of the factors. The four factor scores were averaged to construct an overall motivation score. Cronbach’s coefficient alpha was computed for each of the four factors and the overall score. To answer the two parts of the first research question, two orthogonal contrasts were established a priori. Confidence intervals and effect sizes were computed to help assess the practical significance of the findings. For the second research question, failure and withdrawal rates were com- puted for the control and treatment online groups and compared with the face- to-face group. Because these measures had sample frequencies too small to test for statistical significance, they were combined under the broader classifi- cation of noncompleter and analyzed. Materials Design of the Messages for this Study. The messages were designed using the researchers’ interpretation of the literature surrounding the ARCS model (see Table 1). The mass e-mails used in this study were also based on the researchers’ understanding of what constituted motivational enhancements as well as adapta- tions of earlier works by Moller (1993), Moller and Russell (1994), L. Visser (1998), L. Visser et al. (2002), and Gabrielle (2003). Furthermore, it was the researchers’ desire to create cost-effective, simple, effectual e-mail communications that could be easily adapted to a variety of distance education settings or subjects. In general, the motivational messages were designed using the following template. It should be noted that this template is situational and can (and should) be adapted to fit the given learning environment. Expert consultation from a panel of three professors with extensive (minimum five years) online teaching experience was sought in the development of the template and motivational messages used the study. Suggestions were offered by the experts, and many of their ideas and proposals were incorporated into the final e-mail messages. Introduction. The beginning of the message contained a brief paragraph with an enthusiastic tone of introduction. For instance, “I hope you are doing great! I sent a letter out last week introducing myself and reminding you of what is expected in this class. If you are anything like me, you might have a tendency to procrastinate or have a tough time getting started. Don’t worry—there is still time.” The intent of the opening paragraph was to get the learners’ atten- tion and to convey assurances of personal interest in learner success. Goal Reminders. The intent of the next few paragraphs was to offer goal reminders for the class. For instance, “Ideally, by the end of these first few 168 HUETT ET AL. weeks, you will have completed at least three or four of the SAM Office 2003 assignments” and “Don’t forget the deadline for the Access pathways is April 3rd.” Again, this served to express a personal interest in learner success, offered reminders of personal control, and served to reinforce confidence and satisfaction by giving the students knowledge of what was expected of them. Words of Encouragement. The next paragraph was devoted to general words of encouragement. For instance, “As you complete the assignments this week, I would like to extend hearty congratulations to all of you for your hard work” and “You are almost done with this section of the class, so run for the finish. I have great faith in your continued success. You can do it!” Such sentiments were designed to decrease learner anxiety. Multiple Points of Contact. The final paragraph of the messages served to assure learners and offer multiple contact points for feedback opportunities. For instance, “I am very sure you will be successful. If you ever need my help or have any questions or concerns, please do not hesitate to contact me via e-mail atjXXXXX@xyz.edu. If it is an emergency, I can be reached on my cell at 555–1234.” It was assumed that learner anxiety would decrease if the learners were reminded of the instructor’s approachability and desire to help. RESULTS After computing the four measures of motivation and the overall score for each student, we computed Cronbach’s (1951) coefficient alpha for each fac- tor as an estimate of reliability. The results, as well as a comparison to Keller and Subhiyah (1993) initial reliability results, are found in Table 4. Research Question One The online treatment group had motivational scores on par with the face-to-face classroom group, and both groups seem to show marked differences from the Table 4. Reliability Estimates for Motivational Scales Using Cronbach’s Coefficient Alpha Current Study Keller and Subhiyah (1993) Attention (A) .753 .84 Relevance (R) .804 .84 Confidence (C) .758 .81 Satisfaction (S) .851 .88 Total scale (ARCS) .932 .95 IMPROVING MOTIVATION AND RETENTION 169 online control group not receiving the motivational e-mails. Per the research design, planned orthogonal contrasts were established to determine (a) if moti- vation, as measured by Keller’s ARCS model via the CIS, is statistically dif- ferent between students given motivational e-mails and those who did not and (b) if there is a statistically significant difference in motivation between the online treatment group and the face-to-face classroom. Prior to the analyses, descriptive statistics were run on the four subscales of CIS and the composite scale. Measures of skewness and kurtosis were within the range –1 to +1. Therefore, the assumption that the population from which the samples were selected is normally distributed was tenable (see Table 5). Furthermore, Levene’s Tests for Equality of Variances were computed for the factors with no statistically significant differences found. Therefore, homogeneity of variance was assumed. The results of the planned orthogonal contrasts are found in Table 6. Finally, two measures of practical significance were determined. First, 95% simultaneous confidence intervals for the two contrasts were calculated based on a formula provided by Maxwell and Delaney (2004). Then, effect size estimates for statistically significant results were computed using Cohen’s (1988) d index. The results are also found in Table 6. Research Question Two The control group had a 15.52% withdrawal rate and an 18.97% failure rate compared with a 4.76% withdrawal rate and a 6.35% failure rate for the treat- ment group. The traditional classroom had no withdrawals and a 6.25% failure rate, which was almost identical to that of the treatment. Failure was defined as a student who finished the course but did not earn the minimum number of points required for a passing grade. Specifics on when and why students with- drew from the course were unavailable. Alone, each of these measures had sample frequencies too small to test for statistical significance, so the numbers of withdrawals and failures were combined under the broader classification of noncompleter and analyzed. Results show that the control group had a statisti- cally significant noncompleter rate of 34.5% (9 + 11 / 58) compared with Table 5. Descriptive Statistics for the CIS Subscales and Composite Scale for All Three Groups n Minimum Maximum M SD Skewness Kurtosis A 119 1.25 4.75 3.15 0.72 −.266 −.267 R 119 2.50 5.00 4.37 0.55 −.909 .378 C 119 1.78 5.00 4.14 0.62 −.768 .747 S 119 1.11 5.00 3.82 0.78 −.711 .539 ARCS 119 1.75 4.88 3.87 0.59 −.623 .510 170 HUETT ET AL. 11.1% (3 + 4 / 63) for the treatment (χ2(1, N = 121) = 8.22, p = .004). There was no statistically significant difference between the treatment group and the traditional classroom group on this measure (χ2(1, N = 95) = 0.16, p = .689). DISCUSSION It is encouraging to find that our sample produced reliability measures on par with that of Keller and Subhiyah (1993). Although the subject-to-factor ratio was nearly 30:1, it is possible that low sample sizes (n = 119 as opposed to Keller’s n = 200) enhanced the effects of sampling error and produced esti- mates lower than previous studies. The results of the first planned contrast show that there is a statistically significant difference in means between students receiving the treatment and those who did not for attention (p < .001), confidence (p = .001), satisfaction (p < .001), and the overall measure of motivation (p < .001) but not in rele- vance (p = .122). Furthermore, estimates of effect size for attention (d = 0.94), satisfaction (d = 0.94), and the total ARCS scale (d = 0.85) are large, as suggested by Cohen (1988). Using the same criteria, the mean difference for Table 6. Summary of the Tests of Planned Contrasts Value SE t d.f. p 95% Simultaneous Intervals Cohen’s d Attention (A) Contrast Aa 0.645 0.143 4.504 116 < .001 0.32, 0.97 0.94 Contrast Bb 0.215 0.156 1.377 116 .171 −0.14, 0.57 Relevance (R) Contrast A 0.181 0.116 1.556 116 .122 −0.08, 0.44 Contrast B −0.104 0.127 −0.820 116 .414 −0.39, 0.18 Confidence (C) Contrast A 0.452 0.127 3.564 116 .001 0.16, 0.74 0.71 Contrast B 0.174 0.138 1.257 116 .211 −0.14, 0.49 Satisfaction (S) Contrast A 0.704 0.153 4.604 116 < .001 0.36, 1.05 0.94 Contrast B 0.053 0.167 0.320 116 .750 −0.33, 0.43 Total scale (ARCS) Contrast A 0.495 0.118 4.206 116 < .001 0.23, 0.76 0.85 Contrast B 0.085 0.129 0.658 116 .512 −0.21, 0.38 aContrast A: Is motivation statistically different between students given motivational e-mails (treatment) and those who did not (control)? bContrast B: Is there a statistically significant difference in motivation between the online treatment group and the face-to-face classroom? IMPROVING MOTIVATION AND RETENTION 171 confidence (d = 0.71) could be evaluated as medium-large. Therefore, all sta- tistically significant results showed nontrivial differences between the means of the treatment and control groups. There was a statistically significant difference in every measure of motivation except relevance. Given the nature of the treatment used in this study, this result appears to make sense. Relevance generally addresses the connection between the subject matter and the learner’s ability to find that material useful and personally meaningful. The e-mails were not designed to emphasize how the material learned in the course was readily applicable to the students’ personal or professional situations. Therefore, it is not unexpected that measures of relevance would be nonsignificant. Also, given that the course was required within the degree program and that the subject matter consisted of learning Microsoft Office (a highly relevant and ubiquitous office suite), the lack of variance in relevance between groups is not surprising. Finally, there was no difference detected between the online treatment group and the face-to-face classroom on any measure. Although it would be tempt- ing to try to draw other conclusions about why attention, for instance, had lower measured levels than the other factors (R, C, and S) in all three groups, the survey is designed to be a comparison across groups and not across scales. For this specific study, there was greater student retention as well as a lower student failure rate for the treatment group. Although it would be diffi- cult to claim that a simple e-mail treatment was the only possible cause of the difference in the retention rate between the treatment and control groups, one can speculate that the social aspect or sense of community (like that often present in a face-to-face classroom) created through motivational e-mail com- munications may be part of the necessary structure needed to support online learners’ engagement and persistence (Huett et al. 2007). Given these results, it seems reasonable to promote this treatment to all instructors teaching undergraduates enrolled in the online, entry-level com- puter application course at the university used in this study. With further stud- ies, it might be possible to generalize the findings to further populations. LIMITATIONS AND FUTURE RESEARCH Perhaps the most notable limitation of the study for some would be that a series of e-mail messages presents a rudimentary form of motivational enhancement. The authors acknowledge the complexity of human motivation and do not contend that a single intervention (like motivational e-mails) can address the entire spectrum and inner complexity of motivation. Such was not the intended design of this study. However, one of the perceived benefits of the ARCS model is that it allows for a more prescriptive approach to motiva- tion. According to the model, simple, well-designed, targeted treatments may still return meaningful results when used properly with an appropriate 172 HUETT ET AL. audience. In addition, the content of the e-mail messages as well as the fre- quency of their delivery could leave room for different interpretations by varying researchers. However, given the practical concerns for online educa- tion such as cost, time, and ease of integration, the more narrow approach used in this study is not without merit. Regarding the design of the study, using a convenience sample might have led to contextual effects not detected in the analyses. If we had more course sections, we could have performed a multilevel analysis of the data to attempt to model these effects, but, given the very high effect sizes found in this study, it is doubtful the alternate analysis would have produced contra- dicting results. Perhaps a more precise fill-in-blank style motivational message template could be designed and tested to aid the simplification, standardized develop- ment, and research of the motivational e-mails. However, such a template might lend itself to criticism that one was only scratching the surface of moti- vational complexity or that one was ignoring important contextual and situa- tional factors. Also, when designing messages such as the ones used in this study, it seems reasonable to suggest that some messages could produce greater motivational returns than others and that some participants would respond to certain motivational e-mails that others would not. This makes templated design a difficult proposition. In retrospect, it would have also been potentially beneficial to use an instrument to get a baseline measure of learner motivation before applying the treatment. Getting a read on how motivated learners were before beginning instruction might have helped more clearly explain any changes in motivation. Such a measure is recommended for future studies. Though we were able to achieve significant motivation and retention results delivering the messages at roughly two-week intervals, it remains unclear at this point how frequently messages should be delivered to achieve desired results. It is feasible that fewer or greater numbers of messages could significantly impact results. Future researchers may wish to explore different intervals for message delivery. CONCLUSION It appears that simple, cost-effective, and easy-to-design mass e-mail messages show potential for addressing some of the motivational needs and retention concerns of online students. With their simplicity and ease of use, the motivational e-mails in this study represent another tool that distance educators can employ that complements other motivational efforts. Beyond the statistical findings, there are obvious practical (financial) implications for universities with online programs that wish to improve student motivation, retention, and passing rates with a relatively easy-to-implement strategy. Although this initial study should not be generalized beyond undergraduates IMPROVING MOTIVATION AND RETENTION 173 enrolled in the online, entry-level computer application course at the univer- sity used in this study, the apparent positive results should encourage similar studies for other online courses at other institutions. Additional, ongoing research suggests that motivational communications, such as those used in this study, could have a significant impact on learner motivation and retention in distance learning situations at both the graduate and undergraduate levels. A personal note: Jason Huett would like to dedicate this manuscript to the memory of Dr. Mark Henry Mortensen. Mark, you touched more lives than you ever knew. You were a teacher, a mentor, and a friend. I am a better person and professor for having known you. You died too young, and you are missed. REFERENCES Andreu, M. L. 2002. Developing and implementing local-level retention studies: A challenge for community college institutional researchers. Community College Journal of Research and Practice 26 (4): 333–344. Astleitner, H., and J. Keller. 1995. A model for motivationally adaptive computer-assisted instruction. Journal of Research on Computing in Education 27 (3): 270–280. Bandura, A. 1969. Principles of behavior modification. New York: Holt, Rinehart, & Winston. Bean, J. P., and S. B. Eaton. 2000. A psychological model of college student retention. In Reworking the student departure puzzle, ed. J. M. Braxton, 48–61. Nashville, TN: Vanderbilt University Press. Bickford, N. 1989. The systematic application of principles of motivation to the design of printed instructional materials. Ph.D. diss., Florida State University, Tallahassee. Bohlin, R., and W. Milheim. 1994. Applications of an adult motivational instructional design model. Paper presented at the Association for Educa- tional Communications and Technology, February, Nashville, TN. ChanLin, L. 1994. A case for assessing motivation from learning a computer- assisted instruction. ERIC, ED 376803. Chyung, S. Y. 2001. Systematic and systemic approaches to reducing attrition rates in online higher education. The American Journal of Distance Educa- tion 15 (3): 36–49. Cohen, J. 1988. Statistical power analysis for the behavioral sciences, 2nd ed. Hillsdale, NJ: Erlbaum. Cronbach, L. J. 1951. Coefficient alpha and the internal structure of tests. Psychometrik 16:297–334. Dille, B., and M. Mezack. 1991. Identifying predictors of high risk among community college telecourse students. The American Journal of Distance Education 5 (1): 24–35. 174 HUETT ET AL. Dupin-Bryant, P. A. 2004. Pre-entry variables related to retention in online distance education. The American Journal of Distance Education 18 (4): 199–206. Gabrielle, D. 2003. The effects of technology-mediated instructional strate- gies on motivation, performance, and self-directed learning. Ph.D. diss., Florida State University, Tallahassee. Gagné, R. M. 1985. The conditions of learning, 4th ed. New York: Holt, Rinehart, & Winston, Inc. Huett, J. 2006. The effects of ARCS-based confidence strategies on learner confidence and performance in distance education. Ph.D. diss., University of North Texas, Denton. Huett, J., L. Moller, D. Harvey, D. Holder, V. Godshalk, and M. Engstrom. 2007. Examining the use of learning communities to increase motivation. In Online learning communities, ed. R. Luppicini, 189–204. Charlotte, NC: Information Age. Huett, J., L. Moller, J. Young, M. Bray, and K. Huett. 2008. Supporting the distant student: The effect of ARCS-based strategies on confidence and performance. Quarterly Review of Distance Education 9 (2): 113–126. Keller, J. M. 1987a. Strategies for stimulating the motivation to learn. Perfor- mance and Instruction 26 (8): 1–7. ———. 1987b. The systematic process of motivational design. Performance and Instruction 26 (9): 1–8. ———. 1987c. Development and use of the ARCS model of motivational design. Journal of Instructional Development 10 (3): 2–10. ———. 1999. Using the ARCS motivational design process in computer- based instruction and distance education. New Directions for Teaching and Learning 78:39–47. Keller, J. M., and E. Burkman. 1993. Motivation principles. In Instructional message design: Principles from the behavioral and cognitive sciences, 2nd ed., eds. M. Fleming and W. H. Levie, 3–53. Englewood Cliffs, NJ: Educa- tional Technology. Keller, J. M., and R. Subhiyah. 1993. Manual for the course interest survey (CIS). Tallahassee, FL. Keller, J. M., and K. Suzuki. 1988. Use of the ARCS motivation model in courseware design. In Instructional designs for microcomputer courseware, ed. D. H. Jonassen, 401–434. Hillsdale, NJ: Erlbaum. ———. 2004. Learner motivation and e-learning design: A multinationally validated process. Journal of Educational Media 29 (3): 229–239. Kember, D. 1995. Open learning courses for adults: A model of student progress. Englewood Cliffs, NJ: Educational Technology Publications. Klein, J. D., and E. T. Freitag. 1992. Training students to utilize self-motivational strategies. Educational Technology 32 (3): 44–48. Lee, S., and E. Boling. 1996. Motivational screen design guidelines for effec- tive computer-mediated instruction. Paper presented at the Association for IMPROVING MOTIVATION AND RETENTION 175 Educational Communications and Technology, November, Indianapolis, IN. Lewin, K. 1938. The conceptual representation and the measurement of psychological forces. Durham, NC: Duke University Press. Maslow, A. 1954. Motivation and personality. New York: Harper. Maxwell, S. E., and H. D. Delaney. 2004. Designing experiments and analyz- ing data: A model comparison perspective, 2nd ed. Mahwah, NJ: Erlbaum. Means, T., D. Jonassen, and F. Dwyer. 1997. Enhancing relevance: Embedded ARCS strategies vs. purpose. Educational Technology Research and Devel- opment 45:5–17. Moller, L. 1993. The effects of confidence building strategies on learner moti- vation and achievement. Ph.D. diss., Purdue University, West Lafayette, IN. Moller, L., and J. Russell. 1994. An application the ARCS model confidence building strategies. Performance Improvement Quarterly 7 (4): 54–69. Morris, L., S. Wu, and C. Finnegan. 2005. Predicting retention in online general education courses. The American Journal of Distance Education 19 (1): 23–36. Naime-Diefenbach, B. 1991. Validation of attention and confidence as inde- pendent components of the ARCS motivational model. Ph.D. diss., Florida State University, Tallahassee. Rogers, C. 1951. Client-centered therapy; its current practice, implications and theory. Oxford: Houghton Mifflin. Shellnut, B., A. Knowlton, and T. Savage. 1999. Applying the ARCS model to the design and development of computer-based modules for manufacturing engineering courses. Educational Technology, Research and Development 47:100–110. Skinner, B. F. 1953. Science and human behavior. New York: Free Press. Small, R. V., and M. Gluck. 1994. The relationship of motivational conditions to effective instructional attributes: A magnitude scaling approach. Educa- tional Technology 34 (8): 33–40. Song, S. H. 1998. The effects of motivationally adaptive computer-assisted instruction developed through the ARCS model. Ph.D. diss., Florida State University, Tallahassee. ———. 2000. Research issues of motivation in web-based instruction. Quarterly Review of Distance Education 1 (3): 225–229. Song, S. H., and J. M. Keller. 1999. The ARCS model for developing motivationally-adaptive computer-assisted instruction. Paper presented at the Association for Educational Communications and Technology, February, Houston, TX. Song, S. H., and J. M. Keller. 2001. Effectiveness of motivationally adaptive computer-assisted instruction on the dynamic aspects of motivation. Educa- tional Technology, Research and Development 49:5–22. Spady, W. G. 1970. Dropouts from higher education: An interdisciplinary review and synthesis. Interchange 1:64–85. 176 HUETT ET AL. Suzuki, K., and J. M. Keller. 1996. Applications of the ARCS model in computer-based instruction in Japan. Paper presented at the annual meeting of the Japanese Educational Technology Association, August, Kanazawa, Japan. Tinto, V. 1975. Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research 45:89–125. ———. 1993. Leaving college: Rethinking the causes and cures of student attrition, 2nd ed. Chicago: University of Chicago Press. ———. 2007. Research and practice of student retention: What next? Journal of College Student Retention: Research, Theory, and Practice 8 (1): 1–20. Tolman, E. C. 1932. Purposive behaviour in animals and men. New York: Appleton-Century. Visser, J. 1990. Enhancing learner motivation in an instructor-facilitated learning context. Ph.D. diss., Florida State University, Tallahassee. Visser, J., and J. M. Keller. 1990. The clinical use of motivational messages: An inquiry into the validity of the ARCS model of motivational design. Instructional Science 19:467–500. Visser, L. 1998. The development of motivational communication in distance education support. Den Haag, The Netherlands: CIP-gegevens Koninklijke Bibliotheek. Visser, L., T. Plomp, R. Arimault, and W. Kuiper. 2002. Motivating students at a distance: The case of an international audience. Educational Technology Research and Development 50 (2): 94–110. Visser, L., T. Plomp, and W. Kuiper. 1999. Development research applied to improve motivation in distance education. Paper presented at the Association for Educational Communications and Technology, February, Houston, TX. Woodley, A., P. de Lange, and G. Tanewski. 2001. Student progress in distance education: Kember’s model re-visited. Open Learning 16 (2): 113–131. Yorke, M. 2004. Retention, persistence and success in on-campus higher edu- cation, and their enhancement in open and distance learning. Open Learning 19 (1): 19–32.