Enhancing Quality in Online Learning: Scaffolding Planning and Design Through Proactive Evaluation Distance Education, Vol. 23, No. 2, 2002 Enhancing Quality in Online Learning: Scaffolding Planning and Design Through Proactive Evaluation Rod Sims Learning Services, Deakin University, Australia Graeme Dobbs & Tim Hand Resource Design & Development Services, OTEN DET, Australia ABSTRACT A major issue confronting educators is the extent to which they wish to conform to so-called paradigm shifts in teaching and learning. In the contemporary world of tertiary education these shifts embrace both pedagogy (from instructivist to constructivist) and technology (from classroom to online). As teachers and learners are faced with the potential of these new learning environments, the extent to which learning outcomes are achieved remains a high priority and subject to a wide range of evaluation strategies. Conventionally, evaluation has been positioned at the end of the instructional development cycle, to assess � rst whether or not the creative effort achieved the original product goals and second whether or not the desired learning outcomes were realized. In the context of online teaching and learning environments, however, the level of understanding teachers, learners and developers have of the medium can impact the ultimate effectiveness of the product. This paper articulates an additional dimension to post-development evaluation processes in proposing proactive evaluation, a framework that identi� es critical online learning factors and in� uences that will better inform the planning, design and development of learning resources. This notion of proactive evaluation advocates resource development being undertaken where all planning activities are assessed against the evaluation criteria that would normally be applied during formative assessment. By performing these evaluation checks proactively, online learning resources will, in principle, work � rst time as all relevant factors and issues will have been considered and resolved. More importantly, for those participants who are new to online environments, proactive evaluation will perform a scaffolding and professional development role by enhancing online teaching or learning competencies. Introduction While many practitioners are familiar with the issues and processes associated with the production of online teaching and learning environments, institutions are frequently demanding their im- plementation without necessarily having staff competent in all aspects of online pedagogy. We also still have much to learn about online learning environments, as research studies have demonstrated them to have both positive and negative impacts in terms of effectiveness and achievement of outcomes (Franklin, Peat, Lewis & Sims, 2001). It is therefore critical that ISSN 0158-791 9 print; 1475-0198 online/02/020135-1 4 Ó 2002 Open and Distance Learning Association of Australia, Inc. DOI: 10.1080/0158791 0220000 09169 136 R. Sims et al. online development projects implement quality control processes to ensure that learners are able to participate in effective collaborative discourse. These quality control measures result from adherence to proven materials or instructional development processes, accreditation of technical standards and the evaluation and assessment of the learning environment (see Gore, Bond & Steven, 2000). Evaluation strategies within the educational domain, particularly for distance and online education, are well established and it is important to contextualize their role to accurately position the proactive evaluation framework. In education, the most common forms of quality assessment through evaluation are the formative and summative techniques, which are typically conducted towards the end of the development and delivery process, respectively (Morrison, Ross & Kemp, 2001; Smith & Ragan, 1999). However, other forms of evaluation also contribute to the overall quality process. Eastmond (1994) differentiates the evaluation of programs (for example the success of a pre-service teacher program), projects (assessing the success of activities) and materials (the merit or worth of content items). Using a different schema, Simonson, Smaldino, Abright and Zvacek (2000) refer to � ve components of the evaluation process—accountability (Did the project planners achieve their goals?), effectiveness (Were outcomes achieved?), impact (Did the program make a difference?), organizational context (How did the organization affect the project?) and unanticipated consequences (What happened that was not expected?). Both of these approaches extend the work of Woodley and Kirkwood (1986) who identi� ed evaluation elements relating to project activities, ef� ciency, learning outcomes, programs and policy. While these variations on evaluation tend to conform to the process where data is collected post facto to provide evaluation outcomes, other models include evaluation in the planning and design stage. For example, Albrecht and Bardsley (1994, p. 82) discuss this phase as focusing on “outcomes including student learning, revenue streams, and impact on educational re- sources … the evaluation should drive the planning group back through the process to achieve those outcomes most valued.” Evaluation therefore can take on various forms and be activated during different stages of the overall development process. Where, then, does Proactive Evaluation � t? Owen and Rogers (1999) use the concept of proactive evaluation to refer to activities such as needs assessment, research reviews and identi� cation of exemplary practice, which appear analogous to the instructional problems phase of the instructional design process (Morrison et al., 2001). Within this paper, however, proactive evaluation (which was coined independently from that of Owen & Rogers, 1999) describes a different aspect of the instructional development process. Eastmond (1994, p. 101) cites Scriven (1980) as differenti- ating evaluation as follows: “when the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.” Continuing with this analogy, proactive evaluation is about participants in the design process developing a shared understanding of the ingredients and the recipes to ensure that the “soup” is appropriate for online consumption. Proactive Evaluation Therefore, as an extension to established forms of evaluation, and to provide speci� c support (scaffolding) for the planning and creation of online materials, this paper argues for proactive evaluation to be integrated into this phase of the production process. Through the proactive Enhancing Quality in Online Learning 137 evaluation framework, the production team will focus on the criteria by which the environ- ments and resources might normally be evaluated, thereby ensuring that all factors associated with a successful evaluation are addressed during the planning phase. This process will also pre-empt the possibility of critical, negative formative and summative evaluation. Using proactive evaluation, therefore, environments created by inexperienced people will be more likely to achieve their educational and learning goals as this online pedagogy is integral to all facets of the design, development and implementation process as recommended by Sims (1997). Additionally, the underlying principles assume that the environment for which the develop- ment is being undertaken is not a traditional instructional setting, but an online collaborative place in which the familiar classroom relationships between teacher, learner and content are extended to enable learner-centred activities and participatory partnerships. By supporting proactive evaluation, planning and design activities will not only ensure that all aspects of creating online learning activities and environments are addressed, but that subsequent formative and summative evaluation will be more directed and meaningful. Proactive evalu- ation also focuses the decision-making process on the complex interaction between disci- plinary content, learning outcomes and online, computer-based learning environments. Better understanding and addressing these relationships will consistently produce more effective teaching and learning resources, and the following classi� cation of factors and associated in� uences describes the framework for proactive evaluation in detail. Strategic Intent One of the � rst questions we always ask our clients is “Why are you attempting to place these resources or activities into an online context?”. If they are unable to provide an explicit answer, then we argue that the strategic intent or rationale for the product has not been de� ned adequately. Without a clear understanding of the purpose of the product and the stakeholders who have an investment and/or interest in its outcomes, the chances of success are reduced. Within the tertiary education environment speci� cally, these stakeholders include the adminis- tration, the faculty, the development group and most importantly the students. Without full commitment to the concept from all stakeholders, even with the best intentions, effectiveness in terms of learning outcomes being achieved may not be realized. The critical issue therefore is the extent to which the online component(s) being considered will add value to the teaching and learning process. For example, if an institution decides to “go online” without allocating suf� cient funds to recreate materials so that they will be consistent with learner–computer communication, then the decision could prove extremely costly with few, if any, positive educational outcomes. Indeed, anecdotal feedback is suggest- ing a rebellion against online materials; for example, where they are perceived by faculties as an economic solution to government cutbacks and workload increases. In other cases, where “online” has been introduced to reduce class contact time, student anger has resulted with resentment for being short changed, not to mention issues of quality. Once the strategic intent has been de� ned, members of the design and development team will have a foundation on which to base the various elements of an online learning environment, the � rst of which is content. 138 R. Sims et al. TABLE 1(a). Options affecting online content Static , . Dynamic Predetermined Teacher Learner Captured dialogue and presented contributed contributed (interactivity) Constructed Content de� ned Content de� ned Content de� ned Through Content de� ned and prescribed by and prescribed, and prescribed, collaborative through research the teacher, and but additions or but learner endeavours, by participants does not change modi� cations additions and content material and subsequent during the made by teacher contributions is added to the interpretation and delivery cycle if and when enhance the overall resource construction required resource base base for the program Content Many online projects have focused on the conversion of existing paper-based resources into their digital equivalent, with a proliferation of unit outlines and study guides in either HTML or PDF formats being made available for student access. We take the position that this is not online learning, and if portrayed as such is a misrepresentation of the capabilities and bene� ts of the technology. While such materials may be accessible through the Internet and therefore “online,” until there is an appropriate mix of motivation and engagement it does not equate to “online learning” (Malone & Lepper, 1987; Sims, 2000). We believe that online learning must therefore be conceptualized as an environment that integrates collaboration, communication and engaging content with speci� c group and independent learning activities and tasks. If content materials and learning activities are to be placed online, then signi� cant levels of thought must be placed on the very nature of the medium and the underlying implications for teaching and learning. As represented by the content options identi� ed in Table 1(a), the online environment caters for a range of content formats: from predetermined static elements to the dynamic state where content is sourced, repurposed, constructed and enabled by and for all participants in the learning process. Between these two extremes are the more common exemplars for delivery of course content in terms of resource material being contributed to by both the teacher and the learner. For the teacher new to online learning, maximum exploitation of the online environment means having to reassess the overall approach to the content, how it should be presented or accessed and the relationship between teacher and learner in that process. In addition, the options for unit or course content should be considered in terms of the interaction with the major design issues and their impact on the learning community. As elaborated in Table 1(b), these issues have a signi� cant impact on the presentation of the content and subject matter in the broader learning process. Implicit in this framework is the Enhancing Quality in Online Learning 139 TABLE 1(b). Online content—major components Component Issues Structure If adopting strategies that enable the dynamic construction of organization and knowledge, traditional forms of information presentation may have information to be modi� ed Matches goals and The extent to which program goals and objectives are prede� ned outcomes may be affected by strategies that enable the learner to use knowledge construction techniques Contextual and/or With a dispersed cohort of learners, content must be considered in situated terms of the context in which the learner is situated rather than that of the teacher’s particular experience Information Recognition of the learners’ ability to contribute to the knowledge accuracy, integrity base presents questions as to accuracy and integrity—from whose and totality perspective are these characteristics of the content to be measured and assessed? Accessibility A function of the technology (adaptable for handicapped users, uses appropriate plugins and bandwidth) but also includes accessibility in terms of appropriate language use (terminology, right level and even right language) Extensibility of Is the discipline base so rigid that no options for new content are content considered possible, or can new alternatives be considered for collaboratively constructing and extending the knowledge base? Quality of To what extent can traditional norms for quality of presentation expression be maintained if a more dynamic approach to content is (language, considered appropriate, and what impact might this have on grammar, image roles in the development process? resolution) assumption that content can no longer be seen as being “owned” by the teacher or discipline, but rather as an information base that can be perceived and worked with in many different ways. An illustration of the interplay between content creation and design parameters can be seen when an online environment enables learner creation of content. While the information itself may be linked directly to the discipline, the environment enabling it to be created dynamically may con� ict with accepted organizational or design standards. In developing online content the critical element of the process is to recognize this possibility and to be able to justify why speci� c decisions have been taken. Having established the ways in which content must be understood within the online context, the second factor in the proactive evaluation framework relates to the context or design into which that content will be deployed. 140 R. Sims et al. TABLE 2(a). Options affecting online learning design I ¬ Pedagogy ® C Learning outcome Resources Individual teachers and These options link learning The ways in which media learners have different strategy to outcome and elements are used and the philosophies on the most affect each component extent to which they are appropriate ways that of learning design: accessible will in� uence knowledge is gained and problem solving; declarative the individual components learning acquired. As knowledge; concept learning; of learning design online environments principle learning; procedural can be perceived as learning; cognitive strategies; supporting the attitude & motivation and constructivist paradigm, psychomotor (Smith & Ragan, adopting rigid instructivist 1999) strategies may degrade the overall effectiveness of the encounters experienced by the learners Learning Design The term learning design is used to emphasize the learner-centred environments that online resources can provide. Taking this stance is particularly important because it forces designers to conceptualize the development process from the learner’s perspective rather than that of the content or the teacher. However, this does not preclude developers from adopting an instructivist (I) or presentational strategy compared to a constructivist (C) or generative approach, but does require careful thinking about the learner and the options provided for interacting with the content and their learning partners. As shown in Table 2(a), the design of resources will be in� uenced by the pedagogy, outcome and resources considered appropriate for the task. Within the context of the design process, the team should be able to articulate the underlying pedagogy of the product, the different types of learning being addressed and the ways that media resources will be used to enhance the learning environment. When considered in terms of the speci� c issues that are critical to the overall design task shown in Table 2(b), the complexity of addressing educational and technological elements of the process is further emphasized. Embarking on the design and development of resources for online environments requires new layers of thinking to be added to the well-established principles of course development. Understanding how the content and learning design will impact on the ultimate success of the environments being developed is only part of the overall framework. For teachers and learners encountering the online medium for the � rst time, these concepts must also be linked with the ways that the participants interact with the actual environment, which is the focus of the following two factors. Enhancing Quality in Online Learning 141 TABLE 2(b). Online learning design—major components Component Issues Prior experience Online learning is a new environment, and learners must have the requisite skills to work effectively within this paradigm Approaches to Does the provision of a range of media elements provide learners learning with the resources necessary to meet their approach to learning, and what impact will this set of options have on the overall development effort? Learning The extent to which the environment is designed as a digital environment page or a virtual world will impact on the overall development effort (see Interface Design) Pathways/sequencing The strategies for online learning can establish prede� ned pathways or enable students to explore and discover different facets of the content. Managing these options to minimize information overload becomes critical Outcomes In providing an online environment, are the stakeholders providing for a range of outcomes or are they consistent with prede� ned objectives? Assessment Closely linked to outcomes, are new forms of assessment being considered for the online environment, such as collaborative understanding and concept formation? Level of learning What impact might governmental standards have on the design—and do those standards in� uence or constrain the preferred modes of delivery within the online context? Interface Design The interface between learner and computer is one of the most neglected aspects of online learning, and when coupled with our limited understanding of the complexities of the interactive process (see Sims, 2000), extensive development has to be undertaken to achieve successful and ongoing communication between the learner and the environment. As detailed in Table 3(a), the options available for online productions can range from the non-contextual through to the theatrical, where the learner can be portrayed as an active player in the overall learning process rather than a passive observer (Laurel, 1991; Sims, 2000). Our position is that designers must spend more effort ensuring that learners are integrated into a narrative sequence of the learning process, rather than a familiar solution which sees content being presented in a glamorous and dynamic format but without necessarily achieving engagement with the content through the interface. Overall, the conceptualization of the interface must consider the strategies employed to position the learner within the illusion created by the virtual learning environment; the way in 142 R. Sims et al. TABLE 3(a). In� uences affecting interface design Non-contextual , . Contextual , . Narrative , . Theatrical Interaction Navigation Information design design Input/output design Aesthetics What procedures Have the various How clear are Does moving How does the have been interactivity the options for between “look and feel” employed to options been entering and resources affect contribute to or ensure maximum catered for and accessing continuity of detract from the communication of communicated to content and delivery or communication information? the learner? responses? context? experience? which representations, metaphors, icons are employed to support communication; how design decisions affect connectedness and interference within the learner–computer interface and the extent to which animations and sound effects impact on cognitive load and degradation of learning (Sweller, 1988). The major issues for Interface Design are identi� ed in Table 3(b) and are linked explicitly to the elements of Learning Design, one area that is often neglected by developers new to the creation of online resources. To maximize the online learning experience it is not suf� cient to apply rigorous educational design to content materials, as the means by which that content (resource, activity, conference, reading) is presented to the learner will impact on its overall effectiveness. The factor closely associated with interface design relates to the different ways in which TABLE 3(b). Online interface design—major components Component Issues User comfort— Has appropriate usability testing determined the extent to which connectedness users are able to work with the resources and make the necessary connections between content elements? User control In what ways are users able to control the learning process and link User centred the activities to their own learning requirements? Supports content Has the interface been conceptualized to be consistent with the structure content structure while maintaining acceptable standards? Supports learning Has the interface been designed to be consistent with the particular design approach paradigm employed for the course? Alignment of What strategies have been employed to ensure the mental model of mental models the design group has been effectively communicated to the learner? Customization vs. In what ways can the learner structure the environment to meet their individualization own individual learner needs or preferences? Enhancing Quality in Online Learning 143 TABLE 4. Elements of interactivity Interactivity Motor Cognitive Collaborative Learner–learner The exchange of ideas, resources and information between learners enrolled in a course of study Learner–teacher The exchange of ideas, resources and information between learners and teachers participating in a course of study Learner–content The means by which learners access and make meaning out of Web-enabled content Learner–interface The means by which learners access learning environments and the extent to which they are successful in its navigation Teacher–content The way teachers create and interact with content—see Table 1(a) Teacher–teacher The support provided for community of teachers Content–content To what extent are “intelligent agents” being employed to search and update content materials teachers and learners will communicate with each other and the course resources, a process involving interaction and interactivity. Interactivity Interactivity is about successful communication and, in the context of online learning environments, one of the most crucial success factors. As a component of the human–computer relationship or encounter (Anderson & Garrison, 1998; Hedberg & Sims, 2002; Sims, 2000, 2001), interactivity can include passive presentation, navigation, undirected exploration, directed involvement and speci� c manipulation. The extent to which these constructs of interactivity impact on the continuity of communication between learner and interface, content, other learners or other teachers is critical to the overall effectiveness of the experience and is inextricably linked to the factors and in� uences associated with content, learner design and interface design. Interactivity is not simply a function of computer-based transactions, but a fundamental success factor for teaching and learning, especially when implemented in an online context. In most cases, regardless of any virtual community that exists, the learner will be working independently and therefore the effectiveness of those communications (interactions) will ultimately determine the effectiveness and ef� ciency of the learning environment. As shown in Table 4, elements of interactivity can encompass both human–computer activity and human–human communication. Through the creative process, developers must consider those aspects of both the design and interface that might enhance or impede the success of the different, but often simultaneous, interactions. The ability of the learner to “inhabit” the interactive world presented to them is naturally critical to its success as a learning environment. The way in which the motor (navigational and control interactions), cognitive (engaging and thinking aspects of the interactions) and collaborative (computer-mediated 144 R. Sims et al. TABLE 5. Elements of assessment Assessment Teacher-directed Peer-directed Student-directed Assignments To what extent do assessment items conform to “old standards” and what workload impact does this have on the teacher? Examinations Are examinations required, such as for professional accreditation, or are other performance indicators suf� cient? Project work What options are available for assessment through projects, and which of the participants is responsible for de� ning completion? Work placement Can performance in the workplace ful� l the learning objectives? Authentication Is there concern about the integrity of assessment submissions, or are there other formats that might preclude this operation? interactions with other course participants) elements of an interaction coalesce with the task being undertaken will contribute to the successful engagement of the learner with the activity. To cover all aspects of the teaching and learning environment it is also necessary to focus on the means by which the learning will be assessed, and the extent to which assessment strategies will be created to cater speci� cally for the dynamics of online communities (Palloff & Pratt, 1999). Assessment Much discussion takes place in educational institutions about how best to deploy multiple- choice or short-answer questions in online environments, and what form of authentication should be installed to verify the electronic submission of assignments or completion of remote examinations. However, these strategies seem to contradict those contemporary approaches to learning that advocate active participation by learner and teacher and enable self-assessment and re� ection (Morgan & O’Reilly, 1999). The challenge for planners and designers, therefore, is to determine how they can use online techniques for assessment strategies and at the same time not replicate those strategies designed for the face-to-face environment. As shown in Table 5, assessment can be teacher, peer or student directed and within that context the way in which assessment items are presented becomes critical. The “peer-directed” option provides means for groups to determine and assess the learning outputs whereas the “student-directed” option provides for individuals to de� ne and pursue speci� c learning outcomes. In addition, assessment may also focus on new environments in which the performance data are collected, such as real-world workplace environments. As was identi� ed in the introduction, online teaching and learning environments have particular characteristics that affect both the teacher and the learner. While the discussion to date has focused on the elements of proactive evaluation that impact on support or scaffolding for the teacher, it is also critical that similar consideration is given to the support required for learners who are working collaboratively and online for the � rst time. While many learners new to universities will have excellent computer skills, it is their adaptation to the demands Enhancing Quality in Online Learning 145 TABLE 6. Elements of student support Student support On-campus Mixed-mode Off-campus Auxiliary information How effective are the communities? Communities of Are they encouraged through collaborative activity or learners discouraged due to independent � exibility? Institutional support What expectations do you have for your students? Features Personalization How do you plan to nurture students into the collaborative world of online learning? Security What support personnel and resources have been identi� ed to ensure that students will feel integral to the learning environment? of online learning—independence, collaboration, peer work—that will have signi� cant impact on its success. Student Support Providing appropriate support for the learner cohort is even more critical in the online environment because in many instances they will be working independently in their preferred environment. Even though this environment may include collaborative work, the learner’s only medium of communication is the computer, and therefore support becomes critical to ensure their mental model is consistent with that of the other stakeholders in the process. In addition to the typical help systems, announcements and guides, recent research (Sims, 2000) has suggested that more explicit support is required to bring the learner into the online environ- ment, especially by eliminating assumptions that learners will know what to do and why they are doing it. The environments and issues that impact support are articulated in Table 6. The remaining factors of the proactive evaluation framework focus on organizational elements and the formative and summative evaluation strategies themselves. Utility of Content Within Australia, new digital copyright legislation and the proliferation of digital resources have provided incentives to focus on the international standards for online learning environ- ments. A crucial component of any development exercise, therefore, is to examine the extent to which content can be used in multiple environments (within and outside the product being developed), the means stakeholders might have to customize the materials and the interoper- ability between other learning objects in the wider curriculum. Complicating these factors is the increase in legislative and compliancy conditions; at the time of writing, ensuring online resources do not breach copyright and are accessible for learners with disabilities are part of the quality control process. These aspects are shown in Table 7, with the assumption that compliance achieved by ensuring a robust technological infrastructure underpins the learning environment. 146 R. Sims et al. TABLE 7. Elements of content utility Compliance Copyright Accessibility Infrastructure Multiple use What bene� ts would accrue from a Digital Object Management Customizability System (DOMS)? Interoperability Are the learning objects compliant with international standards? TABLE 8. Factors affecting outcomes Outcomes Program maintenance Quality audit Teacher performance Learning Very simply, did we get it right? Satisfaction What needs to be done to make it better? Results Knowing the parameters that will be used to validate both quantitative and qualitative outcomes of the learning experience will be critical to its ongoing success Outcomes vs. To what extent will the learners continue in the program and objectives develop a sense of lifelong learning? One of the complexities of online development and evaluation is that issues and factors such as accessibility impact on a wide range of environmental and operational elements of online learning. For example, while legislative or policy requirements might insist on online facilities being accessible to people with disabilities, the guidelines will also impact on the look and feel of the interface and ultimately in the communication between learner and people and objects within the learning environment. Outcomes The � nal factor of the proactive evaluation framework is an assessment of the extent to which outcomes have been successfully achieved. For example, measures of learning associated with both intra-curricula and extra-curricula activities; the level of learner satisfaction with the overall experience; the completion rates and the extent to which pass rates and grades are consistent with alternative delivery options. Overall, the design effort needs to include items to enable a comparative analysis of student outcomes in relation to the overall development parameters, as shown in Table 8. Conclusion The capacity of computer-based technology to display combinations of media elements and respond meaningfully to user actions and manipulations has been established for many years. However, the power and capability of the computer to support the learning process is often lost in a maze of marketing publicity and technical gadgetry. Unfortunately, without the requisite Enhancing Quality in Online Learning 147 skills, it has become all too easy to create Web-based materials without understanding the underlying principles of online, interactive, engaging learning. Rather than creating effective learning environments, many development initiatives have proved to be ineffective, with learning activities a confused labyrinth of information, links, collaboration, discussions and navigation. Using the concept of proactive evaluation, a strategy to enable scaffolding and support for teachers and learners new to online environments, the factors and in� uences integral to this framework are considered critical to effective online learning. By focusing on the planning and design phases of the development cycle, proactive evaluation addresses the critical issues associated with the creation of learning resources and environments for delivery in an online context to ensure they will have a greater chance of achieving educational outcomes, with both teachers and learners gaining signi� cant value from their online experiences. Given the extent of this framework, and the importance of understanding the many factors and layers of in� uence that affect the development of effective online resources, perhaps the critical factor is whether online developers and practitioners require a credential to practise. Indeed, the costs of implementation and the risks of failure may well make this an imperative! Acknowledgements Special thanks to Terry Anderson, Peter George, Mike Orey, Daniel Surrey, Karen Handley and Ann Speed of the ITFORUM online community whose feedback and comments during August and September 2001 contributed to the development of this paper. REFERENCES Albrecht, R., & Bardsley, G. (1994). Strategic planning and academic planning for distance education. In B. Willis (Ed.), Distance education: Strategies and tools. Englewood Cliffs, NJ: Educational Technology Publications. Anderson, T., & Garrison, D. R. (1998). New roles for learners at a distance. In C. Gibson (Ed.), Distance learning in higher education: Institutional responses for quality outcomes. Madison, WI: Atwood. Eastmond, N. (1994). Assessing needs, developing instruction, and evaluating results in distance education. In B. Willis (Ed.), Distance education: Strategies and tools. Englewood Cliffs, NJ: Educational Technology Publications. Franklin, S., Peat, M., Lewis, A., & Sims, R. (2001). Technology at the cutting edge: A large scale evaluation of the effectiveness of educational resources. In C. Montgomerie & J. Viteli (Eds.), Proceedings of Ed-Media 2001. World Conference on Educational Multimedia, Hypermedia and Telecommunications. Tampere, Finland, June 25–30, 2001; Association for the Advancement of Computing in Education (AACE). Gore, C., Bond, C., & Steven, V. (2000). Organisational self-assessment: Measuring educa- tional quality in two paradigms. Quality Assurance in Education, 8(2), 76–84. Hedberg, J., & Sims, R. (2002). Speculations on design team interactions. Journal of Interactive Learning Research, 12(2/3), 189–204. Laurel, B. (1991). Computers as theatre. Reading, MA: Addison-Wesley. 148 R. Sims et al. Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learning, and instruction, III: Conative and affective process analysis. Hillsdale, NJ: Lawrence Erlbaum Associates. Morgan, C., & O’Reilly, M. (1999). Assessing open and distance learners. London: Kogan Page. Morrison, G. R., Ross, S. M., & Kemp, J. E. (2001). Designing effective instruction. New York: John Wiley. Owen, J. M., & Rogers, P. J. (1999). Program evaluation: Forms and approache s (2nd ed.). St. Leonards, NSW: Allen & Unwin. Palloff, R. M., & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass. Scriven, M. (1980). Evaluation thesaurus (2nd ed.). Inverness, CA: Edgepress. Simonson, M., Smaldino, S., Abright, M., & Zvacek, S. (2000). Teaching and learning at a distance: Foundations of distance education. Upper Saddle River, NJ: Prentice Hall. Sims, R. (1997). Interactive learning as an “emerging” technology: A reassessment of interactive and instructional design strategies. Australian Journal of Educational Technol- ogy, 13(1), 68–84. Sims, R. (2000). Learners as actors: Strategies for computer-enhanced learning encounters. Unpublished doctoral thesis, University of Wollongong, NSW, Australia. Sims, R. (2001). The online learning alchemist: Preventing gold turning into lead. In C. Montgomerie & J. Viteli (Eds.), Proceedings of Ed-Media 2001. World Conference on Educational Multimedia, Hypermedia and Telecommunications. Tampere, Finland, 25–30 June, 2001; Association for the Advancement of Computing in Education (AACE). Smith, P. L., & Ragan, T. J. (1999). Instructional design. New York: John Wiley. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285. Woodley, A., & Kirkwood, A. (1986). Evaluation in distance learning. Paper 10. Resources in Education. (ERIC Document Reproduction Service No. ED 304 122). Correspondence. Rod Sims, Director, Teaching & Learning Support Unit, Learning Services, Deakin University, Geelong, VIC 3216, Australia. E-mail: rsims@deakin.edu.au Rod Sims is Director of the Teaching & Learning Support Unit with the responsibility to provide educational design, professional development and learning management support to the university community. His research is focused on the learner-computer interface, instructional design and online pedagogy. Graeme Dobbs is Assistant Director OTEN-DE and Manager of Vocational Resources Development, including the Multimedia Unit. He is nationally recognised as an expert in instructional methodologies utilising leading-edge technologies. Tim Hand is Principal Education Of� cer at the Open Training Education Network (OTEN) in Sydney, the largest provider of distance education in Australia. His current research focus is in sustainable production initiatives for the Vocational Education and Training (VET) sector.