Making Use of Scenarios for Achieving
Effective Use in Community Computing Contexts
Roderick L Lee
Pennsylvania State University
Craig H Ganoe
Pennsylvania State University
Wendy A Schafer
Pennsylvania State University
Cecelia B Merkel
Pennsylvania State University
John M. Carroll
Pennsylvania State University
Mary Beth Rosson
Pennsylvania State University
Keywords: Effective Use, Scenarios, Participatory Design
The concept of effective use is gaining currency as a way of thinking about usability in community informatics. Broadly defined, effective use is a group’s capacity to integrate information and communications technologies (ICTs) into its current work practices in order to enhance its goal attainment. However, frameworks and methods for achieving effective use are less clearly defined. This paper combines the concept of scenarios from HCI and decision effectiveness from social psychology in order to identify a design process to enhance effective use in community information systems design projects. Our process for achieving effective use focuses first on the efficacy of scenarios as an anchoring and adjustment heuristic and second as a tool to encourage and support participatory design. This study concludes with suggestions for future research on effective use in community computing.
1. Introduction
Effective use is a concept that is emerging as a potential usability goal in community informatics. However, there is a need for concrete tools to help researchers, practitioners and community groups find ways to work together to achieve effective use. Gurstein (2003) defined effective use as “The capacity and opportunity to successfully integrate ICTs into the accomplishment of self or collaboratively identified goals” (p. 12). This is an important and worthy goal but in this formulation its achievement is still abstract. Participatory Design (PD) shares this goal of finding ways to make users an active part of the design process and is especially useful to community informatics researchers and practitioners in trying to achieve effective use in practice through providing ools that can be used to enhance usability.
Community computing facilitates information dissemination and joint activity pertaining to grass-roots mobilization. Because community settings are less typically used for incubating new computer supported cooperative work (CSCW) applications, less is known about how these environments would inform design interventions. In community computing, there are two central challenges that potentially undermine effective use from a PD perspective. The first arises when working with groups on information systems design projects, in finding ways to actively engage the knowledge, interest and skills of new users. This is compounded in community computing by the broader problem of universal access. This phenomenon is being referred to as the organizational divide where there is a lack of financial resources, time and staff that are familiar with ICTs to allow for the strategic use of technology to advance the organization’s mission. In addition, groups often have competing goals, interests and distinct cultures that can stifle active participation in the design process. In response to this we suggest that scenarios can be used as mechanisms to engage users and facilitate their collective involvement in PD in order to develop requirements consensually.
A second challenge that arises when working with groups on information systems design projects is understanding the factors that undermine or enhance effectiveness. Given that gathering requirements is an ill-structured, knowledge-intensive problem solving activity that involves negotiation, emphasis must be placed on enhancing the decision-making activity with respect to the problem at hand. Drawing on the field of judgment and decision-making, a key issue concerns the choices made after some degree of deliberation. From the standpoint of decision effectiveness, we suggest that scenarios used as an anchoring and adjustment heuristic takes us a step closer to achieving effective use.
In this paper, we demonstrate the usefulness of one tool, scenarios, to help a group come up with requirements for building a Community Information System (CIS). The main purpose and contribution of this paper is to clarify the notion of effective use and to broaden the range of organizational models countenanced in CSCW. This study adds to the extant literature by making explicit the contribution of scenarios in a community computing context.
2. Background: Civic Nexus
This work is motivated by our Civic Nexus project. Civic Nexus is a three-year participatory design project with the goal of working with community groups to facilitate their ability to use and to learn about technology as they pursue existing goals and as they envision new directions for their community. A goal of this project is to assist communities in taking control of technology in order to achieve goals that they see as important.
As an extension of this project we are engaged in an ongoing design collaboration with the Underground Railroad community of practice in Pennsylvania. Our collaboration with the UGRR research community began during the fall of 2003. This community is an IT-marginalized, grass roots, distributed group that spans multiple organizations and geographical regions. Our community partners include the Pennsylvania Historical and Museum Commission (PHMC), Pennsylvania Federation of Museums and Historical Organizations (PFMHO), the Pennsylvania State Library, Center for Anti-Slavery Studies (CASS), the Underground Railroad Research (UGRR) Community in Pennsylvania, and Penn State University faculty and students.
The UGRR community is a group of users who share a common practice in that they all work in some capacity related to the preservation of UGRR history. However, this particular community relies mostly on volunteer effort. The most pressing issues for this group are lack of a technical infrastructure, and limited access to resources, time, and technical expertise. Apart from an initial start-up grant from the Federation of Museums and Historical Organizations to support undergraduate students, all support for the project is volunteer.
The core group of the community has embarked on an ambitious endeavor to develop a novel community information system with a supporting infrastructure in order to share information and knowledge pertaining to the UGRR activity in Pennsylvania and to engage in mutual learning. The community envisions developing a CIS that will connect local pockets of expertise and serve as a gathering place so that isolated professionals can share and develop knowledge pertaining to the UGRR.
There are 67 county historical societies in Pennsylvania and a multitude of other organizations that are actively engaged with research concerning the UGRR. Each holds a unique piece of the rich UGRR history. Currently, only 16 of these county historical societies are actively participating in the UGRR domain. Of these 16 historical societies, only three are listed in the National Park Service’s Underground Railroad Network to Freedom database. One member of the group serves as the community coordinator and boundary spanner. This individual (Sue) is actively engaged in an effort to link these disparate groups covering the entire state.
The following scenario with Sue exemplifies this role. Sue begins the task of soliciting support by physically visiting each region of the state in order to communicate the community vision with other stakeholders who will eventually become users of the system. Sue’s activities take an enormous amount of time and can detract from the team building necessary to complete the design project. Based on this scenario there is a concrete task in engaging the user in deriving a solution to this real problem--managinjg the communication, coordination and cooperation with stakeholders from other organizations. Moreover, the participants need a way to share information with one another and a gathering place where they can discuss this time period that is conducive to the development of this community.
Through scenario-based techniques and PD methods, we have collaboratively decided on three features of the system to support and extend the current work practice. First, the system should include an information portal to provide information to other researchers. Second, the system should consist of a private workspace for the community to link and coordinate unconnected activities pertaining to the UGRR history. Thirdly, for the future there is a plan to incorporate a collaborative mapping feature that enriches the quality of the user’s experience and facilitates complex decision-making with respect to disputed sites. Finally, the system will include an interactive database of 744 slave cases with a “query” functionality that will enable users to identify patterns in the data.
This will add value in a number of ways. First and foremost, the system will provide a forum to publicize an often misunderstood chapter of American History. Moreover, the system will promote awareness, scholarship and facilitate greater understanding of the historical significance of the UGRR movement and its implications for today. As already noted, practical design consideration include how to engage the knowledge skills and interest of novice users and developing the capacity of the community so as to support it in making effective design decisions.
3. Conceptualizing Effective Use
Contemporary work groups and teams in organizations differ dramatically from civic sector groups. However, approaches to usability have been derived mainly from studies of information systems in organizations. Evaluation of design interventions involving work groups and teams are assessed through the construct of team effectiveness. Team effectiveness, which consists of performance and viability, is broadly defined as the quantity and quality of a team’s outputs over time. Drawing on empirical research and theory from social psychology our understanding of effective use is broadened to encompass community viability. We operationalize effective use in the context of team effectiveness as: the capacity, willingness and opportunity to successfully integrate ICTs into the accomplishment of self or collaboratively identified goals over time. Our definition captures the group’s capacity, willingness, and opportunity to integrate technology into their work practice.
In this study, we are employing participatory design methods to involve the users directly in the design process. Participatory Design is a context-specific design approach involving users early in the decision-making process from the initial brainstorming all the way through the design process. Because PD is decision intensive, we need to consider how to enhance the quality of decisions that are made in order to achieve successful outcomes. Therefore, we limit our discussion to one central activity – group decision-making. Huber and McDaniel (1986) define a central activity as one whose successful execution is critical to the accomplishment of goals. Quality decisions in PD assist groups in achieving their goals and enhance effective use. Bad decisions however hinder goal attainment which in turn negatively impacts on effective use.
In general, decision-making is defined as the sensing, exploration and definition of problems and opportunities as well as the generation, evaluation and selection of solutions. The scope of our work is at the community level of analysis. At the community level, decision-making is defined as the process in which a community selects a course of action to respond to both problems and opportunities. However, group decision-making is more complex than individual decision-making for several reasons. First, high performing groups are effective at combining multiple preferences and beliefs. Second, high performing groups must maximize social interaction. Finally, groups must have developed effective norms for managing conflict.
Creativity and innovation are parameters used in the context of decision-making. Creativity is an element of a decision-making process in which the group produces novel and useful ideas in response to problems and opportunities. Novel ideas represent new ways of thinking. Useful ideas have the potential to contribute to effective use. Innovation is the successful implementation of creative ideas. Our goal is to use scenario exercises in order to scaffold the community from creativity to innovation.
4. Scenarios as Anchoring and Adjustment Heuristics
A growing number of scholars have evoked the concept of scenarios as a way to bridge the design-actuality gap and to narrow the specification-implementation gap. Recently, scenarios have been used to anchor problem-based learning in instructional settings and as tools for facilitating participation in community computing contexts. Conversely, a number of scholars have pointed to the importance of scenarios in enhancing usability, increasing our knowledge of a problem space and as tools for inquiry and discussion. We emphasize usability in the context of designing a community information system to support and extend the workplace roles and practices of a geographically dispersed community of practice. This is invariably tied to participatory design efforts in which community groups are made an active part of the design process as a way to bridge differences between the designer, user and system models.
The concept of scenarios emerged from the field of strategic management in the 1940s. In this context, scenarios are used as a tool for change management. Scenarios are work-oriented design objects that are defined as stories about people and their activities. “The defining property of a scenario is that it projects a concrete narrative description of activity that the user engages in when performing a specific task, a description sufficiently detailed that design implication can be inferred and reasoned about”.
In practice it is easy to spend an enormous amount of time generating creative ideas with little innovation or likelihood of implementation. For example, a group could become frozen in the creativity process which limits the opportunity to collectively define problems and identity solutions. Innovation might be problematic in the UGRR community because of the multi-faceted nature of this project. This project includes representatives from diverse organizations. In addition, the project includes a team of researchers from two locations of The Pennsylvania State University’s School of Information Sciences and Technology, as well as undergraduate students in an introductory HCI course. In some cases, this has resulted in a lack of common ground. For example, conflict arose from a lack of understanding of the diverse organizational cultures. One participant referred to this phenomenon as the town and gown culture. In this instance, we did not make it crystal clear to our partners that our interest in the project was limited to investigating computer supported collaboration. Moreover, we did not emphasize that our mission was to assist them in achieving their goals. In order to move from creativity to innovation, it is necessary to enhance common ground. We suggest that scenarios are key to effective communication and decision-making and are essential to achieving effective use.
Scenarios enhance decision-making by providing detailed knowledge of the application domain and provide an anchor for inquiry and discussion. As such, scenarios are malleable tools for decreasing errors in the decision-making process. Errors in decision-making are a result of heuristics and biases. The heuristics and biases paradigm was the dominant approach for decision-making under uncertainty in the1970s and 1980s. Heuristics are cognitive rules of thumb or mental shortcuts that humans use to simplify information processing. We limit our discussion here to the anchoring and adjustment heuristic.
Systematic evidence to support the anchoring effect first appeared in the 1970s. Two problems were identified with the anchoring and adjustment heuristic. First, the anchor might not be appropriate. Second, the adjustment from the anchor might not be sufficient. One way to overcome this is to use scenarios (the anchors) as an initial starting point and then make adjustments (analysis and refinements).
From the standpoint of anchors, scenarios provide a language for community members to express their desires/requirements as well as a language for designers to help community members to see how the system might work and envision possibilities they had not considered. Scenarios also facilitate the convergence of conflicting perspectives between users and designers. Moreover, scenarios are a way to facilitate the effective communication necessary to achieve common ground. We suggest that effective use is best achieved by starting with an initial scenario (anchor) and then adjusting toward the ultimate goal. For example, a user envisions a requirement, embodies it in a scenario of interaction and reflects on it and comes up with new way of thinking about the problem.
5. Scenarios to Encourage and Support Participatory Design
In this study, we took a PD approach where users are not only experimental subjects but also members of the design team and are actively engaged in the decision-making process. Users can be classified within three levels of participation: consultative, consensus or representative. Consultative participants offer opinions; however, they are not involved in the decision-making process. Select members of the 67 historical societies in Pennsylvania will eventually serve as consultative participants. In consensus participation, all stakeholders are involved in the decision-making process. The UGRR project involves stakeholders from multiple organizations. Because the focus of this treatise is at the community level of analysis, representative participation is more conducive to achieving effective use.
In PD, scenarios present a novel approach to be used as a communication tool and as a language for stakeholders with competing interests and experiences to talk with each other. There are many times when competing interests and experiences lead to conflict. This conflict might cause a group to lose its focus and hinder goal attainment. Unmanaged conflict could ultimately derail a design project. Scenarios can assist in keeping the group focused by providing a point of reference and a cognitive aid. This facilitates the communication necessary for stakeholders with differing visions and diverse cultures to achieve common ground. Additionally, scenarios can be used to create simulated worlds in which designers and users can mutually explore design options and make informed decisions. Therefore, scenarios are useful in getting everyone to connect to the design process by illustrating important design decisions in a flexible manner that is understandable to both the user and the designer .
6. Scenarios in the UGRR Project
The primary sources of data were derived from workshops, training sessions, and semi-structured discussions. Workshops and training sessions occurred in downtown Harrisburg at the Penn State University Eastgate Center. We chose this off-campus location in order to make the experience as informal as possible. Through the workshops we were able to highlight areas of conflict and develop strategies for conflict management. Semi-structured discussions occurred at the actual worksite, via telephone and at the university. These discussions were transcribed and then content analyzed. The results of the discussion lead to detailed case material for an undergraduate survey course on human-computer interaction.
6.1 Scenarios in the HCI Curriculum
The first phase of this study involved using the domain problem as case material and service learning opportunities for students in an introductory HCI Course. Each student group was assigned a user population for the group project. They then studied the work practice in order to develop scenarios. Scenarios were used to scaffold the learning experience of students as they acquired discipline specific knowledge, skills and expertise. Through the use of rich descriptive scenarios and as students worked in project teams, learning occurred through the processes of interaction, negotiation and collaboration. The class mainly concentrated on developing problem and activity scenarios. Problem scenarios are used to synthesize field observations and are iteratively refined during the requirements analysis phase. Activity scenarios are used to envision future practices.
Based on feedback from students and course evaluations, scenarios enhanced the students’ satisfaction with the project. For example, one student reported that “Using scenarios helps us to see the user as a real person, putting us in their shoes. This enables us to better determine what the application will be used for, how it will be used, and who the intended audience will be (their background, skill level, and the technology available to them). Personally, our group found scenarios to be useful in helping us to recognize the needs of the users. Interacting with ‘real life’ users helped us to keep in mind that we were developing the system for them, based on their requirements, not our own. This enabled us to consider each user category throughout the project and design the system accordingly. Using scenarios helped us to be able to look back, periodically, to ensure that the system's functionality met the users' requirements and expectations.”
Another student indicated that “Scenarios are extremely useful and force the developers to put themselves in the users' place to design a product that will fulfill their needs. The scenarios allow developers to quickly and efficiently get an idea of the requirements of a product.” A possible explanation could be that scenarios enhanced the decision-making ability by providing detailed knowledge about the application domain and a story in which design decisions could be inferred and reasoned about in the course of their work.
6.2 Scenarios in the UGRR Community
We used short narratives of interaction during our communication with the users. These narratives were used in order to determine the requirements for the aesthetics and navigation of the site. In addition, we are currently working on scenario development and analysis of the requirements for the actual workspace. With respect to the aesthetics and navigation, we conducted workshops in which similar sites were used as anchors. From this interaction, we were able to envision scenarios that were applicable to the design project.
One future application is the development of an interactive mapping feature to facilitate complex decision-making. An example scenario follows.
Sue is a historian in a local community historical society. Sue enters an online forum on the UGRR community network. There is a new thread related to a heated dispute between concerned citizens and community and state officials. The issue at hand is whether or not a sewer line expansion project can begin. The project would require the demolition of a vacant church. Although the church is vacant, Sue remembers stories that the church was indeed part of the UGRR. Sue decides that she would like to get involved; however, the project is scheduled to begin in 24 hours. She quickly recalls that Tim, a researcher at Florida A&M University, conducted research on the A.M.E. churches in Pennsylvania. She emails Tim to inquire about the church. Tim confirms that this is indeed a historic church. In order to persuade interested parties, Tim suggests representing the church on a map. However, both recall problems in mapping historical landmarks on traditional maps.
This problem scenario provides an example application for the workspace. It is in this context that we will focus our attention on effective use and the efficacy of scenarios to facilitate goal attainment. This will require a collaborative mapping tool and shared editor so that these distributed researchers can plan what parts of the site need to be protected along with the use of a Global Positioning System (GPS) so that researchers can collect and share on-site details with the rest of the UGRR community.
7. Conclusions
The UGRR community models many key requirements for collaborative environments and represents a rich test bed to gain a broader understanding of community groups and envision possibilities for CSCW applications. Community computing is more specialized than other domains. Community groups are bounded rationally and may be limited in their ability to make informed technology decisions. They lack the time, resources and technical expertise necessary to participate in novel development projects. This constrains their ability to make optimal choices; instead, they satisfice.
Our analysis of effective use revealed the need to explicate the dimensions of effective use. For example, what motivates groups to participate in collective endeavors? Although we illustrated the efficacy of using scenarios to enhance decision-making and participation in PD, we did not explain the underlying causal factors. The current approach is two-dimensional and suggests that effective use = ƒ (capacity x opportunity). In this model, capacity and opportunity are the key determinants of performance. The opportunity x capacity model does not account for the willingness of the individuals to actively engage in joint activity over time. For this reason, we suggest a broader framework and definition of effective use. We suggest that a three dimensional model that incorporates motivation, moderated by time is a more fruitful approach.
The approach at the very least, should be three-dimensional to include a moderator that indicates that effective use = ƒt(opportunity x capacity x motivation). Such a model is based on the assumption that effective use is a function of opportunity, capacity and motivation moderated by time. Therefore, the concept of effective use is extended and defined as: the capacity, willingness and opportunity to successfully integrate ICTs into the accomplishment of self or collaboratively identified goals over.
In this case study, we suggest that scenarios coupled with participatory design methods can be appropriately malleable tools for facilitating effective use. Our preliminary work suggests that scenarios are powerful tools that can be used as anchoring and adjustment heuristics as well as mechanisms to engage users in PD. Moreover, we suggest that scenarios may also be viable tools for conceptualizing and consequently achieving community effectiveness.
8. Future Research
An area for future research is the creation and maintenance of social capital. Putnam has shown that there has been a decline in social capital. He noted that we need to strengthen connectivity and maintain social networks that help us become more involved in communities in order to achieve social goals. In organizational science and social psychology, social capital is used to explain the joint effects of knowledge distribution and network structure on team performance. At the level of the work group, social capital represents the goodwill derived from the network of relations that can be mobilized to facilitate the pursuit of collective goals and team effectiveness. However, scholars are split between two opposing views: bonding and bridging social capital. Therefore, some have chosen the optimal configuration perspective. What is needed is a model to analyze social capital, learning and effective use at multiple levels. We suggest that activity theory provides a promising approach to producing a more refined analysis of the antecedents and consequences of community effectiveness and learning.
Adler, P., & Kwon, S. (2002). Social capital: Prospects for a new concept. Academy of Management Review, 27(1), 17 - 40.
Carroll, J. (1997). Scenario-based design. In H.M. Helander, T.K. Landauer & P. Prabhu (Eds.), Handbook of Human-Computer Interaction (Vol. Second Completely Revised Edition, pp. 383 - 406). Amsterdam: Elsevier.
Carroll, J. (2001). Community computing as human-computer interaction. Behaviour & Information Technology, 20(5), 307 - 314.
Connolly, T., & Ordonez, L. (2003). Judgment and decision making. Handbook of Psychology, 13, 493-517.
Farooq, U., Merkel, C., Nash, H., Rosson, M., Carroll, J., & Xiao, L. (2005, January 3-6, 2005). Participatory Design as Apprenticeship: Sustainable Watershed Management as a Community Computing Application. Paper presented at the 38th Annual Hawaii International Conference on System Sciences, Waikoloa, Hawaii.
Farooq, U., Merkel, C., Xiao, L., Nash, H., Rosson, M., & Carroll, J. (2006). Participatory design as a learning process: Enhancing community-based watershed management through technology. In The Environmental Communication Yearbook, III. Mahwah, NJ: Lawrence Erlbaum Associates.
George, J., & Jones, G. (2002). Understanding and managing organizational behavior (Third ed.). Upper Saddle River, NJ: Prentice-Hall.
Gurstein, M. (2003). Effective use: A community informatics strategy beyond the digital divide. First Monday, 8(12).
Hertzum, M. (2003). Making use of scenarios: a field study of conceptual design. International Journal of Human-Computer Studies, 58, 215 - 239.
Huber, G., & McDaniel, R. (1986). The Decision Making Paradigm of Organizational Design. Management Science, 32(5), 572 - 589.
Jarke, M., Bui, X., & Carroll, J. M. (1998). Scenario management: An interdisciplinary approach. Requirements Engineering, 3(3 - 4), 155 - 173.
Jarke, M., & Kurki-Suonio, R. (1998). Introduction to the special issue: Scenario management. IEEE Transactions on Software Engineering, 24(12), 1033 - 1035.
Kirschenbaum, J., Kunamneni, R., & Servon, L. (2002). The Organizational Divide. In L. Servon (Ed.), Bridging the Digital Divide: Technology, Community and Public Policy. Malden, MA: Blackwell Publishing.
Kraut, R. (2003). Applying social psychology theory to the problems of group work. In J. M. Carroll (Ed.), HCI Models, Theories, and Frameworks: Towards a Multidisciplinary Science (pp. 325 - 356). New York: Morgan Kaufmann Publishers.
Kyng, M. (1995). Creating contexts for design. In J. M. Carroll (Ed.), Scenario-based Design: Envisioning Work and Technology in Systems Development (pp. 135 - 163). New York: Wiley.
Lesser, E. (Ed.). (2000). Knowledge and Social Capital: Foundations and Applications. Boston, MA: Butterworth-Heinemann.
Mehra, B., Bishop, A., Bazzell, I., & Smith, C. (2002). Scenarios in the Ayfa Project as a participatory action research (PAR) tool for studying information seeking and use across the "digital divide". Journal of the American Society for Information Science and Technology, 53(14), 1259 - 1266.
Merkel, C., Xiao, L., Farooq, U., Ganoe, C., Lee, R., Carroll, J., et al. (2004, July 27 - 31, 2004). Participatory design in community computing contexts: Tales from the field. Paper presented at the Proceedings of the Eighth Biennial Participatory Design Conference, Toronto, Canada.
Mumford, E. (1983). Designing participatively. Manchester: Manchester Business School Publications.
Nahapiet, J., & Ghoshal, S. (1998). Social capital, intellectual capital and the organizational advantage. Academy of Management Review, 22(2), 242 - 266.
Norman, D. (1990). The design of everyday things. New York: Doubleday.
Putnam, R. (1995). Turning in, turning out: The strange disappearance of social capital in America. Political Science & Politics, 28(4), 664 - 684.
Putnam, R. (2000). Bowling alone: The collapse and revival of American community. New York: Simon & Shuster.
Rosenthal, E. (1996). Social networks and performance. Team Performance Management, 3(4), 288 - 294.
Rosson, M., & Carroll, J. (2002). Usability engineering: Scenario-based development of human-computer interaction. New York: Morgan Kaufmann Publishers.
Rosson, M. B., & Carroll, J. M. (1995). Narrowing the specification-implementation gap in scenario-based design. In J. M. Carroll (Ed.), Scenario-based Design: Envisioning Work and Technology in Systems Development (pp. 135 - 163). New York: Wiley.
Schneider, J. (2003). Small, Minority-Based Nonprofits in the Information Age. Nonprofit Management & Leadership, 13(4), 383 - 399.
Shea, G., & Guzzo, R. (1987). Groups as human resources. In K.M Rowland & G. R. Ferris (Eds.), Research in personnel and human resource management (Vol. 5, pp. 323 - 356). Greenwich, CT: JAI Press.
Simon, H. (1973). The structure of ill-structured problems. Artificial Intelligence, 4, 181 - 202.
Sundstrom, E., De Meuse, K., & Futrell, D. (1990). Work teams: Applications and effectiveness. American Psychologist, 45(2), 120 - 133.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124 - 1131.
Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453 - 458.