Research Article
Traversing the Design-Language Divide in the Design and Evaluation of Physical Learning Environments: A Trial of Visual Methods in Focus Groups
Nicolette Lee
Stella Tan
Swinburne University of Technology
Melbourne, Australia
When gathering data for the design and evaluation of physical learning environments, there is a significant challenge in traversing, or translating, architect/designer language and stakeholders’ verbal accounts of their expectations, preferences and experiences. In a series of studies used to provide data for both the evaluation of existing spaces and the design of future spaces, the authors utilized three focus group methods that incorporated elements of participatory design and visual play in activities. This paper describes these methods and the efficacy of each method in the context of physical learning environment evaluation and design.
Over the past decade, shifts in higher education curriculum and pedagogical approaches, along with an increasingly diverse and technology-socialized generation of students, have prompted a re-examination of where, when, and how learning occurs in a university environment. As a result, campus design historically centered on the development of standardized, functional classroom, and lecture spaces has rapidly refocused on informal and technologically-enriched spaces. The discourse, once centered on functional efficiency, security, occupancy, and maintenance, has also refocused to be dominated by the effect of spaces on learning and the user experience.
The ambitious and widespread nature of these redevelopments, and the substantial costs and risks involved, have given rise to a broadening of discussions in the university sector. In these discussions, the evidence base for developing new learning spaces, or lack thereof, has emerged as a key issue. The limitations of evaluation in relation to operational issues such as per capita floor space, cabling requirements for computers, service levels, or utilisation and occupancy rates have also been brought into sharp relief by the new focus on user experience. Indeed, it has increasingly been argued that rigorous and multi-layered evaluations are needed in order to develop evidence-based models and knowledge for the design of learning environments (Johnson, 2005; Johnson & Lomas, 2005; Mirijamdotter, Somerville, & Holst, 2006).
There has also been an expansion in the use of novel methods of data collection for the purposes of learning space evaluation. While surveys have dominated the movement to carry out evaluations of user experience, there has also been evidence of an increase in the use of mixed-methods and more qualitative or interactive approaches to evaluation. These approaches span from participant and non-participant observation to focus groups and the use of informal stakeholder groups during the design phase. Individual studies cite the use of a range of observational activities, including photography, to document the way the space is used (Hunley & Schaller, 2006). Personal reporting, such as the use of user-experience journals and blogs, has also gained in popularity as a means of data collection. In addition to these methods, focus groups appear to have gained significant traction as a preferred approach to engaging small groups of staff and/or students in both pre-design and post-occupancy evaluation (Learning Landscape Project Team, 2008; Lee & Tan, 2011; Woolner et al, 2007; 2009).
Over the past five years, the authors have been working in the design and evaluation of learning spaces at an Australian metropolitan university. Over this period, iterations of studies have been conducted into the needs and experiences of academic staff and students at both pre-design and post-occupancy stages. The intention of these studies was to gain a better understanding the efficacy of designs in relation to user experience, and to utilize post-occupancy evaluations in the development of new design specifications.
Both quantitative and qualitative methods have been used in this process. This paper focuses on the focus group methods used, and in particular, the use of activities to supplement or supplant focus-group discussions. Three types of focus group activity were trialed, each with a particular data outcome in mind. As part of this process, the activities and their usefulness in gaining insights into the needs and perspectives of users were themselves evaluated. This paper describes the rationale for the selection of those methods, the context and details of their implementation, and their efficacy in gathering detailed and useful information around the user experience that could then be fed forward to the next stage of campus design developments.
Focus groups are a widely utilized research method and are often used as part of a mixed-methods approach. Focus groups, as a form of group interview, are commonly used to follow up after the analysis of survey data (Stewart, Shamdasani, & Rook, 2007) to allow researchers to explore emergent themes in greater depth. As Stewart et al point out, “one of the most appealing features of focus groups is their robust versatility for shedding light on almost any topic or issue” (p. 42). Focus groups are also commonly used for evaluative research or, less formally, as a means of gathering general stakeholder feedback on organisational activities. Stewart et al argue that focus group approaches can be useful in both exploratory and confirmation phases of evaluation. In other words, both as a mechanism for identifying stakeholder needs and as part of a review of outcomes.
While focus groups have a number of advantages as a research method, including their ability to provide rich data very quickly (Silverman, 2001), they also have limitations, including the tendency for dominant voices to overtake the discussion, thereby privileging particular cohorts while excluding others (Cohen & Manion, 1997). Woolner et al (2009) also argue that, when used alone, these are severe limitations of verbal interview techniques. They further suggest that visual mediation activities are a valuable alternative for, or addition to, verbal or written data collection techniques. In particular, they argue that visual activities have significant advantages in situations where the participants are likely to be diverse in age, cultural background or linguistic ability (Woolner, et al., 2010). Such activities can take on a range of forms, but typically involve “enhancing the traditional interview through using visual items, such as photographs, pictures or diagrams to mediate interviews and discussions” (Woolner, et al., 2009). The aim of this process is to produce a more interactive and formative process than that possible by verbal interview alone.
The use of visual and interactive methods during the needs analysis stage of evaluation is also reflected in the literature around participatory design. Participatory design is broadly centered on the notion of collaborations between designers, researchers and user-participants (Blomberg & Henderson, 1990; Sanders, Brandt, & Binder, 2010; Zaphiris, Laghos, & Zacharia, 2009). The advantage of participatory design processes over user-centered design (in which the researcher/designer defines the development process and interprets feedback from users) is that, depending upon the degree of interactivity, participants are able to directly engage in the creative processes of designing and/or problem solving. Sanders (2002) argues that the significant advantage of participatory methods is that tacit knowledge can be uncovered in a way not possible with verbal methods (p. 4) and that: “The new rules call for new tools. People want to express themselves and to participate directly and proactively in the design development process” (p.2).
The three focus group methods described in this paper were intended to contribute to the exploratory and/or review stages of evaluation, and to focus participants on their experiences and preferences. Two of the approaches focused on an evaluation of user experience and had previously been used in primary school campus evaluations. The researchers devised the third approach as part of a needs analysis for the design of new learning environments. In the next section, the context of the studies is briefly outlined. This is followed by a summary description of each method, its use in context, and our findings regarding the challenges and benefits in relation to the intended purpose.
At Swinburne University of Technology, major campus developments have been the subject of evaluation at both pre-design and post-occupancy stages since 2007. Campus developments over that period have included a project hub, a student services building, and a nine-story teaching and research building dedicated to technology programs. A second multi-story teaching and research building is currently underway. The authors have undertaken research with faculty, administrative staff and students during the pre-design, building and post-occupancy stages of each of these building projects. Both quantitative and qualitative methods have been utilized together and separately, as appropriate to the research objectives.
The approaches described in this paper arose out of our experiences of carrying out a pre-design needs analysis. While standard focus groups, using semi-structured interview formats, had proven useful in gathering data regarding the overall experience of stakeholders, and some insight into their needs for particular spaces, we found that there were limitations in the data which appeared inherent to the largely interview-based formats we were using. Specifically, that:
- They were inadequate for translation between design and experiential languages
- They yielded little useful data for a deeper understanding of design requirements
- There was a tendency for ‘group think’ or dominant members to overwhelm the data
While the last of these issues, uneven representation of individuals in the data, is well documented in the literature as a limitation of focus groups, the relationship between design evaluation and focus groups has not been thoroughly explored. The core challenge we found in developing meaningful evaluations in the context of physical learning environments was that design was mediated by participants’ capacity to think and talk in design terms and to relate daily experience with campus design issues. This led to a disconnect between the participants’ experiences and involvement, and the actual design process.
As a result, over subsequent iterations of evaluation we trialed three focus group activities with the intention of identifying mechanisms for more meaningful engagement with user-participants.
The three methods were trialed as part of the evaluation of two buildings. The first of these was the Project Hub, a student-focused learning environment located at the main Melbourne campus of the university. The Hub is a learning environment designed for use by final year students undertaking their capstone projects. Students work in this space without supervision; faculty wishing to use the space for tutorials must be invited by their students. The Hub is designed in several zones, each with a specific type of activity in mind. These are: open spaces with group tables, meeting rooms, a computer area, and informal social areas.
Along with a survey covering student perspectives on the design, fit-out and relevance of the Hub design to their learning activities and needs, focus groups were held to gain further insights into the way that the spaces were used and student perceptions of the design of the zones. Participants were final year students drawn from across the disciplines (n-22) and aged 21-27. Two focus group methods were used: diamond ranking and visual mapping.
The first method utilized a visual activity for gaining feedback regarding design preferences for a social learning environment. Originally developed by a British team (Woolner, Clark, & Ulrike, 2008; Woolner, Hall, Wall, & Dennison, 2007) carrying out school consultations for a government building project, the process falls within the “photo elicitation” range of methods. Based on a thinking skills activity, it is used in lieu of a simple question/answer procedure.
The activity involves providing the participants with nine pictures of areas within a learning environment. The group or groups are given time to discuss preferences for particular areas and for what reason. Then, through group consensus, a diamond shaped ranking is produced showing the most favoured areas at the top, to the least favoured areas at the bottom. The groups are then asked to explain why they decided upon the arrangement they chose. Further discussion is used to explore both the areas of consensus and any disagreements within and across groups.
This activity provides a prompt for conversation and gives a greater opportunity for all participants to influence discussions. Woolner et al. argue that using photographs for data collection prompts participants to provide commentary and perspectives that would not otherwise be given, and also mediates understanding more effectively than purely verbal responses. With sufficient numbers, diamond ranking can also be used to generate quantitative data by placing values on each image (James & Thomas, 2008).
We followed the prescribed procedure for diamond ranking, and provided groups of 4-5 participants with nine laminated photographs of the Hub. The outcomes from Groups 1 and 2 are shown as Figure 1, below.
The results of the rankings provided data regarding which areas students preferred in the Hub and, with the two groups independently choosing similar arrangements, affirmed the more and less favoured areas. Following completion of the ranking procedure, the groups were asked to explain their decisions, and to explain where individuals had differed in their preferences. As well as general discussion regarding the most and least preferred areas, deeper data were found when the groups described why they had chosen a particular hierarchy of preference and where differences of opinion had arisen. Individual participants articulated preferences for quiet or social areas, types of aesthetic or responses to furniture, particular technologies and their varying usefulness for specific activities. Areas of agreement and disagreement equally prompted active discussion and detailed explanations.
The diamond ranking gave participants an opportunity to argue for parts of the space, driving discussion around preferences that might have been less balanced without the visual mechanism. From our experience, we felt that the process did enable the kind of visual mediation and enhancement of active engagement described by Woolner et al (2009). In particular, the use of images allowed participants to first discuss their experience with one another without researcher questioning, and then to drive the following discussion. The process of ranking visuals also gave structure to the discussion and allowed participants time to work through their ideas and elaborate on them. As researchers, we were left with a visual and written record of group preferences and their genesis. This, when combined with researcher notes, helped to validate our interpretations of the discussion, and gave depth to the analysis.
The same British team used visual mapping activities to elicit data around patterns of preference and usage. While diamond ranking helps to generate group decisions, visual mapping allows participants to provide data regarding their individual experiences. Each participant is provided with a pen and a map of the space. They are then asked to draw on the map a typical route that they would take when they visit the space, marking locations they spend time in and making comments where appropriate. After completing their maps, participants are asked to explain the route they have drawn. Mapping in this way not only provides a point of discussion but also visual evidence of common trends and individual differences.
For our purposes, the visual mapping tool was slightly modified. Again, we used the activity with groups of four to six students. Each participant group was provided with a different colored pen and a transparent sheet to draw on, which itself was placed on top of a copy of the Hub map. By having each participant draw on the transparent sheets, it was possible to combine each by layering them on top of one another. This provided a visual means of identifying similarities and differences between each participant’s use of the space. After completing their maps, participants were asked to explain the route they had drawn.
The findings from this activity were similar to, and confirmed the general hierarchy of preferences in, the data generated by the groups completing the diamond ranking activity. The findings also provided some new insights into the way that students selected spaces in a particular order, and why. Specifically, in this activity students were more focused on individual choices and what they did on entry to the space than a discussion of the design characteristics of each space. In other words, the discussion tended toward reflective descriptions of their perceptions of the environment as a social space, rather than aesthetic preferences.
Nonetheless, similar to our experience of the ranking procedure, we found that the activity prompted a great deal of discussion and debate, with similarities of choices becoming clear very quickly. The process also prompted a great deal of discussion about the limitations of the space in peak hours, when preferred spaces were not available, and the degree of social engagement in the space. This flagged a pattern of use that was opportunistic (looking for friends) and indicated that the Hub had become a focus of campus visits, something that was not evident in the diamond ranking groups.
However, despite providing more insight into the broader social aspects of space use, in terms of understanding design needs, this method resulted in more general findings and was less effective than the diamond ranking. As we were evaluating the use of a relatively small and defined learning environment, the findings around navigation were also limited. The method is possibly more effective in a more complex and larger environments, where traffic patterns are less able to be observed and recounted through other methods.
The third method we trialed was for the post-occupancy evaluation of a new teaching/research building at the same campus. In this case, the focus groups were convened to identify how the new group-oriented and technology-enabled classrooms were meeting staff and student needs. Because the rooms in question had been piloted with previously un-tested layouts and technology, we were particularly concerned with their usability. In particular, we hoped to gain sufficient understanding of the experience of pilot rooms to improve the designs for future use and/or retrofitting of the current building.
Again, we used a survey to elicit general feedback and a focus group to gain deeper insights into the experience of users. Focus group participants (n=9) were faculty who had used the new rooms during the first semester of their implementation. Participants were aged 32-48 and came from various disciplines, including humanities, business, information technology and engineering.
While all of the methods are potentially useful for gaining insights into how learning environments are experienced, and therefore how new spaces might be designed, this method was more explicitly forward-looking. We described the process to participants as one that was intended to draw on current experiences to identify effective characteristics for classroom design. These findings would then be used to design possible changes to the existing rooms, and for new developments.
For this purpose, and based on concepts drawn from participatory design, we developed a basic design activity to facilitate discussion around classroom fit-out. The activity involved the use of abstract objects as tools to work through the design problems of creating the ”perfect” teaching environment. As with the first two methods, the intention of the activity was to assist participants to articulate their experiences and thoughts, prompted by the questions raised by the activity. We used a collection of everyday items such as pins, string, ice-cream sticks, Lego, beads, play-dough, marker pens, blue-tack and sticky note pads. The scale, color, and number of objects were selected with the intention of providing sufficient and flexible items that would facilitate creative solutions, promote positive play and not confuse or overwhelm participants.
Participants were provided with a collection of these items in groups of two to three. An art board and a selection of play objects were provided to each group. We provided participants with verbal instructions for the activity at each stage. The first instruction was for participants to imagine, if they were given total control, how they would design a teaching room. They were asked to consider the details of such a space, including the kinds of general equipment and furniture they would use, and the placement of each. We then gave participants 15 minutes to create their first design. The second instruction was to consider the use of technology, and 10 minutes were given for this. The final part of the activity was a whole-of-group discussion regarding the choices made, justifications and comparison of similarities and differences between the designs.
The activity prompted a great deal of creative discussion, argument and laughter as participants interpreted the objects and attempted to work through the problems of designing a complete space. The level of interaction was very high as was the level of participant-driven discussion and debate following the activity. We also saw an increased level of creative solutions to problem experiences between stakeholders than that gained in the standard focus group interviews. While the debate was perhaps more wide-ranging than that found in interview-type processes, we were able to intervene with specific questions and clarifications that provided substantial data on the affordances and barriers to teaching and learning in those spaces, and the variation in use from group to group. In particular, we gathered data around the common patterns of one-to-one, one-to-group and group-to-group communications affecting furniture and technology use, particularly the use of multiple screens and their control.
A secondary, but useful, outcome of this process was a formative element. Participants reported gaining a greater understanding of the complexity of designing a space. Participants also reported that they appreciated the new spaces more following attempts to design their preferred environment, as the activity had given them insights into the conflicts between individual needs and the accommodations that needed to be made in designing for multiple uses.
That is not to say the activity was without challenges. Some participants appeared to be intimidated by the openness of the activity and to be confused by the use of abstract items, having expected a question-answer format. Yet, within 10 minutes participants were actively engaged in imagining what the objects could represent and working through the problems. Participants also reported that they would have liked longer than 30 minutes to complete the activity, despite the scheduling challenges involved. The feeling of time constraint may also explain why participants tended to use very basic forms of representation. Most groups used mostly blocks, marker pens, and pipe cleaners, making use of annotation to explain what was being represented (see Figure 4).
Focus groups are time consuming to carry out and analyze. Qualitative data can also suffer from being skewed by small and self-selecting respondents. In our case, the focus groups also suffered from low face-validity in comparison to the companion surveys. Nonetheless, focus groups can be used to achieve in-depth insight, and to allow for exploratory discussions with participants. We had previously carried out focus groups using semi-structured interview formats, but these had also suffered from a lack of depth and, during analysis, required a great deal of researcher interpretation for design purposes. We also found that the most important aspect of the data, why users found some design characteristics useful, was difficult to interpret from verbal responses alone. As a consequence, transfer of evaluation findings from one context to another remained problematic.
As noted by Woolner et al (2009), visual methods are especially effective where verbal or textual information is difficult, particularly when comparing across groups with varying verbal skill (e.g. children and adults). We would further argue that both visual and design-based methods are equally useful where there is a need to mediate between verbal, visual, and spatial languages, and particularly where participants are not familiar with one or more of those languages. We found that the visual and interactive nature of the activities, when used in a focus group context, was an effective mechanism for engaging participants in more complex discussions and facilitated equitable and active contributions from individuals. Specifically, that the visual cues provided a common point of entry into discussion. Further, we found that the visual and design processes used helped to mediate participant capacity to use design language, and the researchers’ capacity to interpret perspectives articulated by participants.
While we found that diamond ranking and visual mapping provided very similar data sets, they also had sufficient differences to provide distinct insights. The evaluative design activity took a generative approach but equally provided rich insights into participant experience and perspectives. These went beyond a measurement of satisfaction with particular learning environments, giving us a greater understanding of why particular designs were preferred and how they facilitated the work carried out by students and staff. When used in combination with survey material, the data supported the transfer of design characteristics to new contexts.
Blomberg, J. L., & Henderson, A. (1990). Reflections on participatory design: Lessons from the Trillium experience. Paper presented at the Conference on Human Factors in Computing Systems, Seattle, Washington.
Cohen, L., & Manion, L. (1997). Research methods in education (4th ed.). London: Routledge.
Hunley, S., & Schaller, M. (2006). Assessing learning spaces. In D. G. Oblinger (Ed.), Learning Spaces. Boulder, California: Educause.
Johnson, C. (2005). Lessons learned from teaching web-based courses: The 7-year itch. Nursing Forum, 40(1), 11-17.
Johnson, C., & Lomas, C. (2005). Design of the learning space: Learning and design principles. Educause Review, 40(4), 16-28.
Learning Landscape Project Team (2008). Learning landscape project: Research methods; 'Movers and Shapers' focus groups. Cambridge University.
Lee, N. & Tan, S. (2011). A comprehensive learning space evaluation model. Canberra: Australian Learning & Teaching Council. Available online at http://www.altc.edu.au/project-comprehensive-learning-space-swinburne-2008
Mirijamdotter, A., Somerville, M., & Holst, M. (2006). An interactive and iterative evaluation approach for creating collaborative learning environment. The Electronic Journal of Information Systems Evaluation, 9(2).
Sanders, E. (2002). From user-centred to participatory design approaches. In J. Frascara (Ed.), Design and the social sciences: Taylor Francis.
Sanders, E., Brandt, E., & Binder, T. (2010). A framework for organising the tools and techniques of participatory design. Paper presented at the Participatory Design Conference, Sydney.
Silverman, D. (2001). Interpreting qualitative data: Methods for analysing talk, text and interaction (2nd ed.). London: Sage.
Stewart, D. W., Shamdasani, P. N., & Rook, D. W. (2007). Focus groups: Theory and practice (2nd ed.). Thousand Oaks, California: Sage.
Woolner, P., Clark, J., Hall, E., Tiplady, L., Thomas, U., & Wall, K. (2010). Pictures are necessary but not sufficient: Using a range of visual methods to engage users about school design. Learning Environments Research, 13(1), 1-22.
Woolner, P., Clark, J., & Ulrike, T. (2008). Using visual activities to mediate a learning conversation about how a school community regards its premises. Paper presented at the BERA, Edinburgh.
Woolner, P., Hall, E., Wall, K., & Dennison, D. (2007). Getting together to improve the school environment: User consultation, participatory design and student voice. Improving Schools, 10(3), 233-248.
Woolner, P., Thomas, U., Todd, L., & Cummings, C. (2009, 15-17 September). How do visually mediated encounters differ from traditional interviews? Paper presented at the First International Visual Methods Conference, University of Leeds.
Zaphiris, P., Laghos, A., & Zacharia, G. (2009). Distributed construction through participatory design Encyclopedia of Information Science and Technology (2nd ed., pp. 1181-1185): IGI Global.