ACTA ACUSTICA UNITED WITH ACUSTICA Vol. 100 (2014) 154 – 165 DOI 10.3813/AAA.918695 A Qualitative Approach to Evaluation of Perceived Qualities of Audio and Video in a Distance Education Context DanNyberg, JanBerg LuleåUniversity ofTechnology. Piteå,Norrbotten, Sweden. [dan.nyberg, jan.berg]@ltu.se Summary This study presents a qualitative method for collecting and analysing data to describe audio and video quality. Used in the social sciences, arts, and humanities, this approach relies on phenomenology and hermeneutics and uses interviewsandquestionnaires toassess theaudioandvideoqualityofmasterclasses inclassicalmusic taught via the Internet. Although this study is only exploratory, it provides evidence that themethod could successfully be used to gather descriptions of perceived qualities. PACS no. 43.71.Gv, 43.72.Kb, 43.75.Cd, 43.75.St 1. Introduction There are numerous contexts inwhich evaluation of audio quality may be performed and several evaluation meth- ods have been developed over the years. Although each method has its ownmerits, specificmethodsmay bemore suitable in specificcontexts.Whenanewcontext emerges, these methods need to be evaluated, refined, and consid- ered. Existing methods, however, mainly rely on experi- mental data that comes from carefully manipulated situ- ations to control experimental variables. Exclusively re- lying on experimental data may not actually reflect the reality that the experiment aims to represent, i.e. the ex- periment’s ecological validity may not be sufficient. The concept of ecological validitywas addressed byBrunswik [1] and later discussed by Gibson [2] who elaborated on examples relating to visual perception (e.g. drawings and how well they represent reality). For reproduced sound, Guastavino et al. [3] suggested that listening tests should be designed to match the aim of the study through gener- ation of stimuli that enable subjects to “treat the test sam- ples as potentially familiar experiences through cognitive processes elaborated in actual situations”. Consequently, due to the risk of not attaining a sufficient level of eco- logical validity, there are situations where alternatives to experimental studies should be considered. Thus, relying onlyonexperimentaldatacanbean issuewhenbothaudio and video qualities are of interest and/orwhen other qual- ity aspects are involved. In addition, most of these meth- odswork by quantifying the listener’s experiences, an ap- proach that limits the type of data collected. Received 3September 2012, accepted 13October 2013. This limitation is imminent in the context of distance education over the Internet, which involves audio, video and real-time interaction. This becomes significantly im- portant in distance music education (MED). Here the re- search has increased over the last ten years [4, 5]. Typi- cally, a MED teacher located at one school teaches a stu- dent located at another school via the Internet, often by means of a video conferencing system [4, 5]. Several findings regarding teaching possibilities and at- titudes have been made in earlier research. Greenberg’s white paper on video conferencing-based distance educa- tion [6] shows that teachers have to adopt their teaching to accommodate; this involves e.g. interactivity betweenpar- ticipants and teaching strategies that have to be matched with the technology. Work by Masum et al. [4] in the project “MusicGrid” adds to this by noting that teaching musicoverdistance requiresboth frequent demonstrations and continuous feedback during the lessons. Findings by [4, 6] also indicate that participants become quickly adapted and comfortablewith the system.Both [4, 6] con- clude that distance education can be cost effective and al- lows users to access more learning experiences without travelling.Shepard [5] takesamore technicalviewandex- plains the need of striving to have high quality audio and video, bidirectional transmission and easy-to-use system in a MED situation. Shepard also gives indications that conferencingsystemsusingMPEG-2encoderanddecoder canwork if used right. Research efforts have also been made by Woszczyk et al. [7] to establish a transparent systemfor connecting two physical locations via a shared electronic virtual space for musical practice. However, few studies specifically focus on how MED students and teachers perceive the audio quality of a particular system. Woszczyk et al. [7] con- clude that inaddition toaudioquality, other factors suchas 154 © S. Hirzel Verlag · EAA Nyberg, Berg: Evaluation of perceived qualities ACTA ACUSTICA UNITED WITH ACUSTICA Vol. 100 (2014) image size of video, tactile/haptic response (low frequen- cies transmitted via separate channels to simulate floor vi- brations) and synchrony between audio and video are in- volved in creating a transparent MED system. They also found thatwhen asynchrony between audio and video oc- curs, participants often only focus on themusic. This paper presents a selected qualitative method for data collection and data analysis that is commonly found in the social sciences, arts, and humanities and applies these methods to evaluate audio and video quality as it is perceived by teachers and students in a non- experimental distance education setting (specifically in a real-life clas- sicalmusic course conducted via the Internet). The terms audio and sound quality will be used inter- changeable during the course of this paper. The term non- experimentalmeans that the researcherhasnocontrolover the parameters such as choice of participants (other than inviting students already enrolled in the classes), where the study took place, transmission quality between the lo- cations, andchoiceofequipment.The term real-lifemeans that the lessons in the classicalmusic course are authentic lessons in which the researcher does not intervene. These qualitative methods are used because the study was con- ducted in non-experimental and real-life conditions. Al- though the results from such a studymay be used to eval- uate the quality of audio equipment, this issuewas not the targetof this study.Rather, this study intends to investigate what type of data a researcher can collect by using these qualitativemethods and to show the applicability of these methods in the field of perceptual quality. 2. Earlier research on perceived audio and video quality Asstated in the introduction,MEDinvolvesmore thanone perceptual qualitymode (e.g. auditory, visual and tactile). Previous research on the quality of perceived audio, which is one of the modalities under investigation, shows that it has been successfully evaluated. These data collec- tionmethodsoftendealwith rating scales [8, 9, 10, 11,12] and descriptive analyses and use the subject’s own vocab- ulary to evaluate the perceived audio quality [13]. What all thesemethods have in common is that they all are con- ducted in laboratory settings and follow an experimental paradigm. Toole [8] as well as Gabrielsson and his co- workers (e.g. [9]) conducted research that comprised lab- oratory experiment using subjects as evaluators of sound quality, inmoreparticular, the soundqualityof loudspeak- ers was evaluated by means of rating scales. The exper- imental paradigm has since then been a predominant ap- proach for audio quality evaluation through subjective as- sessments. Bech and Zacharov give a comprehensive re- viewof different approaches in their book [10]. Since then, several recommendations have been devel- oped to evaluate subjective audio quality with the use of rating scales, for an example ITU-R BS.1116-1 [11] and ITU-R BS.1534-1 [12]. The former focuses only on as- sessment of small impairments of audio quality, while the latter focuses on assessment of intermediate audio qual- ity. There has also been research employing subjects’ own vocabulary in order to find an appropriate descriptive ter- minology that may be used for subsequent quantification of perceived audio characteristics [13, 14]. There are also a variety of methods designed to evalu- ate perceivedvideoquality.Thesedata collectionmethods range from scales [15, 16] to interpretation-based estima- tionsof imagequality [17]orboth [18].Theseapproaches, however, only look at onemodality. In multimodal research, the focus shifts since both au- dio and video are involved in a MED situation. In the multimodal field, several approaches are used to evalu- ate multimodal quality. Some of the approaches use a mixed-methodsapproachwherebothnumericanddescrip- tive data are analysed. For example, Strohmeier et al. [19] used a mixed-methods approach to evaluate multimodal quality perception of audio andvideo, inspired by sensory evaluation methods; they used the method Open Profil- ing ofQuality to understandquality perception. TheOpen Profiling of Quality contained three parts: a method they call psychoperceptual evaluation (the subject uses rating scales to evaluate perceived audio quality); sensory pro- filing (collecting individual quality attributes); and exter- nal preference mapping (constructing links between the psychoperceptual result and the sensory profiling). Using analysis of variance (ANOVA) and Principal Component Analysis (PCA), they analysed thepsychoperceptual eval- uation and sensory profiling, respectively. Jumisko-Pyykkö et al. [20] used a qualitative approach to evaluate audio-visual quality by combining a single-sti- mulusmethod, forqualityevaluation,withsemi-structured interviews, to establish quality evaluation criteria. The data analysis combined a qualitative analysis based on grounded theory and a quantitative statistical analysis, Bayesian modelling. Jumisko-Pyykkö et al. [21] also ap- plied an entirely qualitative approach (semi-structured in- terviews) and compared the qualitative results with quan- titative results in order to understand the relationship be- tween produced and experienced quality in the context of interactive audio-visual systems. The analysis used here also followed the grounded theory approach. This qualita- tive approach showed that itwaspossible to collect 13dif- ferent categories of quality criteria from the subjects and that these categories were related to the quantitative re- sults. This is an indication that a qualitative method can complement quantitativemeasurements in order to under- standmultimodal quality. Other researchers have employed single-method ap- proaches [22, 23, 24, 25] in multimodal quality research. BeerendsandDeCaluwe[22] tested the influenceofvideo quality on perceived audio quality (and vice versa) using a nine-point absolute category rating scale. The data anal- yses were made using ANOVA. Bech et al. [24] analysed the interactionbetweenaudioandvisual factors in thecon- text of a home theatre system. The experiment collected numeric data from rating scales using defined attributes with anchors and the datawere analysed usingANOVA. 155 ACTA ACUSTICA UNITED WITH ACUSTICA Nyberg, Berg: Evaluation of perceived qualities Vol. 100 (2014) All methods presented above have shown to be well suited for the intendedpurposes anddeliveredvaluable re- sults; however, thesemethods aredesigned for experimen- tal or laboratory contexts and some of the methods could be hard to adapt to a real-life situation (non-experimental situation) such as the MED situation. One cannot control all parameters involved in an MED situation, contrary to the experimental research presented above. Clearly, labo- ratory testing cannot completely account for the real life situations found in distant learning MED, a situation that is relatively unstudied and may require new methods to develop a complete picture of its usefulness. 3. A qualitative approach When considering a suitable method in a relatively new, non-experimental, and multi-modal context, such as mas- ter classes in classicalmusic over the internet, another ap- proachmay be required that uses qualitativemethods that focus more on describing and understanding the experi- ence than on quantifying it. An example of viewing sound quality from a some- what different angle has been suggested by Blauert and Jekosch [26]. They present a perceptionism view of the world, a view that relies on the belief that all knowl- edge is based on sensory perception. This view allowed them to create a layer model of audio quality based on the amount of abstraction involved for each judgment of audio quality. The model enabled them to show the sit- uation dependence of different audio qualities based on the degree of abstraction. One of their findings, the qual- ity called“Aural-communicationQuality”,which involves thehighest amountof abstraction, canbe investigatedwith psychological and cognitive tests in real cases such as in tests of usability, comprehension and dialogue qual- ity. The situation dependence in their model relates well to the previously discussed concept of ecological valid- ity, which further strengthens the argument for consider- ing non-experimental studies outside the laboratory when assessing audio quality in certain contexts. This example indicates that it is possible to adapt and employ methods used inotherfields, in this casepsychology, but also in the humanities and social sciences. Since MED also focus on usability, anddialogue,methods related to thosediscussed byBlauert and Jekosch are of interest. The following sections initially present the implications ofusingaqualitativeapproachandsomequalitativepoints of departure, followedbyhowdata collection andanalysis methods may be applied in the MED context, given the qualitative approach. 3.1. Implications of using a qualitative approach When researchers use qualitative methods, they need to consider the possible limitations of such an approach. In qualitative research the question of validity focuses on the employment of the “reduction” process (analysis) that leads toa result andnotwhether thedatacanbe replicated, a commonview issue in thefieldof audioquality research. The qualitative research focuses on howclearly the analy- sis is conducted and described by the researcher and how well the descriptions of the data relates to the collected data. In hermeneutics for an example the hermeneutic cir- cle comes into play, see [27, 28] and in phenomenology, bracketing, phenomenological reduction and horizontali- sation are the major methods employed, see [28, 29]. All thesemethodsaredesigned tounderstand thedata andcre- ating a description of the collected data. One implication of qualitative research is that two re- searchers do not produce exactly the same results when faced with the same task. This poses no problem, how- ever. That is, competent and skilful researchers produce results easily recognized by other researchers. Each ex- ploration/study should bring a different perspective on the phenomenon under study and each perspective creates a more comprehensive picture of the phenomenon [27]. 3.2. Qualitative points of departure Empirical phenomenology and hermeneutics, approaches frequently used in psychology [27, 28], are suitable points of departure for evaluating audio and video quality in a distance education context such as MED conducted via the Internet. Empirical phenomenology has the following goal: To understand the general psychologicalmeaning of some particular human way of being-in-a-situation [. . .] through a number of descriptions of this way of being-in-a-situation from people who have lived through and experience themselves as so involved [27, p.40]. Moustakas, agreeingwithTesch[27], states thatonecan determine the underlying structures of an experience by interpreting the description of a situation where the expe- rience occurs [28]. As for hermeneutics, Moustakas sum- marizes it thisway: [R]eading a text1 so that the intention and meaning behind appearances are fully understood [28, p.9]. ThisviewagreeswithTesch’sexplanationofhermeneu- tics [27]. Hermeneutics looks at human experience by studying the lifeworld2 in order to form a whole, includ- ing a description of experience. Interpretation of an expe- rience shows what is hidden behind a phenomenon [28]. This understanding means that an empirical phenomeno- logicalandhermeneuticdeparturemayhelpevaluateaudio quality, video quality, and their interaction in order to ex- tract, interpret, and describe the experience of the persons involved in aMEDsituation. Other researchers havealreadyappliedgrounded theory in the form of semi-structured interviews to evaluate the perception of audio-visual quality [20, 21] (as discussed 1The term “text” in a hermeneutic sense is very wide and incorporates human action, interview transcripts, and texts [27]. 2Lifeworld is a translation from German, Lebenswelt: ”All the imme- diate experiences, activities, and contacts that make up the world of an individual or corporate life” (New Oxford American Dictionary). 156 Nyberg, Berg: Evaluation of perceived qualities ACTA ACUSTICA UNITED WITH ACUSTICA Vol. 100 (2014) Table I.Top:Questionnaire 1.Bottom:Questionnaire 2:Questions inbold are “yes” and“no”questions and the subject couldonlygive a “yes” or “no” answer. The remaining questions are presented as open-ended questions in the questionnaire. TheX indicates towhich category of persons the questions are directed. 1 What are the positive aspect/aspects of distantmusical learning, based on this occasion? 2 What are the negative aspect/aspects of distantmusical learning, based on this occasion? 3 Explain the sound quality in this carried out lesson. Nr Teacher Student Both Question 1 X Did you change your teachingmethods on this occasion? 2 X Inwhatway did it change? 3 X Inwhatwaywere you able to interact, by speaking and singing/playing,with the teacher/student? 4 X Can you perceive the teacher’s instructions clearly? 5 X If “Yes”, describe in your ownwords the perceived sound. 6 X If “No”, describe in your ownwords the perceived sound. 7 X Can you perceive the music examples from the teacher clearly? 8 X If “Yes”, describe in your ownwords the perceived sound. 9 X If “No”, describe in your ownwords the perceived sound. 10 X Inwhatway did the communication between you and the teacher/studentwork? 11 X What could further enhance the lesson? 12 X Are there any positive aspectswith the use of distance learning? 13 X Are there any negative aspectswith the use of distance learning? 14 X What could be improvedwith the technology based on this lesson, according to your opinion? 15 X What could be improved in the teaching, based on this lesson, according to your opinion? in section2).Theaim in these caseswas to collect andun- derstand the unknownunderlying characteristics ofmulti- modal qualitywhen evaluated quantitatively, Grounded theory, an approach that is often found in so- ciology, generates a theory fromdata [30, 31].When con- ductinggrounded theory research, a researcher shouldcol- lect data on a particular subject/topic before developing theories and hypotheses about it rather than developing theories and hypotheses before collecting data [27]. Grounded theory has similarities to the proposed ap- proachabove,aswewant thedata togive information tous about a phenomenon (in this case audio and video quality and interactionbetween teacherandstudents), butourpur- pose is not to generate a theory or a hypothesis about the phenomenon; ratherour focus is todescribe theperception ofwhat is experienced in this specific context. 3.3. Collection of data The twodata collectionmethods used, questionnaires and interviews, can be applied post the MED situation, thus they do not influence student-teacher interaction. Both these methods are often found in qualitative research. In- terviews are used in phenomenology research and to some extent inhermeneutics [27,28,32].Compared tonumeric- baseddata collectionmethods, bothquestionnaires and in- terviews can provide detailed descriptions of how audio and video quality are perceived, separately and together. Similarly, questionnaires have been used in earlier studies [33, 34, 35, 36] with good results. Open-ended question- naires can gather many responses from the subjects and give anoverviewof thefield under study.Of course, ques- tionnaires can contain questions where the responses are quantitative, e.g., by using scales, lines, checkboxes, etc., but such approaches are not considered here as they rep- resent a methodology outside the scope of this paper. In- terviews provide insight into how subjects experience the learning environment–audio, video, and student-teacher interaction. This approach also allows more flexibility in data collection because follow-up questions can be asked [32],which is not possiblewith questionnaires.Collecting descriptive data from interviews could be beneficial when the researchfield is relatively unexplored [20]. The data collection is designed in the following way: The design can be seen as a funnel, with broad questions at first (Questionnaire 1), secondlymorenarrowquestions (Questionnaire 2) still remaining open with but with cer- tain focus points and thirdly interviews with participants in order to investigatingwhat particular users perceive. In the following sections it is shown how these data collec- tionmethods are applied in aMEDcontext. 3.3.1. Using questionnaires Two sets of questionnaires were distributed during sev- eral MED sessions. Their design was partially influenced by previous studies on distance learning [4, 5]. As parts of the questionnaires included broad and/or open-ended questions, no pre-tests were considered. Questionnaire 1 is a brief questionnaire containing a set of open-ended questions regarding positive and negative aspects of the MED sessions as well as audio and video quality aspects. Based on the answers collected from the first question- naire, seeTable I, a secondmore elaborated questionnaire was designed (Questionnaire 2), see Table I. In this re- spect, Questionnaire 1 partially serves the function of a pilot test. Questionnaire 2 had more specific open-ended 157 ACTA ACUSTICA UNITED WITH ACUSTICA Nyberg, Berg: Evaluation of perceived qualities Vol. 100 (2014) questions and twoyes and no questions that led to follow- up questions (“if Yes, please describe . . .”, “if No, please describe . . .”).Questions1and2 in thequestionnairewere designed to capture the teachers’ perspective and ques- tions 4–9were specifically designed for capturing the stu- dents’perspective.A totalof15questionswereused in the questionnaire. This new set of questions was designed to encourage the participants to reflect on whether they per- ceived the audio quality and video quality as good or bad and why they perceived the qualities in these particular ways. The questions were designed to capture a compre- hensible view of the participant’s perception of the over- all quality of the MED system and the participant’s ex- perience. Data from Questionnaire 2 were systematically analysed (described below). 3.3.2. Using interviews As a next step, personal semi-structured interviews with the participants in theMEDsituation (students and teach- ers) were used to further shed light on the results from the questionnaire and obtain an even more individual de- scription of the experience of perceived quality. An inter- viewguidecombiningbothopen-endedandcircumscribed questionswasused toguideboth the interviewerand inter- viewee to obtain the descriptions of the users’ experience [28]; see Table II for the questions used in the interview guide.Thequestion asked from the interviewguide varied depending onwhether the intervieweewere a teacher or a student. 3.4. Analysis of the collected data A phenomenological and hermeneutical approach is used as the point of departure for the analysis. The analysis focuses on describing the user’s perception of audio and video quality as well as the interaction between teachers and students. This sectionpresents the data analysismeth- ods. 3.4.1. Using questionnaire analysis An ad-hoc analysis is used on the data and includes theme analysis and meaning categorization analysis [32, 37]. In the themeanalysis, themesaresearchedfor in thecollected data [37].The themeanalysis is accomplishedbyfirst list- ingall thecollectedanswersundereachquestionand read- ing all the answers for similarities and common answers [23].This part of the analysis is analysedhermeneutically. The meaning categorization analysis is an analysis that is inspired byVerbal ProtocolAnalysis [38]. Each answer is codedandcounted according to its identifiedproperties. The verbal protocol analysis requires an algorithm or de- scription on how to handle each verbal unit [38, 39]. Berg [14, 39] categorised the data into descriptive features and attitudinal features. Each category was then divided into two subsets. For descriptive features, the units were di- vided intounimodal (only audiomodality) andpolymodal (other sensory modalities). The attitudinal features were divided in the samewaybut into emotional/evaluative atti- tudes or attitudes related to naturalness. The current anal- ysis, however, differs from Berg’s approach: each answer Table II. Questions used in the interview guide. Translated from Swedish. • Howwas the lesson/master class? • On the topic of user freedom – Did you feel limited in your practice? * Is it the technology? * Is it the distance between you and the teacher/ stu- dent? * Is it the lack of presence by the teacher/student? * Is it the communication that poses problems? – If you feel free in your practice * What is it thatmakes you feel free in your practice? * Canyoudo the things that youwant to do? * Towhat extent does the feeling of freedomexist? • Could you complete the master class without being af- fected by the technology/systemused? • Did you perceive the technology as a hindrance or as a tool? • What worked and what didn’t work during the master class? – Technology * Was it good? * Was it bad? * What could be improved? Describe them – Pedagogics * Was it good? * Was it bad? * What could be improved? Describe them • Howdid you perceive the sound quality? – Canyou compare it to a known format/media? • Howdid you perceive the sound and video quality? • If you exclude the video, how did you perceive the sound quality? • If you exclude the sound, how did you perceive the video quality? • Did you perceive any “delay/latency” between the sound and video? – If yes,which camefirst according to you? – Did this delay pose any problem for you? to the open-endedquestions for both its descriptive and its attitudinal featureswas analysed as each answermay con- tain both. Both the descriptive and attitudinal features of each answer were interpreted based on the context (each question) from which they were taken. The sorting pro- cesses were conducted in the following way: Sorting 1– eachanswerwassortedafter its interpreteddescriptive fea- tureusing the labels below; e.g., if the answerwere related to audio quality, it was given a sound quality-related label (Sqr); and Sorting 2–each answer was sorted after its in- terpreted attitudes, i.e., positive, negative, both, or blank. The following labelswere usedwhen sorting each answer into its respective topic and attitude: 158 Nyberg, Berg: Evaluation of perceived qualities ACTA ACUSTICA UNITED WITH ACUSTICA Vol. 100 (2014) Sorting 1. Descriptive features: Sound quality related (Sqr), video quality related (Vqr), sound and video qual- ity related (Sqr&Vqr), communication related (Com) (“communication” here refers to speaking and interaction among the participants), teaching related (Tch), technol- ogy related (Tec), teaching and communication related (Tch&Com), and diverse statements (Div). Sorting 2. Attitudinal features: positive (+), negative (-), positive and negative (+ &-), and attitudes that contained no positive or negative statements (blank). Consequently, sorting 1 in the meaning categorization analysis provides an overview of the number of answers on each perceived quality. The second sorting (Sorting 2) provides anoverviewof the answers’ attitudes (sorting2): positive, negative, both, or blank (general statements). 3.4.2. Using interview analysis The interviewdatawere analysed using an approach com- monly used in phenomenology studies [27, 32], mean- ing condensation [32]. One or several persons can do this type of analysis. When looking at the data from a phe- nomenological point of view, the interviewer and/or re- searcher must set aside his/her pre-understanding of the phenomenon to obtain objectively rich and clear descrip- tions of the phenomenon under study [32]. This is re- ferred to as “bracketing”. The bracketing started during thedesignof the interviewquestionsby formulatingopen- endedquestions to facilitate the subjects’owndescriptions of their experience and continued throughout the analysis process. Meaning condensation concentrates the uttered mean- ing intomore essentialmeanings andcontainsfivegeneral steps: Step 1–reading the interview to obtain an overview of its content, establishing a sense of the whole; Step 2– creating units of meaning (the answers) as the interview subject expresses them; Step 3–creating themes that dom- inate the units ofmeaning; Step 4–asking questions to the units ofmeaning based on the research purpose; and Step 5–creating a summary of the interviews’ central themes andpresenting theminonedescriptive statementper inter- view [27, 32]. The steps of the interviewanalysis used the original method as a baseline for the analysis. The steps used in our analysis are listed below: 1. Transcribing the interview (transcriptionmethods used were structured in colloquial language [32]); 2. Creating units ofmeaning of each transcribed answer; 3. Creating themes; 4. Sorting the units’ answers into corresponding themes; and 5. Summarizing each theme in the text for each interview. All stepsof the evaluationmethodarepresented in ablock diagram in Figure 1. Each step (block) is followed by its result. 4. Evaluation of perceived qualities of a master class The qualitative approach presented in section 3 was de- veloped to evaluate perceived audio quality during real Figure 1. The steps and the results for each step used in the eval- uationmethod. MED situations while considering the perceived video quality and interaction between the teacher and student. The MED situations were real master classes in classi- cal music taught over distance and connected via video conferencing systemsandan IPnetwork (Public Internet). That is, the teacher was at one location and the student at another. This section presents the results from the qualita- tive approach. The study was conducted under non-experimental con- ditions.Measurementsof latencybetween the locationand audio and video were not possible due to inaccessibility tomeasurement equipment.Table IIIprovides information aboutwhat equipmentwas used at each location. 4.1. Participants The master classes were conducted in Oulu (Finland), Helsinki (Finland), Piteå (Sweden), Olos (Finland), and Rovaniemi (Finland) on several occasions during the au- tumn and winter of 2009 and spring and autumn of 2010. The instruments/ensembles used in these master classes wereviolin,Frenchhorn, cello, and stringquartet. Singing was also a part of some classes. The participants in the study were college/ conserva- tory/ universitymusic students and teacherswhowere re- cruited by means of invitations trough their home institu- tions. Hence, they may be regarded as experienced per- formers compared to the general public as well as to mu- sic students at preceding levels of the education system. Their participation was voluntary. In the beginning of the study (Questionnaire1and2, seebelow), teachersandstu- dents directly involved in actively singing or playing their instruments as well as students observing the players par- 159 ACTA ACUSTICA UNITED WITH ACUSTICA Nyberg, Berg: Evaluation of perceived qualities Vol. 100 (2014) Table III. Tablewith the equipment for each location. Location Equipment All locations TandbergMXPEdge 95 video confe- rencing systems, 50-52 inchLCD television screens Piteå 2xNeumannKM184microphones (occasionally aMicrotechGefell UMT70smicrophone) 2xGenelec 1030A speakers Helsinki 2xNeumannTLM-103microphones 2xGenelec 8030A speakers Olos 2xNeumannTLM-103microphones 2xGenelec 8030A speakers Oulu 2xNeumannTLM-103microphones 2xGenelec 8030A speakers Rovaniemi TVspeakers 1xClockaudio limitedC600microphone ticipated. In a later step (Interviews), only active teachers and students partook. 4.2. Procedure The stepsused for evaluationof themaster classeswere as follows: 1. Questionnaire 1 (Brief); 2. Questionnaire 2 (Extended); 3. Ad-hoc analysis of the questionnaires; 4. Interviews; and 5. Analysis of the interviews (meaning condensation). Both questionnaires were completed and the interviews were conducted after each lesson so as not to interrupt theperformanceof the teachers and students.The timebe- tween the closing of the master class and the distribution of questionnaires and/or conducting the interviewswas no longer than one hour. During the first master class, Questionnaire 1 (three open-ended questions) was distributed to obtain a general overviewof the participants’ experiences: their evaluation of the overall sound quality and their evaluation of their overall distance learning experience. A total of six ques- tionnaireswere collected. From this information, a second questionnairewasdesignedanddistributed tocollectmore detailed descriptions from the participants. The question- naires were distributed to all participants (teachers, stu- dents, and observers participating in the master classes). A total of 22 questionnaires were collected over a period of three master classes. The participants were allowed to complete the questionnaires at home for later submission. Hence, the time used by the participants for completion of the forms differed from less than one hour up to sev- eral days. The use of questionnaires is outlined in section 3.3.1. After thedistributionof thequestionnaires, a total of six interviews were conducted by the first author: two inter- viewswith students participating in themaster classes and thismasterclass studyandfour interviewswith twomaster class teachers. Both students had previous experience of master classes and some experience with virtual distance communicationbuthadnoperviousexperiencewithMED situations before this project. Both teachers had some ex- perience of MED situations previous to this project. The interview followed theprinciples outlined in section3.3.2. One of the teachers was interviewed three of the four times in order to collect additional information from each session.The responseswere recordedonanaudio recorder and later transcribed. The second teacherwas interviewed via e-mail and several follow-up e-mails were sent to en- courage further reflection on topics deemed interesting by the interviewer. This e-mail correspondence can be com- parable to a regular interviewbecauseof thenatureof ask- ing follow-upquestions to inquire interesting leads aswell as asking for clarification on certain answers. 4.3. Analysis Thequestionnairedataand interviewdatawasanalysedby using the traditional data analysismethods that goes along with phenomenology and hermeneutics, presented in sec- tion 3.4. For simplicity, one person did the coding and meaning condensation, as the hermeneutical point of ori- gindoesnotpreclude theuseofasingle interpreter (coder) provided that this person is aware of his/her possible pre- conceptions and prejudices.Although, later in the process an investigator triangulationhas alsobeenapplied, i.e. dif- ferent evaluators to review the findings in order to reduce potential bias [40]. 4.4. Results from the analysis 4.4.1. Questionnaire analysis (ad-hoc analysis) The first part of the analysis, finding major trends, gave fourmajor themes/trends.Quotations are translated toEn- glish and originate from Swedish, English and Finnish. The sound quality-related trend contained statements on theperceptionof thesound.Several answers stated that the instruction and music examples played from the teacher/ studentwere perceived as good and clear: “Goodnatural sound”. “Sound was very clear; there were no problems un- derstandingwhat the teacher said”. The sound and video quality-related trend included the perception of an asynchrony between audio and video: “The delay could be shorter”. “The sound and the picture should be in the same time”. The teaching-related trend showed that the participants thought that the distance master classes offered more op- portunities for participating inmaster classes and that they offered the opportunity to play in front of different teach- ers. This trend alsomeant travelwas not required: “The teachingwas excellent”. “Youdon’t have to travel far away to get lessons”. Thecommunication-related trendcontained theperception of having a hard time communicating with the student or the teacher and playing togetherwith the teacher: 160 Nyberg, Berg: Evaluation of perceived qualities ACTA ACUSTICA UNITED WITH ACUSTICA Vol. 100 (2014) “It ismore difficult to communicatewith the teacher”. “There shouldbeaclear signal to the students so that theywould knowwhen to stop playing”. The second part of the ad-hoc analysis – meaning cate- gorisation analysis (Table IV) – shows the total number of responses of the categories. The purpose of the cat- egorisation is to outline the major trends/distribution of collected questionnaire answers. Table IV also shows the distribution of the received answers: the results show that the most frequently occurring responses are positive and are related to teaching. The second largest quantity of re- sponses is positive and related to sound quality. The third largest quantity of responses does not contain any inter- pretedattitudesofpositive,negative,both,orgeneral state- ments (blank). These qualities are related to sound and video quality. The data also show that positive attitudes are most common. The second most common attitude is general statements without any positive/negative or both attitudes in theanswers.The thirdmostcommonattitude is negative and the fourthmost common contains both posi- tiveandnegativeattitudes.Thisanalysisdoesnotallowfor more detailed analysis more then showing rough chunks ofdata andmajor trends.As shownbyRaimbault [41] one runs the risk to overlook certain patterns by looking at the data only from one perspective. Since this research is ex- ploratory, the choice of only looking at the major trends is a conscious decision made by the authors in order to discovermajor tendencies in the teachers and students re- sponses. 4.4.2. Interview analysis (meaning condensation) The descriptive texts collected from each interview (step 5, Interview analysis) are summarized in this section into corresponding themes, i.e., making one descriptive text/ description for each themebasedonall six interviews, one person three times and three other persons, one time each. This strategy is done tomaintain the richness of the infor- mation in the collected data. A total of three themes were created. Perceived audio and video quality The teachers perceive that the video quality is good and works for distancemaster classes. The teachers could also imagine how the sound of each instrument sounded live, based on experience, even though the instruments did not sound natural during the master class. One teacher per- ceived the sound quality as “metallic” and “boring”, but with significant direct sound, no room sound, and good dynamics.The same teacher compared the sound to ahigh fidelityMP3.One teacherperceivedadelaybetweenaudio and video, with the audio leading. With respect to sound quality, one teacher found it hard to distinguish between the system’s limitations and the student’s limitations al- though the teacher could distinguish this difference in a later master class. One teacher could easily see the stu- dent’s playing technique, but could not evaluate it because the soundwas difficult to hear. Table IV. The number of responses in each category from the meaning categorisation analysis from the extensive question- naire survey.Theboldnumbers show the categories attaining the largest number of statements in total and for each attitude. Descriptive Attitudinal features features + - +&- Blank Total Tch 35 4 3 1 43 Sqr 17 4 8 7 36 Sqr&Vqr 3 4 5 14 26 Diverse 6 3 1 8 18 Com 4 5 1 7 17 Vqr 2 1 0 6 9 Tec 0 5 0 1 6 Tch&Com 2 0 0 1 3 Total 69 26 18 45 Both students compared the video quality as equal to or better than aYouTube clip.One student couldonly see the main features but not the contours or the proportions of the image.One student perceived the audio quality as “far away, distant, and a littlemuddy” and compared the audio to a YouTube clip. In addition, the same student found it hard to understand the teacher’s voice. One student com- pared the sound quality to aMP3 coded sound, but worse than a movie although the student could distinguish be- tween a normal spoken voice and a softly spoken voice. One student perceived the delay between audio and video as strange. Perceived problems and possibilities The delay between audio and video and the locations are perceived as a problem thatmakes it hard to communicate with the teacher/student. Small details in themusic disap- peared. Not knowing what is sent to the other locations, both related to video and sound quality, is perceived as a problem. During a distance master class, it is also hard to perceive the playing technique used by the students and how the students control their muscles. Distance learning is also problematic formusic classes because it is difficult for teachers to evaluate their students’ playing techniques andmuscle control. Several topics –controlling tempo, intonation, articula- tion, and phrasing–were adequately dealt with during the classes. According to one teacher, if the delays between audio and video were short and the teachers were aware of it, the delay could be less problematic. Both teachers perceived the technology in an overall positive light. One student and one teacher state that there are indirect bene- fits of distancemaster classes since they providemeans to connectwith teachers/studentswithout traveling. Perceived differences and similarities between regular and distance master classes The perceived similarities were meeting with the student personally and discussions with the student. In addition, fromapedagogicpointofview, thedistant learningmaster classwas similar toa regularmaster class.Twodifferences 161 ACTA ACUSTICA UNITED WITH ACUSTICA Nyberg, Berg: Evaluation of perceived qualities Vol. 100 (2014) were identified: correcting playing techniquewas difficult and verbal explanations rather than hands-on demonstra- tionswere required toexplainnewpositions to the student, a situation that requiredmore time. In addition, creating a relationship with a teacher/student during distance mas- ter classes was more difficult than during a regular mas- ter class. One teacher also stated that a physical/personal contact establishedbefore starting longperiodsofdistance lessonswould be helpful. 5. Discussion 5.1. The results Seen together, the ad-hoc analysis and the meaning con- densation analysis provided an overview of what type of trends/themes exist in the data and how the perceived au- dioquality relates to theperceivedvideoquality and to the interaction between the users. The results from this study can be seen in the following points: • The subjects could complete themaster classeswithout major difficulties. • The audio and video quality was not optimal but suffi- cient. • There was a lack of synchrony between audio and video. • Therewas a perceived delay between locations. • Teachers had to adapt their teaching to accommodate for the systemused. • MEDis perceived as cost-affective. • Participants can access more teaching and learning ex- perienceswith less travelling. These results confirm previous research findings on sev- eral counts.Masum et al. found that teachers and students found the tools (system used) comfortable [4]. This find- ing coincides with the authors’ results; the subjects could complete themaster classeswithout anymajor difficulties. The authors’ results also indicate that the perceived au- dio and video quality in the system is not optimal but is sufficient for this type of music education. There are also problemswith lackof synchronybetweenvideoandaudio and latency between the locations, but the subjects’ state- ments indicate that they can work around these problems andmanageaMEDsituationsuccessfully.Woszczyket al. also report similar results [7]. The results also align with previous research when it comes to teaching. To conduct successful teaching with the use of video conferencing systems, the teacher needs to adapt the content to handle the pedagogical situation [6]. In addition, it is cost-effective to bring teachers and teaching experiences to a large population of students [6]. Clearly, videoconference systems also allow teachers to trainand teach inplacesother than theirhome location [4]. These previous results all coincide with the results from the current study. The implications of these findings will be discussed the section 5.2. The subjects’ initial attitude towards using such a sys- temcould be a bias that affected the results. That is, a stu- dentmay have entered the studywith a preconceived idea about distance learning. Such preconceived attitudes need to be considered when dealing with subjective responses. Before the study, based on their experience of soundqual- ity, the authors expected the subjects to be negative about the sound quality; however, the results did not indicate this. Another possible bias, which may be connected to the positive attitudes collected, is that all the participants volunteered to participate in the study, showing they had an initial interest in MED; that is, the participants were self-selected on some level. 5.2. The methodology Using qualitative methods including analysis shows a po- tential for arriving at a set of data that can be usablewhen evaluating several perceived qualities in one system. This method, with post questionnaires and post interviews, al- lows the participants to complete their task without in- terruptions and encourages the subjects to use their own words to describe their experience. This approach helps create a broad picture of what is happening in the study. Thismethod also allows for a simultaneous description of what is perceived and what is affected even if there are more parameters affecting the perceived total quality than just theaudioquality.Comparing thismethod toother sub- jective assessment methods of audio quality proposed by other studies [10], onecouldnot sayaparticularmethod is better than another method because they collect different setsofdata;however, somesubjectiveassessmentmethods [8, 9, 10, 11, 12, 15] have predetermined verbal descrip- tors and factors that aim at a defined part of the perceived quality. The approach in this paper enables the subjects to reflect on what they have perceived with few restrictions. Thus, the information can shed light onbroader aspects of the perceived quality. This broad approach makes it pos- sible to discover unexpected and possibly important fac- tors, related either to audio quality or to other qualities that affect how a specific situation or implementation is perceived. Hence, factors outside the audio domain may also be considered. On the topic of analysis, the data presented in Table 1 do not say anything in detail about the content of the re- sponses; the data only show the distribution of answers, which demonstrates that simple quantitative observations can also bemade using thismethod.A difficultywith this categorisation/numeric summaryaswithall categorization methods is that one can choose other categories and thus get a different distribution of the answers. In this study, the blank category refers to that no attitudinal response could be discovered. Consequently, blank responses may indicate that no strong emotionswere evoked in thosepar- ticular cases and possibly that the responses were more of a descriptive character. The lack of further detailed in- formation, e.g. in subcategories, is a result of the chosen approach [41]. 162 Nyberg, Berg: Evaluation of perceived qualities ACTA ACUSTICA UNITED WITH ACUSTICA Vol. 100 (2014) Byusingopen-endedquestions in aquestionnaire, a va- riety of answers can be elicited. This strategy can be an advantage if a participant gives an answer that sheds light on a new area that the researcher has overlooked, or it can be fruitless if the participants “don’t know” or do not evenanswer thequestion.Open-endedquestionsuseddur- ing an interview can encourage the interviewee to answer freely while still allowing the interviewer to guide the in- terviewee into interesting topics if they arise. In the lat- ter situation, the interviewer needs to be very responsive to follow-up possibilities and needs to understandwhat to ignore. Such a strategy, of course, is a source of bias that has tobeconsideredcarefully.Researchersusingquestion- naires and conducting several interviews during an evalu- ation processmust be aware that the data size quickly be- comes large and is hard tomanage and time-consuming to analyse. As stated in section 3.3 the study design can be seen as a funnelmoving fromat set of broad answers collected from the questionnaires 1 and 2 to more detailed answers in the interviews.This designwasused since thequestion- naires worked similarly to a pilot study to facilitate nar- rower questions and topics in the interviews. In the meaning condensation of the interviews, each unit of meaning was categorized into themes. This divi- sion made the answers more clear and provided a better overview and a better understanding of the interviewed subject’s perception. The study includes repeated inter- viewswithone teacher.Thiswillunavoidablycreate learn- ing effects that affect the response. Learning effects was not studied separately as the major trends were in focus this time.However, for future studiesof themethod’schar- acteristics, thismaybe of interest. As all data collection was done after the completion of the event, this collection method relies on the subject’s memory. When answering questions after an event rather than during it, some shift in the recollection of perceived sensations may occur. This shift can be a disadvantage compared toothermoredirectmethodsused for ratingand assessing. Small differences between sessions may not be captured,as theywouldbeharder todetectwhenoneexpe- rience is compared to anotherwith a time gap in between. Further studieswith thesemethodsmight be enhanced by a stimulated recall approach by means of video record- ing the master classes allowing the subjects to watch and comment and thus facilitating the subject to recall what happened during themaster class. Because thestudywasconductedduringarealMEDsit- uation, some other limitations did arise. One limitation of themethodology is that in thisparticularcaseonlyoneper- sonconducted the coding.Aspointedout, thehermeneuti- cal approach does not disqualify a single interpreter from doing the coding, provided that this person is aware of his/her possible preconceptions. Though, in several appli- cationsmore thanonepersonperforms codingof the data. In the current study, the findings were discussed among the authors in order to reduce the potential bias andwhen using a phenomenological or hermeneutical approach for the analysis, the methods used requires the researcher to beawareofhis/herownpreconceptionsandprejudicesand exclude them during the analysis. The author performing the analysis was aware of this possible bias (examples in section5.1) during thedesignofquestionsaswell asunder the analysis of the results. As can be seen in section 4, Table III, the equipment was changed occasionally between different locations and sessions. This can, of course, result in a bias, but the pri- maryfocuswasnot to linkasetofexperiences toparticular equipment, but to evaluate the perceived audio and video quality in anecological valid situationduring livedistance master classes. Another restriction was the unavailability ofmeasuring thedelaybetween the locations andbetween audio andvideo.This couldhave shed some light onwhen delaywas present andwhen the delaywas not present. As indicated, the approach used in the current study yields different information compared to most of the pre- viously used methods that evaluate audio quality. Hence, this method cannot be used interchangeably with exist- ing methods to obtain the same type of data. However, by adding information that is not available from other methods, this approachwill increase the knowledge of the subject’s experience. The results may also be used in an exploratory way as a means of observing what subjects perceive as noticeable, which in its turn can be used to develop evaluation scales used in existing methods. The rich verbal data resulting fromaqualitative approachmay provide a more holistic representation of an audio event, improving our understanding of how the event is expe- rienced. For further research on qualitative research ap- plied on perceived sound and video quality the method and frameworkof introspection (examiningonesowncon- scious thought and feelings [42]) might be fruitfully ap- plied. See [42]. The high correspondence between the results from this study and the studies quoted in section 5.1 can be seen as a successful triangulation and a verification of that the method employed in this study enables the extraction of results thathavepreviouslybeen found in similar contexts. In addition, the method allows for unexpected features to be discovered e.g. the interaction influence on perception of audio and video quality, making the audio quality for an example good enough. Themethod is shown also to be potent for extracting similar data from real-life situations asdataextracted fromexperimental conditions.Altogether this strengthens and encourages future development and use of this type ofmethods. Acknowledgement This work is part of the Vi R Music project and was par- tially financed by the European Union program Interreg IV A Nord. The authors would like to thank all the par- ticipants, students, and teachers who participated in the surveys and interviews. A thanks also goes to colleagues, technicians, and staff members who made this work pos- sible. The authors alsowant to express a special thanks to 163 ACTA ACUSTICA UNITED WITH ACUSTICA Nyberg, Berg: Evaluation of perceived qualities Vol. 100 (2014) Dr.AndersPersson, JonasEkeroot, and JonAllan for their valuable comments on the drafts. References [1] E. Brunswik: Organismic achievement and environmental probability. ThePsychologicalReview50 (1943) 255–272. [2] J. J. Gibson: The ecological approach to the visual percep- tion of pictures. Leonardo 11 (1978) 227–235. [3] C. Guastavino, B. F. G. Katz, J.-D. Polack, D. J. Levitin, D.Dubois:Ecological validityof soundscape reproduction. ActaAcustica unitedwithAcustica 91 (2005) 333–341. [4] H.Masum,M.Brooks, J. Spence:MusicGrid:Acase study in broadband video collaboration. http://firstmonday.org/ ojs/index.php/fm/article/view/1238/1158. [5] B. K. Shepard, G. Howe, T. Snook: Internet2 and musical applications. Proceedings,NationalAssociation of schools ofmusic, The 84th annualmeeting, Seattle, USA,Novem- ber 24, 2008. http://www.briankshepard.com/pdf/2008Proceedings.pdf. [6] A.Greenberg,R.Colbert:Navigating theseaof researchon videoconferencing-based distance education. Wainhouse Research, Polycom, Inc., February 2004. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1. 110.8045&rep=rep1&type=pdf. [7] W. Woszczyk, J. Cooperstock, J. Roston, W. Martens: Shake, rattle, and roll: Getting immersed in multisensory, interactive music via broadband networks. Journal of Au- dioEngineeringSociety 53 (2005) 336–344. [8] F.E.Toole:Subjectivemeasurementsof loudspeaker sound quality and listener performance. Journal of Audio Engi- neeringSociety 33 (1985) 2–32. [9] A. Gabrielsson, B. Lindström: Perceived sound quality of high-fidelity loudspeakers. Journal of Audio Engineering Society 33 (1985) 33–53. [10] S. Bech, N. Zacharov: Perceptual audio evaluation – The- ory, method and application. John Wiley & Sons Ltd., Chichester, England, 2006. [11] BS.1116-1:Methods for the subjective assessment of small impairments in audio systems including multichannel sound systems. ITU-RRecommendations, 1994–1997. [12] BS.1534-1:Methods for the subjective assessment of inter- mediatequality levelof codingsystem. ITU-RRecommen- dations, 2001–2003. [13] G. Lorho: Individual vocabulary profiling of spatial en- hancement systems for stereo headphone reproduction. Presented at the 119th AES convention, New York, New YorkUSA,October 7-10, 2005. [14] J. Berg, F. Rumsey: Identification of quality attributes of spatial audio by repertory grid technique. Journal ofAudio EngineeringSociety 54 (2006) 365–379. [15] BT.500-12: Methodology for the subjective assessment of the quality of television pictures. ITU-R Recommenda- tions, 09/2009. [16] K. Seshadrianthan, R. Soundararajan, A. C. Bovik, L. K. Cormack: Study of subjective and objective quality assess- ment of video. IEEE transactions on image processing 19 (2010). [17] J. Radun, T. Leisti, J. Häkkinen, H. Ojanen, J.-L. Olives, T. Vuori, G. Nyman: Content and quality: Interpretation- based estimation of image quality. ACM Transactions on AppliedPerception 4 (2008)Article 21. [18] G. Nyman et al.: What do users really perceive – probing the subjective image quality. Proc. of SPIE-IS&T Elec- tronic Imaging, SPIEVol 6059, 2006, 605902. [19] D.Strohmeier, S. Jumisko-Pyykkö,K.Kunze:Openprofil- ing of quality:Amixedmethod approach to understanding multimodal quality perception. – In:Advances inMultime- dia, Vol 2010. Hindawi PublishingCorporation, 2010,Ar- ticle ID658980. [20] S. Jumisko-Pyykkö, J. Häkkinen, G. Nyman: Experienced quality factors – qualitative evaluation approach to audio- visual quality. Proceedings of SPIE -The International So- cietyofOpticalEngineering, 6507, 2007, art.No. 65070M. [21] S. Jumisko-Pyykkö,U.Reiter,C.Weigel: Producedquality is not perceived quality –Aqualitative approach to overall audiovisual quality. 3DTVConference,May7-9, 2007. [22] J. G. Beerends, F. E. De Caluwe: The influence of video quality on perceived audio quality and vice versa. Journal ofAudioEngineeringSociety 47 (1999). [23] M. P.Hollier, A.N. Rimell: An experimental investigation intomulti-modal synchronisation sensitivity for perceptual model development. Presented at the 105th AES Conven- tion, SanFrancisco,CA, September 26-29, 1998. [24] S. Bech, V. Hansen, W. Woszczyk: Interaction between audio-visual factors in a home theatre system: Experimen- tal results. Presented at the 99th AES Convention, New York,October 6-9, 1995. [25] J. Häkkinen, V. Alatonen, M. Schrader, G. Nyman, M. Lehtonen, J. Takatalo: Qualitative analysis of mediated communication experience. Quality of Multimedia Ex- perience (QoMEX), Second International workshop, June 2010. [26] J. Blauert, U. Jekosch: A layer model of sound quality. Journal ofAudioEngineering 60 (2012). [27] R. Tesch: Qualitative research: analysis types and soft- ware tools. RoutledgeFalmer, 2 park Square, Milton Park, Abingdon,Oxon,Ox144RN, 1990. [28] C. E. Moustakas: Phenomenological research methods. SAGEpublications, Inc.,ThousandOaks,California, 1994. [29] S. B. Marriam: Qualitative research, a guide to design and implementation. Jossey–Bass, San Francisco, CA, US, 2009. [30] B. G. Glaser, A. L. Strauss: The discovery of grounded theory: Strategies for qualitative research. Aldine Publish- ing Company, 200 Saw Mill River Road, Hawthorne, N.Y. 10532, 1967. [31] A. L. Strauss, J. Corbin: Basics of qualitative research – Techniques and procedures for developing grounded the- ory, second edition. SAGEPublications, 2455TellerRoad, ThousandsOaksCalifornia 91320, 1998. [32] S.Kvale, S.Brinkmann: InterViews–Learning the craft of qualitative research interviewing. Second edition. SAGE Publications, Inc., 2455 Teller Road, ThousandOaks, Cal- ifornia 91320, 2009. [33] X.Fang,S.Chan, J.Brzezinski,C.Nair:Developmentofan instrument to measure enjoyment of computer game play. International Journal of Human-Computer Interaction 26 (2010) 868–886. [34] S. Wolfson, G. Case: The effect of sound and colour on responses to a computer game. Interactingwith computers 13 (2000) 183–192. [35] L. E. Nacke, M. N. Grimshaw, C. A. Lindley: More then a feeling: Measurement of sonic user experience and psy- chophysiology in a first-person shooter game. Interacting with computers 22 (2010) 336–343. 164 http://www.ingentaconnect.com/content/external-references?article=0953-5438(2000)13L.183[aid=10203257] http://www.ingentaconnect.com/content/external-references?article=0953-5438(2000)13L.183[aid=10203257] http://www.ingentaconnect.com/content/external-references?article=1610-1928(2005)91L.333[aid=7576207] http://www.ingentaconnect.com/content/external-references?article=0953-5438(2000)13L.183[aid=10203257] http://www.ingentaconnect.com/content/external-references?article=0953-5438(2000)13L.183[aid=10203257] http://www.ingentaconnect.com/content/external-references?article=1610-1928(2005)91L.333[aid=7576207] Nyberg, Berg: Evaluation of perceived qualities ACTA ACUSTICA UNITED WITH ACUSTICA Vol. 100 (2014) [36] S. H. Hsu, M.-H. Wen, M.-C. Wu: Exploring user expe- riences as predictors of mmorpg addiction. Computers & Education 53 (2009) 990–999. [37] G. Ejlertsson: Enkät i praktiken - en handbook i enkät- metodik (andra upplagan). Studentlitteratur, 2005. ISBN 91-44-03164-5. [38] E. Samoyleko, S. McAdams, V. Nosulenko: Systematic analysis of verbalisation produced in comparing musical timbres. International Journal of Psychology 31 (1996) 255–278. [39] J. Berg: Systematic evaluation of perceived spatial quality in surroundsoundsystems. Publication23-4;Doctoraldis- sertation, Department ofMusic andMedia, Piteå, Sweden, 2002. [40] N.K.Denzin,Y.S.Lincoln: Introduction:Entering thefield of qualitative research. – In: Handbook of qualitative re- search. N. K. Denzin, Y. S. Lincoln (eds.). SAGE, Thou- sandOaks,CA, 1994. [41] M. Raimbault: Qualitative judgements of urban sound- scapes: Questionning questionnaries and semantic scales. ActaAcustica unitedwithAcustica 92 (2006) 929–937. [42] P. Vermersch: Introspection as practice. Journal of Con- sciousness Studies 6 (1999) 17–42. 165 http://www.ingentaconnect.com/content/external-references?article=0020-7594(1996)31L.255[aid=7576257] http://www.ingentaconnect.com/content/external-references?article=0020-7594(1996)31L.255[aid=7576257] http://www.ingentaconnect.com/content/external-references?article=1610-1928(2006)92L.929[aid=7576204] http://www.ingentaconnect.com/content/external-references?article=0020-7594(1996)31L.255[aid=7576257] http://www.ingentaconnect.com/content/external-references?article=0020-7594(1996)31L.255[aid=7576257] http://www.ingentaconnect.com/content/external-references?article=1610-1928(2006)92L.929[aid=7576204]