Submitted 11 May 2016
Accepted 1 August 2017
Published 16 October 2017

Corresponding author
Pariya Kashfi,
pariya.kashfi@chalmers.se,
p@puix.org

Academic editor
James Procter

Additional Information and
Declarations can be found on
page 36

DOI 10.7717/peerj-cs.130

Copyright
2017 Kashfi et al.

Distributed under
Creative Commons CC-BY 4.0

OPEN ACCESS

Integrating User eXperience practices
into software development processes:
implications of the UX characteristics
Pariya Kashfi, Agneta Nilsson and Robert Feldt
Department of Computer Science and Engineering, Chalmers University of Technology and Gothenburg
University, Gothenburg, Sweden

ABSTRACT
User eXperience (UX) is a key factor in the success of software systems. Many software
companies face challenges in their work with UX. Existing research does not analyze
UX practices and challenges in relation to other software quality characteristics or,
in particular, in relation to usability. A better understanding of these challenges can
help researchers and practitioners better address them in the future. In this empirical
study, we have interviewed 17 practitioners with different backgrounds and occupations
from eight software development companies. Their responses are coded, and analyzed
with thematic analysis. We report eight themes of challenges that practitioners face
in their work with UX. While some of these challenges partly overlap with those
reported in existing literature about usability or other software quality characteristics,
the participants of our study either view many of the challenges as unique to UX,
or more severe in the case of UX. Although at a superficial level challenges of UX
and other quality characteristics overlap, we differentiate these challenges at a deeper
level through the five main characteristics of UX: subjective, holistic, dynamic, context-
dependent and worthwhile. In particular, we identified that these characteristics have at
least 20 implications (i.e. additional difficulties) for day-to-day work of practitioners.
We found that 11 of these implications have been previously reported in literature.
However, to the best of our knowledge, the remaining nine implications are unique
to our study. These implications can explain why practitioners perceive the challenges
to be more severe than for other quality characteristics. Most importantly, they can
explain the industry’s lopsided focus on the pragmatic aspect of UX. Our findings
can be useful for researchers in identifying new and industry-relevant research areas
and for practitioners to learn from empirically investigated challenges in UX work,
and base their improvement efforts on such knowledge. Identifying and investigating
the overlaps underlines the importance of these challenges, and can also help finding
research areas not only for enhancing UX work but also software quality in general. It
also makes it easier for practitioners to spot, better understand as well as find mitigation
strategies for UX, through learning from past experiences and developments in the area
of software quality.

Subjects Human-Computer Interaction, Software Engineering
Keywords Usability, Software quality, Quality requirements, User experience, Non-functional
requirements

How to cite this article Kashfi et al. (2017), Integrating User eXperience practices into software development processes: implications of
the UX characteristics. PeerJ Comput. Sci. 3:e130; DOI 10.7717/peerj-cs.130

https://peerj.com
mailto:pariya.kashfi@chalmers.se
mailto:p@puix.org
https://peerj.com/academic-boards/editors/
https://peerj.com/academic-boards/editors/
http://dx.doi.org/10.7717/peerj-cs.130
http://creativecommons.org/licenses/by/4.0/
http://creativecommons.org/licenses/by/4.0/
http://dx.doi.org/10.7717/peerj-cs.130


INTRODUCTION
As the software industry has matured, the demands that society puts on the quality
of software systems have increased. It is no longer enough to focus only on the many
functions that a piece of software should supply. To deliver a system that is consistent and
of high quality there are a large number of characteristics that need to be considered (Chung
et al., 2000). Some, such as testability, are internal or relate to the development process
and mainly concern developers, while others such as performance and usability, are critical
for users (International Organisation for Standardisation (ISO), 2011). At an even broader
level, the actual experience of the end users as they interact with the software needs to be
taken into account.

Recently, this widening scope of software quality characteristics has led to the
introduction and study of the concept of user experience (UX). Even though different
definitions of UX exist, they share the same essence: UX is a user’s holistic perception
of functionality and quality characteristics of a piece of software (Hassenzahl, 2003; Wright &
McCarthy, 2010; Jordan, 2002). In general, UX literature emphasizes that assuring efficiency
and effectiveness during use of the software, i.e high usability, does not guarantee that the
end users will have a positive experience (Hassenzahl, 2010). However, the perception of UX
is generally different in academic and industrial contexts: whereas the former concentrates
on hedonic aspects and emotions, the latter focuses more on functionality and usability
issues (Väänänen-Vainio-Mattila, Roto & Hassenzahl, 2008).

Current UX models (e.g., Hassenzahl, 2003; Wright & McCarthy, 2010) differ in their
view on how various underlying elements and processes contribute to forming the end
user’s overall experience with products and services. One of the well-known UX models
is developed by Hassenzahl (Hassenzahl, 2003). It breaks UX down into pragmatic and
hedonic attributes. Pragmatic attributes concern usability and functionality of software
(e.g., clear, supporting, useful and controllable). Hedonic attributes, on the other hand,
concern communicating identity, provoking memories, and providing stimulation
(e.g., outstanding, impressive, exciting, and interesting). These attributes emphasize
individuals’ psychological well-being. An end user’s perception of these attributes leads to
a judgment about the product’s appeal (e.g., ‘‘It is good/bad’’), emotional consequences
(e.g., pleasure, satisfaction) and behavioral consequences (e.g., increased time spent with
the product). While pragmatic attributes concern achieving do-goals, hedonic attributes
concern satisfying be-goals (Hassenzahl, 2003). Do-goals are the concrete outcome that the
end user wishes to achieve whereas be-goals rest in essential human needs. To provide a
better understanding of UX, Hassenzahl (Hassenzahl, 2010) emphasizes five characteristics
of UX: subjective, holistic, dynamic, context-dependent and worthwhile.

All software systems deliver some UX, positive or not, whether the UX has explicitly
been taken into account during development. Research has shown that certain practices
can increase the likelihood of delivering a desirable UX (Hassenzahl, 2010) (hereafter, UX
practices). However, simply applying these practices in isolation is not enough (Abrahão
et al., 2010; Ferreira, Sharp & Robinson, 2012; Ovad & Larsen, 2015). Like methods and
practices used to support other software quality characteristics (Chung et al., 2000), they
need to be integrated into development processes and considered throughout projects.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 2/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Nevertheless, UX is often neglected in software projects and the UX state-of-practice
is immature (Abrahão et al., 2010; Isomursu et al., 2012; Law & Abrahão, 2014). In order
to improve the state-of-practice, we must first understand and address challenges that
practitioners face in their everyday work with UX (hereafter UX challenges). To achieve
this, the distinctions and interrelations between UX, usability and other software quality
characteristics must first be properly analyzed. A number of studies claim to have studied
UX challenges, but their results need to be interpreted with caution because they treat UX
and usability as interchangeable (e.g., Cajander, Lárusdóttir & Gulliksen, 2013; Ardito et al.,
2014; Lanzilotti et al., 2015), or examine UX in isolation so its relation to usability and other
software quality characteristics is not recognized (e.g., Isomursu et al., 2012; Vermeeren et
al., 2010).

Our study aims to complement the current body of knowledge by investigating UX
challenges while explicitly differentiating UX and usability. In doing so, we discuss these
challenges in relation to the unique characteristics of UX. We also contribute to the current
literature by providing an explanation for the industry’s lopsided focus on the pragmatic
aspect of UX. Also, in our analysis, we examine how the handling of UX compares to the
handling of general software quality characteristics, especially usability. Here, we report
our findings and answer the following research questions: what challenges do practitioners
face when integrating UX practices into software development processes and organizations?,
and how do UX challenges relate to challenges of handling software quality characteristics,
in particular usability? Our findings can be useful for researchers in identifying new
and industry-relevant research areas, and for practitioners in learning from empirically
investigated UX challenges, and basing their improvement efforts on such knowledge.

The structure of this paper is as follows: The second section provides background on
the topic and summarizes the related literature. The third section describes our research
methodology and presents the different research sites. The fourth section presents the
results from our study: the identified themes of challenges. The fifth section discusses our
findings and puts them in relation to the current literature. The last section concludes our
study and suggests future research.

BACKGROUND AND RELATED WORK
This section elaborates on the characteristics of UX and their implications for UX work.
Understanding these characteristics is important when analyzing and discussing our
findings on UX challenges because it provides a basis for comparing and contrasting UX
with usability and other software quality characteristics. This section also summarizes
related empirical studies on software quality characteristics, in particular, usability and UX.

UX is dominated by subjective aspects of human perception (Wright & McCarthy,
2010; Hassenzahl, 2010). For instance, one user may perceive particular software features
as simple, novel, and admirable while another may perceive them as complicated and
old. Other software quality characteristics can be treated more objectively, at least in
theory (Kashfi et al., 2016). The term subjective has also been used in requirements literature
to highlight that since quality characteristics are hard to measure, practitioners tend to
judge them based on their personal opinion (Chung et al., 2000; Paech & Kerlow, 2004).

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 3/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Here, we use this term to refer to its meaning in the context of UX. Currently, the most
efficient and feasible approach to measure perception and emotion of users is to directly
gather their opinions, and let them express themselves (Law, Van Schaik & Roto, 2014).
This is often performed through questionnaires or scales (e.g., AttrakDiff, Self-Assessment
Manikin, Affect Gird Zimmermann, 2008). However, to gain reliable results, practitioners
need to gather responses from statistically significant number of heterogeneous users (Law,
Van Schaik & Roto, 2014).

UX is also holistic and not totally reducible to its complexly intertwined
underlying elements (Hassenzahl, 2010). UX emerges from underlying functionality
and quality characteristics of software, and the user’s perception of them in a given
situation (Hassenzahl, 2003). The holistic nature of UX resembles the cross-cutting nature
of other quality characteristics. In both cases, more than one functionality is affected by
the related requirements. However, in the case of UX, this interrelation is more complex,
especially considering that UX is dynamic (aka temporal) and thus a user’s perception of
the UX underlying elements changes over time (Hassenzahl & Tractinsky, 2006).

Although practitioners may manipulate UX of a product through its underlying elements,
they cannot guarantee a certain holistic UX (Wright & McCarthy, 2010; Hassenzahl, 2010).
Practitioners can only to some extent increase the likelihood of delivering a certain
UX (Hassenzahl & Tractinsky, 2006). Consequently, there is no consensus in the community
on how UX should be designed or measured and whether the overall UX of a piece of
software can be predicted by merely manipulating and measuring these elements (Law,
Van Schaik & Roto, 2014). This is explained further below.

UX is dynamic and emerges and changes over time (Hassenzahl, 2010). For example,
over time, the user may find a novel feature as old, or a complex feature as simple. Hence,
in designing and evaluating UX, practitioners should pay certain attention to different
episodes of experience (Hassenzahl, 2010); namely expected experience (before usage),
momentary experience (during usage), remembered experience (shortly after usage) and
accumulated experience (over longer period of use) (Hassenzahl, 2010). Practitioners need
to decide which episodes are more important than others for the software being developed
and why; for instance, for an e-marketing website first impression is more important than
it is for a work application. This knowledge can then help them suggest more suitable
design solutions.

UX research uses the term context-dependent (aka situated) to emphasize that any
experience is unique, unrepeatable, and situated in its context (Wright & McCarthy, 2010).
Nevertheless, experiences can be categorised because their essence is the same, i.e., they
connect to essential human needs or be-goals (Hassenzahl, 2010). Implications of being
context-dependent largely overlap the implications we discussed for the holistic nature
of UX.

UX is worthwhile (aka positive), meaning that it focuses on value and creating desirable
experiences than only preventing negative ones, i.e., the focus of usability (Hassenzahl,
2010). While usability concerns removing problem, frustration, and stress (i.e., negative),
UX is based on the idea that removal of dissatisfaction does not necessarily lead
to satisfaction and pleasure. Through being holistic, UX addresses both satisfiers

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 4/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


(e.g., fulfilled needs, emerging emotions) and dissatisfiers (e.g., usability or technical
problems) (Hassenzahl, 2010). Therefore to a large extent, the implications of being
worthwhile overlap the implications of its holistic nature.

Software organizations often face various challenges when dealing with quality
characteristics. Berntsson Svensson et al. (2012) found that if practitioners lack
understanding and knowledge about software quality characteristics, they tend to
undervalue, and ignore these characteristics during development. Berntsson Svensson et
al. (2012) also report that practitioners are more likely to dismiss those characteristics
that are considered less important, e.g., usability is more dismissed than performance.
Karlsson et al. (2007) show that functional issues are often perceived to be more important
than quality issues. Based on their findings, practitioners find it difficult to deal with the
dependencies between quality requirements and the functionality or other characteristics
of the system.

Limited knowledge and awareness has also been reported as one of the challenges
to usability work (Rosenbaum, Rohn & Humburg, 2000). Bak et al. (2008) report that
developers’ minds are set more on the programming aspect of the product than its
usability, and they often misunderstand the term ‘‘usability evaluation’’ and often relate
it to functionality. Gulliksen et al. (2004) report that limited awareness in different levels
of organizations, in particular at the management level, can lead to down-prioritizing
usability. They therefore suggest increasing knowledge and awareness about usability
among different stakeholders and in various levels of organizations, in particular among
management.

Other studies show that, in general, practitioners find it more difficult to perform
requirements and testing activities for quality characteristics than for functionality (Paech &
Kerlow, 2004; Berntsson Svensson et al., 2012; Chung & do Prado Leite, 2009). For instance,
it is generally more difficult to document quality requirements in a measurable way, or
handle their dependencies to functional requirements (Berntsson Svensson et al., 2012;
Karlsson et al., 2007; Borg et al., 2003; Nuseibeh & Easterbrook, 2000). Borg et al. (2003)
report that lack of competencies to handle quality characteristics in projects often leads to
ignoring related requirement in projects. Even if these requirements are properly quantified
(specified in a measurable manner), they require suitable competencies to be tested and
verified. Similar problems have also been discussed in the usability literature. Practitioners
find it challenging to document measurable usability requirements (Cajander, Lárusdóttir
& Gulliksen, 2013; Gulliksen et al., 2004). Ardito et al. (2014) showed that if practitioners
fail to include usability in requirements documents, these requirements might be ignored
later in testing.

The limited access to competencies and unclear responsibilities are also reported as
challenges to better usability work. For instance, Rosenbaum, Rohn & Humburg (2000)
report that lack of usability professionals is one of the main obstacles organizations face
concerning usability. Boivie, Gulliksen & Göransson (2006) show that even in cases when
organizations have access to the right expertise, usability professionals are not sure about
their responsibilities, and are uncertain as to how to contribute to the projects. Chamberlain,

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 5/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Sharp & Maiden (2006) report that power struggles rise as designers within a project defend
their discipline in response to the decisions made by developers, and vice versa.

Despite the differentiation of UX and usability, only a limited number of empirical
studies have so far analyzed the implications of these differences. One example is the
study by Vermeeren et al. (2010) that compares the challenges of evaluating usability and
UX. They argue that some of UX evaluation methods need to be further improved and
developed for better use in practice. According to Vermeeren et al., there is still a lack of
suitable methods for evaluating UX in earlier development phases or in the period before
actual use (i.e., anticipated use). They also highlight that current methods are not often
practical because they need special expertise, are time consuming, or their data analysis
is hard. Similarly, Isomursu et al. (2012) discuss that practitioners face difficulties in UX
work compared to usability work because they do not have access to tools and methods to
objectively measure UX.

Law, Van Schaik & Roto (2014) explore the basic question of whether UX elements are
measurable. They report that their interviewees expressed skepticism and ambivalence
towards specific UX measures even if attitudes were more positive overall. They note
that practitioners show opposing views on whether UX can or should be divided into
composing elements, or whether it needs to be considered or measured as a whole. Results
from their interviews show three categories of challenges concerning the interplay between
UX evaluation and software development: (i) theoretical (measuring UX holistically
or in elements, and conceptualizing long-lasting versus momentary experience), (ii)
methodological (differing preferences for quantitative versus qualitative data by design- and
engineering-oriented stakeholders), and (iii) practical (lack of knowledge and competencies
for interpreting measurement outcomes). The participants in the Law et al.’s study
emphasized they need to use a variety of media (e.g., video, TV, social media) to develop
the required prototypes for measuring UX and that they often even need more than one
such prototype to gather enough input for design. Their practitioners also argued that UX
measures are essentially prone to fading and fabrication, or that there is a lack of means to
measure the exact emotion of users at each moment (Law, Van Schaik & Roto, 2014).

The work of Law, Van Schaik & Roto (2014) is duplicated in the context of the Latin
American software development industry by Gerea & Herskovic (2007). They conclude
that practical aspects such as cost and time play a more important role in whether or not
practitioners measure UX. Other challenges reported by Gerea et al. include: limited access
to the end users, and lack of knowledge and experience in UX measurement similar to
what Vermeeren et al. (2010) found.

Alves, Valente & Nunes (2014) investigated how UX evaluation is performed in practice
(i.e., by whom, in what phases of software development, and using what tools and methods).
According to their data, in around 50% of the cases UX evaluation is performed without
involving the end users, and often evaluators ‘assume’ what the perception of users will
be. Alves, Valente & Nunes (2014) also report that sometimes evaluations are performed
by software developers that do not necessarily have the required competencies. Also, often
tools and methods are selected based on cost rather than suitability for the project. Alves
el al. used a list of evaluation tools and methods that are mainly usability-specific and

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 6/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


1see http://www.allaboutux.org for more
UX-specific tools and methods.

2Here, we use the terms ‘UX practitioner’
and ‘non-UX practitioners’ to respectively
refer to practitioners who have UX-
related roles and responsibilities in the
organizations and those who do not.

lack UX-specific methods (e.g., Attrakdiff questionnaire (Hassenzahl, Burmester & Koller,
2003)1). This can introduce a risk to their data because practitioners might have preferred
the use of a generic method such as a questionnaire for evaluating UX, without necessarily
acknowledging that such generic methods may purely produce usability-related data if not
used with specific attention to UX-related concepts, such as the non-tasks-related aspect
of use (Hassenzahl, Diefenbach & Göritz, 2010).

Studies also show that the perception of UX is generally different in academic and
industrial contexts. Väänänen-Vainio-Mattila, Roto & Hassenzahl (2008) report that
practitioners still focus more on functionality and usability issues in their UX work.
Similarly, in Kuusinen & Väänänen-Vainio-Mattila’s (2012) study in a large software
organization, ease of use and efficiency were the most often reported sources of good
UX. Hence, these studies conclude that while academia concentrates more on the hedonic
aspect, the industry focuses more on the pragmatic aspect.

As we saw, those studies that explicitly take differences between UX and usability into
account, mainly focus on evaluation activities and the role of UX measures in challenges
practitioners face. They therefore do not sufficiently discuss other aspects of UX work,
for instance requirements or communication and collaboration between UX and non-UX
practitioners.2 In addition, those studies that report software organizations often focus
more on usability than UX rarely provide an explanation for this lopsided focus. Hence,
there is a need for further empirical studies that investigate the role of UX characteristics
and differentiate UX from other software quality characteristics including usability. Our
study is a response to this research gap.

METHODS
We have conducted an explorative, qualitative study (Braun & Clarke, 2006) to investigate
the challenges to integrating UX practices into software development processes. Below, we
detail our research approach by describing the different companies where we conducted
interviews, how the data was collected, and our approach to data analysis.

Research sites
We selected a variety of companies with different characteristics for our study in order
to improve the generalizability of our findings (Runeson & Höst, 2008). Table 1 shows an
overview of these companies and their main characteristics and labels them (A–H) for
easier reference later in the paper.

The companies span various domains (company type) and vary in size (number of
employees). We approached both consultancy and product development companies in
order to cover both perspectives. Both A and E are well-known consultancy companies in
Sweden, while B, C, D, and H are well-known product development companies in Sweden.
Throughout the study, we were introduced to additional companies by our interviewees,
and we also included a number of interviewees from those companies (F and G). Company
F is an international consultancy company, also active in Sweden. Company G is an
American company. Only one of the companies (E) was previously known to the authors
based on previous research collaborations.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 7/40

http://www.allaboutux.org
https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Table 1 Overview of the companies that participated in the study.

Company type Number of employees

(A) IT services, consultancy & outsourcing >130,000
(B) Defense & security ∼14,000
(C) Wireless solutions & network testing ∼1,800
(D) Systems & software development company ∼120
(E) User experience consultancy & training ∼50
(F) IT & management consultancy >1,300
(G) User experience & Usability consultancy 34
(H) Telecommunication ∼114,000

Table 2 The interviewees who participated in this study.

Code Roles & responsibilities Field of education Industrial experience (since)

A-1 Business developer, UX lead & interaction designer Cartoonist 2010
A-2 Interaction designer & UX designer Computer science & interaction design 2005
A-3 Project leader & developer Computer science 2006
A-4 Art director & graphical designer Photography 1996
B-1 Software engineer & project manager Psychology & electrical engineering 2007
B-2 Technical product manager Computer science 2000
B-3 Senior project manager Computer science 1995
B-4 Product manager & project manager Computer science 2006
B-5 Technical project manager & requirements analyst Computer science 2002
B-6 System developer & interaction designer Computer science & interaction design 2011
C-1 UX lead & interaction designer Cognitive science 1996
D-1 System developer & requirements analyst Computer science 2000
E-1 Business developer, UX lead & interaction designer Software engineering 1983
F-1 UX lead & human factors specialist Cognitive science 1989
G-1 Manager, design strategist & UX lead Design 1999
H-1 Manager, design strategist & UX designer Multimedia & interaction design 1997
H-2 UX designer Multimdeia & design 1996

Data collection
The aim of the interviews was presented to our main contacts in each company to assure
selected interviewees were suitable for our research. The selected practitioners (see Table 2)
represent technical (e.g., developers), design (e.g., interaction designers), and management
roles. We had the option to ask for more interviewees, but since the study was explorative,
after 17 interviews we were confident that we had covered the major challenges from a
sufficiently broad range of perspectives.

We conducted semi-structured interviews (Flick, 2009) to collect more of the
interviewees’ viewpoints, which was important to the explorative nature of our study.
The questions covered different phases of development processes, and also the concept of
UX and how it differed from usability in the interviewee’s eye.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 8/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Transcribing the 
interviews

Segmenting the 
interview 

transcripts

e.g. initial codes:
- UX vs. usability
- motives
- definition
- tools and methods

Identifying the key 
points in each 

segment

Coding the 
segments

Conducting the 
interviews

Data collection

Data analysis

Creating the initial 
set of codes

Creating the 
interview guide Literature review

Formulating the 
research question

Preparation 

Creating new 
codes

Categorizing the 
coded segments

Creating the 
themes

a segment per row in the 
excel file

coding was performed in an 
excel file: each row was 
coded, using separate 
codes in different columns

excel macros were used to 
generate separate sheets for 
each code from different 
interview transcripts

Challenge

Code

Key point

Interview 
transcript

Theme

Levels of 
data abstraction

Figure 1 Process of data gathering and analysis.
Full-size DOI: 10.7717/peerjcs.130/fig-1

We prepared an interview guide with a set of pre-designed questions based on the
knowledge gained from literature. In each interview, questions were rephrased, added,
or skipped based on the interviewee’s background and responses. Each interview covered
all phases of software development, the activities performed in each phase, and the tools
and methods applied. Thirteen of the interviews were conducted face-to-face, and four via
video or telephone conference. Each interview lasted between 30–60 min, and was recorded
and transcribed. The interviews were all performed in the spring of 2012.

Data analysis
Figure 1 provides an overview of the data gathering and analysis process. We applied
thematic analysis as described in Braun and Clarke (Braun & Clarke, 2006) to encode and
analyze the interview data.

We segmented the interview transcriptions into meaningful paragraphs or sentences in
a way that each of these segments presented one concept. We then coded these segments
(Braun & Clarke, 2006). We used Microsoft Excel to document the coded data. Each
interview transcript was recorded in a separate sheet. Segments of this transcript was
recorded in separate rows, and different codes were assigned to each segment in separated

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 9/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-1
http://dx.doi.org/10.7717/peerj-cs.130


columns, following that segment. After coding, categories of challenges, i.e., themes,
emerged as we put together similar concepts presented in the coded segments. A mind-map
of challenges and categories was created to better identify and visualize the relations.

We created a pilot list of codes (Braun & Clarke, 2006) that was iteratively refined. The
initial codes captured challenges and their context and included: definition, challenges,
solutions, tools and methods, evaluations, requirements, UX versus usability, and activities.

As the interviewees mentioned various challenges, we asked follow-up questions to better
understand attempted solutions, or previous experiences with usability or other quality
characteristics in relation to similar challenges. When the interviewees used different
terminologies, or had limited knowledge concerning UX or usability, we mapped their
statements to the relevant concepts based on the definitions of the concepts in literature.

For instance this statement: ‘‘There is a bit of confusion in the field and in the company as
well, what’s the difference? design is design’’ (A-1) mapped to these key points: definition of
UX is not clear, and practitioners do not know what UX exactly means. This statement
was then coded as ‘challenge’, and ‘definition’. Segments that related to understanding
and definition of UX, such as this example, resulted in the theme ‘lack of consensus on
definition and construct of UX’.

Threats to validity
Threats to validity are outlined and discussed based on the classification by Runeson
and Höst (Runeson & Höst, 2008): Selection process of subjects for interviews can cause
a threat to construct validity. Selection bias is always present when subjects are not fully
randomly sampled. However, our subjects were selected based on their role, experience
and availability so there is little more we could do to alleviate this threat. The presence
of a researcher may influence the behavior and response of the subjects. This threat was
alleviated somewhat by the guarantee of confidentiality of the data but is an inherent
limitation of the research method used.

In any empirical study, incorrect data is a threat to internal validity. Interviews were
audio recorded to mitigate this threat. The authors also analyzed the material in several
rounds of independent as well as joint sessions to gradually reach consensus on the intended
meaning of the respondents. We also shared the results of our analysis with the interviewees
to validate and confirm the findings.

External validity concerns the ability to generalize the results beyond the actual study.
Since the interviews are just a sample from a potentially very large population, they should
be interpreted with some caution. We sampled a number of different organizations in
different industrial domains to decrease the effect of this threat. In addition, the majority
of the organizations we studied are Swedish (exceptions are G (American), and F (Dutch)),
and the culture of Swedish software industry can have an impact on how the studied
organizations perceive and address UX or other quality characteristics. This distribution
however is not sufficient to draw any conclusions based on the cultural differences of
these companies, and in interpretation of findings this matter needs to be taken into
consideration. Qualitative studies rarely attempt to generalize beyond the actual setting
and are more concerned with explaining and understanding the phenomena under study.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 10/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Theme 1: 
Lack of consensus on 

definition & construct of 
UX

Theme 2: 
Lack of consensus on the 

value of UX

Theme 3: 
Low industrial impact of 

UX models, tools & 
methods 

Theme 4: 
Focus on objectively 

measurable aspects of 
software

Theme 5: 
Difficulties in engineering 
UX-related requirements

Theme 6: 
Focus on evaluating 

functionality & usability,  
not UX

Theme 7: 
Lack of consensus on UX 
related competencies & 

responsibilities

Theme 8: 
Communication & 

collaboration gap between 
UX & non-UX practitioners

Themes of 
challenges

more tactical

more fundamental

inclu
des explains

Figure 2 The identified themes of challenges.
Full-size DOI: 10.7717/peerjcs.130/fig-2

Another concern is that our data gathering was performed in the spring of 2012.
Therefore, our data may not reflect today’s UX state-of-practice in the studied
organizations. However, the data is valid when interpreted in its own time frame. Also,
to minimize the effect of the time frame on our analysis, we have included recent studies
published since 2012 when discussing the results.

Another threat concerns reliability, the extent to which the data and analysis are
dependent on the specific researchers. Although the coding process is performed by the
first author, to improve reliability of the generated themes, the three authors individually
and independently conducted a pilot coding of these segments using an initial coding guide
as explained above. The outcomes of the pilot coding were discussed in several sessions
with all three authors, and the differences in coding were analyzed and resolved. Also, we
had carefully designed the interviews before running them. We also defined the coding
process after the interviews and before analyzing the data. The initial codes were therefore
identified mainly based on observed interview responses. We also ensured the themes are
not imposed on the data rather emerged from it.

RESULTS
This section presents eight themes of challenges that emerged from the interview data (see
Fig. 2). These themes are described and supported by the interviewees’ quotations. Each
quotation is provided with the interviewee’s code, and labeled either ‘SE’ or ‘HCI’ to reflect
the community to which the interviewee belongs. This helps better understanding of the
current state of practice from these two perspectives.

Theme 1. Lack of consensus on definition and construct of UX
Our data shows that practitioners’ understanding of the concept of UX differs and is, in
some cases, even inconsistent and contradictory (see Fig. 3). According to the respondents,
UX is a new concept and there is a lack of general agreement on its meaning in the field in

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 11/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-2
http://dx.doi.org/10.7717/peerj-cs.130


Theme 1: 
Lack of consensus on 
definition & construct 

of UX

Practitioners have 
different 

understandings of 
UX

Practitioners have 
inconsistent & 

contradictory views 
on UX

Customers have 
limited knowledge 

about UX

There is a lack of 
consensus on 

meaning of UX both in 
research & practice

UX is seen as equal 
to usability or IxD

includes

Figure 3 Challenges concerning definition and construct of UX (Theme 1).
Full-size DOI: 10.7717/peerjcs.130/fig-3

general, and among practitioners within organizations in particular. In some practitioners’
view, UX is the same as usability or Interaction Design (IxD) despite the fact that UX
includes the user’s subjective perception and overall experience. The lopsided focus on
usability and IxD can be explained by their relative simplicity: ‘‘I think discussions at large
when it comes to UX design at common ground is still about IxD and usability. Usability is
easy to talk about and everybody understands it.’’ (A-1/HCI).

One of the interviewees referred to UX as ‘‘just another buzz word’’ (E-1/HCI). In her
view, UX contains the same concepts that have been around for a long time under other
names such as usability and emotional design. On the other hand, some practitioners
explicitly differentiated between UX and usability. One of the participants emphasized
that usability is ‘‘a minor subset of UX’’ and added: ‘‘I’ve never, ever called myself a usability
expert, and I would never do that.’’ (H-1/HCI). Similarly, another practitioner stressed that
UX is not about IxD and has a much broader perspective: ‘‘[IxD] is just the end results, we
do not call ourselves interaction designers, because that is only 10% of our work but that is an
important element because that is the most visual part of our work .’’ (F-1/HCI). Similarly,
another practitioner referred to UX as: ‘‘a wholeness with the emotional, social, economical,
functional, and technical parts.’’ (A-4/HCI). Another practitioner described UX as: ‘‘pretty
much everything that affects a user’s interaction with a product.’’ (H-2/HCI). He further
emphasized UX is ‘‘the whole package’’ and usability is only one part of it. A number of
practitioners stated that UX is a ‘broad’ and ‘holistic’ concept covering not only the user
perspective, but also the business perspective. While the latter looks into how the design
contributes to achieving business goals, the former assures satisfying the end users’ needs.
The user perspective includes aspects such as ‘emotions’, ‘values’ and ‘preferences’.

Some practitioners related UX to the ‘why’ behind the functional requirements and
the software in general. For instance, one practitioner described this through three main
questions of ‘what’ to build, ‘how’ to build it, and ‘why’ to build it (A-4/HCI). Some
practitioners related UX to the GUI by stating that it is about ‘‘the cool things, the new
things, the flashy things.’’ (A-3/SE). In their view, UX is mainly about emotions and aesthetics
therefore not applicable to all types of applications, for instance ‘productivity applications’.

In general, the practitioners with technical backgrounds showed limited knowledge
about UX. Their knowledge was often limited to cognitive aspects of design, i.e., usability.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 12/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-3
http://dx.doi.org/10.7717/peerj-cs.130


Theme 2: 
Lack of consensus on 

the value of UX

B2B businesses do 
not value UX enough

UX is often down-
prioritized to 
functionality

Management does 
not value UX, e.g. as 
part of the business 

strategy

Customers down-
proritize UX & do not 

value it

Stakeholders have 
different views on the 

importance of UX

includ
es

Business 
stakeholders do not 

value UX

Cost of UX is not 
justified for 
customers

Figure 4 Challenges concerning value of UX (Theme 2).
Full-size DOI: 10.7717/peerjcs.130/fig-4

One of the technical managers stated: ‘‘ [our customers] talk about increased workload. That
is a negative thing. I don’t know if that qualifies as UX.’’ (B-2/SE). On the other hand, a UX
practitioners emphasized that UX ‘‘goes beyond cognitive aspects of design’’ (E-1/HCI), the
main focus of usability.

The participants also discussed that customers’ limited knowledge about UX is a
challenge. Customers who have heard about UX can be too ambitious regarding emotional
and non-task-related user needs. Customers’ limited knowledge also means that they
often specify the related requirements vaguely and using inconsistent and subjective
terminology. They often indicate a need for quality characteristics such as ‘cool’, ‘fun’,
or ‘high-tech’ mostly because they are affected by such ‘buzz words’ while they do not
necessarily know what these terms actually mean. Practitioners emphasized that to prevent
misunderstandings UX-related requirements should be refined early on to concrete
requirements and specified in a measurable way. Regarding this one of the interviewees said:
‘‘usually they say ‘we want something like that app’, ‘we want it to be cool and high tech’.
Then you have to initiate a dialog to find out what that means for this particular customer.’’
(A-1/HCI).

Theme 2. Lack of consensus on the value of UX
Generally speaking, our data shows that various stakeholders have different views on
whether UX is important or not (see Fig. 4). A group of the interviewees believed UX of
software is increasingly important because of the growing general importance of software in
recent decades. Interactions users had with earlier software were limited to command-line,
and software was mainly built to support existing manual work. Software is now a large
part of life, and users are exposed to a variety of interaction styles (mouse, touch, etc.),
and experiences. This exposure to various software systems affects the experience and

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 13/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-4
http://dx.doi.org/10.7717/peerj-cs.130


expectations of users. Regarding this, one practitioner said: ‘‘Users are meeting a lot of good
things, and they are expecting good things all the time.’’ (E-1/HCI).

According to the practitioners, various businesses are learning from successful products
in the market. This has inspired not only market-driven but also business-to-business
software projects. This is supported by one of the interviewees saying: ‘‘A lot of business-
to-business applications are being informed by business-to-consumer apps.’’ (G-1/HCI). In
particular, in cases where a product has competitors, the motivation to improve UX
increases. One of the practitioners argued that UX is now a ‘differentiator’: ‘‘Today I
think that many companies do usable products, in order to distinguish a brand or a product
we need to add an extra level to the product so that is really what I call UX. We need to take
more things into account, e.g., emotions, and that it needs to look great, and it is not only
about being usable.’’ (A-2/HCI). In market-driven products, branding, emotional concerns,
and relations with end users are important. Therefore, in our participants’ views, these
businesses are often more concerned about UX. Another interviewee argued that for
market-driven software, in particular game development, UX is ‘‘part of the common
practice’’ (A-1/HCI) while this is not the case for business-to-business software. He
further emphasized this approach of market-driven projects should be ‘‘transferred to other
projects’’ as well.

Some of the practitioners emphasized that UX is an important software quality
characteristic that needs to be taken into account more in projects. However, in some
functionality-focused organizations, UX is considered something ‘‘on the side’’ and not a
core concept or value. According to these practitioners, the business units often focus on
functionality and not UX. As one of the interviewees emphasized, in this case the business
unit is not ‘‘that concerned with the look and feel’’ (A-4/HCI). The practitioners were
generally positive that more and more organizations, even the technology-focused ones
are learning about UX and the importance of taking it into account in their products. For
instance, one practitioner said that in their company, business units now show less resistance
towards UX and the importance of UX ‘‘starts to be visible’’ (H-2/HCI) to these units.

In addition, the interviewees emphasized managements’ preferences, values, and
motivations also influence UX work in organizations. Regarding this, one practitioner
stated: ‘‘I think it’s sometimes just the reason people go into business in the first place ...the
people who are in it because they think that their product or service solves a real problem,
they generally care about UX more.’’ (G-1/HCI). This interviewee further discussed how
individuals, in particular higher management, play an important role in whether UX is a
priority in an organization or not. One of the interviewees stated that the inputs from their
UX group in research and development are ignored by business units because some of the
ideas are ahead of their time while these units tend to focus on near future: ‘‘business units
are occupied with their very close, near time results so they look at what they can sell now.’’
(H-2/HCI). He further highlighted that some of these previously rejected UX ideas are now
being incorporated in the products and are getting support from management because
competitors now implement similar ideas.

Another challenge is that customers often resist taking on the costs of performing UX
work. In practitioners’ view, often customers are not aware of UX and its value, and

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 14/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


in particular how it differs from usability and IxD. Regarding difficulties in convincing
customers, a practitioner said: ‘‘It can be hard ...If you have to have three weeks extra to
make the graphics work in a certain way, they might think it’s unnecessary. And maybe the
software’s going to work fine, but if they made these extra efforts or put in this extra amount
of money, they would have actually gone much, much further maybe.’’ (A-4/HCI). As a way
to convince customers to agree to the cost of UX practices, one of the organizations (A)
uses examples of successful products in the market that are known for their good UX (e.g.,
Apple). Another organization (F) uses fixed prices for their projects so that practitioners
can freely spend part of the budget on UX practices. Some practitioners emphasized that
they talk about UX ‘indirectly’. They connect UX to business goals, and argue for UX from
the point of view of the business success, and not the end users. An interviewee motivated
this by saying: ‘‘if you start babbling about usability and strange kind of things, they will say
‘oh! I don’t want to pay for this’.’’ (E-1/HCI).

According to some practitioners, UX practices should be sold to the customer as part
of the contract to assure coverage of the associated costs. This requires showing how such
practices will add value to the software and have a return on investment (ROI) for the
customer. Nevertheless, often presenting a ROI is difficult. This is illustrated by a UX prac-
titioner who said: ‘‘I don’t think you can put ROI into a proposal necessarily. I think that’s
irresponsible, frankly. Because 95% of the time, we don’t understand the true and false nature
of an issue when we’re writing a proposal. It’s only after working with a client for a little bit
of time that we begin to see the nuance there. That usually undercuts any kind of under-
standing that’s used to generate proposed improvements in ROI, for example.’’ (G-1/HCI).

Theme 3. Low industrial impact of UX models, tools and methods
We identified a number of challenges concerning how practitioners apply UX theories,
tools and methods (see Fig. 5). We observed that often practitioners’ knowledge regarding
UX is based on experience and work with similar concepts, not on any specific UX
models or theories. Some practitioners had formal training in usability and IxD and used
that foundation to build UX skills over time in an ad hoc way. For instance, one of the
practitioners stated: ‘‘When I first started in the 90s, there was no such thing as UX or IxD,
actually ... [then] it has been IxD and then sort of merged into UX design.’’ (H-1/HCI).
In general, the interviewees were not familiar with currently popular approaches to UX
and corresponding models, even those interviewees that demonstrated a relatively good
understanding of UX. One interviewee (A-4/HCI) stated that she handles users’ emotions,
values etc. ‘in a non-academic way’, another interviewee (A-2/HCI) said that the way
she handles UX is not formal or ‘by the book’. The practitioners generally showed a
positive attitude towards applying new models, tools, methods and techniques to their
work: ‘‘we are lacking this, so this would be really nice to have more research results that we
could apply.’’ (A-2/HCI). However, often organizations resist introducing new models,
tools, methods, or techniques. Hence, in the studied organizations, practitioners often
only rely on traditional interview and observation techniques in their work with UX.
The interviewees referred to two models they use in their UX work: ‘emotional design’
by Donald Norman and Maslow’s hierarchy of human needs. The latter is used as an

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 15/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Theme 3: 
Low industrial impact 
of UX models, tools & 

methods 

Organisations resist 
to introduce UX tools 

& methods

Current UX theories   
are often unknown to 

practitioners

Knowledge on UX is 
gained individually & 
in an ad hoc way & 
not through theories

includes UX practitioners 
apply only traditional 

tools & methods

Figure 5 Challenges concerning industrial impact of UX theories (Theme 3).
Full-size DOI: 10.7717/peerjcs.130/fig-5

Theme 4: 
Focus on objectively 

measurable aspects of 
software

Stakeholders focus on 
functionality & often 

down-prfioritize quality 
characteristics

Stakeholders focus on 
actual quality characteristics 
& often down-prioritize the 

perceived qualities

Stakeholders focus on 
objectively 

measurable quality 
characteristics 

includes

Too much focus on 
functionalities can lead to 

ignoring UX

The relation of 
functionality and UX 

is often not taken 
into account

Figure 6 Challenges concerning subjective aspects of software (Theme 4).
Full-size DOI: 10.7717/peerjcs.130/fig-6

inspiration when eliciting user needs. The interviewees mentioned that such models can
help create a methodology to work with UX and ‘‘build the right things in the right order’’
(F-1/HCI). Respondents with this type of experience were a clear minority.

Theme 4. Focus on objectively measurable aspects of software
Another identified theme of challenges concerns how the studied organizations handle
subjective aspects of software (see Fig. 6). A group of practitioners emphasized that
the software development and engineering community has traditionally had much greater
focus on software functionality. One practitioner stressed the relation between functionality
and UX as follows: ‘‘The quality of experience is really depending to some extent on how the
functional requirements are met, but also actually on what the functional requirements are,
also just the amount of them.’’ (H-1/HCI). This interviewee further emphasized: ‘‘For us,
technology is the material that we can shape in order to create experiences.’’ The interviewees
emphasized that satisfying functional requirements does not necessarily mean that the
software includes correct or valuable functionality. This is evidenced by one interviewee
saying: ‘‘what do you know when you have signed [the technical specification]? Do you know
that it is a good solution? No! you only know that it meets the functional requirements and to
me it is silly!’’ (E-1/HCI).

In addition, the participants believed that the software community often focuses more
on ‘actual’ than ‘perceived’ quality characteristics of software. While the former concerns

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 16/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-5
https://doi.org/10.7717/peerjcs.130/fig-6
http://dx.doi.org/10.7717/peerj-cs.130


Theme 5: 
Difficulties in 

engineering UX-
related requirements

To-be goals/non-task 
related needs are 
either ignored or 
treated informally

Traceability between UX-
related & business 

requirements is often lost in 
projects

Customers are too 
ambitious regarding 

UX-related 
requirements

Customers use 
wrong terminology 

for UX-related 
requirements

Practitioners lack 
guidelines on mapping 

non-task-related needs to 
measurable requirements

It is difficult to involve 
users to assure 

addressing UX-related 
requirements

includes

Non-task-related needs 
are hard to elicit, refine, 
communicate & agree 

upon

Practitioners lack 
tools & methods to 
deal with non-task-

related needs

Practitioners need 
forms other than text 
to communicate UX-
related requirements

It is difficult to find a 
balance between 

business & technical & 
UX-related requirements

There is a lack of 
consensus on to what 
extent users should be 

involved in projects

User involvement is 
perceived to be time-

consuming

Functional requirements are 
easier to define & translate 

to design solutions than UX-
related requirements

There is a lack of consensus 
on the order of eliciting & 
refining functional vs. UX-

related requirements 

Figure 7 Challenges concerning requirements work (Theme 5).
Full-size DOI: 10.7717/peerjcs.130/fig-7

objectively measurable quality characteristics, the latter concerns how users subjectively
perceive these qualities. For instance, users may perceive a response time of 50 ms (i.e.,
actual performance) as fast or slow (i.e., perceived performance). Regarding the role of
perceived qualities in experience of users an interviewee stated: ‘‘sometimes the perception
of time is more important than the actual time, and these are the things you should pinpoint
[to the stakeholders].’’ (E-1/HCI).

Theme 5. Difficulties in engineering UX-related requirements
Another identified theme of challenges concerns handling UX in requirements work (see
Fig. 7). According to the practitioners, in many cases, the non-task-related needs of users
(i.e., their be-goals) or their emotions are either neglected or treated only informally:
‘‘We’d specify this more in the persona descriptions; for example that this persona needs to,
or wants to experience some kind of things. In the wireframes, we might specify an animation
for example that it should feel smooth or something like that.’’ (A-2/HCI). For instance,
in one of the organizations, emotional design goals are often only documented and
communicated in the form of a ‘‘post-it note on the wall’’, as a reminder. Although the
interviewee emphasized this approach is not optimal and there is a need for more formal
approaches to deal with such requirements: ‘‘it should be good if we could formalize it a little
bit more I think.’’ (A-2/HCI).

The practitioners highlighted that it is an open problem as to how to map these types of
needs to measurable requirements. For instance, one of the interviewees stated: ‘‘I would
say the emotional part of this is very very rarely formally put into words.’’ (A-1/HCI). In

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 17/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-7
http://dx.doi.org/10.7717/peerj-cs.130


particular, when the software product is innovative, these problems are compounded. This
is evidenced by an interviewee saying: ‘‘This is a kind of project where nobody really can tell
how this should be, what it should be like, nobody has done it before, there are no standards
to refer to ...who can specify those [UX-related] requirements? You need to have a certain
quality but what is the suitable level of that quality? We haven’t really found what that level
is.’’ (B-3/SE).

The practitioners generally agreed that to communicate UX-related requirements, they
require forms other than text (e.g., sketches, wireframes). They emphasized ‘‘concrete and
tangible’’ forms can facilitate communicating these intangible and abstract requirements,
and ensure that be-goals as well as do-goals are taken into consideration. Regarding this
an interviewee told us: ‘‘we create ‘mood boards’ where you take an image-driven approach
to the look and feel, and we use references of course, like ‘that app has a good flow in it’, and
‘that app has a good feeling to it’.’’ (A-1/HCI). Practitioners with technical backgrounds
also seemed to have a similar approach to UX-related requirements: ‘‘If the customer said
that they want it to ‘look nice’, then you have to make the graphical design first and then they
can say ‘hey! this looks nice’ and then you have taken care of that requirement.’’ (A-3/SE).

It is important that these non-textual UX-related requirements (e.g., wireframes) are
traceable to other requirements. Regarding this a practitioner stated: ‘‘at the end we show
the wireframes with all kinds of numbers and those numbers are linked to the excel sheets of
the requirements, their descriptions and how they are linked with the CPR and business case.’’
(F-1/HCI). Practitioners related these problems to a lack of competence in dealing with
UX-related requirements within their organizations: ‘‘I think it is largely a competence thing.
Doing emotional aspects of design is quite a new concept, I have only heard about it in the last
year, or last two years maybe, so I do not think that knowledge has really reached the industry
yet.’’ (A-3/SE). UX practitioners believed that handling functionality is comparatively
easy and straightforward. Regarding this, one practitioner highlighted: ‘‘features are
what most project managers, most managers can understand. You can count them, you can
map them to customers, customer dialog for instance, and so forth, and you can compare your
amount of features with the competitors...when it comes to both features and user experience
requirements—features are a bit easier to define somehow, because you can say things like,
‘Right, we need an email component with this capability’’’ (H-1/HCI). As another example,
an interviewee stated: ‘‘Functional requirements are easy to create, to merge into a design;
more emotional things are more difficult.’’ (F-1/HCI).

UX practitioners argued that UX-related requirements should be elicited before
refining functional requirements. For instance, one interviewee stated: ‘‘First you have
to define the business requirements, the user requirements, the IxD, then you can define the
functional requirements.’’ (E-1/HCI). Another practitioner said: ‘‘I think that the functional
requirements should come as a result of a dialog between different types of domains such as
user experience, business, and technology.’’ (H-1/HCI). Nevertheless, according to these
practitioners, such an approach is not common in practice. This is inline with the view of
practitioners with technical backgrounds who believe the requirement process should start
by first eliciting and refining functional requirements and then quality requirements.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 18/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Some practitioners highlighted the challenge of finding a balance between UX-related
needs, business goals, and technological constraints. Regarding the importance of finding
a balance between emotional and business needs, one of the interviewees stated: ‘‘you
can spend a lot of time, thinking about people’s emotions and so on, but if you are going to
succeed you have to look at the business perspective.’’ (E-1/HCI). In addition, according to
the interviewees, while customers should be involved to assure alignment of projects with
business perspectives, the end users should be involved to assure alignment with UX-related
needs, and practitioners require to find a balance between these needs. As an example, an
interviewee stated: ‘‘we can buy the best camera in the world, users will have a better image
quality than they have in reality but it is too expensive today, ....that ruins the business case
for that customer’’ (e.g., B-3/SE).

We observed some inconsistencies in practitioners’ views concerning user involvement.
In favor of involving users an interviewee stated: ‘‘maybe you want to have the end user
involved also with the developers so that developers understand what they are doing, instead
of just following the specifications. I think that would be very very valuable.’’ (A-3/SE). On
the other hand, a project manager stated that involving users in requirements discussions
increases the costs of the projects. Therefore, it is better to negotiate requirements and sign
a contract without involving the users. Regarding this an interviewee stated: ‘‘[they] think
they can say anything during the [requirement] workshop and then get it. It is not the case...
So it leads to lots of long long long discussions afterwards.’’ (B-2/SE). As another example: ‘‘it
usually leads to features that you take on more than what you agreed from the beginning. So
it’s possible that the customer gets a better system but they still don’t pay you more money for
this.’’ (B-1/SE).

One developer stated that they often have less access to the end users, which can
be problematic for their work mainly because this means developers do not often
understand the rationale for the requirements: ‘‘developers don’t get the motivation behind
the requirements because that gets lost during the way. So, the marketing people say that we
must have this requirement, and interaction designers say we must do it this way, so you
have the ‘what’ and the ‘how’ but you never get the ‘why’.’’ (A-3/SE). Some practitioners
emphasized that relying too much on the end users’ opinions might lead to less creativity
in the design work: ‘‘we have a quote sitting on the wall here, it’s from Henry Ford [that says]
‘If I had asked people what they wanted, they would have said a faster horse’.’’ (H-2/HCI).

Theme 6. Focus on evaluating functionality and usability, not UX
We also identified a number of challenges regarding evaluation of UX in the studied
organizations (see Fig. 8). Our data shows that in general, our selected organizations
mainly focus on testing functionality. In addition, similar to the previous theme on
requirements, the interviewees highlighted that limited involvement of the end users in
projects is a challenge to evaluating UX, especially since evaluating UX requires gathering
users subjective perception of the software. We observed that practitioners with technical
backgrounds are often unfamiliar with how their organizations handle UX or even usability
testing. They also showed limited knowledge as to why such evaluations can contribute to
the success of the software.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 19/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Theme 6: 
Focus on evaluating 

functionality & 
usability, not UX

UX evaluation is 
more difficult because 
of its holistic nature & 

broad scope

Practitioners focus 
more on testing 

functionality

Situatedness of UX 
makes it difficult to 

evaluate 

Practitioners find it more 
difficult to evaluate UX  
than usability or other 
quality characteristics

Temporality of UX 
makes it difficult to 

evaluate

Limited resources are 
assigned to UX 

evaluation although it 
requires more resources 

Practitioners find it 
difficult to evaluate non-

task-related/hedonic 
aspect of software

Practitioners have 
limited access to 

tools & methods for 
evaluating UX

includes

It is difficult to involve 
users to evaluate UX, 

in particular its 
subjective elements

Technical stakeholders 
lack knowledge on the 
value of UX evaluation

Technical stakeholders 
lack knowledge on 

how UX evaluation is 
performed

UX evaluation is 
often replaced by 
usability testing

UX evaluation is 
performed informally 
so may not result in 

reliable results

Practitioners require 
more complete 
prototypes to 
evaluate UX

Figure 8 Challenges concerning testing and evaluation (Theme 6).
Full-size DOI: 10.7717/peerjcs.130/fig-8

Generally speaking, UX evaluation is immature in the studied organizations. In projects
with limited time or budget, UX evaluation is either non-existent or rare, especially when
compared to other testing activities: ‘‘we have done much functional testing of course, system
tests etc., but end user testing we have not performed much I’d say.’’ (A-2/HCI). In some
organizations, UX evaluation is basically replaced by usability testing. Regarding this, one
interviewee stated: ‘‘I think user tests tend to be more focused on pure usability. I guess it’s
when you’re releasing the product into the wild, that’s when you start to get maybe the most
valuable feedback or the most truthful ones.’’ (A-4/HCI). According to these practitioners,
usability testing is not enough to evaluate the whole UX of the software. Regarding this, a UX
practitioner stated: ‘‘when you evaluate usability, it’s when you go into the nitty gritty details,
and try to look at efficiency within the user interface. My personal view is that that is not that
relevant. I mean it’s relevant but not in what we do [i.e., UX].’’ (H-2/HCI). To compensate
for limited formal UX evaluations, some practitioners gather informal qualitative user
feedback after release, for instance through comments in the App Store or on social media.

According to the practitioners we spoke with, UX evaluation is limited because it is
difficult compared to evaluating usability or other quality characteristics. Some practitioners

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 20/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-8
http://dx.doi.org/10.7717/peerj-cs.130


related this difficulty to the fact that UX involves emotions, and non-task-related user needs
(i.e., be-goals), and that limited tools and methods exist to support addressing these needs
in evaluations. Emotion can be even impossible to measure using current quantitative
approaches, as one of the practitioners stated: ‘‘it is difficult to sort out or really get the
correct feeling that the user has because they will try to explain it but it perhaps is not the real
emotion that we catch at the end. If we had a method or approach to really get these from the
user that would be great.’’ (A-2/HCI). Similarly, one interviewee said: ‘‘some goals are more
difficult to measure than others, e.g., if this is a feeling thing: ‘I should be very well informed’,
but mostly you can measure [them] in the usage test through observations and interviews’’ (E-
1/HCI). This practitioner further emphasized that they can specify quantitative measures
for UX-related requirements (e.g., ‘‘10 out of 10 users should succeed, and they should
be content’’), but also need to observe users in order to gain better understanding of
the experience: ‘‘can the users perform the tasks? how do they perform the tasks? how do they
feel afterwards? are they content?’’ (E-1/HCI). This interviewee further emphasized that to
measure feelings of users they need ‘a rather complete prototype’. Practitioners highlighted
that despite importance of involving the end users in UX evaluations, it is difficult to access
them: ‘‘I think we should involve the end users a lot more but it is a political issue, the project
needs to convince the customer that this is necessary.’’ (A-2/HCI).

Some of the interviewees related the difficulty in UX evaluation to the holistic
nature of UX. Discussing cases where practitioners take a holistic approach to UX,
one interviewee stated: ‘‘how would you measure that sort of holistic experience throughout
the process of designing it? Because, of course you cannot [implement or design] everything
at the same time, and you know there are so many dependencies. How do you straighten
those out and how do you understand what you’re measuring and not measuring?’’ (H-
1/HCI). This interviewee further emphasized that the broader scope of UX negatively
impacts evaluations: ‘‘Evaluation is a problem of course, cause user experience is much
broader in scope than usability, it’s more difficult to evaluate also. Like the phone example
that I gave you before, how do you pick up on the fact that someone has experienced the
competition unless you ask for it? Can you count on what a person would actually say about
their experience? What if the person doesn’t say anything? That’s a problem, of course,
because UX is much broader in scope, and if you have a wider scope on it, then you have a
much more difficult task to actually frame it in an evaluation phase.’’ (H-1/HCI).

Some practitioners related the difficulty in UX evaluation to the fact that users’
expectations and their perception of a product change over time and are affected by various
factors; e.g., introduction of new technologies or appearance of a competing product. One
of the interviewees highlighted: ‘‘It’s like when you try on new clothes. The shirt you were
wearing going into the dressing room and looked fine, looks shabby when you’ve tried out the
new shirt.’’ (H-1/HCI). The interviewee used this analogy to explain the subjective and
dynamic nature of expectations, and that for each individual a new experience can affect
the user’s perception of other products.

Another problem concerns difficulties in relating the result of laboratory evaluations
to the real experience of users, and the role of context in forming experiences: ‘‘I
think user tests tend to be more focused on pure usability. I guess it’s when you’re releasing

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 21/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Theme 7: 
Lack of consensus on 

UX related competencies 
& responsibilities

There is a lack of 
practitioners with 
multidisciplinary 

competencies needed to 
handle UX

There is a lack of 
consensus on how the 

diverse UX responsibilities 
should be assigned in 

projects

Organisations lack 
competencies for eliciting, 
refining & communicating  

be-goals or non-task-
related needs

UX practitioners often fail 
to gain & control an overall 

view on UX design, 
specially in agile settings 

A diverse set of 
competencies is required 

for better UX work

includes

UX responsibilities are not 
well-defined in 
organziations

UX responsibilities are 
negatively impacted by 
limited resources & low 
management support

Figure 9 Challenges concerning UX-related competencies and responsibilities (Theme 7).
Full-size DOI: 10.7717/peerjcs.130/fig-9

the product into the wild, that’s when you start to get maybe the most valuable feedback or
the most truthful ones. So it’s often good to make some sort of what we call a beta product
and publish it. Let people interact with it. But that’s a luxury also because that takes time.
Often, in many, many cases, you have to just make something work. In an ideal world you
should have a testing phase, of course. To get the users’ perspective. That should be really nice
to have.’’ (A-4/HCI). Another problem concerns the relation between different episodes
of experience, for instance first impression of users and UX after using the software for
a while: ‘‘It’s a fantastic feeling to sit in a brand-new car in a car shop. How do you feel about
the same car if you’ve had it for a year, driving it around with two kids in the backseat, for
instance? Experiencing all of the quirks from the interior, and all the things that don’t work,
the stupid things that make your hands go dirty every time you’re supposed to close a door, for
instance? That’s part of user experience design. So it’s a very floating scale.’’ (H-1/HCI).

Theme 7. Lack of consensus on UX-related competencies and
responsibilities
Another group of challenges concerns the competencies and responsibilities required to
perform UX practices (see Fig. 9). Our data shows that for better UX work, organizations
need to have access to a variety of competencies including brand management, visual
design, usability engineering, interaction design, and emotional design. In particular, to be
able to address the hedonic aspect of UX, it is crucial to have access to competencies for
eliciting, refining, communicating, and testing be-goals or non-task-related needs of users.

A group of interviewees believed that due to the wide spectrum of UX competencies,
it is unlikely to find all of those competencies in any individual practitioner. This group
therefore argued that all of the team members (with complementary competencies) should
jointly take on the UX responsibilities. Regarding this, one of the practitioners said: ‘‘I’m
not sure if that should be a specific role [...] so everybody should have a UX focus now. I’m
not sure if we can have some sort of UX guy [who takes the final decisions].’’ (A-4/HCI). This
interviewee further emphasized that achieving a better UX requires a ‘UX-mindset’ that

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 22/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-9
http://dx.doi.org/10.7717/peerj-cs.130


even the technical roles in the projects (e.g., programmers) should have. On the other
hand, a group of interviewees believed there is a need for specific practitioners with
these multidisciplinary competencies because these competencies are tightly connected
and hard to separate. This group therefore argued for defining specific UX-related roles
and responsibilities in projects. Regarding the importance of access to individuals with
diverse competencies, an interviewee stated: ‘‘I, as an art director, have to have somewhat
deep knowledge about UX, and also IxD ...you can’t separate them.’’ (A-4/HCI). Another
interviewee said: ‘‘You have to be knowledgeable in many areas. You have to be very holistic
in the way you think about things, cause you have to speak with programmers in the language
of programmers, to some extent, at least, to gain their respect, but also you have to be treated
as an equal...You also have to understand business, and that side of software. You have to be
a little bit of everything but you also have to be a specialist in your own domain so it’s kind of
like a juggling act.’’ (H-1/HCI).

Being able to gain an overall view of UX design was one of the concerns highlighted in the
interviews. According to the practitioners, this is often a difficult yet important prerequisite
for creating a coherent UX. As the interviewees stated, to deal with the complexity of today’s
systems, the common approach in the software community is to break down the whole
system into various sub-systems and work on them independently. Such an approach can
harm the UX of the software since often in these cases UX practitioners lose the overall view
on the UX design, how these different sub-systems fit together as a whole, and how they
individually and in combination contribute to the experience of the end users. Regarding
this, one interviewee emphasized: ‘‘you have to tear it down, yeah! but what happens to the
whole? who is going to define the whole?’’ (E-1/HCI). The interviewees also highlighted that
in agile settings, the decision-making process is spread out both over time as well as among
the team members. This further complicates the process of creating a unified and coherent
UX design (e.g., H-1/HCI). Further, agile processes enforce a focus on a few piece(s)
of the design at each iteration. Regarding this, one interviewee said: ‘‘you need to deliver
wireframes for parts of the application but you still do not know how it all will fit together at
the end.’’ (A-2/HCI).

Similar to what we saw for competencies, our data shows that responsibilities of UX
practitioners are also very diverse. As some UX practitioners expressed, they have varying
responsibilities in and contributions to different projects; this depends on factors such
as management support, available resources, and timing. In one organization (C), the
UX team is responsible for handling requirements and feeding them to the development
teams. In another organization (G), the UX group is part of R&D where the group mainly
focuses on future products, and long-term vision of the company. One UX practitioner
from company G described her responsibilities as: ‘‘that can loosely be described as discovery,
research, overall strategy, and then high-level design.’’ (G-1/HCI). In another organization
(H), since the number of practitioners with UX knowledge is low, none of these practitioners
are part of any particular project teams, and are instead shared resources among projects.
UX practitioners are also often responsible for spreading knowledge and awareness about
UX in the organization, or giving support to development teams concerning UX matters.
Regarding this, a practitioner expressed: ‘‘I think on a very high level, our responsibility is to

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 23/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Theme 8: 
Communication & 

collaboration gap between 
UX & non-UX practitioners

UX perspective may 
get lost in later 

development phases

UX work is more visible 
throughout the process 

than usability work

UX-related decisions 
are postponed in 

projects, specially in 
agile settings

Some stakeholders 
believe UX can be 

improved through minor 
changes to the GUI in 

later development phases

Power struggles rise 
between UX & non-

UX practitioners 

Non-UX practitioners 
lack trust in UX 

practitioners since the 
UX field is relatively 

young 

UX practitioners get 
involved in projects late

There may be a 
communication & 
collaboration gap 

between UX and non-UX 
practitioners 

includes

Figure 10 Challenges concerning communication and collaboration between UX and non-UX
practitioners (Theme ).

Full-size DOI: 10.7717/peerjcs.130/fig-10

inform, influence and inspire.’’ (H-1/HCI). He further stated: ‘‘we contribute to the process
by running workshops, by providing provocative questions, or providing examples, engaging in
discussions in which people from other domains dig down really deep into their own layers
of knowledge, and we can ask really simple questions to poke them with our perspective.’’
(H-1/HCI).

Theme 8. Communication and collaboration gap between UX and
non-UX practitioners
This theme of challenges concerns how UX and non-UX practitioners communicate with
each other and collaborate in projects (see Fig. 10). The interviewees generally agreed
that better UX work requires early involvement of UX practitioners in projects. Early
involvement helps UX practitioners get first-hand information about the customer and
the end users. In addition, practitioners get a chance to negotiate trade-offs between
the design solutions and technical constraints. If required, a design concept can then
be updated based on the developers’ feedback. Nevertheless, UX practitioners often are
involved in projects only in later stages. Regarding this a practitioner stated: ‘‘The worst
case is when someone has met a client and talked a lot about the software, then I meet this guy
who has met the client ... then it’s secondhand information and everything gets distorted.’’
(A-4/HCI).

Early involvement is in particular important since in later stages it is difficult or even
impossible to make effective changes to the UX design. As an interviewee stated: ‘‘on the
personal level developers can be offended if someone thinks it’s not a good solution, they can
have a hard time killing their darlings, they could have been thinking of it as a very beautiful

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 24/40

https://peerj.com
https://doi.org/10.7717/peerjcs.130/fig-10
http://dx.doi.org/10.7717/peerj-cs.130


solution they have made with 40,00 lines of code, so they don’t want to throw it away.’’ (H-
2/HCI). Some stakeholders however believe that UX can be improved with just minor
GUI changes in later stages of development, they therefore down-prioritize UX practices
in earlier stages. The interviewees also highlighted that agile processes have a limited focus
on strategic decisions such as the overall UX of the software. Often these strategic decisions
are either ‘skipped’ or postponed since agile methodologies tend to prioritize immediate
and current problem solving: ‘‘agile is a lot about problem solving and that’s what sort of gets
priority.’’ (H-1/HCI).

According to the interviewees, even in cases when UX practitioners are involved in early
phases, they may lose their connection to the project in later phases. To overcome this
problem, the practitioners proposed various solutions. One interviewee, for instance, stated
that they rely on UX advocates in projects: ‘‘We are not UX experts that you hire if you want
to give a problem to somebody, have them go away for three months, and then have them
come back with the solution... If something needs to go into production after our principal
work is finished, you need somebody who’s going to be involved in that process and involved
from the very beginning. So we have a very strong focus on collaboration...Oftentimes, at least
when it comes to transitioning into a production environment, we’re looking for UX savvy
development managers or developers.’’ (G-1/HCI). Another UX practitioner told us that
throughout their projects, they continuously check the status of the UX design: ‘‘we try to
have always at least one person who was part of the original dialog present during the weekly
checkups, and basically just going by the desks and checking, informally.’’ (A-1/HCI).

UX practitioners need to be involved in projects early on, and need to have an active
role throughout the projects which, in our interviewees’ view, can lead to power struggles
between UX and non-UX practitioners: ‘‘sometime it can be perceived as we’re trying to
take control of the situation.’’ (H-1/HCI). This interviewee further emphasized this often
means that working with UX is more difficult than usability: ‘‘I think the reason why it
was easier to work with usability to some extent was that you didn’t take up any space. It was
like being a woman in the early 20th century. You were there, but you didn’t vote, you didn’t
do anything.’’ (H-1/HCI). Similarly another interviewee said: ‘‘There are a lot of strong
stakeholders that are really interested in doing those kind of things, programmers for instance,
who like to be in control.’’ (A-2/HCI).

Power struggles between UX and non-UX practitioners were also attributed to different
motivations of practitioners (e.g., a developer wanting to develop more efficient code vs. a
designer wanting to create a better design). Regarding this, a UX practitioners emphasized:
‘‘Everyone wants to start with their own domain, as soon as possible, from day one. Then, of
course, you have the problem of ownership of the direction, where to go and what to do and
why. That’s something that we struggle with quite a bit.’’ (E-1/HCI). Similarly, another
interviewee said: ‘‘I think there is an ongoing struggle, cause we all have different drivers; like
my driver is always about making technology that is interesting, useful, and brings some sort
of value to the users in the context in which it is being used.’’ (H-1/HCI). This interviewee
further emphasized the situation has changed in their work as they moved from usability
towards UX in recent years: ‘‘Compared to when we only worked with usability, I would
argue that the situation is rather different, and part of it has to do with, being part of the

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 25/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


early phases, but I think the biggest challenge and the biggest struggle nowadays is really the
ownership of decision making, and who calls the shots ...Because usability didn’t disturb the
big process, the big decision-making, the prioritization of what to do, or what constituted a
feature that was needed to be implemented. Usability wasn’t part of that discussion. Usability
was always part of like, how you should make a font, or the colors, like lipstick on the pig!’’
(H-1/HCI).

Better UX work requires regular communication and collaboration among UX and
non-UX practitioners which sometimes can be challenging since these two groups of
practitioners often have different responsibilities, education, motivation, and constraints
in their work. Regarding the importance of communicating with designers a developer
stated: ‘‘I think we need to be working tighter with the design department to help them know
what can be done.’’ (D-1/SE). Similarly, a UX practitioner said: ‘‘we really try to make
this give and take ...we have constant communication, and I will say that we get always input
from developers that we need to consider.’’ (A-2/HCI).

According to the UX practitioners, if they are disconnected from the technical roles,
e.g., developers, they may not be aware of technological constraints when choosing and
developing a design concept: ‘‘honestly the further we are away from the people that actually
build the stuff, we run the real risk of becoming hand waving idiots.’’ (G-1/HCI). Another
UX practitioner stated: ‘‘I realized quite early in my career that I have to communicate with
these guys who program or develop something, and I have to understand what they’re saying.’’
(A-4/HCI). The interviewees however emphasized that overcoming this communication
gap should be a two-way effort: ‘‘I’m not saying that we should be the only ones with this
kind of multidisciplinary approach. I think the other ones should also have that, that’s a big
challenge, I would say.’’ (H-1/HCI). Similarly, another practitioner said: ‘‘So I have to have
some sort of knowledge about the technology because I have to know what my limitations are
.... I have to have some sort of technical know-how so I can communicate with developers. I
expect the same from them. So they realize that the aesthetic choice has to be made and it can
take time.’’ (A-4/HCI).

According to the practitioners, in order to facilitate a better communication, UX
practitioners need to acquire basic knowledge about various technical topics., e.g.,
programming, testing, architecture etc. As emphasized by one respondent: ‘‘You have
to be like knowledgeable in many areas...you have to be very holistic in the way you think
about things, cause you have to speak with programmers in the language of programmers, to
some extent...You also have to understand the business.’’ (H-1/HCI). Similarly, another
practitioner stated: ‘‘you have to speak in an engineering language. That’s a real challenge for
UX work, because you always have to translate it to terms that makes sense to an engineer or
economist.’’ (A-1/HCI).

The respondents also emphasized that communication between UX and non-UX
practitioners can be challenging because of lack of trust in UX practitioners. Regarding this,
one interviewee emphasized: ‘‘we have had problems with some of the developers sometimes.
It has been a bit of conflict. I think we have some work to do to really get a ‘we-feeling’ that
we, together, are developing an awesome product.’’ (C-1/HCI). One practitioner attributed
this lack of trust to the fact that the field of UX is a relatively new field and less established

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 26/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


compared to the technical fields, e.g., SE. This also means that UX practitioners are
often younger and less experienced than non-UX practitioners: ‘‘most UX practitioners
are quite young still, I do not think they have been that long in the market for development of
their competencies yet.’’ (A-3/SE). One UX practitioner emphasized that they can gain more
trust over time and as they accomplish more in their work: ‘‘[over time] we are adding on
the pile of what we would call successful things we have done, and of course that gives us a bit
more ‘trust’ I could say.’’ (H-2/HCI).

DISCUSSION
In this section, we answer the research questions and describe the implications of our
findings for research and practice. We use Hassenzahl’s model of UX and his proposed list
of UX characteristics as an analytical lens to investigate the challenges in greater detail.

In answer to RQ1 (what challenges do practitioners face when integrating UX practices
into software development processes and organizations?), we found 8 themes of challenges
as depicted in Fig. 2. Some of these themes are more fundamental and concern the views,
attitudes and knowledge of stakeholders (Theme 1, Theme 2, Theme 3, and Theme 4) while
some others are more tactical (Theme 5, Theme 6, Theme 7, and Theme 8). There is clearly
a multifaceted and complex set of relations between the themes. These interrelations are
discussed here to the extent supported by our data. More empirical data should be gathered
in the future to elaborate and validate these connections.

At a high level, the more fundamental themes can explain the tactical themes. A limited
knowledge of UX (Theme 1) is likely also connected with the low impact of UX models in
industry (Theme 3). In addition, a lack of common reference and knowledge base can lead
to language gap, meaning that the concepts of UX and UX practices can mean different
things to different people. This gap may contribute to communication problems within
software development organizations (Theme 8).

Through a lopsided emphasis on functionality and actual quality characteristics
(Theme 4), we can explain the identified challenges concerning UX-related requirements
(Theme 5) and testing (Theme 6). Requirements and testing challenges can also be
attributed to limited tools, methods and competencies, and communication issues (Theme
3, Theme 7, and Theme 8). Even if companies intend to focus more on UX, they do not
always have the capability (e.g., competencies) to turn this intention into action. Moreover,
even if UX practitioners with the right competencies are present in an organization, they
often face power struggles and lack of trust therefore fail to effectively communicate and
collaborate with non-UX practitioners.

In answer to RQ2 (how do UX challenges relate to challenges of handling software quality
characteristics, in particular usability? ), we found that there are in fact various overlaps
between the identified themes and challenges reported for handling usability and/or
software quality characteristics in general. However, despite similarities at a superficial
level, we differentiate the challenges through the characteristics of UX: subjective, holistic,
context-dependent, temporal and worthwhile. In particular, through these characteristics, we
highlight the inherent differences between UX and other quality characteristics, including

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 27/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


usability, and explain the lopsided focus of the software industry on usability and the
pragmatic aspect of UX.

While previous empirical studies report that usability is not understood properly,
especially among technical stakeholders (e.g., Bak et al., 2008) we observed that in the
studied organizations practitioners have fair knowledge and awareness about usability. This
can be an indication that industrial knowledge and awareness of usability is improving.
In the case of UX, the same level of maturity however is not yet achieved in industry and
organizations struggle with knowledge and awareness concerning (i) the concept of UX
and how it differs from usability, (ii) the potential of UX, in particular its hedonic aspect to
add value to software, (iii) existing UX theories, tools and methods, and how to apply them
in current development processes, (iv) competencies required for UX work, (v) handling
UX-related responsibilities, and (vi) handling communication and collaboration among
UX and non-UX practitioners.

Previous research shows that usability and purpose of use often dominate over UX
in the software industry (Väänänen-Vainio-Mattila, Roto & Hassenzahl, 2008; Kuusinen
& Väänänen-Vainio-Mattila, 2012). However, an explanation for why this is the case is
missing in the current body of knowledge. We observed a similar trend in the studied
organizations. We additionally found explanations for this lopsided focus by investigating
how the characteristics of UX impact day-to-day work of practitioners. These characteristics
were in fact also highlighted by the interviewees when describing the challenges or
discussing their work with UX mainly in comparison to usability. Some examples of
the interviewees’ quotations in relation to these characteristics are shown in Table 3. It is
not surprising that these characteristics were pointed out only by those interviewees who
have knowledge and experience on UX. Below, we clarify how our practitioners experience
the implications of these characteristics in their work with UX (see Table 4 for a summary).
In total, we identified 20 such difficulties that mainly concern requirements and testing
activities in software development. As shown in Table 4, 11 of these implications have been
previously reported in literature. However, to the best of our knowledge, the remaining nine
implications are unique to our study. Clearly, our study did not focus on identifying these
implications. Further research is needed to provide a more comprehensive understanding
of them, and their interconnections.

The holistic nature of UX requires addressing not only do-goals but also be-goals.
Practitioners however believe that eliciting be-goals and translating them into requirements
and design solutions is more difficult than for functionality or other quality characteristics.
One explanation may be that practitioners disagree on whether they should identify and
refine these be-goals in parallel, before or after other do-goals, including functionality.
Another explanation may be that these practitioners have limited access to guidelines, tools
and methods for treating be-goals in their word.

Previous research (Law, Van Schaik & Roto, 2014) emphasizes that still limited practical
guidelines exist on how to choose suitable UX measures and metrics for UX underlying
elements, and interpret their findings to improve the overall UX. We additionally found
that this challenge also negatively impacts requirements and design. UX requirements,
design and evaluation methods are still developing. Methods such as emocards

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 28/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Table 3 UX characteristic and their implications for the identified challenges as supported by our data.

Characteristic Sample quotes regarding challenges

Subjective ‘‘it is difficult to sort out or really get the correct feeling that the user has because they will try to explain it but it perhaps
is not the real emotion that we catch at the end, if it would be a method or approach to really get these from the user that
would be great.’’ (A-2/HCI)

Holistic ‘‘Functional requirements are easy to create, to merge into a design; more emotional things are more difficult.’’ (F-1/HCI)
Dynamic ‘‘It’s a fantastic feeling to sit in a brand-new car in a car shop. How do you feel about the same car if you’ve had it for a year,

driving it around with two kids in the backseat, for instance? Experiencing all of the quirks from the interior, and all the
things that don’t work, the stupid things that make your hands go dirty every time you’re supposed to close a door, for in-
stance? That’s part of user experience design. So it’s a very floating scale.’’ (H-1/HCI)

Context-dependent ‘‘I think user tests tend to be more focused on pure usability. I guess it’s when you’re releasing the product into the wild,
that’s when you start to get maybe the most valuable feedback or the most truthful ones. So it’s often good to make some sort
of what we call a beta product and publish it. Let people interact with it. But that’s a luxury also because that takes time.
Often, in many, many cases, you have to just make something work. In an ideal world you should have a testing phase, of
course. To get the users’ perspective. That should be really nice to have.’’ (A-4/HCI)

Worthwhile ‘‘if I start with ‘how’, I will never get to the ‘why’. If I start with ‘what’, with just making things, I will totally miss every im-
portant point there is. For me it tends to be very, very useful to focus on the why. So if I can sort of see this why, even if it’s
very, very unclear, I can sort of approach this ‘how’ and ‘what’ in a much better way. (G-1/HCI)

Table 4 In relation to the characteristics of UX, we identified the above 20 implications. These implications can explain the extra difficulties that
practitioners face in their work with UX compared to other quality characteristics, including usability.

UX characteristics Implications (i.e., extra difficulties) as perceived by practitioners

• UX relies on human
subjective perception
• UX is temporal and there is
a complex relation between
various episodes of experience
• UX emerges from
complexly intertwined
underlying elements
• UX includes both
hedonic (be-goals) and
pragmatic (do-goals) aspects
• UX is unique and
situated in the context
• UX is worthwhile, about
adding value, and more than
just preventing problems and
frustrations

• UX work is perceived to be morea visible in the development process than usability work (our finding)
• abstract UX-related needs (e.g., be-goals or emotional needs) are difficult to refine and translate into more
concrete ones (our finding)
• it is hard to translate abstract UX-related needs to concrete design solutions (our finding)
• it is hard to frame UX in evaluations because of all the complex underlying dependencies between its elements
(our finding)
• there is a lack of consensus among practitioners on whether abstract UX-related needs should be elicited
before, after or in parallel with other needs (our finding)
• translating abstract UX-related needs (in particular for the hedonic aspect) to measurable requirements is perceived to be more
difficult than for functionalities or other quality characteristics (our finding)
• compared to the case of usability, more power struggles, and disagreements are perceived to rise between UX
and non-UX practitioners (our findings)
• it is perceived to be more difficult to decide between design alternatives and resolve power struggles and disagreements that can
arise between UX and non-UX practitioners (our findings)
• requirements related to the hedonic aspect of UX are either neglected or treated informally (our findings)
• UX-related needs of users are abstract concepts (literature, our finding)
• practitioners do not have access to standardized and agreed upon set of UX measures and metrics (literature, our finding)
• field studies are crucial in UX work, in particular to gather authentic user experiences but are perceived to be
more resource demanding (literature, our findings)
• creating and using more sophisticated prototypes are vital in UX evaluations but are perceived to
require more resources (literature, our finding)
• practitioners are skeptical about effectiveness and reliability of current UX evaluation methods (literature, our finding)
• practitioners do not always have access to enough users (literature, our finding)
• practitioners do not always have access to sophisticated prototypes in earlier phases (literature, our finding)
• practitioners do not often know how to measure the whole UX in relation to its underlying elements (literature, our finding)
• users’ memory of their experience is prone to fabrication and fading (literature, our finding)
• a deeper understanding of the relation between UX and time is missing (literature, our finding)
• a deeper understanding of the relation between UX and memory is missing (literature, our finding)

Notes.
aAdmittedly, we are not drawing any quantitative conclusion from our qualitative data. Here, we merely aim to emphasize the interviewees’ perception of different issues.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 29/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


(Desmet, Overbeeke & Tax, 2001) for gathering users’ momentary emotions about the
interaction and UX curve (Kujala et al., 2011) for long-term UX evaluation have been
introduced but they are not widely accepted in industrial software development. We
therefore highlight a need for more research on developing industry-relevant guidelines,
tools and methods for identifying and selecting underlying hedonic and pragmatic elements
of UX not only for measurement purposes but also for requirements and design activities.

Practitioners were also generally skeptical about the ability to address the subjective
nature of UX in evaluations. For instance, our interviewees were concerned that what
a user remembers about his or her emotions and shares in interviews or observations
may not reflect the reality. Our interviewee’s concern confirms the findings of Law, Van
Schaik & Roto (2014). Practitioners believed that the most truthful data about experience
and perception of users can only be gathered in the field, and through using a functional
version of the product in a real situation. However, field studies are too resource demanding
(Vermeeren et al., 2010) and not feasible in all projects. We additionally found that despite
skepticism about interviews and observations, these methods are common in industry.
Moreover, organizations often resist introducing more novel tools and methods for various
reasons including the cost of these tools and methods, or lack of knowledge and awareness
about their value.

In addition, our interviewees found it difficult to evaluate UX because of its subjectivity.
They emphasize that while there is a great need to involve the end users in evaluations,
accessing them in projects is often rare. Limited access to the end users is a known and
persistent problem in the industry (Bak et al., 2008). However, we argue that compared
to usability work, UX work is, to a larger extent, impacted by this limitation. According
to the current studies, to reduce the impact of subjectivity on evaluations and to generate
less biased and more reliable results (i) more users need to be involved in evaluations for
statistical significance (Law, Van Schaik & Roto, 2014), and (ii) other stakeholders (e.g.,
developers) cannot act as substitutes for users in evaluations because they are biased
towards the software being developed by their organizations (Locascio et al., 2016). On
the other hand, the limitations of access to end users and competencies can be easier to
overcome in the case of usability through involving limited number of users (e.g., involving
five users in usability tests (Nielsen, 1993)) or using internal stakeholders as test subjects
(Locascio et al., 2016).

Also, our interviewees found it difficult to evaluate subjective and holistic experiences
of users because access to functioning or complete prototypes is often rare in earlier
phases of projects. One solution is to apply evaluation tools and methods that do not
require functioning or complete prototypes but rely on practitioners’ imagination of how
users may perceive the design, which is however an open research topic (Vermeeren et
al., 2010). We also saw practical difficulties concerning handling the relation between
different episodes of experience. It is important to take this relation into account in
evaluations because of the dynamic nature of UX. The practitioners lacked knowledge and
expertise on finding a balance between short- and long-term experiences in both design
and evaluation. Hence, we agree with Law, Van Schaik & Roto (2014) that the community
still lacks enough understanding of the relation between UX, time and memory (the

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 30/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


implications of the subjective and dynamic nature of UX), and suitable UX metrics and
measures that can support this relation are lacking.

The UX practitioners brought up the worthwhile (i.e., positive) nature of UX mainly
when discussing their approach to requirements and design. The UX practitioners believed
this aspect of experience (i.e., the value) or the ‘why’ behind developing the software
for the end users, should be identified before the ‘how’ and ‘what’ that is connected to
usability and functionality. Nevertheless, this view is often in conflict with traditional
and common approaches to software development that focus on identifying functionality
or other quality characteristics (Chung et al., 2000). The practitioners also pointed to the
worthwhile nature of UX when describing their way of working and priorities. While the
UX practitioners emphasize the software value from both the user and business perspective,
non-UX practitioners often tend to merely focus on the business perspective which can
create tensions between UX and non-UX practitioners. In particular, since practitioners still
do not have enough knowledge, or tools and methods to make informed trade-offs between
these two perspectives. For instance, to deliver a certain experience, UX practitioners may
suggest one design solution to support specific user needs, but business stakeholders may
suggest another solution to support specific business goals.

While previous empirical studies report limited access to usability professionals (e.g.,
Bak et al., 2008), the organizations we studied seem to have enough access to usability
competencies which may indicate a general improvement of usability work in the industry.
However, these organizations suffer from limited access to UX competencies, in particular
concerning the hedonic aspect of UX and addressing various be-goals. To overcome the
lack of UX and usability competencies, current research suggests simplifying well-known
testing methods and training developers to perform them (Oevad & Larsen, 2016). While
this approach is shown to be useful in the case of usability and the pragmatic aspect of UX
(Locascio et al., 2016; Oevad & Larsen, 2016), this is not necessarily the case for the hedonic
and more subjective aspect of UX. For instance, as mentioned before, developers’ bias is
shown to compromise the validity of testing results because developers are loyal to the
products they develop and their organizations (Locascio et al., 2016).

In the studied organizations, the responsibilities of usability professionals are well
understood and defined while there are uncertainties concerning the UX-related
responsibilities. In addition, our interviewees perceive that power struggles appear more
often and are more difficult to resolve for UX than for usability. Previous studies report
power struggles between developers and designers as a challenge to UX and usability work
(Chamberlain, Sharp & Maiden, 2006) but do not give explanations for why this is the
case. Here, we give more empirical support for this challenge, and additionally provide
explanations for it.

One explanation can be that UX responsibilities are diverse and can overlap the work
of non-UX practitioners such as business and requirements analysts. Another explanation
is that all stakeholders experience different products and services on a day-to-day basis.
Therefore, non-UX practitioners can easily have opinions about what experiences the
software should deliver, and how they should be delivered, i.e., through which design

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 31/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


solutions. This can lead to power struggles even in cases when there is no overlap between
UX and non-UX responsibilities.

Another explanation may be that UX practitioners should be involved early on and
throughout projects, and their responsibilities involve strategic decision making hence the
work of UX practitioners is more visible in projects as our interviewees emphasized. The
practitioners believed they can partially overcome the power struggles by increasing UX
knowledge and awareness in organizations, and informing other stakeholders about UX
responsibilities (what, why and how of UX practices), especially in relation to the overall
strategies of the organization.

In contrast to what we discussed above for UX, practitioners can test other software
quality characteristics in labs, and even without necessarily using a sophisticated prototype
of the software or even involving the end users. For example, usability can be tested
on simple paper prototypes, using heuristic or other expert evaluations, or performance
problems can be minimized through the use of software performance prediction approaches
in earlier phases (Balsamo et al., 2004). In the case of other software quality characteristics,
the focus is on actual measures and values not the users’ subjective perception of these
values. Additionally, in contrast to UX, other quality characteristics are not dependent
on time. For instance, performance or security measurements will have the same results
even when repeated over time, provided that the software, and the test context (e.g., CPU
load) have not changed. In other words, the difficulties we discussed above concerning
requirements and testing of UX are either less severe or not even present for other quality
characteristics, making their practice less challenging as perceived by our participants.
Limited access to the end users and competencies are also less critical when addressing
usability.

Power struggles can be resolved using objective evidence that shows why an alternative
is better than another. This can for instance be achieved through measurements which is
possible for other quality characteristics, including usability, at least in theory. However,
as we saw, measuring UX is difficult and even impossible in earlier phases. In addition,
UX metrics and measures are not agreed upon or standardized, yet for other quality
characteristics, practitioners have access to standards, e.g., ISO/IEC 9126-4 (International
Organisation for Standardisation (ISO), 2004). Therefore, in the case of UX, it is difficult to
decide among various alternatives in order to resolve the power struggles.

Furthermore, the importance of the pragmatic and hedonic aspects of UX varies in
different projects. For instance, maximizing efficiency is not always relevant for the
software being designed, and not every software is required to support different be-goals
such as curiosity, motivation, interest or fun. Similarly, different characteristics of UX have
different importance in different projects. While the first impression (i.e., temporality)
may be more important for an e-commerce software, an e-learning software may require
more focus on motivation (i.e., situatedness). We however argue that practitioners need
to have sufficient knowledge and awareness on the hedonic and pragmatic aspects, UX
characteristics, and be-goals to be able to make an informed decision regarding their
relevance and importance in various projects.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 32/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Our study contributes to the current body of knowledge through identifying the overlaps
in challenges of UX and other quality characteristics, in particular usability. These overlaps
underline the significance of these challenges and call for more research on how to overcome
them in practice. By highlighting these overlaps, we can help identify research areas for
improving the work of UX as well as software quality in general. We can also make it easier
for practitioners to spot, better understand, as well as find mitigation strategies for UX
challenges by learning from past experiences and developments in the area of software
quality. In addition, our study contributes to the current body of knowledge through
connecting the challenges to the UX characteristics and clarifying the implications of these
characteristics in day-to-day work of practitioners, in particular in comparison to usability
and other quality characteristics.

Based on our work, there are at least three future studies that could be particularly
interesting. First, it would be relevant to conduct an in-depth study to investigate various
approaches that the academics or practitioners propose or apply to address the identified
challenges, most importantly in relation to the characteristics of UX, and different software
development cultures. Second, it would also be valuable to examine the differences between
UX and other quality characteristics, and their associated practices in more depth through
a deeper case study. We have reported on the similarities and differences between UX
and other quality characteristics to the extent supported by our data. Because of the
explorative nature of this study our data does not support details about how the studied
organizations handle various quality characteristics in their day-to-day work. Third, we did
not study how the type of software being developed (e.g., leisure vs work) may impact the
identified challenges in the studied organizations. Investigating the correlation between the
types of software and practices and challenges is an interesting future research direction.
Investigating this correlation can help organizations better understand the cause and effect
of the challenges and better plan for addressing them.

CONCLUSION
Designing and developing for UX is a multidimensional and multidisciplinary activity which
has not yet gained an established position in the software industry. Many development
companies still fail to deliver good UX in their products and services, and practitioners in
these companies face various challenges when dealing with UX throughout their projects.
Although UX is different from usability, many of the present frameworks and guidelines
for UX integration do not clearly separate these two concepts. Moreover, the perception of
UX generally differs in academic and industrial contexts: whereas the former concentrates
on the hedonic aspect and emotions, the latter focuses more on functionality and usability
issues. Our study shows that approaches for integrating UX into software development
require further development. In particular, we showed that the characteristics of UX
(Hassenzahl, 2003), i.e., subjective, holistic, dynamic, context-dependent, and worthwhile
play an important role in shaping UX challenges.

We provided a deeper analysis of UX challenges through the lens of the aforementioned
UX characteristics. These characteristics have at least 20 implications for the day-to-day

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 33/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


work of practitioners. To the best of our knowledge, nine of these implications have
not been previously reported, and are unique to our study. Through these implications,
we explained the extra difficulties practitioners face in their work with UX compared
to usability and other quality characteristics. We could also explain the lopsided focus
of industry on the pragmatic aspect of UX. The related work does not provide such an
analysis nor does it often make such a differentiation between UX and usability except for
the few studies we summarized the related work. We therefore bring depth to the identified
challenges by presenting their relation to the UX characteristics. We hope to have helped
the community to identify ways to systematically improve the current UX.

To make future progress with integrating UX practices into software development
processes, the community needs to take into account these UX characteristics and their
implications for the day-to-day work of practitioners when developing guidelines, tools
and methods to address related challenges. Practitioners need to be aware of the UX
characteristics not only in evaluations but also requirement and design activities. It is also
of crucial importance to understand and address the impact of these characteristics on
communication and collaboration between UX and non-UX practitioners. Admittedly,
the importance of the pragmatic and hedonic aspects of UX is different in projects.
Nevertheless, practitioners need to be aware of the differences between these aspects, and
various UX characteristics to make an informed decision in each project regarding where to
focus and why. The challenges and implications we present in this paper can guide future
research on UX integration and creating more awareness concerning UX, and related
challenges in the software industry. Our findings can also shed light on the more general
problem of how practitioners should integrate new and less mature knowledge areas, of
which software quality characteristics and UX are examples, into software development
processes.

ACKNOWLEDGEMENTS
The authors would like to thank all the organizations and in particular practitioners who
participated in this study.

APPENDIX A. INTERVIEW GUIDE
General questions
• What is your education and work background?
• What is your role in this company?
• How many years have you had this role?
• Do you know any of these terms (see Appendix C)? if yes, how do you apply them in
your work?

• How do you define UX?
• How do you define Usability?

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 34/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Questions related to requirements
• How is the overall requirements process in your company?
• How do you approach functional requirements?
• How do you approach non-functional/quality requirements?
• How do you approach requirements related to UX?
• What challenges do you face in your work regarding requirements related to UX?

Questions related to design
• How is the overall design process in your company?
• How is ‘design’ related to ‘requirements’ in your work?
• What challenges do you face in your work regarding design, in particular in relation to
UX?

Questions related to evaluation or testing
• How is the overall evaluation/testing process in your company?
• How do you test functional requirements?
• How do you test non-functional requirements?
• How do you test UX?
• What challenges do you face in testing UX, or requirements related to UX?

APPENDIX B. CODING GUIDE
• Every segment can have any number of applicable codes
• The codes should be selected from the list below. If a new concept appears in data the
possibility of adding a new code should be discussed among the authors.

• Any uncertainty in coding a segment should be discussed among the authors.

List of codes
1. Challenges
2. Solutions
3. UX
4. Usability
5. UX vs Usability
6. Motives
7. Definition
8. Organization
9. Project
10. Software
11. Process
12. Individuals
13. Tools and methods
14. Roles
15. Responsibilities
16. Collaboration

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 35/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


17. Communication
18. Requirements
19. Evaluations

APPENDIX C. TERMINOLOGY TABLE
The interviewees were asked to specify each and every term they know and whether they
apply it in their work. They were also asked to add any relevant term that is missing from
the list.

• Usability
• User Experience
• Quality in Use (QiU)
• Emotional design
• Pleasurable design
• Aesthetics of design
• Affective computing
• Affective design
• Usability requirements
• UX requirements
• Affective requirements
• Emotional requirements
• User values
• User emotions
• User motivations
• ISO/IEC 9126
• ISO/IEC 25010
• Hedonic and pragmatic
• Instrumental and non-instrumental

ADDITIONAL INFORMATION AND DECLARATIONS

Funding
The authors received no funding for this work.

Competing Interests
The authors declare there are no competing interests.

Author Contributions
• Pariya Kashfi conceived and designed the experiments, performed the experiments,
analyzed the data, contributed reagents/materials/analysis tools, wrote the paper,
prepared figures and/or tables, performed the computation work, reviewed drafts
of the paper.

• Agneta Nilsson and Robert Feldt conceived and designed the experiments, analyzed the
data, wrote the paper, reviewed drafts of the paper.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 36/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130


Data Availability
The following information was supplied regarding data availability:

The raw data includes the interviewee recordings and transcripts which because of
NDA between Chalmers University and collaborative industries (i.e., companies who
participated in the interviewees) are not allowed to be shared with any third party. More
quotations are included and the coding process is elaborated to address this issue partially.

Supplemental Information
Supplemental information for this article can be found online at http://dx.doi.org/10.7717/
peerj-cs.130#supplemental-information.

REFERENCES
Abrahão S, Juristo N, Law EL-C, Stage J. 2010. Interplay between usability and

software development. Journal of Systems and Software 83(11):2015–2018
DOI 10.1016/j.jss.2010.05.080.

Alves R, Valente P, Nunes NJ. 2014. The state of user experience evaluation practice.
In: Proceedings of the 8th nordic conference on human–computer interaction: fun, fast,
foundational (NordiCHI ’14). New York: ACM, 93–102.

Ardito C, Buono P, Caivano D, Costabile MF, Lanzilotti R, Dittrich Y. 2014. Human-
centered design in industry: lessons from the trenches. Computer 47(12):86–89
DOI 10.1109/MC.2014.355.

Bak JO, Nguyen K, Risgaard P, Stage J. 2008. Obstacles to usability evaluation in
practice. In: 5th Nordic conference on human–computer interaction building bridges—
NordiCHI’08. New York: ACM Press, 23.

Balsamo S, Di Marco A, Inverardi P, Simeoni M. 2004. Model-based performance
prediction in software development: a survey. IEEE Transactions on Software
Engineering 30(5):295–310 DOI 10.1109/TSE.2004.9.

Berntsson Svensson R, Gorschek T, Regnell B, Torkar R, Shahrokni A, Feldt R.
2012. Quality requirements in industrial practice—an extended interview study
at eleven companies. IEEE Transactions on Software Engineering 38(4):923–935
DOI 10.1109/TSE.2011.47.

Boivie I, Gulliksen J, Göransson B. 2006. The lonesome cowboy: a study of the usability
designer role in systems development. Interacting with Computers 18(4):601–634
DOI 10.1016/j.intcom.2005.10.003.

Borg A, Yong A, Carlshamre P, Sandahl K. 2003. The bad conscience of requirements
engineering: an investigation in real-world treatment of non-functional require-
ments. In: Proceedings of the 3rd conference on software engineering research and
practice in Sweden (SERPS’03). 1–8.

Braun V, Clarke V. 2006. Using thematic analysis in psychology. Qualitative Research in
Psychology 3(2):77–101 DOI 10.1191/1478088706qp063oa.

Cajander Å, Lárusdóttir MK, Gulliksen J. 2013. Existing but not explicit—the user
perspective in scrum projects in practice. In: Proceedings of the 14th IFIP TC 13

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 37/40

https://peerj.com
http://dx.doi.org/10.7717/peerj-cs.130#supplemental-information
http://dx.doi.org/10.7717/peerj-cs.130#supplemental-information
http://dx.doi.org/10.1016/j.jss.2010.05.080
http://dx.doi.org/10.1109/MC.2014.355
http://dx.doi.org/10.1109/TSE.2004.9
http://dx.doi.org/10.1109/TSE.2011.47
http://dx.doi.org/10.1016/j.intcom.2005.10.003
http://dx.doi.org/10.1191/1478088706qp063oa
http://dx.doi.org/10.7717/peerj-cs.130


international conference human–computer interaction (INTERACT 2013). Berlin,
Heidelberg: Springer, 762–779.

Chamberlain S, Sharp H, Maiden N. 2006. Towards a framework for integrating agile
development and user-centred design. In: Abrahamsson P, Marchesi M, Succi G,
eds. 7th international conference, XP 2006. Lecture notes in computer science, Oulu,
Finland, 143–153.

Chung L, Do Prado Leite JCS. 2009. On non-functional requirements in software
engineering. In: Borgida AT, Chaudhri VK, Giorgini P, Yu E, eds. Conceptual
modeling: foundations and applications, ser. Lecture notes in computer science. Vol.
5600. Berlin Heidelberg: Springer, 363–379.

Chung L, Nixon BA, Yu E, Mylopoulos J. 2000. Non-functional requirements in
software engineering. Boston: Kluwer Academic Publishing.

Desmet P, Overbeeke K, Tax S. 2001. Designing products with added emotional value:
development and application of an approach for research through design. The Design
Journal 4(1):32–47 DOI 10.2752/146069201789378496.

Ferreira J, Sharp H, Robinson H. 2012. Agile development and user experience design
integration as an ongoing achievement in practice. In: Proceedings of the 2012 agile
conference (AGILE 2012). Piscataway: IEEE, 11–20.

Flick U. 2009. An introduction to qualitative research. 4th edition. London: Sage Publica-
tions.

Gerea C, Herskovic V. 2007. Measuring user experience in latin america: an exploratory
survey. In: Proceedings of the Latin American conference on human computer
interaction—CLIHC’15. New York: ACM Press, 1–4.

Gulliksen J, Boivie I, Persson J, Hektor A, Herulf L. 2004. Making a difference: a survey
of the usability profession in Sweden. In: Proceedings of the 3rd Nordic conference on
human–computer interaction (NordiCHI’04). New York: ACM, 207–215.

Hassenzahl M. 2003. The thing and I: understanding the relationship between user and
product. In: Blythe MA, Monk AF, Overbeeke K, Wright PC, eds. Funology: from
usability to enjoyment. Dordrecht: Springer Netherlands, 31–42.

Hassenzahl M. 2010. Experience design: technology for all the right reasons. San Francisco:
Morgan & Claypool.

Hassenzahl M, Burmester M, Koller F. 2003. AttrakDiff: ein Fragebogen zur Messung
wahrgenommener hedonischer und pragmatischer Qualität (Attraktif: a question-
naire for the measurement of perceived hedonic and pragmatic quality). In: Ziegler
J, Szwillus G, eds. Mensch & computer 2003: interaktion in bewegung. Stuttgart: B.G.
Teubner, 187–196.

Hassenzahl M, Diefenbach S, Göritz A. 2010. Needs, affect, and interactive products—
facets of user experience. Interacting with Computers 22(5):353–362
DOI 10.1016/j.intcom.2010.04.002.

Hassenzahl M, Tractinsky N. 2006. User experience—a research agenda. Behaviour &
Information Technology 25(2):91–97 DOI 10.1080/01449290500330331.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 38/40

https://peerj.com
http://dx.doi.org/10.2752/146069201789378496
http://dx.doi.org/10.1016/j.intcom.2010.04.002
http://dx.doi.org/10.1080/01449290500330331
http://dx.doi.org/10.7717/peerj-cs.130


International Organisation for Standardisation (ISO). 2004. ISO 9126: software
engineering—product quality—Part 4: quality in use metrics. Geneva: International
Organisation for Standardisation.

International Organisation for Standardisation (ISO). 2011. ISO 25010: systems and
software engineering—Systems and software Quality Requirements and Evaluation
(SQuaRE)—System and software quality models. Geneva: International Organisation
for Standardisation.

Isomursu M, Sirotkin A, Voltti P, Halonen M. 2012. User experience design goes agile in
lean transformation—a case study. In: Proceedings of the 2012 agile conference (AGILE
2012). Piscataway: IEEE, 1–10.

Jordan PW. 2002. Designing pleasurable products: an introduction to the new human
factors. Boca Raton: CRC Press.

Karlsson L, Dahlstedt Å. G, Regnell B, Natt och Dag J, Persson A. 2007. Requirements
engineering challenges in market-driven software development—an interview
study with practitioners. Information and Software Technology 49(6):588–604
DOI 10.1016/j.infsof.2007.02.008.

Kashfi P, Feldt R, Nilsson A, Svensson RB. 2016. Evidence-based timelines for user
experience software process improvement retrospectives. In: 42nd euromicro
conference on software engineering and advanced applications. Piscataway: IEEE.

Kujala S, Roto V, Väänänen-Vainio-Mattila K, Sinnelä A. 2011. Identifying hedonic
factors in long-term user experience. In: Proceedings of the 2011 conference on
designing pleasurable products and interfaces (DPPI ’11). New York: ACM, 17:1–17:8.

Kuusinen K, Väänänen-Vainio-Mattila K. 2012. How to make agile UX work more
efficient: management and sales perspectives. In: Proceedings of the 7th nordic
conference on human–computer interaction: making sense through design (NordiCHI
’12). New York: ACM, 139–148.

Lanzilotti R, Costabile MF, Ardito C, Informatica D, Aldo B. 2015. Addressing usability
and UX in call for tender for IT products. In: Proceedings of the 15 th IFIP TC 13
international conference human–computer interaction (INTERACT 2015), 1–8.

Law EL-C, Abrahão S. 2014. Interplay between User Experience (UX) evaluation and sys-
tem development. International Journal of Human-Computer Studies 72(6):523–525
DOI 10.1016/j.ijhcs.2014.03.003.

Law EL-C, Van Schaik P, Roto V. 2014. Attitudes towards user experience (UX)
measurement. International Journal of Human-Computer Studies 72(6):526–541
DOI 10.1016/j.ijhcs.2013.09.006.

Locascio J, Khurana R, He Y, Kaye J. 2016. Utilizing employees as usability participants:
exploring when and when not to leverage your coworkers. In: Proceedings of the 2016
CHI conference on human factors in computing systems. New York: ACM, 4533–4537.

Nielsen J. 1993. Usability engineering. Cambridge: Academic Press Inc.
Nuseibeh B, Easterbrook S. 2000. Requirements engineering: a roadmap. In: Proceedings

of the conference on the future of software engineering–ICSE’00. Vol. 1. New York:
ACM Press, 35–46.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 39/40

https://peerj.com
http://dx.doi.org/10.1016/j.infsof.2007.02.008
http://dx.doi.org/10.1016/j.ijhcs.2014.03.003
http://dx.doi.org/10.1016/j.ijhcs.2013.09.006
http://dx.doi.org/10.7717/peerj-cs.130


Oevad T, Larsen LB. 2016. How to reduce the UX bottleneck—train your soft-
ware developers. Behaviour & Information Technology 35(12):1080–1090
DOI 10.1080/0144929X.2016.1225818.

Ovad T, Larsen LB. 2015. The prevalence of UX design in agile development processes in
industry. In: Proceedings of the 2015 agile conference (AGILE 2015). Piscataway: IEEE,
40–49.

Paech B, Kerlow D. 2004. Non-functional requirements engineering—quality is
essential. In: Proceedings of the 10th international working conference on requirements
engineering: foundation for software quality (REFSQ’04), 237–250.

Rosenbaum S, Rohn JA, Humburg J. 2000. A toolkit for strategic usability. In: Proceed-
ings of the SIGCHI conference on Human factors in computing systems (CHI’00). New
York: ACM, 337–344.

Runeson P, Höst M. 2008. Guidelines for conducting and reporting case study re-
search in software engineering. Empirical Software Engineering 14(2):131–164
DOI 10.1007/s10664-008-9102-8.

Väänänen-Vainio-Mattila K, Roto V, Hassenzahl M. 2008. Now Let’s do it in practice:
user experience evaluation methods in product development. In: Proceedings of
extended abstracts on human factors in computing systems (CHI’08). New York: ACM,
3961–3964.

Vermeeren APOS, Law EL-C, Roto V, Obrist M, Hoonhout J, Väänänen-Vainio-Mattila
K. 2010. User experience evaluation methods: current State and Development needs.
In: Proceedings of the 6th nordic conference on human–computer interaction extending
boundaries (NordiCHI ’10). New York: ACM, 521–530.

Wright P, McCarthy J. 2010. Experience-centered design: designers, users, and com-
munities in dialogue, ser. In: Synthesis lectures on human-centered informatics. San
Francisco: Morgan & Claypool.

Zimmermann PG. 2008. Beyond usability-measuring aspects of user experience. PhD
Dissertation, Swiss Federal Institute of Technology Zurich.

Kashfi et al. (2017), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.130 40/40

https://peerj.com
http://dx.doi.org/10.1080/0144929X.2016.1225818
http://dx.doi.org/10.1007/s10664-008-9102-8
http://dx.doi.org/10.7717/peerj-cs.130