key: cord-0044008-impngwp6
authors: Roelens, Ben; Bork, Dominik
title: An Evaluation of the Intuitiveness of the PGA Modeling Language Notation
date: 2020-05-05
journal: Enterprise, Business-Process and Information Systems Modeling
DOI: 10.1007/978-3-030-49418-6_27
sha: 6a85f11347dd88632c5e1128cf674a098724632b
doc_id: 44008
cord_uid: impngwp6

The Process-Goal Alignment (PGA) modeling method is a domain-specific modeling language that aims to achieve strategic fit of the business strategy with the internal infrastructure and processes. To ensure the acceptance and correct understanding of PGA models by business-oriented end-users, an intuitively understandable notation is of paramount importance. However, the current PGA notation was not formally tested up to now. In the paper at hand, we apply an evaluation technique for testing the intuitiveness of domain-specific modeling languages to bridge that research gap. Based on an analysis of the tasks, we propose improvements to six elements of the initial PGA notation. Our research contributes a comprehensive description of the empirical modeling language evaluation, which enables the reproducibility of the evaluation procedure by the conceptual modeling community.

The design of Domain-Specific Modeling Languages (DSMLs) is gaining popularity in the Conceptual Modeling field [9] . DSMLs are specifically designed for a specific purpose (e.g., to analyze and communicate about a problem) in a particular domain [6] . In comparison to General Purpose Modeling Languages (e.g., UML or ER), DSMLs reduce the complexity of the modeling effort by using concepts that are familiar to the intended end-users and by hiding complex model constraints in the tailored meta-model [6] .

The Process-Goal Alignment (PGA) modeling method is a specific DSML, which has the purpose to achieve strategic fit in the business architecture [15] . Strategic fit is an important architectural concern for organizations, as it requires to align the business strategy with the internal infrastructure and processes [7] . This enables companies to adequately react on opportunities and threats in its external environment. The design of the PGA modeling method is the result of different iterations of Action Design Research [17] , which allowed the gradual refinement of its syntax, semantics, and modeling procedure [15] .

One of the design requirements of the PGA method is a clear communication of the organizational strategy to ensure its understanding by business-oriented experts [15] . These experts are not applying the method themselves, but are guided by a modeler, who collects the necessary information and constructs the PGA models. Therefore, ensuring that PGA models can be intuitively understood by business-oriented end-users is of paramount importance to reduce the cognitive load for them. This will foster the use of the models to identify possible organizational improvements. To realize this, the PGA notation was initially guided by the principle of semantic transparency. This principle imposes that the graphical notation of a modeling language element suggests its meaning [10] . However, the intuitiveness of the PGA notation was not tested yet.

This research gap can be solved by an evaluation technique for testing the intuitiveness of DSMLs [3] . The technique comprises a set of tasks which are divided into three phases: (i) term association, (ii) notation association, and (iii) case study. These tasks were conducted by Master students of Ghent University to test the intuitiveness of the PGA notation. Based on an analysis of the results, improvements to the initial notation are proposed.

The paper is structured as follows. Section 2 reviews foundational literature about conceptual modeling, modeling language notations, and the PGA modeling method. Afterwards, Sect. 3 describes how the evaluation technique and the data analysis were performed. In Sect. 4, the results of the analysis are presented, which leads to the proposal of an improved PGA notation in Sect. 5. The paper ends with a reflection and concluding remarks in Sect. 6.

Conceptual modeling concerns the application of abstraction to reduce the complexity of a certain domain for a specific stakeholder purpose. Originally, conceptual modeling was targeting to support human beings for the purposes of a common understanding and communication [11] . In recent years, the scope of the discipline has exceeded this pure representative means toward using the created models as a formalized knowledge base that enables automated processing. Modeling methods are the core of conceptual modeling [8] , which consist of: a modeling language, a modeling procedure, and mechanisms & algorithms. The modeling language encompasses its syntax, semantics, and notation. The syntax defines the grammar of the language, which includes the available concepts and the allowed relationships between them. Semantics refers to the meaning, whereas notation is the visual representation of the language concepts.

Information processing can be divided into two steps [12] : Perceptual Processing (seeing) which is fast and automatic, and Cognitive Processing (understanding) which is slow and resource-intensive. Conceptual models should aim for computational offloading, i.e., replacing some cognitive tasks by perceptual ones. Moody states that "Designing cognitively effective visual notations can [..] be seen as a problem of optimizing them for processing by the human mind" [10, p. 761] .

"The extent to which diagrams exploit perceptual processing largely explains differences in their effectiveness" [10, p. 761 ]. When analyzing the perceptual processing quality of a visual notation, one needs to consider semantic transparency [10] . Semantic transparency is defined as "the extent to which a novice reader can infer the meaning of a symbol from its appearance alone" [10, p. 765 ]. In literature, semantic transparency is often considered synonymous to an intuitive understanding. A notation with a high semantic transparency enables users to infer the meaning of a symbol/model from their working and/or long-term memory. Semantic transparency therefore "plays a crucial role in [...] acceptance" of modeling languages [5, p. 123 ].

PGA has been introduced in [14] and further developed in [15] as a project within the Open Models Laboratory (OMiLAB) [2] . A tool prototype has been realized with the ADOxx meta-modeling platform [1] . To achieve strategic fit in the business architecture, PGA aims at the development of a business architecture heat map following a modeling procedure that consists of three activities: (i) developing a prioritized business architecture hierarchy, (ii) executing the performance measurement, and (iii) performing the strategic fit improvement analysis.

The first step aims to model the creation of value throughout a hierarchical structure of business architecture elements. Based on Strategic Management frameworks, the PGA meta-model incorporates the following elements (i.e., capitalized in the remainder of the text): Activity, Process, Competence, Value Proposition, Financial Structure, Internal Goal, Customer Goal, and Financial Goal. To design an intuitive notation for business-oriented end-users, icons were used to represent these elements. An overview of the initial PGA notation is found in Table 4 . Afterwards, valueStream relations are added between these elements to show the hierarchical value structure. Each valueStream relation is prioritized by using the AHP mechanism (i.e., based on pairwise comparisons) [16] and a color coding with accompanying line texture is used to differentiate between a high (i.e., solid red color), medium (i.e., dashed orange color), or low priority (i.e., dotted green color) w.r.t. their strategic Importance.

The performance measurement mechanism is applied to each business architecture element to identify an appropriate performance indicator, set a performance target and an allowed deviation percentage, and to analyze the actual outcome for each indicator. This enables the differentiation between an excellent, expected, or bad Performance for each element. Following existing heat mapping techniques, bad Performance is visualized by using a solid red, expected Performance by a dashed orange, and excellent Performance by a dotted green color.

The first two activities result in a business architecture heat map (see Fig. 1 for an example 1 ), which can be further analyzed during the strategic fit improvement analysis. The main objective of this analysis is to identify operational changes that could potentially improve the value creation throughout the business architecture to result in a better realization of the strategic objectives. To support this, a critical path can be identified starting from a Goal with a bad Performance (e.g., Improve short-term solvency in Fig. 1 ) as a chain of valueStream relations that have a high or medium Importance and that connect business architecture elements on downstream hierarchical levels of which the Performance can be improved (e.g., Increase current assets, Sale additional products, Operational excellence, and Baking) [15] . A critical path ends at the Activity or Process level of the business architecture (e.g., Preheating), which leads to concrete changes that can be applied to the organizational processes.

In this paper, we applied the evaluation technique of [3] to test the intuitiveness of the PGA notation by the intended end-users 2 . The participants were given 60 min to complete the evaluation questions. This evaluation comprised a set of tasks, which were clustered in three core phases surrounded by an initiation and a conclusion phase (see Fig. 2 ).

Procedure of the evaluation technique [3] Initiation Phase. Participants were briefly introduced to the relevant domain (i.e., strategic fit in the business architecture) and the building blocks of the PGA modeling method without showing any visual aspects like language concepts or sample models. Besides this, some information about the participants was collected. This information comprised demographic aspects (i.e., gender, age) as well as questions regarding the experience of the participant with modeling languages. An informed consent was used to provide the necessary information about voluntary participation to the user study and the anonymous processing of the collected data. Phase 1 -Term Association. Participants were provided terms that refer to names of PGA modeling language concepts. Each participant then individually drafted one or more graphical representations that he/she deems as the most intuitive for the element. Participants received a blank paper with a list of the terms and they were asked to use a variation of colors to perform this task. Phase 2 -Notation Association. Afterwards, samples of the current PGA notation were presented and participants were asked to record up to three intuitive associations that pop out when looking at them. Notably, the notations were presented without any hint of e.g., the name or the semantics of this concept. Furthermore, the concepts forming part of the term association were different to the ones of the notation association to omit hampering intuitiveness. In order to enable a full coverage of the PGA notation, we decided to divide the participants into two groups. Group A had half of the concepts as part of the term association and the other half as part of the notation association. For group B, the order of the phases was the same, but the concepts were divided oppositely between the two tasks. Phase 3 -Case Study. This task included comprehension questions targeting an example of a business architecture heat map (see Fig. 1 ). The legend is added here for clarification but was not provided to the respondents. Each question was oriented towards the identification of particular meta-model elements in the model, e.g., how many elements have a good performance, which type of element is supported by Operational excellence, etc. Concluding Phase. As a last step, participants were asked to provide qualitative feedback and improvement suggestions about the current PGA notation.

Phase 1 -Term Association. For the analysis, one needs to deconstruct the participants' graphical representations according to different visual variables [10] , such as color, shape, icons, texture, or any other visual aspect that pops out. As not all respondents used the full range of visual variables, their relative frequency per meta-model element is analyzed. For example, 75% of the shapes used to depict an Activity is a (rounded) rectangle (see Table 1 ). Phase 2 -Notation Association. The percentage of participants that provided a matching association with the name of the meta-model element is calculated. Besides, the relative rank of this percentage is interesting to analyze, as it is an indicator of the extent to which the PGA element is outperformed by other meta-model constructs. This would show that there is a problem with the perceptual discriminability as the different symbols are not clearly distinguishable from each other in the current notation [10] . Phase 3 -Case Study. Comprehension questions can be answered based on the available information in the sample model. Therefore, it is possible to analyze the given responses and calculate the percentage of participants that provided a correct answer to a certain comprehension question.

All participant responses were digitized and stored in a shared cloud infrastructure. All authors started a pretest for analyzing the results with only a few responses. Afterwards, the gained experience was exchanged to streamline the structure of the analysis, e.g., the visual variables to be applied during the classification of the term associations. Next, the authors independently analyzed all responses, after which the analysis was condensed toward a harmonized result.

The participants were students following a Master level class on IT Management at the Faculty of Economics and Business Administration of Ghent University.

In total, 139 students participated in the user study. The participants were randomly assigned to two different groups (see Sect. 3.1), resulting in 70 participants for group A and 69 for group B. Their average age was 22 years and 41% of them were female. Although the participants were not familiar with the PGA method, 86% had some prior modeling knowledge about ER modeling, 90% about business process modeling, and 34% about ArchiMate. Table 1 shows the deconstruction of the participants' graphical representations according to different visual variables. The relevant variables that were used by participants include color, shape, icons, and text. In this respect, icons are symbols that perceptually resemble the concepts they represent [10] , while shape refers to geometric figures (e.g., square, line). Due to limited space, Table 1 only covers the most used visual variables (i.e., a cumulative frequency of at least 50% if the individual absolute frequency is at least two).

During the term association analysis, some interesting insights are obtained. First, it is observed that the participants dominantly used a blue color to design a graphical representation for a given meta-model element. To efficiently handle a large group of 139 respondents, they were asked to bring a variety of colors. However, we see that the majority of the participants only used one color to draft a notation. This unexpected result is a limitation in the set-up of the evaluation technique.

Concerning shape, some recurring proposals can be seen such as a rectangle, triangle, ellipse, or arrow. The origin of these proposals can be explained by the modeling experience of the participants, which are familiar with ER and business process modeling. Although the intuitiveness of such shapes is quite limited, participants added to them corresponding icons to further shape the meaning of the concepts (e.g., thinking balloon, dollar/euro sign, bull's-eye, graph, exclamation mark, etc.). These icons seem to provide an important instrument for participants to design an intuitive notation.

In line with the principle of Dual Coding [10] , participants used text to complement the proposed graphics, which enabled them to depict the meaning of an element both visually and verbally. In most cases, the text is equal to the first letter of the meta-model element (e.g., C for Competence) or its complete name (e.g., Activity). In other cases, text refers to the content of a meta-model element. This is the case for Financial structure, which is a representation of the cost and revenue structure (e.g., as coded by C & R) that is implemented by an organization.

Finally, number and graphical position were used for specific meta-model elements. More specifically, participants employed spatial enclosure to represent activities as a subset of the overarching process. In this case, we see a dominant proposal of three rectangles connected by arrows. Graphical position is also proposed for the representation of a valueStream. In this case, participants use a hierarchy of arrows to depict the value creation throughout the business architecture. Table 2 shows the results of the notation association task. For each element, the percentage of participants giving a matching association and the relative rank of this association is listed. Important to note here is that the visualization of the valueStream relation (i.e., a non-directed line, see Table 4 ), was not explicitly tested as the meaning of this relation only becomes clear when included in a hierarchical business architecture heat map. The percentage of correct associations ranges between 0% and 36.23%. The PGA concepts Activity (24.29% -rank 1), Process (36.23% -rank 1), Financial Structure (12.75% -rank 3), and Financial Goal (20.29% -rank 2) perform the best as we analyze both the percentage and the relative rank of the correct associations. The notation of the other elements is less intuitive, as the percentages are below 5%. Moreover, some of them are outperformed by other meta-model elements. More specifically, the Competence notation (i.e., a stage icon) is con- 

In Table 3 , the results of the case study are given. To keep this example manageable for participants in the given time, only one type of Goal (i.e., Financial Goal) was included in the sample model. This is the reason why no results are available for Customer and Internal Goal in Table 3 . Although all questions were oriented towards the identification of meta-model elements, partially correct answers could also be identified. These include naming elements at the instance level (e.g., Take sample instead of Activity) or using close synonyms for the meta-model element (i.e., Task as a synonym for Activity). Besides, there was not a question that directly targeted the identification of a valueStream, but problems with the intuitiveness of this relation can be derived from incorrect answers to the questions about the Activity and Value Proposition concept. More specifically, some incorrect answers indicate that the valueStream relation was interpreted in the wrong direction.

Although the mean score of complete correct answers for this task is 41.32%, Table 3 shows that the meaning of the Value Proposition (i.e., 5.04% correct answers) and Importance (i.e., 5.76% correct answers) notation cannot easily be derived from the business architecture heat map. Even if partially correct answers are included, these elements are the two least performing of all PGA concepts with total scores of 21.59% for Value Proposition and 14.39% for Importance. Besides, there seems to be a problem with the intuitiveness of the valueStream notation, which was read in the wrong direction in the Activity and Value Proposition question by respectively 18.71% and 27.34% of the participants. As one can notice, the scores for Financial Structure and Financial Goal are the same, as the identification of these meta-model elements was included in one question during the case study task.

During the conclusion phase, we obtained 104 remarks from 58 unique participants (i.e., a response rate of 41.73%). Of the responses, 45 could be specifically traced back to the PGA meta-model, distributed among the aspects color and line style (24 remarks), Importance (12 remarks), valueStream (5 remarks), and Activity (4 remarks). As can be seen in Table 4 , color and line style refer both to Performance and Importance in the PGA meta-model. We provide illustrative feedback in the following.

-Color & line style: "Using colors is a good idea, it gives a nice and quick overview." "The meaning of the different colors & line styles is not clear." -Importance: "It is not clear what the numbers next to the relations mean." -valueStream: "It is difficult to see where certain value streams go to." -Activity: "The model would improve if the total process of how the organization operates was represented."

We conclude the design cycle by proposing directions for improving the PGA notation. This proposal is based on the combined evaluation results discussed previously. In particular, we distinguish between (i) no change is required and (ii) the suggestion of a new notation. In the first case, no change is required as the results confirm the intuitiveness of the initial notation. In the latter, we use (some of) the suggestions of the participants to propose a new notation. For some elements, this also includes changes aimed at the homogenization of the notations of all PGA elements. For the PGA elements Activity, Process, Financial Structure, Financial Goal, and Performance, we propose to preserve the initial notation based on the analysis of the results. The notation association, case study, and qualitative feedback confirm the intuitiveness of these elements. Moreover, the suggested notations of the term association phase only include generic shapes (i.e., rectangle and ellipse) with no or recurring icons (i.e., dollar/euro sign). Following these suggestions would have a negative impact on the perceptual discriminability between Activity and Process on the one hand, and Financial Structure and Financial Goal on the other hand. Besides, we understand the qualitative feedback about the lack of a complete process description in the PGA models. However, this is a deliberate design choice of the modeling method as the main purpose of the business architecture heat maps is to achieve alignment between the different layers. Therefore, it is not always needed to offer a complete view on the business architecture, as this may hamper the understanding of the models [15] . Performance is an exception in the analysis, as it combines a low score for the notation association (i.e., 0%) with a score of 81.29% for the case study. This can be explained by the fact that Performance is implemented as an attribute to the other PGA meta-model elements. Consequently, the meaning of the color coding only becomes intuitive when implemented in a complete business architecture heat map (see Fig. 1 ). Qualitative feedback further confirmed that the use of color enables to give a nice and quick overview of alignment opportunities in the business architecture.

The main argument to propose a new notation for Competence is the confusion that the initial one causes for end-users. Indeed, during the notation association phase, it became clear that people naturally attach the meaning of Performance to the visualization. Based on the suggestions of the participants during the term association task, we propose a combination of a person and light bulb icon as the new notation (see Table 4 ). This notation should refer to the cognitive abilities that are associated with the definition of a Competence as the internal knowledge, skills and abilities of an organization.

A new notation for Value Proposition is also proposed in Table 4 , as the initial notation was one of the least performing PGA elements during the notation association (i.e., 2.83%) and case study (i.e., 5.04%) tasks. However, the suggested icons by participants do not show a clear preference as they are closely related to financial elements (i.e., dollar/euro or + sign) or cognitive abilities (i.e., light bulb). Therefore, the new notation is a gift that is exchanged between two hands. We believe this provides a more intuitive notation for the products and services that are exchanged between a company and its customers. This proposal is in line with the notation of a Value Proposition in the Business Model Canvas (i.e., a gift icon) [13] . The notation of an Internal Goal needed improvement as respondents confused it with processes during the notation association phase. As the current PGA notation already includes a bull's-eye to represent the goal aspect, the analysis of the term association task did not provide further concrete suggestions. To stress the internal characteristic of the term, it was decided to graphically enclose cog wheel icons (i.e., the initial notation) inside a factory icon (see Table 4 ).

A similar argument can be provided for the new notation of a Customer Goal. This element scored low (i.e., 0%) in the notation association task, which clearly shows that the intuitiveness of the current notation needs improvement. Based on the suggestions of the term association phase, it was decided to clearly represent the customer by a person icon in the new notation. To stress the meaning of a customer, the person is shaking hands with another person, holding a briefcase (see Table 4 ).

Problems with the understanding of the valueStream relation became apparent during the case study, in which a large part of the participants applied it in the wrong direction. Furthermore, some of the qualitative feedback confirms that the direction of the valueStream is not clear. In line with the term association task, this issue is solved by using an arrow in the newly proposed PGA notation. This arrow points towards the element in business architecture heat map, of which the value creation is supported by an element on the downstream hierarchical level.

A last change that is proposed concerns the Importance element. In the initial notation, this attribute was visualized by a colored valueStream accompanied by a certain texture. Furthermore, we added a number (i.e., showing its relative importance) to this visualization as a form of dual coding. However, the results of the different evaluation tasks showed that this notation was poorly understood by participants. This was also confirmed by the qualitative feedback about the confusing color coding and numbers of the valueStream relations. A first improvement could be identified based on the term association phase, of which the results show that an exclamation mark is an intuitive way of representing Importance. We combine this suggestion by replacing the color coding by a different thickness of the valueStream relations. As a result, a valueStream with a high Importance will be depicted by a thick arrow, combined with three exclamation marks. The thickness and number of exclamation marks decreases for a valueStream with a medium or low Importance (see Table 4 ).

This paper describes the execution of an evaluation technique [3] to test the intuitiveness of the initial PGA notation [15] . This evaluation was needed to validate the communication potential of PGA and to improve the understanding and acceptance of the resulting models by business-oriented end-users. The evaluation tasks were performed by 139 Master's students of Ghent University with an elaborate economical background and basic modeling experience. The analysis of these tasks and the qualitative feedback led to the proposal of an alternative notation for six of the 11 elements of the PGA modeling method.

This research is not free from threats to validity [18] . To preserve construct validity, it is important to ensure that the executed tasks are suited to evaluate the intuitiveness of a DSML. Therefore, we applied an existing evaluation technique, for which the origin of the tasks is rigorously substantiated [3] . With respect to internal validity, external factors that influence the results need to be avoided. In this respect, participants were chosen with the same educational background (i.e., Master's students in Business Engineering and Business Administration). Besides this, the participants had similar foreknowledge in conceptual modeling and received a collective introduction to PGA. Furthermore, participation was voluntarily and no compensation was provided. Finally, we used two different randomly assigned groups and divided the PGA concepts between the term association and notation association tasks to mitigate an allocation bias. In this way, we made sure that the terms given during the first task did not influence the associations of the notation association phase. The choice of participants also affects the external validity or generalizability of the results. The students have a strong economic orientation which enabled us to obtain a group of respondents with knowledge and skills that can act as a proxy for business-oriented stakeholders. These stakeholders are the targeted end-users of the PGA modeling method. Nevertheless, the choice for students is an inherent limitation and further research is needed to replicate the evaluation technique with business practitioners. Reliability reflects the degree to which the results could be reproduced by the modeling community. To ensure this, the procedure that was used to apply the evaluation technique and the URL of the evaluation questionnaires can be found in Sect. 3.1. Finally, we added the details about the analysis of the different evaluation tasks in Sect. 3.2.

Future research is needed for the evaluation of the proposed improvements. This includes an experiment, in which the intuitiveness of the initial and newly proposed notation is compared. Such an experiment could be based on recall and comprehension questions, which compare the effectiveness and efficiency of interpreting both versions of the PGA notation [4] . Nevertheless, more research is needed to set-up a rigorous experimental design. In this respect, we are currently implementing the new version of the notation to become part of a future version of the PGA modeling tool. The new tool shall be made available through the PGA project space within the OMiLAB 3 of the Open Models Laboratory [2] .

On a separate research stream, we will investigate possibilities of automating the applied evaluation technique [3] . In this respect, we aim to set up a web-environment that automatically generates the evaluation sheets once the concepts and sample notations are uploaded. Moreover, it shall provide a WYSI-WYG web editor for drawing notations and storing them. To support the analysis of the collected data, this system shall use OpenCV or similar technologies to automatically analyze the created proposals for new notations. Besides this, enabling text analysis could be useful for the results of the notation association task as well as implementing statistical analysis of the responses and the automated generation of evaluation reports. Ultimately, the web-environment will increase the possibilities of testing a modeling language comprehensively, as it enables an efficient set-up, execution, and analysis of the evaluation. Consequently, it will mitigate issues related to the paper-and-pen evaluation of a tool-based modeling language.

ADOxx Metamodelling Platform

An open platform for modeling method conceptualization: the OMiLAB digital ecosystem

Intuitive understanding of domain-specific modeling languages: proposition and application of an evaluation technique

Guidelines for empirical evaluations of conceptual modeling grammars

On the semantic transparency of visual notations: experiments with UML

Domain-specific modeling languages: requirements analysis and design guidelines

Strategic alignment: leveraging information technology for transforming organizations

Metamodelling platforms

Domain-Specific Conceptual Modeling -Concepts, Methods and Tools

The "physics" of notations: toward a scientific basis for constructing visual notations in software engineering

Conceptual Modelling and Telos. Conceptual Modelling, Databases, and CASE: an Integrated View of Information System Development

Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers

The creation of business architecture heat maps to support strategy-aligned organizational decisions

Realizing strategic fit within the business architecture: the design of a process-goal alignment modeling and analysis technique

How to make a decision: the analytic hierarchy process

Action design research

Experimentation in Software Engineering