key: cord-0177967-74xu3ho2 authors: Bieringer, Lukas; Grosse, Kathrin; Backes, Michael; Krombholz, Katharina title: Mental Models of Adversarial Machine Learning date: 2021-05-08 journal: nan DOI: nan sha: c26d491b05efd5b6c5502a4e76ff05f730542f55 doc_id: 177967 cord_uid: 74xu3ho2 Although machine learning (ML) is widely used in practice, little is known about practitioners' actual understanding of potential security challenges. In this work, we close this substantial gap in the literature and contribute a qualitative study focusing on developers' mental models of the ML pipeline and potentially vulnerable components. Studying mental models has helped in other security fields to discover root causes or improve risk communication. Our study reveals four characteristic ranges in mental models of industrial practitioners. The first range concerns the intertwined relationship of adversarial machine learning (AML) and classical security. The second range describes structural and functional components. The third range expresses individual variations of mental models, which are neither explained by the application nor by the educational background of the corresponding subjects. The fourth range corresponds to the varying levels of technical depth, which are however not determined by our subjects' level of knowledge. Our characteristic ranges have implications for the integration of AML into corporate workflows, security enhancing tools for practitioners, and creating appropriate regulatory frameworks for AML. Abstract-Although machine learning (ML) is widely used in practice, little is known about practitioners' actual understanding of potential security challenges. In this work, we close this substantial gap in the literature and contribute a qualitative study focusing on developers' mental models of the ML pipeline and potentially vulnerable components. Studying mental models has helped in other security fields to discover root causes or improve risk communication. Our study reveals four characteristic ranges in mental models of industrial practitioners. The first range concerns the intertwined relationship of adversarial machine learning (AML) and classical security. The second range describes structural and functional components. The third range expresses individual variations of mental models, which are neither explained by the application nor by the educational background of the corresponding subjects. The fourth range corresponds to the varying levels of technical depth, which are however not determined by our subjects' level of knowledge. Our characteristic ranges have implications for the integration of AML into corporate workflows, security enhancing tools for practitioners, and creating appropriate regulatory frameworks for AML. Index Terms-Adversarial ML, Mental Models Adversarial machine learning (AML) studies the reliability of learning based systems in the context of an adversary [1] , [2] , [3] . For example, tampering with some features often suffices to change the classifier's outputs to a class chosen by the adversary [4] , [5] , [6] . Analogously, slightly altering the training data enables the attacker to decrease performance of the classifier [7] , [8] . Another change in the training data allows the attacker to enforce a particular output class when a specified stimulus is present [9] , [10] . Most state-of-the-art attacks and mitigations are in an ongoing arms race [11] , [12] , [13] . Although machine learning (ML) is increasingly used in industry, very little is known about ML security in practice. To tackle this question, we conduct a first study to explore mental models of AML. Mental models are relatively enduring, internal conceptual representations of external systems that originated in cognitive science [14] , [15] . In other security related areas, correct mental models have been found to ease the communication of security warnings [16] or enable users to implement security bestpractices [17] . Mental models also serve to enable better *. First two authors contributed equally. interactions with a given system [18] , or to design better user interfaces [19] . Our methodology builds upon these previous works by using qualitative methods to investigate the perception of vulnerabilities in ML applications. Our findings shed light on four characteristic ranges of practitioners' mental models of AML. The first concerns the separation of AML and standard security. In many cases, the borders between these two fields are blurry: a subject may start talking about evasion and finish the sentence with a reference to cryptographic keys. On the other hand, security threats are often taken for granted, whereas practitioners are less aware of AML attack scenarios. Secondly, we identified functional and structural components with respect to the perception of AML. More concretely, structural components are cognitively put into functional relation within the mental models. Furthermore, our subjects show large variation across their perception of attacks and defenses. These variations are unrelated to security background or other educational factors, and are only partially influenced by different applications of ML. Last but not least, the degree of technical depth in our subjects' mental models differs: Whereas some subjects explained their applications almost at the code-level, others had rather a high level perspective where the ML model was seen as a mere part of the system in question. Our interviews showed that most of our subjects lack an adequate and differentiated understanding to secure ML systems in production. At the same time, more than a third of our participants feels insecure about AML. These concerns seem justified as we found evidence that semiautomated fraud on ML systems takes place in the wild. However, our findings have more practical implications. Our results on mental models allow us to address the current lack of understanding by (I) aligning corporate workflows that enable all actors to understand AML threats with minimal effort, (II) developing tools that help practitioners to assess and evaluate security of ML applications, and (III) drafting regulations that contain adequate security assessments and reduce insecurity about AML. However, more work is needed to understand the individual and shared mental models of practitioners and assess the real world security risks when applying ML. In this section, we review related work on AML and recall different attacks that have recently been discussed. We also review literature on mental models with regard to human-computer interaction, usable security and ML. data design training model deployment poisoning [7] , [20] , backdooring [9] , [10] model stealing [21] model reverse engineering [22] membership inference [23] evasion [4] , [6] adversarial reprogramming [24] adversarial initialization [25] weight perturbations [26] sponge attacks [27] Figure 1: AML threats within the ML pipeline. Each attack is visualized as an arrow pointing from the step controlled to the point where the attack affects the pipeline. For the sake of completeness, we conclude with a description of additional, recent attacks, some of which are part of our questionnaires (Appendix C.3). In adversarial initialization, the initial weights of a neural network 1 are targeted to harm convergence or accuracy during training [25] , [40] . In adversarial reprogramming, an input perturbation mask forces the classifier at test time to perform another classification task than originally intended [24] . For example, a cat/dog classifier is reprogrammed to classify digits. In model reverse engineering, crafted inputs allow to deduce from a trained model the usage of dropout and other architectural choices [22] . Finally, sponge attacks aim to increase energy consumption of the classifier at test time [27] . In general, AML research has been criticized for the limited practical relevance of its threat models [41] , [42] . There is also limited knowledge about which threats are relevant in practice. To the best of our knowledge, only Kumar et al. [43] have studied this question and found that practitioners are most concerned about poisoning and model theft. Yet, in academia, most work focused on evasion so far. To shed more light on AML in practice, we interview industrial practitioners and take a first step towards a theory of mental models of AML. To this end, we now introduce and review mental models. Mental models are relatively enduring and accessible, but limited, internal conceptual representations of external systems [44] that enable people to interact with given systems. Hence, the field of human computer interaction (HCI) studied this concept quite early [45] . Mental models, most recently, saw an increasing relevance in usable security. We now recall prior application scenarios and highlight relevant conceptual contributions in the context of security and ML. Mental models in HCI and usable security. The relevance of mental models has been subject to a lengthy debate in HCI research [46] , [47] . In many cases, the focus was to capture, depict and analyze mental models of specific objects of investigation. Examples of topics include, but are not limited to, the design of online search applications [48] , interface design [49] , and interfaces for blind people [50] . Research in usable security has recently focused on mental models of security in general [18] , [51] , privacy in general [52] , security warnings [16] , incident response [53] , the internet [54] , the design of security dashboards [55] , the Tor anonymity network [19] , privacy and security in smart homes [56] , [17] , encryption [57] , [58] , HTTPS [59] , and cryptocurrency systems [60] . With regard to the respective object of investigation, these contributions paved the way for improvements of user interface designs [19] , adequate security communication [16] , as well as the development of security policies and implementation of best-practices [17] . It has been argued that security mental models contain structural and functional properties [57] . For each application, users develop a cognitive representation of its inherent components, their interconnection and correspondingly possible security threats. This representation helps them to understand where threats could emerge and how they could take effect. Mental models evolve dynamically upon individual interaction with a given application [61] . Mental models in ML. In order to interact with an ML application, humans need a mental model of how it combines evidence for prediction [62] . This is all the more important for ML-based applications which often inherit a certain opacity. As Lage et al. [63] pointed out, the number of necessary cognitive chunks is the most important type of complexity in order to understand applications. During interaction with black-box processes, humans strive for reduced complexity which may lead to the development of inaccurate or oversimplified mental models [64] , [65] . A dedicated line of research therefore elaborates on the relevance and nature of mental models in the context of explainable artificial intelligence. Mental models have been found to serve as scaffolds not only for a given ML application [66] , but also for its embedding in organizational practices [67] . For data science teams, these workflows usually consist of predefined steps ( Figure 1 ) and necessitate interpersonal collaboration. Following Arrieta et al. [68] , we argue that individual collaborators within these teams (e.g. ML engineers, software engineers) develop separate internal representations of a given workflow or application. The need for appropriate mental models thereby increases with the enlarged scope of ML applications [69] and involved stakeholders [70] , [71] . Recent work in this line of research called for qualitative studies at the intersection of the HCI and ML communities, to better understand the cognitive expectations practitioners have on ML systems [64] , [72] . Suchlike studies seem all the more relevant as various industry initiatives propagate a human-centric approach to AI, explicitly referring to mental models. 2 However, the current scientific discourse lacks a dedicated consideration of cognition in AML. In order to fill this gap, we present the first qualitative study to elicit mental models of adversarial aspects in ML. This section describes the design of our semistructured interview study, the drawing task, our recruiting strategy, the participants, and how we analyzed the data. Our methodology was designed to investigate the 2. e.g. https://pair.withgoogle.com/chapter/mental-models/ To assess participants' perceptions, we conducted semi-structured interviews enriched with drawing tasks. We draw inspiration from recent work in usable security which also investigated mental models [57] , [59] . Before the interview, participants were informed about the general purpose of our study and the applied privacy measures during data collection. Participants were also instructed to complete a questionnaire on demographics, organizational background and a self-reflected familiarity with field-related concepts (Appendix C) before the interview. This questionnaire was filled with or without the author's presence. The answers to this questionnaire have later been used to put participants' perceptions in context to their organizational and individual background. The threefold structure of our interviews covered 1) a specification of a given ML project a subject was involved in, 2) the underlying ML pipeline of this project and 3) possible security threats within the project. We chose this approach as the different attack vectors form part of the ML-pipeline as shown in Section 2.1. The detailed interview guideline can be found in Appendix B. As a last step of our interviews, we confronted the subjects with exemplary attacker models for some of the threats considered relevant in industrial application of ML [43] . To assess practitioners' understandings of these threats, study participants had to elaborate on these attack vectors within their specific setup (Appendix C.2). After the interview, and to assess the subjects' knowledge about (A)ML in general, participants had to fill an additional questionnaire after the interview (Appendix C.3). In this questionnaire, we addressed general knowledge in ML and asked for a self-reflected familiarity rating with some of the attacks we discussed in Section 2.1. This questionnaire was handed to the subjects after the interview as to avoid priming. We conducted one pilot interview to evaluate our study design. This first subject met all criteria of our target population in terms of employment, education and prior knowledge. As his explanations and drawings matched our expectations, we only added a specific question regarding the collaborators within a given ML-based project. Each interview lasted approximately 40 minutes and was jointly conducted by the first two authors of this paper. To minimize interviewer biases, we equally distributed the interviews between the two authors: one was the lead interviewer and the other took additional notes and screenshots of the drawing task. Due to the COVID-19 pandemic, all interviews were conducted remotely and relied on a freely available digital whiteboard 3 . Recruitment for a study on applied ML in corporate environments presents a challenge, as only a small proportion of the overall population works with ML. Furthermore, the topic touches compliance and intellectual property of participating organizations. Hence, many companies are skeptical about the exchange with third parties. As a consequence, many current contributions with industrial practitioners as study subjects are conducted by corporate research groups (e.g. [43] , [73] ). We tried to initiate interviews with two multinational companies with more than 140,000 employees each. Unfortunately, both denied our request after internal risk assessments. Therefore, we focused on smaller companies where we could present our research project directly to decision-makers and convince them to participate in our study. We relied on the individual networks of the authors (pilot subject, S11) and public databases 45678 to find potential subjects and used direct-messaging on LinkedIn and emails to get in contact with them. Recruitment of study participants happened in parallel to interview conduction. Some subjects forwarded our interview request to internal colleagues, so that we talked to multiple employees of some participating companies (see Table 1 ). We aimed to recruit experienced and knowledgeable participants and hence our requirements were a background in ML or computer science and positions such as data scientists, software engineers, product managers, or tech leads. We did not require any prior knowledge in security. After 15 recruited subjects, the research team agreed that the interviews saturated, and we stopped recruiting. The subjects were randomly assigned an ID (a number between 1 and 20) which was used throughout our analysis. All participants were offered an euro 20 voucher as compensation for their time. 3 . https://awwapp.com/ 4. https://www.crunchbase.com/ for European companies operating in AI and having raised more than 1 million dollar funding 5. https://www.appliedai.de/hub/2020-ai-german-startup-landscape 6. https://www.forbes.com/sites/alanohnsman/2021/04/26/ ai-50-americas-most-promising-artificial-intelligence-companies/ ?sh=653894c477cf 7. https://www.sitsi.com/top-10-most-promising-european-ai-startups 8. https://siliconcanals.com/news/top-artificial-intelligence-startups-europe/ We summarize demographic information in Table 1 . One subject, S10, did not hand in the questionnaire and is consequently not included in the following statistics. 14 participants identified as male, one identified as female, with an average age of 34 years (standard deviation (STD) 4.27). As intended for a first exploration of practitioners' perception of AML, our sample covered various application domains and organizational roles which we now describe in detail. Education and prior knowledge. The majority of subjects (9 of 14) has a PhD, with all subjects holding some academic degree. Most participants (12 of 14) reported that they had attended lectures or seminars on ML. Roughly half (6 of 14) reported to have a similar background in security. To measure our participants' knowledge in the area of ML, we constructed a questionnaire based on job interview questions 9 for ML (Appendix C.3). Given that participants were not previously informed they had to take a test, we aimed to select a broad range of topics easy to query with multiple choice answers that were not too hard. The questionnaire had 8 questions, with the subjects correctly answering on average 6.64 questions (STD 1.14). Guessing would yield an average of 2.66 correct questions. Thus, while we do not know how reliable our questionnaire estimates ML knowledge, we conclude that all our subjects are indeed knowledgeable in ML. We also sanity checked the knowledge of our subjects in AML. Confirming our expectations, few subjects reported high familiarity, and very recent/less known attacks were rated as unfamiliar. Employment. Regarding the size of the companies, four subjects worked in companies with less than ten employees, five in companies with less than 50 and the remaining six subjects in companies with less than 200 employees. The companies' application areas were as diverse as healthcare, security, human resources, and others. Most subjects were working in their current positions 6 years (STD 4.9). Their roles were diverse: Most subjects (8 of 15) were in managing positions. Three were software or ML engineers, three more were researchers. One of the subjects stated to be both a researcher and a founder. One subject did not report his role. Finally, we asked subjects to report which goals were part of their companies' AI/ML checklist. Almost all subjects (13 of 14) reported that performance mattered in their company. Half (7 of 14) stated that privacy was important. Slightly less than half (6 of 14) focused on explainability and security. Least subjects (4 of 14) listed fairness as a goal in their products. To conclude, when interpreting these numbers, one should keep in mind that not all five goals apply equally to all application domains. Furthermore, our sample is too small to derive per area or per company insights, and we thus leave a detailed analysis for future work. Our analysis adopted an inductive approach, where we followed recent work in social sciences and usable security that constructed theories based on qualitative data [74] , [59] . To distill observable patterns in interview transcripts and drawings, we applied two rounds of open coding. We then performed Strauss and Corbin's descriptive axial coding to group our data into categories and selective coding to relate these categories to our research questions [75] . Throughout the coding process, we used analytic memos to keep track of thoughts about emerging themes. The final set of codes for interview transcripts and drawings is listed in Appendix D. As a first step, the first two authors independently conducted open coding sentence by sentence and sketch by sketch. This allowed for the generation of new codes without predefined hypotheses. Afterwards, the resulting codes were discussed and the research team agreed on adding specific codes for text snippets relating to the confusion of standard security and AML. As a second step, two coders independently coded the data again. After all iterations of coding, conflicts were resolved and the codebook was adapted accordingly. During axial coding, the obtained codes were grouped into categories. The first two authors independently came up with proposed categories which have then been discussed within an in-person meeting. While the grouping was undisputed for some of the categories presented in Appendix D (e.g. AML attacks, pipeline elements), for others the research team decided for (e.g. confusion, relevance) or against (e.g. type of ML model applied) the inclusion of a corresponding category only after detailed discussion. In addition, dedicated codes for the perception of participants (e.g. perceives AML as a feature, not a bug or security issue) were added to the codebook. Once the research team agreed on a final codebook, all transcripts and drawings were coded again using corresponding software. 10 In doing so, we aimed for inferring contextual statements instead of singular entities. The codes and categories served as a baseline for selective coding. Independently, the researchers came up with observations and proposals for specific mental models. Every proposal included a definition of the observation, related codes, exemplary quotes and drawings. The first two authors then met multiple times to discuss the observations and the corresponding relations of codes and categories. During these discussions, the four characteristic ranges of participants' AML perception, described in detail in Section 4, were distilled. We calculated Cohen's kappa [76] to measure the level of agreement among the coders. For drawings, we reached κ = 0.85, and for interview transcripts κ = 0.71. These values indicate a good level of coding agreement since both values are greater than 0.61 [77] . Given the semitechnical nature of our codebook, we consider these values as substantial inter-coder agreement. Irrespective of this and in line with best practices in qualitative research, we believe that it is important to elaborate how and why disagreements in coding arose and disclose the insights gained from discussions about them. Each coder brought a unique perspective on the topic that contributed to a more complete picture. Due to the diverse background of our research team in AML, usable security and economic 10 . Available at https://www.taguette.org/ and https://www.maxqda.com/. geography, most conflicts arose regarding the relevance of technical and organizational elements of transcripts and drawings. These were resolved during conceptual and onthe-spot discussions within the research team. Given previous work on mental models [57] , we expected to find structural and functional properties in our subjects' mental models. Concerning ML, we designed our study in a way that subjects would first visualize their perception of the pipeline and then later add corresponding attacks and defenses. For the pipeline, we expected that participants would name basic steps or components, such as data (collection), training, and testing. In general, we assumed subjects' descriptions would vary in technical depth. Regarding AML, one of our motivations to conduct this study was to learn which knowledge our subjects had. As a recent phenomenon, AML might not be known at all in practice, although practitioners might be aware of attacks relevant to their specific application. In particular, we did not expect practitioners to visualize attacks using a starting and target point, as done in Figure 1 . The ethical review board of our university reviewed and approved our study design. We limited the collection of person-related data as much as possible by assigning IDs to participants that were used throughout the analysis. Since all participants were employed at existing companies and partially shared business-critical information, we aimed to avoid company-specific disclosures in this paper. We complied with both local privacy regulations and the general data protection regulation (GDPR). We identified four characteristic ranges that describe the variation in practitioners' mental models of AML. Our data indicates that the individual perception varies along these ranges; they are no binary features. Figure 2 visualizes the two extremes of each of the four characteristic ranges. Albeit one could reason that each of these properties is more complex than a mere range, we reason that for a first attempt to characterize mental models such a simplified representation is appropriate. As first range, we describe to which degree our subjects mixed standard security and AML concepts. An example is given in Figure 2 for model stealing. One extreme is a subject who distinguished between 'model stealing' and for example a 'code breach', whereas on the other hand, some subjects were concerned about the model being somehow copied. We provide a detailed description of our findings in Section 4.1. The second characteristic range concerns structural components and functional relations between them. Figure 2 shows both extremes for model reverse engineering of a neural network. By crafting inputs, an attacker might deduce architectural choices within the functional structure, whereas on the other hand a hyperparameter from the model could be accessed illicitly. We present our detailed The third range concerns variations in the pipelines, attacks, and defenses described. An example is shown in Figure 2 for poisoning attacks. Here, the attacker either injects specific inputs to the application (triangles and squares in our example), or a general, malicious input. The detailed findings on these individual variations are presented in Section 4.3. The last and fourth characteristic range describes the level of technical depth. Figure 2 depicts the extremes of sophisticated technical depth for membership inference. Some participants explained their setting almost on the code level, whereas others would just utter the high level concern of their data being illicitly accessed. More detailed findings on the corresponding variances are presented in Section 4.4. We will now detail each of the four characteristic ranges and give examples of both interviews and drawings where they occurred. We found that our subjects generally did not distinguish between classical security and AML. Albeit there is a clear distinction in research, it might not matter in practice whether an attacker obtained the data of a company via a social engineering attack, exploiting a security vulnerability, or via a prediction API. On the one hand, the boundary between security and AML often appeared blurry or unclear, with the corresponding concepts intertwined. On the other hand, there were crucial differences in the perception between classical security and AML threats. One difference is that whereas security defenses were often clearly stated as such, AML mitigations 11 were often applied without security incentives. Finally, we find a tendency to not believe in AML threats. Many subjects denied responsibility, doubted an attacker would benefit, or stated the attack does not exist in the wild. There was no such tendency in standard security. 4.1.1. Mingling AML and security. We first provide examples to clarify our observation that security and AML were not distinguished by our participants. Afterwards, we investigate if security and AML are used interchangeably, by investigating the co-occurrence of codes. Vagueness of the boundary between security and AML. There are plenty of examples on vagueness about the boundary between classical security and AML. For example S20 reasoned about evasion: "this would require someone to exactly know how we deploy, right? and, where we deploy to, and which keys we use". At the beginning, the scenario seems unclear, but the reference to (cryptographic) keys or access tokens shows that the subject has moved to classical security. Analogously, when S18 reasoned about membership inference: "but that could be only if you break in [...] if you login in to our computer and then do some data manipulation". Again, this subject was reasoning about failed access control as opposed to an AML attack via an API. Sometimes, ambiguity in naming confused our subjects. For example, S11 thought aloud: "poisoning [...] the only way to install a backdoor into our models would be that we use python modules that are somewhat wicked or have a backdoor". In this case, the term 'backdoor' in our questionnaire triggered a standard security mindset involving libraries in contrary to our original intention to query subjects about neural network backdoors. The same reasoning can also be seen in S11's drawing (compare Figure 3) , where 'backdoor' points to python modules. Finally, S12 stated: "maybe the poisoning will be for the neural network. From our point of view you would have to get through the Google cloud infrastructure". From an AML perspective, the poison is in the data which is uploaded from the user. Yet, the infrastructure is perceived as an obstacle for the attack. Correlations between security and AML attacks. In the previous paragraph, we showed that the boundaries between AML and classical security are blurred in our interviews. Another example is S6 reasoning about IP loss: "we are very much concerned I'd say the models themselves and the training data we have that is a concern if people steal that would be bad". In this case, it is left out how the attack is performed. Analogously, S9 remarked: "We could of course deploy our models on the Android phones but we don't want anybody to steal our models". To investigate whether our subjects are more concerned about some property or feature (data, IP, the model functionality) than about how it is stolen or harmed, we examined the co-occurrence of AML and security codes that refer to similar properties in our interviews. For example, the codes 'model stealing' and 'code breach' both describe a potential loss of the model (albeit the security version is broader). Both codes occur together six times, with 'code breach' being tagged one additional time. Furthermore, the code 'model reverse engineering', listed only two times, occurs both times with both 'model stealing' and 'code breach'. However, not all cases are that clear. For example 'membership inference' and 'data breach' only occur together two times. The individual codes are more frequent, and were mentioned by three ('membership inference') and eleven ('data breach') participants. Analogously, attacks on availability (such as DDoS) in ML and classical security were only mentioned once together. Such attacks were brought up in an ML context twice, in standard security four times. Codes like 'evasion' and 'poisoning', in contrast, are not particularly related to any standard security concern. We conclude that AML and security are not interchangeable in our subjects' mental models to refer to attacks with a shared goal. In the previous subsection, we found that our subjects did not distinguish classical security and AML. To show that this is not true in general, we now focus on the differences between the two topics. To this end, we start with the perception of defenses and then consider the overall perception of threats in AML and security. We conclude with a brief remark on the practical relevance of AML. Defenses. Out of fifteen interviews, in thirteen some kind of defense or mitigation was mentioned; all corresponding interviewees mentioned a security defense (encryption, passwords, sand-boxing, etc). An AML mitigation appeared in eight. In contrast to security defenses, however, AML defenses were often implemented as part of the pipeline, and not seen in relation to security or AML. As an example, S9, S15, and S18 reported to have humans in the loop, however not for defensive purposes. S10 and S16 were aware that this makes an attack more difficult. For example, S16 stated: "maybe this poisoning of the data [...] is potentially more possible. There, we would have to manually check the data itself. We don't [...] blindly trust feedback from the user". Analogous observations hold techniques like explainable models (3 subjects apply, 1 on purpose) or retraining (2 apply, additional 2 as mitigation). For example, S14 said: "when we find high entropy in the confidences of the data [...] for those kind of specific ranges we send them back to the data sets to train a second version of the algorithm". In this case, retraining was used to improve the algorithm, not as a mitigation. We conclude that albeit no definite solution to vulnerability exists, many techniques that increase the difficulty for an attacker are implemented by our subjects. At the same time, many practitioners are unaware which techniques potentially make an attack harder. Perception of threats. There is also a huge difference in the perception of threats in security and AML. In security, threats were somewhat taken for granted. For example, S9 was concerned about security of the server's passwords "because anybody can reverse-engineer or sniff it or something". Analogously, S6 said to pay attention to "the infrastructure so that means that the network the machines but also the application layer we need to look at libraries". On the other hand, almost a third of our subjects (4 of 15) externalized responsibility for AML threats. For example, S3 said their "main vulnerability from that perspective would probably be more the client would be compromised". Analogously, S1 remarked that ML security was a "concern of the other teams". In both cases, the subjects referred to another entity, and reasoned that they were not in charge to alleviate risks. Other reasons not to act include subjects not having encountered an AML threat yet, and concluded AML was not relevant. More concretely, S9 remarked: "we also have a community feature where people can upload images. And there could be some issues where people could try to upload not safe or try to get around something. But we have not observed that much yet. So it's not really a concern, poisoning". Roughly half of the subjects (7 of 15) reported to doubt the attackers' motivation or capabilities in the real world. For example, S1 said: "I have a hard time imagining right now in our use-cases what an attacker might gain from deploying such attacks". S20, who worked in the medical domain, stated: "I'm left thinking, like, why, what could you, achieve from that, by fooling our model. I'm not sure what the benefit is for whoever is trying to do that". Finally, many subjects (9 of 15) believed that they have techniques in place which function as defenses. As an in-depth evaluation of which mitigations are effective in which setting is beyond the scope of this paper, we leave it for future work. Practical relevance of AML. The fact that most subjects did not consider AML threats relevant might simply be an expression of these threats being academic and not occurring in practice. Yet, our interviews showed that there are already variants of AML attacks in the wild. More concretely, S10 stated: "What we found is [...] common criminals doing semi-automated fraud using gaps in the AI or the processes, but they probably don't know what AML, like adversarial machine learning is and that they are doing that. So we have seen plenty of cases are intentional circumventions, we haven't quite seen like systematic scientific approaches to crime". The fact that many of our subjects seemed unconcerned about AML could then be an indicator that harmful AML attacks are (still) rare in practice. On the one hand, classical IT security and AML were mingled in our subjects' mental models: the boundaries between the corresponding threats were often unclear. Yet, security and AML were not interchangeable in our subjects mental models to refer to attacks with a shared goal. Furthermore, security threats were treated differently than AML threats: the latter were often considered less relevant. Finally, as our interviews show, Figure 3 : Drawing of S11. Red markings were added by the subject before, blue after being confronted with selected attacks. Figure 4 : S1 inserted red arrows to indicate attacks. there are already variants of AML attacks in practice. We now turn to more general properties of mental models in AML which we discovered during the interviews. We found structural and functional components in our subjects' mental models. Structural components cover multiple, constituting entities that an individual perceives as relevant within a given application. In interaction with an ML system, functional components describe an individual's perception of the relations between the structural elements. As intended, the structure of our interview and drawing task (Appendix B) allowed to investigate these properties on the level of the ML pipeline, of the attack vectors as well as of the defenses. 4.2.1. ML pipeline. All subjects distinguish clearly separable elements within their ML workflow. The specific composition of these steps defines the structure of a certain ML pipeline. For two participants, this structure reflects the ML pipeline that we introduced in Figure 1 . When asked to sketch the kind of pipeline applied, S4 talked about "data", "training", "testing", and "visualization". We argue that these structural components serve as a scaffold for an individual's mental model. Interestingly, the mental models of 12 out of 15 subjects covered additional components that we did not expect prior to the study. The sketches of S3, S7, and S11 (Figure 3 ), for example, contain explicit elements for data capturing. S1 (Figure 4) , S9, S12, as well as S20 included dedicated elements representing a specific database to their drawing. Five subjects also highlighted structural elements within Figure 5 : Drawing of S16. Colors were added after selected attack were presented to the subject. Red refers to evasion, purple to reverse engineering, blue to membership inference. the deployment environment during the interviews. S14, for example, specified on an API for deployment "on several kinds of hardware architectures". Analogously, S1 described an API that "can be used to allow the user to interact with the models" (Figure 4) . Hence, these structural elements concerning data and deployment seem to be of importance for the corresponding mental models. However, the perception of industrial practitioners does no only focus on these structural components but also covers functional aspects. S6 for instance stated that his ML pipeline "forks into a number of different directions and there are also interactions between the different components". In the corresponding sketch, multiple arrows within and across specific ML models indicate this interconnection of single components. Other drawings include this functional perspective through straight lines connecting the structural components (see Figure 4 , S1), arrows connecting some of the structural components in a subsequent manner (e.g. S14), and arrows connecting all structural components in a subsequent manner (S18 in Figure 6 ). The identified structural and functional components seem to be similarly relevant for mental models on attack vectors. For any kind of ML-specific threat, participants were able to precisely locate where they situated the corresponding, structural starting point. These have been specifically named during the interview and sketched via labelled arrows (e.g. Figure 3 , S11), additional annotations (S11, S15), highlighted parts of potentially vulnerable pipeline components (e.g. Figure 7 , S10) or as entire steps within a given ML workflow that have been marked as vulnerable (S9, S20). Strikingly, we saw a wide overlap in the perception of potential focal starting points for attack vectors. Study participants considered the model itself, the input of their ML pipeline, or the deployment environment to be particularly vulnerable. Figure 5 (S16) shows this for the latter. When confronted with poisoning and reverse engineering attacks, S16 marked the input and output of his pipeline as possible starting points for threats (purple rectangles) and talked about how a competitor could "screw our labeled dataset" or a customer might "ask a lot of questions to the API". However, the perception of attack vectors did also cover functional components. S1, for example, depicted the causal sequence of a "data injection attack" as three consecutive red arrows connecting different components of his ML pipeline (Figure 4 ). This is all the more relevant, as S1 provided such a functional explanation and drawing for each of the attack vectors we presented to him. His mental models, hence, clearly seem to contain functional components. This is also the case for S16, who similarly provided explanations on the functional evolvement of certain attacks within his workflow and even added corresponding functional elements to his sketch (blue and red arrows in Figure 5 ). Although we found participants' defenses explanations and sketches to be rather sparse, structural and functional properties are also relevant for the corresponding mental models. As visible in the sketch of S18, defenses are often thought of as structurally bound to specific components of a workflow/pipeline (Figure 6 , S18). Data (S14), training (S6) and the models themselves (S10) have been specifically named as focal points for implementing defenses. In the case of defenses implemented at the model component, S14 stated to "regularize in a way that makes it less sensitive to an adversary". Hence, these implemented defenses are cognitively attached to the classifier as a focal pipeline component. However, security mental models also contain functional properties. In the case of human-in-the-loop-defenses, for example, S14 stated to send certain classifications "back to the data sets to train a second version of the algorithm" if the output confidence for certain data exhibited high entropy. This is depicted in the corresponding sketch by an arrow pointing from a rectangle with the caption "CPU" at the end of the pipeline to "raw data" (initial step of the pipeline). Similarly, S7, a subject working in video surveillance, explained the defense they had implemented to secure the transfer of input data (from cameras and onsite computers) into their pipeline: "This can only go out, never go in. [...] Nothing from the internet can connect to that server". Industrial practitioners, hence, perceive defenses as containing functional components to unfold their full effect. We conclude that mental models in AML are composed of structural components which are cognitively put into (internal) relation. However, the specific unfolding of these internal conceptual representations seems to depend on the corresponding application and its underlying ML pipeline. There are more sources of Figure 6 : Drawing of S18. Red star indicates the most important component of the pipeline, not an attack. variation in our subjects' mental models, however. We now investigate these variations. Our interviews showed great variation regarding to the threats reported and in which detail, if at all. We investigate possible underlying causes that might influence these differences across subjects, including the prior knowledge and education as well as the subjects' application domain. We start with the variation of mental models across subjects. Perceived relevance of AML. The practitioners differed in the importance they attributed to AML. A third of them (5 of 15) did not mention AML at all before we explicitly asked. Another third reported that they were not very concerned about AML. For example, S1 stated that evasion, or "injecting malicious data to basically make the model [...] predict the wrong things" was "a concern that is not as high on my priority list". S15, analogously, said: "mainly the machine learning pipeline this is the less critical security problem", reasoning that "simply a performance would be unexpected". Yet, over a third (6 of 15) of the participants reported to feel insecure about AML when confronted with the topic. Of these six subjects, two previously showed low priority on AML, and three did not mention AML at all. An example of insecurity is S4, who stated she needed "some more research on it". Some subjects, like S19, were concerned about specific attacks: "I maybe need to learn more about this membership". In summary, some practitioners consider AML threats important, whereas some subjects did not know AML well, and yet others did not consider it an important threat. From each of these three groups, there was at least one subject that felt not well informed. After the interviews (e.g., off the record) some participants stated that their awareness for AML had increased due to the interview. Specificity of attacks. Not only the overall opinion, but also the specificity with which attacks were described varied greatly. On the one hand, S1 (Figure 4 ) added in his drawing text to the starting point of the attack. He also depicted in detail how it propagated through the individual steps using red arrows. On the other hand, S10 (Figure 7 ) only added blue color to denote that an attack is possible at the input or output of the system. Yet, a vague representation in the drawing does not imply a vague description of the attacks. During the interview, S10 stated: "we have to work with the assumption that the data we have [...] may ... contain ... basically unlimited number of modified samples or input data and that we don't know which ones are they and whether they would come in next day or so". This paraphrases, in contrast to the drawing, poisoning fairly accurately. In contrast, S6 described a possible threat more vaguely as "the models themselves and the training data we have, that is a concern if people steal that would be bad". After showing our participants' differences in the perception of AML, we focus on two major points that possibly explain these different perceptions. We first investigate influence of the application setting of the ML projects. Afterwards, we examine the educational background of our subjects as possible explanation. Application setting. We first study the influence of the application domain of each subject. As we expect practitioners in security-related tasks to show different behavior, we divide the subjects into security and non-security groups, starting with subjects working in security-related fields. S10, who worked in a setting with cybersecurity reported: "there is some standard AML attacks on ML you can use, but we design our system knowing that very well; on the other hand, we know that there is no perfect security, so, again defense is in monitoring and vigilance, but it's not something that can be fully automated in our opinion". S10 was in general very sensitive towards AML. S4, also from a cybersecurity setting, was less concerned about evasion: "I can't imagine yet how it can be applied for real life, for example [...] since we are pretty close on our development". Yet, S4 also stated the need to gather more information about AML. Hence, also participants who worked in security-related areas had diverse mental models with respect to concrete attacks. Subjects from non-security fields have similarly diverse mental models. This diversity is also reflected in the drawings. S11 (Figure 3 ) added some attacks (in red) before we provided explanations of evasion, backdooring and membership inference (added in blue). S18 (Figure 6 ), on the other hand, did not add any threats in his drawing. Analogously, opinions also differ in the interviews; e.g., S15 who worked in an non-security setting, was aware of security issues: "one interesting thing of course is that the solution is in some ways constraint by adversarial security considerations so for example you cannot use natural language generation very much because of potential adversarial behavior". On the other hand, and confirming the drawing, S18 reported that "we do not really protect the machine learning part". Prior knowledge. We find no strong relation between education and capability or knowledge about AML in our sample. For example, one subject self-reported high knowledge in AML, but also stated: "maybe the poisoning will be for the neural network". Here, a general attack, poisoning, is related to a specific model (neural networks). On the other end of the spectrum, S9 did not self-report any knowledge about security or AML, but correctly remarked: "Somebody could send us 100.000 images and collect all the results and try to build a model from that". We conclude that none of these properties directly explain the diversity in our subjects' mental models of AML. In this section, we considered the individuality within the mental models of the practitioners that we interviewed. We showed two such examples, one was the concern uttered about AML, the other the specificity of the attacks described. We investigated two possible reasons that could influence mental models, the task at hand and prior education as reported by our subjects. Both properties had a low influence on mental models in our sample. To conclude the section, we have a more general look at the variance of the technical depth of our participants' perceptions. The degree of detail in explanations and drawings defines the technical depth of participants' mental models of AML. We found these to contain in-depth technological descriptions as well as more abstract and ambiguous facets. Along these lines, some mental models focused on information about ML pipelines, whereas others encompassed the embedding system as well. 4.4.1. Sparse technical depth. Some participants showed a stronger focus on higher-level concepts of their MLbased project. Out of 15 subjects, 4 described their ML pipeline as a classifier within the larger project context (for example visible in Figure 5 ). Concerning the general perception of the ML pipeline (Figure 1 ), this seems to affect mainly the relevance of ML-models as such within the pipeline. Although 10 out of 15 subjects talked about models as pipeline components, their explanations remained rather superficial from a technical perspective. The coded text snippets cover terms such as "model" (S4), "classification" (S19) "classifier" (S15), or, at most, a specification of the model type (e.g. "neural network", (S15)). 6 out of 15 subjects did not even include the model to their drawings. Instead, for example, S10 just sketched a rectangle with the caption "detection" and possible inferences about input data indicated by an arrow (Figure 7) . This level of abstraction can also be observed for attack vectors on the ML pipeline. Asked to specify a certain threat model, S19 for example, stated: "It's like everywhere. Internal threats, external threats. Trying to mess with the communication, trying to mess if we model something". However, it remains unclear how such an attack would actually take effect. In a similar manner, S14 explained that an adversary could "try to put some pythons in non conforming ways to trigger networks". From a technical perspective, this is a rather unspecific description of potential security issues within the given project. This seems to also apply for defenses, that our participants' organizations apply to encounter AML-specific security threats. S18, for example, first explained that "the countermeasures are all in the API". Then, only after he opened the corresponding security documentation, this subject was able to provide further details on the implemented defenses. Hence, mental models in AML can entail a certain level of abstraction. However, four participants revealed a clear orientation towards a concrete technological implementation. S9, for example, described the underlying pipeline of his project very detailed and in chronological order. This subject also included hardware/software components in his explanation and precisely defined data pre-processing: "From these 20 million images we have about 500.000 images that are labelled by experts. From those we take labelled images of certain quality and build a data set. [...] We do a lot of stuff where we make sure that, for example, the images from one user don't end up in both training and test. We remove duplicates and lots of other steps. And then we create TensorFlow files from that". Remarkably, such a degree of detail mostly concerned more fine-grained perceptions of the ML pipeline depicted in Figure 1 . S4, for example, clearly differentiated between "model study", "model training", "model testing" and "model validation" in her sketch. In a similar manner, S14 walked us through the logical sequence of his whole workflow. This subject even presented multiple options for internal components (e.g. various kinds of losses) which could be replaced according to the given use case: "Everything is composable, so you can really pick the ones that work best for you". Such an in-depth perception eases the procedural understanding of attack vectors. In doubting the relevance of poisoning in real-world scenarios, S7 stated to "control training [...] so although you decide one day to go one station and feed us with a lot of wrong data, at the end of the year is 1 /365 percent of our training data. So like that would mean nothing to the neural network". Hence, mental models in AML can also come with a sophisticated technical depth. To conclude, industrial practitioners perceive ML systems and components of ML pipelines at a varying level of technical depth. This includes the embedding system, the interconnection of the pipeline's components and corresponding possible security threats. In particular, this relates to the model-only perspective often taken in academic research versus a more system oriented perspective that encompasses the ML model's context as part of a larger system. Similar to Shankar et al. [43] , we find that most of our subjects lack an adequate and differentiated understanding to secure ML systems in production. Given that we found only reports of semi-automated fraud in our sample, the absence of strong AML in practice might explain this lack of knowledge. Yet, in our sample, 6 of 15 subjects directly reported to feel insecure about ML security. We thus now discuss the diverse implications of our study on how to tackle these insecurities and the overall lack of knowledge. We start at the most fine-grained implications concerning AML tools and libraries. We then discuss the implications of our findings for the the embedding of AML in corporate workflows and finish with implications for regulatory frameworks of AML. Enhancing AML libraries and tools. Whereas several subjects reported their software (S9, S14), infrastructure (S14) or service provider (S3, S12, S20), none mentioned a specific tool for assessing security risks. At the same time, many subjects uttered concerns about AML, and it is thus promising that several recent initiatives aim at providing better access to AML. This includes AML libraries 12 , but also overviews like the Adversarial ML Threat Matrix 13 . Our findings have concrete implications for such tools. The confusion between AML and classical security suggests for example that workflows and tools need to either enforce dedicated audits for both classical and AML security or united countermeasures to address both areas jointly. In addition, our findings described as the range of technical depth suggests that tools can/should be developed at different levels of abstraction. Such levels of abstraction can be a very general project context (S6, S11, S16, S20) or at a very low technical depth (S4, S7, S9, S14). Furthermore, the application area and the individual variation help to understand how specific these tools need or are able to be. Embedding AML into corporate workflows. Whereas academia generally studies AML with the perspective of an individual model, in practice, the entire ML pipeline and broader AI workflow need to be considered. In our interviews, for example S6 and S14 described the entire workflow of their AI application, whereas other subjects focused on the ML pipelines. To successfully integrate AML into corporate workflows, however, more effort is needed. All actors working on an ML product need to be able to identify relevant and possible attacks and implementable defenses. Our work shows that it is crucial to provide information that is useful regardless of the level of abstraction or individual variations. Ideally, suchlike information are oriented along the functional and structural properties we found. In general, practitioners should be able to understand AML through minimum viable mental models with the lowest possible number of cognitive chunks [63] , [70] . If necessary, the provided information should be sufficient to develop initial mental models into more accurate internal representations. These internal representations contain then the potential security threats within the corresponding application, and our work provides a first step towards understanding these models. Creating appropriate regulatory frameworks for AML. Lastly, our study has implications for regulatory approaches that enable appropriate security assessments. The individual differences we found imply that regulatory frameworks need to find a way to formally deal with these individual differences with regards to necessary security measures. The currently proposed 'Legal Framework for AI' by the European Commission, for example, differentiates certain types of ML applications of which some are prohibited or classified as high-risk and require a certain risk management. Furthermore, the range we named 'technical depth' is essential to communicate such frameworks to both technical ML practitioners and non-technical stakeholders. In general, to develop and refine adequate mental models, practitioners need to be knowledgeable in AML. Future regulation could incorporate this requirement by providing adequate information at multiple mental abstraction levels [78] . For example, current regulatory drafts like the NIST Taxonomy and Terminology of AML 14 ease a functional understanding of attacks and defenses. This draft explicitly lists references that might help practitioners develop more complex mental models. 15 A similar regulation for privacy, the European general data protection regulation, was often mentioned by our subjects. With the regulation serving as scaffold for their privacy perception, they reported to comply even though we did not ask explicitly about privacy beyond membership inference. Our findings underline the need for additional research at the intersection of AML and cognitive science. In this section, we summarize potential directions of future work. We begin with three aspects of mental models: (I) a refinement of the mental models presented in this paper, (II) their temporal involvement, and (III) usercentric taxonomies. Afterwards, we discuss the usability of AML tools and conclude with the assessment of AML in practice. Refinement of ranges. As stated at the beginning of Section 4, our ranges are a first step towards a refined mental model of AML, and thus more work is needed to shed light on the details of these ranges. This includes, but is not limited, to a deep understanding of why security and AML are mingled, an extensive study on the influence of prior knowledge and different applications. The latter might help us understand why subjects doubt certain attackers, as described in Section 4.1.2, and whether this is justified in some cases. Along these lines, also the externalization of responsibility described in Section 4.1.2 needs a more in depth understanding. Temporal evolvement of mental models in AML. A better understanding about the development of individual mental models could help to assess necessary steps to make practitioners take into account AML. In addition, research on how mental models are shared between various AI practitioners might help to implement adequate defenses within and across corporate workflows. Corresponding starting points can be found in cognitive science [79] , where the convergence of mental models has been studied as a three-phase process of orientation, differentiation and integration [80] . User-centric threat taxonomy. More work is also needed to understand why and how industrial practitioners relate classical security and AML. Here, it could be interesting to consider the taxonomies proposed by Biggio et al. [81] and Barreno et al. [1] . These frameworks seem promising to investigate which specific structural elements practitioners consider relevant for specific attack vectors and how they perceive the causal evolvement of these 14 attacks. In line with recent work by Wang et al. [82] , such user-centric attack taxonomies might help to understand practitioners' reasoning on AML. Utility and usability of AML tools and libraries. Furthermore, we found that practitioners' mental models depend on available and provided information. Future research should therefore elaborate on the needed specificity of the available information. Furthermore, an evaluation of the available AML tools and libraries with regards to capabilities and needs of industrial practitioners might ease their usage across application domains. In line with recent work on fairness [83] and ethics [84] , we consider this crucial for designing usable and accessible tools, corporate guidelines and regulations. AML in the wild. Given the evidence of semiautomated, ML-related fraud, a more detailed assessment of which attacks are conducted in the wild would be greatly beneficial. Future work could investigate this with focus on different groups of ML practitioners, including for example ML engineers, auditors, and researchers. Furthermore, our work outlines that the model perspective usually taken in AML is of limited use in practice. More work is needed to study AML in the context of entire ML pipelines and end-to-end workflows. We followed an inductive approach to investigate mental models through qualitative analysis. Hence, the data collected is self-reported and subjected to a coding process. We continued coding and refining codes until a good level of inter-coder agreement was reached. Nonetheless, all our findings are subject to interpretation which is inherent to qualitative analysis. Finally, due to the COVID-19 pandemic, all interviews were conducted remotely and the interface limitations of the digital whiteboard might have impacted the participants' sketches. With 15 participants, our sample size is rather small and limits the generalizability of our findings. However, given the applied methods and that we reached saturation, the size is indeed acceptable [57] , [19] . All participants were employed at European organizations with <200 employees. This is due to the fact that while several multinational companies stated great interest in our research, they denied participation after internal risk assessments. As mental models of ML systems are always embedded in organizational practices [67] , we strongly encourage future research to assess our findings within larger samples including more variety, for example academics, small and large companies, etc. Finally, despite our efforts, we only managed to recruit one female participant. Hence, we cannot exclude the possibility that our findings are biased. Furthermore, AML itself is a subject of study of which the perception evolves continuously. With an increasing awareness for security within applied machine learning, the findings presented can only be valid temporarily. Machine learning is applied in a wide range of settings. Consequently, not all attacks are relevant within each application domain. For example, a healthcare setting is subjected to other threats than a cybersecurity setting. For the sake of studying abstract ranges of mental models, we did not consider the application in the present work. Yet, we would like to point out the necessity to study this aspect of AML in general. Based on our semi-structured interviews with industrial practitioners, we take a first step towards a theory of mental models of AML. We identified four characteristic ranges of practitioners' mental models. The first range describes the relationship between AML and classical IT security. These two topics were often mingled, yet not used interchangeably by our subjects. The second range confirmed the existence of structural and functional components within mental models of (A)ML. For example, some subjects marked a structural component as a starting point for and attack, whereas other subjects explained the causal steps of the attack. The third range concerns the general variability in our subjects' mental models, which in we found to be independent in our sample from the application domain and reported background knowledge. This included the priority AML has for subjects: whereas some uttered clear concern, other subjects were not worried at all. Finally, the fourth characteristic range describes that industrial practitioners perceive ML-specific threats and defenses at a varying level of technical depth. Whereas some subjects for example explained pipeline elements and attacks almost at the code level, other subjects made only high level references. A clear understanding of the elicited mental models allows to improve information for practitioners and adjustments of corporate workflows. More concretely, our results help to develop tools for practitioners that assess the security of ML. These tools should be incorporated into the ML workflow to ease security evaluation and minimize risks. Finally, regulatory frameworks might reduce uncertainty about AML and increase the awareness for possible security threats. However, a wide range of subsequent research towards an encompassing theory of mental models in AML is still required. Last but nor least, we are convinced that the AML community will benefit from further practical assessment of attacks occurring in the wild, as our work already provides evidence of semiautomated fraud in the wild. We also investigated the familiarity of our subjects with AML attacks. To avoid priming, we asked subjects to rate their familiarity after the interview. As sanity checks, we added two rather unknown terms, adversarial initialization [25] and neural trojans [85] (similar to backdoors). The results are depicted in Figure 8 . Only one subject reported to be familiar with one attack (evasion). In general, most subjects reported to have heard of most common attacks (evasion, poisoning, membership inference, and model stealing). As expected for the sanity check, adversarial initialization and neural trojans were largely unknown. Thank you so much for taking the time to give us your perspective on security in machine learning. This study consists in III parts. Part I aims at exploring your role in ML-projects. Part II addresses the underlying machine learning pipeline. In part III, we want to know how you perceive the security of machine learning. In part II and III, please visualize the topics (and relationships between them) that we ask you about. There are no rules, no wrong way to do it, and don't worry about spelling things perfectly. Nothing is off limits and you can use any feature of the digital whiteboard. After this last part, we will ask you about your knowledge about security of machine learning before this study. • Have you encountered any issues relating to security in the projects you described? • Where in the pipeline did these security-related issues originate? • Can you specify the cause of the security-related issues? • Can you specify how these security-related issues evolve in your pipeline? • Which goal pursues an adversary with a such a threat? • What is the security violation of the threat? • How specific is the depicted threat? • Are you aware of any further possible security threats in the scope of your project or pipeline? • Which countermeasures do you implement against any of the aforementioned threats? Thank you so much for taking the time to give us your perspective on security in machine learning. 3. Questionnaires of our study 3 This attack targets a model during deployment. The goal of the attacker is to fool the model: changing its output significantly by altering the input only slightly. An example is to change a picture containing a dog, present it to a cat-dog-classifier, and the model's output changes from dog to cat. Poisoning. This attack targets the training or optimization phase of the model. The goal of the attacker is to either decrease accuracy significantly, or to install a backdoor. An example is a cat-dog classifier that always classifies images containing a smiley as cat. Privacy/ Membership Inference. This attack targets a model at test-time. The attacker's goal is to identify individual samples from or even the whole training set. An example is to measure the confidence on an input, as some algorithms tend to be more confident on data they have seen during training. Also over-fitting eases to determine what a classifier was trained on. Please answer the following questions about ML. For each question, please tick at least one box. 0/1-loss. Cross-entropy loss. Hinge-loss. Question 2. What is the difference between classification and regression? The kind of labels we fit: reals vs discrete classes. Regression is the name of classification in psychology / medical science. Regression is for discrete labels, classification for real valued ones. Question 3. What is the difference between L 1 and L 2 regularization? L 1 yields sparser solutions. L 2 yields sparser solutions. none -they differ only in few practical applications. Question 4. In the bias-variance trade-off, what does high variance imply? The analyzed data shows high variance. The classifier is overly complex and potentially overfits. The data is likely to be classified fair (e.g., low bias). Question 5. Why is Naive Bayes naive? Due to historic reasons. Due to the assumption that all features are independent. Because the application is simple and straightforward. Question 6. What is cross-validation? Training on one task and then transferring the model to another task. Splitting the dataset and training/evaluating on different subsets. A method to reduce overfitting or choosing hyperparameters. Question 7. What are kernels in machine learning? Essentially similarity functions. A part of SVM, potentially yielding non-linear SVM. A specific instance of a similarity function used in SVM. Question 8. What is pruning? Deletion of for example weights in a model. Deletion of specific points of the data. A technique to get a smaller from a large model with similar performance. To conclude the study, we will ask you to rate your background knowledge on attacks before this study according to the following four classes: • Familiar. The final set of codes for the interviews is depicted in Table 2 . Analogously, the codes for the drawings can be found in Table 3 . Can machine learning be secure Wild patterns: Ten years after the rise of adversarial machine learning A marauder's map of security and privacy in machine learning: An overview of current and future research directions for making machine learning secure and private Adversarial classification Evasion attacks against machine learning at test time Intriguing properties of neural networks Antidote: understanding and defending against poisoning of anomaly detectors Support vector machines under adversarial label noise Backdoor attacks against learning systems Targeted backdoor attacks on deep learning systems using data poisoning Adversarial examples are not easily detected: Bypassing ten detection methods Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples Bypassing backdoor detection algorithms in deep learning Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness Mental Models. Erlbaum Bridging the gap in computer security warnings: A mental model approach i don't own the data": End user perceptions of smart home device data practices and risks Influencing mental models of security: a research agenda New me: Understanding expert and non-expert perceptions and usage of the tor anonymity network Trojaning attack on neural networks Stealing machine learning models via prediction apis Towards reverse-engineering black-box neural networks Membership inference attacks against machine learning models Adversarial reprogramming of neural networks On the security relevance of initial weights in deep neural networks On the robustness of convolutional neural networks to internal architecture and weight perturbations Sponge examples: Energy-latency attacks on neural networks Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers Machine learning with membership privacy using adversarial regularization Attriguard: A practical defense against attribute inference attacks via adversarial machine learning Memguard: Defending against black-box membership inference attacks via adversarial examples Prada: protecting against dnn model stealing attacks Prediction poisoning: Towards defenses against model stealing attacks Maximally fault tolerant neural networks Practical fault attack on deep neural networks Backdooring convolutional neural networks via targeted weight perturbations Model-reuse attacks on deep learning systems Bit error robustness for energy-efficient dnn accelerators Towards certificated model robustness against weight perturbations Bad global minima exist and sgd can reach them. ICML Motivating the rules of the game for adversarial example research Security and machine learning in the real world Adversarial machine learning-industry perspectives Mental models concepts for system dynamics research How to t (r) ap users' mental models Mental models-general introduction and review of their application to human-centred security Mental models: concepts for human-computer interaction research The design of browsing and berrypicking techniques for the online search interface Integrating culture into interface design The design of auditory user interfaces for blind users End user and expert perceptions of threats and potential countermeasures Why doesn't jane protect her privacy? In PETS should i worry?" a cross-cultural examination of account security incident response my data just goes everywhere:" user mental models of the internet and implications for privacy and security Influence of mental models on the design of cyber security dashboards End user security and privacy concerns with smart homes When is a tree really a truck? exploring mental models of encryption Exploring user mental models of end-to-end encrypted communication tools if https were secure, i wouldn't need 2fa"-end user and administrator mental models of https User mental models of cryptocurrency systems -a grounded theory approach Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 'it's reducing a human being to a percentage' perceptions of justice in algorithmic decisions Believe it or not: Designing a human-ai partnership for mixedinitiative fact-checking Human evaluation of models built for interpretability Interpreting interpretability: Understanding data scientists' use of interpretability tools for machine learning Can children understand machine learning concepts? the effect of uncovering black boxes Understanding mental models of ai through player-ai interaction How do data science workers collaborate? roles, workflows, and tools Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai Interpretable decision sets: A joint framework for description and prediction Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs What do we want from explainable artificial intelligence (xai)? -a stakeholder perspective on xai and a conceptual model guiding interdisciplinary xai research Explainable machine learning in deployment Improving fairness in machine learning systems: What do industry practitioners need? In CHI Why do developers get password storage wrong? a qualitative usability study Basics of qualitative research A coefficient of agreement for nominal scales The measurement of observer agreement for categorical data Psychological foundations of explainability and interpretability in artificial intelligence Metaphor no more: A 15-year review of the team mental model construct Merging internal and external processes: Examining the mental model convergence process through team communication Pattern recognition systems under attack: Design issues and research challenges Designing theory-driven user-centric explainable ai The landscape and gaps in open source fairness toolkits. Available at SSRN Surveying the landscape of ethics-focused design methods Neural trojans The authors would like to thank Battista Biggio, Antoine Gautier and Michael Schilling for their helpful feedback. This work was supported by the German Federal Ministry of Education and Research (BMBF) through funding for the Center for IT-Security, Privacy and Accountability (CISPA) (FKZ: 16KIS0753) and by BMK, BMDW and the Province of Upper Austria in the within the COMET program managed by FFG in the COMET S3AI module.