key: cord-0230780-38ia42fr
authors: Dubiel, Mateusz; Yilma, Bereket Abera; Latifzadeh, Kayhan; Leiva, Luis A.
title: A Contextual Framework for Adaptive User Interfaces: Modelling the Interaction Environment
date: 2022-03-31
journal: nan
DOI: nan
sha: 0ec57618b63eb24d2c93c14d6732c6b866cb5109
doc_id: 230780
cord_uid: 38ia42fr

The interaction context (or environment) is key to any HCI task and especially to adaptive user interfaces (AUIs), since it represents the conditions under which users interact with computers. Unfortunately, there are currently no formal representations to model said interaction context. In order to address this gap, we propose a contextual framework for AUIs and illustrate a practical applica- tion using learning management systems as a case study. We also discuss limitations of our framework and offer discussion points about the realisation of truly context-aware AUIs.

Research in Adaptive User Interfaces (AUIs) has provided empirical evidence about the importance of contextual information from the user's environment to aid personalisation of UIs [Shankar et al. 2007 ]. Although there are several works on context-aware AUIs that explore the role of said environment, this research area remains open. In practice, users evolve in a physical space together with other people and devices that often tend to influence the behaviour of the users. Additionally, a physical space by itself has certain rules that govern user behaviour and impose additional constraints on * Both authors contributed equally to this research. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). CHI 22 Adaptive UIs Workshop, April 30th, 2022, New Orleans, LA © 2022 Copyright held by the owner/author(s). how users interact with computers, which ultimately determines how interfaces should be adapted. Nevertheless, classic approaches to AUIs focus on cues and signals solely collected from users to personalise interfaces, largely ignoring contextual information. In this work we propose a general framework which allows to incorporate the user's interaction context (or environment) in AUIs by design.

In order to provide contextual grounding for our framework we first briefly overview related work on AUIs (section 2). Then, in section 3, we formalise the problem by taking a systems theory perspective [Maier 1996] , and present our proposed framework inspired by a classic reinforcement learning architecture [Sutton and Barto 1998; Watkins and Dayan 1992] . In section 4 we present a case study that illustrates a potential application of our framework. We finish with implications of our proposal and provide several points for discussion and future work (section 5).

Many studies in AUIs explore the role of user's feedback on interface personalisation. Some types of feedback collected include: using natural language programming to optimise behaviour of artificial agents and task disambiguation [Jiang et al. 2019] ; introducing head-gesture analysis to facilitate hands-free interaction with Head Mounted displays [Yan et al. 2018] ; enabling silent-speech interaction mode with a mobile device by analysing user's mouth opening degree [Sun et al. 2018] ; utilising users' mouse behaviour to automatically modify layout of a website [Leiva 2012 ]; or using user body postures and physical activity to adjust the features of the surrounding environment such as ambient light, temperature or music [Wang et al. 2019] .

Another line of research focused on system features that can be used for interface adaptation. For instance, Jiang et al. [Jiang et al. 2019] proposed a novel layout method that adds OR-constraints (ORC) to standard constraint-based layout specifications. The proposed method unifies grid layout and flow layout, offering new possibilities for flexible UIs that were not supported by any other layout method. In another study, Swearngin et al. [Swearngin et al. 2020 ] proposed the Scout system to help designers rapidly explore alternative interface layouts through mixed-initiative interaction with high-level constraints (e.g. semantic structure, emphasis, order) and design feedback. arXiv:2203.16882v1 [cs.HC] 31 Mar 2022

Previous formalizations of AUIs have approached the adaptation problem from different angles. One notable example in this area is the Supple system [Gajos and Weld 2004] which framed UI adaptation as an optimisation problem. Supple minimises user's effort by providing interface adaptations that meet device and user capabilities. However, Supple did not account for the modeling of environment features.

Similarly, Bouzit et al. [Bouzit et al. 2017 ] formalised a design space for user interface adaptation, alas without modelling the environment. The proposed framework is based on Perception-Decision-Action cycle that is augmented by Learning-Prediction-Action, allowing for UI designs that are descriptive, comparative, and generative.

On the other hand, Abrahão et al. [Abrahão et al. 2021 ] proposed a conceptual reference framework for intelligent user interface adaptation with a set of conceptual adaptation properties that are crucial for model-based AUIs. While the authors mention an "Environmental Model" as a part of their "Context Model" that affects system adaptation, there is no formulation provided regarding the elements that define the environment and the role of their interactions.

Overall, while previous work on proposed conceptual modelling of the system, to the best of our knowledge, conceptual modelling of the environment has not been proposed yet. Moreover, while taxonomies of interaction environment exist there are currently no formalisation of this concept. Bearing in mind the importance of contextual information that can be obtained from interaction environment and benefits that it can be for development of AUIs, in the current paper we propose a conceptual framework for modelling the environment.

We propose that the environment of the user can be understood as a System-of-Systems (SoS) which contains sub-systems such as the user, the device which implements the adaptation or personalisation, other users and entities in the same physical space with the user that may have a direct or indirect influence on their behaviour or on the personalisation. Each of these sub-systems posses their own systemic properties; i.e. components, objectives, relations, behaviour, structure, interface, environment, and functions [Maier 1996 ]. Taking such a systemic view allows to better understand the coexisting entities, their inter-dependencies, as well as their influence on the environment and vice versa. Thus, we can formalise the problem of UI adaptation as a function of the composing systems, given by

where is the user, is the device to be adapted through its interface, and is the environment potentially influencing the user as an independent system as well as containing subsystems itself, such as other people and devices that may have an impact on the user. Adapting UIs often entails changing some functionalities such as information content, presentation layout, or distinctiveness of an interface to increase its personal relevance to the user. Doing so requires taking into account not only the user but also environmental constraints and potential influences imposed on the user. This essentially means devising an efficient strategy to understand and reason about dynamic interaction responses of the user and, consequently, adapting the UI to the environmental changes. Implementing this, however, is not a trivial task as it requires to derive efficient representations of the user, the coexisting entities, and the environment's state from high-dimensional sensory inputs, and use these information sources to generalise past experiences to new situations in order to better adapt UIs.

Such types of challenging tasks are remarkably solved by humans and other animals through a harmonious combination of reinforcement learning (RL) and hierarchical sensory processing systems [Fukushima and Miyake 1982; Serre et al. 2005 ]. This in particular has inspired the development of several RL algorithms over the years, cf. Nguyen et al. . Early RL algorithms were limited to domains in which useful features could be handcrafted, or to domains with fully-observed low-dimensional state spaces.

Recently, Deep Q-networks (DQNs) can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning [Mnih and Kavukcuoglu 2017] . DQNs have been tested in various complicated tasks and were able to outperform all previous RL algorithms [Silver et al. 2016 [Silver et al. , 2017 . DQNs have also enabled breakthroughs such as "AlphaGO" [Chen 2016] and"AlphaStar" [Arulkumaran et al. 2019] , which have inspired recent work on AUIs in the context of linear menus [Todi et al. 2021 ]. These advancements demonstrate the potential of RL to build intelligent agents by giving them the freedom to learn by exploring their environment and make decisions to take actions that maximise a long term reward.

We believe that RL can be highly beneficial to AUIs, as it allows learning through exploration, unlike classic approaches and supervised methods, that often require large amounts of labelled data and are harder to train with continuous action spaces. Taking this inspiration, in the following we reformulate the goal of AUIs as an RL task by extending the high-level formalisation in Equation 1.

In classic RL, agents interact with their environment through a sequence of observations, actions, and rewards [Watkins and Dayan 1992] . At a given time, an agent takes an observation (i.e., information about the state of the environment) and takes an action that will maximise a long-term reward. The agent then observes the consequence of the action on the state of the environment and the associated reward. It then continues to make decisions about which actions to take in a way that maximises the cumulative future reward. This is done by learning action value function:

which is the maximum sum of rewards discounted by at each time step , achievable by a policy = ( | ), after making an observation of and taking an action . This means that RL agents operate based on a policy to approximate -values (stateaction pairs) that maximise a future reward. Figure 1 illustrates the schematics of the different components in classic RL. We refer the reader to the work of Watkins and Dayan [Sutton and Barto 1998; Watkins and Dayan 1992] for the details on Q-learning and RL. Adopting this to the context of AUIs, the agent corresponds to the device which operates based on some policy . The action taken by the device at any time step corresponds to selecting an optimal UI configuration layout denoted by x = ( 1 , . . . , ) representing a configuration over features. The observation state then corresponds to the combination of the state of the user and the state of the environment . The reward the device receives for selecting a configuration x is the sum of the rewards deemed appropriate for the corresponding states of the user and the environment: = + . Figure 2 illustrates the problem of AUIs as an RL task.

Following the classic RL formulation, at each step the approximation of the optimal Q-value function * will be refined by enforcing the Bellman equation [Watkins and Dayan 1992] , which can be reformulated for our AUI setting by substituting an action with the task of selecting a configuration x given by * ( , x) = E ′ ∼ + max

which states that given any state-configuration pair and x, the maximum cumulative reward achieved is the sum of the reward for that pair, , plus the value of the next state we end up with, ′ . The value at state ′ will be the maximum over all possible configurations x ′ at * ( ′ , x ′ ). Thus, the optimal policy * corresponds to selecting the best UI configuration in any state, as specified by * . In this iterative process, the Bellman equation is used as a valueiteration algorithm to refine * :

where converges to * as approaches to infinity. In the context of AUIs, we are interested in finding an optimal policy on which the device operates in order to select the best possible configuration given the states of the user and the environment. Since users in such settings experience cognitive and physical workload, they respond differently depending on: individual skills, UI familiarity, preferences, etc. In particular, these user states may correspond to implicit interactions which are often hard to detect and analyse. Nevertheless, considerable advances have been made in inferring emotional, cognitive, and behavioural states through response monitoring of various biosignals [Dinh et al. 2020] , for example electrocardiography [Cairns et al. 2016] , electroencephalography [Höller et al. 2017] , or eye-gaze [Menges et al. 2019 ] and mouse [Brückner et al. 2021] movements. Thus, such biosignals can be leveraged as an underlining technique of our approach to iteratively infer user states to find the best personalised UI configuration through an optimal policy given the inferred user states.

With the COVID-19 pandemic, the use of learning management systems has increased dramatically [Szopiński and Bachnik 2022] .

Since these systems can be accessed with different devices (e.g. desktop, mobile, tablet) and in various environments (e.g. home-office, conference rooms, common areas in universities), it is important that the UI can consider these environments for adaptation. For the purpose of demonstrating our proposed framework, we will consider a use case where an AUI accounts for changes in environment in order to improve the user's remote learning experience.

Mark is a student participating in a remote lecture that is broadcasted live on an online learning platform. Mark is attending from his apartment (environment(e)), which he shares with other students. When the lecture starts, Mark joins from his living room using a laptop and listens to the audio via laptop speakers, as he does not have any headset available. The current layout of the content presentation on his laptop (device (d)) corresponds to a configuration (x) chosen by the AUI. Some time later, a persistent drilling noise starts coming out from his neighbour's flat, so the AUI automatically activates closed captions. The captions appear to be too small to read, which makes Mark lean closer to the screen. The AUI detects this movement and modifies the font size to improve readability. After 20 minutes, the drilling noise stops and the QA session starts, so Mark switches on his microphone to ask a question. As he begins to speak, suddenly one of his flatmates enters the living room. The AUI recognises a visual change in the environment and automatically blurs Mark's background to minimise the distracting impact that it may have on other students who are currently attending the lecture. Shortly after the doorbell rings. Mark is now alone in the apartment so he should answer the door. However, since the QA session is still going on, Mark continues listening to it via his mobile phone as he walks towards the door. The AUI modifies the layout to account for the limitations of the small screen display. The lecture concludes as Mark opens the door: it is a deliveryman with the headphones that he recently ordered.

In the presented scenario the changes in the environment such as the drilling noise (acoustic signal), someone entering the room (background visual signal), walking towards the door (posture change), etc. are signals collected from the user as well as the environment that serve as input to interpret the state of user ( ) and environment ( ). In RL, rewards guide the exploratory nature of an agent. Hence, in our proposed framework, the device expects some form of feedback from the user and the environment in the form of a reward (r), either positive or negative, for choosing the content presentation (x) given their observed states. It should be noted that modelling such a reward is rather a challenging task due to difficulties in interpreting the true meaning of multi-modal signals collected from the users and their environment. Nonetheless, the above use case illustrates how modelling and responding to the changes in user environment can contribute to an improved learning experience by using signals capture from the environment to automatically adapt user interface to different interaction circumstances.

Although our framework paves a path towards incorporating the environment in the design of AUIs, the RL formulation suffers from a scalability problem. This is due to the fact that one must compute ( , x) for every state-configuration pair in order to select the best UI configuration. This is computationally infeasible as the configuration space is potentially infinite, only limited by the number of elements that can be adapted. Take for example the domain of web applications: with CSS and JavaScript it is possible to modify any UI element at will [Leiva 2011 ]. Recent research in RL has addressed the scalability problem by using function approximators, typically a neural network, to estimate the action-value function ( , ; ) ≈ * ( , ) where is the trainable parameters (weights) of the neural network [Mnih and Kavukcuoglu 2017] . Deep Q-learning is one of the most commonly used techniques to approximate optimal action-value functions using a neural network. Hence, we can define our Q-function approximator using a neural network too. This means that, in the forward pass while training the network, we use a loss function to minimise the error of the Bellman equation, thereby determining how far ( , x) is from the target * ( , x), given by

Then, then backward pass is a gradient update with respect to the Q-function parameters . It is also evident from recent works that there are different variations of DQN that have enjoyed a huge success in RL tasks such as Actor-Critic methods [Xiang et al. 2019] , which combine DQN with Deep Deterministic Policy-Gradient Algorithms (DDPG) [Lillicrap et al. 2015 ] and a multi-agent version of actor-critic methods [Lowe et al. 2017; Ryu et al. 2020] . We are optimistic that our proposed framework, complemented by such techniques, may overcome scalability issues and open new, interesting research opportunities in AUIs, as it enables adaptation by learning through exploration.

Overall, our proposed contextual framework can bring several practical benefits. Firstly, it allows a contextual modelling approach of AUIs that incorporates environment and accounts for its constituent elements. Secondly, framing the problem as an RL task opens possibility of multidisciplinary research at the intersection of Machine Learning, Cognitive Science and Psychology. Thirdly, if applied in practise, the RL architecture empowers us to learn complex interaction responses of users and their environment through exploration, which, in turn, can lead to design of more seamless and user-friendly AUIs. Furthermore, although we have used a single user setting to illustrate the proposed architecture, it could be extended to more complicated interaction scenarios such as multi-user setting and group adaptation, where single adaptation is applied to multiple users.

Nevertheless, we would like to note that there are several open questions that need to be addressed before the practical implementation of our framework becomes feasible. Firstly, determining the reward function poses a challenge as it requires keeping the user in the loop in order to provide their feedback based on system's actions. Secondly, the system needs to correctly interpret and act on a multitude of signals from the environment that may potentially have contradictory meaning. Thirdly, due to complexity of the environment, the neural network requires to be exposed to a large number of examples which makes training extremely time consuming and negatively impacts scalability of the system. We anticipate that by concerted collaborative efforts, the AUI community will begin to address these challenges to pave way to practical implementation of the proposed contextual framework in the future.

This work was supported by the Horizon 2020 FET program of the European Union through the ERA-NET Cofund funding grant CHIST-ERA-20-BCI-001.

Modelbased intelligent user interface adaptation: challenges and future directions

AlphaStar: An Evolutionary Computation Perspective

The PDA-LPA design space for user interface adaptation

When Choice Happens: A Systematic Examination of Mouse Movement Length for Decision Making in Web Search

A computer-human interaction model to improve the diagnostic accuracy and clinical decision-making during 12-lead electrocardiogram interpretation

The evolution of computing: AlphaGo

Stretchable respiration sensors: Advanced designs and multifunctional platforms for wearable physiological monitoring

Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition

SUPPLE: automatically generating user interfaces

Reliability of EEG Measures of Interaction: A Paradigm Shift Is Needed to Fight the Reproducibility Crisis

ORC layout: Adaptive GUI layout with OR-constraints

Automatic web design refinements based on collective user behavior

Restyling Website Design via Touch-Based Interactions

Continuous control with deep reinforcement learning

Multi-agent actor-critic for mixed cooperative-competitive environments

Architecting Principles for Systems-of-Systems

Improving User Experience of Eye Tracking-Based Interaction: Introspecting and Adapting Interfaces

Methods and apparatus for reinforcement learning

Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications

Multi-agent actor-critic with hierarchical graph attention network

Object recognition with features inspired by visual cortex

User-context for adaptive user interfaces

Mastering the game of Go with deep neural networks and tree search

Mastering the game of go without human knowledge

Lip-interact: Improving mobile device interaction with silent speech commands

Introduction to reinforcement learning

Scout: Rapid exploration of interface layout alternatives through high-level design constraints

Student evaluation of online learning during the COVID-19 pandemic

Adapting User Interfaces with Model-Based Reinforcement Learning

Flextouch: Enabling large-scale interaction sensing beyond touchscreens using flexible and conductive materials

Q-learning

Continuous control with deep reinforcement learning for mobile robot navigation

Headgesture: hands-free input approach leveraging head movements for hmd devices