key: cord-0305307-zr022ets
authors: Schneegass, Christina; Sigethy, Sophia; Eiband, Malin; Buschek, Daniel
title: Comparing Concepts for Embedding Second-Language Vocabulary Acquisition into Everyday Smartphone Interactions
date: 2021-09-16
journal: nan
DOI: 10.1145/3473856.3473863
sha: a7edcf0cb76b1c5f19d5d17f4fd7b74aa734c5e5
doc_id: 305307
cord_uid: zr022ets

We present a three-week within-subject field study comparing three mobile language learning (MLL) applications with varying levels of integration into everyday smartphone interactions: We designed a novel (1) UnlockApp that presents a vocabulary task with each authentication event, nudging users towards short frequent learning sessions. We compare it with a (2) NotificationApp that displays vocabulary tasks in a push notification in the status bar, which is always visible but learning needs to be user-initiated, and a (3) StandardApp that requires users to start in-app learning actively. Our study is the first to directly compare these embedding concepts for MLL, showing that integrating vocabulary learning into everyday smartphone interactions via UnlockApp and NotificationApp increases the number of answers. However, users show individual subjective preferences. Based on our results, we discuss the trade-off between higher content exposure and disturbance, and the related challenges and opportunities of embedding learning seamlessly into everyday mobile interactions.

as lockscreen wallpapers [8] . These concepts aim to increase users' exposure to the learning content by maximising the visibility of the tasks [8] or by exploiting idle moments such as waiting situations [5, 10] . While all these concepts are successfully evaluated individually, there has not yet been a comparative evaluation investigating different levels of embedding learning into everyday smartphone interactions.

To address this gap, we conducted a three-week within-subject field study, which compares users' interactions and exposure to learning content using three learning apps that represent different embedding levels, described as follows:

(1) First, our novel concept called UnlockApp extends the concept of Dearman and Truong's [8] learning wallpaper by connecting a simple multiple-choice vocabulary task that the user can answer by a single button press with the phone unlock action. Although users can still dismiss the learning task, this concept is strongly embedded into the unlock action as the app initiates the learning task. We aim to nudge users toward more frequent learning by lowering the threshold for users to interact with the content.

(2) Second, our notification learning app (NotificationApp) presents a learning task in a continuously shown notification. The notification increases learning task's visibility during interactions with the lockscreen or status bar but still requires users to actively initiate learning.

(3) Third, we implement a StandardApp, following a baseline self-contained and self-initiated learning design.

In our in-the-wild evaluation, we investigate how the different degrees of embedding and self-initiated vs app-initiated exposures impact interactions with the learning content over the day and how users perceive this form of learning in their everyday use.

The results show that the UnlockApp and NotificationApp designs, compared to the StandardApp, significantly increase the number of learning tasks users answer per day. The UnlockApp, in particular, further distributes the learning more evenly across the day. Interestingly, we observe a great variety in participants' subjective perceptions of the three concepts. While some people like that the UnlockApp nudges them to engage with the vocabulary regularly, others favored the NotificationApp, arguing that it is less distracting, especially when unlocking their phone with a specific goal in mind. In contrast, when participants had time to spare, they enjoyed long learning streaks with the StandardApp.

Based on these comparisons, we discuss challenges and opportunities of embedding learning tasks more ubiquitously into peoples' everyday lives. In particular, combinations of features from the different concepts and offering further personalisation options can target individual user preferences and balance the trade-off between nudging users to learn and disturbing routine smartphone tasks.

The ubiquity of mobile devices fosters the shift from learning being a classroom-based activity to an activity that is embedded into our daily life. By applying the micro-learning paradigm, learning content is broken down into small bits of information and presented to learners using simple interaction concepts [3] . Hereby, the learning task is spread out over time. Especially for vocabulary acquisition, a higher number of short interactions should be favored over one long learning streak [7] . By spacing out the learning content, prior work has demonstrated that learners could memorise vocabulary content better than immediate successive presentation [9] . The concept of "spaced-repetition" suggests that repeating learning content according to a repetitive schedule helps consolidate the memory of it [1, 29] . This schedule targets the transient forgetting of information that naturally occurs after a certain amount of time of the information is not rehearsed or applied [28] . As MLL apps require self-directed learning, it needs to be taken into account that the success of autonomous learning and the adherence to rehearsal schedules depends on the consideration of various dimensions (i.e. context of learners, learner interest and motivation, language proficiency, etc.) [21, 23] . Thus, the overall goal of micro-learning is to maximise the degree of exposure with the learning content due to increased presentation and, thus, frequent repetition of content. In particular for the teaching of languages, the application of the micro-learning approach has proven to improve vocabulary acquisition and recall [5, 11, 31] . Prior work presented different ideas for realizing micro-learning in mobile learning. For example, Dearman and Truong [8] implemented a mobile language learning application that can display learning tasks (i.e. vocabulary translations with three multiple-choice options) on the lockscreen. The authors show that users frequently interacted with the application and improved their knowledge of the language over the course of this study. Dingler et al. [10] achieved similar results by implementing and evaluating a vocabulary application that presents tasks in push-notification. Their app "QuickLearn" detects opportune moments for the presentation of the learning content, in particular, when users are bored and use their smartphone without a specific purpose. Dingler et al. [10] concluded that their application increased the number of "quick" learning sessions on the go, which was appreciated by their participants.

Users' interactions with their mobile devices can range in their level of intensity. Prior work has shown that half of our interactions with our smartphones take less than 30 seconds, with only one in ten interactions exceeding four minutes [32] . In particular, many interactions do not even reach the threshold of fifteen seconds, what Ferreira et al. [12] defined as so-called "micro-usage" situations. For example, users briefly reply to a message, check their phone for new push notifications, or want to know the time. For many of those interactions, users do not unlock their device [15, 16] or only use one application after unlocking [24] . However, unlocking the mobile device is still an action that happens frequently throughout the day. Mahfouz et al. [24] report that users unlock their phone on average 46 times, while other studies report an average of between 25 [15] and 47 unlocks [14] per day per participant. As this interaction has no other purpose than the authentication, prior work suggests combining the authentication event with a micro-interaction task, in particular, when the slide-to-unlock authentication is used. Truong et al. [30] emphasised that authentication gestures serve no purpose other than to unlock the phone and outline their potential to serve other tasks such as data collection. Therefore, it has been proposed as a quick and easy method to enter journaling data [33] , perform nutrition tracking [18] , or sleep tracking [6] .

In summary, micro-learning applications for vocabulary-based language learning have proven to increase vocabulary recall [5, 11, 31] and have shown great potential for pervasively embedding learning into everyday smartphone usage [8, 10] . Yet, it remains unclear how best to embed the learning task into everyday device use. This motivates our idea and investigation of our new concept of integrating learning directly into the authentication process, by displaying a vocabulary task every time the phone is unlocked using any form of authentication mechanism. Furthermore, to the best of our knowledge, no comparative analysis has been performed yet to investigate potential differences between the proposed pervasive presentation of vocabulary and a "standard" in-app learning application that requires the user to actively initiate learning. For example, while Dearman and Truong [8] showed a knowledge gain through their wallpaper app, they do not implement a "standard" app as a control condition.

We implemented three concepts with varying levels of integration following existing research ideas presented in Section 2.1, as Android applications (for Android 6.0 and higher). The UnlockApp (see Figure 1a ) presents vocabulary immediately following the unlock action. A word is presented in a foreign language (L2) and the user is asked to choose between two presented translations in their mother tongue (L1) by pressing a button on the screen. We chose to implement a multiple-choice approach with two answer options as it enables quick and simple interaction with the content.

We further implemented a push notification learning app, which presents vocabulary in a pervasively displayed notification in the status bar (named NotificationApp, see Figure 1c , along the concept proposed by Dingler et al. [10] ) and on the lockscreen (see Figure 1b ).

Finally, we implemented a baseline vocabulary app, similar to the interactions with established apps, which the user has to actively start and quit (named StandardApp, see Figure 1d ).

Beyond the different presentations, a key conceptual difference is that the UnlockApp nudges the user to interact with the content, while the NotificationApp and StandardApp require users to actively initiate the learning themselves.

However, in contrast to the StandardApp, the NotificationApp's constant visibility acts as a continuous reminder for users to engage with the learning content.

All three apps include vocabulary allowing for learning Spanish, French, and Swedish with translations into the native language of our University's country. The word list comprised of 446 common nouns (originally from the British National Corpus 1 ) and have been translated to the other languages using Google Translate while removing inaccurate translations and highly ambiguous words, similar to the procedure of Cai et al. [5] and Dingler et al. [10] .

In the background, all three applications are connected to a Firebase database, which stores both the learning content and interaction logging data. In the database, each user is assigned a unique id to ensure anonymity. All three applications are described in more detail in the following.

To integrate the learning task into the authentication action, we display the vocabulary translation interface immediately when the user unlocks the screen (see Figure 1a ). As the current Android versions do not allow a direct manipulation of the lockscreen, we chose to implement a context-registered broadcast receiver. This receiver listens for the unlock event (ACTION_USER_PRESENT indicates that the user is present after the device is woken up 2 ) while running continuously in the background and places the app in the foreground when an unlock occurs. Since it is possible that applications are shut down for battery optimisation purposes, we further included a foreground service that allows for automatic restart of the application.

When an unlock event occurs, the UnlockApp asks the user to translate one word from the native language to one of the three languages of choice -Spanish, French, or Swedish. To keep the learning task as short as possible, the interface only presents the translation task, two potential translation options in the form of buttons. Further, we include a "skip" option to keep learners from guessing the answer, as guessing increases the risk of experiencing the negative testing effect [25] (i.e. remembering the false answer one gave to a multiple-choice question rather than the correct solution). After the user selects one of the two translation options, the UnlockApp displays corrective feedback by highlighting the answer option in either red or green. Providing feedback in multiple-choice learning tasks is essential and counteracts the negative testing effect [4] . After a short delay for users to perceive the feedback, the app moves into the background.

We chose a translucent background to not distract the user too much from the primary task they were about to accomplish when unlocking the phone. Prior research by Hodgetts and Jones [17] has shown that when the primary task was still visible during an interruption, participants are able to resume the task faster. To ensure consistency, we used the Android material design and color palettes.

The UnlockApp further allows to use a fingerprint sensor gesture (swiping left or right on the fingerprint sensor) as input to answer the tasks. We utilise the Fingerprint Gesture Controller accessibility service 3 . This controller registers gestures performed on the fingerprint sensor, independent from sensor location (depending on the model, the sensor can be located on the back of the phone, on the side, or on the front below or on the screen). While prior work has already shown that fingerprint sensor gestures can be a feasible tool for quick input such as unlock journaling [13] , this feature is not yet supported by many Android devices. Thus, we will not evaluate the usage of it in this study.

In this app, learning content is presented in a continuously displayed push notification, both in the (pulled down) status bar of the smartphone (see Figure 1c ) but also on the lockscreen (see Figure 1b) . This enables the user to interact with the content at any point in time and answer as many tasks as they like to, the continuous notification working as an active and constant reminder. The notification displays the task and word to be translated as well as two potential translation options via action buttons 4 . Those buttons are frequently applied by other applications to allow immediate interaction with the app sending the notification. For example, Gmail 5 employs action buttons to enable users to delete or reply to emails directly from the notification. We implemented a broadcast receiver to listen for the input in the notification and check the correctness. In the same style as the UnlockApp, green and red coloring of the two potential translations indicates corrective feedback. Afterward, the notification is updated and a new word is shown. Likewise, when the user dismisses a notification or restarts the app, it presents a new word. The notification itself cannot be removed by the user but remains constantly visible in the pulled down status bar.

In contrast to the UnlockApp and NotificationApp, the StandardApp does not show any continuous reminder of the learning task. The users have to actively open the application and engage with the learning content.

Once the users open the app, they can complete as many learning tasks (i.e. request as many translation tasks) as they like. The appearance of the application's interface is consistent with the interface of the UnlockApp: The app shows the native word, the two translation options in buttons, a dismiss button, and corrective feedback for the task (see Figure 1d ). In contrast to the UnlockApp, the background is fully opaque.

Our evaluation follows a within-subject design, where all participants interact with all three different applications to allow them to draw comparisons. Each app is used over the course of one week (seven days) before switching to the next app. The vocabulary progress is preserved so that people can continue in the next app where they left off in the previous one. The order of the apps is counter-balanced across participants to avoid sequence effects.

In our subsequent analysis, we explore differences in users' interaction with the three apps (independent variable), in particular frequency and duration of interactions, as well as perceived usability and experience (dependent variables).

Based on related work, we particularly evaluate differences among the three apps with regard to how much they expose users to the learning content, stating the following main hypothesis: 1 Embedding vocabulary tasks into everyday smartphone interactions as in the UnlockApp and NotificationApp leads to a higher number of learning tasks solved by users over the course of the study compared to the StandardApp.

Furthermore, we gather individual feedback on participants' preferences and experiences when learning with the three applications.

We provided participants with a detailed installation guide enabling people to download the three applications from the Google Play Store (setup as a non-public test version), after they gave informed consent. The guide also encouraged them to restart the (current) app in case they restart their phones or the app is closed for any other reason and does not restart automatically.

At the beginning of the study, people filled in a questionnaire on demographics and current authentication method, smartphone usage habits, language proficiency, and motivation to learn a new language. In this process, people selected a language to learn with the apps (Spanish, French, or Swedish). We informed participants that the vocabulary taught in the apps is on a beginner's level and thus recommended to choose a language they are not very proficient in.

During the study, we logged data on the users' interaction with the apps, including the number of solved vocabulary tasks, their correctness, response times and task dismissals (in the UnlockApp).

We recruited 30 participants via University mailing lists, social media channels, and word-of-mouth, nineteen identifying as female and eleven as male. Their average age was 29.8 years ( = 15.55, range 19-78 years) and the majority (18) reported to be studying or working full time in jobs such as Engineer, Physicist, Accountant, Doctor, or Dentist, while two were pensioners. Fifteen stated having a high school degree and twelve a university degree (including bachelor, master, or phd). One third (19 people) reported using fingerprint authentication on their smartphones, and seven reported using a PIN or password, two patterns, and two face recognition. All participants stated to use their device at least multiple times a day (15) if not multiple times per hour (15) .

At the time of the study, ten participants confirmed learning or actively improving on a language (four Spanish, three French, two English, and one Japanese). They further stated to be proficient in at least one and up to five foreign languages ( = 3), and reported to have great interest in learning a new language ( = 6.07, = 0.85, Likert-scale from 1="I fully disagree" to 7="I fully agree"). As languages to learn in this study, thirteen people chose Spanish, twelve

French, and five Swedish. For their successful participation all people received a 25 Euro Voucher for an online store, or an equivalent amount of study credit points.

We report on the data of 30 people. However, one person did not complete any of the final questionnaires, and two additional people did not submit the questionnaire for the StandardApp. Therefore, we report the interaction data of 30 and the questionnaire results of 29 people for the UnlockApp and NotificationApp, 27 for the StandardApp, respectively.

For the statistical analysis of the questionnaires, we performed pair-wise exclusion for incomplete data sets.

We used R [27] for significance testing; concretely, (generalised) linear mixed-effects models (LMMs, packages lme4 [2] and lmerTest [22] ). The LMMs accounted for individual differences (participants) and app order via random intercepts. Note that app order was counterbalanced yet we still included it here following best practices. As fixed effects, we included app, plus the number of days since the start of the study. We further plotted people's interactions with the three applications over the course of the whole day and the week of use: Figure 3 visualises the number of tasks solved by the users, indicating that they interacted with each app most frequently on the first day of use. It has to be noted here that participants started the study on different days of the week (Monday: 5, Tuesday: 2, Wednesday: 2, Thursday: 5, Friday: 5, Saturday: 6, Sunday: 5) and the starting weekday then stayed the same for each person across app conditions. Moreover, the plots reveal that interactions with the UnlockApp are scattered more across the time of the day compared to the interactions with the StandardApp and NotificationApp.

Furthermore, the plot matches the results of the LMM that the number of answered questions declined from day one to day seven.

In general, the users answered more than 90% of the learning tasks correctly over all three applications (see Table 1 ). The correctness rates for the StandardApp and NotificationApp are slightly higher than the rates for the UnlockApp. A Friedman test revealed no significant difference ( > .05) for the task correctness of the answers with regard to the three applications. For the UnlockApp, we can further measure the time between when the learning task is presented and the moment the answer is recorded. On average, it took participants 3.4 seconds to answer a task using the UnlockApp ( = 1.02).

The time to complete the task for incorrect answers is higher ( = 3.98, = 2.71) compared to correctly answered tasks ( = 3.38, = 1.02).

The UnlockApp enabled users to dismiss a learning task in case they did not want to answer it. During the seven study days, users chose to skip between zero and 80 tasks, with an average of 9.6 skips ( = 14.48, = 4). Similar suggestions were stated for the UnlockApp: Participants wished for free text entry opportunities (P22), pronunciation support (P9, P26), and grammar knowledge (P3, P5, P7, P19). Since vocabulary in itself is not sufficient to learn a new language, P6 considers the app a nice addition for people currently attending language learning courses and P19 recommends using it to freshen up a language. Regarding the overall concept of combining the learning task with authentication, many participants stated positive impressions: While P9 highlighted the simple design, P19 emphasises that "it's good that you always have to solve at least one task and therefore learn continuously". Further, P6 stated to like the app and in particular the idea, similar to P8. However, P8 adds that "[...] it bothers me that I have to answer the task or dismiss the app when I just quickly want to do something on my phone". P20 shares the experience and describes that when they just quickly want to use the phone, they are "not focused enough to answer the questions conscientiously".

To address this issue, other participants suggested including more personalisation options. For example, P7 wishes for a feature to adjust the number of vocabulary tasks presented at each authentication (which was fixed to one in the study), describing that if a user currently has more time to learn, then they could increase the number of words presented per unlock event. Further, P15 proposes to define time intervals during the day in which the app is active (P15).

Our user study was conducted during the COVID-19 pandemic, probably leading to anomalies in users' daily routines, mobility, and smartphone usage due to lockdown and work-from-home phases. All these factors potentially influence the use of mobile language learning applications; thus, we can not ensure the generalisability of our results. However, we included a control condition via the StandardApp.

The focus of our evaluation lies on the users' experiences and users' interaction with the three apps of varying integration into smartphone usage. Although our applications do not constitute an exhaustive representation of all MLL applications, we chose these three as a concise comparison of how different levels of embedding could be implemented.

Further, with our focus on user's interaction we can not report on actual vocabulary retention. However, as retention of vocabulary is increased by frequent interaction with the content [7, 9] , we expect our applications to positively impact people's vocabulary recall and recognition. Nonetheless, actual vocabulary retention, particularly concerning the consolidation properties of the apps over time, needs to be further evaluated in future work.

By providing people the three applications to use for each seven days during this user study, we gained interesting insights into users' learning behavior. We decided in favor of a shorter within-subject study over a longer betweensubject study to enable users to draw comparisons among their interaction with the three apps and to make the best use of the limited sample size. However, we are aware that our study only presents a narrow view onto users' actual behavior. As the users show a great diversity in individual preferences, only a long-term evaluation with a larger sample will be able to show which behaviours will prevail in users' daily lives. Our study data shows a more extensive usage of each application on the first day, declining over the seven days of usage. This pattern could indicate a form of curiosity or novelty effect. Educational technology research has shown that accustomisation with new learning technology can negatively impact learners' preferences regarding technology-based learning [20] and reduce users' motivation [19] .

In contrast to the StandardApp, the UnlockApp's concept partly counteracts this decline by continuously engaging the learner in solving vocabulary tasks with each authentication event. However, a future long-term evaluation in particular of the new UnlockApp concept is required to reveal the strength of the novelty effect in everyday usage.

Our analysis confirms our initial hypothesis, showing that the NotificationApp and UnlockApp result in a higher number of solved vocabulary tasks per participant. Thus, we conclude that the learning content exposure was higher for these two apps when compared to the baseline, the StandardApp. Furthermore, the presentation of the vocabulary tasks after the authentication leads to a more spread out exposure to the tasks across the day, confirming Dingler et al. [10] 's conclusion that proactively triggering vocabulary learning sessions can effectively space out learning. Further, the continuous presentation of content through the NotificationApp allows for short but also extensive learning sessions (cf. the outlier P4). The similar distribution of correctly and incorrectly answered tasks across the three applications gives no indication that users might have been less focused when learning with the UnlockApp or NotificationApp.

Additionally, we observed that users occasionally skip tasks with the UnlockApp. This suggests that even though the skipping requires the same amount of effort as selecting an answer (one button press), the users can judge if they have the mental capacity and/or time to engage in the learning or not.

The subjective feedback revealed individual differences regarding people's attitudes toward the three applications:

While many state to like the concepts of the NotificationApp, opinions on the UnlockApp are mixed. Specifically, some people felt distracted by the UnlockApp's vocabulary presentation when they unlocked their phone with a specific task in mind. People do not oppose the concept in general but express their need for further personalisation. Suggested adaptation features include the definition of learning time frames for the UnlockApp or adjusting the number of tasks presented with each authentication event. Moreover, we see potential for automated mechanisms that learn from people's interactions with the UnlockApp (in particular dismissals) and adjust the presentation accordingly. I.e. the app should stop promoting users with learning tasks after unlock events at times during the day when they are frequently dismissed.

Based on the results here, we deem it useful to discuss possible extensions of the UnlockApp concept to further authentication methods: Concretely, the majority of participants in our study used fingerprint authentication to unlock (yet with phones that do not support fingerprint gestures). Our app and concept already support fingerprint gestures to respond to vocabulary tasks (see implementation section), which can help to embed the learning task even more implicitly into the authentication (i.e. no switch from fingerprint to touch required). Prior work has found that using fingerprint sensor gestures for journaling is perceived faster and less intrusive compared to notifications [13] . Both touch input and fingerprint gestures might also be combined with unlocking methods that do not require further input themselves, such as face unlock, or when Android's smart screen lock feature is enabled (i.e. no unlock at home). Beyond this, we already gathered experiences with knowledge-based authentication methods: One third of our sample stated to use PIN, password, or pattern authentication. In these cases, the UnlockApp also presents the task immediately after the unlock event.

Our application focused on the presentation of vocabulary translations to reduce the complexity of the content. With this, we aim to simultaneously increase the control of potential effects among the three apps. The participants of our study strongly emphasised the demand for additional features such as grammar knowledge or pronunciation exercises in open comments, and compared our application to applications available on the market. To accommodate for people's need for more complex learning tasks, we see three potential solutions: (1) By combining the embedding concepts with a fully-featured learning app, we could address users' preferences for long learning streaks when they have time to spare (i.e., with the StandardApp), during which the app could also teach grammar or pronunciation knowledge.

The differentiation between two complexity levels could also be realized by (2) differentiating between short and long learning sessions as suggested by Cai et al. [5] (e.g., by exploiting different types of waiting situations). Additionally, the UnlockApp could refresh users' vocabulary knowledge by continuously presenting translation tasks after each authentication to engage users with a language on a daily basis. Option (3) is to investigate extensions of learning tasks that are short enough to be embedded into the UnlockApp or NotificationApp. These tasks need to be solvable with simple interactions and with low effort and time. Possible examples include fill-in-the-blank tasks offering a word in two tenses or with two suffix options.

We have compared three basic concepts for embedding vocabulary learning tasks into everyday smartphone use. When similar ideas were proposed in the literature, these individual concepts had been evaluated independently of each other.

Thus, in this paper, our main contribution to the literature is their direct empirical comparison in a single study.

In this light, our key result and conclusion is that embedding vocabulary learning tasks into everyday smartphone interactions -concretely: the status-bar notifications or the unlock procedure -significantly increases exposure over a dedicated self-initiated learning app.

To critically reflect on this finding, we might ask if this embedding and increased exposure is "worth it" for users, considering potential downsides such as distraction or longer unlock interactions. Based on our results and user feedback here, we conclude that the overall answer is yes, yet our insights point towards individual differences: Both embedding approaches (notification, unlock) were favoured by a considerable proportion of people, who provided various concrete reasons for their choices.

In the broader context of the literature, the results from our direct comparison here thus support such embedding and micro-learning concepts, and therefore strengthen their motivation and this line of research overall. Practically speaking, our findings motivate that language learning apps should consider offering these embedding concepts to users, for example, as complementary widgets to their main app content.

Looking ahead, our findings encourage further investigations in this direction: For example, future work could explore the design space of unlock integration in detail (e.g. supporting further learning tasks and unlock mechanisms), also to improve the individually perceived trade-offs of learning and disturbance (e.g. via personalised and context-aware adaptations). Also, learning tasks could be connected to other smartphone actions, such as giving additional purpose to less productive actions defined by the user (e.g. starting a social media app).

Institute for Digital Transformation (bidt).

The effect of spaced repetition on meaningful retention

Fitting Linear Mixed-Effects Models Using lme4

Mobile Learning with Micro-content: A Framework and Evaluation

Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing

Wait-Learning: Leveraging Conversational Dead Time for Second Language Education

SleepTight: Low-Burden, Self-Monitoring Technology for Capturing and Reflecting on Sleep Behaviors

Untangling the benefits of multiple study opportunities and repeated testing for cued recall

Evaluating the Implicit Acquisition of Second Language Vocabulary Using a Live Wallpaper

Effects of variable encoding and spaced presentations on vocabulary learning

Language Learning On-the-Go: Opportune Moments and Design of Mobile Microlearning Sessions

MicroMandarin: Mobile Language Learning in Context

Contextual Experience Sampling of Mobile Application Micro-Usage

Exploring the Use of Fingerprint Sensor Gestures for Unlock Journaling: A Comparison With Slide-to-X

It'sa hard lock life: A field study of smartphone (un) locking behavior and risk perception

Mobile Device Usage Characteristics: The Effect of Context and Form Factor on Locked and Unlocked Usage

A large-scale, long-term analysis of mobile device usage characteristics

Contextual cues aid recovery from interruption: The role of associative activation

Harnessing the "Ambience" of the Mobile-Phone Lockscreen for Ultra-Lite Logging

Learner motivation and e-learning design: A multinationally validated process

Student responses to computers: A longitudinal study

Chapter One: Language learning defined by time and place: A framework for next generation designs. In Left to my own devices: Learner autonomy and mobile-assisted language learning

lmerTest Package: Tests in Linear Mixed Effects Models

Self-directed use of mobile devices for language learning beyond the classroom. ReCALL

Android users in the wild: Their authentication and usage behavior

Memorial consequences of answering SAT II questions

Best practice guidance for linear mixed-effects models in psychological science

2020. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing

The seven sins of memory: insights from psychology and cognitive neuroscience

Enhancing human learning via spaced repetition optimization

Slide to X: Unlocking the Potential of Smartphone Unlocking

Augmenting the Web for Second Language Vocabulary Learning

Fast App Launching for Mobile Devices Using Predictive User Context

Examining Unlock Journaling with Diaries and Reminders for In Situ Self-Report in Health and Wellness