Frontline Learning Research Vol.6 No. 2 (2018) 57
-71
ISSN 2295-3159
a Leibniz-Institut für
Wissensmedien, Germany
Article received 14 May 2018 / revised 25 September/ accepted 26 September/ available online 7 December
During the last decade the combined recording of eye-tracking data and electroencephalographic (EEG) data has led to the methodology of fixation-related potentials analysis (FRP). This methodology has been increasingly and successfully used to study EEG correlates in the time domain (i.e., event-related potentials, ERPs) of cognitive processing in free viewing situations like text reading or natural scene perception. Basically, fixation-onset serves as time-locking event for epoching and analysing the EEG data. In this article the methodology of fixation-related frequency band power analysis (FRBP) is proposed and conceptually outlined to study cognitive load and affective variations in learners during free viewing situations of multimedia learning materials (i.e., combinations of textual and pictorial elements). The EEG alpha frequency band power at parietal electrodes may serve as a valid measure of cognitive load, whereas the frontal alpha asymmetry may serve as a measure of affective variations. I will introduce and motivate the measures and the methodology and discuss methodological challenges and potential ways to overcome them. The methodology is frontline for learning research, first, as to date the EEG has been rarely used to study design effects of multimedia learning materials and second, as fixation-related EEG data analysis has rarely been done focussing on the frequency domain (i.e., FRBP). Despite methodological challenges still to be solved, FRBP may provide a more in-depth picture of cognitive processing during multimedia learning compared to eye-tracking data or EEG data in isolation and thus may help clarifying effects of multimedia design decisions.
Keywords: EEG; eye-tracking; fixation-related EEG data analysis; EEG alpha frequency band power; multimedia
There is general agreement in instructional psychology that an adequate design of multimedia learning material (i.e., combinations of text and picture) is crucial for learning success (e.g., Mayer, 2009). This is because the design of multimedia learning material can alter the amount of additional, extraneous cognitive load (CL) imposed on the learner, either, in case of "good" design by freeing working memory resources, or, in case of "bad" design, by depleting working memory resources, potentially leading to an overload-situation hampering successful learning (see theoretical accounts like the Cognitive Load Theory, Sweller, van Merrienboer, & Paas, 1998; or the Cognitive Theory of Multimedia Learning, Mayer, 2009). Various multimedia design principles have been described that may alter learners' extraneous CL (Mayer & Fiorella, 2016; Mayer & Moreno, 2003).
However, it still remains a matter of research of how exactly CL is influenced by certain multimedia elements and multimedia design decisions. For example, how exactly adding decorative pictures to learning materials (i.e., texts) influence cognitive (and affective) processing and consequently alter learning outcomes is still a matter of debate (for a comprehensive review see Rey, 2012). Such decorative pictures (or graphical elements) adjacent to textual information in multimedia learning materials that are only loosely content-related (e.g., a picture of a lightning stroke adjacent to a text describing the meteorological formation of thunderstorms) have been termed pictorial seductive details (Harp & Mayer, 1998). Pictorial seductive details have resulted in mixed effects on learning outcomes, ranging from beneficial effects (Schneider, Nebel, & Rey, 2016), to no-effects (Park & Lim, 2007), and even detrimental effects (Harp & Mayer, 1998; Mayer & Fiorella, 2016). While the beneficial effects might be explained by affective processes with the pictures positively altering the learners' motivational state (Knörzer, Brünken, & Park, 2016; Lenzner, Schnotz, & Müller, 2013; Magner, Schwonke, Aleven, Popescu, & Renkl, 2014; Schneider et al., 2016), detrimental effects might be explained by increased extraneous CL due to the additional, yet irrelevant pictorial information (Mayer & Fiorella, 2016; Mayer & Moreno, 2003), and due to effects of distraction away from and interruption of the learning process (i.e., schema construction; Schneider, Dyrna, Meier, Beege, & Rey, 2017).
Clearly, in order to unravel potential reasons for the disparate effects of pictorial seductive details, adequate process measures (i.e., online measures) are necessary to better understand the effects of multimedia design elements on cognitive and affective processing. The electroencephalogram (EEG) and more precisely the EEG alpha and theta frequency band power might serve as such adequate, promising process measures. Especially, the methodology of fixation-related EEG frequency band power analysis (FRBP) might allow studying the EEG data in ecological valid task settings, that is, in free viewing situations of multimedia learning material, and may allow identifying which multimedia elements (i.e., text or picture) alter CL to what extent. In the next section, I will introduce the EEG alpha (and theta) frequency band power as potential and promising measures of CL and affective processing. I will then describe the FRBP methodology on a conceptual level and address open methodological challenges as well as possible approaches to overcome them. Note that the main purpose of the current article is to conceptually propose the FRBP analysis as a promising, new methodological account for multimedia research by providing an overview of the EEG frequency band power as a measure of CL and affective processing and by making readers aware of potentials as well as weaknesses of the FRBP methodology, yet without discussing practicalities and single methodological challenges in depth. The article may nevertheless serve as a valid and helpful primer for future research using the FRBP methodology in the context of multimedia materials.
Identifying adequate process measures still is an important and general matter of debate in instructional psychology (Brünken, Seufert, & Paas, 2010; Paas, Tuovinen, Tabbers, & van Gerven, 2003). Traditionally, most research focused on outcome measures like learning success, retention, or transfer of the learned knowledge. Subjective rating scales are used to assess the learners' invested effort during learning (e.g., Hart & Staveland, 1988; Klepsch, Schmitz, & Seufert, 2017; Paas, 1992) with effort sought to be directly related to CL and hence working memory load (Schnotz & Kürschner, 2007). One general drawback of subjective rating scales is that they only allow assessing CL in hindsight and for longer time periods. Whether participants report an averaged impression of CL or certain load-peaks remains elusive (Schmeck, Opfermann, van Gog, Paas, & Leutner, 2015). More importantly, motivational factors might confound the subjective ratings of CL (Schnotz et al., 2009). Consequently, the use of objective, online process measures of CL during learning have been proposed (Antonenko, Paas, Grabner, & van Gog, 2010; Brünken, Plass, & Leutner, 2003; Korbach, Brünken, & Park, 2017; Paas et al., 2003). For example, performance in a parallel secondary task can be used to assess the current CL of the primary task (Brünken et al., 2003; Park & Brünken, 2015). However, the secondary task may unintentionally interfere with the primary task. In contrast, physiological measures like pupil dilation or the EEG, and, more specifically, the EEG alpha (8 – 13 Hz) frequency band power at parietal electrodes and the theta (4 – 6 Hz) frequency band power at frontal electrodes may serve as valid measures of CL overcoming the aforementioned limitations (Antonenko et al., 2010; Beatty & Lucero-Wagoner, 2000). In addition, the frontal alpha asymmetry (FAA; Smith, Reznik, Stewart, & Allen, 2017) might serve as an index of affective effects in multimedia task materials.
The human EEG is typically recorded via several (e.g., 32 or 64) electrodes that are placed on the scalp at different positions consistently defined by the (extended) international 10-20 system (Jasper, 1958) using electrode caps with predefined slots for the electrodes (see Figure 1 for an exemplary electrode layout). After applying electrode gel to reduce the electrical impedance between the electrodes and the skin, the raw EEG can be measured. The EEG is the amplified recording of the small electrical currents generated within the brain by the summed post-synaptic electrical potentials of large neuronal assemblies consisting of several millions of neurons (pyramidal cells of the cortex) that are oriented in parallel and synchronously active (for reviews see Cohen, 2017; Jackson & Bolger, 2014; Olejniczak, 2006). The typical sampling rate of an EEG recording device is 500 or 1000 Hz (i.e., one measurement point every two or each millisecond, respectively). The EEG thus reflects the oscillatory activity of specific neuronal populations with excellent time resolution, yet the spatial resolution is rather low (in the range of centimeter; Olejniczak, 2006).
Figure 1. Schematic head (nose up) with exemplary electrode layout. Highlighted are representative electrode positions for measuring CL (blue color) or affective processing (red color). Note. Electrodes outside the head are due to the projection of 3D locations onto 2D space.
After some data preprocessing steps (e.g., filtering, artefact removal; for guidelines see Picton, 2000), the recorded EEG data can be analyzed in the time domain and/or in the frequency domain. For analyses in the time domain, parts of the EEG (i.e., epochs) time-locked to certain events (e.g., stimulus-onsets) are averaged across several trials (i.e., repetitions of a specific event) to increase the signal-to-noise ratio. This procedure results in so-called event-related potential (ERP) curves, that is, positive and negative deflections (i.e., components) that have been linked quite specifically to certain cognitive processes (for a comprehensive review see Münte, Urbach, Düzel, & Kutas, 2000). For analyses in the frequency domain, the spectrum is calculated for the EEG epochs of interest (e.g., using fast-fourier transforms, FFT, or wavelet analysis; Cohen, 2014). This results in power values for the different frequencies contained in the EEG signal (and also phase information, which however is beyond the scope of the current article; the interested reader may refer to Bastiaansen, Mazaheri, & Jensen, 2012; Cohen, 2014). The more neurons are synchronously active (i.e., 'firing') at a specific frequency in response to an event, the higher is the measured power (i.e., amplitude to the square) at this specific frequency. Therefore, an increase in frequency band power after an event is generally termed event-related synchronization (ERS), whereas a decrease in frequency band power is termed event-related desynchronization (ERD; Pfurtscheller & Lopes da Silva, 1999). The amount of change in ERD or ERS in relation to an event can be expressed as percentage of change using the ERD/ERS%-formula given in Pfurtscheller & Lopes da Silva (1999; see also Antonenko et al., 2010). Five frequency bands have traditionally been differentiated in the EEG reflecting functionally different oscillatory neuronal activity: Slow oscillatory activity in the delta (< 3 Hz) range, oscillatory activity in the theta (4 – 6 Hz), alpha (8 – 13 Hz), and beta (13 – 24 Hz) range, and fast oscillatory activity in the gamma (> 40 Hz) range (for reviews see Bastiaansen et al., 2012; Krause, 2003). In the context of multimedia research oscillatory activity (and hence, power) in the theta and alpha frequency band is of specific interest as will be detailed below. Slow (delta) and fast (gamma) oscillatory activity are rather prone to artefacts (e.g., slow drifts or muscle activity) requiring highly controlled lab settings and rather simple, basic tasks and task materials (see, e.g., Bastiaansen et al., 2012), thus being rather inadequate for multimedia research. Oscillatory activity in the beta band has mainly been attributed to reflect activity of the motor cortex (Pfurtscheller, Zalaudek, & Neuper, 1998), but might also be interesting for studying cognitive processes (Engel & Fries, 2010).
Importantly, the EEG alpha and theta frequency band power might be used as reliable process measures of CL in multimedia research. It has been consistently observed that the EEG alpha frequency band power at parietal electrodes decreases for increasing CL whereas the frontal-central EEG theta frequency band power increases for increasing CL (Gevins & Smith, 2000; Kretzschmar et al., 2013; Palomäki, Kivikangas, Alafuzoff, Hakala, & Krause, 2012; Pesonen, Hämäläinen, & Krause, 2007; Scharinger, Soutschek, Schubert, & Gerjets, 2015; 2017). Functionally, the alpha ERD (i.e., the decreasing alpha power) has been interpreted to index cortical activity related to attentional and memory processing (Klimesch, 1999; Krause, 2003). According to Klimesch (1999) the alpha frequency band might be functionally divided further in an upper alpha (10 – 13 Hz) frequency band that might mainly be related to (semantic) memory processing and a lower alpha (8 – 10 Hz) frequency band that might mainly be related to attentional processes. Neural activity in the EEG theta frequency band has been associated predominantly with processes of working memory and cognitive control (Itthipuripat, Wessel, & Aron, 2013; Nigbur, Ivanova, & Stürmer, 2011; Sauseng et al., 2002; Sauseng, Griesmayr, Freunberger, & Klimesch, 2010). Note however, that the functional relationship between EEG frequency band power and specific cognitive processes is not as clear and well established as the relationship between certain ERP components of the EEG and cognitive processes (Krause, 2003). However, in contrast to ERPs that require a highly controlled, artificial task environment, especially EEG alpha frequency band power has been shown to reliably index CL also in complex task materials like those of instructional psychology (Antonenko & Niederhauser, 2010; Gerlic & Jausovec, 2001; Scharinger, Kammerer, & Gerjets, 2015; 2016). Importantly, changes in the EEG frequency band power index fluctuations of CL with high temporal acuity.
Furthermore, apart from indexing CL the EEG alpha frequency band power might also be used to assess emotional and motivational aspects of stimulus-processing. The frontal alpha asymmetry (FAA) may serve as such a measure. The FAA is a relational measure of the EEG alpha (8 – 13 Hz) frequency band power over the left-frontal hemisphere compared to the right-frontal hemisphere, commonly calculated as the difference between corresponding electrodes (e.g., F4 minus F3; Smith, Reznik, Stewart, & Allen, 2017). The FAA has its origin in cortical activity of the prefrontal cortex. The prefrontal cortex plays an important role in affective processing and emotion regulation showing specific left and right hemispheric lateralization effects for emotions and affective stimulus content (Demaree, Everhart, Youngstrom, & Harrison, 2005; Dixon, Thiruchselvam, Todd, & Christoff, 2017). Although still matter of debate, greater left than right hemispheric cortical activity has been associated with approach motivation and hence predominantly emotionally positively connoted stimuli, whereas greater right hemispheric cortical activity has been associated with withdrawal motivation and hence predominantly emotionally negatively connoted stimuli (Ahern & Schwartz, 1985; Harmon-Jones, Gable, & Peterson, 2010). Most EEG studies so far assessed the FAA during rest conditions after emotion induction procedures (e.g., Coan & Allen, 2004). Yet, recent research indicated that the FAA might be also used during task performance when emotional stimuli are presented for a brief period of time (Schöne, Schomberg, Gruber, & Quirin, 2016; Weinreich, Stephani, & Schubert, 2016). Thus, potentially, the FAA might be used to study affective aspects of multimedia. Whether the FAA can be used in such a way for complex multimedia materials has however to be studied further.
The methodology of fixation-related EEG data analysis may allow analyzing the EEG frequency band power in free viewing situations of multimedia task materials when certain elements (i.e., areas of interest, AOIs) are fixated and thus may allow to assess how CL (or in case of the FAA, affective processing) is altered by specific multimedia elements like pictorial seductive details. In the next section, I will give a brief overview on the methodology, then conceptually pointing out methodological challenges that one has to be aware of.
Eye-tracking is increasingly used in research on instructional multimedia materials to study underlying cognitive processes during learning (e.g., Eitel, Scheiter, & Schüler, 2012; Hyönä, 2010; Jarodzka, Holmqvist, & Gruber, 2017; Mayer, 2010; Schüler, 2017; Van Gog & Scheiter, 2010; for a recent overview on the literature see Alemdag & Cagiltay, 2018). Eye-tracking data shows the movement of the eyes, basically differentiating between saccades (i.e., the eyes quickly moving) to elements of the visual scenery and fixations (i.e., the eyes practically at rest) on elements of the visual scenery (for a comprehensive discussion of different eye-tracking patterns see Holmqvist & Andersson, 2017). It is generally agreed on, that during saccades the visual information intake is not possible (e.g., Kok & Jarodzka, 2017a), whereas it is possible (and most of the time takes place) during fixations. It has been shown that the fixation patterns vary depending on the task materials and the given task (e.g., in picture viewing, Yarbus, 1967, or in reading, Strukelj & Niehorster, 2018; for reviews see Kowler, 2011; Rayner, 1998; 2009), indicating a plausible link between fixations and cognitive processing (Just & Carpenter, 1976; 1980). Note however, this link might not always be straightforward. Visual attention might be slightly ahead of what is fixated (depending on the task, up to 250 ms; Deubel, 2008). In addition, due to peripheral viewing especially in reading more elements than those fixated at might be processed (Baccino & Manunta, 2005; Dimigen, Kliegl, & Sommer, 2012; Rayner, 2009). On the contrary, what is fixated at might not always be (consciously) processed. This might be the case during periods of mind-wandering (Foulsham, Farley, & Kingstone, 2013), or if visual elements are not task relevant (as indicated in change blindness paradigms; e.g., Triesch, Ballard, Hayhoe, & Sullivan, 2003). Nevertheless, fixations have been used as a valid proxy reflecting the individuals' structure of the information intake (and procesing) in free viewing situations and hence as triggers for defining epochs in time for which the EEG data can be analysed.
In free viewing or free reading situations no specific stimulus-onset exists. However, the fixation-onset can be used as the event for which the EEG data is time-locked, epoched, and analysed to (depending on the concrete research question, some studies also used saccade-onset as time-locking event; cf. Dimigen, Sommer, Hohlfeld, Jacobs, & Kliegl, 2011). Most studies so far have studied the EEG in the time domain, that is, fixation-related potentials (FRPs), for example in reading research (Dimigen, et al., 2011; Frey, Lemaire, Vercueil, & Guérin-Dugué, 2018; Henderson, Luke, Schmidt, & Richards, 2013; Hutzler et al., 2007; Kliegl, Dambacher, Dimigen, & Sommer, 2014; Kornrumpf, Dimigen, & Sommer, 2017; Kornrumpf, Niefind, Sommer, & Dimigen, 2016; Léger et al., 2014; Niefind & Dimigen, 2016; Weiss, Knakker, & Vidnyánszky, 2016), natural scene perception (Giannini, Alexander, Nikolaev, & van Leeuwen, 2018; Simola, Le Fevre, Torniainen, & Baccino, 2015; Simola, Torniainen, Moisala, Kivikangas, & Krause, 2013), visual search (Brouwer, Hogervorst, Oudejans, Ries, & Touryan, 2017; Kamienkowski, Ison, Quiroga, & Sigman, 2012; Kaunitz et al., 2014; Ries, Touryan, Ahrens, & Connolly, 2016; Winslow et al., 2010), decision making (Frey et al., 2013), and human-computer interaction (Léger et al., 2014). Despite methodological challenges (see below) these studies concurrently report FRP-effects comparable to classical ERP-effects (e.g., N400-like effects for word predictability; Dimigen et al., 2011), thus validating the basic feasibility and meaningfulness of fixation-related EEG data analyses.
Only few studies so far have analysed the EEG data in the frequency domain using FRBP. For example, Scharinger and colleagues (Scharinger et al., 2015, Experiment 1) used FRBP to study CL during hypertext-like reading, comparing several parts of a text that defined two different areas of interest (AOIs): AOIs of parts of the text where participants simply read and AOIs of parts of the text where participants additionally had to perform hyperlink-like selection processes. As hypothesized, the CL was higher when participants had to perform hyperlink-like selection processes in addition to purely text reading, indicated by decreased EEG alpha frequency band power. Interestingly, this result was also confirmed by the pupil dilation data, with the pupil showing a larger diameter (i.e., higher CL) for parts of the text with hyperlink-like selection processes as compared to purely text reading. In a second experiment (Scharinger et al., 2015, Experiment 2) the results could be replicated using classical response-locked EEG data analysis instead of fixation-related EEG data analysis, thus underlining the validity of the fixation-related EEG data analysis methodology. Another study by Scharinger and colleagues (Scharinger et al., 2016) used FRBP to study CL (as indexed by the parietal EEG alpha frequency band power) in free viewing and evaluating of search engine result pages. This study indicated that a perfect hit (i.e., a semantically and lexically matching search result for a specific search query) resulted already during initial fixations in decreased EEG alpha frequency band power as compared to search results that are no, or no perfect matches for a given search query. This has been interpreted as indicating that the best hit is recognized early in time and potentially then more thoroughly processed (resulting in increased CL) as compared to other, less fitting search results. Finally, Vignali and colleagues (Vignali, Himmelstoss, Hawelka, Richlan, & Hutzler, 2016) used FRBP to study semantic violations in sentences in free reading situations. They observed decreased lower-beta band (13-18 Hz) power for fixations of semantically unrelated words as compared to semantically related words, also indicating higher CL.
To sum up, studies so far indicated the principal feasibility and validity of the methodology of fixation-related EEG data analysis. Combining eye-tracking and the EEG in research on instructional multimedia materials seems to be promising for two reasons. First, it allows to study the EEG during learning with multimedia task materials in task settings of high ecological validity (i.e., 'realistic' task materials, with texts and pictures presented simultaneously on one screen). Without this methodology, text and pictures would have to be presented separately in time (i.e., in rather artificial sequences) to create stimulus-onsets for the EEG data analysis. Second, the EEG data might be used as an additional measure for triangulating the meaning of observed eye-movement patterns as has been proposed for verbal data (Kok & Jarodzka, 2017b). For example, the EEG might help differentiating whether longer fixations might indicate increased CL (König et al., 2016; Reichle & Reingold, 2013) or individuals' expertise (Bertram, Helle, Kaakinen, & Svedström, 2013; Reingold & Sheridan, 2011), or it might help differentiating whether learners are still working on a (difficult) task or mind-wandering (Foulsham et al., 2013). Yet, some methodological challenges remain that one has to be aware of.
There are some challenges that one has to be aware of when analysing fixation-related EEG data. First, the EEG and eye-tracking data has to be synchronized. This can be done by regularly sending triggers (e.g., each second) during data recording to both recording devices (i.e., the EEG and the eye-tracker). Based on these triggers the two data streams can then be synchronized offline using for example the toolbox EEGLAB (Delorme & Makeig, 2004) with the EYE-EEG plugin (Dimigen et al., 2011). The synchronisation of the EEG and eye-tracking data includes matching the sampling rates of both data streams (which is for the EEG typically at 500 Hz or 1000 Hz and for current remote eye-tracking devices at 120 Hz or 250 Hz), either by up- or down-sampling.
Once the EEG data and the eye-tracking data are synchronized there remain several challenges that have to be taken into consideration. These include the correction of eye-movement artefacts, dealing with overlapping EEG data segments due to different and rather short fixation durations, and the selection of an adequate baseline (for a comprehensive discussion of these challenges and potential solutions the interested reader may refer to Baccino, 2011; Dimigen et al., 2011; Nikolaev et al., 2016; the purpose of the current article is to make readers aware of potentials as well as weaknesses of the methodology, yet without discussing single methodological challenges in depth).
Eye-movements (i.e., saccades and blinks) alter the electric fields around the eyes and consequently confound the raw EEG, especially at frontal electrodes (Iwasaki et al., 2005). Thus, eye-movement artefacts may either mask the EEG correlates of interest or, worse, may induce a systematic error. For example, in multimedia the eye-movement patterns vary between viewing of textual and pictorial elements. Thus, when interested in fixation-related EEG data for multimedia, that is, when comparing EEG data for text reading and picture viewing, the EEG may be confounded by different eye-movement patterns. However, several methodologies exist to clean the raw EEG data from eye-movement artefacts (for reviews see Croft & Barry, 2000; Islam, Rastegarnia, & Yang, 2016). For example, independent component analysis (ICA) can be used to reliably identify and correct for eye-movement artefacts (Chaumon, Bishop, & Busch, 2015; Delorme, Sejnowski, & Makeig, 2007; Jung et al., 2000; Zhou & Gotman, 2009). Noteworthy, especially in the context of fixation-related EEG data analysis the use of ICA has been shown to result in adequately cleaned EEG, outperforming other methodologies like standard regression-based data cleaning (Henderson et al., 2013; Hutzler et al., 2007).
While a variety of methodologies exists for cleaning EEG data from eye-movement artefacts, the varying and rather short length of fixations is another challenge for fixation-related EEG analysis that one has to be aware of. For example, during reading typical fixations last on average between 200-250 ms (e.g., Dimigen et al., 2011). This is problematic when analysing the EEG data in the time domain as later components in the FRP of a current fixation (e.g., the N400) might be overlapped (i.e., confounded) by early components of the FRP of the following fixation. In FRP analysis very short fixations are therefore excluded from analysis (e.g., fixations < 80 ms; Frey et al., 2018). Several statistical methods (e.g., regression-based models) have been proposed to deal with the potentially confounding effect of overlapping EEG data segments (Baccino, 2011; Dimigen et al., 2011; Frey et al., 2018; Nikolaev, Pannasch, Ito, & Belopolsky, 2014). It has also been proposed to compare only those data sequences of comparable fixation-patterns between task conditions of interest to minimize potentially confounding effects due to different fixation durations and hence differently overlapping EEG segments (Nikolaev et al., 2016). While this proceeding might be feasible for task conditions that are quite comparable (e.g., both within the domain of reading), for multimedia task materials with texts and pictures it might be impossible to adequately match the fixations used for analysis due to the different fixation patterns for reading and picture viewing. Moreover, in FRBP analysis overlapping EEG data epochs due to short fixation durations might frequently occur. This is because, the length of the EEG data epoch defines the possible frequency resolution of the calculated spectrum. For example, in the theta frequency range (4 – 6 Hz) one oscillation lasts at minimum 250 ms. Thus, when interested in the theta frequency band power an analysis window of at least 250 ms would be necessary. Commonly, an analysis window including several oscillations of the specific frequency of interest is recommended for calculating the spectrum (i.e., an analysis window of at least 500 ms length). Consequently, one has to be aware of the constraint that FRBP analysis is seldom suitable for analysing the EEG data for single fixations (unless their duration is long enough). Yet for research on multimedia task materials interested in CL this constraint might not be of too much relevance, as the AOIs of interest (i.e., parts of the text versus the pictures) might generally comprise several fixations that one could summarize, resulting in EEG data epochs long enough for analysis (see Figure 2). Importantly, one still would be able to differentiate between first and later visits of an AOI. Nevertheless, a comparison of the EEG in classical sequential stimulus presentation paradigms (e.g., in multimedia research by presenting text and pictures sequentially) with fixation-related paradigms might be necessary to further validate the reliability of the FRBP analysis when used in new experimental paradigms or for new, complex multimedia task materials.
Figure 2. Left part: Schematic illustration of the problem of overlapping EEG data segments (marked by the lightning symbol) in case of short fixations and a FRBP analysis based on single fixations (i.e., using very small AOIs). Right part: Schematic illustration of EEG data segments aligned to larger AOIs which would be typically used in multimedia research. Longer EEG segments could be used without overlapping. Note. The EEG and eye-tracking data has been artificially combined and does not show data of a real person.
Another, however easily addressable challenge of overlapping EEG segments in fixation-related EEG data analysis are specific EEG correlates at the very beginning of a stimulus (e.g., when the text for reading is shown on the screen for the first time). To avoid stimulus-presentation associated event-related EEG correlates potentially masking fixation-related EEG correlates, the first 700 ms of stimulus presentation (i.e., the first few fixations) should be excluded from EEG data analysis (Dimigen et al., 2011).
Finally, defining an adequate baseline for EEG data analysis is not trivial in free viewing situations (Dimigen et al., 2011; Nikolaev et al., 2016). In classical EEG data analysis typically a pre-stimulus baseline is used for baseline correction in order to reduce slow currency drifts (e.g., due to fatigue) in the EEG epochs used for analysis (Picton, 2000). As in fixation-related EEG data analysis the pre-fixation baseline is contaminated by saccadic activity, it has been proposed to alternatively use a short time-frame directly after fixation-onset as baseline (e.g., Baccino, 2011) or to use a global, pre-stimulus baseline (i.e., a baseline at the beginning of the task; Nikolaev et al., 2016). To date, there is no clear recommendation of what time interval is best suited as baseline in fixation-related EEG data analysis. Choosing an adequate baseline may largely depend on the concrete task design. Potentially, in FRBP it might also be possible to report absolute power values (i.e., to avoid using a baseline) if the experimental conditions (i.e., the AOIs for which the EEG data is analyzed and compared) are fully permutated with respect to spatial and temporal positions (i.e., when spatial or timing issues can be excluded as potential confounds of the data).
Despite the remaining challenges of using the methodology of fixation-related EEG data analysis, the methodology clearly is at the frontline of learning research. EEG (alpha and theta) frequency band power may help gaining deeper insight in the cognitive processing of multimedia elements (i.e., text and picture and resulting CL or affective effects). Thus, FRBP may substantially contribute to a better understanding of multimedia design effects like pictorial seductive details and consequently may foster better instructional design.
Ahern, G. L., & Schwartz, G. E. (1985). Differential
lateralization for positive and negative emotion in the human
brain: EEG spectral analysis. Neuropsychologia, 23(6),
745–755. https://doi.org/10.1016/0028-3932(85)90081-8
Alemdag, E., & Cagiltay, K. (2018). A systematic review of eye
tracking research on multimedia learning. Computers &
Education, 125(July), 413–428.
https://doi.org/10.1016/j.compedu.2018.06.023
Antonenko, P., & Niederhauser, D. S. (2010). The influence of
leads on cognitive load and learning in a hypertext environment. Computers
in Human Behavior, 26(2), 140–150.
https://doi.org/10.1016/j.chb.2009.10.014
Antonenko, P., Paas, F., Grabner, R., & van Gog, T. (2010).
Using electroencephalography to measure cognitive load. Educational
Psychology Review, 22(4), 425–438.
https://doi.org/10.1007/s10648-010-9130-y
Baccino, T. (2011). Eye movements and concurrent event-related
potentials’: Eye fixation-related potential investigations in
reading. In S. Liversedge, I. Gilchrist, & S. Everling (Eds.),
Oxford Handbook of Eye Movements(pp. 857–870). Oxford,
UK: Oxford University Press.
Baccino, T., & Manunta, Y. (2005). Eye-fixation-related
potentials: Insight into parafoveal processing. Journal of
Psychophysiology, 19(3), 204–215.
https://doi.org/10.1027/0269-8803.19.3.204
Bastiaansen, M., Mazaheri, A., & Jensen, O. (2012). Beyond
ERPs: Oscillatory Neuronal Dynamics. In S. J. Luck & E. S.
Kappenman (Eds.), The Oxford Handbook of Event-Related
Potential Components(pp. 31–50). Oxford, UK: Oxford
University Press.
Beatty, J., & Lucero-Wagoner, B. (2000). The pupillary system.
In J. T. Cacioppo, L. G. Tassinary, & G. Berndtson (Eds.), Handbook
of psychophysiology(2nd ed., pp. 142–162). Cambridge, UK:
Cambridge University Press.
Bertram, R., Helle, L., Kaakinen, J. K., & Svedström, E.
(2013). The Effect of Expertise on Eye Movement Behaviour in
Medical Image Perception. PLoS ONE, 8(6),
e66169. https://doi.org/10.1371/journal.pone.0066169
Brouwer, A.-M., Hogervorst, M. A., Oudejans, B., Ries, A. J.,
& Touryan, J. (2017). EEG and Eye Tracking Signatures of
Target Encoding during Structured Visual Search. Frontiers in
Human Neuroscience, 11, 1–11.
https://doi.org/10.3389/fnhum.2017.00264
Brünken, R., Plass, J. L., & Leutner, D. (2003). Direct
measurement of cognitive load in multimedia learning. Educational
Psychologist, 38(1), 53–61.
https://doi.org/10.1207/s15326985ep3801_7
Brünken, R., Seufert, T., & Paas, F. (2010). Measuring
cognitive load. In J. L. Plass, R. Moreno, & R. Brünken
(Eds.), Cognitive Load Theory(pp. 181–202). Cambridge:
Cambridge University Press.
https://doi.org/10.1007/978-1-4419-8126-4_6
Chaumon, M., Bishop, D. V. M., & Busch, N. A. (2015). A
practical guide to the selection of independent components of the
electroencephalogram for artifact correction. Journal of
Neuroscience Methods, 250 , 47–63.
https://doi.org/10.1016/j.jneumeth.2015.02.025
Coan, J. a, & Allen, J. J. B. (2004). Frontal EEG asymmetry as
a moderator and mediator of emotion. Biological Psychology,
67(1–2), 7–49.
https://doi.org/10.1016/j.biopsycho.2004.03.002
Cohen, M. X. (2014). Analyzing neural time series data:
theory and practice.Cambridge, MA: MIT Press.
https://doi.org/10.1007/s13398-014-0173-7.2
Cohen, M. X. (2017). Where Does EEG Come From and What Does It
Mean? Trends in Neurosciences, 40(4), 208–218.
https://doi.org/10.1016/j.tins.2017.02.004
Croft, R. J., & Barry, R. J. (2000). Removal of ocular
artifact from the EEG: a review. Neurophysiologie
Clinique/Clinical Neurophysiology, 30 (1), 5–19.
https://doi.org/10.1016/S0987-7053(00)00055-1
Delorme, A., & Makeig, S. (2004). EEGLAB: An open source
toolbox for analysis of single-trial EEG dynamics including
independent component analysis. Journal of Neuroscience
Methods, 134(1), 9–21.
https://doi.org/10.1016/j.jneumeth.2003.10.009
Delorme, A., Sejnowski, T., & Makeig, S. (2007). Enhanced
detection of artifacts in EEG data using higher-order statistics
and independent component analysis. NeuroImage,34(4),
1443–1449. https://doi.org/10.1016/j.neuroimage.2006.11.004
Demaree, H. A., Everhart, D. E., Youngstrom, E. A., &
Harrison, D. W. (2005). Brain lateralization of emotional
processing: historical roots and a future incorporating
"dominance". Behavioral and Cognitive Neuroscience Reviews,
4(1), 3–20. https://doi.org/10.1177/1534582305276837
Deubel, H. (2008). The time course of presaccadic attention
shifts. Psychological Research,72(6), 630–640.
https://doi.org/10.1007/s00426-008-0165-3
Deubel, H., & Schneider, W. X. (1996). Saccade target
selection and object recognition: Evidence for a common
attentional mechanism. Vision Research, 36(12),
1827–1837. https://doi.org/10.1016/0042-6989(95)00294-4
Dimigen, O., Kliegl, R., & Sommer, W. (2012). Trans-saccadic
parafoveal preview benefits in fluent reading: A study with
fixation-related brain potentials. NeuroImage, 62(1),
381–393. https://doi.org/10.1016/j.neuroimage.2012.04.006
Dimigen, O., Sommer, W., Hohlfeld, A., Jacobs, A. M., &
Kliegl, R. (2011). Coregistration of eye movements and EEG in
natural reading: Analyses and review. Journal of Experimental
Psychology. General, 140(4), 552–572.
https://doi.org/10.1037/a0023885
Dixon, M. L., Thiruchselvam, R., Todd, R., & Christoff, K.
(2017). Emotion and the prefrontal cortex: An integrative review.
Psychological Bulletin, [Epub], 1–61.
https://doi.org/10.1037/bul0000096
Eitel, A., Scheiter, K., & Schüler, A. (2012). The time course
of information extraction from instructional diagrams. Perceptual
and Motor Skills, 115(3), 677–701.
https://doi.org/10.2466/22.23.PMS.115.6.677-701
Engel, A. K., & Fries, P. (2010). Beta-band oscillations -
signalling the status quo? Current Opinion in Neurobiology,
20(2), 156–165.
https://doi.org/10.1016/j.conb.2010.02.015
Foulsham, T., Farley, J., & Kingstone, A. (2013). Mind
wandering in sentence reading: Decoupling the link between mind
and eye. Canadian Journal of Experimental Psychology/Revue
Canadienne de Psychologie Expérimentale , 67(1),
51–59. https://doi.org/10.1037/a0030217
Frey, A., Ionescu, G., Lemaire, B., López-Orozco, F., Baccino, T.,
& Guérin-Dugué, A. (2013). Decision-making in information
seeking on texts: An eye-fixation-related potentials
investigation. Frontiers in Systems Neuroscience, 7(39).
https://doi.org/10.3389/fnsys.2013.00039
Frey, A., Lemaire, B., Vercueil, L., & Guérin-Dugué, A.
(2018). An eye fixation-related potential study in two reading
tasks: reading to memorize and reading to make a decision. Brain
Topography, 1–21. https://doi.org/10.1007/s10548-018-0629-8
Gerlic, I., & Jausovec, N. (2001). Differences in EEG power
and coherence measures related to the type of presentation: Text
versus multimedia. Journal of Educational Computing Research,
25 (2), 177–195.
http://dx.doi.org/10.2190/YDWY-U3FJ-4LY4-LYND
Gevins, A., & Smith, M. E. (2000). Neurophysiological measures
of working memory and individual differences in cognitive ability
and cognitive style. Cerebral Cortex, 10(9),
829–839. https://doi.org/10.1093/cercor/10.9.829
Giannini, M., Alexander, D. M., Nikolaev, A. R., & van
Leeuwen, C. (2018). Large-scale traveling waves in EEG activity
following eye movement. Brain Topography, 1–15.
https://doi.org/10.1007/s10548-018-0622-2
Harmon-Jones, E., Gable, P. A., & Peterson, C. K. (2010). The
role of asymmetric frontal cortical activity in emotion-related
phenomena: A review and update. Biological Psychology, 84(3),
451–462. https://doi.org/10.1016/j.biopsycho.2009.08.010
Harp, S. F., & Mayer, R. E. (1998). How seductive details do
their damage: A theory of cognitive interest in science learning.
Journal of Educational Psychology,90(3),
414–434. https://doi.org/10.1037/0022-0663.90.3.414
Hart, S. G., & Staveland, L. E. (1988). Development of
NASA-TLX (Task Load Index): Results of empirical and theoretical
research. In P. A. Hancock & N. Meshkati (Eds.), Human
Mental Workload(pp. 139–183). Amsterdam, NL.
https://doi.org/10.1016/s0166-4115(08)62386-9
Henderson, J. M., Luke, S. G., Schmidt, J., & Richards, J. E.
(2013). Co-registration of eye movements and event-related
potentials in connected-text paragraph reading. Frontiers in
Systems Neuroscience, 7(28).
https://doi.org/10.3389/fnsys.2013.00028
Holmqvist, K., & Andersson, R. (2017). Eye tracking: A
comprehensive guide to methods, paradigms, and measures .
Lund, Sweden: Lund Eye-Tracking Research Institute.
Hutzler, F., Braun, M., Võ, M. L.-H., Engl, V., Hofmann, M.,
Dambacher, M., Leder, H., & Jacobs, A. M. (2007). Welcome to
the real world: validating fixation-related brain potentials for
ecologically valid settings. Brain Research, 1172,
124–9. https://doi.org/10.1016/j.brainres.2007.07.025
Hyönä, J. (2010). The use of eye movements in the study of
multimedia learning. Learning and Instruction, 20(2),
172–176. https://doi.org/10.1016/j.learninstruc.2009.02.013
Islam, M. K., Rastegarnia, A., & Yang, Z. (2016). Methods for
artifact detection and removal from scalp EEG: A review. Neurophysiologie
Clinique/Clinical Neurophysiology, 46 (4–5),
287–305. https://doi.org/10.1016/j.neucli.2016.07.002
Itthipuripat, S., Wessel, J. R., & Aron, A. R. (2013). Frontal
theta is a signature of successful working memory manipulation. Experimental
Brain Research, 224(2), 255–262.
https://doi.org/10.1007/s00221-012-3305-3
Iwasaki, M., Kellinghaus, C., Alexopoulos, A. V., Burgess, R. C.,
Kumar, A. N., Han, Y. H., Lüders, H. O., & Leigh, R. J.
(2005). Effects of eyelid closure, blinks, and eye movements on
the electroencephalogram. Clinical Neurophysiology, 116(4),
878–885. https://doi.org/10.1016/j.clinph.2004.11.001
Jackson, A. F., & Bolger, D. J. (2014). The neurophysiological
bases of EEG and EEG measurement: A review for the rest of us. Psychophysiology,
51(11), 1061–1071. https://doi.org/10.1111/psyp.12283
Jarodzka, H., Holmqvist, K., & Gruber, H. (2017). Eye tracking
in educational science : Theoretical frameworks and research
agendas. Journal of Eye Movement Research, 10(1),
1–18. https://doi.org/10.16910/jemr.10.1.3
Jasper, H. H. (1958). The ten-twenty electrode system of the
International Federation. Electroencephalography and Clinical
Neurophysiology, 10, 371–375.
Jung, T., Makeig, S., Humphries, C., Lee, T., McKeown, M. J.,
Iragui, I., & Sejnowski, T. J. (2000). Removing
Electroencephalographic aretfacts by blind source seperation. Psychophysiology,37(2),
163–178. https://doi.org/10.1111/1469-8986.3720163
Just, M. A., & Carpenter, P. A. (1976). Eye fixations and
cognitive processes. Cognitive Psychology, 8(4),
441–480. https://doi.org/10.1016/0010-0285(76)90015-3
Just, M. A., & Carpenter, P. A. (1980). A theory of reading:
From eye fixations ot comprehension. Psychological Review,
87(4), 329–354.
https://doi.org/10.1037/0033-295X.87.4.329
Kamienkowski, J. E., Ison, M. J., Quiroga, R. Q., & Sigman, M.
(2012). Fixation-related potentials in visual search: A combined
EEG and eye tracking study. Journal of Vision, 12(7),
4–4. https://doi.org/10.1167/12.7.4
Kaunitz, L. N., Kamienkowski, J. E., Varatharajah, A., Sigman, M.,
Quiroga, R. Q., & Ison, M. J. (2014). Looking for a face in
the crowd: Fixation-related potentials in an eye-movement visual
search task. NeuroImage, 89, 297–305.
https://doi.org/10.1016/j.neuroimage.2013.12.006
Klepsch, M., Schmitz, F., & Seufert, T. (2017). Development
and validation of two instruments measuring intrinsic, extraneous,
and germane cognitive load. Frontiers in Psychology, 8,
1–18. https://doi.org/10.3389/fpsyg.2017.01997
Kliegl, R., Dambacher, M., Dimigen, O., & Sommer, W. (2014).
Oculomotor control, brain potentials, and timelines of word
recognition during natural reading. In Current Trends in Eye
Tracking Research(pp. 141–155). Cham: Springer
International Publishing.
https://doi.org/10.1007/978-3-319-02868-2_10
Klimesch, W. (1999). EEG alpha and theta oscillations reflect
cognitive and memory performance: A review and analysis. Brain
Research Reviews, 29(2–3), 169–195.
https://doi.org/10.1016/S0165-0173(98)00056-3
Knörzer, L., Brünken, R., & Park, B. (2016). Facilitators or
suppressors: Effects of experimentally induced emotions on
multimedia learning. Learning and Instruction, 44,
97–107. https://doi.org/10.1016/j.learninstruc.2016.04.002
König, P., Wilming, N., Kietzmann, T. C., Ossandon, J. P., Onat,
S., Ehinger, B., Gameiro R. R., & Kaspar, K. (2016). Eye
movements as a window to cognitive processes. Journal of Eye
Movement Research, 9(5), 1–16.
https://doi.org/10.16910/jemr.9.5.3
Kok, E. M., & Jarodzka, H. (2017a). Before your very eyes: the
value and limitations of eye tracking in medical education. Medical
Education, 51(1), 114–122.
https://doi.org/10.1111/medu.13066
Kok, E. M., & Jarodzka, H. (2017b). Beyond your very eyes: eye
movements are necessary, not sufficient. Medical Education,
51(11), 1190–1190. https://doi.org/10.1111/medu.13384
Korbach, A., Brünken, R., & Park, B. (2017). Measurement of
cognitive load in multimedia learning: a comparison of different
objective measures. Instructional Science.
https://doi.org/10.1007/s11251-017-9413-5
Kornrumpf, B., Dimigen, O., & Sommer, W. (2017).
Lateralization of posterior alpha EEG reflects the distribution of
spatial attention during saccadic reading. Psychophysiology,
54(6), 809–823. https://doi.org/10.1111/psyp.12849
Kornrumpf, B., Niefind, F., Sommer, W., & Dimigen, O. (2016).
Neural correlates of word recognition: A systematic comparison of
natural reading and rapid serial visual presentation. Journal
of Cognitive Neuroscience, 28(9), 1374–1391.
https://doi.org/10.1162/jocn_a_00977
Kowler, E. (2011). Eye movements: The past 25 years. Vision
Research, 51(13), 1457–1483.
https://doi.org/10.1016/j.visres.2010.12.014
Krause, C. M. (2003). Brain electric oscillations and cognitive
processes. In K. Hugdahl (Ed.), Neuropsychology and
Cognition. Experimental Methods in Neuropsychology (21st
ed., pp. 111–130). Boston, MA: Kluwer Academic Publishers Group.
Kretzschmar, F., Pleimling, D., Hosemann, J., Füssel, S.,
Bornkessel-Schlesewsky, I., & Schlesewsky, M. (2013).
Subjective impressions do not mirror online reading effort:
Concurrent EEG-eyetracking evidence from the reading of books and
digital media. PloS ONE, 8(2), e56178.
https://doi.org/10.1371/journal.pone.0056178
Léger, P. M., Titah, R., Sénecal, S., Fredette, M., Courtemanche,
F., Labonte-Lemoyne, Él., & De Guinea, A. O. (2014). Precision
is in the eye of the beholder: Application of eye fixation-related
potentials to information systems research. Journal of the
Association for Information Systems, 15, 651–678.
http://dx.doi.org/10.17705/1jais.00376
Lenzner, A., Schnotz, W., & Müller, A. (2013). The role of
decorative pictures in learning. Instructional Science,
41(5), 811–831. https://doi.org/10.1007/s11251-012-9256-z
Magner, U. I. E., Schwonke, R., Aleven, V., Popescu, O., &
Renkl, A. (2014). Triggering situational interest by decorative
illustrations both fosters and hinders learning in computer-based
learning environments. Learning and Instruction,29,
141–152. https://doi.org/10.1016/j.learninstruc.2012.07.002
Mayer, R. E. (2009). Multimedia Learning(2nd ed.). New
York, NY: Cambridge University Press.
Mayer, R. E. (2010). Unique contributions of eye-tracking research
to the study of learning with graphics. Learning and
Instruction, 20(2), 167–171.
https://doi.org/10.1016/j.learninstruc.2009.02.012
Mayer, R. E., & Fiorella, L. (2016). Principles for reducing
extraneous processing in multimedia learning: coherence,
signaling, redundancy, spatial contiguity, and temporal contiguity
principles. In R. Mayer (Ed.), The Cambridge Handbook of
Multimedia Learning(pp. 279–315). Cambridge: Cambridge
University Press. https://doi.org/10.1017/CBO9781139547369.015
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce
cognitive load in multimedia learning.Educational Psychologist,
38(1), 43–52. https://doi.org/10.1207/S15326985EP3801_6
Münte, T. F., Urbach, T. P., Düzel, E., & Kutas, M. (2000).
Event-related brain potentials in the study of human cognition and
neuropsychology. Handbook of Neuropsychology, 1,
1–97.
Niefind, F., & Dimigen, O. (2016). Dissociating parafoveal
preview benefit and parafovea-on-fovea effects during reading: A
combined eye tracking and EEG study. Psychophysiology, 53(12),
1784–1798. https://doi.org/10.1111/psyp.12765
Nigbur, R., Ivanova, G., & Stürmer, B. (2011). Theta power as
a marker for cognitive interference. Clinical Neurophysiology,
122 (11), 2185–2194.
https://doi.org/10.1016/j.clinph.2011.03.030
Nikolaev, A. R., Meghanathan, R. N., & van Leeuwen, C. (2016).
Combining EEG and eye movement recording in free viewing: Pitfalls
and possibilities. Brain and Cognition, 107,
55–83. https://doi.org/10.1016/j.bandc.2016.06.004
Nikolaev, A. R., Pannasch, S., Ito, J., & Belopolsky, A. V.
(2014). Eye movement-related brain activity during perceptual and
cognitive processing. Frontiers in Systems Neuroscience(8).
https://doi.org/10.3389/fnsys.2014.00062
Olejniczak, P. (2006). Neurophysiologic basis of EEG. Journal
of Clinical Neurophysiology,23(3), 186–189.
https://doi.org/10.1097/01.wnp.0000220079.61973.6c
Paas, F. G. (1992). Training strategies for attaining transfer of
problem-solving skill in statistics: A cognitive-load approach. Journal
of Educational Psychology,84(4), 429–434.
https://doi.org/10.1037/0022-0663.84.4.429
Paas, F., Tuovinen, J. E., Tabbers, H., & van Gerven, P. W. M.
(2003). Cognitive load measurement as a means to advance cognitive
load theory. Educational Psychologist, 38(1),
63–71. https://doi.org/10.1207/S15326985EP3801_8
Palomäki, J., Kivikangas, M., Alafuzoff, A., Hakala, T., &
Krause, C. M. (2012). Brain oscillatory 4-35 Hz EEG responses
during an n-back task with complex visual stimuli. Neuroscience
Letters, 516 (1), 141–145.
https://doi.org/10.1016/j.neulet.2012.03.076
Park, B., & Brünken, R. (2015). The rhythm method: A new
method for measuring cognitive load. An experimental dual-task
study. Applied Cognitive Psychology, 29(2),
232–243. https://doi.org/10.1002/acp.3100
Park, S., & Lim, J. (2007). Promoting positive emotion in
multimedia learning using visual illustrations sanghoon park and
jung lim. Journal of Educational Multimedia & Hypermedia,
16 (2), 141–162.
Pesonen, M., Hämäläinen, H., & Krause, C. M. (2007). Brain
oscillatory 4-30 Hz responses during a visual n-back memory task
with varying memory load. Brain Research, 1138,
171–177. https://doi.org/10.1016/j.brainres.2006.12.076
Pfurtscheller, G., & Lopes da Silva, F. H. (1999).
Event-related EEG/MEG synchronization and desynchronization: Basic
principles. Clinical Neurophysiology, 110(11),
1842–1857. https://doi.org/10.1016/S1388-2457(99)00141-8
Pfurtscheller, G., Zalaudek, K., & Neuper, C. (1998).
Event-related beta synchronization after wrist, finger and thumb
movement. Electroencephalography and Clinical Neurophysiology,
109 (2), 154–160.
https://doi.org/10.1016/S0924-980X(97)00070-2
Picton, T. W. (2000). Guidelines for using human event-related
potentials to study cognition: Recording standards and publication
criteria. Psychophysiology, 37(2), 127–152.
https://doi.org/10.1111/1469-8986.3720127
Rayner, K. (1998). Eye movements in reading and information
processing: 20 years of research. Psychological Bulletin,
124(3), 372–422.
https://doi.org/10.1037/0033-2909.124.3.372
Rayner, K. (2009).Eye movements and attention in reading, scene
perception, and visual search. Quarterly Journal of
Experimental Psychology (Vol. 62).
https://doi.org/10.1080/17470210902816461
Reichle, E. D., & Reingold, E. M. (2013). Neurophysiological
constraints on the eye-mind link. Frontiers in Human
Neuroscience, 7(July), 1–6.
https://doi.org/10.3389/fnhum.2013.00361
Reingold, E. M., & Sheridan, H. (2011). Eye movements and
visual expertise in chess and medicine. In S. P. Liversedge, I. D.
Gilchrist, & S. Everling (Eds.), The Oxford Handbook of
Eye Movements(pp. 528–550). Oxford University Press.
https://doi.org/10.1093/oxfordhb/9780199539789.013.0029
Rey, G. D. (2012). A review of research and a meta-analysis of the
seductive detail effect. Educational Research Review, 7
(3), 216–237. https://doi.org/10.1016/j.edurev.2012.05.003
Ries, A. J., Touryan, J., Ahrens, B., & Connolly, P. (2016).
The impact of task demands on fixation-related brain potentials
during guided search. PloS One, 11(6),
e0157260. https://doi.org/10.1371/journal.pone.0157260
Sauseng, P., Griesmayr, B., Freunberger, R., & Klimesch, W.
(2010). Control mechanisms in working memory: A possible function
of EEG theta oscillations. Neuroscience and Biobehavioral
Reviews, 34 (7), 1015–1022.
https://doi.org/10.1016/j.neubiorev.2009.12.006
Sauseng, P., Klimesch, W., Gruber, W., Doppelmayr, M., Stadler,
W., & Schabus, M. (2002). The interplay between theta and
alpha oscillations in the human electroencephalogram reflects the
transfer of information between memory systems. Neuroscience
Letters, 324(2), 121–124.
https://doi.org/10.1016/S0304-3940(02)00225-2
Scharinger, C., Kammerer, Y., & Gerjets, P. (2015). Pupil
dilation and EEG alpha frequency band power reveal load on
executive functions for link-selection processes during text
reading. PLOS ONE, 10(6), e0130608.
https://doi.org/10.1371/journal.pone.0130608
Scharinger, C., Kammerer, Y., & Gerjets, P. (2016).
Fixation-related EEG frequency band power analysis: A promising
neuro-cognitive methodology to evaluate the matching-quality of
web search results? In C. Stephanidis (Ed.), HCI
International 2016 Posters’ Extended Abstracts, Part I (pp.
245–250). Cham, Switzerland: Springer International Publishing.
https://doi.org/10.1007/978-3-319-40548-3_41
Scharinger, C., Soutschek, A., Schubert, T., & Gerjets, P.
(2015). When flanker meets the n-back: What EEG and pupil dilation
data reveal about the interplay between the two central-executive
working memory functions inhibition and updating. Psychophysiology,
52(10), 1293–1304. https://doi.org/10.1111/psyp.12500
Scharinger, C., Soutschek, A., Schubert, T., & Gerjets, P.
(2017). Comparison of the working memory load in N-back and
working memory span tasks by means of EEG frequency band power and
P300 amplitude. Frontiers in Human Neuroscience, 11(6).
https://doi.org/10.3389/fnhum.2017.00006
Schmeck, A., Opfermann, M., van Gog, T., Paas, F., & Leutner,
D. (2015). Measuring cognitive load with subjective rating scales
during problem solving: differences between immediate and delayed
ratings. Instructional Science,43(1), 93–114.
https://doi.org/10.1007/s11251-014-9328-3
Schneider, S., Dyrna, J., Meier, L., Beege, M., & Rey, G. D.
(2017). How Affective Charge and Text–Picture Connectedness
Moderate the Impact of Decorative Pictures on Multimedia Learning.
Journal of Educational Psychology.
https://doi.org/10.1037/edu0000209
Schneider, S., Nebel, S., & Rey, G. D. (2016). Decorative
pictures and emotional design in multimedia learning. Learning
and Instruction, 44(March), 65–73.
https://doi.org/10.1016/j.learninstruc.2016.03.002
Schnotz, W., Fries, S., & Horz, H. (2009). Motivational
aspects of cognitive load theory. In M. Wosnitza, S. A.
Karabenick, A. Efklides, & P. Nenniger (Eds.),
Contemporary motivation research. From global to local
perspectives (pp. 69–96). Göttingen, Germany: Hogrefe
& Huber.
Schnotz, W., & Kürschner, C. (2007). A Reconsideration of
Cognitive Load Theory. Educational Psychology Review, 19(4),
469–508. https://doi.org/10.1007/s10648-007-9053-4
Schöne, B., Schomberg, J., Gruber, T., & Quirin, M. (2016).
Event-related frontal alpha asymmetries: electrophysiological
correlates of approach motivation. Experimental Brain
Research, 234(2), 559–567.
https://doi.org/10.1007/s00221-015-4483-6
Schüler, A. (2017). Investigating gaze behavior during processing
of inconsistent text-picture information: Evidence for
text-picture integration. Learning and Instruction, 49,
218–231. https://doi.org/10.1016/j.learninstruc.2017.03.001
Simola, J., Le Fevre, K., Torniainen, J., & Baccino, T.
(2015). Affective processing in natural scene viewing: Valence and
arousal interactions in eye-fixation-related potentials. NeuroImage,
106, 21–33.
https://doi.org/10.1016/j.neuroimage.2014.11.030
Simola, J., Torniainen, J., Moisala, M., Kivikangas, M., &
Krause, C. M. (2013). Eye movement related brain responses to
emotional scenes during free viewing. Frontiers in Systems
Neuroscience, 7(41).
https://doi.org/10.3389/fnsys.2013.00041
Smith, E. E., Reznik, S. J., Stewart, J. L., & Allen, J. J. B.
(2017). Assessing and conceptualizing frontal EEG asymmetry: An
updated primer on recording, processing, analyzing, and
interpreting frontal alpha asymmetry. International Journal
of Psychophysiology, 111, 98–114.
https://doi.org/10.1016/j.ijpsycho.2016.11.005
Strukelj, A., & Niehorster, D. C. (2018). One page of text:
Eye movements during regular and thorough reading, skimming, and
spell checking. Journal of Eye Movement Research, 11(1),
1–22. https://doi.org/10.16910/jemr.11.1.1
Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C.
(1998). Cognitive architecture and instructional design. Educational
Psychology Review, 10(3), 251–296.
https://doi.org/10.1023/a:1022193728205
Triesch, J., Ballard, D. H., Hayhoe, M. M., & Sullivan, B. T.
(2003). What you see is what you need. Journal of Vision,
3(1), 86–94. https://doi.org/10.1167/3.1.9
Van Gog, T., & Scheiter, K. (2010). Eye tracking as a tool to
study and enhance multimedia learning. Learning and
Instruction,20 (2), 95–99.
https://doi.org/10.1016/j.learninstruc.2009.02.009
Vignali, L., Himmelstoss, N. A., Hawelka, S., Richlan, F., &
Hutzler, F. (2016). Oscillatory brain dynamics during sentence
reading: A fixation-related spectral perturbation analysis. Frontiers
in Human Neuroscience, 10, 1–13.
https://doi.org/10.3389/fnhum.2016.00191
Weinreich, A., Stephani, T., & Schubert, T. (2016). Emotion
effects within frontal alpha oscillation in a picture oddball
paradigm. International Journal of Psychophysiology, 110,
200–206. https://doi.org/10.1016/j.ijpsycho.2016.07.517
Weiss, B., Knakker, B., & Vidnyánszky, Z. (2016). Visual
processing during natural reading. Scientific Reports, 6,
1–16. https://doi.org/10.1038/srep26902
Winslow, B., Carpenter, A., Flint, J., Wang, X., Tomasetti, D.,
Johnston, M., & Hale, K. (2010). Combining EEG and eye
tracking: Using fixation-locked potentials in visual search. Journal
of Eye Movement Research, 6(4), 1–11.
https://doi.org/10.16910/jemr.6.4.5
Yarbus, A. L. (1967). Eye movements and vision. New
York, NY, USA: Plenum Press.
https://doi.org/10.1016/0028-3932(68)90012-2
Zhou, W., & Gotman, J. (2009). Automatic removal of eye
movement artifacts from the EEG using ICA and the dipole model. Progress
in Natural Science, 19(9), 1165–1170.
https://doi.org/10.1016/j.pnsc.2008.11.013