Original Research

Examining the internal structure of the Executive Functioning Inventory amongst South African students

Candice Britz, Casper J.J. van Zyl

Received: 13 Mar. 2020; Accepted: 29 July 2020; Published: 21 Sept. 2020

Copyright: © 2020. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

The role of executive functions in everyday life can hardly be overstated. Its influence ranges from pathological behaviour on the negative side, to quality of life on the positive side of human functioning. Assessment of executive functions includes both objective and subjective measures, which include self-report measures. Most self-report measures, however, were developed for use in clinical populations. The Executive Functioning Inventory (EFI) is a brief self-report measure developed for use in healthy populations. Psychometrically, the measure appears to function reasonably well in American and European populations; however, its internal structure is yet to be examined in South Africa. The aim of this study was to evaluate the internal consistency reliability, item functioning and factor structure of the EFI in this context. The data (n = 1904) were collected amongst students at a large urban university of the Gauteng province of South Africa. McDonald’s omega reliability estimates were mostly satisfactory with some exceptions, ranging between 0.59 and 0.76. A five-factor model consistent with a multidimensional view of executive functioning found modest support in this data. With the exception of two items, item response theory analysis further found the items of the EFI to function well on their respective subscales. Overall, the results were largely consistent with previous findings, providing initial support for its use in South Africa, especially, for research studies seeking a brief index of executive functioning or as part of a comprehensive assessment of executive functioning, if required.

Keywords: executive functioning; self-report; reliability; validity; confirmatory factor analysis.

Introduction

Executive functioning (EF) is an umbrella term for the capacity to create, sustain and shift mental sets (Suchy, 2009, 2016). Broadly, it refers to a set of top-down cognitive processes responsible for the management of coordinated thought and action (Gray-Burrows et al., 2019). There appears to be some consensus amongst researchers that there are essentially three core domains of EF (Diamond, 2013; Gray-Burrows et al., 2019; Miyake & Friedman, 2012). These include: inhibition (i.e. referring to cognitive and behavioural restraint and the determination of attentional focus); updating or working memory (i.e. momentarily holding information in memory for later processing) and set shifting (i.e. cognitive flexibility required to switch between mental tasks and operations). In combination, they facilitate several critical capabilities such as reasoning, generating goals and plans along with the ability to sustain attention and motivation to see them through (Aron, 2008). It also includes the mental flexibility required to adjust goals and plans in the event of changing circumstances. This family of behaviours are conscious and effortful and are in contrast to intuitive, instinctive, routine, automatic or otherwise overlearned behaviours (Diamond, 2013).

Whilst there may be broad consensus on the core functions, there is still no single definition or universally adopted conceptualisation of executive function (for reviews see Goldstein, Naglieri, Princiotta, & Otero, 2014; McCloskey & Perkins, 2013). Further, EF is considered a multidimensional construct rather than a single unitary trait. (McCloskey, Perkins, & Van Divner, 2009). Thus, with more than 30 definitions of EF (Goldstein et al., 2014) and as many constructs hypothesised to be contained under this umbrella (McCloskey & Perkins, 2013), it should be clear that EF refers to an array of complex, multidimensional cognitive processes and abilities (Otero & Barker, 2014).

Executive functions are predominately associated with the prefrontal cortex and associated areas (Jacobs, Anderson, & Anderson, 2008; Otero & Barker, 2014). Its developmental progression is prolonged, starting in infancy and continuing to adulthood (De Luca & Leventer, 2008). Early research on executive functions and the parts of the brain they are associated with, involve the well-known story of Phineas Gage, a man who suffered severe damage to his ventromedial prefrontal cortex, which had particularly interesting effects on his executive functions (Barkley, 2012). In subsequent years, interest in EF has continued to increase. This is not surprising, as the relevance of EF can hardly be overstated.

Executive functions play a role in just about every domain of life (Diamond, 2013). For example, they have been investigated in the context of school readiness (Morrison, Ponitz, & McClelland, 2010), school success (Borella, Carretti, & Pelgrina, 2010), job success (Bailey, 2007), romantic relationships (Eakin et al., 2004), health behaviours (Crescioni et al., 2011; Miller, Barnes, & Beaver, 2011), criminal and other potentially threatening behaviours (Broidy et al., 2003; Denson, Pederson, Friese, Hahm, & Roberts, 2011) and even quality of life studies (Brown & Landgraf, 2010; Davis, Marra, Najafzadeh, & Liu-Ambrose, 2010). Not to mention its importance to mental health. For example, EF has been implicated in schizophrenia, obsessive-compulsive disorder, depression, addictions, attention deficit hyperactivity and conduct disorder, to name but a few mental health problems where it has been implicated (Goldstein & Naglieri, 2014).

Importantly, evidence suggests that conditions of disadvantage in early life are associated with adverse cognitive development from childhood through adolescence (Berthelsen, Hayes, White, & Williams, 2018; Hackman & Farah, 2009; Hackman, Farah, & Meaney, 2010; Hackman, Gallop, Evans, & Farah, 2015; McEwen & Gianaros, 2010; Sheridan, Sarsour, Jutte, D’Esposito, & Boyce, 2012). This is particularly relevant to South Africa when considering the disadvantaged circumstances in which many children are raised that render them particularly vulnerable to deficits in EF.

As mentioned earlier, executive functions cover many constructs and behaviours. This has given rise to a number of different approaches to its measurement (Egger, De Mey, & Janssen, 2007; Spinella, 2005). Broadly, these can be categorised into subjective and objective measures of executive functions (Smithmyer, 2013). For example, a well-known objective assessment is the Wisconsin Card Sorting Test. It assesses inhibition and mental flexibility as it requires an individual to maintain a task set, to be flexible in response to feedback and to avoid perseveration by inhibiting prior incorrect responses (Salthouse, Atkinson, & Berish, 2003). Another common objective measure is the Stroop Test. This measure requires inhibiting an overlearned response in order to engage with an incongruent stimulus (MacLeod, 1991). Verbal fluency tests are another important class of objective measures. These require participants to generate several items related to some category, whilst observing and evading replication and using different retrieval strategies (Strauss, Sherman, & Spreen, 2006).

In contrast, subjective measures allow individuals to report on various aspects of EF, which provides an indication of their competence in complex, daily problem-solving activities (Toplak, West, & Stanovich, 2012), commonly referred to as self-rated executive function (SREF) measures. A well-known example includes the Behaviour Rating Inventory of Executive Functioning (BRIEF). This instrument assesses EF behaviours in children and adolescents at home and school environments. There are two versions of this measure. One requires parents and teachers to complete separate forms and the other is self-report (BRIEF-SR; Guy, Isquith, & Gioia, 2004; Toplak et al., 2012). Other self-report measures of EF include the Barkley Deficits in Executive Function Scale – Children and Adolescents (Barkley, 2012), the Delis Rating of Executive Functions (D-REF; Delis, 2012) and the Comprehensive Executive Function Inventory (CEFI; Naglieri & Goldstein, 2013).

However, there are limitations to all measures of EF (Smithmyer, 2013). For example, some do not map well to real world settings, whilst others measure only a single aspect of EF, and some – indeed most – were developed for use in clinical populations. Owing to such limitations, Spinella (2005) undertook development of the Executive Functioning Inventory (EFI), a self-report measure that seeks to index a broad spectrum of executive functions within a healthy population (Egger et al., 2007). In contrast to objective assessments of EF, self-report measures have the added advantage of being cost-effective and easy to administer.

The EFI contains 27 items and 5 subscales, namely Motivational Drive (MD), Impulse Control (IC), Empathy (EM), Organisation (ORG) and Strategic Planning (SP). Using parallel analysis, Spinella (2005) found a five-factor model as best representing the data, which is consistent with the theoretical model. Collectively these factors accounted for 49.7% of total variance (Spinella, 2005). This five-factor structure has also found support in subsequent research (Janssen, De Mey, & Egger, 2009; Smithmyer, 2013). In a second-order factor analysis, Spinella (2005) also found three higher-order factors and argued that this model is consistent with the way executive functions have been associated with the dorsolateral (SP, ORG), orbitofrontal (IC, EM ) and the anterior cingulated (MD scale) regions of the brain (Cummings, 1993). As a reviewer correctly pointed out, such models reflect a time when executive function theories still mirrored functional divisions of the frontal lobes. A view no longer accepted today (Chung, Weyandt, & Swentosky, 2014; Otero & Barker, 2014). Indeed, this model has not found support in other studies examining higher-order models of the EFI. For example, Janssen et al. (2009) only found support for two higher-order factors.

Whilst there appears to be some support for the reliability and construct validity of the EFI, to the authors’ knowledge, the EFI has not been examined for use within the South African context. The purpose of this study is to investigate the internal psychometric properties of the EFI amongst university students for use in this setting. Specifically, its reliability, factor structure and item functioning given its susceptibility to variation when used in different populations. It is therefore important to examine these properties of the EFI in this context.

Method

Participants

The data that were analysed for the study were collected as part of a larger project that explored wellness within an urban African context. Participants were 1904 undergraduate psychology students (mean = 20.07 years, SD = 2.3 years) at a large urban university in the Gauteng province of South Africa. The majority of participants (76%) were women. Participants’ home languages included: isiZulu (19.8%), isiXhosa (5.8%), English (22.7%), isiNdebele (11.9%), Sepedi (11.3%), Sesotho (7.9%), Setswana (10.5%), Siswati (5%), Afrikaans (4.5%), Tshivenda (3.3%), Xitsonga (6.5%) and unspecified (0.8%).

Instruments

Executive Function Index

The EFI consists of 27 items. The items are divided into five subscales, namely MD, ORG, IC, EM, and SP, consisting of four, five, five, six and seven items, respectively. The items of the MD scale assess behavioural drive, activity level and interest in novelty (e.g. ‘I have a lot of enthusiasm to do things’). Organisation items assess the ability to carry out organised goal-directed behaviour through functions such as multitasking, sequencing and holding information in mind to make decisions (e.g. ‘I have trouble when doing two things at once’). The IC scale measures self-inhibition, risk-taking and social conduct (e.g. ‘I take risks, sometimes for fun’). The EM scale addresses a person’s concern for the well-being of others, pro-social behaviour and a cooperative attitude (e.g. ‘I take other people’s feelings into account when I do something’). Finally, the SP scale consists of items addressing a tendency to think ahead, plan and use strategies (e.g. ‘I think about the consequences of an action before I do it’) (Spinella, 2005).

Data analysis

Reliability analysis

Three measures of reliability were computed, namely Cronbach’s alpha, Guttman 6 and McDonald’s omega. This allows for a broad consideration of the EFI’s reliability. Cronbach’s alpha and Guttman 6 are provided as they are well known by practitioners and researchers alike, especially Cronbach’s alpha. However, both have several limitations, such as the fact that they do not reflect the actual structure of a test (Bentler, 2008; Revelle & Zinbarg, 2009; Sijtsma, 2009). Also, for Cronbach’s alpha, the fact that it assumes tau-equivalence, which is mostly violated, means it will underestimate the reliability of a psychological measure (Revelle & Condon, 2019; Sijtsma, 2009). By contrast, McDonald’s omega is a latent variable modelling approach to reliability estimation, which models the structure of a test (McDonald, 1999; Revelle & Condon, 2019). As such, inferences regarding reliability will be primarily based on the results from this method.

Confirmatory factor analysis

In line with the theory informing the development of the EFI, a five-factor and a three-factor higher-order confirmatory factor model was tested, reflecting the multidimensional nature of EF according to Spinella (2005). The analysis was computed using the ‘lavaan’ package (Rosseel, 2012) in R (R Core Team, 2019). Several goodness-of-fit indices were considered to evaluate the model, including the comparative fit index (CFI; Bentler, 1990), Tucker–Lewis index (TLI; Tucker & Lewis, 1973), root mean square error of approximation (RMSEA; Steiger & Lind, 1980) and the standardised root mean square residual (SRMR). Satisfactory fit is typically reflected by CFI and TLI values greater than 0.95 and less than 0.08 for RMSEA and SRMR (Hu & Bentler, 1999). Weighted least squares mean and variance corrected estimation (WLSMV) was used given its performance on ordered categorical data relative to maximum likelihood (ML) estimation (Beauducel & Herzberg, 2006).

Item response theory analysis

Both one parameter logistic model (1PL; Rasch) and two parameter logistic (2PL; Graded Response) models were computed to more closely examine item functioning on the subscales of EFI. Winsteps version 4.5.4 was used for Rasch rating scale analysis (1960, 1980) and the ‘mirt’ package (Chalmers, 2012), version 1.32.1, was used for Graded Response Modelling (GRM; Samejima’s 1969, 1997, 2013) with the R-programming language (R Core Team, 2019). As Rasch models philosophically require data to fit the model, infit mean-square values of less than 0.60 and greater than 1.40 criteria were considered for misfit on the Likert type items of the EFI (Bond & Fox, 2007).

Procedure

The ethics committee of the Department of Psychology and Faculty of Humanities at a large urban university in South Africa granted permission for data collection. All the participants were informed about the purpose of the study and had to provide informed consent. Participants were informed that they could withdraw themselves from the study at any time if they so wished and all data will be kept private and confidential. Email was the primary format used to relay all necessary information to participants. Furthermore, a link was emailed to participants directing them to the questionnaire containing demographic questions and the psychological measures. The study was limited to current students (student numbers required) from the relevant institution. Non-students were not eligible for participation. No incentives were offered for participation. The findings were used for research purposes only.

Ethical consideration

Permission to conduct the study was obtained from the ethics committee of the Faculty of Humanities at a large urban university in South Africa (ethical clearance number: EC010562016).

Results

Descriptive statistics summarising the subscales of the EFI along with bivariate correlations are reported in Table 1. The correlations are all statistically significant ranging between 0.06 and 0.42. However, the sample size is quite large, and the available statistical power enables the identification of very small and practically insignificant associations as statistically significant, for example, the associations between MD and IC (r = 0.06) and ORG and EM (r = 0.11).

Reliability analysis

Reliability estimates for each of the subscales are reported in Table 2. Three types of internal consistency reliability were computed: McDonalds omega (total), Guttman 6 and Cronbach’s alpha to allow a broad consideration of reliability. McDonald’s omega is a latent variable-based method to compute reliability and is used for inference in this study, given some of the limitations of Cronbach’s alpha and Guttman 6 mentioned earlier (Revelle & Condon, 2019). Inspection of Table 2 shows McDonald’s omega coefficients ranging between 0.59 and 0.76. It is interesting to note that both Cronbach’s alpha and Guttman 6 estimates were lower than McDonald’s omega coefficients.

Confirmatory factor analysis

Confirmatory factor analytic (CFA) results for the models tested are presented in Table 3. The EFI is a multidimensional measure, meaning that these scales in combination represent the EF required for a range of behaviours, such as planning and goal achievement. Thus, EF is not a unidimensional latent construct underlying the scales of the EFI; hence, a five-factor higher-order model was not tested. The CFA results for the five-factor model, reported in Table 3, are mixed when considering the goodness-of-fit values. Whilst the absolute fit indices (RMSEA and SRMR) are satisfactory, the incremental fit values (CFI and TLI) are not. The same is true for the three-factor higher order model, for which the goodness-of-fit is generally weaker compared to the five-factor model.

Factor loadings for the stronger model (five-factor) are reported in Table 4. In general, most items had satisfactory loadings (ranging from 0.50 to 0.74), although there are eight items with relatively weak loadings (ranging from 0.27 to 0.48), which would influence model fit. Inspection of residual correlations and the modification indices suggested that adjustments can be made to improve the model; however, the correlated errors did not have sufficient content overlap that would justify amendments to the model. Thus, no model re-specifications were considered.

Item response theory analysis

Both Rasch and Graded Response models were applied separately to the subscales of the EFI. The items of each scale mostly fit the Rasch model well, with only item 12 (infit mean square = 1.62) of the EM subscale overfitting the model. Graded response model analysis further suggests that this is because of a relatively weak discrimination (‘a’) parameter of 0.640. These results are not overly problematic as it suggests the item is just not contributing much new information to the subscale.

Whilst not exceeding the Rasch cut-off threshold for misfit, item 4 on the MD subscale also had a high mean square infit value (1.34) relative to the other items comprising the construct. Graded response model analysis further suggested that item 4 in particular, functioned quite poorly, with improperly ordered option characteristic curves (OOC) that arguably contribute more noise than signal to the measurement of MD. This can be seen in the bottom right of Figure 1 where the OCCs for the MD items are displayed. Whilst such problematic items may be less influential overall in scales with many items, this subscale is the shortest on the EFI, and contains only four items in total. Both items 4 and 12 are the only reversed scored items on their respective subscales.

FIGURE 1: Option category curves for the items of the Motivational Drive subscale.

Discussion

The aim of this study was to examine the psychometric evidence for the EFI amongst South African students. Specifically, its internal consistency reliability, factor structure and item response functioning. Previous studies have found reasonable support for the reliability and factor structure of the EFI in different populations, however, no research has been conducted in South Africa.

The reliability results for the subscales of the EFI were mostly satisfactory, with McDonald’s omega coefficients ranging between 0.59 and 0.76, although MD (ω = 0.59) and IC (ω = 0.64) were somewhat weaker than expected. The Cronbach’s alpha estimates in this study are largely similar to those by Janssen et al. (2009) who reported estimates ranging between 0.63 and 0.69. However, both these studies found one scale each to be notably weaker; IC (0.41) was weaker in the study by Janssen et al. (2009), whilst MD (0.59) was weaker in the present study. Both these studies were conducted on university samples with very similar age and gender representation, although the present study is larger and culturally much more diverse. Spinella’s (2005) initial alpha estimates were somewhat stronger than the present results, whereas in contrast, a later study by Smithmyer (2013) reported slightly weaker results in general compared with the present study. Interestingly, Smithmyer (2013) found SP (0.49) and ORG (0.59) to have the weakest reliability amongst the five subscales, neither of which were flagged for weak reliability in previous work. Overall, there appears to be some fluctuation across reliability estimates in the literature, although results are reasonable in general. All studies – with the exception of Spinella (2005) – were conducted amongst university students, so it remains unclear to what degree these fluctuations are attributable to random sample variation. With respect to the previous samples, however, the present study was unique with regard to its diverse cultural representation. Nonetheless, whilst some weak results were observed for most scales of the EFI at least once in different studies, it is arguably a good thing that no one scale was consistently flagged as problematic. The fact that most estimates were Cronbach’s alpha coefficients – which typically underestimates true reliability – suggests that the EFI’s true reliability is under-reported in the literature and supports the notion that the EFI’s reliability is mostly acceptable.

Turning to the confirmatory factor analysis, the results for both the five-factor and three-factor higher-order models were somewhat ambiguous when considering their goodness-of-fit values. Whilst the absolute fit indices were satisfactory in general, the incremental fit values were weaker. However, when comparing the five-factor and three-factor higher-order models, the former had better fit compared with the latter. As such, the factor loadings of only the five-factor model were presented for consideration. Most items had satisfactory loadings on their expected factors, although there were eight items with relatively weak loadings, which influenced model fit. These results provide support for Spinella’s (2005) theoretical model, with five separate constructs contributing to the assessment of executive function, although model fit was modest. These results are also consistent with Smithmyer (2013), who found support for a five-factor solution. In their evaluation of the Dutch translated version of the EFI, Janssen et al. (2009) were also able to replicate the EFI structure proposed by Spinella (2005), although three items (3, 9, 10) of the SP scale had primary loadings on a separate factor in a principal components analysis. In contrast, these items had satisfactory loadings in the present study, whilst item 13 had a relatively weak loading on this factor.

Compared with the five-factor model, the three-factor higher-order model found weak support in this study. This model corresponds to three major regions of the prefrontal cortex (Cummings, 1993), for which Spinella (2005) reported some support. Like this study, Janssen et al. (2009) also found little support for a three-factor higher order model.

Item response theory analysis showed that the items of the EFI generally function appropriately in their respective subscales. This view is supported by both Rasch rating scale and graded response models. Only the two reversed scored items, 4 and 12, on the MD and EM subscales, respectively, were found to function quite poorly. It is recommended that researchers using the EFI in South Africa carefully examine the impact of these items in their own work. It would also be important to see if these items emerge as problematic in future studies conducted in this context to determine the robustness of the present results.

Some limitations of the present study should be noted. Although the data were collected in a diverse sample of students within an urban setting, its conclusions are necessarily limited to this population. As such, results cannot be generalised to the South African population broadly. Further, as Spinella (2005) indicated, the EFI is potentially sensitive to cross-cultural factors as well as differences in age, gender and education levels. This is important when considering the EFI – or any measure of EF – for use in the South African context. As mentioned before, conditions of disadvantage are known to affect the development of executive functions, and this should be borne in mind given the socio-economic disparities across racial groups that remains present in South African society. Future research is required to explore the degree to which scores on the EFI are invariant across relevant demographic strata.

Conclusion

The EFI is a self-report measure of several constructs representing aspects of everyday EF. The present study examined the internal consistency reliability, factor structure and item functioning of the EFI amongst university students in South Africa. Results show that the measure has mostly acceptable internal consistency reliability, and the confirmatory factor analysis found modest support for a five-factor model consistent with previous work (Spinella, 2005). Items of the EFI also appear to function mostly well on their respective subscales. Overall, the findings offer preliminary evidence that the EFI can be used effectively in student populations of South Africa as a brief self-report indicator of EF, noting the weaknesses and limitations described in this article. However, if a comprehensive assessment of EF is required, the EFI should be supplemented by additional measures (i.e. objective measures) along with other clinical information where relevant.

Acknowledgements

The authors have declared that no competing interests exist.

Both authors have contributed equally to this work.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Data are available on request.

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

References