Original Research

The impact of COVID-19 on psychometric assessment across industry and academia in South Africa

Mandy Wigdorowitz, Pakeezah Rajab, Tasneem Hassem, Neziswa Titi

Received: 15 Oct. 2020; Accepted: 05 Mar. 2021; Published: 31 Mar. 2021

Copyright: © 2021. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

The coronavirus disease 2019 (COVID-19) pandemic had changed the world in unexpected ways and psychometric assessment was no exception. Despite the advancements made in online psychometric assessment implementation, the authors of this commentary reflected on their own experiences in the context of the psychology profession in South Africa, where psychology professionals had been faced with the dilemma of halting, postponing or adapting assessments for remote implementation. Remote implementation had many challenges, notably shifting the logistics, financial burden and accountability onto the test-taker. In addition, when implementing remote testing, considerations of supervised and unsupervised testing need to be considered in terms of flexibility, control, test-taker comfort, standardisation, costs, ethical concerns and crisis management. Whilst in the private sector, remote psychometric assessment had been met with resilience and innovation, in academia, remote psychometric research was faced with unique challenges which affect all aspects of the research process and access to participation. Across both industry and academia where psychometric assessments were conducted, the scores and results need to be interpreted with reflection and caution as the pandemic had led to an increase in psychological distress in addition to the unique contextual challenges that South Africa already faced.

Keywords: academia; COVID-19 pandemic; industry; mental health; psychometric assessment; South Africa; remote testing.

On 26 March 2020, all South Africans were directly affected by the coronavirus disease 2019 (COVID-19) pandemic when a nation-wide lockdown was imposed to curb the spread of the disease, following an announcement by the World Health Organization (WHO; Cucinotta & Vanelli, 2020). This brought various economic and academic activities to a near standstill. One area within psychology that was directly affected was the administration of psychometric assessments, both across industry (private and public sectors) and academia. This is because the options for psychometric evaluation and in-person testing abruptly changed, whereby new ways of performing daily and structured operations were adjusted with unprecedented speed and agility in order to adhere to the lockdown regulations. Whilst the implementation of assessments is generally dynamic and constantly improving, with widespread consideration for in-person and online procedures, it has been advised that, in certain cases, examiner–examinee contact is necessary to carry out valid and reliable testing (International Test Commission, 2005, 2010; Naglieri et al., 2008). This is largely dependent on the psychological construct being measured and for what purpose it is to be used. Whilst, for instance, a metric of depression may be obtained via an online questionnaire, an in-depth cognitive evaluation may require an in-person approach.¹

South African psychology professionals (PPs) – which include all those with a psychology registration under the auspices of the Health Professions Council of South Africa – working with assessments were faced with the dilemma to either postpone or halt assessment administration or adapt and shift the assessments to an online platform, where possible. For those implementing the latter solution, two options became available. Firstly, supervised remote testing is conducted, whereby the test-taker (TT) completes the assessments whilst guided by a PP via a web-based interface. Secondly, unsupervised remote testing is conducted, where the TT independently completes the assessments under no PP guidance (see Table 1 for considerations for each testing scenario). With working-from-home as the imposed ‘new normal’, guidance was available for PPs to adopt a virtual exchange, by means of tele-psychology (Chipise, Wassenaar, & Wilkinson, 2019; Evans, 2018). Yet, focus was placed on appropriate tele-practice across a range of psychological offerings with limited emphasis on assessment. Crucially, Evans (2018) noted that when psychological assessments were conducted via tele-psychology, the integrity of the assessment needed to be preserved and the recommended administrative procedures were followed. This sentiment is shared by the American Psychological Association in its guidance on psychological tele-assessment during COVID-19, but it caveats that where administrative procedures are now unable to:

[M]imic or at least approximate the standardized protocols presented in test manuals… psychologists should take steps to collect data that are as high quality as possible and use caution and clinical expertise when interpreting those data. (Wright, Mihura, Pade, & McCord, 2020, para. 40)

As such, in conjunction with adjusting assessment practices, there are psychological effects that should be acknowledged which may have an effect on both TTs and PPs involved in the assessment process. Widespread psychological distress has been reported as a serious consequence of this pandemic (United Nations, 2020), and alarmingly, mental health-related conditions prior to the pandemic already result in annual economic losses in excess of $1 trillion, and globally, more than 80% of individuals with a mental health need, do not receive any form of quality or affordable care (WHO, 2019). The South African population is no exception to experiencing dire mental health outcomes. In an online survey aimed to gauge the mental health of South Africans as a consequence of the pandemic, the South African Depression and Anxiety Group (2020) reported that the effects of COVID-19 and the imposed lockdown restrictions were linked to an alarming elevation in people reporting mental health symptoms, stemming from financial distress to familial living conditions and an overall increase in general anxieties. Moreover, many South Africans live below the poverty line, making depression and anxiety an expected reality in the country (Francis & Webster, 2019). The coronavirus disease 2019 was therefore an additional obstacle to pre-existing adversities experienced by numerous citizens (Mayosi et al., 2012; Prince et al., 2007) and had a significant impact on how psychometric assessment practices were carried out.

In this commentary, the authors, who have diverse backgrounds in academia and the private sector, provide their reflections on psychometric assessment across industry and academia from a psychology praxis perspective. We invite PPs to think together towards more innovative and adaptable approaches for assessment across various sectors in South Africa that take into consideration the constraints to access and affordability of psychometric assessments, and the mental health effects of COVID-19 on TTs and PPs, which ultimately accentuate existing inequalities.

Assessment in industry

A search for selection and development assessments will quickly demonstrate that all local assessment providers have online platforms with immediate reporting functionalities hosting an array of ability, personality, interest and values assessments. Online administration of assessments, many of which are unsupervised (such as personality and values questionnaires) was therefore not something PPs in this field, especially those in the private sector, necessarily needed to adjust to. This is to say, for a number of years already, many of these assessments have been standardised and were available to be administered electronically, so at the face-value, it can be assumed that the psychometric integrity of these assessments is intact. However, we were faced with questions that a technical manual written prior to the pandemic could not have answered: Should PPs working in the corporate sector adjust their interpretations with reference to the impact that the pandemic may have? Are such adjustments even possible given the unparalleled experience of testing under such conditions? It is impossible to exhaustively answer these questions at present, given the copious factors at play, even when little change needs to be imposed for administering certain assessments, but they should be deliberated on in accordance with the progression of assessment throughout- and post-pandemic.

It has become exceedingly challenging for the administration of cognitive and ability-based assessments, where traditionally, unsupervised remote testing was rarely employed as a suitable option in South Africa (cf. Brearly et al., 2017). Whilst most assessments used for development purposes, such as personality and values assessments which are seen as lower ‘risk’, may be less influenced by time, context and supervision of test-taking, it is best practice to complete cognitive assessments, in particular, in the morning and without distraction, especially if the results will be used for selection and succession decisions (Ngo, Biss, & Hasher, 2018; Sievertsen, Gino, & Piovesan, 2016). Normally, in in-person settings, PPs ensure that these assessments are proctored to confirm the TT’s identity, ensure no cheating and sharing of items and provide administrative and query support. With the expansion of video-conferencing options, PPs are still able to supervise some cognitive assessments, but there are aspects that the PP has lost control over, which could still ultimately result in inaccurate scores.

Completing assessments at home impedes the home-work boundary and shifts the logistical onus (stable Internet connection, quiet setting), financial burden (electricity and data charges) and accountability (self-paced completion, distractions) onto the TT. For unsupervised remote testing, TTs can complete assessments outside of typical working hours, which could also negatively affect the integrity of the assessment. For instance, it is impossible to know whether the TT or someone else completed the assessment (which has always been a concern with unsupervised testing), or whether the items are viewed by non-intended audiences, such as spouses or roommates who are in the room at the time of assessment completion, and/or other staff members in the case of shared email addresses or shared screens if the TT chooses to complete the assessment between meetings and forgets to sign off.

Despite the changed context in which assessments must be conducted, it is clear that assessment practices within industry have adjusted, and although there are specific limitations to remote testing, overall, the private sector has exhibited resilience and innovation in dealing with assessment administration during the pandemic.

Assessment in academia

Psychometric assessment in academia has faced unique challenges too. Psychology professionals in academia utilising assessments for research purposes have had to adapt, modify or discontinue their research. If suitable assessment adaptation was not a viable option, the research needed to be altered, possibly leading to new aims and research questions, or stopped altogether, regardless of the amount of data already collected. Any of these changes could result in drastic theoretical, methodological and/or ethical implications both central to the research study itself and more peripherally, concerning the research process. Of significance is how the pandemic has affected all aspects of the research process, including the timeline and completion of the research, availability and duration of funding, degree or grant completion requirements and deadlines, research aims and hypotheses, ethical amendments and clearance, participant testing environment and additional cost implications (Hedding, Greve, Breetzke, Nel, & Van Vuuren, 2020). Prior to determining the most suitable remote research strategy, the researcher would also need to consider the implications of both supervised and unsupervised remote testing (see Table 1) and the impact each of these settings may have on the above-mentioned aspects of the research.

In South Africa, a particular concern with research involving assessments – both generally and more crucially during and post the pandemic – is the availability of sufficient funding as well as concerns about budget cuts to higher education (see Naidu & Dell, 2020). Research grants and adaptation of assessments would be contingent on the funding available for a given project, as, for example, various experts may need to be consulted in order to ensure optimal functionality of the assessment in line with test manual guidance. In addition, the researcher has to consider the psychological impact and financial burden placed on participants. Unlike industry, where the TTs choose to complete assessments for selection processes or are expected to partake in assessments as part of development initiatives, TTs in academia are mostly unpaid volunteers with little incentive to complete assessments. Thus, adding further obstacles, such as data costs and connectivity issues in more rural areas, may reduce the number of participants either willing or able to partake in remote research. These factors will further aggravate the financial strain on a population already dealing with financial difficulties (Francis & Webster, 2019). These socio-economic circumstances inadvertently result in a less diverse and representative sample, as they limit access to participation in research amongst certain social groups in the country.

Although access limitations have been accentuated in academic research because of the economic inequalities re-introduced by the pandemic, one possible benefit is that researchers can gain insight into the feasibility and psychometric integrity of remote assessment.

Concluding remarks

The COVID-19 pandemic almost brought economic and academic activities to a standstill in a country already confronted with pervasive inequality and adversity (Francis & Webster, 2019). Participation in psychometric assessment during this time, either through industry or in an academic setting, may only be suitably accessible to those with some economic and social privileges. It is, therefore, essential that psychometric assessments, and the methodological and ethical principles that accompany them, consider how the contextual landscape of South Africa has changed and continues to evolve over the course of the pandemic. Equally worth consideration is what the long-term effects will be once the pandemic is brought under control.

It is undeniable that the pandemic has fostered creativity in the way PPs adapt and administer psychometric assessments to TTs, but in implementing such innovation, it is important to recognise the specific limitations and possible social, economic and mental health effects that may emerge beyond what is normally acknowledged as key observations within the field (Irwing, Booth, & Hughes, 2018). Particular focus should be placed on equitable access, affordability and the psychological well-being of PPs and especially TTs, given the existing pressures involved in completing an assessment and performing well in it. Test-takers’ psychological well-being is an important factor that must be taken into account when selecting participants and interpreting their assessment results, given that the effects of the pandemic may further negatively affect the social, economic and infrastructural conditions in which South Africans live, quite possibly inflating depression and anxiety rates amongst normal samples. We, therefore, recommend that PPs administering assessments are attentive to possible stressors experienced by TTs, as a temporary or long-term consequence of the pandemic, which could affect performance (Wright et al., 2020).

Psychology professionals need to reconsider the tools they draw on to evaluate certain constructs, and in doing so, should be sensitive to the diversity of TTs in terms of their scores as well as their personal and social characteristics. The pandemic has also created an opportunity for test developers to demonstrate the stability of their measures, with avenues for future, post-COVID research to be conducted on the same assessments that have been used before and during the pandemic. In a post-COVID world, it would further benefit the profession to reflect on the changes enforced during this period and consider their impacts; that is, what changes will remain after the pandemic and in what ways will assessment practices shift or how might they return to the pre-COVID standard.

Acknowledgements

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

M.W. conceptualised the presented article structure and wrote the manuscript together with P.R., T.H. and N.T.

This article followed all ethical standards for research without direct contact with human or animal subjects.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

The authors confirm that the data supporting the findings of this study are available within the article.

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

References

Footnote

1. However, see Brearly et al. (2017) for a review on neuropsychological test administration by video-conference.