key: cord-1055360-0gx1pgpu authors: Hevia, Felipe J.; Tristan, Samana Vergara-Lope; Velásquez-Durán, Anabel; del Campo, David Calderón Martín title: Estimation of the fundamental learning loss and learning poverty related to COVID-19 pandemic in Mexico date: 2021-11-18 journal: Int J Educ Dev DOI: 10.1016/j.ijedudev.2021.102515 sha: 02688ec940360902aa11fff02429b0e619717dd1 doc_id: 1055360 cord_uid: 0gx1pgpu There is evidence of learning loss due to the COVID-19 pandemic, but there is no related information for Mexico. The objective of the study is to estimate learning loss in reading and numeracy in Mexico. We compared the results of two household surveys conducted in 2019 and 2021. We interviewed 3,161 children between 10 and 15 years. We estimated a learning loss according with SES in a range from 0.45-0.34 SD in reading and 0.82-0.62 SD in mathematics by COVID-19 pandemic, and an increase in learning poverty in a range of 25.7%-15.4% in reading and 29.8%-28.8% in numeracy. Gaps in fundamental learning by gender and SES increased. There is an urgent need develop a clear strategy to perform personalized diagnoses and implement remedial courses to address learning loss. The COVID-19 pandemic, and the associated school closures, have represented one of the most critical educational emergencies of the last 100 years. As we will see below, evidence has been generated regarding the adverse effects prolonged school closures had on fundamental learning. This information is critical for education systems to develop strategies to help children recover from lost learning. For this reason, it is necessary to know in greater detail how much has been lost and how far behind children will be when they return to school. In this sense, the research problem is the lack of information on the effects of the COVID-19 pandemic on fundamental learning in reading and arithmetic in Mexico. The closure of schools associated with the COVID-19 pandemic is related to learning loss and an increase in learning poverty. In Addition, Mexico is a country that kept its schools closed for many weeks, which has increased the educational inequality gap and deepened the school dropout crisis, making it increasingly difficult to maintain educational quality. There are simulations regarding the impact of this closure on learning, but there is no information based on empirical evidence. The COVID-19 pandemic has had very adverse effects on education, with school closures that have affected more than 190 million students worldwide (UNESCO, 2021) . The regions most affected by the pandemic in terms of health and education are Latin America and the Caribbean (Rieble-Aubourg & Viteri, 2020; World Bank, 2021) . This region has not only experienced a very high concentration of deaths concerning its population (John Hopkins University, 2021) but has kept schools closed longer than many other regions (UNESCO, 2021) . As has been widely analyzed, school closures and health and economic crises increase the inequality gap, the risk of dropout, educational exclusion, and, as we will see in this article, learning loss (Fore, 2021; GEM-UNESCO, 2020; United Nations, 2020; World Bank, 2020) and learning poverty Azevedo & Montoya, 2021; World Bank, 2019a) . J o u r n a l P r e -p r o o f in academic progress, most commonly due to extended gaps or discontinuities in a student's education" (Huong et al., 2020, p. sp) . Before the pandemic, learning loss was associated with summer vacations, when a drop in academic performance and a widening achievement gap among students were observed (Kuhfeld, 2019) . However, with the pandemic, interest in measuring learning loss has increased, intending to reduce learning loss and inequalities in learning and dropout Donnely & Patrinos, 2021; Huong et al., 2020; Iqbal et al., 2020; Kaffenberger, 2020; Pier et al., 2021) . At the beginning of the pandemic, statistical estimates were made to predict these losses. For example, Iqbal et al. (2020) estimated a loss of between 0.3 and 0.9 quality-adjusted years of schooling according to data from 157 countries. Azevedo et al. (2020) , through a simulation, estimated that after five months of school closure due to COVID-19, there could be a loss of 0.6 quality-adjusted years of schooling, an increase of 25% in "insufficient" level scores on standardized tests, and an increase in learning poverty, especially in the poorest countries. These projections have been complemented with empirical analyses in developed countries, which have estimated the learning loss in various contexts, including basic and higher education. Donnely and Patrinos (2021) identified studies that show learning loss in at least some of the students analyzed. In basic education, Maldonado, and De Witte (2021) identified losses of 0.19 SD in mathematics in Belgium and 0.29 SD in Denmark for 6th graders in elementary school. Engzell et al. (2020) analyzed the learning loss in mathematics, reading, and writing in the Netherlands in boys and girls between eight and 11 years old and estimated a loss of 0.08 SD between 4th and 6th grade in these domains. Kuhfeld et al. (2020) , in a sample of 4.4 million students aged three to eight years in the United States, estimated losses in mathematics of 5-10 percentage points, with no differences in reading, using an at-home online test (MAP Growth Assessment). Gore et al. (2021) analyzed the closure of schools in Australia and found that after 8-10 weeks of closure, there were losses in mathematics and reading in three elementary schools. The study by Schult et al. (2021) of 5th graders in Germany found that after 8.5 weeks of closure, there was a loss of 0.03-0.09 SD in mathematics and 0.07 SD in reading. Only one study, by Tomasik et al. (2020) , found no significant differences in mathematics in a sample of 25,685 students in Switzerland after eight weeks of school closure (Donnely & Patrinos, 2021, p. 149 ). J o u r n a l P r e -p r o o f These empirical analyses have been published only for high-income countries with a relatively short closure of schools. Thus, except for the simulations, there are no estimates of the learning loss in middle-income countries or with more extended school closings. For this reason, the first objective of this article is to estimate learning losses due to the COVID-19 pandemic in southern Mexico. On the other hand, just before the pandemic, the UNESCO Institute for Statistics and the World Bank coined the concept of learning poverty. Learning poverty is the inability to read and understand a simple text by ten years. This indicator combines schooling and learning indicators: it begins with the proportion of children who have not achieved minimum reading proficiency (as measured in schools) and is adjusted by the proportion of children who are not in school (and are assumed to be unable to read proficiently) (World Bank, 2019a, p. 6 ). This concept allows the inclusion of some of the results of the Citizen-Led Assessment, which was developed in the global south to generate evidence on the difference between schooling and learning (schooling is not learning) (Pritchett, 2013) . Additionally, it allows an understanding of the intensity of the learning crisis in the countries of the global south (Banerji, 2013; Banerji et al., 2013; Hevia & Vergara-Lope, 2020; Munene, 2016) and is a pioneering home-based assessment for investigating the learning of all children, whether they attend a school or not (Banerji, 2017; Mugo et al., 2015; PAL, 2018; Ruto, 2013; Wilson, 2018) . The Independent Learning Measurement Project (MIA, for its acronym in Spanish), which conducts Citizen-Led Assessments in southern Mexico, generated information on the learning lag between 2015 and 2019 for various states in southeastern Mexico (Hevia & Vergara-Lope, 2019; Vergara-Lope et al., 2017; . However, there is no information on the effects of school closures on learning poverty in Mexico. Therefore, the second objective of this article is to estimate the learning poverty in southern Mexico. The effects of COVID-19 in Mexico have been profound in health, economics, and education, as we will review in this document. Regarding health, a total of 326,612 excess deaths were J o u r n a l P r e -p r o o f recorded in 2020 (Mejía et al., 2021) , and as of May 2021, there were 223,068 deaths. In the economic sphere, the GDP fell by 8.5% (INEGI, 2021a) . Regarding education, between March 2020 and May 2021, schools were closed for 48 weeks. According to the National Institute of Statistics and Geography (Instituto Nacional de Estadística y Geografía -INEGI), in the 2019-2020 school year, 740,000 students between 3 and 29 years old (2.2%) did not finish the school year, 58.9% for reasons associated with COVID-19 and 8.9% for lack of money or resources. Of these, 28.8% dropped out of school because they had lost contact with their teachers or could not do homework, 22.4% dropped out because someone in the household was left without a job or had a reduced income, and 17.7% left because they lacked a computer, other device, or Internet connection (INEGI, 2021b, p. 12) . Therefore, for the 2020-2021 school year, 5.2 million students between 3 and 29 years old (9.6%) were not enrolled in school due to the COVID-19 pandemic and a lack of economic resources (INEGI, 2021b) . This situation is similar to the estimates by the Secretariat of Public Education (SEP, for its acronym in Spanish), which, at the beginning of the 2020-2021 school year, indicated that approximately 10% of students at the primary level and 8% at the high school level had dropped out (Arellano, 2020) . The "Learn at home" strategy continued to deliver its original program and study plan by remote means, initially via the internet and, during the 2020-2021 school year, via television (SEP, 2020b). According to a telephone survey conducted by the Research Institute for Development with Equality (Instituto de Investigaciones para el Desarrollo con Equidad -EQUIDE), in May 2020, only 60% of households reported having seen or heard about this program, and the proportion decreased according to socioeconomic status (SES) (Pérez & Gaitan, 2020, pp. 3-4) . Despite this, according to the government, this strategy allowed eight out of 10 students to establish constant communication with their teachers during the 2019-2020 cycle and nine out of 10 students to "maintain their learning" (SEP, 2020a). This last statement, however, was only supported by an opinion poll. As of September 2021, no document or research would allow an estimation of the effect of the pandemic on learning. For these reasons, the objectives of this article are to estimate the learning loss and learning poverty in Mexico. To meet the objectives of this research, the results of two household surveys that used the same tools to measure fundamental learning are compared. The first was developed by the J o u r n a l P r e -p r o o f and its results were published in February 2019 (MIA, 2019) . In this survey, 3,438 individuals between 7 and 17 were interviewed, and the MIA tool was applied to measure reading and basic mathematics . The second survey was implemented by the Center for Educational and Social Studies (Centro de Estudios Educativos y Sociales -CEES) and Mexicanos Primero in May 2021 in the states of Campeche and Yucatan. In Addition, in this survey, 2,000 individuals between 10 and 15 were interviewed, and a questionnaire adapted from MIA that included the same items that were applied in 2019. In the 2019 survey, the total number of households in the states of Yucatan, Quintana Roo, and Campeche was considered the population universe, according to the results of the 2015 inter census survey (INEGI, 2015) . Therefore, the sample design was defined with a probability of 95% (z = 1.96), a margin of error of +/-2%, a power of 0.8, and a type I error of 0.03. Based on this calculation, a sample of 2,564 homes was selected. Multistage, probabilistic, stratified cluster sampling was performed (Hernández Sampieri et al., 2001; Kerlinger & Lee, 2002) . The final unit of selection was the home, and the observation units were boys, girls, and young people between 7 and 17 years old who lived in the selected home. For the selection of homes, the sampling procedure involved four steps: 1. Selection of clusters or primary sampling units through stratified systematic sampling and with probability proportional to size, which in this case was the municipalities that make up each electoral district J o u r n a l P r e -p r o o f 2. Selection of the secondary sampling units, which in this case were rural localities and primary geostatistical areas (Áreas Geoestadística Básica-AGEB) for urban areas 3. Random selection of blocks within the cluster 4. Systematic selection of homes in the blocks All boys and girls between 7 and 17 years old who lived in the selected households and wanted to participate in the study were interviewed. The information was collected from 2,564 households in 39 urban and rural locations in 20 different municipalities. The surveys were performed by 293 volunteers who received six effective hours of prior training. In the 2021 survey, to calculate the representative sample, the universe of observations corresponded to the list of beneficiaries of scholarships from the Benito Juárez Welfare Program (Programa Nacional de Becas para el Bienestar Benito Juárez) for the 2019 fiscal year, which was the last year for which the scholarships were available. Therefore, the sample has a probability of 95% (z = 1.96) with a margin of error of +/-3%, a power of 0.8, and a type I error of 0.03 and included 2,037 boys and girls. In Addition, the selection of municipalities and localities was proportional to size, and individuals from 20 municipalities were interviewed (CEES, 2021) . Although both surveys are random, they were designed to identify households, and all children living in that household were interviewed. The following procedures were carried out to ensure the comparability of the bases. First, a Socio-Economic Status-SES index was generated composed of the schooling of the head of household and household characteristics: number of rooms, internet connectivity, and the existence of a car. Using Pearson's correlation, this index has a moderate and significant correlation with the Diaz-Acosta et al. Second, using normal quartiles, four levels were generated: high, medium-high, medium-low, and low SES. Third, we compared means by age between 2019 and 2021 for each SES level, finding no significant differences (Table 1) Source: Authors' analysis. For reading and mathematics, the MIA tool was applied grade. In mathematics, number recognition, addition and subtraction are aligned to 2nd grade and division and problems to 4th grade. To score the instrument, each item has a value of one point, so the scores range from 0-5 in reading and mathematics . The tool's reliability was calculated using Cronbach's alpha coefficient; internal consistency indexes of 0.82 in general, 0.78 for reading, and 0.74 for mathematics were obtained, which are considered acceptable levels (Celina Oviedo & Campo Arias, 2005) . During the 2021 administration, item 4 (History) was not applied; therefore, the analyses were performed using four reading and five mathematics items. Diverse descriptive and inferential statistical tests were used for the analysis. First, we used descriptive statics to determine fundamental learning in reading and mathematics. Second, we used Pearson correlations and the Spearman test to identify the factors associated with J o u r n a l P r e -p r o o f educational achievement. Third, means were also compared using parametric (Student's t-test) and non-parametric test (U-Mann-Whitney), and the effect size was calculated using Cohen's d (Ellis, 2010; Sullivan & Feinn, 2012) . These procedures were performed using the statistical package Stata 14.1. The results are presented in three sections: the first shows a big picture of the learning loss and estimates its effect cording by SES; in the second, the increase in learning poverty is estimated; and in the third, comparisons are made according to gender. In reading, there was a learning loss for all items and at all ages and SES groups analyzed. In this sense, early ages (10 and 11 years) and Low SES show the steepest drops. For example, at age 10, on the Comprehension item, the falls are 25% in Low SES and 15% in High SES. The accumulated results shows that the most critical reduction was on comprehension in every SES quartile (see Table 2 ). Source: Authors' analysis. When we compare reading means, we can first observe a learning loss in all socio-economic levels. And secondly, the losses are largest at the lowest SES ( Figure 1 ). The Table 3 shows a comparison of groups in reading, using the Mann-Whitney test for nonparametric data. There were statistically significant differences between 2019 and 2021 in all the SES levels. Using d-Cohen, we estimate ranging from 0.45 SD at Low SES to 0.35 SD at High SES in reading (see Table 3 ). Table 4 ). Source: Authors' analysis. When we compare mathematic means, we can first observe a learning loss in all socioeconomic levels. And secondly, the losses are higher at the lowest SES ( Figure 2 ). J o u r n a l P r e -p r o o f Source: Authors' analysis. Source: Authors' analysis. The Table 5 shows a comparison of groups in mathematics, using the Mann-Whitney test for non-parametric data. There were statistically significant differences between 2019 and 2021 in all the SES levels. Using d-Cohen, we estimate ranging from 0.83 SD at Low SES to 0.63 SD at High SES in mathematics (see Table 5 ). Note: *** sig. > 0.001. Source: Authors' analysis. Learning poverty in reading is defined as the inability of a 10-year-old child to read and understand a text adequately (World Bank, 2019a, p. 6). From Figure 3 , three observations can be made: 1) there was a significant increase in learning poverty in all SES quartiles. As figure 3 J o u r n a l P r e -p r o o f a lower rate. (Figure 4) . Finally, this section presents the comparisons between groups according to gender. The gaps between 2019 and 2021 are compared by gender according to with SES quartiles. As Table 6 shows, in Low SES, both boys and girls presented significant decreases in learning. In mathematics the losses are more unequal, in Low SES girls lost more (0.93 SD), while in High SES boys lost more (0.72 SD) (Table 5) . Note: *** sig. > 0.001; * sig. > 0.05. Source: Authors' analysis. The results are shown here reflect an educational emergency because of the COVID-19 pandemic and the closure of schools for more than 48 weeks in Mexico. The effects of the pandemic have manifested in a widespread way in both genders, in all the ages studied and for children with different SES. Although there were already severe lags in fundamental learning in reading and mathematics (2019 measurement), the loss that occurred after more than 12 months of school closures was evident (2021 measurement) , and much more severe in the low SES level. The results regarding the learning loss and the increase in learning poverty in reading and mathematics are consistent with both empirical analyses conducted in other parts of the world (Donnely & Patrinos, 2021) and existing estimates and simulations . These results also show a trend similar to the Comprehensive Learning Assessment (Diagnóstico Integral de los Aprendizajes) conducted by the government of Chile. Furthermore, this study shows that 43% of students are below the expected level in mathematics, and 60% are below the expected level in reading (Agencia de Calidad de la Educación, 2021) , confirming that the learning loss due to school closures is a generalized phenomenon. The magnitude of the learning loss forces governments and society in general to generate an urgent strategy of return to classes that has two priorities: the generation of diagnostic methods, training, and personalized evaluations that give teachers as specific a picture as possible of the diversity of learning levels that they will find when their students return to school. For this purpose, valid and reliable tools, such as those of the MIA Project, can be used to obtain immediate information regarding the reading and mathematics levels of each J o u r n a l P r e -p r o o f child with a few minutes of application; such information can be used to make decisions about the needs of a particular child and implement a leveling strategy based on the child's learning and not on his or her grade in school. Second, remedial, and leveling courses should be designed and implemented to address this learning loss, as recommended by the United Nations (2020) and prominent international organizations. The "Teaching at the Right Level" (TaRL) strategy seems to be particularly relevant and effective at both the international level (Banerjee, 2016) and in Latin American, where the MIA project has been able to evaluate remedial courses that show positive and significant effects for addressing this lag (Hevia & Vergara-Lope, 2021) . The implementation of remedial courses implies difficulties guaranteeing minimum infrastructure conditions and adequate health safety protocols in each school. In Addition, it is necessary working on comprehensive recovery that does not neglect the socioemotional and mental health situation of teachers, families, and the children themselves. Comparative experience recommends incorporating modifications to the curriculum to provide greater flexibility to teachers, avoid content saturation and incorporate formative assessment for high-impact decisions, such as student promotion. A recent report from the region highlights that, among the fifteen countries participating in the Latin American Laboratory for Assessment of the Quality of Education, only Brazil, Peru, and Mexico have not modified their curricula in this sense (UNESCO/OREALC 2021). In sum, in a comprehensive strategy of recovery, reinforcement, and leveling, diagnostic assessment is the necessary step to identify and remedy setbacks in equity and quality of education and to guide specific, non-homogeneous interventions required to achieve the fundamental learning that was not achieved (UNESCO/OREALC 2021). However, the learning lag is not a problem that can be solved in a few weeks or with the use of leveling courses alone; in this sense, together with an emerging strategy, we agree with the United Nations (2020) that there is a need to develop fairer and more egalitarian educational systems that leave no one behind. Resultados Diagnóstico Integral de Aprendizaje Asociación Mexicana de Agencias de Inteligencia y Mercado de Opinión El inmenso reto de frenar el abandono escolar Simulating the Potential Impacts of COVID-19 School Closures on Schooling and Learning Outcomes: A Set of Global Estimates (Policy Research Working Paper Núm. 9284) Building Back Better After COVID-19: The Importance of Tracking Learning Inequality. Data for Sustainable Development A systematic review of factors linked to poor academic performance of disadvantaged students in science and math's in schools The Birth of ASER. Learning Curve When Schooling Doesn't Mean Learning The Annual Status of Education Report Nota técnica cálculos de muestra Yucatán y Campeche 2021 Aproximación al uso del coeficiente alfa de Cronbach Medición simplificada del nivel socioeconómico en encuestas breves: Propuesta a partir de acceso a bienes y servicios Learning loss during COVID-19: An early systematic review The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results Learning Loss Due to School Closures During the COVID-19 Pandemic Los niños y niñas no pueden permitirse otro año sin escuela The impact of COVID-19 on student learning in New South Wales primary schools: An empirical study. The Australian Educational Researcher Metodología de la investigación Evaluaciones educativas realizadas por ciudadanos en México: Validación de la Medición Independiente de Aprendizajes Educational Accountability or Social Accountability in Education? Similarities, Tensions, and Differences (Working Paper Núm. 5; p. 64). Medición Independiente de Aprendizajes Particularidades, límites y potencialidades de las evaluaciones dirigidas por ciudadanos en América Latina Efectos de cursos remediales en aprendizajes y equidad durante la COVID-19 en México 2020 |, & Learning | 0. (2020, mayo 18). The COVID-19 induced learning loss -What is it and how it can be mitigated? The Education and Development Forum Comunicado de prensa núm. 97/21 29 de enero de 2021 Encuesta para la Medición del Impacto COVID-19 en la Educación (ECOVID-ED) 2020. Nota técnica We should avoid flattening the curve in education -Possible scenarios for learning loss during the school lockdowns COVID-19 Map Modeling the Long-Run Learning Impact of the COVID-19 Learning Shock: Actions to (More Than) Mitigate Loss Investigación del comportamiento: Método de investigación en ciencias sociales Surprising New Evidence on Summer Learning Loss Learning during COVID-19: Initial findings on students' reading and math achievement and growth The Effect of School Closures on Standardized Student Test Outcomes Estimación del exceso de mortalidad por todas las causas durante la pandemia del Covid-19 en México. Salud Pública de México Resultados medición A Call to Learning Focus in East Africa: Uwezo's Measurement of Learning in Kenya, Tanzania and Uganda Achieving Education for All: Dilemmas in System-wide Reforms and Learning Outcomes in Africa Usos desiguales de Universidad Iberoamericana COVID-19 and the Educational Equity Crisis Evidence on Learning Loss from the CORE Data Collaborative. Policy Analysis for California Education The Rebirth of Education: Schooling Ain't Learning Educación más allá del COVID-19 Information on learning outcomes as a catalyst to spur citizens to act: Examples from Tanzania, Uganda and Kenya Did Students Learn Less During the COVID-19 Pandemic? Reading and Mathematics Competencies Before and After the First Pandemic Wave Iniciará el Ciclo Escolar 2020-21 con el modelo de aprendizaje a distancia Aprende en Casa II Using Effect Size-Or Why the P Value Is Not Enough Educational gains of in-person vs. distance learning in primary and secondary schools: A natural experiment during the COVID-19 pandemic school closures in Switzerland Global monitoring of school closures caused by COVID-19 A un año del comienzo de la pandemia: Continuidad educativa y evaluación en América Latina y el Caribe en Education during COVID-19 and beyond United Nations La Reforma Educativa: ¿Ha dado resultados? Construcción de línea base para una evaluación independiente de aprendizajes y factores asociados al logro educativo. En Premio Nacional de Investigación Social y de Opinión Pública Evaluación alternativa de competencias básicas de lectura y aritmética y análisis de factores asociados en Yucatán How Civil Society Can Supply Rigorous Data for the SDGs: The Citizen-Led Assessment Approach. Data for Sustainable Development Learning Poverty World Bank Mexico. Learning Poverty Brief The COVID-19 Pandemic: Shocks to Education and Policy Responses Acting Now to Protect the Human Capital of Our Children: The Costs of and Response to COVID-19 Pandemic's Impact on the Education Sector in Latin America and the Caribbean We would like to thank the "Instituto Campechano" and "IEPAC" who helped recruit and train volunteers for the data collection in 2019.