key: cord-0596520-ro1qo4k6
authors: Gonzalez, T.; Rubia, M. A. de la; Hincz, K. P.; Comas-Lopez, M.; Subirats, L.; Fort, S.; Sacha, G. M.
title: Influence of COVID-19 confinement in students performance in higher education
date: 2020-04-20
journal: nan
DOI: nan
sha: 95a01431d2948b87a8080151ede1b5c25966a134
doc_id: 596520
cord_uid: ro1qo4k6

This study explores the effects of COVID-19 confinement in the students performance in higher education. Using a field experiment of 458 students from three different subjects in Universidad Autonoma de Madrid (Spain), we study the differences in assessments by dividing students into two groups. The first group (control) corresponds to academic years 2017/2018 and 2018/2019. The second group (experimental) corresponds to students from 2019/2020, which is the group of students that interrupted their face-to-face activities because of the confinement. The results show that there is a significant positive effect of the COVID-19 confinement on students performance. This effect is also significative in activities that did not change their format when performed after the confinement. We find that this effect is significative both in subjects that increased the number of assessment activities and subjects that did not change the workload of students. Additionally, an analysis of students learning strategies before confinement shows that students did not study in a continuous basis. Based on these results, we conclude that COVID-19 confinement changed students learning strategies to a more continuous habit, improving their efficiency. For these reasons, better scores in students assessment are expected due to COVID-19 confinement that can be explained by an improvement in their learning performance.

3 changes in the common format of the evaluation tools. On the contrary, students can obtain lower grades what, again, could be related to the evaluation format or to a less effective autonomous learning due to the change in teaching method.

The objective of this article is to reduce the uncertainty in the assessment process in higher education during the COVID-19 pandemic. To achieve this goal, we analyze students' learning strategies before and after confinement.

Altogether, our data indicates that autonomous learning in this scenario has increased students' performance and higher scores should be expected. We also discuss the reasons underneath this effect. We present a study that involves more than 450 students enrolled in 3 subjects from different degrees from the Universidad Autónoma de Madrid (Spain) during three academic years, including data obtained in the 2019/2020 academic year, where the restrictions due to the COVID-19 pandemic has been applied.

E-learning has experienced significant change due to the exponential growth of the internet and information technology [3] . New e-learning platforms are being developed for tutors to facilitate assessments and for learners to participate in lectures [4 -5] . Both assessment processes and self-evaluation have been proven to benefit from technological advancement. Even courses that include all the contents online such as Massive Open Online Courses (MOOCs) [6 -7] have been also become popular. The inclusion of e-Learning tools in higher education implies that a greater amount of information can be analyzed, improving teaching quality [8 -9 -10] . In recent years, many studies have been performed analyzing the advantages and challenges of massive data analysis in higher education [11] . For example, a study of Gasevic et al [12] indicates that time management tactics had significative correlations with academic performance. Jovanovic et al also demonstrated that assisting students in their management of learning resources is critical for a correct management of their learning strategies in terms of regularity [13] .

Related to autonomous learning, many studies have been performed regarding the concept of self-regulated learning (SRL), in which students are active and responsible for their own learning process [14 -15] as well as being knowledgeable, self-aware and able to select their own approach to learning [16 -17] . Some studies indicated that SRL significantly affected students' academic achievement and learning performance [18 -19 -20] . Researchers indicated that students with strongly developed SRL skills were more likely to be successful both in classrooms [21] and online learning [22] . These studies and the development of adequate tools for evaluation and self-evaluation of learners have become especially necessary in the COVID-19 pandemic in order to guarantee good performance in e-learning environments [23] .

Linear tests, which require all students to take the same assessment in terms of the number and order of items during a test session, are among the most common tools used in computer-based testing. Computer adaptive test (CAT), based on item response theory, was formally proposed by Lord in 1980 [24 -25 -26 ] to overcome the shortfalls of the 4 linear test. CAT allows dynamic changes for each test item based on previous answers of the student [27] . More advanced CAT platforms use personalization to individual learner´s characteristics by adapting questions and providing tailored feedback [28] . Research contains numerous examples of assessment tools that can guide students [29 -30 -31 ] and many advances have been also developed in the theoretical background of CAT [32] . In this aspect, advantages offered by CAT go beyond simply providing a snapshot score [33] , as is the case with linear testing. Some platforms couple the advantages of CAT-specific feedback with multistage adaptive testing [34] . The use of CAT is also increasingly being promoted in clinical practice to improve patient quality of life. Over decades, different systems and approaches based on CAT have been used in the educational space to enhance the learning process [35 -36] . Considering the usage of CAT as a learning tool, establishing the knowledge of the learner is crucial for personalizing subsequent question difficulty. CAT does have some negative aspects such as continued test item exposure, which allows learners to memorize the test answers and share them with their peers [37 -38] . As a solution to limit test item exposure, a large question bank has been suggested. This solution is unfeasible in most cases, since majority of the CAT models already require more items than comparable linear testing [39] .

The aim of this study is to identify the effect of COVID-19 confinement on students' performance. This main objective leads to the first hypothesis of this study which can be formulated as H1: COVID-19 confinement has a significative effect in students' performance. The confirmation of this hypothesis should be done discarding any potential side effect such as students cheating in their assessment process related to the distant learning. Moreover, a further analysis should be done to investigate what factors of COVID-19 confinement are responsible of the change.

A second hypothesis is H2: COVID-19 confinement has a significative effect in the assessment process. The aim of the project was therefore to investigate the following questions:

1. Is there any effect (positive or negative) of COVID-19 confinement in students' performance?

2. Is it possible to be sure that COVID-19 confinement is the origin of the different performance (if any)?

3. What are the reasons of the differences (if any) in students' performance?

What are the expected effects of the differences in students' performance (if any) in the assessment process?

We have used two online platforms. The first one is e-valUAM [40] , an online platform that aims to increase the quality of tests by improving the objectivity, robustness, security and relevance of content. e-valUAM implements all the CAT tests described in the following sections. The second online platform used in this study is the Moodle platform provided by the Biochemistry Department from Universidad Autónoma de Madrid, where all the tests that do not use adaptive questions are implemented. Adaptive tests have been used in the subjects "Applied Computing" and "Design of Water Treatment Facilities".

Traditional tests have been used in the subject "Metabolism".

Let us consider a test composed by NQ items. In the most general form, the normalized grade Sj obtained by a student in the j-attempt will be a function of the weights of all the questions α and the normalized scores ψ (Sj=Sj(α,ψ)), and can be defined as:

where the ψi is defined as

where δ is the Kronecker delta, Ai the correct answer and Ri the student's answer to the i-question. By using this definition, we limit ψi to only two possible values: 1 and 0; ψi =1 when the student's answer is correct and ψi=0 when the student gives a wrong value. This definition is valid for both open answer and multiple-choice tests. In the case of multiplechoice test with NR possible answers, ψi can be reduced to consider the random effect. In this case:

Independently of using equation 2 or 3, to be sure that Sj(α, ψ) is normalized (i.e. 0<Sj (α, ψ)<1), we must impose the following additional condition on α:

In the context of needing a final grade (FG) between 0 and a certain value M, which typically takes values such as 10 or 100, we just need to rescale the Sj(α, ψ) value obtained in our model by a factor K, i.e. FGj=K Sj(α, ψ).

We will now include the option of having questions with an additional parameter L, which will be related to the level of relevance of the question. L is a number that we will assign to all the questions included in the repository of the test (i.e. the pool of questions from where the questions of a j-test will be selected). The concept of relevance can take different significances depending on the context and the opinion of the teachers. In our model, the questions with lower L values will be shown initially to the students, when the students answer correctly a certain number of questions with the lower L value, the system starts proposing questions from the next L value. By defining NL as the number of possible L values, the L value that must be obtained in the k-question of the j-test can be defined as:

where trunc means the truncation of the value between brackets. It is worth noting that Lk is proportional to the sum of the student's answers to all the previous questions in the test. This fact means that, in our model, the Lk depends on the full history of answers given by the student. Lk is inversely proportional to NQ, which means that it takes a higher number of correct answers to increase Lk. Once Lk is defined, a randomly selected question is shown to the student.

Another important fact that implies the use of equation 5 in the adaptive test is that we will never have Lk<Lk-1. In other words, once a student starts answering questions from a certain L value, they will never go back to the previous one. In Fig 1) .

Green arrows indicate the path students take when they answer a question correctly. Students follow red arrows when an incorrect answer is given. Grades are represented at the bottom of the figure, improving with each step to the right.

MA-Ts are used for evaluation and self-evaluation of theoretical contents. In this case, we use a standard format 7 where a single correct answer must be chosen from a pool of possible answers shown by the system. Since students can sometimes answer correctly by a random selection, equation 3 must be used when evaluating the score of any item. The format of the MA-T in our CAT method requires the following elements: shows the time remaining to finish the test.

To train the contents of numerical problems, an OA-T was developed in which the statements include at least one parameter that will change its value with each execution of the application. This kind of question requires the following elements to be created:

-Statement with explicit indication of the modifiable parameter(s).

-Minimum and maximum values of each modifiable parameter.

-Programming code (Matlab in e-valUAM) that calculates the solution to the problem.

-Level of the item.

As with MA-T, there is also a possibility of including multimedia files in the item. However, in this type of question, the multimedia option is only available in the statement as the answer is numerical and included by the user in all cases. In this case the interface has a space for entering the numerical answer. In this figure we also show the feedback of the application when an incorrect previous answer has been introduced.

Traditional tests have been used in the subject "Metabolism" from the Human Nutrition and Dietetics Degree.

The course contents are divided in 6 parts corresponding with different aspects of human metabolism. Each part is taught by face-to-face lectures followed by different on-line activities that the students perform in the Moodle platform and that are later discussed between them and with the professor in face-to-face workshops. The on-line activities in each part of the course performed after the corresponding lecture are, in this order: "Exercises Workshop", "Discussion Workshop", 8 "Self-assessment" and "Test". Except for the Discussion Workshops, discussed below, all activities are on-line questionnaires (multiple-choice and cloze questions) that the students must perform in Moodle, achieving a score automatically calculated and showed that will be part of their continuous assessment. Exercises Workshops and Selfassessment are done by the students without supervision of the tutor, whereas the Tests are done on-line but under controlled conditions, just in the same way as a regular examination. The students should perform a total of 15 on-line activities: 5 Exercises Workshops, 6 Self-assessment activities and 4 Tests. Discussion Workshops are practical cases that must be discussed by the students and sent to the tutor by Moodle. As these activities do not obtain an automatic score but require a correction by the professor, they have not been considered in this study.

The control group of our study is formed by the students of "Applied Computing" and "Metabolism" 

The experimental group of our study are students of "Applied Computing" and "Metabolism" from academic year 2019/2020. In the longitudinal study of "Design of Water Treatment Facilities", experimental group corresponds to the third stage of the study.

To answer our research questions, we first set up an experiment that obtain accurate measurements of the autonomous learning activities both in the control and experimental group. The high accuracy in the measurements in the autonomous learning activities is achieved by using the adaptive tools previously described both in learning and assessment. In our experiment, students know from the beginning the evaluation format (and e-Learning tools), available for their autonomous learning. This fact is extremely motivating for students to use the tool and perform the tests. In Fig   2 we show the procedure used in our experiments to measure the autonomous learning. There are two kinds of contents in the subjects included in the experiment: theory and numerical problems. In the case of theory, students can use the e-valUAM platform through MA-T. The final evaluation is performed by OA-T. In the case of numerical problems, both learning and evaluation use OA-T. In this case, the motivation of students for using the platform is even higher because 9 their final scores will be obtained by exactly the same tool they use in their autonomous learning. This system is applied in all the experiments related to analyse the autonomous learning activity.

In the case of "Design of Water Treatment Facilities", an additional longitudinal study has been performed as is shown in Fig 2. This study has been performed in three stages. In the first stage, the self-evaluation material was presented to the students right before the evaluation test. In the second stage, the self-evaluation material was available 3 weeks before. To manage the theory contents, a MA-T with three possible answers to each question and only one correct, has been created as well. In the first stage, with the test available only the day before the exam, students had at their disposal a MA-T composed of 16 questions and with a time limit of 15 minutes. In the second stage, the time available was increased to 30 minutes and the question number to 18. These changes are related to the nature of the subject contents and do not have a significant impact on the study. The OA-T tests had 6 and 9 questions in the first and second stage, respectively.

In order to maintain consistency in the subsequent study, an additional effort has been made to keep the difficulty of the questions as similar as possible.

Stage 3 had no new material added, instead, it used all the material created in stages 1 and 2. Stage 3 corresponds to the time window between the end of classes and the last exam. In this stage, all the material was available from the beginning. The protocol used in this stage only varied from stage 2 in the inclusion of a reward for students who used the 10 application 3 or more different days. The reward was related to a bonus in the final grades.

Once the correct measure of the autonomous learning is ensured, as explained in the previous section, we must develop an experiment to describe the effect of COVID-19 confinement. This experiment compares results of control and experimental groups in all the assessment process. There are two stages that must be considered to determine the effect of the COVID-19 confinement. The first one corresponds to the period without confinement in the course 2019/2020

(before March 11), when all the measurable activities are performed in similar conditions for experimental and control groups. The second stage corresponds to the period of COVID-19 confinement (after March 11), where some measurable activities are performed in a different format and statistical differences can be found by comparing experimental and control groups.

In the case of "Applied Computing", students use the e-valUAM application in a continuous basis as it is available from the beginning of the course. They use the same OA-T for training and for their final exam, therefore the students use the application often from the beginning of the course. For this reason, in "Applied Computing", data analysis is performed over a continuous self-evaluation process over a single test along the whole academic year. The test performed by the students did not change its format and questions in the whole three years under study.

In the case of "Metabolism", the confinement started just before the beginning of Part IV of the subject.

Therefore, in this second stage the scheduled in-person lecturers from Part IV where recorded on video by the tutor and provided to the students Afterwards, they had the opportunity to perform the corresponding on-line continuous assessment activities, just in the same way as they would have done in normal conditions (as these activities were always not face-to-face). As this subject already has an important number of on-line, not face-to-face, activities programmed previously to the cancellation of face-to-face teaching, it is an extremely valuable tool to analyse the effect of the confinement on students' performance.

In 

In this subject we have analyzed students' performance expressed as their score (1-10) obtained in all the tests performed in self-evaluation. Those tests (OA-T format) were available from the beginning of the academic year and students used them in a continuous basis. First, we tested if data followed a Gaussian distribution by a D'Agostino and

Pearson omnibus normality test [41] . As the data did not pass the normality test, we used non-parametric statistical tests to compare the data from the 3 academic years. Means were compared by a Kruskal-Wallis test [42] and, when differences were found, by a Mann-Whitney post hoc test [43] to determine which pairs of means were different. All the statistical analysis was performed using GraphPad Prism 6 (GraphPad Software, La Jolla, California, USA). Data is presented as mean±SD and statistical significance was set at p<0.05.

Here, we have analysed 2 variables, the score (0-10) obtained by the students in the different activities and the proportion of students that pass the activity (score≥5), from 2019/2020 and the 2 previous academic years. Firstly compare the results from the present year with those from the 2 previous. When data from each activity followed a normal distribution, we compared the 3 means by a 1-way ANOVA followed by a an unpaired 2-tailed t-test; when data did not follow a normal distribution, we compared the 3 means with a Kruskal-Wallis test followed by a Mann-Whitney post hoc test. In order to compare the proportion of students that pass the activities (score≥5) in the different years, we performed a z-test (after confirming that our data met the central limit theorem). All the statistical analysis was performed using GraphPad Prism 6. Data is presented as mean±SD and statistical significance was set at p<0.05.

In order to substantiate these related samples of results, we performed a Wilcoxon Signed Rank test to compare two related samples that cannot be assumed to be normally distributed. In the experiments related to this subject, the same students were involved in the two samples. Data is presented as mean±SD and statistical significance was set at p<0.05.

This study with real students took place during 

The data for this study has been obtained from real students enrolled in the "Metabolism" course from the Human Nutrition and Dietetics Degree from Universidad Autónoma de Madrid, Spain. This is a 6 ECTS compulsory 

The experience with real students took place during the 2017-2018 academic year. 23 students were involved in the fourth-year optative subject "Design of Water Treatment Facilities". This subject was taught through theoretical and practical classes in the classroom and laboratory, respectively. Our study focused on the theoretical part of the subject.

This course corresponds to 6 ECTS and belongs to the Chemical Engineering Degree in the Faculty of Sciences of the Universidad Autónoma de Madrid, Spain.

In this section, we analyze students' self-learning strategies in the years that were not influenced by COVID-19

(2017/2018 and 2018/2019). However, since data is very similar in both courses, we will focus con 2017/2018 for clarity.

All statistical results can be also applied to 2018/2019. In those years, theoretical lessons were taught in the classroom and practical lessons in the computer laboratories with the physical presence of a teacher. Practical lessons always used the e-valUAM platform and the adaptive test described previously.

In Fig 4a, we show the scores obtained by all the students in the course 2017/2018 in all the attempts made with the adaptive tests. Attempts are ordered chronologically. A score of -1 in the figure implies that the student did not finish the test and did not take a numerical score. Those tests have been included in the figure as a part of student's autonomous work and must be considered as time of learning; however, will be excluded from score data analysis. There are some effects in the figure that must be considered. First, there are certain scores more repeated than others (2.5, 5 and 7.5), which correspond to a jump in the level (L) (level 1: 0-2.5, level 2: 2.5-5, level 3: 5-7.5 and, level 4: 7.5-10), meaning that students are not able to answer more difficult questions yet. Another effect that can be easily seen in the figure is a region where a huge number of tests had a score -1 (between 600 and 700 approximately). This effect appears because, in those days, teachers asked students to find the most difficult questions in the test. For that reason, students interrupted the tests when they found those questions.

However, the most interesting effect, which can be observed in 

To study the effects of a rewarded scenario, we have developed an experiment in the academic year 2017/2018 for students from the subject "Design of Water Treatment Facilities". We have used 3 stages (fully described in Materials and methods section): stage 1, where the self-evaluation tests have been made available for 1 day; stage 2, where the selfevaluation tests have been made available for 3 weeks; and stage 3, where self-evaluation tests were available for a long time with a reward related to their use. Fig 5 shows the attempts distribution in all the stages. Focusing on the MA-T, 13/20 students have made more attempts in the first stage than in the second, which is striking considering that in the second stage application was available for longer. Three students have not made any attempt in either of the 2 stages.

As for the group of students who have used the application for more than one day, we noted only 3 individuals working for at least 3 days. Considering that the application has been available for 3 weeks in this second stage, these results show the very low willingness of the students to work continuously. Focusing on the OA-T, this effect is even more significant, as only two cases have been recorded in which students have used the application for more than one day.

Increasing student motivation and analyzing the results of a more persistent use of the tool is the objective of stage 3.

In Fig 5 we can see the distribution of attempts made by the students in the whole study (3 stages). There is a fourth stage in the figure that corresponds to an additional exam made by the students that did not pass the previous ones.

Since only a few of them were involved in this fourth stage, it was excluded from the analysis. As we can see in the figure, attempts from stage 1 were limited to a couple of days, which corresponds to the short period when the test was available.

Some attempts can be found in a wider window in stage 2. However, a much higher number of attempts extended in time can be found in stage 3, which corresponds to the rewarded stage. In order to measure the use of the tool over time, we summarized the use of the system by students in stage 2 and 3 by comparing the number of days the students attempted the stage 2 MA-T and OA-T tests during the 3 weeks leading up to the exams given after each stage. It is interesting to note that, during stage 3, six more students utilized the system than in stage 2; however, the total attempts reduced between the 2 stages from 123 to 102. This translates to a 23% increase in the average use of the application over time from 1.9 days in stage 2 to 2.4 days in stage 3 combined with a reduction of attempts per student from 9.5 in stage 2, to 5.4 in stage 3. In the earlier stage, fewer students used the application intensively by making multiple attempts a day. Students who logged in at the later stage were more likely to use the application once or twice a day over several days because of the reward related to this behavior. Out of a total sample of 19 students who used the tool in in stage 3, 5 were eliminated by the test as they accessed the tests the same number of days in both stages. Then, for the use of application by the remaining 14 students the W-value was of 15.5, with the critical value of W=25 at p.0.05. Therefore, the result is significant.

We have studied two cases related with two different and common scenarios in COVID-19 distant learning. The first one is related to subjects where teachers increased the number of tasks that students should perform autonomously.

The second one is related to subjects where all the autonomous tasks are just the same as in previous courses. In this case, only face-to-face sessions have been replaced by distant learning activities such as online sessions or recorded classes.

In respectively. Normalizing with the number of students, we find 13.87 and 16.1 test/student respectively. This is a 17 difference of 2.23 test/student in a period of 3.5 months, which means a difference of less than 1 test/student each month.

The conditions after the COVID-19 confinement radically changed in the course 2019/2020 because students were required to perform a higher number of tests each week. As we can see in Fig When comparing the mean scores from the period before confinement (Fig 7) , we found that they were statistically different between the 3 academic years , differences being between the mean score from the 2019/2020 between both periods (before and after confinement). We have seen also an increase in the mean score differences after the confinement. 

This study has been performed with students from the subject "Metabolism", which is a subject where no additional tasks have been imposed to students because of COVID-19 confinement. We have analysed students' score and proportion of students that pass the activity (score≥5). As it can be observed in Fig 8a, the scores obtained by the students in the different activities along the course were consistently similar in both years of the control group, with the only exception of the second activity where there was a slightly decrease in year 2018/2019 (p<0.05). Similarly, the proportion of students with a score≥5 was very similar in the control group (p>0.05) (Fig 8b) . Therefore, students' performance in the previous 2 years (control group) were not significantly different, allowing us to compare with them the results from 2019/2020. Scores obtained by the students in 2019/2020 academic year before confinement were similar to those obtained by students from the control group, although some differences were found at the beginning of the course (activities number 2, 3 and 4, p<0.05). We think that this could be due to the adaptation of the students to the course and the new assessment methodology, being afterwards similar again. However, after the end of face-to-face teaching and beginning of confinement, students' scores were significantly higher than in the previous academic years (Fig 8a) . Moreover, we also found higher scores and an increase in students that pass the activity number 9, which is a test (Figs 8a and 8b ). In the present academic course (2019/2020), this test has been performed by the students confined at their homes, in contrast to previous years where it was performed in the classroom under the supervision of the tutor. In addition, we have analysed the proportion of students that perform the activities in the different courses (Fig 8c) , finding that this proportion is very similar, with only a difference in one activity. 

The aim of this study was to identify the effect of COVID-19 confinement on students' performance. Therefore, we conducted an experiment among 450 students from three subjects in different degrees of higher education at Universidad Autónoma de Madrid, Spain. The results of this study answer our 4 research questions:

1. Is there any effect (positive or negative) of COVID-19 confinement in students' performance? 21 2. Is it possible to be sure that COVID-19 confinement is the origin of the different performance (if any)?

3. What are the reasons of the differences (if any) in students' performance?

4. What are the expected effects of the differences in students' performance (if any) in the assessment process?

With respect to research question 1, the results show that there is a significant positive effect of COVID-19

confinement on students' performance. The results indicate that students obtain better scores in all kind of tests that are performed after the beginning of confinement. Different sources of error have been removed from our study by including only subjects that fit the following requirements for the last three academic years:

-Same teaching methodology and teachers all the years.

-Same assessment process all the years, including both distant and on-site activities. year, we have two periods with different conditions. The first period is taught in the same conditions as in previous years (before confinement). In the second period, those conditions dramatically changed, and all the teaching and learning activities are limited to distant learning. The results of our studies clearly indicate that there are significant differences in students' performance after the confinement that cannot be found before in the same year or when comparing to the previous academic years.

At this point, it is clear that the variable that correlates with the change in students' performance is the beginning of confinement. However, we cannot stablish yet if the difference is due to the: -New learning methodology.

-New assessment process.

The problem with confinement is that not only learning and teaching strategies should be modified, but also the assessment process as it cannot be done face-to-face. There are some concerns related to these new assessment methods such as the opportunity of cheating by the students. This is the reason why we have chosen only subjects that include several tests that have not been modified because of the confinement. The results of our study show that students have a significant improvement in their scores also in tests that were performed in distant format also in previous years.

Moreover, this improvement is only significant when comparing data after the confinement (i.e. there are not significant differences in tests in distant format that were performed before the confinement). These findings reveal that the new assessment process cannot be the reason of the improvement in students' performance because they also get better scores when the format of the assessment does not change. For these reasons, we establish that the new learning methodology 22 is the main reason for the changes detected in students' performance after the confinement. Now, we shall discuss research question 3 (What are the reasons of the differences (if any) in students' performance?). We have proven that something has changed in students learning methodology. The question is which is the common element in those methodologies that impulse the improvement in the learning process.

We have analysed data from two different subjects that used two very different learning strategies after the confinement. In one of them, additional e-Learning tasks were imposed to the students. Theoretical lessons were replaced by writing documents. In the other one, no additional e-Learning tasks were imposed, and theoretical lessons were replaced by multimedia classes. In both cases, we have found a significant increment in students' performance in the evaluation tests after the confinement. It seems that students' performance is increased independently of the learning strategies followed by teachers. Since we have established that the assessment process cannot be responsible of the differences, and we cannot find any common element in the learning methodologies, we must think on a general change in the autonomous learning process.

We have also analyzed data from previous years in two subjects that demonstrate that students do not work in a continuous basis. In both subjects, students work hard only the last days before the final exams. In one subject, we have found that more than 33% of the autonomous work were done the previous 5 days to the final exam. In the other subject, students only used the e-Learning material the last 2 days before the final exam, even when they were provided three weeks in advance. In this study, we have also found that students easily change their learning behavior and study continuously when a reward is offered. This extra motivation dramatically changed their learning strategy and students worked in a much wider time window, increasing their performance. An increasing in the students' performance due to an adequate time management in the learning is well-stablished in the literature [12 -13] .

In the present COVID-19 confinement, students can find by themselves many different motivations (rewards) to work in a continuous basis. First, the confinement is a new scenario that has never been faced by the students. For these reasons, students do not have any previous experience to use as a reference in their learning process. Without any previous reference, students should be confident that they are following the course correctly and therefore, work continuously to be sure that they do not miss any important content. Another interpretation is that they are afraid of missing the academic year because of the COVID-19 confinement and they work harder to overcome any difficulty. Finally, students may be motivated by their intrinsic responsibility in a very confused situation and work hard to contribute as much as they can

to solve the problems that higher education is facing. Most probably, different students will find different motivations in this new scenario (probably a combination of many). We conclude that there is a real and measurable improvement in the students' learning performance that we believe can guarantee the good progress of this academic year despite the COVID-19 confinement.

Answering question 4, we have demonstrated that students get better grades in activities that did not change 23 their format after the COVID-19 confinement. Moreover, we have demonstrated that there is an improvement in their learning performance. In conclusion, higher scores are expected due to the COVID-19 confinement that can be directly related to a real improvement in students learning.

Tracking e-learning through published papers: A systematic review

A comparative analysis of forums and wikis as tools for online collaborative learning

Designing videogames to improve students' motivation

Investigating variation in learning processes in a FutureLearn MOOC

The MOOC model for digital practice. SSHRC Application, Knowledge Synthesis for the Digital Economy

Teamwork Assessment in Collaborative Projects Through Process Mining Techniques

Assessment of collaborative learning experiences by graphical analysis of wiki contributions

Mining theory-based patterns from Big data: Identifying self-regulated learning strategies in Massive Open Online Courses

Complexity leadership in learning analytics: Drivers, challenges and opportunities

Analytics of time management strategies in a flipped classroom

Predictive power of regularity of pre-class activities in a flipped classroom

Comparing students' self-discipline and self-regulation measures and their prediction of academic achievement

Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects

The metacognitive control components of self-regulated learning

Student differences in self-regulated learning: Relating grade, sex, and giftedness to self-efficacy and strategy use

Using formative assessment to influence self-and co-regulated learning: the role of evaluative judgement

Theorizing and researching levels of processing in self-regulated learning

Developing Web-based assessment strategies for facilitating junior high school students to perform selfregulated learning in an e-Learning environment

Applying a web-based training to foster self-regulated Learning-Effects of an intervention for large numbers of participants. The Internet and Higher Education

Comparing online and blended learner's self-regulated learning strategies and academic performance. The Internet and Higher Education

Learning style based individualized adaptive e-learning environments: Content analysis of the articles published from

Fundamentals of item response theory

Applications of item response theory to practical problems

Computerized adaptive testing: a primer

Using response times to detect aberrant responses in computerized adaptive testing

Exploring feedback and student 25 characteristics relevant for personalizing feedback strategies

Adaptive Tests as a Tool for Evaluating Work Groups in Engineering

An ensemble approach in converging contents of LMS and KMS. Education and Information Technologies

Experience applying Language Processing techniques to develop educational software that allow active learning methodologies by advising students

Stochastic programming for individualized test assembly with mixture response time models

Computer Adaptive Testing: A Primer

Computer-Adaptive Testing: Implications for Students' Achievement, Motivation, Engagement, and Subjective Test Experience

The effects of computer self-efficacy, training satisfaction and test anxiety on attitude and performance in computerized adaptive testing

Adaptive Model for Computer-Assisted Assessment in Programming Skills

Strategies for controlling testlet exposure rates in computerized adaptive testing systems

Overexposure and underexposure of items in computerized adaptive testing (Measurement and Research Department Reports 2001-1

Modelling experts' behavior with e-valUAM to measure computer science skills

Tests for Departure from Normality. Empirical Results for the Distributions of b2 and √ b1

Use of ranks in one-criterion variance analysis

On a test whether one of two random variables is stochastically larger than the other

This work has been financed by the project Erasmus+ 2017-1-ES01-KA203-038266 Project of the European Union: "Advanced Design of e-Learning Applications Personalizing Teaching to Improve Virtual Education".