key: cord-0059849-4vy7ovj4
authors: Rossiter, John Anthony
title: Concepts of Threshold Assessment for a First Course in Control Engineering
date: 2020-09-09
journal: CONTROLO 2020
DOI: 10.1007/978-3-030-58653-9_1
sha: dba3068b7f9cceb9f4e690735cbeaba985fc2877
doc_id: 59849
cord_uid: 4vy7ovj4

This paper focuses on the combined challenge of encouraging students to engage with learning and assessment of their competence levels. A core challenge for many staff is the need to distinguish different levels of learning, and to evidence core competence clearly, especially for students with lower marks. This paper proposes a novel assessment strategy which separates core competencies from the more challenging application of the learning in engineering problem solving. The assessment design is efficient for staff and students and allows a reduction in student stress levels while simultaneously giving them strong incentives to adopt good working practices. Evaluation evidence is given to demonstrate the efficacy of the approach.

The most stressful part of a student journey is the assessment. Potentially future career options and a sense of self-worth are tied into the marks that students achieve. Nevertheless, despite the huge importance of good performance in assessment, the majority of students are quite poor in study skills and time management and the consequence is an increased stress at the end of year exams and a prevalence of cramming as opposed to real learning.

Many strategies have considered how to support students in better learning models and a particularly popular one [2, 4, 9, 12, 18] from the early 2000s is the use of many low weighted computer quizzes. The idea is that the small marks available in each quiz give students an incentive to engage with the quiz content and thus, implicitly, to learn the core requirements of a course in regular manner. However, even though largely successful, these quizzes create stresses of their own:

1. Despite the low weighting (often 2-3% per quiz) students get fixated on the actual mark for each quiz and even argue about something that, in effect may be worth only 0.02% of the module mark.

2. Because the actual mark is recorded, students are under pressure (stress) to perform high quality work every couple of weeks in modules where this is deployed and thus may feel under continuous stress. 3 . Some students complain that the quizzes are simplistic as they do not reward working and penalise incorrect answers with a zero mark.

So, while successful to some extent in encouraging engagement, a typical implementation can add significantly to student stress. An alternative model may provide the quizzes as formative feedback opportunities only e.g. [13, 19] ; that is, students take the quizzes, or indeed use other resources [5, 7] , to self-assess their progress but receive no marks for doing so. The author has found through experience that this model does not work for many students, irrespective of the quality of the resources, with two main negatives.

1. Most students do not do what is considered as optional [9] and thus fall behind in their studies with a knock on effect of increasing dissatisfaction and student struggles/disengagement. 2. Irrespective of the support provided, students seem unhappier with 'formative feedback' provision, even though this is intended to remove stress by allowing practice and learning without the performance being assessed formally. They want a more tangible reward for doing something.

Hence, this paper aims to improve on the model of regular computer assessment by considering mechanisms for reducing the stress in this process. It so happens that accreditation processes [1, 6, 20 ] also form a strong motivation for supporting the proposal. Accreditors put a lot of emphasis on evidence that students who pass a course have met all the learning outcomes; without this evidence that cannot accredit the degree as they argue they do not have evidence that the students have achieved the requisite skills. One downside of this mode of thinking, certainly within the UK, is that accreditors increasingly require that students achieve a good mark in every assessment component of a module (courseworks, exams, laboratories) because, if these contain distinct learning outcomes, then we can only demonstrate the student is competent, if they pass them all. One can argue about the truth or not of the accreditors' assumptions, but they make the rules and, in simple terms, we must provide clear evidence that students have passed all the learning outcomes for a (every) module if they are to be accredited.

So, in summary this paper focuses on two parallel requirements:

1. Encourage students to engage with module material on a regular basis and develop good study skills. 2. Demonstrate, efficiently, that students have met all the learning outcomes for a module to meet accreditation requirements.

It is emphasised that this paper does not discuss assesssments used to distinguish between different levels of passing performance. Section 2 introduces the concept of learning outcomes and their relevance to degree education and from this makes an assessment design proposal. Section 3 discusses the proposal and its link to accreditation and Sect. 4 gives some evaluation data for the proposal of this paper based on a pilot.

It is noted at the outset that accreditating bodies are inconsistent in their requirements for evidence of learning outcome (LO) achievement. For example, if a module is assessed by examination only, they are happy to accept that a student receiving a pass mark on the examination has met all the LO, as long as the examination assesses all the LO. However, all academics know that many students question spot and do not answer many parts of an examination paper, so a pass mark does not mean that have passed all the different elements but rather, in all likelihood they have done quite well on some questions. Despite this inconsistency where this is a single module assessment, in the UK accrediting bodies [20] (whose policies are sanctioned in liaison with other international accrediting bodies) will put undue focus on the need to pass all individual module assessments, where there are several components. This is probably because it may be clearer how the LO are separated into the different assessment components. This requirement puts huge stresses on both students and examination boards as they now have to, not only consider the student overall mark but also the component marks; in many cases students with a high overall module mark such as 15% over a pass mark, are still recorded as fail due to a single component being just a few percent below the pass mark.

The context of this paper takes account of this accreditation issue and thus, some aspects may not be fully applicable elsewhere. A module leader would like an efficient mechanism for demonstrating that passing students have met all the LO and, the argument here, is that can usefully be separated from performance assessment whereby we aim to distinguish different levels of passing performance.

In summary then, the proposal is to develop a two fold assessment strategy.

1. Pass/fail (or threshold) components which students MUST pass to demonstrate they have met all the LO for the module. 2. Classification components which are used to distinguish performance of different levels; a student with bare pass may score zero on these components as long as they have passed all the threshold components.

The design of the latter assessments is not discussed here, suffice to say they need not, and indeed should not, contain any elementary parts.

A competent chartered engineer needs to be skilled in applying their knowledge to solve problems and indeed, for many challenging problems, may also need to demonstrate imagination and creativity. Conversely, if one considers a single topic (for example electrical circuits), it may be sufficient for most engineers to have only foundational knowledge of core components and circuit analysis as this topic may have very little relevance to their job. For topics that are not their speciality, it is not reasonable to expect engineers to undertake problem solving and thus, for accreditation purposes, basic competence only is sufficient.

A core question for academics is to decide: what constitutes a basic level of knowledge and/or to what extent must students be able to apply that knowledge to meet accreditation requirements? This paper assumes a graduated level of ability, something like the following (it is understood that such things often cannot be tied down precisely so this list is intended to be indicative).

-A bare pass can be achieved by demonstrating knowledge or recall of core information and the selection and application of simple but fundamental computations linked to a topic. -For a medium level of pass, in addition to the above core skills, students should demonstrate the ability to perform straightforward applications of some of the module knowledge for problem solving. -For a strong pass, students should accurately solve most straightforward applications they have seen before within a module, and also tackle more difficult computations/analysis for some but not all topics in the module. -For a first class mark, say top 10-15%, students should accurately solve all applications they have seen before within a topic to a high degree of technical depth and also some slightly more challenging applications with novel aspects. -For a excellent mark (top few students only), in addition to the above, students should be able to select appropriate techniques and apply creativity in problem solving for scenarios they have not encountered before.

This paper forcuses on the first item in the list and thus requires a definition of the core information and computations that underpin a module and suffice to demonstrate a pass level of performance.

This paper focuses on a first course in control [17] and thus one may wish to consider, what would constitute a basic level of competence for this type of topic? Some suggestions for discussion are given in Table 1 , although clearly benchmarking this is context/institution/degree dependent in general. The list is not intended to be complete but rather is illustrative.

Most modules will have tutorial sheets, quizzes and assignments and these are often designed with the performance assessment in mind, that is to distinguish different levels of student ability. Hence a tutorial sheet will begin with threshold 

Similarly, an assignment may have a straightforward introductory part followed by increasingly challenging application components. The main proposal here is to separate out the threshold components entirely, that is to have entirely distinct threshold tutorial sheets/quizzes/laboratories. In all likelihood these already exist so to some extent it is simply a reorganisation of how material is presented to the students. However, herein lies the major change which will facilitate a reduction in staff marking time. The aim of threshold assessment is to award pass or fail, that is, not to award a mark. Hence, marking of these components is a binary decision and can be done very efficiently.

-A student attending a laboratory is judged either to have met the required skills or not. No mark is awarded. -A student taking a tutorial/test on threshold concepts (Table 1 ) is judged to pass or fail only, again no mark is awarded. Indeed, most straightforward skills can be assessed with simple binary decisions: (i) correct calculation; (ii) multi-choice questions; (iii) etc. In Sheffield we intend to use a pass mark of 70% on a large number of straightforward questions.

During the trial it was agreed that students scoring less than 70% would receive the actual mark, that is, they still receive some credit for the knowledge and skills they have demonstrated.

In summary, as part of the planning for the module delivery, the module leader ensures the core LO (at a pass/fail level only) are included in some threshold assessments. Students are expected to pass these assessments and, should they pass them all they will pass the module. Assessment will largely be automatic/binary as considering pass/fail competence only and thus can be done very efficiently.

1. Hand written tutorial sheets with binary marking schemes can be assessed quickly by a teaching assistant or a staff member or peer or an automated system, especially if the answers are entered on a standard proforma. 2. Computer quiz environments are very efficient at assessing binary decisions and once written, can be used in subsequent years. These environments will usually link to the student database adding another efficiency gain, that is, no manual handling of the assessment.

A second motivation for so-called threshold assessment is to reduce student stress levels. One might think this sounds perverse: how does adding several small assessments reduce overall stress? In simple terms the argument is that the assessments are pass/fail, that is, students are not expected to do their very best work and rather need only show that they have mastered the basics; the more advanced application of knowledge is not included. This reduces the pressure to over-prepare and any concerns about whether the mark should be 80% rather than 76% as both are awarded a pass, that is, the same mark. The positive aspect of this is that students have marks in the bag which they have merited by doing what all conscientious students should be doing anyway, keeping up with lectures and tutorial sheets at least to a basic level. In truth, we would expect students to be working harder than is required to just pass the threshold assessments. Students can then approach their more challenging assignments and end of year exams with the confidence that they have already passed the module, can be accredited and can progress. Moreover, by keeping up with basic level skills one would expect students to progress better and be more content with the more challenging aspects.

A secondary aspect is how the threshold assessments are delivered. Clearly the most important aspect is that the students develop and demonstrate basic competence, but without feeling overly stressed by the process. One stress alleviator is to emphasise that it does not matter whether it takes a student numerous attempts to achieve this, as these assessments do not influence classification. Consequently, the author gives students multiple attempts, typically 3-5, so that students who fail at the first attempt need not be stressed in the knowledge they can revise a bit more and try again. It should be emphasised that this multiple attempt approach is most efficiently delivered through a computer quiz environment as this entails no extra load to the staff member and moreover, these environments allow random number allocation and random question selection so that the student will get a similar but different test each time.

Alternatives favoured by some colleagues, and used by the author for one class, are the use of straightforward short in-class tests. These can be marked, by peers, at the end of the class as the marking scheme is very simple, thus saving marking time. Students who fail this test have a week to hand in a perfectly correct solution to the test. Again, as perfection is now expected, these later submissions will be very quick to handle and forces students to, at the very least, write down and engage with perfect solutions.

A pilot of the proposed strategy was undertaken in the Autumn of 2019 and data was collected about both student perceptions and performance across 3 modules: (i) year 2 chemical engineers; (ii) year 2 general engineers and (iii) year 1 aerospace, bioengineering and systems engineers. This section summarises the quantitative data and gives some conclusions. We will add some qualitative data to the final paper, if the students provide this in the end of module feedback forms. Some early face to face feedback suggests that the ability to demonstrate basic competence was hugely important to self-confidence for some students, which also ameliorates stress significantly.

From a staff perspective, using a virtual learning environment and colours, it is easy to get an overview of how the class is doing as a whole and also to pick out individuals who may be struggling. Figure 1 shows progress in week 8 for the smallest cohort, from which it is clear which student is totally disengaged (row 24) and needs following up: -Assessments 1, 2, 3 and 5 should be complete and the RED squares show students who did not pass this. -Assessments 4 and 6 are in progress so you can see which students have still to complete.

The remainder of this section presents a subset of the data collected from the students of the three different modules in the trial. It is noted that the profiles are similar across all modules. Questions used a typical Likert scale, that is: 1. definitely agree, 2. agree, 3. neutral, 4. disagree, 5. definitely disagree. The questions are given in the figure legends, see Figs. 2, 3, 4, 5 and 6. A basic summary is:

1. Figure 2 indicates that the assessment regime helped the large majority of students with their time management and keeping on top of their studies. 2. Figure 3 reinforces the evidence seen in previous years; if the quizzes do not carry a mark, most students will not use them despite their formative value. 3. Figures 4 and 5 indicate that, despite having an assessment every 1-2 weeks, actually most students felt the approach reduced their stress levels. 4. Figures 6 and 7 suggest that students like this assessment approach and would like it to be retained. Fig. 4 . Having multiple attempts along with the need to only score 70% helped reduce the stress of these assessments so I could focus on learning rather than getting stressed about trying to achieve the highest score possible.

Allowing me to achieve a significant percentage of the module mark by Christmas has helped reduce my overall stress linked to assessment and progress at University. 7 . I would prefer that there were no threshold assessments, that is the quizzes did not count in the module mark, and the end of year examination was worth more.

This paper has proposed an assessment regime which clearly separates basic learning outcomes from more advanced ones and thus allows students to demonstrate they deserve to pass a module/course in a less stressful manner, with only classification assessments being done at the end of a module. The preliminary evaluation demonstrates that the students appreciate the approach, even though, by their own tacit admission, it means they are working harder than they would if all interim assessment and feedback was formative. A core advantage is that the students are engaging earlier with mastering foundational concepts and the implicit hope is that this will enable them to engage better with the more challenging aspects of the module later in term. Indeed, the anecdotal evidence is that lecture attendance later in term is higher this year than in previous years, and this is likely linked to students still being able to follow what is going on. We are waiting for a discussion with accreditors (scheduled for May 2020 but postponed due to COVID-19) on their reception to this strategy, for example, do they welcome it and can this make their quality assurance easier? In fact, recent announcements from accrediting bodies suggests that this approach is likely to be actively encouraged and indeed the author's University has adopted it in year 1 for the academic year 2019-20 in response to the COVID crisis.

ABET: Accreditation in the USA

Experiences of using computer assisted assessment in engineering mathematics

Peer instruction: ten years of experience and results

The effectiveness of a computer assisted learning programme in engineering mathematics

MOOC on control of mobile robots

MATLAB assessment for final year modules

Case studies of the resources students use

Experiences in the use of web-based delivery for first year engineers

Engaging engineers in learning

Applications of online pedagogy to a first year blended learning module using a VLE

A blended learning module design approach to engage new students

Typical control curricula and using software for teaching/assessment: a UK perspective

Using quizzes instead of paper based exams to assess control topics

Opportunities and good practice in control education: a survey

A first course in feedback, dynamics and control: findings from an online pilot survey for the IFAC community

Implementation of computer assisted assessment: lessons from the literature