key: cord-0068263-5tztga01 authors: O’Sullivan, Linda; Casey, Deirdre; Crowley, Julie title: Asynchronous online mathematics learning support: an exploration of interaction data to inform future provision date: 2021-09-28 journal: nan DOI: 10.1093/teamat/hrab016 sha: 83e7b808cbab75b06a1e62dd46824cabfc484dde doc_id: 68263 cord_uid: 5tztga01 Over the past 30 years, higher education institutions (HEIs) worldwide have been grappling with the difficulties experienced by many students entering higher education due to their poor pre-entry core mathematical skills. In the Republic of Ireland and the UK, the provision of mathematics learning support (MLS) is the approach most commonly adopted by HEIs to deal with this problem, providing free one-to-one mathematical support and/or workshops for students. However, despite the availability of such supports and research that suggests that engagement with these supports can have a positive impact on student retention and progression, ensuring high levels of student engagement with these supports remains a significant challenge. In more recent years, some institutions have started to provide online MLS which takes many different forms but mainly provide links to websites, revision notes, past exam papers, etc. In light of the COVID-19 pandemic and the need for many to resort to remote teaching, it would appear the successful provision of online mathematics support is set to become increasingly important over the coming years. This paper will examine student interaction data with one such online MLS provision, hosted within a virtual learning environment. The interaction data will be analysed in order to gain a better understanding of the level of student engagement with this resource and the content most frequently accessed. This analysis will be used to inform the future development and enhancement of the resource so as to encourage student engagement. For more than 30 years now, mathematics educators in the Republic of Ireland have expressed concern about the difficulties experienced by large numbers of students in numerate disciplines in higher education due to their poor pre-entry mathematical skills (Department of Mathematics and Computing/Cork Regional Technical College, 1985; Hurley & Stynes, 1986) . Similar concerns have also been expressed in the UK, Australia and the USA (Hourigan & O'Donoghue, 2007) so much so that this phenomenon is now commonly referred to in the literature as the 'mathematics problem' (Lawson et al., 2003; Gill et al., 2010) . From a higher education perspective, the 'mathematics problem' is a catch-22 situation. On the one hand, governments far and wide are attempting to ensure economic viability by widening participation in higher education and are providing funding to higher education institutions (HEIs) based on the number of students attracted and retained. While on the other hand, HEIs are attempting to compete for this funding but are not in a position to demand the level of mathematical competency deemed necessary to guarantee students' success in their chosen discipline. As a result, HEIs have been forced to put a range of mechanisms in place to do the best with the students they have in order to meet the demands of government, retain these students and maintain the associated funding. Generally, these measures have included changes to the curriculum, introduction of bridging courses, grouping students by ability and increasing contact hours. In addition, diagnostic testing is sometimes used to monitor the changing nature of student intake, to identify gaps in prior knowledge, to identify at risk students and to inform the development of appropriate materials in learning support centres. However, an increasingly common approach across many HEIs has been the provision of mathematics learning support (MLS) often provided through dedicated physical spaces which are often referred to as MLS Centres (MLSC) (Lawson et al., 2001 (Lawson et al., , 2003 Cronin et al., 2016) . MLS is an umbrella term for a wide range of mathematical provision and is interpreted as a facility offered to all students, whose programme of study requires some level of mathematical proficiency, which is provided in addition to their regular programme of study, i.e., timetabled lectures, laboratories, tutorials, etc. (Lawson et al., 2003) . While the provision of MLS in higher education is at this stage quite extensive (Lawson et al., 2003; Cronin et al., 2016) and the need for mathematics support is set to continue for the foreseeable future (Lawson & Croft, 2015) , their future is far from secure. For example, MLS funding on the island of Ireland is often of a non-regular nature which has the potential to be withdrawn at any stage (Mac an Bhaird et al., 2013; Cronin et al., 2016) . Hence, there can be significant differences in MLS in terms of setup and mode of operation largely due to the variety of funding models in place-i.e., in some HEIs, MLS is funded centrally, while in others, it is funded by departments, and in some cases, it has no funding at all. As a result, some MLS provisions have a dedicated room to house their activities while others use general purpose classrooms, some employ dedicated MLS staff to provide support to students while others use existing teaching staff or post-graduate students and some provide printed resources and textbooks while others provide a variety of online resources including video. However, while the setup and operational aspects of MLS may differ, the nature of the support is similar in that most operate a mixture of one-to-one support either on a drop-in basis or through prebooked appointments or alternatively offer a timetabled series of workshops on fundamental topics. Unfortunately, while there is much research to suggest that MLS can have a positive impact on student retention and progression (Lee et al., 2008; Mac an Bhaird et al., 2009) , there is a huge problem with student non-engagement with these services, in particular by those students most in need, i.e., with weak mathematical backgrounds (Cronin et al., 2016) . As mentioned previously, a common feature of MLS in many HEIs has been the provision of one-to-one support either through pre-booked appointments or on a drop-in basis. This approach, while highly valued among those students availing of the service, can be quite expensive to provide due to administrative overheads associated with the former and costs associated with supplying adequate levels of support for the latter due to uncertain and fluctuating levels of demand. A study into student non-engagement with MLS found that the reasons for student nonengagement are quite varied and complex but generally those from weaker mathematical backgrounds tend to cite issues to do with a fear of asking for help and the structure of the MLS such as lack of information about the MLSC, not knowing its location, unsuitable opening hours and problems with how specific services are provided (Mac an Bhaird et al., 2013) . Therefore, given the precarious nature of funding for MLS (Mac an Bhaird et al., 2013; Cronin et al., 2016) to maintain MLS provision going forward HEIs have had to take steps to address some of these issues to ensure they are making the most of the funding they are allocated, they are reaching as many of their student body as possible and that the approaches they adopt are sustainable in the face of reduced, or even worse, no funding in the future. One approach that has the potential to address many of these issues is the provision of some level of asynchronous online MLS (OMLS). From a student perspective, this type of support should appeal as follows: students do not have to ask for help-they can just utilize the service when and if they need it; they do not have to physically go to a location to avail of help they can access it online; and there is no issue with opening times as this type of support is available on demand when and if it is needed. From a HEI's perspective, this would appear to be quite a cost-effective model as while it requires resourcing to set up and maintain such an online service, it is potentially a much more sustainable approach going forward. In addition, in light of the COVID-19 pandemic and the need for many to resort to remote teaching, it would appear the successful provision of OMLS is set to become increasingly important to HEIs over the coming years. A survey of MLS provision on the island of Ireland (Cronin et al., 2016) showed that almost half of those institutions surveyed offered some form of online mathematics provision. However, that, which was considered to correspond to online mathematics provision, took many forms with many providing revision notes, links to websites or video tutorials via either a dedicated website or an offering through their virtual learning environment (VLE). Clearly, this indicates that there is a huge potential to enhance and expand this type of mathematics provision to better meet the needs of students, but to achieve this, it is necessary to understand what students value and utilize most in such an environment. While much research has been conducted around MLS provision in higher education, from evaluating the impact of such provision (Mac an Bhaird et al., 2009; Gill et al., 2010) to identifying reasons for non-engagement (Mac an Bhaird et al., 2013) and developing the best practice for MLS provision (Lawson et al., 2003) , similar research around OMLS provision is scarce. Results of a recent survey on OMLS presence in the Republic of Ireland and the UK found that there were very low levels of evaluation of OMLS provision and student engagement with these services (Mac an Bhaird et al., 2020) . Unlike physical MLSC, where the capturing of accurate engagement data can be difficult (Cronin & Meehan, 2015; Cronin et al., 2019) , OMLS have, by their nature, a wealth of access and usage data available which is automatically generated and stored as students interact with the resource. If harnessed and analysed correctly these data could enable providers to gain a better understanding of student behaviour and take appropriate steps to optimize their online offering to maximize student engagement. In terms of online course design, research suggests course structure, content presentation and opportunities for collaboration and interaction, and timely feedback are key considerations to enhance student learning and engagement (Lister, 2014) . Given that OMLS can be presented as an online self-directed asynchronous course, it is anticipated that many of these principles will also apply to the design of OMLS. In this paper, we will focus on Maths Online, an OMLS provision at Cork Institute of Technology (CIT) offered via the institute's VLE, Canvas. We will examine how learning analytics (SoLAR, 2021), in the form of descriptive analytics, can be used to gain a better understanding of student engagement with the resource and how this information can be used, going forward, to help inform the design of the resource and potentially enhance engagement. We will outline how the relevant data were extracted from the VLE and analysed. We will also share key findings from this analysis in the hope that they may be of assistance to other HEIs who are considering, or who are in the process of developing, OMLS. CIT provides MLS through what is known as its Academic Learning Centre (ALC). The ALC provides free support to students in mathematics and statistics as well as other subjects including physics, computing, economics, chemistry, accounting, mechanics and academic writing. Prior to COVID-19, this support took the form of timetabled drop-in subject support sessions which ran throughout the academic year. Two staff members are responsible, on a job share basis, for the day-to-day running of the ALC with subject specific support provided by academic staff. In addition to physical on campus face-to-face MLS sessions, CIT's mathematics department have, over the past number of years, developed an OMLS for its students called Maths Online to act as a supplement for students studying mathematics or statistics modules. The origins of Maths Online can be traced back as far as 1993 when a set of text-based units were developed as part of a CIT initiative, in the area of open and distance learning research, to innovate in education for quality and access. Subsequently in 2003, Maths Online was created with these units and other resources implemented in CIT's then VLE, Blackboard. This version of Maths Online was available to CIT students for almost 20 years housing notes and auto-corrected multiple-choice questions on a range of key topics in mathematics and statistics including algebra, trigonometry, calculus, descriptive statistics and probability. As of January 2019, Maths Online is now implemented as a standalone resource through Canvas, CIT's new VLE. Currently, it provides a range of useful mathematical resources such as exercises, solutions and quizzes which have evolved over the past 20 years. The content is organized by topic, i.e., algebra, statistics and probability, functions, trigonometry and calculus and faculty, i.e., Business & Humanities or Science & Engineering. In addition, students can download mathematical software needed for their studies, i.e., MAPLE, SPSS or Minitab, and links are provided to useful mathematical websites. Other features include an announcements feature, for broadcasting information to all students enrolled in Maths Online; a discussion area, to enable interactive communications between ALC staff and students and more recently; and, as a result of the COVID-19 pandemic, a facility to book online mathematics support sessions with a member of the MLS team (Casey, 2020) . It should be noted that the software downloads are the only downloadable content in Maths Online. The resource is primarily aimed at first/second year non-specialist mathematics students, i.e., students who study a mathematics or statistics module, provided by CIT's mathematics department, as part of their primary degree. Hence, at the start of each academic year all first/second year students studying such modules are automatically enrolled in Maths Online. However, as Maths Online is potentially useful to all students, who may be experiencing difficulty with mathematics or statistics, there is also a self-enrol feature which enables any student, at any stage in their studies, to utilize the resource. Recent enhancements to Maths Online include the development and inclusion of video solutions to past exam paper questions for a specific module, Statistics & Financial Maths for Business (STAT6011), which appeared problematic to many students, and the incorporation of practice exercises in a variety of topics using Numbas, the freely available e-assessment tool for mathematics developed at Newcastle University (Carroll et al., 2017) . In Maths Online, these practice exercises are organized by topicnamely algebra, differentiation, logarithms, indices, percentages, matrices, descriptive statistics and summary statistics. These topics were identified by MLS support lecturers as key areas where students would benefit from additional support. Inclusion of such a facility enables students to practice particular types of mathematical problems in a given topic, receive instant feedback on their submission and advice on where they may have gone wrong, and they then have the opportunity to attempt other similar autogenerated questions. The rationale behind both of these enhancements was to make the resource more student-centred with the goal of enhancing student engagement and learning. Research has shown that for students' assessment drives learning, students can benefit from answering practice questions and that students need timely feedback on their learning to succeed (Cepeda et al., 2008; Gibbs, 2010; Hopkins et al., 2016) . Hence, it was thought that students might be more inclined to engage with the Maths Online resource if they felt there was something more tangible in it for them. What better way to achieve this than through the provision of video solutions to past exam paper questions together with an opportunity to practice questions and receive instantaneous feedback? By providing this type of content, it is hoped that students will become self-regulated learners capable of identifying their learning attainments/deficiencies and take appropriate action, if necessary, by availing of further Maths Online content or synchronous MLS via the ALC. While the potential of both enhancements to improve student learning is clear, evidence of student engagement is needed to justify any further development as a significant amount of time and effort is required to develop such content (Cronin & Meehan, 2020) . Given that Maths Online is implemented through a VLE, any user who interacts with the resource will leave digital trace data behind them that can be extracted and analysed to provide valuable insights into user behaviour and content/feature usage. In this paper, we explored Maths Online's Canvas interaction data to gain a greater understanding of the following: • How many students have accessed Maths Online? • How often did students access content within Maths Online? • How much time overall did students spend using Maths Online? • What Maths Online content did students most frequently access? Having answers to these questions should feed into more informed decision-making about the design of the resource that may enable enhanced engagement. In addition, it will provide a baseline against which the effectiveness of any future development of the resource can be measured. The Canvas VLE automatically provides an Access Report for each student enrolled in Maths Online which shows a summary of student engagement with the resource. Figure 1 shows an example of one such Access Report, it indicates the type of content (as an icon), the name of the content, the number of times the student viewed the content, the number of times the student participated in the content (if applicable such as posting to a discussion) and the last time the student viewed the content. Unfortunately, Canvas only provides this Access Report on a student-by-student basis and does not automatically provide a facility to download all of the Access Reports for all students enrolled in Maths Online. However, to gain a better understanding of overall student engagement with Maths Online content an analysis of this data would be necessary. To obtain the dataset of interest, it was necessary to install a script within Maths Online, developed by a member of the Canvas Community (Jones, 2020) . When run, this script obtained a list of all students enrolled in Maths Online, from the time the resource was created on Canvas to the time at which the script was run, it retrieved the relevant Access Report for each of these students and compiled all of the Access Reports into a single comma separated values (CSV) file. It should be noted that the resultant CSV file did not contain data for students enrolled in Maths Online who had not viewed resource content or participated in the resource. Therefore, to obtain a complete dataset for all students enrolled in Maths Online, an additional Python script was developed, by the first author, which utilized the Canvas API to retrieve the list of students enrolled in Maths Online and which saved these data to a Student Enrolment CSV file. For each student, the student enrolment CSV contained a row consisting of the following data: • User ID -a unique Canvas ID assigned to each user; • SIS_User_ID -what the student information system (SIS) knows the user by (in the case of CIT, it's the student's CIT student ID that is used to uniquely identify a student); • Display Name -the name of the student displayed in Canvas which is normally in First-Last name format; • Email -is the student email address, in the case of CIT it's the student's CIT email address; • Created_at -is when the student's Canvas account was created. It should be noted, from a Maths Online perspective, data available in Canvas in relation to the origin of the student is quite limited. For example, it is not possible to determine the faculty or discipline the student comes from, the stage of their studies, the mathematics or statistics modules they may be studying or even to distinguish between those who were automatically enrolled and those who self-enrolled. While such data are not available within Canvas itself, it is anticipated that much of this data is available within the CIT's SIS which, going forward, we would hope to gain access to for future work so that datasets can be cross-referenced, and even more detailed analyses carried out. Maths Online has been hosted within the Canvas VLE since January 2019 and both aforementioned scripts were run in May 2020. Hence, this analysis is based on approximately 17 months of operational data. The Student Enrolment CSV file contained 4457 records while the Access Report CSV file contained 17202 records. To verify the scripts had run correctly, a small number of students were randomly selected and their data in Canvas were cross-checked against the corresponding CSV files for accuracy. In each case, it was found the data matched, and therefore, we could conclude that the scripts run were valid. Both CSV files were then imported into Microsoft Excel for further analysis. In advance of any detailed analysis, however, some data cleansing was necessary, this can be summarized as follows. • In the Student Enrolment dataset, it was found, by examining the Email field, that there were 22 CIT staff who had enrolled as students; hence, their data were removed together with the corresponding 226 records from the Access Report dataset which corresponded to these users. • In the Access Report dataset there were 53 records corresponding to students who did not exist in the Student Enrolments dataset. These records appeared to correspond to students who had withdrawn from CIT, so these records were also removed. Hence, the final Student Enrolment and Access Report datasets contained 4435 and 16923 records respectively representing the students enrolled in Maths Online and their corresponding Access Report data. Determining how many students had accessed Maths Online required the cross-referencing of the Student Enrolment and Access Report datasets-students for which there was no data in the Access Report dataset could be said to have never accessed Maths Online, while students for which data existed in the Access Report dataset could be said to have accessed Maths Online at least once. The results are reported in Table 1 and, considering Maths Online is an optional resource for students to engage with, the figure of 72% would seem like a high proportion of students to have accessed the resource. In order to obtain a greater understanding of the level of engagement by those students that had accessed Maths Online, we attempted to gauge how frequently students had accessed the resource's content. Unfortunately, the Access Report dataset provided quite limited data in this respect. However, it does contain the following fields for each student and each piece of content they accessed within the resource. • Last Access -the last date and time the student viewed or participated in the specified content, as per the Canvas Student Access Report. • First Access -the first date and time the student viewed or participated in the specified content, as per the Canvas Student Access Report. • Last Activity -displays the date and time of a student's most recent interaction with Maths Online in general and not a specific piece of content. Hence, for a given student the value of this field is the same for all records in the Access Report dataset associated with that student. Within Canvas, this field is not shown on the Student Access Report but is instead available on the Canvas People pagea page which displays high level information about the students enrolled in Maths Online including name, student ID, last activity and total activity and is the location from where individual Canvas Student Access reports can be accessed. This time stamp is updated when the student navigates through the resource using the resource's Navigation menu, replies to a discussion or submits an assignment or quiz, or when they access resource files and pages. It should be noted that for some records this field was empty, as Last Activity does not get updated when students interact with Canvas using mobile devices, and in these cases, Last Activity was assumed to be the same as Last Access. Although these fields contain data relating to interactions that as such occurred at different levels, with Last Access and First Access being at the content level and Last Activity at the resource level, these fields provide three separate time stamps from which a rough estimate of the number of days each student had engaged with Maths Online can be determined. This was achieved by comparing the Last Access, First Access and Last Activity dates for each row in the Access Report dataset as follows: • if all three fields contained the same date, then this was considered to indicate engagement on at most 1 day; • if only two of the three fields contained the same date, then this was considered to indicate engagement on at least two different days; • if all three fields contained different dates, then this was considered to indicate engagement on three or more different days. When these data were grouped by student, it was possible to assign an overall category for the number of days of engagement to the student as an indicator of their level of activity within the resource, by determining the maximum number of days the student had accessed any content within Maths Online. From Fig. 2 , it can be seen that 73% (2234) of students accessed content on more than one day-with only 29% (917) accessing content on three or more days. This would appear to present a slightly different picture of student engagement with Maths Online, in that it suggests for many students, engagement with the resource can be quite limited. Hence, we would argue first impressions are extremely important so as to ensure students get a clear understanding of what relevant support is available and can easily locate the content they require from their very first interaction with the resource. The Access Report dataset contains a field called Total Activity which shows, for each student, how long, in decimal hours, they have interacted with Maths Online. This value is the same for every row in the Access Report dataset corresponding to that student and it is the same value as that displayed for the student within the Total Activity column of the People page in Canvas, a page that provides a tabular display of information about the users of the resource. Figure 3 shows that most students, 2661 (83%) of the 3204 students, using Maths Online used it for 30 min or less. Hence, since the window of opportunity to gain students attention appears to be quite limited, this would suggest the design of the resource is important and that the content and features we hope students will find beneficial should be clearly signposted. The Access Report dataset contains a number of fields for each record that are useful in discovering which content students are viewing, namely • Category which corresponds to the type of content which can have the following values: announcements -corresponding to the Announcements area of Canvas for Maths Online; -external_tools -corresponding to any external tools used, e.g., Numbas, within Maths Online; topics -corresponding to the Discussions area of Canvas and specific Discussion topics for Maths Online; wiki -which corresponds to specific pages created by the developers of Maths Online. • Title -the name of the content. • Views -the number of times the student viewed that type of content. Prior to conducting any detailed analysis, some further minor data cleansing was necessary as closer examination of the data revealed it included rows pertaining to content that, if retained, might provide a false impression of student content usage, i.e., a custom page that was developed to act as the Maths Online home page and various files representing embedded images on this and some other pages. Table 2 provides a summary of the content viewed by category and shows overall there have been 39161 views of Maths Online content by students. Of these views, the Home area accounts for 14131 (36%) of views which is not surprising as it is the home/landing page for the resource and is the default page displayed to any student accessing Maths Online. This is followed by the categories wiki (24%) and topics (21%) with the remaining categories (modules, external_tools, pages, announcements, grades, files, roster and syllabus) accounting for the balance of the views (15%). From an initial review of the titles within each category, it was apparent that of all the categories the most important were wiki, topics, external tools and files as the views associated with the remaining categories represented nothing more significant than views of the relevant area in Canvas. Hence, it was decided to focus on these four categories and determine the total number of views for each category by title. The results of this analysis are illustrated as a treemap diagram in Fig. 4 , where the colour and size of each rectangle is proportional to the number of views of the given of content by category and title respectively. From Fig. 4 , it can clearly be seen that for • wiki: the most frequently viewed pages corresponded to pages containing software downloads like Minitab, MAPLE and SPSS; pages corresponding to a specific module (STAT6011) which appeared problematic to many students and which contains video solutions to questions from a past exam paper; and some generic pages containing information and advice. • topics: the Discussions area, which shows what discussions are available, was viewed most frequently and this was followed by views of individual discussion threads like booking a oneto-one online appointment with an ALC Maths lecturer, asking a question and accessing Numbas. • external_tools: the Numbas LTI (Numbas Learning Tool Interoperability), a tool that runs entirely on a CIT server enabling students to access exercises created in Numbas directly from within Maths Online and saves attempt data on the server, was viewed most frequently. • files: the most frequently viewed file was a particular Excel file about the standard normal distribution. Together, Table 2 and Fig. 4 provide some interesting insights. • The importance of the design of the resource's front/home page, as it accounts for 36% of content views. This area above any other would appear to present resource developers with the best opportunity to set out the stall for the resource and direct students to the content and features they may find most useful. • In terms of custom content created by the resource developers, it was interesting to observe that the pages containing video solutions developed to questions from a past examination paper for a statistics module, STAT6011, proved relatively popular representing 3197 (8%) of the content views. However, it should be noted we can only say these pages were viewed and cannot guarantee that the videos themselves were viewed. However, this still suggests that although the development of these types of resources can be time consuming, they may be an effective way of capturing students' attention and getting them to engage with the resource. In the future, to get a clearer insight into the video content most frequently viewed within Match Online, then it might be wise to create a YouTube channel where all video content could be uploaded, as unlisted videos which could be embedded in Maths Online as necessary. This would then mean that YouTube analytics could be used, in place of Canvas analytics, to provide a more accurate picture of engagement levels with the associated video content. • The topics category, which corresponds to the discussions area of Canvas, features quite prominently representing 8411 (21%) of content views. In Maths Online, discussion topics are created by the resource developers but are organized as threaded discussions which means that students can create, edit and delete discussion topics and reply to discussion posts from other students. However, on review of the various discussion threads total participation levels in each discussion were relatively low, i.e., less than 10 in each case. In addition, when the actual discussions were reviewed it was observed that it was the resource developers who responded to student queries rather than students responding to each other. This suggests, given the large number of views the Discussions area received, that students were expecting to find something useful in this area and perhaps this is something that could be leveraged more going forward to create a more interactive type of resource. • Under the external_tools category, there is some evidence to suggest that Numbas was being used, as the Numbas LTI represented 1260 (3%) of content views which seems quite low in comparison to the video solutions to past exam papers. This suggests that this feature may not be being used as effectively as anticipated and may warrant some further attention. Perhaps, this is due to lack of awareness of the availability of such a feature among students which could be overcome by advertising this feature more widely to students on the resource home page, or signposting students to Numbas exercises by module, or perhaps aligning this feature in some way with video solutions to past exam papers. These proposals may enhance engagement by helping students navigate to the Numbas exercises most relevant to their needs more easily. In addition, it is clear that there are significant limitations to the data available in Canvas regarding Numbas in that it is not possible to directly determine which Numbas exercises are being attempted from the Canvas access data alone. However, these data are available on a CIT server as the Numbas LTI, handle attempt data and store it on this server and might be worth analysing for more detailed insights in any future work. • The navigation links on the Canvas Navigation Menu that students do not need direct access to, such as Files, Pages, Grades, People and Syllabus, should be hidden as they may only serve to distract and confuse students-currently these links represent 1732 (4%) of the content views. Initial analysis of student interaction data with Maths Online appeared to show quite high engagement levels and suggests that there is a substantial audience for this form of OMLS. However, while the engagement levels observed were encouraging, it must be acknowledged that some of this could be attributable to the automatic enrolment of students in the resource rather than to students actively seeking out OMLS for themselves. Closer examination of the interaction data provided some insight into the level of student engagement and showed most students used Maths Online for 2 h or less, over the entire data collection period, with the majority accessing the resource on two or more days. However, it was also noted that a significant number of students were found to have accessed the resource over just 1 day. This suggests that while large numbers of students do access Maths Online, the time spent using the resource can be quite low and quite a significant number of students do not return after their initial engagement. Therefore, if developers of OMLS want students to engage with their resource for longer periods of time or return more frequently, it would appear that the design of the resource is important as students need to get a good first impression. However, it should be noted that utilizing time-related metrics, such as frequency of access and the amount of time spent using the OMLS, as measures of student engagement does have certain limitations. This is because these metrics are largely dependent on the nature of the OMLS and whether its content is static, i.e., content that could perhaps be bulk downloaded by the student such as notes, or whether it is more dynamic, i.e., content that requires the student to interact with the resource such as auto-corrected exercises. Overall, while Maths Online contains both types of content, it would be considered to contain more dynamic than static content and hence it was felt that the use of such metrics were reasonable in this case. Analysis of content usage for this resource suggests a number of ways of enhancing student engagement with OMLS. Firstly, in terms of overall design of an OMLS, the importance of the resource's home page and having a streamlined navigation menu within the VLE was clearly evident from the large percentage of views recorded for these content categories. We would suggest these areas should be the focus of any resource design efforts as they present the greatest opportunity to capture students' attention. They should provide clarity about what the resource has to offer and provide clear signposting to useful content and features. Secondly, rather than creating a resource with generic content on various mathematical topics it seems that there is greater potential to engage students if the resource contains more targeted tailor-made content, such as video solutions to past exam paper questions, which students can directly relate to. Thirdly, when designing OMLS some consideration should be given to the inclusion of a discussion forum feature as this analysis indicated a significant level of interest from students, based on the percentage of views recorded, for this type of feature, albeit as observers rather than participants. Further research is necessary to determine exactly how best to leverage this type of feature to enhance student engagement and encourage greater interaction, both between lecturer and student and between students themselves. But it is clear, this feature if utilized correctly has the potential to create a very different type of resource, i.e., one that is more student-centred and that could potentially transform how students perceive and interact with the resource. Finally, the inclusion of opportunities for students to self-assess and the provision of immediate feedback and advice, such as through the use of Numbas exercises, are important within OMLS. However, the availability of this type of feature is pointless unless students are aware of it. Each of the above suggested enhancements align closely with those identified in the literature as key considerations when designing online courses, i.e., course structure, content presentation, opportunities for collaboration and interaction and timely feedback (Lister, 2014) . Future work will involve combining these findings with information gathered through student and tutor feedback to design improvements and assess impact. In addition, given the limited nature of data available within the VLE in relation to student engagement with certain types of content and student origin, other sources of data will also be leveraged, e.g., YouTube video analytics and Numbas exercise attempt data captured by the Numbas LTI and stored on the CIT server and data from CIT's SIS. This should enable more detailed analyses to be performed and provide even greater insights into student engagement. The 'mathematics problem' is set to remain for the foreseeable future and if anything could be even worse in the coming years as more and more students present with mathematics learning deficiencies due to unavoidable gaps in their learning as a result of the COVID-19 pandemic. Hence, student nonengagement with traditional MLS provision in the form of MLSC must be tackled and it would appear that OMLS has real potential. Not only does this form of support overcome some of the common issues cited by students for non-engagement with MLS such as fear of asking for help and location/time constraints, but it also has the potential to reach a far greater number of students than traditional MLS. In this paper, student interaction data with an OMLS provision at one Irish HEI was explored in some detail. This OMLS was one in which the target audience were non-specialist mathematics students who were automatically enrolled at the start of each academic year and which also provided a self-enrol option for any other student interested in the resource. Until this research was conducted, the resource developers lacked a clear understanding of the level of student engagement with the resource and whether students understood the content and features the resource offered. Our preliminary analysis of the interaction data provides some valuable insights into this form of OMLS which may prove useful to other HEI's who are considering, or are in the process of developing, such a resource. Numbas as an engagement tool for first-year Business Studies students Emergency pivot to online academic learning support: crisis or opportunity? AISHE-J All Irel Spacing effects in learning: a temporal ridgeline of optimal retention: research article An audit of Mathematics Learning Support provision on the island of Ireland in 2015. Dublin: National Forum for the Enhancement of Teaching and Learning in Higher Education Analysis using natural language processing of feedback data from two mathematics support centres The development and evolution of an advanced data management system in a Department of Mathematics and Computing/Cork Regional Technical College (1985) Report on the basic Mathematical Skills Test of first year students in Cork RTC in 1984 Using Assessment to Support Student Learning The origins, development and evaluation of mathematics support services Spaced retrieval practice increases college students' short-and long-term retention of mathematics knowledge Mathematical under-preparedness: the influence of the pre-tertiary mathematics experience on students' ability to make a successful transition to tertiary level mathematics courses in Ireland Basic mathematical skills of UCC students Obtaining and using Access Report data for an entire course Mathematics Support-past and present and, most importantly future Good Practice in the Provision of Mathematics Support Centres. Stats and OR Network: LTSN Maths After the diagnostic test-what next? MSOR Connections Predicting performance of first year engineering students and the importance of assessment tools therein Design of e-learning and online courses: a literature analysis trends in the design of e-learning and online learning Student non-engagement with mathematics learning supports The impact of the mathematics support centre on the grades of first year students at the National University of Ireland Maynooth Mathematics support centres' online presence: provision in Ireland and the UK What is learning analytics? Society for Learning Analytics Research (SoLAR Prior to joining CIT, she worked as a statistician in a variety of sectors including research, pharmaceuticals and electronics. She is currently seconded to the Teaching and Learning Unit in CIT where she is involved in and coordinates projects and research related to the enhancement of teaching, learning, assessment and student engagement She is also a qualified secondary school teacher, obtaining her Postgraduate Diploma in Education from University College Cork in 2008. She is currently coordinator of the Academic Learning Centre in CIT having previously worked as a mathematics tutor in the centre Julie Crowley received her PhD in Mathematics from University College Cork in 2010 and is a mathematics lecturer in CIT since 2004. She is secretary of the Irish Mathematics Learning Support Network, a member of the Education subcommittee of the Irish Mathematical Society and chair of the Mathematics Learning Community in CIT. She conducts research in Mathematics Education