key: cord-0937585-n3frejn3
authors: Mubin, Omar; Alnajjar, Fady; Shamail, Abdullah; Shahid, Suleman; Simoff, Simeon
title: The new norm: Computer Science conferences respond to COVID-19
date: 2020-11-27
journal: Scientometrics
DOI: 10.1007/s11192-020-03788-9
sha: c6dcb14cf18f4f6952b24dd42718004167acdd53
doc_id: 937585
cord_uid: n3frejn3

The disruption from COVID-19 has been felt deeply across all walks of life. Similarly, academic conferences as one key pillar of dissemination and interaction around research and development have taken a hit. We analyse an interesting focal point as to how conferences in the area of Computer Science have reacted to this disruption with respect to their mode of offering and registration prices, and whether their response is contingent upon specific factors such as where the conference was to be hosted, its ranking, its publisher or its original scheduled date. To achieve this, we collected metadata associated with 170 conferences in the area of Computer Science and as a means of comparison; 25 Psychology conferences. We show that conferences in the area of Computer Science have demonstrated agility and resilience by progressing to an online mode due to COVID-19 (approximately 76% of Computer Science conferences moved to an online mode), many with no changes in their schedule, particularly those in North America and those with a higher ranking. Whilst registration fees have lowered by an average of 42% due to the onset of COVID-19, conferences still have to facilitate attendance on a large scale due to the logistics and costs involved. In conclusion, we discuss the implications of our findings and speculate what they mean for conferences, including those in Computer Science, in the post-COVID-19 world. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s11192-020-03788-9) contains supplementary material, which is available to authorized users.

Academic conferences are one of the primary forms of dissemination of research output and scientific work. Particularly in the discipline of Computer Science, many conferences are highly prestigious with the reputation of some on par with peer reviewed journals (Vardi 2009; Vrettas and Sanderson 2015) ; some notable conferences having acceptance rates as low as 20% (Fathalla et al. 2017) . Typically, proceedings of computer science conferences archive main track submissions as full papers normally in the range of 6 to 10 double column pages which are reviewed in their entirety and not as abstracts (Caires 2015) , unlike other fields such as Business or Design. Further, bibliometric studies have shown that Computer Science researchers publish much more in conferences as compared to journals (Franceschet 2010) . This focus has resulted in the establishment of conference rankings in Computer Science (Li et al. 2018) , for instance, the popular CORE Rankings Portal. 1 Whilst such rankings present specific perspectives on the proxies for quality of a research venue, respective conference ranks are utilised by researchers in their personal profiles against their conference publications, alongside with the acceptance rate of respective conferences. Refereed conferences have a plethora of benefits over more traditional forms of research output (Green 2017) , including the opportunity to network professionally with peers (through discussions in the main track but also through the numerous parallel tracks, as well as through the interactions at workshops and tutorials), present incremental updates to research (such as work in progress projects (Mubin et al. 2018) , which may otherwise be "light" for journal manuscripts) and position papers, and expand the professional, cultural and personal benefits of international travel. From the above, it is clear and evident that the significant benefits of conferences emerges from the opportunity to travel and interact with peers and potential collaborators. With the emergence of the SARS-CoV-2 (COVID-19) virus as a pandemic in February 2020 (Helmy et al. 2020) , conferences have been severely impacted due to several logistical constraints. International travel has been curtailed and incoming travellers to various countries have been forced to quarantine for 14 days. In the early days of the pandemic, travel bans were also in effect (Gössling et al. 2020) . Public health responses have been in place, including restrictions on social distancing and gatherings, which has directly affected academic conferences. As such, due to the organisational overhead, impracticality and the associated high risk of holding a face to face conference, most organisers were either forced to hold conferences virtually at short notice (by no means an easy task (Houston 2020)), postpone them (wait it out) or cancel them for the year. Some communities understand and acknowledge virtual conferences as "the new norm" and have already proposed guidelines, lessons learnt and experiences of organising remote academic events, meetings or conferences (Weissgerber et al. 2020; Vervoort et al. 2020) . Others have already begun to advocate for the advantages that virtual conferences provide over in person attendance (Klöwer et al. 2020) . Some literature from the medical discipline indicates that the COVID-19 responses from the community in establishing research conferences include scalable approaches such as both synchronous (Zoom, YouTube) and asynchronous mediums (Slack and the like) (Rose et al. 2020 ) as well as online simulations (Gottlieb et al. 2020 ). There has also been mention of some medical conferences offering free attendance; but no number or proportion has been given to this so far (Speirs 2020) . The impact of COVID-19 on research conferences while established and understood from the literature above, is still missing a more quantitative data-driven perspective. Further most of the investigations are based on the impact on a single conference edition (Bilas et al. 2020) , with no overview at a discipline level, including the discipline of Computer Science-the one close to the hearts of the authors. A data generated overview can inform the community on what successful actions and contingencies were taken in response to the disruption caused by the pandemic and allow organisers of future conferences to respond accordingly in similar situations. In particular, considering the discipline of Computer Science is expected to actively pursue the development of new technology, which will also include integrated virtual conference platforms. Therefore, it may not be wrong to assume that Computer Science conferences could have been more robust in their response to the pandemic. Studying how Computer Science conferences have been impacted by COVID-19 will most certainly be revealing, particularly in comparison to another discipline, which we chose in our case to be Psychology. The response of a conference to any unforeseen and drastic (natural or unnatural) event is most likely contingent upon a number of factors, pertaining mostly to its profile and attributes. For example, conferences with higher budgets are likely to be more robust in their response. We anticipate that hence the response of each (Computer Science) conference to COVID-19 would not have been the same and dependent on a number of predictors such as where the conference was to be held, when it was to be held, how much operating budget it had and the reputation or ranking of the conference. Therefore, as such we aimed to quantitatively determine how (and to what extent such as, virtual or hybrid; on schedule, cancelled or postponed, if latter by how many days) Computer Science conferences were impacted by the pandemic, what action they took in response and whether the response was determined by internal factors such as when the conference was supposed to take place, its ranking and publisher or external factors such as the host country. Further, we also aimed to understand the financial impact on conferences by determining, the revised registration costs of conferences and whether substantial discounts were offered. The response of research conferences to COVID-19 may also be contingent on their available operating budget. Conferences with a flexible budget could be more robust in response to rescheduling, transition to alternate modes and generous reduction in participant registration fees. Corporate sponsorship is one of the primary sources of income for research conferences (Pierce et al. 2020 ); therefore we also aimed to measure if the number of sponsors would promote a reduction in registration fees. Further, as mentioned prior, indexing and archival is a key component of Computer Science conferences, therefore we anticipated that conferences would still have costs emanating from publisher fees. Lastly, we also aimed to highlight our results by bench-marking them against another discipline (case in point Psychology). In summary, our investigation contributes towards Computer Science and Scientomterics literature by presenting a data-driven understanding of the multi-pronged impact of COVID-19 on Computer Science research conferences. This once in a century pandemic will most certainly change the face of research conferences for the unforeseen future or until a recognised vaccine is available; hence such an investigation is timely and essential.

To analyse the exact responses taken by Computer Science conferences in response to COVID-19, we collected meta data pertaining to these conferences. Thereafter, we analysed the data to quantitatively assess the impact of COVID-19.

In order to acquire a large breadth and pool of Computer Science conferences we collated conferences across two renowned publishers in the discipline (with their reputation and standing also evidenced from literature (Halpern and Lagoze 1999; Wallach 2011) ); Institute of Electrical and Electronics Engineers (IEEE) 2 and Association for Computing Machinery (ACM). 3 Three conference listings were identified; the conference calendar of IEEE Computer Society, 4 upcoming conferences as identified by ACM (including special interest groups like SIGGRAPH, SIGKDD), 5 and the range of conferences from ACM's special interest group SIGCHI. 6 Data related to the noted conferences was extracted during the last week of July. Further, in order to benchmark our observations with respect to Computer Science conferences, we also extracted meta level information about conferences from the discipline of Psychology from a variety of sources. 789 This was done in the first week of August 2020.

Metadata associated with the conferences was obtained by thoroughly scanning the website of each conference. For each conference the following data was manually scrapped:

1. Name of the conference. 2. Host country. 3. Region (or continent of the host country). 4. Original Starting date of the conference, recorded as month only. 5. Duration of the conference. 6. Current status of the conference; which was further split into 6 mutually exclusive sub-categories:

(a) Cancelled (b) no change in date or mode mentioned (c) no change in date but changed to a hybrid mode (where authors were given the option of presenting papers online or in person, as convenient) (d) no change in date but changed to a virtual mode (e) postponed, no change in mode mentioned (f) postponed, change to virtual mode 7. In the case a conference was postponed, we noted by how many days and whether the new date was made available. Any conference, which was cancelled, held on the 2 https ://www.ieee.org/. 3 https ://www.acm.org/. 4 https ://www.compu ter.org/confe rence s/confe rence s-impac ted-by-covid -19. 5 https ://www.acm.org/upcom ing-confe rence s. 6 https ://sigch i.org/confe rence s/upcom ing-confe rence s/. 7 https ://www.psych ology .org.au/Train ing-and-caree rs/APS-Congr ess-and-confe rence s. 8 https ://www.noldu s.com/blog/psych ology -confe rence s-2020. 9 https ://confe rence monke y.org/top/psych ology /confe rence s. originally scheduled date (in whatever format) or postponed indefinitely was assigned a missing value for this category. 8. Registration Fees (in USD using the foreign currency exchange rate as at August 16), namely:

(a) Original Author Fees (if available and clearly indicated as pre ; where this fee is intended to cover the publication of an accepted paper in the conference proceedings. (b) Revised Author Fees (if available and clearly indicated as a response to ; where this fee is intended to cover the publication of an accepted paper in the conference proceedings. 9. The number of sponsors of the conference. 10. The Publisher of the conference (ACM, IEEE, or both). 11. CORE Ranking of the conference 10 (where available). 12. Online platform used for the conference for the purposes of presenting the accepted papers.

The conference registration prices were also normalised over duration of the conference, under the assumption that conferences of a longer duration would have additional expenses. The normalised registration prices are hereon in reported in the units USD/day. All codes had a missing value or not applicable category. Two coders from the authorship team discussed the codes and independently assessed and scrapped the categories. Any disagreements in category definition were resolved by consultation. Since the data was objective, inter-rater reliability was not computed. Data was first collected in Excel and then transferred to SPSS for further analysis.

As mentioned in the previous section, we collected data for a total of 170 Computer Science conferences and 25 Psychology conferences. We discuss results obtained across both disciplines separately, first for the Computer Science conferences. We now report on the descriptive results of Computer Science conferences followed by various association checks.

The pool of 170 Computer Science conferences were observed to be spread across the entire globe with 34.7% intended to be hosted in Europe, 28.2% in Asia and 23.5% in North America. The original dates of the conference were also evenly spread out (16.5% commencing in August, 12.9% in May and September, 11.2% in July, 10.0% in June, 8.8% in November, 7.1 % in April and so on). The average conference duration was 3.84 days ( N = 167 ). 69 conferences were observed to have a CORE ranking (7.1% A*, 12.9% A, 13.5% B and 7.1% C). The remaining 101 conferences did not have a CORE ranking available.

A total of 100 Computer Science conferences (58.8%) changed to a virtual mode (with no change in date). 19 (11.2%) were postponed with no clear change in mode mentioned; 17 (10%) offered a hybrid mode (with no change in date); 13 (7.6%) postponed and changed to a virtual mode; 13 (7.6%) did not state any change in date or mode and 8 (4.7%) conferences were cancelled. We were able to compute the average postponement period as 141.2 days. This data was available for a total of 20 conferences only and as mentioned prior these 20 conferences excluded those which did not change their scheduled date, got cancelled or were postponed indefinitely. One conference was postponed (essentially rescheduled) by exactly a year. On average, registration fees pre COVID-19 were 714.1 USD for authors, normalised fees by duration of conference were on average 210 USD ( N = 63 and 60 respectively), whereas they dropped to 306.9 USD, 85.2 USD/day in response to and during COVID-19 ( N = 91 ). Student registration fees were on average 177.6 USD, 51 USD/day in response to and during COVID-19 ( N = 75 ), whilst just attending the conference virtually costed on average 142.9 USD, 42 USD/day ( N = 68 ). We were able to compute the registration difference for authors from prior and during COVID-19 and this turned out to be an average decrease of 307.82 USD, 90.75 USD/day ( N = 33 ) or approximately 42%. 8 conferences (from these 33) did not change their registration fees whereas one conference decreased their author fees by 1650 USD (which was the highest difference).

We also report on the observed frequencies of publishers. A total of 93 conferences had their (or intended to have their) proceedings archived in ACM (54.7%), followed by IEEE with 60 conferences (35.3%). 8 conferences were published (or intended to be published) both in IEEE and ACM and 9 conferences were noted to either not explicitly mention an archival repository or refer to one that was neither ACM or IEEE. The average sponsors per Computer Science conference was 4.53. 10 conferences were noted to have no sponsor and 1 conference was observed to have 48 sponsors. By studying the conference websites we also extracted their choice of presentation platform-particularly for those that had gone online. 76 conferences from the 130 identified as "virtual" in some form did not clearly specify their choice of platform (58%). From those that did, 34 conferences specified Zoom as amongst one of their choices of teleconferencing platforms. Other popular platforms were CEvent (9), Whova (6), YouTube (4) and Slack (4). A summary of the descriptive results is provided in the form of a table (see "Appendix 1").

Our study had broadly selected or identified four predictors; namely region, publisher, ranking and the originally scheduled month of the conference. Before analysing their effect on the COVID-19 response strategy of every conference, we conducted association checks amongst the predictors as a means to test independence. Primarily, we considered ranking as the major factor, given that ranking of an academic conference can be governed by a number of variables. We were not overly worried with regards to high association amongst region and other variables as the host region of a conference typically rotates every year.

To establish the independence of our predictors, Chi-Square tests of association (referred further as Chi-Square tests) were run. There was not enough evidence in the data to suggest an association between region and ranking; originally scheduled month and ranking; and publisher and ranking.

Thereafter, we performed association checks between our predictors and the response strategy of the conferences in our data. In order to compensate for the unequal groups and low sample size within our study we conducted appropriate non parametric tests studying the impact of Region, Publisher, Ranking and Original date of the conference on the status of the conference as well as its registration costs and the number of days a conference was postponed by. For the categorical variable of conference status we report on Chi-Square tests and for the continuous variables of registration and postponement period Kruskal-Wallis tests were performed. Primarily, only significant results are reported hereunder ( p < 0.05 ); see "Appendix 2" for details, bar one exception.

Association testing as evidenced from a chi-square test revealed that there was a significant association between the month the conference was originally scheduled in and the status of the conference; see Fig. 1 ). Conferences that were originally scheduled in May were more likely to be postponed (adjusted residual z = 2.7 ) as compared to the other months. Conferences that were originally scheduled in August were more likely to persist with the originally scheduled date and at the same time provide hybrid presentation options (adjusted residual z = 4.0 ), in comparison to the other months. Further association testing revealed that there was a significant relationship between region and current status of the conference. Asian conferences provided significantly more hybrid options on the originally scheduled date (adjusted residual z = 3.3 ), whereas North American conferences were significantly more likely to be held in a virtual mode on the originally scheduled dates (adjusted residual z = 3.1 ) in comparison to the other regions (see Fig. 2) . A Kruskal-Wallis test revealed that the region had a significant impact on student registration prices during and in response to COVID-19. Posthoc Fig. 1 Response of computer science conferences between February to December 2020 to the COVID-19 pandemic by originally scheduled date pairwise comparisons revealed that Asian conferences had significantly higher student registration costs during and in response to COVID-19 as compared to North America ( p = 0.02 , mean = 246.24 USD vs. 58.13 USD). A Kruskal-Wallis test also revealed that the region had a significant impact on audience or viewer registration prices in response to COVID-19. Again, posthoc pairwise comparisons revealed that Asian The significance results for this particular association are reported (see "Appendix 2") to complement the afore-mentioned figure. A Kruskal-Wallis test revealed that the type of publisher had a significant effect on pre-COVID-19 author registration prices. Posthoc comparisons illustrated that IEEE conferences had significant higher author registration costs (pre COVID-19) as compared to ACM conferences ( p < 0.001 , mean = 1142.11 USD vs. 582.97 USD). Further, a Kruskal-Wallis test revealed that the type of publisher had a significant effect on viewer registration prices in response to COVID-19. Posthoc comparisons illustrated that ACM conferences had significantly higher viewer registration costs (in response to COVID-19) as compared to IEEE conferences ( p = 0.003 , mean = 196.88 USD vs. 47.83 USD). A final chi-square test indicated that there was a significant association between the ranking of the conference and the current status in terms of it's scheduling ; see Fig. 4) . A* conferences were significantly more likely to to persist with their original dates but go virtual (adjusted residual z = 2.3 ), in comparison to the other ranked conferences. C ranked conferences were significantly more likely to be cancelled (adjusted residual z = 3.1 ), in comparison to the other ranked conferences.

We repeated 4 Kruskal-Wallis tests, one for each predictor (region, originally scheduled month, publisher and ranking) using the 4 normalised registration prices as the measurements. In general, identical trends were observed as in the afore-mentioned results for absolute registration prices. We also computed Spearman's rho correlation coefficients for the number of sponsors with the registration variables as well as the number of days a conference was postponed by. None of these were significant. Lastly, none of the attributes of the conference had an impact on the duration of postponement (which was determined through the Kruskal-Wallis tests jointly with registration fees). 

Data from the 25 Psychology conferences was extracted as a means to provide a reference point for our results obtained from the analysis of Computer Science conferences. From the pool of 25 Psychology conferences, 10 (40%) decided to change to a virtual mode (with no change in date). 9 conferences were postponed (36%) with no clear change in mode mentioned. The average postponement period was 264 days ( N = 6 ). Registration prices pre-COVID-19 for authors were on average 489 USD, 152 USD/day (N=8) and in response to 278 USD, 92/day USD ( N = 11 ) . Student registration costs in response to COVID-19 were on average 133 USD, 38 USD/day ( N = 9 ). Attending conferences virtually on average costed 219 USD, 55 USD/day ( N = 6 ). Two conferences were explicitly made free for all attendees in response to COVID-19. The average sponsor count for Psychology conferences was 1.68.

In our analysis, we have attempted to discern the impact of COVID-19 on Computer Science conferences and their associated response. Our results show that Computer Science conferences were in general quick to move to some form of a virtual mode (130 out of 170), with only a few taking the drastic steps of cancellation or postponement. Interestingly, host region, conference ranking and the month the conference was originally scheduled in all seem to have an influence on the contingency plans of the conference. Asian conferences were more likely to provide face to face options, in comparison to North American conferences; the latter were quicker to move to a fully virtual mode. This can be attributed to the prevailing state of the pandemic in the respective continents, as some countries in Eastern Asia had a more robust (and successful) response to COVID-19 (Lu et al. 2020 ). An additional factor could be the access to technological support for running full scale virtual conference, especially the capability to integrate different platforms for covering the spectrum of conference activities in a perceptually single environment. Similarly, the prestige and reputation of highly ranked conferences could have meant that there was little flexibility for them to reschedule or cancel; say in comparison to lower ranked conferences. The impact of the the scheduled month of the conference was in line with the current condition of the pandemic, which was at its worst in March and April. Conferences later in the year appeared to be more hopeful of avoiding rescheduling as well as providing in person attendance opportunities. Even though conferences had moved in droves to an online mode, we did not observe a similar transition to cheaper registration prices, particularly for the "attendance only" category, for example for Asian conferences and ACM sponsored conferences. IEEE conferences offered cheaper attendance only prices, whilst author registration prices were initially higher. The number of sponsors interestingly had no association with any of the registration prices, even though it is reasonable to assume that sponsors would provide a monetary benefit to the organisers. For those conferences who advertised a revised registration breakdown, more than one quarter did not drop their author registration fees. This can be attributed to financial limitations of host organisations who for the sake of contingency measures, insurance coverage (in COVID-19) and reduction in risk would not want to take a chance with last minute withdrawals or no-shows (even if authors were presenting online) and advocate for a surplus budget. Archiving accepted papers with publishers can be a costly affair (Van Noorden 2013) and organising committees would have been wary of this. In the near future as university budgets tighten further due to COVID-19 (Ahlburg 2020) and the demand for large scale and affordable (virtual) attendance increases, the onus will fall on the publishing companies to reduce their printing and archival costs.

A comparison of the response strategy of Computer Science and Psychology conferences indicates that conferences in the discipline of Computer Science appear to be more robust in their response and COVID-19 contingency plans. Relatively, Psychology conferences were more prone to postponement and cancellations. More than 65% of the total sample of Computer Science conferences moved online and on their originally scheduled dates. On the surface, it does appear that Psychology conferences were slower to adapt to online mode. Further investigations are required to determine if this was due to any obstacles while using or acquiring teleconferencing software and experience with virtual conference environments. It does appear that Psychology conferences had lower operating budgets as their registration costs both pre and in response to COVID-19 were in general lower than Computer Science conferences. This is also exemplified by two examples of Psychology conferences offering no registration fees whatsoever (apparently not even for authors). Psychology conferences also had significantly fewer sponsors than Computer Science conferences. However, since we cannot comment on the archival protocol in the field of Psychology and whether this attracts exorbitant costs, it will be hard to interpret the low operating costs for Psychology conferences. Financial limitations faced by Psychology conferences could have also been an hindrance in recruiting event management or teleconferencing software. For Computer Science conferences, the Zoom software emerged as the most popular (synchronous) platform of choice. Asynchronous platforms (such as Slack or Discord) were rarely mentioned amongst the Computer Science conferences that we analysed. Such platforms could serve to eradicate many logistical issues such as time zone differences which attendees would face in online conferences (Harabor and Vallati 2020) . In the conferences in the second half of the year, event management environments like Whova 11 have appeared to provide substantial integration.

At this juncture, we would also like to comment on our approach, methodology and ensuing limitations of our analysis. Our sample size whilst favourable does not encapsulate the entire spectrum of Computer Science conferences. Further, publishers like Springer were not explicitly accounted for. ACM and IEEE are the primary publishers in the area of Computer Science and while we believe they provide sufficient coverage of the field, extending the sample may be worthwhile. In addition, our scrapping methodology was entirely based on what was presented and mentioned on the conference website. Some details may have been hidden, not disclosed publicly and only provided via internal communication to registered participants. We also did not account for whether the conference had already taken place or will take place in the future; given that we scrapped data in July/ August 2020. Therefore conferences may have changed their delivery details or registration costs after we had collated the data. We did, however, record the "original" starting date of the conferences as a means to measure the "reaction time" of conferences to COVID-19. Since the primary source of our data for every conference was its website, we did not have access to other meaningful attributes, such as how many papers a conference had accepted or was intending to accept. The size of the program will most certainly have an impact on the response strategy employed by the organisers. We also did not record if individual tracks (such as posters or full papers) were delivered in specific modes. These are matters of complementary study as well as the impact of the different levels of networking and contacts with industry at the virtual environments in the conferences.

While from the presented analysis it may seem that the COVID-19 pandemic has negatively impacted conferences, it would be unfair to present the situation as a completely foregone conclusion against virtual conferences. In fact, this analysis identifies some of the underpinning factors, including and extending to virtual conferences, of the agility and resilience of conferences as specialised events for the interactive dissemination of research, with the benefit of social interactions for many attendees. The research community needs academic conferences in one form or the other. From a financial perspective, virtual conferences should be cheaper to organise as compared to face to face conferences, notwithstanding the reduction in carbon footprint (due to reduced air travel) and addressing environmental sustainability. 12 Typically, organizers move from focusing on venues, food and ushers among other things-to focusing on the selection of a suitable online platform to organize the presentations, discussion rooms, on-line industry exhibitions, poster sessions and other means for intensive networking and interaction during the conference. We also believe that virtual conferences tend to be more inclusive. Attendees do not have to worry about securing travel funds nor the tiring and uncertain process of visa applicationswhich further allows attendees from underrepresented areas to come forth and be part of the discussions, including students and early-career researchers. Such an extension has the potential to change the fabric of the discussions and the spectra of the ideas presented. Some disciplines have reported a record increase in (online) delegates over conventional editions of the conference in years gone by (Speirs 2020) . Further, virtual conferences are proposing various techniques to enhance the interactivity of their sessions, these include releasing recordings, proceedings and supplementary videos of their presentations. This allows attendees to focus on the topics they are interested in and catch up on everything else that they may have missed during the conference (due to logistical issues such as time zone differences, personal commitments, poor bandwidth, etc). With the great benefits of organising virtual conferences also comes various challenges and obstacles (Zargaran et al. 2020) . The scientific community is not used to organising virtual conferences on such a large scale and predicaments related to organisation, cyber security, lack of networking and technology have been witnessed. The uncertainty towards organisation of virtual conferences was also exemplified in our results. For those conferences ( N = 33 ) where there was a clearly advertised and revised registration structure, on average author registration costs fell by about 42%; however there were still about a quarter of the conferences ( N = 9∕33 ) who did not decrease author registration costs. As the world becomes more resilient, and develops and learns to live in this "new norm", and/or until a COVID-19 vaccine is widely available, perhaps a hybrid or a co-located approach (conference held simultaneously across several locations) for research conferences would be the most appropriate. The lessons rapidly learned in virtual and hybrid conference organisation will be essential in addressing key issues of environmental sustainability.

The disruption, caused by COVID-19 has had a profound and deep influence on the entire globe, across the economy and society in general. With extensive bans on gatherings, COVID-19 has also affected research conferences for the foreseeable future. The scientific community is fully cognizant of this change and acknowledges that organising a conference in these times is an arduous task (Misa et al. 2020 ). Therefore, we find a prevalence of guidelines introduced for virtual conferences (such as from ACM) (Wang et al. 2020 ). Our analysis has provided a glance onto the agility and resilience in the way conferences in Computer Science have responded to the pandemic both in terms of the mode of offering and registration prices. We also attempted to ascertain if the response was dependent on aspects such as where the conference was intended to be scheduled, its ranking or the publisher associated with the conference. In general, we have illustrated that Computer Science conferences were robust in their response and swiftly moved their entire programs to some form of an online mode. These alternative modes enrich the toolbox of conference organisers in terms of the range of flexibility of conference activities and scenarios. With the public health response to COVID-19 expected to persist and whilst the world waits for the availability of a vaccine, "the new norm" of virtual conferences is here to stay.

Impact of region on viewer registration prices Kruskal-Wallis 2 (N = 64, 3) = 18

Association between publisher and status of conference Chi-square 2 (N = 161, 10) = 15

Impact of publisher on author registration prices Kruskal-Wallis 2 (N = 57, 2) = 23

Impact of publisher on viewer registration prices Kruskal-Wallis 2 (N = 67, 2) = 11

Association between ranking and status of conference Chi-square 2 (N = 69, 15) = 27

Covid-19 and UK universities

The eurosys 2020 online conference: Experience and lessons learned

Again, the role of conference papers in computer science and informatics

Analysing scholarly communication metadata of computer science events

The role of conference publications in CS

Pandemics, tourism and global change: A rapid assessment of covid-19

Rethinking residency conferences in the era of covid-19

Purpose and practice of medical research meetings. Introduction to Research Methodology for Specialists and

The computing research repository: Promoting the rapid dissemination and archiving of computer science research

Organising a successful AI online conference

The covid-19 pandemic: A comprehensive review of taxonomy, genetics, epidemiology, diagnosis, treatment, and control

Lessons of covid-19: Virtual conferences

An analysis of ways to decarbonize conference travel after covid-19

The impact of conference ranking systems in computer science: A comparative regression analysis

Weathering covid-19 storm: Successful control measures of five Asian countries

Lessons learned organizing the PAM 2020 virtual conference

Tracking the follow-up of work in progress papers

Conferences in an era of expensive carbon

Physically distant, educationally connected: Interactive conferencing in the era of covid-19

Reflections on the upsurge of virtual cancer conferences during the covid-19 pandemic

The true cost of science publishing

Conferences vs. journals in computing research

Brave new world: Virtual conferencing and surgical education in the coronavirus disease

Conferences versus journals in computer science

Rebooting the CS publication process

Organizing virtual conferences through mirrors: The ACM e-energy 2020 experience

Point of view: Mitigating the impact of conference and travel cancellations on researchers' futures

Covid-19: A unique opportunity to upgrade medical conferences