key: cord-0914634-gha0k8ma authors: Gravano, David M.; Chakraborty, Uttara; Pesce, Isabella; Thomson, Michael title: Solutions for Shared Resource Lab Remote Quality Control and Instrument Troubleshooting during a Pandemic date: 2020-11-24 journal: Cytometry A DOI: 10.1002/cyto.a.24266 sha: 1d7cefc77d40d29a1744611dd08325979d5ea7ed doc_id: 914634 cord_uid: gha0k8ma The COVID‐19 pandemic has dramatically affected shared resource lab (SRL) staff in‐person availability at institutions globally. This article discusses the challenges of ensuring reliable instrument performance and quality data output while facility staff and external service provider on‐site presence is severely limited. Solutions revolve around the adoption of remote monitoring and troubleshooting platforms, provision of self‐service troubleshooting resources specific to facility instruments and workflows, development of an assistance contact policy, and ensuring efficiency of limited in‐person staff time. These solutions employ software and hardware tools that are already in use or readily available in the SRL community, such as remote instrument access tools, video hosting and conferencing platforms, and ISAC shared resources. © 2020 International Society for Advancement of Cytometry The ability of end-users to perform instrument QC and identify, diagnose, and correct issues is of critical importance during a pandemic. Self-service troubleshooting can be considered the first line of defense when concerns arise. Instrument vendors have also made great advances in userfriendliness of instrument operation and QC, facilitating enduser independence (3). Establishing a system within SRLs to empower end-users to assist in these roles will be advantageous as more SRLs strive to provide expanded services and throughput without a commensurate expansion of staffing (4) (5) (6) . However, SRL managers must also set clear limits on which procedures should be attempted by users and which should require staff consultation. The foundation of effective collaboration between SRL stakeholders is a solid communication strategy (7) . While face-to-face communication and demonstrations may have comprised a large portion of SRL user and staff interactions, alternative means of communication have increased in importance. SRLs should not rely on one single communication platform, but rather spread their messaging across multiple modalities including email listservs, SRL websites, physical signage, lab management software platforms, presentations, social media, and personal communication. Despite reliance on multiple means of communication, it is paramount that SRL managers ensure the information presented is current and consistent. An essential component of user independence and selfreliance is a robust training program and accessible standard operating procedures (SOPs). Considerations for user and staff training during COVID-19 is covered in a concurrent article in this issue (8) . Two important resources SRLs should develop and distribute are (1) written instrument SOPs that include QC and troubleshooting considerations and (2) video SOPs. The format and content of SOPs have been covered extensively in the ISAC SRL Best Practices publication (9) . A new resource developed by the ISAC SRL Services Committee is the SRL SOP Repository (https://archive.org/details/@isac_ srl). This repository contains SOPs on instrument operation, QC, and troubleshooting for a variety of cytometers across multiple institutions. The materials uploaded to the repository are reviewed for appropriateness by the SRL Services Committee, ensuring this resource serves as a powerful, contemporary, and high-quality template for SOP development. In addition to SOPs followed under normal operational conditions, SOPs generated during the pandemic to address issues such as reduced in-person staffing, workspace decontamination, and access and spacing restrictions must be considered and are covered in another article in this issue (10) . As a supplement to written SOPs, video SOPs of basic instrument setup, QC, and troubleshooting have proven to be extremely useful for visually oriented SRL users. Video SOPs functionally distinguish themselves from the many theoretical training videos that discuss the techniques of cytometry in abstract terms. Video SOPs should present a brief step-bystep process for instrument startup, QC, shutdown, and/or troubleshooting of commonly encountered issues. Care must be taken to ensure the video SOPs do not contradict written SOPs. Video SOPs should have similar information to written SOPs including: • Title slide or accompanied written description • Purpose • Date of establishment • Revision number and date of revision • Table of contents-included as a timestamp of content transitions • Link or reference to a written SOP While some video SOPs may be helpful to users across institutions, it is advisable for each facility to generate videos that accurately reflect each facility's instruments and SOPs. Videos should be housed on reliable and accessible web video hosting platforms or a secure local source if public viewability is not desired. YouTube serves as a good hosting platform due to its ubiquitous nature and free channel hosting ability. For instruments that are not networked, storage of video files in a readily accessible location on the workstation is also an effective option. Developing detailed SOPs is critical for self-service operation for users. However, equally important is making SOPs accessible and known to users. A major component of this will come in the training stages of instrument use where trainees should be instructed on how to locate and utilize instrument SOPs (8) . Printed, laminated SOPs may be affixed on or near each instrument to ensure ease of access when troubleshooting needs arise. Additionally, if using video SOPs, placing an easy-access link to them on the workstation computer as well as on the SRL website is recommended. SRL users should perform QC prior to use of SRL instrumentation. Each SRL must develop QC procedures for each of their instrument types guided by accepted best practices (11) (12) (13) (14) . For each instrument, clear criteria for determining if the instrument has passed QC must be communicated to users. This will entail reference to a baseline performance report often generated using fluorescent bead standards and comparison of measures such as coefficient of variation values for each channel. It is important that the result of this metric is relayed back to SRL staff in order to confirm instrument performance specifications are passing standards set by both the SRL and instrument manufacturer. User QC measurement can be accomplished by: • Ensuring the SOP and reagents for QC are available for ease of use • Providing users with metrics for determining QC pass/fail • Facilitating user submission of the QC report to the SRL staff for review and record keeping • Following a workflow that is provided by SRL Staff (Fig. 1) Each SRL must clarify which procedures and troubleshooting measures are allowable to be performed by their users, and this must be effectively communicated to them. Care must be taken when balancing the need to facilitate self-service with the risk that users may worsen issues, jeopardize the integrity of their data, or damage instrumentation. It is advisable to develop a decision tree for troubleshooting that guides users through the proper sequence of interventions and allowed self-service measures. When self-service is not appropriate, Health Insurance Portability and Accountability Act QC quality control SOP standard operating procedure SRL shared resource laboratory SRL staff must be able to support user needs through a combination of in-person and remote support. As both SRL staff and end users may be subjected to local travel restrictions, room occupancy limits, and curfews, it is imperative that clear scheduling of times when SRL staff will be on site is communicated (15) . This will assist in both scheduling of experiments for which SRL staff will be required, when instruments may need to be taken offline for hands-on troubleshooting or maintenance, and when QC can be verified in person. Required information includes: • When staff will be physically present on site • Which staff operated instruments will be available when staff are on site • Which instruments will be unavailable when staff are on site • Which instruments cannot be simultaneously utilized due to proximity The purpose of this is to allow the SRL staff to perform robust QC, maintenance as required and provide in-person assistance that has been requested by end-users. While remote support by facility staff may have been commonplace for many SRLs prior to COVID-19, the pandemic has made it an increasingly essential service. A successful SRL must be extremely responsive and be able to utilize technological aids to best serve users remotely. It is imperative that SRLs clarify to users how their remote support functions and make that information widely accessible. This information should be available to all SRL users on the SRL website and include the following information: • The hours of remote support availability and by whom • Contact information for remote support • Estimated response times • What technological aids may be required for remote support Remote support aids-Hardware While some facilities may find that telephone, text, or email are sufficient to handle most issues that arise for users when staff are remote, other problems can be more effectively solved with the aid of advanced technological support. Workflow of user-performed instrument quality control. When users are entrusted to perform instrument QC, the workflow is laid out in the SOP with a decision tree to guide the user through the QC process. In case of a QC fail, the user shall contact the SRL staff who can perform remote troubleshooting to achieve a QC pass or decide what further action to take if QC continues to fail. Webcams. Generally, workstations connected to instruments do not have integrated webcams. Therefore, to provide remote troubleshooting solutions, SRLs can invest in webcams to promote efficient visualization of system components between users and off-site SRL staff. The desirable key features of webcams for remote troubleshooting are: • High definition camera-HD or Full HD • Robust autofocus • Built-in Microphone for two-way communication • At least 2 m of cable to allow the on-site user to provide a real time visualization of the entirety of the instrument Webcams with the above-described features are usually available at low to moderate cost, but in the COVID-19 environment may be difficult to source due to increased demand and supply chain issues. This lack of availability is a potential limiting factor for SRLs that need to provide remote support. Figure 2A illustrates the use of a webcam providing a constant video feed, enabling the remote SRL staff to observe and instruct on-site personnel how to perform basic optical adjustments to a flow cytometer. Dedicated troubleshooting smart device. Another solution to visualizing cytometry hardware issues is through the use of tablets or phones. While nearly every SRL user will have a personal smartphone, there are challenges to relying on their device to connect. These challenges include: • Incompatible operating systems or installed apps • Unreliability of mobile data networks or Wi-Fi to user devices • Unfamiliarity of staff or users with each other's technology • Institutional regulations applicable to personal mobile devices When instrument troubleshooting is needed, it can already be a highly stressful and frustrating situation that should not be compounded by technological issues with the remote support tools. For this reason, each SRL is advised to develop a SOP for remote troubleshooting that utilizes workstation-connected hardware/software or a SRL-provided troubleshooting device. This will be highly dependent on institutional preferences and support options of various remote meeting platforms, as discussed below. The remote troubleshooting SOP should be extensively tested between facility staff under as diverse circumstances as possible prior to adoption with users. Below is an example of a solution to minimize software and hardware compatibility issues: • SRL provides a single dedicated mobile troubleshooting device, for example, a tablet or smartphone that is connected to your institutional Wi-Fi network • Ensure that the troubleshooting device is stored in an assigned area, always charged, and the login credentials are easily available to all users • Provide a sturdy protective case for the device to avoid accidental damage by users and that can be decontaminated per laboratory protocol • Install facility-preferred troubleshooting platform and make sure it is easy to access • Develop a remote device troubleshooting SOP and make it readily available to users in proximity to the device, on the SRL website, and/or via other communication methods Web-based meeting and sharing tools are now a ubiquitous part of life: they are used daily for personal and/or business use. Considering institutional preferences and software licenses, personal preferences, and end-user platform familiarity, there are many meeting and sharing platforms that can be chosen by SRLs. Another aspect to consider is whether the platforms work on local network or must communicate with external servers. Here we briefly discuss some of the platforms regularly utilized in SRLs with specific attention to their features in the context of remote troubleshooting and QC (Table 1) . This is not an exhaustive list, but rather a reflection of the predominant SRLutilized platforms during the COVID-19 pandemic. Generally, these platforms offer online meetings that can facilitate audiovisual communication between SRL users and staff and enable sharing of the entire desktop, certain applications, and files. These platforms also allow remote access and control of instrument workstations, which are useful features to effectively troubleshoot instrument issues. In addition, when operated through a workstation, these platforms can permit a HD camera to be hand operated by the user, enabling SRL staff to assist in troubleshooting instrument hardware issues. Most platforms work across numerous operating systems and devices including Windows, macOS, Linux, iOS, iPadOS, and Android. However, it is important to verify the compatibility of these software platforms with versions of operating systems running in instrument computers. Legacy operating systems such as Windows 7 and Windows XP may not support these platforms. Regarding security and privacy, these platforms allow secure meetings with data encryption, password, and access controls. AnyDesk. AnyDesk is a remote desktop software and the Enterprise version has the key advantage of working solely on a local network and not having to communicate with external servers. This configuration may be advantageous to SRLs that handle sensitive data. Google Meet. As a response to the COVID-19 pandemic, Google Meet advanced features such as screen sharing and remote control, that previously were accessible only to enterprise and education customers via G Suite, became available to the general public via installation of the Google Meet Chrome extension. Microsoft's cloud-based identity and access management service, Azure Active Directory. For remote desktop control in Microsoft Teams, as in Zoom, SRL users can give remote control to staff only following desktop sharing. TeamViewer. TeamViewer is a free platform (for noncommercial use) that allows full remote control of instrument workstations. TeamViewer connections are fully encrypted and this platform implements password encryption with the secure remote password (SRP) protocol. A new random password is generated for each session. However, with an account, SRL staff can consistently connect to saved partner devices without the need to re-enter session passwords. TeamViewer Pilot. TeamViewer has recently released an augmented reality application called TeamViewer Pilot. The user is able to share a video feed from their mobile device that can be annotated by both the SRL staff member and user in real time with augmented reality overlay tools, such as drawings, arrows, and text (Fig. 2B ). This can be effectively used to visualize and correct issues with instrument hardware. Webex. Webex is a platform that allows real-time HD video sharing across workstations and smart devices. It has rigorous data security and enables tools such as screen and file sharing, an interactive whiteboard which can be stored on the cloud. SRLs may find it especially useful if their institution already relies on Cisco services for their IT infrastructure. Zoom. Zoom has the benefit of being a broadly familiar platform that is user intuitive and has readily available online resources such as a help center, live and recorded trainings, and a blog. The SRL user can hand over control of a workstation to staff; however, remote control is not allowed when using an iPad or Android device. For effective troubleshooting, a meeting and sharing platform should offer not only efficient and secure video communication but also secure remote access and control. A key limitation to utilization of web-based support is the prevalence of Remote support tools for instrument troubleshooting. (A) A webcam connected to the instrument workstation allows users and SRL staff to troubleshoot instrument hardware issues without the need for additional mobile devices. (B) Applications, such as TeamViewer Pilot, on a dedicated troubleshooting device enable SRL staff to remotely diagnose and troubleshoot issues using tools such as augmented reality. This is especially relevant when instrument workstations operate on outdated operating systems or are not networked. antiquated operating systems or lack of connectivity of many cytometer workstations. Institutional IT departments should always be consulted prior to networking workstations to ensure they comply with their security policies. All remote connectivity tools used should also comply with local IT department guidelines. Due to the plethora of meeting and sharing platforms available, each SRL should choose the most appropriate one based on features, institutional preferences, and compatibility with end-users. It is important to consider that the one-on-one privacy expectation during video sessions between users and off-site staff may be difficult to enforce. Remote access and control of cytometer workstations is extremely useful. However, it must be used cautiously due to the sensitive nature of the resident data or other proprietary information. For this reason, remote access to workstations may be limited to facility staff or vendor support. SRLs recognize the importance of skilled external service providers to ensure dependable operation of their cytometry instrumentation. Whether the SRL utilizes the instrument manufacturer, third party service providers, or a combination of the two, this pandemic has necessitated the development of procedures for facilitating safe in-person service visits as well as an increased reliance on remote vendor support. Options for External Support While many SRLs have delayed nonessential services such as preventative maintenance visits, the need for emergency service has continued as SRLs continue to serve their clients. Emergency in-person service visits are continuing in most • What instrument diagnostics can be collected? • To what extent can the service provider remotely control the system? • Can software be remotely managed and updated? • Cost and Availability: • Is the remote service platform availability geographically restricted? • What are vendor technical support hours and how do they align with various time zones? • Is remote service only available to customers with active service contracts? • Is there a fee for remote service? Table 2 highlights these key considerations regarding remote service platforms from several major cytometer manufacturers. All vendors included provide secure platforms that are internationally available with the purchase of an instrument service contract. Instances of successful utilization of vendor remote support platforms have exemplified how instrument manufacturers have supported SRLs during the COVID-19 pandemic. Where allowed on-site, service engineers have successfully followed institutional requirements such as appropriate use of Personal Protective Equipment. However, travel restrictions have required some services to occur fully remotely. Highlighting the importance of support via remote, web-based troubleshooting to assist SRL operations, many instrument manufacturers have used virtual platforms such as BD Assurity Linc (BD Biosciences), BeckmanConnect (Beckman Coulter), and BriCare (BioRad). Others such as Luminex Corporation, Thermo Fisher Scientific, and Cytek Biosciences have used third party communication web tools for secure audio and video interactions during instrument troubleshooting ( Table 2 ). In a specific example, full remote support was required during the installation of a BD FACSAria III cell sorter at the National Center for Primary Care and Allergy Research at the University of Sri Jayewardenepura, Sri Lanka. This was accomplished by BD Biosciences, along with their local channel partners, using a combination of tools to facilitate the entire installation process. These tools included remote desktop control for continuous instrument monitoring and the use of a third-party platform, Help Lightning, to create a virtual interactive presence utilizing on-site mobile devices (Fig. 3A) . The two software platforms allowed a remote engineer to collaborate with on-site technicians to effectively facilitate remote instrument and workstation management, in addition to the live video feed to successfully install the instrument. A combination of different web-based tools (Microsoft Teams and BeckmanConnect) have been utilized by Beckman Coulter for instrument troubleshooting their customers' SRLs ( Fig. 3B ). Remote vendor service is important beyond a restricted travel environment, including situations where in-person QC and instrument troubleshooting are fundamental responsibilities of the SRL. Here, we propose a three-tiered system toward providing continuity of instrument performance during a pandemic. First, SRLs must empower users to know which tasks are allowed to be performed in the absence of on-site staff and provide reliable SOPs to accomplish them. Second, SRLs must provide staff support, both in-person and remotely, when self-service is not appropriate. And third, SRLs must work with external providers to ensure safe and reliable service options that leverage vendor expertise and remote service tools. During a fluid situation, SRLs must continuously evolve their strategies to address QC and troubleshooting based on their institutional guidance and conditions on the ground as the pandemic progresses. Feedback from users, staff, and other stakeholders is an important component of this process. Communication and solicitation of feedback from these groups in the form of email, surveys, and advisory committee recommendations should guide the decision making of SRL managers. The COVID-19 pandemic has forced SRLs to confront operational challenges extremely rapidly. It has made obvious the need for remote support options and has galvanized SRLs to broadly adopt new SOPs and communication technologies. While these needs existed before COVID-19, this pandemic has provided the spark for many SRLs to strengthen their operations and establish policies and procedures that will serve them well into the future. Moving academic research forward during COVID-19 A joint action in times of pandemic: The German BioImaging recommendations for operating imaging core facilities during the SARS-Cov-2 emergency Current and future challenges in cytometry: Reports of the CYTO 2018 Conference Workshops Current and future challenges in cytometry: Reports of the CYTO 2019 Conference Workshops A framework for managing core facilities within the research enterprise Metrics for success: Strategies for enabling core facility performance and assessing outcomes Operational changes in a shared resource laboratory with the use of a product lifecycle management approach: A case study SRL staff and user training in a global pandemic International Society for Advancement of Cytometry (ISAC) flow cytometry shared resource laboratory (SRL) best practices Modifying regulatory practices to create a safe and effective working environment within a shared resource lab (SRL) during a global pandemic The flow cytometry shared resource laboratory: Best practices to assure a high-quality, cost-effective partnership with biomedical research laboratories Flow cytometry method validation protocols Standardization, calibration, and control in flow cytometry A universal metric for evaluating, optimising and benchmarking the performance of a Research Technology Platform (RTP) Operational issues: Lessons learnt during the 'Initial Global Lockdown of 2020 The authors thank Gen Engelhardt Creative for assistance with graphic design. The authors further thank BD Biosciences, India and Italy and Beckman Coulter, India for providing information on their remote service platforms. All authors are ISAC Shared Resource Lab Emerging Leaders. David Gravano: Project administration; writing-original draft; writing-review and editing. Uttara Chakraborty: Writing-original draft; writing-review and editing. Isabella Pesce: Writing-original draft; writing-review and editing. Michael Thomson: Supervision; writing-original draft; writing-review and editing.