key: cord-0069371-wg4tu1bg
authors: nan
title: Pathology Visions 2020: Through the Prism of Innovation
date: 2021-09-27
journal: J Pathol Inform
DOI: 10.4103/2153-3539.326643
sha: cf700f5b08f7eca120ba727afb7a302f0d64865c
doc_id: 69371
cord_uid: wg4tu1bg

nan

Michael Rivers, DPA President, kicked off the meeting with a thoughtful and stimulating opening session. In the keynote address, "Going beyond Sight, Revealing Multiplex Biology," we heard the exciting advances pioneered by Dr. Jared Burks of MD Anderson Cancer Center. A plenary presentation from Dr. Joel Saltz of Stony Brook University introduced us to "Artificial Intelligence-Driven Pathomics: The Next Diagnostic Frontier." The virtual format allowed attendees to preview, on their own time, the many presentations by distinguished speakers in both the Clinical Track and the Education and Research Track. The Q&A sessions, which were held in real time, provided the opportunity for attendees to interact and engage with speakers as well as each other. Vendor workshops and showcase Q&A sessions enhanced the dissemination of knowledge about the latest offerings on display.

Attendees from all over the world including the United States, Canada, Europe, Australia, South America, and Asia came together to hear the Chairs of four major US academic institutions who discuss "Implementing and Adopting Digital Pathology for Clinical Diagnostics" as well as an international expert panel comment on their "Perspective of DP/AI/ML and a Snapshot of the Response to the COVID-19 Pandemic."

The regulatory and standards update provided the latest information on changes in the DP field as well as the DPA's continued endeavors with the FDA. The Connectathon panel, in collaboration with Digital Imaging and Communications in Medicine, gave us hope for a more open and interactive future.

The 45 scientific posters introduced the latest advances in the field, and attendees had the opportunity to virtually meet budding talent in DP investigation. We congratulate this year's poster award winners:

Thank you to the members of this year 's Program Committee who stopped at nothing to ensure the success of this meeting, and a very special thanks to Ms. Abbey Norris, who went above and beyond the call of duty to transition this meeting to the virtual platform midway through the planning process. It is very energizing to see the resilience of the DP/AI community thriving in the face of such a challenge. The time of unleashing the power of DP/AI to benefit precision medicine has arrived. We hope you will find the information in the abstracts useful. We strive to be your best partner in DP/AI education.

Background: The Regulatory and Standards Taskforce (R&S TF) has the mission to advance digital pathology by bringing clarity to the regulatory pathway for digital pathology including its evolution and creating awareness thereof and working toward the development and adoption of standards as well as promoting interoperability in digital pathology for clinical use. Collaboration between Digital Pathology Association and Alliance for Digital Pathology: Both aim to move digital pathology forward and ultimately improve patient care. One would wonder what the differences between the two are, and we would further explain the differences and how these are complementary to each other. The R&S TF has its role in the competitive space and focuses mainly on (1) developing general principles for regulatory clearance, (2) creating clarity around regulations for digital pathology devices, and (3) aligning around standards such as Digital Imaging and Communications in Medicine (DICOM) and color, to facilitate regulatory pathways for medical devices. The alliance has its role in the precompetitive space and focuses mainly on regulatory science, with the aim to (1) account for the patient perspective by including patient advocacy, (2) investigate and develop methods and tools for the evaluation of effectiveness, safety, and quality to specify risks and benefits in the precompetitive phase, (3) help strategize the sequence of clinically meaningful deliverables, (4) encourage and streamline the development of groundtruth data sets for machine learning model development and validation, and (5) clarify regulatory pathways by investigating relevant regulatory science questions. [1] Opportunities for Digital Pathology Association and Alliance to Collaborate on Medical Device Development Tools: Medical device development tool (MDDT) can be used in the development and evaluation of medical devices. The MDDT program provides benefits such as (1) increase predictability in device evaluation and regulatory decision-making; (2) efficiency because multiple sponsors can use it for their device to show safety and effectiveness; (3) bridging the gap between R&D using regulatory science; and (4) collaboration in a noncompetitive setting where parties could collaborate to expedite development, validation, and use of an MDDT. [2] Accomplishments and Roadmap: Standardization plays a critical role in the development of artificial intelligence (AI) algorithms, such as image standards and displays. We are focusing on supporting ongoing standardization efforts and participate in integrating the healthcare enterprise and DICOM Working Group 26 (WG26) as well as the International Electrotechnical Commission WG 51 (medical image display systems, evaluation methods, and on-site testing). In addition, we actively contribute to the first Virtual Connectathon at Pathology Visions 2020 which leads to improved understanding of standards. General Principles for Verification and Validation in Modification to a Software as Medical Device based on Artificial Intelligence: Modification to a Software as Medical Device introduces a new risk that could cause harm or requires a new control measure to prevent harm requiring a new premarket submission in the USA. Standards such as DICOM as well as how specifications of the original input cleared/authorized/approved device are described could improve regulatory process. Remote Sign-Out using Whole-Slide Imaging Devices: With the current COVID-19 pandemic, CMS has issued a waiver that allows for remote sign-out of cases [3] and the FDA has provided guidance [4] that enables remote use of digital pathology systems not cleared for this type of use. The DPA has the ultimate goal to use Real-World Date (1) to support the practice of remote sign-out beyond the pandemic and (2) to change FDA's guidance "Technical Performance Assessments" to allow an open system approach. The DPA has executed a survey to describe the current landscape of WSI remote use. As a next step, a clinical study is a set-up to generate valid Real-World Evidence on the performance of remote case sign-out in the clinical setting. The data will be analyzed on concordance between the remote sign-out diagnosis and a subsequent quality control second read.

references Background: Multiplexed tissue imaging supporting a comprehensive understanding of the cellular organization is the new normal and reveals the biology of the tissue ecosystem, reveals opportunistic disease progression, and helps explain successful or failed drug treatments in creating and supporting personalized medicine. Methods: Digital pathology is a required component for these applications and opens the door to standardized quantitative methods of these tissues. This is a massive expansion to the field of pathology, the science of the causes and effects of diseases. This expansion is modernizing pathology and incorporating immunology and spatial and higher-order mathematics, all while being rooted in the world of histotechnology. Conclusions: Why is any of this important and is it worth the promise? What are the multiplexed options and how do they compare to the nonimaging alternatives? What can really be accomplished with these technologies and how complicated are they? Ultimately, what does it all mean, why is it worth the effort, and where are the pitfalls? By utilizing digital pathology data, one can choose the technology or assay that is most suited for the research question, and then, utilizing classical histotechnology methods brings these data together to answer robust and unique medical questions. The human eye limitations prohibit it from accomplishing what is possible digitally once one is aware of what is possible and understands the unique nature of cellular organization. When the registration is done by a human controlling the graphical user interface, the tedious task can be time-consuming and therefore limit the sample size required for regulatory purposes. Methods: In this study, a software tool was developed for determining the pixel-level equivalence between two viewers. Given two viewers opening the same digital slide, the tool automatically registers the image pair by a series of zooming and panning operations conducted by sending keyboard and mouse events to the graphical user interface. The differences between the two registered images are then computed for each pixel. Experiments were conducted to test three freely available third-party WSI viewers with 100 regions of interest.

The results show that some WSI viewers exhibit greater discrepancies, which originate from not only the color processing and image compression pipeline but also the tile stitching algorithm. Conclusions: The tool was demonstrated to be useful for measuring pixel-level equivalence between different viewer products and also detecting programming errors in the development phase.

Background: It has become increasingly clear that the need for huge datasets represents a bottleneck for applications of artificial intelligence, such as distinguishing between classes. Another problem is that, in most cases, there are substantially fewer annotated target lesions than normal tissues for comparison. While deep-learning algorithms have successfully tackled problems previously thought to be solely the domain of humans (thus, the term "artificial intelligence"), this dependence on massive amounts of data is one of the major ways in which computer models of the brain are deficient. Methods: The various strategies by which humans learn from small datasets are analyzed. Humans utilize large numbers of specialized neural nets arranged in both linear and parallel fashion, each solving a restricted classification problem. They rely on local error corrections such as Hebbian learning ("neurons that fire together wire together") as compared to the nonlocal back-propagation used in most implementations of artificial neural nets. They incorporate reinforcement into their learning strategy. For these reasons, toddlers are easily capable of classifying objects after one or only a few examples. Show them a banana once and they will recognize the next example of a banana. Results: Based upon the above considerations, the currently available best strategies are found to be transfer learning, zero short learning and Siamese networks, one class models, and generative networks. In addition, these strategies provide methods for dealing with weakly annotated images as well as class imbalance. It is shown that reinforcement learning can be applied to image classification by replacing loss functions with reward functions, providing additional approaches. Conclusion: It is found that neither an extensive mathematical background nor advanced programming skills are needed to make this subject accessible to pathologists with limited background in the nuts and bolts of machine learning. However, some familiarity with the basic principles of deep learning will be helpful and will be briefly reviewed. The purpose of this presentation is to enable better communication and interaction between pathologists and computer scientists, as well as to understand the limitations of machine learning, in general. These results suggest that AI can accurately assess CD8-topology on multiple tumor types while avoiding interpathologist variation from manual scoring. Future work aims to leverage this capability to more efficiently study CD8-topology and its role in treatment outcomes and mechanisms of action.

Background: The utility of artificial intelligence (AI) in pathology is expanding at staggering rates. Natural language processing (NLP) is an ideal tool to reliably and methodically review vast volumes of data. We reviewed hundreds of thousands of relevant articles and established connections and insights between image analysis (IA) and pathology using NLP/AI technologies. Methods: Using the E-Fetch PubMed article metadata tool, we identified relevant articles with 205 pathology-and AI-related key terms. Our NLP engine was tuned to recognize terms in context with inferences to automatically abstract key clinical and IA concepts from the unstructured text in over 200,000 scholarly articles from 2000 to present and to filter articles with over 50 connections between the pathology and AI terms to filter to 74,151 reviewed articles. Results: Over 200,000 published articles revealed thousands of connections between IA and pathology concepts. Visualization of these concept connections facilitated observations of at least 10 core research focuses (i.e., immunohistochemical: breast pathology; mitoses: disease progression). Conclusions: Rapid increases in the published articles make it challenging to stay "up to date" in our ever-expanding body of medical research. Using NLP it was possible to structure the text of hundreds of thousands of articles into minable data elements in seconds. The relationships between the structured data concepts allowed the authors to establish trends. For example, how often specific clinical concepts were combined in a publication like1) HER2, algorithm and breast or; 2) CAD, pulmonary and CT. Learning when clinical concepts are combined across so many publications would not be reasonable with manual review. Furthermore, the analysis exposed existing opportunities to further apply AI to underevaluated pathological topics. We share our findings to demonstrate the progress of AI in pathology and the understudied areas, which could be the focus for on-going work.

Background: Augmented reality microscopy (ARM) is a new technology that converts the conventional light microscope into a platform for digital pathology applications while avoiding the need first to photograph or digitize slides. ARM allows a pathologist to view a glass slide through the eyepiece of the microscope and simultaneously view computergenerated output, such as annotations, morphometrics, image analysis, and artificial intelligence algorithms, being projected as an overlay directly on the optical view of the glass slide. ARM further enables the functionality of pathology informatics by potentially collecting previously unattainable metadata from the microscope stage. Such metadata include objective magnification, correlating the slide label with the histopathology image, and tracking the XY movement of the glass slide on the stage. Methods: ARM was the subject of several recent studies including its use as a tool for real-time telepathology, scoring of Ki-67 neuroendocrine tumors with QuPath algorithms, quantification of nonalcoholic fatty liver disease utilizing deep-learning AI algorithms, and tracking movement of the slide on the stage during routine pathology workflow to determine best pathology practices. Results: As opposed to conventional digital pathology technologies such as whole-slide imaging, ARM is quicker to use, provides for the real-time and seamless integration of algorithms without the need to first acquire digital images, may be cheaper than high-capacity whole-slide scanners, and does Journal of Pathology Informatics S8 not require special technical skills to operate. Conclusion: The microscope is still extensively used by pathologists and will likely maintain certain optical and workflow advantages over screen-based diagnosis. ARM can therefore fill the role of a bridging technology, which both maintains and digitizes microscope-based workflow. Looking to the future, ARM is a technology that can allow pathology laboratories to utilize both microscopy and WSI in the training and deployment of AI algorithms, synchronize diagnostic data with the laboratory information system, and automate additional mundane tasks within existing pathology workflow.

Yair Rivenson 1,2 , W. Dean Wallace 3 , Aydogan Ozcan 1,4,5,6

Background: The emergence of digital pathology has created new possibilities of computer-aided diagnoses. However, digitizing pathology still has high but not directly reimbursable infrastructural costs. We wish to discuss the role that novel technological advancements in the field of virtual staining can play in catalyzing the adoption of digital pathology by disrupting the histological staining process, a mostly overlooked, yet critical, part of the histopathology workflow. Methods: Virtual staining uses deep learning to digitally stain a label-free image of a tissue section. The label-free tissue section is imaged with a contrast generating microscopic imaging technique, such as autofluorescence, quantitative phase imaging, and multispectral or bright-field imaging. The framework relies on availability of accurately registered label-free (inputs) and histologically stained images (labels) and uses a generative adversarial network to facilitate the training of a deep neural network. Results: Virtual staining from label-free tissue was demonstrated in blind studies to be equivalent of quality (or sometimes subjectively better) to standard histological staining techniques. In a second study, the same conclusion was demonstrated for diagnosing purposes by "blinded" board-certified pathologists. Conclusions: Virtual staining has many advantages, such as tissue preservation, enabling multiple (virtual or physical and virtual) stains on the same tissue section, and stain color standardization, thus eliminating intra-and inter-laboratory staining variability. Other than direct benefits to patients, the transition from chemical staining will substantially reduce histology laboratory operating costs, staff exposure to chemicals, and supply chain dependence.

Background: Deep-learning methods are having a profound impact on the performance efficiency and reliability of digital pathology analytic methods. The generation and use of whole-slide images (WSIs) are becoming routine and inexpensive, where we can feasibly expect that almost all biopsy and excision specimens from cancer patients will be digitized and virtually available as WSIs in the near future. Many clinical and epidemiologic studies are extensively using WSIs alongside the greater acceptance and adoption of digital pathology in clinical settings. By leveraging advances in computational power, speed, and the development of open-source convolutional network architectures, it is now possible to reliably analyze complex WSIs of cancer to characterize areas containing tumor-infiltrating lymphocytes, identify regions of different types and subtypes of tumors, and segment and classify many kinds of cells. These analytic methods are increasingly being adopted to provide nuanced information about cancer phenotype through quantitative pathology, leading to the emergence of Pathomics for clinical research and surveillance studies. Methods: We have developed a variety of open-source Pathomics tissue analytics models for WSIs that perform (1) quantification of tumorinfiltrating lymphocytes for multiple tumor types, (2) nuclear segmentation for multiple tumor types, (3) segmentation of breast, prostate, nonsmall cell lung, and pancreatic cancer, and (4) segmentation and subclassification of histologic subtypes of prostate and nonsmall cell lung cancer adenocarcinoma. We have also generated a variety of related data products. [1] [2] [3] [4] [5] Results: Programs, models, and generated data products for our published and formally validated models (e.g., tumor infiltrating lymphocyte analysis, breast cancer tumor segmentation, and nuclear segmentation for multiple tumor types) can be found on Github https://github.com/SBU-BMI/ histopathology_analysis. Additional programs, models, and datasets will be added to this site after being published and validated. Conclusions: Pathomics analyses will play increasingly pivotal roles in clinical research, epidemiologic, and surveillance studies. Publicly accessible open-source code and models capable of carrying out core Pathomics tasks have the potential to dramatically advance our understanding of human pathobiology on a large scale alongside the increasing availability of WSIs. As methods continue to emerge, validated data products can be used to refine and develop more advanced methods and ensemble models. In this abstract, we describe available open-source code and deep-learning computer vision models to perform fundamentally important tasks with Pathomics. Deep embedding, or feature vectors, provided by pretrained deep artificial neural networks, has become a dominant source for image representation in digital pathology. Their contribution to the performance of image analysis can be improved through fine-tuning. One might even train a deep network from scratch with the histopathological images, a highly desirable option that is generally impeded in pathology by lack of labeled images and the computational expense. We propose KimiaNet that employs the topology of the DenseNet with four dense blocks, fine-tuned, and trained with histopathological images in different configurations. We used more than 240,000 image patches with 1000 × 1000 pixels acquired at ×20 through our proposed "high-cellularity mosaic" approach to enable the usage of weak labels of 7126 whole-slide images of formalin-fixed paraffin-embedded human pathology samples publicly available through The Cancer Genome Atlas (TCGA) repository. We tested KimiaNet using three public datasets, namely TCGA, endometrial cancer images, and colorectal cancer images, by evaluating the performance of search and classification when corresponding features of different networks are used for image representation.

Background: Whole-slide imaging (WSI) provides remote viewing capabilities and enables use of image analysis tools, but adds cost and work, delays access to diagnostic information, introduces image artifacts, and does not address shortcomings of traditional processing. Interest in "slidefree" histology techniques stems from standard histological limitations: slow, labor-intensive processes, associated with artifacts, process errors, and requiring technical skill that is increasingly in short supply. However, the general perception is that "slide-free" technologies are prone to being slow or costly and critically produce inferior image quality, necessitating traditional processing for definitive diagnosis. Journal of Pathology Informatics S10 Trust, Leeds, UK, 2 University of Leeds, Leeds, UK. E-mail: bethany.williams2@nhs.net

The COVID-19 pandemic has forced pathologists and laboratory professionals to consider working in new ways to meet workforce challenges. In this session, I outline and discuss the national guidance on emergency remote digital reporting published by the Royal College of Pathologists in response to the pandemic. This includes an approach to risk assessment of the home reporting of digital slides and a number of practical resources, including a point-of-use quality assurance test for display screens. In this presentation, the benefits of the digital pathology (DP) workflow will be highlighted: lean approach, barcoding, and tracking are the words that pathologists should learn to implement the digital workflow. The possibility for remote working using secure VPN connection allowed us to manage cases and render diagnosis even during the stressing COVID period. The post-COVID DP workflow is characterized by the possibility to access to cases from everywhere and at any time with the possibility of using artificial intelligence tool. Suggestions how to implement a DP workflow, guidelines, and regulatory problems will be addressed.

The following four topics were discussed at the meeting. First, the Japanese Society of Digital Pathology (JSDP) has 19 years of history. The organization is composed of pathologists, data scientists, IT engineers, vendors, pathology technicians, and other academic members who are interested in digital pathology. JSDP serves as a companion organization of the Japanese Society of Pathology (JSP), and all pathology members belong to JSP. During JSDP's history, we have published several white papers and a guideline draft in the area of telepathology, digital pathology, and artificial intelligence (AI). The second topic was the current status of digital pathology under the COVID-19 pandemic in Japan. The effect of COVID-19 in Japan has been moderate compared to US/EU countries. In the big cities such as capital Tokyo, the effect of a case number for both histology and cytology was large, and there was a significant decrease of cases from early March to late June; however, rural area has not been affected much and remote diagnosis of the rural area has not to change case volume compared to the last year. The third topic was the current status of AI in pathology in Japan. Currently, to establish pathology -AI, there are four major groups such as the Cabinet Office of the Japanese Government, JSP, RIKEN, and NEDO. They aim to create AI-driven hospitals, databases and platforms, explainable models, and collaborative models between human and AI models. As a use case in Japan, a digital pathology network initiated by Nagasaki University was introduced. The successful creation of diagnostic and educational networks among 15 institutes throughout Japan was introduced, and it was emphasized that education has been the key to success. AI has been implemented into clinical practice in that network for the scaling of tumor cellularity and the detection of mycobacteria. Background: The advent of whole-slide imaging (WSI) scanners in recent years has propelled digital pathology to a new level and helped pathologists store, view, and share slides more easily and efficiently. Henry Ford Health System's Pathology and Lab Medicine Department has been a pioneer in this realm and is continually integrating more digital pathology tools as part of its lean culture. The Department successfully utilized digital pathology and WSI in multidisciplinary tumor boards, off-site intraoperative and intradepartmental consultations, and performance improvement program presentations, researches, and didactics such as digital gross and unknown conferences. This study investigates how WSI remote access impacts pathology residents' onsite work hours, time management, and well-being. Methods: We surveyed all residents (n = 14) before granting them VPN enabled remote access to WSI, and we followed up with another survey 1 year later. Results: Initial survey showed that 100% of residents used onsite WSI. All residents predicted having remote access to be beneficial, as 79% of them stayed after hours to review slides for unknown conferences (79%) and tumor boards' presentations (64%).

One-year follow-up survey showed that 57% of the residents (n = 8) knew how to scan slides and 64% found that knowledge helpful. Over the past year, 71% (n = 10) accessed WSI remotely and 69% found it helpful. The remote access was for reviewing performance improvement program, unknowns, and tumor boards' slides and research. When asked about the process, 21% felt that it was complicated [main issues in Figure 1 ] and 57% felt that additional training and incorporation of WSI in daily activities would be helpful. Conclusion: Granting remote access to WSI improves residents' time management and wellbeing by reducing onsite time spent for reviewing and assigning scanned slides. Constant training is needed to harness those benefits. Based on our resident group experience, we propose to have a written protocol with pictures to provide residents with step-by-step training on how to scan and assign scanned slides; access them remotely and troubleshoot issues; improve quality of images, storage space, and duration to encourage residents to use this technology to build a virtual library; and provide additional opportunities for WSI integration

Background: Tall cell variant (TCV) is an aggressive subtype of papillary thyroid carcinoma (PTC). In 1976, TCV was 

Journal of Pathology Informatics S12 defined as PTC predominantly composed of cells whose height is at least twice their width, called tall cells. [1] The inherent subjectivity of TCV diagnoses and inter-observer variability allow deep-learning methods to classify TCV objectively. In this work, we develop a high-level TCV classifier as an initial step to provide an objective TCV diagnosis and enable robust measurement of cell morphology. Methods: We utilized eight TCV and eight classical PTC hematoxylin and eosin-stained whole-slide images (WSIs) to build a classifier model using deep neural networks (DNNs) inspired by Resnet50. [2] This model focuses on tiles that are relevant to TCV morphology features by automatically evaluating individual tall cells. We can associate the percentage of tall cells to tumor regions to predict the case label for a given slide. Results: The model is successfully able to predict tumor versus normal at ×10 with 92% accuracy and tall cell versus not tall cell features at ×40 with 75% accuracy. Further assessment was performed on high-magnification tiles to extract morphological features of individual tall cells. These features discriminate tiles with tall cells from other cells [ Figure 1 ]. Conclusions: We propose a framework that employs a combination of a DNN classifier and automatic morphological assessment to create a potential morphological signature for TCV. The success of this model potentially provides a more objective method for an interpretable classification of TCV. references 1 Background: National Pathology Imaging Co-operative (NPIC) is an NHS program for the development and evaluation of digital pathology and the use of artificial intelligence (AI) to speed up the diagnosis of diseases such as cancer. The program is led by Leeds Teaching Hospitals NHS Trust and includes a network of nine NHS hospitals, seven universities, and ten industry partners including Leica, Roche, and Sectra. NPIC is part of a UKwide network of Digital Pathology and Imaging AI Centres of Excellence, supported by £37 m of government funding and £11 m industry contribution. Methods: The program will consider the end-to-end process of AI development including deployment of digital pathology, data ethics, data sharing practices, patient and public engagement activities, interoperability and standards (Digital Imaging and Communications in Medicine [DICOM]), quality assurance, clinical validation, and clinical exemplars for the real-world evaluation of digital pathology. Results: NPIC has established an infrastructure in the North of England, created over 25 new jobs and working with over a number of new partners. The program is deploying digital pathology across NHS laboratories in the North of England; at Leeds Teaching Hospitals, 100% of glass slides are scanned and available for primary diagnosis with 1400 glass slides and 1 TB image data per day. The new generation of Leica GT450 scanners has been installed. Finally, progress has been made with regionwide capturing of COSD synoptic cancer reports (mTuitive), National Pathology Exchange system, and migration to regional Sectra PACS solution. A further five sites will go-live by 2021 and remaining sites will be fully digitized by the end of 2023.

A number of publications to highlight the benefits of digital pathology are available, and further works in the evaluation of quality assurance (QA) tools-scanner calibration tools, QA display tools, stain quantification, DICOM conformance specification, and testing are in progress. Recruitment of an expert PPI panel, a literature review of clinical adoption of AI, and a project initiation for clinical exemplar in lung, breast, and skin cancer have commenced. A new international training center will be opened by 2021. NPIC at scale will see a rapid deployment of scanners across over 40 hospitals, including all 29 pathology networks in the whole country, fully digitizing cases over a population of 6 million people, scanning over 2.4 million images/3 petabytes of data per year in a single-vendor neutral archive. Conclusions: NPIC builds on the existing fully digital laboratory at Leeds and will create a globally leading infrastructure for digital pathology and AI. The AI technologies, being developed through the training of algorithms on diagnostic data, will provide further patient benefits, such as streamlining the diagnostic process and providing a platform for further research and innovation.

Identifying which individuals will develop tuberculosis (TB) remains a problem due to limited animal models and computational approaches. To meet these needs, we show that diversity-outbred (DO) mice reflect human-like genetic diversity and develop human-like lung granulomas when infected with Mycobacterium tuberculosis (M.tb). This led us to apply deep learning to identify supersusceptibility from hematoxylin and eosin lung sections using only clinical outcomes (supersusceptible or not-supersusceptible) as class labels. We developed, validated, and tested a multiple instance learning model that diagnosed supersusceptibility with high accuracy (91.50% ± 4.68%) similar to the results by two expert pathologists (94.95% and 94.58%). Two boardcertified veterinary pathologists (GB) examined the imaging biomarker and determined the model was making decisions using karyorrhectic and pyknotic nuclear debris. Finally, the imaging biomarker was quantified in lung sections from >400 DO mice and data input for genetic mapping. This identified unique locations on the genome significantly associated with the imaging biomarker, i.e., granuloma necrosis. Preliminary inspection of the gene regions suggests that granuloma necrosis may be controlled by genes in the region of the mouse major histocompatibility locus. Overall, these findings provide a novel means to convert visual patterns into data suitable for rigorous statistics. This is a major step forward for TB research as granuloma necrosis cannot be accurately quantified by pathologists, and there are no validated or consensus surrogate biomarkers of granuloma necrosis that can be substituted for this important morphological tissue change. There is an urgent need for widespread cancer diagnosis in lowresource settings, especially in contrast to areas with developed healthcare systems. According to a study in The Lancet, in the U.S., there is one pathologist for every 20,000 individuals, while in Sub-Saharan Africa, there is only one for every million. In addition, current telepathology system for cancer diagnosis mostly relies on pathologists performing remotely, which is low-throughput and requires more time and resources. With the growth of telepathology, remote diagnosis becomes a viable solution to address the lack of skilled pathologists in developing regions. Here, we present a cost-efficient device that incorporates embedded deep learning to achieve realtime, point-of-care diagnosis of whole pathological slides. We achieve this with a low-cost, three-dimensional printable microscope that uses the Raspberry Pi camera module to capture high-resolution images of slides up to ×200. Then, using a weakly-supervised deep-learning model run on the inexpensive NVIDIA Jetson Nano, the device is able to accurately classify the whole slide without any pixel-level annotations. Furthermore, the model's attention-based approach to diagnosis allows us to generate human-interpretable heatmaps displaying the regions the most influential to the model's diagnosis.

Our device also incorporates a touch screen and batteries to increase accessibility as an easy-to-use and low maintenance device while still maintaining an efficient runtime. Overall, we demonstrate that the device is capable of achieving accurate, high-throughput, and interpretable cancer diagnoses in lowresource settings.

Robin Dietz 1 , Jianyu Rao 1

Background: Immunohistochemical staining for p16 protein is frequently used in cervical biopsies. Block-positive p16 staining suggests prior infection with human papillomavirus (HPV) and precancerous potential. We sought to train a deeplearning algorithm to predict p16 staining in cervical biopsies using hematoxylin and eosin (H and E) tiles extracted from digital whole-slide images. Methods: 44 slides were scanned at ×40 magnification using an Aperio AT2 scanner. H and E and p16 whole-slide images were aligned using QuPath software. [1] Epithelial regions on the H and E slides were correspondingly annotated as positive or negative with p16 immunostaining as ground-truth. Thirty-eight slides (73,817 image tiles) were used in the training and validation sets of a 10-layer neural network run for 50 epochs using GPUs in Google Colab. [2] Six slides (11,217 tiles) were held out as a test set on original images [ Figure 1 ]. Results: The model achieved 95.7% accuracy in the training set and 93.7% in the validation set. The testing set with six original images achieved an area under receiver operating characteristic of 0.75 [ Figure 2 ]. The optimal threshold value was very low at 0.01 (out of 1 Background: The placenta is the first organ to form and performs the functions of the lung, gut, kidney, and endocrine systems. Abnormalities in the placenta are the cause of or reflect most abnormalities in gestation and can have lifelong consequences for the mother and infant. Placental villi undergo a complex, but reproducible sequence of maturation across the first trimester. Abnormalities of villous maturation are a feature of gestational diabetes and preeclampsia, among others, but show significant interobserver variability. Methods: Machine learning has emerged as a powerful tool for research in pathology. To overcome and take advantage of the heterogeneity within the placenta, we developed GestaltNet which emulates human attention to high-yield areas and aggregation across regions. We used this network to estimate the gestational age (GA) of scanned placental slides and compared it to a similar network lacking the attention and aggregation functions.

In the test set, GestaltNet showed a higher r 2 (0.9444 vs. 0.9220) than the usual network. The mean average error (MAE) between the estimated and actual GA was also better in the GestaltNet (1.0847 vs. 1.4505 weeks). On whole-slide images, we found the attention sub-network discriminates areas of terminal villi from other placental structures. Using this behavior, we estimated GA for 36 whole slides not previously seen by the model. In this task, similar to that faced by human pathologists, the model showed an r 2 of 0.8859 with an MAE of 1.3671 weeks. Conclusions: We show that villous maturation is machine recognizable. Machine-estimated GA is useful when GA is unknown or to study the abnormalities of villous maturation, including those in gestational diabetes or preeclampsia. GestaltNet, by incorporating human-like behaviors of attention and aggregation, points toward a future of truly whole-slide digital pathology.

Background: There is increasing interest in merging molecular "omic" data (e.g., DNA mutations) with histopathological findings. Whole-slide imaging (WSI) systems enable such integrative studies. However, commercial WSI solutions are highly expensive. We custom-built a WSI station in our laboratory using low-cost hardware to enable an endto-end digital pathology (DP) workflow 

The mitotic index is one of the main components for determining the histoprognostic of cancer in many organs, namely breast cancer and peritoneal mesothelioma considered in this study. The conventional method of manually counting mitosis in histological images is tedious and subject to inter-and intra-observer variability. The automatic detection of mitosis is a promising solution but presents several algorithmic and numerical challenges, mainly because of the colorimetric, morphological, and structural variability of these cells. This variability is intrinsic to the biological development of mitosis. Each mitotic phase (prophase, metaphase, anaphase, and telophase) gives rise to a different shape of the nuclei. For example, in telophase, a mitotic cell has two distinct nuclei completely separated but should be counted as single mitosis. To this, structural variability is added the colorimetric variability introduced by the image acquisition process, including sample cutting, staining, and digitalization. Materials and Methods: We propose a high-precision method using two consecutive neural networks, the first aims at segmenting all cells in the image and the second step aims at sorting cells into two categories -mitosis/nonmitosis. The analysis is therefore performed in two steps. Cell Segmentation Step: First, we apply a stain normalization method [1] to reduce staining variability between slides. We then use a particular neural network architecture called U-net [2] for cell segmentation.

We use a U-net version inspired from the article [3] for The evaluation of this model on 20% of the collected dataset gives us very encouraging results, i.e., a receiver operating characteristic-area under the curve close to 0.98. Testing on a larger basis with images from multiple medical centers is required to verify the model performance 11. Discussion/ Conclusion: The next step in this study is to evaluate this approach on a larger dataset from different centers to verify its generalization to new data and other organs. Our future work will also be devoted to evaluating the use of such an algorithm in a medical routine.

references Background: Incorporation of digital teaching materials into residency training programs has followed a slow adoption curve expected for many new technologies over the past two decades. The COVID-19 pandemic dramatically shifted the paradigm for many resident teaching modalities as teaching institutions instituted social-distancing measures to prevent spread of the novel coronavirus. The impact of this shift on pathology trainee education has not been well studied. Methods: We conducted an online survey of pathology trainees, program directors, and faculty to assess pre-and post-COVID-19 use of, and response to, various digital pathology modalities. Responses were solicited through both social media and directed appeals. Results: A total of 261 respondents (112 faculty, 52 program directors, and 97 trainees) reported a dramatic and significant increase in the use of digital pathology-related education tools. A significant majority of faculty and program directors agreed that this shift had adversely affected the quality (59% and 62%, respectively) and effectiveness (66%) of their teaching. This perception was similar among learners relative to the impact on quality (59%) and effectiveness (64%) of learning. A majority of respondents (70%-92%) anticipate that their use of digital pathology education tools will increase further or remain the same post-COVID.

The global COVID-19 pandemic created a unique opportunity and challenge for pathology training programs. Digital pathology resources were accordingly readily adopted to continue supporting educational activities. The learning curve and utilization of this technology were perceived to impair the quality and effectiveness of teaching and learning. Since the use of digital tools appears poised to continue to grow post-COVID-19, challenges due to impaired quality and effectiveness for teaching and learning pathology will need to be addressed. 

Background: Mature B-cell neoplasms are common hematopoietic malignancies, but detailed molecular characteristics have not been fully understood. Next-generation sequencing (NGS) has become more widely used to characterize the molecular genetics of hematopoietic malignancies. Here, we present a bioinformatic platform to integrate NGS assay data to characterize copy number alteration and mutational signatures of mature B-cell neoplasms. Methods: Fifty-five samples from 53 patients were sequenced by an NGS system (Illumina Inc., CA, USA). Thirty of 55 samples were chronic lymphocytic leukemia/small lymphocytic lymphoma, 8 diffuse large B-cell lymphoma, 5 mantle cell lymphoma, and 5 plasma cell myeloma. The NGS data with 1425-gene assay capacity were processed and integrated by an in-house custom bioinformatic pipeline that combines eight genomic programs and allows visualization and analysis through cBioPortal platform (MSKCC, NY, USA). For each tumor, mutation spectrum was compared to copy number alteration if present. Thirty mutational signatures (data available in 45 samples), each of which represents a unique pattern of Watson-Crick base pair change and attributes to a specific biological origin, were analyzed by MuSiCa-R application (GPtoCRC Research Group, Spain). Results: There were 226 significantly mutated genes identified in 32 of 55 samples (range: 1-75 per sample). The most common mutant genes included TP53 (14.6%), CHEK2 (12.7%), MDC1 (10.9%), ATM (9.1%), MSH3 (9.1%), and KMT2D (9.1%). Copy number gain was present in 182 genes, while copy number loss was present in 155 genes. TP53 mutation was frequently associated with copy number alteration (6 of 8 mutant genes), while none of the seven CHEK2 and six MDC1 mutant genes were accompanied by any copy number alteration. The most frequent mutational signatures included #3, #1, and #15, which were attributed to proposed etiologies of BRCA1/2, age, and defective DNA mismatch repair, respectively. Conclusions: An in-house custom bioinformatics pipeline has been established to characterize genomic features of mature B-cell neoplasms and can be potentially applied to other tumor types. Interestingly, mutational signatures of mature B-cell neoplasm have been found to be associated with BRCA pathway, suggesting that B-cells may share similar pathogenesis of DNA double-strand break repair as is found more commonly in other cancer types. Pathologists may be prone to interobserver variations in mitoses counts due to subjectivity and accepting this while making an AI model can help in setting realistic goals for production-ready models. In our future studies, we aim to increase the scale in terms of the numbers of training and testing data.

Background: Color calibration is complex. The method by which whole-slide imager (WSI) systems observe color is different from the way the human eye does; thus, images need to be processed to interpret these differences. The standard method of determining the difference between color truth and the WSI image is imaging a calibrated color slide and comparing spectral variations via CIE ΔE standards. Further complexity is added to this process when the WSI manufacturer must determine what colors the scanner requires for adequate calibration. The next issue is to determine how much variation is acceptable for the human eye and computer algorithms. The construction of the calibration slide must also be considered. Methods: Hologic's digital cytology system utilizes the Sierra slide to establish the truth for color calibration. The WSI instrument scans the Sierra slide, and the WSI variation (ΔE) is calculated by comparison to the Sierra spectral data. With the difference quantified, a specific instrument ICC profile can be generated. The ICC profile adjusts the WSI scanner digital image to ground truth color. Results: Results indicate that the Hologic's digital imager (WSI) when using Sierra slide reduces color variation to less than 2 ΔE for intended color range. When the digital image is passed into either a review station or an algorithm platform, the color fidelity is maintained. Conclusions: WSI systems seeking usage for primary diagnosis require color calibration. The Sierra slide provides acceptable color calibration for the Hologic's cytology WSI system. The amount of spectral variation (ΔE) acceptable for WSI is subjective based on many factors, with most literature citing ΔE of 5 or less. Calibration minimizes system to system variability, producing repeatable color output regardless of system age or optical component variation. Background: In the field of histopathology, pathologists diagnose patients by assessing imaged tissue sections. Even with the pathologists' trained eye, there is a significant risk for misdiagnosis. For decades, the hematoxylin and eosin (H and E) stain has been a standard way to visualize the morphology of cells. A common way to specifically detect proteins of interest is the chromogenic diaminobenzidine (DAB) labeling combined with a counterstain to visualize the cell nuclei. However, this method suffers from a narrow dynamic range, problems with quantification, and difficulties with multiplexing and colocalization. Techniques based on immunofluorescence offer the possibility for a quantitative readout but suffer from photobleaching, background fluorescence, and broad excitation-emission bands, leading to optical crosstalk in multiplexed applications. Here, we present a technology based on upconversion nanoparticles (UCNPs) allowing us to overcome major problems associated with commonly used imaging techniques. UCNPs are excited in the near-infrared and emit photons with higher energy (photon upconversion). This process drastically reduces the measurement background by removing any autofluorescence of the matrix and minimizing light scattering. Together with a remarkable photostability, UCNPs are also a perfect candidate for applications needed for long exposure times. Methods: Novel luminescent UCNPs combined with a prototype imaging device were used to detect selected markers, e.g., Her2, in the human tissue. Formalin-fixed paraffin-embedded breast cancer cell line and human breast cancer tissue were sectioned and labeled using an autostainer. The UCNP-based upconversion imaging of the human tissue sections was compared with standard DAB-based labeling. To improve the scanning speed, pulsed laser excitation and time-gated detection were explored. In addition, the combination of UCNP labeling with H and E co-staining and co-imaging was investigated. Results: Images obtained with our novel device proved that in-house-synthesized antibody-UCNP conjugates are an excellent label for the detection of cancer markers in human tissue sections. Our data show that UCNPs do not alter the bright-field images of the counterstain and hence do not interfere with standard tissue evaluation by a pathologist. In addition, bright-field and luminescent images can be merged to provide a better understanding of tissue morphology.

The emerging field of UCNP-based labeling techniques offers new possibilities for more accurate diagnosis by combining the advantage of H and E staining with specific labeling of a marker of interest. The high-contrast images of the UCNP labeling -generated by our scanning device -set the foundation for generating ground truth for machine learning Introduction: New automated image analysis methods are continuously being developed to improve objectivity and efficiency and to address the error rate associated with human medical image interpretation. Here, the performance of a supervised convolutional neural network-based algorithm pretrained to classify 40 skin tumor types is evaluated via comparison to diagnoses issued by two senior board-certified pathologists not involved in algorithm creation in classifying six representational diagnostic entities in dermatopathology. Hypothesis: Deep-learning multiclass classification algorithms can reach clinical-grade accuracy in differential diagnostics within histopathology in real-world pathology environment. Methods: Hematoxylin and eosin-stained glass slides for 300 biopsy cases (punch, shave, and excisional biopsies) of face, neck, back, and arms; 87% of the cases were from Scottlab Pathology, Ltd, from Caucasian population, 9% of the cases were from Tanzania, and 4% of the cases were from Rwanda, from African population. The slides had notable variability in quality of staining and contained artifacts such as folds and ink markings. The cases had confirmed diagnoses of either basal cell carcinoma (BCC) (34%), nodular melanoma (20%), lentigo maligna melanoma (6%), superficial spreading melanoma (5%), dysplastic nevus (3%), intradermal nevus (26%), or compound nevus (6%), initially diagnosed by light microscopy and confirmed by board-certified pathologists. Glass slides were scanned using high-and low-resolution scanners set at ×40 magnification. The scanned whole-slide imagers (WSIs) were then independently interpreted by the Mechanomind image recognition algorithm at the University of Chicago, Ingalls Memorial Hospital Medical campus, and classified into one of three diagnostic classes: BCC, melanoma, or nevus. Each WSI took 10-20 s to analyze. Two senior board-certified pathologists served as the primary evaluators.

The sensitivity of the image recognition algorithm was 97.8% for the 91 melanoma cases, 100% for the 107 nevus cases, and 99.0% for the 102 BCC cases. The corresponding specificities were 97.5% for the melanoma and nevus cases and 100% for BCC cases. One of the cases initially misdiagnosed by the pathologist as melanoma was correctly recognized by the algorithm as a nevus. Conclusion: While additional studies with more lesion types, more cases, and more pathologists are needed, high specificity and sensitivity algorithms appear to be ready for use in screening, quality assurance, and workload distribution.

Background: Studies have demonstrated that the morphology on hematoxylin and eosin (HE) whole-slide images can be histologically classified by computer vision algorithms and has the potential to indicate the presence of genetic drivers. However, these algorithms need a large amount of annotated data. Whole-slide images often have gigapixel resolution, which makes it difficult to perform manual annotation.

In this work, we employed a GAN via a novel interactive representation learning patch-level annotation framework on whole-slide images of non-small cell lung cancer adenocarcinoma (NSCLA) to classify the lepidic histological subtype pattern of NSCLA. Methods: The pretrained acronym "GAN" stands for generative adversarial network was developed by computational pathologists at the University of Glasgow. The test set data source consisted of 437 lung images obtained from PathLink, which is a tissue and image repository associated with Vanderbilt's data warehouses. Five NSCLA lepidic subtype whole-slide HE images were extracted from this PathLink cohort. These images were then annotated with QuPath segmentation API to establish the ground truth. The images were divided into patches of 224 × 224 with 50% overlap. For data augmentation, the images were rotated. This resulted in 218,000 ground truth patches for the interactive patch-level annotation framework. Foreground annotation selections designated lepidic morphology while background annotation selections designated nonlepidic morphology. The real-time classification occurred via logistic regression in the Napari segmentation "API" stands for application programming interface. Results: Generalization performance of the GAN interactive patch-level annotation model was measured against ImageNet dataset weights derived from a ResNet50 model the original weights of developed GAN. Accuracy, precision, recall, area under the curve, and the F1 score were calculated for evaluating performance relative to the number of foreground and background annotations for each trial. Five annotation trials were performed for real-time annotation of two lepidic image for each dataset weights. Conclusion: Both the GAN and ImageNet models were able to distinguish significant amount of NSCLA lepidic subtype tissue. However, the model with ImageNet-based weights was prone to creating more false positives and required more foreground and background patch annotations to reach the target F1 score of 0.70 compared to model with the GAN. Background: Deep learning has excelled in a wide range of tasks in computational pathology, including but not limited to the characterization of both common and rare morphological phenotypes and the prediction of molecular alterations from histology. However, large datasets of diverse, high-quality annotated training examples are often necessary to build robust and accurate deep learning-based models for real-world test scenarios. Pooling medical data from multiple institutions can alleviate this challenge but is constrained by privacy concerns associated with sharing sensitive patient information, in addition to the technical challenges of transferring and storing large whole-slide image files at the terabyte or even petabyte scale. On the other hand, federated learning is a promising field of machine-learning research that allows algorithms to learn from multiple decentralized data sources without requiring training data to be directly shared. Methods: We introduce a federated-learning framework for computational pathology on whole-slide images using attention-based multiple instances learning for weakly-supervised learning and random noise generation for preserving differential privacy. We evaluate our approach on two different classification problems using large multicenter datasets, including renal cell carcinoma histologic subtyping and invasive breast carcinoma histologic subtyping. We additionally demonstrate that our approach can be used for survival prediction and patient stratification from whole-slide images. Results: Our proposed privacypreserving federated learning framework can be used to develop accurate, interpretable deep-learning models from distributed data sources for both classification and survival prediction. Conclusion: We showed our attention-based multiple instance learning inspired framework is applicable to both classification and survival prediction using whole slide images and slide-level labels only, and when combined with federated learning, enables accurate deep learning-based computational pathology models to be developed from multiple data sources while preserving differential privacy for each site.

Background: The Department of Pathology installed a slide scanning microscope for remote interpretation of frozen sections. We summarize the process and the challenges for our institution. Methods: A validation study set of 40 frozen section cases was digitalized using a Sakura Visiontek M6 digital microscope robotic imaging system. The cases were selected to represent a variety of organ systems. All slides were scanned at ×20, apart from touch preparations, which were scanned at ×40. Eight surgical pathologists reviewed the scans, independently, and their interpretations were compared to the final diagnosis rendered on the permanent sections. Major discrepancies were defined at the level of benign versus malignant, while minor discrepancies were defined as not giving enough information about the diagnosis or an inaccurate diagnosis in a family of benign or malignant neoplasms. Results: The rate of major discrepancies ranged from 0 to 2 (average of 2.2%), minor discrepancies from 0 to 3 (average of 4.7%), and deferrals from 1 to 4 (average of 6.3%). Two cases accounted for the majority of major discrepancies; one salivary gland and one bone tumor. Gynecological pathology and liver pathology were the most prevalent minor discrepancies (47% and 20%, respectively). Conclusion: A digital slide system appears to perform relatively well, but inferior to conventional slides. The rates of discrepancy did not appear to be affected by years of experience. There are some limitations to the design of this study which may affect the overall concordance. Background: The advent of highly multiplexed methods for antibody staining, including both imaging mass cytometry and multiplex fluorescence, allows for precise understanding of relationships between multiple protein biomarkers. These methods have advanced the capability to perform "tissue cytometry"-flow cytometry-like analysis on tissue sections. However, a major challenge for effective implementation of cytometric analysis on multiplex images remains the availability of robust segmentation algorithms to break the image into subregions representing single cells. Methods: Our core facility, working in close collaboration with several laboratories, has developed image segmentation pipelines for handling immune cell segmentation that takes into account the biological domain knowledge to assist with cell segmentation at the individual cell scale. Utilizing the background immunological understanding of cell surface markers obtained from flow cytometric analysis, the segmentation is guided in such a way as to avoid the appearance of "biologically impossible" marker combinations.

Biologically-guided segmentation has successfully been applied to a number of tissues stained with multiplex imaging mass cytometry approaches. Validation against manual annotations reveals an improvement in the accuracy of segmentation over traditional image analysis methodologies. Data visualization techniques adopted from flow cytometric analysis, including scatterplots alongside manually labeled cell populations, allow for accurate determination of image-based thresholds for cell identification. This modified strategy allows for a more thorough interrogation of the resulting dataset, assessing not only distinct cellular populations but also spatial relationships between distinct populations, providing a robust methodology to assess immune cell subpopulations in a spatial context.

Background: Recent advancements in digital pathology facilitate the understanding toward the role of tumor microenvironment (TME) in governing the triple-negative breast cancer (TNBC) progression. Assisted by image analysis and spatial statistics, key information about the spatial heterogeneity within the TME can be extracted. These analyses have been applied to CD8+ T-cells; however, quantitative analyses of other important markers and their correlations have not until now been possible. Methods: In this study, we developed a computational pathology pipeline for characterizing the spatial arrangements of five immune markers (CD3, CD4, CD8, CD20, and FoxP3), and then, the functionality is tested on 25 whole-slide images from patients with TNBC. To start, immune marker-labeled cells are segmented and colocalized. Then, this information is converted to point patterns. Subsequently, invasive front (IF), central tumor (CT), and normal tissue (N) are annotated. At this point, intratumoral heterogeneity can be analyzed. The pipeline is then repeated for all specimens to capture intertumoral heterogeneity. Results: In this study, we observe both intra-and inter-tumoral heterogeneities for all specimens across all five immune markers. Compared to CT and N, IF tends to associate with higher densities of immune cells and overall larger variations in spatial model fitting parameters and higher density in cell clusters and hotspots. Conclusions: Results suggest a distinct role of IF in the tumor immunoarchitecture. Although the sample size is limited in the study, the computational workflow could be readily reproduced and scaled due to its automatic nature. Importantly, the value of the workflow also lies in its potential to be linked to treatment outcomes and identification of predictive biomarkers for responders/nonresponders and its application to parameterization and validation of computational immunooncology models. This study provides a comprehensive analysis of immunologic heterogeneity in murine tumor models. We anticipate that these results will provide guidelines for designing preclinical studies for drug discovery and development.

The study of the heterogeneous tumor microenvironment has significant clinical implications. New assays to guide immunotherapy require accurate characterization of interactions between multiple phenotypes (e.g., tumor cells and different types of immune cells) in the tumor microenvironment.

Multiplexed immunofluorescence (IF) enables us to understand the complexity and heterogeneity of the immune context of tumor microenvironments and its influence on response to immunotherapies. In this project, we report the development of an automated digital pathology algorithm for biomarker classification using a deep-learning approach. The biomarker used is PD1 which is primarily expressed on lymphocytes.

We develop an end-to-end deep-learning algorithm to perform both cell detection and classification using only point labels.

To perform better localization of cells, we use repel coding which is an enhanced center coding of cells. To the best of our knowledge, repel coding is used for the first time for deep learning-based classification of biomarker on IF images. We have validated our algorithm on multiple IF-stained tissue types, which includes gastric, pancreas, lung, breast, colon, and bladder. The algorithm achieves more than 85% accuracy on all tissue types. Image analysis is motivated by the desire to extract relevant biological data from digital images to answer underlying questions about the pathogenesis of disease and other biological phenomena. With the advent of automated wholeslide scanning came the demand for image analysis software equipped with machine learning and artificial intelligence. This brings new issues and obstacles in image analysis and for those who perform it. Here, we present common difficulties in image acquisition and analysis on a cohort of tissue samples processed through the Analytic Microscopy Core at Moffitt Cancer Center. We compare pros and cons between two types of image analysis software on images acquired on an Aperio AT2 whole-slide scanner. We provide a realistic workflow which addresses and circumvents common bottlenecks in data exploitation from pre-to post-acquisition. In addition, we offer strategies to overcome complications such as stitching artifacts from image acquisition, tissue heterogeneity and damage, variability in stain penetrance, and border definition during segmentation, which all result in variation between samples. Being mindful of these obstacles and verifying data postprocessing can lead to robust, reliable, reproducible results and biologically relevant data.

The Grundium Ocus is a compact portable digital microscope and slide scanner ideal for desktop use [ Figure 1 ]. It permits whole-slide imaging (WSI) as well as robotic microscopy. We compared the performance of this device to a conventional light microscope with glass slides for second opinion teleconsultation and telecytology. Methods:

Ocus was compared to glass slide reads with a 2-week washout period for three use cases: (1) second opinion teleconsultation of gastrointestinal pathology cases (WSI), (2) second opinion teleconsultation of cytology cases (WSI), and (3) Maccabi has deployed both Galen Prostate and Galen Breast as a quality control (QC) system on all new prostate and breast biopsies entering the laboratory. The system raises an alert whenever it encounters a discrepancy between the automated analysis and the original diagnosis, prompting a second human review. Conclusions: The importance of accurate diagnosis in prostate and breast biopsies, together with the growing shortage of pathologists, makes a QC system as this extremely useful for diagnostic accuracy and safety. To the best of our knowledge, these are the first AI-based prostate and breast diagnostic systems running in a live clinical setting. In this talk, we will discuss the workflow in the laboratory and performance of the algorithms.

A pathologist's process of making a diagnosis is focused, holistic, and robust. A pathologist fully inspects all slides, carefully attends to informative regions within the slides, jointly considers all informative regions as context, and integrates available complementary clinical information or modalities. This process has been incredibly challenging to approximate with artificial intelligence (AI). It is an important goal because an AI that can accurately predict disease state and connect pathologists with similar patient cases has the promise of broadening consensus in patient care worldwide. Here, we propose an AI method that attempts to model aspects of a pathologist's diagnostic workflow. The AI considers one or more photomicrographs to describe a case, attends to regions of interest within the photomicrograph(s), jointly considers all regions of interest for an overall prediction, and includes possibly-missing covariates such as tissue type. Here, we refine our prior deep-learning technique that used photomicrographs shared on social media to predict disease state (nonneoplastic, benign, or malignant). [1] Like the simplest baselines in our prior work, [1] our proposed method uses 2412 hand-engineered features to represent a 224 × 224 region of interest in a photomicrograph, with additional inputs for ten tissue types and a marker-mention covariate [ Figure 1 ]. We sample regions of interest in a 5 × 5 grid throughout a photomicrograph and feed these to a deep neural network. Each of these regions is fed through a fully-connected layer, and an attention-weighted sum is calculated across the regions, where all attention weights are normalized to sum to one. This attention process selects for informative parts of the slide that contribute to the disease state (e.g., malignancy), while selecting against less informative covariates (e.g., tissue type) as inputs. This network then learns an attention-weighted sum of image inputs as a set representation of a patient case. Like our prior work, [1] the network ultimately learns to predict disease state validation accuracy approaches our prior work [1] of 60.16% (dotted red line). Solid blue line indicates validation accuracy progression through training Figure 3 : Search performance precision@k = 1 on the test set is 0.4461, which is better than chance performance from our prior work (0.3967 ± 0.0044) [1] Journal of Pathology Informatics S30 parts of the slide that do not contribute (e.g., slide background, normal tissue, pen, and emojis). Subsequent fully-connected layers integrate covariates, and a final prediction of disease state is made. Compared to our prior work, our new deep-learning method's performance approaches similar validation accuracy [60.16% vs. 54.66%, Figure 2 ]. Following an 80% training, 10% validation, and 10% test set split of our data, we report a test set accuracy of 52.63% (without covariates, this 52.63% drops to 43.94%). These accuracies are better than chance (41.01% ± 1.50% accuracy, random forest permutation test, 10-fold cross validation). Area under receiver operating characteristic is 0.6722, which is weak, but better than chance (0.5003 ± 0.0149, permutation test). Compared to our prior work, [1] our proposed deep-learning method is a strikingly simpler baseline in that we use only 2412 hand-engineered features to represent an image, rather than convolutional features from a ResNet-50. [2] In the future, we expect to improve performance further by using ResNet-50 features, in a manner similar to our prior work. [1, 3] As a search baseline, we report precision@k = 1 of 0.4461 [ Figure  3 ]. This was better than a simple L1 norm of the 2412 features (0.4143 ± 0.0071, 10-fold cross validation), but there is room to improve. In contrast, our prior work reported precision@k = 1 of 0.7618 ± 0.0018 (via leave-one-pathologist-out cross validation), but this involved a complex blend of features, deep neural network ensembles, and a random forest. In the future, we expect our proposed attention-based methods to become an important component of a more complex histopathology search and disease prediction system. In conclusion, our simple proposed attention-based deep-learning method promises to become an important component of more sophisticated methods for disease prediction and search on social media, such as those in our prior work. [1] We believe that this is the first use of attention-guided deep learning on histopathology data from social media. In the future, we expect this to improve the utility of "pathobot," our AI-driven histopathology search tool on social media. [1] Introduction: Training machine-learning models for analyzing digital pathology images requires a large set of manually labeled ground truth, which is tedious and time-consuming to collect. Herein, we develop an active-learning system that guides the user to label samples that contribute the most to the learning performance of the model. This way, the model can be trained using less labeled data. Furthermore, the system enables users to train or optimize classifier models while iteratively collecting the ground truth. Active learning requires several different components including high-resolution image storage and retrieval, data visualization, interactive data correction, and image analysis for algorithm training and inferencing. This framework is available within dPath, a Roche Tissue Diagnostics Computational Pathology Research Platform. Methods: The designed system can be employed by pathologists or imaging scientists to collect the ground truth and train classifier models in an iterative manner. Different conventional machine learning or deep-learning models can be trained using this system. The user starts with labeling cells or regions of the tissue in the images. The labeled examples are used to train a model or optimize a pretrained model. The classification results and corresponding certainty heatmap are visualized. The user then labels more training samples from the most uncertain regions and retrains the classifier. This iteration continues until the model reaches the desired performance and can be deployed. A high-performance image server handles the requests for images from the front-end application. Using Roche's high-performance cluster, results from training and inferencing the model are generated and stored at scale in a database. APIs enable the interaction between the machine learning engine and the database. The dPath platform integrates all of these components and allows the user to train models within a web browser. Results: For proof of concept, we successfully trained a model to detect macrophages in multiplex immunofluorescence stained tissue images on dPath. The active-learning system enabled us to train a classifier with half the size of the labeled data needed in random sample selection approach. Conclusion: The digital pathology active learning system enables the end-user to optimize a classifier in an efficient and interactive manner.

Journal of Pathology Informatics S31 Introduction: Multiplex immunofluorescence (IF) staining of tumor specimens enables simultaneous assessment of different biomarkers in the tumor microenvironment (TME), which provides insights into tumor progression and responses to immunotherapy approaches. Digital pathology algorithms can quantify the expression and co-expression of biomarkers and characterize their spatial relationships in the TME on a cell-by-cell basis. These algorithms usually rely on manual identification of the tumor area in whole-slide images (WSIs) by a pathologist. Necrotic regions are also manually annotated to be excluded from phenotype detection and reporting. A necrotic region is a spectrum of morphological changes that follow cell death in living tissue. The variation in the size, shape, and number of necrotic regions across WSIs makes their manual annotation an error-prone and time-consuming task. Herein, we developed an automated deep learning-based algorithm for segmentation of necrosis regions in IF-stained WSIs. Methods: The deep-learning architecture that we employed is U-Net, which is a fully convolutional network and is fast to train and performs well on small training datasets. Our dataset consists of WSIs of colorectal tumor sections stained with a 5plex IF assay for CD3, CD8, CD68, PD-L1, and PanCK represented by Cy5, R6G, R610, FAM, and DCC fluorophores and DAPI as a counterstain. The necrotic regions are annotated on WSIs by a pathologist, and ground truth segmentation masks are generated. To train the model and assess its mask prediction performance, we used the following quantitative features: sum of binary cross-entropy loss and dice coefficient. Results: We used 28 WSIs (24 for training and 4 for testing) and selected random regions of interest (ROIs) on them. Then, we extracted image patches of fixed size (512 × 512 pixels) from ROIs (2126 patches for training and 230 for testing). We used DAPI and PanCK channels as the input to the U-Net model and ground-truth masks as the corresponding mask labels. From our initial experiments, we were able to achieve acceptable mask segmentation predictions both quantitatively and visually. Conclusion: Automated identification of necrotic regions by deep-learning algorithms enables faster and more reliable assessment of IF-stained WSIs by eliminating the need for manual annotation from pathologists. Background: Pathology is switching from an "analog" to a "digital" era. Digitalization is not limited to slide scanning; artificial intelligence (AI)-based tools and computer-aided diagnoses are increasingly used. In this context, pathologists will benefit from adjustments to training, enabling them to understand and use these technologies properly. Here, we report on our preliminary implementation of a computational pathology (CP) training program at Roche. Methods: Veterinary pathology (VP) and computer vision (CV) specialists worked together to improve joint knowledge in these fields. The work was divided into (1) review available resources, (2) identify and outline the scientific questions to be addressed with AI, and (3) "hands-on" assessment of the acquired knowledge by means of close VP/CV interaction in algorithm development or testing of commercially available software. Results: The review process led to creating a digital interactive AI digital glossary, with videos and links to external resources. During the "hands-on" experience with use case examples, python programming skills or in-depth knowledge of commercially available AI tools were acquired. Conclusions: The glossary was a useful resource for the pathology and informatics groups that will be extended to cover Unix and relevant commercial software. In our experience, Unix knowledge is a must-have in the CP algorithm development. Python knowledge improves pathologist/computer scientist communication and helps in developing complex tools; nevertheless, commercial tools may offset programming basic daily-based tasks.

Journal of Pathology Informatics S33

were 84%-100%. Conclusions: The proposed method has been shown to perform well in identifying RBC artifacts in multiplexed immunofluorescence images. Meanwhile, more diversified ground truth data including negative samples improved the overall performance. Background: Among the genitourinary cancers, prostate cancer (PCa) is considered pauci immune, and it is well documented that not only the presence or absence but also the spatial and contextual arrangements of tumor-infiltrating lymphocytes (TILs) is the determinant of outcome. The objective of this study is to illustrate detailed and contextual evaluation of TILs on whole slides and corresponding tissue microarray (TMA) sections. Methods: A total of 80 PCa cases with their corresponding TMA slides were used for this analysis. For each case, a single most representative hematoxylin and eosin slide was scanned at ×40 using XY scanner. Tumor from the corresponding block was represented in triplicate on a TMA using 1 mm needle. An image-based algorithm was developed using QuPath-0.2.0-m8 software to assess both intratumoral TILs (IT-TILs) and peritumoral TILs (PT-TILs). Since the TMA sections were taken from the center of the tumor nodule, only IT-TILs could be assessed. We compared density of lymphocytes between the numbers calculated from WSI and TMA, respectively, using Spearman's rank correlation analysis and paired t-test. We also perform subgroup analyses based on different pretreatment PSA levels, clinical stages, pathological stages, biopsy Gleason scores, and therapy outcome. Results:

We present the first spatial quantitative study of TILs in 80 primary prostate adenocarcinomas. In most patients, both PT-TILs and IT-TILS could be assessed by WSI; however, in the TMAs, the PT-TILs could not be evaluated at all due to systematic selection of intratumoral region during construction.

No meaningful correlation was observed between density of lymphocytes calculated from WSI and TMA, respectively (r = −0.014). Conclusion: TMA-based quantitative immune cell counts were validated by automated paired TIL quantification in whole-slide cohorts. Our results suggest that wholeslide imaging by virtual microscopy is irreplaceable for TIL quantification, as a potential biomarker predicting and monitoring PCa treatment response. TMA analysis does not provide information of tumor microenvironment and TIL at peritumoral compartment of prostate cancer. Even while using three cores from each sample to construct TMA blocks, it is not sufficient to allow for coverage of biologic heterogeneity of the infiltrating lymphocytes in the intratumoral compartment of prostate adenocarcinomas. One potential solution might be the creation of image microarrays that would allow for capturing all of a tumor's morphologic variation on a single slide.

Color normalization in digital histopathology images

Convolutional Networks for Biomedical Image Segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention

Deep watershed transform for instance segmentation

Olaleke O. Folaranmi 17 , Jerad M. Gardner

Department of Pathology, BC Cancer, British Columbia, Canada, 6 Department of Pathology

Deep Residual Learning for Image Recognition

Data-efficient and weakly supervised computational pathology on wholeslide images

Deep Learning for Health Informatics

A dynamic Bayesian networks to predict sequences of organ failures in patients admitted to ICU

Measuring severity: How sick is sick? How well is well?

Domino effect principle

Use of health information technology to reduce diagnostic errors

A theory of medical decision making and health: Fuzzy trace theory

Background: CorePlus Servicios Clínicos y Patológicos in Puerto Rico, a leading pathology and clinical laboratory, handles 53,000 accessions annually, of which ~6.4% are prostate core needle biopsies (PCNBs), with ~46% diagnosed with cancer. Methods: Ibex Medical Analytics developed Galen™ Prostate, an artificial intelligence (AI)-based solution, to detect and grade prostate cancer in addition to other cell types and features such as perineural invasion, highgrade prostatic intraepithelial neoplasia, and inflammation. The algorithm was trained on >1 M image samples from multiple institutes manually annotated by senior pathologists. Galen Prostate is the first AI-based solution for cancer detection used in routine clinical practice, is deployed in pathology laboratories worldwide, and has identified missed cancers both retrospectively in France and prospectively in Israel. Results: A total of 101 retrospective prostate cases from CorePlus comprising a total of 1279 hematoxylin and eosin slides were analyzed by the algorithm, demonstrating high specificity and sensitivity for cancer detection (96.9% and 96.5%, respectively) and an area under the curve of 0.901 for differentiating low-grade (Gleason 6) from high-grade (Gleason 7+) cancer. The diagnosis of 11 slides was revised subsequent to algorithmic results. After validation, CorePlus deployed Galen prostate as quality control (QC) system on all new PCNBs. The system raises an alert for discrepancies between the algorithmic analysis and the pathologist, prompting a second human opinion. Conclusions: An AIbased QC system is extremely useful for diagnostic accuracy and safety. To the best of our knowledge, this is the first AIbased digital pathology diagnostic system deployed in a US laboratory and used in routine clinical practice. Background: In the last 20 years, digital pathology emerged as an important tool for pathologists and has been critical for the continuation of clinical and scholarly work during the COVID-19 pandemic. Before the pandemic, we had successfully deployed whole-slide imaging (WSI) and other digital pathology tools for daily sign-out, education, and research at our institution. As such, we were well positioned to adapt to the changes in practice imposed by the pandemic. Methods: We conducted a survey of our pathologists and trainees to gather information about their pre-and post-COVID-19 use of digital pathology tools. Results of the survey were analyzed and representatives from each subspecialty commented on unique aspects of their workflow. Results and Conclusions: Before the COVID-19 pandemic, utilization of digital pathology tools was variable among pathologists and trainees. The most significant change occurred in the adaptation of digital pathology for faculty-trainee sign-out and educational conferences. Before the pandemic, most faculty-trainee interactions were in-person, utilizing glass slides and a multi-headed microscope. The pandemic, however, suspended all in-person sign-out and resulted in a greater use of digital pathology tools. Survey results demonstrate that both pathologists and trainees were satisfied with remote sign-out during this time, but most preferred returning to in-person interactions once restrictions are lifted. With the CMS waiver for off-site primary sign-out, some faculty utilized digital pathology tools from home. While most favored permanently allowing off-site primary sign-out, most indicated that they would use it sparingly or in unique circumstances once activities returned to normal. Background: Risk severity-ranked biomarkers are early "trigger signals" of interactive, critical organ/system (COS) dysfunction. [1] The first dysfunctional COS rapidly induces multiple critical organ/system failure syndrome (MOFS), [2, 3] and therapies are often too late, if MOFS already exists. Quantified, significantly abnormal (SA), biomarkers signify cell and tissue damage in each COS. Color-coded SA biomarkers are visual alerts for the Domino effect, where one COS failure affects others. [4] Risk pattern severity levels detect cell and tissue damage in the first dysfunctional COS that causes sequential MOFS. [2] Methods: Signal detection theory finds optimal combination of SA diagnostic biomarkers, quantified and clustered into "risk signal patterns" versus "noise" for each COS. Predictive diagnostic analytics isolates high-risk SA biomarker patterns in multimorbid patients, "trigger signals" of poor outcomes. Graphic reports of colorcoded risk patterns are perceived faster by the physician's brain than numbers. [5, 6] Results: Figure 1 shows the new patientspecific report. Conclusions: Multimorbid patient's risk for MOFS accelerates when pathologic biologic, toxic, physical, or endogenous agents cause vascular endothelial damage and immunologic dysregulation. Laboratory Medicine (LM) is the first to detect SA biomarkers and quantitate (rank) their severity with predictive diagnostic risk analytics. LM defines a patient's COS "point of maximal clinical risk" with color-coded SA. Visual diagnostic predictive informatics is a proactive LM patient safety tool for early detection of high-risk patients by translating data into knowledge with visual images. However, red blood cell (RBC) artifacts generated by autofluorescence interfere with images analysis in multiple ways, which can cause: (1) false detection of nuclei in the DAPI channel; (2) misclassification of individual biomarkers; and (3) incorrect segmentation of tumor areas. The objective of this study was to identify RBC artifacts to improve the classification and segmentation accuracy of the biomarkers. Methods: As purely intensity-based methods for RBC identification have limited accuracy, a deep convolution neural network-based method for image segmentation (U-Net) was explored because it incorporates intensity as well as texture features through learning from image data. Field-ofview images were obtained from the whole slides, on which ground-truth region masks were marked for RBC artifacts and non-RBC objects. The image patches of size 256 × 256 were extracted from the field-of-view color images, which were generated from the original multiplexed immunofluorescence image. The details of training configurations are as follows: loss function is binary cross-entropy, and optimizer is Adam. Three experiments were performed: (1) using a single tissue indication (bladder, 152 patches) to train and test the network.;

(2) using a combination of six tissue indications (colon, lung, breast, pancreas, gastric, and bladder, 275 patches); (3) combining six indications and negative patches where no RBC signal present (344 patches). In all experiments, 80% of the data were used for training and 20% of the data were used for testing. Results: Compared to the annotated ground truth, the training accuracy in the three experiments was 93.8%, 93.9%, and 95.2%, respectively. The training losses were 0.142, 0.142, and 0.113, respectively. The accuracy for the test data in the three experiments was 85.6%, 85.5%, and 88.3%, respectively. The losses for the test data were 0.149, 0.148, and 0.119, respectively. Four field-of-view images were randomly selected from the test dataset; the pixel-to-pixel agreement between the identified RBC mask and the ground-truth mask