key: cord-0058578-pb4cy2pb authors: Dirvanauskas, Darius; Maskeliūnas, Rytis; Raudonis, Vidas; Misra, Sanjay title: Embryo Spatial Model Reconstruction date: 2020-08-26 journal: Computational Science and Its Applications - ICCSA 2020 DOI: 10.1007/978-3-030-58814-4_65 sha: e80845cce5f7d5f2fadf0167c719f6367f7cc0ef doc_id: 58578 cord_uid: pb4cy2pb Time lapse microscopy offered new solutions to study embryo development process. It allows embryologist to monitor embryo growth in real time and evaluate them without interfering into their growth environment. Embryo evaluation during growth process is one of the key criteria in embryo selection for fertilization. Live embryo monitoring is time consuming and new tools are offered to automate part of process. Our proposed algorithm gives new possibilities for embryo monitoring. It uses embryo images which are taken from different embryo layers, extracts embryo cell features and returns metrical evaluation to compare different embryos. High number of extracted features shows embryo fragmentation. Other tool which we present is spatial embryo model. Features extracted from embryo layers are combined together to spatial model. It allows embryologist to examine embryo model and compare different layers in one space. The obtained spatial embryo model will be later used to develop new algorithms for embryo analysis tasks. Different artificial fertilization methods, such as insemination or in vitro fertilization (IVF) are used by fertility doctors to reach successful pregnancies. If medicaments do not help IVF is used to treat infertility. Multiple embryos are grown in vitro at the same time. One of IVF success factors is selection of the most viable embryo [1] . Embryos for insemination are selected in visual inspection performed by embryologists. It is prone to error and requires time. In one cycle of IVF the pregnancy success rate can be as high as 60%. In some cases fertilization fails repeatedly and patients need multiple IVF cycles [2] . Time-lapse microscopy (TLM) has provided new tools for embryo image inspection. These machines are used to continuously monitor embryos and capture images in different layers. The aim is to inspect embryo's cell shape and measure development stage duration without removing from their growth environment. Embryo growth to 4-cell stage could last up to 48 h and 72 h to 8-cell stage. In most cases duration of 2-cell and 4-cell stages defines quality of the embryo and could be used as one of the decision factors for embryo selection [3] . To decrease human error rate automated tools are required to accurately track duration of development stages for multiple embryos at the same time. In this study, we present a method for embryo's spatial model reconstruction. This spatial model will allow embryologist to analyze embryo features from different layer in one space. Different image analysis methods are already applied to embryo analysis. Real time embryo division monitoring techniques capture division time [4] . Different solutions using conditional random field [5] and VisSeg model [6] were proposed for cell counting. A. Khan applied deep learning techniques to estimate number of cells in embryo [7] . Other study applied deep learning to develop automated embryo grading system [8] . Fully convolutional neural network model was used for embryo inner mass segmentation [9] . Set of neural networks were trained to predict quality of embryos [10] . Embryo cell localization was achieved by using linear chain Markov model [11] . Three dimensional morphological segmentation using watershed based algorithm was presented [12] . Machine learning methods were used to develop model which helps to select potential embryo for implantation [13] . UNet architecture was proposed for blastomere centroids localization [14] . Dual ResUNet model which avoids loss of spatial and identity was proposed for zebrafish embryo segmentation [15] . High speed camera and deep learning model are used for real time cell population analysis [16] . Combination of Gaussian distribution and K-Means method was proposed for white blood cell segmentation [17] . Circular Hough transform was performed to count blood cells in microscopic images [18] . Different machine learning classifiers were used to classify blood cells for leukemia diagnosis [19] . Image based data analysis techniques for better results were presented in J. Caicedo and et al. study [20] . Deep learning architectures for medical image segmentation and classification were discussed in other study [21] . Machine learning algorithms were used to create image based cell sorting model [22] . There is lack of studies which proposes spatial embryo analyses algorithms. In this work, we aim to develop a method which would prepare embryo's spatial model and analyze its quality. Combination of image processing algorithm were used to extract embryo cell features. Images captured in different embryo layers were used to construct three dimensional model. This model will be a tool for embryologists to analyze spatial model and compare separate layers in one space. Embryo images used in our study were taken with Esco Global incubator series, called Miri TL (timelapse) [23] . The embryo image sets were registered in the German, Chinese, and Singapore clinics. No identity data were ever provided to the authors of this paper. Embryo database consists of more than 26,000 image sets with a resolution of 600 × 600 pixels. One image set contains 7 photos taken from different embryo layers. Images are obtained from 24 different growing embryos in Esco Global incubator. Esco incubator culturing conditions were: temperature of 37°C, stable level of 5% CO2, and the controllable values of nitrogen and oxygen mixture. The embryos were photographed in a culture coin dish made from polypropylene with a neutral media pH = 7. The inverted microscopy principle was used, with 20x lenses, without zoom, with focusing and with field of view of 350 um. An example of embryo images in the Esco dataset is presented in Fig. 1 . The embryo feature extraction and spatial model generation was done on an Intel i5-4570 CPU with a GeForce 1060 GPU and 8 GB of RAM. Our proposed model process embryo images in multiple steps (see Fig. 2 ). User selects embryo sample for analysis and model loads images from seven embryo layers. Images are converted to grayscale format and smoothed to reduce noise level. After that, in vitro localization detects its contour in every layer's image to find most accurate contour for every layer. Background, which is outside in vitro contour, is removed. In embryo contour detection, cell edge and embryo fragmentation features are extracted. These features are used in next steps to compare embryos and prepare their spatial model. In spatial embryo model reconstruction our proposed method uses extracted features. It takes these features and plots them as 3D scatter model. Embryologist could use this model to evaluate and compare different embryo layers in one space. This comparison helps if cells grow equally in different layers. Spatial model have options to hide layers, change distance between layers and make X and Y axis cuts (see Fig. 3 ). Our tool gives new possibilities for embryo analysis. It transfers information from 2D images to 3D spatial plot. In layer feature comparison step embryo cell contour and fragmentations are counted as features. These features are added together and considered as base value for embryo evaluation. If test embryo feature number exceeds base value, it is considered that embryo's fragmentation rate is too high. This embryo comparison gives embryologists additional information for embryo evaluation. For model evaluation extracted features are compared to base value. This comparison defines embryo fragmentation level. Embryo spatial model evaluation is done by embryologist visual inspection. At first in vitro localization part algorithm tries to find circle which define in vitro contour the best. Loaded seven embryo layer images are converted to grayscale format. To reduce noise, images are smoothed with 3 × 3 averaging filter. Filter takes average of pixels under kernel area and modifies central pixel value with the average. Circle Hough Transformation is used to find circle, which defines in vitro location [24] . A circle with radius R and center (a, b) can be described with the following parametric equations. x = a + R cos(θ ) (1) To reduce computation time we limit radius with lower and upper values. If multiple circles are detected, algorithm averages their radius and center. After circles are detected in every layer, algorithm averages their radius and centers to return parameters into one circle. If circle is not detected in layer, that layer is skipped and not used for circle parameter calculation. Background view which is outside detected circle is removed as not needed information. View inside in vitro is used in next step to extract embryo features. Active Canny edge detection method is used to get embryo cell contour. In our model threshold values are calculated and adjusted according to pixels median of filtered image. Threshold lower and upper values are calculated using Eqs. 3 and 4, there σ = 0.33 and x is the median of the single channel pixel intensities. Our proposed model allows embryologists to evaluate embryo fragmentation number in different embryo layers. For our experiment we took 24 exemplary embryos while they evolved for 48 h. Average feature extraction duration 1.3 s. The following Table 1 gives average values, which could be used as a base value to evaluate embryo fragmentation. If test embryo feature value is significantly higher than base value in presented table we can predict that test embryo has higher number of fragmentation. After this evaluation we can see in Fig. 4 that embryos with bigger number of cells have higher feature number. This is caused, because of additional cell contour, which appears in two and four cell samples. From the graphic we can see that first layer feature number is 5-7% lower than feature number in third and fourth layer. It means that at lowest layer less features are detected and number increases till third layer. From fourth to seventh layer feature number decreases. This pattern allows to separate embryos which feature number is distributed differently compared to base values. In our database we only had embryos which started to mutate at two cell stage. Therefore we were not able to compare one and four cell healthy embryo feature number to mutated ones. We took test embryos which started to mutate in two cell stage to compare their feature level with base value. Results showed that average extracted feature number from these embryos there 1964. It is 37% higher compared to normal two cell embryo. We were not able to compare one and four cell mutated embryos, because our database did not had them. In Fig. 5 three different development stage embryos are presented. These plots allow embryologists to examine embryo cell contour in every layer. It is seen that full embryo contour is not extracted in every layer. Some of it blends with background and is not separated in contour detection step. Advantages of this plot is that layers can be stacked one on each other and it allows to analyze how embryo contour differs in each layer. We used localization and edge detection methods to extract embryo features from their seven layer images. These features are presented as spatial plot. Embryologists can examine plot by rotating embryo 3D model. This model can be dynamically changed by making X and Y axis cuts and changing distance between layers or turning them off. Prepared spatial model could be utilized for embryo 3D model printing. Features extracted from 24 sample embryos were averaged to make base values to which test embryo could be compared. If feature number in test embryo is higher than base value, we can predict higher fragmentation rate in that embryo. Feature numbers do not spread evenly in every layer. First layer has smaller number of features, which increases up to third layer. From third to seventh layer feature number decreases. This feature spread pattern could be used as template to which we can compare test embryo feature spread. Our database had only embryos which started to mutate in two cell stage. Therefore we were not able to carry out all three stages embryo base value comparison to mutated ones. In our study we found embryo feature base value that increases while embryo grows. Average feature numbers in different stages are: 1 cell -1328; 2 cell -1423; 4 cell -1558. Features extracted from mutated embryo which mutation started at two cell stage gave 37% higher feature number compared to healthy one. Which factors are most predictive for live birth after in vitro fertilization and intracytoplasmic sperm injection (IVF/ICSI) treatments? Analysis of 100 prospectively recorded variables in 8,400 IVF/ICSI single-embryo transfers Investigation and treatment of repeated implantation failure following IVF-ET Bovine embryo morphology and evaluation Automated embryo stage classification in time-lapse microscopy video of early human embryo development Automated monitoring of human embryonic cells up to the 5-cell stage in time-lapse microscopy images Segmentation and quantitative analysis of individual cells in developmental tissues Deep convolutional neural networks for human embryonic cell counting Using deep learning with large dataset of microscope images to develop an automated embryo grading system Inner cell mass segmentation in human HMC embryo images using fully convolutional network Artificial intelligence techniques for embryo and oocyte classification A linear chain markov model for detection and localization of cells in early stage embryo development 3DMMS: robust 3D membrane morphological segmentation of C. elegans embryo Selection of single potential embryo to improve the success rate of implantation in IVF procedure using machine learning techniques Blastomere cell counting and centroid localization in microscopic images of human embryo Zebrafish embryo vessel segmentation using a novel dual ResUNet model Real-time image processing for microscopy-based label-free imaging flow cytometry in a microfluidic chip Detection of white blood cell cancer using image processing Blood cell count using digital image processing Computer-aided acute lymphoblastic leukemia diagnosis system based on image analysis Data-analysis strategies for image-based cell profiling Deep learning for medical image processing: overview, challenges and the future Intelligent image-activated cell sorting Machine Vision: Theory, Algorithms, Practicalities Acknowledgments. The authors also would like to thank Esco Global for kindly provided embryo image dataset and ŽivilėČerkienė for her expert opinion and helpful comments. Ethics. The permit for ethical studies for using the human subject related materials was issued by the Ethics Committee of the Faculty of Informatics, in Kaunas University of Technology. The number of a permit is: IFEP201706-3.