key: cord-0231008-t6avgalh authors: Zhao, Tianrui; Ourselin, Sebastien; Vercauteren, Tom; Xia, Wenfeng title: Seeing through multimode fibers with real-valued intensity transmission matrices date: 2020-02-29 journal: nan DOI: nan sha: befa550ef1743187dd3b17f11a2129dc8e0c816e doc_id: 231008 cord_uid: t6avgalh Image transmission through multimode optical fibers has been an area of immense interests driven by the demand for miniature endoscopes in biomedicine and higher speed and capacity in telecommunications. Conventionally, a complex-valued transmission matrix is obtained experimentally to link the input and output light fields of a multimode fiber for image retrieval, which complicates the experimental setup and increases the computational complexity. Here, we report a simple and high-speed method for image retrieval based on our discovery of a pseudo-linearity between the input and output light intensity distributions of multimode fibers. We studied the impact of several key parameters to image retrieval, including image pixel count, fiber core diameter and numerical aperture. We further demonstrated that a wide variety of input binary images could be faithfully retrieved from measured output speckle patterns using this method, promising to be useful for highly miniaturized endoscopy in biomedicine and spatial-mode-division multiplexing in telecommunications. Multimode optical fibers (MMFs) have been increasingly attractive for applications in biomedical endoscopy and telecommunications, owing to the capability of transporting light via a large number of transverse optical modes. For biomedical endoscopy, the number of transverse modes in an MMF represents the number of pixels in the images. Compared to multi-core coherent fiber bundles that are commonly used in biomedical endoscopy, MMFs are significantly more cost-effective, and the effective pixel density in an MMF can be 1-2 orders of magnitudes greater [1] [2] . For telecommunications, MMFs are attractive due to the potential of multiplexing data within the large number of modes. However, light propagation in MMFs suffers from modal dispersion and mode coupling; For example, when projecting an image pattern onto the proximal fiber tip, the light couples into different modes with different propagation constants and thus forms a random-like speckle pattern at the distal end [1, 3, 4] . Therefore, the propagation characteristics of an MMF are required for faithfully retrieving input patterns from measured speckle patterns. Wavefront shaping has been an emerging technology for controlling light transport in disordered media [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] . A number of research groups [7] [8] [9] [10] [11] have studied the transmission matrix (TM) theory for image transmission through disordered media such as MMFs. In this theory, the disordered medium is characterized by a complex-valued matrix, which connects the input and output light fields with complex-valued transmission constants that represent the changes of light field during the light transport [6, [8] [9] [10] [11] . Therefore, the input light field can be calculated when the TM and the output light field (phase and amplitude) are known. However, conventional cameras are only able to capture the light intensity, and the phase information is usually obtained using holographical methods with a complex optical reference arm that can degrade the system stability [8] [9] [10] [11] . Recently, image retrievals through disordered media using only the intensity information of the output light field were achieved with deep learning [17-21] and model-based methods [22] , allowing a simpler optical system compared to those of TM-based methods. With deep learning, a large set of input-output image pairs were used to train a multi-layer neural network for predicting input images such as handwritten digits [17] and letters [19] , and Quickdraw objects [18] . However, the requirements for large datasets and iterative optimization processes resulted in prolonged time for data acquisition and neural network training. Further, the performance of the trained neural network is likely to depend on similarities between the testing and training datasets. Most recently, a model-based method was used to characterize an MMF [22] . With this method, a set of input images were projected onto one end of an MMF whilst the intensity values of the outputs at the other end were converted into amplitude (square root of the intensity) information with zero phase. With these input-output pairs, an iterative algorithm was used to obtain a complex-valued matrix as the inverse TM of the fiber. Different from the deep learning methods, this matrix explicitly linked the input images with the output speckles with a physically-informed model and hence allowed the retrieval of images of complex natural scenes [22] . However, large datasets and time-consuming iterative optimization were required for calculating the complex-valued matrix, which limited its applicability. Thus, a simple and high-speed approach to characterize MMFs is highly desired. In this work, we developed, for the first time to the best of our knowledge, a method to characterize an MMF with a real-valued intensity transmission matrix (ITM), which connects the input and output light intensities and hence to retrieve input images from the intensity values of the output speckles. This method is based on our discovery of a pseudo-linearity between the input and output intensity distributions of MMFs. Importantly, as calculating real-valued ITMs was achieved without a time-consuming iterative process or a large database, this method allowed a high-speed characterization to be performed within ~16 s (~8.2 s for data acquisition and ~7.8 s for ITM calculation) for retrieving images with 1024 pixels and hence paved the way for the practical uses of MMFs in endoscopy and telecommunications applications. A number of experiments were performed to investigate the impact of several key parameters to the performance of image retrieval. The relationship between the switched 'ON' pixel count of input images (J) to the pseudo-linearity is shown in Fig. 1 . With a constant input pixel count (1024), the correlation coefficient, indicating the fidelity of image retrieval, increased rapidly from ~5% to ~90% with J increasing from 32 to 384, and remained largely consistent with J in the range of 384 to 896. With J increasing from 896 to 1024, the correlation coefficient declined rapidly, which can be attributed to the loss of low spatial frequency components due to the diffraction of the DMD micromirrors. This suggests that the pseudo-linearity is dependent on the J values. One possible explanation is that as the input binary patterns for characterization had 50% pixels with "1" value, the resulting ITM was more suitable for retrieving input images with a similar intensity distribution as those used in characterization. However, further investigations are required to better understand this dependency. The impact of total input pixel count (N) is shown in Fig. 2 . With the same input image pattern, the input images with more pixels had lower correlation coefficients. The fiber core . To achieve a greater pixel count while maintaining good fidelity, MMFs with larger core diameters and/or numerical apertures could be used as they support larger numbers of transverse modes. However, it is notable that this will result in increased computational time for fiber characterization and image reconstruction, which could be mitigated with parallel computing. The impact of the variability of input image patterns is show in Fig. 3 . The fidelity of the retrieved images was weakly dependent on the input patterns: correlation coefficients varied from 91.76% for a handwritten digit to 97.62% for a random binary pattern. After binarization, correlation coefficients for all the types of input images increased and were greater than 99% in most of the cases. With a higher number of input pixels (64×64), the correlation coefficient varied from 74.84% for the handwritten digit to 90.91% for the random binary pattern, respectively (see Supplement 1 Fig. S2 ). Two videos showing retrieved images from a series of output speckles through the same fiber are shown in Visualizations 1, 2. Compared with other existing methods, our approach has a number of distinct advantages. First, the fiber characterization process with our method requires both a short time for calculation and data acquisition. In contrast, both the training process of deep learning-based methods and model-based algorithms rely on iterative optimization that is usually very timeconsuming (several hours). Furthermore, they require large training datasets and hence a relatively long data acquisition time. This is problematic when repeated fiber characterization process is required due to speckle decorrelations. Second, the retrieved image quality with our method is weakly dependent on the types of input images. However, deep learning-based approaches are most likely to work only for images that are similar to the training datasets [17] [18] [19] [20] [21] . Last, the experimental setup is much simpler compared to those of holographical methods with complex optical reference arms that can degrade the stability of the system. Similar to other methods, our method also suffers from speckle decorrelations induced by fiber deformations. One mitigation solution could be to integrate an MMF within a rigid medical catheter or needle for biomedical applications. Recently, it was reported that with the knowledge of the fiber shape, TM of the fiber can be corrected to compensate deformations-induced speckle decorrelations [1] . A similar approach could be used in the future to correct for the deformation-induced changes in ITM for image retrieval. In conclusion, we developed a method to measure the light intensity transmission characteristics for an MMF with a real-valued ITM, and with this, input light intensity distributions can be reconstructed from the measured output light intensity distributions. This method enables a high-speed MMF characterization, and high-fidelity image retrievals using a simple measurement setup, and thus could be useful for several applications in biomedical endoscopy and telecommunications. The Here we defined an ITM that connected the intensities of the input patterns and the output speckles as: where I p m represented the intensity value at the m th pixel in the p th output speckle, and each column of this matrix represented a vectorized input pattern. As all micromirrors were so that the ITM can be obtained as: where In the second step, the ITM was used to reconstruct input images from the output speckles using linear inversion as: where Iimage is the intensity distribution of the retrieved input pattern, Iout is the intensity distribution of output speckle measured by the camera at the distal fiber tip, Iin is the intensity distribution of the ground truth. Here we define this approximate linear relationship between the input and output intensities as a pseudo-linearity. To study the relationship between the intensity distribution of the input image to the pseudo- Seeing through chaos in multimode fibers Ultrathin endoscopes based on multicore fibers and adaptive optics: a status review and perspectives Long-range spatio-temporal correlations in multimode fibers for pulse delivery Principal modes in multimode fibers: exploring the crossover from weak to strong mode coupling Focusing coherent light through opaque strongly scattering media Controlling waves in space and time for imaging and focusing in complex media Measuring the transmission matrix in optics: an approach to the study and control of light propagation in disordered media Shaping the light transmission through a multimode optical fiber: complex transformation analysis and applications in biophotonics Exploiting multimode waveguides for pure fiber-based imaging Digital confocal microscopy through a multimode fiber Scanner-free and wide-field endoscopic imaging by using a single multimode optical fiber Focusing through dynamic tissue with millisecond digital optical phase conjugation Photoacoustically guided wavefront shaping for enhanced optical focusing in scattering media Focusing and compression of ultrashort pulses through scattering media High-speed scattering medium characterization with application to focusing light through turbid media King's College London, 4th Floor, Lambeth Wing St Thomas' Hospital London, London SE1 7EH, United Kingdom *Corresponding author: *wenfeng.xia@kcl.ac.uk This document provides supplementary information to "Seeing through multimode fibers with real-valued intensity transmission matrix The results of image retrievals through three different fibers with the same ground truth and varying total pixel counts. Top; Ø105 µm Some examples of image retrievals through a multimode fiber with 32×32-pixel random patterns. The fiber has a core diameter of 200 µm, a NA of 0 T.Z. and W.X. conceived of and performed the study, and wrote the paper. W.X. and T.V. supervised the project. S.O. provided scientific mentorship throughout the project. All the authors discussed the results and commented on the manuscript. The authors declare that there are no conflicts of interest. T.V. holds shares from Mauna Kea Technologies, Paris, France, which, however, did not support this work. The data and scripts that support the findings of this study are available from the corresponding author on reasonable request.