key: cord-0517437-9d0nau7w authors: Hosseinabadi, A. H. Hadi; Salcudean, S. E. title: Force Sensing in Robot-assisted Keyhole Endoscopy: A Systematic Survey date: 2021-03-20 journal: nan DOI: nan sha: 9398988e13643ffb58fb440017c43d6e6e9f263d doc_id: 517437 cord_uid: 9d0nau7w Instrument-tissue interaction forces in Minimally Invasive Surgery (MIS) provide valuable information that can be used to provide haptic perception, monitor tissue trauma, develop training guidelines, and evaluate the skill level of novice and expert surgeons.Force and tactile sensing is lost in many Robot-Assisted Surgery (RAS) systems. Therefore, many researchers have focused on recovering this information through sensing systems and estimation algorithms. This article provides a comprehensive systematic review of the current force sensing research aimed at RAS and, more generally, keyhole endoscopy, in which instruments enter the body through small incisions. Articles published between January 2011 and May 2020 are considered, following the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. The literature search resulted in 110 papers on different force estimation algorithms and sensing technologies, sensor design specifications, and fabrication techniques. In Minimally Invasive Surgery (MIS), surgical access is provided through small incisions or natural orifices in the body. A surgical instrument is operated by the surgeon for tissue manipulation. Compared to open surgery, MIS provides less tissue trauma, postoperative pain, patient discomfort, wound complications and immunological response stress Wottawa et al. (2016) , lower risk of infection Soltani-Zarrin et al. (2018) and blood loss Dai et al. (2017) , shorter hospital stay , faster recovery Lee et al. (2015) , and improved cosmetics Aviles et al. (2016) all of which lead to improved therapeutic outcome and efficiency Otte et al. (2016) and lower morbidity and morality Aviles et al. (2017) making MIS cost-effective Faragasso et al. (2014) . Nonetheless, the ergonomically cumbersome posture increases surgeon fatigue. The limited instrument dexterity and visual perception of the scene Haghighipanah et al. (2017) ; Haouchine et al. (2018) , and the non-intuitive hand-eye coordination due to fulcrum motion reversal decrease accuracy and contribute to surgeon fatigue Hadi Hosseinabadi et al. (2019) . The high level of psychomotor skills needed increases the operation time and require a longer learning curve Shahzada et al. (2016) . The sense of touch is reduced by friction in the access port and instrument mechanism. In Robotic MIS (RMIS), the surgical instrument is controlled by a robotic manipulator and operated by a remote surgeon. The robotic operation restores hand-eye coordination Aviles et al. (2015a) and innovations in tool design improve dexterity leading to improved ergonomics that reduce surgeon fatigue Stephens et al. (2019) ; . The enhanced 3D surgical vision, automatic movement transformations, fine motions, filtering of physiological hand tremor and motion scaling lead to improved surgery precision Sang et al. (2017) . However, the surgeon is isolated from the surgical site by robotic manipulators that do not provide the haptic perception Juo et al. (2020) . This deprives the surgeon of a rich source of information. Thus, many studies are targeted towards the reconstruction and evaluation of haptic feedback. Haptics can be either tactile or kinesthetic Okamura (2009) . The tactile perception is through the cutaneous receptors in the skin which can sense, for example, texture or temperature Mack et al. (2012) . The kinesthetic force feedback is perceived by mechanoreceptors in the muscle tendons to detect force, position and velocity information about objects Juo et al. (2020) . Traditionally surgeons use palpation to characterize tissue properties, detect nerves and arteries Bandari et al. (2017) , and identify abnormalities such as lumps and tumors Lv et al. (2020) ; Puangmali et al. (2012) . Moreover, the surgeons rely on the sense of touch to regulate the applied forces. Excessive forces can lead to tissue trauma, internal bleeding, and broken sutures. Insufficient forces however can lead to loose knots and poor sutures. Sang et al. (2017) ; Li et al. (2016) . Direct Force Feedback (FF) and Sensory Substitution (SS) are the most common approaches of presenting surgeons with force information. While the direct method provides the most intuitive interaction Li and Hannaford (2017) , it is the most challenging one to implement, as it requires a method of force sensing and a safe and robust teleoperation interface for force reflection. A compromise between transparency and stability of different teleoperation frameworks is reported by Hashtrudi-Zaad and Salcudean (2001) . In sensory substitution, visual, auditory, or vibrotactile signals provide haptic perception to the surgeon. While safety can be easily guaranteed, this method can cause discomfort, distraction, and cognitive overload. In general, visual methods are shown to be the most effective feedback modality. Abdi et al. (2020) In summary, the introduction of haptic perception is proven to decrease operation time Abiri et al. (2017) , facilitate training, improve accuracy, and enhance patient safety for novice surgeons in complex tasks Juo et al. (2020) . More experienced surgeons learn to infer force information from visual cues such as the tissue and instrument deformations and the stretch in sutures Aviles et al. (2017) . Additionally, force information can be used to automate surgical robot tasks in dynamic and unstructured environments Kuang et al. (2020) , to identify tissues in real time, to create tissue-realistic models and simulators for training Stephens et al. (2019) , and to perform surgical skills assessment Soltani-Zarrin et al. (2018) . An extensive review of haptic perception and its efficacy in RMIS is presented by Amirabdollahian et al. (2018) . This review concluded that while there is a consensus on the need for haptic and tactile feedback, no commercial system is yet available that addresses this need. More recently, El Rassi and El Rassi (2020) presented a brief overview of haptic feedback in teleoperated robotic surgery. Overtoom et al. (2019) and Rangarajan et al. (2020) surveyed virtual haptics in surgical simulation and training. The latter followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines to identify the relevant literature. The authors similarly affirm the efficacy of haptic feedback in surgical education. None of the publications above review the developments in the field of force sensing and estimation. Abdi et al. (2020) reviewed research since 2000 on the efficacy of haptic feedback in teleoperated medical interventions. The authors present a concise overview of the force-sensing literature with 44 references cited over a wide range of medical applications. Although the review provides a general understanding of the challenges and complexities in instrument-tissue force measurement, it is not a comprehensive presentation of the prominent developments and the articles were subjectively selected with no evaluation criteria. Additionally, the records were only classified based on the sensing technology and the sensor location; However, the instrument's dexterity level, the sensing Degrees of Freedom (DoFs), and the performance measures were not compared. A comparison of its references with the records cited in our review shows an overlap of only 20 out of 110 papers. reviewed tactile sensing literature over the past twenty years. It also includes some literature on force-sensing in neurosurgery and microsurgery procedures. Although the authors presented a comprehensive review with 121 references, a comparison of the included articles with the records in this paper shows an overlap of only 8 out of 110 papers which are mostly on developments related to the gripping force sensing. This article is a systematic review based on the PRISMA guidelines that expands on the sensor design requirements and presents the most recent developments in force sensing and estimation in keyhole endoscopy. We discuss how research has evolved over the past decade and provide suggestions for future research directions. The closest publications to our review are the surveys by Puangmali et al. Puangmali et al. (2008) and Trejos et al. Trejos et al. (2010) which were published about a decade ago, and therefore there are no overlapping papers with those reviews. A systematic survey was conducted by following the PRISMA guidelines (see Figure 1 ) and it was based on Google Scholar, Web-of-Science, PubMed, and IEEE Xplore Digital Library repositories. The period for the review is over the past decade, from January 2011 until May 2020. The following keywords were used for identification: Force sensing, Kinesthetic, Tactile, Haptics, Minimally Invasive Surgery, MIS, MIRS, Robot-Assisted, RMIS, RAS, RAMIS, Laparoscopy, and Endoscopy. For every year, the first 20 pages of search results in Google Scholar were surveyed (total of 2000 records). The same approach was used for the identification of records through the other repositories (PubMed: 213, and IEEEXplore: 40) . For screening, the duplicates were removed and the identified records were skimmed through to mark the ones that are relevant to keyhole endoscopy. The articles that refer to force sensing in microsurgery, neurosurgery, and needle insertion were excluded because they involve a different set of requirements and challenges. Specifically, microsurgical instruments such as those used in neurosurgery and retinal surgery Gonenc et al. (2017) have a much smaller diameter (less than 2 mm) and do not require an articulated wrist, which complicates the actuation system and sensors' power and signal routing. Moreover, briefly discussed the force-sensing literature in microsurgery, neurosurgery, and needle insertion. 114 articles were found eligible for a complete review. Throughout the review, the references of the selected papers were surveyed and the relevant articles that were not initially identified were added, thus increasing the total number of eligible records to 129. The work progressions and duplicate publications were removed to lead to the 110 articles included in this survey. The included articles are tabulated for an easier comparison of the method, the sensor location, the sensing DoFs, the dexterity of the instrument under study, and the results. The Dexterity Index (DI) for different instruments is defined according to the Table 1 and Figure 2 . Depending on the sensor location, the sensing DoFs are defined as instrument or wrist tri-axial forces (F X , F Y , F Z ) and moments (M X , M Y , F Z ), and the gripper normal (F N ), shear (F S ), and pull (F P ) forces as depicted in Figure 3 . In summarizing the results, the following acronyms were used: ACC: Accuracy, ERR: Maximum absolute error, MAE: Mean-Absolute-Error, NRMSE: Normalized-Root-Mean-Square-Error, RES: Resolution, RMSE: Root-Mean-Square-Error, RNG: Range, and SENS: Sensitivity. DoF, Range, Resolution, Accuracy, Bandwidth and Sampling Rate Grasping + Axial rotation 3 Grasping + Flexion 4 Grasping + Flexion + Axial rotation 5 Grasping + Flexion + Abduction 6 Grasping + Flexion + Abduction + Axial rotation . Deformations in the sensor structure or displacements in its components are the physical surrogates that are monitored for force estimation. Thus, there are always trade-offs between the sensor's structural rigidity, resolution and sensitivity, and range Puangmali et al. (2012 Kim et al. (2017) in the range of 0.5 to 200 N increasing to 15-27% below 0.5 N Hwang and Lim (2017) which can be considered as a requirement on the sensor accuracy. The human's temporal resolution is 320 Hz for force discrimination and up to 700 Hz for vibration detection Puangmali et al. (2012) . However, the desired bandwidth of the force sensor is usually dictated by the application (FF, SS, vibration detection, etc.) and desired noise and resolution performance. A sample rate of 500 Hz is considered appropriate for direct force feedback applications Jones et al. (2017) . Sample rates as low as 30 Hz can be effective in visual SS modality. Surgical instruments are inserted into the body through a cylindrical port of 12-15 mm in diameter Spiers et al. (2015) ; Li et al. (2015b) . The outside diameter of the instrument is desired to be less than 10 mm Shi et al. (2019) . The sensor should be lightweight to not significantly increase the instrument inertia. The operation rooms are filled with equipment that can cause electromagnetic interference, and the electrocautery tools operate at high voltages Lim et al. (2014) ; Seok et al. (2019) . Thus, the sensors require insulation for electrostatic protection and shielding against electromagnetic interference Peña et al. (2018) . The sensors that enter the body also require sealing against humidity and debris ingression Trejos et al. (2014) . Surgical instruments are cleaned and sterilized for reuse; the former refers to removing debris from the device and the latter is the elimination of microorganisms that can cause disease Trejos et al. (2014) . The common sterilization methods are plasma and gamma radiation, the use of chemicals (alcohol, ethylene oxide or formaldehyde), and steam sterilizations . Steam sterilization is the fastest and the most preferred method Soltani-Zarrin et al. (2018) which is performed in an autoclave at 120-135°C, 207 kPa and 100% humidity for 15-30 minutes Spiers et al. (2015) ; Zhao and Nelson (2015) . This harsh environment can be destructive to many transducers, signal conditioning electronics, wire insulations, bondings, and coatings. The sensors for use in MIS must abide by ISO 10993 which entails a series of standards for evaluating the biocompatibility of medical devices Trejos et al. (2014) . For biocompatibility electrical components often require coatings that interfere with sterilizability Spiers et al. (2015) . Instruments are disposed after 10 to 15 uses due to accelerated cable fatigue Anooshahpour et al. (2014) ; Kim et al. (2018a) ; Xue et al. (2018) . The EndoWrist instruments retail at $2k-$5k Spiers et al. (2015) . If the sensor is integrated into the instrument and is to be disposed, it should not increase the instrument price significantly. An adaptable solution that can be easily used on different instruments is desirable. The sensors can be placed in the instrument mounting interface, the instrument base, proximal (outside the body) and distal (inside the body) shafts, the actuation mechanism (cables/rod), the trocar mount and its distal end, the articulated wrist, and the gripper jaws ( Figure 4 ). While the size, sterilizability, biocompatibility, and insulation requirements are more relaxed for the sensors placed outside Lv et al. (2020) showed that they do not provide high resolution in the axial direction unless the structure is modified to amplify axial strains. The sensors in the instrument shaft and trocar cannot independently measure the gripping force He et al. (2014) and measuring cable tensions cannot provide information on the axial force. The sensors integrated into the trocar and instrument interface are usually adaptable Wang et al. (2014) . Sensors placed at the gripper jaw provide the most accurate readings and have the most stringent design constraints. They are difficult to fabricate, package, mount Kim et al. (2018b) ; He et al. (2014) , and shield Seok et al. (2019) ; Wang et al. (2014) and have limited adaptability which makes them cost-prohibitive for disposable instruments Xue et al. (2019) . Additionally, the electronics are usually placed away from the transducer which deteriorates the Signal to Noise Ratio (SNR) Suzuki et al. (2018) . Placing the force sensor at the grasper may also conflict with functional requirements for monopolar or bipolar cautery instruments He et al. (2014) . Figure 5 summarizes the severity level of different sources that contribute to the sensing inaccuracy as a function of the sensor location (scale of 1 to 3; 1 is minimum, 3 is maximum and is no effect). It also compares how stringent the listed design requirements are for each sensor location (scale of 1 to 3; 1 is the least, 3 is the most and refers to not a requirement). The distribution of the records included in this survey as a function of the sensor locations and the sensing technologies are shown in the same figure. It is evident that the sensorless techniques have the majority of publications over the past ten years. Additionally, the Micro-Electro-Mechanical (MEM) and Fiber Brag Grating (FBG) technologies have been widely adopted in the fabrication of miniature transducers that can be integrated into the gripper jaws. An overview of different transduction technologies is presented in the next section. Sensing technologies and the corresponding number of articles in this survey is shown in Figure 6 . Sensorless refers to the case where the sensors used for force estimation are already inherent in the surgical robot Stephens et al. (2019) . In model-based approaches, the sensors are the encoders and the motor current measurements. In the visionbased techniques, the sensor is the visual feedback of the surgical site through mono or stereo cameras. Model-based techniques can be categorized into 1) analytical models developed based on first principles, 2) disturbance observers and Kalman filters that utilize a dynamic model and the control loop commands and feedback signals, and 3) data-driven models which consider the instrument as a black-box and fit a quantitative model to a customized set of input and outputs. The model-based literature is summarized in Tables 2 and 3. They include 15 analytical models, eight of which are physics-based and 7 studies use observers or Kalman filters. There are 5 articles on the use of data-driven models. The accurate dynamic model of the surgical instruments is challenging to obtain due to the many sources of nonlinearities e.g. friction, backlash Sang et al. (2017) , tendons compliance Anooshahpour et al. (2014) and creep Haghighipanah et al. (2017) , elastic deformations, actuators performance variations (the motors' brush conductivity and change in the armature winding resistance) Li et al. (2016) , hysteresis Anooshahpour et al. (2016) , inertia, and gravity Wang et al. (2014) . Additionally, any model relies on a set of measurements (calibration or training set) that are usually taken at the beginning and used throughout the estimation. It is experimentally shown that the tool behavior changes with time which deteriorates the estimation accuracy Kong et al. Proposed a three step Robust Reaction Torque Observer (RRTO): 1) Cancel the error in estimated torque (overshoot correction) 2) Identify and compensate the inertial torque component in the drive train (inertia compensation) 3) Estimate the gripping torque. Plot comparison, not quantified. 3-Anooshahpour et al. Proposed two quasi-static models on cable dynamics (Pull & Pull-Push) that take tendons friction & compliance into account. A linear combination of the two models provided a close estimation of the output gripping torque. Grip torque EndoWrist Needle Driver / 6 Plot comparison, not quantified. The gripper was actuated by a pneumatic catheter balloon to provide a uniform gripping force. The pneumatic pressure was monitored to estimate the grip force. (2015) A wrist actuation design using planetary gears was proposed to decouple the motions in different DoFs. The motor currents were used to estimate the forces in static and dynamic scenarios. Proposed the compensation of the gripping torque which was experimentally identified as a function of the instrument posture. The experiments were on: EndoWrist 1) ProGrasp 2) Large Needle Driver 3) Dissecting forceps. EndoWrist instruments / 6 1) ERR < 10.69% 2) ERR < 13.03% 3) ERR < 16.25% The articulated wrist was replaced by a machined spring. The instrument was pneumatically driven. A 3-DoF continuum model of the spring distal joint and the pneumatic pressure were used for force estimation. Forceps tri-axial Proposed the use of Sliding Perturbation Observer (SPO) to estimate the reaction force. The presented method compensated for the Coulomb friction. Evaluated two approaches for force estimation on the 3 rd link of the Raven-II system: 1) Further expanded on the approach in Li et al. (2016) by adding cable tension estimation. 2) The force was estimated by measuring the cable stretch using a linear encoder. Raven-II 10 mm gripper / 6 ERR 1) < 4N, 2) < 3N RNG: 0-10N #2 Provided better estimation at lower forces 14- Li and Hannaford (2017) Used the Gaussian Progress Regression (GPR) supervised learning approach because of its ability to deal with uncertainties and nonlinearity. The model inputs were motors encoder, velocity, and current. Raven-II 10 mm gripper / 6 Developed the dynamic model of one jaw by using the Benson model to describe the dry friction. The parameters were experimentally identified for an instrument designed based on the concept in Zhao and Nelson (2015) . Evaluated motor current command and measurement, and differential gearbox as proximal torque surrogates and used Neural Networks (NNs) to estimate the distal gripping torque considering all three surrogates as inputs. (2019) The performance of NNs, decision tree, random forest, and support vector machine models were compared in the angle and gripping torque estimation of each jaw. It concluded that the NN estimations were reliable when trained and tested on each jaw, on the same tool, and withing the frequency of the training data. Lee et al. (2015) showed that for the same input force by the surgeon, the grip force of the daVinci EndoWrist grasper can vary up to 3.4 times depending on its posture. As a result, despite the extensive research work, force estimations that rely on dynamic models do not provide highly reliable results yet, especially in the instrument's lateral direction Fontanelli et al. (2017) . In comparison, the data-driven techniques based on supervised learning Li and Hannaford 2019)) with the video stream as inputs to estimate the instrument-tissue interaction forces. The vision-based techniques are robust to many sources of inaccuracy listed in Figure 5 . However, they can be affected by the instrument occlusion, smoke and changes in the tissue properties, lighting conditions and camera orientation. The estimation update rate cannot be faster than the video frame rate which is usually 30 Hz. This limitation makes the vision-based approached not suitable for FF applications in which the control loop is desired to execute faster than 500 Hz Jones et al. (2017) . The current literature highlights that force estimation through video processing is easier in pushing tasks (characterized by smooth deformations) than those produced by pulling tasks that are characterized by irregular tissue deformations due to grasping Marban et al. (2019) . Force estimation based on using Optical Coherence Tomography (OCT) as the reference sensor is proposed by Otte et al. (2016) and Gessert et al. (2018) . OCT images provide volumetric data with a resolution of a few micrometers in which the tissue compression and subsurface deformations can be reflected. Thus, they contain a richer (2011) Image processing algorithms were utilized for suture strain estimation by identifying the suture line and tracking the displacement of markers. The achieved resolution in strain estimation was two orders of magnitudes smaller than the known strain to failure of most suture materials (20+%). / Suture pull Force magnitude Mono Strain resolution of 0.2% and 0.5% was achieved in one-marker tracking on stationary suture and twomarker tracking on moving suture, respectively. Kim et al. (2012) The soft-tissue deformation was obtained by processing the stereoscopic depth image as a surface mesh. It was compared against the original organ shape from preoperative images. A spring damper model was used for force estimation. / Tissue push and pull Gripper tri-axial forces No results were presented 3-Noohi et al. (2014) A virtual template, based on assuming soft tissue local deformation to be a smooth function, was used to estimate the tissue deformation without a-priori knowledge of its original shape. The force magnitude was estimated by using a biomechanical model. / Tissue push Gripper tri-axial forces In force magnitude: This was an extension to Aviles et al. (2016) where the proposed RNN+LSTM architecture was extended to three-axis force components. Z-axis was normal to the tissue, X and Y axes were planar with the tissue surface. signal space compared to the mono and stereo visions that provide only the surface information. Strain gauges are the most commonly used transducers for force sensing Yu et al. (2018a) . They are accurate and small and can be designed in different configurations for multi-axis force sensing. Although the transducers are low cost with a price of $10-$25 per unit Srivastava et al. (2016) , they require special surface preparation, adhesives, and coatings for optimal performance that increases the assembly and integration cost Li et al. (2017b) . When used in Wheatstone bridge arrangements, they require multiple wires for connection that makes packaging difficult for quick and seamless integration with surgical instruments Shi et al. Trejos et al. (2014 Trejos et al. ( , 2017 conducted an extensive study on biocompatible adhesives and coatings that can withstand the harsh environment during steam sterilization. However, none of the combinations showed reliable measurements after seven cycles. Tables 6 to 8 summarizes the articles which (2017) It is similar to the overcoat method by Shimachi et al. (2003) . The instrument was mounted to an inner tube that was attached to a 6 axis F/T sensor. / Instrument interface Optical methods use light intensity (e.g. photodiodes, phototransistors), frequency (e.g. Fiber Brag Gratings), or phase (e.g. interferometry) modulation for force measurement. The optical signal can be locally converted to electric signals, or be transferred with fibers for distal processing. Placing the electronics away from the instrument tip makes sterilization easier. The optical fibers are flexible, scalable, biocompatible, electrically passive, insensitive to electromagnetic noise and thus MRI compatible Peña et al. (2018) , durable against high radiation Lim et al. (2014) , immune to water Song et al. (2014) , corrosion-resistant Shi et al. (2019) , and low cost . However, optical fibers cannot be routed into small bending radii Trejos et al. (2014) . Additionally, The presence of small and intricate parts can make fabrication and assembly of fiberbased sensors costly Li et al. (2017b) . The Light Intensity Modulation (LIM) based sensors are vulnerable to light intensity variations due to the temperature or fiber bending Lv et al. (2020) . This can be improved by normalizing the optical signal against the emitted power Hadi Hosseinabadi et al. (2019) . Alternatively, a redundant strain-free fiber can be used to compensate for the effect of temperature or other sources of uncertainty Puangmali et al. (2012) ; Song et al. (2014) . Table 9 summarizes the articles that address MIS force estimation based on LIM. The FBG sensors are wave-length coded and insensitive to the changes in the light intensity. FBGs are very sensitive, have calibration consistency, and exhibit high SNR which provide repeatable and high-resolution strain measurements Shahzada et al. (2016) . Multiple gratings can be accommodated into one fiber Soltani-Zarrin et al. (2018) simplifying the design and signals routing. Thus, they are also used in shape sensing Lv et al. (2020) . Nonetheless, FBGs require interrogators for signal processing which the commercial systems cost between $10k to $100k Yurkewich et al. (2014) . The articles which used FBGs for MIS force estimation are summarized in Table 10 . Capacitive methods are attractive solutions for high resolution and compact force sensor designs. Compared to strain gauges, they provide limited hysteresis in microscale and increased sensitivity Sang et al. (2017) . However, they have a limited range Li et al. (2017b) and are prone to thermal and humidity drift . The change in capacitance can be due to the change of the overlapping area or the distance between the two electrodes; the latter provides higher sensitivity and a more linear response Kim et al. (2017) . The commercially available Capacitance to Digital Converter (CDC) chips such as the AD7147 from Analog Devices significantly simplify the signal processing, which was believed to be challenging for capacitive transducers Trejos et al. (2014) . However, they provide a low sampling rate. Table 11 lists the articles that are based on the capacitive transduction principle. The Two optical FBGs were integrated into the forceps. Each fiber had two gratings for measuring the mechanical strain (on the surface) and for temperature compensation (at the center of the bending neutral axis). / Gripper The force sensor had a miniature flexure based on a Sarrus mechanism to achieve high axial sensitivity and a large measurement range. Two sensors consisting of a triangular prism shape and two capacitive-type transducers with an elastomeric polymer dielectric were integrated into the grasper. Molding was used to fabricate a prototype. / Gripper (tip) Gripper normal (F N ), shear (F S ), pull (F P ), grip (F G ) forces CD RMIS instrument for RAVEN-II / 6 RES: Two sensors with 3 electrodes and common grounds were integrated into the Gripper. The dielectric was air and the signal processing electronics were local. Extension of Kim et al. (2018b) with humidity compensation. An AC shield was added to minimize the temperature effects on parasitic capacitance. An anodizing process was applied for electric insulation. The sensors range was extended to 20 N. / Gripper (base) Gripper normal (F N ), shear (F S ), pull (F P ), grip (F G ) forces, axial torque (T P ) CD RMIS instrument for S-surge robot / 6 RNG: ±20 N sterilizability and biocompatibility of the existing literature are not evaluated. MEM sensors (Table 12 ) operate based on the same physical principles discussed so far. However, MEM fabrication techniques such as deposition, etching, and lithography allow for the cost-effective production of small, fully-integrated, monolithic sensors Radó et al. (2018) with reduced lead time in prototypes and high throughput batch volumes Gafford et al. (2014) . Typically, MEMS do not require manual assembly, bonding, and alignment, and provide functional devices after the fabrication process Gafford et al. (2013) . By utilizing MEMS technology, it is possible to develop smart parts (e.g. grippers) with integrated sensing capability for micromanipulation Pandya et al. (2014) . Biocompatible coatings can be added to MEM sensors for biomedical applications. Piezoelectric transducers do not require an external power supply and have high stiffness Li et al. (2017b) . However, they are subject to charge leakage and are not suitable for low frequency and static loads Sang et al. (2017) . They are also sensitive to temperature. Piezoresistive transducers used in force-sensitive resistors are scalable with low hysteresis and noise Bandari et al. (2017) . Nonetheless, their linear response is limited to a small range and they drift under constant load Juo et al. (2020) . They do not have the challenges associated with the integration of strain gauges, are relatively insensitive to humidity, and can be used in high temperatures above 170°C Li et al. (2017b) . Shape Memory Alloys (SMAs) like Nitinol have a higher gauge factor compared to common metallic strain gauges and provide a larger range due to their stretchability. SMAs require an insulating coating for use on conductive surfaces. They can be clamped at two ends and do not require a backing material with special surface preparation. They are low cost and available at diameters as small as few microns Srivastava et al. (2016) . Quantum tunneling composite (QTC) pills are flexible polymers that act as insulators in resting-state but increase conductivity when compressed. They are very sensitive, provide a wide dynamic range, and are low cost (less than 1$/pill). However, they are temperature sensitive and inaccurate in dynamic loading applications Novoseltseva (2018). Recently, vibration frequency and phase shifting due to an applied force have been measured for force estimation by the use of accelerometers. This approach is slow as it needs a few vibration cycles to generate stable and repeatable signals Kuang et al. (2020) . In keyhole endoscopy, the surgeon's interaction with the surgical site is via slender instruments that are inserted into the body through small incisions. Despite the many benefits to the patient, the operation is more challenging for the surgeon due to the instruments' limited dexterity, fulcrum motion reversal, uncomfortable posture, and limited visual presentation. Additionally, the surgeon's force perception is affected by the forces between the instrument and the skin and the instrument's dynamics. The adoption of robotic and computer vision technologies resolves the limitations above and significantly improves the accuracy and efficiency in RMIS. However, most telesurgical systems completely isolate the surgeon from the tissue through the local/remote architecture of robotic telemanipulation. This deprives the A slender shaft was excited by using a vibration motor. The structure's triaxial acceleration signals in time-domain showed discernible ellipse-shaped profiles when a force was applied. The acceleration profiles were characterized via regression to estimate the direction and magnitude of the applied force. surgeon of the rich information in palpation and direct interaction with the tissue. Without force feedback, the interaction of the surgeons with the environment is not as intuitive as direct manipulation and therefore extensive training is required. Moreover, the lack of haptic feedback leads to a higher risk of errors and longer task completion time, up to 2 orders of magnitude in complex tasks Hannaford et al. (1991) , which may lead to higher surgery costs. One active research stream in the field of robotic surgery is improving the sense of telepresence for the surgeons, also known as "transparency". Direct force feedback is the most intuitive approach to improve transparency. For a fully transparent haptic experience, reliable interaction force sensing at the surgeon console and the instrument-tissue interface is required. This is in addition to a safe bilateral teleoperation architecture, and a local manipulator that is capable of reflecting the force commands, known as a haptic display. The extensive literature on haptic control indicates a trade-off between transparency and stability Hashtrudi-Zaad and Salcudean (2001) . Alternatively, sensory substitution was proposed instead of haptic feedback, in the form of visual, auditory, or vibrotactile cues of force information. Although the safety can be easily guaranteed in systems with SS, it is not intuitive and can cause cognitive overload for the surgeon. The SS methods can also be used in MIS systems because no robotic manipulator is required for force reflection. The efficacy of different haptic feedback modalities in improving the surgical training and surgeon performance metrics has been studied extensively Amirabdollahian et al. (2018) ; Rangarajan et al. (2020) ; Overtoom et al. (2019) ; Abdi et al. (2020) . It is shown that a transparent haptic experience and visual feedback of force information improve the performance metrics and shorten the training time for novice surgeons in complex tasks. Apart from haptic feedback, the instrument-tissue interaction forces can be used for tissue damage monitoring, surgical skills assessment, development of surgical training guidelines, and to automate tasks. Extensive research has been conducted to estimate or sense the instrument-tissue interaction forces. The functional requirements depend on the application. While it is not necessary to estimate the tissue forces precisely to provide an appropriate haptic experience Jones et al. (2011) , the bandwidth and sampling rate are important requirements to ensure low latency and smooth interaction with the remote environment. The sampling rate and bandwidth are less critical in SS. Sensorless approaches utilize the information that is already available in the robotic manipulator; the axes positions and velocities, motors currents, and visual display of the surgical theatre. With the exponential growth, over the past decade, in the available computational power to researchers, data-driven approaches based on supervised learning Li and Hannaford (2017) ; Aviles et al. (2017) have been widely adopted. Among them, neural networks have shown promising results when trained and used on one particular instrument. However, they require a long and computationally-expensive training phase that is yet clinically-prohibitive. The training is based on a set of measurements at the beginning of the surgery that is used afterward for force estimation throughout the entire surgery. Proposed approaches that have an instrument's operational parameters as inputs, do not consider the variations between instruments and the change of instruments behavior throughout its use Kong et al. (2018) . Considering how the research direction has evolved over the past decade, experimentation with different model architectures, development of efficient training, and identification methods that can be automatically performed at the system start-up Spiers et al. (2015) , improving the computation time and incorporation of online adaptation techniques are attractive research areas to be further investigated. Moreover, all the existing literature uses the information at the patient manipulator for force estimation, but the inclusion of the operating parameters at the surgeon console may also improve the quality of force estimation. The sensor design is another avenue towards collecting force data at the instrument tip. The sensor can be located inside or outside the body. The closer the sensor is to the instrument tip the more accurate the measurements are. However, the size, biocompatibility, sterilizability, insulation, and sealing requirements are more stringent when such an approach is followed. Design proposals for sensor integration into the instrument tip have limited adaptability because the instruments for different types of surgery have different shapes at the tip (e.g. EndoWrist cautery forceps, graspers, dissectors, needle drivers, etc.). Therefore, a custom sensor needs to be designed for every instrument which increases the development, fabrication, and maintenance costs. A variety of transduction principles, including resistive, capacitive, optical, piezoelectric, and magnetic have been used in the development of sensing solutions for minimally invasive procedures. While strain gauges are still the most commonly used transducers, the study by Trejos et al. Trejos et al. (2017) showed that biocompatible adhesives and coatings can only survive a maximum of 6 steam sterilization cycles. Considering that the instruments are typically used 10 times before disposed, this would lead to a 40% increase in the cost of the instruments with integrated strain gauges. Additionally, the installation of strain gauges is labor-intensive that contributes to increased cost. A comparison of the publications summarized in this article with the surveys by Puangmali et al. (2008) and Trejos et al. (2010) indicates a noticeable shift towards utilizing FBG and MEMS technologies for the development of gripper integrated miniature sensors ( Figure 5 ). FBGs are compact, sterilizable, biocompatible, electrically passive, and immune to electromagnetic noise. They provide high sensitivity with sub-micron resolution and can have multiple gratings embedded in one fiber which simplifies fiver and therefore optical signal management. While the commercial interrogators are expensive, there are signal conditioning solutions proposed to decrease the electronics cost Yurkewich et al. (2014) . The developments in MEMS technology have overcome the barrier of scale and cost in the fabrication of delicate miniature sensors. Additionally, MEMS sensors typically do not need manual assembly and can be integrated into the desired application after production. Another observable trend is the utilization of datadriven regression approaches for sensor calibration. Models based on neural networks and other supervised learning methods such as Gaussian Process Regression have shown unprecedented performance in handling nonlinearities and uncertainties in sensor calibration. Compared to the surgical instruments, the transducers show a more consistent response and do not need regular calibration unless removed and reintegrated. Efficient calibration approaches that can be quickly and automatically performed without operator intervention (e.g based on payload estimation) would benefit the RMIS systems. Haptics in Teleoperated Medical Interventions: Force Measurement, Haptic Interfaces and Their Influence on Users Performance Estimation of Tool-Tissue Forces in Robot-Assisted Minimally Invasive Surgery Using Neural Networks Multi-Modal Haptic Feedback for Grip Force Reduction in Robotic Surgery Visual-Perceptual Mismatch in Robotic Surgery Prevalence of Haptic Feedback in Robot-Mediated Surgery: A Systematic Review of Literature Quasi-Static Modeling of the da Vinci Instrument Classical Preisach Model of Hysteretic Behavior in a da Vinci Instrument Sensorless Force Estimation Using a Neuro-Vision-Based Approach for Robotic-Assisted Surgery Exploring the Effects of Dimensionality Reduction in Deep Networks for Force Estimation in Robotic-Assisted Surgery Towards Retrieving Force Feedback in Robotic-Assisted Surgery: A Supervised Neuro-Recurrent-Vision Approach Force-Feedback Sensory Substitution Using Supervised Recurrent Learning for Robotic-Assisted Surgery A Recurrent Neural Network Approach for 3D Vision-Based Force Estimation Miniature Tri-Axial Force Sensor for Feedback in Minimally Invasive Surgery Miniaturized Optical Force Sensor for Minimally Invasive Surgery with Learning-Based Nonlinear Calibration Tactile Sensors for Minimally Invasive Surgery: A Review of the State-of-the-Art, Applications, and Perspectives Hybrid Piezoresistive-Optical Tactile Sensor for Simultaneous Measurement of Tissue Stiffness and Detection of Tissue Discontinuity in Robot-Assisted Minimally Invasive Surgery Real-Time Measurement of the Tool-Tissue Interaction in Minimally Invasive Abdominal Surgery: The First Step to Developing the Next Generation of Smart Laparoscopic Instruments Three-axis Force Sensor With Fiber Bragg Grating Grasper Integrated Tri-axial Force sensor System for Robotic Minimally Invasive Surgery Providing Haptic Feedback in Robot-Assisted Minimally Invasive Surgery: A Direct Optical Force-Sensing Solution for Haptic Rendering of Deformable Bodies A review of Haptic Feedback in Tele-Operated Robotic Surgery Novel Uniaxial Force Sensor Based on Visual Information for Minimally Invasive Surgery A novel force sensing integrated into the trocar for minimally invasive robotic surgery A monolithic Approach to Fabricating Low-Cost, Millimeter-Scale Multi-Axis Force Sensors for Minimally-Invasive Surgery Force-Sensing Surgical Grasper Enabled by Pop-up Book MEMS A Novel Pneumatic Force Sensor for Robot-Assisted Surgery Force estimation from OCT volumes using 3D CNNs Vision-based Deformation Recovery for Intraoperative Force Estimation of Tool-Tissue Interaction for Neurosurgery 3-DOF Force-Sensing Motorized Micro-Forceps for Robot-Assisted Vitreoretinal Surgery Optical Force Sensing In Minimally Invasive Robotic Surgery Utilizing Elasticity of Cable-Driven Surgical Robot to Estimate Cable Tension and External Force Vision-Based Force Feedback Estimation for Robot-Assisted Surgery Using Instrument-Constrained Biomechanical Three-Dimensional Maps A Pneumatically Driven Surgical Manipulator with a Flexible Distal Joint Capable of Force Sensing Analysis of Control Architectures for Teleoperation Systems with Impedance/Admittance Master and Slave Manipulators A Fiberoptic Force-Torque Sensor for Minimally Invasive Robotic Surgery Force Sensing of Multiple-DOF Cable-Driven Instruments for Minimally Invasive Robotic Surgery Design and evaluation of 2-DOF compliant forceps with force-sensing capability for minimally invasive robot surgery Grip Force Estimation of Laparoscope Surgical Robot based on Neural Network Optimized by Genetic Algorithm Inferring Interaction Force from Visual Information without Using Physical Force Sensors Development of a Standalone Surgical Haptic Arm A Soft Multi-Axial Force Sensor to Assess Tissue Properties in Real-Time Center for Advanced Surgical and Interventional Technology Multimodal Haptic Feedback for Robotic Surgery Implementation of Surgical Robot Instrument with a Force Feedback A Modular Force-Controlled Robotic Instrument for Minimally Invasive Surgery -Efficacy for Being Used in Autonomous Grasping Against a Variable Pull Force A 3-DOF Sensor to Estimate the Force Applied to the Tip of a Surgical Instrument Development of Surgical Forceps Integrated with a Multi-axial Force Sensor for Minimally Invasive Robotic Surgery A Surgical Palpation Probe with 6-Axis Force/Torque Sensing Capability for Minimally Invasive Surgery Sensorized Surgical Forceps for Robotic-Assisted Minimally Invasive Surgery Force Sensor Integrated Surgical Forceps for Minimally Invasive Robotic Surgery Image-Based Force Estimation of Deformable Tissue Using Depth Map for Single-Port Surgical Robot Da Vinci tool torque mapping over 50,000 grasps and its implications on grip force estimation accuracy Vibration-Based Multi-Axis Force Sensing: Design, Characterization, and Modeling A Grasping Forceps with a Triaxial MEMS Tactile Sensor for Quantification of Stresses on Organs Pneumatic-Type Surgical Robot End-Effector for Laparoscopic Surgical-Operation-by-Wire A Grip Force Model for the da Vinci End-effector to Predict a Compensation Force A Laparoscopic Grasping Tool with Force Sensing Capability Preliminary Design of Multi-Axial Contact Force Sensor for Minimally Invasive Robotic Surgery Grasper Achieving Haptic Perception in Forceps Manipulator Using Pneumatic Artificial Muscle Miniature 6-axis Force/Torque Sensor for Force Feedback in Robot-Assisted Minimally Invasive Surgery Design and Performance Evaluation of a 3-Axis Force Sensor for MIS Palpation A novel 4-DOF Surgical Instrument with Modular Joints and 6-Axis Force Sensing Capability Development of an Inexpensive Tri-Axial Force Sensor for Minimally Invasive Surgery Gaussian Process Regression for Sensorless Grip Force Estimation of Cable-Driven Elongated Surgical Instruments Dynamic Modeling of Cable Driven Elongated Surgical Instruments for Sensorless Grip Force Estimation Grip Force Measurement of Forceps with Fibre Bragg Grating Sensors A High-Precision and Miniature Fiber Bragg Grating-Based Force Sensor for Tissue Palpation During Minimally Invasive Surgery Interactive Force-Sensing Feedback System for Remote Robotic Laparoscopic Surgery Scaling Method for Force Feedback of Forceps Manipulator Based on Beam Theory Estimation of Interaction Forces in Robotic Surgery using a Semi-Supervised Deep Neural Network Model A Recurrent Convolutional Neural Network Approach for Sensorless Force Estimation in Robotic Surgery Visual Measurement of Suture Strain for Robotic Surgery Single-Use Haptic Palpation Probe for Locating Subcutaneous Blood Vessels in Robot-Assisted Minimally Invasive Surgery An Actuated Force Feedback-Enabled Laparoscopic Instrument for Robotic-Assisted Surgery MEMS 6-axis Force-Torque Sensor Attached to the Tip of Grasping Forceps for Identification of Tumor in Thoracoscopic Surgery Using Monocular Images to Estimate Interaction Forces During Minimally Invasive Surgery Force Feedback for the Patient Side Manipulator of the daVinci Research Kit Haptic Feedback in Robot-Assisted Minimally Invasive Surgery Evaluation of Torque Measurement Surrogates as Applied to Grip Torque and Jaw Angle Estimation of Robotic Surgical Tools Towards Force Sensing Based on Instrument-Tissue Interaction Haptic Feedback, Force Feedback, and Force-Sensing in Simulation Training for Laparoscopy: A Systematic Overview MEMS Based Low Cost Piezoresistive Microcantilever Force Sensor and Sensor Module Printing Strain Gauges on Intuitive Surgical da Vinci Robot End Effectors State-of-the-Art in Force and Tactile Sensing for Minimally Invasive Surgery Miniature 3-axis Distal Force Sensor for Minimally Invasive Surgical Palpation 3D force sensors for laparoscopic surgery tool Actual Reaction Force Separation Method of Surgical Tool by Fuzzy Logic Based SMCSPO Systematic Review of Virtual Haptics in Surgical Simulation: A Valid Educational Tool? A Novel Device for Measuring Forces in Endoluminal Procedures External force estimation and implementation in robotically assisted minimally invasive surgery A Force-Sensing Surgical Tool with a Proximally Located Force/Torque Sensor Prototype of Instrument for Minimally Invasive Surgery with 6-Axis Force Sensing Capability Disposable Force Sensing Clip for Robotic Surgical Instruments Compensation of Environmental Influences on Sensorized-Forceps for Practical Surgical Tasks Optical Fibers and Sensors for Medical Diagnostics and Treatment Applications XVI High-Sensitivity Fiber Bragg Grating-Based Distal Force Sensor for Laparoscopic Surgery Measurement of Force Acting on Surgical Instrument for Forcefeedback to Master Robot Console Development of a 2-DOF Sensorized Surgical Grasper for Grasping and Axial Force Measurements Optical Fiber Bragg Grating (FBG) Force Reflection Sensing System of Surgical Tool for Minimally Invasive Surgery Investigating Remote Sensor Placement for Practical Haptic Sensing with EndoWrist Surgical Tools Design of an Ultra Thin Strain Sensor Using Superelastic Nitinol for Applications in Minimally Invasive Surgery Conditions for Reliable Grip Force and Jaw Angle Estimation of da Vinci Surgical Tools Development and Testing of Force-Sensing Forceps Using FBG for Bilateral Micro-Operation System Experimental Characterization of a Tactile Sensor for Surgical Applications Information Technology, Communication and Control, Environment and Management Grasping Force Estimation in Robotic Forceps Using a Soft Pneumatic Actuator with a Built-in Sensor A Dual-Arm 7-Degrees-of-Freedom Haptics-Enabled Teleoperation Test Bed for Minimally Invasive Surgery A Sterilizable Force-Sensing Instrument for Laparoscopic Surgery Design and Evaluation of a Sterilizable Force Sensing Instrument for Minimally Invasive Surgery Force Sensing and Its Application in Minimally Invasive Surgery and Therapy: A Survey Estimation of the Grasping Torque of Robotic Forceps Using the Robust Reaction Torque Observer Design and Testing of a Pressure Sensing Laparoscopic Grasper Indirect Measurement of Pinch and Pull Forces at the Shaft of Laparoscopic Graspers Design of a Slave Arm of a Surgical Robot System to Estimate the Contact Force at the Tip of the Employed Instruments A Clamping Force Estimation Method Based on a Joint Torque Disturbance Observer Using PSO-BPNN for Cable-Driven Surgical Robot End-Effectors External Force Self-Sensing based on Cable-Tension Disturbance Observer for Surgical Robot End-Effector Force-Sensing Sleeve for Laparoscopic Surgery Novel Force-Sensing System for Minimally Invasive Surgical Instruments Evaluating Tactile Feedback in Robotic Surgery for Potential Clinical Application Using an Animal Model Model-Based Grasp Force Estimation for Minimally Invasive Surgery An Estimation Method of Grasping Force for Laparoscope Surgical Robot Based on the Model of a Cable-Pulley System Design and Evaluation of FBG-Based Tension Sensor in Laparoscope Surgical Robots Sliding Perturbation Observer Based Reaction Force Estimation Method of Surgical Robot Instrument for Haptic Realization Design and Static Calibration of a Six-Dimensional Force/Torque Sensor for Minimally Invasive Surgery External Force Sensing Based on Cable Tension Changes in Minimally Invasive Surgical Micromanipulators Design and Realization of Forceps with 3-D Force Sensing Capability for Robot-Assisted Surgical System Low-Cost Force-Sensing Arthroscopic Tool Using Threaded Fiber Bragg Grating Sensors Sensorless Force Sensing for Minimally Invasive Surgery Amir Hossein Hadi Hosseinabadi would like to NSERC for his Canada Graduate Scholarship-Doctoral scholarship. Professor Salcudean gratefully acknowledges infrastructure support from CFI and funding support from NSERC and the Charles Laszlo Chair in Biomedical Engineering.