See beyond the visible in endoscopy
through augmented perception, reasoning, modelling and action
Coordinator (person and lab): Sandrine Voros, TIMC-IMAG / CAMI group
CAMI Partners : LTSI laboratory (Rennes)
Started: October 2016 (1st recruitment)
Motivations and objectives:
Surgical endoscopy is a minimally invasive surgical technique for surgeries on abdominal organs. This type of surgery offers numerous benefits for the patient (less bleeding, transfusions, shorter intervention time), but it increases the complexity of the surgeon’s task, since he doesn’t have a direct vision of the operative site. To assist the surgeon in his surgical gesture, two types of robotic systems have been developed: robotic endoscope holders (such as Endocontrol’s ViKY® robot), and complete telesurgery systems (such as Intuitive Surgical’s da Vinci ™ robot), which offer a comfort close to open surgery in a minimally invasive environment.
Today, only the da Vinci™ system is used on a large scale in hospitals (~5000 systems installed worldwide as of 2017), however, its clinical added value for the patient remains controversial compared to its cost [1].
On the other hand, CAMI systems allow to compute the distance between the performed surgical gesture and the surgical planning (e.g. control the trajectory of a drilling tool), and thus offer the opportunity to actually quantify the Delivered Medical Service to patients. While these distance measurements and their link to clinical outcomes are relatively easy to make for bone surgery (in short, planning can be made on pre-operative images, and the tools trajectories are somewhat simple), surgical endoscopy is a more complex case. Indeed, it concerns soft tissues, so if pre-operative data is available, it rapidly becomes false; moreover, defining the “good clinical practice” is more complex than comparing the actual (often straight) trajectory of a drilling tool to its planned trajectory.
4 needs must be fulfilled to provide a CAMI system for surgical endoscopy
-
Capture in real-time the position of the surgeon’s instruments and information about the targets of the surgical gesture (anatomical structures)
-
Transform this information into metrics that characterize the surgical practice
-
Establish quantified definitions of the “good surgical practice” that the surgeon wants to apply to treat his patient optimally
-
Provide the surgeon with user interfaces allowing him to visualize the information that will help him minimize the distance between his surgical practice and the gold standard practice(s)
We (TIMC-IMAG) have ongoing projects aiming at offering solutions for those needs in the context of surgical endoscopy.
Regarding augmented perception and reasoning, we have proposed a method for image-based tool tracking which won the 2015 MICCAI EndoVis Grand Challenge (tool tracking category). We have also developed a prototype of a “global vision system” (GVS) which adds mini-cameras to the classical endoscope, in order to increase the laparoscopic field of view and help the surgeon better visualize the surgical scene.
Lastly we are currently working on devices and methods to allow fusion of laparoscopic and ultrasound images during radical prostatectomy (one of the most challenging urologic interventions).
These devices and methods will allow to solve needs 1) and 2) with a better perception of the environment and a fused environment where metrics can be extracted in a single referential to capture surgical gestures in their context.
Regarding need 3), we currently work on more upstream activities led in collaboration with the LTSI in Rennes concerning the modelling of surgical procedures to better understand and define “good surgical practice”.
Our objective in the frame of this integrated project is to foster on those existing bricks in order to provide bimodal navigation systems (GVS or ultrasound + endoscopic images) that are ready for -clinical evaluations. Regarding the modelling of surgical procedures, our objective is to demonstrate that exploiting such augmented information in addition to the workflow of surgical activities can enhance the understanding of surgical expertise.
Main results and on-going work
A postdoctoral fellow (Sinara Vijayan, recruited in July 2017) will work on the bimodal navigation systems. All the hardware for the GVS has been developed and is available. The main challenge will be to conceive and perform the pre-clinical experiments mandatory for a risk analysis and for the preparation of a first clinical trial. Anticipated bottleneck are the biocompatibility of the device, and finding the right criteria to assess the delivered medical service of the device. Regarding the US/endoscopic fusion, the hardware is available, except for the Ultrasonix Ultrasound machine which is under purchase (since 0ctober 2016). The main challenge will be to create adequate visualization based on the fused data, including exploiting specific US modes (such as Doppler for the visualization of neurovascular bundles).
A postdoctoral fellow (Katia Charrières), recruited in June 2017 will focus on the enhancement on the endoscopic tool tracking method, using deep learning approaches. Cases where our tool tracking approach fails to detect instruments will be analyzed. Anticipated challenges will be the capacity to provide generic enough solutions to cover the variability of laparoscopic images and the capacity to provide a “lightweight” enough approach to be compatible with the real-time constraints.
A PhD student (Arthur Derathe) has been recruited in October 2016. The PhD is performed in collaboration with the LTSI in Rennes and a Digestive Surgeon of the Grenoble CHU (Dr. Fabian Reche, also a PhD in the TIMC-IMAG lab). The clinical target of the work is gastric bypass laparoscopic surgery, surgery of which Dr. Fabian Reche is a renowned expert. The objective of this PhD is to combine classical workflow annotations (surgical activities) with contextual information extracted from the laparoscopic video feed in order to assess the quality of surgical procedures, according to “quality” criteria identified by Dr. Fabian Reche. Katia Charrières will also participate in this axis thanks to her know-how in machine learning.
Transfer: N/A for the moment
Main publications: N/A for the moment
[1] Yaxley, John W et al. Robot-assisted laparoscopic prostatectomy versus open radical retropubic prostatectomy: early outcomes from a randomised controlled phase 3 study. The Lancet , Volume 388 , Issue 10049 , 1057 – 1066.
