Skip to main content

Advertisement

SpringerLink
Log in

Automated Surgical-Phase Recognition for Robot-Assisted Minimally Invasive Esophagectomy Using Artificial Intelligence

  • Thoracic Oncology
  • Published:
Annals of Surgical Oncology Aims and scope Submit manuscript

Abstract

Background

Although a number of robot-assisted minimally invasive esophagectomy (RAMIE) procedures have been performed due to three-dimensional field of view, image stabilization, and flexible joint function, both the surgeons and surgical teams require proficiency. This study aimed to establish an artificial intelligence (AI)-based automated surgical-phase recognition system for RAMIE by analyzing robotic surgical videos.

Methods

This study enrolled 31 patients who underwent RAMIE. The videos were annotated into the following nine surgical phases: preparation, lower mediastinal dissection, upper mediastinal dissection, azygos vein division, subcarinal lymph node dissection (LND), right recurrent laryngeal nerve (RLN) LND, left RLN LND, esophageal transection, and post-dissection to completion of surgery to train the AI for automated phase recognition. An additional phase (“no step”) was used to indicate video sequences upon removal of the camera from the thoracic cavity. All the patients were divided into two groups, namely, early period (20 patients) and late period (11 patients), after which the relationship between the surgical-phase duration and the surgical periods was assessed.

Results

Fourfold cross validation was applied to evaluate the performance of the current model. The AI had an accuracy of 84%. The preparation (p = 0.012), post-dissection to completion of surgery (p = 0.003), and “no step” (p < 0.001) phases predicted by the AI were significantly shorter in the late period than in the early period.

Conclusions

A highly accurate automated surgical-phase recognition system for RAMIE was established using deep learning. Specific phase durations were significantly associated with the surgical period at the authors’ institution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Siegel RL, Miller KD, Fuchs HE, Jemal A. Cancer statistics, 2021. CA Cancer J Clin. 2021;71:7–33.

    Article  Google Scholar 

  2. Kitagawa Y, Uno T, Oyama T, et al. Esophageal cancer practice guidelines 2017 edited by the Japan Esophageal Society: part 1. Esophagus. 2019;16:1–24.

    Article  Google Scholar 

  3. Straatman J, Van Der Wielen N, Cuesta MA, et al. Minimally invasive versus open esophageal resection: three-year follow-up of the previously reported randomized controlled trial: the TIME trial. Ann Surg. 2017;266:232–6.

    Article  Google Scholar 

  4. Kataoka K, Takeuchi H, Mizusawa J, et al. A randomized phase III trial of thoracoscopic versus open esophagectomy for thoracic esophageal cancer: Japan Clinical Oncology Group Study (JCOG) 1409. Jpn J Clin Oncol. 2016;46:174–7.

    Article  Google Scholar 

  5. Kernstine KH. The first series of completely robotic esophagectomies with three-field lymphadenectomy: initial experience. Surg Endosc Other Interv Tech. 2008;22:2102.

    Article  CAS  Google Scholar 

  6. Park SY, Kim DJ, Yu WS, Jung HS. Robot-assisted thoracoscopic esophagectomy with extensive mediastinal lymphadenectomy: experience with 114 consecutive patients with intrathoracic esophageal cancer. Dis Esophagus. 2016;29:326–32.

    Article  CAS  Google Scholar 

  7. van der Sluis PC, Ruurda JP, Verhage RJJ, et al. Oncologic long-term results of robot-assisted minimally invasive thoraco-laparoscopic esophagectomy with two-field lymphadenectomy for esophageal cancer. Ann Surg Oncol. 2015;22:S1350–6. https://doi.org/10.1245/s10434-015-4544-x.

    Article  PubMed  Google Scholar 

  8. Luketich JD, Pennathur A, Awais O, et al. Outcomes after minimally invasive esophagectomy: review of over 1000 patients. Ann Surg. 2012;256:95–103.

    Article  Google Scholar 

  9. Suda K, Ishida Y, Kawamura Y, et al. Robot-assisted thoracoscopic lymphadenectomy along the left recurrent laryngeal nerve for esophageal squamous cell carcinoma in the prone position: technical report and short-term outcomes. World J Surg. 2012;36:1608–16.

    Article  Google Scholar 

  10. Morimoto Y, Kawakubo H, Ishikawa A, et al. Short-term outcomes of robot-assisted minimally invasive esophagectomy with extended lymphadenectomy for esophageal cancer compared with video-assisted minimally invasive esophagectomy: a single-center retrospective study. Asian J Endosc Surg. 2021;15:270–8.

    Article  Google Scholar 

  11. van der Sluis PC, van der Horst S, May AM, et al. Robot-assisted minimally invasive thoracolaparoscopic esophagectomy versus open transthoracic esophagectomy for resectable esophageal cancer: a randomized controlled trial. Ann Surg. 2019;269:621–30.

    Article  Google Scholar 

  12. Tagkalos E, van der Sluis PC, Berlth F, et al. Robot-assisted minimally invasive thoraco-laparoscopic esophagectomy versus minimally invasive esophagectomy for resectable esophageal adenocarcinoma, a randomized controlled trial (ROBOT-2 trial). BMC Cancer. 2021;21:1060.

    Article  CAS  Google Scholar 

  13. Chao YK, Li ZG, Wen YW, et al. Robotic-assisted esophagectomy vs video-assisted thoracoscopic esophagectomy (REVATE): study protocol for a randomized controlled trial. Trials. 2019;20:1–8.

    Article  Google Scholar 

  14. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–8.

    Article  CAS  Google Scholar 

  15. Misawa M, Kudo SE, Mori Y, et al. Artificial intelligence-assisted polyp detection for colonoscopy: initial experience. Gastroenterology. 2018;154:2027-29.e3.

    Article  Google Scholar 

  16. Hirasawa T, Aoyama K, Tanimoto T, et al. Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images. Gastric Cancer. 2018;21:653–60.

    Article  Google Scholar 

  17. Zhao W, Yang J, Sun Y, et al. 3D deep learning from CT scans predicts tumor invasiveness of subcentimeter pulmonary adenocarcinomas. Cancer Res. 2018;78:6881–9.

    Article  CAS  Google Scholar 

  18. Takeuchi M, Seto T, Hashimoto M, et al. Performance of a deep learning-based identification system for esophageal cancer from CT images. Esophagus. 2021;18:612–20.

    Article  Google Scholar 

  19. Garrow CR, Kowalewski KF, Li L, et al. Machine learning for surgical phase recognition: a systematic review. Ann Surg. 2021;273:684–93.

    Article  Google Scholar 

  20. Bodenstedt S, Wagner M, Katić D, et al. Unsupervised temporal context learning using convolutional neural networks for laparoscopic workflow analysis. Retrieved 20 April 2022 at http://arxiv.org/abs/1702.03684.

  21. Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N. Statistical modeling and recognition of surgical workflow. Med Image Anal. 2012;16:632–41.

    Article  Google Scholar 

  22. Guédon ACP, Meij SEP, Osman KNMMH, et al. Deep learning for surgical phase recognition using endoscopic videos. Surg Endosc. 2021;35:6150–7.

    Article  Google Scholar 

  23. Hashimoto DA, Rosman G, Witkowski ER, et al. Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann Surg. 2019;270:414–21.

    Article  Google Scholar 

  24. Kitaguchi D, Takeshita N, Matsuzaki H, et al. Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: experimental research. Int J Surg. 2020;79:88–94.

    Article  Google Scholar 

  25. Rice TW, Patil DT, Blackstone EH. 8th Edition AJCC/UICC staging of cancers of the esophagus and esophagogastric junction: application to clinical practice. Ann Cardiothorac Surg. 2017;6:119–30.

    Article  Google Scholar 

  26. Czempiel T, Paschali M, Keicher M, et al. TeCNO: surgical-phase recognition with multi-stage temporal convolutional networks. Lect Notes Comput Sci LNCS. 2020;12263:343–52.

    Article  Google Scholar 

  27. Hernandez JM, Dimou F, Weber J, et al. Defining the learning curve for robotic-assisted esophagogastrectomy. J Gastrointest Surg. 2013;17:1346–51.

    Article  Google Scholar 

  28. Park SY, Kim DJ, Kang DR, Haam SJ. Learning curve for robotic esophagectomy and dissection of bilateral recurrent laryngeal nerve nodes for esophageal cancer. Dis Esophagus. 2017;30:1–9.

    Article  CAS  Google Scholar 

  29. Ward TM, Hashimoto DA, Ban Y, et al. Automated operative phase identification in peroral endoscopic myotomy. Surg Endosc. 2021;35:4008–15.

    Article  Google Scholar 

  30. Kitaguchi D, Takeshita N, Matsuzaki H, et al. Deep learning-based automatic surgical step recognition in intraoperative videos for transanal total mesorectal excision. Surg Endosc. 2022;36:1143–51.

    Article  Google Scholar 

  31. Levin M, McKechnie T, Khalid S, Grantcharov TP, Goldenberg M. Automated methods of technical skill assessment in surgery: a systematic review. J Surg Educ. 2019;76:1629–39.

    Article  Google Scholar 

Download references

Acknowledgment

We thank Kumiko Motooka, a staff member at the Department of Surgery in Keio University School of Medicine, for her help with the preparation of this report.

Author information

Author notes

  1. Masashi Takeuchi and Kosuke Saito contributed equally to this article.

Authors and Affiliations

  1. Department of Surgery, Keio University School of Medicine, Tokyo, Japan

    Masashi Takeuchi MD, PhD, Hirofumi Kawakubo MD, PhD, Kosuke Saito MSc, Yusuke Maeda MD, PhD, Satoru Matsuda MD, PhD, Kazumasa Fukuda PhD, Rieko Nakamura MD, PhD & Yuko Kitagawa MD, PhD

Authors
  1. Masashi Takeuchi MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hirofumi Kawakubo MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kosuke Saito MSc
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yusuke Maeda MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Satoru Matsuda MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kazumasa Fukuda PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rieko Nakamura MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yuko Kitagawa MD, PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hirofumi Kawakubo MD, PhD.

Ethics declarations

Disclosures

Yuko Kitagawa received lecture fees from Chugai Pharmaceutical Co., Ltd., Taiho Pharmaceutical Co., Ltd, Asahi Kasei Pharma Corporation, Otsuka Pharmaceutical Factory, Inc., Shionogi & Co., Ltd., Nippon Covidien, Inc., Ono Pharmaceutical Co., Ltd., and Bristol-Myers Squibb K.K. Yuko Kitagawa was supported by grants from Chugai Pharmaceutical Co., Ltd., Taiho Pharmaceutical Co., Ltd, Yakult Honsha Co. Ltd., Asahi Kasei Co., Ltd., Otsuka Pharmaceutical Co., Ltd., Takeda Pharmaceutical Co., Ltd., Ono Pharmaceutical Co., Ltd., Tsumura & Co., Kyouwa Hakkou Kirin Co., Ltd., Dainippon Sumitomo Pharma Co., Ltd., EA Pharma Co., Ltd., Astellas Pharma, Inc., Toyama Chemical Co., Ltd., Medicon, Inc., Kaken Pharmaceutical Co. Ltd., Eisai Co., Ltd., Otsuka Pharmaceutical Factory, Inc., Teijin Pharma Limited, Nihon Pharmaceutical Co., Ltd., and Nippon Covidien, Inc. Yuko Kitagawa held an endowed chair provided by Chugai Pharmaceutical Co., Ltd. and Taiho Pharmaceutical Co., Ltd, outside the submitted work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (DOCX 59 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Takeuchi, M., Kawakubo, H., Saito, K. et al. Automated Surgical-Phase Recognition for Robot-Assisted Minimally Invasive Esophagectomy Using Artificial Intelligence. Ann Surg Oncol 29, 6847–6855 (2022). https://doi.org/10.1245/s10434-022-11996-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1245/s10434-022-11996-1

Search

Navigation