, projecting the coordinates of the annotations to the endoscopic view based on the registration result between the US image and the endoscopic camera

, Endoscopic image augmentation: (1) annotating distal and proximal margin (green) of the simulated tumor on 'rectal wall, US images using crosshairs

, generating resection line (red) 10 mm below the most distal margin

, During the experiment, the surgeon marked the proximal and distal end of the simulated tumor and the distal resection margin on the rectum phantom by using the implemented AR framework, as shown in Figure 4.17 (a). Then, the surgeon made straight cuts on these marks

J. M. Abeysekera, M. Najafi, R. Rohling, and S. E. Salcudean, Calibration for Position Tracking of Swept Motor 3-D Ultrasound, Ultrasound in Medicine and Biology, vol.40, issue.6, pp.1356-1371, 2014.

A. Agustinos, R. Wolf, J. A. Long, P. Cinquin, and S. Voros, Visual servoing of a robotic endoscope holder based on surgical instrument tracking, IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics, pp.13-18, 2014.
URL : https://hal.archives-ouvertes.fr/hal-01164895

S. Atallah, S. W. Larach, and J. R. Monson, Stereotactic navigation for TAMIS-TME. Minimally Invasive Therapy and Allied Technologies, vol.25, pp.271-277, 2016.

R. T. Azuma, A survey of augmented reality, Presence: Teleoperators and Virtual Environments, vol.6, issue.4, pp.355-385, 1997.

S. J. Baek, C. H. Kim, M. S. Cho, S. U. Bae, H. Hur et al., Robotic surgery for rectal cancer can overcome difficulties associated with pelvic anatomy, Surgical Endoscopy, vol.29, issue.6, pp.1419-1424, 2015.

C. Bergmeir, M. Seitel, C. Frank, R. De-simone, H. Meinzer et al., Comparing calibration approaches for 3D ultrasound probes, International Journal of Computer Assisted Radiology and Surgery, vol.4, pp.203-213, 2009.

S. Bernhardt, S. A. Nicolau, L. Soler, and C. Doignon, The status of augmented reality in laparoscopic surgery as of 2016, Medical Image Analysis, vol.37, pp.66-90, 2017.

C. Bichlmeier, F. Wimmer, S. M. Heining, and N. Navab, Contextual anatomic mimesis hybrid in-situ visualization method for improving multi-sensory depth perception in medical augmented reality, 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp.129-138, 2007.

W. Birkfellner, J. Hummel, W. , and E. , Tracking devices, Image-Guided Interventions, pp.23-45, 2008.

R. G. Blanco, J. Califano, B. Messing, J. Richmon, J. Liu et al., Transcervical ultrasonography is feasible to visualize and evaluate base of tongue cancers, PLoS ONE, vol.9, issue.1, p.87565, 2014.

B. F. Branstetter, T. M. Blodgett, L. A. Zimmer, C. H. Snyderman, J. T. Johnson et al., Head and neck malignancy: is PET/CT more accurate than PET or CT alone?, Radiology, vol.235, issue.2, pp.580-586, 2005.

F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre et al., Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries, CA: A Cancer Journal for Clinicians, vol.68, issue.6, pp.394-424, 2018.

X. Chen, L. Xu, Y. Wang, H. Wang, F. Wang et al., Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display, Journal of Biomedical Informatics, vol.55, pp.124-131, 2015.

A. Cheng, X. Guo, H. Kang, B. Tavakoli, J. U. Kang et al., Concurrent photoacoustic markers for direct three-dimensional ultrasound to video registration, SPIE 8943, Photons Plus Ultrasound: Imaging and Sensing, vol.8943, p.89435, 2014.

C. L. Cheung, C. Wedlake, J. Moore, S. E. Pautler, and T. M. Peters, Fused video and ultrasound images for minimally invasive partial nephrectomy: A phantom study, International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.408-415, 2010.

H. Choi, B. Cho, K. Masamune, M. Hashizume, and J. Hong, An effective visualization technique for depth perception in augmented reality-based surgical navigation, International Journal of Medical Robotics and Computer Assisted Surgery, vol.12, issue.1, pp.62-72, 2016.

D. R. Clayburgh, J. K. Byrd, J. Bonfili, and U. Duvvuri, Intraoperative Ultrasonography during Transoral Robotic Surgery, Rhinology and Laryngology, vol.125, issue.1, pp.37-42, 2016.

R. M. Comeau, A. F. Sadikot, A. Fenster, and T. M. Peters, Intraoperative ultrasound for guidance and tissue shift correction in image-guided neurosurgery, Medical Physics, vol.27, issue.4, pp.787-800, 2000.

S. F. Coquia, U. M. Hamper, M. E. Holman, M. R. Dejong, R. M. Subramaniam et al., Visualization of the oropharynx with transcervical ultrasound, American Journal of Roentgenology, vol.205, issue.6, pp.1288-1294, 2015.

K. Daniilidis, Hand-eye calibration using dual quaternions, The International Journal of Robotics Research, vol.18, pp.286-298, 1999.

K. Daniilidis and E. Bayro-corrochano, The dual quaternion approach to hand-eye calibration, Proceedings -International Conference on Pattern Recognition, vol.1, pp.318-322, 1996.

J. Demongeot, A. Lontos, and E. Promayon, Discrete Mesh Approach in Morphogenesis Modelling: the Example of Gastrulation, Acta Biotheoretica, vol.64, issue.4, pp.427-446, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01498621

S. C. Desai, C. Sung, and E. M. Genden, Transoral robotic surgery using an image guidance system, The Laryngoscope, vol.118, issue.11, pp.2003-2005, 2008.

P. R. Detmer, G. Bashein, T. Hodges, K. W. Beach, E. P. Filer et al., 3D ultrasonic image feature localization based on magnetic scanhead tracking: in vitro calibration and validation, Ultrasound in Medicine and Biology, vol.20, issue.9, pp.923-936, 1994.

L. R. Dice, Measures of the amount of ecologic association between species, Ecology, vol.26, issue.3, pp.297-302, 1945.

M. Dubuisson and A. K. Jain, A modified Hausdorff distance for object matching, 12th International Conference on Pattern Recognition, pp.566-568, 1994.

S. Engelhardt, R. De-simone, S. Al-maisary, S. Kolb, M. Karck et al., Accuracy evaluation of a mitral valve surgery assistance system based on optical tracking, International Journal of Computer Assisted Radiology and Surgery, vol.11, issue.10, pp.1891-1904, 2016.

C. Fakhry, N. Agrawal, J. Califano, B. Messing, J. Liu et al., The use of ultrasound in the search for the primary site of unknown primary head and neck squamous cell cancers, Oral Oncology, vol.50, issue.7, pp.640-645, 2014.

D. G. Farwell, J. Phipps, D. Gorpas, D. Ma, J. Bec et al., Augmented fluorescence lifetime imaging for enhanced tumor margin delineation during trans-oral robotic surgery, Proceedings of the 9th International Conference on Head and Neck Cancer, 2016.

A. Fedorov, R. Beichel, J. Kalpathy-cramer, J. Finet, J. C. Fillion-robin et al., 3D Slicer as an image computing platform for the quantitative imaging network, Magnetic Resonance Imaging, vol.30, issue.9, pp.1323-1341, 2012.

M. Feuerstein, T. Mussack, S. Heining, and N. Navab, Intraoperative laparoscope augmentation for port placement and resection planning in minimally invasive liver resection, IEEE Transactions on Medical Imaging, vol.27, issue.3, pp.355-369, 2008.

M. Feuerstein, T. Reichl, J. Vogel, J. Traub, and N. Navab, Magneto-optical tracking of flexible laparoscopic ultrasound: model-based online detection and correction of magnetic tracking errors, IEEE Transactions on Medical Imaging, vol.28, issue.6, pp.951-967, 2009.

G. Fichtinger, A. Deguet, K. Masamune, E. Balogh, and G. Fischer, Image overlay guidance for needle insertion in CT scanner, IEEE Transactions on Biomedical Engineering, vol.52, issue.8, pp.1415-1424, 2005.
URL : https://hal.archives-ouvertes.fr/inria-00590185

J. M. Fitzpatrick, J. B. West, and C. R. Maurer, Predicting error in rigid-body point-based registration, IEEE Transactions on Medical Imaging, vol.17, issue.5, pp.694-702, 1998.

A. M. Franz, T. Haidegger, W. Birkfellner, K. Cleary, T. M. Peters et al., Electromagnetic tracking in medicine -A review of technology, validation, and applications, IEEE Transactions on Medical Imaging, vol.33, issue.8, pp.1702-1725, 2014.

K. Gavaghan, T. Oliveira-santos, M. Peterhans, M. Reyes, H. Kim et al., Evaluation of a portable image overlay projector for the visualisation of surgical navigation data: Phantom studies, International Journal of Computer Assisted Radiology and Surgery, vol.7, issue.4, pp.547-556, 2012.

G. Gerig and F. Klein, Fast contour identification through efficient hough transform and simplified interpretation strategy, International Conference on Pattern Recognition, pp.498-500, 1986.

F. Greif, D. Aranovich, N. Hananel, M. Knizhnik, and A. Belenky, Intraoperative ultrasound in colorectal surgery, Journal of Clinical Ultrasound, vol.37, issue.7, pp.375-379, 2009.

N. Haouchine, J. Dequidt, I. Peterlik, E. Kerrien, M. Berger et al., Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery, IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp.199-208, 2013.
URL : https://hal.archives-ouvertes.fr/hal-00842855

P. Hsu, R. W. Prager, N. E. Houghton, A. H. Gee, and G. M. Treece, Accurate fiducial location for freehand 3D ultrasound calibration, Ultrasonic Imaging and Signal Processing, vol.6513, p.651315, 2007.

Y. Hu, H. U. Ahmed, Z. Taylor, C. Allen, M. Emberton et al., MR to ultrasound registration for image-guided prostate interventions, Medical Image Analysis, vol.16, issue.3, pp.687-703, 2012.

C. H. Huh, M. S. Bhutani, E. B. Farfán, and W. E. Bolch, Individual variations in mucosa and total wall thickness in the stomach and rectum assessed via endoscopic ultrasound, Physiological measurement, vol.24, issue.4, pp.15-22, 2003.

H. G. Kenngott, M. Wagner, M. Gondan, F. Nickel, M. Nolden et al., Real-time image guidance in laparoscopic liver surgery: First clinical experience with a guidance system based on intraoperative CT imaging, Surgical Endoscopy, vol.28, issue.3, pp.933-940, 2014.

N. K. Kim, M. J. Kim, L. H. Yun, S. K. Sohn, M. et al., Comparative study of transrectal ultrasonography, pelvic computerized tomography, and magnetic resonance imaging in preoperative staging of rectal cancer, Diseases of the Colon and Rectum, vol.42, issue.6, pp.770-775, 1999.

M. E. Kisansa and S. Andronikou, Ultrasound imaging of tongue malignancy, International Journal of Case Reports and Images, vol.8, issue.1, pp.1-6, 2017.

K. Konishi, M. Hashizume, M. Nakamoto, Y. Kakeji, I. Yoshino et al., Augmented reality navigation system for endoscopic surgery based on three-dimensional ultrasound and computed tomography: Application to 20 clinical cases, International Congress Series, vol.1281, pp.537-542, 2005.

J. B. Kruskal and R. A. Kane, Intraoperative US of the liver: techniques and cilnical applications, RadioGraphics, vol.26, issue.4, pp.1067-1084, 2006.

T. Lange, S. Eulenstein, M. Hiinerbein, and P. Schlag, Vessel-based non-rigid registration of MR / CT and 3D ultrasound for navigation in liver surgery, Computer Aided Surgery, vol.8, issue.5, pp.228-240, 2003.

H. E. Liu, E. Auvinet, J. Giles, F. Rodriguez, and Y. Baena, Augmented reality based navigation for computer assisted hip resurfacing: a proof of concept study, Annals of Biomedical Engineering, vol.46, issue.10, pp.1595-1605, 2018.

W. P. Liu, S. Reaugamornrat, A. Deguet, J. M. Sorger, J. H. Siewerdsen et al., Toward intraoperative image-guided transoral robotic surgery, Journal of robotic surgery, vol.7, issue.3, pp.217-225, 2013.

W. P. Liu, J. D. Richmon, J. M. Sorger, M. Azizian, and R. H. Taylor, Augmented reality and cone beam CT guidance for transoral robotic surgery, Journal of Robotic Surgery, vol.9, issue.3, pp.223-233, 2015.

X. Liu, S. Kang, W. Plishker, G. Zaki, T. D. Kane et al., Laparoscopic stereoscopic augmented reality: toward a clinically viable electromagnetic tracking solution, Journal of Medical Imaging, vol.3, issue.4, p.45001, 2016.

S. M. Lulich, Registration and fusion of 3D head-neck MRI and 3D/4D tongue ultrasound, The Journal of the Acoustical Society of America, vol.144, issue.3, pp.1904-1904, 2018.

A. Malti and J. P. Barreto, Robust hand-eye calibration for computer aided medical endoscopy, 2010 IEEE International Conference on Robotics and Automation, pp.5543-5549, 2010.

B. Marques, F. Roy, N. Haouchine, E. Jeanvoine, S. Cotin et al., Framework for augmented reality in Minimally Invasive laparoscopic surgery, 17th International Conference on E-Health Networking, Application and Services (HealthCom), pp.22-27, 2015.
URL : https://hal.archives-ouvertes.fr/hal-01315574

Y. Masutani, T. Dohi, F. Yamane, H. Iseki, and K. Takakura, Augmented reality visualization system for intravascular neurosurgery, Computer Aided Surgery, vol.3, issue.5, pp.239-247, 1998.

P. Mountney, J. Fallert, S. Nicolau, L. Soler, and P. W. Mewes, An augmented reality framework for soft tissue surgery, International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.423-431, 2014.

M. Nakamoto, K. Nakada, Y. Sato, K. Konishi, M. Hashizume et al., Intraoperative magnetic tracker calibration using a magneto-optic hybrid tracker for 3-D ultrasound-based navigation in laparoscopic surgery, IEEE Transactions on Medical Imaging, vol.27, issue.2, pp.255-270, 2008.

M. Nicolaou, A. James, B. P. Lo, A. Darzi, Y. et al., Invisible shadow for navigation and planning in minimal invasive surgery, International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.25-32, 2005.

A. Patsias, L. Giraldez-rodriguez, A. D. Polydorides, R. Richards-kortum, S. Anandasabapathy et al., Feasibility of transoral roboticassisted high-resolution microendoscopic imaging of oropharyngeal squamous cell carcinoma, Head & Neck, vol.37, issue.8, pp.99-102, 2015.

T. C. Poon and R. N. Rohling, Comparison of calibration methods for spatial tracking of a 3-D ultrasound probe, Ultrasound in Medicine & Biology, vol.31, issue.8, pp.1095-1108, 2005.

B. C. Porter, D. J. Rubens, J. G. Strang, J. Smith, S. Totterman et al., Three-dimensional registration and fusion of ultrasound and MRI using major vessels as fiducial Markers, IEEE Transactions on Medical Imaging, vol.20, issue.4, pp.354-359, 2001.

R. W. Prager, R. N. Rohling, A. H. Gee, and L. Berman, Rapid calibration for 3-D freehand ultrasound, Ultrasound in Medicine and Biology, vol.24, issue.6, pp.855-869, 1998.

P. Pratt and A. Arora, Transoral robotic surgery: image guidance and augmented reality, ORL, vol.80, pp.204-212, 2018.

P. Pratt, E. Mayer, J. Vale, D. Cohen, E. Edwards et al., An effective visualisation and registration system for image-guided robotic partial nephrectomy, Journal of Robotic Surgery, vol.6, issue.1, pp.23-31, 2012.

K. Prokopetc, T. Collins, and A. Bartoli, Automatic detection of the uterus and fallopian tube junctions in laparoscopic images, International Conference on Information Processing in Medical Imaging, pp.552-563, 2015.

G. A. Puerto-souza, J. A. Cadeddu, and G. L. Mariottini, Toward long-term and accurate augmented-reality for monocular endoscopic videos, IEEE Transactions on Biomedical Engineering, vol.61, issue.10, pp.2609-2620, 2014.

S. Reaungamornrat, W. P. Liu, A. S. Wang, Y. Otake, S. Nithiananthan et al., Deformable image registration for cone-beam CT guided transoral robotic base of tongue surgery, Physics in Medicine & Biology, vol.58, issue.14, pp.4951-4979, 2013.

M. Rosenthal, A. State, J. Lee, G. Hirota, J. Ackerman et al., Augmented reality guidance for needle biopsies: an initial randomized, controlled trial in phantoms, Medical Image Analysis, vol.6, issue.3, pp.313-320, 2002.

F. Rousseau, P. Hellier, and C. Barillot, Confhusius: a robust and fully automatic calibration method for 3D freehand ultrasound, Medical Image Analysis, vol.9, issue.1, pp.25-38, 2005.
URL : https://hal.archives-ouvertes.fr/inserm-00136520

E. Rullier, Q. Denost, V. Vendrely, A. Rullier, L. et al., Low rectal cancer: classification and standardization of surgery, Diseases of the Colon and Rectum, vol.56, issue.5, pp.560-567, 2013.

G. F. Schwartz, B. B. Goldberg, M. D. Rifkin, and S. E. Orazio, Ultrasonography: an alternative to x-ray-guided needle localization of nonpalpable breast masses, Surgery, vol.104, issue.5, pp.870-873, 1988.

J. Shen, N. Zemiti, J. Dillenseger, and P. Poignet, Fast and simple automatic 3D ultrasound probe calibration based on 3D printed phantom and an untracked marker, pp.878-882, 2018.
URL : https://hal.archives-ouvertes.fr/lirmm-01828628

B. Sigel, D. G. Spigos, P. E. Donahue, R. Pearl, G. L. Popky et al., , 1979.

, Intraoperative ultrasonic visualization of biliary calculi, Current surgery, vol.36, issue.3, pp.158-159

T. Simpfendörfer, M. Baumhauer, M. Müller, C. N. Gutt, H. P. Meinzer et al., Augmented reality visualization during laparoscopic radical prostatectomy, Journal of Endourology, vol.25, issue.12, pp.1841-1845, 2011.

R. Singla, P. Edgcumbe, P. Pratt, C. Nguan, and R. Rohling, Intra-operative ultrasound-based augmented reality guidance for laparoscopic surgery, Healthcare Technology Letters, vol.4, issue.5, pp.204-209, 2017.

D. Teber, S. Guven, T. Simpfendörfer, M. Baumhauer, E. O. Güven et al., Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? preliminary in vitro and in vivo results, European Urology, vol.56, issue.2, pp.332-338, 2009.

B. E. Treeby and B. T. Cox, k-Wave: MATLAB toolbox for the simulation and reconstruction of photoacoustic wave fields, Journal of Biomedical Optics, vol.15, issue.2, p.21314, 2010.

B. M. Trotta, C. S. Pease, J. J. Rasamny, P. Raghavan, and S. Mukherjee, Oral cavity and oropharyngeal squamous cell cancer: key imaging findings for staging and treatment planning, RadioGraphics, vol.31, issue.2, pp.339-354, 2011.

R. Tsai and R. Lenz, A new technique for fully autonomous and efficient 3D robotics hand/eye calibration, IEEE Transactions on Robotics and Automation, vol.5, issue.3, pp.345-358, 1989.

O. Ukimura, M. Nakamoto, Y. Sato, M. Hashizume, T. Miki et al., Augmented reality for image-guided surgery in urology, New Technologies in Urology, pp.215-222, 2010.

K. M. Van-abel and E. J. Moore, Surgical management of the base of tongue, Operative Techniques in Otolaryngology -Head and Neck Surgery, vol.24, pp.74-85, 2013.

M. Vermaas, F. Ferenschild, C. Verhoef, J. Nuyttens, A. Marinelli et al., Total pelvic exenteration for primary locally advanced and locally recurrent rectal cancer, European Journal of Surgical Oncology, vol.33, issue.4, pp.452-458, 2007.

M. A. Vitrani, M. Baumann, D. Reversat, G. Morel, A. Moreau-gaudry et al., Prostate biopsies assisted by comanipulated probe-holder: first in man, International Journal of Computer Assisted Radiology and Surgery, vol.11, issue.6, pp.1153-1161, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01312297

F. Volonté, F. Pugin, P. Bucher, M. Sugimoto, O. Ratib et al., Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: Not only a matter of fashion, Journal of Hepato-Biliary-Pancreatic Sciences, vol.18, issue.4, pp.506-509, 2011.

Q. Wang, A. Khanicheh, D. Leiner, D. Shafer, and J. Zobel, Endoscope field of view measurement, Biomedical Optics Express, vol.8, issue.3, pp.1441-1454, 2017.

C. Wengert, M. Reeff, P. C. Cattin, and G. Székely, Fully automatic endoscope calibration for intraoperative use, Bildverarbeitung für die Medizin, pp.419-423, 2006.

E. Wild, D. Teber, D. Schmid, T. Simpfendörfer, M. Müller et al., Robust augmented reality guidance with fluorescent markers in laparoscopic surgery, International Journal of Computer Assisted Radiology and Surgery, vol.11, issue.6, pp.899-907, 2016.

J. T. Yen, J. P. Steinberg, and S. W. Smith, Sparse 2-D array design for real time rectilinear volumetric imaging, IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, vol.47, issue.1, pp.93-110, 2000.

L. Zhang, M. Ye, P. Chan, Y. , and G. , Real-time surgical tool tracking and pose estimation using a hybrid cylindrical marker, International journal of computer assisted radiology and surgery, vol.12, issue.6, pp.921-930, 2017.

Z. Zhang, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.22, 2000.

. Shen, Intraoperative ultrasound-based augmented reality guidance, 2019.
URL : https://hal.archives-ouvertes.fr/lirmm-02105828

. Shen, Transrectal ultrasound image-based real-time augmented reality guidance in robot-assisted laparoscopic rectal surgery: a proof-of-concept study, International Journal of Computer Assisted Radiology and Surgery (IJCARS, 2019.
URL : https://hal.archives-ouvertes.fr/lirmm-02381958

. Shen, Intraoperative ultrasonography-based augmented reality for application in image guided robotic surgery. Computer 32nd Assisted Radiology and Surgery (CARS), International Journal of Computer Assisted Radiology and Surgery (IJCARS), vol.13, pp.45-46, 2018.
URL : https://hal.archives-ouvertes.fr/lirmm-01828409

. Shen, Fast and simple automatic 3D ultrasound probe calibration based on 3D printed phantom and an untracked marker, 40th IEEE Conference on Engineering in Medicine and Biology Conference (EMBC), pp.878-882, 2018.
URL : https://hal.archives-ouvertes.fr/lirmm-01828628

. Shen, Intraoperative 3D ultrasound probe calibration, Recherche en Imagerie et Technologies pour la Santée, p.2017, 2017.

. Shen, Augmented perception in transoral robotic surgery for tongue base cancer, 7th Joint Workshop on New Technologies for Computer/Robot Assisted Surgery, p.2017, 2017.

. Shen, Augmented reality visualization based on 3D ultrasonography, 2017.
URL : https://hal.archives-ouvertes.fr/lirmm-02105826