VerroTouch: Haptic Instrument Vibration Feedback For Robotic Minimally Invasive Surgery
VerroTouch unobtrusively measures the contact vibrations of the robotic instruments and enables the surgeon to feel and hear these naturalistic cues.
(Top) The vibrations a surgeon feels when a laparoscopic tool grasps a needle. (Middle) The vibrations by a robotic surgery instrument grasping a needle. (Bottom) The vibrations delivered to the hand of the surgeon grasping a needle with an augmented robot.
Robotic minimally invasive surgery (RMIS) enables surgeons to manipulate structures deep within the body via tiny incisions. Though the robotic tools mimic human hand motions, surgeons operate without the haptic (kinesthetic and tactile) cues they are accustomed to receiving in open and laparoscopic surgery.
The Penn Haptics Group has recently introduced a method for adding tactile and audio feedback of tool vibrations to RMIS systems, creating sensations similar to what one feels and hears when using a laparoscopic tool. This method aims to make RMIS more immersive and natural.
Our system, which we have named VerroTouch, measures the vibrations caused by tool contacts and immediately recreates them at the master console for the surgeon to hear and feel. This augmentation enables the surgeon to feel the texture of rough surfaces, the start and end of contact with manipulated objects, and other important tactile events. While it does not provide low frequency forces, we believe vibrotactile feedback will be useful for surgical task training and evaluation, and our studies have thus far supported this hypothesis.
Three-axis accelerometers attached to the patient-side robot manipulators measure the tool contacts that occur duing surgery. These sensors install within the sterile drapes on da Vinci systems, thus requiring no sterilization.
Acceleration signals are then conditioned and processed to drive audio and haptic feedback.
Voice coil actuators mounted near the surgeon’s hands generate vibrotactile, haptic feedback of the instrument vibrations, allowing the surgeon to feel the contacts. Speakers can also be used to provide audio feedback of instrument vibrations.
- Significantly preferred over a non-haptic system by surgeons at all levels of training.
- Feels natural and does not interfere with robot operation.
- Improves surgeon awareness of the robotic instruments and their interactions with the surgical field.
- Provides feedback even when tools are obscured or off camera.
- Built from widely available low-cost components.
- Installs within the sterile drapes on da Vinci systems, requiring no sterilization.
- Works "in vivo" -- verified in a porcine model and in recordings of live human surgeries.
- Enhances procedural awareness for all individuals in the OR via audio cues.
- Accelerates training of residents learning RMIS.
- Enables automatic quantitative surgical skill evaluation.
Recently, Prof. Katherine J. Kuchenbecker explained our haptic feedback system to David Pogue for a PBS NOVA scienceNOW segment.
An overview of the system technology and several applications for measuring and recreating surgical instrument vibrations for training and skill assessment.
Execution of box trainer tasks. The audio of this video was recorded using our vibration sensors and gives an example of the audio feedback our method is capable of providing to users.
In vivo system validation with a porcine model.
In vivo instrument vibration recordings from human patients.
Installation of the system components on a da Vinci S.
- PBS inside NOVA (November 2012) -- Augmented reality with a sense of a touch
- Urology Times Article (October 2011) -- Study: Vibrotactile feedback enhances robotic surgery
- W. McMahan, E. D. Gomez, L. Chen, K. Bark, J. C. Nappo, E. I. Koch, D. I. Lee, K. R. Dumon, N. N. Williams, and K. J. Kuchenbecker, A practical system for recording instrument interactions during live robotic surgery. Journal of Robotic Surgery, In Press, Published online April 2013. (paper) (video)
- K. Bark, W. McMahan, A. Remington, J. Gewirtz, A. Wedmid, D. I. Lee, and K. J. Kuchenbecker. In Vivo Validation of a System for Haptic Feedback of Tool Vibrations in Robotic Surgery. Surgical Endoscopy, Volume 27(2):656-664, July 2012. (paper) (video)
- W. McMahan, J. Gewirtz, D. Standish, P. Martin, J. A. Kunkel, M. Lilavois, A. Wedmid, D. I. Lee, and K. J. Kuchenbecker. Tool Contact Acceleration Feedback for Telerobotic Surgery. IEEE Transactions on Haptics, Special Issue on Haptics in Medicine and Clinical Skill Acquisition. Volume 4(3):210-220, July 2011. (paper) (video1) (video2)
Conference Papers and Abstracts
- K. Bark, E. D. Gomez, C. Rivera, W. McMahan, A. C. Remington, K. M. Murayama, D. I. Lee, K. R. Dumon, N. N. Williams, and K. J. Kuchenbecker. Surgical instrument vibrations are a construct-valid measure of technical skill in robotic peg transfer and suturing tasks. In Proceedings, Hamlyn Symposium on Medical Robotics, pages 50–51, July 2012. (pdf)
- W. McMahan, K. Bark, J. Gewirtz, D. Standish, P. D. Martin, J. A. Kunkel, M. Lilavois, A. Wedmid, D. I. Lee, K. J. Kuchenbecker. Tool Vibration Feedback May Help Expert Robotic Surgeons Apply Less Force During Manipulation Tasks. In Proceedings, Hamlyn Symposium on Medical Robotics, pages 37–38, June 2011. (pdf)
- K. J. Kuchenbecker, J. Gewirtz, W. McMahan, D. Standish, P. Martin, J. Bohren, P. J. Mendoza, and D. I. Lee. VerroTouch: High-frequency acceleration feedback for telerobotic surgery. In Proceedings, EuroHaptics, pages 189-196, July 2010. (pdf)
- W. McMahan, E. D. Gomez, L. Chen, K. Bark, J. C. Nappo, E. I. Koch, D. I. Lee, K. Dumon, N. Williams, and K. J. Kuchenbecker. A practical system for recording instrument interactions during live robotic surgery. In Proceedings, Medicine Meets Virtual Reality (MMVR), February 2013.
- E. D. Gomez, K. Bark, C. Rivera, W. McMahan, A. C. Remington, D. I. Lee, N. N. Williams, K. M. Murayama, K. R. Dumon, and K. J. Kuchenbecker. Construct validity of instrument vibrations as a measure of robotic surgical skill. Accepted for presentation at the American College of Surgeons (ACS) Clinical Congress, 2012. Journal of American College of Surgeons, volume 215, pages S119-120, 2012.
- E. D. Gomez, K. Bark, W. McMahan, C. Rivera, A. Remington, D. I. Lee, and K. J. Kuchenbecker. VerroTouch: Detection of instrument vibrations for haptic feedback and skill assessment in robotic surgery. Accepted for presentation at the Annual Meeting of the Society of American Gastroinestinal and Endoscopic Surgeons (SAGES), March 2012.
- W. McMahan, J. Gewirtz, D. Standish, P. D. Martin, J. A. Kunkel, M. Lilavois, A. Wedmid, D. I. Lee, and K. J. Kuchenbecker. VerroTouch: Vibrotactile feedback for robotic minimally invasive surgery. Accepted for presentation at the Annual Meeting of the American Urological Association (AUA). The Journal of Urology, 185(4, Supplement):e373, May 2011. Best Poster in Session.
- K. J. Kuchenbecker, J. Gewirtz, W. McMahan, D. Standish, J. Bohren, P. Martin, A. Wedmid, P. J. Mendoza, and D. I. Lee. VerroTouch: A vibrotactile feedback system for minimally invasive robotic surgery. Accepted for presentation at the 28th World Congress of Endourology, number PS8-14, September 2010.
This work was supported by the Pennsylvania Department of Health via Health Research Formula Funds, by the National Science Foundation via grant #IIS-0845670, by a Translational Research Award from the Coulter Foundation, and by the University of Pennsylvania. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding organizations.