Deep learning robotic guidance for autonomous vascular access

This is the blogpost for the paper “Deep learning robotic guidance for autonomous vascular access”.

Written by Alvin I. Chen, Max L. Balter , Timothy J. Maguire and Martin L. Yarmush.

Introduction

Problem statement

This paper focused on designing a autonomous robotic device for peripheral vessel cannulations, blood drawing and fluid infusion utilizing recurrent fully convolutional neural network (Rec-FCN).

State of the art

Autonomous medical robots have been developed with outstanding perception and dexterity [1]. For conducting clinical procedures, peripheral vascular interventions are indispensably required [2]. Since infusion time and cannulation attempts affects significantly the efficiency of medical therapies [3], it is of vital significance to develop more precise and consistent vascular access methods.

For positioning guidance, imaging technologies have been developed in mainly four categories: tactile pressure-based imaging, optical coherence tomography and photoacoustic tomography, near-infrared (NIR) optical imaging [4] and ultrasound (US) imaging [5-8].

For robotic approaches, duplex US vessel imaging [9][10], monocular NIR imaging [11], NIR stereo imaging [12][13] and multimodal imaging [14][15] have been adapted in robotic systems for vessel insertion, which are, however, still lack of closed-loop guidance and full autonomy.

Methodology

Deep learning architecture

In this paper, we adapted deep recurrent fully convolutional encoder-decoder network (Rec-FCN) [16-18] in training models for NIR (Fig. 1) and US (Fig. 2) guidance.

Figure 1 Rec-FCN model on NIR image sequences.

Figure 2 Rec-FCN model on B-mode and Color Doppler image (CDI) sequences.

1. Structure

In the standard structure of Rec-FCN, we introduced residual connections [19] in each convolutional block and incorporated skip connections [16] between the encoder and decoder layers. Through evaluation on three recurrent units (RNN [20], LSTM [21], GRU [22][23]) for temporal inference, we utilized GRU in the study.

2. Loss function

The multi-task loss functions for the two networks were given by [24-26]:

3. Datasets

The training and testing dataset for the first network was processed from NIR stereo image sequences of left and right forearm vessels from 9 participants [27]. The second network was pre-trained on two public US image datasets [28][29] and fine-tuned on transverse 2D US images of peripheral upper-extremity vessels from nine subjects [27]. All the datasets were split into 0.5 s sequences.

4. Other techniques

• Data augmentation: random rotation, horizontal and vertical flips, cropping and scaling, and gain and contrast adjustment
• Temporal augmentation: alternating the direction of the sequences between epochs and by applying window warping [28]
• Algorithm: standard stochastic gradient descent algorithm with the Adam optimizer66 and L2 weight decay regularization [29]
• Implementation: the TensorFlow library [30]

Bimodal NIR and US imaging

In the work, we used complementary sequences from NIR and US imaging (Fig. 3). Arm motions are estimated based on frame-to-frame non-rigid registration of the predicted segmentations and used to continuously update the robot trajectory before and during cannulation.

Figure 3 NIR and US image analysis for each step

Performance evaluation

(1) Error metric:

• Modified hausdorff distance (mm)

• Dice score

• Jaccard index

(2) Estimation criterion

• Error rate
• (expanded) Confusion matrix
• Precision-Recall and F1-score

Experimental setup

Robotic tracking

• Subject: random arm movements of 13 volunteers inside the device workspace for 60 s

• Tracking procedure: randomly select a target cannulation site, track robotically under NIR and US image guidance and maintain the target vessel at the field center.

In vitro and in vivo studies

1. Vessel segmentation and identification on human forearms

2. Cannulatoin on in vitro phantom models

• Optimization of robotic cannulation parameters (Fig. 4a) and investigation on in vitro tissue-mimicking models (Fig. 4b)

(a) Optimization of robotic insertion parameters

(b) Optimization of parameters in constructing 16 in vitro models

Figure 4: Experiments on parameters for robotic cannulation

• Comparison of robotic cannulation (n = 320) to manual cannulation without image guidance (n = 160), with NIR guidance (n = 160) and with US guidance (n = 160)

2. Blood drawing and fluids infusion in rats

• Comparison of two NIR imaging methods and of two US imaging qualities respectively
• Procedure: manually determine the cannulation sites with its closest segmented vessel as target. After successful insertion, collect 250 µl of blood and infuse 250 µl saline.

Results

End-to-end workflow for robotic vascular access (Fig. 5)

Figure 5: Workflow sequence for robotic cannulations

Automated vessel segmentation

Rec-FCN architecture have revealed high similarity to manual assessment from experts by vessel identification (Fig. 6). In terms of pixel-wise prediction accuracy, Rec-FCN models have overall outperformed the standard FCN. With regard to modified hausdorff distance, the predictions under US imaging were more accurate than under NIR imaging (Fig. 7), with examples of vessel segmentation in Fig. 8, 9.

Figure 6: Vessel identification by robotic and expert

Figure 7: Comparison of FCN and Rec-FCN for vessel segmentation and NIR stereo arm surface reconstruction

Figure 8: Vein and artery segmentation from B-mode and CDI sequences

Stereo reconstruction from NIR image sequences

The 3D surface map of vessel segmentation based on NIR stereo image sequences was reconstructed with great performance (Fig. 13).

Figure 9: Vessel segmentation and disparity estimation on NIR image sequences

Vessel classification from duplex US sequences

By binary vessel classification, the performance of CDI clutter filtering [31], Rec-FCN predictions from B-mode and concatenated two-channel duplex images increased orderly (Fig. 10a,b), which also proved the positive effect of recurrent units on classification performance.

(a) Receiver operating characteristic (ROC)

                     (b) Precision-recall

Figure 10: Performance comparsion of CDI, FCN, Rec-FCN, FCN-CDI by vessel classification

Robotic tracking and motion compensation

• Rec-FCN predictions from image sequences achieved lower tracking error than that from single image frames (Fig. 11).

Figure 11: Absolute error of robot tracking

• Robotic tracking would confront larger tracking errors when motion velocity rose (Fig. 12) and show a delay in trajectory over a velocity limit (Fig. 13).

Figure 12: Absolute translational and rotational errors during robotic tracking

Figure 13: Representative trajectories from robotic tracking and ground-truth

• Frame-to-frame motion estimates had obvious correlation with true displacements (Fig. 14) and could indicate a sudden change of velocity (Fig. 15a,b).

Figure 14: Correlation of predicted and ground-truth motions

(a) ROC and precision-recall by classifying sudden arm movements

(b) Reliability of detecting sudden arm movements

Figure 15: Sudden motion detection during robotic tracking

In vitro vessel insertion and in vivo fluid exchange

Insertion parameters that maximized robotic performance varied across different tissue conditions (Fig. 16). Specifically, among the 15 tissue properties we could find the most influential parameters (Fig. 17).

Figure 16: Effects of robotic insertion parameters on success rates in phantoms

Figure 17: Effects of phantom material properties on autonomous cannulation

With regard to first-stick success, the robot approaches outperformed significantly manual cannulation (Fig. 18a), blood collection and fluid delivery (Fig. 19) with and without NIR or US guidance. Moreover, robotic insertions were more consistent and particularly efficient in difficult conditions (Fig. 18b).

(a) With regard to operators

(b) With regard to difficulty levels

Figure 18: Comparison of manual unassisted, NIR-, US-guided and robotic cannulation by vascular access attempts

Figure 19: Comparison of manual unassisted, NIR-, US-guided and robitic by blood collection and fluid delivery

Furthermore, robotic cannulation showed its potential of identifying failure in vessel insertion (Fig. 20).

Figure 20: ROC and precision-recall of reliability in predicting failed access attempts

Discussion and recommendation

Further clinical assessment and applications in real-world scenarios

As robotic cannulation could reduce therapeutic risks with its higher success rate and consistency in vascular access [32], it is worthwhile to conduct clinical assessment among all possible populations and translate the paradigm into surgical assistance. Moreover, the capability of robot systems to identify different vessel branches could be utilized for surgeries in deep anatomy. In emergency conditions, robotic technologies could help gain vascular access faster.

Safety problem

In future, robot safety facilities could be further developed based on the present robot paradigm which was able to detect sudden arm motions, excess forces and failed access attempts during cannulation.

Robotic tracking

In this work, robot trajectories are updated according to pose predictions from deep learning network. Further researches could dive into learning-based strategies on trajectories compared to traditional robotic motion control.

Autonomy

Further promotions on autonomic robotic guidance for vascular access could consider the selection of the cannulation site as well as the flexibility by difficult anatomical structures.

Conclusion

In a word, the present work not only provides a medical robot utilizing recurrent fully convolutional neural network on near-infrared and ultrasound imaging sequences on automated vessel segmentation, 3D reconstruction, vessel identification as well as fluids drawing and infusion, but also suggests the feasibility of autonomic robotic systems outperforming professional practitioners in vascular interventions, particularly in difficult anatomical and physiological conditions.

Student review

This paper arranged a series of researches and devised a holonomic robotic system for the vascular access tasks on the basis of multidisciplinary knowledges and technologies. Most impressive was the successful implementation of full convolutional neural network prediction models which updated its knowledge along image sequences and compensated the motion during robotic tracking. Moreover, the integration of NIR and duplex US imaging guidance in robot system enhanced its capability of conducting a complete procedure of autonomous vessel cannulation and fluid exchange with high accuracy and consistency. Last but not least, the robot has shown its reliability of sudden motion and excessive force detection and been equipped with instantaneous needle disposal mechanism, which could prevent injuries to great extent.

One incompleteness lay in the insufficient empirical data. For example, the trained network under NIR guidance could probably be limited to a specific portion of the whole demographical spectrum, since the training data were acquired from only nine participants. Similarly, the robot tracking performance was only tested on 13 human beings and the in vivo studies involved only 20 rats. What’s more, as instantaneous movement led to an interrupt in robotic trajectories, the cannulation procedure might cost more execution time if the sensors are too sensitive.

As for the future researches, the characteristic of this robot device can be further evaluated and optimized through adequate clinical data and thus be implemented in surgical tasks. Furthermore, its sensitivity of accelerated speed and force detection can be adapted to security facilities of other autonomous robotic systems, which needs a lot of experiments and arrangement on the cooperation among sensors and actuators. Based on the experiments on different parameters of cannulation and artificial organs, the future robotic system could perform more precise guidance with optimized hyperparameters.

Reference

[1] Yang, G. Z. et al. Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci. Robot. 2, eaam8638 (2017).

[2] Niska, R., Bhuiya, F. & Xu, J. National hospital ambulatory medical care survey: 2010 emergency department summary. Natl Health Stat. Report 2010, 1–31 (2010).

[3] Sampalis, J. S., Lavoie, A., Williams, J. I., Mulder, D. S. & Kalina, M. Impact of on-site care, prehospital time, and level of in-hospital care on survival in severely injured patients. J. Trauma 32, 252–261 (1993).

[4] Bashkatov, A. N., Genina, E. A., Kochubey, V. I. & Tuchin, V. V. Optical properties of human skin, subcutaneous and mucous tissues in the wavelength range from 400 to 2,000 nm. J. Phys. D 38, 2543–2555 (2005).

[5] Lamperti, M. et al. International evidence-based recommendations on ultrasound-guided vascular access. Intensive Care Med. 38, 1105–1117 (2012).

[6] Egan, G. et al. Ultrasound guidance for difficult peripheral venous access: systematic review and meta-analysis. Emerg. Med. J. 30, 521–526 (2013).

[7] Seto, A. H. et al. Real-time ultrasound guidance facilitates femoral arterial access and reduces vascular complications: FAUST (Femoral Arterial Access with Ultrasound Trial). JACC Cardiovasc. Interv. 3, 751–758 (2010).

[8] Stolz, L. A., Stolz, U., Howe, C., Farrell, I. J. & Adhikari, S. Ultrasound-guided peripheral venous access: a meta-analysis and systematic review. J. Vasc. Access 16, 321–326 (2015).

[9] Hong, J., Dohi, T., Hashizume, M., Konishi, K. & Hata, N. An ultrasound-driven needle-insertion robot for percutaneous cholecystostomy. Phys. Med. Biol. 49, 441–455 (2004).

[10] de Boer, T., Steinbuch, M., Neerken, S. & Kharin, A. Laboratory study on needle–tissue interaction: toward the development of an instrument for automated venipuncture. J. Mech. Med. Biol. 7, 325–335 (2007).

[11] Carvalho, P ., Kesari, A., Weaver, S., Flaherty, P . & Fischer, G. Robotic assistive device for phlebotomy. In Proc. ASME 2015 International Design and Engineering Technical Conferences & Computers and Information in Engineering Conference Vol. 3, 47620 (ASME, 2015).

[12] Brewer, R. Improving Peripheral IV Catheterization Through Robotics—From Simple Assistive Devices to a Fully Autonomous System (Stanford University, 2015).

[13] Chen, A. I., Nikitczuk, K., Nikitczuk, J., Maguire, T. J. & Y armush, M. L. Portable robot for autonomous venipuncture using 3D near infrared image guidance. Technolog y 1, 72–87 (2013).

[14] Harris, R., Mygatt, J. & Harris, S. System and methods for autonomous intravenous needle insertion. US patent 9,364,171 (2011).

[15] Balter, M. L., Chen, A. I., Maguire, T. J. & Y armush, M. L. Adaptive kinematic control of a robotic venipuncture device based on stereo vision, ultrasound, and force guidance. IEEE Trans. Ind. Electron. 64, 1626–1635 (2017).

[16] Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 640–651 (2017).

[17] Valipour, S., Siam, M., Jagersand, M. & Ray, N. Recurrent fully convolutional networks for video segmentation. In Proc. 2017 IEEE Conference on Applications of Computer Vision 26–36 (IEEE, 2017).

[18] Ronneberger, O., Fischer, P . & Brox, T. U-Net: convolutional networks for biomedical image segmentation. Med. Image Comput. Comput. Interv. 9351, 234–241 (2015).

[19] He, K., Zhang, X., Ren, S. & Sun, J. Identity mappings in deep residual networks. In Proc. 14th European Conference on Computer Vision 630–645 (Springer, 2016).

[20] Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).

[21] Shi, X. et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In Proc. 28th International Conference on Neural Information Processing Systems 802–810 (MIT Press, 2015).

[22] Cho, K. et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proc. 2014 Conference on Empirical Methods in Natural Language Processing 1724–1734 (ACL, 2014).

[23] Chung, J., Çağlar, G., Cho, K. & Bengio, Y . Empirical evaluation of gated recurrent neural networks on sequence modeling. Preprint at https://arxiv.org/abs/1412.3555 (2014).

[24] Zhao, H., Gallo, O., Frosio, I. & Kautz, J. Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3, 47–57 (2017).

[25] Sudre, C. H., Li, W ., Vercauteren, T., Ourselin, S. & Jorge Cardoso, M. in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (eds Cardoso M. et al.) 240–248 (Springer, 2017).

[26] Chambolle, A., Caselles, V ., Novaga, M., Cremers, D. & Pock, T. in Theoretical Foundations and Numerical Methods for Sparse Recovery (ed. Fornasier, M) 263–340 (2010)

[27] Chen, A. I., Balter, M. L., Maguire, T. J. & Y armush, M. L. 3D near infrared and ultrasound imaging of peripheral blood vessels for real-time localization and needle guidance. In Medical Image Computing and Computer-Assisted Interventations Vol. 9902, 130–137 (Springer, 2016).

[28] Říha, K. Artery Databases (Brno University of Technology, 2014); http://splab.cz/wp-content/uploads/2014/05/ARTERY_TRANSVERSAL.zip

[29] Zukal, M., Beneš, R., Číka, I. P . & Říha, K. Ultrasound Image Database (Brno University of Technology, 2013); http://splab.cz/wp-content/uploads/2013/11/us_images.zip

[30] Le Guennec, A., Malinowski, S. & Tavenard, R. Data augmentation for time series classification using convolutional neural networks. Preprint at https://halshs.archives-ouvertes.fr/halshs-01357973 (2016).

[31] Bjærum, S., Torp, H. & Kristoffersen, K. Clutter filter design for ultrasound color flow imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 49, 204–216 (2002).

[32] Galena, H. J. Complications occurring from diagnostic venipuncture. J. Fam. Pract. 34, 582–584 (1992).