Robotics is using deep learning technology to accurately navigate the indoor environment, detect and track objects of interest, and move without collision. However, the increasing complexity of deep learning makes it challenging to adapt to these workloads on embedded systems. Although you can make a trade-off between accuracy and the size of the deep learning model, in most robotic applications, sacrificing accuracy to meet real-time requirements is often counterproductive.

Ease of use and deployment makes the NVIDIA Jetson platform the right choice for developers, researchers, and manufacturers to build and deploy robots such as jetbot, mushr, and mitracecar. In this paper, we propose a deep learning model for classification, object detection and human pose estimation on Jetson. We also provide an ros2 node to monitor various resources and operating parameters of Jetson during deployment. Ros2 provides a lightweight implementation because it eliminates the dependency of bridge nodes and provides various advantages in embedded systems.

We use the existing NVIDIA framework to deploy the deep learning model, such as tensorrt, to improve the model reasoning performance. We have also integrated the NVIDIA deepstream SDK and ROS 2 so that you can perform stream aggregation and batch processing, and deploy various AI models for classification and object detection, including resnet18, mobilenetv1 / V2, SSD, Yolo, fasterrcnn. In addition, we also implement ros2 nodes for developers around the world based on popular Jetson projects such as trt\u pose and jetson\u stats. Finally, we provide a GitHub repository for each of the applications mentioned above, including ros2 nodes and docker files, so that you can easily deploy nodes on the Jetson platform. For details on each project, see the following sections.

Ros2 node for human posture estimation

ros2_ trt_ The pose package is based on TRT_ Pose, which can perform pose estimation on the Jetson platform. The repository provides two trained models for pose estimation using resnet18 and densenet121. In order to understand human posture, the pre training model infers 17 body parts according to the categories of coco data sets.

ROS2-TensorRT. png

Figure 1 ros2 \ u TRT_ Description and output of the pose package.

Here is ros2_ trt_ Main functions of the pose package:

Publish pose_ Msgs, such as count of person and person_ id 。 For each person_ ID, which publishes 17 body parts.

Provide startup files for use and visualization on rviz2:

Image message

Visual markers: body_ joints , body_ skeleton

Contains the docker image based on Jetson, which is easy to install and use.

ROS 2 package for pytorch and tensorrt

ROS-2-TensorRT-classification. jpg

Figure 2. ros2 trt _ Classification and TRT_ Package description of the detection node.

Using pytorch, there are two packages for classification and detection, and each package implements the corresponding TRT version. These four software packages are a good starting point for roboticists using ros2 to start using pytorch for in-depth learning.

Tensorrt has been integrated into the package with the help of torch2trt to speed up inference. It generates a runtime engine that is optimized according to the network architecture and deployment devices.

The main features of these software packages are as follows:

For classification, you can choose from a variety of Imagenet pre training models, including resnet18, alexnet, squezenet, and resnet50.

For detection, SSD based on mobilenetv1 is currently supported for training on coco data sets.

The TRT package provides a significant acceleration in the execution of reasoning compared to the pytorch model, which executes reasoning directly on the GPU.

The reasoning results are published in the form of visual graphics.

When you run the node, a window is also displayed that visualizes the reasoning results.

Jetson based docker images and boot files are provided for ease of use.

ROS 2 node for deepstream SDK

ROS-2-DeepStream-publisher-node. png

Figure 3 package description of ROS 2 deepstream publisher node.

The deepstream SDK provides a complete stream analysis toolkit for building end-to-end AI based solutions using multi-sensor processing, video and image understanding. It supports popular object detection and segmentation models, such as the most advanced SSD, Yolo, fasterrcnn and maskrcnn.

NVIDIA provides ROS 2 nodes to perform two reasoning tasks according to the deepstream Python apps project, as shown below:

Target detection: four types of objects are detected: vehicle, person, roadsign and twowheeler.

Attribute classification: vehicle objects have three types of attributes: color, make, and type.

These publisher nodes receive single or multiple video streams as input from cameras or files. They perform reasoning and publish the results of detection and classification to different topics. We also offer to subscribe to these topics and_ An example of a ros2 subscriber node that displays results in msgs format. Each inference task also generates a visual window with bounding boxes and labels around the detected objects. Additional reasoning tasks and custom models can be integrated with the deepstream software architecture provided in this project.

In the video, the console at the bottom shows the average rate (in Hz) at which multi stream publishing nodes publish classified output.

vision_ Sample classification output in msgs classification2d format:

[vision_ msgs. msg. ObjectHypothesis(id=’silver’, score=0.7280375957489014), vision_ msgs. msg. ObjectHypothesis(id=’toyota’, score=0.7242303490638733), vision_ msgs. msg. ObjectHypothesis(id=’sedan’, score=0.6891725063323975)]

ROS 2 Jetson statistics

ros2_ jetson_ The stats package is a community building package for monitoring and controlling your Jetson devices. It runs on your terminal and provides a python package for easy integration in Python scripts. Using ros2_ jetson_ Stats library to build ROS 2 diagnostic messages and services.

ros2_ jetson_ The stats package has the following ROS 2 diagnostic messages:

GPU / CPU usage percentage

EMC / swap / memory status (% usage)

SoC Power and temperature

You can now control the following from the ROS 2 command line:

(ezx29 and speed:

Power model (nvpmodel)

jetson_ clocks

You can also provide a parameter to set the frequency of reading diagnostic messages.

For details, see nvidia-ai-iot/ros2_ jetson_ stats GitHub repo 。

ROS 2 container for Jetson

In order to easily run different versions of ROS 2 on Jetson, NVIDIA has released various dockerfiles and built scripts for ROS 2 eloquent and fox, as well as ROS melody and noetic. These containers provide an automated and reliable way to install ROS or ROS 2 on Jetson and build your own ROS based applications.

Because elocquent and melody are already ubuntu18. 04 provides pre built packages, so these versions of ROS are installed into containers by dockerfiles. On the other hand, foxy and noetic are built from the source code inside the container, because these versions are all for Ubuntu 20. 04 pre built. For containers, ROS or ROS 2 using these versions are the same, regardless of the underlying OS distribution.

To build the container, run jetpack4 at. Clone repo on the Jetson device of version 4 or later, and then start the ROS build script:

$ git clone $ cd jetson-containers $ 。/ scripts/docker_ build_ ros. sh all # build all: melodic, noetic, eloquent, foxy $ 。/ scripts/docker_ build_ ros. sh melodic # build only melodic $ 。/ scripts/docker_ build_ ros. sh noetic # build only noetic $ 。/ scripts/docker_ build_ ros. sh eloquent # build only eloquent $ 。/ scripts/docker_ build_ ros. sh foxy # build only foxy

This command creates a container with the following tags:





For example, to start the ROS 2 foxy container, run the following command:

$ sudo docker run –runtime nvidia -it –rm –network host ros:foxy-ros-base-l4t-r32.4.4

In addition to other hardware accelerators (such as video encoders and decoders) on the Jetson device, the use of the – runtime NVIDIA flag automatically enables GPU pass through in the container. To transfer Mipi CSI cameras in a container, include the following flags:

–volume /tmp/argus_ socket:/tmp/argus_ socket

To transfer the v4l2 USB camera in the container, install the required /dev/video* device when starting the container:

–device /dev/video0

NVIDIA Omniverse Isaac simulation platform for ROS developers

The NVIDIA Isaac simulation toolkit built on the Omniverse NVIDIA platform brings some useful improvements over the existing robot workflow:

It leverages Omniverse’s highly accurate physical simulations and realistic ray tracing graphics, and integrates directly with industry-leading physical frameworks, such as the NVIDIA PhysX SDK for rigid body dynamics.

It refocuses on interoperability, deep integration with NVIDIA Isaac SDK, and extension of ROS.

It is easy to extend. Through its Python based scripting interface, it allows you to adapt to your own unique use cases.

It is built for deployment. Its architecture supports workflows on local workstations and works with NVIDIA NGC through the cloud.

About author

Amey Kulkarni is a development technology engineer at NVIDIA, focusing on effectively deploying deep learning solutions to the edge. Before joining NVIDIA in 2019, he completed his PhD in computer engineering from the University of Maryland. His research interests are the deployment of digital signal processing, computer vision and deep learning systems on embedded platforms.

Rishabh Chadha is NVIDIA’s embedded Engineering Intern – AI. He focuses on integrating deep learning and robotics framework for NVIDIA Jetson platform. He will graduate from Worcester Institute of technology in 2021 with a master’s degree in robotics. His interests mainly include deep learning, medical imaging and robot perception.

Asawaree Bhide is an AI embedded Engineering Intern of NVIDIA, dedicated to optimizing and deploying the deep learning model on edge devices. She is currently pursuing a master’s degree in computer science at Georgia Institute of technology and is interested in solving complex perceptual tasks that are navigated autonomously by specific agents. Tomasz Lewicki is an embedded engineering intern at NVIDIA. He holds a master’s degree in computer engineering from San Jose State University and a bachelor’s degree in robotics engineering from Warsaw University of technology. His interests focus on computer vision and deep learning of robot applications.

Reviewed by: Guoting

Leave a Reply

Your email address will not be published. Required fields are marked *