High precision real-time reasoning is a challenging task, especially in the low visibility environment. With the NVIDIA Jetson embedded platform, the recently concluded Defense Advanced Research Projects Agency (DARPA) subterranean challenge (subt) team can detect objects of interest with high accuracy and throughput. In this article, we will introduce the results, systems and challenges faced by the team in the last stop of the system competition.
Subt challenge is an international robot competition organized and coordinated by DARPA. The competition encourages researchers to develop new methods for robots to map, navigate and search environments that pose challenges such as low visibility, hazards, unknown maps or poor communication infrastructure.
2019 coronavirus disease includes three preliminary circuit events: tunnel circuit, urban circuit and cave circuit (cancelled due to the covid-19 pandemic), and the final comprehensive challenge course. Each track and final are held in different environments and different terrains. According to the organizers of the event, the competition will be held in three different stages, and will be held in KY in September 2021. Louisville held its last match.
Subt challenge’s competitors use NVIDIA technology to meet their hardware and software requirements. The team uses the desktop / server GPU to train the model deployed on the robot using NVIDIA Jetson embedded platform to detect interesting artifacts and objects in real time, which is the main standard for determining the winning team. Five out of seven competitors also use the Jetson platform for real-time target detection.
The subt challenge is inspired by the real scenarios that the first responder faces during a search and rescue operation or disaster response.
The most advanced methods developed through this competition will help to reduce the risk of casualties of search and rescue personnel and first aid personnel when exploring unknown underground environment. In addition, the automatic robot will assist the staff to explore the environment, find survivors, objects of interest, and enter places at risk to humans.
Figure 1. DARPA underground challenge explores innovative methods and new technologies for mapping, navigating and searching complex underground environments. – Picture provided by DARPA.
The competition involves various technical challenges, such as dealing with unknown, unstructured and uneven terrain that robots may not be able to easily manipulate.
These environments typically do not have any infrastructure to communicate with the central command. From the perspective of perception, the visibility of these environments is very low, and the robot must find the artifacts and objects of interest.
The task of the competitive team is to meet these challenges by developing new sensor fusion methods and developing new or modified existing robot platforms with different abilities to locate and detect objects of interest.
Cerberus team (a cooperative walking and flying robot for autonomous exploration in underground environment) is a consortium of universities and industrial organizations around the world.
The team participated in the competition with four quadruped robots named anymal, five UAVs with variable size and payload capability, which are mainly manufactured in-house, and a roaming robot in the form of a super giant robot. In the final of the competition, the team finally used four animal robots and super giant robots for exploration and artifact detection.
Each animal robot is equipped with two CPU based computers and an NVIDIA Jetson AgX Xavier. The rover robot is equipped with NVIDIA GTX 1070 GPU.
Cerberus team uses the improved version of you only look one (Yolo) model for target detection. The model uses two NVIDIA RTX 3090 GPUs to train on 40000 tagged images.
Before deploying to Jetson for real-time reasoning, tensorrt is used to further optimize the trained model. Jetson AgX Xavier can reason at a collective frequency of 20 Hz. In the finals of the competition, Cerberus team took the lead in discovering 23 of the 40 cultural relics in the environment and won the first place.
Cerberus team also used GPU to draw topographic elevation map and train the movement strategy controller of the anymal quadruped robot. Use Jetson AgX Xavier to draw elevation map in real time. The mobile strategy training of anymal robot in rough terrain is completed offline using desktop GPU.
Team co stars
Led by researchers from NASA’s Jet Propulsion Laboratory (JPL) in Southern California and other university and industrial collaborators, the team cooperation underground autonomous robot (co star) won the 2020 competition, which focuses on exploring the complex underground urban environment.
They also successfully participated in the 2021 mixed artificial and natural environment competition, ranking fifth. The co starring team took part in the competition with four positions, four husky robots and two UAVs.
In the last round, due to unexpected hardware problems, the team finally used one spot and three husky robots. Each robot is equipped with a CPU based computer and an NVIDIA Jetson AgX Xavier.
For target detection, the team uses RGB and thermal images. They used a medium variant of the Yolo V5 model to process high-resolution images for real-time inference. The team trained two different models to reason between captured RGB and thermal images.
The image-based model uses about 54000 tagged frames for training, while the thermal image model uses about 2400 tagged images for training. In order to train the model on their customized dataset, the team co star used the pre trained Yolo V5 model on the coco dataset, and used the NVIDIA transmission learning Toolkit (called Tao Toolkit) for transmission learning.
The model is trained using two internally deployed NVIDIA A100 GPUs and an AWS instance consisting of eight V100 GPUs. Before deploying the model on Jetson AgX Xavier, the team prunes the model using tensorrt.
With this setting, the team cooperation star can infer RGB images received by five realsense cameras and images received by one thermal camera at a frequency of 28 Hz. In the last run, the robot can detect all 13 workpieces in the designated area. Exploration time is limited due to deployment delays caused by unexpected hardware problems at the deployment site.
Equipped with NVIDIA Jetson platform and NVIDIA GPU hardware, teams competing in DARPA SUT events can effectively train models for real-time reasoning, solve challenges brought by underground environment and accurate target detection.
MITESH Patel is the developer relations manager of NVIDIA. He works with higher education researchers to implement their ideas using the NVIDIA SDK and platform. Prior to joining NVIDIA, he was a senior research scientist of Fuji Xerox Palo Alto laboratory Co., Ltd. and was committed to developing indoor localization technology for asset tracking of hospitals and delivery vehicle tracking of manufacturing facilities. MITESH received his PhD in robotics from the automated systems center (CAS) at the University of technology, Sydney, Australia in 2014.
Reviewed by: Guoting