How to deploy the DepthVista (TOF Camera) SDK for Isaac SDK 2021.1 on a Jetson board

As you may know, Time of Flight cameras leverage the light detection technique wherein they illuminate the object with a laser and a sensor sensitive to the laser’s wavelength to capture the reflected light. Then, the camera’s sensor calculates the distance between the sensor and the object by using the time difference between the light emission and when the light returns to the camera.

Owing to their ability to accurately measure depth, TOF cameras are one of the best choices for enabling seamless navigation in Autonomous Mobile Robots (AMRs) such as goods-to-person robots, pick and place robots, patrol robots, harvesting robots, automated weeders, etc. TOF cameras also come with a few more advantages compared to other depth mapping technologies such as stereo and structured light.

To learn more about how a time of flight camera works, how it compares with its alternatives, and how it helps Autonomous Mobile Robots see, please visit the below articles:

• What is a TOF sensor? What are the key components of a TOF camera?
• How Time-of-Flight (TOF) compares with other 3D depth mapping technologies
• How does an Autonomous Mobile Robot use time of flight technology?

To address the rising demand for time of flight cameras in the AMR and other automated vehicles market, e-con Systems recently launched DepthVista – a 3D time of flight camera with an accuracy of <1%.

In this article, we do a deep dive into the steps involved in deploying the DepthVista SDK for Isaac SDK 2021.1 on a Jetson board (To learn more about the NVIDIA Isaac ROS, please check out the article What is the NVIDIA® Isaac ROS? How to get started with it?)

But first, let us learn why DepthVista is the perfect camera solution for your depth mapping needs.

Why DepthVista is your TOF camera of choice

e-con Systems designed and developed DepthVista, a Time-of-Flight camera that provides depth information and RGB data in one frame. This helps in ensuring simultaneous depth measurement and object recognition. DepthVista is built for precise depth measurement as it is equipped with a 3D depth sensor that streams video data at a high resolution of 640 x 480 @ 30fps. It also houses an AR0234 color global shutter sensor from onsemi® that streams HD and FHD @30fps.

This 3D Time of Flight camera comes with two operational modes – the close range mode and the far mode. The former facilitates depth measurement from 0.2m to 1.2 m, while the latter operates from 1m to 6m.  Also, DepthVista uses a VCSEL of 850nm, which is safe for human eyes and enables your system to operate even in absolute darkness.

Please check out the below video to learn more about the features and applications of DepthVista:

3D Time-of-Flight (ToF) camera for accurate 3D depth imaging

Prerequisites for deploying DepthVista (TOF Camera) SDK for Isaac SDK 2021.1

Let us now get into the detailed process involved in configuring the DepthVista SDK for Isaac SDK 2021.1 on a Jetson board. Here’s a list of what you need for this:

• Isaac sdk 2021.1
• Jetson Devices: Jetson AGX Xavier, Jetson Xavier NX or Jetson TX2, Jetson Nano 2GB/4GB
• Jetpack version: Jetpack 4.5.1
• Desktop computer with Ubuntu 18.04 + NVIDIA GPU and a compute capability of 6.1

How to install the DepthVista (TOF Camera) SDK for Isaac SDK 2021.1

Installing Isaac SDK on desktop:

Download the Isaac SDK from the link https://developer.nvidia.com/isaac-sdk-20211 and extract the SDK in a preferred location (example: tar xf isaac-sdk-20210609-e336b5195.tar.xz -C ~/isaac).

Installing dependencies on desktop:

Install dependencies at the root folder of the Isaac SDK  repository by running the following command:
user@desktop:~/isaac/engine$ ./engine/build/scripts/install_dependencies.sh

Deploying DepthVista (TOF Camera) SDK for Isaac SDK 2021.1

First, you should download the DepthVista TOF Camera SDK from the git repository – https://github.com/econsystems/depthvista-isaac-package.git to a temporary location. Then, make sure to follow the below steps in order to deploy the DepthVista TOF Camera SDK for Isaac SDK 2021.1.

1. 1. Go to the DepthVista TOF Camera SDK downloaded path
   2. Run the deploy script with the Isaac SDK 2021.1 installed path

user@desktop:~/depthvistacamera_sdk$ chmod +x ./deploy.sh
user@desktop:~/depthvistacamera_sdk$ ./deploy.sh ~/isaac


How to deploy and run the SDK on Jetson boards

Now, let’s find out how to deploy an application from your desktop machine to the Jetson board and how to effectively run the same. It’s important to note that the Isaac SDK fully supports cross-compilation for Jetson. Hence, compiling the source code on Jetson itself is not recommended.

And remember that the first two of the below steps need to be run only once.

1. Make sure you have an SSH key on your desktop machine.

2. Copy your SSH identity to Jetson using the user name and IP you use to log on the robot with a command similar to the following:
   user@desktop:~/isaac/sdk$ ssh-copy-id <username_on_jetson>@<jetson_ip>
   where <username_on_jetson> is your user name on the Jetson, and <jetson_ip> is the IP address of the Jetson.

3. To run the DepthVista camera sample application on Jetson, first deploy the package to Jetson with the following command:
   user@desktop:~/isaac/sdk$ ./../engine/engine/build/deploy.sh --remote_user
<username_on_jetson> -p //apps/samples/depthvista_camera:depthvista_camera-pkg -d jetpack45 -h <jetson_ip>
   where <username_on_jetson> is your user name on the Jetson, and <jetson_ip> is the IP address of the Jetson.

4. Login to Jetson to run the application:
   user@desktop:~/isaac$ ssh JETSON-USER@JETSON-IP

5. Go to the deployment folder and run the application:
   user@jetson:~/$ cd deploy/user/depthvista_camera-pkg
user@jetson:~/deploy/user/depthvista_camera-pkg$ sudo ./apps/samples/depthvista_camera/depthvista_camera

   Here “user” is the user name you use on your host system.

6. Once the application is running, connect to it in your browser and inspect the running application with websight. To do so, navigate to http://JETSON-IP:3000 in your browser.

Published nodes of the DepthVista camera

e-con’s DepthVista camera is capable of providing color image in UYVY format and the depth value in meters and IR images (grey scale).

The color image is published in the node “/camera/depthvista/rgb”.

There are two modes in depth, SHORT and LONG ranges, which can be configured dynamically. When SHORT range is configured, the depth image is published in the node “/camera/depthvista/depth_short” and in the case of LONG range, the depth image is published in the node “/camera/depthvista/depth_long”.

The IR image is published in the node “/camera/depthvista/ir”.

The camera’s extrinsic and intrinsic details are published in the below nodes:

• Camera pair extrinsics (TOF-to-RGB): “/camera/depthvista/extrinsics”
• RGB intrinsics: “/camera/depthvista/rgb_intrinsics”
• Depth intrinsics: “/camera/depthvista/depth_intrinsics”

Dynamic re-configuration settings

The run-time parameter(s) is listed below:

TOF Range mode: 0 (short 0.2 to 1.2 meters), 1 (long 1.0 to 6 meters)

Final thoughts

For more than a decade, e-con Systems has been at the forefront of innovation in bringing depth cameras to the market. In fact, we were one of the first companies in the world to launch a stereo camera for embedded vision applications. Today, we have evolved to offer a variety of 3D depth cameras like DepthVista that deliver truly path-breaking embedded vision experiences. Visit our page to see the full list of 3D Depth cameras.

If you are looking for help in integrating e-con’s Time of Flight camera into your application, please write to us at camerasolutions@e-consystems.com.