What are RGBD cameras? Why RGBD cameras are preferred in some embedded vision applications? In recent years, 3D vision technologies have evolved significantly improving their accuracy, range, low light performance, etc. And there has been a surge in demand for depth sensing in many embedded vision applications. Some of the most popular applications where a depth camera is used are autonomous mobile robots, automated guided vehicles, remote patient monitoring, and people counting systems. While we all understand what a depth camera is, the concept of an RGB depth camera (or RGBD camera) is a mystery to many. In this blog post, we will attempt to learn what an RGBD camera is and why it is a game-changer in some of the most popular embedded vision applications.
What is an RGB camera?
Before we learn about an RGBD camera, we need to first understand what an RGB camera is. The term RGB refers to the color model in which the red, green, and blue primary colors of light are added to create different colors that we perceive. An RGB camera is used to deliver colored images of people and objects by capturing light in red, green, and blue wavelengths (RGB). This camera utilizes visible light with a wavelength ranging from 400 to 700nm.
What is an RGBD camera?
An RGBD camera is a type of depth camera that provides both depth (D) and color (RGB) data as the output in real-time. Depth information is retrievable through a depth map/image which is created by a 3D depth sensor such as a stereo sensor or time of flight sensor. RGBD cameras are able to do a pixel-to-pixel merging of RGB data and depth information to deliver both in a single frame. For instance, DepthVista – a time of flight based RGBD camera from e-con Systems – has the capability to deliver both the depth and RGB data in a single frame.
Figure 1 – DepthVista: 3D time of flight RGBD camera from e-con Systems
For the purpose of understanding how this works better, please have a look at the below figure that illustrates a sample application used to stream image and depth data from DepthVista. Here, the camera displays a depth frame (640 x 480), an IR frame (640 x 480) and an RGB frame (640 x 480) at 30 fps along with a 3D viewer.
Figure 2 – sample application streaming RGB and depth data from e-con’s DepthVista
Why RGBD camera is one of the most preferred cameras in certain embedded vision applications?
Adding RGB data to a 3D depth sensing camera’s point cloud or depth map allows to pinpoint objects more effectively for pattern recognition or detection. This is particularly useful in applications that need to identify the type and nature of objects in the scene along with measuring the depth to them. And the ability to deliver these two types of data in a single frame makes the camera even more suitable for applications such as facial recognition based anti-spoofing systems and people counting devices. To learn more about how RGB depth cameras can enhance the performance and effectiveness of more such embedded vision applications, we recommend you check out the two articles given below:
- How does a Time-of-Flight camera make remote patient monitoring more secure and private?
- How does an Autonomous Mobile Robot use time of flight technology?
Introduction to ToF Technology in RGBD Cameras
Time-of-Flight (ToF) technology in RGBD cameras offers accurate depth measurement by calculating the time it takes for light to travel from the camera to the object and back. This technology enhances depth accuracy, especially in low-light or challenging environments.
Basically, RGBD cameras combine RGB (color) data with ToF depth sensing to deliver detailed 3D imaging and accurate spatial mapping, which makes them ideal for use cases like robotics, gesture recognition, and 3D modeling. ToF technology also offers fast, real-time processing, ensuring superior performance in dynamic environments.
RGBD camera with a combination of Time of flight and RGB sensors
A time of flight based RGBD camera is one of the most common types of RGBD cameras that uses a ToF 3D sensor for depth measurement and an RGB sensor for capturing color information. The ToF sensor calculates the distance based on the total flight time of the light (considering the velocity of light) which can be done even in complete darkness.
To learn more about the concept of ToF and how ToF camera systems work, have a look at the article What is a Time-of-Flight sensor? What are the key components of a Time-of-Flight camera? If you are interested in learning how time of flight compares with other depth-sensing technologies, please read the article How Time-of-Flight (ToF) compares with other 3D depth mapping technologies.
The following block diagram shows the architecture of a ToF + RGB camera:
Popular Applications of RGBD Cameras
Automated parking systems
RGBD cameras provides accurate depth and object detection for automated parking systems to ensure effective space identification and vehicle navigation. It enables smart, hands-free parking in tight environments, reducing the risk of accidents.
Agriculture monitoring drones
RGBD cameras in drones help monitor crops, combining depth data to assess plant growth with RGB data for color-based health analysis. It supports better decision-making in irrigation, pest control, and harvesting.
Warehouse inventory management systems
In warehouse logistics, RGBD cameras enhance real-time inventory management by capturing 3D data on stock levels and locations. That way, it becomes easy to automate inventory counts and reduce errors in tracking misplaced items.
Autonomous Guided Vehicles (AGVs)
RGBD cameras enable AGVs to navigate factory floors or warehouses with precision. They capture 3D data that allows these vehicles to identify obstacles while being able to navigate around them, ensuring smooth operation even in dynamic environments.
Autonomous Mobile Robots (AMRs)
AMRs use RGBD cameras for spatial awareness to autonomously navigate environments like warehouses or hospitals. Depth data helps these robots detect obstacles and plan paths, while RGB imaging ensures object identification, improving the efficiency of tasks such as picking, packing, or material handling.
Robotic pick-and-place systems
RGBD cameras support robots in accurately identifying and handling objects on production lines. The depth and color information equip the robots to gauge size, position, and orientation, ensuring precision in sorting and assembling items.
e-con Systems’ ToF-based RGB depth camera
DepthVista from e-con Systems is an RGB depth camera that uses a CCD sensor for 3D depth measurement and the AR0234 color global shutter sensor from Onsemi for capturing RGB data. This RGBD camera uses a VCSEL of 850nm for high-speed data communication and is capable of streaming VGA (640X480) at 30 fps. The 2D RGB video is integrated with the 3D depth video to create the colored point cloud (a collection of points in three-dimensional space). Then the post processing of 3D depth data can be performed using post processing filters such as planarization, temporal filter, edge detection, and spatial filter. The depth frame with all the post processing filters applied is shown in the following figure:
Figure 4 – Comparison between a normal depth frame and a depth frame with all the post processing filters applied
To learn more about the features, benefits, and target applications of the camera, please visit DepthVista’s product page. You could also watch the below video to get a quick overview of the product.
3D Time-of-Flight (ToF) camera for accurate 3D depth imaging
Browse our Camera Selector page to view e-con Systems’ complete portfolio.
If you require help integrating RGBD cameras into your embedded vision applications, please write to camerasolutions@e-consystems.com.
References:
Prabu is the Chief Technology Officer and Head of Camera Products at e-con Systems, and comes with a rich experience of more than 15 years in the embedded vision space. He brings to the table a deep knowledge in USB cameras, embedded vision cameras, vision algorithms and FPGAs. He has built 50+ camera solutions spanning various domains such as medical, industrial, agriculture, retail, biometrics, and more. He also comes with expertise in device driver development and BSP development. Currently, Prabu’s focus is to build smart camera solutions that power new age AI based applications.