How to achieve 4K@30fps recording in NVIDIA Jetson platforms

“We need to record 4K@30fps seamlessly without losing any frames in H.264 format.”

We get this requirement from all our customers who use 4K cameras like e-CAM130A_CUXVR for their surveillance applications. While our cameras can stream 4K@30fps in the MIPI interface and NVIDIA Jetson platforms also support hardware encoding for the same, there are certain limitations in the existing application software. Especially in cases when Gstreamer handles the frames for H.264 encoding.

In this blog, you get to see how we helped one of our customers to overcome these limitations and achieve 4K@30 fps recording with H.264 format in NVIDIA Jetson platforms.

Video recording using Gstreamer: The conventional approach

While recording the videos using the Gstreamer pipeline or the Gstreamer application, the camera frames must be passed to multiple software stack/plugins for video conversion, video encoding and muxing. Hence, frame drop issues may be encountered during recording. However, you can access the HW encoder through Gstreamer – as shown in the diagram below.

In the conventional Gstreamer pipeline, the captured frames are copied into the new buffer (by local memcpy) and wrapped with Gstreamer metadata for processing and transferring content between plugins. For example, you can find the memory copy in gstomxh264enc plugin. This will decrease the performance – causing frame drops. The conventional Gstreamer recording pipeline in Jetson NVIDIA platforms is showcased in the below diagram.

A quicker and more convenient approach by e-con Systems

e-con has developed a Gstreamer app source to overcome this major issue – using NVIDIA Tegra Multimedia APIs. Tegra Multimedia APIs are a collection of lower-level APIs that support flexible application development by providing better control over the underlying hardware blocks. NVIDIA Jetson platforms have their own hardware encoding and decoding blocks – directly accessible via these APIs.

You can use these APIs to execute video format conversion, scaling, cropping, encoding, and decoding in the hardware memory (memory copy done by hardware) to improve overall performance. The ability to access the hardware blocks directly improves camera streaming and recording performance. 

It is important to note that the multimedia APIs are separate from the GStreamer framework (which provides high-level APIs).

When recording the videos using the Tegra multimedia application, the camera frames received from the v4l2 stack is directly processed in the hardware encoder driver. This eliminates the CPU buffer usage to transfer the frames via the Gstreamer stack to the hardware driver, as discussed above.

You can access the hardware encoder through multimedia APIs, as shown in the below diagram.

e-con Systems’ Gstreamer app source can also perform the conversion and encoding – using multimedia APIs. The captured camera frames are converted to I420 format and then encoded to H264 format using multimedia APIs. It is mixed with the MP4 format by leveraging the Gstreamer mp4mux plugin. This method eliminates the frame drop since most of the conversion and compression are completed in the multimedia API stack. It means that the Gstreamer stack concentrates only on recording.

The below diagram illustrates e-con Systems’ unique approach.

When we use multiple cameras for recording, we notice that the frame drop occurs due to the maximum bandwidth utilization as far as hardware is concerned. But the performance is better when compared to conventional Gstreamer for the same number of cameras.

If you need this Gstreamer app source and the sample application source or have any questions about our approach, we’d be more than happy to help.

Related cameras

• e-CAM80_CUNX – Sony 4K Camera for NVIDIA® Jetson Xavier™ NX
• e-CAM130A_CUXVR – Four Synchronized 4K Cameras for Jetson AGX Xavier™
• Multi-Camera solution with 180° image stitching – Jetson AGX Xavier

Please write to camerasolutions@e-consystems.com.