Today, in industries like retail and sports broadcasting, there has been a surge in demand for multiple cameras to capture the entire scene in a single frame. The deciding factor in these cases ends up being the ability to synchronize multiple cameras perfectly. However, traditional multi-camera set-ups faced many issues concerning transitions and synchronization. Hence, it ended up causing poor audio quality and shoddy camera angles.
This is a strict no-no, especially for cutting-edge retail devices like smart carts. Since the accuracy of imaging determines the quality of information available for edge-based AI inferencing, the effectiveness of the multi-camera system has a clear and direct impact on end-customer experiences.
In this blog, let’s discuss the burgeoning need for smooth multi-camera synchronization to help retailers build truly customer-friendly smart carts and autonomous checkout systems. We will also find out about the nuances of implementing it the right way for your retail applications.
What is multi-camera synchronization?
Multi-camera synchronization starts at the hardware frame-level by connecting all the cameras through the “Master”- “Slave” configuration. You must have control over individual cameras and be in a position to select the number of cameras to be streamed. Such systems can ensure multi-image capturing, multi-video recording, and multi-network streaming.
The synchronization between multiple cameras occurs using hardware and software trigger modes, which will be explained later in this blog. Both methods have their own advantages and limitations. Your selection should be based on the specific goals that your application is looking to achieve.
Two major challenges of multi-camera synchronization
When it comes to integrating a multi-camera solution, there are predominantly two practical challenges that come in the way:
- The readout from the sensors should happen simultaneously to achieve a perfect frame during the image capture.
- The latency between the different cameras must be less than or around 10 milliseconds.
Both these challenges have to be overcome in a simple multi-camera system as well as a 180-degree stitching camera where multiple images have to be combined to form a 180-degree view of the scene. We will discuss how to address these issues in an upcoming section. But first, let us look at why multi-camera synchronization is critical in cameras for autonomous checkout systems and smart carts.
Why multi-camera synchronization is invaluable for smart carts
Multiple cameras operate together to identify the product in a smart cart. The images taken from different cameras at different angles should be in sync. This is critical for the object detection algorithm to identify the product easily.
It’s invaluable because traditional cash counter checkout systems have already become passe in today’s age of contactless experiences. The average retail customer is starting to prefer walking into a store, picking what they want to buy and walking out again without any physical interactions. Smart carts, leveraging customer identification, tracking, and product recognition algorithms, make sure that this is a reality. But your choice of the camera is crucial because if the smart cart or autonomous checkout system doesn’t work properly, it leads to bitter customer and retail staff experiences and unnecessary time delays.
How to implement multi-camera synchronization in smart carts the right way
A multi-camera system if not implemented properly in a smart cart can lead to increased error rates and hence the need for more frequent interventions from the retail staff. In this section, let us learn how to do a multi-camera implementation the right way.
For example, if you require a solution involving six cameras to cover your entire retail cart, you can connect four USB cameras to cover the corners of the cart and two in the center of the cart to create the perfect image. The synchronization between the cameras is for a single interface – either USB or MIPI. Hence, you have to design and architect your smart cart device in a way that it can support both USB and MIPI cameras.
Implementing multi-camera synchronization involves configuring either a hardware trigger or a software trigger mode. Let’s now look at how these are done in the USB camera firmware and MIPI camera driver.
Using the hardware trigger
While using hardware trigger in the application, a PWM (Pulse Width Modulation) signal of 20Hz with 10% duty cycle is produced externally, which is used to trigger the sensor. The USB and MIPI cameras are put in “slave” mode. Basically, readout from the sensor doesn’t happen since there is no external PWM signal.
Since all the cameras are connected to the same PWM source, you can be sure that there are no lags during readout between the cameras. Furthermore, it means that the multi-camera system is bound to yield a properly synchronized frame.
It’s pertinent to note that the limitation in hardware trigger is that there is a need for an external PWM circuit to control the cameras.
Using the software trigger
In the software trigger mode, a customized firmware is developed to send a frame whenever the host places a request. So, the camera is in a continuously running state, and upon receiving the host’s request through HID command, it sends the frame enabling proper synchronization.
Latency tends to be a challenge while using a software trigger to synchronize different interfaces (in this case – four USB cameras). The latency between sending a request to all cameras depends on the host device’s performance. Therefore, in the case of low-performing host devices, the latency is very likely to vary.
How to overcome latency challenges
We have found that whether using a hardware or software trigger, the latency generally varied in the range of 10 milliseconds, which is great for smart carts.
But in the solution setup we spoke about earlier, four USB cameras are used in the embedded platform, which would limit the USB bandwidth. To overcome this, you can increase the line time of sensor readout to utilize the available bandwidth. By doing so, you can also increase the blanking between the frame readouts. So, it helps optimize the bandwidth utilization while avoiding frame corruption.
Synchronized global shutter multi-camera system from e-con Systems
As an OEM Camera Solution Provider, e-con Systems has developed a portfolio of embedded multi-camera solutions for an array of retail applications like smart carts and autonomous checkout systems. One of our best-selling products e-CAM217_CUMI0234_MOD – Full HD color global shutter camera module – is an ideal vision solution for multi-camera synchronization. The camera comes in both USB and MIPI variants. The MIPI variant is compatible with the most popular processor platforms and carrier boards such as the NVIDIA Jetson series, NXP i.MX8, Raspberry Pi 4, Connect Tech’s Photon AI dev kit, AVerMedia EN715 carrier board, and Diamond Systems’ FLOYD carrier board.
e-CAM217_CUMI0234_MOD is based on the AR0234 global shutter sensor from Onsemi and can be connected to the processors mentioned above for maximum performance. It provides Full HD resolution in UYVY format at 60 FPS – with perfectly synced images and a latency of just 10 milliseconds.
To learn more about the features of this Full HD color global shutter camera module, check out e-CAM217_CUMI0234_MOD’s product page.
You could also watch the below videos to learn more about the USB and Xavier NX variants of this camera:
Other multi-camera solutions from e-con Systems
- e-CAM130A_CUXVR – Four Synchronized 4K Cameras for Jetson AGX Xavier™
- Multi-Camera solution with 180° image stitching – Jetson AGX Xavier
- e-CAM25_CUXVR – Six Synchronized Global Shutter Cameras for Jetson AGX Xavier™
- SurveilsQUAD – Sony IMX290 Synchronized Multi-Camera System
- e-CAM30A_HEXCUTX2 – Six Synchronized Full HD Cameras for NVIDIA® Jetson AGX Xavier™/Jetson™ TX2
If you are looking for help in integrating high-performance multi-camera systems into your smart carts or any other applications, please write to us at camerasolutions@e-consystems.com.
Vinoth Rajagopalan is an embedded vision expert with 15+ years of experience in product engineering management, R&D, and technical consultations. He has been responsible for many success stories in e-con Systems – from pre-sales and product conceptualization to launch and support. Having started his career as a software engineer, he currently leads a world-class team to handle major product development initiatives