An ISP is a key component in an embedded camera system since a sensor provides the output only in the RAW format. An ISP (Image Signal Processor) is a dedicated processor that converts this RAW image data into a high quality workable output through various processes such as noise reduction, lens shading correction, gamma correction, auto exposure and auto white balance etc.
That said, a logical question that might arise is why don’t image sensors come integrated with an ISP. Why don’t sensor manufacturers such as Sony, ON Semiconductor, or OmniVision consider adding value to their sensor products by integrating a dedicated ISP?
Well, it got me curious too.
So, in this article I attempt to break down the reasons for sensor manufacturers not offering a combination of a sensor and ISP as an integrated solution.
A look back before we start
Before we start looking at the reasons, a bit of history might help.
When we say that sensors don’t come with ISPs, was it always the case? Not really.
Sensor manufacturers used to offer ISPs as well in the past. The last known image sensor that continues to have an integrated ISP and is still selling in volumes is OmniVision’s OV5640, which is a 1/4 inch 5MP camera. However, as some of you may be already aware, OmniVision has decided not to launch image sensors with integrated ISP beyond 5MP anymore.
Reasons for image sensors not offered with an ISP anymore
Image sensors do not come integrated with ISPs anymore because of predominantly 2 reasons:
- Growth of microprocessors with built-in ISP
- Product developers wanting to choose the ISP based on tuning requirements
Let us look at each of these in detail.
Growth of microprocessors with built-in ISP
Earlier, processors didn’t have a built-in ISP. But today, most of the modern manufacturers like Qualcomm, NXP, and NVIDIA offer an ISP with their microprocessors. Due to this, sensor manufacturers do not want to add to their product cost by integrating an ISP within. Since an ISP is not a value add anymore, sensor manufacturers do not want to run the risk of losing a design win or a bulk order on budget or cost grounds.
Product developers wanting to choose the ISP based on tuning requirements
The second significant reason for manufacturers to come up with raw Bayer filter sensors is that many product developers and design engineers want to pick the ISP on their own based on the tuning they would require. It is also dependent on the features and the interfacing supported by the ISP.
Different ISPs come with different feature sets, and picking and integrating an ISP many a times requires a thorough understanding of the sensor as well. One of e-con Systems’ key strengths is having a suite of products that come with different ISPs for different types of applications.
For example, Hyperyon – a 2MP ultra low light USB camera based on the Sony STARVIS IMX290 sensor – uses the Socionext ISP. On the other hand, e-CAM21_CUTX2 – Sony STARVIS IMX290 camera for NVIDIA Jetson TX2 – uses the NVIDIA Jetson ISP. While Hyperyon is targeted at low-light HDR USB 2.0 applications, e-CAM21_CUTX2 is suited only for low-light MIPI CSI-2 based applications.
External ISP vs Internal ISP
Now that we understand that an ISP comes integrated with a processor (which is when it is called an internal ISP), we need to look at why we might need an external ISP in some cases.
It is well understood that USB cameras definitely need an external ISP. So, the question of choice between an internal and external ISP arises only in the case of other cameras.
Even though processors come with an internal ISP, they have not yet become as sophisticated as external ISPs. External ISPs tend to offer more flexibility and array of features in comparison with internal ISPs. In complex use cases where multiple cameras (say as many as 6) need to be synchronized, an external ISP is recommended for better image output.
Also, some product developers who use NVIDIA processors do not want to use the internal ISP since it consumes additional GPU bandwidth for processing. They would instead prefer to use the processor for algorithm processing alone with the help of GPUs.
Hope you got a fair understanding of why today ISPs come integrated with processors and not image sensors. Also, the choice between an internal and external ISP pretty much depends on your application. The more complex your application, the higher the need for an external ISP.
In case you wish to learn more about ISP tuning, and the significance of using an external ISP, have a look at the article
Camera ISP and the significance of using an external ISP in imaging solutions.
If you have any queries on this topic, please write to us at camerasolutions@e-consystems.com. We would be more than happy to guide you.
Prabu is the Chief Technology Officer and Head of Camera Products at e-con Systems, and comes with a rich experience of more than 15 years in the embedded vision space. He brings to the table a deep knowledge in USB cameras, embedded vision cameras, vision algorithms and FPGAs. He has built 50+ camera solutions spanning various domains such as medical, industrial, agriculture, retail, biometrics, and more. He also comes with expertise in device driver development and BSP development. Currently, Prabu’s focus is to build smart camera solutions that power new age AI based applications.