An ISP is a key component in an embedded camera system since a sensor provides the output only in the RAW format. An ISP (Image Signal Processor) is a dedicated processor that converts this RAW image data into a high-quality workable output through various processes such as noise reduction, lens shading correction, gamma correction, auto exposure, auto white balance, etc.
That said, a logical question that might arise is why image sensors don’t come integrated with an ISP. Why don’t sensor manufacturers such as Sony, ON Semiconductor, or OmniVision consider adding value to their sensor products by integrating a dedicated ISP?
Well, it got me curious too.
So, in this article I attempt to break down the reasons for sensor manufacturers not offering a combination of a sensor and ISP as an integrated solution.
A look back before we start
Before we start looking at the reasons, a bit of history might help.
History of Image Signal Processors
When we say that sensors don’t come with ISPs, was it always the case? Not really.
Sensor manufacturers used to offer ISPs as well in the past. The last known image sensor that continues to have an integrated ISP and is still selling in volumes is OmniVision’s OV5640, which is a 1/4-inch 5MP camera. However, as some of you may be already aware, OmniVision has decided not to launch image sensors with integrated ISP beyond 5MP anymore.
Why Image Sensors are No Longer Offered with an ISP
Image sensors do not come integrated with ISPs anymore because of predominantly 2 reasons:
- Growth of microprocessors with built-in ISP
- Product developers wanting to choose the ISP based on tuning requirements
Growth of microprocessors with built-in ISP
Earlier, processors didn’t have a built-in ISP. But today, most of the modern manufacturers like Qualcomm, NXP, and NVIDIA offer an ISP with their microprocessors. Due to this, sensor manufacturers do not want to add to their product cost by integrating an ISP within. Since an ISP is not a value add anymore, sensor manufacturers do not want to run the risk of losing a design win or a bulk order on a budget or cost grounds.
Product developers wanting to choose the ISP based on tuning requirements
The second significant reason for manufacturers to come up with raw Bayer filter sensors is that many product developers and design engineers want to pick the ISP on their own based on the tuning they would require. It is also dependent on the features and the interfacing supported by the ISP.
Different ISPs come with different feature sets, and picking and integrating an ISP many times requires a thorough understanding of the sensor as well. One of e-con Systems’ key strengths is having a suite of products that come with different ISPs for different types of applications.
For example, Hyperyon – a 2MP ultra low light USB camera based on the Sony STARVIS IMX290 sensor – uses the Socionext ISP. On the other hand, e-CAM21_CUTX2 – Sony STARVIS IMX290 camera for NVIDIA Jetson TX2 – uses the NVIDIA Jetson ISP. While Hyperyon is targeted at low-light HDR USB 2.0 applications, e-CAM21_CUTX2 is suited only for low-light MIPI CSI-2 based applications.
External ISP vs Internal ISP
Now that we understand that an ISP comes integrated with a processor (which is when it is called an internal ISP), we need to look at why we might need an external ISP in some cases.
It is well understood that USB cameras definitely need an external ISP. So, the question of choice between an internal and external ISP arises only in the case of other cameras.
Even though processors come with an internal ISP, they have not yet become as sophisticated as external ISPs. External ISPs offer more flexibility and an array of features in comparison with internal ISPs. In complex use cases where multiple cameras (say as many as 6) need to be synchronized, an external ISP is recommended for better image output.
Also, some product developers who use NVIDIA processors do not want to use the internal ISP since it consumes additional GPU bandwidth for processing. They would instead prefer to use the processor for algorithm processing alone with the help of GPUs.
However, the choice between an internal and external ISP pretty much depends on your application. The more complex your application, the higher the need for an external ISP.
Want to learn more about ISP tuning, and the significance of using an external ISP? Please read: read this blog.
Addressing the Need for Specialized ISP Solutions: RGB-IR Sensor Use Cases
While traditional RGB sensors remain the standard, the industry is increasingly turning to RGB-IR sensors for applications that demand both visible and near-infrared (NIR) imaging capabilities. These sensors use a modified Bayer pattern to capture both visible and IR light, making them ideal for applications like in-cabin automotive monitoring systems, which require accurate detection of occupants and activities during both day and night.
However, processing RGB-IR sensor data requires a distinct ISP pipeline. Most ISPs are designed for standard RGB data, which makes it challenging to efficiently handle RGB-IR outputs. This is where e-con Systems’ FPGA + ISP (Existing RGB ISP) solution comes into play. Powered by our patented RGB-IR separation algorithm, this approach bridges the gap between sensors and traditional ISPs. By optimizing costs and simplifying integration, it delivers robust imaging performance without the need for a completely new ISP design.
e-con Systems’ TintE™, a FPGA-based Image Signal Processor
e-con Systems has 20+ years of experience in designing, developing, and manufacturing OEM cameras. Our extensive experience in ISP tuning and image processing is showcased in TintE™, our FPGA-based Image Signal Processor (ISP) that meets unique image enhancement requirements.
We provide a complete, ready-to-deploy imaging pipeline that seamlessly integrates with various applications, ensuring great image quality. The pipeline features adjustable components like debayering, auto white balance, auto exposure, and gamma correction. Furthermore, TintE™ can be implemented on different FPGA platforms, delivering consistent, high-quality imaging across a range of system configurations while maintaining performance.
Some of its features include low latency, support for 4k 60 fps, support for crop and binning, scalability up to 8k (MIPI, USB, PCI), support for 2, 4, 8 pixels per clock, and more.
Browse our Camera Selector Page to check out our complete portfolio
If you need help integrating an external ISP-based camera into your embedded vision application, please write to camerasolutions@e-consystems.com
Prabu is the Chief Technology Officer and Head of Camera Products at e-con Systems, and comes with a rich experience of more than 15 years in the embedded vision space. He brings to the table a deep knowledge in USB cameras, embedded vision cameras, vision algorithms and FPGAs. He has built 50+ camera solutions spanning various domains such as medical, industrial, agriculture, retail, biometrics, and more. He also comes with expertise in device driver development and BSP development. Currently, Prabu’s focus is to build smart camera solutions that power new age AI based applications.