The appropriate image sensor can bring more flexibility for Embedded designers, save the cost of bill of materials, and reduce the occupied area of lighting and optical components. It also allows designers to choose a large number of affordable image signal processors with optimized deep learning function from the consumer market without facing more complexity.
According to reports, new imaging applications are booming, from “industry 4.0” cooperative robots, to UAVs for fire fighting or agriculture, to biometric face recognition, to hand-held medical devices in home care points. A key factor in the emergence of these new applications is that embedded vision is more pervasive than ever before. Embedded vision is not a new concept; it just defines a system, which includes a visual setup to control and process data without an external computer. It has been widely used in industrial quality control, the most familiar example is “smart camera”.
In recent years, due to the development of affordable hardware devices in the consumer market, these devices have greatly reduced the cost of bill of materials (BOM) and product volume compared with the previous programs using computers. For example, small system integrators or OEMs can now purchase single board computers or modular systems such as NVIDIA Jetson in small quantities, while larger OEMs can directly obtain image signal processors such as Qualcomm snapdragon or Intel movidius myriad 2. At the software level, the market software library can speed up the development of special vision system and reduce the difficulty of configuration, even for small batch production.
The second change driving the development of embedded vision system is the emergence of machine learning, which enables the neural network in the laboratory to receive training, and then directly upload to the processor, so that it can automatically identify features and make decisions in real time.
It is very important for imaging enterprises facing these high growth applications to provide solutions for embedded vision systems. Image sensor plays an important role in large-scale introduction because it can directly affect the performance and design of embedded vision system. The main driving factors of image sensor can be summarized as smaller size, weight, power consumption and cost, which is called “swap-c” (decreasing size, weight, power and cost).
1. Cost reduction is very important
The accelerator of new embedded vision applications is the price to meet the market demand, and the cost of vision system is a major constraint to achieve this requirement.
1.1 optical cost savings
The first way to reduce the cost of the vision module is to reduce the size of the product. There are two reasons: first, the smaller the pixel size of the image sensor, the more chips can be made on the wafer; on the other hand, the sensor can use smaller and lower cost optical components, both of which can reduce the inherent cost. For example, Teledyne e2v’s Emerald 5m sensor reduces the pixel size to 2.8 μ m, enabling the s-port (M12) lens to be used on the 5 megapixel global shutter sensor, resulting in direct cost savings – the entry-level M12 lens costs about $10, while the larger C-Port or F-Port lens costs 10 to 20 times as much. So reducing the size is an effective way to reduce the cost of embedded vision system.
For image sensor manufacturers, this reduced optical cost has another impact on the design, because generally speaking, the lower the optical cost, the less ideal the incident angle of the sensor. Therefore, low-cost optics need to design a specific displacement microlens above the pixel to compensate for the wide-angle distortion and focusing light.
1.2 low cost interface of sensor
In addition to optical optimization, the choice of sensor interface also indirectly affects the cost of vision system. Mipi csi-2 interface is the most suitable choice for cost saving (it was originally developed by MIPI Alliance for mobile industry). It has been widely adopted by most ISPs and has begun to be adopted in the industrial market because it provides a low-cost system on chip (SOC) or system on module (SOM) integration from NXP, NVIDIA, Qualcomm or Intel. A CMOS image sensor with Mipi csi-2 sensor interface is designed. Without any intermediate converter bridge, the data of the image sensor is directly transmitted to the host SOC or SOM of the embedded system, thus saving the cost and PCB space. Of course, this advantage is more prominent in the embedded system based on multi-sensor (such as 360 degree panoramic system).
However, these benefits are limited because the connection distance of Mipi interface is limited to 20cm, which may not be the best in remote settings where the sensor is far away from the host processor. In these configurations, at the expense of miniaturization, the camera board solution with longer integrated interface is a better choice. Some off the shelf solutions can be integrated. For example, camera boards of industrial camera manufacturers (such as FLIR, AVT, Basler, etc.) can be used in Mipi or usb3 interfaces, which can reach a range of more than 3m to 5M.
1.3 reduce development cost
Rising development costs are often a challenge when investing in new products; it can cost millions of dollars on one-time development costs and put pressure on time to market. For embedded vision, this pressure becomes greater, because modularity (that is, whether the product can switch to use multiple image sensors) is an important consideration for integrators. Fortunately, by providing a certain degree of cross compatibility between sensors, for example, by defining a family of components sharing the same pixel architecture to have stable photoelectric performance, by sharing a single front-end mechanism with a common optical center, and by simplifying evaluation, integration, and supply chain with compatible PCB components, development costs are reduced.
To simplify the design of camera board (even for multiple sensors), there are two ways to design sensor package. Pin to pin compatibility is the preferred design for camera board designers, because it enables multiple sensors to share the same circuit and control, making the assembly completely unaffected by PCB design. Another option is to use size compatible sensors, so that multiple sensors can be used in the same PCB, but it also means that they may have to cope with the differences in the interface and wiring of each sensor.
Figure 1 the image sensor can be designed to provide pin compatibility (left) or size compatibility (right) to achieve proprietary PCB layout design
2. Energy efficiency provides better ability to work alone
Micro battery powered devices are the most obvious applications that benefit from embedded vision because external computers prevent any portable applications from happening. In order to reduce the energy consumption of the system, image sensor now contains a variety of functions, so that system designers can save power.
From the perspective of sensors, there are many ways to reduce the power consumption of embedded vision system without losing the frame rate. The simplest way is to minimize the dynamic operation of the sensor itself at the system level by using standby or idle mode for as long as possible. In standby mode, the power consumption of the sensor is reduced to less than 10% of the working mode by closing the simulation circuit. In idle mode, the power consumption is reduced by half, and the sensor can restart to acquire images in a few microseconds.
Another way to save energy in sensor design is to use advanced lithography node technology. The smaller the technology node is, the smaller the voltage required by the conversion transistor is. Because the power consumption is proportional to the voltage, the power consumption can be reduced. So the pixels produced with 180nm technology 10 years ago not only reduced the transistor to 110Nm, but also reduced the voltage of digital circuit from 1.9V to 1.2V. The next generation of sensors will use 65nm technology nodes to make embedded vision applications more energy efficient.
Finally, by choosing the right image sensor, the energy consumption of LED lamp can be reduced under some conditions. Some systems have to use active lighting, such as 3D map generation, action pauses, or simply use sequential pulses to specify wavelengths to enhance contrast. In these cases, reducing the noise of image sensor in low brightness environment can achieve lower power consumption. By reducing the sensor noise, engineers can decide to reduce the current density or the number of LED lights integrated into the embedded vision system. In other cases, when image capture and led flicker are triggered by external events, selecting the appropriate sensor readout structure can significantly save power. Using the traditional shutter sensor, the LED light must be fully opened when the frame is fully exposed, while the global shutter sensor allows the LED light to be opened only in a certain part of the frame. Therefore, in the case of intra pixel correlated double sampling (CDS) application, the global shutter sensor can be used to replace the rolling shutter sensor, which can save the lighting cost and keep the same low noise as the CCD sensor used in the microscope.
3. The function on chip paves the way for the visual system of application program design
Some extended concepts of embedded vision lead us to fully customize the image sensor and integrate all processing functions (system on chip) in 3D stacking mode to optimize performance and power consumption. However, the cost of developing this kind of product is very high, and it is not impossible to achieve this level of integration in the long run. Now we are in a transitional stage, including embedding some functions directly into the sensor, so as to reduce the computing load and speed up the processing time.
For example, in the application of bar code reading, Teledyne e2v company has patented technology, which adds an embedded function including a proprietary bar code recognition algorithm into the sensor chip. This algorithm can find out the bar code position in each frame, so that the image signal processor only needs to focus on these areas, and improve the efficiency of data processing.
Figure 2 Teledyne e2v snappy 5 megapixel chip, automatic identification of barcode position
Another feature that reduces processing load and optimizes “good” data is Teledyne e2v’s patented fast exposure mode, which enables the sensor to automatically correct exposure time to avoid saturation when lighting conditions change. This feature optimizes processing time because it adapts to light fluctuations in a single frame, and this fast response minimizes the number of “bad” images the processor needs to process.
These features are usually specific and require a good understanding of the customer’s application. As long as you know enough about the application, you can design a variety of other functions on chip to optimize the embedded vision system.
4. Reduce the weight to fit the minimum application space
Another major requirement of embedded vision system is to be able to cooperate with small space or small weight, so as to be used for handheld devices and / or to extend the working time of battery driven products. This is the reason why most embedded vision systems now use low resolution small optical format sensors with only 1mp to 5MP.
Reducing the size of pixel chip is only the first step to reduce the occupied area and weight of image sensor. The current 65nm process allows us to reduce the global shutter pixel size to 2.5 μ m without compromising the photoelectric performance. This production process enables CMOS image sensors such as full HD global shutter to meet the requirements of less than 1 / 3 inch in the mobile phone market.
Another main technology to reduce the weight and occupied area of sensor is to reduce the package size. Chip level packaging has grown rapidly in the market in the past few years, especially in mobile, automotive electronics and medical applications. Compared with the traditional ceramic land grid array (CLGA) package commonly used in the industrial market, the chip level fan out package can realize higher density connection, so it is an excellent solution to the challenge of lightweight and miniaturization of embedded system image sensor. For example, in the emerald 2m image sensor chip level package of Teledyne e2v, the side height is only half that of the ceramic package, while the size is reduced by 30%.
Fig. 3 Comparison of CLGA package (left) and wafer level fan out organic package (right) for the same chip, the latter can reduce the space occupation, thickness and cost
Looking to the future, we expect that the new technology can further realize the smaller sensor size required by embedded vision system.
3D stack is an innovative technology for semiconductor device production. Its principle is to make various circuit chips on different wafers, and then stack and interconnect them by using copper to copper connection and through silicon vias (TSV) technology. Because 3D stack is a multi-layer stacked chip, it allows the device to achieve smaller footprint than traditional sensors. In the 3D stack sensor, the readout and processing chip can be placed under the pixel chip and row decoder. In this way, the space occupying size of the sensor is reduced due to the reduced readout and processing chip, and more processing resources can be added to the sensor to reduce the load of the image signal processor.
Fig. 4 the 3D chip stack technology helps to realize the overlapping of pixel chips, simulation and digital circuits, and even the combination of additional processing chips for special applications, reducing the sensor area
However, there are still some challenges to make 3D stack technology widely used in the image sensor market. First of all, it is an emerging technology. Second, it costs more than three times more than traditional technology because of the additional process steps. Because 3D overlay will be the choice of high performance or very small footprint embedded vision system.
In summary, embedded vision system can be summarized as a “lightweight” vision technology, which can be used in different types of enterprises including OEM, system integrator and standard camera manufacturer. “Embedded” is a general description that can be used for different applications, so it is impossible to list its features. However, there are several applicable rules for optimizing embedded vision system, that is, generally speaking, the market driving force does not come from super fast speed or super high sensitivity, but from size, weight, power consumption and cost. Image sensor is the main driver of these conditions, so we need to choose the appropriate image sensor carefully in order to optimize the overall performance of embedded vision system. The appropriate image sensor can bring more flexibility for Embedded designers, save the cost of bill of materials, and reduce the occupied area of lighting and optical components. It also allows designers to choose a large number of affordable image signal processors with optimized deep learning function from the consumer market without facing more complexity.
Editor in charge: PJ