Embedded system can be any system based on microprocessor, which can complete specific work and can be seen everywhere, such as cars, kitchens, consumer electronics, medical devices, etc; Computer vision uses digital processing and intelligent algorithms to understand images or videos. It has been studied for a long time, but it is still in the ascendant.
The term “embedded vision” refers to the use of computer vision technology in embedded systems. In other words, “embedded vision” refers to an embedded system that extracts its meaning from visual input. Similar to the popularity of wireless communication technology in the past 10 years, embedded vision technology is expected to be widely used in the next 10 years.
Obviously, embedded vision technology can bring great value to a variety of applications (Figure 1). Two examples are Mobileye’s vision based assisted driving system, which is used to help prevent vehicle accidents; And Mg International’s swimming pool safety system to prevent swimming drowning. Some pure geeks invented the laser mosquito gun of intelligent ventures, which is designed to prevent people from contracting malaria.
Figure 1. Embedded vision starts with computer vision applications, including assembly line inspection, optical character recognition, robotics, monitoring and military systems. However, in recent years, the need to reduce costs and improve performance has accelerated the wide application of this technology in various other markets.
Just as high-speed wireless interconnection started from high-cost novel technology, so far, embedded vision technology is generally only used in complex and expensive systems, such as surgical robots for hair transplantation, quality control and inspection systems in manufacturing industry, etc.
The development of digital integrated circuit is the key factor for the development of high-speed wireless technology from novelty to mainstream. When the chip speed is fast enough, the cost is low enough and the energy efficiency is high enough, high-speed wireless technology will become the mainstream technology in the market. Today, people can buy broadband wireless modems for less than $100.
Similarly, the progress of digital chip technology has paved the way for the mass application of embedded vision (Fig. 2). Similar to wireless communication, embedded vision needs strong processing power – in particular, more and more applications use high-resolution cameras and start using multiple cameras. It is a great challenge to realize such processing capacity at a sufficiently low cost, so as to promote mass application. The fact is that embedded vision applications need strong programmability, which further increases the difficulty of this challenge. In wireless applications, the standard means that the baseband algorithm between different mobile phones will not change much. Compared with it, in embedded vision applications, it is possible to get better results – realize more valuable functions through unique algorithms.
Figure 2. Embedded visual aid system involves hardware, semiconductor and software component suppliers, subsystem developers, system integrators and end users, as well as basic research to achieve future breakthroughs. This paper mainly focuses on the embedded visual algorithm processing technology shown in the figure.
With embedded vision, the industry has entered a “virtuous circle”, which is the characteristic of many other digital signal processing (DSP) applications. At present, although there are few chips specially used for embedded vision applications, these applications increasingly use high-performance and cost-effective processing chips developed for other applications, including digital signal processor, CPU, FPGA and GPU. These chips have higher and higher programmable performance per unit cost and unit power. Therefore, they can support the realization of large quantities of embedded visual products. These mass applications have also attracted more attention from silicon wafer providers, who will provide better performance, higher efficiency and programmable processing capability.
As mentioned earlier, visual algorithms usually require strong computing power. Of course, all embedded systems are generally limited by strict cost and power requirements. In other DSP application fields, such as digital wireless communication, chip designers use special coprocessors and accelerators to complete the demanding processing tasks required by the application, and meet the requirements of high performance, low cost and low power consumption, so as to solve this problem. However, chip users generally cannot program these coprocessors and accelerators.
Wireless applications can generally accept these advantages and disadvantages. Wireless application standards mean that there are strong commonalities between the algorithms used by different device designers. However, in visual applications, there are no standard constraints on the selection of algorithms. On the contrary, many methods can be selected to solve a special visual problem. Therefore, visual algorithms are very diverse and often change rapidly with time. As a result, compared with applications such as digital wireless and consumer video devices based on compression technology, visual applications do not tend to use non programmable accelerators and coprocessors.
However, it is difficult to achieve high performance, low cost, low power consumption and programmable functions at the same time. Special purpose hardware can usually achieve high performance at low cost, but its programmability is weak. General purpose CPU has programmable ability, but its performance is poor, its cost performance is not high, and its energy efficiency is low. Highly demanding embedded vision applications typically use multiple processing units in combination, for example, may include:
● general purpose CPU for heuristic complex decision, network access, user interface, storage management and overall control, etc.
● high performance digital signal processor for real-time medium rate processing, less complex algorithms, etc.
● one or more highly parallel engines for simple algorithm pixel rate processing.
Any processor can be used for embedded vision in theory. At present, the most likely types are:
● high performance embedded CPU
● combination of special standard products (ASSP) and CPU
● graphics processing unit (GPU) with CPU
● digital signal processor with accelerator and CPU
● mobile “application processor”
● field programmable gate array (FPGA) with CPU
Various processors and their key advantages and disadvantages in embedded vision applications
High performance embedded CPU
In many cases, the embedded CPU can not provide enough performance to implement the vision algorithm with high requirements – nor can it meet the performance requirements at an acceptable price or power consumption. Generally, memory bandwidth is a key performance bottleneck, because visual algorithms usually use a large amount of data and do not repeatedly access the same data. The memory system of embedded CPU can not be designed to adapt to this kind of data flow. However, similar to most processors, the performance of embedded CPU is gradually enhanced over time. In some cases, it can provide sufficient performance.
Where possible, there is good reason to run visual algorithms on the CPU. Firstly, most embedded systems need CPU to realize various functions. If the required visual functions can be realized by this CPU, it reduces the complexity of the system compared with the multiprocessor solution. Moreover, most visual algorithms were developed on PC at the beginning, using general CPU and its related software development tools. The similarity between PC CPU and embedded CPU (and its related tools) means that compared with other types of embedded visual processors, it is generally easier to implement visual algorithms on embedded CPU. Finally, embedded CPU is usually simpler to use than other types of embedded visual processors because of its relatively intuitive architecture, mature tools and other application development basic support platforms, such as operating system.
ASSP with CPU
ASSP is a dedicated and highly integrated chip, customized for special or professional applications. ASSP can use CPU or a separate CPU chip. By virtue of specialization, ASSP usually has excellent cost and energy efficiency compared with other types of processing solutions. Among other technologies, ASSP improves efficiency by using dedicated coprocessors and accelerators. Moreover, since ASSP mainly focuses on professional applications, a large number of application software is usually required.
This specialization enables ASSP to achieve high efficiency, but it also brings great limitations: lack of flexibility. ASSP designed for one application can not be used for other applications, or even applications related to the target application. ASSP uses a unique architecture and is more difficult to program than other types of processors. In fact, some ASSPS do not support user programming. Another consideration is risk. ASSP is usually provided by small suppliers, which may increase the risk that it is difficult to provide chips, or it is unable to provide follow-up products to help system designers update their design, and designers have to design from scratch.
GPU with CPU
GPU is mainly used for 3D graphics, and more and more are used to realize other functions, such as visual applications. At present, the GPU of personal computer tends to be programmable, and can complete other functions in addition to 3D graphics. This type of GPU is called “general purpose GPU”, or “GPGPU”. GPU has strong parallel processing ability. They are unique on personal computers. You can use GPU software development tools for free. Programming from GPGPU is not very complicated. For these reasons, developers who develop their computer vision algorithm for the first time on PC usually use GPU as the parallel processing engine. They need to speed up the execution of the algorithm for the purpose of simulation or prototype development.
GPU is tightly integrated with general-purpose CPU, sometimes on the same chip. However, one limitation of GPU chips is that the types of CPUs that can be integrated are limited, and the CPU operating systems that support such integration are also limited. At present, low-cost and low-power GPUs designed for products such as smart phones and tablets can be provided. However, these GPUs are generally not GPGPUs, so it is very difficult to use them in other applications except 3D graphics.
Digital signal processor with accelerator and CPU
Digital signal processor is a microprocessor specially used for signal processing algorithms and applications. For the signal processing and other tasks of the core of visual application, this specialization makes the efficiency of digital signal processor much higher than that of general CPU. Moreover, compared with other types of parallel processors, digital signal processors are relatively mature and more convenient to use.
However, although the performance and efficiency of digital signal processor in visual algorithm are higher than that of general CPU, it is still difficult to provide sufficient performance to meet the algorithm requirements. For this reason, DSP generally needs one or more auxiliary coprocessors. Therefore, a typical DSP chip in vision application includes CPU, digital signal processor and multiple coprocessors. This heterojunction can produce good performance and high efficiency, but it is also difficult to program. In fact, DSP vendors generally do not support users to program coprocessors; Instead, the coprocessor runs the software function library developed by the chip supplier.
Mobile application processor
Mobile “application processor” is a highly integrated chip system, which is generally mainly designed for smart phones rather than other applications. Application processors usually include high-performance CPU cores and various special coprocessors, such as digital signal processor, GPU, video processing unit (VPU), 2D graphics processor, image acquisition processor, etc.
These chips are designed specifically for battery powered applications, so they are very energy efficient. Moreover, due to the increasing importance of applications around smart phones and tablets, mobile application processors generally have a strong basic software development support platform, including low-cost development circuit boards, Linux and Android ports. However, as discussed in the previous chapters on digital signal processors, the special coprocessors in application processors are generally not user programmable, which limits their development in visual applications.
FPGA with CPU
FPGA is a flexible logic chip, which can be reconfigured at gate level and module level. This flexibility enables users to customize the computing structure to meet the application requirements at any time. It also supports the selection of I / O interfaces and on-chip and off-chip devices that meet the application requirements. It can customize the computing structure, combine a large number of resources in modern FPGA, and realize high performance and good cost performance and energy efficiency ratio at the same time.
However, using FGPA is actually a hardware design function, not a software development work. Generally, the register transfer level (RTL) uses hardware description language (Verilog or VHLD) to design FPGA. The register transfer level is a very low abstraction level. Compared with other types of processors discussed in this paper, this makes FPGA design very time-consuming and expensive.
Nevertheless, it is more and more convenient to use FPGA, which is caused by many factors. First, the so-called “IP module” Library – Reusable FPGA design component library, which has become more and more powerful. In some cases, these libraries can directly meet the requirements of visual algorithm. In other applications, they also support functions such as video I / O port or scan line buffer. Moreover, FGPA suppliers and their partners provide more and more reference designs – reusable system design using FPGA for professional applications. Finally, using high-level synthesis tools, designers use high-level language to realize vision and other algorithms in FPGA, and the efficiency is getting higher and higher. Users can implement CPU with relatively low performance in FPGA. Moreover, in a small number of applications, FPGA manufacturers have integrated high-performance CPU into the device.
Source: China Electronics Network