Electronic enthusiast network reported (Wen / Huang Jingjing) on September 6, vivo officially released its self-developed ISP chip v1.
Vivo has been working on image technology for a long time, and has established an image track including night view, portrait, motion anti shake, zoom and video. In the night scene, vivo and its partners have developed a new CFA technology to enhance the sensitivity, which is applied to imx598x CMOS; The image system jointly developed with Zeiss has realized the optical design of super large aperture in limited space, and realized Zeiss optical lens and T-type coating on vivo mobile phone for the first time, which has greatly improved the glare of night scene and significantly improved the shooting effect of night scene. In addition, there are micro pan tilt stabilizers and 50 mm focal length portrait lenses, which are constantly refined in anti shake.
Then, with the continuous improvement of image technology, whether algorithm is important or hardware is important, the answer is mainly interpreted in three aspects. They are image chip, optical device and software algorithm.
Vivo heavy release V1 ISP chip
Du Yuanjia, director of vivo image algorithm, believes that starting from the needs of consumers, people have more and more expectations for mobile image effects in complex scenes. For example, people don’t want noise in night scenes; The requirements for video shooting are constantly improving. In order to provide users with more creative space, mobile image will face a new evolution. The mobile recording world will develop from static to dynamic, help users express in multiple dimensions and give more creation to customers.
In terms of imaging speed and real-time, under the current computing power of mobile phones, it takes half a second to one second to process a photo. In terms of 30 frames of video, the algorithm time of each frame cannot exceed 33 milliseconds. There is a 30 times speed difference. 60 frame and 12 frame video are more demanding. In the past, mobile phones used to shoot one frame of content per second. Now they have to calculate at least 30 frames.
Energy consumption is another problem faced by image functions. The increase of operation density and the decrease of power consumption are a pair of contradictions. We must find a way of large computation and high-speed processing. We believe that only by using customized chips, cooperating with the software algorithm of the main chip, and combining software and hardware, can the power load of the main chip be released and the user experience be greatly improved.
Vivo did not launch customized chips for the first time. Customized hifi chips were introduced as early as vivo x1 \ xplay. In 2017, customized DSP image chips were put into x9 plus.
Today, after vivo spent 24 months, invested 300 R & D team and deeply cooperated with mobile phone SOC manufacturers, VI released the first self-developed professional image chip v1.
VI is a professional image chip of vivo leading and open service high-speed computing imaging. It is a fully customized integrated circuit chip with special specifications. It is located between the main chip and the display panel, and can be matched with different main chips and displays. It adopts high-speed two-way interconnection with the main chip ISP for data interaction with very low delay. It can expand the ISP’s high-speed imaging computing power, release the ISP load of the main chip, and serve the needs of users in preview and video recording.
Different from GPU and DSP, professional customized chips have the characteristics of high performance, low delay and low power consumption when dealing with specific functions. There is an exponential improvement in energy efficiency ratio. For example, suppose the CPU is a comprehensive salesman, GPU and DSP make some reservations according to the business type, and the complex business is still handed back to the CPU. V1 defines the business type for everyone, divides them into groups and counters, and processes them at the same time. Therefore, it can not only process complex operations with high bandwidth like CPU, but also process in parallel like GPU and DSP.
In order to maximize the synchronous processing capacity, vivo innovated and optimized the internal data storage architecture and high-speed read-write circuit. Achieve an equivalent 32 megabytes of large on-chip cache. Although on-chip cache is expensive, it can support more computing density and data volume. At present, the cache of flagship desktop computer processors is 16 to 24 megabytes.
For example, for the shooting of night scene video, noise reduction and frame insertion are carried out through v1. The algorithm module with large amount of calculation, namely MEMC technology, can be used for motion compensation. With the assistance of V1, the main chip runs MEMC denoising and frame insertion of Full HD 60 frames with low power consumption during low light recording, which improves the effect of night scene image. And cooperate with the noise reduction function of the main chip to carry out secondary brightness and secondary noise reduction.
Thus, the effect that 1 + 1 of main chip ISP and customized ISP V1 is greater than 2 can be obtained. Since the software algorithm is transferred to V1 dedicated hardware circuit, the power consumption of V1 dedicated hardware circuit is reduced by 50% when processing the same imaging calculation algorithm at high speed.
The imaging system and optical devices are inseparable. The ability of each device jointly affects the imaging effect. Therefore, no matter how the shooting effect is improved, it still can not completely solve the problems such as glare, ghost and poor edge image quality. It needs to be improved by improving the quality of the whole optical system.
Zhu Panpan, director of optical devices, introduced that vivo chose high-specification glass lenses to replace plastic lenses to achieve ultra-low dispersion, high transmittance and strong thermal stability. The color difference of the field of view is about 70% lower than the design value of the previous line. Abbe number is an indicator of color ability. The higher the value, the purer the color. The Abbe number of glass lenses used by vivo is as high as 81.6, and the Abbe number of professional cameras is between 80-90.
By introducing SWC coating technology, the reflected light is suppressed on the lens surface, which is a technology based on the principle of bionics. After SWC coating, the reflectivity is reduced to 0.1%, which significantly improves the image and improves glare and ghost.
ALD coating atomic layer deposition process allows atoms to be deposited layer by layer through special chemical reaction to form a film layer of about 0.4 nm each, reducing the average reflection to 0.25%, almost no angle offset, better consistency between the center and edge, and further effectively improving the ghost problem. ALD technology is exclusive to vivo, the first in the industry, and has been mature for mass production.
In addition, for the petal ghost, it absorbs the red light wavelength between the filter and the sensor. Vivo adopts the pigment spin coating process. It is also the first manufacturer in China to use the pigment spin coating process.
Improvement of algorithm part — faithfully restore the natural color seen by human eyes
Vivo’s color system is based on the understanding of users’ color needs. It is positioned as fresh, bright and textured. At the same time, professional creators will more faithfully restore the natural colors seen by human eyes.
Wang Zhaofeng, vivo image product manager, said that because everyone has different understanding of color, vivo selected 140 color cards with richer color blocks for fine standard definition, and selected 80 real scenes for real scene effect acceptance. With the help of more refined color debugging algorithm, the supporting effect is realized. After rough debugging and fine adjustment, the hue accuracy of photos is finally improved Δ E converged by about 15.5%, leading the industry in the smaller Δ E and still good saturation.
Vivo selects the biotar lens represented by Zeiss biotar 58 / 2.0 and biotar75 / 1.5, gives the rotating out of focus imaging characteristics an excellent “atmosphere” for portrait shooting, and reproduces the classic Zeiss portrait style on the mobile phone with the support of vivo’s software and hardware in the portrait field.
In order to realize multi-dimensional image experience, vivo also strengthens AI face enhancement algorithm, delay photography function, log mode, super night scene rawhdr3.0 algorithm, superraw, etc.
Mobile photography enters the era of hardware level algorithm
Answer the initial question, is mobile photography hardware or algorithm to determine the future? Vivo believes that image technology research and development is a systematic technical project. Just like the human eye, the picture everyone sees is the result of signal transmission to the optic nerve through the photoreceptor cells of the eye, which is finally transmitted to the visual center of the brain and processed by the brain. The same is true for mobile phones. Through the joint cooperation of platforms, devices, algorithms and other aspects, the final imaging quality is determined. Algorithms and hardware are indispensable. Through V1 chip, vivo believes that the next trend in the mobile imaging industry will be the “era of hardware level algorithms”.