Traditionally, computing tasks in automobiles are performed by microcontroller units (MCU) and application processors (AP). A typical mid-range car can contain 25 to 35 mcu/aps, while a luxury car may use 70 or more. More and more vehicles need extremely complex and computationally intensive functions to complete tasks such as advanced driving assistance system (ADAS), Infotainment, control, network and security. Many of these applications involve machine vision in the form of image and video processing and artificial intelligence (AI).
It is difficult for a single processor architecture to handle all the electrical interfaces and protocols required for peripherals such as sensors, cameras, and displays. In addition, in many cases, these processors simply cannot meet the extreme computing needs of tasks such as machine vision and artificial intelligence.
In order to solve this complexity, automotive system designers are turning to field programmable gate array (FPGA), instead of replacing the existing mcu/ap, but acting as a bridge between them and other devices, and unloading communication and other computing intensive tasks.
Since FPGAs can be programmed to support various electrical interfaces and protocols, they can act as a bridge between mcu/ap and sensors, cameras and displays. In addition, because they can perform computation and operations in a massively parallel manner, FPGAs can be used to perform compute intensive visual processing and AI tasks, freeing up host processors for other activities.
This paper discusses the processing requirements of modern automobile, and describes some automobile applications that FPGA can solve. Then it introduces some sample FPGAs of lattice semiconductor, and shows how to use them to solve connection, processing and security problems. Relevant development boards are also provided to help designers get started.
Target automobile application of FPGA
In order to support its ADAS function, today’s vehicles use many sensors outside the vehicle, including camera, radar, lidar and ultrasonic detector. In many cases, it is necessary to obtain data from different sensors, preprocess these data (remove noise and format them as needed), and use sensor fusion to combine the data, so that the uncertainty of the obtained information is lower than if the data from different sensors would be used alone. In many cases, AI applications are used to analyze data, make decisions, and take appropriate actions.
A relatively new trend is the deployment of electronic (also known as “digital”) mirrors. In this case, a wide-angle high-resolution camera is installed in the rear window. The video stream from the camera is displayed on the digital display screen, replacing the traditional mirror, so as to form a clear rear view, which is not obstructed by the rear seat passengers. In some cases, the video stream from the mirror mounted camera may be merged with the video stream from the rear window camera. These three information are “spliced together” to provide a single image on the ultra wide electronic mirror, so that the driver has a higher degree of situational awareness of what is happening around the vehicle.
Another recent trend is the deployment of interior cameras on the instrument panel, on the steering column or integrated into the middle of the rear-view mirror (conventional or electronic). When used in conjunction with AI, these interior mirrors can be used for various tasks, such as identifying who is sitting in the driver’s seat and adjusting the mirrors and mirrors accordingly. In addition to monitoring drivers to ensure that they pay attention to the road rather than dozing off, such a system can also look for signs of sleepiness, as well as medical problems or pain, such as seizures or heart attacks, and take appropriate action. These actions may include activating the hazard warning lights, applying the brakes, and guiding the vehicle to the side of the road.
For video based applications, in some cases, it is necessary to split a single video input into multiple streams; In other cases, the design requirement may be to aggregate multiple video streams into one.
With the increasing popularity of electric vehicles (EV), it is necessary to monitor and control the motor, and monitor and manage the charging process to achieve the longest battery life.
Most importantly, many cars today are beginning to support 5g or v2x, where v2x (“vehicle to anything”) refers to the communication between the vehicle and any other entity that may affect (or be affected by) the vehicle, from roadside infrastructure to other vehicles. In addition to this connectivity, security is also required to prevent vehicles from being hacked.
It is important to remember that not all FPGAs are suitable for automotive applications. The automotive electronics Committee (AEC) was originally an organization established by Chrysler, Ford and GM in the 1990s to establish general parts qualification and quality system standards. One of the most frequently cited AEC documents is aec-q100, “integrated circuit stress test certification based on failure mechanism”.
IATF 16949:2016 is a technical specification designed to develop a quality management system, provide continuous improvement, emphasize defect prevention, and reduce changes and waste in the automotive industry supply chain and assembly process. IATF 16949:2016 is based on the ISO 9001 standard and was created by the international automotive working group (IATF) and the ISO Technical Committee.
Electronic system suppliers in the automotive market increasingly require semiconductor suppliers to provide products that meet the aec-q100 standard and can prove the iso/ts-16949 certification of their quality system.
Select the correct FPGA for operation
FPGA is very flexible, but different device families provide various combinations of capabilities and functions, making it more suitable for specific tasks. For example, in embedded vision applications, modern cameras and displays usually use Mipi interfaces. Both Mipi csi-2 (camera / sensor) and DSI (display) protocols use a communication physical layer (PHY) called d-phy. Traditional mcu/ap may not support this interface, but some FPGAs support it, such as lattice semiconductor’s crosslink NX embedded vision and processing FPGA.
Automotive system developers who want to evaluate crosslink NX FPGAs will find the combination of lifcl-vip-si-evn crosslink NX VIP sensor input board (Figure 1) and lf-evdk1-evn modular embedded vision kit very interesting (the former can be used as the input board of the latter). In addition to the crosslink NX FPGA, the sensor input board is equipped with four 13 megapixel Sony imx258 CMOS Mipi image sensors, which support 4k2k @ 30 frames per second (FPS) or 1080p @ 60 FPS. It also supports easy sensor connection through three independent PMOD interfaces.
Figure 1: the crosslink NX VIP sensor input board that can be input as an embedded visual development kit contains a crosslink NX FPGA and supports the aggregation of four Mipi Sony imx258 image sensors. (picture source: lattice semiconductor)
For computing intensive applications that also require high i/o bandwidth, such as AI for gesture recognition and control, speech recognition and control, human presence detection, occupant recognition, driver monitoring and other tasks, lattice’s ecp5 FPGA has up to 3.2 gbit/ serializer / deserializer (SerDes), and each device in the dual channel block has up to four channels to achieve higher granularity, up to 85K look-up table (LUT), Enhanced digital signal processing (DSP) blocks provide 2x resource improvement for symmetric filters and single event disturbance (SEU) mitigation support. These FPGAs also provide programmable i/o support for lvcmos 33/25/18/15/12, XGMII, LVTTL, LVDS, bus LVDS, 7:1 LVDS, LVPECL, and Mipi d-phy i/o interfaces.
An example ecp5 device is the lfe5u-85f-6bg554c, which has 84000 logic elements, 3.75 megabits (Mbits) of ram and 259 i/os. Lfe5um-45f-versa-evng ecp5 versa development kit is also of interest (Figure 2). The board adopts half length PCI Express (PCIe) dimensions, allowing designers to evaluate the key connection characteristics of ecp5 FPGAs, including PCIe, Gigabit Ethernet (GBE), DDR3 and general SerDes performance.
Figure 2: the ECPE versa development kit uses half length PCI Express dimensions to enable designers to evaluate the key connection characteristics of ecp5 FPGAs, including PCIe, GBE, DDR3 and universal SerDes performance. (picture source: lattice semiconductor)
Protect the car with FPGA
Security threats from hackers are increasing, and new violations continue to occur. In the case of cars, cyber attacks can lead to loss of control, injury or death of passengers and pedestrians, and damage to cars, other vehicles and property.
A large part of automobile safety solutions is to establish the root of trust (ROT); That is, hardware resources that can always be trusted in the system. One solution is FPGA based hardware rot (hrot), such as the solution provided by lattice’s machxo3d series devices. In addition to a large amount of LUT resources and a large amount of i/o, these flash based devices also provide instant boot and hot plug functions. General applications include gluing logic, bus bridge, bus interface, motor control, power on control and other control logic applications.
Of particular interest is that machxo3d is the only FPGA with dual boot function and less than 10k LUT. It is equipped with a hard immutable security engine certified by the National Institute of standards and Technology (NIST). This enables machxo3d to appear as an hrot of the vehicle in the form of a system that starts first and then shuts down. After the system is powered on, machxo3d first checks to ensure that it is running the verified firmware. It then checks the firmware of other devices in the system. If any component in the system (including itself) is attacked or damaged, machxo3d will reject the suspected firmware and reload the component with a known good, verified firmware image.
For developers interested in evaluating machxo3d based designs, the lcmxo3d-9400hc-d-evn machxo3d development board provides an extensible prototype design platform (Figure 3). The board has l-asc10 (analog sensing and control) hardware management device, universal i/o interface for Arduino and raspberry PI boards, two Hirose fx12-40 connector positions (DNI), aardvark connector (DNI) and 128 Mbit serial peripheral interface (SPI) flash memory, and has quadruple read and take function.
Figure 3: the machxo3d development board has a machxo3d FPGA, an l-asc10 (analog sensing and control) hardware management device, supports Arduino and raspberry PI boards, two Hirose fx12-40 connector locations (DNI), an Aardvark connector, and a usb-b connection for device programming. (picture source: lattice semiconductor)
The board is a 4 x 6 inch form factor with USB mini-b connectors for power supply and programming, and multiple connector locations that support Arduino, aardvark, fx12, Hirose, and raspberry PI. USB cable and quick start guide are included.
Modern automotive electronics need more and more sensors, electrical interfaces and protocols, and have corresponding requirements for processing capacity and bandwidth. Artificial intelligence and machine vision processing, as well as the increase of security requirements, have complicated the implementation of solutions using classic MCU or AP methods.
As shown in the figure, by properly applying FPGA, designers can increase a certain degree of flexibility and processing capacity to bridge different processing environments, perform sensor aggregation and fusion functions, meet i/o bandwidth requirements, perform calculations and operations in a large-scale parallel manner, and release host processors for other activities.
In terms of security, the flash based FPGA with dual boot function and NIST certified immutable security engine can act as the hrot of the car, and ensure that it and other devices only run authenticated firmware, so as to prevent hackers from damaging the car system by password
Reviewed and edited by: Fu Qianjiang