On Wednesday (February 26th), the California Department of Motor Vehicles released the annual report on automatic driving in California, detailing the driving conditions of the automatic driving companies approved in California last year and how long the safety operators of each company need to take over a car. The annual report of the February 26th long time driving test is also available.
This “breakaway report” provides a rare opportunity to take a look at the way the autonomous driving companies that are testing on public streets work.
Unfortunately, after reading these reports, we can hardly measure how far away we are from the era of autonomous driving.
First of all, companies use different terms to explain all kinds of separations. In addition, the report only covers the testing in California, while most of the big players have been tested in other places at the same time, such as waymo’s test in Phoenix, Argo’s test in Pittsburgh and Miami, and the test in Las Vegas by amber.
More importantly, disengagement is not a good way to measure the progress of autonomous driving. It is also not suitable for comparison between companies, because different companies have different test locations. For example, cruise is in San Francisco with complex road conditions, and waymo is in the quiet suburbs. Different companies follow different agreements. Some companies require safe drivers to take over vehicles when there are emergency vehicles around or near the campus. Thus, the vehicles will also have system separation in the place where they can normally drive.
The best way to reduce detachment rate is to accumulate mileage in some simple and well researched areas, but this is the worst way to improve the automatic driving system. Waymo said Wednesday (26 February) that the California report did not “provide insight into its autonomous driving program” or “differentiate its performance in the field of autonomous driving from other companies.”.
So how do these companies themselves track their technology progress?
Some measures are simple. If the company’s vision system can only detect 98% of pedestrians now, its machine learning algorithm may need to learn more cases, which is expected to exceed 99.99%.
Matt Johnson Roberson, CEO of refraction AI, said the company checks these statistics at least once a month, including the frequency of computer system crashes and the reliability of these vehicles following software instructions. In Ann Arbor, Michigan, the company is building a small self driving car that can deliver food along the bicycle lanes.
Although the start-up and its competitors have their own unique way of measuring progress, most companies seem to focus more on the mileage that can really drive safely than the total mileage that the company tests.
Step one: think about what you have to do to drive your car. It takes decades for an autopilot to go to all parts of the world at any time. Most developers are targeting market segments with geographical location, road type and driving conditions.
Cruise’s autopilot will have to deal with the entire San Francisco traffic, which means that its autopilot must master all the driving skills of human drivers. It should successfully deal with unprotected left turn, four way parking, roundabout and crazy steep streets. The objectives of Optimus ride and voyage are relatively low, mainly in retirement communities and other restricted areas with low performance requirements.
People can tabulate the performance needed for autonomous driving, just like the syllabus, and then train cars. Some new start-ups start with basics, such as writing code to teach cars to identify and maintain lanes.
Then, add lane changes, merge on the freeway, or slow down to get others into the lane. Whenever you need to change the control software of the car, you can first try it in the computer simulation to see how it works and find out the loopholes.
Then, it is applied to a vehicle and tested on a dedicated lane under controlled conditions. Once confirmed, it can enter the public road test. Waymo has traveled 20 million miles in the real world and more than 10 billion in the virtual world.
As each feature improves, “you can start to cross them off the list,” says Don Burnett of Kodiak robotics, an automated truck company. “How many features do you have to implement?” What functions have been implemented? This is a very good indicator of progress for a company.
At the same time, the company can further strengthen the realized functions. If you’re studying lane change, you’ll start with a scene where there are no other vehicles around, focusing on driving tracks and speeds similar to human drivers. (again, this is done first in a simulated environment and then in the real world.)
Then you can add some cars to the scene, and then add more cars. At this time, the system must decide when the workshop distance is safe. Finally, it’s like a human driver acquiesces another driver to merge into the driveway.
Once you cross out all the subtle functions, you have a “fully functional” system. But the height of these barriers also helps explain why so many self driving companies are looking for more limited business models, such as trucks on high-speed roads and fixed route buses.
Elon Musk, the always confident CEO of Tesla, is one of the few who claims to have won. “I think our fully automatic driving vehicle will be fully functional this year.” “This means that this year, cars will be able to find you in the parking lot, pick you up and take you all the way to your destination without any human intervention,” Musk said in early 2019 But, he later explained in a conference call, “being fully functional means it has the opportunity to take you from home to work without intervention.”
However, there is still a big gap between “complete function” and “complete task”. Take Tesla’s smart call, launched last September, for example, which automatically drives the car from the parking space to the owner’s location. At present, the evidence shows that it works in most cases, but sometimes it can’t distinguish asphalt, grass or ice, and sometimes it gets stuck on the garage door.
So once you add functionality to your computer code base, you have to make sure it works in as many situations as possible. Chris Urmson, who led waymo in the early years, said this reflected the importance of simulation. Mr. Hamson is now CEO of Aurora, which is developing autonomous driving technology for a variety of applications, including truck transportation.
Last year, when Mr. Hamson’s team studied unprotected curves, they first sent human drivers on a fact finding mission. They are interested in sampling the diversity of life, such as how fast or how slow human drivers are at different intersections, how seriously a truck affects the vision of a car.
They load the survey results into the simulation software, and then change the details by “blurring”, making subtle changes to the position, speed, etc. of other participants. Mr. Hamson said Aurora had conducted more than two million simulations and polished the system before turning left in real traffic scenes.
Then they drove the autopilot car to the street and verified the knowledge acquired by the computer in the real world. During this process, the Aurora safety operator noticed some unusual situations and moments. Vehicle behavior is different from what they expected, which usually leads to the separation of the automatic driving system.
However, instead of focusing on the number of times to manually control the vehicle, the company’s engineers use these moments as materials to conduct more simulations, more fuzzy tests and more adjustments to improve the performance of the vehicle.
At some point in the future, Hamson and his team will announce that their system has proven its skills in enough scenarios to drive into the real world without human security officers. Different companies will announce it at different times, because no one can reach an agreement on this controversial issue. To what extent is it safe? Regulators can’t tell.
The Federal Department of transportation provides only vague guidelines for the development of safety systems. Many states in the United States welcomed the development of autonomous driving software, but did not impose any technical requirements.
In this regard, California is particularly prominent: more than 60 companies have been approved to test autonomous driving technology in the state, but only five companies have obtained the permission of California Public Utilities Commission (CPUC) to carry out passenger transportation in the state.
Bryant Walker Smith, a professor of autonomous vehicle policy at the University of South Carolina School of law, said: ‘don’t expect this loose schedule to change.’. These autopilot run complex software in complex environment. Regulators and the public do not have enough expertise, resources or time to fully understand how all these things work.
No company is likely to drive as long as a human driver in order to provide data to prove that their car can. That means everyone has to take a big step, or at least a small step, in faith, Smith said. “It depends on whether the company that develops and deploys autonomous driving is trustworthy.”
Refraction AI’s autopilot is unlikely to cause serious injuries, because the speed of these vehicles is between 10 and 12 miles per hour. Therefore, the team can bypass the safety factor and consider another indicator: single transportation cost.
Recently, the company’s engineers spent about a month studying four-way parking. Johnson Robertson said they made the autopilot reach a “no failure” level, but in fact it was because it was too conservative to wait for seven or eight minutes to act. So they decided to avoid the problem altogether, change the route, or have someone remotely control it. Remote operation is an important tool for autopilot system, but it has not been paid enough attention to. )
This is feasible, because the future of refraction AI does not depend on mastering the tricky four-way parking technology. The company’s only important measure is whether it can get University of Michigan students to eat hot hamburgers and chips.