Overview
How can lawmakers, regulators, or the general public obtain useful information on the safety level of autonomous vehicles on the road today? Currently, there is no consistent, industry-wide practice for reporting ADS (automated driving system) performance metrics. That is where the Automated Vehicle Safety Consortium (AVSC) comes in.
What Is The AVSC?
The AVSC is an industry program of SAE Industry Technologies Consortia (SAE ITC). AVSC’s core members are Aurora, Cruise, GM, Lyft, Motional, Torc Robotics, Uber, Volkswagen, Waymo, and Zoox. Consortia members commit to applying their experience and combined knowledge to earn public confidence in the safe operation of SAE Level 4 and Level 5 automated vehicles, including both light-duty passenger and cargo vehicles.
Earlier in 2023, AVSC released its Best Practice for Developing ADS Safety Performance Thresholds Based on Human Driving Behavior, also called AVSC00012202308. The AVSC publication explains a process that automated driving system-dedicated vehicle (ADS-DV) developers can follow to establish reference metrics (aka baseline safety data) for safe human driving in particular scenarios.
With those metric reference values in hand, developers can set safety performance targets (aka quantitative performance baselines) for the same driving behaviors performed by their vehicles. Those performance baselines can then inform decisions made by Congress, regulators, and the like.
Primary Procedure Points
AVSC00012202308 builds on previous Best Practices offered by the Consortia, specifically AVSC0006202103 and AVSC0008202111 – all of which are technology-neutral documents that provide key considerations for safely deploying automated vehicles on public roads.
According to the AVSC, the reference values are based on safe, naturalistic driving studies (NDS) performed by manufacturers or third parties and are typical of safe human-driving maneuvers. AVSC recommends this procedure for obtaining reference values from human driving data:
#1: Determine the scenario and the behavior of interest.
#2: Filter data from naturalistic data studies.
#3: Analyze data and perform tests.
#4: Use results to inform performance metrics and set performance targets.
“This process enables manufacturers to enhance the safety performance of ADS fleets by aligning with human-relative benchmarks and considerations,” AVSC wrote in the best practice.
Data Helps Tell The Story
I was curious to know how this process would play out in an ADS–DV lab, so I connected with Matthew Chee, principal engineer at AVSC. “Let’s start with step one and say, for example, the scenario or behavior of interest is how a vehicle passes a cyclist on the side of the road,” Chee said. “The manufacturer would collect their own data on vehicles passing cyclists or buy data that has been recorded by others.”
For step two, he explained, the manufacturer would filter the data. If they only want data on vehicles passing cyclists on city streets, for example, they would filter for that. For step three, he continued, they would analyze the data. The AVSC best practice offers possible methods for doing so, including looking at things like statistical significance, hypothesis testing, and other relevant testing.
Finally, he explained, they would pull out numbers that show, for example, that in over 50 percent of the cases of good driving behavior, human drivers slow down to below Y miles per hour when passing a cyclist on city streets. They might also see that in over 60 percent of the cases, human drivers leave a gap of X feet between their vehicle and the cyclist when passing.
“That data becomes the manufacturer’s reference values,” Chee said. “From there, the manufacturer decides if they want to match human driving behavior or act more cautiously.”
More Transparent Communication
Chee emphasized what’s important is that by following this process, lawmakers, regulators, and the public can understand how a manufacturer obtained those reference values against which they are comparing the performance of their vehicles. “Otherwise, there is no universal determination for safety performance thresholds,” he said. “With the resulting performance baseline data in hand, developers can describe to Congress, the general public, and other relevant parties how their automated driving systems behave compared to human drivers.”
The Consortia cautions that when using naturalistic driving studies for this purpose, ADS developers should consider only good human driving behavior, as these studies inherently also include data on poor human driving behavior. “To ensure safety, ADS developers must incorporate various data sources, define parameter sets, and establish safety thresholds throughout the design, construction, verification, and validation (V&V) processes,” the AVSC authors wrote.
Deeper Understanding & Greater Balance
The proposed approach involves comparing the on-road driving behavior of an ADS to that of human drivers, aiming to gain a deeper understanding of ADS behavior and its implications, the AVSC authors explain. The assessment centers on the predictability and similarity of ADS behavior relative to observed human driving behavior, they added. “By adopting this approach, a more comprehensive evaluation of the performance of ADS fleets can be achieved,” they wrote. “This empowers developers to identify areas for improvement and potentially unknown unsafe behaviors.”
Reference points for assessing the relative safety performance of ADS fleets are crucial, AVSC further explained. “The safety performance of some ADS behaviors can then be measured and compared to NDS data from human drivers to help characterize the socially acceptable balance between safety, lawful driving, efficiency, and comfort,” they wrote.
According to Chee, the overarching idea of this and the AVSC’s other best practices is to offer a foundation for continuous improvement across the industry to help promote safe development over time.
Vital Puzzle Piece
Where are we today on the ability to discern the safety level of robotaxis and other self-driving vehicles? How does this most recent AVSC best practice fit into the automated driving safety ecosystem? As the industry improves its safety processes across the board, AVSC00012202308 provides a piece of the safety puzzle.
This particular best practice concentrates on whether an SAE Level 4 and Level 5 vehicle’s normal driving behavior matches that of a responsible human driver. It (intentionally) does not cover other safety aspects, such as responding to equipment or road infrastructure failures. And that’s okay – it’s just important to be aware there are intended limitations and that this is an important piece of safety rather than all of safety.
AVSC best practice documents and publications, both past and current, can be downloaded via the Consortium’s official website.
