Within the scope of AVs, the risk = frequency x severity approach is sometimes infeasible and possibly even misleading. This can happen when there is no way to acquire the data necessary for estimating a usefully narrow interval around the frequency of a rare but consequential event, or because the frequency is subject to high and unpredictable variance. In both cases, naively applying this equation can lead to underestimation of risk. In the presence of the kinds of pressures described in this article, when faced with such estimation challenges and when required to use this kind of risk framework, employees could be compelled to err towards a lesser rather than a maximal frequency estimation (especially if frequency is estimated subjectively), as in these cases the uncertainty around frequency could cause it to essentially become the highest possible level, resulting in even more engineering work to get in the way of tight deadlines.
It might be worth considering that safety certification and operational approval is not necessarily binary. By properly assigning risk into frequency and severity, as is done routinely in military design reviews, it may be possible to proceed cautiously enough by identifying and mitigating consequential risks. For example, if AV safety performance at railroad crossings is questionable, then a combination of geofencing and design/staffing for active interdiction at crossings might provide requisite safety assurance for the planned test objectives. It is unlikely that any test program undisciplined enough to either merely drive anywhere without restrictions or test without defined and achievable test objectives will yield acceptable safety or public confidence in safety. Autonomous driving until unacceptable crashes occur will never be enough.
As for industry acceptance of death by AV, the firearms industry experience may be illustrative. The annual number of deaths from firearms in the USA is roughly 10% of the number of industry employees. ( In 2021 - 48,830 deaths/375,819 employees, [https://www.pewresearch.org/short-reads/2023/04/26/what-the-data-says-about-gun-deaths-in-the-u-s/ : https://www.nssf.org/articles/firearm-industry-full-time-jobs-reach-375819-in-2021/ ]) It seems that corporate America and its employees are quite comfortable with one death (and probably 10 serious injuries) for every 10 FTE. That is a pretty grim precedent for the AV industry ethos. Accepting that would require similar scope of self-delusion about the benefits and jeopardies of the AV technology, but that self-delusion seems pretty well established and heavily supported by the industry.
Great description of how some of this process could occur in a business - I think it applies as much to a startup as to a public company. Part of the problem for many of these companies is that when they claim to be prioritizing safety, or even measuring it, it's superficial metrics like disengagements or incidents per distance traveled or per trip, rather than really building a safety case from the ground up and ensuring robust design and testing practices.
This is very realistic characterization of how corporations evaluate business risk.
Their cognitive biases make them downplay the business risk of negative safety outcomes.
"Safety First" is a slogan but it only gets lip service. You can see this trait outside the AV business too.
For example, corporations have become very IT-dependent, but the decisionmakers don't understand the business risk of poor cybersecurity hygiene. A "Department of Homeland Security" study found that 90% of the cyber-intrusion incidents could be traced to well-known weaknesses and vulnerabilities in their software - issues that do not require HiTech solutions. Of course, adding the quality criteria would increase their acquisition cost, but they do not have a good understanding of the downstream cost of intrusions. The easily-quantifiable present cost outweighs the hard-to-quantify future costs.
Wow. The safety manager you describe is the lived experience of one of my students (probably more) a few years ago. This clip from one of my lectures was developed based on their experience. Very similar to your post.
Within the scope of AVs, the risk = frequency x severity approach is sometimes infeasible and possibly even misleading. This can happen when there is no way to acquire the data necessary for estimating a usefully narrow interval around the frequency of a rare but consequential event, or because the frequency is subject to high and unpredictable variance. In both cases, naively applying this equation can lead to underestimation of risk. In the presence of the kinds of pressures described in this article, when faced with such estimation challenges and when required to use this kind of risk framework, employees could be compelled to err towards a lesser rather than a maximal frequency estimation (especially if frequency is estimated subjectively), as in these cases the uncertainty around frequency could cause it to essentially become the highest possible level, resulting in even more engineering work to get in the way of tight deadlines.
It might be worth considering that safety certification and operational approval is not necessarily binary. By properly assigning risk into frequency and severity, as is done routinely in military design reviews, it may be possible to proceed cautiously enough by identifying and mitigating consequential risks. For example, if AV safety performance at railroad crossings is questionable, then a combination of geofencing and design/staffing for active interdiction at crossings might provide requisite safety assurance for the planned test objectives. It is unlikely that any test program undisciplined enough to either merely drive anywhere without restrictions or test without defined and achievable test objectives will yield acceptable safety or public confidence in safety. Autonomous driving until unacceptable crashes occur will never be enough.
As for industry acceptance of death by AV, the firearms industry experience may be illustrative. The annual number of deaths from firearms in the USA is roughly 10% of the number of industry employees. ( In 2021 - 48,830 deaths/375,819 employees, [https://www.pewresearch.org/short-reads/2023/04/26/what-the-data-says-about-gun-deaths-in-the-u-s/ : https://www.nssf.org/articles/firearm-industry-full-time-jobs-reach-375819-in-2021/ ]) It seems that corporate America and its employees are quite comfortable with one death (and probably 10 serious injuries) for every 10 FTE. That is a pretty grim precedent for the AV industry ethos. Accepting that would require similar scope of self-delusion about the benefits and jeopardies of the AV technology, but that self-delusion seems pretty well established and heavily supported by the industry.
Great description of how some of this process could occur in a business - I think it applies as much to a startup as to a public company. Part of the problem for many of these companies is that when they claim to be prioritizing safety, or even measuring it, it's superficial metrics like disengagements or incidents per distance traveled or per trip, rather than really building a safety case from the ground up and ensuring robust design and testing practices.
This is very realistic characterization of how corporations evaluate business risk.
Their cognitive biases make them downplay the business risk of negative safety outcomes.
"Safety First" is a slogan but it only gets lip service. You can see this trait outside the AV business too.
For example, corporations have become very IT-dependent, but the decisionmakers don't understand the business risk of poor cybersecurity hygiene. A "Department of Homeland Security" study found that 90% of the cyber-intrusion incidents could be traced to well-known weaknesses and vulnerabilities in their software - issues that do not require HiTech solutions. Of course, adding the quality criteria would increase their acquisition cost, but they do not have a good understanding of the downstream cost of intrusions. The easily-quantifiable present cost outweighs the hard-to-quantify future costs.
Wow. The safety manager you describe is the lived experience of one of my students (probably more) a few years ago. This clip from one of my lectures was developed based on their experience. Very similar to your post.
https://youtube.com/clip/UgkxtLsfgvyTAonHvKRDAgfLX0LIczF1Njvq?si=D91vlr3fR5se2Nln
Art imitates life.