The New GM Cruise Safety Plan
What is in -- and missing from -- Steve Kenner's safety plan for Cruise?
Last week Steve Kenner, Cruise’s chief safety officer, made some big reveals about their safety plan at TRB ARTS in San Diego. You can see coverage here (paywall): https://www.autonews.com/mobility-report/steve-kenner-outlines-safety-overhaul-gms-cruise (My comments below are primarily based on that written report. If I’m missing something big I invite Cruise to share the presentation slides or additional written information publicly.)
According to the article, the major themes for safety at Cruise are:
(1) ‘Role model driver’ instead of ‘better-than-average-driver’. This is a good aspirational goal. People clearly expect better of computer drivers. The approach is apparently going to consist of:
“Benchmarks” that sound a lot like what other companies call scenarios, such as left turns, with metrics indicating performance. Perhaps broader scope than strictly scenarios, but smells that way for now.
“Brittleness” metrics such as frequency of “glitches” that can detect service disrupts such as mass strandings.
Generally these sound like KPIs that emphasize avoiding embarrassing social media videos and headlines. This is a must-have to repair their public reputation.
(2) Third-Party Review. They promise to hire an independent evaluator to conduct a safety review before driver-out operations. I applaud this step, because you simply won’t get safety without independent oversight. My money is on a reasonable selection of TUV-SUD, but we’ll see who they pick. The question here is what criteria will be used for the evaluation? Will it be a more generic process audit, or a deep look at the technical parts of the product as well? And will they make a report finding summary public so we know the scope of the review as well as conclusions?
Recall that Cruise said once before they’d get an independent look at safety processes, but that promise evaporated as weeks went by after the crash. Let’s hope it sticks this time.
(3) Collaborative Data Sharing. MITRE is apparently standing up an autonomous ground vehicle data sharing initiative. This is great, and I hope other companies join.
Apparently there are other things in the works. But I’m reluctant to read much into anything said that hasn’t made its way into the press or written public documents. Let’s hope Cruise says more over time.
So what’s missing — besides a lot more detail so we can understand what Cruise is really going to be doing? At a high level some things I have still not heard are:
Whether they are going to say more about internal safety process improvement. At one extreme it is possible their safety processes were near-perfection before the tragic pedestrian dragging mishap and their previous CEO simply over-rode safety to speed up deployment. At the other extreme is a lot more work needed on safety processes and staffing. Which is it, and what’s the plan?
What exactly is that safety case they’re talking about? In practice some people mean a rigorous analysis, while others are just waving their hands.
How will they know their metrics are actually predictive of safety? What they are talking about sounds a lot like lagging KPI indicators about driving quality. But there is a lot more to safety at scale than careful driving, including especially fault management and safely responding to surprises such as unknown unknowns that show up in the environment. How will they know their metrics have predictive value for driver-out safety? If metrics are not directly tied to specific claims in a safety case, it is difficult to know if they have any predictive value for when the time comes to scale up.
What exactly will the criteria be for driver-out operation? Aurora seems to be signaling that they are waiting for 100% completion of their safety case (especially evidence) and tracking that percentage. What will the Cruise story be? If Cruise says they are going to transition to “driver-out testing” as they did last time around, to me that is a huge red flag. You don’t take the safety driver out until you’re sure the result will be safe. But we don’t know their transition plan, so we’ll have to wait and see.
Certainly this was better than previous corporate-style messaging that was more about regulatory capture than achieving safety. But I find it somewhat odd that they haven’t coordinated a more public release of the messaging delivered at the ARTS conference. (Which, BTW, had a lot of regulators in attendance.) Hopefully we’ll hear more soon.
An interesting question is when/how they try to regain their permits in California. The Quinn Emanuel report appeared to list the criteria for that (see footnote 52 on document page 77 / Acrobat page 81). While other states are much more permissive, I would think regaining regulatory trust in California would send a strong message of positive change.
I hope Cruise is able to get themselves on a reasonable track to safety. A big lesson that should be learned is that small-scale lack of mishaps can easily be more about getting lucky than being safe. But safety lives in the long tail events, and getting lucky doesn’t scale.
The US can benefit from more competition able to actually deploy safe autonomous vehicles. I wish Cruise the best for their rebirth. Now we wait to see how things go.
Saying the CEO overrode safety is accurate but not precise. There were many middle managers, many of whom are still there and have since promoted to director roles, that told me to my face that there was far too much money on the line to abide by the operational safety checks we had ourselves set. Benchmarks routinely qualified or outright ignored to preserve deployment velocity. Cruise is facing all of the same incentives now (I.e., get to market as quickly as possible) with all the same leadership in all the same places. One can only hope they don’t cut all the same corners.