The future of self-driving cars is here, but it's not as smooth a ride as we might have hoped. Despite the progress, these autonomous vehicles still need a human touch to keep them on track. The reality check: self-driving cars are not as independent as we'd like to believe.
At a recent industry gathering, The Autonomous 2025 in Vienna, a former Navy pilot turned academic, Mary Cummings, dropped some truth bombs. She argued that while self-driving cars are a reality, they're far from perfect. And here's where it gets controversial: these cars need human supervisors, or as Cummings puts it, "babysitters," to step in when things get tricky.
"Self-driving cars hallucinate," she warned. These hallucinations are more than just a glitch; they can lead to serious accidents. Cummings cited an incident where a Tesla suddenly decelerated from 65 mph to zero, causing an eight-car pileup. The reason? The car "saw" something that wasn't there.
And this is the part most people miss: these hallucinations are a fundamental technical issue that's blocking large-scale deployment. It's not just about the technology; it's about understanding the limitations of AI. "AI doesn't think. It doesn't imagine, it doesn't know," Cummings emphasized. She warned against relying solely on generative AI for safety cases, highlighting the need for human oversight.
Cummings reserved her strongest criticism for vision-only autonomous systems. She argued that no robot can reliably navigate using a single sensor modality. And when it comes to highway driving, no self-driving vehicle company has demonstrated consistent, safe operation at high speeds. "Every self-driving car company needs human babysitters," she said, "because they all require some level of human input."
The industry is aware of these challenges. The Automated Vehicle Safety Consortium has released a best practice document distinguishing between remote assistance and remote driving. Remote assistance is about providing information to help the vehicle continue its journey when it encounters a tricky situation. Remote driving, on the other hand, is when a remote driver takes over the dynamic driving task in real-time.
Cummings cautioned against offshoring remote-operations staff, pointing to the risks of signal delay and latency. She shared an example of a Waymo vehicle being broadsided due to a delayed remote-assist signal. "Nothing good comes out of parking your remote operators halfway around the world," she said.
So, where does that leave us? Well, the notion of fully self-driving cars is still more of a dream than a reality. As Cummings put it, "At best, you are a human-babysat self-driving car company." TTTech Auto's CEO, Stefan Poledna, echoed this sentiment, arguing that some form of human involvement is currently unavoidable.
Poledna highlighted the challenges of intermittent supervision, especially in scenarios where human drivers are physically present in the vehicle. He suggested that a Level 3 or Level 4 system could keep the vehicle safe by stopping or pulling over and asking for human assistance when uncertain. In that sense, the human becomes the onboard babysitter.
Cummings brought up the Swiss cheese model of accident causation, emphasizing the need for redundancy in safety-critical autonomy. She shared her experience as a lead expert in a court case where Tesla was ordered to pay $243 million in damages for a fatal crash linked to "bad AI technology." The verdict, she said, revealed the industry's tendency to cut corners on testing and training neural nets.
"Safety-critical autonomy demands redundancy," she stressed. "We can't build safe chips," Poledna agreed, arguing that today's software and chip technologies fall short of delivering a single monolithic component up to the safety level of a human. System-level redundancy across sensors, compute, communication, and actuation is essential, he said.
Cummings acknowledged her cautious perspective but reaffirmed her belief in the potential of self-driving vehicles, especially in the context of robotic shuttles. She urged industry collaboration, saying, "America is hyper-competitive; companies refuse to work together, and they refuse to work with regulators. This is not the path forward."
Sharing data, especially on AI hallucination, is key to bringing safer autonomous systems to market, she said.
So, what do you think? Are self-driving cars ready for prime time, or do we still need to babysit them? Let's discuss in the comments!
