LLMs provide brains for smart self-driving cars: paper suggests

A new study demonstrates large language models can serve as effective decision-makers for autonomous vehicles, thinking logically about complex scenarios.

A group of researchers from Tsinghua University, the University of Hong Kong, and the University of California, Berkeley, have investigated autonomous driving technology using large language models (LLMs) for high-level decision-making. Their new paper, published on the preprint server arXiv on Oct. 4, demonstrates that LLMs can successfully comprehend traffic scenarios, make reasoned judgments adhering to rules, and provide clear explanations for their decisions.

While existing autonomous driving systems based on deep learning have shown promise, they still face challenges when dealing with rare events and providing interpretability. The paper reads:

“LLMs can think like humans, and reason about new scenarios by combining common sense, and the visible thinking process makes them strongly interpretable. […] The reasoning skills and interpretability of LLMs help overcome the limitations of current learning-based [autonomous driving] systems regarding adaptability and transparency.”

To leverage the strengths of LLMs, the researchers devised a structured thought process to manage the reasoning steps. The LLM gathers pertinent information, evaluates the driving scenario, and provides high-level action guidance. These textual decisions are then converted into parameters that direct the low-level controller. Extensive experiments highlight the considerable performance gains using this approach.

Compared to reinforcement learning and optimization methods, the LLM-enhanced system achieved substantially lower costs and higher safety across diverse driving tasks, including intersections, roundabouts, and emergency maneuvers. The paper claims:

“This paper presents an initial step toward leveraging LLMs as effective decision-makers for intricate [autonomous driving] scenarios in terms of safety, efficiency, generalizability, and interoperability.”

Beyond quantifiable metrics, the LLM reportedly demonstrated situational awareness and adaptability resembling human drivers. For example, it appropriately slowed down when another vehicle had the right-of-way, rather than optimizing for efficiency alone.

While still in an early research stage, this pioneering work lays the foundation for continued advances. The paper concludes:

“We aspire for it to serve as inspiration for future research in this field.”

Featured Image Credit: istockphoto; Pexels; Thank you!

The post LLMs provide brains for smart self-driving cars: paper suggests appeared first on ReadWrite.