What’s Under the ‘Hood’ of Self-Driving Cars?

Headed by tech giants like Google and Yandex, the rise of automated vehicles seems inevitable, but what are the intellectual mechanics at work?

A vehicle equipped with Pony.ai's self-driving technology is parked at the company's office in Fremont, Calif., on June 17, 2021. Nathan Frandino/Reuters

By Olga Megorskaya

Waymo, a unit of Google’s Alphabet Inc., and Yandex Self-Driving Group, a division of the Russian-based Yandex corporation, are among more than a dozen leading names in automated vehicle software and hardware. The former recently launched a pilot self-driving taxi program in San Francisco, the latter has been testing its automated cars worldwide for the past several years and has self-driving rovers on several college campuses in the U.S. Not to be outdone, the Chinese tech company Baidu Inc., along with the Toyota-backed Chinese self-driving startup, Pony.ai, is set to debut a 100-car fleet of paid driverless taxis in Beijing in 2022, and has preliminary plans to launch a similar program in California in the same year (see link below).

The main challenge to such vehicles’ broader applicability, however, is training the AI that powers self-driving to the point of flawlessness.

It’s All About Data

Software and data engineers train the algorithms that lie at the core of artificial intelligence, and for self-driving cars, data relating to the roads and driving conditions on and around them is key for navigation. To do so, AI must first learn what this data means, and this is where “data labeling” plays a critical role. Labeling data (essentially attaching a meaning to it) is the first step in creating this kind of full-fledged AI. Such labels, or meanings, must be informative, discriminating, and independent. They also need to be precise and correspond to reality (the ground truth), which is why data labeling is a human-based process—often a tedious and arduous one, requiring thousands of people to examine and annotate (to label a vehicle as either a truck or a car, for example, or to distinguish among traffic light colors).