Challenges in Autonomous Vehicle Testing and Validation
A study, prepared by Strategy Analytics, predicts autonomous vehicles will create a massive economic opportunity that will scale from $800 billion in 2035 (the base year of the study) to $7 trillion by 2050. An estimated 585,000 lives could be saved due to autonomous vehicles between 2035 and 2045, the study predicts.
While science fiction movies of the 20th century predicted that we’d have flying cars by now, it appears that there are a different set of overarching themes in the auto industry currently on the cusp of becoming a widespread reality. Electric power and the discussion around emissions, has arguably received the most media attention, however other trends such as vehicle sharing, and vehicles being connected to the internet and each other are also becoming prominent. A yearly updating model, where both software and hardware (particularly in fleets), will also become relevant to reflect rapid changes in technology. The final trend is perhaps one with the most technological challenges, and the greatest implication for changing how humans interact with cars (and transport in general): autonomous vehicles.
An autonomous vehicle (also called a driver-less or self-driving vehicle) is one that senses its environment and operates without any human involvement. Companies such as Waymo, Uber, Tesla, and a myriad of others have been testing solutions for many years, however the task of testing and validating driverless vehicle technology comes with an array of challenges. Coupled with the fact that lives are at stake (both for passengers and pedestrians), there is little room for error. So what are the challenges facing autonomous vehicle testing and validation?
Let’s start with the issue of gathering enough data, which needs to be collected both from vehicle sensors and from external sources, which is then used to train AI models. While real-world testing offers valuable feedback, there is the difficulty of scaling, where it is physically infeasible to run the millions of miles of tests required to gather enough data. Additionally, there is a need for test commonality, whereby identical tests are run with different types of vehicles. So how does such a large number of tests get carried out in an appropriate time frame? The answer lies in parallelized computer simulations, where, in the case of Waymo, they can achieve around 20 million miles a day in their Carcraft simulation platform (the equivalent of 100 years of driving on public roads). Waymo Driver, the company’s autonomous vehicle software suite, has reportedly accumulated over 15 billion simulated autonomous miles as of April 2020.
The AI that drives a vehicle needs to also take into account various nuances during the vehicle’s operation, including the comfort of its passengers, which can sometimes be an overlooked factor during the testing process, which is why Waymo takes into account what it calls “Comfort Metrics”. Jonathan Karmel, Product Lead, Simulation & Automation at Waymo explained to VentureBeat that “Some of the key components are things like acceleration and deceleration, and we want to receive that information into simulation to predict what we think a rider or driver reaction would have been in the real world. There’s a machine learning model to predict what those reactions are in (Carcraft).”
While comfort is important, safety is essential. Autonomous vehicles sometimes struggle during adverse weather conditions, and when approaching unusual objects on the road ahead. In 2016 a Tesla operating in self-driving mode crashed into an overturned white truck trailer on the highway, and another Tesla in 2018 slammed into a parked fire truck. In both situations, the car’s sensors failed to ‘see’ the objects ahead. This notion of ‘seeing’ the road and approaching objects becomes even harder during bad weather. An autonomous vehicle uses multiple systems to drive itself, including GPS, traditional cameras, radar and LIDAR (a technology that bounces lasers off the surrounding environment). The LIDAR system can lose accuracy when operating through raindrops and snowflakes, and the car’s cameras can be blocked by fog or heavy snow. Potential solutions to these problems include creating overlapping, redundant systems on the car itself (in case one is inhibited), along with cars communicating with each other, and even embedding sensors in the pavement to feed data to the surrounding vehicles.
Developing the necessary hardware and software, along with the planning and manpower required to continually test the technology, requires huge capital investments. It is estimated that Waymo has spent around $3.5 billion so far, while Cruise has spent over $1 billion. Uber’s spend, as of 2019, has exceeded $1 billion, although the company has subsequently received investment and guidance from Toyota. While cost is often a natural obstacle for many pioneering technologies, the ongoing COVID-19 pandemic has put an added strain on many autonomous vehicle testing operations, with various companies (including Uber and Cruise) halting their physical testing.
Assuming the autonomous vehicle industry overcomes its technological and economic challenges, there is one final obstacle to navigate once the technology is perfected: public trust. After a series of high-profile incidents, for example in March 2018 in which a self-driving Uber car struck a pedestrian who was walking across the road with a bicycle, public sentiment is skeptical of this emerging technology. According to a 2019 AAA survey, a worrying 71% of people are afraid to drive in fully driver-less vehicles. However earlier this month Waymo opened a small number of autonomous ride-hailing cars to a limited number of customers in Phoenix, with operations slowly ramping up over the coming weeks to anyone who downloads the smartphone app in the area.
Request a demo and see it Titan in action