Car software is riddled with bugs

This piece in five points:

• Does it matter these days how much horsepower a car has? For some, sure. Yet software performance is probably a lot more important when you consider that a modern vehicle boasts an amount of lines of programming language well in excess of 100 million.

• Traditionally, car manufacturers design and build their most important components themselves. With software, the situation is different because that has never been their area of expertise. Much software development, including cyber security, is outsourced.

• However, there is a change in this: as part of a major catch-up drive, manufacturers are investing heavily in developing their own software. A number of groups have even set up special subsidiaries for this purpose.

• Automotive software is extensively tested, yet it is packed with - mostly harmless - bugs. In general, software contains an error in an estimated 1 in 20 lines of code. This is partly because the automotive industry is fiercely competitive and engineers are under pressure to introduce innovative gadgets as quickly as possible. 'Ship now, fix later' is a common credo in the industry.

• If something is wrong with your car, it is increasingly likely to be due to defective software. Reports of motorists going crazy from (dangerous) software bugs, software-related recalls, flawed cyber security and hacks, as well as mass damage claims are hard to keep up with.

That a top executive of a major car group like Volkswagen ever in prison would end up as a result of widespread software tampering under his leadership, few could have imagined beforehand. It happened to Oliver Schmidt. With 'Dieselgate' VW and also other carmakers showed their ugly side. The story is well known: programmers were found to have worked on the sly for years on specific software to outsmart emissions tests. When cars were on the test bench, they knew they had to emit less exhaust than they normally would on the road. Lawmakers were completely unaware that this was possible.

The scandal that came to light in September 2015 has still not died down. Millions of cars were recalled worldwide, while dragging court cases in several countries resulted in sky-high fines and some prison sentences. Just last June, eight years after the initial revelations, other top executives got suspended prison sentences for their part in the massive rigging.

Last week, the judge in Amsterdam several claims organisations as representative advocates in their legal battle around tampering software in Peugeot, Citroën and Opel diesel cars. In the next phase of this case, the court will determine whether owners of such models have received a defective product that does not comply with the law, and they are therefore entitled to compensation.

The question still arises whether it would not have been cheaper and smarter, instead of investing so much in software to get out of compliance, to simply make the cars comply with environmental standards (lower emissions). And phase out any performance loss as a result, including through overall software development.

Millions of lines of programming language

Because it is clear that there is a world to be gained (but therefore also lost) with software. For years, everything in cars has been about software. With more than 100 million lines of programming language all "doing" something, a modern car is just a little less complex than a mouse's genome with 120 million base pairs in its DNA, according to a scientist in 2016. Now, seven years later, that number of rules will have increased significantly, especially given increasingly sophisticated driver assistance systems.

A vehicle's software architecture consists of a mix of free open source code, whether ready-made code from commercial parties or not, previously self-written and out-developed legacy code plus new self-written code. Car manufacturers are currently concerned on a bill from Brussels that the European Parliament will vote on next month. They fear that the so-called Regulation cyber resilience - which will provide general requirements and standards for software cybersecurity - will complicate the work of the open source community and make developers of such software (e.g. the Eclipse Foundation and the parties united within COVESA, the Connected Vehicle Systems Alliance) will refrain from further cooperation with the auto industry for fear of being held responsible for (not) meeting the obligations of the new regulations.

Outsource

Much software development, including cyber security, is outsourced by car manufacturers. In doing so, they do to a large extent direct themselves, accurately specify their programming requirements and, of course, ensure that the intellectual property over the source code falls into their hands. Some of hundreds of high-tech companies which may or may not focus exclusively on automotive software are Continental and its subsidiary Electrobit, Hitachi and his daughter GlobalLogic, Harman (daughter of Samsung), Intellias, Bosch, Perforce, IPG Automotive and, for example Keysight. Well-known semiconductor manufacturers such as NXP and Infineon are (indirectly) closely involved in software development for connected cars.

These types of companies mainly focus on the core software which takes care of the actual operation of all systems in the vehicle. Mostly, however, media attention focuses on the in-dash infotainment system and the extent to which car manufacturers rely on two of the best-known tech giants, namely Apple and Google, to develop it (more on this in a forthcoming article).

Catch-up

Traditionally, car manufacturers design and build their most important parts themselves. After all, that which is of crucial strategic importance and which allows you to stand out from the competition is something you want to keep in your own hands as much as possible. The paradox is that this is different with software. Software has never been the area of expertise of car companies.

The main exception to this is Tesla, which from the outset was made up of computer scientists who do everything self-coding and also approach things in a very different way than is traditionally done in the automotive industry. [To the huge data streams of all the Tesla's driving around and overloading the current network Incidentally, last month Tesla CEO Elon Musk announced the construction of a supercomputer of your own ('Dojo') worth $1 billion].

Switch to the state of mind which is more peculiar to Tesla, did not go smoothly for many traditional car companies. Yet they are all now playing catch-up, not least also to cope with new Chinese competitors - some of which are still barely inferior to Tesla. Especially given the increasing number of electric cars, manufacturers are increasing their desire to be able to develop crucial software themselves. They are investing in capabilities and talent, creating whole new industries according to their ambitions. Stellantis, for instance, as the parent company of 16 car brands, is creating one software development hub after another. There are now eight in eight different countries worldwide.

Other random examples of subsidiaries entirely focused on software include the Renault Software Factory, BMW Car IT (''We live and breathe software') and CARIAD, the Volkswagen Group's software and technology company that employs thousands of people. Partnerships with software companies are all-too-common: for example, CARIAD works through the joint venture e.solutions for example, again working closely with Elektrobit. Honda just struck join forces with the also Japanese software group SCSK.

Acquisitions are also the order of the day: a few years ago, for example, Volkswagen acquired telematics specialist WirelessCar over from Volvo and the autonomous driving platform Argo AI came owned by both Volkswagen and Ford. Meanwhile, the Argo AI lifted in name. Equally, the technology is being used by both carmakers in-house expanded.

Mercedes announced earlier this year the introduction of the Mercedes-Benz operating system (MB.OS) to: a chip-to-cloud-architecture with comprehensive software features for all future vehicles. Whereas the manufacturer previously relied on collaborations - including with semiconductor manufacturer Nvidia for software for automated driving - the group is now emphatically taking matters into its own hands. Last year, Mercedes achieved sales of over a billion dollars from the overall use of software in cars. MB.OS is to ensure that those sales reach around ten billion by 2030.

Bugs

Of course, the average motorist does not care about all this. They just want their car to get from A to B in a decent and comfortable way. To ensure this, cars are tested extensively. First in simulations, where at some point you can skip the long stretches on the motorway where nothing actually happens, and pre-programme unusual traffic situations or scenarios that would probably not occur after millions of kilometres in reality, to see how the software reacts to them. One company that is adept at this and has realistic test rigs and systems is AVL from Graz (Austria). Then a large number of test kilometres on a racetrack or public road are covered to make sure no strange things happen. And then the software is basically ready for use.

So much, in very condensed form, for the rosy story. The fact is that reporting on motorists going crazy of (dangerous) software bugs in their cars, about software-related recalls, lack of cybersecurity and hacks as well as mass claims barely keeping up. And to think that The Economist here thirteen years ago already wrote about - Probably not the first.

If there is something wrong with your car, it is increasingly likely that it is due to software playing tomfoolery. In general, software contains an error in an estimated 1 in 20 lines of code. Coding is and remains largely human work. It sounds a bit worse than it really is, but viewed this way, a vehicle is a system riddled with hairline cracks.

Technical debt

It can take years for an idea for a new feature in a car is well and truly realised and marketed. Manufacturers are increasingly trying to shorten the duration of such development cycles. Within the fierce competition raging in the automotive sector, engineers are under pressure to introduce innovative gadgets as quickly as possible. A significant amount of technical debt is then almost inevitable. Technical debt amounts to half-baked software solutions, patches and short cuts rather than a solid and mature script. These solutions are often the result of the (forced) choice to prioritise speed over thoroughness. 'Ship now, fix later' is a principle widely used across the software industry.

Intentions to rectify such half-sloppiness later are far from being met. In this sense, technical debt is a debt on the future. All sorts of safety nets are put in place to intercept erroneous behaviour, and usually things go well because most bugs in the code are harmless. But not always: there may be special circumstances where a part of the software is called upon to exhibit quirks. Certain bits of code - which did not cause any problems during testing - are woken up and executed, so to speak, and then you may just find yourself as a motorist with a nasty bug to deal with.

In many cases, those bugs causing a car to panic out of nowhere, as it were, and do crazy, unexpected things. For example, taking 'corrective' action where it is not wanted or needed at all. For example, braking sharply just because someone is neatly cycling next to the vehicle, or, for example, entering a tunnel and it is dark overnight. [For a year or two now, signs have been posted at the Westerscheldetunnel near Terneuzen, among other places, instructing motorists to use the Adaptive Cruise Control to be switched off because otherwise there is a risk of cars in the tunnel applying full brakes, with dire consequences].

Who is responsible in case of, say, an accident caused by faulty car software? It's a logical question. Only the answer is less obvious. According to insurer Arag in fact, it may well be the motorist. Moreover, the manufacturer may be off the hook if it was not possible to detect the defect when the car was sold.

For the record: in all cases, these are cars that have passed type approval and thus have been declared safe to drive. For now, that inspection only marginally looks at the functioning of the software, if only because the inspection authorities (in the Netherlands, the RDW) are often not well-equipped to do so. Discussions between stakeholders are ongoing to change this over time.

Not directly dangerous, but annoying: infotainment systems also turn out to be not always flawless. For instance, there are stories of dashboard screens freezing, flickering or going black; or, for instance, music suddenly stopping, or suddenly blaring far too loudly from the speakers. Unlike the problems described above, you can sometimes fix these kinds of bugs yourself.

So much software on wheels naturally makes one think about cyber security, and hence privacy. What about that? Errors and shortcomings can be rectified through updates, whether or not over the air (i.e. over the internet). However, there are several snags in that too. These subtopics related to connected cars are the focus of upcoming articles.

Different accounts

Finally, a side observation regarding in-vehicle software. Whereas it is common to log into a PC or, say, a streaming service like Netflix with your own account, the multi-user concept has strangely not yet reached the automotive sector. The possibility of allowing different riders to create different personal accounts has not yet been introduced. In other words, the car does not adapt to the user; as ever, the user has to adapt the car to himself.

Different accounts would not only be convenient in view of individual preferences and settings that would jump once (playlists, music volume, temperature, position of mirrors and seats, navigation to frequently visited destinations, etc.). It would also ensure 'data hygiene' in vehicles that leave a huge trail of data. For instance, contact lists and other imported material from different users' smartphones would not get mixed up in the infotainment system. An additional benefit would be that if you forget to delete your data in the case of a rental or leased car, the next driver would not be able to access your data.

Logging into a personal account obviously requires some form of identification (which takes a bit more time when driving away), but there are several solutions to that. Car manufacturers could at least build the multi-user concept into their software without too much trouble. But they are hierarchical organisations where even the - often more conservative and design-focused - top must believe in such innovations. And must be convinced that motorists increasingly want to use their car in a way that is compatible with other products and digital services.