What happens when six active maintainers manage a codebase that is used in hundreds of products that maintain large parts of our digital everyday life without being paid in their free time? We had this impressively demonstrated shortly before Christmas 2021 with the Log4Shell vulnerability. This vulnerability in the Java logging library Log4j made it clear to almost everyone who has anything to do with managing source code or administering software how dangerous software dependencies in their own code can be.
From 2012 to 2018 Fabian A. Scherschel wrote daily as an editor for heise online and c’t, first in London in English and later in German from Hanover. Since 2019 he has been reporting on IT security, operating systems, open source software and video games as a freelance author and independent podcaster.
In an online dialogue between Christian Grobmeier from the Log4j project and the head of the Open Source Security Foundation (OpenSSF), Brian Behlendorf, the Federal Office for Information Security (BSI) asked on Tuesday what we can learn from the consequences of this vulnerability.
The software supply chain is at risk
Behlendorf from the OpenSSF, a project under the auspices of the Linux Foundation, emphasized that Log4Shell has shown above all how shaky the security structure of the “software supply chain” is. Because that’s exactly how open source developers should think now, says Behlendorf. What is often simply referred to as software dependencies or dependencies is an integral part of the end product supply chain. Developers can no longer just embed random code in their projects just because a library already exists for it; instead, they would have to think about where this software came from, whether it was really necessary and – ultimately – whether it was safe.
Grobmeier from the Log4j development team, who looks at the software dependencies from the other side, also sees major problems here. Currently, 34 percent of the daily downloads of Log4j versions are vulnerable to the CVE-2021-44228 vulnerability. A vulnerability that has now been patched for more than 15 months. Despite this, the processes involved in distributing the Java logging library – what Behlendorf calls the software supply chain – result in loads of other software still embedding vulnerable versions of Log4j. The nature of modern software development, in this case Java, makes it impossible to prevent other projects from downloading vulnerable versions of Log4j and using them in their own code.
Fundamental problem with open source development
This vulnerability in the supply chain is primarily a problem with open source software. But since, as Behlendorf points out, open-source software is in almost every product that uses some kind of software these days, the problem is one that affects society as a whole. According to Behlendorf, the US government understood the problem after Log4Shell at the latest. The White House is trying to better secure this part of the digital infrastructure. In the US, people are thinking about new laws that make software companies financially responsible for security gaps.
Behlendorf emphasizes that the OpenSSF is committed to ensuring that the end providers of software products in particular are held liable. This is the only way to prevent responsibility being shifted to the developers in open source projects. Log4j developer Grobmeier emphatically agreed to this condition: As an unpaid developer who works on Log4j in his spare time, he must immediately withdraw from development if he can be legally prosecuted for security gaps. The risk for the livelihood of his family is too great for him.
That’s probably how most volunteer open source developers will think. And companies that pay their developers to work on open source projects in order to give back to the community projects that are used in their own products will probably think twice about this commitment if suddenly claims for damages for any programming errors are in the room.
After audits, also pay the project developers
Behlendorf stressed that it was just as important to give open source programmers a helping hand. That is one of the tasks of the Open Source Security Foundation. The software that runs large parts of the public Internet must be subjected to rigorous security audits. But that alone is not enough, says Behlendorf. You not only have to pay the security researchers who carry out these audits, but also the developers who then plug the gaps found.
Here, too, Grobmeier from the Log4j team agreed: You can talk about hobby developers who already sacrifice a significant part of their free time to develop inglorious software that is used by half the world, but actually hardly anyone is interested because it is “boring”. and “not really cool” now also expect to become security experts. People like him simply don’t have time for that alongside their job and software development.
After the big Log4Shell panic, he watched a presentation at the Blackhat security conference, where a few years earlier there had been reports of exactly the kind of JNDI injection attacks that were fatal to Log4j, said Grobmeier. Up to this point he had not even known what the blackhat conference was. Nobody reported these findings to the Log4j team between this blackhat presentation and the discovery of the Log4Shell vulnerability. The disaster made it clear to him how important security is in software development. During the BSI discussion, he went to great lengths to encourage other open source developers to accept assistance such as that from OpenSSF with open arms.
Don’t think outside the box
Unfortunately, it was not clear at the BSI event what is being done in Germany to prevent or at least cushion the consequences of security gaps such as Log4Shell in the future. All participants agreed that the security of open source projects is extremely important as a fundamental part of the supply chain of almost all software products.
In online workshops like the one described here, the BSI makes an effort to bring together stakeholders from business, government and public life – but on Tuesday evening it became clear that many of those involved were not aware of the explosive nature of the topic for society as a whole. A society that unanimously – on the part of the economy, the government and also the press – repeatedly and loudly advocates accelerating digitization in almost all areas of public life should actually face security problems such as those examined here in a two-hour dialogue tackle with the highest priority.
Everything on the net without a net and false bottom
Anyone who sees cashless payment transactions as the only alternative, wants to make large parts of the traffic infrastructure dependent on digital control units, unconditionally connects critical infrastructure and wants to move the economy to the cloud by hook or by crook, should actually consider beforehand how the foundations for these plans can be secured can. Corresponding questions on our part were, however, repeatedly ignored in the discussion.
It’s good to know that the Log4j project now has its security issues under control. And it’s also encouraging to find out what plans the OpenSSF is pursuing to help out similar projects before the big bang causes the developers to cancel their Christmas holidays because the entire Internet is at a standstill.
But it seems as if no one has yet drawn the consequences for society as a whole for digitization from the Log4Shell vulnerability. If events with the participation of core developers of important open source projects under the aegis of the BSI already lack this security-first approach, it will hardly be granted to large parts of civil society.
(fab)
