The software supply chain is a serious concept and one that demands some attention. In recent years, it is at the heart of the Rugged DevOps initiative supported by the principles and thought leadership from Deming. And while stumbling through Deming principles and discussing the Software Supply Chain with Rugged DevOps leaders, mostly Josh Corman, a main concept became really controversial - the notion of embracing Fewer, Better Suppliers. This post captures an exploration of this concept and some lessons learned. Let’s start with a little bit of history, a couple examples about why this concept is important and then discuss what we learned from a recent experiment.
15 years ago, technology was very different. The dot-com bubble busted. Soon thereafter, Web 2.0 was born with the intention to increase user-generated content, interoperability, and user friendly experiences. And then in 2001, 17 men sat around a table declaring their dislike for slow Software Development practices and gave rise to the Agile Manifesto. In 2004, Cloud computing emerged to support DevOps, and skip forward 12 years - here we are, Rugged DevOps.
Today, technology mostly requires re-use of components and software defined environments. Open source and commercial products are commonly incorporated into larger packaged software, incorporated into micro-services, and bundled to meet customer demands. Now integration of components is common across the technology industry to support customer demands at speed and scale. Yet - at the same time, so many more workloads are unknowingly falling prey to a greater number of security defects. Software professionals are without the resources they need to easily evaluate and determine whether the libraries and components they have chosen can be hardened.
So why is this? Imagine for a moment that your job is to develop a web site and use a common web framework along with Open Source components to achieve innovating against customer demands quickly. And in this framework, there are a few common components which have been around for decades achieving a stable status. The DevOps team uses these components knowing full well that the community is aggressive and works hard to build good software components. But hidden further in the framework are other libraries that these projects rely on. Quickly you begin to realize that the complexity of inheritance driven software creates complexity well beyond what most developers can research themselves during the design and build phase of Continuous Software Delivery. Here are a couple examples of highly complex components that are widely used within many higher level frameworks. These examples will give you some idea of why these issues exist and why they are difficult to eradicate.
In 2000, ASN.1 became an interesting framework with all of two books written and very few experts. Yet, it was commonly used and still is within a variety of projects. ASN.1 was interesting and has some sequence recursion problems that are hard to implement securely. More importantly, the main libraries for it are used at a low level within a variety of software components. The University of OULU Security Programming group uncovered several issues within the framework and the SNMP implementation of it in the early 2000’s publishing their work and later more issues were uncovered. ASN.1 has been at the heart of well over a decade and a half of serious vulnerabilities that allow for anything from buffer overflows to remote code execution in many platforms (Windows, Linux, etc.) and libraries such as OpenSSL.
In 2010, Struts2 framework vulnerability was made famous at the Pwnie awards in Las Vegas. It was CVE-2010-1870. It was very serious, allowing attackers to gain remote access to an environment with a single HTTP request and 5 special parameters set. Ouch! It was difficult to test and many companies had included it in their software and services. Worse, it was complicated and the release voting process took far too long. During that year, many fell victim and there were a variety of patch notifications that resulted from the inclusion of this framework in other software and services. Big software platforms had been vulnerable for a long period of time because patches were delayed. CVE-2010-1870 is still showing up in large environments.
Earlier this year, I had a chance to really think about this problem at great length and believe that we must find some means of achieving Fewer, Better Suppliers while still balancing the demands of developing Awesome, Innovative Software at Speed and Scale. While talking with Josh Corman, we discovered that DevSecOps work has been squarely pitted between these two outcomes. But more importantly, Sonatype has developed a tool to help the community understand the complexity of the inherited security defects being included in the software supply chain by most organizations. It provides a visual depiction of how software security defects get embedded deep into the software deployed by most organizations.
As a security professional, I love this tool! As a DevSecOps professional, I want to figure out how to use it to help DevOps teams make better decisions and know what to fix, when. As you can imagine, I’m super excited to find someone to help me understand how to implement this tool in my DevSecOps environment. And well, who doesn’t love to bang their head against the wall? I’ve been looking around for people who have attempted to choose a Fewer, Better Suppliers path and well wouldn’t you know it I haven’t found many embarking on this journey. So in the true spirit of understanding what its going to take to achieve this nirvana, we decided to run an experiment. And as you might have guessed, we’ve learned a few things and wanted to share the journey.
Experiment: Determine if we can persuade software developers to use fewer components
We worked with several teams to discuss using fewer components that they have to manage and maintain. The feedback was unanimous. We learned that DevOps teams want to use the tools they know already work. And they weren’t willing to give up any of these tools without replacements. But we did gain an insight that made us hopeful. Teams were interested in making sure they could stay ahead of the Red Team or worse External Attackers and were willing to use a tool to discover and patch their tools faster. In fact, with high fidelity remediation requests and virtual patching options, these teams were very interested in helping to ensure the environment they support is hardened and checks built into the Continuous Delivery pipeline. More importantly, these teams were very interested in increasing their visibility into hidden security defects within their software stacks.
This was very interesting feedback, because we weren’t sure we would get much traction on reducing the number of suppliers, which we didn’t. But we did learn that we could indeed gain traction with DevOps teams by increasing visibility so that defect resolution could be faster and planned within their release cycle.