It’s sprint 9 out of the planned 10 for your next major website release. Your web development team is sprinting to launch the new version of your site with an early September deadline. You want to have the site live and running for at least two months before Black Friday and the big shopping season to shake out any bugs. Rather than code out all the functionality your site needs from scratch, the dev team leverages a wide variety of JavaScript libraries that it finds on GitHub. “Are all the libraries tested and validated?” you ask the team. They assure you that they all checked out.
A few weeks after launch you get a notice from an information security research team that they have spotted a credit card skimmer on your site. However, you don’t recognize the JavaScript library that the security researchers say has been compromised; it wasn’t on the final list you got from your development team. It turns out one of the teams made a last minute switch because the library they had originally chosen lacked some key capabilities they needed.
Now you have a giant mess on your hands with angry customers and a massive code audit for the entire site looming. You were a victim of “‘Shadow Code.” And you are not alone.
The Rise of Shadow Code
Shadow Code is the software development equivalent of Shadow IT. In Shadow IT, employees use cloud services and software that is not approved, monitored or supported by IT. Shadow IT can range from unauthorized SaaS subscriptions to AWS cloud servers spun up on a credit card without oversight or compliance.
Shadow Code is when developers include in applications third-party software code sometimes without approval or any safety validation. They do this because it is usually quicker than writing the functionality from scratch. The particular piece of Shadow Code may solve a problem more precisely or it may appear to be a better fit with other application components. Some examples of this can be shopping cart software, payment systems or responsive design requirements. Sometimes, too, developers mistakenly include shadow code that is fake but has a similar name (this is referred to as typosquatting).
In truth, Shadow Code has been around since the dawn of software. Most code is useful in more than one application, so naturally the reuse and copying of code has been a long accepted practice. What has recently changed is the volume of third-party code in use and the increasingly critical function that these third-party code libraries play in applications. Front-end applications, like websites, mostly consist of third-party code - some more than 70%. The primary form of third-party code used today are JavaScript libraries, although third-party code is used in all other modern languages including Ruby, Python and Golang. For websites, developers use third-party JavaScript libraries for almost all sensitive jobs including processing payment information, customer log-ins and more. Numerous SaaS providers (including HUMAN) use JavaScript to extend application functionality out to customer websites. A big portion of JavaScript library usage, however, is for more mundane tasks like form validation, providing the correct date and time, and font delivery.
These third-party libraries and tools can make up a considerable percentage of middleware code that connects the front-end to the back-end databases and core application servers. Package managers allow websites to download multiple libraries or packages regularly and keep them all maintained via a centralized service.
Inviting Strangers Into Your House
The trouble with all of these third-party libraries and tools is that using them is akin to inviting strangers into your house. In fact, you may have a party where you only know one third of the guests. They are in your house, and you hope they don’t steal from you or break things.
Some guests we know are likely to be very well behaved. For example, Facebook creates and maintains the popular React JavaScript library and framework used to build web applications. Because it has a large community and is backed by Facebook, you can be fairly certain it is secure and well-maintained.
With other guests it may be harder to know for sure. There are hundreds of popular JavaScript libraries delivering useful functionality that are rarely updated and have maintainers who are individuals that may be hard to reach in case of a breach. The library may work quite well but it presents a real security risk. In other instances, multiple maintainers have privileges over the software code of a library and the version changes are not tracked closely. That can allow subtle changes to slip in unnoticed that transform an otherwise upstanding library into a malicious one.
Then there are libraries that, while maintained, are not secured to your standards. There have been a number of instances where a frequently-maintained but insecure third-party service was hacked. Large brands using this library were compromised as a result, exposing their customers to real risks. With third-party services, you are only as strong as your weakest link.
The bottom line of Shadow Code? As the website owner, you are fully liable for any breach or security issue that unauthorized or poorly chosen libraries may introduce, maliciously or unintentionally, to your customers and users.
Fixing Shadow Code
Just like Shadow IT, heavy-handed enforcement will upset your developers. Keep in mind that they are using Shadow Code to improve their speed and productivity, not out of any malicious intent. Here are some basic steps:
- Set up quick approval and gate processes for any library or external code you plan to incorporate. This is designed to make sure it checks basic boxes around security and maintainer visibility and responsiveness. If you see unanswered issues sitting for months in the project’s GitHub repository, you know things are not good.
- Run all third-party libraries and code through detection and verification tools that examine the code for signs of unauthorized modification. There are excellent services that do this and there are also open source tools that assign a unique identifier (a hash number) to the contents of a library which will change if the code is modified. Browsers can ask for the hash and verify it's the right one before loading the code.
- Get acquainted with the concept of Content Security Policy (CSP) and how it can help build a better security posture for your website. A good CSP can prevent common attacks launched from compromised third-party Javascript libraries such as cross-site scripting (XSS), clickjacking and other code injection attacks. True, CSP management is manual and time-consuming but you can mitigate many of the risks of Shadow Code with some simple CSP rules.
- Make sure you have clear monitoring and alerting capabilities in runtime, where the code is actually executing. The technology should be able to, for example, detect if a new form was added to your website or if a new “buy” button is appearing for a shopping cart that wasn’t there before - all signs that you have been a victim of Magecart attacks and other compromises of your third-party code. This monitoring must work both in staging and production environments. Ideally, you want to catch problems in staging before they go live.
A Major Shift
Fixing Shadow Code requires a major shift in how we develop software. Ignoring the problem is not an option; compliance rules are getting tighter, legal and business risk growing greater. Fortunately, the fix does not have to be painful. Companies that adopt modern DevOps practices can easily enforce review of any new libraries as a mandatory step in CI/CD pipelines. In addition, there are developer tools that can monitor libraries and third-party code entering the pipeline and ensuring those tools pass basic security checks, and track code changes to those libraries over time. Lastly, there are now a new generation of machine learning tools that can study the behaviors of your code in the staging or production environments and note any changes in how the site loads content, forms or pages to identify anomalies that may be introduced through Shadow Code.
The reality is, every website owner is dealing with some Shadow Code. Development teams are focused on solving problems and sometimes the easiest and fastest way to do that is to go outside approved channels to get things done quickly. Unfortunately, this may be putting their business and their customers at risk. It’s up to website owners to put in place the processes that prevent Shadow Code and enable their development teams to get third-party code approved quickly and cleanly as part of the normal DevOps cycle. This is the best way to preserve the agility and value of leveraging all the useful code already in existence without exposing your business and your customers to unnecessary risks.
