Posted under: Heavy Research
In our first paper on ‘Building Security Into DevOps’, given the ‘newness’ of DevOps for most of our readers, we included a discussion on the foundational principles and how DevOps is meant to help tackle numerous problems common to software delivery. Please refer to that paper is you want more detailed background information. For our purposes here we will discuss just a few principles that directly relate to the integration of security teams and testing with DevOps principles. These concepts lay the foundations for addressing the questions we raised in the first section, and readers will need to understand these as we discuss security tooling and approaches in a DevOps environment.
DevOps and Security
Build Security In
It is a terrible truth, but wide use of application security techniques within the code development process is relatively new. Sure, the field of study is decades old, but application security was more often bolted on with network or application firewalls, not baked into the code itself. Security product vendors discovered that understanding application requests in order to detect and then block attacks is incredibly difficult to do outside the application. It is far more effective to fix vulnerable code and close off attack vectors when possible. Add-on tools are getting better – and some work inside the application context – but better to address the issues in the code when possible.
A central concept when building security in is ‘shift left’, or the idea that we integrate security testing earlier within the Software Development Lifecycle (SDLC) – the phases of which are typically listed left to right as design, development, testing, pre-production and production. Essentially we shift more resources away from production on the extreme right, and put more into design, testing and development phases. Born out of lean manufacturing, Kaizen and Deming’s principles, these ideas have been proven effective, but typically applied to the manufacture of physical goods. DevOps has promoted use in software development, demonstrating we can improve security at a lower cost by shifting security defect detection earlier in the process.
Automation is one of the keys to success for most firms we speak with, to the point that the engineering teams often equate DevOps and Automation as synonymous. The reality is the cultural and organizational changes that come with DevOps are equally important, it’s just that automation is sometimes the most quantifiable benefit.
Automation brings speed, consistency and efficiency to all parties involved. DevOps, like Agile, is geared towards doing less, better, and faster. Software releases occur more regularly, with less code change between them. Less work means better focus, more clarity of purpose with each release, resulting in fewer mistakes. It also means it’s easier to rollback in the event of mistakes. Automation helps people get their jobs done with less hands-on work, but as automation software does exactly the same things every time, consistency is the most conspicuous benefit.
The place where automation is first applied, where the benefits of automation are most pronounced, are the application build servers. Build servers (e.g.: Bamboo, Jenkins, ), commonly called Continuous Integration (CI) servers, automatically construct an application – and possibly the entire application stack – as code is changed. Once the application is built, these platforms may also launch QA and security tests, kicking back failed builds to the development team. Automation benefits other facets of software production, including reporting, metrics, quality assurance and release management, but security testing benefits are what we are focused on in this research.
On the outset this may not seem like much; calling security testing tools instead of manually running the tests. That perspective misses the fundamental benefits of automated security testing. Automation is how we ensure that each update to software includes security tests, ensuring consistency. Automation is how we help avoid mistakes and omissions common with repetitive and – let’s be totally transparent here – boring manual tasks. But most importantly, as security teams are typically outnumbered by developers at a ratio of 100 to one, automation is the key ingredient to scaling security coverage without having to scale security personnel headcount.
A key DevOps principle is to break down silos and have better cooperation between developers and supporting QA, IT, security and other teams. We have heard this idea so often that it sounds cliché, but the reality is few in software development actually made changes to implement this idea. Most DevOps centric firms are changing development team composition to include representatives from all disciplines; that means every team has someone who knows a little security and/or represents security interest, even on small teams. And for those that do, they realize the benefits not just of better communication, but true alignment of goals and incentives. Development in isolation is incentivized to write new features. Quality Assurance in isolation is incented to get code coverage for various tests. When everyone on a team is responsible for the successful release of new software, there is a change in priorities and changes to behavior.
This item remains a bit of a problem for many of the firms we have interviewed. The majority of firms we have spoken with are large in size, with hundreds of development teams located in different countries, some of which are third party (re: external) consultants. It is hard to get consistency across all of these teams, and even harder to get general participation. Managerial structure is set up so development managers manage developers, not IT personnel. The management tools for feature tracking, trouble-ticketing, resource allocation are geared towards a siloed structure. And many leading security tools are set to analyze and report defects to security professionals, not developers or IT personnel who resolve an issue. Progress is still measured in feature outputs and code coverage, and bonuses awarded accordingly.
The point here is this cultural change, and the great benefits derived, are not realized without some changes to the supporting systems and structures. This is a very hard adjustment, one where the various managers are all looking to implement policies as if they have full oversight, missing the point that they too need to adopt the ‘one team’ approach with their peers to effectively enact changes.
Security Practitioners and Application Security
Why security folks struggle with DevSecOps, and even application security in general, as they do not have backgrounds in software development. Most security practitioners come from a network security background, and many CISOs we speak with are more risk and compliance focused, so there is a general lack of understanding of software development. This lack of knowledge of development tooling and processes, along with common challenges developers are trying to overcome, means security teams seldom understand why automated build servers, central code repositories, containers, Agile and DevOps have caught fire and widely adopted in a very short time. Here we discuss some of the drivers for changes in development practices and the key areas security teams need to understand when trying to get a handle on application security.
- Knowledge of Process: We are not here to teach the nuances of development process, but we want to point out the reasons why processes change: Speed. Waterfall, Spiral, Prototype Evolutions, Extreme Programming, Agile and Agile with Scrum are all process variations made over the last 20 years. Each came with the same goals: Reduce complexity (i.e.: simplify requirements) and speed up software delivery. When you understand that the majority of changes to how we build software over the last 20 years is to address these two goals, you start to understand the process itself is not important; the goals of delivering better software, faster is. Daily scrums, bi-weekly software delivery (i.e.: Sprints), Kanban, Agile, test driven development and automated build servers are tools to advance the state of the art. So it is critical for security professionals to understand that security testing and policies should embrace these same ideals. And lastly, DevOps is process independant; you can embrace DevOps and still have a waterfall process, but certainly DevOps fits more naturally with Agile.
- Knowledge of Tools: Software development leverages many tools to manage code and process. Of these, the two most important to security are code repositories and code build tools. Repositories like Git essentially manage application code, giving developers a shared location to store code, track versions and changes. Others, like Docker Registry, are specifically used for containers. These tools are essential for developers to manage the code they are building, but also important to security as it provides a place where code can be inspected. Build servers like Jenkins and Bamboo automate the build, testing and delivery of code. But rather than at a component or module level, they are typically employed for full-app-stack testing. Developers and quality assurance teams use the build server to launch functional, regression and unit testing; security teams should also leverage these build servers to integrate security testing (.e.g.: SAST, DAST, Composition Analysis, security unit tests) so it fits within the same build process, and used all of the same management and communications tools. It is important for security teams to understand which tools the development team uses, and who controls those resources, and arrange the integration of security testing.
- Everything Is Code: Applications are software. This is fairly well understood, but what is not understood that in many environments – especially public cloud – that your servers, networks, messaging, IAM and every other bit of the infrastructure may be defined as configuration scripts, templates or application code. IT teams now define an entire data center with a template comprised of a few hundred lines of script. The idea for security practitioners is twofold: security policies can also be defined in scripts/code, and that you can examine code repositories to ensure the templates, scripts and code are secure before they are run. This is a fundamental change to how security audits may be conducted.
- Open Source: Open source software plays a huge part in application development, and is so universally embraced in the development community it is almost impossible to find a new application development project that does not leverage it. This means that a large portion of your code may not be tested in the way you think it is, or developers may intentionally use old, vulnerable versions. Why? Because the old version works with their code. If they change the library it may break the application and require more work. Developers are incented to get code working and we have witnessed heroic efforts on their part to avoid new (i.e.: patched) open source versions for the sake of stability. So we want you to come away with two points: You need to test open source code before it hits production, and you need to ensure that developers do not surreptitiously swap out trusted versions of open source libraries for older, probably vulnerable, versions.
- Tooling and Dev ‘Buy In’: The first step most security teams take when introducing security into application development is to run static analysis scans. The good part is that most security practitioners know what SAST is and what it does. The bad part is that security started with older SAST tools that are slow, produce output only intelligible by security folks, created ‘false positive’ alerts, and and they do not have the critical APIs needed to fully integrate with the rest of the build process. All told, their efforts were developer hostile, and most development teams reacted in kind by ignoring the scans or removing the tools from the build process. Two key aspects here: you want to select tools that operationally fit the development model (faster, easier, better), and use tools that are actually effective. Left to their own decision making process, developers will always choose the easiest tool to integrate, not the most effective security scanning tool. It is important that the security team is part of the security tool selection process to ensure security scans are providing adequate analysis.
- Security Friction & Cultural Dynamics: Most application security teams are playing catch-up. Development is (usually) already agile, and if some of your development organizations are embracing DevOps, it’s possible IT and QA are also agile. This means security folks are the non-agile anomaly; anything you do or ask for adds time and complexity, the antithesis of software engineering goals. This topic is so important that I have added the entire next section, ‘Scaling Security’, on methods to address the cultural friction between security and development.
- SDLC and S-SDLC: Many application security teams approach application security by looking at the Software Development LifeCycle (SDLC), with the goal of applying some form of security analysis in each phase of the lifecycle. A secure-SDLC (S-SDLC) will typically include threat modeling during design, composition analysis in development, static analysis during the build phase, and any number of tests pre-production. This is an excellent way to set up a process independant application security program. And as many large organizations come to understand, each of your development teams employ a slightly different process, and it is entirely possible your company uses every known development process in existence. This is a huge headache, but the S-SDLC becomes your yardstick: Use the S-SDLC as the policy template, and then map the security controls to the fit within the different processes.
As we mentioned in the introductory section, most security teams are vastly outnumbered. As an example, I spoke with three midsized firms this week; the development personnel ranged in number from 800-2000, while the security teams ranged in size from 12 to 25. Of the security personnel, they typically had two or three individuals with a background in application security. While they may be as rare as unicorns, that does not mean they have magical powers to cover all development operations, so they need to learn ways to scale their experience across the enterprise. Further, they need to do wit in a way that meshes with development ideals, getting the software development teams to perform the security controls they designs. Here are several methods that work.
- Automation: We have already discussed automation to some degree so I will keep it short here. Automation is how security analysis occurs faster, more often and without direct operation by security teams. Security tools that perform automated analysis, either out of the box or custom checks of your design, is critical in order to scale across multiple development teams. And yes, this means that for every build pipeline you have in your company you will have to get tools integrated with that pipeline, so this takes time. And this means that not only is the scan automated, but the distribution of the results is integrated with other tools/processes. This is how teams scale, and instrumental for the next two bullet items on our list.
- Failing the Build: Development and security teams commonly have friction between them. Security typically provides development managers security scan results with thousands of defects. Development managers interpret that as saying “Dude, your code sucks, what’s wrong with you, fix it now!” One of the ways to reduce the friction between the two groups is to take the output from static or dynamic scans, discuss the scope of the problem, what critical defects mean, and come to an agreement on what is reasonable to fix in the mid term. Once everyone is in agreement about what a critical issue is and in what time period is reasonable to fix it in, you instruct the security tools to fail a build when critical bugs are discovered. While this process takes some time to implement, and some pain to work through, it changes the nature of the relationship between security and development. No longer is it security saying the code is defective, it’s an unbiased tool reporting a defect to development. Security is no longer the bad guy standing in the way of progress, rather development is now having to meet a new quality standard, one focused on code quality as it relates to security defects. It also changes the nature of the relationship as developers often need assistance in understanding the nature of the defect, look for ways to tackle classes of defects instead of individual bugs, and developers go to security for assistance. Failing the build creates a sea-change in the relationship between the two groups. It takes time to get there, and any security tooling the produces false positives magnifies the difficulty in getting this change in place, but this step is critical for DevOps teams.
- Metrics: Metrics are critical to understand the scope of application security issues present and security tools are how you will collect most metrics. Even of you do not fail the build, and even if the results are not shared with anyone outside security, integrating security testing into build servers and code repositories is critical to gaining visibility and metrics. These metrics will help you decide where to spend your budget, be it additional tooling, developer education, or runtime protection. And these metrics will be your guide to the effectiveness of the tools, education and runtime protection you implement. Without metrics, you’re just guessing.
- Security Champions: One of the most effective methods I have discovered to scale security is to deputize willing developers – those that have an active interest in security – to be a ‘security champion’ on their development team. Most developers have some interest in security, and they know security education makes them more valuable to the company, and that often means raises. For security, this means you have a liaison on a security team, one you can ask questions of and one who will come to you if they have questions. Typically security teams cultivate these relationships through education, having a ‘center of excellence’ where developers and security pros can ask questions (i.e.: Slack channels), sending the developers to security conferences, or simply sponsoring events like lunches where security topics are discussed. Regardless of how you do it, this is an excellent way to scale security without scaling security headcount, and we recommend you set aside some budget and resources as it returns far more benefits than it costs.
- Education: If you want to have developers understand security and threats to applications, educate them. Engineering leads and VPs of Engineering, because of the expense on headcount, are usually under tight restrictions for educational budgets. To fill the gaps, it is not uncommon for security teams to shoulder the expense of educating select developers on skills the organization is lacking. Sometimes it is done through the purchase of security related CBTs, sometimes it is the purchase of professional services provided by security tool vendors, and sometimes it is specific classes from SANS or other institutes. Understanding how to remediate application security issues, security reference architectures, how to perform threat modeling, and how to use security tools are all common.
This series is going to be a bit longer than most. Over the last few years we have acquired considerable research. And while we will attempt to be concise, there is simply a lot of material to cover in order to address the questions from section one.
Next up I will discuss putting together a secure SDLC, and the integration of security testing in the development process.