Posted under: Heavy Research
As mentioned in an earlier section, DevOps is not all about tools and technology, but much of its success is how people work within this model. We have already gone into great detail about tools and process, and we approached much of the content from the perspective of security practitioners getting onboard with Devops. And since this paper is more geared towards helping security folks, here we outline their role in a DevOps environment. We hope this summation assists you in working with these other teams and reducing friction.
And while we have intentionally called this research paper Enterprise DevSecOps, keep in mind your development and IT counterparts think there is no such thing. To them security becomes part of the operational process of integrating and delivering code. We call security out as a separate thing because, while woven into the DevOps framework, it’s substantially more difficult for security personnel to fit in. You need to look at how you can improve delivery of secure code without waste and without introducing bottlenecks in a development process you may not be intimately familiar with. The good news is that security fits nicely within a DevOps model, but you need to tailor things to work within the automation and orchestration employed by your organization to be successful.
The Security Pro’s Responsibilities
Learn the DevOps model: We have not even touched the theory and practice of DevOps in this paper. There is a lot for you to learn on base concepts and practices. To work in a DevOps environment you need to understand what it is and how it works. You need to understand cultural and philosophical changes, and how they effect process, tooling and priorities. You will need to understand your companies approach to best integrate security tooling and metrics. Once you understand the mechanics of the development team, you’ll have a better idea of how to work with them, in context of their process.
Learn how to be agile: Your participation in a DevOps team means you need to fit into DevOps — not the other way around. The goal of DevOps is fast, faster, fastest: small iterative changes that offer quick feedback, ultimately reducing Work In Progress (WIP). Small, iterative changes to improve security fit this model. You will prioritize things that make the delivery of secure software over the delivery of new features, and that is often a huge undertaking to change a longstanding culture of ‘feature first’. You need to adjust requirements and recommendations so they fit into the process, often simplified into small steps, with enough information for the tasks to be both automated and monitored. You can recommend manual code reviews or fuzz testing, so long as you understand where they fit within the process, and what can — and cannot — gate releases.
Educate: Our experience shows that one of the best ways to bring a development team up to speed in security is training: in-house explanations or demonstrations, third-party experts to help with application security or threat modeling, eLearning, or various commercial courses. The historical downside has been cost, with many classes costing thousands of dollars. You’ll need to evaluate how best to use your resources — the answer typically includes some eLearning for all employees, and select people attending classes and then teaching peers. On-site experts can be expensive, but an entire group can participate in training.
Grow your own support: There is simply no way for security teams to scale out their knowledge without help. This does not mean hundreds of new security employees, it means deputizing developers to help with product security. Security teams are typically small and outnumbered by developers 100:1. What’s more, security people are not present at most development meetings, so they lack visibility in day-to-day agile activities. To help extend the reach of the security team, see if you can get someone on each development team – what is called a ‘security champion’ – to act as a security advocate. This helps not only extend the security team’s reach, but also expand security awareness.
Help DevOps team understand threats: Most developers don’t fully grasp how attackers approach attacking a system, or what it means when a code or SQL injection attack is possible. The depth and breadth of security threats is outside their experience, and most firms do not teach threat modeling. The OWASP Top Ten is a good guide to the types of code deficiencies that plague development teams, but you should map these threats back to real-world examples, show the
damage that a SQL injection attack can cause, and explain how a Heartbleed type vulnerability can completely expose customer credentials. Think of these real-world use cases as ‘shock and awe’, which goes a long way to help developers
Advise on remediation practices: Your security program is inadequate if it simply says to “encrypt data” or “install WAF”. All too often, developers and IT have a singular idea of what constitutes security, centered on a single tool they want to set and forget. Help build out the elements of the security program, including both in-code enhancements and supporting tools. Teach how those each help to address specific threats, and offer help with deployment and policy setup. In the past, firms used to produce code style guides to teach younger developers what properly written code looks like. Typically these are not on line. Consider a wiki page for security coding practices and other reference materials that are easily discovered and easily readable by people who do not have a security background.
Help evaluate security tools: It is unusual for people outside security to fully understand what security tools do, or how they work. So you can help in two ways; first, help developers select tools. Misconceptions are rampant, and not just because vendors over-promise capabilities. Additionally it is uncommon for developers to evaluate code scanners, activity monitors, or even patch management systems effectiveness. In your role as advisor it is your responsibility to help DevOps understand what the tools can provide and what fits within your testing framework. Sure, you might not be able to evaluate the quality of the API, but you can tell when a product fails to deliver meaningful results. Second, you should help position the expenditure as its not always clear to the people holding the purse strings how specific tools address security and compliance requirements. You should specify functional and reporting requirements for the tool that meet the business needs.
Help with priorities: During our research we had many security pros tell us that all vulnerabilities started looking like high priorities, and it was incredibly difficult to differentiate a vulnerability that had impact on the organization and one that did not. The field of exposure analysis is outside the skill set of developers. You need to help fill this gap because not every vulnerability poses real risk. And security folks have a long history of
sounding like the terrorism threat scale, with vague warnings about “severe risk” and “high threat levels”. None of these warnings are valuable without mapping a threat to possible exploitations, what the exploit might mean to the business, or what you can do to address — and reduce — risks. For example you might be able to remediate a critical application vulnerability in code, patch supporting systems, disable the feature if it’s not critical, block with IDS or firewalls, or even filter with WAF or RASP technologies. Or cases where code exploitation cannot actually harm the business, so ‘do nothing’ is the right answer. Rather than the knee-jerk “OMG! Fix it now!” reaction we have historically seen, there are typically several options to address a vulnerability, so presenting tradeoffs to a DevOps team allows them to select the best fit.
Write tests: DevOps has placed some operations and release management personnel in the uncomfortable position of having to learn to script, code, and place their work open for public review. It pushes people outside their comfort zones in the short term, but that is a key part of building a cohesive team in the medium term. It is perfectly acceptable for security folks to contribute tests to the team: scans that validate certificates, checks for known SQL
injection attacks, open source tools for locating vulnerabilities, and so on. If you’re worried about it, help out and integrate unit and regression tests. Integrate and ingratiate! To participate in a DevOps team, where automation plays a key part, it’s likely you will need to know how to write scripts or templates. The good news is that your policies are now embodied into the definition of the environment. Don’t be afraid that you don’t know source code control or the right format for the scripts; this is an area where developers are usually keen to assist. Learning a bit on the scripting side before your tests can be integrated into the build and deployment servers, but you’ll do more than preach security — you can contribute!