Injecting Security into a Devops pipeline | DevSecOps
Since forever, security researchers emphasized on the importance of integrating security from the early stages of development. In accordance, a system is considered secure if it was built with security in mind.
We're seeing companies adopt devops without prior knowledge. Which resulted in developers and managers looking at it as magic. You push your code to source control and Voila! you have it in production. Such approach creates a gap between teams and makes software engineers understand less their code and how it operates.
In this Netflix talk, Dave Hahn talks about how they don't do devops in their company, they look more at their culture and leaning toward the "you built it, you run it" approach.
Devsecops tries to bridge that gap indirectly. By making security everyone's job you force your different departments to act as a single unit and make a developer see the impact of his code in production.
Your primary focus should reside in protecting the end user. Ensure confidentiality of their information, keeping it intact and always available.
I've drawn a typical CI/CD chain with security implementation to help you better visualize the final ecosystem.
These are not the best tools for every context. For the sake of this guide, i used the ones i like or have experimented with.
We'll discuss every stage of the application's life cycle and introduce security in it.
With every java release oracle publishes a security's developer guide. This guide helps you to understand the Java security technology, tools, and implementations of commonly used security algorithms, mechanisms, and protocols on the Java Platform.
But how do you ensure developers are following these guides? Here we introduce the software composition analysis.
Let's say you're working on a spring boot project and using maven to manage it. The dependencies you're using are written by a company or an individual and they are free of charge. Therefore no one can blame the author for not testing their code base or updating it with every new technology breach out there.
As a result, SCA was introduced with the exponential growth of opensource, allowing developers to understand their open-source inventory and if they represent any threat to the security of their product. We'll look at two approaches.
Maven dependency check: A maven plugin introduced by the Open Web Application Security Project [OWASP] to help discover known vulnerabilities in your dependencies. It is easy to integrate and can have a significant impact on the final product.
For instance, we can take the example of Equifax, a company that had failed to patch the Apache Struts vulnerability (CVE-2017-5638) which exposed data of over 140 million US consumers and forced the company to pay a fee of 700 million US dollars. One thing to keep in mind is that you can tolerate some vulnerabilities depending on the context of your project. For instance, if a vulnerability affects windows users and you're running your web application in a linux machine, then you can tolerate it.
Pre-commit Hooks: When using git, you can benefit from the hooks feature. Git hooks are scripts that Git executes before or after events such as: commit, push, and receive. Git hooks are a built-in feature and you don't need to download anything. you can find them in your .git/ folder of any project. There are tools like findsecbugs or scripts made by the community that can help you enforce code quality standards on yourself or your team through these hooks.
Here you define your workflow, do you have functional tests? do you deploy to staging environments? What is your definition of done?
There are many areas to cover in this stage, we'll only focus on security for the sake of simplicity.
When testing a web application for example, generally there are two approaches.
Static analysis: In this stage we use tools like sonarqube for code reviews . To go through our source code and detect some patterns for know vulnerabilities. Things like hard-coded passwords, sql injections.. Or simply help our pentesting team to look for severe vulnerabilities more efficiently.
Sonarqube helps you enforce a quality gate for your project and uses a variety of tools to perform tests and provide feedback on how you can optimize your code. You can even use it from your IDE as a developer and take your development game to the next level.
Dynamic analysis: This is the "be mean to your application stage". We run our application in a sandbox environment and perform penetration testing through tools such as OWASP ZAP jenkins plugin.
Or just call in the pentesting team if you can afford the down time.
Doing this stage is time extensive and depending on your business and how much time you can tolerate. You can do things like running them behind the scenes and letting the pipeline finishes. Maybe do these tests once a day or once a week.
We use a base image to build our container, ranging from debian, to ubuntu to nginx. Let's see a graph of known vulnerabilities in some of the most popular base images.
Hardening a docker image is a tedious job because they are weak in the security department due to docker's design. A container can provide root access to the host system to an attacker with too much ease. From there they can take down your entire infrastructure or access another container running your database.
Like we checked our dependencies for know vulnerabilities we can do the same with our docker images. Using tools like Clair or Anchore Engine in our pipelines.
Orchestration systems such as Kubernetes provide the means with which you can protect your containers through abstractions or sandboxing, forcing networking rules or maybe using a selinux right from your deployments. It's up to you to decide what best suits your needs.
The kubernetes team provides great insight on how you can secure your cluster and the deployments on top of it.
In this article i used kube-bench which Checks whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.
Combined with Hashicorp's vault to store kubernetes secrets in a secure manner. You can use it for things like mTLS and passwords rotations..
For example we can check the health of our nodes, as illustrated below.
The Prometheus operator integrates seamlessly with our cluster. Providing metrics without making the application aware of its existence. Right off the shelf, you have access to all of these metrics below
Yet if your application exports Prometheus metrics it would be even much better and insightful.
Another great tool is Weave Scope by weaveworks. IT automatically detects processes, containers, hosts. Written in bible.js this tool is a work of art.
We're seeing companies adopt devops without prior knowledge. Which resulted in developers and managers looking at it as magic. You push your code to source control and Voila! you have it in production. Such approach creates a gap between teams and makes software engineers understand less their code and how it operates.
In this Netflix talk, Dave Hahn talks about how they don't do devops in their company, they look more at their culture and leaning toward the "you built it, you run it" approach.
Devsecops tries to bridge that gap indirectly. By making security everyone's job you force your different departments to act as a single unit and make a developer see the impact of his code in production.
Security is not hacking!
A statement i needed to make before discussing anything. Hacking is an art that requires a deep understanding of the target system and a wild imagination. A field i respect and admire, yet it represents a needle in a haystack. As a post action, it should be used as a last line of defense.Your primary focus should reside in protecting the end user. Ensure confidentiality of their information, keeping it intact and always available.
I've drawn a typical CI/CD chain with security implementation to help you better visualize the final ecosystem.
These are not the best tools for every context. For the sake of this guide, i used the ones i like or have experimented with.
We'll discuss every stage of the application's life cycle and introduce security in it.
Plan
The best way to introduce security in the early stages of development is by planning it as a functional requirement. Do you use UML to model your application? take a look at UMLsec. Which is an extension to the unified modeling language.Develop
Once we figured out the architecture of our product and divided tasks between developers here we introduce security best practices for developers. We'll take JAVA as a reference language for the rest of this guide. You can apply any of the strategies discussed here in any programming language.With every java release oracle publishes a security's developer guide. This guide helps you to understand the Java security technology, tools, and implementations of commonly used security algorithms, mechanisms, and protocols on the Java Platform.
But how do you ensure developers are following these guides? Here we introduce the software composition analysis.
Let's say you're working on a spring boot project and using maven to manage it. The dependencies you're using are written by a company or an individual and they are free of charge. Therefore no one can blame the author for not testing their code base or updating it with every new technology breach out there.
As a result, SCA was introduced with the exponential growth of opensource, allowing developers to understand their open-source inventory and if they represent any threat to the security of their product. We'll look at two approaches.
Maven dependency check: A maven plugin introduced by the Open Web Application Security Project [OWASP] to help discover known vulnerabilities in your dependencies. It is easy to integrate and can have a significant impact on the final product.
 |
| Source |
For instance, we can take the example of Equifax, a company that had failed to patch the Apache Struts vulnerability (CVE-2017-5638) which exposed data of over 140 million US consumers and forced the company to pay a fee of 700 million US dollars. One thing to keep in mind is that you can tolerate some vulnerabilities depending on the context of your project. For instance, if a vulnerability affects windows users and you're running your web application in a linux machine, then you can tolerate it.
Pre-commit Hooks: When using git, you can benefit from the hooks feature. Git hooks are scripts that Git executes before or after events such as: commit, push, and receive. Git hooks are a built-in feature and you don't need to download anything. you can find them in your .git/ folder of any project. There are tools like findsecbugs or scripts made by the community that can help you enforce code quality standards on yourself or your team through these hooks.
Test
It's a great idea to get developers to optimize their code, yet they can bypass these standards and decide to do things the easy way. Here comes the job of a Continuous integration tool. Using gitlab runners, jenkins, bamboo or any other tool, you can enforce your company's standards and provide a transparent view to what everyone is doing.Here you define your workflow, do you have functional tests? do you deploy to staging environments? What is your definition of done?
There are many areas to cover in this stage, we'll only focus on security for the sake of simplicity.
When testing a web application for example, generally there are two approaches.
Static analysis: In this stage we use tools like sonarqube for code reviews . To go through our source code and detect some patterns for know vulnerabilities. Things like hard-coded passwords, sql injections.. Or simply help our pentesting team to look for severe vulnerabilities more efficiently.
Sonarqube helps you enforce a quality gate for your project and uses a variety of tools to perform tests and provide feedback on how you can optimize your code. You can even use it from your IDE as a developer and take your development game to the next level.
 |
| Source |
 |
| Source |
Or just call in the pentesting team if you can afford the down time.
Doing this stage is time extensive and depending on your business and how much time you can tolerate. You can do things like running them behind the scenes and letting the pipeline finishes. Maybe do these tests once a day or once a week.
Build
Now that we finished doing our tests we're going to package our application. Docker is becoming the de-facto in the upcoming years so we'll look what we can do here from a security perspective.We use a base image to build our container, ranging from debian, to ubuntu to nginx. Let's see a graph of known vulnerabilities in some of the most popular base images.
 |
| Source: Synk |
Hardening a docker image is a tedious job because they are weak in the security department due to docker's design. A container can provide root access to the host system to an attacker with too much ease. From there they can take down your entire infrastructure or access another container running your database.
Like we checked our dependencies for know vulnerabilities we can do the same with our docker images. Using tools like Clair or Anchore Engine in our pipelines.
Orchestration systems such as Kubernetes provide the means with which you can protect your containers through abstractions or sandboxing, forcing networking rules or maybe using a selinux right from your deployments. It's up to you to decide what best suits your needs.
Deploy
Kubernetes is no perfect system. Up to today, 22 vulnerabilities were discovered. Since kubernetes treats our compute resources as a single unit, controlling its API or etcd can result in taking control of our entire cluster. |
| Source |
The kubernetes team provides great insight on how you can secure your cluster and the deployments on top of it.
In this article i used kube-bench which Checks whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.
Combined with Hashicorp's vault to store kubernetes secrets in a secure manner. You can use it for things like mTLS and passwords rotations..
Monitor
Once the application is tested and deployed, we need to continuously monitor it. We can integrate tools such as prometheus and grafana to provide visual metrics of what is happening in our cluster.For example we can check the health of our nodes, as illustrated below.
The Prometheus operator integrates seamlessly with our cluster. Providing metrics without making the application aware of its existence. Right off the shelf, you have access to all of these metrics below
Yet if your application exports Prometheus metrics it would be even much better and insightful.
Another great tool is Weave Scope by weaveworks. IT automatically detects processes, containers, hosts. Written in bible.js this tool is a work of art.
Comments
Post a Comment