In the age of cloud and serverless computing, microservices, and domain-driven design, there are many buzzwords beyond those just mentioned that one should definitely be familiar with. One extremely important aspect of software development today is DevOps.
DevOps stands for the integration of software development (Dev) and operations (Ops) into a closed process, where the entire development lifecycle is kept as short and efficient as possible. DevOps represents a methodology where software development can be developed and operated holistically and agilely. Due to its widespread popularity, DevOps has become a leading paradigm for technology companies and IT departments in recent years.
Without going into a detailed explanation of DevOps as a methodology, I would like to address a specific further development: DevSecOps. Since DevOps only focuses on the integration of the development and operation process, an extremely important aspect of any software development is lost: the security aspect.
DevSecOps as an Extension of DevOps
DevSecOps extends the DevOps methodology by incorporating the security aspect, thus better addressing the requirements of modern software development. By integrating security-relevant requirements into the entire lifecycle of the process, these requirements are also captured and considered.
The Cloud Native Glossary has its own entry on DevSecOps and defines the term as follows:
“The term DevSecOps refers to a cultural merger of the development, operational, and security responsibilities. It extends the DevOps approach to include security priorities with minimal to no disruption in the developer and operational workflow. Like DevOps, DevSecOps is a cultural shift, pushed by the technologies adopted, with unique adoption methods.”
According to the CNCF, DevOps as well as DevSecOps are not just methodologies, but also a complete cultural shift. It talks about the responsibilities in terms of development, operations, and security. Security is another component and a priority that guarantees minimal to no interruptions in the development and operational phases.
Snyk, a well-known cloud-native player with a product range specializing in security, also defines DevSecOps in its own way. Here is their definition:
“The definition of DevSecOps Model, at a high-functioning level, is to integrate security objectives as early as possible in the lifecycle of software development. While security is “everyone’s responsibility,” DevOps teams are uniquely positioned at the intersection of development and operations, empowered to apply security in both breadth and depth.”
Snyk emphasizes the importance of integrating security into the software development lifecycle from the very beginning so that this aspect can be supported by all stakeholders. Both this definition and the one mentioned earlier make it clear that DevSecOps is about considering security as an essential part of all considerations throughout the entire process. The responsibility and implementation of security-relevant points thus lies with the entire Dev(Sec)Ops team and is a core competency in all relevant activities.
Here is a Venn diagram, on the left the DevOps process without security and on the right the same process with the security aspect. This serves as a simple visualization of what the “Sec” in DevSecOps stands for.
Figure: Source: Snyk
Benefits of DevSecOps
Why would anyone want to do DevSecOps at all? In case the benefits of a security-aware DevOps process are not intuitively clear, I would like to briefly list the points here and explain why it makes sense to consider DevSecOps as a full-fledged part of DevOps.
Figure: Source: Snyk
- Faster software delivery: The speed of software delivery is improved when security is integrated into the development process. Bugs are identified and fixed before deployment, allowing developers to focus on functionality.
- Improved security focus: Security is an important aspect from the very beginning. A shared responsibility model ensures that security is tightly integrated - from design and development to securing production workloads.
- Cost optimization: By identifying vulnerabilities and bugs before deployment, risk and operating costs are reduced exponentially.
- Added value for DevOps: Improving the overall security posture as part of a shared responsibility culture is achieved by integrating security practices into DevOps. According to the Snyk/Puppet 2020 DevSecOps Insights Report, this is true for established DevSecOps organizations.
- Improved integration and speed of security measures: The cost and time required to deliver secure software is reduced because security controls do not have to be implemented retrospectively.
- Overall economic success: Increased confidence in the security of the developed software and the acceptance of new technologies enable improved revenue growth and expanded business offerings.
The DevSecOps Flow
Figure: Source: Snyk
Shift-left and shift-right
Much has been written about the benefits of performing security assessments early in the software development lifecycle ("Shift Left") before vulnerabilities find their way into production.
However, DevSecOps must also extend to production environments ("Shift Right") for four reasons:
- Most attacks occur in production.
- Scanning source code cannot provide the same comprehensive insights as observing the application while it is running in production.
- Some applications running in production may not have gone through the development environment, so they never had a chance to be scanned by security tools in your development environment.
- To detect new zero-day vulnerabilities, existing applications in your production environment need to be monitored.
In the Age After Log4j
Thoughtworks, one of the most well-known technology consultancies and inventor of the Technology Radar format, from which we previously drew a deviation from CNCF as a reference, regularly shares its opinions on general trends and market developments. In an article titled “Macro trends in the tech industry | March 2022” from early 2022, Thoughtworks dedicated an entire paragraph to the question of how the tech world will deal with open-source projects after the zero-day vulnerability Log4j.
After explaining how open source, especially GNU/Linux, has established itself in the IT world and now accounts for 90% of cloud servers, as well as how widespread open-source software is in various areas of software development, Log4j made it clear that the aspect of the origin and quality as well as development and maintenance of an open-source project is just as relevant to corporate security as potentially homemade security vulnerabilities.
The article also quotes Steve Marquess, whose quote I would like to mention here.
“It takes nerves of steel to work for many years on hundreds of thousands of lines of very complex code, with every line of code you touch visible to the world, knowing that code is used by banks, firewalls, weapons systems, web sites, smart phones, industry, government, everywhere. Knowing that you’ll be ignored and unappreciated until something goes wrong.”
CNCF has its own Cloud Native End-User Tech Radar
Inspired by the Thoughtworks Technology Radar, the CNCF has created its own variation of it to provide a visual overview of technology trends.
Figure: CNCF Technology Radar 2021
The visible results are based on around 150 reviews from a CNCF survey conducted in 2021. A distinction is made between the categories “Adopt”, “Trial”, and “Assess”. “Adopt” are technologies that have already established themselves sufficiently and can be integrated into one’s own DevSecOps process without hesitation. “Trial” are technologies that one should try out. “Assess” are technologies that one should keep an eye on, but have not yet reached the maturity level to use them without hesitation.
Many well-known tools can be found in the “Adopt” area, including names like Sonarqube, Istio, HashiCorp Vault, Terraform, ArgoCD, and OPA. These technologies should be familiar to any seasoned DevOps professional. Lesser-known candidates can be found in the “Assess” area. It is precisely these technologies that could bring about the desired innovation in the field of DevSecOps in the future.
Here is a brief overview of the listed “Assess” technologies and why they might be interesting for DevSecOps:
Cilium
Cilium is a Kubernetes component that makes the network and connectivity more secure. It is based on the modern GNU/Linux kernel feature eBPF, which is a kind of bytecode interpreter at the kernel level. With eBPF, you can run workloads in isolation at the kernel level and execute kernel functions outside of their intended operation. This allows Cilium to process network traffic directly in the kernel, bypassing many detours, which is more efficient and secure.
HashiCorp Sentinel
Sentinel is a policy-as-code software and comes from the well-known and leading cloud-native software company HashiCorp. It allows you to integrate policies into your own DevOps process and have them versioned, reviewed, and tested like application code.
Trivy
Trivy is a vulnerability scanner and allows you to check a wide range of artifacts for security vulnerabilities.
All of these technologies should be evaluated separately and are perfect for further technological improvements. Check back with us regularly to see if we have already prepared the tool in the meantime. 🤓
Case Study: Aqua Trivy
Trivy is an interesting project that performs vulnerability scanning of artifacts. In Trivy’s own words, the application defines itself as follows:
Trivy (tri pronounced like trigger, vy pronounced like envy) is a simple and comprehensive vulnerability/misconfiguration/secret scanner for containers and other artifacts. Trivy detects vulnerabilities of operating system packages (Alpine, RHEL, CentOS, etc.) and language-specific packages (Bundler, Composer, npm, yarn, etc.). In addition, Trivy scans infrastructure as code (IaC) files such as Terraform and Kubernetes to detect potential configuration issues that put your deployments at risk of attack. Trivy also scans for hard-coded secrets such as passwords, API keys, and tokens. Trivy is easy to use. Just install the executable and you are ready to scan. All you need to do to scan is specify a target, such as a container image name.
Installation and Usage
Trivy can be easily installed via the package manager of your choice. Here on macOS simply with brew:
|
|
The syntax of trivy is as follows:
|
|
For a container image, it could look like this:
|
|
On the official website, there is also an animated GIF that illustrates its use quite well. This is what it might look like:
Figure: Source: Trivy
A Short List of Best Practices
I haven’t come up with the best practices myself, but I’m relying on collective experience. On Devops.com there is a list of points that I think makes a lot of sense, and I’d like to go through them.
- Secure the development process: The first step of DevSecOps is to secure the CI/CD pipelines. This means that only authorized developers have access to the Git repositories and changes to the code must be approved by designated reviewers via pull requests. This ensures that the code base itself is reviewed for security-relevant aspects and that new code goes through the same process.
- Protect the production environment: The production environment is where the application is deployed and made available to the end customer. Therefore, it is crucial that the production environment is made as secure as possible. One way to achieve this is to divide it into different so-called tiers. If one of these tiers is compromised, other tiers are available to deploy the application.
- Introduce least-privilege principles: It makes sense to design access rights to DevOps resources in such a way that users are granted the least possible privileges. This principle is called least-privilege. The reason for this is that your own developers and stakeholders ultimately pose the greatest security risk. The reason for this is not necessarily that employees act maliciously, but that it is not always easy to work in a security-conscious manner and to be aware of all security vulnerabilities.
- Use role-based access control (RBAC): Role-based access control, also known as RBAC, is a method of authorizing access to resources based on roles rather than individual users. The advantage of this is that the roles must be defined, e.g. “developer” or “tester”, and users can be assigned these roles. When a DevOps resource is requested, it is checked whether the user has the role required for the resource. RBAC has the advantage that potential damage can be proactively limited, as only users can compromise resources for which they also have the appropriate role.
- Encrypt sensitive data: Any data that could potentially be used to harm or impersonate individuals should be classified as sensitive. This could include, for example, credit card numbers or social security numbers. Sensitive data must be encrypted accordingly. One of several ways to do this is to encrypt with PGP (Pretty Good Privacy), a combination of public and private keys to encrypt data. There are many other ways to encrypt.
- Use multi-factor authentication: Multi-factor authentication (MFA), also known as 2-factor authentication when only 2 channels are used for authentication, is a method to increase security when logging in and authenticating to DevOps resources. This helps if an unauthorized user tries to log in to a cloud platform with a legitimate password because the password has been compromised.
- Use secrets management tools: The code base may also contain sensitive data, such as API keys or user data for interfaces. These should not be stored unsecured in a Git repository, even if access to the repositories is regulated. A secrets management tool such as HashiCorp Vault or AWS Secrets Manager can help here. This allows these secrets to be protected accordingly and only used where they are needed.
- Promote security awareness: An important part of DevSecOps is, of course, the mentality and culture of the employees. It is important to create awareness of security and to promote continuous training in this area. This includes training with certificates or customized training for the internal infrastructure and the DevOps environment.
- Use a Web Application Firewall (WAF): A Web Application Firewall (WAF) is a firewall specifically designed for web applications that regulates access to web server backend systems. It helps to intercept malicious incoming traffic. There are many WAF solutions, often used in combination with a reverse proxy such as NGINX or HAProxy.
- Conduct regular security audits: Regular audits are a central process in a DevOps environment, especially with regard to security aspects. They help to identify and fix vulnerabilities. These audits can cover the entire DevOps process or be divided into smaller, dedicated units such as penetration tests or simple code reviews.
- Use Intrusion Detection and Prevention Systems (IDPS): Intrusion Detection and Prevention Systems (IDPS) are designed to detect and block malicious activity. They can protect physical and virtual resources. There are many IDPS solutions such as Snort, Suricata, or Bro. Often IDPS systems are used as part of Security Information and Event Management Systems (SIEM).
- Design a Disaster Recovery Plan (DRP): In case something goes wrong and systems are unintentionally compromised, it makes sense to have a backup plan in the form of a Disaster Recovery Plan (DRP). This plan serves as a guide for what to do in the event of an outage. It contains a list of steps and conditions under which these steps should be executed. DRPs also contain contact information and detailed instructions on how to restore or rebuild systems.
- Use logging and monitoring tools: Logging and monitoring are fundamental components of any DevOps process, but in terms of security, traceability is an important aspect. In the event of an attack, you want to be able to track exactly what happened in order to close the security gap. There are many well-known solutions used by large companies, such as Splunk, Nagios, or the ELK Stack.
- Conduct regular penetration tests: Penetration tests are security tests that attempt to gain unauthorized access to a target system. Potential attack possibilities are simulated in a controlled manner in order to assess known and unknown attack vectors as best as possible. Penetration tests are part of a security audit and should be carried out regularly. Specialized penetration testers are often hired for this purpose. Well-known tools in this area include, for example, Metasploit.
- Use Access Control Lists (ACLs): Access control lists, also known as ACLs, are lists of rules that allow or restrict access to DevOps resources. All rules are checked, e.g. whether the incoming IP address is on a blacklist. If no rule applies, access is granted.
So, those were 15 best practices in a nutshell. It definitely makes sense to look at each point individually and familiarize yourself with the principles. Security is not a static state, but changes over time and adapts to new conditions. It is therefore important to read and educate yourself regularly to stay up to date. The points listed above provide a good insight into the security aspect of a DevSecOps solution.
Closing Remarks
At b-nova, DevOps is one of our passions. And we have also come to the opinion that you can only do real DevOps in combination with the security aspect, i.e. as DevSecOps. This is also one of the reasons why we are increasingly pursuing security topics and proactively integrating them into our daily routine and knowledge pool.
It’s best to check out our TechUps on security to stay on top of DevSecOps.
Links and Further Resources
DevSecOps | Cloud Native Glossary
DevSecOps, September 2021 | Cloud Native End User Tech Radar
DevSecOps Overview | Snyk Resources
What is DevSecOps? And what you need to do it well | dynatrace
DevSecOpsTools | The DevSecOp tools that secure DevOps workflows | Atlassian
15 DevSecOps Best Practices | DevOps.com
Container Security Scanning with Trivy and Azure DevOps | Liam Gulliver’s Blog
This techup has been translated automatically by Gemini