Building a Continuous Secure Delivery Pipeline

In today’s fast paced software development world, we have seen teams facing difficulties keeping up with security requirements. Regular security breach in news highlights how a simple security miss can result into big financial and reputation loss.

To solve this problem, we tried to integrate security as an agile engineering practice, similar to pairing or TDD at ThoughtWorks. In this talk we will speak about challenges teams face to include security as a practice. We will share some of the lessons learned, tools and techniques to help teams build a continuous delivery pipeline which has security at its core. We will also talk about how a continuously evolving threat model helps team to bake security in the product instead of bloating on in later.

After attending this talk, you will learn the shift in mindset required to have security at the core of delivery pipeline. You will also learn some tools and techniques to be included in your development and delivery workflow to help build security in. Finally, you will learn how having a continuously evolving threat-model can help mitigate security risks.

Transcript

1. Building a Continuous Secure Delivery Pipeline Shirish Padalkar @truthyvalue
3. “I feel like we talk about dev and ops working better together, but we kind of ignore security and security teams.” Pete Cheslock “Why We Can't Have Nice Things”, DevOpsDays Austin, 2015
4. https://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/
7. The Security Sandwich } Threat modelling Security discussions Penetration testing A lot of changes Who is taking care of security?
8. Security by checklist
9. Security by throwing things over the wall
10. Source - https://flic.kr/p/JfzVd9 The Security Theatre
11. Security Team’s perspective
12. No upfront design • Upfront design is a big no-no in agile / lean teams • When to make sure the design is secure? • How do you review the design? • Design isn’t done • No design document to be handed off • Design is constantly changing along with the code and requirements • Lean teams want to build MVP, fail fast
13. • Good for scaling and maintaining domain boundaries • Bring operational complexity • Big attack surface • No obvious security “choke point” • Different tech stack ➟ difficult to standardise security practices • No standard logging by default Microservices
14. • Standard for deployment on the cloud • Package and deploy all of the runtime dependencies • Eliminates the “works on my machine” problem • Provide some isolation and security protection by default • How to manage secrets inside image? • Dealing with container breakouts • How do you know if container image to be trusted? Containers
15. Source: The State of Open Source Security Report, 2019 by Snyk
16. Delivery at speed needs Security at speed
18. Shift security left
19. Write code and tests CI/CD Test Release Design
20. Time Cost of defect Write code and tests CI/CD Test Release Design
21. DevOps 㱺 DevOpsSec
22. DevOps 㱺
24. Source - “DevOpsSec” by Jim Bird, page 19
25. Before we begin
26. Security is like an onion. Not because it’s makes you cry.
27. Security can’t get in the way of delivery.
28. Pre-development
29. • Make it easy to write secure code and difficult to make mistakes • Build on top of secure libraries and frameworks • Build security in upfront and try to make it invisible to developers • Define a security low bar which all projects needs to meet ◦ All password must be encrypted ◦ Every engineer must have 2FA enabled • Provide tools which identifies if an insecure dependency is introduced Secure defaults
30. • Get security requirement as part of each story • Mark security sensitive stories and get them reviewed by security specialist • Write security acceptance criteria • Write abuse(r) / “evil user” stories / “misuse cases” Security Requirements
31. Abuse cases As an attacker I want to perform a SQL injection attack against login form So that I can login as an admin without correct credentials
32. • What assumptions are you making about callers? • Can you trust the data you are getting? • What happens if something fails? 5 minute threat modelling
33. 5 minute threat modelling
35. During Development
36. • Static Application Security Testing (SAST) • Designed to analyse source code • Some tools are starting to move into the IDE SAST tools
37. • Scales well ◦ Can be run on lots of software ◦ Can be run repeatedly • Useful for things such as buffer overflows, SQL Injection Flaws • Output is good for developers ◦ Highlights the precise source files, line numbers SAST tools - Strengths
38. • Not all security vulnerabilities are easy to find automatically ◦ Authentication problems ◦ Access control issues ◦ Insecure use of cryptography, etc. • High numbers of false positives ◦ Some commercial tools do a better job at this • Still need to 'prove' that an identified security issue is an actual vulnerability. • Many of these tools have difficulty analysing code that can't be compiled. SAST tools - Weaknesses
39. IDE plugins - Snyk https://blog.jetbrains.com/idea/2019/03/catching-vulnerabilities-instantly-in-your-intellij-idea-environment/
40. IDE plugins - FindSecBugs https://find-sec-bugs.github.io/
41. Pre-commit hooks https://github.com/thoughtworks/talisman
42. Dependency Checker https://www.owasp.org/index.php/OWASP_Dependency_Check
43. Dependency check as service https://github.blog/2017-11-16-introducing-security-alerts-on-github/
44. Vulnerability Scanner https://github.com/hawkeyesec/scanner-cli
45. Vulnerability Scanner as service https://docs.gitlab.com/ee/user/application_security/sast/index.html
46. SAST - checkmarx (commercial) https://www.checkmarx.com/products/static-application-security-testing/
47. SAST - Fortify (commercial) https://www.microfocus.com/en-us/products/static-code-analysis-sast/overview
48. • Pairing with security specialist on security sensitive stories • Code reviews Pairing
49. During Testing
50. • Dynamic Application Security Testing (DAST) • Designed to scan web applications, normally from the outside • Great to catch low hanging fruits ◦ eg. missing CORS headers • Can’t validate logic flaw automatically DAST tools
51. DAST - Burp https://portswigger.net/burp
52. DAST - ZAP https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project
53. • We can’t rely fully on SAST and DAST tools • Need to write own tests for validating business logic flaws Automated tests
54. bdd-security https://github.com/continuumsecurity/bdd-security
55. • gem install gauntlt • easy-to-read language • Adaptors for ◦ nmap ◦ cURL ◦ sqlmap ◦ etc. • http://gauntlt.org/ Gauntlt # nmap-simple.attack Feature: simple nmap attack to check for open ports Background: Given "nmap" is installed And the following profile: | name | value | | hostname | example.com | Scenario: Check standard web ports When I launch an "nmap" attack with: """ nmap -F <hostname> """ Then the output should match /80.tcp\s+open/ Then the output should not match: """ 25\/tcp\s+open """
56. • Do as exploratory testing • Not effective as a control gate in CI/CD world • Don’t forget to automate after discovering vulnerability! Manual security testing
57. Securing Infrastructure
58. • Must have to fix security problems at one place propagate everywhere • Compliance using code • Can write security tests for ◦ unnecessary services are disabled ◦ ports that do not need to be open are indeed not open ◦ Review permissions on sensitive files and directories Infrastructure as Code
59. Infrastructure Hardening
60. • Denial - No solution • Reluctance - Emailing encrypted containers • Bargaining - Team password managers • Acceptance - Encrypted DVCS • Progress - Orchestration • Mastery - Secret Service 6 stages of secret management
61. Secret Service https://www.vaultproject.io/
62. Continuous Monitoring https://github.com/Netflix/security_monkey
63. Continuous Monitoring
64. Toggles
65. Summary
66. • Delivery at speed needs security at speed • Security can’t get in the way of delivery • More often than not, the security team isn’t available to look at everything. We need to do more! • Prefer built-in security over bloat-in security • Introduce transparent tools which breaks your CD pipeline if vulnerabilities are introduced • Everybody is responsible for security. Take aways
67. • IDE plugins ◦ Snyk ◦ FindSecBugs • Talisman • Dependency Checker • Hawkeye • Checkmarx • Fortify Snapshot • DAST • Burp Suite • Zed Attack Proxy • Automated tests • Bdd-security • Gauntlt • dev-sec.io • Hashicorp Vault • Security Monkey • Splunk Security • Toggles
68. Source - “DevOpsSec” by Jim Bird, page 33
69. Source - “DevOpsSec” by Jim Bird, page 33
70. Source - “DevOpsSec” by Jim Bird, page 33
71. Books
72. Acknowledgements Sara Diaz and Cade Cairns
73. Questions? shirishp@thoughtworks.com https://twitter.com/truthyvalue https://www.linkedin.com/in/shirishpadalkar/
74. Thank you! https://www.thoughtworks.com/careers/jobs