Building a Secure Continuous Delivery Pipeline - Medly.Tech

In this talk, we will speak about challenges teams face to include security as a practice. We will share some of the lessons learned, tools and techniques to help teams build a continuous delivery pipeline which has security at its core. We will also talk about how a continuously evolving threat model helps team to bake security in the product instead of bloating on later.

Transcript

1. 1 Building a Continuous Secure Delivery Pipeline Shirish Padalkar @truthyvalue
2. 2 About Me • Shirish Padalkar / शि रीष पाडळकर • Developer, Agile Coach, Security Specialist • Principal Engineer @ Medly • Security / DevSecOps practice lead
3. 3 Current state
5. “I feel like we talk about dev and ops working better together, but we kind of ignore security and security teams.” Pete Cheslock “Why We Can't Have Nice Things”, DevOpsDays Austin, 2015
6. 6 https://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/
9. 9 The Security Sandwich } Threat modeling Security discussions Penetration testing A lot of changes Who is taking care of security?
10. 10 Security by checklist
11. 11 Security by throwing things over the wall
12. Source - https://flic.kr/p/JfzVd9 The Security Theatre
13. 13 Security team’s perspective
14. 14 No upfront design • Upfront design is a big no-no in agile / lean teams • When to make sure the design is secure? • How do you review the design? • Design isn’t done • No design document to be handed off • Design is constantly changing along with the code and requirements • Lean teams want to build MVP, fail fast
15. 15 Microservices • Good for scaling and maintaining domain boundaries • Bring operational complexity • Big attack surface • No obvious security “choke point” • Different tech stack ➟ difficult to standardise security practices • No standard logging by default
16. 16 Containers • Standard for deployment on the cloud • Package and deploy all of the runtime dependencies • Eliminates the “works on my machine” problem • Provide some isolation and security protection by default • How to manage secrets inside image? • Dealing with container breakouts • How do you know if container image to be trusted?
17. 17 Source: The State of Open Source Security Report, 2020 by Snyk
18. 18 Delivery at speed needs Security at speed
20. 20 Shift security left
21. 21 Write code and tests CI/CD Test Release Design
22. 22 Time Cost of defect Write code and tests CI/CD Test Release Design
23. 23 DevOps 㱺 DevSecOps
24. DevOps 㱺 24 Dev Sec Ops Sec Ops Dev Ops Dev Sec
26. 26 Source - “DevOpsSec” by Jim Bird, page 19
27. 27 Before we begin
28. Security is like an onion. Not because it’s makes you cry.
29. 29 Security can’t get in the way of delivery.
30. 30 Pre-development
31. 31 Security Champions Program • Every project has one nominated security champion • Single point of contact for the team from security perspective • Security Champion • Has strong engineering skills • Has visibility into key decisions of the project • Is an influential team member • Security team mentors this person about security • Goals of security champion • Educate the team about security thinking • Own project’s Threat Model • Manage known vulnerabilities
32. 32 Secure defaults • Make it easy to write secure code and difficult to make mistakes • Build on top of secure libraries and frameworks • Build security in upfront and try to make it invisible to developers • Provide tools which identifies if an insecure dependency is introduced • Define a security low bar which all projects needs to meet • All password must be encrypted • Every engineer must have 2FA enabled • Etc.
33. 33 Gather Security Requirements • Get security requirement as part of each story • Mark security sensitive stories and get them reviewed by security specialist • Write security acceptance criteria • Write abuse(r) / “evil user” stories / “misuse cases”
34. 34 Abuse cases As an attacker I want to perform a SQL injection attack against login form So that I can login as an admin without correct credentials
35. 35 5 minute threat modeling • What assumptions are you making about callers? • Can you trust the data you are getting? • What happens if something fails? https://martinfowler.com/articles/agile-threat-modelling.html
37. 37 During development
38. 38 SAST Tools • Static Application Security Testing (SAST) • Designed to analyse source code • Some tools are starting to move into the IDE
39. 39 SAST Tools - Strengths • Provides immediate feedback • Scales well • Can be run on lots of software • Can be run repeatedly • Useful for things such as buffer overflows, SQL Injection Flaws • Output is good for developers • Highlights the precise source files, line numbers
40. 40 SAST Tools - Weaknesses • Not all security vulnerabilities are easy to find automatically • Authentication problems • Access control issues • Insecure use of cryptography, etc. • High numbers of false positives • Some commercial tools do a better job at this • Still need to 'prove' that an identified security issue is an actual vulnerability. • Many of these tools have difficulty analysing code that can't be compiled.
41. 41 SAST Tools - IDE plugins https://plugins.jetbrains.com/plugin/10972-snyk-vulnerability-scanner
42. 42 SAST Tools - IDE plugins https://find-sec-bugs.github.io/
43. 43 Pre-commit hooks https://github.com/thoughtworks/talisman
44. 44 SCA Tools - Dependency Checker https://owasp.org/www-project-dependency-check/
45. 45 SCA Tools - Dependency Check as a Service
46. 46 Pairing with security specialist • Pairing with security specialist on security sensitive stories • Helps understanding “why”s, not just “how”s • Code reviews
47. 47 During testing
48. 48 Vulnerability Scanners - Snyk https://github.com/snyk/snyk
49. 49 Vulnerability Scanners - Snyk https://github.com/snyk/snyk
50. 50 Vulnerability Scanning as a Service https://docs.github.com/en/code-security/code-scanning
51. 51 DAST tools • Dynamic Application Security Testing (DAST) • Designed to scan web applications, normally from the outside • Great to catch low hanging fruits • eg. missing CORS headers • Can’t validate logic flaw automatically
52. 52 DAST - Burp https://portswigger.net/burp
53. 53 DAST - OWASP Zed Attack Proxy (ZAP) https://www.zaproxy.org/
55. 55 Automated tests • We can’t rely fully on SAST and DAST tools • Sometimes we need to write own tests for validating business logic flaws
56. 56 Semgrep https://semgrep.dev/
57. 57 Semgrep https://semgrep.dev/
58. 58 Gauntlt • Framework that enables security testing that is usable by devs, ops and security. • easy-to-read language • Easily hooks into testing tools and processes • Adaptors for • nmap • cURL • sqlmap • etc. • http://gauntlt.org/ # nmap-simple.attack Feature: simple nmap attack to check for open ports Background: Given "nmap" is installed And the following profile: | name | value | | hostname | example.com | Scenario: Check standard web ports When I launch an "nmap" attack with: """ nmap -F <hostname> """ Then the output should match /80.tcp\s+open/ Then the output should not match: """ 25\/tcp\s+open """
59. 59 Manual Security Testing • Do as exploratory testing • Not effective as a control gate in CI/CD world • Pen tests takes too long to setup, run and review • Don’t forget to automate after discovering vulnerability!
60. 60 Securing Infrastructure
61. 61 Infrastructure as Code • Must have to fix security problems at one place propagate everywhere • Compliance using code • Can write security tests for • unnecessary services are disabled • ports that do not need to be open are indeed not open • Review permissions on sensitive files and directories
62. 62 Static code analysis - tfsec https://tfsec.dev/
63. 63 Container Image Scanning - Clair https://github.com/quay/clair
64. 64 Policy As Code - Hashicorp Sentinel https://www.hashicorp.com/sentinel
65. 65 Open Policy Agent (OPA) https://www.openpolicyagent.org/
66. 66 Managing Secrets 6 stages of secret management • Denial - No solution • Reluctance - Emailing encrypted secrets • Bargaining - Team password managers • Acceptance - Encrypted DVCS • Progress - Orchestration • Mastery - Secret Service https://www.vaultproject.io/
67. 67 Infrastructure Hardening https://dev-sec.io/ and https://github.com/dev-sec
68. 68 Cloud Specific solutions https://aws.amazon.com/guardduty/
69. 69 Feature Toggles • Gives you control to turn functionality on or off dynamically • Super handy if something goes wrong • https://writeabout.net/2016/12/21/there-is-no-devops-without-feature-flags/
70. 70 Summary
71. 71 Take aways • Delivery at speed needs security at speed • Security can’t get in the way of delivery • More often than not, the security team isn’t available to look at everything. We need to do more! • Prefer built-in security over bloat-in security • Introduce transparent tools which breaks your CD pipeline if vulnerabilities are introduced • Everybody is responsible for security.
72. 72 Snapshot of tools • IDE plugins • Snyk • FindSecBugs • Talisman • Dependency Checker • Github Dependabot • Snyk Scanner • DAST • Burp Suite • Zed Attack Proxy • Automated tests • Semgrep • Gauntlt • tfsec • Hashicorp Sentinel • Clair Image Scanner • Open Policy Agent • Hashicorp Vault • dev-sec.io
73. 73 Source - “DevOpsSec” by Jim Bird, page 33
74. 74 Books
75. 75 Interested in implementing this? We are hiring! shirish.padalkar@medly.com https://apply.medly.careers/ Our Values • Empathy • Make realistic tradeoffs • Credibility • “Enablers of security” than owners of security • Reward good security practices • Helping people who ask is always our highest priority
76. 76 Questions? shirish.padalkar@medly.com @truthyvalue https://www.linkedin.com/in/shirishpadalkar/
77. Thank you, 77