Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a Continuous Secure Delivery Pipeline

Building a Continuous Secure Delivery Pipeline

In today’s fast paced software development world, we have seen teams facing difficulties keeping up with security requirements. Regular security breach in news highlights how a simple security miss can result into big financial and reputation loss.

To solve this problem, we tried to integrate security as an agile engineering practice, similar to pairing or TDD at ThoughtWorks. In this talk we will speak about challenges teams face to include security as a practice. We will share some of the lessons learned, tools and techniques to help teams build a continuous delivery pipeline which has security at its core. We will also talk about how a continuously evolving threat model helps team to bake security in the product instead of bloating on in later.

After attending this talk, you will learn the shift in mindset required to have security at the core of delivery pipeline. You will also learn some tools and techniques to be included in your development and delivery workflow to help build security in. Finally, you will learn how having a continuously evolving threat-model can help mitigate security risks.


Shirish Padalkar

May 16, 2019


  1. Building a Continuous Secure Delivery Pipeline Shirish Padalkar @truthyvalue

  2. None
  3. “I feel like we talk about dev and ops working

    better together, but we kind of ignore security and security teams.” Pete Cheslock “Why We Can't Have Nice Things”, DevOpsDays Austin, 2015
  4. https://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/

  5. None
  6. None
  7. The Security Sandwich } Threat modelling Security discussions Penetration testing

    A lot of changes Who is taking care of security?
  8. Security by checklist

  9. Security by throwing things over the wall

  10. Source - https://flic.kr/p/JfzVd9 The Security Theatre

  11. Security Team’s perspective

  12. No upfront design • Upfront design is a big no-no

    in agile / lean teams • When to make sure the design is secure? • How do you review the design? • Design isn’t done • No design document to be handed off • Design is constantly changing along with the code and requirements • Lean teams want to build MVP, fail fast
  13. • Good for scaling and maintaining domain boundaries • Bring

    operational complexity • Big attack surface • No obvious security “choke point” • Different tech stack ➟ difficult to standardise security practices • No standard logging by default Microservices
  14. • Standard for deployment on the cloud • Package and

    deploy all of the runtime dependencies • Eliminates the “works on my machine” problem • Provide some isolation and security protection by default • How to manage secrets inside image? • Dealing with container breakouts • How do you know if container image to be trusted? Containers
  15. Source: The State of Open Source Security Report, 2019 by

  16. Delivery at speed needs Security at speed

  17. None
  18. Shift security left

  19. Write code and tests CI/CD Test Release Design

  20. Time Cost of defect Write code and tests CI/CD Test

    Release Design
  21. DevOps 㱺 DevOpsSec

  22. DevOps 㱺

  23. None
  24. Source - “DevOpsSec” by Jim Bird, page 19

  25. Before we begin

  26. Security is like an onion. Not because it’s makes you

  27. Security can’t get in the way of delivery.

  28. Pre-development

  29. • Make it easy to write secure code and difficult

    to make mistakes • Build on top of secure libraries and frameworks • Build security in upfront and try to make it invisible to developers • Define a security low bar which all projects needs to meet ◦ All password must be encrypted ◦ Every engineer must have 2FA enabled • Provide tools which identifies if an insecure dependency is introduced Secure defaults
  30. • Get security requirement as part of each story •

    Mark security sensitive stories and get them reviewed by security specialist • Write security acceptance criteria • Write abuse(r) / “evil user” stories / “misuse cases” Security Requirements
  31. Abuse cases As an attacker I want to perform a

    SQL injection attack against login form So that I can login as an admin without correct credentials
  32. • What assumptions are you making about callers? • Can

    you trust the data you are getting? • What happens if something fails? 5 minute threat modelling
  33. 5 minute threat modelling

  34. None
  35. During Development

  36. • Static Application Security Testing (SAST) • Designed to analyse

    source code • Some tools are starting to move into the IDE SAST tools
  37. • Scales well ◦ Can be run on lots of

    software ◦ Can be run repeatedly • Useful for things such as buffer overflows, SQL Injection Flaws • Output is good for developers ◦ Highlights the precise source files, line numbers SAST tools - Strengths
  38. • Not all security vulnerabilities are easy to find automatically

    ◦ Authentication problems ◦ Access control issues ◦ Insecure use of cryptography, etc. • High numbers of false positives ◦ Some commercial tools do a better job at this • Still need to 'prove' that an identified security issue is an actual vulnerability. • Many of these tools have difficulty analysing code that can't be compiled. SAST tools - Weaknesses
  39. IDE plugins - Snyk https://blog.jetbrains.com/idea/2019/03/catching-vulnerabilities-instantly-in-your-intellij-idea-environment/

  40. IDE plugins - FindSecBugs https://find-sec-bugs.github.io/

  41. Pre-commit hooks https://github.com/thoughtworks/talisman

  42. Dependency Checker https://www.owasp.org/index.php/OWASP_Dependency_Check

  43. Dependency check as service https://github.blog/2017-11-16-introducing-security-alerts-on-github/

  44. Vulnerability Scanner https://github.com/hawkeyesec/scanner-cli

  45. Vulnerability Scanner as service https://docs.gitlab.com/ee/user/application_security/sast/index.html

  46. SAST - checkmarx (commercial) https://www.checkmarx.com/products/static-application-security-testing/

  47. SAST - Fortify (commercial) https://www.microfocus.com/en-us/products/static-code-analysis-sast/overview

  48. • Pairing with security specialist on security sensitive stories •

    Code reviews Pairing
  49. During Testing

  50. • Dynamic Application Security Testing (DAST) • Designed to scan

    web applications, normally from the outside • Great to catch low hanging fruits ◦ eg. missing CORS headers • Can’t validate logic flaw automatically DAST tools
  51. DAST - Burp https://portswigger.net/burp

  52. DAST - ZAP https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project

  53. • We can’t rely fully on SAST and DAST tools

    • Need to write own tests for validating business logic flaws Automated tests
  54. bdd-security https://github.com/continuumsecurity/bdd-security

  55. • gem install gauntlt • easy-to-read language • Adaptors for

    ◦ nmap ◦ cURL ◦ sqlmap ◦ etc. • http://gauntlt.org/ Gauntlt # nmap-simple.attack Feature: simple nmap attack to check for open ports Background: Given "nmap" is installed And the following profile: | name | value | | hostname | example.com | Scenario: Check standard web ports When I launch an "nmap" attack with: """ nmap -F <hostname> """ Then the output should match /80.tcp\s+open/ Then the output should not match: """ 25\/tcp\s+open """
  56. • Do as exploratory testing • Not effective as a

    control gate in CI/CD world • Don’t forget to automate after discovering vulnerability! Manual security testing
  57. Securing Infrastructure

  58. • Must have to fix security problems at one place

    propagate everywhere • Compliance using code • Can write security tests for ◦ unnecessary services are disabled ◦ ports that do not need to be open are indeed not open ◦ Review permissions on sensitive files and directories Infrastructure as Code
  59. Infrastructure Hardening

  60. • Denial - No solution • Reluctance - Emailing encrypted

    containers • Bargaining - Team password managers • Acceptance - Encrypted DVCS • Progress - Orchestration • Mastery - Secret Service 6 stages of secret management
  61. Secret Service https://www.vaultproject.io/

  62. Continuous Monitoring https://github.com/Netflix/security_monkey

  63. Continuous Monitoring

  64. Toggles

  65. Summary

  66. • Delivery at speed needs security at speed • Security

    can’t get in the way of delivery • More often than not, the security team isn’t available to look at everything. We need to do more! • Prefer built-in security over bloat-in security • Introduce transparent tools which breaks your CD pipeline if vulnerabilities are introduced • Everybody is responsible for security. Take aways
  67. • IDE plugins ◦ Snyk ◦ FindSecBugs • Talisman •

    Dependency Checker • Hawkeye • Checkmarx • Fortify Snapshot • DAST • Burp Suite • Zed Attack Proxy • Automated tests • Bdd-security • Gauntlt • dev-sec.io • Hashicorp Vault • Security Monkey • Splunk Security • Toggles
  68. Source - “DevOpsSec” by Jim Bird, page 33

  69. Source - “DevOpsSec” by Jim Bird, page 33

  70. Source - “DevOpsSec” by Jim Bird, page 33

  71. Books

  72. Acknowledgements Sara Diaz and Cade Cairns

  73. Questions? shirishp@thoughtworks.com https://twitter.com/truthyvalue https://www.linkedin.com/in/shirishpadalkar/

  74. Thank you! https://www.thoughtworks.com/careers/jobs