1
Building a
Continuous Secure Delivery Pipeline
Shirish Padalkar
@truthyvalue
Slide 2
Slide 2 text
2
About Me
● Shirish Padalkar /
शि
रीष पाडळकर
● Developer, Agile Coach, Security Specialist
● Principal Engineer @ Medly
● Security / DevSecOps practice lead
Slide 3
Slide 3 text
3
Current state
Slide 4
Slide 4 text
4
Slide 5
Slide 5 text
“I feel like we talk about dev and ops working better together,
but we kind of ignore security and security teams.”
Pete Cheslock
“Why We Can't Have Nice Things”, DevOpsDays Austin, 2015
9
The Security Sandwich
}
Threat modeling
Security discussions
Penetration testing
A lot of changes
Who is taking care of security?
Slide 10
Slide 10 text
10
Security by checklist
Slide 11
Slide 11 text
11
Security by throwing things over the wall
Slide 12
Slide 12 text
Source - https://flic.kr/p/JfzVd9
The Security Theatre
Slide 13
Slide 13 text
13
Security team’s perspective
Slide 14
Slide 14 text
14
No upfront design
● Upfront design is a big no-no in agile / lean teams
● When to make sure the design is secure?
● How do you review the design?
● Design isn’t done
● No design document to be handed off
● Design is constantly changing along with the code and requirements
● Lean teams want to build MVP, fail fast
Slide 15
Slide 15 text
15
Microservices
● Good for scaling and maintaining domain boundaries
● Bring operational complexity
● Big attack surface
● No obvious security “choke point”
● Different tech stack ➟ difficult to standardise security practices
● No standard logging by default
Slide 16
Slide 16 text
16
Containers
● Standard for deployment on the cloud
● Package and deploy all of the runtime dependencies
● Eliminates the “works on my machine” problem
● Provide some isolation and security protection by default
● How to manage secrets inside image?
● Dealing with container breakouts
● How do you know if container image to be trusted?
Slide 17
Slide 17 text
17
Source: The State of Open Source Security Report, 2020 by Snyk
Slide 18
Slide 18 text
18
Delivery at speed
needs
Security at speed
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
20
Shift security left
Slide 21
Slide 21 text
21
Write code
and tests
CI/CD Test Release
Design
Slide 22
Slide 22 text
22
Time
Cost of defect
Write code
and tests
CI/CD Test Release
Design
Slide 23
Slide 23 text
23
DevOps 㱺 DevSecOps
Slide 24
Slide 24 text
DevOps 㱺
24
Dev Sec Ops
Sec Ops Dev
Ops Dev Sec
Slide 25
Slide 25 text
25
Slide 26
Slide 26 text
26
Source - “DevOpsSec” by Jim Bird, page 19
Slide 27
Slide 27 text
27
Before we begin
Slide 28
Slide 28 text
Security is like an onion.
Not because it’s makes you cry.
Slide 29
Slide 29 text
29
Security can’t get in the
way of delivery.
Slide 30
Slide 30 text
30
Pre-development
Slide 31
Slide 31 text
31
Security Champions Program
● Every project has one nominated security champion
● Single point of contact for the team from security perspective
● Security Champion
● Has strong engineering skills
● Has visibility into key decisions of the project
● Is an influential team member
● Security team mentors this person about security
● Goals of security champion
● Educate the team about security thinking
● Own project’s Threat Model
● Manage known vulnerabilities
Slide 32
Slide 32 text
32
Secure defaults
● Make it easy to write secure code and difficult to make mistakes
● Build on top of secure libraries and frameworks
● Build security in upfront and try to make it invisible to developers
● Provide tools which identifies if an insecure dependency is introduced
● Define a security low bar which all projects needs to meet
● All password must be encrypted
● Every engineer must have 2FA enabled
● Etc.
Slide 33
Slide 33 text
33
Gather Security Requirements
● Get security requirement as part of each story
● Mark security sensitive stories and get them reviewed by security specialist
● Write security acceptance criteria
● Write abuse(r) / “evil user” stories / “misuse cases”
Slide 34
Slide 34 text
34
Abuse cases
As an attacker
I want to perform a SQL injection attack against login form
So that I can login as an admin without correct credentials
Slide 35
Slide 35 text
35
5 minute threat modeling
● What assumptions are you making about
callers?
● Can you trust the data you are getting?
● What happens if something fails?
https://martinfowler.com/articles/agile-threat-modelling.html
Slide 36
Slide 36 text
36
Slide 37
Slide 37 text
37
During development
Slide 38
Slide 38 text
38
SAST Tools
● Static Application Security Testing (SAST)
● Designed to analyse source code
● Some tools are starting to move into the IDE
Slide 39
Slide 39 text
39
SAST Tools - Strengths
● Provides immediate feedback
● Scales well
● Can be run on lots of software
● Can be run repeatedly
● Useful for things such as buffer overflows, SQL Injection Flaws
● Output is good for developers
● Highlights the precise source files, line numbers
Slide 40
Slide 40 text
40
SAST Tools - Weaknesses
● Not all security vulnerabilities are easy to find automatically
● Authentication problems
● Access control issues
● Insecure use of cryptography, etc.
● High numbers of false positives
● Some commercial tools do a better job at this
● Still need to 'prove' that an identified security issue is an actual vulnerability.
● Many of these tools have difficulty analysing code that can't be compiled.
Slide 41
Slide 41 text
41
SAST Tools - IDE plugins
https://plugins.jetbrains.com/plugin/10972-snyk-vulnerability-scanner
Slide 42
Slide 42 text
42
SAST Tools - IDE plugins
https://find-sec-bugs.github.io/
46
Pairing with security specialist
● Pairing with security specialist on security sensitive stories
● Helps understanding “why”s, not just “how”s
● Code reviews
50
Vulnerability Scanning as a Service
https://docs.github.com/en/code-security/code-scanning
Slide 51
Slide 51 text
51
DAST tools
● Dynamic Application Security Testing (DAST)
● Designed to scan web applications, normally from the outside
● Great to catch low hanging fruits
● eg. missing CORS headers
● Can’t validate logic flaw automatically
55
Automated tests
● We can’t rely fully on SAST and DAST tools
● Sometimes we need to write own tests for validating business logic flaws
Slide 56
Slide 56 text
56
Semgrep
https://semgrep.dev/
Slide 57
Slide 57 text
57
Semgrep
https://semgrep.dev/
Slide 58
Slide 58 text
58
Gauntlt
● Framework that enables security testing that is
usable by devs, ops and security.
● easy-to-read language
● Easily hooks into testing tools and processes
● Adaptors for
● nmap
● cURL
● sqlmap
● etc.
● http://gauntlt.org/
# nmap-simple.attack
Feature: simple nmap attack to check for open ports
Background:
Given "nmap" is installed
And the following profile:
| name | value |
| hostname | example.com |
Scenario: Check standard web ports
When I launch an "nmap" attack with:
"""
nmap -F
"""
Then the output should match /80.tcp\s+open/
Then the output should not match:
"""
25\/tcp\s+open
"""
Slide 59
Slide 59 text
59
Manual Security Testing
● Do as exploratory testing
● Not effective as a control gate in CI/CD world
● Pen tests takes too long to setup, run and review
● Don’t forget to automate after discovering vulnerability!
Slide 60
Slide 60 text
60
Securing Infrastructure
Slide 61
Slide 61 text
61
Infrastructure as Code
● Must have to fix security problems at one place propagate everywhere
● Compliance using code
● Can write security tests for
● unnecessary services are disabled
● ports that do not need to be open are indeed not open
● Review permissions on sensitive files and directories
64
Policy As Code - Hashicorp Sentinel
https://www.hashicorp.com/sentinel
Slide 65
Slide 65 text
65
Open Policy Agent (OPA)
https://www.openpolicyagent.org/
Slide 66
Slide 66 text
66
Managing Secrets
6 stages of secret management
● Denial - No solution
● Reluctance - Emailing encrypted secrets
● Bargaining - Team password managers
● Acceptance - Encrypted DVCS
● Progress - Orchestration
● Mastery - Secret Service
https://www.vaultproject.io/
Slide 67
Slide 67 text
67
Infrastructure Hardening
https://dev-sec.io/ and https://github.com/dev-sec
Slide 68
Slide 68 text
68
Cloud Specific solutions
https://aws.amazon.com/guardduty/
Slide 69
Slide 69 text
69
Feature Toggles
● Gives you control to turn functionality on or off dynamically
● Super handy if something goes wrong
● https://writeabout.net/2016/12/21/there-is-no-devops-without-feature-flags/
Slide 70
Slide 70 text
70
Summary
Slide 71
Slide 71 text
71
Take aways
● Delivery at speed needs security at speed
● Security can’t get in the way of delivery
● More often than not, the security team isn’t available to look at everything. We
need to do more!
● Prefer built-in security over bloat-in security
● Introduce transparent tools which breaks your CD pipeline if vulnerabilities are
introduced
● Everybody is responsible for security.
75
Interested in implementing this?
We are hiring!
[email protected]
https://apply.medly.careers/
Our Values
● Empathy
● Make realistic tradeoffs
● Credibility
● “Enablers of security” than owners of security
● Reward good security practices
● Helping people who ask is always our highest
priority