Continuous Deployment - Speaker Deck

Slide 1

Slide 1 text

Continuous Deployment The New #1 Security Feature Nick Galbreath http://www.client9.com/ [email protected] @ngalbreath Security BSides Los Angeles Hermosa Beach Aug 16, 2012

Slide 2

Slide 2 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Continuous Deployment? #1 Security Feature?

Slide 3

Slide 3 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath The latest version of these slides http://www.client9.com/20120816/

Slide 4

Slide 4 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath whoami • Director of Engineering, Etsy • enterprise, fraud, security, fun • But.. on very good terms.  In fact Etsy sponsored my trip here • VP Engineering, “Company Conﬁdential” • Stay tuned for details Nick Galbreath www.client9.com @ngalbreath

Slide 5

Slide 5 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Context Alert! • My background is software development • Mostly in public, web-facing applications • Everything from C to PHP • Your mileage may vary if you are in different industries, with different risk proﬁles

Slide 6

Slide 6 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Problem Statement • Web App vulnerabilities aren’t conceptually that hard and should be easy to deal with. • In spite (or because) of our efforts, security is an “end of line” process or whack-a-mole • Security education has been at best marginally useful to developers (in the large, your organization may be different). • How can we get ever get ahead?

Slide 7

Slide 7 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath How did we get here?

Slide 8

Slide 8 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath The Software Product Model Code ﬂows to functional groups. • Product Managers spec code • Engineers write code • QA tests code • Security tests code • Release engineers package code • Operations runs code

Slide 9

Slide 9 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath High Distribution Cost The Software Product Model is designed for applications where the cost of distribution is high. Where “high” might be measure by risk, money, time, resources, customer annoyance. • Retail, CD/DVDs • Embedded or Exotic Hardware • Safety, Medical or Defense Systems • Operating Systems (phone or desktop) • Your Homework (1-time deploy)

Slide 10

Slide 10 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath SPM -SDLC Ops Release QA QA Development Bug Fix / Slush New Specs Commits time in weeks or months freeze Specs

Slide 11

Slide 11 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath SPM-Production Changes to Production Major Release Minor Releases Major Release BigBang BigBang New Features going live are 100% correlated with volume of changes to production.

Slide 12

Slide 12 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Nothing wrong with this given the requirements. Given high distribution costs, it makes sense to front-load the development process

Slide 13

Slide 13 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath WebApps Y2K • Mostly followed software product model since that’s all we knew • High barrier to entry • Specialized Hardware, software, people to get started • Lots of engineering to keep things running and scaling.

Slide 14

Slide 14 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath True Story • “Can’t push out the spelling error change since it’s too risky” • “That code has already been through QA, it’s locked down.” • “Product has to prioritize that, else we aren’t touching it.” This Smells

Slide 15

Slide 15 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath WebApps 2012 • Almost no barrier to entry • Commodity hardware • “Learn PHP in 24 hours” • Scaling problems can be outsourced (sorta)

Slide 16

Slide 16 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath WebApps 2012 and Cost of Distribution • Moving a few megabytes from source control to a few machines in production over a 1Gb or 10Gb link. • In other words... free!

Slide 17

Slide 17 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Given this and competitive /customer expectations, it’s not unreasonable to expect an SDLC moves faster than the Software Product Model

Slide 18

Slide 18 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath On the other hand, WebApps 2012 have very different failure cases

Slide 19

Slide 19 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath The Nature of Failure • WebApps 2012 are data-driven. • and frequently have APIs, user-generated content, social features (unexpected use cases, new problems) • Failure might be due to algorithm problems, but... • ...more likely it’s bad user input, bad data in database, or operational load. • This means data added in the past might cause problems in the future. Complicated!

Slide 20

Slide 20 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath When SPM meets WebApp2012 • There is a long time between code-written to code- deployed. This “non-value added” steps for what should be low-cost changes. • Might be weeks or months before code deployed. • Feedback loop between code in dev and code in production broken. • When the bug/security report comes in, it’s likely the engineer is on a different project. • Any wonder that engineers don’t care for operations or security?

Slide 21

Slide 21 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Hypothesis • It is impossible to simulate the production environment in development, either to operational differences or data differences. • No amount of QA or Security Testing can prove you don’t have bugs, vulnerabilities, or won’t cause severe operational problems. • You have bugs and vulnerabilities right now your site.

Slide 22

Slide 22 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Conclusion: Your Screwed!

Slide 23

Slide 23 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath • Company wide push to move faster • Being a bottleneck isn’t acceptable. • Nor is giving up or saying “need more resources” • Engineers disengaged • Looming security disaster awaits • Whack-a-Mole doesn’t scale

Slide 24

Slide 24 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath If we want to ﬁx Security, we have to ﬁx Development.

Slide 25

Slide 25 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Continuous Deployment A System of Software Production Characterized By Numerous Small Changes to the Production Environment  or That Crazy Shit That Etsy Does. And Google. And Facebook. And Amazon. And Twitter. And NetFlix.   So maybe not that crazy.

Slide 26

Slide 26 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath CD -Changes to Production New Features are not correlated with volume of changes to production new feature new feature

Slide 27

Slide 27 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Developers are responsible and conﬁdent in their code. In Production

Slide 28

Slide 28 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath What If You Had a Button that said DEPLOY This button logs who performed the change, and what the change was, but no other rules or controls. •Pushes whatever is on HEAD/TRUNK to production. •In about a minute. •Anyone is allowed to push it.

Slide 29

Slide 29 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 1: Fear • Likely no one is going to push it since they are afraid they’ll break something. • Meanwhile un-deployed code keeps piling up. ex. New hires are terriﬁed of deploying an... HTML change! “but I don’t want to break Etsy!”

Slide 30

Slide 30 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 2: First Attempt • At some point, some brave sole will put their code on TRUNK, and push the button. • It’s likely someone else tells them that their feature blew up the site or doesn’t work, and to please role it back.

Slide 31

Slide 31 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 3: With Graphs • The developer learns that they’d don’t know how the code runs in production and they need some way of understanding how it works. • Enter Graphite/Ganglia/StatsD!  http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/ • Make it free to monitor anything in the application and expose that to everyone.

Slide 32

Slide 32 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 4: Push It • Repushing out code with ﬁx, still causes some problem as witness by a graph falling off a cliff, but the developer was aware of it and was able to role back.

Slide 33

Slide 33 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 5: Isolation • Hmmm, the developer in reviewing the code notices that actually they are pushing a few bugs ﬁxes, and some other minor features. • Maybe just pushing out a single bug ﬁx one at time to help isolate the problem.

Slide 34

Slide 34 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 6: Success! • Yes! The developer pushed code and ﬁxed a bug and made the site just that much better. • The secret about continuous deployment is small deltas that you or anyone can understand easily.

Slide 35

Slide 35 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 7: Dark Pushes • Now that the developer got the bugs out of the way, it time for the feature. • Let’s push out all the supporting ﬁles. By themselves they do nothing. By getting these out ﬁrst, you isolate them as being “unlikely to cause a site problem” • Also now that they are on the trunk, others can look at them (easily).

Slide 36

Slide 36 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 8: Rampups • Now it’s time to get that feature live. • Instead of a Big Bang, he’ll put a ‘rampup’ in the code. This will control how many people on the site will get the new feature. • Maybe start with “employees only” so his team can test in production. • Start at 1%, 5%, 10% and make sure things work, graphs are stable and work up to 100%. • if problem, can rampdown or turn off.

Slide 37

Slide 37 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 8: Eliminate • Along the way you’ll get burned by little things, so, we’ll • A development environment that mimics prod as close as possible (won’t be exact) • Fast and stable unit and functional tests that are easy to run. If they are slow and ﬂakey, no one will use them • Eliminate stupid bugs with commit or pre-commit static analysis. • Move QA/Security/Release checks as close as possible to the developer.

Slide 38

Slide 38 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 9: Communicate • As more people get use to it, you’ll need a way of co-ordinating releases among people. • IRC works well • Need set of conventions that match your risk levels. • At least developers are talking about releases!

Slide 39

Slide 39 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Take 10: Learn • No doubt along the way, serious mistakes will be made. Complex system failures will happen. • Learn from them. Do Post-Mortems. Do Root-Cause Analysis. • Recount what happened. • 99.99999999% of problems are caused by mistakes not maliciousness • How can the environment be changed so it doesn’t happen again? • Publish the results.

Slide 40

Slide 40 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath But What About...

Slide 41

Slide 41 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath What About That Guy Who Pushes at 3AM • That Guy who pushes at 3AM, and something goes wrong and wakes up all of operations with pagers going off will quickly learn this was a bad idea. • It’s about courtesy and respect. • Of course there are off-hours exception, in which That Guy should pre-inform everyone.

Slide 42

Slide 42 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath What about code reviews? • Yup, do them • Nothing here precludes code reviews. • In fact, it’s frequently easier to do since the reviewer doesn’t have to dork around with branches or tags.... they have all the dark code already on Trunk/Head • .. and the reviews are smaller and faster

Slide 43

Slide 43 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath What about security reviews? • Yup do ‘em. • Nothing here eliminates your existing review cycle.

Slide 44

Slide 44 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath What about Agile Methods? • (everyone does “agile differently” so hard to qualify this). • Agile methods frequently work to improve the spec-writing / development cycle • Or the spec / dev / qa cycle • But code still pools up waiting to go to production.

Slide 45

Slide 45 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath What about Customer Service? Do they freak out with all the changes? • Remember, most changes either do nothing, or are trivial or are minor. • Feature launches always need to be co- ordinated with customer service (from audience question)

Slide 46

Slide 46 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath So Why Did I Tell You All This?

Slide 47

Slide 47 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath That Engineer who previously didn’t push code is now sensitized that their code has consequences and are responsive to fix it. It’s amazing how interested engineers become in security when you find problems with their code  when they are able to fix quickly themselves.

Slide 48

Slide 48 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Security Fixes can go out quickly. In addition, you know ﬁxes can go out since they happen every day.

Slide 49

Slide 49 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath You can repurpose the QA stack, graphing and log analysis for attack detection and vulnerability prevention. Need ideas? Check out these other presentations on fraud and security by Etsy: http://slidesha.re/IMaavq http://slidesha.re/JGaU2s http://slidesha.re/KPvHYu http://slidesha.re/Kw5zdV

Slide 50

Slide 50 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath While there is always whack- a-mole, you can focus on being a service organization and work on engineering to be secure by default.

Slide 51

Slide 51 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath New Roles, Less Silos • Developers: works with operations • QA: works on making systems to empower people to write tests, static analysis, in-house consultancy on good test design • Release: tools to push code to production, system images. • Security: in house consultancy, security engineering, secure by default, detection

Slide 52

Slide 52 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath So Continuous Deployment is Only for Websites?

Slide 53

Slide 53 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Google Chrome • Really made updates painless for the consumer. • Frequent changes “regularly” -- maybe not continuous but way faster than normal software product • Multiple channels of releases. • Conﬁg ﬂags can turn on or off experimental features. • Works so well, many others are copying this approach.

Slide 54

Slide 54 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Apps • Due to cost of deployment being high  (e.g. due to approval from Apple) • And due to diversity of destination (how many different types of hardware will it run on), hard to predict how well it work. • Put as much as you can into the release • Then read conﬁgs from internet to light up or turn off features

Slide 55

Slide 55 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Chip Design • After this talk, I met an engineer who does hardware design. • All changes are tiny and then tested, then committed. • Any change too big is rejected. • Learned the hard way that big changes are impossible to understand and test.

Slide 56

Slide 56 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath So What Now?

Slide 57

Slide 57 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Security is in a Good Position to Force Change • Security bridges multiple disciplines: ops, dev, qa, release, business. Unique position to make change. • When breach happens (in whatever the layer), we need to patch fast. • I hope that is not controversial.

Slide 58

Slide 58 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Start with the Deploy Button

Slide 59

Slide 59 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath It will change your SDLC more resilient more secure

Slide 60

Slide 60 text

Nick Galbreath BSidesLA Aug16,2012 @ngalbreath Continuous Deployment The New #1 Security Feature