KubeCon EU 2017: Dancing on the Edge of a Volcano

Slide 1

Slide 1 text

Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)

Slide 2

Slide 2 text

Dancing on the Edge of a Volcano Kubernetes Security Processes Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)

Slide 3

Slide 3 text

Today's Major Parts ● Context Setting ● Open Source and Security Disclosure ● Kubernetes Security Process ● Helping Out

Slide 4

Slide 4 text

Today's Major Parts ● Context Setting ● Open Source and Security Disclosure ● Kubernetes Security Process ● Helping Out

Slide 5

Slide 5 text

Computer Security in 2017 BP

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Why is Server Security Important?

Slide 10

Slide 10 text

3,424,000,000 Internet Users Source: http://www.internetlivestats.com/internet-users/

Slide 11

Slide 11 text

29,000,000 Software Developers and IT Practitioners Source: https://www.infoq.com/news/2014/01/IDC-software-developers

Slide 12

Slide 12 text

We are Outnumbered!

Slide 13

Slide 13 text

238,975,082 New Internet Users in 2016 Source: http://www.internetlivestats.com/internet-users/

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Documents Commerce Communications

Slide 16

Slide 16 text

~100,000,000 Servers Worldwide

Slide 17

Slide 17 text

3 Per Person In the Software and IT Industry

Slide 18

Slide 18 text

100+ Per Person At the Internet Giants

Slide 19

Slide 19 text

Kubernetes.

Slide 20

Slide 20 text

Today's Major Parts ● Context Setting ● Open Source and Security Disclosure ● Kubernetes Security Process ● Helping Out

Slide 21

Slide 21 text

Challenges with Open Source and Security Disclosure JF

Slide 22

Slide 22 text

An open process Need to be open to vendors and community but also keep information from those who can do harm before bugs are fixed.

Slide 23

Slide 23 text

GPG: Do you even?

Slide 24

Slide 24 text

Can we make everyone happy? Researchers, community, vendors… All sides want something different.

Slide 25

Slide 25 text

Users Expectations - Want software without bugs - Want to be able to upgrade without breaking changes - Want to be let know when security fixes are made so they can act on them accordingly

Slide 26

Slide 26 text

Original Reporter Expectations - Want “a human to say acknowledged” - Want updates on status, transparency - Want bug to be fixed in a certain timeframe

Slide 27

Slide 27 text

Project Expectations - Want bugs to be fixed quickly and embargo to be kept secret until time of disclosure - Want upgrades to remain seamless - Want to continue to instill trust with users

Slide 28

Slide 28 text

How to get people to upgrade? - Challenges with making sure people know - Use mailing lists, establish a pattern for the project that allows people to follow along - How to handle if very severe… pray you don’t have someone make up a fun name

Slide 29

Slide 29 text

Challenge: name the vulnerability

Slide 30

Slide 30 text

How bugs get reported... I found a bug… should I be evil or let upstream know... Let me google “X security disclosure” or I’ll just try to email security@

Slide 31

Slide 31 text

How bugs get reported... I’ll lookup the project’s recommended method for disclosures I found a bug… I’ll file an issue on the repo...

Slide 32

Slide 32 text

I found a bug… I’ll file an issue on the repo... - Not everyone can recognize the severity of a bug at first glance. - Allow for mistakes and have a process for handling them in a way that it will not confuse the original reporter or cause unnecessary attention. Mistakes happen

Slide 33

Slide 33 text

Learning from Other Communities BP

Slide 34

Slide 34 text

The Common Stuff

Slide 35

Slide 35 text

General Recommendations We Got - Make documentation Googleable (X security disclosure) - Have a dedicated response team - Coordinate via a mailing list - Don't force the use of GPG - Have internal documentation on the process - Use a severity leveling system - Request CVEs when appropriate*

Slide 36

Slide 36 text

Docker: Document everything JF

Slide 37

Slide 37 text

Docker: Tooling for everything JF

Slide 38

Slide 38 text

Linux Kernel: Don't Negotiate Set timeline expectations; don't negotiate on embargos BP

Slide 39

Slide 39 text

Node.js: Help the Ecosystem BP Fix the core but also help track the ecosystem and notify

Slide 40

Slide 40 text

Go: Early warning systems JF Give people a heads up that a vulnerability is coming

Slide 41

Slide 41 text

Go: Small team, explicit contacts Mailing lists fail, people don't read email, list everyone. JF

Slide 42

Slide 42 text

Today's Major Parts ● Context Setting ● Open Source and Security Disclosure ● Kubernetes Security Process ● Helping Out

Slide 43

Slide 43 text

Where we are today BP

Slide 44

Slide 44 text

Kubernetes Normal Release Process BP

Slide 45

Slide 45 text

Kubernetes Release Process k8s 1.5.x k8s 1.6.0 k8s 1.6.x Bug fixes & security patches Features Features k8s 1.7.0 k8s 1.7.x Bug fixes & security patches Features

Slide 46

Slide 46 text

Kubernetes Release Process k8s 1.5.x k8s 1.6.0 k8s 1.6.x Bug fixes & security patches Features Features Bug fixes & security patches ~9 Months*

Slide 47

Slide 47 text

The Kubernetes v1.6 Release Schedule ● … 4-6 weeks of coding … ● Jan 30th (Monday) v1.6.0-alpha.1 ● Feb 13th (Monday) v1.6.0-alpha.2 ● Feb 20th (Monday) Cut release-1.6 branch and v1.6.0-beta.0 ● Feb 27th (Monday) Start code freeze ● Feb 28th (Tuesday) v1.6.0-beta.1 ● March 7th (Tuesday) - v1.6.0-beta.2 ● March 14th (Tuesday) Lift code freeze and v1.6.0-rc.1 ● March 22nd (Wednesday) - v1.6.0

Slide 48

Slide 48 text

The Release Team ● Release Management Team Lead ● Release Branch Manager ● Docs Lead ● Features Lead ● Bug Triage Lead ● Test Infra Lead ● Automated Upgrade Testing Lead ● Manual Upgrade Testing Lead ● Testing Lead ● Product Lead

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

Lots of moving parts

Slide 51

Slide 51 text

So lots and lots of testing...

Slide 52

Slide 52 text

Kubernetes Security Release Process JF

Slide 53

Slide 53 text

Bug report Fix team summoned Bug is reported through the disclosure process. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions

Slide 54

Slide 54 text

Bug report Fix team summoned The “Fix Team” is organized and a “Lead” makes sure relevant engineers are included in the discussion. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions

Slide 55

Slide 55 text

Bug report Fix team summoned Patches are posted to a private repo. They are tested and reviewed there by the “fix team.” within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions

Slide 56

Slide 56 text

Bug report Fix team summoned “Fix Lead” emails kubernetes-announce and kubernetes-security-announce informing users a fix will be made available on a certain date. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions

Slide 57

Slide 57 text

Bug report Fix team summoned If the bug is deemed critical enough, downstream distributions will be given notice about the bug and patches ahead of time. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions

Slide 58

Slide 58 text

Bug report Fix team summoned Release is cut from the private repo and patches to the public repo are made after binaries are available. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions

Slide 59

Slide 59 text

Scoring Issues w/ CVSS

Slide 60

Slide 60 text

Scoring Issues w/ CVSS

Slide 61

Slide 61 text

5.0/10 Heartbleed

Slide 62

Slide 62 text

Why is Heartbleed only a 5.0? CVSS 2.0 scoring is failing to take into account quite a few factors: 1. It’s a target of opportunity for attackers: 30-70% of the Internet was effected 2. It’s being actively and successfully exploited on the Internet 3. It’s easy to exploit

Slide 63

Slide 63 text

7.8/10 Dirty COW

Slide 64

Slide 64 text

Areas for Improvement.

Slide 65

Slide 65 text

Rotating the Responsibilities - Challenges - Responsible security process requires non-public work - Running the process is not glamorous - Incentives - Many organizations rely on Kubernetes; the work is important - Critical security announcement will be highly referenced - Next Steps - We need a way of making decisions! Kubernetes Governance Discussion concludes BP

Slide 66

Slide 66 text

Communication to Users - Send an email to kubernetes-announce and call it a day?! - How do you make announcements immediately actionable? - Should vendors be informed ahead of time with a fix? 1 JF

Slide 67

Slide 67 text

Today's Major Parts ● Context Setting ● Open Source and Security Disclosure ● Kubernetes Security Process ● Helping Out

Slide 68

Slide 68 text

Do Something. BP

Slide 69

Slide 69 text

Some Homework - Join Kubernetes Announce! - Review the Kubernetes Security Process - Governance discussion to figure out - Help improve https://github.com/kubernetes/test-infra

Slide 70

Slide 70 text

Thank You! Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)