KubeCon EU 2017: Dancing on the Edge of a Volcano

KubeCon EU 2017: Dancing on the Edge of a Volcano

2786cdedd6e0eaa34b64b17e1cea81b9?s=128

Brandon Philips

March 31, 2017
Tweet

Transcript

  1. Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)

  2. Dancing on the Edge of a Volcano Kubernetes Security Processes

    Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)
  3. Today's Major Parts • Context Setting • Open Source and

    Security Disclosure • Kubernetes Security Process • Helping Out
  4. Today's Major Parts • Context Setting • Open Source and

    Security Disclosure • Kubernetes Security Process • Helping Out
  5. Computer Security in 2017 BP

  6. None
  7. None
  8. None
  9. Why is Server Security Important?

  10. 3,424,000,000 Internet Users Source: http://www.internetlivestats.com/internet-users/

  11. 29,000,000 Software Developers and IT Practitioners Source: https://www.infoq.com/news/2014/01/IDC-software-developers

  12. We are Outnumbered!

  13. 238,975,082 New Internet Users in 2016 Source: http://www.internetlivestats.com/internet-users/

  14. None
  15. Documents Commerce Communications

  16. ~100,000,000 Servers Worldwide

  17. 3 Per Person In the Software and IT Industry

  18. 100+ Per Person At the Internet Giants

  19. Kubernetes.

  20. Today's Major Parts • Context Setting • Open Source and

    Security Disclosure • Kubernetes Security Process • Helping Out
  21. Challenges with Open Source and Security Disclosure JF

  22. An open process Need to be open to vendors and

    community but also keep information from those who can do harm before bugs are fixed.
  23. GPG: Do you even?

  24. Can we make everyone happy? Researchers, community, vendors… All sides

    want something different.
  25. Users Expectations - Want software without bugs - Want to

    be able to upgrade without breaking changes - Want to be let know when security fixes are made so they can act on them accordingly
  26. Original Reporter Expectations - Want “a human to say acknowledged”

    - Want updates on status, transparency - Want bug to be fixed in a certain timeframe
  27. Project Expectations - Want bugs to be fixed quickly and

    embargo to be kept secret until time of disclosure - Want upgrades to remain seamless - Want to continue to instill trust with users
  28. How to get people to upgrade? - Challenges with making

    sure people know - Use mailing lists, establish a pattern for the project that allows people to follow along - How to handle if very severe… pray you don’t have someone make up a fun name
  29. Challenge: name the vulnerability

  30. How bugs get reported... I found a bug… should I

    be evil or let upstream know... Let me google “X security disclosure” or I’ll just try to email security@
  31. How bugs get reported... I’ll lookup the project’s recommended method

    for disclosures I found a bug… I’ll file an issue on the repo...
  32. I found a bug… I’ll file an issue on the

    repo... - Not everyone can recognize the severity of a bug at first glance. - Allow for mistakes and have a process for handling them in a way that it will not confuse the original reporter or cause unnecessary attention. Mistakes happen
  33. Learning from Other Communities BP

  34. The Common Stuff

  35. General Recommendations We Got - Make documentation Googleable (X security

    disclosure) - Have a dedicated response team - Coordinate via a mailing list - Don't force the use of GPG - Have internal documentation on the process - Use a severity leveling system - Request CVEs when appropriate*
  36. Docker: Document everything JF

  37. Docker: Tooling for everything JF

  38. Linux Kernel: Don't Negotiate Set timeline expectations; don't negotiate on

    embargos BP
  39. Node.js: Help the Ecosystem BP Fix the core but also

    help track the ecosystem and notify
  40. Go: Early warning systems JF Give people a heads up

    that a vulnerability is coming
  41. Go: Small team, explicit contacts Mailing lists fail, people don't

    read email, list everyone. JF
  42. Today's Major Parts • Context Setting • Open Source and

    Security Disclosure • Kubernetes Security Process • Helping Out
  43. Where we are today BP

  44. Kubernetes Normal Release Process BP

  45. Kubernetes Release Process k8s 1.5.x k8s 1.6.0 k8s 1.6.x Bug

    fixes & security patches Features Features k8s 1.7.0 k8s 1.7.x Bug fixes & security patches Features
  46. Kubernetes Release Process k8s 1.5.x k8s 1.6.0 k8s 1.6.x Bug

    fixes & security patches Features Features Bug fixes & security patches ~9 Months*
  47. The Kubernetes v1.6 Release Schedule • … 4-6 weeks of

    coding … • Jan 30th (Monday) v1.6.0-alpha.1 • Feb 13th (Monday) v1.6.0-alpha.2 • Feb 20th (Monday) Cut release-1.6 branch and v1.6.0-beta.0 • Feb 27th (Monday) Start code freeze • Feb 28th (Tuesday) v1.6.0-beta.1 • March 7th (Tuesday) - v1.6.0-beta.2 • March 14th (Tuesday) Lift code freeze and v1.6.0-rc.1 • March 22nd (Wednesday) - v1.6.0
  48. The Release Team • Release Management Team Lead • Release

    Branch Manager • Docs Lead • Features Lead • Bug Triage Lead • Test Infra Lead • Automated Upgrade Testing Lead • Manual Upgrade Testing Lead • Testing Lead • Product Lead
  49. None
  50. Lots of moving parts

  51. So lots and lots of testing...

  52. Kubernetes Security Release Process JF

  53. Bug report Fix team summoned Bug is reported through the

    disclosure process. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
  54. Bug report Fix team summoned The “Fix Team” is organized

    and a “Lead” makes sure relevant engineers are included in the discussion. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
  55. Bug report Fix team summoned Patches are posted to a

    private repo. They are tested and reviewed there by the “fix team.” within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
  56. Bug report Fix team summoned “Fix Lead” emails kubernetes-announce and

    kubernetes-security-announce informing users a fix will be made available on a certain date. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
  57. Bug report Fix team summoned If the bug is deemed

    critical enough, downstream distributions will be given notice about the bug and patches ahead of time. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
  58. Bug report Fix team summoned Release is cut from the

    private repo and patches to the public repo are made after binaries are available. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
  59. Scoring Issues w/ CVSS

  60. Scoring Issues w/ CVSS

  61. 5.0/10 Heartbleed

  62. Why is Heartbleed only a 5.0? CVSS 2.0 scoring is

    failing to take into account quite a few factors: 1. It’s a target of opportunity for attackers: 30-70% of the Internet was effected 2. It’s being actively and successfully exploited on the Internet 3. It’s easy to exploit
  63. 7.8/10 Dirty COW

  64. Areas for Improvement.

  65. Rotating the Responsibilities - Challenges - Responsible security process requires

    non-public work - Running the process is not glamorous - Incentives - Many organizations rely on Kubernetes; the work is important - Critical security announcement will be highly referenced - Next Steps - We need a way of making decisions! Kubernetes Governance Discussion concludes BP
  66. Communication to Users - Send an email to kubernetes-announce and

    call it a day?! - How do you make announcements immediately actionable? - Should vendors be informed ahead of time with a fix? 1 JF
  67. Today's Major Parts • Context Setting • Open Source and

    Security Disclosure • Kubernetes Security Process • Helping Out
  68. Do Something. BP

  69. Some Homework - Join Kubernetes Announce! - Review the Kubernetes

    Security Process - Governance discussion to figure out - Help improve https://github.com/kubernetes/test-infra
  70. Thank You! Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)