Boston DevOps Meetup: Agility vs Resilience post Heartbleed

On April 7th Heartbleed was made public knowledge and so a race began to patch the internet. Those using open source software had to take a look at how their stack relied upon OpenSSL and it's sundry of related packages.

DevOps had to weigh the risks of tearing down production systems and endangering SLA's or risking customer data.
Moving forward it has changed how we test and trust software, seek out early warning and run DevOps.

At the heart of the Agile methodology is the ability to make changes to the front and backend of the system and see that code in production relatively quickly. This is in clear opposition to the rigors of compliance.

Transcript

1. Agility vs. Resilience
2. Agility vs. Resilience
3. LUMINOSO 3 Agility vs. Resilience Luminoso Technologies Inc. ! Welcome! Boston DevOps Meetup 5/20/2/14
4. LUMINOSO 4 Agility vs. Resilience Luminoso Technologies Inc. ! ! Paul Wagner pwagner@luminoso.com @pwgnr
5. LUMINOSO 5 Agility vs. Resilience Luminoso Technologies Inc. A bit about Luminoso! •  Luminoso is a cloud based multilingual text analytics solution that provides businesses, researchers, and decision-makers with the ability to quickly analyze and derive insights from large amounts of unstructured text, especially survey responses, consumer feedback, and social media. •  Luminoso enables Fortune 1000 companies to tap into customer experience, fine- tune marketing strategy, elevate customer service and evolve product offerings. •  The intuitive, easy to use nature of the Dashboard, paired with the core technology’s unrivaled ability to understand human language, makes it essential for efficient and impactful text analysis. BLOGS – FORUMS – CALL CENTER TRANSCRIPTS – FOCUS GROUP TRANSCRIPTS SURVEYS – REVIEWS – EMAILS – ARTICLES SOCIAL MEDIA
6. LUMINOSO 6 Agility vs. Resilience Luminoso Technologies Inc.
7. LUMINOSO 7 Agility vs. Resilience Luminoso Technologies Inc. Source:  h*p://xkcd.com/1354/  
8. LUMINOSO Agility vs. Resilience •  12/31/2011 11PM Heartbleed checked into OpenSSL •  11/2013 Intelligence agencies rumored to be using HB •  3/21/14 “Discovered” by Neal Mehta of Google Security •  < 3/31/14 someone tells CloudFlare •  4/1/14 OpenSSL team is informed •  4/2/14 Codenomicon separately discovers the issue •  4/3/14 Akamai patches servers •  4/5/14 Codenomicon purchases the heartbleed.com domain •  4/6/14 RedHat and other vendors are notified •  4/7/14 HeartBleed is announced Source:  h*p://www.smh.com.au/it-­‐pro/security-­‐it/heartbleed-­‐disclosure-­‐?meline-­‐who-­‐knew-­‐what-­‐and-­‐when-­‐20140415-­‐zqurk.html   Heartbleed Timeline! 8 Luminoso Technologies Inc.
9. LUMINOSO 9 Agility vs. Resilience Luminoso Technologies Inc. Where were you?
10. LUMINOSO 10 Agility vs. Resilience Luminoso Technologies Inc. Who markets a bug?! Codenomicon…  that’s  who.  
11. LUMINOSO 11 Agility vs. Resilience Luminoso Technologies Inc. Bad bugs are discovered everyday!
12. LUMINOSO Agility vs. Resilience Who knew?! •  Google (3/21 or prior) •  CloudFlare (3/31 or prior) •  OpenSSL (4/1) •  Codenomicon (4/2) •  National Cyber Security Centre Finland (4/3) •  Akamai (4/4 or earlier) •  Facebook (?) ?! 12 Luminoso Technologies Inc. Source:  h*p://www.smh.com.au/it-­‐pro/security-­‐it/heartbleed-­‐disclosure-­‐?meline-­‐who-­‐knew-­‐what-­‐and-­‐when-­‐20140415-­‐zqurk.html  
13. LUMINOSO Agility vs. Resilience Who didn’t know?! •  AWS •  Yahoo •  Twitter •  Canonical •  Cisco •  Juniper •  GitHub •  Pinterest •  Flickr •  GoDaddy •  DropBox •  Luminoso You get the idea.! Source:  h*p://www.smh.com.au/it-­‐pro/security-­‐it/heartbleed-­‐disclosure-­‐?meline-­‐who-­‐knew-­‐what-­‐and-­‐when-­‐20140415-­‐zqurk.html   ?! Luminoso Technologies Inc. 13
14. LUMINOSO 14 Agility vs. Resilience Luminoso Technologies Inc. Testing 1/3   #  Quick  and  dirty  demonstra?on  of  CVE-­‐2014-­‐0160  by  Jared  Stafford  (jspenguin@jspenguin.org)   Which worked… Sort of…
15. LUMINOSO 15 Agility vs. Resilience Luminoso Technologies Inc. Testing 2/3! https://filippo.io/Heartbleed/
16. LUMINOSO 16 Agility vs. Resilience Luminoso Technologies Inc. Testing 3/3!
17. LUMINOSO 17 Agility vs. Resilience Luminoso Technologies Inc. Confidential Remediation! String of issues hit. •  OS versions •  Package dependencies •  Our SaaS providers •  Testing •  Certificate revocation Heartbleed  released  
18. LUMINOSO 18 Agility vs. Resilience Luminoso Technologies Inc. How to fix? Patch
19. LUMINOSO 19 Agility vs. Resilience Luminoso Technologies Inc. How to fix?! Disable heartbeat
20. LUMINOSO 20 Agility vs. Resilience Luminoso Technologies Inc. How to fix?! Remove OpenSSL and use something else
21. LUMINOSO 21 Agility vs. Resilience Luminoso Technologies Inc. How to fix?! Replace vulnerable attack surfaces with patched systems
22. LUMINOSO 22 Agility vs. Resilience Luminoso Technologies Inc. How to fix?! Reissue those certificates
23. LUMINOSO 23 Agility vs. Resilience Luminoso Technologies Inc. How to fix?! As  of  5/9/14   Source:  h*p://news.netcraX.com/archives/2014/05/09/keys-­‐leX-­‐unchanged-­‐in-­‐many-­‐heartbleed-­‐replacement-­‐cer?ficates.html  
24. LUMINOSO 24 Agility vs. Resilience Luminoso Technologies Inc. Bad bugs are discovered everyday!
25. LUMINOSO 25 Agility vs. Resilience Luminoso Technologies Inc. Emerging issues! What uses OpenSSL? OpenVPN Nginx FTPS
26. LUMINOSO 26 Agility vs. Resilience Luminoso Technologies Inc. Emerging issues Monitoring the dependencies of a whole stack is difficult.
27. LUMINOSO 27 Agility vs. Resilience Luminoso Technologies Inc. Emerging issues! Dependency Graphs? JuJu, Beanstalk, Salt, Puppet, Chef, etc have limitations and become dependencies themselves. Docker isn’t ready for production.
28. LUMINOSO 28 Agility vs. Resilience Luminoso Technologies Inc. Emerging issues! Getting early warning
29. LUMINOSO 29 Agility vs. Resilience Luminoso Technologies Inc. Emerging issues! SaaS products that the Business uses
30. LUMINOSO 30 Agility vs. Resilience Luminoso Technologies Inc. Emerging issues! Availability vs Data Security
31. LUMINOSO 31 Agility vs. Resilience Luminoso Technologies Inc. Discussion time! ! ! ! !
32. LUMINOSO 32 Agility vs. Resilience Luminoso Technologies Inc. Agility vs. Resilience ! Security  (system  down)   Up4me  (99.999%)   Trust   SLA’s   A  true  fix   A  hack   Documented  fix   Less  disrup?on   Patch  opportunity   Key  resources  will  get  to  it   2nd  level  a*ack   Data  loss  is  cost  of  doing  business   Happy  customers  
33. LUMINOSO 33 Agility vs. Resilience Luminoso Technologies Inc. Organizational Challenges! What SaaS is your company using? Who can authorize extensive down time? How are customers notified? Who can shut down production?
34. LUMINOSO 34 Agility vs. Resilience Luminoso Technologies Inc. Agility vs. Resilience ! Ultimately It depends on your business model
35. LUMINOSO 35 Agility vs. Resilience Luminoso Technologies Inc. How can an organization avoid the next Heartbleed?! Contribute “The  key  here  is  about  actually  commi`ng  resources.    It’s  oXen  amazing  (to  me)  how  quickly  the  commitment  of  actual  resources   will  focus  an  otherwise  overly  broad  discussion.    If  you  start  by  asking  people  to  list  all  of  the  OSS  technology  that  is  cri?cal  to  the   business,  you  will  likely  end  up  with  a  massive  list.    Now  tell  them  that  they  are  going  to  have  to  commit  headcount,  budget  to  support   every  piece  of  technology  on  the  list  (and  jus?fy  it)  –  it  will  likely  shrink  rapidly.”   -­‐  Adam  Comerford
36. LUMINOSO 36 Agility vs. Resilience Luminoso Technologies Inc. How can an organization avoid the next Heartbleed?! Create plans of action and checklists
37. LUMINOSO 37 Agility vs. Resilience Luminoso Technologies Inc. How can an organization avoid the next Heartbleed?! Get on the relevant mailing lists and read them
38. LUMINOSO 38 Agility vs. Resilience Luminoso Technologies Inc. How can an organization avoid the next Heartbleed?! Map the Network artisanal vLANs and all
39. LUMINOSO 39 Agility vs. Resilience Luminoso Technologies Inc. How can an organization avoid the next Heartbleed?! SAML 2.0 Integration
40. LUMINOSO 40 Agility vs. Resilience Luminoso Technologies Inc. How can an organization avoid the next Heartbleed?! Watch out for commits on NYE
41. LUMINOSO 41 Agility vs. Resilience Luminoso Technologies Inc.
42. LUMINOSO 42 Agility vs. Resilience Luminoso Technologies Inc. Hypothetical worst case next Heartbleed?! Hardware?
43. LUMINOSO 43 Agility vs. Resilience Luminoso Technologies Inc. Hypothetical worst case next Heartbleed? systemd iptables