Heartbleed at Acquia - Speaker Deck

Heartbleed at Acquia

by Marc Seeger

Slide 1

Slide 1 text

Marc Seeger (@rb2k)  Boston Devops Meetup  May 20th 2014 at

Slide 2

Slide 2 text

Act 1: Technology

Slide 3

Slide 3 text

How it all started 7:24 PM

Slide 4

Slide 4 text

How it all started 7:30 PM

Slide 5

Slide 5 text

How it all started 7:26 PM

Slide 6

Slide 6 text

How it all started 7:33 PM

Slide 7

Slide 7 text

How it all started

Slide 8

Slide 8 text

Quick risk assessment Lucid: [00:35:27] [email protected]:~# openssl version OpenSSL 0.9.8k 25 Mar 2009 ! Precise: [00:34:37] [email protected]:~# openssl version OpenSSL 1.0.1 14 Mar 2012

Slide 9

Slide 9 text

Where’s Waldo OpenSSL 8000 EC2 Machines: - 99.9% of them puppetized - Candidates: - Balancers - SVN Servers - Appliances - ELBs - 3rd party AMIs - Unique little snowﬂakes  (Jira, Crucible,…)

Slide 10

Slide 10 text

Let the patching begin

Slide 11

Slide 11 text

Rollout Australia: ! Con: - Spiders - Snakes ! Pro: - Ops is awake

Slide 12

Slide 12 text

Rollout

Slide 13

Slide 13 text

Scan www

Slide 14

Slide 14 text

Waiting on ELBs…

Slide 15

Slide 15 text

Internal Certiﬁcates

Slide 16

Slide 16 text

Suddenly: “reverse” Heartbleed

Slide 17

Slide 17 text

Act 2: Communication

Slide 18

Slide 18 text

Internal • Pre-determined chat rooms • Dial-in conference bridges • A communication plan Thanks SSAE-16, PCI and FedRAMP… I guess :)

Slide 19

Slide 19 text

Statuspage + Twitter * Powered by StatusPage.io *

Slide 20

Slide 20 text

Documentation https://docs.acquia.com/articles/heartbleed-acquia-cloud

Slide 21

Slide 21 text

Proactive communication Phone calls by Acquia support, TAMs, …

Slide 22

Slide 22 text

Since then: Post mortem

Slide 23

Slide 23 text

Since then: Incident Commander (shamelessly stolen from Heroku) http://en.wikipedia.org/wiki/Incident_command_system