So you've been hacked. Now what?

How to handle an exploit and/or data protection violation, in the wake of the GDPR regulations.

Transcript

1. Frédéric G. MARAND
 http://www.osinet.fr/ So you’ve been hacked…
 now what ? 1
2. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 Table of Contents 1 Intro : setting the stage 2 Snapshotting 3 Maintaining presence 4 Crisis communication 5 Rebuild, don’t repair 6 Using forensics tools 7 Back online 2
3. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1 
 Intro X
4. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1 Some fact checking first • In this room … • Whose site has been hacked already ? 3 Each question is harder than the previous one.
5. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1 Some fact checking first • In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? 4
6. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1 Some fact checking first • In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ? 5
7. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1 Some fact checking first • In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ? • Who actually has a GDPR-compliant business continuity plan ? 6
8. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1 Some fact checking first • In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ? • Who actually has a GDPR-compliant business continuity plan ? • Who read node 2365547 ? 7
9. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 8 • It’s the basis for this presentation. • On the plus side, it can be regularly updated by the community • On the minus site, it’s less detailed
10. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.2 Can you say that again ? 9 I.A.N.A.L. (and you probably ain’t either)
 So be sure to get one ! Tip 1 Non-law professionals are usually not allowed to provide legal help. Tip 2 And even where allowed, it’s not a good idea. Tip 3 Get one long before that day so you can negotiate the agreement
11. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.3 Whence do I speak ? • Drupal org member since 2005 (fgm) • Drupal consultant, not a site building agency • Worked on fixing broken(-in) sites since 2008 • Auditing • Fixing technical flaws • Addressing intrusions / exploits • Mostly Media and Government sites (.fr) • « Provisional member » on the Security Team 10
12. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.4 Choose your own adventure • 10:00 The daily scrum has just begun. • 10:01 Phones rings : someone noticed your site has been defaced and is warning you • 10:02 Twitter and Reddit start buzzing • 10:05 Phones ring all over the place, journalists and CxO types on the other end, asking about GDPR Art. 33/34 ; your mailbox is filling with warnings • What is your next step ? 11 • Variant: only tech noticed. Less stress, but same requirements • art 33 : notify control authority • art 34: notify affected individuals
13. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.5 Get ready • Notepad 1 : discovery log • all your work steps, to the
 tiniest • all your findings / observations • with timestamps and numbers 12 • Notepad 2 : remedies ideas • all your ideas for fixing the breach • all your ideas for further hardening • cross-refer notepad 1 numbers Tip1 : same pad, start from the end Tip2 : paper, not digital
14. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 2
 Snapshots X
15. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 2.1 Forensic copy : why ? • Obvious temptation: • restore & resume • Still vulnerable • So you need to diagnose • Analyzing means modifying • So preserve the « crime scene » • Snapshot everything 13 Technically, you’re more vulnerable than originally, since someone actually knows that: • you are affected by the vulnerability • you did not fix it
16. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 2.2 Snapshots : pull the plug ? • Prevents interference • Shutdown handlers, SIGPWR • Self-destructing code on network loss • Easy on VMs, cloud instances • No need to pull the plug 14 • But… • Bare remote servers • Further data loss • Journaled FS • Databases • Service interruption Example met in 05/2018: the Jenkins exploit code removed itself from disk immediately after launch, to avoid detection by disk scanners, and reloaded via crontab. Scanning the disk after pulling the plug would have shown nothing.
17. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 2.3 Snapshots : everything == ? • Not just the main DB
 • Reverse Proxy logs • Web fronts • DB servers • File servers • Also…
 • External logs (SaaS) • External transactions • FW/IDS/WAF logs 15 The site might be just another attack vector, not the main target • Case in point in the Jenkins hack: the bot scanned the disk for usernames / ssh keys / host names, uploading them to its control server and attempting exploit itself. • Are you sure your servers only store the PUBLIC ssh keys, no PRIVATE ? Even the bastion host ? On all user accounts ?
18. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3 
 Presence X So, what kind of presence to maintain after the exploit ?
19. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3.1 Maintaining presence #1 • Pros • Don’t tip off hackers: you’re one step ahead until they notice • Keep on generating short- term value
 • Cons • Damage increase • Responsibility • Legal (GDPR) • Financial • Moral 16 As though intrusion had not been detected
20. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3.2 Attacker workflows (intermezzo) Modern: multistep flow • Break in • Dig for gold • Implant zombie • Wait for implant migration to archives • Activate • Profit
 Amateur: Need for Speed • Use exploit ASAP • While it lasts • Usually least loss (miners) Art crime: hide the act • Valuable content • Identity data • Close the door 17 (➡ forensics) (➡ restore)
21. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3.3 Maintaining presence #2 • Limited static site • Best with prior work • Minimal subset • Possibly taken from RP cache • Very little load • Can run off RP heads
 • Working limited (read-only) site • Alternate infra • Alternate tech • Updates ? • Content created during crisis 18 Safe fallback mode Always easier if planned from the onset. Can help with GDPR Art 41.2b
22. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3.4 Maintaining presence #3 • Social networks and status sites • Always there • Also authoritative for audience • Still needs some preparation • Accounts access • Inclusion in long-term communication • B2C: Facebook / Instagram pages • B2B: Twitter, status sites 19 When all else fails https://xenomorph01.deviantart.com/art/Deadpool-tis-but-a-scratch-479338640 LinkedIn ? Doesn’t feel appropriate, but might.
23. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 4
 Communication X
24. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 4.1 Communicating : upstream • Internal stakeholders • DPO if any personal data are involved • CxO level in most cases • Gag orders ? Protection may exist. But follow the rules. • France: whistleblower protection in Sapin 2 (limited) • Italy (banks): Dec. 385 01/09/93 sect 52bis • US (contractors): Anti-SLAPP • Many countries have similar rules 20
25. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 4.2 : Communication : C-level • DPO (start here), Legal counsel • Crisis Management specialists (costs…) • Law enforcement • National data protection agencies within 72h (GDPR Art. 33, Rec. 85) • EU countries typically have« cybercrime » units like (FR) ANSSI • Other sites • On same server • On same network • Online business partners ➡ GDPR: Data Processors or 3rd parties ? 21 * The 72h limit is only for sensitive data or high risk exploits * Business partners also includes clients
26. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 4.3 Communication : privacy • Often personal data leaks • will happen, or... • unprovable they did not • Operational constraints • Commerce : PCI/DSS 
 (12 steps etc) • Health : (US) HIPAA 
 Subtitle D E2.80.93 • Public image damage control
 • GDPR Conflicting rules • 72h delay for sensitive exploits (Art. 33.1), or justification needed • Police operations need time. Per GDPR, police prevails (Rec. 86) 22
27. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5 
 Rebuild X
28. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5.1 Rebuild : keep, rollback or ..? • Restore and restart same ? • Still just as vulnerable, just more so • Keep and fix ? • lots of time and effort reviewing • never completely trusted : not just Drupal
 • Throw away ? • Event sites, past lines of biz, post-M&A... • Can a static version suffice ? • From RP snapshots : recent content • Best: rebuild from sources + restore backups 23 Best = slide 5.3
29. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5.2 Rebuild : restore content • Needs backups from before the hack • Do you know when it happened ? Hint: « modern workflow » • GFS, continuous incremental, 15 min ? • How much can you lose ? • FLOSS solutions : Amanda, Bacula, custom… • Unprepared emergency ? • Preproduction, CI builds… 24 (➡ modern workflow) • In the « modern workflow » attacker doesn’t move until a long time has elapsed, increasing the cost of rebuild because of new content (e.g. transactions) creation, and the likelihood backups will be erased. • If there is no code in the DB and everything is industrialized and config always moves up and content down, reinstall is a breeze • Zmanda provides EE for Amanda
30. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5.3 Rebuild : sources + export • Easy and reliable, but assumes : • Code-driven development process • Reliable data export system in place • Accessible content exports (e.g. flat files, journals) • Content + assets repositories • Still need to add the fixes • Delay can be a problem on high-volume sites • Bulk handling, Incremental loading 25
31. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5.4 Rebuild : other cases • Ad hoc « traditional » build process • Longer, less reliable • Too long to be a chance to fix the process • From scratch • Too long in most (all ?) cases • Do it as a complement after the fix • Not NOW 26
32. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6 
 Forensics X
33. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6 Forensics : switching hats 27
34. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.1 Forensics : first, think ! • How did you discover the hack ? • What did it take to succeed ? • Cast your net wide, think big • « Unlikely » vs « impossible » • Priority : • Easiest attacks first • OWASP 10 • GIYF : search for notepad 1 patterns 28 
 « Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. » 
 
 Arthur Conan Doyle «The sign of the four »

35. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 • /anything/ may be erased after success • But most of the time, not /everything/ will • Anything you do leaves its own traces • Work on copies of the snapshots • You can restart from fresh copies anytime • There maybe more than one exploit • « Cleaner exploits » after Drupalgeddon 1 • « art crime » workflow 6.2 Forensics : keep in mind 29 (➡art crime) * |v’x - vx|Δpx ≍ ℏ/Δt : Observer effect equation, Niels Bohr, 1928 (ℏ : Planck constant) * https://en.wikipedia.org/wiki/Observer_effect_(physics)
36. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.3 Forensics : classic flaws • Code files : • lax permissions • filesystem traversal issues • Remote payload execution by upload • Nginx without extra hardening • .htaccess won’t do much good • In-DB PHP • PHP module • Eval-uated code 30
37. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.4 Forensics : outer droppings • Filesystem : • <user>/www-data outside /sites • www-data/www-data suspicious • « x » bit (0111) on files below docroot • timestamps • ts(outside sites/*/files) = ts(install) • ts(exploits) > ts(install) • meld with a fresh build from sources • Also check outside docroot / project root 31
38. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.5 Forensics : Drupal modules • Code signing/diffing : • Hacked! Limitations • D7 : md5check, file_integrity • Finding DB PHP • OSInet QA (github) • Misc • security_review 32 Hacked won’t help with custom code: compare with VCS
39. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.6 Forensics : DB scans • Quick wins : • users_field_data.email != users_field_data.init (users on D6/D7) • review roles, accounts with admin roles • On corporate sites, users_field_data.email domains • match users accounts with SSO data, directories • Diff DB snapshot with live • D7: menu_router : file_put_contents, assert • D8: use vendor/bin/drupal debug:router instead of DB • Altova DatabaseSpy content compare 33
40. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.7 Forensics : session data • Sessions should be in persistent storage • Remember when you pulled the plug • Were your sessions in Memcache ? in-memory Redis ? • Quick checks: • sessions.timestamp vs users_field_data : created / changed / access / login • for intranets : sessions.hostname 34
41. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.8 Forensics : logs • You use off-site logs, right ? • SaaS : Loggly, Logmatic, Logsene, Logz.io, Papertrail, Scalyr…. • Homegrown remote ELK • GDPR • Data Processor constraints (Rec. 81) • Record of processing activities (Art. 30)
 • Still on-site ? Read-write ? • dblog {watchdog} • syslog → redirects chain • mongodb_watchdog • redis_watchdog • GELF/Graylog, Logstash • Application/WS logs 35
42. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.9 Forensics : sleuth tools • Software • Guidance Software : Encase • AccessData : Forensics Toolkit (FTK) • Certified consultants • National cybercrime units 36 cybercrime units may help, or suggest approved consultants: tools better used by pros.
43. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7
 Back online X
44. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.1 Live again : restoring production • On preprod, recheck notepad 1 findings vs new build • Usually, reset passwords. • On D8, use mass_pwreset • On D7/MySQL: • update users 
 set pass = concat(‘ZZZ', sha(concat(pass, md5(rand())))); • Prepare the GDPR Art. 34 exploit report • Prepare marketing/social copy • Prepare for the future 37
45. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.2 L8R : future-readiness 38
46. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.3 L8R: be prepared • Developer education on security • Security Team mailing list • https://twitter.com/drupalsecurity • https://www.drupal.org/security/rss.xml • http://crackingdrupal.com/ ? • National support • GDPR requires national authorities to provide education (Art. 57, Rec. 122, 132) • Exists outside GDPR (FR: ANSSI MOOC)
 https://secnumacademie.gouv.fr/ 39
47. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.4 L8R : disaster prevention • Security process • Analyse sec. releases to understand fixes • Look for similar flaw in custom code • Take part in Drupal core / contrib to acquire expertise • Quality process • Systematic peer code reviews • Code-driver maintenance + dev process • Automatic quality tools in CI • Contrib updates scheduling 40
48. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.5 Continuous improvement • « You can’t improve what you don’t measure » • Get time metrics from notepad 1 • Build contingency plan from notepad 2 • Plan for periodic intrusion simulations • GDPR Art. 32.1d: « …the controller and the processor shall implement […] a process for regularly testing, assessing and evaluating the effectiveness of technical and organisational measures for ensuring the security of the processing » 41
49. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 42 
 Drupal, faster, safer http://www.osinet.fr/ @OSInet https://www.linkedin.com/company/osinet/ https://goo.gl/X9LEzz Rate this talk: