So you've been hacked. Now what?

150

So you've been hacked. Now what?

How to react after a hack successfully exploited a server, and possibly breached access to Personally Identifiable Data, in the wake of the GDPR.

This is the presentation-ready version, directly usable for presenting. The other version of this presentation includes the presenter notes.

9770614c66b56331b6947c79b622a7af?s=128

Frédéric G. MARAND

June 08, 2018
Tweet

Transcript

  1. 2.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 Table

    of Contents 1 Intro : setting the stage 2 Snapshotting 3 Maintaining presence 4 Crisis communication 5 Rebuild, don’t repair 6 Using forensics tools 7 Back online 2
  2. 3.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1

    Some fact checking first • In this room … • Whose site has been hacked already ? 3
  3. 4.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1

    Some fact checking first • In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? 4
  4. 5.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1

    Some fact checking first • In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ? 5
  5. 6.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1

    Some fact checking first • In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ? • Who actually has a GDPR-compliant business continuity plan ? 6
  6. 7.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.1

    Some fact checking first • In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ? • Who actually has a GDPR-compliant business continuity plan ? • Who read node 2365547 ? 7
  7. 9.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.2

    Can you say that again ? 9 I.A.N.A.L. (and you probably ain’t either)
 So be sure to get one !
  8. 10.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.3

    Whence do I speak ? • Drupal org member since 2005 (fgm) • Drupal consultant, not a site building agency • Worked on fixing broken(-in) sites since 2008 • Auditing • Fixing technical flaws • Addressing intrusions / exploits • Mostly Media and Government sites (.fr) • « Provisional member » on the Security Team 10
  9. 11.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.4

    Choose your own adventure • 10:00 The daily scrum has just begun. • 10:01 Phones rings : someone noticed your site has been defaced and is warning you • 10:02 Twitter and Reddit start buzzing • 10:05 Phones ring all over the place, journalists and CxO types on the other end, asking about GDPR Art. 33/34 ; your mailbox is filling with warnings • What is your next step ? 11
  10. 12.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 1.5

    Get ready • Notepad 1 : discovery log • all your work steps, to the
 tiniest • all your findings / observations • with timestamps and numbers 12 • Notepad 2 : remedies ideas • all your ideas for fixing the breach • all your ideas for further hardening • cross-refer notepad 1 numbers
  11. 13.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 2.1

    Forensic copy : why ? • Obvious temptation: • restore & resume • Still vulnerable • So you need to diagnose • Analyzing means modifying • So preserve the « crime scene » • Snapshot everything 13
  12. 14.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 2.2

    Snapshots : pull the plug ? • Prevents interference • Shutdown handlers, SIGPWR • Self-destructing code on network loss • Easy on VMs, cloud instances • No need to pull the plug 14 • But… • Bare remote servers • Further data loss • Journaled FS • Databases • Service interruption
  13. 15.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 2.3

    Snapshots : everything == ? • Not just the main DB
 • Reverse Proxy logs • Web fronts • DB servers • File servers • Also…
 • External logs (SaaS) • External transactions • FW/IDS/WAF logs 15 The site might be just another attack vector, not the main target
  14. 16.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3.1

    Maintaining presence #1 • Pros • Don’t tip off hackers: you’re one step ahead until they notice • Keep on generating short- term value
 • Cons • Damage increase • Responsibility • Legal (GDPR) • Financial • Moral 16 As though intrusion had not been detected
  15. 17.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3.2

    Attacker workflows (intermezzo) Modern: multistep flow • Break in • Dig for gold • Implant zombie • Wait for implant migration to archives • Activate • Profit
 Amateur: Need for Speed • Use exploit ASAP • While it lasts • Usually least loss (miners) Art crime: hide the act • Valuable content • Identity data • Close the door 17 (➡ forensics) (➡ restore)
  16. 18.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3.3

    Maintaining presence #2 • Limited static site • Best with prior work • Minimal subset • Possibly taken from RP cache • Very little load • Can run off RP heads
 • Working limited (read-only) site • Alternate infra • Alternate tech • Updates ? • Content created during crisis 18 Safe fallback mode Always easier if planned from the onset. Can help with GDPR Art 41.2b
  17. 19.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 3.4

    Maintaining presence #3 • Social networks and status sites • Always there • Also authoritative for audience • Still needs some preparation • Accounts access • Inclusion in long-term communication • B2C: Facebook / Instagram pages • B2B: Twitter, status sites 19 When all else fails https://xenomorph01.deviantart.com/art/Deadpool-tis-but-a-scratch-479338640
  18. 20.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 4.1

    Communicating : upstream • Internal stakeholders • DPO if any personal data are involved • CxO level in most cases • Gag orders ? Protection may exist. But follow the rules. • France: whistleblower protection in Sapin 2 (limited) • Italy (banks): Dec. 385 01/09/93 sect 52bis • US (contractors): Anti-SLAPP • Many countries have similar rules 20
  19. 21.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 4.2

    : Communication : C-level • DPO (start here), Legal counsel • Crisis Management specialists (costs…) • Law enforcement • National data protection agencies within 72h (GDPR Art. 33, Rec. 85) • EU countries typically have« cybercrime » units like (FR) ANSSI • Other sites • On same server • On same network • Online business partners ➡ GDPR: Data Processors or 3rd parties ? 21
  20. 22.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 4.3

    Communication : privacy • Often personal data leaks • will happen, or... • unprovable they did not • Operational constraints • Commerce : PCI/DSS 
 (12 steps etc) • Health : (US) HIPAA 
 Subtitle D E2.80.93 • Public image damage control
 • GDPR Conflicting rules • 72h delay for sensitive exploits (Art. 33.1), or justification needed • Police operations need time. Per GDPR, police prevails (Rec. 86) 22
  21. 23.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5.1

    Rebuild : keep, rollback or ..? • Restore and restart same ? • Still just as vulnerable, just more so • Keep and fix ? • lots of time and effort reviewing • never completely trusted : not just Drupal
 • Throw away ? • Event sites, past lines of biz, post-M&A... • Can a static version suffice ? • From RP snapshots : recent content • Best: rebuild from sources + restore backups 23
  22. 24.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5.2

    Rebuild : restore content • Needs backups from before the hack • Do you know when it happened ? Hint: « modern workflow » • GFS, continuous incremental, 15 min ? • How much can you lose ? • FLOSS solutions : Amanda, Bacula, custom… • Unprepared emergency ? • Preproduction, CI builds… 24 (➡ modern workflow)
  23. 25.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5.3

    Rebuild : sources + export • Easy and reliable, but assumes : • Code-driven development process • Reliable data export system in place • Accessible content exports (e.g. flat files, journals) • Content + assets repositories • Still need to add the fixes • Delay can be a problem on high-volume sites • Bulk handling, Incremental loading 25
  24. 26.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 5.4

    Rebuild : other cases • Ad hoc « traditional » build process • Longer, less reliable • Too long to be a chance to fix the process • From scratch • Too long in most (all ?) cases • Do it as a complement after the fix • Not NOW 26
  25. 28.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.1

    Forensics : first, think ! • How did you discover the hack ? • What did it take to succeed ? • Cast your net wide, think big • « Unlikely » vs « impossible » • Priority : • Easiest attacks first • OWASP 10 • GIYF : search for notepad 1 patterns 28 
 « Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. » 
 
 Arthur Conan Doyle «The sign of the four »

  26. 29.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 •

    /anything/ may be erased after success • But most of the time, not /everything/ will • Anything you do leaves its own traces • Work on copies of the snapshots • You can restart from fresh copies anytime • There maybe more than one exploit • « Cleaner exploits » after Drupalgeddon 1 • « art crime » workflow 6.2 Forensics : keep in mind 29 (➡art crime)
  27. 30.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.3

    Forensics : classic flaws • Code files : • lax permissions • filesystem traversal issues • Remote payload execution by upload • Nginx without extra hardening • .htaccess won’t do much good • In-DB PHP • PHP module • Eval-uated code 30
  28. 31.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.4

    Forensics : outer droppings • Filesystem : • <user>/www-data outside /sites • www-data/www-data suspicious • « x » bit (0111) on files below docroot • timestamps • ts(outside sites/*/files) = ts(install) • ts(exploits) > ts(install) • meld with a fresh build from sources • Also check outside docroot / project root 31
  29. 32.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.5

    Forensics : Drupal modules • Code signing/diffing : • Hacked! Limitations • D7 : md5check, file_integrity • Finding DB PHP • OSInet QA (github) • Misc • security_review 32
  30. 33.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.6

    Forensics : DB scans • Quick wins : • users_field_data.email != users_field_data.init (users on D6/D7) • review roles, accounts with admin roles • On corporate sites, users_field_data.email domains • match users accounts with SSO data, directories • Diff DB snapshot with live • D7: menu_router : file_put_contents, assert • D8: use vendor/bin/drupal debug:router instead of DB • Altova DatabaseSpy content compare 33
  31. 34.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.7

    Forensics : session data • Sessions should be in persistent storage • Remember when you pulled the plug • Were your sessions in Memcache ? in-memory Redis ? • Quick checks: • sessions.timestamp vs users_field_data : created / changed / access / login • for intranets : sessions.hostname 34
  32. 35.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.8

    Forensics : logs • You use off-site logs, right ? • SaaS : Loggly, Logmatic, Logsene, Logz.io, Papertrail, Scalyr…. • Homegrown remote ELK • GDPR • Data Processor constraints (Rec. 81) • Record of processing activities (Art. 30)
 • Still on-site ? Read-write ? • dblog {watchdog} • syslog → redirects chain • mongodb_watchdog • redis_watchdog • GELF/Graylog, Logstash • Application/WS logs 35
  33. 36.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 6.9

    Forensics : sleuth tools • Software • Guidance Software : Encase • AccessData : Forensics Toolkit (FTK) • Certified consultants • National cybercrime units 36
  34. 37.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.1

    Live again : restoring production • On preprod, recheck notepad 1 findings vs new build • Usually, reset passwords. • On D8, use mass_pwreset • On D7/MySQL: • update users 
 set pass = concat(‘ZZZ', sha(concat(pass, md5(rand())))); • Prepare the GDPR Art. 34 exploit report • Prepare marketing/social copy • Prepare for the future 37
  35. 39.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.3

    L8R: be prepared • Developer education on security • Security Team mailing list • https://twitter.com/drupalsecurity • https://www.drupal.org/security/rss.xml • http://crackingdrupal.com/ ? • National support • GDPR requires national authorities to provide education (Art. 57, Rec. 122, 132) • Exists outside GDPR (FR: ANSSI MOOC)
 https://secnumacademie.gouv.fr/ 39
  36. 40.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.4

    L8R : disaster prevention • Security process • Analyse sec. releases to understand fixes • Look for similar flaw in custom code • Take part in Drupal core / contrib to acquire expertise • Quality process • Systematic peer code reviews • Code-driver maintenance + dev process • Automatic quality tools in CI • Contrib updates scheduling 40
  37. 41.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 7.5

    Continuous improvement • « You can’t improve what you don’t measure » • Get time metrics from notepad 1 • Build contingency plan from notepad 2 • Plan for periodic intrusion simulations • GDPR Art. 32.1d: « …the controller and the processor shall implement […] a process for regularly testing, assessing and evaluating the effectiveness of technical and organisational measures for ensuring the security of the processing » 41
  38. 42.

    ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 42

    
 Drupal, faster, safer http://www.osinet.fr/ @OSInet https://www.linkedin.com/company/osinet/ https://goo.gl/X9LEzz Rate this talk: