So you've been hacked. Now what?

How to react after a hack successfully exploited a server, and possibly breached access to Personally Identifiable Data, in the wake of the GDPR.

This is the presentation-ready version, directly usable for presenting. The other version of this presentation includes the presenter notes.

Transcript

1. Frédéric G. MARAND

   http://www.osinet.fr/
   So you’ve been hacked…

   now what ?
   1
   

   View Slide

2. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
   Table of Contents
   1 Intro : setting the stage
   2 Snapshotting
   3 Maintaining presence
   4 Crisis communication
   5 Rebuild, don’t repair
   6 Using forensics tools
   7 Back online
   2
   

   View Slide

3. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
   1.1 Some fact checking first
   • In this room …
   • Whose site has been hacked already ?
   3
   

   View Slide

4. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
   1.1 Some fact checking first
   • In this room …
   • Whose site has been hacked already ?
   • Who feels ready to face a hacked server ?
   4
   

   View Slide

5. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
   1.1 Some fact checking first
   • In this room …
   • Whose site has been hacked already ?
   • Who feels ready to face a hacked server ?
   • Who actually has a business continuity plan ?
   5
   

   View Slide

6. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
   1.1 Some fact checking first
   • In this room …
   • Whose site has been hacked already ?
   • Who feels ready to face a hacked server ?
   • Who actually has a business continuity plan ?
   • Who actually has a GDPR-compliant business continuity plan ?
   6
   

   View Slide

7. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
   1.1 Some fact checking first
   • In this room …
   • Whose site has been hacked already ?
   • Who feels ready to face a hacked server ?
   • Who actually has a business continuity plan ?
   • Who actually has a GDPR-compliant business continuity plan ?
   • Who read node 2365547 ?
   7
   

   View Slide

8. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
   8
   

   View Slide

9. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
   1.2 Can you say that again ?
   9
   I.A.N.A.L.
   (and you probably ain’t either)

   So be sure to get one !
   

   View Slide

10. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.3 Whence do I speak ?
    • Drupal org member since 2005 (fgm)
    • Drupal consultant, not a site building agency
    • Worked on fixing broken(-in) sites since 2008
    • Auditing
    • Fixing technical flaws
    • Addressing intrusions / exploits
    • Mostly Media and Government sites (.fr)
    • « Provisional member » on the Security Team
    10
    

    View Slide

11. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.4 Choose your own adventure
    • 10:00 The daily scrum has just begun.
    • 10:01 Phones rings : someone noticed your site has been defaced and
    is warning you
    • 10:02 Twitter and Reddit start buzzing
    • 10:05 Phones ring all over the place, journalists and CxO types on the
    other end, asking about GDPR Art. 33/34 ; your mailbox is filling with
    warnings
    • What is your next step ?
    11
    

    View Slide

12. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.5 Get ready
    • Notepad 1 : discovery log
    • all your work steps, to the

    tiniest
    • all your findings /
    observations
    • with timestamps and
    numbers
    12
    • Notepad 2 : remedies ideas
    • all your ideas for fixing
    the breach
    • all your ideas for further
    hardening
    • cross-refer notepad 1
    numbers
    

    View Slide

13. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    2.1 Forensic copy : why ?
    • Obvious temptation:
    • restore & resume
    • Still vulnerable
    • So you need to diagnose
    • Analyzing means modifying
    • So preserve the « crime scene »
    • Snapshot everything
    13
    

    View Slide

14. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    2.2 Snapshots : pull the plug ?
    • Prevents interference
    • Shutdown handlers, SIGPWR
    • Self-destructing code on
    network loss
    • Easy on VMs, cloud instances
    • No need to pull the plug
    14
    • But…
    • Bare remote servers
    • Further data loss
    • Journaled FS
    • Databases
    • Service interruption
    

    View Slide

15. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    2.3 Snapshots : everything == ?
    • Not just the main DB

    • Reverse Proxy logs
    • Web fronts
    • DB servers
    • File servers
    • Also…

    • External logs (SaaS)
    • External transactions
    • FW/IDS/WAF logs
    15
    The site might be just another attack vector, not the main target
    

    View Slide

16. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3.1 Maintaining presence #1
    • Pros
    • Don’t tip off hackers: you’re
    one step ahead until they
    notice
    • Keep on generating short-
    term value

    • Cons
    • Damage increase
    • Responsibility
    • Legal (GDPR)
    • Financial
    • Moral
    16
    As though intrusion had not been detected
    

    View Slide

17. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3.2 Attacker workflows (intermezzo)
    Modern: multistep flow
    • Break in
    • Dig for gold
    • Implant zombie
    • Wait for implant migration to
    archives
    • Activate
    • Profit

    Amateur: Need for Speed
    • Use exploit ASAP
    • While it lasts
    • Usually least loss (miners)
    Art crime: hide the act
    • Valuable content
    • Identity data
    • Close the door
    17
    (➡ forensics)
    (➡ restore)
    

    View Slide

18. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3.3 Maintaining presence #2
    • Limited static site
    • Best with prior work
    • Minimal subset
    • Possibly taken from RP cache
    • Very little load
    • Can run off RP heads

    • Working limited (read-only) site
    • Alternate infra
    • Alternate tech
    • Updates ?
    • Content created during
    crisis
    18
    Safe fallback mode
    Always easier if planned from the onset. Can help with GDPR Art 41.2b
    

    View Slide

19. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3.4 Maintaining presence #3
    • Social networks and status sites
    • Always there
    • Also authoritative for audience
    • Still needs some preparation
    • Accounts access
    • Inclusion in long-term communication
    • B2C: Facebook / Instagram pages
    • B2B: Twitter, status sites
    19
    When all else fails
    https://xenomorph01.deviantart.com/art/Deadpool-tis-but-a-scratch-479338640
    

    View Slide

20. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    4.1 Communicating : upstream
    • Internal stakeholders
    • DPO if any personal data are involved
    • CxO level in most cases
    • Gag orders ? Protection may exist. But follow the rules.
    • France: whistleblower protection in Sapin 2 (limited)
    • Italy (banks): Dec. 385 01/09/93 sect 52bis
    • US (contractors): Anti-SLAPP
    • Many countries have similar rules
    20
    

    View Slide

21. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    4.2 : Communication : C-level
    • DPO (start here), Legal counsel
    • Crisis Management specialists (costs…)
    • Law enforcement
    • National data protection agencies within 72h (GDPR Art. 33, Rec. 85)
    • EU countries typically have« cybercrime » units like (FR) ANSSI
    • Other sites
    • On same server
    • On same network
    • Online business partners ➡ GDPR: Data Processors or 3rd parties ?
    21
    

    View Slide

22. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    4.3 Communication : privacy
    • Often personal data leaks
    • will happen, or...
    • unprovable they did not
    • Operational constraints
    • Commerce : PCI/DSS 

    (12 steps etc)
    • Health : (US) HIPAA 

    Subtitle D E2.80.93
    • Public image damage control

    • GDPR Conflicting rules
    • 72h delay for sensitive
    exploits (Art. 33.1), or
    justification needed
    • Police operations need time.
    Per GDPR, police prevails
    (Rec. 86)
    22
    

    View Slide

23. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5.1 Rebuild : keep, rollback or ..?
    • Restore and restart same ?
    • Still just as vulnerable, just
    more so
    • Keep and fix ?
    • lots of time and effort
    reviewing
    • never completely trusted : not
    just Drupal

    • Throw away ?
    • Event sites, past lines of biz,
    post-M&A...
    • Can a static version suffice ?
    • From RP snapshots : recent
    content
    • Best: rebuild from sources +
    restore backups
    23
    

    View Slide

24. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5.2 Rebuild : restore content
    • Needs backups from before the hack
    • Do you know when it happened ? Hint: « modern workflow »
    • GFS, continuous incremental, 15 min ?
    • How much can you lose ?
    • FLOSS solutions : Amanda, Bacula, custom…
    • Unprepared emergency ?
    • Preproduction, CI builds…
    24
    (➡ modern workflow)
    

    View Slide

25. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5.3 Rebuild : sources + export
    • Easy and reliable, but assumes :
    • Code-driven development process
    • Reliable data export system in place
    • Accessible content exports (e.g. flat files, journals)
    • Content + assets repositories
    • Still need to add the fixes
    • Delay can be a problem on high-volume sites
    • Bulk handling, Incremental loading
    25
    

    View Slide

26. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5.4 Rebuild : other cases
    • Ad hoc « traditional » build process
    • Longer, less reliable
    • Too long to be a chance to fix the process
    • From scratch
    • Too long in most (all ?) cases
    • Do it as a complement after the fix
    • Not NOW
    26
    

    View Slide

27. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6 Forensics : switching hats
    27
    

    View Slide

28. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.1 Forensics : first, think !
    • How did you discover the hack ?
    • What did it take to succeed ?
    • Cast your net wide, think big
    • « Unlikely » vs « impossible »
    • Priority :
    • Easiest attacks first
    • OWASP 10
    • GIYF : search for notepad 1
    patterns
    28
    

    « Once you eliminate the
    impossible, whatever remains,
    no matter how improbable,
    must be the truth. » 

    

    Arthur Conan Doyle «The sign of the four »

    

    View Slide

29. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    • /anything/ may be erased after success
    • But most of the time, not /everything/ will
    • Anything you do leaves its own traces
    • Work on copies of the snapshots
    • You can restart from fresh copies anytime
    • There maybe more than one exploit
    • « Cleaner exploits » after Drupalgeddon 1
    • « art crime » workflow
    6.2 Forensics : keep in mind
    29
    (➡art crime)
    

    View Slide

30. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.3 Forensics : classic flaws
    • Code files :
    • lax permissions
    • filesystem traversal issues
    • Remote payload execution by upload
    • Nginx without extra hardening
    • .htaccess won’t do much good
    • In-DB PHP
    • PHP module
    • Eval-uated code
    30
    

    View Slide

31. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.4 Forensics : outer droppings
    • Filesystem :
    • /www-data outside /sites
    • www-data/www-data suspicious
    • « x » bit (0111) on files below docroot
    • timestamps
    • ts(outside sites/*/files) = ts(install)
    • ts(exploits) > ts(install)
    • meld with a fresh build from sources
    • Also check outside docroot / project root
    31
    

    View Slide

32. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.5 Forensics : Drupal modules
    • Code signing/diffing :
    • Hacked! Limitations
    • D7 : md5check, file_integrity
    • Finding DB PHP
    • OSInet QA (github)
    • Misc
    • security_review
    32
    

    View Slide

33. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.6 Forensics : DB scans
    • Quick wins :
    • users_field_data.email != users_field_data.init (users on D6/D7)
    • review roles, accounts with admin roles
    • On corporate sites, users_field_data.email domains
    • match users accounts with SSO data, directories
    • Diff DB snapshot with live
    • D7: menu_router : file_put_contents, assert
    • D8: use vendor/bin/drupal debug:router instead of DB
    • Altova DatabaseSpy content compare
    33
    

    View Slide

34. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.7 Forensics : session data
    • Sessions should be in persistent storage
    • Remember when you pulled the plug
    • Were your sessions in Memcache ? in-memory Redis ?
    • Quick checks:
    • sessions.timestamp vs users_field_data : created / changed / access /
    login
    • for intranets : sessions.hostname
    34
    

    View Slide

35. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.8 Forensics : logs
    • You use off-site logs, right ?
    • SaaS : Loggly, Logmatic,
    Logsene, Logz.io, Papertrail,
    Scalyr….
    • Homegrown remote ELK
    • GDPR
    • Data Processor constraints
    (Rec. 81)
    • Record of processing
    activities (Art. 30)

    • Still on-site ? Read-write ?
    • dblog {watchdog}
    • syslog → redirects chain
    • mongodb_watchdog
    • redis_watchdog
    • GELF/Graylog, Logstash
    • Application/WS logs
    35
    

    View Slide

36. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.9 Forensics : sleuth tools
    • Software
    • Guidance Software :
    Encase
    • AccessData : Forensics
    Toolkit (FTK)
    • Certified consultants
    • National cybercrime units
    36
    

    View Slide

37. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.1 Live again : restoring production
    • On preprod, recheck notepad 1 findings vs new build
    • Usually, reset passwords.
    • On D8, use mass_pwreset
    • On D7/MySQL:
    • update users 

    set pass = concat(‘ZZZ', sha(concat(pass, md5(rand()))));
    • Prepare the GDPR Art. 34 exploit report
    • Prepare marketing/social copy
    • Prepare for the future
    37
    

    View Slide

38. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.2 L8R : future-readiness
    38
    

    View Slide

39. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.3 L8R: be prepared
    • Developer education on security
    • Security Team mailing list
    • https://twitter.com/drupalsecurity
    • https://www.drupal.org/security/rss.xml
    • http://crackingdrupal.com/ ?
    • National support
    • GDPR requires national authorities to
    provide education (Art. 57, Rec. 122, 132)
    • Exists outside GDPR (FR: ANSSI MOOC)

    https://secnumacademie.gouv.fr/
    39
    

    View Slide

40. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.4 L8R : disaster prevention
    • Security process
    • Analyse sec. releases to
    understand fixes
    • Look for similar flaw in
    custom code
    • Take part in Drupal core /
    contrib to acquire expertise
    • Quality process
    • Systematic peer code reviews
    • Code-driver maintenance +
    dev process
    • Automatic quality tools in CI
    • Contrib updates scheduling
    40
    

    View Slide

41. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.5 Continuous improvement
    • « You can’t improve what you don’t measure »
    • Get time metrics from notepad 1
    • Build contingency plan from notepad 2
    • Plan for periodic intrusion simulations
    • GDPR Art. 32.1d: « …the controller and the processor shall
    implement […] a process for regularly testing, assessing and evaluating
    the effectiveness of technical and organisational measures for ensuring
    the security of the processing »
    41
    

    View Slide

42. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    42
    

    Drupal, faster, safer
    http://www.osinet.fr/
    @OSInet
    https://www.linkedin.com/company/osinet/
    https://goo.gl/X9LEzz
    Rate this talk:
    

    View Slide