$30 off During Our Annual Pro Sale. View Details »

So you've been hacked. Now what?

So you've been hacked. Now what?

How to handle an exploit and/or data protection violation, in the wake of the GDPR regulations.

Frédéric G. MARAND

June 08, 2018
Tweet

More Decks by Frédéric G. MARAND

Other Decks in Technology

Transcript

  1. Frédéric G. MARAND

    http://www.osinet.fr/
    So you’ve been hacked…

    now what ?
    1

    View Slide

  2. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    Table of Contents
    1 Intro : setting the stage
    2 Snapshotting
    3 Maintaining presence
    4 Crisis communication
    5 Rebuild, don’t repair
    6 Using forensics tools
    7 Back online
    2

    View Slide

  3. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1 

    Intro
    X

    View Slide

  4. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.1 Some fact checking first
    • In this room …
    • Whose site has been hacked already ?
    3
    Each question is harder than the previous one.

    View Slide

  5. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.1 Some fact checking first
    • In this room …
    • Whose site has been hacked already ?
    • Who feels ready to face a hacked server ?
    4

    View Slide

  6. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.1 Some fact checking first
    • In this room …
    • Whose site has been hacked already ?
    • Who feels ready to face a hacked server ?
    • Who actually has a business continuity plan ?
    5

    View Slide

  7. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.1 Some fact checking first
    • In this room …
    • Whose site has been hacked already ?
    • Who feels ready to face a hacked server ?
    • Who actually has a business continuity plan ?
    • Who actually has a GDPR-compliant business continuity plan ?
    6

    View Slide

  8. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.1 Some fact checking first
    • In this room …
    • Whose site has been hacked already ?
    • Who feels ready to face a hacked server ?
    • Who actually has a business continuity plan ?
    • Who actually has a GDPR-compliant business continuity plan ?
    • Who read node 2365547 ?
    7

    View Slide

  9. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    8
    • It’s the basis for this presentation.
    • On the plus side, it can be regularly updated by the community
    • On the minus site, it’s less detailed

    View Slide

  10. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.2 Can you say that again ?
    9
    I.A.N.A.L.
    (and you probably ain’t either)

    So be sure to get one !
    Tip 1 Non-law professionals are usually not allowed to provide legal help.
    Tip 2 And even where allowed, it’s not a good idea.
    Tip 3 Get one long before that day so you can negotiate the agreement

    View Slide

  11. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.3 Whence do I speak ?
    • Drupal org member since 2005 (fgm)
    • Drupal consultant, not a site building agency
    • Worked on fixing broken(-in) sites since 2008
    • Auditing
    • Fixing technical flaws
    • Addressing intrusions / exploits
    • Mostly Media and Government sites (.fr)
    • « Provisional member » on the Security Team
    10

    View Slide

  12. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.4 Choose your own adventure
    • 10:00 The daily scrum has just begun.
    • 10:01 Phones rings : someone noticed your site has been defaced and
    is warning you
    • 10:02 Twitter and Reddit start buzzing
    • 10:05 Phones ring all over the place, journalists and CxO types on the
    other end, asking about GDPR Art. 33/34 ; your mailbox is filling with
    warnings
    • What is your next step ?
    11
    • Variant: only tech noticed. Less stress, but same requirements
    • art 33 : notify control authority
    • art 34: notify affected individuals

    View Slide

  13. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    1.5 Get ready
    • Notepad 1 : discovery log
    • all your work steps, to the

    tiniest
    • all your findings /
    observations
    • with timestamps and
    numbers
    12
    • Notepad 2 : remedies ideas
    • all your ideas for fixing
    the breach
    • all your ideas for further
    hardening
    • cross-refer notepad 1
    numbers
    Tip1 : same pad, start from the end
    Tip2 : paper, not digital

    View Slide

  14. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    2

    Snapshots
    X

    View Slide

  15. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    2.1 Forensic copy : why ?
    • Obvious temptation:
    • restore & resume
    • Still vulnerable
    • So you need to diagnose
    • Analyzing means modifying
    • So preserve the « crime scene »
    • Snapshot everything
    13
    Technically, you’re more vulnerable than originally, since someone actually knows that:
    • you are affected by the vulnerability
    • you did not fix it

    View Slide

  16. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    2.2 Snapshots : pull the plug ?
    • Prevents interference
    • Shutdown handlers, SIGPWR
    • Self-destructing code on
    network loss
    • Easy on VMs, cloud instances
    • No need to pull the plug
    14
    • But…
    • Bare remote servers
    • Further data loss
    • Journaled FS
    • Databases
    • Service interruption
    Example met in 05/2018: the Jenkins exploit code removed itself from disk immediately after launch, to avoid detection by disk scanners, and reloaded
    via crontab. Scanning the disk after pulling the plug would have shown nothing.

    View Slide

  17. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    2.3 Snapshots : everything == ?
    • Not just the main DB

    • Reverse Proxy logs
    • Web fronts
    • DB servers
    • File servers
    • Also…

    • External logs (SaaS)
    • External transactions
    • FW/IDS/WAF logs
    15
    The site might be just another attack vector, not the main target
    • Case in point in the Jenkins hack: the bot scanned the disk for usernames / ssh keys / host names, uploading them to its control server and attempting
    exploit itself.
    • Are you sure your servers only store the PUBLIC ssh keys, no PRIVATE ? Even the bastion host ? On all user accounts ?

    View Slide

  18. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3

    Presence
    X
    So, what kind of presence to maintain after the exploit ?

    View Slide

  19. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3.1 Maintaining presence #1
    • Pros
    • Don’t tip off hackers: you’re
    one step ahead until they
    notice
    • Keep on generating short-
    term value

    • Cons
    • Damage increase
    • Responsibility
    • Legal (GDPR)
    • Financial
    • Moral
    16
    As though intrusion had not been detected

    View Slide

  20. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3.2 Attacker workflows (intermezzo)
    Modern: multistep flow
    • Break in
    • Dig for gold
    • Implant zombie
    • Wait for implant migration to
    archives
    • Activate
    • Profit

    Amateur: Need for Speed
    • Use exploit ASAP
    • While it lasts
    • Usually least loss (miners)
    Art crime: hide the act
    • Valuable content
    • Identity data
    • Close the door
    17
    (➡ forensics)
    (➡ restore)

    View Slide

  21. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3.3 Maintaining presence #2
    • Limited static site
    • Best with prior work
    • Minimal subset
    • Possibly taken from RP cache
    • Very little load
    • Can run off RP heads

    • Working limited (read-only) site
    • Alternate infra
    • Alternate tech
    • Updates ?
    • Content created during
    crisis
    18
    Safe fallback mode
    Always easier if planned from the onset. Can help with GDPR Art 41.2b

    View Slide

  22. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    3.4 Maintaining presence #3
    • Social networks and status sites
    • Always there
    • Also authoritative for audience
    • Still needs some preparation
    • Accounts access
    • Inclusion in long-term communication
    • B2C: Facebook / Instagram pages
    • B2B: Twitter, status sites
    19
    When all else fails
    https://xenomorph01.deviantart.com/art/Deadpool-tis-but-a-scratch-479338640
    LinkedIn ? Doesn’t feel appropriate, but might.

    View Slide

  23. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    4

    Communication
    X

    View Slide

  24. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    4.1 Communicating : upstream
    • Internal stakeholders
    • DPO if any personal data are involved
    • CxO level in most cases
    • Gag orders ? Protection may exist. But follow the rules.
    • France: whistleblower protection in Sapin 2 (limited)
    • Italy (banks): Dec. 385 01/09/93 sect 52bis
    • US (contractors): Anti-SLAPP
    • Many countries have similar rules
    20

    View Slide

  25. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    4.2 : Communication : C-level
    • DPO (start here), Legal counsel
    • Crisis Management specialists (costs…)
    • Law enforcement
    • National data protection agencies within 72h (GDPR Art. 33, Rec. 85)
    • EU countries typically have« cybercrime » units like (FR) ANSSI
    • Other sites
    • On same server
    • On same network
    • Online business partners ➡ GDPR: Data Processors or 3rd parties ?
    21
    * The 72h limit is only for sensitive data or high risk exploits
    * Business partners also includes clients

    View Slide

  26. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    4.3 Communication : privacy
    • Often personal data leaks
    • will happen, or...
    • unprovable they did not
    • Operational constraints
    • Commerce : PCI/DSS 

    (12 steps etc)
    • Health : (US) HIPAA 

    Subtitle D E2.80.93
    • Public image damage control

    • GDPR Conflicting rules
    • 72h delay for sensitive
    exploits (Art. 33.1), or
    justification needed
    • Police operations need time.
    Per GDPR, police prevails
    (Rec. 86)
    22

    View Slide

  27. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5

    Rebuild
    X

    View Slide

  28. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5.1 Rebuild : keep, rollback or ..?
    • Restore and restart same ?
    • Still just as vulnerable, just
    more so
    • Keep and fix ?
    • lots of time and effort
    reviewing
    • never completely trusted : not
    just Drupal

    • Throw away ?
    • Event sites, past lines of biz,
    post-M&A...
    • Can a static version suffice ?
    • From RP snapshots : recent
    content
    • Best: rebuild from sources +
    restore backups
    23
    Best = slide 5.3

    View Slide

  29. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5.2 Rebuild : restore content
    • Needs backups from before the hack
    • Do you know when it happened ? Hint: « modern workflow »
    • GFS, continuous incremental, 15 min ?
    • How much can you lose ?
    • FLOSS solutions : Amanda, Bacula, custom…
    • Unprepared emergency ?
    • Preproduction, CI builds…
    24
    (➡ modern workflow)
    • In the « modern workflow » attacker doesn’t move until a long time has elapsed, increasing the cost of rebuild because of new content (e.g.
    transactions) creation, and the likelihood backups will be erased.
    • If there is no code in the DB and everything is industrialized and config always moves up and content down, reinstall is a breeze
    • Zmanda provides EE for Amanda

    View Slide

  30. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5.3 Rebuild : sources + export
    • Easy and reliable, but assumes :
    • Code-driven development process
    • Reliable data export system in place
    • Accessible content exports (e.g. flat files, journals)
    • Content + assets repositories
    • Still need to add the fixes
    • Delay can be a problem on high-volume sites
    • Bulk handling, Incremental loading
    25

    View Slide

  31. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    5.4 Rebuild : other cases
    • Ad hoc « traditional » build process
    • Longer, less reliable
    • Too long to be a chance to fix the process
    • From scratch
    • Too long in most (all ?) cases
    • Do it as a complement after the fix
    • Not NOW
    26

    View Slide

  32. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6

    Forensics
    X

    View Slide

  33. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6 Forensics : switching hats
    27

    View Slide

  34. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.1 Forensics : first, think !
    • How did you discover the hack ?
    • What did it take to succeed ?
    • Cast your net wide, think big
    • « Unlikely » vs « impossible »
    • Priority :
    • Easiest attacks first
    • OWASP 10
    • GIYF : search for notepad 1
    patterns
    28

    « Once you eliminate the
    impossible, whatever remains,
    no matter how improbable,
    must be the truth. » 


    Arthur Conan Doyle «The sign of the four »


    View Slide

  35. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    • /anything/ may be erased after success
    • But most of the time, not /everything/ will
    • Anything you do leaves its own traces
    • Work on copies of the snapshots
    • You can restart from fresh copies anytime
    • There maybe more than one exploit
    • « Cleaner exploits » after Drupalgeddon 1
    • « art crime » workflow
    6.2 Forensics : keep in mind
    29
    (➡art crime)
    * |v’x - vx|Δpx ≍ ℏ/Δt : Observer effect equation, Niels Bohr, 1928 (ℏ : Planck constant)
    * https://en.wikipedia.org/wiki/Observer_effect_(physics)

    View Slide

  36. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.3 Forensics : classic flaws
    • Code files :
    • lax permissions
    • filesystem traversal issues
    • Remote payload execution by upload
    • Nginx without extra hardening
    • .htaccess won’t do much good
    • In-DB PHP
    • PHP module
    • Eval-uated code
    30

    View Slide

  37. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.4 Forensics : outer droppings
    • Filesystem :
    • /www-data outside /sites
    • www-data/www-data suspicious
    • « x » bit (0111) on files below docroot
    • timestamps
    • ts(outside sites/*/files) = ts(install)
    • ts(exploits) > ts(install)
    • meld with a fresh build from sources
    • Also check outside docroot / project root
    31

    View Slide

  38. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.5 Forensics : Drupal modules
    • Code signing/diffing :
    • Hacked! Limitations
    • D7 : md5check, file_integrity
    • Finding DB PHP
    • OSInet QA (github)
    • Misc
    • security_review
    32
    Hacked won’t help with custom code: compare with VCS

    View Slide

  39. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.6 Forensics : DB scans
    • Quick wins :
    • users_field_data.email != users_field_data.init (users on D6/D7)
    • review roles, accounts with admin roles
    • On corporate sites, users_field_data.email domains
    • match users accounts with SSO data, directories
    • Diff DB snapshot with live
    • D7: menu_router : file_put_contents, assert
    • D8: use vendor/bin/drupal debug:router instead of DB
    • Altova DatabaseSpy content compare
    33

    View Slide

  40. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.7 Forensics : session data
    • Sessions should be in persistent storage
    • Remember when you pulled the plug
    • Were your sessions in Memcache ? in-memory Redis ?
    • Quick checks:
    • sessions.timestamp vs users_field_data : created / changed / access /
    login
    • for intranets : sessions.hostname
    34

    View Slide

  41. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.8 Forensics : logs
    • You use off-site logs, right ?
    • SaaS : Loggly, Logmatic,
    Logsene, Logz.io, Papertrail,
    Scalyr….
    • Homegrown remote ELK
    • GDPR
    • Data Processor constraints
    (Rec. 81)
    • Record of processing
    activities (Art. 30)

    • Still on-site ? Read-write ?
    • dblog {watchdog}
    • syslog → redirects chain
    • mongodb_watchdog
    • redis_watchdog
    • GELF/Graylog, Logstash
    • Application/WS logs
    35

    View Slide

  42. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    6.9 Forensics : sleuth tools
    • Software
    • Guidance Software :
    Encase
    • AccessData : Forensics
    Toolkit (FTK)
    • Certified consultants
    • National cybercrime units
    36
    cybercrime units may help, or suggest approved consultants: tools better used by pros.

    View Slide

  43. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7

    Back online
    X

    View Slide

  44. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.1 Live again : restoring production
    • On preprod, recheck notepad 1 findings vs new build
    • Usually, reset passwords.
    • On D8, use mass_pwreset
    • On D7/MySQL:
    • update users 

    set pass = concat(‘ZZZ', sha(concat(pass, md5(rand()))));
    • Prepare the GDPR Art. 34 exploit report
    • Prepare marketing/social copy
    • Prepare for the future
    37

    View Slide

  45. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.2 L8R : future-readiness
    38

    View Slide

  46. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.3 L8R: be prepared
    • Developer education on security
    • Security Team mailing list
    • https://twitter.com/drupalsecurity
    • https://www.drupal.org/security/rss.xml
    • http://crackingdrupal.com/ ?
    • National support
    • GDPR requires national authorities to
    provide education (Art. 57, Rec. 122, 132)
    • Exists outside GDPR (FR: ANSSI MOOC)

    https://secnumacademie.gouv.fr/
    39

    View Slide

  47. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.4 L8R : disaster prevention
    • Security process
    • Analyse sec. releases to
    understand fixes
    • Look for similar flaw in
    custom code
    • Take part in Drupal core /
    contrib to acquire expertise
    • Quality process
    • Systematic peer code reviews
    • Code-driver maintenance +
    dev process
    • Automatic quality tools in CI
    • Contrib updates scheduling
    40

    View Slide

  48. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    7.5 Continuous improvement
    • « You can’t improve what you don’t measure »
    • Get time metrics from notepad 1
    • Build contingency plan from notepad 2
    • Plan for periodic intrusion simulations
    • GDPR Art. 32.1d: « …the controller and the processor shall
    implement […] a process for regularly testing, assessing and evaluating
    the effectiveness of technical and organisational measures for ensuring
    the security of the processing »
    41

    View Slide

  49. ©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
    42

    Drupal, faster, safer
    http://www.osinet.fr/
    @OSInet
    https://www.linkedin.com/company/osinet/
    https://goo.gl/X9LEzz
    Rate this talk:

    View Slide