How to handle an exploit and/or data protection violation, in the wake of the GDPR regulations.
Frédéric G. MARAND http://www.osinet.fr/So you’ve been hacked… now what ?1
View Slide
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0Table of Contents1 Intro : setting the stage2 Snapshotting3 Maintaining presence4 Crisis communication5 Rebuild, don’t repair6 Using forensics tools7 Back online2
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01 IntroX
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.1 Some fact checking first• In this room …• Whose site has been hacked already ?3Each question is harder than the previous one.
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.1 Some fact checking first• In this room …• Whose site has been hacked already ?• Who feels ready to face a hacked server ?4
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.1 Some fact checking first• In this room …• Whose site has been hacked already ?• Who feels ready to face a hacked server ?• Who actually has a business continuity plan ?5
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.1 Some fact checking first• In this room …• Whose site has been hacked already ?• Who feels ready to face a hacked server ?• Who actually has a business continuity plan ?• Who actually has a GDPR-compliant business continuity plan ?6
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.1 Some fact checking first• In this room …• Whose site has been hacked already ?• Who feels ready to face a hacked server ?• Who actually has a business continuity plan ?• Who actually has a GDPR-compliant business continuity plan ?• Who read node 2365547 ?7
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.08• It’s the basis for this presentation.• On the plus side, it can be regularly updated by the community• On the minus site, it’s less detailed
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.2 Can you say that again ?9I.A.N.A.L.(and you probably ain’t either) So be sure to get one !Tip 1 Non-law professionals are usually not allowed to provide legal help.Tip 2 And even where allowed, it’s not a good idea.Tip 3 Get one long before that day so you can negotiate the agreement
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.3 Whence do I speak ?• Drupal org member since 2005 (fgm)• Drupal consultant, not a site building agency• Worked on fixing broken(-in) sites since 2008• Auditing• Fixing technical flaws• Addressing intrusions / exploits• Mostly Media and Government sites (.fr)• « Provisional member » on the Security Team10
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.4 Choose your own adventure• 10:00 The daily scrum has just begun.• 10:01 Phones rings : someone noticed your site has been defaced andis warning you• 10:02 Twitter and Reddit start buzzing• 10:05 Phones ring all over the place, journalists and CxO types on theother end, asking about GDPR Art. 33/34 ; your mailbox is filling withwarnings• What is your next step ?11• Variant: only tech noticed. Less stress, but same requirements• art 33 : notify control authority• art 34: notify affected individuals
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.01.5 Get ready• Notepad 1 : discovery log• all your work steps, to the tiniest• all your findings /observations• with timestamps andnumbers12• Notepad 2 : remedies ideas• all your ideas for fixingthe breach• all your ideas for furtherhardening• cross-refer notepad 1numbersTip1 : same pad, start from the endTip2 : paper, not digital
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.02 SnapshotsX
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.02.1 Forensic copy : why ?• Obvious temptation:• restore & resume• Still vulnerable• So you need to diagnose• Analyzing means modifying• So preserve the « crime scene »• Snapshot everything13Technically, you’re more vulnerable than originally, since someone actually knows that:• you are affected by the vulnerability• you did not fix it
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.02.2 Snapshots : pull the plug ?• Prevents interference• Shutdown handlers, SIGPWR• Self-destructing code onnetwork loss• Easy on VMs, cloud instances• No need to pull the plug14• But…• Bare remote servers• Further data loss• Journaled FS• Databases• Service interruptionExample met in 05/2018: the Jenkins exploit code removed itself from disk immediately after launch, to avoid detection by disk scanners, and reloadedvia crontab. Scanning the disk after pulling the plug would have shown nothing.
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.02.3 Snapshots : everything == ?• Not just the main DB • Reverse Proxy logs• Web fronts• DB servers• File servers• Also… • External logs (SaaS)• External transactions• FW/IDS/WAF logs15The site might be just another attack vector, not the main target• Case in point in the Jenkins hack: the bot scanned the disk for usernames / ssh keys / host names, uploading them to its control server and attemptingexploit itself.• Are you sure your servers only store the PUBLIC ssh keys, no PRIVATE ? Even the bastion host ? On all user accounts ?
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.03 PresenceXSo, what kind of presence to maintain after the exploit ?
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.03.1 Maintaining presence #1• Pros• Don’t tip off hackers: you’reone step ahead until theynotice• Keep on generating short-term value • Cons• Damage increase• Responsibility• Legal (GDPR)• Financial• Moral16As though intrusion had not been detected
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.03.2 Attacker workflows (intermezzo)Modern: multistep flow• Break in• Dig for gold• Implant zombie• Wait for implant migration toarchives• Activate• Profit Amateur: Need for Speed• Use exploit ASAP• While it lasts• Usually least loss (miners)Art crime: hide the act• Valuable content• Identity data• Close the door17(➡ forensics)(➡ restore)
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.03.3 Maintaining presence #2• Limited static site• Best with prior work• Minimal subset• Possibly taken from RP cache• Very little load• Can run off RP heads • Working limited (read-only) site• Alternate infra• Alternate tech• Updates ?• Content created duringcrisis18Safe fallback modeAlways easier if planned from the onset. Can help with GDPR Art 41.2b
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.03.4 Maintaining presence #3• Social networks and status sites• Always there• Also authoritative for audience• Still needs some preparation• Accounts access• Inclusion in long-term communication• B2C: Facebook / Instagram pages• B2B: Twitter, status sites19When all else failshttps://xenomorph01.deviantart.com/art/Deadpool-tis-but-a-scratch-479338640LinkedIn ? Doesn’t feel appropriate, but might.
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.04 CommunicationX
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.04.1 Communicating : upstream• Internal stakeholders• DPO if any personal data are involved• CxO level in most cases• Gag orders ? Protection may exist. But follow the rules.• France: whistleblower protection in Sapin 2 (limited)• Italy (banks): Dec. 385 01/09/93 sect 52bis• US (contractors): Anti-SLAPP• Many countries have similar rules20
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.04.2 : Communication : C-level• DPO (start here), Legal counsel• Crisis Management specialists (costs…)• Law enforcement• National data protection agencies within 72h (GDPR Art. 33, Rec. 85)• EU countries typically have« cybercrime » units like (FR) ANSSI• Other sites• On same server• On same network• Online business partners ➡ GDPR: Data Processors or 3rd parties ?21* The 72h limit is only for sensitive data or high risk exploits* Business partners also includes clients
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.04.3 Communication : privacy• Often personal data leaks• will happen, or...• unprovable they did not• Operational constraints• Commerce : PCI/DSS (12 steps etc)• Health : (US) HIPAA Subtitle D E2.80.93• Public image damage control • GDPR Conflicting rules• 72h delay for sensitiveexploits (Art. 33.1), orjustification needed• Police operations need time.Per GDPR, police prevails(Rec. 86)22
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.05 RebuildX
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.05.1 Rebuild : keep, rollback or ..?• Restore and restart same ?• Still just as vulnerable, justmore so• Keep and fix ?• lots of time and effortreviewing• never completely trusted : notjust Drupal • Throw away ?• Event sites, past lines of biz,post-M&A...• Can a static version suffice ?• From RP snapshots : recentcontent• Best: rebuild from sources +restore backups23Best = slide 5.3
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.05.2 Rebuild : restore content• Needs backups from before the hack• Do you know when it happened ? Hint: « modern workflow »• GFS, continuous incremental, 15 min ?• How much can you lose ?• FLOSS solutions : Amanda, Bacula, custom…• Unprepared emergency ?• Preproduction, CI builds…24(➡ modern workflow)• In the « modern workflow » attacker doesn’t move until a long time has elapsed, increasing the cost of rebuild because of new content (e.g.transactions) creation, and the likelihood backups will be erased.• If there is no code in the DB and everything is industrialized and config always moves up and content down, reinstall is a breeze• Zmanda provides EE for Amanda
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.05.3 Rebuild : sources + export• Easy and reliable, but assumes :• Code-driven development process• Reliable data export system in place• Accessible content exports (e.g. flat files, journals)• Content + assets repositories• Still need to add the fixes• Delay can be a problem on high-volume sites• Bulk handling, Incremental loading25
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.05.4 Rebuild : other cases• Ad hoc « traditional » build process• Longer, less reliable• Too long to be a chance to fix the process• From scratch• Too long in most (all ?) cases• Do it as a complement after the fix• Not NOW26
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06 ForensicsX
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06 Forensics : switching hats27
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06.1 Forensics : first, think !• How did you discover the hack ?• What did it take to succeed ?• Cast your net wide, think big• « Unlikely » vs « impossible »• Priority :• Easiest attacks first• OWASP 10• GIYF : search for notepad 1patterns28 « Once you eliminate theimpossible, whatever remains,no matter how improbable,must be the truth. » Arthur Conan Doyle «The sign of the four »
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0• /anything/ may be erased after success• But most of the time, not /everything/ will• Anything you do leaves its own traces• Work on copies of the snapshots• You can restart from fresh copies anytime• There maybe more than one exploit• « Cleaner exploits » after Drupalgeddon 1• « art crime » workflow6.2 Forensics : keep in mind29(➡art crime)* |v’x - vx|Δpx ≍ ℏ/Δt : Observer effect equation, Niels Bohr, 1928 (ℏ : Planck constant)* https://en.wikipedia.org/wiki/Observer_effect_(physics)
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06.3 Forensics : classic flaws• Code files :• lax permissions• filesystem traversal issues• Remote payload execution by upload• Nginx without extra hardening• .htaccess won’t do much good• In-DB PHP• PHP module• Eval-uated code30
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06.4 Forensics : outer droppings• Filesystem :• /www-data outside /sites• www-data/www-data suspicious• « x » bit (0111) on files below docroot• timestamps• ts(outside sites/*/files) = ts(install)• ts(exploits) > ts(install)• meld with a fresh build from sources• Also check outside docroot / project root31
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06.5 Forensics : Drupal modules• Code signing/diffing :• Hacked! Limitations• D7 : md5check, file_integrity• Finding DB PHP• OSInet QA (github)• Misc• security_review32Hacked won’t help with custom code: compare with VCS
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06.6 Forensics : DB scans• Quick wins :• users_field_data.email != users_field_data.init (users on D6/D7)• review roles, accounts with admin roles• On corporate sites, users_field_data.email domains• match users accounts with SSO data, directories• Diff DB snapshot with live• D7: menu_router : file_put_contents, assert• D8: use vendor/bin/drupal debug:router instead of DB• Altova DatabaseSpy content compare33
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06.7 Forensics : session data• Sessions should be in persistent storage• Remember when you pulled the plug• Were your sessions in Memcache ? in-memory Redis ?• Quick checks:• sessions.timestamp vs users_field_data : created / changed / access /login• for intranets : sessions.hostname34
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06.8 Forensics : logs• You use off-site logs, right ?• SaaS : Loggly, Logmatic,Logsene, Logz.io, Papertrail,Scalyr….• Homegrown remote ELK• GDPR• Data Processor constraints(Rec. 81)• Record of processingactivities (Art. 30) • Still on-site ? Read-write ?• dblog {watchdog}• syslog → redirects chain• mongodb_watchdog• redis_watchdog• GELF/Graylog, Logstash• Application/WS logs35
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.06.9 Forensics : sleuth tools• Software• Guidance Software :Encase• AccessData : ForensicsToolkit (FTK)• Certified consultants• National cybercrime units36cybercrime units may help, or suggest approved consultants: tools better used by pros.
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.07 Back onlineX
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.07.1 Live again : restoring production• On preprod, recheck notepad 1 findings vs new build• Usually, reset passwords.• On D8, use mass_pwreset• On D7/MySQL:• update users set pass = concat(‘ZZZ', sha(concat(pass, md5(rand()))));• Prepare the GDPR Art. 34 exploit report• Prepare marketing/social copy• Prepare for the future37
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.07.2 L8R : future-readiness38
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.07.3 L8R: be prepared• Developer education on security• Security Team mailing list• https://twitter.com/drupalsecurity• https://www.drupal.org/security/rss.xml• http://crackingdrupal.com/ ?• National support• GDPR requires national authorities toprovide education (Art. 57, Rec. 122, 132)• Exists outside GDPR (FR: ANSSI MOOC) https://secnumacademie.gouv.fr/39
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.07.4 L8R : disaster prevention• Security process• Analyse sec. releases tounderstand fixes• Look for similar flaw incustom code• Take part in Drupal core /contrib to acquire expertise• Quality process• Systematic peer code reviews• Code-driver maintenance +dev process• Automatic quality tools in CI• Contrib updates scheduling40
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.07.5 Continuous improvement• « You can’t improve what you don’t measure »• Get time metrics from notepad 1• Build contingency plan from notepad 2• Plan for periodic intrusion simulations• GDPR Art. 32.1d: « …the controller and the processor shallimplement […] a process for regularly testing, assessing and evaluatingthe effectiveness of technical and organisational measures for ensuringthe security of the processing »41
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.042 Drupal, faster, saferhttp://www.osinet.fr/@OSInethttps://www.linkedin.com/company/osinet/https://goo.gl/X9LEzzRate this talk:
fgm
aoshun7
yokawasa
poteboy
line_developers_tw
niftycorp
onlinesalon
cybozuinsideout
takesection
muziyoshiz
colopl
oracle4engineer
bengo4com
orderedlist
aarron
trishagee
zakiyama
nonsquared
jonyablonski
3n
pauljervisheath
62gerente
chriscoyier
reverentgeek
sugarenia