Slide 1
Slide 1 text
Hijacking
syscalls with
(m)Ruby
Franck Verrot -- #RubyKaigi2016
Slide 2
Slide 2 text
A few words about me
Slide 3
Slide 3 text
About me
フランク
フランス人
Living in Walnut Creek, California
Working for Omada Health in San Francisco
My Japanese skills are still embarrassing
Loves embedding Ruby in things
Slide 4
Slide 4 text
Loves embedding mruby
2014: Holycorn: Foreign Data Wrappers for PostgreSQL
Slide 5
Slide 5 text
Loves embedding mruby
CREATE EXTENSION holycorn;
CREATE SERVER holycorn_server\
FOREIGN DATA WRAPPER holycorn;
CREATE FOREIGN TABLE holytable \
(some_date timestampz) \
SERVER holycorn_server
OPTIONS (wrapper_path '/tmp/source.rb');
2014: Holycorn: Foreign Data Wrappers for PostgreSQL
Slide 6
Slide 6 text
Loves embedding mruby
λ psql
psql (9.4.1)
Type "help" for help.
franck=# SELECT * FROM holytable;
some_date
---------------------
2015-06-21 22:39:24
2015-06-21 22:39:24
2015-06-21 22:39:24
...
(10 rows)
2014: Holycorn: Foreign Data Wrappers for PostgreSQL
Slide 7
Slide 7 text
Loves embedding mruby
2014: Holycorn: Foreign Data Wrappers for PostgreSQL
Slide 8
Slide 8 text
Loves embedding mruby
2014: Holycorn: Foreign Data Wrappers for PostgreSQL
2015: Virtual Table generator in SQLite (prototype) [1]
sql> SELECT ... FROM my_virtual_table_in_ruby ...
[1] The Virtual Table Mechanism Of SQLite: https://www.sqlite.org/vtab.html
Slide 9
Slide 9 text
Loves embedding mruby
2014: Holycorn: Foreign Data Wrappers for PostgreSQL
2015: Virtual Table generator in SQLite (prototype) [1]
sql> SELECT ... FROM my_virtual_table_in_ruby ...
2016: Redis module [2] (WIP)
redis> my_custom.command foo bar baz
[1] The Virtual Table Mechanism Of SQLite: https://www.sqlite.org/vtab.html
[2] Redis Loadable Module System: http://antirez.com/news/106
Slide 10
Slide 10 text
Loves embedding mruby
2014: Holycorn: Foreign Data Wrappers for PostgreSQL
2015: Virtual Table generator in SQLite (prototype) [1]
sql> SELECT ... FROM my_virtual_table_in_ruby ...
2016: Redis module [2] (WIP)
redis> my_custom.command foo bar baz
2017: Experimenting mruby clustering
Goal: Distribute data and computations across a set
of connected mruby nodes
[1] The Virtual Table Mechanism Of SQLite: https://www.sqlite.org/vtab.html
[2] Redis Loadable Module System: http://antirez.com/news/106
Slide 11
Slide 11 text
What will we learn?
What’s a syscall again?
Hijacking syscalls, what for?
Integrating with mruby
Side note about performance
Slide 12
Slide 12 text
What is a system
call?
“A system call, sometimes
referred to as a kernel call, is
a request in a Unix-like
operating system made via a
software interrupt by an
active process for a service
performed by the kernel.”
Slide 13
Slide 13 text
“The interface between
your programs, and the OS”
Slide 14
Slide 14 text
“The interface between
low-level libs, and the OS
Slide 15
Slide 15 text
What is a system call?
A syscall in action
ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
Linux, MINIX, BSDs, Darwin/OSX/macOS
(g)libc, ...
Slide 16
Slide 16 text
λ ruby -e'File.open("my_file", "w+"); gets'
What is a system call?
Slide 17
Slide 17 text
λ ruby -e'File.open("my_file", "w+"); gets'
What is a system call?
^Z
[1] + 42 suspended ruby […]
Slide 18
Slide 18 text
LSOF(8) LSOF(8)
NAME
lsof - list open files
SYNOPSIS
lsof ...
DESCRIPTION
Lsof revision 4.87 lists on its standard output file
information about files opened by processes for the
following UNIX dialects:
...
An open file may be a regular file, a
directory, a block special file, a character special
file, an executing text reference, a library, a stream
or a network file (Internet socket, NFS file or UNIX
domain socket.) A specific file or all the files in a
file system may be selected by path.
...
λ sudo lsof -p 42
λ ruby -e'File.open("my_file", "w+"); gets'
What is a system call?
^Z
[1] + 42 suspended ruby […]
Slide 19
Slide 19 text
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 5909 franck cwd DIR 1,4 544 48200426 /private/tmp
ruby 5909 franck txt REG 1,4 13112 14342662 ...
ruby 5909 franck txt REG 1,4 2185464 14341452 ...
...
ruby 5909 franck 0u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 1u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 2u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 3 PIPE 0x77ae2a821f8b77d3 16384 ->0x77ae2a821f8b7893
ruby 5909 franck 4 PIPE 0x77ae2a821f8b7893 16384 ->0x77ae2a821f8b77d3
ruby 5909 franck 5 PIPE 0x77ae2a821f8b7e93 16384 ->0x77ae2a821f8b62d3
ruby 5909 franck 6 PIPE 0x77ae2a821f8b62d3 16384 ->0x77ae2a821f8b7e93
ruby 5909 franck 7u REG 1,4 0 53279582 /private/tmp/my_file
λ sudo lsof -p 42
λ ruby -e'File.open("my_file", "w+"); gets'
What is a system call?
^Z
[1] + 42 suspended ruby […]
Slide 20
Slide 20 text
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 5909 franck cwd DIR 1,4 544 48200426 /private/tmp
ruby 5909 franck txt REG 1,4 13112 14342662 ...
ruby 5909 franck txt REG 1,4 2185464 14341452 ...
...
ruby 5909 franck 0u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 1u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 2u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 3 PIPE 0x77ae2a821f8b77d3 16384 ->0x77ae2a821f8b7893
ruby 5909 franck 4 PIPE 0x77ae2a821f8b7893 16384 ->0x77ae2a821f8b77d3
ruby 5909 franck 5 PIPE 0x77ae2a821f8b7e93 16384 ->0x77ae2a821f8b62d3
ruby 5909 franck 6 PIPE 0x77ae2a821f8b62d3 16384 ->0x77ae2a821f8b7e93
ruby 5909 franck 7u REG 1,4 0 53279582 /private/tmp/my_file
λ sudo lsof -p 42
λ ruby -e'File.open("my_file", "w+"); gets'
What is a system call?
^Z
[1] + 42 suspended ruby […]
Slide 21
Slide 21 text
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 5909 franck cwd DIR 1,4 544 48200426 /private/tmp
ruby 5909 franck txt REG 1,4 13112 14342662 ...
ruby 5909 franck txt REG 1,4 2185464 14341452 ...
...
ruby 5909 franck 0u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 1u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 2u CHR 16,3 0t22126 643 /dev/ttys003
ruby 5909 franck 3 PIPE 0x77ae2a821f8b77d3 16384 ->0x77ae2a821f8b7893
ruby 5909 franck 4 PIPE 0x77ae2a821f8b7893 16384 ->0x77ae2a821f8b77d3
ruby 5909 franck 5 PIPE 0x77ae2a821f8b7e93 16384 ->0x77ae2a821f8b62d3
ruby 5909 franck 6 PIPE 0x77ae2a821f8b62d3 16384 ->0x77ae2a821f8b7e93
ruby 5909 franck 7u REG 1,4 0 53279582 /private/tmp/my_file
λ sudo lsof -p 42
λ ruby -e'File.open("my_file", "w+"); gets'
What is a system call?
^Z
[1] + 42 suspended ruby […]
Slide 22
Slide 22 text
What is a system call?
A syscall in action
ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
Slide 23
Slide 23 text
What is a system call?
A syscall in action
ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
process
Slide 24
Slide 24 text
What is a system call?
ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
process OS
A syscall in action
Slide 25
Slide 25 text
What is a system call?
A syscall in action
ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
process OS
open(“my_file”, …)
Slide 26
Slide 26 text
What is a system call?
A syscall in action
ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
process OS
open(“my_file”, …)
fd = 7
Slide 27
Slide 27 text
What is a system call?
A syscall in action
ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
process OS
open(“my_file”, …)
fd = 7
write(7, “hello!”, …)
Slide 28
Slide 28 text
What is a system call?
A syscall in action
ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
process OS
open(“my_file”, …)
fd = 7
write(7, “hello!”, …)
size = 6
Slide 29
Slide 29 text
Hijacking
syscalls, what
for?
Slide 30
Slide 30 text
Hijacking syscalls, what for?
Fault injection
A “Chaos Monkey” [1] type of tool, for living processes
○ Simulate memory allocation issues
○ Simulate slow/throttled IO
○ Simulate network issues (lost packets, netsplits, …)
○ Simulate failing devices
○ Memory corruption/loss
○ ...
[1] https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
Slide 31
Slide 31 text
Hijacking syscalls, what for?
Real-time / interactive monitoring of a Ruby app
● “open”
● “socket”
● “read”/”write”
● “truncate”
● “unlink”
● …
Slide 32
Slide 32 text
Hijacking syscalls, what for?
Doing it better than FakeFs/Timecop
What’s wrong with tools that monkey-patch Ruby?
● Brittle
○ Doesn’t prevent custom Ruby code from
reimplementing the forbidden calls
● Break POLS
○ Shelling out (backticks, execv* functions, ...) is a way
to escape those restrictions
Slide 33
Slide 33 text
Hijacking syscalls, what for?
Securing Ruby apps
Slide 34
Slide 34 text
Wat?
Slide 35
Slide 35 text
Hijacking syscalls, what for?
Securing Ruby apps
Slide 36
Slide 36 text
Hijacking syscalls, what for?
Securing Ruby apps is kinda tricky…
● Gems
○ Install is insecure
○ Running the legitimate code of a gem is insecure
(signing gems doesn’t really help either)
● Some defaults are kind of… ¯\_(ツ)_/¯
○ Hijacking built-in gem commands is pretty easy [1]
● Code audit is near impossible
○ Tracking self-modifiable code is … hard [2]
○ Some regulations require companies to show they’ve run an audit of
their gems
[1] http://franck.verrot.fr/blog/2015/04/21/hijacking-gem-commands/
[2] https://speakerdeck.com/benjaminleesmith/hacking-with-gems-rulu-2013
Slide 37
Slide 37 text
Hijacking syscalls, what for?
Securing Ruby apps is kinda tricky…
● Way more robust options
○ LXC
○ Docker
○ Linux capabilities
○ Solaris Zones
○ ...
Slide 38
Slide 38 text
How can one
replace syscalls?
Slide 39
Slide 39 text
LD_PRELOAD=
DYLD_LIBRARY_PATH=
Slide 40
Slide 40 text
Library preloading
DLSYM(3) Linux Programmer's Manual
NAME
dlsym, dlvsym - obtain address of a symbol in a shared object or executable
SYNOPSIS
#include
void *dlsym(void *handle, const char *symbol);
...
DESCRIPTION
The function dlsym() takes a "handle" of a dynamic loaded shared
object returned by dlopen(3) along with a null-terminated symbol
name, and returns the address where that symbol is loaded into
memory. If the symbol is not found, in the specified object or any
of the shared objects that were automatically loaded by dlopen(3)
when that object was loaded, dlsym() returns NULL. (The search
performed by dlsym() is breadth first through the dependency tree of
these shared objects.)
Slide 41
Slide 41 text
Library preloading
Slide 42
Slide 42 text
Library preloading
Non-intrusive code injection
● Does not require re-compilation
● Can be done for every lib or system calls
○ Replace glib’s malloc by jemalloc
● Brings dynamism when none’s been built in
usage: LD_PRELOAD=my_new_malloc.so ruby ...
Slide 43
Slide 43 text
mruby
Wrapping up those syscalls
Slide 44
Slide 44 text
Why using (m)ruby?
● It’s Ruby ❤
Slide 45
Slide 45 text
Why using (m)ruby?
● It’s Ruby ❤
● Higher-level interface
○ External DSL to the C implementation
Slide 46
Slide 46 text
mruby as an
External Domain Specific Language
TRUNCATE(2) Linux Programmer's Manual
NAME top
truncate, ftruncate - truncate a file to a specified length
SYNOPSIS top
#include
#include
int truncate(const char *path, off_t length);
int ftruncate(int fd, off_t length);
Slide 47
Slide 47 text
mruby as an
External Domain Specific Language
int truncate(const char *path, off_t length);
int ftruncate(int fd, off_t length);
Slide 48
Slide 48 text
Library preloading
W
iring
m
ruby
... mrb_funcall(vm, syscall_obj, “read”, …)
Slide 49
Slide 49 text
Why using (m)ruby?
● It’s Ruby ❤
● Higher-level interface
○ External DSL to the C implementation
Slide 50
Slide 50 text
Why using (m)ruby?
● It’s Ruby ❤
● Higher-level interface
○ External DSL to the C implementation
● Embedding mruby
○ Produces low-level artifacts
■ Static library
■ Shared library
Slide 51
Slide 51 text
Creating an embeddable
mruby
Slide 52
Slide 52 text
Creating an embeddable mruby
● mruby-cli
○ Establishes the base project tree
λ ~/mruby-cli -s my_mruby_app
create .gitignore
create mrbgem.rake
create build_config.rb
create Rakefile
create Dockerfile
create docker-compose.yml
create tools/
create tools/my_mruby_app/
create tools/my_mruby_....c
create mrblib/
create mrblib/my_mruby_app.rb
create mrblib/my_mruby_app/
...
Slide 53
Slide 53 text
Creating an embeddable mruby
λ docker-compose run compile
Building compile
Step 1 : FROM hone/mruby-cli
# Executing 2 build triggers...
Step 1 : WORKDIR /home/mruby/code
---> Using cache
Step 1 : ENV GEM_HOME
/home/mruby/.gem/
---> Using cache
---> 968244bb0bc5
Successfully built 968244bb0bc5
● mruby-cli
○ Establishes the base project tree
○ Docker-based solution
Slide 54
Slide 54 text
Creating an embeddable mruby
● mruby-cli
○ Establishes the base project tree
○ Docker-based solution
○ Cross-compiles to a variety of environments
λ ls mruby/build/
host
i386-apple-darwin14
i686-pc-linux-gnu
i686-w64-mingw32
X86_64-apple-darwin14
x86_64-pc-linux-gnu
x86_64-w64-mingw32
...
λ file ./mruby/build/x86_64-w64-mingw32/bin/my_mruby_app.exe
my_mruby_app.exe: PE32+ executable for MS Windows (console)
Mono/.Net assembly
Slide 55
Slide 55 text
Creating an embeddable mruby
$(CC) f1.o f2.o ... \
./mruby/build/host/lib/libmruby.a
Slide 56
Slide 56 text
Why using (m)ruby?
● It’s Ruby ❤
● Higher-level interface
○ External DSL to the C implementation
● Embedding mruby
○ Produces low-level artifacts
■ Static library
■ Shared library
Slide 57
Slide 57 text
Why using (m)ruby?
● It’s Ruby ❤
● Higher-level interface
○ External DSL to the C implementation
● Embedding mruby
○ Produces low-level artifacts
■ Static library
■ Shared library
○ Highly modular
○ MultiVM in a single process
Slide 58
Slide 58 text
Performance That’s where it gets
ugly
Slide 59
Slide 59 text
Experimental conditions
Slide 60
Slide 60 text
Performance
Test program (IO intensive)
Slide 61
Slide 61 text
Performance
The hardware
λ system_profiler SPHardwareDataType
Hardware Overview:
Processor Name: Intel Core i5
Processor Speed: 2.7 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache (per Core): 256 KB
L3 Cache: 3 MB
Memory: 8 GB
...
Slide 62
Slide 62 text
Performance
Naive implementation
(1 VM per syscall)
Slide 63
Slide 63 text
Performance
Naive implementation
(1 VM per syscall)
λ make bench_bare => 45.5 msec
λ make bench_naive => 1090.0 msec
± 24x slower than bare process
Slide 64
Slide 64 text
Performance
Optimized implementation
(1 VM per subject under test)
Slide 65
Slide 65 text
Performance
Optimized implementation
(1 VM per subject under test)
Slide 66
Slide 66 text
Performance
Results
λ make bench_bare => 45.5 msec
λ make bench_naive => 1090.0 msec
Previous results
Slide 67
Slide 67 text
Performance
Results
λ make bench_bare => 45.5 msec
λ make bench_naive => 1090.0 msec
λ make bench_optimized => 83.7 msec
2x slower than bare process
> 10x faster than 1 VM per syscall
Slide 68
Slide 68 text
Performance
Future work
● Add DTrace probes to mruby
○ Try to understand mruby better
● Precompile scripts with mrbc
○ Only addresses startup time
Slide 69
Slide 69 text
Conclusion
Slide 70
Slide 70 text
Conclusion
● Replacing system calls is kinda fun (but tricky)
● Still a bit fragile
○ Kernel extension: run mruby at the kernel level
○ Brittle, but was easy to set up
● mruby is my preferred Ruby
○ Lightweight
○ Flexible