RubyKaigi 2016 - Hijacking syscalls with Ruby

Transcript

1. Hijacking syscalls with (m)Ruby Franck Verrot -- #RubyKaigi2016
2. A few words about me
3. About me フランク フランス人 Living in Walnut Creek, California Working for Omada Health in San Francisco My Japanese skills are still embarrassing Loves embedding Ruby in things
4. Loves embedding mruby 2014: Holycorn: Foreign Data Wrappers for PostgreSQL
5. Loves embedding mruby CREATE EXTENSION holycorn; CREATE SERVER holycorn_server\ FOREIGN DATA WRAPPER holycorn; CREATE FOREIGN TABLE holytable \ (some_date timestampz) \ SERVER holycorn_server OPTIONS (wrapper_path '/tmp/source.rb'); 2014: Holycorn: Foreign Data Wrappers for PostgreSQL
6. Loves embedding mruby λ psql psql (9.4.1) Type "help" for help. franck=# SELECT * FROM holytable; some_date --------------------- 2015-06-21 22:39:24 2015-06-21 22:39:24 2015-06-21 22:39:24 ... (10 rows) 2014: Holycorn: Foreign Data Wrappers for PostgreSQL
7. Loves embedding mruby 2014: Holycorn: Foreign Data Wrappers for PostgreSQL
8. Loves embedding mruby 2014: Holycorn: Foreign Data Wrappers for PostgreSQL 2015: Virtual Table generator in SQLite (prototype) [1] sql> SELECT ... FROM my_virtual_table_in_ruby ... [1] The Virtual Table Mechanism Of SQLite: https://www.sqlite.org/vtab.html
9. Loves embedding mruby 2014: Holycorn: Foreign Data Wrappers for PostgreSQL 2015: Virtual Table generator in SQLite (prototype) [1] sql> SELECT ... FROM my_virtual_table_in_ruby ... 2016: Redis module [2] (WIP) redis> my_custom.command foo bar baz [1] The Virtual Table Mechanism Of SQLite: https://www.sqlite.org/vtab.html [2] Redis Loadable Module System: http://antirez.com/news/106
10. Loves embedding mruby 2014: Holycorn: Foreign Data Wrappers for PostgreSQL 2015: Virtual Table generator in SQLite (prototype) [1] sql> SELECT ... FROM my_virtual_table_in_ruby ... 2016: Redis module [2] (WIP) redis> my_custom.command foo bar baz 2017: Experimenting mruby clustering Goal: Distribute data and computations across a set of connected mruby nodes [1] The Virtual Table Mechanism Of SQLite: https://www.sqlite.org/vtab.html [2] Redis Loadable Module System: http://antirez.com/news/106
11. What will we learn? What’s a syscall again? Hijacking syscalls, what for? Integrating with mruby Side note about performance
12. What is a system call? “A system call, sometimes referred to as a kernel call, is a request in a Unix-like operating system made via a software interrupt by an active process for a service performed by the kernel.”
13. “The interface between your programs, and the OS”
14. “The interface between low-level libs, and the OS
15. What is a system call? A syscall in action ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }' Linux, MINIX, BSDs, Darwin/OSX/macOS (g)libc, ...
16. λ ruby -e'File.open("my_file", "w+"); gets' What is a system call?
17. λ ruby -e'File.open("my_file", "w+"); gets' What is a system call? ^Z [1] + 42 suspended ruby […]
18. LSOF(8) LSOF(8) NAME lsof - list open files SYNOPSIS lsof ... DESCRIPTION Lsof revision 4.87 lists on its standard output file information about files opened by processes for the following UNIX dialects: ... An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket.) A specific file or all the files in a file system may be selected by path. ... λ sudo lsof -p 42 λ ruby -e'File.open("my_file", "w+"); gets' What is a system call? ^Z [1] + 42 suspended ruby […]
19. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 5909 franck cwd DIR 1,4 544 48200426 /private/tmp ruby 5909 franck txt REG 1,4 13112 14342662 ... ruby 5909 franck txt REG 1,4 2185464 14341452 ... ... ruby 5909 franck 0u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 1u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 2u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 3 PIPE 0x77ae2a821f8b77d3 16384 ->0x77ae2a821f8b7893 ruby 5909 franck 4 PIPE 0x77ae2a821f8b7893 16384 ->0x77ae2a821f8b77d3 ruby 5909 franck 5 PIPE 0x77ae2a821f8b7e93 16384 ->0x77ae2a821f8b62d3 ruby 5909 franck 6 PIPE 0x77ae2a821f8b62d3 16384 ->0x77ae2a821f8b7e93 ruby 5909 franck 7u REG 1,4 0 53279582 /private/tmp/my_file λ sudo lsof -p 42 λ ruby -e'File.open("my_file", "w+"); gets' What is a system call? ^Z [1] + 42 suspended ruby […]
20. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 5909 franck cwd DIR 1,4 544 48200426 /private/tmp ruby 5909 franck txt REG 1,4 13112 14342662 ... ruby 5909 franck txt REG 1,4 2185464 14341452 ... ... ruby 5909 franck 0u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 1u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 2u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 3 PIPE 0x77ae2a821f8b77d3 16384 ->0x77ae2a821f8b7893 ruby 5909 franck 4 PIPE 0x77ae2a821f8b7893 16384 ->0x77ae2a821f8b77d3 ruby 5909 franck 5 PIPE 0x77ae2a821f8b7e93 16384 ->0x77ae2a821f8b62d3 ruby 5909 franck 6 PIPE 0x77ae2a821f8b62d3 16384 ->0x77ae2a821f8b7e93 ruby 5909 franck 7u REG 1,4 0 53279582 /private/tmp/my_file λ sudo lsof -p 42 λ ruby -e'File.open("my_file", "w+"); gets' What is a system call? ^Z [1] + 42 suspended ruby […]
21. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 5909 franck cwd DIR 1,4 544 48200426 /private/tmp ruby 5909 franck txt REG 1,4 13112 14342662 ... ruby 5909 franck txt REG 1,4 2185464 14341452 ... ... ruby 5909 franck 0u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 1u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 2u CHR 16,3 0t22126 643 /dev/ttys003 ruby 5909 franck 3 PIPE 0x77ae2a821f8b77d3 16384 ->0x77ae2a821f8b7893 ruby 5909 franck 4 PIPE 0x77ae2a821f8b7893 16384 ->0x77ae2a821f8b77d3 ruby 5909 franck 5 PIPE 0x77ae2a821f8b7e93 16384 ->0x77ae2a821f8b62d3 ruby 5909 franck 6 PIPE 0x77ae2a821f8b62d3 16384 ->0x77ae2a821f8b7e93 ruby 5909 franck 7u REG 1,4 0 53279582 /private/tmp/my_file λ sudo lsof -p 42 λ ruby -e'File.open("my_file", "w+"); gets' What is a system call? ^Z [1] + 42 suspended ruby […]
22. What is a system call? A syscall in action ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }'
23. What is a system call? A syscall in action ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }' process
24. What is a system call? ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }' process OS A syscall in action
25. What is a system call? A syscall in action ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }' process OS open(“my_file”, …)
26. What is a system call? A syscall in action ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }' process OS open(“my_file”, …) fd = 7
27. What is a system call? A syscall in action ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }' process OS open(“my_file”, …) fd = 7 write(7, “hello!”, …)
28. What is a system call? A syscall in action ruby -e'File.open("my_file", "w+") { |f| f.write "hello!" }' process OS open(“my_file”, …) fd = 7 write(7, “hello!”, …) size = 6
29. Hijacking syscalls, what for?
30. Hijacking syscalls, what for? Fault injection A “Chaos Monkey” [1] type of tool, for living processes ◦ Simulate memory allocation issues ◦ Simulate slow/throttled IO ◦ Simulate network issues (lost packets, netsplits, …) ◦ Simulate failing devices ◦ Memory corruption/loss ◦ ... [1] https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
31. Hijacking syscalls, what for? Real-time / interactive monitoring of a Ruby app • “open” • “socket” • “read”/”write” • “truncate” • “unlink” • …
32. Hijacking syscalls, what for? Doing it better than FakeFs/Timecop What’s wrong with tools that monkey-patch Ruby? • Brittle ◦ Doesn’t prevent custom Ruby code from reimplementing the forbidden calls • Break POLS ◦ Shelling out (backticks, execv* functions, ...) is a way to escape those restrictions
33. Hijacking syscalls, what for? Securing Ruby apps
34. Wat?
35. Hijacking syscalls, what for? Securing Ruby apps
36. Hijacking syscalls, what for? Securing Ruby apps is kinda tricky… • Gems ◦ Install is insecure ◦ Running the legitimate code of a gem is insecure (signing gems doesn’t really help either) • Some defaults are kind of… ¯\_(ツ)_/¯ ◦ Hijacking built-in gem commands is pretty easy [1] • Code audit is near impossible ◦ Tracking self-modifiable code is … hard [2] ◦ Some regulations require companies to show they’ve run an audit of their gems [1] http://franck.verrot.fr/blog/2015/04/21/hijacking-gem-commands/ [2] https://speakerdeck.com/benjaminleesmith/hacking-with-gems-rulu-2013
37. Hijacking syscalls, what for? Securing Ruby apps is kinda tricky… • Way more robust options ◦ LXC ◦ Docker ◦ Linux capabilities ◦ Solaris Zones ◦ ...
38. How can one replace syscalls?
39. LD_PRELOAD= DYLD_LIBRARY_PATH=
40. Library preloading DLSYM(3) Linux Programmer's Manual NAME dlsym, dlvsym - obtain address of a symbol in a shared object or executable SYNOPSIS #include <dlfcn.h> void *dlsym(void *handle, const char *symbol); ... DESCRIPTION The function dlsym() takes a "handle" of a dynamic loaded shared object returned by dlopen(3) along with a null-terminated symbol name, and returns the address where that symbol is loaded into memory. If the symbol is not found, in the specified object or any of the shared objects that were automatically loaded by dlopen(3) when that object was loaded, dlsym() returns NULL. (The search performed by dlsym() is breadth first through the dependency tree of these shared objects.)
41. Library preloading
42. Library preloading Non-intrusive code injection • Does not require re-compilation • Can be done for every lib or system calls ◦ Replace glib’s malloc by jemalloc • Brings dynamism when none’s been built in usage: LD_PRELOAD=my_new_malloc.so ruby ...
43. mruby Wrapping up those syscalls
44. Why using (m)ruby? • It’s Ruby ❤
45. Why using (m)ruby? • It’s Ruby ❤ • Higher-level interface ◦ External DSL to the C implementation
46. mruby as an External Domain Specific Language TRUNCATE(2) Linux Programmer's Manual NAME top truncate, ftruncate - truncate a file to a specified length SYNOPSIS top #include <unistd.h> #include <sys/types.h> int truncate(const char *path, off_t length); int ftruncate(int fd, off_t length);
47. mruby as an External Domain Specific Language int truncate(const char *path, off_t length); int ftruncate(int fd, off_t length);
48. Library preloading W iring m ruby ... mrb_funcall(vm, syscall_obj, “read”, …)
49. Why using (m)ruby? • It’s Ruby ❤ • Higher-level interface ◦ External DSL to the C implementation
50. Why using (m)ruby? • It’s Ruby ❤ • Higher-level interface ◦ External DSL to the C implementation • Embedding mruby ◦ Produces low-level artifacts ▪ Static library ▪ Shared library
51. Creating an embeddable mruby
52. Creating an embeddable mruby • mruby-cli ◦ Establishes the base project tree λ ~/mruby-cli -s my_mruby_app create .gitignore create mrbgem.rake create build_config.rb create Rakefile create Dockerfile create docker-compose.yml create tools/ create tools/my_mruby_app/ create tools/my_mruby_....c create mrblib/ create mrblib/my_mruby_app.rb create mrblib/my_mruby_app/ ...
53. Creating an embeddable mruby λ docker-compose run compile Building compile Step 1 : FROM hone/mruby-cli # Executing 2 build triggers... Step 1 : WORKDIR /home/mruby/code ---> Using cache Step 1 : ENV GEM_HOME /home/mruby/.gem/ ---> Using cache ---> 968244bb0bc5 Successfully built 968244bb0bc5 • mruby-cli ◦ Establishes the base project tree ◦ Docker-based solution
54. Creating an embeddable mruby • mruby-cli ◦ Establishes the base project tree ◦ Docker-based solution ◦ Cross-compiles to a variety of environments λ ls mruby/build/ host i386-apple-darwin14 i686-pc-linux-gnu i686-w64-mingw32 X86_64-apple-darwin14 x86_64-pc-linux-gnu x86_64-w64-mingw32 ... λ file ./mruby/build/x86_64-w64-mingw32/bin/my_mruby_app.exe my_mruby_app.exe: PE32+ executable for MS Windows (console) Mono/.Net assembly
55. Creating an embeddable mruby $(CC) f1.o f2.o ... \ ./mruby/build/host/lib/libmruby.a
56. Why using (m)ruby? • It’s Ruby ❤ • Higher-level interface ◦ External DSL to the C implementation • Embedding mruby ◦ Produces low-level artifacts ▪ Static library ▪ Shared library
57. Why using (m)ruby? • It’s Ruby ❤ • Higher-level interface ◦ External DSL to the C implementation • Embedding mruby ◦ Produces low-level artifacts ▪ Static library ▪ Shared library ◦ Highly modular ◦ MultiVM in a single process
58. Performance That’s where it gets ugly
59. Experimental conditions
60. Performance Test program (IO intensive)
61. Performance The hardware λ system_profiler SPHardwareDataType Hardware Overview: Processor Name: Intel Core i5 Processor Speed: 2.7 GHz Number of Processors: 1 Total Number of Cores: 2 L2 Cache (per Core): 256 KB L3 Cache: 3 MB Memory: 8 GB ...
62. Performance Naive implementation (1 VM per syscall)
63. Performance Naive implementation (1 VM per syscall) λ make bench_bare => 45.5 msec λ make bench_naive => 1090.0 msec ± 24x slower than bare process
64. Performance Optimized implementation (1 VM per subject under test)
65. Performance Optimized implementation (1 VM per subject under test)
66. Performance Results λ make bench_bare => 45.5 msec λ make bench_naive => 1090.0 msec Previous results
67. Performance Results λ make bench_bare => 45.5 msec λ make bench_naive => 1090.0 msec λ make bench_optimized => 83.7 msec 2x slower than bare process > 10x faster than 1 VM per syscall
68. Performance Future work • Add DTrace probes to mruby ◦ Try to understand mruby better • Precompile scripts with mrbc ◦ Only addresses startup time
69. Conclusion
70. Conclusion • Replacing system calls is kinda fun (but tricky) • Still a bit fragile ◦ Kernel extension: run mruby at the kernel level ◦ Brittle, but was easy to set up • mruby is my preferred Ruby ◦ Lightweight ◦ Flexible
71. ありがとう ございます！ Franck Verrot franck@verrot.fr @franckverrot