Darwin Breakdown

DTrace + Darwin/BSD = <3

Transcript

1. Darwin Breakdown Vladimir Kirillov iPhoneDevCamp 2012
2. Goals • trace through the whole system • dig into the implementation details • discover lots of interesting stuff • take advantage of awesome tools • DTrace, GDB, a text editor • Find out how to run UIKit-based Applications within iOS Simulator without Xcode
3. DTrace • Dynamic Tracing Framework, first released for Solaris 10 in 2004, open sourced in 2005 • a set of kernel modifications to hot-patch program text at run time • a programming language for control • a library to collect execution state and analyze it • provides observability across the entire software stack
4. D • awk-like programming language • a set of statements (data collection/processing ops) executed when a probe (a dynamic breakpoint) hits and a predicate matched • compiled into DIF objects • safe byte-code, interpreted by the kernel • only forward branches! (means no loops) • like BPF • but not for network packets
5. # cat example.d int64_t passed; BEGIN { ! trace("started"); ! passed = 0; } tick-10s {} tick-10s { ! passed = walltimestamp; } END /passed != 0/ { ! printf("stopped after 10 seconds (at %Y)", passed); }
6. # dtrace -eSs example.d ... DIFO 0x0x10d52cd60 returns D type (integer) (size 4) OFF OPCODE INSTRUCTION 00: 29050001 ldgs DT_VAR(1280), %r1 ! DT_VAR(1280) = "passed" 01: 25000002 setx DT_INTEGER[0], %r2 ! 0x0 02: 0f010200 cmp %r1, %r2 03: 13000006 bne 6 04: 0e000001 mov %r0, %r1 05: 11000007 ba 7 06: 25000101 setx DT_INTEGER[1], %r1 ! 0x1 07: 23000001 ret %r1 NAME ID KND SCP FLAG TYPE passed 500 scl glb r D type (integer) (size 8) ...
7. # dtrace -s example.d dtrace: script './example.d' matched 4 probes CPU ID FUNCTION:NAME 1 1 :BEGIN hello world! 0 178912 :tick-10s 0 178912 :tick-10s ^C 0 2 :END stopped after 10 seconds (at 2012 Feb 10 23:01:52)
8. DTrace Providers • fbt: arbitrary function boundary tracing in kernel text • relies on correct function prologue/epilogue ABI • fasttrap: arbitrary instruction in user program text • like debuggers • sdt/usdt: statically defined tracing • arbitrary traps into DTrace (mostly for convenience and stability), replaced by nops when inactive • several special/custom providers (tick-Ns, profile-N, syscall and wrappers around above)
9. # dtrace -l | wc -l 178274 # ps -ef | wc -l 122 # nm /mach_kernel | wc -l 16110 ...
11. Run > iPhone 5.0 Simulator • What happens after the program gets compiled? • Xcode should probably pop the Simulator • And execute the App binary
12. • Add a NSLog statement... Tracing Xcode (if you had the source)
13. • Add a NSLog statement... Tracing Xcode (if you had the source) • Recompile • Add more • Watch, grep, try again
14. Tracing Xcode (naively) • Attach GDB to the process •break execve •break posix_spawn •continue •... •print (char *)$rdi # or $rsi for posix_spawn • What if it spawns some other process which does the job? • What if they use any kind of IPC?
15. Tracing Xcode • In UNIX all process images start from either execve or posix_spawn syscalls # man -k 'new process' execsnoop(1m) - snoop new process execution. Uses DTrace fork(2) - create a new process newproc.d(1m) - snoop new processes. Uses DTrace vfork(2) - spawn new process in a virtual memory efficient way
16. # execsnoop UID PID PPID ARGS 501 5962 3391 git 501 3391 342 Xcode 501 3391 342 Xcode 501 5963 3391 git 501 5965 40 SFLIconTool 501 5964 342 iPhone Simulator 501 5966 5964 launchctl 501 40 1 coreservicesd 501 5974 342 vot 501 5975 342 aggregated 501 5970 342 mediaremoted 501 5972 342 itunescloudd 501 5968 342 ubd 501 5973 342 BTServer 501 5969 342 mstreamd 501 5976 5964 SimulatorBridge 501 5971 342 locationd 501 5977 342 securityd 501 5978 342 itunesstored 205 5979 1 locationd 501 5980 342 apsd 501 5981 342 installd 501 5982 342 lsd 501 5983 5964 SpringBoard 501 5984 342 AppleIDAuthAgent 0 5985 1 xpchelper 501 5986 342 iPhoneDevCamp
17. # execsnoop UID PID PPID ARGS 501 5962 3391 git 501 3391 342 Xcode 501 3391 342 Xcode 501 5963 3391 git 501 5965 40 SFLIconTool 501 5964 342 iPhone Simulator 501 5966 5964 launchctl 501 40 1 coreservicesd 501 5974 342 vot 501 5975 342 aggregated 501 5970 342 mediaremoted 501 5972 342 itunescloudd 501 5968 342 ubd 501 5973 342 BTServer 501 5969 342 mstreamd 501 5976 5964 SimulatorBridge 501 5971 342 locationd 501 5977 342 securityd 501 5978 342 itunesstored 205 5979 1 locationd 501 5980 342 apsd 501 5981 342 installd 501 5982 342 lsd 501 5983 5964 SpringBoard 501 5984 342 AppleIDAuthAgent 0 5985 1 xpchelper 501 5986 342 iPhoneDevCamp ... Where are the arguments?
18. # cat /usr/bin/execsnoop # Written using DTrace (Solaris 10 3/05). /* ... */ syscall::execve:return, syscall::posix_spawn:return /* ... */ { /* ... */ printf("%s\n", curpsinfo->pr_psargs); } # ack -a pr_psargs /usr/lib/dtrace /usr/lib/dtrace/darwin.d 158:! char pr_psargs[80];! /* initial characters of arg list */ 197:! pr_psargs = P->p_comm; /* XXX omits command line arguments XXX */ 238:! pr_psargs = xlate <psinfo_t> ((struct proc *)(T->task- >bsd_info)).pr_psargs; /* XXX omits command line arguments XXX */ # vim ~/execsnoop.real.d
19. •execve(const char *path, char *const argv[], char *const envp[]); • Kernel must copy this array to own address space • Let’s snoop argv it after it gets copied • Time to read the kernel :-) •http://opensource.apple.com/tarballs/xnu/ xnu-1699.24.23.tar.gz Snooping exec
20. Snooping exec • Interesting functions found in bsd/kern/kern_exec.c • they read argv into kernel space struct image_params::ip_startargv • those functions are static • vanilla /mach_kernel does not have them in the symbol list • DTrace fbt does not see them • recompiling a kernel is a bad idea for many reasons • want a struct image_params *!
21. Snooping exec int posix_spawn(proc_t ap, struct posix_spawn_args *uap, int32_t *retval) { ! /* ... */ ! char *bufp = NULL; ! struct image_params *imgp; ! /* ... */ ! /* ! * Allocate a big chunk for locals instead of using stack since these ! * structures are pretty big. ! */ ! MALLOC(bufp, char *, (sizeof(*imgp) + sizeof(*vap) + sizeof(*origvap)), M_TEMP, M_WAITOK | M_ZERO); ! imgp = (struct image_params *) bufp; ! /* ... */ }
22. execsnoop.real.d fbt::__mac_execve:entry, fbt::posix_spawn:entry { self->want_malloc = 1; } /* * First _MALLOC call inside execve/posix_spawn allocates memory * for struct image_params, which will later be used to store * pointers to copied in argv vector. */ fbt::_MALLOC:return /self->want_malloc == 1/ { self->imgp = (struct image_params *)arg1; self->want_malloc = 0; }
23. execsnoop.real.d /* * At this point we know that the ip_startargv and friends are * filled in. */ proc:::exec-success { this->arglen = self->imgp->ip_endargv - self->imgp->ip_startargv; this->arg = self->imgp->ip_startargv; printf("[%d->%d] ", ppid, pid); } proc:::exec-success /this->arglen > 0/ { printf("%s ", stringof(this->arg)); this->arglen -= strlen(stringof(this->arg)) + 1; this->arg += strlen(stringof(this->arg)) + 1; }
24. execsnoop.real.d #define ITER() \ proc:::exec-success \ /this->arglen > 0/ \ { \ printf("%s ", stringof(this->arg)); \ this->arglen -= strlen(stringof(this->arg)) + 1; \ this->arg += strlen(stringof(this->arg)) + 1; \ } ITER() ITER() ITER() ITER() ITER() ITER() ITER() ITER()
25. # dtrace -Cs ~/execsnoop.real.d [3391->6555] git status --porcelain [3391->6555] dvtexec /tank/proger/dev/iosdev 75 77 0 14 ... [342->6564] itunescloudd [342->6560] ubd [342->6556] iPhone Simulator.app/Contents/MacOS/iPhone Simulator -SessionOnLaunch NO [40->6557] SFLIconTool com.apple.recentitems RecentApplications [6556->6558] launchctl load -S Aqua /var/folders/vc/_v_13s_j21j0__xg849gybq40000gn/T/com.apple [40->6559] lssave 0 [342->6565] BTServer [342->6567] aggregated [342->6566] vot -x [342->6562] mediaremoted [342->6563] locationd [6556->6568] iPhone Simulator.app/Contents/MacOS/SimulatorBridge 6556 [342->6561] mstreamd [342->6569] itunesstored [342->6570] securityd [6556->6575] SpringBoard -SBDisableAutoDim YES -SBAutoLockTime -1 -SBAutoDimTime -1 -SBDontLoc [1->6571] locationd [342->6572] apsd [342->6573] installd -t 30 [342->6574] lsd [1->6577] xpchelper [342->6576] AppleIDAuthAgent [342->6578] iPhoneDevCamp [3391->6580] sh /Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/gdb --arch=i38 [6581->6582] arch [6580->6586] dirname /Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/gdb [3391->6580] gdb-i386-apple-darwin --arch i386 --interp= | ruby -ne 'l = []; $_.split(" ").each_with_index{|s,i| l << (i == 1 ? s.split("/")[-1] : s)}; puts l.join("
26. Let’s try just execute it # pwd /tank/proger/Library/Application Support/iPhone Simulator/5.0/Applications/ 9F9A9793-CFE9-4B3E-A767-EFB66D55A99A # ./iPhoneDevCamp.app/iPhoneDevCamp dyld: Library not loaded: /System/Library/Frameworks/UIKit.framework/UIKit Referenced from: /tank/proger/Library/Application Support/iPhone Simulator/5.0/ Applications/9F9A9793-CFE9-4B3E-A767-EFB66D55A99A/./iPhoneDevCamp.app/ iPhoneDevCamp Reason: image not found [1] 7017 trace trap ./iPhoneDevCamp.app/iPhoneDevCamp # env DYLD_ROOT_PATH=/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/ iPhoneSimulator5.0.sdk ./iPhoneDevCamp.app/iPhoneDevCamp 2012-02-11 01:17:12.547 iPhoneDevCamp[7029:fb03] Warning: CFFIXED_USER_HOME is not set! It should be set to the simulated home directory. 2012-02-11 01:17:12.551 iPhoneDevCamp[7029:fb03] Warning: IPHONE_SIMULATOR_ROOT is not set! It should be set to the path of the SDK. Terminating since there is no system event server. (Run the EventPump or pass the argument "-RegisterForSystemEvents" if you want to run without SpringBoard.
27. Let’s try just execute it # pwd /tank/proger/Library/Application Support/iPhone Simulator/5.0/Applications/ 9F9A9793-CFE9-4B3E-A767-EFB66D55A99A # env DYLD_ROOT_PATH=/Developer/Platforms/iPhoneSimulator.platform/ Developer/SDKs/iPhoneSimulator5.0.sdk \ ./iPhoneDevCamp.app/iPhoneDevCamp -RegisterForSystemEvents 2012-02-11 01:17:19.143 iPhoneDevCamp[7032:fb03] Warning: CFFIXED_USER_HOME is not set! It should be set to the simulated home directory. 2012-02-11 01:17:19.148 iPhoneDevCamp[7032:fb03] Warning: IPHONE_SIMULATOR_ROOT is not set! It should be set to the path of the SDK. Couldn't find any font cache file. 2012-02-11 01:17:19.185 iPhoneDevCamp[7032:fb03] nil passed to [UILabel setFont:] and [UIButtonLabel defaultFont] is also nil. Don't know what to do, so leaving font as (null) 0.000000 ...
28. Let’s try just execute it # pwd /tank/proger/Library/Application Support/iPhone Simulator/5.0/Applications/ 9F9A9793-CFE9-4B3E-A767-EFB66D55A99A # env DYLD_ROOT_PATH=/Developer/Platforms/iPhoneSimulator.platform/ Developer/SDKs/iPhoneSimulator5.0.sdk \ ./iPhoneDevCamp.app/iPhoneDevCamp -RegisterForSystemEvents 2012-02-11 01:17:19.143 iPhoneDevCamp[7032:fb03] Warning: CFFIXED_USER_HOME is not set! It should be set to the simulated home directory. 2012-02-11 01:17:19.148 iPhoneDevCamp[7032:fb03] Warning: IPHONE_SIMULATOR_ROOT is not set! It should be set to the path of the SDK. Couldn't find any font cache file. 2012-02-11 01:17:19.185 iPhoneDevCamp[7032:fb03] nil passed to [UILabel setFont:] and [UIButtonLabel defaultFont] is also nil. Don't know what to do, so leaving font as (null) 0.000000 ... Works, but no UI and SpringBoard.
29. Replicating the environment • GDB is enough to retrieve it • nice CLI to attach to a process and examine state • uses ptrace(2) interface • supports scripting • can implement various data structure traversals (lists, trees, hashes, etc)
30. # cat gdbenv set shlib-path-substitutions / /Developer/Platforms/ iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator5.0.sdk/ define penv ! set $env = (char **)environ ! set $i = 02 ! while $env[$i] != 0 ! ! p $env[$i] ! ! set $i = $i+1 ! end end
31. # pid=$(pgrep iPhoneDevCamp); gdb -q -x ./gdb_sim "$(ps -o comm= -p $pid)" -p $pid 0x9184bc22 in mach_msg_trap () (gdb) penv $1 = 0xbffff7e8 "PATH=/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin" $2 = 0xbffff81e "TMPDIR=/tank/proger/Library/Application Support/iPhone Simulator/5.0/Applications/9F9A9793-CFE9-4B3E-A767-EFB66D55A99A/tmp" $3 = 0xbffff899 "SHELL=/bin/zsh" $4 = 0xbffff8a8 "HOME=/tank/proger/Library/Application Support/iPhone Simulator/5.0/Applications/9F9A9793-CFE9-4B3E-A767-EFB66D55A99A" ...
32. • An operating system facility which allows processes to talk to each other • Darwin has many options: • BSD files (including pipes, FIFOs, sockets, etc) • SYSV Shared Memory (ipcs(1), shm_open(2), etc) • Mach ports • the core for {CF,NS}{Mach,Message}Port • NSDistibutedNotificationCenter • Mach Interface Generator mig(1) • Mach Bootstrap launchd(1) • XPC IPC
33. (very brief) Mach Ports Theory • Ports are unidirectional channels • a message queue • only one task has a receive right for it • many tasks can have send rights to it • Mach task (struct task) has an associated port namespace (struct ipc_space) • each port has a 32-bit number in this namespace • Ports are used by the system to represent system resources (each port may be a ‘reference to some object’)
34. % class-dump /Developer/Platforms/iPhoneSimulator.platform/ Developer/Library/PrivateFrameworks/ DVTiPhoneSimulatorRemoteClient.framework/Versions/A/ DVTiPhoneSimulatorRemoteClient | grep launch - (BOOL)_launchSimulatorApplicationGivingError:(id *)arg1; - (BOOL)_launchSimulatorGivingError:(id *)arg1; - (BOOL)_launchSimulatorApplicationGivingError:(id *)arg1 sessionOnLaunch:(BOOL)arg2; ... PS: Hint for the eager
35. Introspecting Mach Soon