ASAN integreation with Buildservice

How to integrate Address Sanitizer with Build Service

Transcript

1. ASAN JUDO THE GENTLE WAY OF BREAKING C/C++ CODE Victor Pereira (vpereira@suse.de)
2. AGENDA 1. Motivation 2. ASAN introduction and main problem with C/C++ 3. SDL for packages/packagers 4. Integration with OBS 5. ASAN Tumbleweed? 6. next steps and ideas
3. MOTIVATION Identify possibile security vulnerabilities in C/C++ For all packages In a non-intrusive way make it part of our development process
4. THE PROBLEM WITH C/C++... Vast majority of security issues come from bad C/C++ memory management memory corruption like buffer over ows, double free, use after free, out of bound reads, ...
5. IF IT WASN'T BAD ENOUGH.. Access to invalid memory is "unde ned behavior" So worse than abort is if the software keep running ... How to prevent it?
6. DRUM ROLL... https://www.rust-lang.org/
7. ADDRESS SANITIZERS (ASAN) AddressSanitizer (or ASan) is a programming tool that detects memory corruption bugs such as buffer over ows or use-after-free AddressSanitizer is based on compiler instrumentation and directly-mapped shadow memory. AddressSanitizer is available for Clang and GCC. Unde ned behaviors are aborted On average, the instrumentation increases processing time by about 73% and memory usage by 340%
8. ... INSTRUMENTATION INCREASES PROCESSING TIME BY ABOUT 73% AND MEMORY USAGE BY 340%
9. USING OBS TO FIND SECURITY ISSUES
11. ... MEMORY USAGE BY 340%
13. ADDRESS SANITIZER (ASAN) IN A NUTSHELL CFLAGS="-fsanitize=address" LDFLAGS="-fsanitize=address" "Acceptable overhead" Performance degradation and lot of memory (problem for OBS) It works - USUALLY - out of the box Bugs and false positive with ASAN are rare
14. CONCLUSION ABOUT ASAN It nd bugs [1] A "Must" if you want to run AFL [2] Unde ned behavior are aborted As anti-exploitation measure, it isn't a valid option Maybe it even introduces new vulnerabilities [1]: https://blog.hboeck.de/archives/868-How- Heartbleed-couldve-been-found.html [2]: https://fuzzing-project.org/tutorial2.html
16. SDL(C) 101 SDL(C) stands for Security Development life cycle Normally it starts in the Planning and Requirement phase Threat modeling happens in the architecture and design phase The Test cases are de ned before coding. Coding phase Testing Release and Maintenance
17. SDL IS NICE BUT... We downstream production ready open source projects Which we have little or no in uence in upstream Upstream has it's own (if any) software development methodology We have 10k+ packages
18. SDL FOR DOWSTREAM PACKAGES There is no Planning and Requirements step (Maybe FATE/ECO) There is no Threat modeling There is (almost) no coding There is Packaging There is building There is test case de nitions There is testing There is release and maintenance
19. SO THE IDEA IS: Jump in the testing/release step 1. Enable globally ASAN in a OBS project without change (almost) any package spec 2. Build the project 3. Check the failures, looking for possible security issues 4. Open an AUDIT bug for security team 5. if con rmed, handle it as security incident 6. Con gure a Factory ISO with our ASAN repository 7. Run it with OpenQA
20. INTEGRATING ASAN WITH OBS OBS stands for openbuild service tool used to build our packages and products We want to avoid change every single RPM spec le WARNING: some OBS/RPM kung-fu knowledge is needed
21. Miyamoto Musashi Which we translate into: Anonymous developer/netadmin from one thing, know ten thousand things From one problem, learn ten thousand things
22. FIRST TRY: SET CFLAGS AND LDFLAGS ON OBS PROJECT CONFIGURATION Set "-fsanitize=address" in opt ags [1] Opt ags exports compiler ags to the build. They will only have an effect when the spec le is using $RPM_OPT_FLAGS. Not all packages use it It doesn't set the LDFLAGS (maybe xable with -Wl ...) Reference: http://openbuildservice.org/help/manuals/obs-
23. reference-guide/cha.obs.build_con g.html
24. SECOND TRY: OVERWRITE %CONFIGURE MACRO overwrite %con gure rpm macro in the project con guration append -fsanitize=address to CFLAGS, LDFLAGS and CXXFLAGS it works, but not all packages call %con gure.. Autotools is just one of the many other tools (i.e Cmake, scons, etc)
25. Macros: %_configure ./configure %configure \ CFLAGS="-fsanitize=address ${CFLAGS:-%optflags}" ; export CFLAGS ; \ CXXFLAGS="-fsanitize=address ${CXXFLAGS:-%optflags}" ; export CXXFLAGS LDFLAGS="-fsanitize=address ${LDFLAGS}" ; export LDFLAGS ; \ FFLAGS="${FFLAGS:-%optflags}" ; export FFLAGS ; \ %{_configure} --host=%{_host} --build=%{_build} \\\ --program-prefix=%{?_program_prefix} \\\ --disable-dependency-tracking \\\ --prefix=%{_prefix} \\\ --exec-prefix=%{_exec_prefix} \\\ --bindir=%{_bindir} \\\ --sbindir=%{_sbindir} \\\ --sysconfdir=%{_sysconfdir} \\\ --datadir=%{_datadir} \\\
26. THIRD TRY: GCC WRAPPER Small shell script which set the gcc ags and call gcc it works for almost 70% of all packages thats the approach I'm using today 85% of packages successfully compiled
27. GCC WRAPPER #!/bin/sh # it points to gcc/gcc-c++ binary REAL=.... if ! echo "$@" | grep -q ‘__KERNEL__\|-nostdlib’; then $REAL -fsanitize=address -fno-common –U_FORTIFY_SOURCE "$@ else $REAL "$@" fi
28. GCC WRAPPER SETUP gcc.spec le: %post -n gcc-sanitizer echo "GCC_VERSION=%{gcc_version}" > /etc/gcc-sanitizer.defau rm /usr/bin/gcc rm /usr/bin/g++ rm /usr/bin/cc ln -sf /usr/bin/gcc-sanitizer /usr/bin/gcc ln -sf /usr/bin/gcc-sanitizer /usr/bin/g++ ln -sf /usr/bin/gcc-sanitizer /usr/bin/cc
29. PROBLEMS FOUND Lot of packages with memory leak Test cases normally are normally not well written Memory usage is higher than normal con gure scripts bad assumptions (pthreads for example) Libtool custom memory allocators
30. PROBLEMS FOUND: LIBTOOL When linking shared libs, libtool lter unknown ags from LDFLAGS for libtool, ASAN ags are unknown it was xed in upstream didn't have time to investigate it further
31. PROBLEMS FOUND: PTHREAD libasan provides pthread_create() but not full API con gure script check for pthread_create() and assume lpthread isnt needed it breaks all packages which need pthreads solution: always link against pthreads :)
32. PROBLEMS FOUND: CUSTOM MEMORY ALLOCATORS OpenSSL or GLib2 implement their own memory allocators ASAN is just able to detect problems with malloc/free, new/delete allocators Workaround would be to try to con gure the application to use the standard allocators
33. PROBLEM FOUND: STRANGE COMBINATIONS ASAN executable, non-ASAN library: works ASAN library, non-ASAN executable: break OBS package ordering jiu-jitsu
34. ASAN TUMBLEWEED: THE TRUE PATH TO ENLIGHTENMENT?
35. The short answer is no Google advices against to put in production ASAN uses enviroment variables without checking for secure execution of setuid binaries It may introduce new vulnerabilities ASAN_OPTIONS='suppressions="/foo root:passwdhash:12345:0::::: bar" log_path=foo' ./suid-binary
36. it writes to foo.$PID using escalated priviledge regular user can write into root owned les before run it, create foo.{1,2,3,..PID_MAX_LIMIT} symlinks to /etc/shadow run it writes in /etc/shadow: AddressSanitizer: failed to read suppressions file '/foo root:passwdhash:12345:0::::: bar'
37. CONFIGURE YOUR PROJECT TO USE IT:
38. EXAMPLE OF STACK TRACE FOUND ON OBS
40. EXAMPLE OF STACK TRACE FOUND ON OBS
41. NEXT STEPS AND IDEAS?
42. Integrate it with Factory and SLES13 If its too heavy, maybe just the setuid or packages which runs as root? Test it with openQA x bugs with gcc wrapper and toolchain Run AFL fuzzer against the RPM with ASAN enabled Set services running on ASAN (nginx, apache, post x, exim, etc) i.e set up ASAN enabled post x and pipe all my spam folder through this mail server i.e set up ASAN enabled apache and let if face the wild internet
43. CONCLUSIONS Mix between Hacweek, daily business and subbotnik if it nd bugs, great! dont put it in production, unless you know what you are doing help is needed!
44. THANK YOU!