The Hidden Treasure of Crash Reports?

Sadly, nobody really loves crash reports, but I'm here to change that!

This crash course in crash reports will highlight how these often overlooked files are an invaluable source of information, capable of revealing malware infections, exploitation attempts, or even buggy (exploitable?) system code. Such insights are critical for defense and offense, empowering us to either protect or exploit macOS systems.

To start, we will explain exactly how to understand the structure and information provided in a crash report. Then, we'll show how this information, which often serves as little more than a digital breadcrumb, can however ultimately reveal the exact cause of the crash. Of course, this journey requires a solid understanding of reverse engineering, so we'll briefly touch on topics such as disassembling and debugging ARM64.

Next, we'll apply what we've learned to work through various real-life crashes that revealed flaws such as uninitialized pointers, use-after-frees, and heap overflows. And yes, some still exist on macOS even today.

Transcript

1. The Hidden Treasure …...of Crash Reports?
2. % WHOAMI Objective-See Foundation Patrick Wardle
3. ...who loves crash reports!? 🙋 A QUESTION: What are Crash Reports? "System & user reports about apps/processes that crash" -Apple
4. WHAT YOU WILL LEARN Uncovering malware, bugs, & more! Understanding crash reports (on macOS) Although the talk is focused on understanding crash reports, we'll also touch on topics such as reverse-engineerings, vulnerabilities, & more. ...and come away with a deep appreciation (love?) crash reports!
5. WHO CARES ABOUT CRASH REPORT? ...(almost) everybody, really really should! Security Teams (detect malware & exploits) Developers (fix bugs) Hackers (find bugs) Users (report bugs) Crash Report
6. USERS & DEVELOPERS to understand & get bugs fixed "will be fixed in the next update" :) } bug report w/ crash report
7. SECURITY TEAMS to find (failed) exploits ...0days? "By September 2008 we had built a system that screened millions of crashes for security exploits. On September 25th a crash came in that got my attention--an exploit in netapi32.dll..." The attackers had a remote code execution vulnerability that affected every version of Windows, gave them full control at SYSTEM level rights, left almost no forensic footprint, and could be used anonymously from anywhere on the Internet. Their exploit was 95% reliable. Almost perfect. Almost." -John Lambert/MSRC ranked #45,000 with "exactly 2 hits ever" 0-day (patched as MS08-067)
8. (IOS-FOCUSED) SECURITY STARTUPS ...are all about crash reports (on iOS) ZecOps (acquired by Jamf, for $45M+) iVerify (just raised $12M series A)
9. MALWARE DETECTION PRODUCTS detect malware, as malicious code often crashes! Remember Stuxnet? ...uncovered via crashes ♥ 💥 OSX.ZuRu (originally undetected) "procName" : "UltraEdit", "procPath" : "/Users/USER/Downloads/UltraEdit.app/Contents/MacOS/UltraEdit", "exception" : { "codes":"0x0000000000000001, 0x0000000000000018", "rawCodes":[1,24], "type":"EXC_BAD_ACCESS", "signal":"SIGSEGV", "subtype":"KERN_INVALID_ADDRESS at 0x0000000000000018" } buggy & thus crashes!
10. MALWARE ANALYSTS to analyze malware patch ...to test! LockBit Ransomware ...crashes on macOS
11. HACKERS & INTELLIGENCE AGENCIES fingerprint systems / find exploitable bugs "The automated crash reports are a 'neat way' to gain 'passive access' to a machine, the presentation continues. This passive access to error messages provides valuable insights into problems with a targeted person's computer and, thus, information on security holes that might be exploitable for planting malware or spyware on the unwitting victim's computer." -spiegel.de leaked image "to gather detailed information to better exploit your machine"
12. THE SOURCE OF (ABSOLUTE) TRUTH! example: CrowdStrike on MSN Claim: A Microsoft bug Claim: A NULL-ptr dereference ...(re)tweeted over 25K times! 🤦 confirmed by CrowdStrike & Microsoft Fact: OOB read in CS's driver Incorrect analysis
13. Crash Report Basics
14. WHAT IS A CRASH REPORT What are Crash Reports? "System & user reports about apps/processes that crash" -Apple Understanding reports ...takes some knowledge!
15. WHERE ARE CRASH REPORTS STORED? as .ips files in "DiagnosticReports" directories /Library/Logs/DiagnosticReports/*.ips ~/Library/Logs/DiagnosticReports/*.ips crash reports file extension macOS's Console app
16. AN CRASH DEMO ...and a walk thru int main(int argc, const char * argv[]) { char* a = NULL; *a = 0x41; } 01 02 03 04 05 06 % ./"I Will Crash" zsh: segmentation fault ./"I Will Crash" Translated Report (Full Report Below) ------------------------------------- Process: I Will Crash [30670] Path: /Users/USER/Library/Developer/Xcode/.../Build/Products/Debug/I Will Crash Identifier: I Will Crash Version: 1.0 Code Type: ARM-64 (Native) Parent Process: zsh [26035] Responsible: Terminal [48970] User ID: 501 Date/Time: 2024-04-20 10:41:16.0317 +0900 OS Version: macOS 13.4 (22F66) Buggy code ...run crash crash report Crash report → → 💥
17. CRASHED THREAD ...and exception information Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000 Exception Codes: 0x0000000000000001, 0x0000000000000000 Crashed Thread Exception Information } Exception type Exception specifics (e.g. NULL ptr deref)
18. STACK BACKTRACE from faulting instruction back to callers Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 I Will Crash 0x10458ff78 main + 44 1 dyld 0x18c867f28 start + 2236 .... Stack Frame # Binary Return Address* 0 I Will Crash 0x10458FF78 1 dyld 0x18C867F28 *Frame 0: Faulting address Address of "faulting instruction" location of instruction that caused exception 💥 0x10458ff78 stack (back)trace
19. (CRASHED) THREAD STATE which includes all registers & their values Thread 0 crashed with ARM Thread State (64-bit): x0: 0x0000000000000001 x1: 0x000000016b873818 x2: 0x000000016b873828 x3: 0x000000016b8738f0 x4: 0x0000000000000000 x5: 0x0000000000000000 x6: 0x0000000000000000 x7: 0x0000000000000000 ... x28: 0x0000000000000000 fp: 0x000000016b873580 lr: 0x000000018c867f28 sp: 0x000000016b873550 pc: 0x000000010458ff78 cpsr: 0x80001000 far: 0x0000000000000000 esr: 0x92000046 (Data Abort) byte write Translation fault Register Description x0 - x7 arg 0, ...arg 7 (x0, also return value from function) pc (faulting) instruction lr return address fp / sp stack frame / stack pointer Arm Assembly Internals & Reverse Engineering Author: Maria Markstedter (🦊)
20. LOADED BINARIES including their (ASLR'd) address & path Binary Images: 0x10458c000 - 0x10458ffff I Will Crash (*). ~/Library/Developer/Xcode/DerivedData/.../Debug/I Will Crash 0x18c862000 - 0x18c8f0553 dyld (*) /usr/lib/dyld in-memory address (start, end) Rebase (to 0x10458c000), so disassembly matches the addresses in the crash report
21. ADDRESS OF FAULTING INSTRUCTION location of instruction that caused the exception main: 0x10458ff4c sub sp, sp, #0x40 0x10458ff50 stp fp, lr, [sp, #0x30] 0x10458ff54 add fp, sp, #0x30 0x10458ff58 mov w8, #0x0 0x10458ff5c str w8, [sp, #0x30 + var_1C] 0x10458ff60 stur wzr, [fp, var_4] 0x10458ff64 stur w0, [fp, var_8] 0x10458ff68 stur x1, [fp, var_10] 0x10458ff6c str xzr, [sp, #0x30 + var_18] 0x10458ff70 ldr x9, [sp, #0x30 + var_18] 0x10458ff74 mov w8, #0x41 0x10458ff78 strb w8, [x9] .... 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 "I Will Crash" disassembly Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 I Will Crash 0x10458ff78 main + 44 1 dyld 0x18c867f28 start + 2236 } int main(...) { char* a = NULL; *a = 0x41; 01 02 03 04 in C (for reference) Thread 0 crashed with ARM Thread State x0: 0x0000000000000001 ... x8: 0x0000000000000041 x9: 0x0000000000000000 pc: 0x000000010458ff78
22. A Windows Crash Report ...courtesy of CrowdStrike
23. CRASH: CROWDSTRIKE'S CSAGENT.SYS ...let's start with the crash dump Faulting instruction: mov r9d, dword ptr[r8] thread state (registers) Invalid address: r8 = 0xffff9c8e'0000008a 💥 unmapped! 💥
24. CRASH DUMP & DISASSEMBLY ANALYSIS revealed: out of bounds memory read ... BSOD! mov r8d, [rsi+4] cmp r8d, 0FEh movzx r11d, r8b mov rax, [rdx+8] mov r8, [rax+r11*8] test r8, r8 jz short leave mov r9d, [r8] 01 02 03 04 05 06 07 08 09 10 11 Register Value Description RAX 0xffff868f`7d1a7200 Array (of pointers) R11 0x14 (20d) Index (OOB) R8 0xffff9c8e`0000008a Array[Index] 💥 Array (RAX) RAX + R11*8 Array[Index] }valid (in bounds) invalid (OOB) Update from CrowdStrike ...confirms our analysis 💥
25. ...and my mom asks me for my autograph RESULTS: CLAIMS DEBUNKED, TRUTH REVEALED! Invited on "Good Morning America" 📺 my mom thinks i'm cool! (or maybe is trolling me)
26. macOS Crashes solely from my computer ( ...and solely from normal use! )
27. HOW TO GENERATE CRASH REPORTS? ...just use macOS (or, write security tools) 💥 💥 💥 👨💻 👨💻 Patrick's code ...crashes all the things !
28. CRASH: LULU ...with a EXC_BAD_ACCESS / SIGBUS Process: com.objective-see.lulu.extension [87832] Path: /Library/SystemExtensions/*/com.objective-see.lulu.extension Crashed Thread: 5 com.apple.NSXPCConnection.user.VBG97UB4TA.com.objective-see.lulu.87821 Exception Type: EXC_BAD_ACCESS (SIGBUS) Exception Codes: KERN_PROTECTION_FAILURE at 0x000000016ba9ffe0 Exception Codes: 0x0000000000000002, 0x000000016ba9ffe0 }deep, repeating call stack SIGBUS + Repeating call stack = "Stack exhaustion" (due to recursion?)
29. WHY? recursion, triggered stack exhaustion -(NEFilterNewFlowVerdict*)processEvent:(NEFilterFlow*)flow { ... [self processRelatedFlow:process.key]; 01 02 03 -(void)processRelatedFlow:(NSString*)key { flows = self.relatedFlows[key]; for(NSInteger i = flows.count - 1; i >= 0; i--) { flow = flows[i]; [flows removeObjectAtIndex:i]; [self processEvent:flow] ... 01 02 03 04 05 06 07 08 09 10 ... 300+ flows triggers stack exhaustion "related" flows fix: non-recursion
30. CRASH: YARA ...with a "Code Signature Invalid" pmapped_file->data = mmap(0, pmapped_file->size, PROT_READ, MAP_PRIVATE, pmapped_file->file, offset); 01 02 💥 Crashed when mapped bytes are read (scanned) if binary has invalid signature 0x000000019f55536c ldr w4, [x19] ; file 0x000000019f555370 mov x0, #0x0 ; address: 0x0 0x000000019f555374 mov w2, #0x1 ; protections: PROT_READ 0x000000019f555378 mov w3, #0x6002 ; flags: MAP_PRIVATE + MAP_RESILIENT_CODESIGN + MAP_RESILIENT_MEDIA 0x000000019f55537c mov x5, x20 ; offset 0x000000019f555380 bl imp___auth_stubs__mmap 01 02 03 04 05 06 07 08 ...compare with Apple's implementation found in (/System/Library/PrivateFrameworks/yara.framework/yara)
31. RESOLVED! bug report, & patch + #define MAP_EXTRA_FLAGS MAP_RESILIENT_CODESIGN pmapped_file->size, PROT_READ, - MAP_PRIVATE, + MAP_PRIVATE | MAP_EXTRA_FLAGS, pmapped_file->file, offset); ... 01 02 03 04 05 06 07 08 09 yara's patch yara now invokes mmap (on macOS) with MAP_RESILIENT_CODESIGN ...so, no more crashes!
32. CRASH: ADOBE CRASH REPORTER(!) ...with the release a NULL object Process: Creative Cloud Helper [85153] Identifier: com.adobe.ccd.helper Version: 6.2.0.554 (6.2.0) Code Type: ARM-64 (Native) Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_BREAKPOINT (SIGTRAP) Exception Codes: 0x0000000000000001, 0x000000018a26605c Application Specific Information: *** CFRelease() called with NULL *** Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 CoreFoundation 0x18a26605c CFRelease.cold.1 + 16 1 CoreFoundation 0x18a088a3c CFRelease + 132 ... 5 AdobeCrashReporter 0x1017044ac CRUtils::GetLoggedInUserName() + 60 6 AdobeCrashReporter 0x10170dae8 AdobeCrashReporterInitialize + 64 stack trace show call originated in Adobe's Crash Reporter library ?
33. WHY? an API result not checked, then (always) freed void GetLoggedInUserName_block_invoke(void * _block) { ... x0 = SCDynamicStoreCreate(0x0, @"GetConsoleUser", ...); x21 = r0; x0 = SCDynamicStoreCopyConsoleUser(...); x20 = r0; CFRelease(x21); ... 01 02 03 04 05 06 07 08 09 10 SCDynamicStoreCreate can return NULL, ...here this is not checked & it's always eventually freed! Thus when NULL is returned, CFRelease will crash 💥 fix: check for NULL (as explained by chatGPT) 💥
34. CRASH: OBJ-SEE'S DO NOT DISTURB with a invalid ptr in an Apple's security API? Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: SEGV_NOOP at 0x0000000000000001 Crashed Thread: 6 Application Specific Information: Attempted to dereference garbage pointer 0x1. Originated at or in a subcall of -[FrameworkInterface initIdentity:] Thread 6 Crashed: 0 Security 0xfffe7192df0e SecError 1 Security 0xfffe7189b668 SecCDSAKeyCopyPublicKey(OpaqueSecKeyRef*) 2 Security 0xfffe71751f69 SecKeyCopyPublicKey 3 dnd 0x107464fc1 SecIdentity.deleteIdentity() 4 dnd 0x10743d87d DNDIdentity.deleteIdentity(deleteAssociatedCA:) 5 Do Not Disturb 0x2073e765c -[FrameworkInterface initIdentity:] guard let key = privateKey, if let pubKey = SecKeyCopyPublicKey(key) { .... } 01 02 03 04 DnD code (occasionally) triggered the crash crash in Apple APIs ?
35. WHY? "garbage pointer" in Apple's SecError function SecError: 0xfffe7192deaf mov r11, rsi ; 2nd arg into r11 0xfffe7192df06 test r11, r11. ; 2nd arg NULL? ...leave 0xfffe7192df09 je leave ... 0xfffe7192df0e mov rdx, qword [r11] ; *(2nd arg) 01 02 03 04 05 06 07 SecError in Security.framework faulting instruction Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: SEGV_NOOP at 0x0000000000000001 Attempted to dereference garbage pointer 0x1. Thread 6 Crashed: 0 Security 0xfffe7192df0e SecError bool SecError(OSStatus status, CFErrorRef *error, CFStringRef format, ...) { ... if (error) CFErrorRef previousError = *error; 01 02 03 04 SecError's source code (instruction that triggers crash) map to source code
36. WHY? an error object is never initialized! static SecKeyRef SecCDSAKeyCopyPublicKey(SecKeyRef privateKey) { CFErrorRef *error; BEGIN_SECKEYAPI(SecKeyRef, NULL) ... END_SECKEYAPI 01 02 03 04 05 06 #define END_SECKEYAPI }\ ... catch (const CommonError &err) { \ if (err.osStatus() != CSSMERR_CSP_INVALID_DIGEST_ALGORITHM) { \ ... SecError(status, error, CFSTR("%s"), ... ... 01 02 03 04 05 06 07 08 As Xcode points out, pointer is never initialized!
37. CRASH: KERNEL (VIA OBJ-SEE'S RANSOMWHERE?) accessing invalid memory triggered a page fault *** Panic Report *** panic(cpu 1 caller 0xffffff8008605ecd): Kernel trap at 0xffffff800892544b, type 14=page fault registers: CR0: 0x0000000080010033, CR2: 0xffffff803db4f000, CR3: 0x000000044d97f05c, CR4: 0x00000000001626e0 RAX: 0x0000000000000001, RBX: 0xffffff803db4eff0, RCX: 0x0000000000000000, RDX: 0x0000000000000010 RSP: 0xffffff9222ac3d20, RBP: 0xffffff9222ac3e60, RSI: 0xffffff803db4f000, RDI: 0xffffff803433a2e8 R8: 0x0000000000000000, R9: 0xffffff80448b16e8, R10: 0x0000700001f2f4c0, R11: 0xffffff802f59d4e8 R12: 0xffffff802813e458, R13: 0x000000000000000e, R14: 0xffffff8034339db0, R15: 0x1575312836070096 RFL: 0x0000000000010202, RIP: 0xffffff800892544b, CS: 0x0000000000000008, SS: 0x0000000000000010 Fault CR2: 0xffffff803db4f000, Error code: 0x0000000000000000, Fault CPU: 0x1, PL: 0, VF: 1 Backtrace (CPU 1), Frame : Return Address 0xffffff9222ac39b0 : 0xffffff80084f210c 0xffffff9222ac3a30 : 0xffffff8008605ecd 0xffffff9222ac3c10 : 0xffffff80084a3743 0xffffff9222ac3c30 : 0xffffff800892544b 0xffffff9222ac3e60 : 0xffffff80089de83b ... address of unmapped memory kernel crash report
38. WHY? bug when checking if path is NULL-terminated? *** Panic Report *** panic(cpu 1 caller 0xffffff8008605ecd): Kernel trap at 0xffffff800892544b, type 14=page fault audit_arg_sockaddr: ... 0xffffff800892544b cmp byte ptr[rbx+r13+2], 0 01 02 03 04 void audit_arg_sockaddr(struct kaudit_record *ar, struct vnode *cwd_vp, struct sockaddr *sa) { int slen; struct sockaddr_un *sun; bcopy(sa, &ar->k_ar.ar_arg_sockaddr, sa->sa_len); switch(sa->sa_family) { case AF_UNIX: sun = (struct sockaddr_un *)sa; slen = sun->sun_len - offsetof(struct sockaddr_un, sun_path); if (sun->sun_path[slen] != 0) { .... 01 02 03 04 05 06 07 08 09 10 11 12 13 14 💥 mapping disasm to the source code
39. ( when +1 hits an unmapped page) an off by one, triggers a page fault Off by One *** Panic Report *** ... Fault CR2: 0xffffff803db4f000 WHY? 0xffffff803db4f000 💥
40. APPLE'S "FIX" ...introduced a kernel information leak! void audit_arg_sockaddr(struct kaudit_record *ar, struct vnode *cwd_vp, struct sockaddr *sa) { ... case AF_UNIX: if (sun->sun_len > offsetof(struct sockaddr_un, sun_path)) { /* Make sure the path is NULL-terminated */ strlcpy(path, sun->sun_path, sizeof(path)); ... 01 02 03 04 05 06 07 # hexdump /var/audit/20170406055225.not_terminated 00000110 2f 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 00000120 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 * 000001d0 41 41 41 41 41 41 41 90 99 0b 0f 07 54 38 c4 ba 000001e0 22 83 3b 9e 56 d5 e0 00 info leak
41. ANOTHER BUG CLOSE BY? yes! a kernel heap overflow void audit_arg_sockaddr(struct kaudit_record *ar, struct vnode *cwd_vp, struct sockaddr *sa) { ... bcopy(sa, &ar->k_ar.ar_arg_sockaddr, sa->sa_len); case AF_UNIX: 01 02 03 04 05 06 128 bytes can be > than 128 bytes! 🤦
42. Incident Identifier: 9EE5610B-7A0C-4558-895F-CF876DEB6B07 Hardware Model: iPhone9,1 Process: MobileSMS [10417] Path: /Applications/MobileSMS.app/MobileSMS ... Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000 ... Thread 6 name: Dispatch queue: com.apple.ResponseKit Thread 6 Crashed: 0 CoreFoundation 0x0000000182922efc 0x182909000 + 106236 1 CoreEmoji 0x00000001886b2354 0x1886a6000 + 50004 2 CoreEmoji 0x00000001886b2354 0x1886a6000 + 50004 3 CoreEmoji 0x00000001886b2c80 0x1886a6000 + 52352 Thread 6 crashed with ARM Thread State (64-bit): x0: 0x0000000000000000 x1: 0x00000001add1ad38 ... x20: 0x00000001add1ad38 x21: 0x0000000000000000 ... sp: 0x000000016f1a5b00 pc: 0x0000000182922efc CRASH: (REMOTE) IOS ...with a NULL ptr deref. during emoji processing involves emojis? 🤗
43. Thread 6 crashed with ARM Thread State: x0: 0x0000000000000000 x21: 0x0000000000000000 pc: 0x0000000182922efc WHY? a NULL pointer is passed to Apple's CFStringCompare _CFStringCompare: ... 0000000182922edc mov x21, x0 ... 0000000182922efc ldr x8, [x21] 💥 Null-pointer Deference (x21 is NULL) CFStringCompare is invoked with NULL ...but why?
44. WHY? because of Apple's censorship! int <redacted>_186b5a2ec { ... x20 = CFLocaleCopyCurrent(); x19 = CFLocaleGetValue(x20, kCFLocaleCountryCode); ... x0 = CFStringCompare(x19, @"CN", 0x0); 01 02 03 04 05 06 07 08 compare the locale with "CN" !? ( 💥 will crash if locale is NULL) getting locale can fail (return NULL) ...but this is not checked ! "A Remote iOS Bug: Apple wrote code to appease the Chinese government ...it was buggy" objective-see.org/blog/blog_0x34.html 🇹🇼
45. Recent Crashes ...that revealed 0days in macOS !
46. ------------------------------------- Translated Report (Full Report Below) ------------------------------------- Process: Keychain Access [49865] Path: /System/Applications/Utilities/ Keychain Access.app/Contents/MacOS/Keychain Access Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x302d302d302d3035 -> 0x0000302d302d3035 (possible pointer authentication failure) Exception Codes: 0x0000000000000001, 0x302d302d302d3035 CRASH: APPLE'S KEYCHAIN ACCESS APP with PAC failure after SecCertificateGetData? Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 CoreFoundation 0x189742684 CFRetain + 48 1 Security 0x18c461f78 SecCertificateCopyData + 36 2 Security 0x18c67d4c4 SecCertificateCreateItemImplInstance + 60 3 Security 0x18c41c2a4 SecCertificateGetData + 144 4 Keychain Access 0x102944be0 0x102910000 + 216032 app's entitlements ...juicy ! stack backtrace
47. SecCertificateGetData invoked with a key! (lldb)* thread #1, stop reason = breakpoint 1.1 Security`SecCertificateGetData: -> 0x18c41c214 <+0>: pacibsp 0x18c41c218 <+4>: stp x24, x23, [sp, #-0x40]! Target 0: (Keychain Access) stopped. (lldb) po $x0 <SecCDSAKeyRef 0x600001ce8510: algorithm id: 1, class=1, algorithm=2a, usage=800001ff attrs=39> the parameter passed to SecCertificateGetData is a key The SecCertificateGetData takes a SecCertificateRef, not a SecKeyRef! 🤦 WHY? } 1st arg: SecCertificateRef 2nd arg: CSSM_DATA struct
48. Process: sysextd [38621] Path: /System/Library/Frameworks/SystemExtensions.framework/Versions/A/Helpers/sysextd User ID: 0 Crashed Thread: 2 Dispatch queue: sysextd.extension_manager Exception Type: EXC_BREAKPOINT (SIGTRAP) Exception Codes: 0x0000000000000001, 0x0000000191eed1e8 Thread 2 Crashed:: Dispatch queue: sysextd.extension_manager 0 libswiftCore.dylib 0x191eed1e8 _assertionFailure(_:_:file:line:flags:) + 268 1 sysextd 0x10095bde0 0x100918000 + 277984 CRASH: APPLE'S SYSTEM EXTENSION DAEMON with an assertion failure validating an extension token = [Client_connection auditToken]; clientInfo = getClientInfo(auditToken, ...); if(NULL == clientInfo) { Swift.assertionFailure("Fatal error",..., sysextd/daemon_ipc_nsxpc.swift, ...); asm { ud2 }; } 01 02 03 04 05 06 💥 👨💻
49. IMPACT prevent security tools from loading! architectures = getArchitectures() if(0 == architectures.count) //"assertion failure: \\\"archs.count\\\" -> %lld" _os_crash_msg(); asm { brk #0x1 } } 01 02 03 04 05 06 struct fat_header { uint32_t magic; uint32_t nfat_arch; }; mach-o/fat.h CA FE BA BE 00 00 00 00 01 00 00 07 % log stream --predicate 'process=="sysextd"' sysextd: realizing target path: file:///Applications/PoC.app/Contents/Library/SystemExtensions/foo.bar.extension.systemextension/ sysextd: assertion failure: "archs.count" -> 0 💥 sysextd disassembly set arch count to 0 ...block (other) security tools from loading
50. Process: nesessionmanager [504] Path: /usr/libexec/nesessionmanager User ID: 0 Crashed Thread: 1 Dispatch queue: NESMProviderManager queue Exception Type: EXC_BAD_ACCESS (SIGBUS) Exception Codes: UNKNOWN_0x105 at 0x00000000dac11a30 Exception Codes: 0x0000000000000105, 0x00000000dac11a30 CRASH: APPLE'S NETWORK EXTENSION SESSION MANAGER with an invalid memory access in respondsToSelector Thread 1 Crashed:: Dispatch queue: NESMProviderManager queue 0 libobjc.A.dylib 0x1893448c0 objc_opt_respondsToSelector + 28 ... 5 Foundation 0x18a89635c -[NSString initWithFormat:] + 52 6 nesessionmanager 0x102e72870 0x102e64000 + 59504 stack backtrace NSString initWithFormat: calls respondsToSelector() Crashes in this function generally due to corrupted / invalid object !
51. WHY? "object type" confusion when building a log msg adr x0, #0x102f2afb8 ; @"%@ %@ %@ %s" adr x1, #0x102f2afd8 ; @"SYSEXT_INVALID_MACH_SERVICE_NAME" bl NEResourcesCopyLocalizedFormatString 01 02 03 from NetworkExtension.framework/Versions/A/Resources/ Localizable.loctable returns: "System extension %@ has an invalid %@ key in its Info.plist: The value of the %@ key must be prefixed with one of the App Groups in the %@ entitlement." adr x8, #0x104f6fb94 ; "com.apple.security.application-groups" ... bl sub_1000a31e0 ; NSString initWithFormat: ... 01 02 03 💥 The code asked for string with format specifiers (place holders) "%@ %@ %@ %s", ....but got back one back with "%@ %@ %@ %@" A "C"-string (%s) ...not an string object (%@)
52. A CLOSER LOOK ...a debugger confirms the issue bl sub_1000a31e0 ; NSString initWithFormat: ... 01 💥 (lldb) po [$x0 className] NSPlaceholderString (lldb) x/s $x1 0x1d652a4e8: "initWithFormat:" (lldb) po $x2 System extension %@ has an invalid %@ key in its Info.plist: The value of the %@ key must be prefixed with one of the App Groups in the %@ entitlement. (lldb) x/4gx $sp 0x16b26e520: 0x00000001462065f0 0x0000000104df6ff8 0x16b26e530: 0x0000000104df6ff8 0x0000000104defb94 (lldb) po [0x0000000104defb94 className] error: Execution was interrupted, reason: EXC_BAD_ACCESS (code=261, address=0xdac11a30). (lldb) x/s 0x0000000104defb94 0x104defb94: "com.apple.security.application-groups" %@ is the format string place holder for an Obj-C object { values to "fill" format string the final argument is a C-string, not a string object! confirming in a debugger
53. IMPACT unloads all network extensions as well! % log stream --level debug --predicate="subsystem='com.objective-see.lulu'" com.objective-see.lulu.extension: [com.objective-see.lulu:extension] method '-[FilterDataProvider stopFilterWithReason:completionHandler:]' invoked with 1 com.objective-see.lulu.extension: [com.objective-see.lulu:extension] reason: NEProviderStopReasonUserInitiated malware can trigger the unloading of all network extensions ...or disable the network! } (Ransomware) prevent backups? (Other malware) prevent cloud-based analysis, certificate revocation checks, etc? LuLu (macOS firewall)
54. AND MORE CRASH REPORTS ...so many more! 😬
55. Crash Report Monitoring ...for automated collection & analysis
56. IN A FEW STRAIGHTFORWARD STEPS ...enumerate, monitor, collect, process Enumerate crash report directories Monitor each (via Endpoint Security) 💥 Collect & analyze...
57. es_client_t client; es_event_type_t events[] = {ES_EVENT_TYPE_NOTIFY_CLOSE}; es_new_client(&client, ^(es_client_t *client, const es_message_t *message) { //TODO: handle event }); es_unmute_all_target_paths(client); es_invert_muting(client, ES_MUTE_INVERSION_TYPE_TARGET_PATH); for(NSString* directory in directories) { es_mute_path(client, directory.UTF8String, ES_MUTE_PATH_TYPE_TARGET_PREFIX); } es_subscribe(client, events, sizeof(events)/sizeof(events[0])); 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 WATCH EACH CRASH REPORT DIRECTORY ...via Endpoint Security and "mute inversion" Specify events of interest (e.g. file close) Create endpoint security client Setup mute inversion Watch each crash report directory Subscribe! ...to specified events on specified dirs.
58. ANALYSIS most crash reports: "junk" ...& should be ignored Process: Xcode [1021] Path: /Applications/Xcode.app/Contents/MacOS/Xcode Exception Type: EXC_BREAKPOINT (SIGTRAP) Exception Codes: 0x0000000000000001, 0x00000001a4929ee0 Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libswiftCore.dylib 0x1a4929ee0 _assertionFailure(_:_:file:line:flags:) + 268 1 SourceEditor 0x121299a20 SourceEditorLineAnnotationDropdown.layoutInfoForIndexPath(_:) + 440 2 SourceEditor 0x121299ef4 SourceEditorLineAnnotationDropdown.dropDownItemTextDidEndEditing... Process of interest? Crash on the main (UI) thread? Crash due to an assertion? (e.g. UI crashes, generally need user input)
59. Conclusions ...& take aways
60. Takeaways Find bugs ...and fix || exploit Uncover malware & exploits } 💥 Crash reports are your (best?) friends! ...and much more!
61. ...so who loves crash reports now!? 🙋
62. Interested in Learning More? read, "The Art of Mac Malware" book(s) "The Art of Mac Malware" free @ https://taomm.org Coming soon! Vol. II: (programmatic) detection
63. Objective-See Foundation 501(c)(3) learn more our community efforts ...& support us! 🥰 The Objective-See Foundation objective-see.org/about.html #OBTS Conference College Scholarships Diversity Programs ("Objective-We")
64. Mahalo to the "Friends of Objective-See"