Practical ransomware detection on macOS (via maths, not AI)

Ransomware remains a serious threat to macOS users, especially as Apple devices continue to gain traction in the enterprise. Despite its many variants, most ransomware shares a fundamental behaviour: rapidly transforming user files into encrypted data.

This talk presents a practical, maths-driven approach to generically detecting and stopping ransomware on macOS. While this approach builds on established ideas, it is now far more practical with Apple's Endpoint Security framework. By leveraging Endpoint Security and entropy-based analysis, we can distinguish benign file activity from encryption and further differentiate encrypted data from compressed data using simple mathematical heuristics. This approach avoids signatures and machine learning, instead relying on observable behaviour and measurable properties of data.

We will walk through the implementation in a free, open-source tool, including real-world testing, and demonstrate the approach against macOS ransomware samples. We will also acknowledge limitations and possible avenues for evasion.

The result is a lightweight, explainable, and practical method for detecting ransomware on macOS using maths, not AI.

Transcript

1. Practical Ransomware Detection (via math, not AI) Patrick Wardle, Objective-See Foundation
2. % WHOAMI Patrick Wardle Building core macOS detection & agent monitoring components Objective-See Foundation "Art of Mac Malware" Book Series DoubleYou
3. Imagine… A brand new, undetected, metamorphic ransomware that targets your macOS users… …yet there is a way to stop it with no prior knowledge of the malware! …intrigued!? 🤔
4. Today, we'll explore macOS ransomware, past and present… …and present a practical approach to generic protection! (macOS) Ransomware an Idea Implementation Results
5. macOS Ransomware
6. "FBI" "ransomware" …though really is just browser-based "scareware" 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 <script type="text/javascript"> show_count = 150; for (i = 0; i < show_count; i++) { frame = document.createElement("iframe"); … frame.setAttribute("src", "close.html"); document.body.appendChild(frame); } … function areYouSure() { str = 'YOUR BROWSER HAS BEEN LOCKED…'; alert(str); return str; } } window.onbeforeunload = areYouSure; Javascript logic Ransom message Floods the victim with alerts, …creating an illusion that ransomware has locked the computer
7. FileCoder …yes, ransomware, but incomplete % file FileCoder FileCoder: Mach-O 64-bit executable x86_64 01 02 03 04 05 06 07 08 09 10 a native mach-O binary <p>Don't you worry! <span style="font-weight:600;">We can help you restore your documents.</span> As a matter of fact, we are the only ones that can help you with that. For a small fee we are willing to restore all your documents and guarantee you that it will <span style="font-weight:600;">never</span> happen again!</p> } Ransom message "FileCoder encrypts only its own files" -Kaspersky + its encryption key is static & stored within the binary 🤪
8. Gopher …proof of concept, to prove a (valid) point "Its goal is to demonstrate that there are really no special barriers in OS X against crypto ransomware. This menace hasn't arrived to OS X purely because of laziness from malware authors and scale, since it should be magnitudes more profitable against Windows targets." -Pedro Vilaca encrypting documents 01 02 03 04 05 06 07 08 09 10 11 12 NSString *extension = [[NSURL fileURLWithPath:filePath] pathExtension]; if ([extension isEqualToString:@"docx"]) [targetFiles addObject:filePath]; for (id object in targetFiles) { ... fd = open([object UTF8String], O_RDWR); unsigned char *source_buf = mmap(0, filestat.st_size, PROT_READ, MAP_SHARED, fd, 0); unsigned char *buf = malloc(filestat.st_size + crypto_box_SEALBYTES); crypto_box_seal(buf, source_buf, filestat.st_size, session_pub_key); ... } Encryption logic (via libsodium)
9. KeRanger the first, in-the-wild functioning ransomware for Macs % codesign -dvvv /Volumes/Transmission/Transmission.app … Identifier=org.m0k.transmission Format=app bundle with Mach-O thin (x86_64) TeamIdentifier=Z7276PX673 Hacked: Transmission (BT client) 01 02 03 04 05 06 07 08 09 10 11 12 Signed (but infected) Transmission app int main(int argc, char **argv) { payloadPath = appResourcesPath(argv[0]) + "/General.rtf"; executablePath ="~/Library/kernel_service"; payload = readFile(payloadPath); writeFile(executablePath, payload); chmod(executablePath, OWNER_EXECUTE); system(executablePath); return NSApplicationMain(argc, argv); } main() copies General.rtf to ~/Library/kernel_service ...and executes it
10. KeRanger a closer look at its encryption Retrieves RSA public key from C&C Encrypts files under /Users & /Volumes % lldb ~/Library/kernel_service ... Process 1343 stopped kernel_service`encrypt_file: -> 0x1000013c0 <+0>: pushq %rbp 0x1000013c1 <+1>: movq %rsp, %rbp Target 0: (kernel_service) stopped. Input / Output (lldb) x/s $rdi 0x304167ed0: "/Users/user/.DS_Store" (lldb) x/s $rsi 0x304167ad0: "/Users/user/.DS_Store.encrypted" Observing encryption (in a debugger)
11. LockBit …from Windows, to macOS (arm64) [19:16:45][74073472l][+] Launch parameters: ./locker_Apple_M1_64 -i '/Users/user/Downloads/ lock_me_up' -m 16 -w 0 -b 0 -r 0 -l 1 -n 0 -d 1 -e '' -s 10 -p test -o 0 -t 0 -f 1 -a 0 -z 0 -y 0 ~~~~~~~~~~~~~~~Hardware~~~~~~~~~~~~~~~~~~ [+] Start encrypting file /Users/user/Downloads/lock_me_up/DazzleSpy.zip [+] Start encrypting file /Users/user/Downloads/lock_me_up/DazzleSpy.zip... [+] End file /Users/user/Downloads/lock_me_up/DazzleSpy.zip size 223464 time 1672883981 is encrypted. Checksum after encryption 3269819564 … arm64 Read more: "The LockBit ransomware comes for macOS" objective-see.org/blog/blog_0x75.html
12. And all others …EvilQuest, MacRansom, etc... MacRansom EvilQuest TurtleRansom
13. Towards Generic Ransomware Detection
14. An Observation (#1) "the cost of waiting" (aka why normal detection methods fail) (New) ransomware can destroy file access before detection signatures arrive! …real world consequences
15. An Observation (#2) All ransomware behaves the same (encrypts files), ...giving us a generic detection heuristic !
16. A (proposed) heuristic Monitor for: The rapid creation …of encrypted files …by an untrusted process Can we efficiently implement this, and generically thwart most (all?) macOS ransomware?
17. Implementation via Endpoint Security and math! Monitor for: The rapid creation …of encrypted files …by an untrusted process } Endpoint Security (e.g file/process events) and detecting encrypted files !? ...via math !
18. Endpoint Security (ES) Events ...that you can subscribe to for your own "client" Events found in an es_event_type_t structure, in EndpointSecurity/ESTypes.h % grep "ES_EVENT_TYPE_" MacOSX.sdk/usr/include/EndpointSecurity/ESTypes.h | wc -l 167 over 160 events you can (mostly) subscribe to (though some, only on newer versions of macOS) Events of interest (for ransomware detection): ES_EVENT_TYPE_NOTIFY_CLOSE/ES_EVENT_TYPE_NOTIFY_RENAME notify a client that a file was just closed or renamed ES_EVENT_TYPE_NOTIFY_EXEC notify a client that a process was just started
19. Creating an ES Client with a event (callback) handler block @import EndpointSecurity; es_client_t* client = nil; int main(int argc, const char * argv[]) { //callback block es_handler_block_t handler = ^(es_client_t *client, const es_message_t *msg) { //TODO: add code to parse/process message }; //create ES client es_new_client(&client, handler); es_new_client can return: } 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 ES_NEW_CLIENT_RESULT_SUCCESS ES_NEW_CLIENT_RESULT_ERR_NOT_ENTITLED ES_NEW_CLIENT_RESULT_ERR_NOT_PERMITTED ES_NEW_CLIENT_RESULT_ERR_NOT_PRIVILEGED es_new_client
20. Subscribing specify events of interest and subscribe 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 int main(int argc, const char * argv[]) { //events of interest es_event_type_t events[] = {ES_EVENT_TYPE_NOTIFY_CLOSE}; //number of events int count = sizeof(events)/sizeof(events[0]); //client creation/callback block removed //subscribe es_subscribe(client, events, count); //run loop (don't want to exit) [[NSRunLoop currentRunLoop] run]; } } Specify events & subscribe es_subscribe once subscribed, events will be delivered to your handler !
21. The ES Message and the event specific structure it contains 01 02 03 04 05 06 07 //callback block es_handler_block_t handler = ^(es_client_t *client, const es_message_t *msg) { //TODO: add code to parse message }; $ less EndpointSecurity/ESMessage.h ... typedef struct { uint32_t version; struct timespec time; uint64_t mach_time; uint64_t deadline; es_process_t * process; uint8_t reserved[8]; es_action_type_t action_type; union { es_event_id_t auth; es_result_t notify; } action; es_event_type_t event_type; es_events_t event; uint64_t opaque[]; } es_message_t; 'responsible' process event + event data es_message_t struct + es_events_t event data is event-type specific ! for example: ES_EVENT_TYPE_NOTIFY_CLOSE is delivered w/ an es_event_close_t
22. The ES Handler Block ...invoked for each matching event (e.g. file close) 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 //handler block es_handler_block_t handler = ^(es_client_t *client, const es_message_t *msg) { es_process_t* process = msg->process; NSString* path = convertStringToken(&process->executable->path); NSLog(@"new event (%d) from %@", event, path); u_int32_t event = msg->event_type; if(event == ES_EVENT_TYPE_NOTIFY_CLOSE) { //ignore non-modified files if(!message->event.close.modified) return; }; } Responsible process Ignore: trusted processes event-specific logic (file close) //TODO: is file encrypted? (example) handler/callback logic Great, now we can efficiently monitor for the creation of files ...but we don't care about all files! Ignore: non-modified files Ignore: non-encrypted files
23. Why "untrusted" processes? …as trusted programs can create encrypted files don't flag (would be false positives) Cloud-sync tools Anti-virus tools Backup tools all can rapidly create encrypted files All are notarized ..and from trusted companies
24. How to classify an "untrusted" process? …(for now) not platform binary && not notarized ES messages (e.g. file close) …contain responsible process 01 02 03 04 05 06 07 08 09 typedef struct { audit_token_t audit_token; pid_t ppid; … uint32_t codesigning_flags; bool is_platform_binary; bool is_es_client; … } es_process_t; part of OS Platform binary (flag) 01 02 03 04 05 06 07 08 09 10 11 //ad hoc binaries aren't notarized if(kSecCodeSignatureAdhoc & csFlags) return NO; //gotta have hardened runtime to get notarized if(!(kSecCodeSignatureRuntime & csFlags)) return NO; "is notarized" ? static SecRequirementRef notarizedReq = nil; SecRequirementCreateWithString(CFSTR("notarized"), kSecCSDefaultFlags, &notarizedReq); return (errSecSuccess == SecCodeCheckValidity(dynamicCode, kSecCSDefaultFlags, notarizedReq)); Notarization check
25. A tangent: Is this file encrypted?
26. How to classify a file as encrypted? …some thoughts on cipher text What does cipher text look like? Encrypted data looks random vs. This "signal" can survive changes in algorithm or malware family!
27. Encrypted files …have high entropy (byte distribution) H = −Σ pᵢ log₂(pᵢ) Shannon entropy Measure the distribution of the 256 possible byte values (0 -> 255) Use base-2 logarithms for a maximum of 8 bits per byte } One repeated byte → entropy 0 All 256 byte values equally distributed → entropy 8 ...encrypted data often has entropy close to 8
28. High Entropy …also found in compressed files File Algorithm (Encryption / Compression) Entropy test.html None 5.12 test.encrypted AES 7.98 test.lzma lzma 7.87 test.zip deflate 7.93 test.rar rar 7.99 test.png deflate 7.91 ...entropy alone cannot reliably separate compression from encryption 😣
29. Compressed vs. Encrypted …via Chi-square statistic & Monte Carlo π estimation compressed data should have some patterns? "Differentiate Encryption From Compression Using Math" www.devttys0.com/2013/06/differentiate-encryption-from-compressionusing-math } Chi-square: how far the byte frequencies deviate from a uniform distribution Monte Carlo: estimates π from byte-derived coordinates ...their combination adds evidence beyond entropy !
30. Chi-square measures deviation from a uniform byte distribution χ² = Σ (observed − expected)² / expected Chi-square Count occurrences of each byte value, 0–255 Compare each observed count w/ expected count, N/256 Sum the normalized squared differences Large deviations suggest a non-uniform distribution (for example, plaintext or structured/compressed data)
31. Monte Carlo π estimation measures randomness by using byte pairs to approximate π π estimate = 4 × (points inside/total points) Monte Carlo (π estimation) Interpret groups of bytes as (x,y) coordinate pairs Count the points that fall inside the quarter-circle Compare the resulting estimate with π A large estimation error suggests a non-uniform distribution (for example, plaintext or structured/compressed data)
32. Test Results …before / after encryption (Gopher) File (plain/ compressed) Entropy χ² statistic π error test.html 5.12 385,760.14 27.32% test.png 7.91 7,383.25 3.04% test.zip 7.93 5,482.31 1.05% Plain/Compressed files File (encrypted) Entropy χ² statistic π error test.html.enc 7.99 236.03 0.23% test.png.enc 7.99 250.05 1.30% test.zip.enc 7.99 259.99 0.33% Encrypted files (Gopher)
33. Test Results …encrypted (KeRanger) File (encrypted) Entropy χ² statistic π error test.png.enc 7.99 235.03 0.15% test.zip.enc 7.99 256.71 0.79% test.html.enc 7.99 127.28 0.01% A different ransomware, produces the same useful signal ! (low Chi-square statistic, and low Monte-Carlo π error)
34. The final algorithm …to classify files as "are encrypted" entropy ≥ 7.95 AND ( π error ≤ 0.5% OR (π error ≤ 1.5% AND χ² ≤ 400) ) High entropy Very accurate π estimate and or Fairly accurate π estimate plus a near-uniform byte distribution
35. RansomWhere? (free/open-source) generic ransomware protection! objective-see.org/products/ransomwhere.html full source: github.com/objective-see/RansomWhere
36. RansomWhere? vs. …all the ransomware! Gopher KeRanger LockBit
37. Limitations No security tool is a panacea! ...and targeted attempts to bypass a specific tool (such as RansomWhere?), will likely succeed ! RansomWhere? limitations: Detections are (slightly) reactive Platform binaries are fully trusted Files must retain encryption (e.g. no base64) ...most of these could be addressed !
38. "HeartBreaker" 💔 Metamorphic Ransomware!?
39. Discovery …uploaded to VirusTotal (May 2025) Submitted: 2025-05-19 (Pakistan) "Heartbreaker" (still, 0 detections) Discovered by M. Carney Megan Carney: "Lessons from Threat Hunting for Malware Triage" (#OBTS v9)
40. A brief triage code signing, strings, & more % file heartbreaker heartbreaker: Mach-O 64-bit executable arm64 % codesign -dvv heartbreaker Identifier=heartbreaker-e2c97a640967e3f6 Format=Mach-O thin (arm64) CodeDirectory v=20400 size=58520 flags=0x20002(adhoc,linker-signed) hashes=1825+0 Signature=adhoc adhoc-signed, arm64 binary % strings heartbreaker Encrypting file: HEARTBREAKER WARNING! Your system has been encrypted! Please send 0.003 BTC ... [dependencies] aes-gcm = "0.10.3" mimicore = { git = "https://github.com/saaaarwar/mimicore" } Writing source to: modified_source.rs Writing Cargo.toml to: Writing build.rs to: Running cargo build Saving new executable Transforming source code METAMORPHIC_MARKER_START METAMORPHIC_MARKER_END } Ransom files? Dependencies? Metamorphism?
41. Definitely Ransomware! encrypts all files in ~/Downloads # FileMonitor.app/Contents/MacOS/FileMonitor -filter heartbreaker "event" : "ES_EVENT_TYPE_NOTIFY_OPEN", "file" : { "destination" : "/Users/user/Downloads/file.txt", … "event" : "ES_EVENT_TYPE_NOTIFY_WRITE", "file" : { “destination" : "/Users/user/Downloads/file.txt", … "event" : "ES_EVENT_TYPE_NOTIFY_WRITE", "file" : { "source" : "/Users/user/Downloads/file.txt", "destination" : “/Users/user/Downloads/file.txt.heartbreaker", Observing encryption Ransom message Enumerate all files in ~/Downloads Encrypt each file with AES-256-GCM Rename it to: <original-name>.heartbreaker
42. A closer look …at its cryptography 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 Hardcoded key!? 🤦 let key_str = String::from("g_w`:D2Xq6Yxez-5{#3mfd=VcB.Qhsub"); let key = aes_gcm::Key::<Aes256Gcm>::from_slice(...); let cipher = Aes256Gcm::new(key); fn encrypt_file(path: &PathBuf, cipher: &Aes256Gcm) -> io::Result<()> { let mut file = File::open(path)?; let mut contents = Vec::new(); file.read_to_end(&mut contents)?; let mut nonce = [0u8; 12]; OsRng.fill_bytes(&mut nonce); let ciphertext = cipher.encrypt(...)?; let mut file = OpenOptions::new().write(true).truncate(true).open(path)?; file.write_all(&nonce)?; file.write_all(&ciphertext)?; rename(path, new_name)?; } Generates a fresh 12-byte nonce Encrypt using AES-256-GCM w/ 32-byte key Writes: nonce || ciphertext || 16-byte GCM tag
43. Decryption? …simple, since the key is static & hardcoded 01 02 03 04 05 06 07 08 09 SUFFIX = ".heartbreaker" KEY = b"g_w`:D2Xq6Yxez-5{#3mfd=VcB.Qhsub" def decrypt_one(source: Path, apply: bool, overwrite: bool) -> str: target = output_path(source) blob = source.read_bytes() nonce, ciphertext_and_tag = blob[:12], blob[12:] plaintext = AESGCM(KEY).decrypt(nonce, ciphertext_and_tag, None) A heartbreaker decryptor % python3 decrypt_heartbreaker.py file.txt.heartbreaker DECRYPTING: file.txt.heartbreaker RECOVERED: file.txt.heartbreaker - > file.txt (19 bytes) % cat file.txt Aloha MacSysAdmins!
44. Metamorphism? be the change you want to see in the world! % strings heartbreaker ... METAMORPHIC_MARKER_START METAMORPHIC_MARKER_END Embedded strings Metamorphism: a technique where malware rewrites its own code each time it replicates (or runs), so every new copy looks different from the last while doing the same thing. Looks different, but logical "same same"
45. Metamorphism be the change you want to see in the world! 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 fn rewrite() -> std::io::Result<()> { strings containing its own embedded source ! let r_5mXvNym5J = include_str!("main.rs"); let r_hG2gBre = include_str!("../build.rs"); ... let r_KdRnzX = format!( r#" [package] … [dependencies] aes-gcm = "0.10.3" mimicore = {{ git = "https://github.com/saaaarwar/mimicore" }} … mimicore::morph( &r_5mXvNym5J, &r_KdRnzX, &r_hG2gBre, r_TzUgnNR02NxZOF, exe_name, )?; … metamorphic engine? } self-rebuilding metamorphic pipeline
46. The Metamorphic Engine? mimicore isn't available, but metacore is (same same?) metacore (github.com/saaaarwar/metacore) "A dynamic and extensible framework for generating and transforming Rust source code with built-in support for randomization and junk code insertion with helper functions"
47. The Metamorphic Engine and what actions does it perform? Locates a designated mutation region (between "METAMORPHIC_MARKER_START/END") Randomizes identifiers Mutates selected strings Changes function structure (control-flow obf.) Generates junk functions Shuffles function order …then, to (re)compile runs: cargo build --release
48. But our detection approach is not signature based... ...so (meta)morph all you want !?
49. HeartBreaker vs. RansomWhere? With no a priori knowledge of the threat, RansomWhere? was able to detect it actions & block it!
50. Conclusions
51. All ransomware behaves the same where it matters, ...giving us a generic detection heuristic ! Thwart ransomware by monitoring for: The rapid creation …of encrypted files …by an untrusted process ...heuristics ftw !
52. Inviting You To "Objective by the Sea" the only Apple-specific security conference #OBTS v9 November, Hawaii objectivebythesea.org
53. Mahalo to the "Friends of Objective-See"
54. Practical Ransomware Detection RESOURCES: blog.malwarebytes.org/threat-analysis/2013/07/fbi-ransomware-now-targetingapples-mac-os-x-users/ securelist.com/blog/research/66760/unfinished-ransomware-for-macos-x unit42.paloaltonetworks.com/new-os-x-ransomware-keranger-infectedtransmission-bittorrent-client-installer/