Killing bugs ... one vulnerability report at a time

Sandro tells the story of how he and his security team came across vulnerabilities in FreeSWITCH, verified the security issues in lab environment, reported them upstream and finally worked with the FreeSWITCH developers to get them fixed. We will explore where security testing has value, that software can greatly benefit from this process of vulnerability reporting, fixing, etc., and also talk a bit about the process, and help you to better understand the impact of such vulnerabilities.

Transcript

1. Killing bugs … one vulnerability report at a time the story behind the latest FreeSWITCH advisories Sandro Gauci, 2021-10-28 Enable Security
2. Introduction
3. Who am I? Hi! Sandro Gauci here Author of SIPVicious OSS - security testing toolset for SIP In Infosec / Cyber security since > 20 years Last 13 years doing offensive security and leading Enable Security We focus on VoIP and WebRTC security i.e. pentesting, security audits, developing security tools, consultancy & training
4. Team effort Everything we do is a team effort $i = $we - credit goes to the whole team Collaboration with Alfred Farrugia And Pinaki Mondal who was with us for a few months last year
5. Introducing SIPVicious PRO We are developing a new security toolset, SIPVicious PRO The vulnerabilities that we found in FreeSWITCH had to do with it Completely new toolset, different than the open-source SIPVicious While SIPVicious OSS focuses on SIP, SIPVicious PRO covers RTC in general
6. About this presentation Security advisories that we released this week Vulnerabilities in FreeSWITCH that were xed with the latest release FreeSWITCH 1.10.7 We cover how these vulnerabilities were discovered, reported and xed
7. What advisories? FreeSWITCH vulnerable to SIP digest leak for con gured gateways FreeSWITCH susceptible to Denial of Service via SIP ooding FreeSWITCH does not authenticate SIP MESSAGE requests, leading to spam and message spoo ng FreeSWITCH does not authenticate SIP SUBSCRIBE requests by default FreeSWITCH susceptible to Denial of Service via invalid SRTP packets
8. About these advisories Two of them are quite similar, the authentication issues for MESSAGE and SUBSCRIBE So I originally thought I would be telling four different stories
9. Fancy story titles SIP digest leak but not as you (might) know it The mystery of the anonymous SIP messages Foom! Flooded out of memory Your encrypted call is important to us
10. But, no! I realized something: everything is connected! Cannot talk about the process of nding one bug without mentioning another
11. So I have one long story to tell …
12. Starting position Ignorance: until recently we had no practical experience with FreeSWITCH Always on our radar but never actually con gured one None of our main clients back then seemed to use it in the products/services that we were testing
13. Through years past In 2018, we wrote this bash script
14. Result when run: 2018-03-25 20:56:55.095342 [NOTICE] switch_core_state_machine.c:690 Hangup sofia/internal/1000@172.17.0.2:5060 [CS_NEW][WRONG_CALL_STATE] 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 850 (sofia/internal/1000@172.17.0.2:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY] 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 847 (sofia/internal/1000@172.17.0.2:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY] 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 849 (sofia/internal/1000@172.17.0.2:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY] 2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1683 Session 848 (sofia/internal/1000@172.17.0.2:5060) Ended 2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY] ./__entrypoint.sh: line 15: 11 Killed /usr/local/freeswitch/bin/freeswitch
15. DDoS simulator vs FreeSWITCH A year later we were testing our DDoS simulator. Tested against FreeSWITCH produced this: 2019-03-27 15:42:20.812017 [CONSOLE] switch_core.c:2455 [This app Best viewed at 160x60 or more..] 2019-03-27 15:42:20.814316 [INFO] switch_time.c:615 Clock synchronized to system time. freeswitch@ddos-simulator> Killed root@ddos-simulator:/usr/local/freeswitch/bin#
16. Dangerous Demos at KW2019
17. We demonstrated our DDoS simulator against a target Unknown target, happened to be FusionPBX Based on FreeSWITCH The target lasted a few seconds
18. A few months later We were curious about FusionPBX, installed on a VPS Tested our tools .. then this happened
20. Something else this time too We ran our SIP method enumeration tool and saw the following
21. ➜ sipvicious sip enum methods udp://target:5060 ... - target: udp://target:5060 - status: security issue, insecure method/s detected results: - responses: REGISTER - supported (401 Unauthorized) SUBSCRIBE - supported (202 Accepted) NOTIFY - supported (481 Subscription Does Not Exist) PUBLISH - supported (200 OK) MESSAGE - supported (202 Accepted) INVITE - supported (407 Proxy Authentication Required) OPTIONS - supported (200 OK) BYE - supported (481 Call Does Not Exist) CANCEL - supported (481 Call/Transaction Does Not Exist) ACK - not supported (no response) PRACK - supported (481 No such response) INFO - supported (200 OK) REFER - supported (420 Bad Extension) UPDATE - supported (481 Call Does Not Exist) issues:
22. What is SIP method enumeration? Tool sends different SIP message types to the target Tells you how it responds If any of the responses are 200 or 202, that means that no authentication is required Usually a problem for servers on the Internet
23. Research time Retrospectively, these were obvious red ags But we were full of doubt and denial
24. Our questions Were we doing anything wrong in our tests? What is the impact of these issues? Are these issues that appear in lab environment only? Is this speci c to FusionPBX? Who do we contact?
25. Back in 2017 We had contacted someone associated with the FreeSWITCH community and eventually submitted a security report but nothing came out of it.
26. Busy times So more time passed
27. The bright future of 2020 Learning about con guration options related to authentication in FreeSWITCH By June we had a working setup and demo showing two things in default FreeSWITCH con guration: MESSAGE allowed spoo ng of messages and spam messages SUBSCRIBE could be used to monitor call events
28. 0:00 / 0:46
29. Related discoveries Around this time we noticed a potential client was also vulnerable Because they did use FreeSWITCH
30. How widespread was this issue? Shodan is useful to nd IP addresses of FreeSWITCH systems across the world
32. Busy times So more time passed
33. FreeSWITCH 1.10.6 released Late March 2021 We remembered that FreeSWITCH needed attention Had they xed any of the issues we had seen but not yet reported? Started peeking into the code and con guration changes
34. A sleepless night Couldn’t sleep on the night of 7th / 8th April Did what any other hacker would do - review the latest commits Noticed that the ag auth-subscriptions was added to the default con guration Good but not good enough - upgrades would not be affected
35. Boredom and curiosity Let’s test the digest leak tool (not a recommended insomnia remedy) Starts a call using SIP INVITE When the other party tries to hangup with a BYE message … Asks the other party to authenticate by sending a 407 SIP response with a challenge
36. Nothing happened The following log line caught my eye: 2021-04-07 22:43:29.502943 [WARNING] sofia_reg.c:2642 Cannot locate any authentication credentials to complete an authentication request for realm '"192.168.188.128"'
37. FreeSWITCH did process our 407 .. But did not nd credentials that match What if it did nd matching credentials? We tracked the code and realized that you need to setup a gateway Uncommented the default gateway and ran it again
38. sipvicious sip crack digest udp://192.168.188.128:5080 -e 1000 INFO[2021-04-08 01:03:04] started digest leak tool on udp://192.168.188.128:5080 INFO[2021-04-08 01:03:04] call picked up by sip:1000@192.168.188.128 INFO[2021-04-08 01:03:05] received BYE, challenging that with a 407 WARN[2021-04-08 01:03:05] digest leaked: response: 04b472d51cf56b05f8c40d30670b7eac, realm=192.168.188.128, nonce=aC20, uri=sip:57878777@192.168.188.69:60196;transport=udp, method=BYE, username=cluecon INFO[2021-04-08 01:03:05] BYE received, terminating call WARN[2021-04-08 01:03:08] security issue detected: digest leaked INFO[2021-04-08 01:03:08] test complete - target: udp://192.168.188.128:5080 - status: security issue, digest leaked results: N/A issues: - digestleak: response: 04b472d51cf56b05f8c40d30670b7eac, realm=192.168.188.128, nonce=aC20, uri=sip:57878777@192.168.188.69:60196;transport=udp, method=BYE, username=cluecon
39. The next day Wrote the drafts for two advisories: Digest leak vulnerability Anonymous/unauthenticated MESSAGE vulnerability Curiosity left from the night before Tried our SIP ooding tool on FreeSWITCH’s TCP port
40. Out of memory error - killing FreeSWITCH ubuntu-bionic kernel: [ 4205.446584] Out of memory: Kill process 22590 (freeswitch) score 939 or sacrifice child ubuntu-bionic kernel: [ 4205.449845] Killed process 22590 (freeswitch) total- vm:10484768kB, anon-rss:7894136kB, file-rss:0kB, shmem-rss:0kB ubuntu-bionic kernel: [ 4205.720680] oom_reaper: reaped process 22590 (freeswitch), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
41. Making contact Suddenly we had 4 vulnerabilities for FreeSWITCH Worked on our reports especially reproduction steps with clear instructions But who should we contact?
42. The OpenSIPIt connection OpenSIPIt had just happened and we had met some of the lead developers So we asked Maksym (organiser), which redirected us to David Duffett We ended up getting in touch with Chris Rienzo
43. No security reporting process in place Chris told us that there was no security report handling in place Similar situation before when reporting a vulnerability to Coturn They had used the Github security advisory feature FreeSWITCH did the same
44. First report in Sent our rst report Not for the rst two vulnerabilities that we discovered Digest leak - because It was the easiest for us to polish FreeSWITCH developers to reproduce
45. Next reports Took a bit longer to send What took so long? Let’s see what a security report consists of
46. The description
47. Needs to accurately describe what we’re doing and the result Needs to answer: what’s the big deal?
49. We tried to give as much information to make it clear that memory was not being freed up in the case of the SIP ood DoS attack.
51. The impact section should answer two main questions: How bad could it be? - worst case scenario How easy is it to pull off?
52. Not all “impact” is obvious DoS in RTC systems is critical The SUBSCRIBE issue took longest to gure out the full impact
54. The most important part of each advisory The steps need to be clear Scripts need to work properly When we fail here, developers will tell us: we cannot reproduce the issue the issue is not as bad
56. Solution / mitigation Guides developers towards what we think are good solutions There are many solutions that do not work so well Or introduce new problems
57. ClueCon! We made sure that all 4 advisories were sent before con rming our presentation Two main conditions FreeSWITCH developers had to be OK with us talking about the vulnerabilities The security xes had to be made available before we talk about the details No pressure :-)
58. What’s next? Some issues were easy to reproduce Others were easy to x Not necessarily both at the same time
59. Di culty 1 The digest leak issue was easy to reproduce. Fixing needed various changes in the codebase.
60. Di culty 2 We needed discussions and a call to make sure that the developers could see exactly what we were seeing We were noticing memory consumption that was never freed after the attack The developers initially did not see this behaviour We did have better tools (SIPVicious PRO) They had the simple scripts that we included in our report
61. Solution: communication! To address these issues, we needed better communication.
62. Testing the security xes Developers started making their xes available to us We tested to make sure that the xes were correct Sometimes patches do not address the vulnerabilities that we report
63. SIP ood xes The rst patch did not actually x the issue We created a new video demo for the developers showing that we could still kill the server Second x was provided the next day which did indeed protect against the attack
64. That fth report - what about it? We were developing the WebRTC component of SIPVicious PRO i.e. support for SIP over Websocket and DTLS-SRTP FreeSWITCH turned out to be the easiest system to test against for our needs While testing SIPVicious PRO’s SRTP code, we saw the following FreeSWITCH error messages
65. 2021-08-11 07:59:06.923256 [INFO] sofia.c:10362 sofia/internal/99988801120715425438@207.154.254.38 receiving invite from 185.44.76.73:56030 version: 1.10.6-release git 1ff9d0a 2021-03-25 13:16:09Z 64bit call-id: 1184444165-1963300639-337710667 2021-08-11 07:59:12.443278 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 10 errors 2021-08-11 07:59:12.643181 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 20 errors 2021-08-11 07:59:12.843235 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 30 errors 2021-08-11 07:59:13.043233 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 40 errors 2021-08-11 07:59:13.243215 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 50 errors 2021-08-11 07:59:13.443220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 60 errors 2021-08-11 07:59:13.643232 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 70 errors 2021-08-11 07:59:13.843220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 80 errors 2021-08-11 07:59:14.043188 [WARNING] switch rtp.c:6332 SRTP audio unprotect failed
66. Our bug nds your bug Rookie mistake - sending incorrectly encrypted SRTP packets But what if we forced this error on other people’s calls? Only one way to nd out
67. BYE sip:1001@192.168.0.112:59438;transport=wss SIP/2.0 Via: SIP/2.0/WSS 192.168.0.106:7443;rport;branch=z9hG4bKgejmX1Ka5F2ZB Max-Forwards: 70 From: <sip:1000@192.168.0.106>;tag=t9trvZ2D9cDtg To: <sip:1001@192.168.0.106>;tag=jNWhecoTN9IR4Zv4 Call-ID: mkBDUpsWEr9CdCqD CSeq: 39402862 BYE User-Agent: FreeSWITCH-mod_sofia/1.10.7-dev+git~20210727T192513Z~432bfc0c45~64bit Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY, PUBLISH, SUBSCRIBE Supported: timer, path, replaces Reason: SIP;cause=613;text=\"SRTP_READ_ERROR\" Content-Length: 0
68. What does this mean? An attacker could simply spray the FreeSWITCH server with invalid SRTP packets This error is then generated which terminates the ongoing encrypted calls Luckily, this issue was rather easy to address - x is to simply ignore SRTP errors Sent out our last advisory
69. Finally, ClueCon! This week a new FreeSWITCH release was published - 1.10.7 Contains the security xes for these vulnerabilities We published our advisories too
70. Take-away
71. What did we learn? We might not always immediately understand the vulnerabilities that we nd Sometimes we might not even immediately realize that we found something (e.g. the SRTP issue) Writing buggy software helps us nd security issues in other people’s software Without security testing tools, nding such issues is much harder
72. More of that Security testing tools need to be exible and robust for security research and testing It is important that we contribute by sharing our ndings with the developers Clear reports are critical if we are to be taken seriously and actually want to help developers
73. Lessons for Open-Source developers The problem of security vulnerabilities affects all software Handling security reports is one of the better solutions Github’s security advisory feature is quite nice Allows private forks so that security xes can be made available to the security researchers for testing
74. FreeSWITCH users Understanding the what is behind the security xes helps understand if the issue affects your systems We encourage you to upgrade
75. This is the end
76. Thanks! FreeSWITCH developers for working with us And inviting us to talk about it! Alfred Farrugia and Pinaki Mondal for their work
77. General references Our advisories: FreeSWITCH advisories: OpenSIPIt Our Dangerous Demo at Kamailio World 2019 https://github.com/EnableSecurity/advisories https://github.com/signalwire/freeswitch/security/advisories SIPVicious PRO About Github security advisories
78. Get in touch & References Email: Enable Security: Communication Breakdown blog @ rtcsec.com sandro@enablesecurity.com https://www.enablesecurity.com Subscribe to the RTCSec Newsletter