Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Killing bugs ... one vulnerability report at a time

Killing bugs ... one vulnerability report at a time

Sandro tells the story of how he and his security team came across vulnerabilities in FreeSWITCH, verified the security issues in lab environment, reported them upstream and finally worked with the FreeSWITCH developers to get them fixed. We will explore where security testing has value, that software can greatly benefit from this process of vulnerability reporting, fixing, etc., and also talk a bit about the process, and help you to better understand the impact of such vulnerabilities.

Sandro Gauci

October 29, 2021

More Decks by Sandro Gauci

Other Decks in Technology


  1. Killing bugs … one vulnerability report at a time the

    story behind the latest FreeSWITCH advisories Sandro Gauci, 2021-10-28 Enable Security
  2. Who am I? Hi! Sandro Gauci here Author of SIPVicious

    OSS - security testing toolset for SIP In Infosec / Cyber security since > 20 years Last 13 years doing offensive security and leading Enable Security We focus on VoIP and WebRTC security i.e. pentesting, security audits, developing security tools, consultancy & training
  3. Team effort Everything we do is a team effort $i

    = $we - credit goes to the whole team Collaboration with Alfred Farrugia And Pinaki Mondal who was with us for a few months last year
  4. Introducing SIPVicious PRO We are developing a new security toolset,

    SIPVicious PRO The vulnerabilities that we found in FreeSWITCH had to do with it Completely new toolset, different than the open-source SIPVicious While SIPVicious OSS focuses on SIP, SIPVicious PRO covers RTC in general
  5. About this presentation Security advisories that we released this week

    Vulnerabilities in FreeSWITCH that were xed with the latest release FreeSWITCH 1.10.7 We cover how these vulnerabilities were discovered, reported and xed
  6. What advisories? FreeSWITCH vulnerable to SIP digest leak for con

    gured gateways FreeSWITCH susceptible to Denial of Service via SIP ooding FreeSWITCH does not authenticate SIP MESSAGE requests, leading to spam and message spoo ng FreeSWITCH does not authenticate SIP SUBSCRIBE requests by default FreeSWITCH susceptible to Denial of Service via invalid SRTP packets
  7. About these advisories Two of them are quite similar, the

    authentication issues for MESSAGE and SUBSCRIBE So I originally thought I would be telling four different stories
  8. Fancy story titles SIP digest leak but not as you

    (might) know it The mystery of the anonymous SIP messages Foom! Flooded out of memory Your encrypted call is important to us
  9. But, no! I realized something: everything is connected! Cannot talk

    about the process of nding one bug without mentioning another
  10. Starting position Ignorance: until recently we had no practical experience

    with FreeSWITCH Always on our radar but never actually con gured one None of our main clients back then seemed to use it in the products/services that we were testing
  11. Result when run: 2018-03-25 20:56:55.095342 [NOTICE] switch_core_state_machine.c:690 Hangup sofia/internal/[email protected]:5060 [CS_NEW][WRONG_CALL_STATE]

    2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 850 (sofia/internal/[email protected]:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/[email protected]:5060 [CS_DESTROY] 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 847 (sofia/internal/[email protected]:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/[email protected]:5060 [CS_DESTROY] 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 849 (sofia/internal/[email protected]:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/[email protected]:5060 [CS_DESTROY] 2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1683 Session 848 (sofia/internal/[email protected]:5060) Ended 2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/[email protected]:5060 [CS_DESTROY] ./__entrypoint.sh: line 15: 11 Killed /usr/local/freeswitch/bin/freeswitch
  12. DDoS simulator vs FreeSWITCH A year later we were testing

    our DDoS simulator. Tested against FreeSWITCH produced this: 2019-03-27 15:42:20.812017 [CONSOLE] switch_core.c:2455 [This app Best viewed at 160x60 or more..] 2019-03-27 15:42:20.814316 [INFO] switch_time.c:615 Clock synchronized to system time. freeswitch@ddos-simulator> Killed root@ddos-simulator:/usr/local/freeswitch/bin#
  13. We demonstrated our DDoS simulator against a target Unknown target,

    happened to be FusionPBX Based on FreeSWITCH The target lasted a few seconds
  14. A few months later We were curious about FusionPBX, installed

    on a VPS Tested our tools .. then this happened
  15. Something else this time too We ran our SIP method

    enumeration tool and saw the following
  16. ➜ sipvicious sip enum methods udp://target:5060 ... - target: udp://target:5060

    - status: security issue, insecure method/s detected results: - responses: REGISTER - supported (401 Unauthorized) SUBSCRIBE - supported (202 Accepted) NOTIFY - supported (481 Subscription Does Not Exist) PUBLISH - supported (200 OK) MESSAGE - supported (202 Accepted) INVITE - supported (407 Proxy Authentication Required) OPTIONS - supported (200 OK) BYE - supported (481 Call Does Not Exist) CANCEL - supported (481 Call/Transaction Does Not Exist) ACK - not supported (no response) PRACK - supported (481 No such response) INFO - supported (200 OK) REFER - supported (420 Bad Extension) UPDATE - supported (481 Call Does Not Exist) issues:
  17. What is SIP method enumeration? Tool sends different SIP message

    types to the target Tells you how it responds If any of the responses are 200 or 202, that means that no authentication is required Usually a problem for servers on the Internet
  18. Our questions Were we doing anything wrong in our tests?

    What is the impact of these issues? Are these issues that appear in lab environment only? Is this speci c to FusionPBX? Who do we contact?
  19. Back in 2017 We had contacted someone associated with the

    FreeSWITCH community and eventually submitted a security report but nothing came out of it.
  20. The bright future of 2020 Learning about con guration options

    related to authentication in FreeSWITCH By June we had a working setup and demo showing two things in default FreeSWITCH con guration: MESSAGE allowed spoo ng of messages and spam messages SUBSCRIBE could be used to monitor call events
  21. Related discoveries Around this time we noticed a potential client

    was also vulnerable Because they did use FreeSWITCH
  22. How widespread was this issue? Shodan is useful to nd

    IP addresses of FreeSWITCH systems across the world
  23. FreeSWITCH 1.10.6 released Late March 2021 We remembered that FreeSWITCH

    needed attention Had they xed any of the issues we had seen but not yet reported? Started peeking into the code and con guration changes
  24. A sleepless night Couldn’t sleep on the night of 7th

    / 8th April Did what any other hacker would do - review the latest commits Noticed that the ag auth-subscriptions was added to the default con guration Good but not good enough - upgrades would not be affected
  25. Boredom and curiosity Let’s test the digest leak tool (not

    a recommended insomnia remedy) Starts a call using SIP INVITE When the other party tries to hangup with a BYE message … Asks the other party to authenticate by sending a 407 SIP response with a challenge
  26. Nothing happened The following log line caught my eye: 2021-04-07

    22:43:29.502943 [WARNING] sofia_reg.c:2642 Cannot locate any authentication credentials to complete an authentication request for realm '""'
  27. FreeSWITCH did process our 407 .. But did not nd

    credentials that match What if it did nd matching credentials? We tracked the code and realized that you need to setup a gateway Uncommented the default gateway and ran it again
  28. sipvicious sip crack digest udp:// -e 1000 INFO[2021-04-08 01:03:04] started

    digest leak tool on udp:// INFO[2021-04-08 01:03:04] call picked up by sip:[email protected] INFO[2021-04-08 01:03:05] received BYE, challenging that with a 407 WARN[2021-04-08 01:03:05] digest leaked: response: 04b472d51cf56b05f8c40d30670b7eac, realm=, nonce=aC20, uri=sip:[email protected]:60196;transport=udp, method=BYE, username=cluecon INFO[2021-04-08 01:03:05] BYE received, terminating call WARN[2021-04-08 01:03:08] security issue detected: digest leaked INFO[2021-04-08 01:03:08] test complete - target: udp:// - status: security issue, digest leaked results: N/A issues: - digestleak: response: 04b472d51cf56b05f8c40d30670b7eac, realm=, nonce=aC20, uri=sip:[email protected]:60196;transport=udp, method=BYE, username=cluecon
  29. The next day Wrote the drafts for two advisories: Digest

    leak vulnerability Anonymous/unauthenticated MESSAGE vulnerability Curiosity left from the night before Tried our SIP ooding tool on FreeSWITCH’s TCP port
  30. Out of memory error - killing FreeSWITCH ubuntu-bionic kernel: [

    4205.446584] Out of memory: Kill process 22590 (freeswitch) score 939 or sacrifice child ubuntu-bionic kernel: [ 4205.449845] Killed process 22590 (freeswitch) total- vm:10484768kB, anon-rss:7894136kB, file-rss:0kB, shmem-rss:0kB ubuntu-bionic kernel: [ 4205.720680] oom_reaper: reaped process 22590 (freeswitch), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  31. Making contact Suddenly we had 4 vulnerabilities for FreeSWITCH Worked

    on our reports especially reproduction steps with clear instructions But who should we contact?
  32. The OpenSIPIt connection OpenSIPIt had just happened and we had

    met some of the lead developers So we asked Maksym (organiser), which redirected us to David Duffett We ended up getting in touch with Chris Rienzo
  33. No security reporting process in place Chris told us that

    there was no security report handling in place Similar situation before when reporting a vulnerability to Coturn They had used the Github security advisory feature FreeSWITCH did the same
  34. First report in Sent our rst report Not for the

    rst two vulnerabilities that we discovered Digest leak - because It was the easiest for us to polish FreeSWITCH developers to reproduce
  35. Next reports Took a bit longer to send What took

    so long? Let’s see what a security report consists of
  36. Needs to accurately describe what we’re doing and the result

    Needs to answer: what’s the big deal?
  37. We tried to give as much information to make it

    clear that memory was not being freed up in the case of the SIP ood DoS attack.
  38. The impact section should answer two main questions: How bad

    could it be? - worst case scenario How easy is it to pull off?
  39. Not all “impact” is obvious DoS in RTC systems is

    critical The SUBSCRIBE issue took longest to gure out the full impact
  40. The most important part of each advisory The steps need

    to be clear Scripts need to work properly When we fail here, developers will tell us: we cannot reproduce the issue the issue is not as bad
  41. Solution / mitigation Guides developers towards what we think are

    good solutions There are many solutions that do not work so well Or introduce new problems
  42. ClueCon! We made sure that all 4 advisories were sent

    before con rming our presentation Two main conditions FreeSWITCH developers had to be OK with us talking about the vulnerabilities The security xes had to be made available before we talk about the details No pressure :-)
  43. What’s next? Some issues were easy to reproduce Others were

    easy to x Not necessarily both at the same time
  44. Di culty 1 The digest leak issue was easy to

    reproduce. Fixing needed various changes in the codebase.
  45. Di culty 2 We needed discussions and a call to

    make sure that the developers could see exactly what we were seeing We were noticing memory consumption that was never freed after the attack The developers initially did not see this behaviour We did have better tools (SIPVicious PRO) They had the simple scripts that we included in our report
  46. Testing the security xes Developers started making their xes available

    to us We tested to make sure that the xes were correct Sometimes patches do not address the vulnerabilities that we report
  47. SIP ood xes The rst patch did not actually x

    the issue We created a new video demo for the developers showing that we could still kill the server Second x was provided the next day which did indeed protect against the attack
  48. That fth report - what about it? We were developing

    the WebRTC component of SIPVicious PRO i.e. support for SIP over Websocket and DTLS-SRTP FreeSWITCH turned out to be the easiest system to test against for our needs While testing SIPVicious PRO’s SRTP code, we saw the following FreeSWITCH error messages
  49. 2021-08-11 07:59:06.923256 [INFO] sofia.c:10362 sofia/internal/[email protected] receiving invite from version:

    1.10.6-release git 1ff9d0a 2021-03-25 13:16:09Z 64bit call-id: 1184444165-1963300639-337710667 2021-08-11 07:59:12.443278 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 10 errors 2021-08-11 07:59:12.643181 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 20 errors 2021-08-11 07:59:12.843235 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 30 errors 2021-08-11 07:59:13.043233 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 40 errors 2021-08-11 07:59:13.243215 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 50 errors 2021-08-11 07:59:13.443220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 60 errors 2021-08-11 07:59:13.643232 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 70 errors 2021-08-11 07:59:13.843220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 80 errors 2021-08-11 07:59:14.043188 [WARNING] switch rtp.c:6332 SRTP audio unprotect failed
  50. Our bug nds your bug Rookie mistake - sending incorrectly

    encrypted SRTP packets But what if we forced this error on other people’s calls? Only one way to nd out
  51. BYE sip:[email protected]:59438;transport=wss SIP/2.0 Via: SIP/2.0/WSS;rport;branch=z9hG4bKgejmX1Ka5F2ZB Max-Forwards: 70 From: <sip:[email protected]>;tag=t9trvZ2D9cDtg

    To: <sip:[email protected]>;tag=jNWhecoTN9IR4Zv4 Call-ID: mkBDUpsWEr9CdCqD CSeq: 39402862 BYE User-Agent: FreeSWITCH-mod_sofia/1.10.7-dev+git~20210727T192513Z~432bfc0c45~64bit Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY, PUBLISH, SUBSCRIBE Supported: timer, path, replaces Reason: SIP;cause=613;text=\"SRTP_READ_ERROR\" Content-Length: 0
  52. What does this mean? An attacker could simply spray the

    FreeSWITCH server with invalid SRTP packets This error is then generated which terminates the ongoing encrypted calls Luckily, this issue was rather easy to address - x is to simply ignore SRTP errors Sent out our last advisory
  53. Finally, ClueCon! This week a new FreeSWITCH release was published

    - 1.10.7 Contains the security xes for these vulnerabilities We published our advisories too
  54. What did we learn? We might not always immediately understand

    the vulnerabilities that we nd Sometimes we might not even immediately realize that we found something (e.g. the SRTP issue) Writing buggy software helps us nd security issues in other people’s software Without security testing tools, nding such issues is much harder
  55. More of that Security testing tools need to be exible

    and robust for security research and testing It is important that we contribute by sharing our ndings with the developers Clear reports are critical if we are to be taken seriously and actually want to help developers
  56. Lessons for Open-Source developers The problem of security vulnerabilities affects

    all software Handling security reports is one of the better solutions Github’s security advisory feature is quite nice Allows private forks so that security xes can be made available to the security researchers for testing
  57. FreeSWITCH users Understanding the what is behind the security xes

    helps understand if the issue affects your systems We encourage you to upgrade
  58. Thanks! FreeSWITCH developers for working with us And inviting us

    to talk about it! Alfred Farrugia and Pinaki Mondal for their work
  59. General references Our advisories: FreeSWITCH advisories: OpenSIPIt Our Dangerous Demo

    at Kamailio World 2019 https://github.com/EnableSecurity/advisories https://github.com/signalwire/freeswitch/security/advisories SIPVicious PRO About Github security advisories
  60. Get in touch & References Email: Enable Security: Communication Breakdown

    blog @ rtcsec.com [email protected] https://www.enablesecurity.com Subscribe to the RTCSec Newsletter