Save 37% off PRO during our Black Friday Sale! »

Killing bugs ... one vulnerability report at a time

Killing bugs ... one vulnerability report at a time

Sandro tells the story of how he and his security team came across vulnerabilities in FreeSWITCH, verified the security issues in lab environment, reported them upstream and finally worked with the FreeSWITCH developers to get them fixed. We will explore where security testing has value, that software can greatly benefit from this process of vulnerability reporting, fixing, etc., and also talk a bit about the process, and help you to better understand the impact of such vulnerabilities.

192a78d487076b95fb3562ad601a1535?s=128

Sandro Gauci

October 29, 2021
Tweet

Transcript

  1. Killing bugs … one vulnerability report at a time the

    story behind the latest FreeSWITCH advisories Sandro Gauci, 2021-10-28 Enable Security
  2. Introduction

  3. Who am I? Hi! Sandro Gauci here Author of SIPVicious

    OSS - security testing toolset for SIP In Infosec / Cyber security since > 20 years Last 13 years doing offensive security and leading Enable Security We focus on VoIP and WebRTC security i.e. pentesting, security audits, developing security tools, consultancy & training
  4. Team effort Everything we do is a team effort $i

    = $we - credit goes to the whole team Collaboration with Alfred Farrugia And Pinaki Mondal who was with us for a few months last year
  5. Introducing SIPVicious PRO We are developing a new security toolset,

    SIPVicious PRO The vulnerabilities that we found in FreeSWITCH had to do with it Completely new toolset, different than the open-source SIPVicious While SIPVicious OSS focuses on SIP, SIPVicious PRO covers RTC in general
  6. About this presentation Security advisories that we released this week

    Vulnerabilities in FreeSWITCH that were xed with the latest release FreeSWITCH 1.10.7 We cover how these vulnerabilities were discovered, reported and xed
  7. What advisories? FreeSWITCH vulnerable to SIP digest leak for con

    gured gateways FreeSWITCH susceptible to Denial of Service via SIP ooding FreeSWITCH does not authenticate SIP MESSAGE requests, leading to spam and message spoo ng FreeSWITCH does not authenticate SIP SUBSCRIBE requests by default FreeSWITCH susceptible to Denial of Service via invalid SRTP packets
  8. About these advisories Two of them are quite similar, the

    authentication issues for MESSAGE and SUBSCRIBE So I originally thought I would be telling four different stories
  9. Fancy story titles SIP digest leak but not as you

    (might) know it The mystery of the anonymous SIP messages Foom! Flooded out of memory Your encrypted call is important to us
  10. But, no! I realized something: everything is connected! Cannot talk

    about the process of nding one bug without mentioning another
  11. So I have one long story to tell …

  12. Starting position Ignorance: until recently we had no practical experience

    with FreeSWITCH Always on our radar but never actually con gured one None of our main clients back then seemed to use it in the products/services that we were testing
  13. Through years past In 2018, we wrote this bash script

  14. Result when run: 2018-03-25 20:56:55.095342 [NOTICE] switch_core_state_machine.c:690 Hangup sofia/internal/1000@172.17.0.2:5060 [CS_NEW][WRONG_CALL_STATE]

    2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 850 (sofia/internal/1000@172.17.0.2:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY] 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 847 (sofia/internal/1000@172.17.0.2:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY] 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 849 (sofia/internal/1000@172.17.0.2:5060) Ended 2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY] 2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1683 Session 848 (sofia/internal/1000@172.17.0.2:5060) Ended 2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY] ./__entrypoint.sh: line 15: 11 Killed /usr/local/freeswitch/bin/freeswitch
  15. DDoS simulator vs FreeSWITCH A year later we were testing

    our DDoS simulator. Tested against FreeSWITCH produced this: 2019-03-27 15:42:20.812017 [CONSOLE] switch_core.c:2455 [This app Best viewed at 160x60 or more..] 2019-03-27 15:42:20.814316 [INFO] switch_time.c:615 Clock synchronized to system time. freeswitch@ddos-simulator> Killed root@ddos-simulator:/usr/local/freeswitch/bin#
  16. Dangerous Demos at KW2019

  17. We demonstrated our DDoS simulator against a target Unknown target,

    happened to be FusionPBX Based on FreeSWITCH The target lasted a few seconds
  18. A few months later We were curious about FusionPBX, installed

    on a VPS Tested our tools .. then this happened
  19. None
  20. Something else this time too We ran our SIP method

    enumeration tool and saw the following
  21. ➜ sipvicious sip enum methods udp://target:5060 ... - target: udp://target:5060

    - status: security issue, insecure method/s detected results: - responses: REGISTER - supported (401 Unauthorized) SUBSCRIBE - supported (202 Accepted) NOTIFY - supported (481 Subscription Does Not Exist) PUBLISH - supported (200 OK) MESSAGE - supported (202 Accepted) INVITE - supported (407 Proxy Authentication Required) OPTIONS - supported (200 OK) BYE - supported (481 Call Does Not Exist) CANCEL - supported (481 Call/Transaction Does Not Exist) ACK - not supported (no response) PRACK - supported (481 No such response) INFO - supported (200 OK) REFER - supported (420 Bad Extension) UPDATE - supported (481 Call Does Not Exist) issues:
  22. What is SIP method enumeration? Tool sends different SIP message

    types to the target Tells you how it responds If any of the responses are 200 or 202, that means that no authentication is required Usually a problem for servers on the Internet
  23. Research time Retrospectively, these were obvious red ags But we

    were full of doubt and denial
  24. Our questions Were we doing anything wrong in our tests?

    What is the impact of these issues? Are these issues that appear in lab environment only? Is this speci c to FusionPBX? Who do we contact?
  25. Back in 2017 We had contacted someone associated with the

    FreeSWITCH community and eventually submitted a security report but nothing came out of it.
  26. Busy times So more time passed

  27. The bright future of 2020 Learning about con guration options

    related to authentication in FreeSWITCH By June we had a working setup and demo showing two things in default FreeSWITCH con guration: MESSAGE allowed spoo ng of messages and spam messages SUBSCRIBE could be used to monitor call events
  28. 0:00 / 0:46

  29. Related discoveries Around this time we noticed a potential client

    was also vulnerable Because they did use FreeSWITCH
  30. How widespread was this issue? Shodan is useful to nd

    IP addresses of FreeSWITCH systems across the world
  31. None
  32. Busy times So more time passed

  33. FreeSWITCH 1.10.6 released Late March 2021 We remembered that FreeSWITCH

    needed attention Had they xed any of the issues we had seen but not yet reported? Started peeking into the code and con guration changes
  34. A sleepless night Couldn’t sleep on the night of 7th

    / 8th April Did what any other hacker would do - review the latest commits Noticed that the ag auth-subscriptions was added to the default con guration Good but not good enough - upgrades would not be affected
  35. Boredom and curiosity Let’s test the digest leak tool (not

    a recommended insomnia remedy) Starts a call using SIP INVITE When the other party tries to hangup with a BYE message … Asks the other party to authenticate by sending a 407 SIP response with a challenge
  36. Nothing happened The following log line caught my eye: 2021-04-07

    22:43:29.502943 [WARNING] sofia_reg.c:2642 Cannot locate any authentication credentials to complete an authentication request for realm '"192.168.188.128"'
  37. FreeSWITCH did process our 407 .. But did not nd

    credentials that match What if it did nd matching credentials? We tracked the code and realized that you need to setup a gateway Uncommented the default gateway and ran it again
  38. sipvicious sip crack digest udp://192.168.188.128:5080 -e 1000 INFO[2021-04-08 01:03:04] started

    digest leak tool on udp://192.168.188.128:5080 INFO[2021-04-08 01:03:04] call picked up by sip:1000@192.168.188.128 INFO[2021-04-08 01:03:05] received BYE, challenging that with a 407 WARN[2021-04-08 01:03:05] digest leaked: response: 04b472d51cf56b05f8c40d30670b7eac, realm=192.168.188.128, nonce=aC20, uri=sip:57878777@192.168.188.69:60196;transport=udp, method=BYE, username=cluecon INFO[2021-04-08 01:03:05] BYE received, terminating call WARN[2021-04-08 01:03:08] security issue detected: digest leaked INFO[2021-04-08 01:03:08] test complete - target: udp://192.168.188.128:5080 - status: security issue, digest leaked results: N/A issues: - digestleak: response: 04b472d51cf56b05f8c40d30670b7eac, realm=192.168.188.128, nonce=aC20, uri=sip:57878777@192.168.188.69:60196;transport=udp, method=BYE, username=cluecon
  39. The next day Wrote the drafts for two advisories: Digest

    leak vulnerability Anonymous/unauthenticated MESSAGE vulnerability Curiosity left from the night before Tried our SIP ooding tool on FreeSWITCH’s TCP port
  40. Out of memory error - killing FreeSWITCH ubuntu-bionic kernel: [

    4205.446584] Out of memory: Kill process 22590 (freeswitch) score 939 or sacrifice child ubuntu-bionic kernel: [ 4205.449845] Killed process 22590 (freeswitch) total- vm:10484768kB, anon-rss:7894136kB, file-rss:0kB, shmem-rss:0kB ubuntu-bionic kernel: [ 4205.720680] oom_reaper: reaped process 22590 (freeswitch), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
  41. Making contact Suddenly we had 4 vulnerabilities for FreeSWITCH Worked

    on our reports especially reproduction steps with clear instructions But who should we contact?
  42. The OpenSIPIt connection OpenSIPIt had just happened and we had

    met some of the lead developers So we asked Maksym (organiser), which redirected us to David Duffett We ended up getting in touch with Chris Rienzo
  43. No security reporting process in place Chris told us that

    there was no security report handling in place Similar situation before when reporting a vulnerability to Coturn They had used the Github security advisory feature FreeSWITCH did the same
  44. First report in Sent our rst report Not for the

    rst two vulnerabilities that we discovered Digest leak - because It was the easiest for us to polish FreeSWITCH developers to reproduce
  45. Next reports Took a bit longer to send What took

    so long? Let’s see what a security report consists of
  46. The description

  47. Needs to accurately describe what we’re doing and the result

    Needs to answer: what’s the big deal?
  48. None
  49. We tried to give as much information to make it

    clear that memory was not being freed up in the case of the SIP ood DoS attack.
  50. None
  51. The impact section should answer two main questions: How bad

    could it be? - worst case scenario How easy is it to pull off?
  52. Not all “impact” is obvious DoS in RTC systems is

    critical The SUBSCRIBE issue took longest to gure out the full impact
  53. None
  54. The most important part of each advisory The steps need

    to be clear Scripts need to work properly When we fail here, developers will tell us: we cannot reproduce the issue the issue is not as bad
  55. None
  56. Solution / mitigation Guides developers towards what we think are

    good solutions There are many solutions that do not work so well Or introduce new problems
  57. ClueCon! We made sure that all 4 advisories were sent

    before con rming our presentation Two main conditions FreeSWITCH developers had to be OK with us talking about the vulnerabilities The security xes had to be made available before we talk about the details No pressure :-)
  58. What’s next? Some issues were easy to reproduce Others were

    easy to x Not necessarily both at the same time
  59. Di culty 1 The digest leak issue was easy to

    reproduce. Fixing needed various changes in the codebase.
  60. Di culty 2 We needed discussions and a call to

    make sure that the developers could see exactly what we were seeing We were noticing memory consumption that was never freed after the attack The developers initially did not see this behaviour We did have better tools (SIPVicious PRO) They had the simple scripts that we included in our report
  61. Solution: communication! To address these issues, we needed better communication.

  62. Testing the security xes Developers started making their xes available

    to us We tested to make sure that the xes were correct Sometimes patches do not address the vulnerabilities that we report
  63. SIP ood xes The rst patch did not actually x

    the issue We created a new video demo for the developers showing that we could still kill the server Second x was provided the next day which did indeed protect against the attack
  64. That fth report - what about it? We were developing

    the WebRTC component of SIPVicious PRO i.e. support for SIP over Websocket and DTLS-SRTP FreeSWITCH turned out to be the easiest system to test against for our needs While testing SIPVicious PRO’s SRTP code, we saw the following FreeSWITCH error messages
  65. 2021-08-11 07:59:06.923256 [INFO] sofia.c:10362 sofia/internal/99988801120715425438@207.154.254.38 receiving invite from 185.44.76.73:56030 version:

    1.10.6-release git 1ff9d0a 2021-03-25 13:16:09Z 64bit call-id: 1184444165-1963300639-337710667 2021-08-11 07:59:12.443278 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 10 errors 2021-08-11 07:59:12.643181 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 20 errors 2021-08-11 07:59:12.843235 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 30 errors 2021-08-11 07:59:13.043233 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 40 errors 2021-08-11 07:59:13.243215 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 50 errors 2021-08-11 07:59:13.443220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 60 errors 2021-08-11 07:59:13.643232 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 70 errors 2021-08-11 07:59:13.843220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 80 errors 2021-08-11 07:59:14.043188 [WARNING] switch rtp.c:6332 SRTP audio unprotect failed
  66. Our bug nds your bug Rookie mistake - sending incorrectly

    encrypted SRTP packets But what if we forced this error on other people’s calls? Only one way to nd out
  67. BYE sip:1001@192.168.0.112:59438;transport=wss SIP/2.0 Via: SIP/2.0/WSS 192.168.0.106:7443;rport;branch=z9hG4bKgejmX1Ka5F2ZB Max-Forwards: 70 From: <sip:1000@192.168.0.106>;tag=t9trvZ2D9cDtg

    To: <sip:1001@192.168.0.106>;tag=jNWhecoTN9IR4Zv4 Call-ID: mkBDUpsWEr9CdCqD CSeq: 39402862 BYE User-Agent: FreeSWITCH-mod_sofia/1.10.7-dev+git~20210727T192513Z~432bfc0c45~64bit Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY, PUBLISH, SUBSCRIBE Supported: timer, path, replaces Reason: SIP;cause=613;text=\"SRTP_READ_ERROR\" Content-Length: 0
  68. What does this mean? An attacker could simply spray the

    FreeSWITCH server with invalid SRTP packets This error is then generated which terminates the ongoing encrypted calls Luckily, this issue was rather easy to address - x is to simply ignore SRTP errors Sent out our last advisory
  69. Finally, ClueCon! This week a new FreeSWITCH release was published

    - 1.10.7 Contains the security xes for these vulnerabilities We published our advisories too
  70. Take-away

  71. What did we learn? We might not always immediately understand

    the vulnerabilities that we nd Sometimes we might not even immediately realize that we found something (e.g. the SRTP issue) Writing buggy software helps us nd security issues in other people’s software Without security testing tools, nding such issues is much harder
  72. More of that Security testing tools need to be exible

    and robust for security research and testing It is important that we contribute by sharing our ndings with the developers Clear reports are critical if we are to be taken seriously and actually want to help developers
  73. Lessons for Open-Source developers The problem of security vulnerabilities affects

    all software Handling security reports is one of the better solutions Github’s security advisory feature is quite nice Allows private forks so that security xes can be made available to the security researchers for testing
  74. FreeSWITCH users Understanding the what is behind the security xes

    helps understand if the issue affects your systems We encourage you to upgrade
  75. This is the end

  76. Thanks! FreeSWITCH developers for working with us And inviting us

    to talk about it! Alfred Farrugia and Pinaki Mondal for their work
  77. General references Our advisories: FreeSWITCH advisories: OpenSIPIt Our Dangerous Demo

    at Kamailio World 2019 https://github.com/EnableSecurity/advisories https://github.com/signalwire/freeswitch/security/advisories SIPVicious PRO About Github security advisories
  78. Get in touch & References Email: Enable Security: Communication Breakdown

    blog @ rtcsec.com sandro@enablesecurity.com https://www.enablesecurity.com Subscribe to the RTCSec Newsletter