Killing bugs ... one vulnerability report at a time

Sandro tells the story of how he and his security team came across vulnerabilities in FreeSWITCH, verified the security issues in lab environment, reported them upstream and finally worked with the FreeSWITCH developers to get them fixed. We will explore where security testing has value, that software can greatly benefit from this process of vulnerability reporting, fixing, etc., and also talk a bit about the process, and help you to better understand the impact of such vulnerabilities.

Transcript

1. Killing bugs …
   one vulnerability report
   at a time
   the story behind the latest FreeSWITCH advisories
   Sandro Gauci,
   2021-10-28
   Enable Security
   

   View Slide

2. Introduction
   

   View Slide

3. Who am I?
   Hi! Sandro Gauci here
   Author of SIPVicious OSS - security testing toolset for SIP
   In Infosec / Cyber security since > 20 years
   Last 13 years doing offensive security and leading Enable Security
   We focus on VoIP and WebRTC security i.e. pentesting, security
   audits, developing security tools, consultancy & training
   

   View Slide

4. Team effort
   Everything we do is a team effort
   $i = $we - credit goes to the whole team
   Collaboration with Alfred Farrugia
   And Pinaki Mondal who was with us for a few months last year
   

   View Slide

5. Introducing SIPVicious PRO
   We are developing a new security toolset, SIPVicious PRO
   The vulnerabilities that we found in FreeSWITCH had to do with it
   Completely new toolset, different than the open-source SIPVicious
   While SIPVicious OSS focuses on SIP, SIPVicious PRO covers RTC
   in general
   

   View Slide

6. About this presentation
   Security advisories that we released this week
   Vulnerabilities in FreeSWITCH that were xed with the latest
   release
   FreeSWITCH 1.10.7
   We cover how these vulnerabilities were discovered, reported and
   xed
   

   View Slide

7. What advisories?
   FreeSWITCH vulnerable to SIP digest leak for con gured gateways
   FreeSWITCH susceptible to Denial of Service via SIP ooding
   FreeSWITCH does not authenticate SIP MESSAGE requests,
   leading to spam and message spoo ng
   FreeSWITCH does not authenticate SIP SUBSCRIBE requests by
   default
   FreeSWITCH susceptible to Denial of Service via invalid SRTP
   packets
   

   View Slide

8. About these advisories
   Two of them are quite similar, the authentication issues for
   MESSAGE and SUBSCRIBE
   So I originally thought I would be telling four different stories
   

   View Slide

9. Fancy story titles
   SIP digest leak but not as you (might) know it
   The mystery of the anonymous SIP messages
   Foom! Flooded out of memory
   Your encrypted call is important to us
   

   View Slide

10. But, no!
    I realized something: everything is connected!
    Cannot talk about the process of nding one bug without
    mentioning another
    

    View Slide

11. So I have one long
    story to tell …
    

    View Slide

12. Starting position
    Ignorance: until recently we had no practical experience with
    FreeSWITCH
    Always on our radar but never actually con gured one
    None of our main clients back then seemed to use it in the
    products/services that we were testing
    

    View Slide

13. Through years past
    In 2018, we wrote this bash script
    

    View Slide

14. Result when run:
    2018-03-25 20:56:55.095342 [NOTICE] switch_core_state_machine.c:690 Hangup
    sofia/internal/1000@172.17.0.2:5060 [CS_NEW][WRONG_CALL_STATE]
    2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 850
    (sofia/internal/1000@172.17.0.2:5060) Ended
    2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel
    sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY]
    2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 847
    (sofia/internal/1000@172.17.0.2:5060) Ended
    2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel
    sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY]
    2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 849
    (sofia/internal/1000@172.17.0.2:5060) Ended
    2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel
    sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY]
    2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1683 Session 848
    (sofia/internal/1000@172.17.0.2:5060) Ended
    2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1687 Close Channel
    sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY]
    ./__entrypoint.sh: line 15: 11 Killed
    /usr/local/freeswitch/bin/freeswitch
    

    View Slide

15. DDoS simulator vs
    FreeSWITCH
    A year later we were testing our DDoS simulator. Tested against
    FreeSWITCH produced this:
    2019-03-27 15:42:20.812017 [CONSOLE] switch_core.c:2455
    [This app Best viewed at 160x60 or more..]
    2019-03-27 15:42:20.814316 [INFO] switch_time.c:615 Clock synchronized to system time.
    freeswitch@ddos-simulator> Killed
    root@ddos-simulator:/usr/local/freeswitch/bin#
    

    View Slide

16. Dangerous Demos at
    KW2019
    

    View Slide

17. We demonstrated our DDoS simulator against a target
    Unknown target, happened to be FusionPBX
    Based on FreeSWITCH
    The target lasted a few seconds
    

    View Slide

18. A few months later
    We were curious about FusionPBX, installed on a VPS
    Tested our tools .. then this happened
    

    View Slide

19. View Slide

20. Something else this time
    too
    We ran our SIP method enumeration tool and saw the following
    

    View Slide

21. ➜ sipvicious sip enum methods udp://target:5060
    ...
    - target: udp://target:5060
    - status: security issue, insecure method/s detected
    results:
    - responses:
    REGISTER - supported (401 Unauthorized)
    SUBSCRIBE - supported (202 Accepted)
    NOTIFY - supported (481 Subscription Does Not Exist)
    PUBLISH - supported (200 OK)
    MESSAGE - supported (202 Accepted)
    INVITE - supported (407 Proxy Authentication Required)
    OPTIONS - supported (200 OK)
    BYE - supported (481 Call Does Not Exist)
    CANCEL - supported (481 Call/Transaction Does Not Exist)
    ACK - not supported (no response)
    PRACK - supported (481 No such response)
    INFO - supported (200 OK)
    REFER - supported (420 Bad Extension)
    UPDATE - supported (481 Call Does Not Exist)
    issues:
    

    View Slide

22. What is SIP method
    enumeration?
    Tool sends different SIP message types to the target
    Tells you how it responds
    If any of the responses are 200 or 202, that means that no
    authentication is required
    Usually a problem for servers on the Internet
    

    View Slide

23. Research time
    Retrospectively, these were obvious red ags
    But we were full of doubt and denial
    

    View Slide

24. Our questions
    Were we doing anything wrong in our tests?
    What is the impact of these issues?
    Are these issues that appear in lab environment only?
    Is this speci c to FusionPBX?
    Who do we contact?
    

    View Slide

25. Back in 2017
    We had contacted someone associated with the FreeSWITCH
    community and eventually submitted a security report but nothing came
    out of it.
    

    View Slide

26. Busy times
    So more time passed
    

    View Slide

27. The bright future of 2020
    Learning about con guration options related to authentication in
    FreeSWITCH
    By June we had a working setup and demo showing two things in
    default FreeSWITCH con guration:
    MESSAGE allowed spoo ng of messages and spam
    messages
    SUBSCRIBE could be used to monitor call events
    

    View Slide

28. 0:00 / 0:46
    

    View Slide

29. Related discoveries
    Around this time we noticed a potential client was also vulnerable
    Because they did use FreeSWITCH
    

    View Slide

30. How widespread was this
    issue?
    Shodan is useful to nd IP addresses of FreeSWITCH systems across
    the world
    

    View Slide

31. View Slide

32. Busy times
    So more time passed
    

    View Slide

33. FreeSWITCH 1.10.6
    released
    Late March 2021
    We remembered that FreeSWITCH needed attention
    Had they xed any of the issues we had seen but not yet reported?
    Started peeking into the code and con guration changes
    

    View Slide

34. A sleepless night
    Couldn’t sleep on the night of 7th / 8th April
    Did what any other hacker would do - review the latest commits
    Noticed that the ag auth-subscriptions was added to the
    default con guration
    Good but not good enough - upgrades would not be affected
    

    View Slide

35. Boredom and curiosity
    Let’s test the digest leak tool (not a recommended insomnia
    remedy)
    Starts a call using SIP INVITE
    When the other party tries to hangup with a BYE message …
    Asks the other party to authenticate by sending a 407 SIP
    response with a challenge
    

    View Slide

36. Nothing happened
    The following log line caught my eye:
    2021-04-07 22:43:29.502943 [WARNING] sofia_reg.c:2642
    Cannot locate any authentication credentials to
    complete an authentication request for realm
    '"192.168.188.128"'
    

    View Slide

37. FreeSWITCH did process
    our 407
    .. But did not nd credentials that match
    What if it did nd matching credentials?
    We tracked the code and realized that you need to setup a
    gateway
    Uncommented the default gateway and ran it again
    

    View Slide

38. sipvicious sip crack digest udp://192.168.188.128:5080 -e 1000
    INFO[2021-04-08 01:03:04] started digest leak tool on udp://192.168.188.128:5080
    INFO[2021-04-08 01:03:04] call picked up by sip:1000@192.168.188.128
    INFO[2021-04-08 01:03:05] received BYE, challenging that with a 407
    WARN[2021-04-08 01:03:05] digest leaked: response: 04b472d51cf56b05f8c40d30670b7eac,
    realm=192.168.188.128, nonce=aC20,
    uri=sip:57878777@192.168.188.69:60196;transport=udp, method=BYE, username=cluecon
    INFO[2021-04-08 01:03:05] BYE received, terminating call
    WARN[2021-04-08 01:03:08] security issue detected: digest leaked
    INFO[2021-04-08 01:03:08] test complete
    - target: udp://192.168.188.128:5080
    - status: security issue, digest leaked
    results:
    N/A
    issues:
    - digestleak:
    response: 04b472d51cf56b05f8c40d30670b7eac, realm=192.168.188.128, nonce=aC20,
    uri=sip:57878777@192.168.188.69:60196;transport=udp, method=BYE, username=cluecon
    

    View Slide

39. The next day
    Wrote the drafts for two advisories:
    Digest leak vulnerability
    Anonymous/unauthenticated MESSAGE vulnerability
    Curiosity left from the night before
    Tried our SIP ooding tool on FreeSWITCH’s TCP port
    

    View Slide

40. Out of memory error - killing
    FreeSWITCH
    ubuntu-bionic kernel: [ 4205.446584] Out of memory: Kill process 22590 (freeswitch)
    score 939 or sacrifice child
    ubuntu-bionic kernel: [ 4205.449845] Killed process 22590 (freeswitch) total-
    vm:10484768kB, anon-rss:7894136kB, file-rss:0kB, shmem-rss:0kB
    ubuntu-bionic kernel: [ 4205.720680] oom_reaper: reaped process 22590 (freeswitch),
    now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
    

    View Slide

41. Making contact
    Suddenly we had 4 vulnerabilities for FreeSWITCH
    Worked on our reports especially reproduction steps with clear
    instructions
    But who should we contact?
    

    View Slide

42. The OpenSIPIt connection
    OpenSIPIt had just happened and we had met some of the lead
    developers
    So we asked Maksym (organiser), which redirected us to David
    Duffett
    We ended up getting in touch with Chris Rienzo
    

    View Slide

43. No security reporting
    process in place
    Chris told us that there was no security report handling in place
    Similar situation before when reporting a vulnerability to Coturn
    They had used the Github security advisory feature
    FreeSWITCH did the same
    

    View Slide

44. First report in
    Sent our rst report
    Not for the rst two vulnerabilities that we discovered
    Digest leak - because
    It was the easiest for us to polish
    FreeSWITCH developers to reproduce
    

    View Slide

45. Next reports
    Took a bit longer to send
    What took so long?
    Let’s see what a security report consists of
    

    View Slide

46. The description
    

    View Slide

47. Needs to accurately describe what we’re doing and the result
    Needs to answer: what’s the big deal?
    

    View Slide

48. View Slide

49. We tried to give as much information to make it clear that memory was
    not being freed up in the case of the SIP ood DoS attack.
    

    View Slide

50. View Slide

51. The impact section should answer two main questions:
    How bad could it be? - worst case scenario
    How easy is it to pull off?
    

    View Slide

52. Not all “impact” is obvious
    DoS in RTC systems is critical
    The SUBSCRIBE issue took longest to gure out the full impact
    

    View Slide

53. View Slide

54. The most important part of
    each advisory
    The steps need to be clear
    Scripts need to work properly
    When we fail here, developers will tell us:
    we cannot reproduce the issue
    the issue is not as bad
    

    View Slide

55. View Slide

56. Solution / mitigation
    Guides developers towards what we think are good solutions
    There are many solutions that do not work so well
    Or introduce new problems
    

    View Slide

57. ClueCon!
    We made sure that all 4 advisories were sent before con rming
    our presentation
    Two main conditions
    FreeSWITCH developers had to be OK with us talking about
    the vulnerabilities
    The security xes had to be made available before we talk
    about the details
    No pressure :-)
    

    View Slide

58. What’s next?
    Some issues were easy to reproduce
    Others were easy to x
    Not necessarily both at the same time
    

    View Slide

59. Di culty 1
    The digest leak issue was easy to reproduce. Fixing needed various
    changes in the codebase.
    

    View Slide

60. Di culty 2
    We needed discussions and a call to make sure that the
    developers could see exactly what we were seeing
    We were noticing memory consumption that was never freed after
    the attack
    The developers initially did not see this behaviour
    We did have better tools (SIPVicious PRO)
    They had the simple scripts that we included in our report
    

    View Slide

61. Solution: communication!
    To address these issues, we needed better communication.
    

    View Slide

62. Testing the security xes
    Developers started making their xes available to us
    We tested to make sure that the xes were correct
    Sometimes patches do not address the vulnerabilities that we
    report
    

    View Slide

63. SIP ood xes
    The rst patch did not actually x the issue
    We created a new video demo for the developers showing that we
    could still kill the server
    Second x was provided the next day which did indeed protect
    against the attack
    

    View Slide

64. That fth report - what
    about it?
    We were developing the WebRTC component of SIPVicious PRO
    i.e. support for SIP over Websocket and DTLS-SRTP
    FreeSWITCH turned out to be the easiest system to test against
    for our needs
    While testing SIPVicious PRO’s SRTP code, we saw the following
    FreeSWITCH error messages
    

    View Slide

65. 2021-08-11 07:59:06.923256 [INFO] sofia.c:10362
    sofia/internal/99988801120715425438@207.154.254.38 receiving invite from
    185.44.76.73:56030 version: 1.10.6-release git 1ff9d0a 2021-03-25 13:16:09Z 64bit
    call-id: 1184444165-1963300639-337710667
    2021-08-11 07:59:12.443278 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed
    with code 21 () 13 bytes 10 errors
    2021-08-11 07:59:12.643181 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed
    with code 21 () 13 bytes 20 errors
    2021-08-11 07:59:12.843235 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed
    with code 21 () 13 bytes 30 errors
    2021-08-11 07:59:13.043233 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed
    with code 21 () 13 bytes 40 errors
    2021-08-11 07:59:13.243215 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed
    with code 21 () 13 bytes 50 errors
    2021-08-11 07:59:13.443220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed
    with code 21 () 13 bytes 60 errors
    2021-08-11 07:59:13.643232 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed
    with code 21 () 13 bytes 70 errors
    2021-08-11 07:59:13.843220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed
    with code 21 () 13 bytes 80 errors
    2021-08-11 07:59:14.043188 [WARNING] switch rtp.c:6332 SRTP audio unprotect failed
    

    View Slide

66. Our bug nds your bug
    Rookie mistake - sending incorrectly encrypted SRTP packets
    But what if we forced this error on other people’s calls?
    Only one way to nd out
    

    View Slide

67. BYE sip:1001@192.168.0.112:59438;transport=wss SIP/2.0
    Via: SIP/2.0/WSS 192.168.0.106:7443;rport;branch=z9hG4bKgejmX1Ka5F2ZB
    Max-Forwards: 70
    From: ;tag=t9trvZ2D9cDtg
    To: ;tag=jNWhecoTN9IR4Zv4
    Call-ID: mkBDUpsWEr9CdCqD
    CSeq: 39402862 BYE
    User-Agent: FreeSWITCH-mod_sofia/1.10.7-dev+git~20210727T192513Z~432bfc0c45~64bit
    Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER,
    NOTIFY, PUBLISH, SUBSCRIBE
    Supported: timer, path, replaces
    Reason: SIP;cause=613;text=\"SRTP_READ_ERROR\"
    Content-Length: 0
    

    View Slide

68. What does this mean?
    An attacker could simply spray the FreeSWITCH server with invalid
    SRTP packets
    This error is then generated which terminates the ongoing
    encrypted calls
    Luckily, this issue was rather easy to address - x is to simply
    ignore SRTP errors
    Sent out our last advisory
    

    View Slide

69. Finally, ClueCon!
    This week a new FreeSWITCH release was published - 1.10.7
    Contains the security xes for these vulnerabilities
    We published our advisories too
    

    View Slide

70. Take-away
    

    View Slide

71. What did we learn?
    We might not always immediately understand the vulnerabilities
    that we nd
    Sometimes we might not even immediately realize that we found
    something (e.g. the SRTP issue)
    Writing buggy software helps us nd security issues in other
    people’s software
    Without security testing tools, nding such issues is much harder
    

    View Slide

72. More of that
    Security testing tools need to be exible and robust for security
    research and testing
    It is important that we contribute by sharing our ndings with the
    developers
    Clear reports are critical if we are to be taken seriously and
    actually want to help developers
    

    View Slide

73. Lessons for Open-Source
    developers
    The problem of security vulnerabilities affects all software
    Handling security reports is one of the better solutions
    Github’s security advisory feature is quite nice
    Allows private forks so that security xes can be made available to
    the security researchers for testing
    

    View Slide

74. FreeSWITCH users
    Understanding the what is behind the security xes helps
    understand if the issue affects your systems
    We encourage you to upgrade
    

    View Slide

75. This is the end
    

    View Slide

76. Thanks!
    FreeSWITCH developers for working with us
    And inviting us to talk about it!
    Alfred Farrugia and Pinaki Mondal for their work
    

    View Slide

77. General references
    Our advisories:
    FreeSWITCH advisories:
    OpenSIPIt
    Our Dangerous Demo at Kamailio World 2019
    https://github.com/EnableSecurity/advisories
    https://github.com/signalwire/freeswitch/security/advisories
    SIPVicious PRO
    About Github security advisories
    

    View Slide

78. Get in touch & References
    Email:
    Enable Security:
    Communication Breakdown blog @ rtcsec.com
    sandro@enablesecurity.com
    https://www.enablesecurity.com
    Subscribe to the RTCSec Newsletter
    

    View Slide