Tales from the Ops Side - Denial of Sleep

Tales from the Ops Side - Denial of Sleep

In the first few months of 2016, our network was subjected to several seemingly unrelated DDoS attacks launched against various clients. Continuing our Tales from the Ops side series, these stream of events robbed us of our sleep, time, and money, and included extortion letters, bitcoin, and even encounters with the FBI.


Hany Fahim

May 29, 2017


  1. None
  2. Denial of Sleep Tales from the Ops Side By Hany

    Fahim Founder and CEO @iHandroid @vmfarms
  3. 100% True* * Names have been changed. Like all our

    tales, the following tale is
  4. Saturday, October 17, 2015 When it all began

  5. Alert - 10:26 pm ET Saturday night. Received multiple pages

    for downed hosts. Usually symptomatic of a network outage of sorts. Got on the phone with our upstream (Tier 1). We were under a DDoS attack.
  6. Attack! Size 10 GBits Target Alpha Client Duration 7 min

    Signature Distributed UDP-Based Random src/dest ports
  7. Russian Federation - 28.3% Romania - 26.4% Ukraine - 17.0%

    Bulgaria - 5.7% Andorra - 3.8% Sweden - 3.8% October 17, 2015
  8. Network Tiers Tier 1

  9. Tier 1 was Overwhelmed Tier 1

  10. Network Connections 5 things are required to establish a network

    connection: Source IP Source Port Destination IP Destination Port Protocol (TCP, UDP, etc…)
  11. TCP vs. UDP TCP is stateful: TCP requires a handshake

    to be negotiated. All packets must be acknowledged. Easy to track connection lifecycle. High overhead.
  12. TCP vs. UDP UDP is stateless: UDP does not require

    a handshake. Packets don’t have to be acknowledged. Difficult to track lifecycle. Low overhead. udp User Datagram Protocol
  13. Common Attacks Because of its stateless nature, UDP-based attacks very

    common and very damaging. TCP attacks are still plentiful (SYN Floods).
  14. Shields Up! Asked Tier 1 to block all UDP traffic

    destined for Alpha Client. Successfully mitigated the attack in 7 min. Kept block in place for 24 hrs after attack subsided.
  15. What can you do to prevent this from happening again?

  16. CloudFlare is a CDN and security company. DNS-based Proxy Service.

    Acts as a go-between yourself and end-users. Worked with Alpha Client to implement CloudFlare
  17. CloudFlare acts as a proxy Client CloudFlare IP Original IP

    Original IP is hidden.
  18. CloudFlare sees many attacks Client Client Client

  19. Tuesday, December 8, 2015

  20. Alert - 3:47pm ET Middle of the work day. Multiple

    pages for downed hosts. Called Tier 1 immediately. We were under attack again.
  21. Attack! Size 12 GBits Target Beta Client Duration 25 min

    Signature Distributed UDP-Based All port 0
  22. Russian Federation - 45.8% Ukraine - 6.0% Brazil - 5.4%

    Czechia - 5.4% Romania - 5.4% Poland - 4.8% December 8, 2015
  23. Shields Up! Port 0 is an invalid port. Due to

    the signature, it was easy to setup a filter. Asked Tier 1 to block all UDP/0 traffic cluster-wide. Successfully mitigated the attack in 25 min. Kept block in place for 24 hrs after attack subsided.
  24. + Attacks can happen anytime and to anyone. Using a

    partnership, bring discounts to clients. Includes a free tier. Offered to help in the migration.
  25. Herd Immunity In order to be effective, most clients need

    to implement. Attack surface area is reduced with each adoption. Not always easy to implement (latency). Can’t force adoption. Marketing, phone calls, education were the only tools.
  26. Wednesday, January 20, 2016

  27. 9:59am ET - 2 Attacks! Size 30 + 20 GBits

    Target Gamma Client
 + Upstream’s Router Duration 76 min Signature Distributed UDP-Based Mostly port 0 + some 53 (DNS)
  28. Russian Federation - 51.2% Ukraine - 14.4% Czechia - 6.9%

    Poland - 6.2% Romania - 5.0% Moldova, Republic of - 2.8% January 20, 2016
  29. Shields Up! Asked Tier 1 to block all UDP/0 cluster-wide.

    Took a lot longer to mitigate (76 min) - more affected systems. Worked with Gamma Client to implement CloudFlare.
  30. udp User Datagram Protocol Block all UDP Most web-based applications

    use TCP (HTTP). Essential services like DNS (53) and NTP (123) use UDP. Made the call to block all UDP, with the exception of NTP and some DNS. Permanent block.
  31. Tuesday, January 26, 2016 6 days later

  32. 5:14pm ET Attack! Size 8 GBits Target Alpha Client Duration

    4 min Signature Distributed UDP-Based Mostly port 0 + some random
  33. Brazil - 19.7% Ukraine - 8.2% Argentina - 4.9% Thailand

    - 4.9% Iran, Islamic Republic of - 3.3% China - 3.3% January 26, 2016
  34. Russian Federation - 51.2% Ukraine - 14.4% Czechia - 6.9%

    Poland - 6.2% Romania - 5.0% Moldova, Republic of - 2.8% January 20, 2016 (previous)
  35. Shields Up? Why did we go down? UDP should be

    blocked. 8 Gbits is smallest so far.
  36. Tier 2 Went Down Upstream’s upstream (Tier 2) went down.

    Doesn’t make sense. Should be able to withstand 8 Gbits
 (survived 30 Gbits).
  37. Tier 1 Tier 2 More Tiers Upstream’s upstream went down.

  38. Border Gateway Protocol (BGP) A name is who you want

    (DNS). An address is where it is (IP Address). A route is how to get there (BGP).
  39. BGP is like GPS for packets Like Google Maps or

    Waze. Routes are dynamic. Key difference: Routes are based on cost, not efficiency.
  40. BGP

  41. BGP Paths are based on least cost. SRC

  42. Brazil - 19.7% Argentina - 4.9%
 Chile - 3.3% Ecuador

    - 1.6% Colombia - 1.6% January 26, 2016 South America
  43. South American Routes Due to cost (agreements), traffic from South

    American countries came inbound from a different link (preferred route). Link was only 10 Gbits (attack was 8 Gbits). Still vulnerable!
  44. South American Routes 10 Gbits

  45. More Questions Alpha Client implemented CloudFlare back in October. Attack

    targeted proxied IP. Bypassed CloudFlare.
  46. Bypassing CloudFlare Client CloudFlare IP Original IP Attackers targeted “secret”

    Original IP.
  47. Not-so-secret Secret? Origin IP leaked. Any number of ways to

    discover proxied IP: Stale/leaked DNS records. Many services keep track of historical DNS entries. Same attacker (same signature), old IP may be recorded.
  48. CloudPiercer.org A service to discover exposure to IP leaks.

  49. Rotate IP Rotate out proxied/origin IP in secret. Original IP

    is tainted. Permanently null-route IP.
  50. Quite Frustrated 4 attacks in 3 months, 2 within a

    week. We were more vulnerable than we thought. CloudFlare works, but requires herd immunity. Blocking is even not enough. Sources matter. Need to “nuke it from orbit.”
  51. Traffic Scrubbing Looked at purchasing traffic scrubbers. Very large. Very

    expensive. Very complicated.
  52. DDoS Scrubbing Services Companies specialize in this field. Own scrubbers

    and big pipes, and have trained staff. Downside: Still expensive. Not in our DC.
  53. DDoS Scrubbers SRC

  54. DDoS Scrubbers SRC SRC SRC

  55. Cat Herding 4 parties involved: Us, Tier 1, Tier 2,

    and Scrubbing Service. Take months to implement. Best Hope. "Only way to be sure.”
  56. Vacation February and March off.

  57. Monday, April 18, 2016

  58. Alpha Client receives Ransom Note

  59. We are Armada Collective. Most importantly, we have launched largest

    DDoS in Swiss history and one of the largest DDoS attacks ever. Search for "ProtonMail DDoS" All your servers will be DDoS-ed starting Monday (April 25) if you don't pay protection fee - exactly 11.41 Bitcoins (CAD $6,193) @ 17RBypDd7p62Jum8uN51rKUJfMWez98yeh If you don't pay by Monday, attack will start, yours service going down permanently price to stop will increase to 20 BTC and will go up 10 BTC for every day of attack. This is not a joke. Our attacks are extremely powerful - peak over 1 Tbps. Do not reply, we will not read. Pay and we will know its you. AND YOU WILL NEVER AGAIN HEAR FROM US!
  60. What do you do? Gave us 7 days. Do you

    take this seriously? Do you pay them? Scrubbing Service was nearly online, but still not ready. Contacted all parties and began seeking fast solutions. Clock was ticking.
  61. Friday, April 22, 2016 T-Minus 3 days

  62. 9:31am ET Attack! Size 15 GBits Target Delta Client Duration

    25 min Signature Distributed UDP-Based Started port 0 + changed to NTP
  63. China - 32.2% Singapore - 7.8% Brazil - 6.7% Colombia

    - 6.7% Iran, Islamic Republic of - 6.7% India - 6.7% April 22, 2016
  64. Gamma Client receives Ransom Note

  65. We are a team of highly skilled individual security professionals

    who are trying to make the internet a safer place. We stumbled on your site and started digging. Our data base expert was able to dump your entire customer database in a matter of minutes. The data has not been released yet. We will tell you the vulnerabilities you have to patch and release your site once you pay our consultation fee of $300 USD. You can send BTC to the following address : 1KRQ6LVBFDGn26cdn6FF5hBckNPsPPJLax After payment is received we will not only stop all attacks but will tell you how to stop them in the future and add you to a blacklist so other groups leave you alone. Contacting your hosting provider will not stop the attack. The authorities cant stop the attack. IT companies will waste a ton of your money and still not be able to mitigate. We have the resources of some nation states. Lets not waste time, you have money to make!
  66. Shields Up! Attacker adapted within 2 minutes. Targeted the only

    UDP protocol still open: NTP. Permanently null-route IP to protect network. Worked with Gamma Client to implement CloudFlare and rotated IP.
  67. Sunday, April 24, 2016 T-Minus 1 day

  68. Route Change DDoS Scrubber “advertises” for Alpha Client’s IP.

  69. Monday, April 25, 2016 Zero Hour

  70. Nothing Happens.

  71. No attack transpired. Huge relief. Was it a bluff? Did

    they detect the route change? Was it a copy-cat?
  72. CloudFlare Blog Post Posted on the same day.

  73. Tuesday, April 26, 2016 The very next day.

  74. Scrubbing For All Decided to implement scrubber service cluster-wide. Hard

    financial decision. DDoS attacks were extremely harmful. Saw no other choice.
  75. Almost a year later…

  76. Thursday, February 16, 2017

  77. Received voicemail from “Agent Michael”, claiming to be FBI. Investigating

    DDoS attacks from mid-2016. Would like to chat.
  78. Is this for real? Called FBI Headquarters, asked to be

    routed to Agent Michael. Agent Michael was legit.
  79. Investigation June 2016 period. Series of attacks targeting San Diego-based

    software company. Have a suspect. He’s American!
  80. Investigation Attacks were distributed and UDP-based. Mostly port 0, but

    can vary. Our IPs showed up on some seized systems.
  81. Sounds Familiar? Same signature! Agent Michael was interested in everything

    we had. With permission, sent everything over, including ransom notes.
  82. Same Attacker! BTC addresses matched! Agent Michael was happy. Allowed

    him to expand time range for investigation.
  83. Monday, March 13, 2017 Agent Michael forwards FBI Press Release

  84. –FBI Press Release - Friday March 3, 2017 “Florida Man

    Arrested for Forcing a
 San Diego Company’s Website Off-Line”
  85. Gerald “Jerry” M. McTear III Age 29 From Ft. Myers,

    Florida. Arrested on charges of fraud and various computer crimes. Facing maximum 17 yrs in prison, USD $750k in fines.
  86. 4 clients, 5 attacks, 6 months Over 60 Countries

  87. The End?

  88. Questions? Psst… We’re Hiring! By Hany Fahim Founder and CEO

    @iHandroid @vmfarms