Tales from the Ops Side - Denial of Sleep

In the first few months of 2016, our network was subjected to several seemingly unrelated DDoS attacks launched against various clients. Continuing our Tales from the Ops side series, these stream of events robbed us of our sleep, time, and money, and included extortion letters, bitcoin, and even encounters with the FBI.

Transcript

2. Denial of Sleep Tales from the Ops Side By Hany Fahim Founder and CEO @iHandroid @vmfarms
3. 100% True* * Names have been changed. Like all our tales, the following tale is
4. Saturday, October 17, 2015 When it all began
5. Alert - 10:26 pm ET Saturday night. Received multiple pages for downed hosts. Usually symptomatic of a network outage of sorts. Got on the phone with our upstream (Tier 1). We were under a DDoS attack.
6. Attack! Size 10 GBits Target Alpha Client Duration 7 min Signature Distributed UDP-Based Random src/dest ports
7. Russian Federation - 28.3% Romania - 26.4% Ukraine - 17.0% Bulgaria - 5.7% Andorra - 3.8% Sweden - 3.8% October 17, 2015
8. Network Tiers Tier 1
9. Tier 1 was Overwhelmed Tier 1
10. Network Connections 5 things are required to establish a network connection: Source IP Source Port Destination IP Destination Port Protocol (TCP, UDP, etc…)
11. TCP vs. UDP TCP is stateful: TCP requires a handshake to be negotiated. All packets must be acknowledged. Easy to track connection lifecycle. High overhead.
12. TCP vs. UDP UDP is stateless: UDP does not require a handshake. Packets don’t have to be acknowledged. Difficult to track lifecycle. Low overhead. udp User Datagram Protocol
13. Common Attacks Because of its stateless nature, UDP-based attacks very common and very damaging. TCP attacks are still plentiful (SYN Floods).
14. Shields Up! Asked Tier 1 to block all UDP traffic destined for Alpha Client. Successfully mitigated the attack in 7 min. Kept block in place for 24 hrs after attack subsided.
15. What can you do to prevent this from happening again?
16. CloudFlare is a CDN and security company. DNS-based Proxy Service. Acts as a go-between yourself and end-users. Worked with Alpha Client to implement CloudFlare
17. CloudFlare acts as a proxy Client CloudFlare IP Original IP Original IP is hidden.
18. CloudFlare sees many attacks Client Client Client
19. Tuesday, December 8, 2015
20. Alert - 3:47pm ET Middle of the work day. Multiple pages for downed hosts. Called Tier 1 immediately. We were under attack again.
21. Attack! Size 12 GBits Target Beta Client Duration 25 min Signature Distributed UDP-Based All port 0
22. Russian Federation - 45.8% Ukraine - 6.0% Brazil - 5.4% Czechia - 5.4% Romania - 5.4% Poland - 4.8% December 8, 2015
23. Shields Up! Port 0 is an invalid port. Due to the signature, it was easy to setup a filter. Asked Tier 1 to block all UDP/0 traffic cluster-wide. Successfully mitigated the attack in 25 min. Kept block in place for 24 hrs after attack subsided.
24. + Attacks can happen anytime and to anyone. Using a partnership, bring discounts to clients. Includes a free tier. Offered to help in the migration.
25. Herd Immunity In order to be effective, most clients need to implement. Attack surface area is reduced with each adoption. Not always easy to implement (latency). Can’t force adoption. Marketing, phone calls, education were the only tools.
26. Wednesday, January 20, 2016
27. 9:59am ET - 2 Attacks! Size 30 + 20 GBits Target Gamma Client
 + Upstream’s Router Duration 76 min Signature Distributed UDP-Based Mostly port 0 + some 53 (DNS)
28. Russian Federation - 51.2% Ukraine - 14.4% Czechia - 6.9% Poland - 6.2% Romania - 5.0% Moldova, Republic of - 2.8% January 20, 2016
29. Shields Up! Asked Tier 1 to block all UDP/0 cluster-wide. Took a lot longer to mitigate (76 min) - more affected systems. Worked with Gamma Client to implement CloudFlare.
30. udp User Datagram Protocol Block all UDP Most web-based applications use TCP (HTTP). Essential services like DNS (53) and NTP (123) use UDP. Made the call to block all UDP, with the exception of NTP and some DNS. Permanent block.
31. Tuesday, January 26, 2016 6 days later
32. 5:14pm ET Attack! Size 8 GBits Target Alpha Client Duration 4 min Signature Distributed UDP-Based Mostly port 0 + some random
33. Brazil - 19.7% Ukraine - 8.2% Argentina - 4.9% Thailand - 4.9% Iran, Islamic Republic of - 3.3% China - 3.3% January 26, 2016
34. Russian Federation - 51.2% Ukraine - 14.4% Czechia - 6.9% Poland - 6.2% Romania - 5.0% Moldova, Republic of - 2.8% January 20, 2016 (previous)
35. Shields Up? Why did we go down? UDP should be blocked. 8 Gbits is smallest so far.
36. Tier 2 Went Down Upstream’s upstream (Tier 2) went down. Doesn’t make sense. Should be able to withstand 8 Gbits
 (survived 30 Gbits).
37. Tier 1 Tier 2 More Tiers Upstream’s upstream went down.
38. Border Gateway Protocol (BGP) A name is who you want (DNS). An address is where it is (IP Address). A route is how to get there (BGP).
39. BGP is like GPS for packets Like Google Maps or Waze. Routes are dynamic. Key difference: Routes are based on cost, not efficiency.
40. BGP
41. BGP Paths are based on least cost. SRC
42. Brazil - 19.7% Argentina - 4.9%
 Chile - 3.3% Ecuador - 1.6% Colombia - 1.6% January 26, 2016 South America
43. South American Routes Due to cost (agreements), traffic from South American countries came inbound from a different link (preferred route). Link was only 10 Gbits (attack was 8 Gbits). Still vulnerable!
44. South American Routes 10 Gbits
45. More Questions Alpha Client implemented CloudFlare back in October. Attack targeted proxied IP. Bypassed CloudFlare.
46. Bypassing CloudFlare Client CloudFlare IP Original IP Attackers targeted “secret” Original IP.
47. Not-so-secret Secret? Origin IP leaked. Any number of ways to discover proxied IP: Stale/leaked DNS records. Many services keep track of historical DNS entries. Same attacker (same signature), old IP may be recorded.
48. CloudPiercer.org A service to discover exposure to IP leaks.
49. Rotate IP Rotate out proxied/origin IP in secret. Original IP is tainted. Permanently null-route IP.
50. Quite Frustrated 4 attacks in 3 months, 2 within a week. We were more vulnerable than we thought. CloudFlare works, but requires herd immunity. Blocking is even not enough. Sources matter. Need to “nuke it from orbit.”
51. Traffic Scrubbing Looked at purchasing traffic scrubbers. Very large. Very expensive. Very complicated.
52. DDoS Scrubbing Services Companies specialize in this field. Own scrubbers and big pipes, and have trained staff. Downside: Still expensive. Not in our DC.
53. DDoS Scrubbers SRC
54. DDoS Scrubbers SRC SRC SRC
55. Cat Herding 4 parties involved: Us, Tier 1, Tier 2, and Scrubbing Service. Take months to implement. Best Hope. "Only way to be sure.”
56. Vacation February and March off.
57. Monday, April 18, 2016
58. Alpha Client receives Ransom Note
59. We are Armada Collective. Most importantly, we have launched largest DDoS in Swiss history and one of the largest DDoS attacks ever. Search for "ProtonMail DDoS" All your servers will be DDoS-ed starting Monday (April 25) if you don't pay protection fee - exactly 11.41 Bitcoins (CAD $6,193) @ 17RBypDd7p62Jum8uN51rKUJfMWez98yeh If you don't pay by Monday, attack will start, yours service going down permanently price to stop will increase to 20 BTC and will go up 10 BTC for every day of attack. This is not a joke. Our attacks are extremely powerful - peak over 1 Tbps. Do not reply, we will not read. Pay and we will know its you. AND YOU WILL NEVER AGAIN HEAR FROM US!
60. What do you do? Gave us 7 days. Do you take this seriously? Do you pay them? Scrubbing Service was nearly online, but still not ready. Contacted all parties and began seeking fast solutions. Clock was ticking.
61. Friday, April 22, 2016 T-Minus 3 days
62. 9:31am ET Attack! Size 15 GBits Target Delta Client Duration 25 min Signature Distributed UDP-Based Started port 0 + changed to NTP
63. China - 32.2% Singapore - 7.8% Brazil - 6.7% Colombia - 6.7% Iran, Islamic Republic of - 6.7% India - 6.7% April 22, 2016
64. Gamma Client receives Ransom Note
65. We are a team of highly skilled individual security professionals who are trying to make the internet a safer place. We stumbled on your site and started digging. Our data base expert was able to dump your entire customer database in a matter of minutes. The data has not been released yet. We will tell you the vulnerabilities you have to patch and release your site once you pay our consultation fee of $300 USD. You can send BTC to the following address : 1KRQ6LVBFDGn26cdn6FF5hBckNPsPPJLax After payment is received we will not only stop all attacks but will tell you how to stop them in the future and add you to a blacklist so other groups leave you alone. Contacting your hosting provider will not stop the attack. The authorities cant stop the attack. IT companies will waste a ton of your money and still not be able to mitigate. We have the resources of some nation states. Lets not waste time, you have money to make!
66. Shields Up! Attacker adapted within 2 minutes. Targeted the only UDP protocol still open: NTP. Permanently null-route IP to protect network. Worked with Gamma Client to implement CloudFlare and rotated IP.
67. Sunday, April 24, 2016 T-Minus 1 day
68. Route Change DDoS Scrubber “advertises” for Alpha Client’s IP.
69. Monday, April 25, 2016 Zero Hour
70. Nothing Happens.
71. No attack transpired. Huge relief. Was it a bluff? Did they detect the route change? Was it a copy-cat?
72. CloudFlare Blog Post Posted on the same day.
73. Tuesday, April 26, 2016 The very next day.
74. Scrubbing For All Decided to implement scrubber service cluster-wide. Hard financial decision. DDoS attacks were extremely harmful. Saw no other choice.
75. Almost a year later…
76. Thursday, February 16, 2017
77. Received voicemail from “Agent Michael”, claiming to be FBI. Investigating DDoS attacks from mid-2016. Would like to chat.
78. Is this for real? Called FBI Headquarters, asked to be routed to Agent Michael. Agent Michael was legit.
79. Investigation June 2016 period. Series of attacks targeting San Diego-based software company. Have a suspect. He’s American!
80. Investigation Attacks were distributed and UDP-based. Mostly port 0, but can vary. Our IPs showed up on some seized systems.
81. Sounds Familiar? Same signature! Agent Michael was interested in everything we had. With permission, sent everything over, including ransom notes.
82. Same Attacker! BTC addresses matched! Agent Michael was happy. Allowed him to expand time range for investigation.
83. Monday, March 13, 2017 Agent Michael forwards FBI Press Release
84. –FBI Press Release - Friday March 3, 2017 “Florida Man Arrested for Forcing a
 San Diego Company’s Website Off-Line”
85. Gerald “Jerry” M. McTear III Age 29 From Ft. Myers, Florida. Arrested on charges of fraud and various computer crimes. Facing maximum 17 yrs in prison, USD $750k in fines.
86. 4 clients, 5 attacks, 6 months Over 60 Countries
87. The End?
88. Questions? Psst… We’re Hiring! By Hany Fahim Founder and CEO @iHandroid @vmfarms