Handling Large Amounts of Traffic on the Edge

Slide 1

Slide 1 text

Handling Large Amounts of Traffic on the Edge Helen Tabunshchyk Cloudflare

Slide 2

Slide 2 text

A bit of context about the work I do

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

CDN • Moving content physically closer to visitors with our CDN • Intelligent caching • Unlimited DDOS mitigation • Unlimited bandwidth at flat pricing with free plans What does Cloudflare do? Website Optimisation • Making web fast and up to date for everyone. • TLS 1.3 (with 0-RTT) • HTTP/2 + QUIC • Server push • AMP • Origin load-balancing • Smart routing • Workers • Post quantum crypto • Many more DNS • Cloudflare is the fastest managed DNS providers in the world. • 1.1.1.1 • 2606:4700:4700::1111 • DNS over TLS

Slide 5

Slide 5 text

What is Cloudflare? • We serve more web traffic than Twitter, Amazon, Apple, Instagram, Bing, & Wikipedia combined. • Anytime we push code, it immediately affects over 200 million web surfers. • Every day, more than 10,000 new customers sign-up for Cloudflare service. • Every week, the average Internet user touches us more than 500 times.

Slide 6

Slide 6 text

What is Cloudflare? • 152 data centres in 74 countries • More than 10 million domains • 10% of all Internet requests • 7.4M requests per second on average, 10M at peak • 1.6M DNS queries per second • 2.8 billion people served each month • Biggest DDoS attack - 942 Gbps • 15 Tbps network capacity and growing

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Life of a packet

Slide 9

Slide 9 text

A long time ago in a galaxy far, far away...

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

The OPTE Project Internet 2015 Map North America (ARIN) Europe (RIPE) Latin America (LACNIC) Asia Pacific (APNIC) Africa (AFRINIC) “Backbone” (highly connected networks) http://www.opte.org

Slide 12

Slide 12 text

Load Balancing Between Data Centres • Locality and congestion control • DNS • BGP • Anycast https://www.cloudflare.com/learning/dns/what-is-dns/

Slide 13

Slide 13 text

http://www.computerhistory.org/atchm/the-two-napkin-protocol

Slide 14

Slide 14 text

Types of Routing

Slide 15

Slide 15 text

Okay, our little packet is inside the DC

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Problems 1. Uneven load Image credit: https://landing.google.com/sre/book/chapters/load-balancing-datacenter.html

Slide 18

Slide 18 text

Problems 2. Different kinds of traffic

Slide 19

Slide 19 text

Problems 3. Per packet load balancing Image credit: https://flic.kr/p/imuUKx

Slide 20

Slide 20 text

Problems 4. Heterogeneous hardware Image credit: computer animated film Madagascar

Slide 21

Slide 21 text

Problems 5. Locality (e.g. for cache) and transport affinity Image credit: https://www.flickr.com/photos/10361931@N06/4259933727/

Slide 22

Slide 22 text

Problems 6. DDoS Image credit: Lockheed Martin F-22 Raptor https://youtu.be/UxgiBJATe9M

Slide 23

Slide 23 text

Problems 7. Group change Image credit: https://flic.kr/p/9Q8yS4

Slide 24

Slide 24 text

Problems 7. Graceful connection draining Image credit: German air force

Slide 25

Slide 25 text

Load balancing techniques

Slide 26

Slide 26 text

ECMP ID (packet) mod N, ID - some function that produces connection ID, e.g. 5-tuple flow; N - the number of configured backends. Uneven load Different kinds of traffic Per packet load balancing Heterogeneous hardware Transport affinity DDoS Group change Graceful connection draining

Slide 27

Slide 27 text

ECMP-CH populating the ECMP table not simply with next-hops, but with a slotted table that's made up of redundant next-hops Uneven load Different kinds of traffic Per packet load balancing Heterogeneous hardware Transport affinity DDoS Group change Graceful connection draining

Slide 28

Slide 28 text

Stateful Load Balancing Uneven load Different kinds of traffic Per packet load balancing Heterogeneous hardware Transport affinity DDoS Group change Graceful connection draining

Slide 29

Slide 29 text

Google Maglev

Slide 30

Slide 30 text

Daisy Chaining a.k.a Beamer https://www.usenix.org/conference/nsdi18/presentation/olteanu https://github.com/Beamer-LB • Beamer muxes do not keep per-connection state; each packet is forwarded independently. • When the target server changes, connections may break. • Beamer uses state stored in servers to redirect stray packets.

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Daisy Chaining a.k.a Beamer https://www.usenix.org/conference/nsdi18/presentation/olteanu https://github.com/Beamer-LB Uneven load Different kinds of traffic Per packet load balancing Heterogeneous hardware Transport affinity DDoS Group change Graceful connection draining Performance Spoilers: could be even better

Slide 36

Slide 36 text

Also FPGA Packet Processing

Slide 37

Slide 37 text

Fun (?) Facts https://www.fastly.com/blog/anatomy-an-iot-botnet-attack

Slide 38

Slide 38 text

An average IoT device gets infected with malware and launches an attack within 6 minutes of being exposed to the internet.

Slide 39

Slide 39 text

Over the span of a day an average of over 400 login attempts per device; 66 percent of them on average are successful.

Slide 40

Slide 40 text

Over the span of a day, IoT devices are probed for vulnerabilities 800 times per hour.

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Types of Attacks

Slide 43

Slide 43 text

Volumetric Attack Protocol Attack Application Attack What is it? Saturating the bandwidth of the target. Exploiting a weakness in the Layer 3 and Layer 4 protocol stack. Exploiting a weakness in the Layer 7 protocol stack. How does it cripple the target? Blocks access to the end-resource Consume all the processing capacity of the attacked-target or intermediate critical resources. Exhaust the server resources by monopolising processes and transactions. Examples NTP Amplification, DNS Amplification, UDP Flood, TCP Flood, QUIC HelloRequest amplification Syn Flood, Ping of Death, QUIC flood HTTP Flood, Attack on DNS Services

Slide 44

Slide 44 text

DDoS Mitigation

Slide 45

Slide 45 text

Disclaimer

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

• Low overhead sandboxed user-defined bytecode running in kernel • Written in a subset of C, compiled by clang llvm • It can never crash, hang or interfere with the kernel negatively • If you run Linux 3.15 or newer, you already have it • Great intro from Brendan Gregg: http://www.brendangregg.com/ebpf.html BPF and eBPF

Slide 49

Slide 49 text

https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/

Slide 50

Slide 50 text

Gatebot

Slide 51

Slide 51 text

iptables • Initially the only tool to filter traffic • Leveraged modules ipsets, hashlimit, connlimit • xt_bpf module allowed to specify complex filtering rules • But soon we started experiencing IRQ storms during big attacks • All CPUs were busy dropping packets, userspace applications were starving of CPU

Slide 52

Slide 52 text

Userspace Offload a.k.a. Kernel Bypass • Based on SolarFlare EF_VI • Network traffic is offloaded to userspace before it hits the Linux network stack • Allows to run BPF in userspace • An order of magnitude faster than iptables (5M pps) • Requires one or more CPUs to busy poll the NIC event queue • Reinjecting packets in the network stack is expensive • HARDWARE DEPENDANT

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

XDP to the rescue!

Slide 55

Slide 55 text

https://www.iovisor.org/technology/xdp XDP Packet Processing Overview +AF_XDP since 4.19

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

https://blog.cloudflare.com/how-to-drop-10-million-packets/

Slide 62

Slide 62 text

But… DPDK?

Slide 63

Slide 63 text

• Allows option of busy polling or interrupt driven networking • No need to allocate huge pages • Dedicated CPUs are not required, user has many options on how to structure the work between CPUs • No need to inject packets into the kernel from a third party user space application • No special hardware requirements • No need to define a new security model for accessing networking hardware • No third party code/licensing required https://github.com/iovisor/bpf-docs/blob/master/Express_Data_Path.pdf Advantages of XDP over DPDK

Slide 64

Slide 64 text

A perfect match: XDP both for load balancing and DDoS mitigation <3

Slide 65

Slide 65 text

XDP L4LB with daisy chaining using encapsulation Uneven load Different kinds of traffic Per packet load balancing Heterogeneous hardware Transport affinity DDoS Group change Graceful connection draining Performance

Slide 66

Slide 66 text

And they lived happily ever after

Slide 67

Slide 67 text

Bonus part

Slide 68

Slide 68 text

(Quick UDP Internet Connections)

Slide 69

Slide 69 text

https://blog.cloudflare.com/the-road-to-quic/

Slide 70

Slide 70 text

https://blog.cloudflare.com/the-road-to-quic/

Slide 71

Slide 71 text

L4LB to the rescue!

Slide 72

Slide 72 text

@advance_lunge Thank you! Image credit: https://www.pinterest.com/pin/4011087146768540