Cloud Infrastructure Interconnect
with Wireguard and OSPF
Date Huang 黃宇強
tjjh89017 [at] hotmail.com
2021/08/01
Slide 2
Slide 2 text
About me
Date Huang 黃宇強
● 2019 OpenInfra Day Taiwan Speaker
○ Massive Bare-Metal Operating System Provisioning Improvement
● 2019 OSC Tokyo Fall Speaker
● 2019 COScon ‘19 Speaker
● 2019 Hong Kong Open Source Conference Speaker
○ De-centralized Bare-Metal Operating System Provisioning
● 2018 ISC High Performance Project Poster Demo
○ The Design and Implementation of Bare Metal Cluster
Deployment Using BitTorrent
● 2017 Open Source Summit North America co-Speaker
○ Building Cloud Infra using cost-effective ARM Boards
● 2017 OpenStack Day Taiwan Speaker
○ Combine Continuous Integration (CI) with OpenStack
● 2016 OpenStack Day Taiwan Invited Speaker
○ OpenStack on ARM64
Open Source Porject
● EZIO
● STUNMESH-go
2
DWDM Solution brief - (1)
● Dense Wavelength Division
Multiplexing (DWDM)
● Fiber-Optic Communications
● Distance: 40, 80, 120KM or
more
● Merge multiple signal into
single paired fiber
5
Slide 6
Slide 6 text
DWDM Solution brief - (2)
● Rent Dark Fiber between different
location
● Connect each others with DWDM
Solution
6
Slide 7
Slide 7 text
DWDM Solution brief - (3)
● Pros
○ Large Bandwidth
○ Full control
● Cons
○ Very Expensive
○ Very Complex Configuration
○ Hard to Use and Maintain
7
Slide 8
Slide 8 text
MPLS VPN brief - (1)
● Multi-Protocol Label Switching
(MPLS)
● Using “Label” to determine the
route
● Provided by ISP
8
Slide 9
Slide 9 text
MPLS VPN brief - (2)
● Pros
○ Convenience
○ Easy to Use and Maintain
● Cons
○ Expensive
○ Less Bandwidth
9
Slide 10
Slide 10 text
SD-WAN IPsec brief - (1)
● Connect all sites with IPsec tunnel
via Broadband or Mobile Network
● Rich Redundancy between
multiple WAN type to Internet
○ Primary: Broadband Network
○ Backup: Mobile Network
● Need Fixed Public IP in most of
time
○ e.g. AWS VPC Customer Gateway
(IPsec) need Public IPs in both side
10
Slide 11
Slide 11 text
SD-WAN IPsec brief - (2)
● Pros
○ Simple
○ Cheap for larger bandwidth
○ Convenience
○ Easy to Use and Maintain
● Cons
○ Need Fixed Public IP at
Centralized Site
○ Centralized Architecture
11
Slide 12
Slide 12 text
Wireguard with
OSPF ● What is Wireguard
● Why Wireguard
● What is OSPF
● Why OSPF
12
Slide 13
Slide 13 text
What is Wireguard
● Simple, Fast, Modern, Secure Tunnel
● Fast without any hardware acceleration
● Support Windows, Linux, MacOS, Android, iOS
● Algo
○ Curve25519
○ ChaCha20
○ Poly1305
13
Slide 14
Slide 14 text
Why Wireguard
● UDP based VPN Protocol
○ NAT and Firewall Traversal Persistence
● Built-in Keepalive
● Built-in Roaming
○ Auto adjust remote peer connection info
● Performance better than OpenVPN and IPsec (AES)
○ Wireguard have good performance without hardware crypto engine
○ Suitable to Embedded system or Network Box
● Much Simpler Configuration than OpenVPN
○ OpenVPN need 5KB config file
○ Wireguard only need several bytes
● Encapsulate IPv6-in-IPv4, IPv4-in-IPv6
14
Slide 15
Slide 15 text
What is OSPF
● Dynamic Route
○ Don’t need to bother to setup static route when new peer is added
● Built-in Keepalive
○ Fast re-route to redundant route when link status or route is changed
● Fast Convergence in small scale network
15
Slide 16
Slide 16 text
Why OSPF
● Built-in Keepalive
○ Wireguard didn’t have link status
○ It will need to send a packet to remote peer and check if receving reply to check link
status
○ OSPF Hello Packet could test link status and check remote OSPF routing engine status
16
Slide 17
Slide 17 text
Wireguard with OSPF
● Fast and Simple VPN tunnel
● NAT Traversal Persistence
● Dynamic Route in Full Mesh Topology
● Fast Re-route
● Auto Check Link and Route Status
17
Mobile Network
● Through CGNAT
○ Full Cone NAT
○ Translate into Same Port and IP mapping when Destation IP and Port are different
● NAT Session to allow ingress traffic
○ Record Src IP, Src Port, Dst IP, Dst Port
○ Allow Firewall Rule
28
Slide 29
Slide 29 text
29
No Session From Site B to Site A
Deny Site A to Site B
Even Site A know NAT mapping about Site B
It still cannot connect to Site B
Slide 30
Slide 30 text
UDP Hole Punching
● Let two clients exchange connection info and try to connect to each other
via 3rd party server
● STUN: Session Traversal Utilities for NAT
● STUN is common implement for UDP Hole Punching (RFC 5389)
30
Slide 31
Slide 31 text
https://bford.info/pub/net/p2pnat/
31
Slide 32
Slide 32 text
STUNMESH-go
● Wireguard helper tool to get through Full-Cone NAT
○ IP and Port translation mapping will be same even different destination
● Written in Golang
● Multiple Architecture Support
○ X86_64, MIPS
● Standalone Executable
○ Don’t need to care about library dependency
● Inspired by wireguard-p2p project
● Open Source
○ https://github.com/tjjh89017/stunmesh-go
○ GPLv2 or later
● Tested with
○ UBNT ER-X v2.0.8-hotfix.1 and Wireguard v1.0.20210424
○ VyOS 1.4-rolling-202105200417
32
Slide 33
Slide 33 text
STUNMESH-go
● Get Public IP and Port after CG-NAT translation
○ cBPF Filter to receive packet from same UDP port as Wireguard used
○ Raw Socket to construct STUN 5389 request packet to same UDP port
● Encrypt Public Info with Wireguard Curve25519 Key
● Save Ciphertext into Cloudflare TXT Record
● Query TXT Record from Cloudflare
● Decrypt Ciphertext and Update Wireguard Peer Endpoint
● Usually, Only need to run once when initiating connection in first time
○ Or disconnect in the same time
33
Slide 34
Slide 34 text
Network between
Your Cloud and Office
Back to here
34
Slide 35
Slide 35 text
35
Slide 36
Slide 36 text
36
Slide 37
Slide 37 text
37
Slide 38
Slide 38 text
38
Slide 39
Slide 39 text
39
Slide 40
Slide 40 text
40
Slide 41
Slide 41 text
41
Slide 42
Slide 42 text
42
Slide 43
Slide 43 text
43
Slide 44
Slide 44 text
44
Slide 45
Slide 45 text
45
Slide 46
Slide 46 text
46
Slide 47
Slide 47 text
47
Slide 48
Slide 48 text
More than L3
Tunnel
48
Slide 49
Slide 49 text
More than L3 Tunnel
● Wireguard only encapsulate L3 packet into tunnel
○ Start from IP header
○ Without L2 MAC address, VLAN
● Some Protocols or Operations will need L2
○ VM migration, two sites need to be in same L2 if VMs connect to each other with same
subnet IPs
49
Slide 50
Slide 50 text
More than L3 Tunnel - L2 Tunnel
● VXLAN
● NVGRE
● HARD to control Broadcast, Unknown Unicast, Multicast in L2 tunnel
○ Proxy ARP
○ Static MAC table
○ Static ARP table
○ BGP-EVPN
50