IPsec mesh network: Perfect for the cloud?

IPsec mesh network: Perfect for the cloud?

Velocity 2015 session: http://velocityconf.com/devops-web-performance-2015/public/schedule/detail/41454
Video: https://www.youtube.com/watch?v=320KfBkGNyY

Private networks are traditionally assumed secure, and traffic crossing the public internet is secured via VPN. For data center-to-data center traffic, that involves a site-to-site tunnel with a VPN concentrator on either side of the encrypted tunnel.

This hub-and-spoke architecture presents a simple solution for protecting network traffic over the internet, but making that solution highly available involves added complexity. Multiple tunnels need to exist, with traffic load balanced across them. Automation must exist to detect down (and half down) links and redirect traffic. Also, these tunnels should scale up as the amount of traffic passing between data centers increases. And this solution does nothing to protect network traffic on the private network—a private network that is increasingly managed by cloud providers and shared with other companies.

When traffic is flowing over networks that we don’t manage (both over the WAN and the LAN), it is time to rethink our network security practices. By using DevOps practices in our network systems, PagerDuty was able to get rid of the hub-and-spoke model and instead use an IPSec mesh architecture. Each server in our system establishes a secure association with its peer and transmits all traffic using IPSec transport. Each host manages encryption and decryption of its own traffic, so our ability to protect that traffic naturally scales up as we add new infrastructure.

This talk will focus on how we implemented that model on our Linux fleet. We will dig into the details of our configuration including the policies we use, and the encryption and authentication mechanisms in place. We will talk about how this model performs on our systems and the impact it has on the production workload. Finally, we will discuss how it handles failure, bugs we’ve found along the way, and how we see this model changing as our infrastructure continues to grow.

In the end, I hope to have given everyone a better understanding of how VPNs work, and how through combining the development and operations disciplines we can produce a solution that was previously considered impractical.

A97a75c945507f70992f579a730b0657?s=128

Doug Barth

May 28, 2015
Tweet

Transcript

  1. 4.

    5/29/15 IPSEC MESH NETWORK: PERFECT FOR THE CLOUD? Traditional setup

    PagerDuty’s setup Rollout Tales from production Should you do this?
  2. 7.

    5/29/15 IPSEC MESH NETWORK: PERFECT FOR THE CLOUD? TLS landmines

    Finagle’s TLS + IPsec IPsec only TLS penalty in MySQL
  3. 12.

    5/29/15 Hosting setup Cloud providers only Multi-datacenter deployment Several different

    technologies Percona XtraDB Cluster Zookeeper Cassandra Ruby/Rails Scala/Finagle nginx IPSEC MESH NETWORK: PERFECT FOR THE CLOUD? US-WEST-2 US-WEST- 1 LINODE
  4. 13.

    5/29/15 PagerDuty’s networking goals Encrypted by default Failures at the

    instance level (not the DC level) Throughput scales as our infrastructure grows IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  5. 14.

    5/29/15 Mesh network Every box handles its own encryption Policy

    enforcement distributed CM makes this manageable IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  6. 16.

    5/29/15 # This configures how Phase 1 key exchange occurs.

    We keep the remote end # anonymous so we don't have to bounce racoon (and therefore lose our SAs) when # new boxes are added. remote anonymous { … Phase 1 IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  7. 17.

    5/29/15 exchange_mode main; Phase 1 IPSEC MESH NETWORK: PERFECT FOR

    THE CLOUD? POLICY DHE AUTH INITIATOR RESPONDER
  8. 19.

    5/29/15 Phase 1 IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?

    # Per connection PSK # box01 10.0.0.1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 50.0.0.1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef # box02 10.0.0.2 0x0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef 50.0.0.2 0x0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
  9. 20.

    5/29/15 # EC2 doesn’t route ESP nat_traversal force; Phase 1

    IPSEC MESH NETWORK: PERFECT FOR THE CLOUD? UDP ESP
  10. 22.

    5/29/15 dpd_delay 20; Phase 1 IPSEC MESH NETWORK: PERFECT FOR

    THE CLOUD? INITIATOR RESPONDER R-U-THERE R-U-THERE-ACK
  11. 23.

    5/29/15 # This configures the SA parameters. Again, anonymous so

    we don't need to # bounce racoon when new boxes are added. sainfo anonymous { pfs_group modp3072; encryption_algorithm aes; authentication_algorithm hmac_sha256; compression_algorithm deflate; lifetime time 8 hours; } Phase 2 IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  12. 24.

    5/29/15 # WAN spdadd 50.0.0.1 10.0.0.2 any -P out ipsec

    esp/transport//require; spdadd 10.0.0.2 50.0.0.1 any -P in ipsec esp/transport//require; spdadd 10.0.0.1 50.0.0.2 any -P out ipsec esp/transport//require; spdadd 50.0.0.2 10.0.0.1 any -P in ipsec esp/transport//require; # LAN spdadd 10.0.0.1 10.0.0.2 any -P out ipsec esp/transport//require; spdadd 10.0.0.2 10.0.0.1 any -P in ipsec esp/transport//require; SPD — WAN & LAN encryption IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  13. 25.

    5/29/15 spdadd 0.0.0.0/0 0.0.0.0/0[22] tcp -P out prio def +

    100 none; spdadd 0.0.0.0/0 0.0.0.0/0[22] tcp -P in prio def + 100 none; spdadd 0.0.0.0/0[22] 0.0.0.0/0 tcp -P out prio def + 100 none; spdadd 0.0.0.0/0[22] 0.0.0.0/0 tcp -P in prio def + 100 none; SPD — SSH excluded IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  14. 26.

    5/29/15 # Exclude ICMP from IPsec. # # Having ICMP

    encrypted makes it difficult for us to investigate networking # issues. traceroute between machines doesn't work because traceroute doesn't # realize that the TTL expired packet referencing the UDP-encap packet is meant # for it. A similar issue exists for mtr. spdadd 0.0.0.0/0 0.0.0.0/0 icmp -P out prio def + 100 none; spdadd 0.0.0.0/0 0.0.0.0/0 icmp -P in prio def + 100 none; SPD — ICMP excluded too IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  15. 27.

    5/29/15 pd-sync-policies IPSEC MESH NETWORK: PERFECT FOR THE CLOUD? Found

    2589 existing policies, 1718 existing point-to-point policies Loading policies from ["/etc/ipsec-tools.conf", "/etc/ipsec-tools.d/*.conf"] Found 1718 policies in the config file Found 0 changed policies Found 0 new policies Found 0 old policies setkey returned successfully
  16. 34.

    5/29/15 $ sudo setkey -DP # SPD entries $ sudo

    racoonctl -l show-sa isakmp # Phase 1 relationships $ sudo setkey -D # Phase 2 relationships $ sudo ip xfrm state # Phase 2 relationships (newer format) Scraping lots of metrics IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  17. 36.

    5/29/15 $ cat /proc/net/xfrm_stat XfrmInError 0 XfrmInBufferError 0 XfrmInHdrError 0

    XfrmInNoStates 1238 XfrmInStateProtoError 8 XfrmInStateModeError 0 XfrmInStateSeqError 500 XfrmInStateExpired 0 XfrmInStateMismatch 0 XfrmInStateInvalid 0 XfrmInTmplMismatch 0 XfrmInNoPols 0 CONFIG_XFRM_STATISTICS IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  18. 38.

    5/29/15 Lessons learned AES works well, even without hardware acceleration

    Network probing is more difficult Path MTU breaks after route timeout. Still investigating. IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  19. 39.

    5/29/15 Ordering issues in DPD DPD is on phase 1

    relationship No liveness check on phase 2 relationships SAs can get out of sync Requires manually clearing the relationships IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  20. 40.

    5/29/15 Linux 3.7 - 3.13 xfrm4_gc_thresh 2.6 used a large

    dynamic value 3.7 switched to static value of 1024 3.13 bumped to 32K after performance issues IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?
  21. 42.

    5/29/15 IPSEC MESH NETWORK: PERFECT FOR THE CLOUD? “Corruption during

    AES encryption in Xen v4.1 or v3.4 paravirtual guests running a Linux 3.0+ kernel, combined with the lack of TCP checksum validation in IPSec Transport mode, which leads to the admission of corrupted TCP data on a ZooKeeper node, resulting in an unhandled exception from which ZooKeeper is unable to recover. Jeez. Talk about a needle in a haystack…”
  22. 44.

    5/29/15 Need an agent Automatically manage policies Handle metrics collection

    and emission Implement phase 2 liveness checks GC old relationships IPSEC MESH NETWORK: PERFECT FOR THE CLOUD?