Kubernetes and Networks - why is this so dang hard?

A brief look at models for integrating Kubernetes clusters into existing networks.

Transcript

1. Kubernetes and networks
   Why is this so dang hard?
   Tim Hockin
   @thockin
   v5
   

   View Slide

2. Kubernetes clusters are made up of nodes
   ● Machines - virtual or physical
   Those nodes exist on some network
   Pods run on those nodes
   Pods get IP addresses
   “Network model” describes how those pod IPs integrate with the
   larger network
   What does “network model” mean?
   

   View Slide

3. Start with a “normal” cluster
   

   View Slide

4. Network: 10.0.0.0/8
   

   View Slide

5. Network: 10.0.0.0/8
   Cluster: 10.0.0.0/16
   

   View Slide

6. NOTE: It’s not *required* that
   a cluster be a single IP range,
   but it’s common and makes
   the pictures easier
   

   View Slide

7. Network: 10.0.0.0/8
   Cluster: 10.0.0.0/16
   Node1:
   IP: 10.240.0.1
   Node2:
   IP: 10.240.0.2
   

   View Slide

8. Network: 10.0.0.0/8
   Cluster: 10.0.0.0/16
   Node1:
   IP: 10.240.0.1
   Pod range: 10.0.1.0/24
   Node2:
   IP: 10.240.0.2
   Pod range: 10.0.2.0/24
   

   View Slide

9. Network: 10.0.0.0/8
   Cluster: 10.0.0.0/16
   Node1:
   IP: 10.240.0.1
   Pod range: 10.0.1.0/24
   Node2:
   IP: 10.240.0.2
   Pod range: 10.0.2.0/24
   NOTE: Different
   Ranges
   

   View Slide

10. NOTE: It’s not *required* that
    nodes have a predefined IP
    range, but it’s common and
    makes the pictures easier
    

    View Slide

11. Network: 10.0.0.0/8
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-b:
    10.0.1.2
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    

    View Slide

12. Pods get IPs from the node’s
    IP range (again, usually but
    not always)
    

    View Slide

13. Kubernetes demands that
    pods can reach each other
    

    View Slide

14. Network: 10.0.0.0/8
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-b:
    10.0.1.2
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    

    View Slide

15. Kubernetes does not say
    anything about things outside
    of the cluster
    

    View Slide

16. Network: 10.0.0.0/8
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-b:
    10.0.1.2
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Other:
    10.128.1.1
    ?
    

    View Slide

17. Multi-cluster makes it even
    more confusing
    

    View Slide

18. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.1.0.0/16
    Node1:
    IP: 10.240.0.3
    Pod range: 10.1.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.1.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    ?
    ?
    

    View Slide

19. Network models
    (not exhaustive)
    

    View Slide

20. Fully-integrated (aka flat)
    

    View Slide

21. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.1.0.0/16
    Node1:
    IP: 10.240.0.3
    Pod range: 10.1.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.1.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    

    View Slide

22. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.1.0.0/16
    Node1:
    IP: 10.240.0.3
    Pod range: 10.1.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.1.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    NOTE: Different
    Ranges
    

    View Slide

23. Each node owns an IP range
    from the larger network
    Everyone on the network
    knows how to deal with that
    (or the network deals with it
    for them)
    

    View Slide

24. Good when:
    ● IP space is available
    ● Network is programmable / dynamic
    ● Need high integration / performance
    ● Kubernetes is a large part of your footprint
    

    View Slide

25. Bad when:
    ● IP fragmentation / scarcity
    ● Hard-to-configure network infrastructure
    ● Kubernetes is a small part of your footprint
    

    View Slide

26. Fully-isolated
    

    View Slide

27. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.1.0.0/16
    Node1:
    IP: 10.240.0.3
    Pod range: 10.1.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.1.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    

    View Slide

28. No connectivity from inside to
    outside or vice-versa!
    

    View Slide

29. In fact, you can re-use all of
    the IPs
    

    View Slide

30. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    NOTE: Same
    Range
    

    View Slide

31. In fact, they are basically on
    different networks
    

    View Slide

32. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

33. May be easier to reason about
    security boundaries
    

    View Slide

34. Good when:
    ● Don’t need integration
    ● IP space is scarce / fragmented
    ● Network is not programmable / dynamic
    

    View Slide

35. Bad when:
    ● Need communication across a cluster-edge
    

    View Slide

36. Island mode
    

    View Slide

37. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.1.0.0/16
    Node1:
    IP: 10.240.0.3
    Pod range: 10.1.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.1.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    gateway
    gateway
    gateway
    

    View Slide

38. Ingress and egress traffic
    goes thru one or more
    abstract “gateways” (more on
    that later)
    

    View Slide

39. You can re-use the Pod IPs (a
    major motivation for this
    model), but node IPs come
    from the larger network
    

    View Slide

40. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    gateway
    gateway
    gateway
    NOTE: Same
    Range
    

    View Slide

41. Can be implemented as an
    overlay network or not
    

    View Slide

42. Another way to think of this:
    clusters have a private
    network for their pods; nodes
    have one leg in the main
    network and one leg in the
    cluster network
    

    View Slide

43. Cluster A pods
    10.0.0.0/16
    Cluster B pods
    10.0.0.0/16
    “Main” network
    10.0.0.0/8
    Cluster A
    Nodes
    Cluster B
    Nodes
    “Hole”
    10.0.0.0/16
    Other
    Any pod can reach the “main” network by masquerading as
    its node, but not vice-versa (except via a gateway)
    

    View Slide

44. Good when:
    ● Need some integration
    ● IP space is scarce / fragmented
    ● Network is not programmable / dynamic
    

    View Slide

45. Bad when:
    ● Need to debug connectivity
    ● Need direct-to-endpoint communications
    ● Need a lot of services exposed (especially
    non-HTTP)
    ● Rely on client IPs for firewalls
    ● Large number of nodes
    

    View Slide

46. Various forms of “gateway”
    

    View Slide

47. Gateway: nodes
    

    View Slide

48. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

49. Ingress: Service NodePorts
    

    View Slide

50. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

51. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

52. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

53. Node uses IP dst_port to
    route to correct service
    

    View Slide

54. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

55. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

56. You can ingress L4 into an L7
    proxy and forward from there
    (e.g. Ingress controllers)
    

    View Slide

57. Egress: IP Masquerade
    (aka SNAT)
    

    View Slide

58. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

59. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

60. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

61. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

62. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

63. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

64. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    

    View Slide

65. SNAT obscures client IP
    (Traffic from pods on a node
    comes from the node’s IP)
    

    View Slide

66. Gateway: VIP (ingress)
    

    View Slide

67. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    VIP
    VIP
    

    View Slide

68. Similar to NodePort, but node
    uses IP dst_ip to route
    

    View Slide

69. Still needs something like
    SNAT to egress
    

    View Slide

70. Gateway: Proxy (ingress)
    

    View Slide

71. Network: 10.0.0.0/8
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1: 10.0.1.0/24 Node2: 10.0.2.0/24
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.0.0.0/16
    Node1: 10.1.1.0/24 Node2: 10.1.2.0/24
    Node1:
    IP: 10.240.0.3
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Proxy
    Proxy
    

    View Slide

72. Can either route to NodePort
    or directly to pod IPs
    (e.g. proxy has special config
    to “get onto the island”)
    

    View Slide

73. Still needs something like
    SNAT to egress
    

    View Slide

74. There’s a LOT more to know
    about ingress (for another
    presentation)
    

    View Slide

75. Options for egress are poorly
    explored, so far
    

    View Slide

76. Archipelago
    (aka “bigger islands”)
    

    View Slide

77. Network: 10.0.0.0/8
    Cluster: 10.0.0.0/16
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.1.0.0/16
    Node1:
    IP: 10.240.0.3
    Pod range: 10.1.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.1.2.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    gateway
    

    View Slide

78. Flat within the archipelago
    

    View Slide

79. Network: 10.0.0.0/8
    Cluster: 10.0.0.0/16
    Other:
    10.128.1.1
    Cluster: 10.0.0.0/16
    Node1:
    IP: 10.240.0.1
    Pod range: 10.0.1.0/24
    Node2:
    IP: 10.240.0.2
    Pod range: 10.0.2.0/24
    Pod-a:
    10.0.1.1
    Pod-c:
    10.0.2.1
    Pod-d:
    10.0.2.2
    Pod-b:
    10.0.1.2
    Cluster: 10.1.0.0/16
    Node1:
    IP: 10.240.0.3
    Pod range: 10.1.1.0/24
    Node2:
    IP: 10.240.0.4
    Pod range: 10.1.1.0/24
    Pod-a:
    10.1.1.1
    Pod-c:
    10.1.2.1
    Pod-d:
    10.1.2.2
    Pod-b:
    10.1.1.2
    gateway
    NOTE: Different
    Ranges
    

    View Slide

80. Can’t reuse pod IPs between
    clusters, but can between
    archipelagos
    

    View Slide

81. Island mode to the rest of the
    network
    

    View Slide

82. Archipelago A pods
    10.0.0.0/14
    Archipelago B pods
    10.0.0.0/14
    “Main” network
    10.0.0.0/8
    Cluster A
    Nodes
    Cluster C
    Nodes
    “Hole”
    10.0.0.0/14
    Other
    Cluster B
    Nodes
    Cluster D
    Nodes
    

    View Slide

83. Can be implemented as an
    overlay network or not
    

    View Slide

84. Good when:
    ● Need high integration across clusters
    ● Need some integration with non-kubernetes
    ● IP space is scarce / fragmented
    ● Network is not programmable / dynamic
    

    View Slide

85. Bad when:
    ● Need to debug connectivity
    ● Need direct-to-endpoint communications
    ● Need a lot of services exposed to non-k8s
    ● Rely on client IPs for firewalls
    ● Large number of nodes across all clusters
    

    View Slide

86. Gateway options are similar
    to plain island mode
    

    View Slide

87. Which one should you use?
    

    View Slide

88. There is no “right answer”.
    You have to consider the
    tradeoffs.
    Sorry.
    

    View Slide