Illustrated Guide To Kubernetes Networking

A short walk through of some ideas around container networking.

Transcript

1. Google Cloud Platform
   An Illustrated Guide to
   Kubernetes Networking
   Tim Hockin
   Senior Staff Software Engineer
   @thockin
   

   View Slide

2. Google Cloud Platform
   Layer 2: ethernet
   

   View Slide

3. Google Cloud Platform
   node-a
   node-c
   node-b
   node-d
   192.168.1.1/16
   11:22:33:44:55:01
   192.168.1.2/16
   01:23:45:67:89:02
   192.168.1.3/16
   11:22:33:44:55:03
   192.168.1.4/16
   01:23:45:67:89:04
   L2
   switch
   

   View Slide

4. Google Cloud Platform
   node-a
   node-c
   node-b
   node-d
   L2
   to: 192.168.1.3
   from: 192.168.1.1
   GET / 192.168.1.1/16
   11:22:33:44:55:01
   192.168.1.2/16
   01:23:45:67:89:02
   192.168.1.3/16
   11:22:33:44:55:03
   192.168.1.4/16
   01:23:45:67:89:04
   

   View Slide

5. Google Cloud Platform
   node-d
   node-b
   node-a
   node-c
   L2
   to:
   from: 11:22:33:44:55:01
   who has 192.168.1.3?
   to: 192.168.1.3
   from: 192.168.1.1
   GET / 192.168.1.1/16
   11:22:33:44:55:01
   192.168.1.2/16
   01:23:45:67:89:02
   192.168.1.3/16
   11:22:33:44:55:03
   192.168.1.4/16
   01:23:45:67:89:04
   “ARP request”
   

   View Slide

6. Google Cloud Platform
   node-a
   node-c
   node-b
   node-d
   L2
   to: 192.168.1.3
   from: 192.168.1.1
   GET / 192.168.1.1/16
   11:22:33:44:55:01
   192.168.1.2/16
   01:23:45:67:89:02
   192.168.1.3/16
   11:22:33:44:55:03
   192.168.1.4/16
   01:23:45:67:89:04
   to: 11:22:33:44:55:01
   from: 11:22:33:44:55:03
   I have 192.168.1.3
   “ARP response”
   

   View Slide

7. Google Cloud Platform
   node-a
   node-c
   node-b
   node-d
   L2
   to: 192.168.1.3
   via: 11:22:33:44:55:03
   from: 192.168.1.1
   GET /
   192.168.1.1/16
   11:22:33:44:55:01
   192.168.1.2/16
   01:23:45:67:89:02
   192.168.1.3/16
   11:22:33:44:55:03
   192.168.1.4/16
   01:23:45:67:89:04
   

   View Slide

8. Google Cloud Platform
   node-a
   root netns
   eth0: 192.168.1.1/16
   11:22:33:44:55:01
   L2 with containers
   cbr0: 10.0.1.1/24
   ctr-1
   eth0: 10.0.1.2/24
   ctr-2
   eth0: 10.0.1.3/24
   ctr-3
   eth0: 10.0.1.4/24
   

   View Slide

9. Google Cloud Platform
   node-a
   192.168.1.1/16
   11:22:33:44:55:01
   node-c
   node-b
   node-d
   192.168.1.2/16
   01:23:45:67:89:02
   192.168.1.3/16
   11:22:33:44:55:03
   192.168.1.4/16
   01:23:45:67:89:04
   L2
   ctr-1 10.0.1.2
   aa:bb:cc:dd:e1:01
   ctr-2 10.0.3.2
   aa:bb:cc:dd:e3:02
   

   View Slide

10. Google Cloud Platform
    node-a
    192.168.1.1/16
    11:22:33:44:55:01
    node-c
    node-b
    node-d
    192.168.1.2/16
    01:23:45:67:89:02
    192.168.1.3/16
    11:22:33:44:55:03
    192.168.1.4/16
    01:23:45:67:89:04
    L2
    ctr-1 10.0.1.2
    aa:bb:cc:dd:e1:01
    ctr-2 10.0.3.2
    aa:bb:cc:dd:e3:02
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    

    View Slide

11. Google Cloud Platform
    node-a
    192.168.1.1/16
    11:22:33:44:55:01
    node-c
    node-b
    node-d
    192.168.1.2/16
    01:23:45:67:89:02
    192.168.1.3/16
    11:22:33:44:55:03
    192.168.1.4/16
    01:23:45:67:89:04
    L2
    ctr-1 10.0.1.2
    aa:bb:cc:dd:e1:01
    ctr-2 10.0.3.2
    aa:bb:cc:dd:e3:02
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    to:
    from: aa:bb:cc:dd:e1:01
    who has 10.0.3.2?
    “ARP request”
    

    View Slide

12. Google Cloud Platform
    node-a
    192.168.1.1/16
    11:22:33:44:55:01
    node-c
    node-b
    node-d
    192.168.1.2/16
    01:23:45:67:89:02
    192.168.1.3/16
    11:22:33:44:55:03
    192.168.1.4/16
    01:23:45:67:89:04
    L2
    ctr-1 10.0.1.2
    aa:bb:cc:dd:e1:01
    ctr-2 10.0.3.2
    aa:bb:cc:dd:e3:02
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    to: aa:bb:cc:dd:e1:01
    from: 11:22:33:44:55:03
    I have 10.0.3.2
    “proxy ARP
    response”
    

    View Slide

13. Google Cloud Platform
    node-a
    192.168.1.1/16
    11:22:33:44:55:01
    node-c
    node-b
    node-d
    192.168.1.2/16
    01:23:45:67:89:02
    192.168.1.3/16
    11:22:33:44:55:03
    192.168.1.4/16
    01:23:45:67:89:04
    L2
    ctr-1 10.0.1.2
    aa:bb:cc:dd:e1:01
    ctr-2 10.0.3.2
    aa:bb:cc:dd:e3:02
    to: 10.0.3.2
    via: 11:22:33:44:55:03
    from: 10.0.1.2
    GET /
    

    View Slide

14. Google Cloud Platform
    Layer 3 - IP
    

    View Slide

15. Google Cloud Platform
    node-a
    node-c
    node-b
    node-d
    192.168.1.1/32 192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    gateway
    

    View Slide

16. Google Cloud Platform
    node-a
    node-c
    node-b
    node-d
    192.168.1.1/32 192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    to: 192.168.1.3
    from: 192.168.1.1
    GET /
    

    View Slide

17. Google Cloud Platform
    node-a
    node-c
    node-b
    node-d
    192.168.1.1/32 192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    to: 192.168.1.3
    from: 192.168.1.1
    GET /
    

    View Slide

18. Google Cloud Platform
    node-a
    node-c
    node-b
    node-d
    192.168.1.1/32 192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    to: 192.168.1.3
    from: 192.168.1.1
    GET /
    routing decision,
    static or learned
    (e.g. BGP)
    

    View Slide

19. Google Cloud Platform
    node-a
    192.168.1.1/32
    node-c
    node-b
    node-d
    192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    

    View Slide

20. Google Cloud Platform
    node-a
    192.168.1.1/32
    node-c
    node-b
    node-d
    192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    

    View Slide

21. Google Cloud Platform
    node-a
    192.168.1.1/32
    node-c
    node-b
    node-d
    192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    

    View Slide

22. Google Cloud Platform
    node-a
    192.168.1.1/32
    node-c
    node-b
    node-d
    192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    routing decision,
    static or learned
    (e.g. BGP)
    

    View Slide

23. Google Cloud Platform
    node-a
    192.168.1.1/32
    node-c
    node-b
    node-d
    192.168.1.2/32
    192.168.1.3/32 192.168.1.4/32
    L3
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    

    View Slide

24. Google Cloud Platform
    Overlays
    Q: When should I use an overlay?
    A: When nothing else works, or when you have
    specific reasons to want it (e.g. the added value
    of management)
    

    View Slide

25. Google Cloud Platform
    node-a
    root netns
    eth0: 192.168.1.1/16
    Overlay (e.g. flannel, weave)
    cbr0: 10.0.1.1/24
    ctr-1
    eth0: 10.0.1.2/24
    ctr-2
    eth0: 10.0.1.3/24
    ctr-3
    eth0: 10.0.1.4/24
    flannel0: 10.0.1.254/16
    

    View Slide

26. Google Cloud Platform
    node-a
    root netns
    eth0: 192.168.1.1/16
    cbr0: 10.0.1.1/24
    ctr-1
    eth0: 10.0.1.2/24
    ctr-2
    eth0: 10.0.1.3/24
    ctr-3
    eth0: 10.0.1.4/24
    flannel0: 10.0.1.254/16
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

27. Google Cloud Platform
    node-a
    root netns
    eth0: 192.168.1.1/16
    cbr0: 10.0.1.1/24
    ctr-1
    eth0: 10.0.1.2/24
    ctr-2
    eth0: 10.0.1.3/24
    ctr-3
    eth0: 10.0.1.4/24
    flannel0: 10.0.1.254/16
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

28. Google Cloud Platform
    node-a
    root netns
    eth0: 192.168.1.1/16
    cbr0: 10.0.1.1/24
    ctr-1
    eth0: 10.0.1.2/24
    ctr-2
    eth0: 10.0.1.3/24
    ctr-3
    eth0: 10.0.1.4/24
    flannel0: 10.0.1.254/16
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

29. Google Cloud Platform
    node-a
    root netns
    eth0: 192.168.1.1/16
    cbr0: 10.0.1.1/24
    ctr-1
    eth0: 10.0.1.2/24
    ctr-2
    eth0: 10.0.1.3/24
    ctr-3
    eth0: 10.0.1.4/24
    flannel0: 10.0.1.254/16
    to: 192.168.1.3
    from: 192.168.1.1
    encap:
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

30. Google Cloud Platform
    node-a
    192.168.1.1/16
    node-c
    node-b
    node-d
    192.168.1.2/16
    192.168.1.3/16 192.168.1.4/16
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    Overlay (e.g. flannel, weave)
    

    View Slide

31. Google Cloud Platform
    node-c
    root netns
    eth0: 192.168.1.3/16
    cbr0: 10.0.3.1/24
    ctr-4
    eth0: 10.0.3.2/24
    ctr-5
    eth0: 10.0.3.3/24
    ctr-6
    eth0: 10.0.3.4/24
    flannel0: 10.0.3.254/16
    to: 192.168.1.3
    from: 192.168.1.1
    encap:
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

32. Google Cloud Platform
    node-c
    root netns
    eth0: 192.168.1.3/16
    cbr0: 10.0.3.1/24
    ctr-4
    eth0: 10.0.3.2/24
    ctr-5
    eth0: 10.0.3.3/24
    ctr-6
    eth0: 10.0.3.4/24
    flannel0: 10.0.3.254/16
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

33. Google Cloud Platform
    node-c
    root netns
    eth0: 192.168.1.3/16
    cbr0: 10.0.3.1/24
    ctr-4
    eth0: 10.0.3.2/24
    ctr-5
    eth0: 10.0.3.3/24
    ctr-6
    eth0: 10.0.3.4/24
    flannel0: 10.0.3.254/16
    to: 10.0.3.2
    from: 10.0.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

34. Google Cloud Platform
    Overlays - the hard part
    

    View Slide

35. Google Cloud Platform
    node-a
    192.168.1.1/16
    node-c
    non-node
    node-d
    192.168.1.2/16
    192.168.1.3/16 192.168.1.4/16
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    Overlay (e.g. flannel, weave)
    

    View Slide

36. Google Cloud Platform
    node-a
    192.168.1.1/16
    node-c
    non-node
    node-d
    192.168.1.2/16
    192.168.1.3/16 192.168.1.4/16
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    to: 10.0.3.2
    from: 192.168.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

37. Google Cloud Platform
    node-a
    192.168.1.1/16
    node-c
    non-node
    node-d
    192.168.1.2/16
    192.168.1.3/16 192.168.1.4/16
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    to: 10.0.3.2
    from: 192.168.1.2
    GET /
    Overlay (e.g. flannel, weave)
    

    View Slide

38. Google Cloud Platform
    node-a
    192.168.1.1/16
    node-c
    non-node
    node-d
    192.168.1.2/16
    192.168.1.3/16 192.168.1.4/16
    ctr-1 10.0.1.2
    ctr-2 10.0.3.2
    to: 10.0.3.2
    from: 192.168.1.2
    GET /
    ?!?!
    Overlay (e.g. flannel, weave)
    

    View Slide

39. Google Cloud Platform
    We need a bridge between the
    physical and overlay networks...
    

    View Slide

40. Google Cloud Platform
    We need a bridge between the
    physical and overlay networks...
    • could: route to nodes
    • could: route to 1 or more bridge machines
    • could: run flannel on client machines
    

    View Slide

41. Google Cloud Platform
    We need a bridge between the
    physical and overlay networks...
    • could: route to nodes
    • could: route to 1 or more bridge machines
    • could: run flannel on client machines
    • see “When should I use an overlay?”
    

    View Slide