Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Refresh DNS Infrastructure with Modern Datacenter Network

Refresh DNS Infrastructure with Modern Datacenter Network

JANOG48 Meeting
https://www.janog.gr.jp/meeting/janog48/linedns-en/

Verda Network Development Team, LINE Corporation
Kawakami Kento

LINE Developers
PRO

July 16, 2021
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. Refresh DNS Infrastructure
    with Modern Datacenter Network
    KAWAKAMI KENTO, VERDA NETWORK DEVELOPMENT TEAM, LINE CORPORATION

    View Slide

  2. Agenda • LINE Private Cloud
    • Previous DNS redundancy methods
    • IP ANYCAST with CLOS Network

    View Slide

  3. • Network software engineer
    • DNS
    • OpenStack
    • Kubernetes
    • Kawakami Kento
    • LINE Corporation
    • Network Development Team
    • LINE New grad in 2019
    2

    View Slide

  4. LINE Private Cloud
    3

    View Slide

  5. Our Services
    B
    B A
    4

    View Slide

  6. DNS as a Service
    Verda DNSDashboard
    5

    View Slide

  7. DNS Architecture
    Designate
    l ')#846?
    l *& ! A5
    DNS Relay
    l DesignateDNS CacheDNS Auth
    *;9:.3
    DNS Auth
    l 0@2+$("-<

    DNS Cache
    l Full resolver =4,@%"
    B>@B7/13
    *
    %"
    6

    View Slide

  8. DNS Architecture
    Designate
    l ')#846?
    l *& ! A5
    DNS Relay
    l DesignateDNS CacheDNS Auth
    *;9:.3
    DNS Auth (VRRP => IP ANYCAST)
    l 0@2+$("-<

    DNS Cache (HWLB => IP ANYCAST)
    l Full resolver =4,@%"
    B>@B7/13
    *
    %"
    7

    View Slide

  9. Background of the replacement
    8
    • (#+/'0,2M@
    • 2 I5JDNS$named.conf>E9

    • 8D!4
    • Physical MachineM@
    • PM$.&0;.!)F
    • VerdaAVM
    8G
    • DNS Auth/CacheBH4• *2"2 7O:DNR
    • BH4• C2%-SQ
    • CLOS NetworkA8G
    • CLOS Network*. 6=8D

    View Slide

  10. VRRP(Virtual Router Redundancy Protocol)
    • VIP6APR Request6

    Primary #:*
    ,/9$*
    • DNS Auth /9$>?
    +A
    • DNS Auth('.
    Service
    • DoS('-5Service"&
    <3

    • @8
    • Act-Stb)2
    • *
    • L2)2=0
    • Act-Stb47!
    <3%;1
    9

    View Slide

  11. HWLB(Hardware Load Balancer)
    • HWLB1I #*!#
    ($E74
    • DNS [email protected] 6K
    • DNS Auth -0D32
    /C;A:8LK
    • )J,
    • &)HWLB4
    • H>
    • #?=DNS'*"*
    B :5
    • VIP(%)HWLB+
    <

    • EoL
    • EoS
    10

    View Slide

  12. VRRP HWLB
    Pros
    Cons
    • -*'
    • Act-Act&$ "


    • Act-Act&$+
    • L2#,
    • Act-Stb !
    %
    )(
    • -*'
    11

    View Slide

  13. CLOS Network
    • LeafSpine $08
    *2
    *5!
    • !%6+
    • Spine7#
    Leaf&3"1#
    • Spine/ Super-Spine7#
    4':!
    • East-West (
    • ECMP(Equal Cost Multi Path)
    )<
    9,3" -
    .;
    12
    [1]: LINE
    https://www.janog.gr.jp/meeting/janog43/program/line/

    View Slide

  14. Full L3 CLOS Network
    • ToR SwitchHypervisoreBGP
    Peering
    • Hypervisor FRRCLOSVM

    • Hypervisor
    VM IP
    /32
    13

    View Slide

  15. BGP advertisement from VM
    • VMFRRHypervisorFRR
    Peering
    • HypervisorFRR ToR


    14

    View Slide

  16. Advertise DNS VIP(1/3)
    • BGP
    VM
    VMDNSVIP
    15

    View Slide

  17. Advertise DNS VIP(2/3)
    • BGPVM

    VM DNSVIP
    • DNSVMHypevisor

    • DNSVM VIP
    16

    View Slide

  18. Advertise DNS VIP(3/3)
    • BGP %"VM
    $#VMDNS VIP"
    • DNS VMHypevisor !

    • DNS VMVIP"
    • ECMP#

    • CLOS NetworkBest Path
    %
    17

    View Slide

  19. Benefit of IP ANYCAST for DNS(1/4)


    • ECMP
    • L3 Switch!
    )

    • CLOS Network %
    $(
    • L2 network#'
    • BGP& Full L3
    Network "
    18

    View Slide

  20. Benefit of IP ANYCAST for DNS(2/4)
    • Act-Act41(0!
    !$'*3
    • CLOS&
    IP ANYCST;

    ECMP9 !
    :<+2
    • % $$)8BGP9
    ;'*3
    • >/ NW6- ! =.
    ,5#7
    "% $;'*3
    19

    View Slide

  21. Benefit of IP ANYCAST for DNS(3/4)
    • DNS Auth DNS Cache!
    (
    • VRRP HWLB "
    *'

    • IP ANYCAST DNS Auth/Cache
    !$)"%&#
    20

    View Slide

  22. Benefit of IP ANYCAST for DNS(4/4)
    • VIP&
    • IP"$
    !DNS Cache


    %#
    21

    View Slide

  23. VRRP HWLB
    Pros
    Cons
    • -*'
    • Act-Act&$ "


    • Act-Act&$+
    • L2#,
    • Act-Stb !
    %
    )(
    • -*'
    22

    View Slide

  24. DNS Server VM
    • DNS VM

    • FRR



    • DNS Server
    • Bind
    • NSD
    • Prometheus Exporters
    • node_exporter
    • bind_exporter
    • nsd_exporter
    • Health Check Daemon
    • Next Page=>
    23

    View Slide

  25. DNS Server VM Traffic
    • DNSVMInterface2
    • Mgmt Interface
    • Prometheus scrape
    • ssh


    • Service Interface
    • DNS Query
    • BGP
    24

    View Slide

  26. Monitoring
    25
    • Prometheus
    • Scrape exporters
    • DNSExporter
    +'
    • blackbox_exporter
    • !Exporter"
    #0
    • DNS QueryVIP
    )(%
    DNS$*

    • ! -,DNS

    /.&

    View Slide

  27. Health Check Daemon
    • Health Check DaemonDNSVIP -

    +)%

    • VMHV5. !

    BGPDown(Service Out%
    • DaemonDNS2&%
    • DNS -
    DNS Query,*
    • TTL: 1
    • Destination: VIP
    • 6
    • 31
    • Down#/
    "7$40'
    26

    View Slide

  28. VRRP HWLB
    Pros
    Cons
    • -*'
    • Act-Act&$ "


    • Act-Act&$+
    • L2#,
    • Act-Stb !
    %
    )(
    • -*'
    27

    View Slide

  29. Disadvantage of IP ANYCAST
    • DNS 9@/

    VM><&=?B
    • BGP Best Path %F:;
    • E#76* VM350%
    D0
    • NW(8"2VM.8
    )?B
    • -5%F'+ @/4C
    • BGP%F3A1$,
    E%F'+!
    VM1100%
    D0
    VM1 VM2 VM3
    28

    View Slide

  30. • DNSCN4IP ANYCASTS=J,/#&I<
    • !.*/$9B
    • CLOSBVMO5; -) $H76P
    • ,1$'1!D9B
    • VM8V1&06P
    • VIP+"*/$9B
    • IP ANYCASTCN4%,/#&
    • BGPBest Path8VFM
    NW:E2>
    (.1 1QT
    • IP ANYCASTK3@URLKAQT
    • Health Check Daemon?G
    Conclusion
    29

    View Slide

  31. Discussion
    • DCDNS
    • DNS

    • CLOS NW
    30

    View Slide

  32. Related Documents
    1. LINE
    !#$
    https://www.janog.gr.jp/meeting/janog43/program/line/
    2.
    "
    https://dnsops.jp/event/20210625/13-kosaka.pdf
    31

    View Slide