Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Container Communication on lattice

Kenta Shinohara
September 11, 2015

Container Communication on lattice

Kenta Shinohara

September 11, 2015
Tweet

More Decks by Kenta Shinohara

Other Decks in Research

Transcript

  1. Copyright@2015 NTT corp. All Rights Reserved.
    第28回PaaS勉強会
    @sinohara
    Container Communication on lattice

    View Slide

  2. Copyright@2015 NTT corp. All Rights Reserved.
    自己紹介
    @sinohara
    NTTでNWの研究開発してます
    (本業はクラウドではありません)
    PaaS勉強中です!

    View Slide

  3. Copyright@2015 NTT corp. All Rights Reserved.
    今日のトピック
    Lattice Appのコンテナ間通信ってどうやるの?
    どこまで出来るの?

    View Slide

  4. Copyright@2015 NTT corp. All Rights Reserved.
    https://github.com/cloudfoundry-incubator/diego-design-notes
    ↓この黄枠がlattice
    http://www.slideshare.net/jacopen/diego-45603123
    http://www.slideshare.net/jacopen/lattice-47937770
    Lattice解説資料(過去のPaaS勉強会より)

    View Slide

  5. Copyright@2015 NTT corp. All Rights Reserved.
    動機、と、(ちょっと)宣伝
    24/365無停止を実現するためのソフトウェア技術
     Consistent-Hashベースのスケール性能に優れた負荷分散構成
     障害時にもステート/キャッシュを失わないレプリケーション
     大規模災害でもサービスを止めないDisaster Recovery
    通信システム以外で面白い使い方、出来ない??
    NetroSphere構想:キャリアネットワークのあり方を変革する新R&Dコンセプトを策定
    http://www.ntt.co.jp/news2015/1502/150219a.html

    View Slide

  6. Copyright@2015 NTT corp. All Rights Reserved.
    要するに
    アイデア募集中!
     頻繁にDBに書きに行く必要がある企業情報システムとか
     スピード!リアルタイム!なソーシャルゲームとか
     課金処理とかやりたいeコマースとか
    ちょっと尖ったアプリケーション開発に如何ですか?
    × =?
    lattice

    View Slide

  7. Copyright@2015 NTT corp. All Rights Reserved.
    Virtual NW
    VM
    boot
    image
    Location #1
    Management
    boot
    image
    boot
    image
    Policy
    VM
    boot
    image
    VNF
    Manager
    (SBY)
    Application
    deployment
    VM VM
    DP MW DP MW
    LB
    (SBY)
    SIP SIP
    VIM
    Orchestrator
    (SBY)
    VM
    boot
    image
    Location #2
    Management
    VM
    boot
    image
    VM
    VIM
    Orchestrator
    (ACT)
    VNF
    Manager
    (ACT)
    boot
    image
    DP MW
    SIP LB
    (ACT)
    *DP MW = Distributed processing middleware
    VM
    DP MW
    SIP
    data
    data
    data
    data R
    O R
    O
    O R
    O R
    O
    * = Data (original)
    R
    * = Data (replica)
    VNF
    Cloud Band Node
    (Alcatel-Lucent)
    Distributed processing platform
    including NTT R&D technology
    (FUJITSU)
    NFVI
    MAGONIA構成例(内部動作)
    Question:これ、latticeでも出来るの?
    クラスタメンバ(VM)間で
    ステート/キャッシュの
    レプリケーションを実施
    ETSI NFV ISG Proofs of Concept #12: Demonstration of multi-location, scalable, stateful Virtual Network Function
    http://nfvwiki.etsi.org/index.php?title=Demonstration_of_multi-location,_scalable,_stateful_Virtual_Network_Function

    View Slide

  8. Copyright@2015 NTT corp. All Rights Reserved.
    今日のトピック(再掲)
    Lattice Appのコンテナ間通信ってどうやるの?
    どこまで出来るの?
    ⇒latticeを色々触ってみて、ネットワーク周りの
    仕組みを分かった範囲で解説します。

    View Slide

  9. Copyright@2015 NTT corp. All Rights Reserved.
    LATTICEの検証内容
    •Lattice Cluster Deployment
    • https://github.com/cloudfoundry-incubator/lattice#clustered-deployment
    • provider = AWS (東京リージョン)
    • lattice version = v0.2.5 + v0.4.0(new!)
    • terraform version = 0.5.3
    • docker version = 1.7.1
    •Lattice cellのNW構造チェック
    • ひたすらifconfig
    •php::http_get(another_container_IP)で
    コンテナ間通信が出来るか確認
    • docker image : kshino/lattice-php-cluster
    • phpinfo()でIPアドレスを確認
    • $_ENV[“CF_INSTANCE_IP”] : VMのIPアドレス
    • $_SERVER[“SERVER_ADDR”] : コンテナのIPアドレス
    • HTTP_GETで?ip=xxx.xxx.xxx.xxxを渡すと、http_get(xxx.xxx.xxx.xxx)が動く

    View Slide

  10. Copyright@2015 NTT corp. All Rights Reserved.
    LATTICE DEPLOYMENT(V0.2.5)
    # Installing docker
    sudo apt-get install -y lxc-docker
    # Installing terraform
    wget -q https://dl.bintray.com/mitchellh/terraform/terraform_0.5.3_linux_amd64.zip -P $HOME/pkg
    sudo unzip -qn $HOME/pkg/terraform_0.5.3_linux_amd64.zip -d /usr/bin/
    # Installing lattice
    git clone https://github.com/cloudfoundry-incubator/lattice.git
    cd lattice
    git checkout v0.2.5
    cd
    # Installing lattice CLI(ltc)
    sudo wget -q https://lattice.s3.amazonaws.com/releases/v0.2.5/linux-amd64/ltc -P /usr/bin
    sudo chmod +x /usr/bin/ltc
    # Deploying lattice cluster
    mkdir $HOME/tf
    cp $HOME/lattice/terraform/aws/example/lattice.aws.tf $HOME/tf/
    cd $HOME/tf
    vi lattice.aws.tf
    terraform get -update
    terraform apply
    AWS access_key, secret_key, .pemの設定
    Based on Ubuntu 14.04

    View Slide

  11. Copyright@2015 NTT corp. All Rights Reserved.
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.21
    eth0
    10.0.1.193
    54.92.69.178
    54.92.69.172
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    V0.2.5
    Cell-brain
    54.92.69.42
    Lattice構築時

    View Slide

  12. Copyright@2015 NTT corp. All Rights Reserved.
    Container #1 Container #2 Container #3 Container #4
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.21
    w0m6uagc6vub-0
    w0m6uagc6vub-1
    10.254.0.1
    w0m6uagc6vuc-0
    w0m6uagc6vuc-1
    10.254.0.5
    wb-0m6uf716cb10
    10.254.0.2
    w0m6uf716lku-0
    w0m6uf716lku-1
    10.254.0.1
    wb-0m6uf716cb20
    10.254.0.6
    w0m6uf716lkv-0
    w0m6uf716lkv-1
    10.254.0.5
    Eth0
    10.0.1.193
    54.92.69.178
    54.92.69.172
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    wb-0m6uagc6m510
    10.254.0.2
    wb-0m6uagc6m500
    10.254.0.6
    kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
    V0.2.5
    Cell-brain
    54.92.69.42
    ltc create

    View Slide

  13. Copyright@2015 NTT corp. All Rights Reserved.
    Container #1 Container #2 Container #3 Container #4
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.36
    w0m7o5jmuq2r-0
    w0m7o5jmuq2r-1
    10.254.0.1
    w0m7o5jmuq2s-0
    w0m7o5jmuq2s-1
    10.254.0.5
    wb-0m7oavp54l10
    10.254.0.2
    w0m7oavp5co6-0
    w0m7oavp5co6-1
    10.254.0.1
    wb-0m7oavp54l20
    10.254.0.6
    w0m7oavp5co7-0
    w0m7oavp5co7-1
    10.254.0.5
    Eth0
    10.0.1.87
    52.69.211.8
    54.65.44.124
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    wb-0m7o5jmuii20
    10.254.0.2
    wb-0m7o5jmuii30
    10.254.0.6
    kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
    V0.4.0
    Cell-brain
    54.92.58.70
    (v0.4.0も同じ)

    View Slide

  14. Copyright@2015 NTT corp. All Rights Reserved.
    Container #1 Container #2 Container #3 Container #4
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.21
    w0m6uagc6vub-0
    w0m6uagc6vub-1
    10.254.0.1
    w0m6uagc6vuc-0
    w0m6uagc6vuc-1
    10.254.0.5
    wb-0m6uf716cb10
    10.254.0.2
    w0m6uf716lku-0
    w0m6uf716lku-1
    10.254.0.1
    wb-0m6uf716cb20
    10.254.0.6
    w0m6uf716lkv-0
    w0m6uf716lkv-1
    10.254.0.5
    Eth0
    10.0.1.193
    54.92.69.178
    54.92.69.172
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    wb-0m6uagc6m510
    10.254.0.2
    wb-0m6uagc6m500
    10.254.0.6
    kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
    V0.2.5
    Cell-brain
    54.92.69.42
    HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5
    (php) http_get(http://10.254.0.5/phpinfo.php)
    疎通確認方法

    View Slide

  15. Copyright@2015 NTT corp. All Rights Reserved.
    DEMO

    View Slide

  16. Copyright@2015 NTT corp. All Rights Reserved.
    • 同一Cell(VM)内であれば、コンテナのIP
    addressを指定すればOK!
    • App URLにも繋がるけど、コンテナの狙い撃ち
    は不可
    • 他Appsのコンテナにも繋げてしまいます・・
    • [v0.4.0] Cell-VMでapache2が動いてる模様
    Container #1 Container #2 Container #3 Container #4
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.21
    w0m6uagc6vub-0
    w0m6uagc6vub-1
    10.254.0.1
    w0m6uagc6vuc-0
    w0m6uagc6vuc-1
    10.254.0.5
    wb-0m6uf716cb10
    10.254.0.2
    w0m6uf716lku-0
    w0m6uf716lku-1
    10.254.0.1
    wb-0m6uf716cb20
    10.254.0.6
    w0m6uf716lkv-0
    w0m6uf716lkv-1
    10.254.0.5
    Eth0
    10.0.1.193
    54.92.69.178
    54.92.69.172
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    wb-0m6uagc6m510
    10.254.0.2
    wb-0m6uagc6m500
    10.254.0.6
    kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
    V0.2.5
    Cell-brain
    54.92.69.42
    HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5
    (php) http_get(http://10.254.0.5/phpinfo.php)
    疎通確認方法

    View Slide

  17. Copyright@2015 NTT corp. All Rights Reserved.
    Cell(VM)間は?

    View Slide

  18. Copyright@2015 NTT corp. All Rights Reserved.
    ・・次回作にご期待ください
    m(_ _)m

    View Slide

  19. Copyright@2015 NTT corp. All Rights Reserved.
    いくつか候補が有ります
    FanNetworking

    View Slide

  20. Copyright@2015 NTT corp. All Rights Reserved.
    FanNetworking
    •同一のClass B-NW内でOverlay + Tunneling
    •コンテナIPのネゴシエーションがVM内で完結
    •/8なので1 Cellあたり最大255個
    •コンテナのIPの共有は別途必要
    •現在v0.3.0/Ubuntu image(GCE/AWS)有
    172.16.0.0/16
    172.16.0.0/16
    172.16.3.5
    172.16.23.37
    10.3.5.x/8
    10.23.37.x/8
    VM#1
    VM#2
    Overlay-NW
    Class-B NW
    https://wiki.ubuntu.com/FanNetworking
    https://launchpad.net/ubuntu/+source/ubuntu-fan
    https://insights.ubuntu.com/2015/06/24/ubuntu-fan-images/

    View Slide

  21. Copyright@2015 NTT corp. All Rights Reserved.
    •etcdでコンテナ用アドレス空間を共有管理
    •VMごとにアドレス空間を切り出し
    •flanneldを通じVM間で通信
    https://github.com/coreos/flannel/blob/master/packet-01.png
    https://github.com/coreos/flannel

    View Slide

  22. Copyright@2015 NTT corp. All Rights Reserved.
    まとめ
    Lattice Appのコンテナ間通信の仕組みについて、
    分かったことを紹介
    Cell内のコンテナ間通信はPrivate IPで。
    Cellをまたがるコンテナ間通信については
    To Be Continued….
    アイデア募集中!

    View Slide