Container Communication on lattice

Transcript

1. Copyright@2015 NTT corp. All Rights Reserved.
   第28回PaaS勉強会
   @sinohara
   Container Communication on lattice
   

   View Slide

2. Copyright@2015 NTT corp. All Rights Reserved.
   自己紹介
   @sinohara
   NTTでNWの研究開発してます
   （本業はクラウドではありません）
   PaaS勉強中です！
   

   View Slide

3. Copyright@2015 NTT corp. All Rights Reserved.
   今日のトピック
   Lattice Appのコンテナ間通信ってどうやるの？
   どこまで出来るの？
   

   View Slide

4. Copyright@2015 NTT corp. All Rights Reserved.
   https://github.com/cloudfoundry-incubator/diego-design-notes
   ↓この黄枠がlattice
   http://www.slideshare.net/jacopen/diego-45603123
   http://www.slideshare.net/jacopen/lattice-47937770
   Lattice解説資料(過去のPaaS勉強会より)
   

   View Slide

5. Copyright@2015 NTT corp. All Rights Reserved.
   動機、と、（ちょっと）宣伝
   24/365無停止を実現するためのソフトウェア技術
    Consistent-Hashベースのスケール性能に優れた負荷分散構成
    障害時にもステート/キャッシュを失わないレプリケーション
    大規模災害でもサービスを止めないDisaster Recovery
   通信システム以外で面白い使い方、出来ない？？
   NetroSphere構想：キャリアネットワークのあり方を変革する新R&Dコンセプトを策定
   http://www.ntt.co.jp/news2015/1502/150219a.html
   

   View Slide

6. Copyright@2015 NTT corp. All Rights Reserved.
   要するに
   アイデア募集中！
    頻繁にDBに書きに行く必要がある企業情報システムとか
    スピード！リアルタイム！なソーシャルゲームとか
    課金処理とかやりたいeコマースとか
   ちょっと尖ったアプリケーション開発に如何ですか？
   × =？
   lattice
   

   View Slide

7. Copyright@2015 NTT corp. All Rights Reserved.
   Virtual NW
   VM
   boot
   image
   Location #1
   Management
   boot
   image
   boot
   image
   Policy
   VM
   boot
   image
   VNF
   Manager
   (SBY)
   Application
   deployment
   VM VM
   DP MW DP MW
   LB
   (SBY)
   SIP SIP
   VIM
   Orchestrator
   (SBY)
   VM
   boot
   image
   Location #2
   Management
   VM
   boot
   image
   VM
   VIM
   Orchestrator
   (ACT)
   VNF
   Manager
   (ACT)
   boot
   image
   DP MW
   SIP LB
   (ACT)
   *DP MW = Distributed processing middleware
   VM
   DP MW
   SIP
   data
   data
   data
   data R
   O R
   O
   O R
   O R
   O
   * = Data (original)
   R
   * = Data (replica)
   VNF
   Cloud Band Node
   (Alcatel-Lucent)
   Distributed processing platform
   including NTT R&D technology
   (FUJITSU)
   NFVI
   MAGONIA構成例（内部動作）
   Question：これ、latticeでも出来るの？
   クラスタメンバ(VM)間で
   ステート／キャッシュの
   レプリケーションを実施
   ETSI NFV ISG Proofs of Concept #12: Demonstration of multi-location, scalable, stateful Virtual Network Function
   http://nfvwiki.etsi.org/index.php?title=Demonstration_of_multi-location,_scalable,_stateful_Virtual_Network_Function
   

   View Slide

8. Copyright@2015 NTT corp. All Rights Reserved.
   今日のトピック(再掲)
   Lattice Appのコンテナ間通信ってどうやるの？
   どこまで出来るの？
   ⇒latticeを色々触ってみて、ネットワーク周りの
   仕組みを分かった範囲で解説します。
   

   View Slide

9. Copyright@2015 NTT corp. All Rights Reserved.
   LATTICEの検証内容
   •Lattice Cluster Deployment
   • https://github.com/cloudfoundry-incubator/lattice#clustered-deployment
   • provider = AWS (東京リージョン)
   • lattice version = v0.2.5 + v0.4.0(new!)
   • terraform version = 0.5.3
   • docker version = 1.7.1
   •Lattice cellのNW構造チェック
   • ひたすらifconfig
   •php::http_get(another_container_IP)で
   コンテナ間通信が出来るか確認
   • docker image ： kshino/lattice-php-cluster
   • phpinfo()でIPアドレスを確認
   • $_ENV[“CF_INSTANCE_IP”] ： VMのIPアドレス
   • $_SERVER[“SERVER_ADDR”] ： コンテナのIPアドレス
   • HTTP_GETで?ip=xxx.xxx.xxx.xxxを渡すと、http_get(xxx.xxx.xxx.xxx)が動く
   

   View Slide

10. Copyright@2015 NTT corp. All Rights Reserved.
    LATTICE DEPLOYMENT(V0.2.5)
    # Installing docker
    sudo apt-get install -y lxc-docker
    # Installing terraform
    wget -q https://dl.bintray.com/mitchellh/terraform/terraform_0.5.3_linux_amd64.zip -P $HOME/pkg
    sudo unzip -qn $HOME/pkg/terraform_0.5.3_linux_amd64.zip -d /usr/bin/
    # Installing lattice
    git clone https://github.com/cloudfoundry-incubator/lattice.git
    cd lattice
    git checkout v0.2.5
    cd
    # Installing lattice CLI(ltc)
    sudo wget -q https://lattice.s3.amazonaws.com/releases/v0.2.5/linux-amd64/ltc -P /usr/bin
    sudo chmod +x /usr/bin/ltc
    # Deploying lattice cluster
    mkdir $HOME/tf
    cp $HOME/lattice/terraform/aws/example/lattice.aws.tf $HOME/tf/
    cd $HOME/tf
    vi lattice.aws.tf
    terraform get -update
    terraform apply
    AWS access_key, secret_key, .pemの設定
    Based on Ubuntu 14.04
    

    View Slide

11. Copyright@2015 NTT corp. All Rights Reserved.
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.21
    eth0
    10.0.1.193
    54.92.69.178
    54.92.69.172
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    V0.2.5
    Cell-brain
    54.92.69.42
    Lattice構築時
    

    View Slide

12. Copyright@2015 NTT corp. All Rights Reserved.
    Container #1 Container #2 Container #3 Container #4
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.21
    w0m6uagc6vub-0
    w0m6uagc6vub-1
    10.254.0.1
    w0m6uagc6vuc-0
    w0m6uagc6vuc-1
    10.254.0.5
    wb-0m6uf716cb10
    10.254.0.2
    w0m6uf716lku-0
    w0m6uf716lku-1
    10.254.0.1
    wb-0m6uf716cb20
    10.254.0.6
    w0m6uf716lkv-0
    w0m6uf716lkv-1
    10.254.0.5
    Eth0
    10.0.1.193
    54.92.69.178
    54.92.69.172
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    wb-0m6uagc6m510
    10.254.0.2
    wb-0m6uagc6m500
    10.254.0.6
    kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
    V0.2.5
    Cell-brain
    54.92.69.42
    ltc create
    

    View Slide

13. Copyright@2015 NTT corp. All Rights Reserved.
    Container #1 Container #2 Container #3 Container #4
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.36
    w0m7o5jmuq2r-0
    w0m7o5jmuq2r-1
    10.254.0.1
    w0m7o5jmuq2s-0
    w0m7o5jmuq2s-1
    10.254.0.5
    wb-0m7oavp54l10
    10.254.0.2
    w0m7oavp5co6-0
    w0m7oavp5co6-1
    10.254.0.1
    wb-0m7oavp54l20
    10.254.0.6
    w0m7oavp5co7-0
    w0m7oavp5co7-1
    10.254.0.5
    Eth0
    10.0.1.87
    52.69.211.8
    54.65.44.124
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    wb-0m7o5jmuii20
    10.254.0.2
    wb-0m7o5jmuii30
    10.254.0.6
    kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
    V0.4.0
    Cell-brain
    54.92.58.70
    (v0.4.0も同じ)
    

    View Slide

14. Copyright@2015 NTT corp. All Rights Reserved.
    Container #1 Container #2 Container #3 Container #4
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.21
    w0m6uagc6vub-0
    w0m6uagc6vub-1
    10.254.0.1
    w0m6uagc6vuc-0
    w0m6uagc6vuc-1
    10.254.0.5
    wb-0m6uf716cb10
    10.254.0.2
    w0m6uf716lku-0
    w0m6uf716lku-1
    10.254.0.1
    wb-0m6uf716cb20
    10.254.0.6
    w0m6uf716lkv-0
    w0m6uf716lkv-1
    10.254.0.5
    Eth0
    10.0.1.193
    54.92.69.178
    54.92.69.172
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    wb-0m6uagc6m510
    10.254.0.2
    wb-0m6uagc6m500
    10.254.0.6
    kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
    V0.2.5
    Cell-brain
    54.92.69.42
    HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5
    (php) http_get(http://10.254.0.5/phpinfo.php)
    疎通確認方法
    

    View Slide

15. Copyright@2015 NTT corp. All Rights Reserved.
    DEMO
    

    View Slide

16. Copyright@2015 NTT corp. All Rights Reserved.
    • 同一Cell(VM)内であれば、コンテナのIP
    addressを指定すればOK！
    • App URLにも繋がるけど、コンテナの狙い撃ち
    は不可
    • 他Appsのコンテナにも繋げてしまいます・・
    • [v0.4.0] Cell-VMでapache2が動いてる模様
    Container #1 Container #2 Container #3 Container #4
    LATTICE ARCHITECTURE
    Cell-1
    Cell-0
    eth0
    10.0.1.21
    w0m6uagc6vub-0
    w0m6uagc6vub-1
    10.254.0.1
    w0m6uagc6vuc-0
    w0m6uagc6vuc-1
    10.254.0.5
    wb-0m6uf716cb10
    10.254.0.2
    w0m6uf716lku-0
    w0m6uf716lku-1
    10.254.0.1
    wb-0m6uf716cb20
    10.254.0.6
    w0m6uf716lkv-0
    w0m6uf716lkv-1
    10.254.0.5
    Eth0
    10.0.1.193
    54.92.69.178
    54.92.69.172
    Region = ap-northeast-1
    Virtual Machine Container Network Interface
    description Network Tunnel
    wb-0m6uagc6m510
    10.254.0.2
    wb-0m6uagc6m500
    10.254.0.6
    kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster
    V0.2.5
    Cell-brain
    54.92.69.42
    HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5
    (php) http_get(http://10.254.0.5/phpinfo.php)
    疎通確認方法
    

    View Slide

17. Copyright@2015 NTT corp. All Rights Reserved.
    Cell(VM)間は？
    

    View Slide

18. Copyright@2015 NTT corp. All Rights Reserved.
    ・・次回作にご期待ください
    m(_ _)m
    

    View Slide

19. Copyright@2015 NTT corp. All Rights Reserved.
    いくつか候補が有ります
    FanNetworking
    

    View Slide

20. Copyright@2015 NTT corp. All Rights Reserved.
    FanNetworking
    •同一のClass B-NW内でOverlay + Tunneling
    •コンテナIPのネゴシエーションがVM内で完結
    •/8なので1 Cellあたり最大255個
    •コンテナのIPの共有は別途必要
    •現在v0.3.0／Ubuntu image(GCE/AWS)有
    172.16.0.0/16
    172.16.0.0/16
    172.16.3.5
    172.16.23.37
    10.3.5.x/8
    10.23.37.x/8
    VM#1
    VM#2
    Overlay-NW
    Class-B NW
    https://wiki.ubuntu.com/FanNetworking
    https://launchpad.net/ubuntu/+source/ubuntu-fan
    https://insights.ubuntu.com/2015/06/24/ubuntu-fan-images/
    

    View Slide

21. Copyright@2015 NTT corp. All Rights Reserved.
    •etcdでコンテナ用アドレス空間を共有管理
    •VMごとにアドレス空間を切り出し
    •flanneldを通じVM間で通信
    https://github.com/coreos/flannel/blob/master/packet-01.png
    https://github.com/coreos/flannel
    

    View Slide

22. Copyright@2015 NTT corp. All Rights Reserved.
    まとめ
    Lattice Appのコンテナ間通信の仕組みについて、
    分かったことを紹介
    Cell内のコンテナ間通信はPrivate IPで。
    Cellをまたがるコンテナ間通信については
    To Be Continued….
    アイデア募集中！
    

    View Slide