Container Communication on lattice

Transcript

1. Copyright@2015 NTT corp. All Rights Reserved. 第28回PaaS勉強会 @sinohara Container Communication on lattice
2. Copyright@2015 NTT corp. All Rights Reserved. 自己紹介 @sinohara NTTでNWの研究開発してます （本業はクラウドではありません） PaaS勉強中です！
3. Copyright@2015 NTT corp. All Rights Reserved. 今日のトピック Lattice Appのコンテナ間通信ってどうやるの？ どこまで出来るの？
4. Copyright@2015 NTT corp. All Rights Reserved. https://github.com/cloudfoundry-incubator/diego-design-notes ↓この黄枠がlattice http://www.slideshare.net/jacopen/diego-45603123 http://www.slideshare.net/jacopen/lattice-47937770 Lattice解説資料(過去のPaaS勉強会より)
5. Copyright@2015 NTT corp. All Rights Reserved. 動機、と、（ちょっと）宣伝 24/365無停止を実現するためのソフトウェア技術  Consistent-Hashベースのスケール性能に優れた負荷分散構成  障害時にもステート/キャッシュを失わないレプリケーション  大規模災害でもサービスを止めないDisaster Recovery 通信システム以外で面白い使い方、出来ない？？ NetroSphere構想：キャリアネットワークのあり方を変革する新R&Dコンセプトを策定 http://www.ntt.co.jp/news2015/1502/150219a.html
6. Copyright@2015 NTT corp. All Rights Reserved. 要するに アイデア募集中！  頻繁にDBに書きに行く必要がある企業情報システムとか  スピード！リアルタイム！なソーシャルゲームとか  課金処理とかやりたいeコマースとか ちょっと尖ったアプリケーション開発に如何ですか？ × =？ lattice
7. Copyright@2015 NTT corp. All Rights Reserved. Virtual NW VM boot image Location #1 Management boot image boot image Policy VM boot image VNF Manager (SBY) Application deployment VM VM DP MW DP MW LB (SBY) SIP SIP VIM Orchestrator (SBY) VM boot image Location #2 Management VM boot image VM VIM Orchestrator (ACT) VNF Manager (ACT) boot image DP MW SIP LB (ACT) *DP MW = Distributed processing middleware VM DP MW SIP data data data data R O R O O R O R O * = Data (original) R * = Data (replica) VNF Cloud Band Node (Alcatel-Lucent) Distributed processing platform including NTT R&D technology (FUJITSU) NFVI MAGONIA構成例（内部動作） Question：これ、latticeでも出来るの？ クラスタメンバ(VM)間で ステート／キャッシュの レプリケーションを実施 ETSI NFV ISG Proofs of Concept #12: Demonstration of multi-location, scalable, stateful Virtual Network Function http://nfvwiki.etsi.org/index.php?title=Demonstration_of_multi-location,_scalable,_stateful_Virtual_Network_Function
8. Copyright@2015 NTT corp. All Rights Reserved. 今日のトピック(再掲) Lattice Appのコンテナ間通信ってどうやるの？ どこまで出来るの？ ⇒latticeを色々触ってみて、ネットワーク周りの 仕組みを分かった範囲で解説します。
9. Copyright@2015 NTT corp. All Rights Reserved. LATTICEの検証内容 •Lattice Cluster Deployment • https://github.com/cloudfoundry-incubator/lattice#clustered-deployment • provider = AWS (東京リージョン) • lattice version = v0.2.5 + v0.4.0(new!) • terraform version = 0.5.3 • docker version = 1.7.1 •Lattice cellのNW構造チェック • ひたすらifconfig •php::http_get(another_container_IP)で コンテナ間通信が出来るか確認 • docker image ： kshino/lattice-php-cluster • phpinfo()でIPアドレスを確認 • $_ENV[“CF_INSTANCE_IP”] ： VMのIPアドレス • $_SERVER[“SERVER_ADDR”] ： コンテナのIPアドレス • HTTP_GETで?ip=xxx.xxx.xxx.xxxを渡すと、http_get(xxx.xxx.xxx.xxx)が動く
10. Copyright@2015 NTT corp. All Rights Reserved. LATTICE DEPLOYMENT(V0.2.5) # Installing docker sudo apt-get install -y lxc-docker # Installing terraform wget -q https://dl.bintray.com/mitchellh/terraform/terraform_0.5.3_linux_amd64.zip -P $HOME/pkg sudo unzip -qn $HOME/pkg/terraform_0.5.3_linux_amd64.zip -d /usr/bin/ # Installing lattice git clone https://github.com/cloudfoundry-incubator/lattice.git cd lattice git checkout v0.2.5 cd # Installing lattice CLI(ltc) sudo wget -q https://lattice.s3.amazonaws.com/releases/v0.2.5/linux-amd64/ltc -P /usr/bin sudo chmod +x /usr/bin/ltc # Deploying lattice cluster mkdir $HOME/tf cp $HOME/lattice/terraform/aws/example/lattice.aws.tf $HOME/tf/ cd $HOME/tf vi lattice.aws.tf terraform get -update terraform apply AWS access_key, secret_key, .pemの設定 Based on Ubuntu 14.04
11. Copyright@2015 NTT corp. All Rights Reserved. LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.21 eth0 10.0.1.193 54.92.69.178 54.92.69.172 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel V0.2.5 Cell-brain 54.92.69.42 Lattice構築時
12. Copyright@2015 NTT corp. All Rights Reserved. Container #1 Container #2 Container #3 Container #4 LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.21 w0m6uagc6vub-0 w0m6uagc6vub-1 10.254.0.1 w0m6uagc6vuc-0 w0m6uagc6vuc-1 10.254.0.5 wb-0m6uf716cb10 10.254.0.2 w0m6uf716lku-0 w0m6uf716lku-1 10.254.0.1 wb-0m6uf716cb20 10.254.0.6 w0m6uf716lkv-0 w0m6uf716lkv-1 10.254.0.5 Eth0 10.0.1.193 54.92.69.178 54.92.69.172 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel wb-0m6uagc6m510 10.254.0.2 wb-0m6uagc6m500 10.254.0.6 kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster V0.2.5 Cell-brain 54.92.69.42 ltc create
13. Copyright@2015 NTT corp. All Rights Reserved. Container #1 Container #2 Container #3 Container #4 LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.36 w0m7o5jmuq2r-0 w0m7o5jmuq2r-1 10.254.0.1 w0m7o5jmuq2s-0 w0m7o5jmuq2s-1 10.254.0.5 wb-0m7oavp54l10 10.254.0.2 w0m7oavp5co6-0 w0m7oavp5co6-1 10.254.0.1 wb-0m7oavp54l20 10.254.0.6 w0m7oavp5co7-0 w0m7oavp5co7-1 10.254.0.5 Eth0 10.0.1.87 52.69.211.8 54.65.44.124 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel wb-0m7o5jmuii20 10.254.0.2 wb-0m7o5jmuii30 10.254.0.6 kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster V0.4.0 Cell-brain 54.92.58.70 (v0.4.0も同じ)
14. Copyright@2015 NTT corp. All Rights Reserved. Container #1 Container #2 Container #3 Container #4 LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.21 w0m6uagc6vub-0 w0m6uagc6vub-1 10.254.0.1 w0m6uagc6vuc-0 w0m6uagc6vuc-1 10.254.0.5 wb-0m6uf716cb10 10.254.0.2 w0m6uf716lku-0 w0m6uf716lku-1 10.254.0.1 wb-0m6uf716cb20 10.254.0.6 w0m6uf716lkv-0 w0m6uf716lkv-1 10.254.0.5 Eth0 10.0.1.193 54.92.69.178 54.92.69.172 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel wb-0m6uagc6m510 10.254.0.2 wb-0m6uagc6m500 10.254.0.6 kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster V0.2.5 Cell-brain 54.92.69.42 HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5 (php) http_get(http://10.254.0.5/phpinfo.php) 疎通確認方法
15. Copyright@2015 NTT corp. All Rights Reserved. DEMO
16. Copyright@2015 NTT corp. All Rights Reserved. • 同一Cell(VM)内であれば、コンテナのIP addressを指定すればOK！ • App URLにも繋がるけど、コンテナの狙い撃ち は不可 • 他Appsのコンテナにも繋げてしまいます・・ • [v0.4.0] Cell-VMでapache2が動いてる模様 Container #1 Container #2 Container #3 Container #4 LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.21 w0m6uagc6vub-0 w0m6uagc6vub-1 10.254.0.1 w0m6uagc6vuc-0 w0m6uagc6vuc-1 10.254.0.5 wb-0m6uf716cb10 10.254.0.2 w0m6uf716lku-0 w0m6uf716lku-1 10.254.0.1 wb-0m6uf716cb20 10.254.0.6 w0m6uf716lkv-0 w0m6uf716lkv-1 10.254.0.5 Eth0 10.0.1.193 54.92.69.178 54.92.69.172 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel wb-0m6uagc6m510 10.254.0.2 wb-0m6uagc6m500 10.254.0.6 kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster V0.2.5 Cell-brain 54.92.69.42 HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5 (php) http_get(http://10.254.0.5/phpinfo.php) 疎通確認方法
17. Copyright@2015 NTT corp. All Rights Reserved. Cell(VM)間は？
18. Copyright@2015 NTT corp. All Rights Reserved. ・・次回作にご期待ください m(_ _)m
19. Copyright@2015 NTT corp. All Rights Reserved. いくつか候補が有ります FanNetworking
20. Copyright@2015 NTT corp. All Rights Reserved. FanNetworking •同一のClass B-NW内でOverlay + Tunneling •コンテナIPのネゴシエーションがVM内で完結 •/8なので1 Cellあたり最大255個 •コンテナのIPの共有は別途必要 •現在v0.3.0／Ubuntu image(GCE/AWS)有 172.16.0.0/16 172.16.0.0/16 172.16.3.5 172.16.23.37 10.3.5.x/8 10.23.37.x/8 VM#1 VM#2 Overlay-NW Class-B NW https://wiki.ubuntu.com/FanNetworking https://launchpad.net/ubuntu/+source/ubuntu-fan https://insights.ubuntu.com/2015/06/24/ubuntu-fan-images/
21. Copyright@2015 NTT corp. All Rights Reserved. •etcdでコンテナ用アドレス空間を共有管理 •VMごとにアドレス空間を切り出し •flanneldを通じVM間で通信 https://github.com/coreos/flannel/blob/master/packet-01.png https://github.com/coreos/flannel
22. Copyright@2015 NTT corp. All Rights Reserved. まとめ Lattice Appのコンテナ間通信の仕組みについて、 分かったことを紹介 Cell内のコンテナ間通信はPrivate IPで。 Cellをまたがるコンテナ間通信については To Be Continued…. アイデア募集中！