Container Communication on lattice

by Kenta Shinohara

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Copyright@2015 NTT corp. All Rights Reserved. https://github.com/cloudfoundry-incubator/diego-design-notes ↓この黄枠がlattice http://www.slideshare.net/jacopen/diego-45603123 http://www.slideshare.net/jacopen/lattice-47937770 Lattice解説資料(過去のPaaS勉強会より)

Slide 5

Slide 5 text

Copyright@2015 NTT corp. All Rights Reserved. 動機、と、（ちょっと）宣伝 24/365無停止を実現するためのソフトウェア技術  Consistent-Hashベースのスケール性能に優れた負荷分散構成  障害時にもステート/キャッシュを失わないレプリケーション  大規模災害でもサービスを止めないDisaster Recovery 通信システム以外で面白い使い方、出来ない？？ NetroSphere構想：キャリアネットワークのあり方を変革する新R&Dコンセプトを策定 http://www.ntt.co.jp/news2015/1502/150219a.html

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Copyright@2015 NTT corp. All Rights Reserved. Virtual NW VM boot image Location #1 Management boot image boot image Policy VM boot image VNF Manager (SBY) Application deployment VM VM DP MW DP MW LB (SBY) SIP SIP VIM Orchestrator (SBY) VM boot image Location #2 Management VM boot image VM VIM Orchestrator (ACT) VNF Manager (ACT) boot image DP MW SIP LB (ACT) *DP MW = Distributed processing middleware VM DP MW SIP data data data data R O R O O R O R O * = Data (original) R * = Data (replica) VNF Cloud Band Node (Alcatel-Lucent) Distributed processing platform including NTT R&D technology (FUJITSU) NFVI MAGONIA構成例（内部動作） Question：これ、latticeでも出来るの？クラスタメンバ(VM)間でステート／キャッシュのレプリケーションを実施 ETSI NFV ISG Proofs of Concept #12: Demonstration of multi-location, scalable, stateful Virtual Network Function http://nfvwiki.etsi.org/index.php?title=Demonstration_of_multi-location,_scalable,_stateful_Virtual_Network_Function

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Copyright@2015 NTT corp. All Rights Reserved. LATTICEの検証内容 •Lattice Cluster Deployment • https://github.com/cloudfoundry-incubator/lattice#clustered-deployment • provider = AWS (東京リージョン) • lattice version = v0.2.5 + v0.4.0(new!) • terraform version = 0.5.3 • docker version = 1.7.1 •Lattice cellのNW構造チェック • ひたすらifconfig •php::http_get(another_container_IP)でコンテナ間通信が出来るか確認 • docker image ： kshino/lattice-php-cluster • phpinfo()でIPアドレスを確認 • $_ENV[“CF_INSTANCE_IP”] ： VMのIPアドレス • $_SERVER[“SERVER_ADDR”] ：コンテナのIPアドレス • HTTP_GETで?ip=xxx.xxx.xxx.xxxを渡すと、http_get(xxx.xxx.xxx.xxx)が動く

Slide 10

Slide 10 text

Copyright@2015 NTT corp. All Rights Reserved. LATTICE DEPLOYMENT(V0.2.5) # Installing docker sudo apt-get install -y lxc-docker # Installing terraform wget -q https://dl.bintray.com/mitchellh/terraform/terraform_0.5.3_linux_amd64.zip -P $HOME/pkg sudo unzip -qn $HOME/pkg/terraform_0.5.3_linux_amd64.zip -d /usr/bin/ # Installing lattice git clone https://github.com/cloudfoundry-incubator/lattice.git cd lattice git checkout v0.2.5 cd # Installing lattice CLI(ltc) sudo wget -q https://lattice.s3.amazonaws.com/releases/v0.2.5/linux-amd64/ltc -P /usr/bin sudo chmod +x /usr/bin/ltc # Deploying lattice cluster mkdir $HOME/tf cp $HOME/lattice/terraform/aws/example/lattice.aws.tf $HOME/tf/ cd $HOME/tf vi lattice.aws.tf terraform get -update terraform apply AWS access_key, secret_key, .pemの設定 Based on Ubuntu 14.04

Slide 11

Slide 11 text

Copyright@2015 NTT corp. All Rights Reserved. LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.21 eth0 10.0.1.193 54.92.69.178 54.92.69.172 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel V0.2.5 Cell-brain 54.92.69.42 Lattice構築時

Slide 12

Slide 12 text

Copyright@2015 NTT corp. All Rights Reserved. Container #1 Container #2 Container #3 Container #4 LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.21 w0m6uagc6vub-0 w0m6uagc6vub-1 10.254.0.1 w0m6uagc6vuc-0 w0m6uagc6vuc-1 10.254.0.5 wb-0m6uf716cb10 10.254.0.2 w0m6uf716lku-0 w0m6uf716lku-1 10.254.0.1 wb-0m6uf716cb20 10.254.0.6 w0m6uf716lkv-0 w0m6uf716lkv-1 10.254.0.5 Eth0 10.0.1.193 54.92.69.178 54.92.69.172 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel wb-0m6uagc6m510 10.254.0.2 wb-0m6uagc6m500 10.254.0.6 kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster V0.2.5 Cell-brain 54.92.69.42 ltc create

Slide 13

Slide 13 text

Copyright@2015 NTT corp. All Rights Reserved. Container #1 Container #2 Container #3 Container #4 LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.36 w0m7o5jmuq2r-0 w0m7o5jmuq2r-1 10.254.0.1 w0m7o5jmuq2s-0 w0m7o5jmuq2s-1 10.254.0.5 wb-0m7oavp54l10 10.254.0.2 w0m7oavp5co6-0 w0m7oavp5co6-1 10.254.0.1 wb-0m7oavp54l20 10.254.0.6 w0m7oavp5co7-0 w0m7oavp5co7-1 10.254.0.5 Eth0 10.0.1.87 52.69.211.8 54.65.44.124 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel wb-0m7o5jmuii20 10.254.0.2 wb-0m7o5jmuii30 10.254.0.6 kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster V0.4.0 Cell-brain 54.92.58.70 (v0.4.0も同じ)

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Copyright@2015 NTT corp. All Rights Reserved. • 同一Cell(VM)内であれば、コンテナのIP addressを指定すればOK！ • App URLにも繋がるけど、コンテナの狙い撃ちは不可 • 他Appsのコンテナにも繋げてしまいます・・ • [v0.4.0] Cell-VMでapache2が動いてる模様 Container #1 Container #2 Container #3 Container #4 LATTICE ARCHITECTURE Cell-1 Cell-0 eth0 10.0.1.21 w0m6uagc6vub-0 w0m6uagc6vub-1 10.254.0.1 w0m6uagc6vuc-0 w0m6uagc6vuc-1 10.254.0.5 wb-0m6uf716cb10 10.254.0.2 w0m6uf716lku-0 w0m6uf716lku-1 10.254.0.1 wb-0m6uf716cb20 10.254.0.6 w0m6uf716lkv-0 w0m6uf716lkv-1 10.254.0.5 Eth0 10.0.1.193 54.92.69.178 54.92.69.172 Region = ap-northeast-1 Virtual Machine Container Network Interface description Network Tunnel wb-0m6uagc6m510 10.254.0.2 wb-0m6uagc6m500 10.254.0.6 kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster V0.2.5 Cell-brain 54.92.69.42 HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5 (php) http_get(http://10.254.0.5/phpinfo.php) 疎通確認方法

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Copyright@2015 NTT corp. All Rights Reserved. FanNetworking •同一のClass B-NW内でOverlay + Tunneling •コンテナIPのネゴシエーションがVM内で完結 •/8なので1 Cellあたり最大255個 •コンテナのIPの共有は別途必要 •現在v0.3.0／Ubuntu image(GCE/AWS)有 172.16.0.0/16 172.16.0.0/16 172.16.3.5 172.16.23.37 10.3.5.x/8 10.23.37.x/8 VM#1 VM#2 Overlay-NW Class-B NW https://wiki.ubuntu.com/FanNetworking https://launchpad.net/ubuntu/+source/ubuntu-fan https://insights.ubuntu.com/2015/06/24/ubuntu-fan-images/