Share
Embed
Reasonable Auto Scaling CI with AWS Spot Instance
Gitlab CIとAWS Spot Instanceを使ってオートスケールなCIを実現しよう
YutaTetsuka
October 13, 2017
Transcript
43 slides
-
Reasonable Auto Scaling CI with AWS Spot Instance GitLab Meetup
Tokyo #4 Yuta Tetsuka 2017-10-12 Last update: 2017-10-13 -
Yuta Tetsuka - JustSystems corporation joined in 2012 - Team
smilezemi (smile-zemi.jp) - Android engineer - Qiita: @tetsukay 1 About me 2 -
Before CI Environments - Gitlab CI - AWS on-demand Instance
- m4.xlarge - 4core 16GB $0.258 / h 6 -
Problems PS: 2017-10-13 - CI is slow - Avg 26min
- Can't concurrent build - 2 builds… 55min - 4 builds… 180min (timeout) - High Cost - 72 USD / month - 280 hours / month - (It was not as good as I thought it was.) 7 -
Auto scaling GitLab CI - Use AWS Spot Instance -
by docker-machine - Create Instance per CI 11 -
Spot Instance merit - Low cost - about 15%〜20% of
on-demand price - e.g. c4.4xlarge - On-demand: $1.008 / h - Spot: $0.182 / h - per-second billing model 14 -
Appropriate scene - Batch Job - Data Analyze - Temporary
Tasks - etc... 15 -
Spot Instance demerit 2 - Shut down if it soars
above the bid price - Time required to start instance - about 1〜2 min 17 -
After CI Environments - Gitlab CI Auto Scaling - AWS
Spot Instance - c4.4xlarge - 16core 30GB $0.182 / h (minimum) - Bid $0.5 - CI Time 25min -> 14min - Unlimited Concurrent CI!! - PS 2017-10-13: (until aws limits, normally 5 concurrents) - PS 2017-10-13: My case is 20 18 -
Runner - Instance Type - t2.micro (1core 1GB) - 1.7
Yen / h - Executor - docker+machine 20 -
Install docker-machine # Mac $ curl -L https://github.com/docker/machine/releases/download/v0.12.2/docker-machine-`uname -s`-`uname -m`
>/usr/local/bin/docker-machine && \ chmod +x /usr/local/bin/docker-machine # Windows with Git bash $ if [[ ! -d "$HOME/bin" ]]; then mkdir -p "$HOME/bin"; fi && \ curl -L https://github.com/docker/machine/releases/download/v0.12.2/docker-machine-Windows-x86 _64.exe > "$HOME/bin/docker-machine.exe" && \ chmod +x "$HOME/bin/docker-machine.exe" # Linux $ curl -L https://github.com/docker/machine/releases/download/v0.12.2/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/local/bin/docker-machine 21 -
Test docker-machine docker-machine create \ --driver amazonec2 \ --amazonec2-region ap-northeast-1
\ --amazonec2-request-spot-instance \ --amazonec2-spot-price 0.05 \ --amazonec2-ami=ami-2a69be4c \ --amazonec2-instance-type m4.large \ --amazonec2-private-address-only \ --amazonec2-vpc-id vpc-xxxxx \ --amazonec2-subnet-id subnet-xxxxx \ test-instance-name 22 Result -
Register Runner gitlab-runner register --non-interactive \ --url https://gitlab.example.com/ci \ --registration-token
[TOKEN] \ --executor "docker+machine" \ --name "gitlab-ci-auto-scaling" \ --docker-image "ubuntu" \ --machine-machine-driver "amazonec2" \ --machine-machine-name "gitlab-ci-%s" \ --machine-machine-options "amazonec2-ami=ami-2a69be4c" \ --machine-machine-options "amazonec2-region=ap-northeast-1" \ --machine-machine-options "amazonec2-request-spot-instance" \ --machine-machine-options "amazonec2-use-ebs-optimized-instance=true" \ --machine-machine-options "amazonec2-instance-type=c4.4xlarge" \ --machine-machine-options "amazonec2-spot-price=0.20" \ --machine-machine-options "amazonec2-vpc-id=[IF NEED]" \ --machine-machine-options "amazonec2-subnet-id=[IF NEED]" \ --machine-machine-options "amazonec2-ssh-keypath-id=[IF NEED]" \ --machine-machine-options "amazonec2-keypair-name=[IF NEED]" 23 -
Before - Runner Instance… - m4.xlarge $0.258 - 280 hours
/ month - 280 * 0.258 = $72.24 25 -
After - Runner Instance… - t2.micro $0.0152 - 280 hours
/ month - 280 * $0.0152 = $4.256 - CI Instances - c4.4xlarge $0.1825 (minimum) - (about) 200 CI / month - 14 min * 200 = 2800 min / month - 46.67 hours * (about) $0.2 = $9.33 26 -
Conclusion - c4.4xlarge is powerful ! - We could upgrade
CI machine specs - Unlimited concurrent build ! - Very low cost ! 27 -
ライセンスの数え方 - Admin Area -> Users のアクティブユーザ が必要ライセンス数 - Jenkinsビルド用等に用意されたBotユー
ザもライセンスが必要 - StatusがBlocked なユーザは不要 30 -
ライセンスをオーバーすると? - そのまま利用可能 - ユーザを追加できなくなるということはない - True-Up model(後述)に基づき精算 - (追記:
2017-10-13) ライセンス追加発注も可とのこと 32 -
1人が複数のサーバに属している - Gitlab1, Gitlab2 があり,Aという人が二つ に属している場合 - ライセンス数は1でOK - 「実際の利用者」をカウントすれば良い
33 弊社は用途別にたくさん GitLabが立っているので助かります … -
True-Up Modelとは? - 公式の説明が少しわかりにくい 35 The True-Up model seems complicated,
can you illustrate? If you have 100 active users today, you should purchase a 100 user subscription. Suppose that when you renew next year you have 300 active users (200 extra users). When you renew you pay for a 300 user subscription and you also pay the full annual fee for the 200 users that you added during the year. -
追加発注モデル PS: 2017-10-13 - 人数が増えたタイミングで,その分を追加購入する形 - 月割となるので,True-Up Modelよりオトク - ライセンスが増えた段階で購入・新ライセンスファ
イルの適用が必要になり,True-Up Modelより手 間 41 -
弊社では - GitLab Users APIにAdminでアクセス - 複数のGitLabからユーザ取得 - ユーザをマージししてユニークユーザ数に 加工
- Cronで月1計測 - PS: 2017-10-13 (たぶん)True-Up Model 42