FASTER! FASTER!! FASTER!!! -accelerate pipeline-

20190227 - @Quoine Concourse CI/CD Meetup Tokyo #8 !

Transcript

1. FASTER! FASTER!! FASTER!!! and more ... 2019.02.27 @ Concourse Meetup
2. ߕ൹ ྒྷഅ KAI RYOMA @LEGNOH Platform Engineer
3. LAST YEAR'S ADVENT CALENDAR ➤ 2019೥ͷCI/CDΛ઎͏ - Qiita ➤ https://qiita.com/legnoh/items/159cc15540ae7957b2de ➤ Internal Concourse owner
4. TODAY'S THEME
5. ACCELERATE CI PIPELINE TODAY 's THEME
6. FOR EXAMPLE... ➤ npm test & audit & upload coverage file jobs: - name: gotcha plan: - get: repo - task: unit-test-and-audit config: platform: linux image_resource: type: docker-image source: repository: node tag: 10 inputs: - name: repo outputs: - name: out run: dir: repo path: /bin/sh args: - -c - | npm install npm test npm run-script make-coverage npm audit cd .. mv repo/coverage/coverage-final.json out/`basename \`pwd\``.json - put: minio params: file: out/*.json
7. SIMPLE, BUT... ➤ 2m... (in my company's environment) jobs: - name: gotcha plan: - get: repo - task: unit-test-and-audit config: platform: linux image_resource: type: docker-image source: repository: node tag: 10 inputs: - name: repo outputs: - name: out run: dir: repo path: /bin/sh args: - -c - | npm install npm test npm run-script make-coverage npm audit cd .. mv repo/coverage/coverage-final.json out/`basename \`pwd\``.json - put: minio params: file: out/*.json
8. FASTER! #1
9. #1: AGGREGATE ➤ task:unit-test-and-audit executes it in circuit jobs: - name: gotcha plan: - get: repo - task: unit-test-and-audit config: platform: linux image_resource: type: docker-image source: repository: node tag: 10 inputs: - name: repo outputs: - name: out run: dir: repo path: /bin/sh args: - -c - | npm install npm test npm run-script make-coverage npm audit cd .. mv repo/coverage/coverage-final.json out/`basename \`pwd\``.json - put: minio params: file: out/*.json
10. DIVIDE, DIVIDE, DIVIDE.... ➤ example plan: - get: repo - task: prepare ... run: dir: repo path: /bin/sh args: - -c - | npm install - aggregate: - task: unit-test ... run: dir: repo path: /bin/sh args: - -c - | npm test - task: audit ... run: dir: repo path: /bin/sh args: - -c - | npm audit - do: - task: make-coverage ... run: dir: repo path: /bin/sh args: - -c - | npm run-script make-coverage npm audit cd .. mv repo/coverage/coverage-final.json out/`basename \`pwd\``.json - put: minio params: file: out/*.json Parallel execution! circuit
11. USE AGGREGATE IN 1ST ACTION! ➤ "aggregate" block is the best solution in acceleration! ➤ quite and quick effective ➤ customizable on your pipeline ⚒
12. FASTER!!! #2
13. #4: USE CACHES: BLOCK ➤ npm install executes over the internet... jobs: - name: gotcha plan: - get: repo - task: unit-test-and-audit config: platform: linux image_resource: type: docker-image source: repository: node tag: 10 inputs: - name: repo outputs: - name: out run: dir: repo path: /bin/sh args: - -c - | npm install npm test npm run-script make-coverage npm audit cd .. mv repo/coverage/coverage-final.json out/`basename \`pwd\``.json - put: minio params: file: out/*.json install every time...
14. #2: STORE CACHES! ➤ use 1st time's cache in 2nd~ times! jobs: - name: gotcha plan: - get: repo - task: unit-test-and-audit config: platform: linux image_resource: type: docker-image source: repository: node tag: 10 inputs: - name: repo ɹɹɹɹɹɹcaches: - path: repo/node_modules outputs: - name: out run: dir: repo path: /bin/sh args: - -c - | npm install npm test npm run-script make-coverage npm audit cd .. mv repo/coverage/coverage-final.json out/`basename \`pwd\``.json - put: minio params: file: out/*.json store caches in next time job
15. FASTER!!! #3
16. #3: USE FAST PACKAGE MANAGER ➤ Node.js: ➤ npm → yarn ➤ PHP: ➤ use local composer repository (packagist.jp) ➤ prestissimo (parallel downloader) ➤ FYI: ޫ஗͍໰୊Λࠀ෰ͯ͠composerΛ10ഒ଎ͨ͘͠࿩ - Mercari Engineering Blog > time npm install npm WARN accelerate-pipeline@1.0.0 No description ... npm install 2.51s user 0.35s system 116% cpu 2.449 total > time yarn install yarn install v1.13.0 ... ✨ Done in 0.21s. yarn install 0.49s user 0.10s system 122% cpu 0.476 total
17. FASTER!! #4
18. #4: CHANGE TO LIGHT DOCKER IMAGE jobs: - name: gotcha plan: - get: repo - task: unit-test-and-audit config: platform: linux image_resource: type: docker-image source: repository: node tag: 10 inputs: - name: repo outputs: - name: out run: dir: repo path: /bin/sh args: - -c - | npm install npm test npm run-script make-coverage npm audit cd .. mv repo/coverage/coverage-final.json out/`basename \`pwd\``.json - put: minio params: file: out/*.json official tag : FROM buildpack-deps
 (debian) > docker images | grep node node 10 8fc2110c6978 3 weeks ago 897MB node 10-alpine fe6ff768f798 3 weeks ago 70.7MB
19. #4: USE ALPINE! ➤ SIZE jobs: - name: gotcha plan: - get: repo - task: unit-test-and-audit config: platform: linux image_resource: type: docker-image source: repository: node tag: 10-alpine inputs: - name: repo outputs: - name: out run: dir: repo path: /bin/sh args: - -c - | npm install npm test npm run-script make-coverage npm audit cd .. mv repo/coverage/coverage-final.json out/`basename \`pwd\``.json - put: minio params: file: out/*.json change to -alpine image! > docker images | grep node node 10 8fc2110c6978 3 weeks ago 897MB node 10-alpine fe6ff768f798 3 weeks ago 70.7MB
20. YOU MIGHT BE CAREFUL... ➤ If you run apps in CF(or another Image), 
 it's better to run CI in same env. ➤ node-gyp's build was often broken in another env... ➤ Java is not pb!? ➤ Trade-off between CI speed and CI reliability ➤ Image Caches is active in 2nd time!!
21. RESULT... x7 FASTER✈ (123s → 17s)
22. IF YOUR PIPELINE IS SLOW... ➤ 1: use "aggregate:" ➤ 2: use "caches:" ➤ 3: use more faster package manager(plugin) ➤ 4: use light Docker Image
23. PR
24. JAPANESE TUTORIAL! ➤http://concoursetutorial-ja.cfapps.io/ ➤PullRequest is welcome!
25. QUESTION?
26. IMAGE BY ➤ ͔Θ͍͍ϑϦʔૉࡐू ͍Β͢ͱ΍ ➤ https://www.irasutoya.com/