無限にスケールする上に自律的なJenkinsに見る夢~AWS篇~ / Dream of autonomous and infinite scalable Jenkins with AWS

Kyotoなんか #4の資料です

Transcript

1. ແݶʹεέʔϧ͢Δ্ʹ

   ࣗ཯తͳJenkinsʹݟΔເ

   ~AWSἫ~
   KyotoͳΜ͔ #4
   @moznion
   

   View Slide

2. @moznion
   Software engineer
   Hatena Intern 2013
   

   View Slide

3. Jenkins
   

   View Slide

4. Jenkins
   - CI (Continuous Integration) Platform
   - Workflow
   - ศརͳcron
   

   View Slide

5. Why Jenkins?
   

   View Slide

6. Why Jenkins?
   - ΋ͱ΋ͱ͸ (Ұ෦͸ࠓ΋) CircleCIΛ࢖͍ͬͯͨ
   - Job਺͕૿͑ΔʹͭΕ٧·Δqueue
   - ϓϥϯ্͛Δͱߴ͍ΜεΑ……
   - ΋ͬͱϚονϣͳϚγϯͰϏϧυ͍ͨ͠ͱ͍͏ཉ
   - ͦΖͦΖͪΌΜͱͨ͠σϓϩΠϑϩʔͱ͔Λ੔͍͑ͨ
   -
   

   View Slide

7. Why Jenkins?
   - ΋ͱ΋ͱ͸ (Ұ෦͸ࠓ΋) CircleCIΛ࢖͍ͬͯͨ
   - Job਺͕૿͑ΔʹͭΕ٧·Δqueue
   - ϓϥϯ্͛Δͱߴ͍ΜεΑ……
   - ΋ͬͱϚονϣͳϚγϯͰϏϧυ͍ͨ͠ͱ͍͏ཉ
   - ͦΖͦΖͪΌΜͱͨ͠σϓϩΠϑϩʔͱ͔Λ੔͍͑ͨ
   - ΋ͱ΋ͱJenkinsΛεέʔϧͤ͞Δํ๏Λ஌͍ͬͯͨ
   

   View Slide

8. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
   

   View Slide

9. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
   - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ
   - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠……
   - Slave nodeͷӡ༻େม͗͢Δ໰୊
   - Master InstanceࢮͶͳ͍໰୊
   - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)
   

   View Slide

10. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
    - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ
    - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠……
    - Slave nodeͷӡ༻େม͗͢Δ໰୊
    - Master InstanceࢮͶͳ͍໰୊
    - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)
    

    View Slide

11. ฒྻjob਺্͛ΔͷϚδَ໳ʂ
    

    View Slide

12. ฒྻjob਺্͛ΔͷϚδَ໳
    SlaveΛ଍͢ͷ͸؆୯ʂʂʂʂ
    

    View Slide

13. ฒྻjob਺্͛ΔͷϚδَ໳
    - slave଍͢ͷ͸؆୯ͳΜ͚ͩͲ……
    - ୯७ʹ؅ཧίετ͕૿͑Δͧʂ
    - ʮ͜ͷslaveͩͱͳ͔ͥςετ͕ίέ·͢ʯ
    

    View Slide

14. ฒྻjob਺্͛ΔͷϚδَ໳
    - slave଍͢ͷ͸؆୯ͳΜ͚ͩͲ……
    - ΦʔτεέʔϦϯάΛͪ·ͪ·΍͍ͬͯ͘ඞཁ͕͋Γ……
    - ΠϯελϯεΛ༡͹͓ͤͯ࣌ؒ͘͸ແ͍
    - CI͸Α΄ͲϓϩδΣΫτཱ͕ͯࠐΜͰͳ͍ݶΓ৸ͯΔ͕࣌ؒ
    ࢧ഑త
    - ׂͱઃఆ͕೉͍͠ (ίπ͕͍Δ)
    

    View Slide

15. ͏͏……
    

    View Slide

16. զʑ͸΄ͬͱ͍ͯ΋ແݶʹϏϧυ͕εέʔϧ͢Δ
    Jenkins͕΄͍͚ͩ͠ͳΜ͡Ό……
    

    View Slide

17. ͦΜͳ΋ͷ͸……
    ͋Δʂʂʂ
    

    View Slide

18. ͦΜͳ΋ͷ͸……
    ͋Δʂʂʂ
    

    View Slide

19. AWS CodeBuild
    

    View Slide

20. AWS CodeBuild
    - ෼୯Ґͷ࣌ؒି͠
    - ೚ҙͷίϯςφΛ࣮ߦͰ͖Δ
    - ͦͷίϯςφ্ͰϏϧυ΍Βςετ΍Β͢Δ
    - (LambdaΑΓ΋͕͔͔࣌ؒΔλεΫͷ࣮ߦ؀ڥʹ΋……)
    

    View Slide

21. AWS CodeBuild
    - Jenkins Plugin͕͋Δ (AWSۘ੡)
    - https://github.com/awslabs/aws-codebuild-jenkins-plugin
    - JenkinsͰड͚ͨτϦΨͰCodeBuildΛൃՐͰ͖Δ
    

    View Slide

22. https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/
    

    View Slide

23. https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/
    

    View Slide

24. https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/
    ࣮࣭ແݶʂʂʂʂ
    

    View Slide

25. ΋ͨΒ͞ΕͨԸܙ
    - slave nodeͷϝϯς஌Βͣ
    - ΨϯΨϯฒྻϏϧυ͢Δͧʂʂʂ
    - ࣌ؒି͠ (= ࢖ͬͨ෼͚ͩ) ͷྉۚઃఆͳͷͰ

    ࣗ෼ͰΦʔτεέʔϧॲཧΛ͠ͳͯ͘Α͍
    

    View Slide

26. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
    - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ
    - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠……
    - Slave nodeͷӡ༻େม͗͢Δ໰୊
    - Master InstanceࢮͶͳ͍໰୊
    - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)
    

    View Slide

27. Master Instance͕ਆ֨Խ͢Δ
    

    View Slide

28. Master Instanceਆ֨Խ
    - Jenkinsͷmaster data͸masterϚγϯͷ*ϩʔΧϧετϨʔδ*ʹ

    อଘ͞ΕΔ
    - Πϯελϯε͕ࢮΜͩΒऴΘΓ
    - ͔͠͠ࢮ͸ආ͚ΒΕͳ͍
    - ͔ͩΒόοΫΞοϓΛऔͬͨΓ͢Δ
    - e.g. https://papix.hatenablog.com/entry/2016/02/09/101918
    - όοΫΞοϓ͔ΒͷϦετΞ͸࣮ࡍ໘౗
    - ָ͍͠࢓ࣄͰ΋ͳ͍……
    - μ΢ϯλΠϜग़Δ͠……
    

    View Slide

29. Master Instanceਆ֨Խ
    - ݁Ռͱͯ͠ຖ೔ଉࡂΛفΔ೔ʑ……
    - ӡѱ͘Πϯελϯε͕ࢮΜͩΒσΠϦʔόοΫΞοϓ
    ͔ΒϦετΞ
    - σΠϦʔόοΫΞοϓͱࢮ͵ؒࡍͱͷࠩ෼ʹ͍ͭͯ͸
    ఘΊΔ͔͠ແ͍……
    

    View Slide

30. ͏͏……
    

    View Slide

31. զʑ͸ৎ෉Ͱεέʔϧ͢Δ
    Jenkins͕΄͍͚ͩ͠ͳΜ͡Ό……
    

    View Slide

32. ͦΜͳ΋ͷ͸……
    ͋Δʂʂʂ
    

    View Slide

33. ͦΜͳ΋ͷ͸……
    ͋Δʂʂʂ
    

    View Slide

34. Amazon Elastic File System
    

    View Slide

35. AWS Elastic File System (EFS)
    - AmazonͷnfsΈ͍ͨͳ΍ͭ
    - Instanceʹmountͯ͠࢖͏
    - EBS + nfsతͳ……
    - ϘϦϡʔϜαΠζ͕ࣗಈͰ֦ுɾॖখ͢Δ
    - Disk full஌Βͣͷੜ׆
    

    View Slide

36. AWS Elastic File System (EFS)
    - $JENKINS_HOMEΛEFSϘϦϡʔϜ্ʹ͢Δ
    - Πϯελϯε͕ࢮΜͰ΋৽͍͠Πϯελϯεʹ

    ͦͷϘϦϡʔϜΛϚ΢ϯτ͢Δ
    - => Πϯελϯε͸લੈͷهԱΛҾ͖ܧ͗෮׆ʂʂʂ
    

    View Slide

37. mount
    Working...
    

    View Slide

38. ❌mount
    Instance down...
    

    View Slide

39. ❌ New instance
    coming up...
    

    View Slide

40. ❌ mount
    Works fine!!
    

    View Slide

41. AWS Elastic File System (EFS)
    - EBSͰ΋ྑ͍ͷͰ͸ʁ
    - ͸͍ʂɹEBSͷ΄͏͕͍҆͠ʂʂ
    - ͨͩEFS͸ϘϦϡʔϜࣗಈ֦ு͕خ͍͠
    - (͏͔ͬΓdisk fullʹͳΒͳ͍)
    - ͋ͱEFSͩͱmulti masterΈ͍ͨͳ͜ͱ΋Ͱ͖Δ (ޙड़)
    

    View Slide

42. Jenkins Multi Master
    - ଟ෼Ͱ͖Δ (௕࣌ؒӡ༻ͯ͠ͳ͍)
    - ͔͠͠CodeBuildͱڞଘ͍ͯ͠Δ؀ڥͰMulti Masterʹ͢Δ

    ϝϦοτ΄΅ͳ͘ͳ͍Ͱ͔͢ʁ
    - σϓϩΠ࣌ʹҰॠ2୆masterΛཱ͓͍ͯͯͯɼݹ͍ํΛ

    ࡴ͢Έ͍ͨͳ௿μ΢ϯλΠϜσϓϩΠ޲͚ͷ࢖͍ํ͸Մೳ
    

    View Slide

43. ΋ͨΒ͞ΕͨԸܙ
    - ΧδϡΞϧʹΠϯελϯε͕ࢮΜͰ΋҆৺ʂ
    - ͱ͸͍͑όοΫΞοϓ͸औͬͯΔ
    - ݁Ռͱͯ͠ΠϯελϯεͷೖΕସָ͕͑ʹͳͬͨ
    - ετϨʔδ΋ࣗಈ֦ு͢ΔͷͰDisk full஌Βͣ
    

    View Slide

44. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
    - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ
    - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠……
    - Slave nodeͷӡ༻େม͗͢Δ໰୊
    - Master InstanceࢮͶͳ͍໰୊
    - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)
    

    View Slide

45. ࣾձ໰୊: Jenkins͓͡͞ΜͷͳΓखෆ଍
    

    View Slide

46. Jenkinsͷӡ༻Λ୭͔ʹԡ͠෇͚ΔͷΛ΍ΊΔ
    - ୭Ͱ΋ӡ༻Ͱ͖ΔΑ͏ʹ͢Δ
    - શਓຽJenkins͓͡͞Μ/͓͹͞ΜԽ
    - σϓϩΠ΍ϝϯς͸ۃྗলྗԽ͢Δ
    

    View Slide

47. Jenkinsͷӡ༻Λ୭͔ʹԡ͠෇͚ΔͷΛ΍ΊΔ
    - Amazon Elastic Beanstalk + Docker
    - Dockerfile + Dockerrun.aws.jsonʹΑΔߏ੒ͷίʔυԽ
    - .ebextensionsʹΑΔࣗಈԽ
    - EFS mountͳͲ
    - https://github.com/moznion/jenkins_eb_efs_codebuild_example
    

    View Slide

48. - Elastic Beanstalkͷmanaged updateͰࣗಈతʹύον͕౰ͨΔ
    - ͍ͭͰʹΠϯελϯεͷೖΕସ͑΋ߦΘΕΔ
    - EFS؀ڥԼͳͷͰΠϯελϯεೖΕସ͑΋ָʑࣗಈԽ
    - جຊతʹӡ༻͸์ஔͯ͠·͢
    - EFSͷϘϦϡʔϜࣗಈ֦ுͷύϫʔͰ΋͋Δ
    ΋ͨΒ͞ΕͨԸܙ
    

    View Slide

49. ·ͱΊ
    

    View Slide

50. ·ͱΊ
    - JenkinsͷslaveϊʔυΛAWS CodeBuildʹҕৡ͢Δ͜ͱͰ࣮࣭
    ແݶʹฒྻϏϧυ਺͕εέʔϧ͢ΔΑ͏ʹͳΓ·ͨ͠
    - JenkinsͷετϨʔδΛEFSʹ͢Δ͜ͱͰϚελʔσʔλͷ
    ϩόετԽΛߦ͍·ͨ͠
    - Elastic BeanstalkΛ༻͍ͯJenkinsΛӡ༻͢Δ͜ͱͰӡ༻ͷ௿
    ίετԽΛ͠·ͨ͠
    - Α͔ͬͨͰ͢Ͷ
    

    View Slide

51. Q?
    

    View Slide

52. Wait...
    

    View Slide

53. ͜ͷJenkinsʹ͸໰୊͕͋Δ
    - EFS͸஗͍ʂʂʂʂʂ
    - Jenkins͕git clone͢ΔͱϚδ͔͔࣌ؒΓ·͢
    - git cloneΛճආ͢Δํ๏͸͍·ͷͱ͜Ζແ͍……
    - shallow cloneΛ༗ޮʹ࢖͏
    - sparse checkoutΛ༗ޮʹ࢖͏
    - ͳͲͷ޻෉͕ඞཁ
    - ͦ΋ͦ΋CodeBuild͕git clone͢ΔΜͩ͠jenkins͸͠ͳͯ͘

    ྑ͍Μ͚ͩͲ……
    - ͋Δ͍͸EBSΛ࢖͏……
    

    View Slide

54. ຊ౰ʹJenkins͸ඞཁʁ
    - ͭ͡͸AWS CodeBuild͸࠷ۙΊͬͪΌͰ͖͕Αͯ͘ɼ

    ୯ମͰ΋े෼࢖͑Δ
    - GitHubͷhookड͚ΕΔ͠
    - Commit statusΛม͑Δػೳ·Ͱ͍͍ͭͯΔ
    - Ϗϧυཤྺ΋ҰཡͰ͖Δ͠
    - CloudWatch LogsʹϏϧυͷϩά΋ग़Δ
    - JenkinsෆཁͰ͸ʁʁʁʁɹ͜ΕͰे෼Ͱ͸ʁʁʁʁ
    - ͸͍
    

    View Slide

55. ຊ౰ʹJenkins͸ඞཁʁ
    - ࣗ༝ͳWebhookΛड͚Δඞཁ͕͋Δͱ͔……
    - ͋ͱ௨஌ͱ͔Ͷʂ
    - ͋Δ͍͸΋ͬͱଞʹϏϧυύΠϓϥΠϯΛܨ
    ͛Δඞཁ͕͋Δͱ͔……
    - ͦ͏͍͏ͱ͖͸΂ΜΓͰ͢
    - (͋ͱ͸·͋Ұཡੑ͕Α͍) (ॾઆ͋Γ·͢)
    

    View Slide

56. Q?
    

    View Slide