Upgrade to Pro — share decks privately, control downloads, hide ads and more …

無限にスケールする上に自律的なJenkinsに見る夢~AWS篇~ / Dream of autonomous and infinite scalable Jenkins with AWS

moznion
August 18, 2018

無限にスケールする上に自律的なJenkinsに見る夢~AWS篇~ / Dream of autonomous and infinite scalable Jenkins with AWS

Kyotoなんか #4の資料です

moznion

August 18, 2018
Tweet

More Decks by moznion

Other Decks in Technology

Transcript

  1. ແݶʹεέʔϧ͢Δ্ʹ

    ࣗ཯తͳJenkinsʹݟΔເ

    ~AWSἫ~
    KyotoͳΜ͔ #4
    @moznion

    View Slide

  2. @moznion
    Software engineer
    Hatena Intern 2013

    View Slide

  3. Jenkins

    View Slide

  4. Jenkins
    - CI (Continuous Integration) Platform
    - Workflow
    - ศརͳcron

    View Slide

  5. Why Jenkins?

    View Slide

  6. Why Jenkins?
    - ΋ͱ΋ͱ͸ (Ұ෦͸ࠓ΋) CircleCIΛ࢖͍ͬͯͨ
    - Job਺͕૿͑ΔʹͭΕ٧·Δqueue
    - ϓϥϯ্͛Δͱߴ͍ΜεΑ……
    - ΋ͬͱϚονϣͳϚγϯͰϏϧυ͍ͨ͠ͱ͍͏ཉ
    - ͦΖͦΖͪΌΜͱͨ͠σϓϩΠϑϩʔͱ͔Λ੔͍͑ͨ
    -

    View Slide

  7. Why Jenkins?
    - ΋ͱ΋ͱ͸ (Ұ෦͸ࠓ΋) CircleCIΛ࢖͍ͬͯͨ
    - Job਺͕૿͑ΔʹͭΕ٧·Δqueue
    - ϓϥϯ্͛Δͱߴ͍ΜεΑ……
    - ΋ͬͱϚονϣͳϚγϯͰϏϧυ͍ͨ͠ͱ͍͏ཉ
    - ͦΖͦΖͪΌΜͱͨ͠σϓϩΠϑϩʔͱ͔Λ੔͍͑ͨ
    - ΋ͱ΋ͱJenkinsΛεέʔϧͤ͞Δํ๏Λ஌͍ͬͯͨ

    View Slide

  8. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊

    View Slide

  9. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
    - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ
    - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠……
    - Slave nodeͷӡ༻େม͗͢Δ໰୊
    - Master InstanceࢮͶͳ͍໰୊
    - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)

    View Slide

  10. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
    - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ
    - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠……
    - Slave nodeͷӡ༻େม͗͢Δ໰୊
    - Master InstanceࢮͶͳ͍໰୊
    - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)

    View Slide

  11. ฒྻjob਺্͛ΔͷϚδَ໳ʂ

    View Slide

  12. ฒྻjob਺্͛ΔͷϚδَ໳
    SlaveΛ଍͢ͷ͸؆୯ʂʂʂʂ

    View Slide

  13. ฒྻjob਺্͛ΔͷϚδَ໳
    - slave଍͢ͷ͸؆୯ͳΜ͚ͩͲ……
    - ୯७ʹ؅ཧίετ͕૿͑Δͧʂ
    - ʮ͜ͷslaveͩͱͳ͔ͥςετ͕ίέ·͢ʯ

    View Slide

  14. ฒྻjob਺্͛ΔͷϚδَ໳
    - slave଍͢ͷ͸؆୯ͳΜ͚ͩͲ……
    - ΦʔτεέʔϦϯάΛͪ·ͪ·΍͍ͬͯ͘ඞཁ͕͋Γ……
    - ΠϯελϯεΛ༡͹͓ͤͯ࣌ؒ͘͸ແ͍
    - CI͸Α΄ͲϓϩδΣΫτཱ͕ͯࠐΜͰͳ͍ݶΓ৸ͯΔ͕࣌ؒ
    ࢧ഑త
    - ׂͱઃఆ͕೉͍͠ (ίπ͕͍Δ)

    View Slide

  15. ͏͏……

    View Slide

  16. զʑ͸΄ͬͱ͍ͯ΋ແݶʹϏϧυ͕εέʔϧ͢Δ
    Jenkins͕΄͍͚ͩ͠ͳΜ͡Ό……

    View Slide

  17. ͦΜͳ΋ͷ͸……
    ͋Δʂʂʂ

    View Slide

  18. ͦΜͳ΋ͷ͸……
    ͋Δʂʂʂ

    View Slide

  19. AWS CodeBuild

    View Slide

  20. AWS CodeBuild
    - ෼୯Ґͷ࣌ؒି͠
    - ೚ҙͷίϯςφΛ࣮ߦͰ͖Δ
    - ͦͷίϯςφ্ͰϏϧυ΍Βςετ΍Β͢Δ
    - (LambdaΑΓ΋͕͔͔࣌ؒΔλεΫͷ࣮ߦ؀ڥʹ΋……)

    View Slide

  21. AWS CodeBuild
    - Jenkins Plugin͕͋Δ (AWSۘ੡)
    - https://github.com/awslabs/aws-codebuild-jenkins-plugin
    - JenkinsͰड͚ͨτϦΨͰCodeBuildΛൃՐͰ͖Δ

    View Slide

  22. https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/

    View Slide

  23. https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/

    View Slide

  24. https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/
    ࣮࣭ແݶʂʂʂʂ

    View Slide

  25. ΋ͨΒ͞ΕͨԸܙ
    - slave nodeͷϝϯς஌Βͣ
    - ΨϯΨϯฒྻϏϧυ͢Δͧʂʂʂ
    - ࣌ؒି͠ (= ࢖ͬͨ෼͚ͩ) ͷྉۚઃఆͳͷͰ

    ࣗ෼ͰΦʔτεέʔϧॲཧΛ͠ͳͯ͘Α͍

    View Slide

  26. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
    - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ
    - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠……
    - Slave nodeͷӡ༻େม͗͢Δ໰୊
    - Master InstanceࢮͶͳ͍໰୊
    - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)

    View Slide

  27. Master Instance͕ਆ֨Խ͢Δ

    View Slide

  28. Master Instanceਆ֨Խ
    - Jenkinsͷmaster data͸masterϚγϯͷ*ϩʔΧϧετϨʔδ*ʹ

    อଘ͞ΕΔ
    - Πϯελϯε͕ࢮΜͩΒऴΘΓ
    - ͔͠͠ࢮ͸ආ͚ΒΕͳ͍
    - ͔ͩΒόοΫΞοϓΛऔͬͨΓ͢Δ
    - e.g. https://papix.hatenablog.com/entry/2016/02/09/101918
    - όοΫΞοϓ͔ΒͷϦετΞ͸࣮ࡍ໘౗
    - ָ͍͠࢓ࣄͰ΋ͳ͍……
    - μ΢ϯλΠϜग़Δ͠……

    View Slide

  29. Master Instanceਆ֨Խ
    - ݁Ռͱͯ͠ຖ೔ଉࡂΛفΔ೔ʑ……
    - ӡѱ͘Πϯελϯε͕ࢮΜͩΒσΠϦʔόοΫΞοϓ
    ͔ΒϦετΞ
    - σΠϦʔόοΫΞοϓͱࢮ͵ؒࡍͱͷࠩ෼ʹ͍ͭͯ͸
    ఘΊΔ͔͠ແ͍……

    View Slide

  30. ͏͏……

    View Slide

  31. զʑ͸ৎ෉Ͱεέʔϧ͢Δ
    Jenkins͕΄͍͚ͩ͠ͳΜ͡Ό……

    View Slide

  32. ͦΜͳ΋ͷ͸……
    ͋Δʂʂʂ

    View Slide

  33. ͦΜͳ΋ͷ͸……
    ͋Δʂʂʂ

    View Slide

  34. Amazon Elastic File System

    View Slide

  35. AWS Elastic File System (EFS)
    - AmazonͷnfsΈ͍ͨͳ΍ͭ
    - Instanceʹmountͯ͠࢖͏
    - EBS + nfsతͳ……
    - ϘϦϡʔϜαΠζ͕ࣗಈͰ֦ுɾॖখ͢Δ
    - Disk full஌Βͣͷੜ׆

    View Slide

  36. AWS Elastic File System (EFS)
    - $JENKINS_HOMEΛEFSϘϦϡʔϜ্ʹ͢Δ
    - Πϯελϯε͕ࢮΜͰ΋৽͍͠Πϯελϯεʹ

    ͦͷϘϦϡʔϜΛϚ΢ϯτ͢Δ
    - => Πϯελϯε͸લੈͷهԱΛҾ͖ܧ͗෮׆ʂʂʂ

    View Slide

  37. mount
    Working...

    View Slide

  38. ❌mount
    Instance down...

    View Slide

  39. ❌ New instance
    coming up...

    View Slide

  40. ❌ mount
    Works fine!!

    View Slide

  41. AWS Elastic File System (EFS)
    - EBSͰ΋ྑ͍ͷͰ͸ʁ
    - ͸͍ʂɹEBSͷ΄͏͕͍҆͠ʂʂ
    - ͨͩEFS͸ϘϦϡʔϜࣗಈ֦ு͕خ͍͠
    - (͏͔ͬΓdisk fullʹͳΒͳ͍)
    - ͋ͱEFSͩͱmulti masterΈ͍ͨͳ͜ͱ΋Ͱ͖Δ (ޙड़)

    View Slide

  42. Jenkins Multi Master
    - ଟ෼Ͱ͖Δ (௕࣌ؒӡ༻ͯ͠ͳ͍)
    - ͔͠͠CodeBuildͱڞଘ͍ͯ͠Δ؀ڥͰMulti Masterʹ͢Δ

    ϝϦοτ΄΅ͳ͘ͳ͍Ͱ͔͢ʁ
    - σϓϩΠ࣌ʹҰॠ2୆masterΛཱ͓͍ͯͯͯɼݹ͍ํΛ

    ࡴ͢Έ͍ͨͳ௿μ΢ϯλΠϜσϓϩΠ޲͚ͷ࢖͍ํ͸Մೳ

    View Slide

  43. ΋ͨΒ͞ΕͨԸܙ
    - ΧδϡΞϧʹΠϯελϯε͕ࢮΜͰ΋҆৺ʂ
    - ͱ͸͍͑όοΫΞοϓ͸औͬͯΔ
    - ݁Ռͱͯ͠ΠϯελϯεͷೖΕସָ͕͑ʹͳͬͨ
    - ετϨʔδ΋ࣗಈ֦ு͢ΔͷͰDisk full஌Βͣ

    View Slide

  44. Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊
    - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ
    - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠……
    - Slave nodeͷӡ༻େม͗͢Δ໰୊
    - Master InstanceࢮͶͳ͍໰୊
    - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)

    View Slide

  45. ࣾձ໰୊: Jenkins͓͡͞ΜͷͳΓखෆ଍

    View Slide

  46. Jenkinsͷӡ༻Λ୭͔ʹԡ͠෇͚ΔͷΛ΍ΊΔ
    - ୭Ͱ΋ӡ༻Ͱ͖ΔΑ͏ʹ͢Δ
    - શਓຽJenkins͓͡͞Μ/͓͹͞ΜԽ
    - σϓϩΠ΍ϝϯς͸ۃྗলྗԽ͢Δ

    View Slide

  47. Jenkinsͷӡ༻Λ୭͔ʹԡ͠෇͚ΔͷΛ΍ΊΔ
    - Amazon Elastic Beanstalk + Docker
    - Dockerfile + Dockerrun.aws.jsonʹΑΔߏ੒ͷίʔυԽ
    - .ebextensionsʹΑΔࣗಈԽ
    - EFS mountͳͲ
    - https://github.com/moznion/jenkins_eb_efs_codebuild_example

    View Slide

  48. - Elastic Beanstalkͷmanaged updateͰࣗಈతʹύον͕౰ͨΔ
    - ͍ͭͰʹΠϯελϯεͷೖΕସ͑΋ߦΘΕΔ
    - EFS؀ڥԼͳͷͰΠϯελϯεೖΕସ͑΋ָʑࣗಈԽ
    - جຊతʹӡ༻͸์ஔͯ͠·͢
    - EFSͷϘϦϡʔϜࣗಈ֦ுͷύϫʔͰ΋͋Δ
    ΋ͨΒ͞ΕͨԸܙ

    View Slide

  49. ·ͱΊ

    View Slide

  50. ·ͱΊ
    - JenkinsͷslaveϊʔυΛAWS CodeBuildʹҕৡ͢Δ͜ͱͰ࣮࣭
    ແݶʹฒྻϏϧυ਺͕εέʔϧ͢ΔΑ͏ʹͳΓ·ͨ͠
    - JenkinsͷετϨʔδΛEFSʹ͢Δ͜ͱͰϚελʔσʔλͷ
    ϩόετԽΛߦ͍·ͨ͠
    - Elastic BeanstalkΛ༻͍ͯJenkinsΛӡ༻͢Δ͜ͱͰӡ༻ͷ௿
    ίετԽΛ͠·ͨ͠
    - Α͔ͬͨͰ͢Ͷ

    View Slide

  51. Q?

    View Slide

  52. Wait...

    View Slide

  53. ͜ͷJenkinsʹ͸໰୊͕͋Δ
    - EFS͸஗͍ʂʂʂʂʂ
    - Jenkins͕git clone͢ΔͱϚδ͔͔࣌ؒΓ·͢
    - git cloneΛճආ͢Δํ๏͸͍·ͷͱ͜Ζແ͍……
    - shallow cloneΛ༗ޮʹ࢖͏
    - sparse checkoutΛ༗ޮʹ࢖͏
    - ͳͲͷ޻෉͕ඞཁ
    - ͦ΋ͦ΋CodeBuild͕git clone͢ΔΜͩ͠jenkins͸͠ͳͯ͘

    ྑ͍Μ͚ͩͲ……
    - ͋Δ͍͸EBSΛ࢖͏……

    View Slide

  54. ຊ౰ʹJenkins͸ඞཁʁ
    - ͭ͡͸AWS CodeBuild͸࠷ۙΊͬͪΌͰ͖͕Αͯ͘ɼ

    ୯ମͰ΋े෼࢖͑Δ
    - GitHubͷhookड͚ΕΔ͠
    - Commit statusΛม͑Δػೳ·Ͱ͍͍ͭͯΔ
    - Ϗϧυཤྺ΋ҰཡͰ͖Δ͠
    - CloudWatch LogsʹϏϧυͷϩά΋ग़Δ
    - JenkinsෆཁͰ͸ʁʁʁʁɹ͜ΕͰे෼Ͱ͸ʁʁʁʁ
    - ͸͍

    View Slide

  55. ຊ౰ʹJenkins͸ඞཁʁ
    - ࣗ༝ͳWebhookΛड͚Δඞཁ͕͋Δͱ͔……
    - ͋ͱ௨஌ͱ͔Ͷʂ
    - ͋Δ͍͸΋ͬͱଞʹϏϧυύΠϓϥΠϯΛܨ
    ͛Δඞཁ͕͋Δͱ͔……
    - ͦ͏͍͏ͱ͖͸΂ΜΓͰ͢
    - (͋ͱ͸·͋Ұཡੑ͕Α͍) (ॾઆ͋Γ·͢)

    View Slide

  56. Q?

    View Slide