無限にスケールする上に自律的なJenkinsに見る夢~AWS篇~ / Dream of autonomous and infinite scalable Jenkins with AWS

Slide 1

Slide 1 text

ແݶʹεέʔϧ͢Δ্ʹ  ࣗ཯తͳJenkinsʹݟΔເ  ~AWSἫ~ KyotoͳΜ͔ #4 @moznion

Slide 2

Slide 2 text

@moznion Software engineer Hatena Intern 2013

Slide 3

Slide 3 text

Jenkins

Slide 4

Slide 4 text

Jenkins - CI (Continuous Integration) Platform - Workflow - ศརͳcron

Slide 5

Slide 5 text

Why Jenkins?

Slide 6

Slide 6 text

Why Jenkins? - ΋ͱ΋ͱ͸ (Ұ෦͸ࠓ΋) CircleCIΛ࢖͍ͬͯͨ - Job਺͕૿͑ΔʹͭΕ٧·Δqueue - ϓϥϯ্͛Δͱߴ͍ΜεΑ…… - ΋ͬͱϚονϣͳϚγϯͰϏϧυ͍ͨ͠ͱ͍͏ཉ - ͦΖͦΖͪΌΜͱͨ͠σϓϩΠϑϩʔͱ͔Λ੔͍͑ͨ -

Slide 7

Slide 7 text

Why Jenkins? - ΋ͱ΋ͱ͸ (Ұ෦͸ࠓ΋) CircleCIΛ࢖͍ͬͯͨ - Job਺͕૿͑ΔʹͭΕ٧·Δqueue - ϓϥϯ্͛Δͱߴ͍ΜεΑ…… - ΋ͬͱϚονϣͳϚγϯͰϏϧυ͍ͨ͠ͱ͍͏ཉ - ͦΖͦΖͪΌΜͱͨ͠σϓϩΠϑϩʔͱ͔Λ੔͍͑ͨ - ΋ͱ΋ͱJenkinsΛεέʔϧͤ͞Δํ๏Λ஌͍ͬͯͨ

Slide 8

Slide 8 text

Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊

Slide 9

Slide 9 text

Jenkinsӡ༻ʹࡍͯ͠ग़͘Θ͢໰୊ - ฒྻ࣮ߦJob਺ͷ૿Ճʹ൐͏ੑೳྼԽ - ߜΔͱbuild଴ͪͷΩϡʔ͕٧·Δ͠…… - Slave nodeͷӡ༻େม͗͢Δ໰୊ - Master InstanceࢮͶͳ͍໰୊ - ͓ئ͍͠·͢೴ͷڪා (™ @kyanny)

Slide 10

Slide 10 text

Slide 11

Slide 11 text

ฒྻjob਺্͛ΔͷϚδَ໳ʂ

Slide 12

Slide 12 text

ฒྻjob਺্͛ΔͷϚδَ໳ SlaveΛ଍͢ͷ͸؆୯ʂʂʂʂ

Slide 13

Slide 13 text

ฒྻjob਺্͛ΔͷϚδَ໳ - slave଍͢ͷ͸؆୯ͳΜ͚ͩͲ…… - ୯७ʹ؅ཧίετ͕૿͑Δͧʂ - ʮ͜ͷslaveͩͱͳ͔ͥςετ͕ίέ·͢ʯ

Slide 14

Slide 14 text

ฒྻjob਺্͛ΔͷϚδَ໳ - slave଍͢ͷ͸؆୯ͳΜ͚ͩͲ…… - ΦʔτεέʔϦϯάΛͪ·ͪ·΍͍ͬͯ͘ඞཁ͕͋Γ…… - ΠϯελϯεΛ༡͹͓ͤͯ࣌ؒ͘͸ແ͍ - CI͸Α΄ͲϓϩδΣΫτཱ͕ͯࠐΜͰͳ͍ݶΓ৸ͯΔ͕࣌ؒ ࢧ഑త - ׂͱઃఆ͕೉͍͠ (ίπ͕͍Δ)

Slide 15

Slide 15 text

͏͏……

Slide 16

Slide 16 text

զʑ͸΄ͬͱ͍ͯ΋ແݶʹϏϧυ͕εέʔϧ͢Δ Jenkins͕΄͍͚ͩ͠ͳΜ͡Ό……

Slide 17

Slide 17 text

ͦΜͳ΋ͷ͸…… ͋Δʂʂʂ

Slide 18

Slide 18 text

ͦΜͳ΋ͷ͸…… ͋Δʂʂʂ

Slide 19

Slide 19 text

AWS CodeBuild

Slide 20

Slide 20 text

AWS CodeBuild - ෼୯Ґͷ࣌ؒି͠ - ೚ҙͷίϯςφΛ࣮ߦͰ͖Δ - ͦͷίϯςφ্ͰϏϧυ΍Βςετ΍Β͢Δ - (LambdaΑΓ΋͕͔͔࣌ؒΔλεΫͷ࣮ߦ؀ڥʹ΋……)

Slide 21

Slide 21 text

AWS CodeBuild - Jenkins Plugin͕͋Δ (AWSۘ੡) - https://github.com/awslabs/aws-codebuild-jenkins-plugin - JenkinsͰड͚ͨτϦΨͰCodeBuildΛൃՐͰ͖Δ

Slide 22

Slide 22 text

https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/

Slide 23

Slide 23 text

https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/

Slide 24

Slide 24 text

https://aws.amazon.com/jp/blogs/devops/simplify-your-jenkins-builds-with-aws-codebuild/ ࣮࣭ແݶʂʂʂʂ

Slide 25

Slide 25 text

΋ͨΒ͞ΕͨԸܙ - slave nodeͷϝϯς஌Βͣ - ΨϯΨϯฒྻϏϧυ͢Δͧʂʂʂ - ࣌ؒି͠ (= ࢖ͬͨ෼͚ͩ) ͷྉۚઃఆͳͷͰ  ࣗ෼ͰΦʔτεέʔϧॲཧΛ͠ͳͯ͘Α͍

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Master Instance͕ਆ֨Խ͢Δ

Slide 28

Slide 28 text

Master Instanceਆ֨Խ - Jenkinsͷmaster data͸masterϚγϯͷ*ϩʔΧϧετϨʔδ*ʹ  อଘ͞ΕΔ - Πϯελϯε͕ࢮΜͩΒऴΘΓ - ͔͠͠ࢮ͸ආ͚ΒΕͳ͍ - ͔ͩΒόοΫΞοϓΛऔͬͨΓ͢Δ - e.g. https://papix.hatenablog.com/entry/2016/02/09/101918 - όοΫΞοϓ͔ΒͷϦετΞ͸࣮ࡍ໘౗ - ָ͍͠࢓ࣄͰ΋ͳ͍…… - μ΢ϯλΠϜग़Δ͠……

Slide 29

Slide 29 text

Master Instanceਆ֨Խ - ݁Ռͱͯ͠ຖ೔ଉࡂΛفΔ೔ʑ…… - ӡѱ͘Πϯελϯε͕ࢮΜͩΒσΠϦʔόοΫΞοϓ ͔ΒϦετΞ - σΠϦʔόοΫΞοϓͱࢮ͵ؒࡍͱͷࠩ෼ʹ͍ͭͯ͸ ఘΊΔ͔͠ແ͍……

Slide 30

Slide 30 text

͏͏……

Slide 31

Slide 31 text

զʑ͸ৎ෉Ͱεέʔϧ͢Δ Jenkins͕΄͍͚ͩ͠ͳΜ͡Ό……

Slide 32

Slide 32 text

ͦΜͳ΋ͷ͸…… ͋Δʂʂʂ

Slide 33

Slide 33 text

ͦΜͳ΋ͷ͸…… ͋Δʂʂʂ

Slide 34

Slide 34 text

Amazon Elastic File System

Slide 35

Slide 35 text

AWS Elastic File System (EFS) - AmazonͷnfsΈ͍ͨͳ΍ͭ - Instanceʹmountͯ͠࢖͏ - EBS + nfsతͳ…… - ϘϦϡʔϜαΠζ͕ࣗಈͰ֦ுɾॖখ͢Δ - Disk full஌Βͣͷੜ׆

Slide 36

Slide 36 text

AWS Elastic File System (EFS) - $JENKINS_HOMEΛEFSϘϦϡʔϜ্ʹ͢Δ - Πϯελϯε͕ࢮΜͰ΋৽͍͠Πϯελϯεʹ  ͦͷϘϦϡʔϜΛϚ΢ϯτ͢Δ - => Πϯελϯε͸લੈͷهԱΛҾ͖ܧ͗෮׆ʂʂʂ

Slide 37

Slide 37 text

mount Working...

Slide 38

Slide 38 text

❌mount Instance down...

Slide 39

Slide 39 text

❌ New instance coming up...

Slide 40

Slide 40 text

❌ mount Works fine!!

Slide 41

Slide 41 text

AWS Elastic File System (EFS) - EBSͰ΋ྑ͍ͷͰ͸ʁ - ͸͍ʂɹEBSͷ΄͏͕͍҆͠ʂʂ - ͨͩEFS͸ϘϦϡʔϜࣗಈ֦ு͕خ͍͠ - (͏͔ͬΓdisk fullʹͳΒͳ͍) - ͋ͱEFSͩͱmulti masterΈ͍ͨͳ͜ͱ΋Ͱ͖Δ (ޙड़)

Slide 42

Slide 42 text

Jenkins Multi Master - ଟ෼Ͱ͖Δ (௕࣌ؒӡ༻ͯ͠ͳ͍) - ͔͠͠CodeBuildͱڞଘ͍ͯ͠Δ؀ڥͰMulti Masterʹ͢Δ  ϝϦοτ΄΅ͳ͘ͳ͍Ͱ͔͢ʁ - σϓϩΠ࣌ʹҰॠ2୆masterΛཱ͓͍ͯͯͯɼݹ͍ํΛ  ࡴ͢Έ͍ͨͳ௿μ΢ϯλΠϜσϓϩΠ޲͚ͷ࢖͍ํ͸Մೳ

Slide 43

Slide 43 text

΋ͨΒ͞ΕͨԸܙ - ΧδϡΞϧʹΠϯελϯε͕ࢮΜͰ΋҆৺ʂ - ͱ͸͍͑όοΫΞοϓ͸औͬͯΔ - ݁Ռͱͯ͠ΠϯελϯεͷೖΕସָ͕͑ʹͳͬͨ - ετϨʔδ΋ࣗಈ֦ு͢ΔͷͰDisk full஌Βͣ

Slide 44

Slide 44 text

Slide 45

Slide 45 text

ࣾձ໰୊: Jenkins͓͡͞ΜͷͳΓखෆ଍

Slide 46

Slide 46 text

Jenkinsͷӡ༻Λ୭͔ʹԡ͠෇͚ΔͷΛ΍ΊΔ - ୭Ͱ΋ӡ༻Ͱ͖ΔΑ͏ʹ͢Δ - શਓຽJenkins͓͡͞Μ/͓͹͞ΜԽ - σϓϩΠ΍ϝϯς͸ۃྗলྗԽ͢Δ

Slide 47

Slide 47 text

Jenkinsͷӡ༻Λ୭͔ʹԡ͠෇͚ΔͷΛ΍ΊΔ - Amazon Elastic Beanstalk + Docker - Dockerfile + Dockerrun.aws.jsonʹΑΔߏ੒ͷίʔυԽ - .ebextensionsʹΑΔࣗಈԽ - EFS mountͳͲ - https://github.com/moznion/jenkins_eb_efs_codebuild_example

Slide 48

Slide 48 text

- Elastic Beanstalkͷmanaged updateͰࣗಈతʹύον͕౰ͨΔ - ͍ͭͰʹΠϯελϯεͷೖΕସ͑΋ߦΘΕΔ - EFS؀ڥԼͳͷͰΠϯελϯεೖΕସ͑΋ָʑࣗಈԽ - جຊతʹӡ༻͸์ஔͯ͠·͢ - EFSͷϘϦϡʔϜࣗಈ֦ுͷύϫʔͰ΋͋Δ ΋ͨΒ͞ΕͨԸܙ

Slide 49

Slide 49 text

·ͱΊ

Slide 50

Slide 50 text

·ͱΊ - JenkinsͷslaveϊʔυΛAWS CodeBuildʹҕৡ͢Δ͜ͱͰ࣮࣭ ແݶʹฒྻϏϧυ਺͕εέʔϧ͢ΔΑ͏ʹͳΓ·ͨ͠ - JenkinsͷετϨʔδΛEFSʹ͢Δ͜ͱͰϚελʔσʔλͷ ϩόετԽΛߦ͍·ͨ͠ - Elastic BeanstalkΛ༻͍ͯJenkinsΛӡ༻͢Δ͜ͱͰӡ༻ͷ௿ ίετԽΛ͠·ͨ͠ - Α͔ͬͨͰ͢Ͷ

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Wait...

Slide 53

Slide 53 text

͜ͷJenkinsʹ͸໰୊͕͋Δ - EFS͸஗͍ʂʂʂʂʂ - Jenkins͕git clone͢ΔͱϚδ͔͔࣌ؒΓ·͢ - git cloneΛճආ͢Δํ๏͸͍·ͷͱ͜Ζແ͍…… - shallow cloneΛ༗ޮʹ࢖͏ - sparse checkoutΛ༗ޮʹ࢖͏ - ͳͲͷ޻෉͕ඞཁ - ͦ΋ͦ΋CodeBuild͕git clone͢ΔΜͩ͠jenkins͸͠ͳͯ͘  ྑ͍Μ͚ͩͲ…… - ͋Δ͍͸EBSΛ࢖͏……

Slide 54

Slide 54 text

ຊ౰ʹJenkins͸ඞཁʁ - ͭ͡͸AWS CodeBuild͸࠷ۙΊͬͪΌͰ͖͕Αͯ͘ɼ  ୯ମͰ΋े෼࢖͑Δ - GitHubͷhookड͚ΕΔ͠ - Commit statusΛม͑Δػೳ·Ͱ͍͍ͭͯΔ - Ϗϧυཤྺ΋ҰཡͰ͖Δ͠ - CloudWatch LogsʹϏϧυͷϩά΋ग़Δ - JenkinsෆཁͰ͸ʁʁʁʁɹ͜ΕͰे෼Ͱ͸ʁʁʁʁ - ͸͍

Slide 55

Slide 55 text

ຊ౰ʹJenkins͸ඞཁʁ - ࣗ༝ͳWebhookΛड͚Δඞཁ͕͋Δͱ͔…… - ͋ͱ௨஌ͱ͔Ͷʂ - ͋Δ͍͸΋ͬͱଞʹϏϧυύΠϓϥΠϯΛܨ ͛Δඞཁ͕͋Δͱ͔…… - ͦ͏͍͏ͱ͖͸΂ΜΓͰ͢ - (͋ͱ͸·͋Ұཡੑ͕Α͍) (ॾઆ͋Γ·͢)