データ分析基盤Crois
by
Recruit Communications
Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
σʔλੳج൫$SPJT
Slide 2
Slide 2 text
ΞδΣϯμ
Slide 3
Slide 3 text
1. Crois 2. Crois 3. 4. 5. 15
Slide 4
Slide 4 text
$SPJTͷ֓ཁ
Slide 5
Slide 5 text
17
Slide 6
Slide 6 text
o D n T mf , , ,, , T D T m e c a T T D r k k 18
Slide 7
Slide 7 text
@7 8-,. •5C3B <'$%* +4 / !& " 9A •>1!) :62?; # $()/ •&'$' =0 * & +49A 19
Slide 8
Slide 8 text
Crois •" • • •!#API 20
Slide 9
Slide 9 text
22
Slide 10
Slide 10 text
Slide 11
Slide 11 text
= = =
Slide 12
Slide 12 text
= = =
Slide 13
Slide 13 text
(&= = %$"= #'!
Slide 14
Slide 14 text
$SPJTͷΈ
Slide 15
Slide 15 text
32
Slide 16
Slide 16 text
• YAML tasks: - name: task-1 module: sleep params: hoge: fuga - name: task-2 module: sleep 33
Slide 17
Slide 17 text
AWS BatchStep Functions •AWS Batch • •Step Functions • 34
Slide 18
Slide 18 text
SFN •11SFN StateMachine tasks: - name: task-1 module: sleep = 35
Slide 19
Slide 19 text
SFN = Lambda Lambda Lambda 36
Slide 20
Slide 20 text
SFN = AWS Batch submit-job Lambda Lambda Lambda 37
Slide 21
Slide 21 text
SFN = AWS Batch ! ! ! describe-job Lambda Lambda Lambda 38
Slide 22
Slide 22 text
SFN = AWS Batch " $ #! SFN $ Lambda Lambda Lambda 39
Slide 23
Slide 23 text
• •SFNParallel tasks: - name: task-1 module: sleep - parallel: - tasks: - name: task-3 module: sleep - tasks: - name: task-4 module: sleep - tasks: - name: task-5 module: sleep = 40
Slide 24
Slide 24 text
SFN(Parallel) 3 Lambda 41
Slide 25
Slide 25 text
Lambda SFN (Parallel) flatten 42
Slide 26
Slide 26 text
AWS Batch module agent- downloader Instance mount agent Module run.sh execute data- bucket download upload secrets Agent download Agent project-role 43
Slide 27
Slide 27 text
SFNBatch9, %(" •&$1+5E ? •'#( - • 500DF&$1++ 67 !%(" •&$ACAPI>8 •DynamoDB.4@2; •AWS Batch))B3/0 •Fargate= :<* 44
Slide 28
Slide 28 text
46
Slide 29
Slide 29 text
•Crois " $ • KMSIAM • Decrypt! project-role api project-key #%! other-role #%! 47
Slide 30
Slide 30 text
#% •"S3(. !" ! $ •S3 21&), -' • ! IAM! +*01 agent data- bucket download upload project-key project-role Container 01/, 48
Slide 31
Slide 31 text
• ./%"+& •(- #' •IAM! *, api + project-key agent project-role Container $ *,)& 49
Slide 32
Slide 32 text
50
Slide 33
Slide 33 text
API09>=2 •AWS Batch •RegisterJobDefinition@B)( "5- D% • &!%* •Step Functions •4/8:API'!&$!? • #!A3 •IAM RoleKMS Key Policy •Role61,2030;<KMS Key PolicyC.Role* • &!%+7 51
Slide 34
Slide 34 text
SFN &*!+' •SFN Parallel $% &*! +') •Parallel" &*!# (+ 52 Lambda
Slide 35
Slide 35 text
SFN"& '# •S3 ! 53 Lambda Lambda S3%) $ "& '#(
Slide 36
Slide 36 text
։ൃϑϩʔͷ
Slide 37
Slide 37 text
• = In,Out '&Docker •#" ECR#" •ECRpush ! %$ api #" module- registry 55
Slide 38
Slide 38 text
D<5103? >6 •)*-510/% " /!#IT' +/$C •@>6;E: •GitHub, Docker, =79A OK •CI(Drone)84CGF2 •1,(#,1)*-51&. B 56
Slide 39
Slide 39 text
(& •1 ! • ")'* • Drone(CI))'* • /% build, push# • git diff+ • , .- module- registry Drone ghe hook (&$ push push git diff & docker build 57
Slide 40
Slide 40 text
ར༻ࣄྫ
Slide 41
Slide 41 text
BS@Q2A=)-!$'* •CroisP1;4 9MBS@Q2A= )-!$'* •Web(-<1LG • 7O8#6K> • BS@Q+#/ ,%0 • @QFILN= • @QD+#/H ER •CroisAPIC? BS@Q<1: 3CroisJ • -&."50'-C? 59
Slide 42
Slide 42 text
2:#,' •< 2:*2:# ' •Crois( 4 • &<2: %+ • $;"8)!70 0 • 2: ' • -.< 53 •69$/12769 60
Slide 43
Slide 43 text
ىͬͨ͜มԽ
Slide 44
Slide 44 text
2-1.0 •$"+5')#%&!( Crois2- •Crois 4 /* •Crois 4 3, 62
Slide 45
Slide 45 text
.# •Crois *&, ) •+(/'! "$ %- 63
Slide 46
Slide 46 text
͋Γ͕ͱ͏͍͟͝·ͨ͠ We are hiring https://www.rco.recruit.co.jp/career/engineer/
Slide 47
Slide 47 text
+-4' •& •AWS Batch, Step Functions6.0 E/@1"!# 8A •https://www.rco.recruit.co.jp/career/e ngineer/blog/detail_of_job_scheduler_ on_aws/ • < *:4' •)CD3?7= B> 2,(;95 $% %1)C# Crois •http://hatenanews.com/articles/2019/ 03/04/103000 65