データ分析基盤Crois

Machine Learning Casual Talks #10 での発表資料です。

リクルートの横断組織であるリクルートコミュニケーションズで開発しているデータ分析基盤のCroisについて紹介させていただきました。

Transcript

1. σʔλ෼ੳج൫$SPJT
2. ΞδΣϯμ
3. 1. Crois 2. Crois 3. 
    4.   5.  15
4. 
   $SPJTͷ֓ཁ
5. 17 
   
6. 
      o D n T mf , , ,, , T D T m e c a T T D r k k 18
7. @7 8-,. •5C3B <'$%* +4 /  !& " 9A •>1!) :62?; # $()/ •&'
   $' =0  * & +49A 19
8. Crois  • " •  •
   
      
   
   •!#API 20
9. 22
   
10. 
     
    
11. =
     =
    
    = 
    
12. =
     =
    
    = 
    
13. (&=  =  %$"=  
    
    #'!
14. 
    $SPJTͷ࢓૊Έ
15. 32 
    
16.   •  
      YAML tasks: - name: task-1 module: sleep params: hoge: fuga - name: task-2 module: sleep 33
17. 
    
      AWS BatchStep Functions •AWS Batch •  •Step Functions •
    
     34
18. 
    
    SFN •
    
    11SFN StateMachine    tasks: - name: task-1 module: sleep = 35
19. SFN
    = Lambda Lambda Lambda 36
20. SFN
    =  AWS Batch 
       submit-job Lambda Lambda Lambda 37
21. SFN
    = 
     AWS Batch 
     !   !  !  
     describe-job Lambda Lambda Lambda 38
22. SFN 
    =   AWS Batch " $  #!  SFN $ Lambda Lambda Lambda 39
23.  
    
    •  •SFNParallel  tasks: - name: task-1 module: sleep - parallel: - tasks: - name: task-3 module: sleep - tasks: - name: task-4 module: sleep - tasks: - name: task-5 module: sleep = 40
24. SFN
    (Parallel) 
    3 Lambda 41
25. Lambda SFN 
    (Parallel)    flatten 42
26. AWS Batch 
    module  agent- downloader Instance mount agent Module run.sh execute data- bucket download upload secrets Agent 
     
       download Agent project-role 43
27. SFN Batch9, %(" •&$1+5E ? •
    '#( - • 500DF &$1++ 67   !%(" •&$ACAPI> 8  •DynamoDB.4@2;  •AWS Batch))B3/0  •Fargate= :<* 44
28. 46
       
29.  •Crois 
    " $ • KMS
    IAM
     •
     
     Decrypt! project-role api project-key  #%! other-role #%! 47
30.  #% •"S3(. !
    " ! $ •S3 21&), -' • ! IAM! +* 01  agent data- bucket download upload project-key project-role Container 01/, 48
31. 
     • ./%"+& •(- #' • IAM
    !  *,  api + project-key agent project-role Container $ *,)& 49
32. 50 
    
33. API09>=2  •AWS Batch •RegisterJobDefinition@B) ( "5- D%
    • &!%* •Step Functions •4/8:API'
    !&$!?  • #
    !A3  •IAM RoleKMS Key Policy •Role61,2030;< KMS Key PolicyC.Role*  • &!%+7 51
34. SFN  &*!+' •SFN Parallel $%   &*! +')
    •Parallel"  &*!#  (+
     52 Lambda   
    
35. SFN"& '# •S3
     ! 53 Lambda Lambda   S3%) $   "& '#(
36. 
    ։ൃϑϩʔͷ޻෉
37. 
     •
     = In,Out '&Docker  •
    #" ECR#" •ECR push   ! %$ api #" module- registry 55
38. D<5103? >6 •)*
    -510/%   "
    /!#IT' +/$
    C • @>6 ;E: •GitHub, Docker, =79A  OK •CI(Drone)84 C GF2  •1,(#,1)*
    -51&.
    B 56
39.  (& •1   ! • 
    ")'*  • Drone(CI))'* • /%   build, push# • git diff+ • ,  .- module- registry Drone ghe hook (&$ push push git diff & docker build 57
40. 
    ར༻ࣄྫ
41. BS@Q2A=)-!$'
    * •CroisP1;4 9MBS@Q2A= )-!$'
    * •Web(-<1LG • 7O8#
    6K> • BS@Q+#/ ,
    %0 • @QFILN= • @QD+#/H ER •CroisAPIC? BS@Q<1: 3CroisJ  • 
    -&."50'- C? 59
42. 
    2:#,' •< 2:*
    2:# '  •Crois( 4  • &<2: %+ • $;"8)!70 0  • 
    2: ' • -.< 53 •69$/12769 60
43. 
    ىͬͨ͜มԽ
44. 2-1.0 •$"+5')#%&!( Crois2-  •Crois 4     /* •Crois 4   3, 62
45.  .# •Crois *&,    ) • 
    +(/'! "$ %- 63
46. ͋Γ͕ͱ͏͍͟͝·ͨ͠ We are hiring https://www.rco.recruit.co.jp/career/engineer/
47. +-4' •& •AWS Batch, Step Functions6.0 E/@1"!# 8A •https://www.rco.recruit.co.jp/career/e ngineer/blog/detail_of_job_scheduler_ on_aws/ • < *:4' •)CD
    3?7= B> 2,(;95 $% %1)C# Crois •http://hatenanews.com/articles/2019/ 03/04/103000 65