データ分析基盤Crois

 データ分析基盤Crois

Machine Learning Casual Talks #10 での発表資料です。

リクルートの横断組織であるリクルートコミュニケーションズで開発しているデータ分析基盤のCroisについて紹介させていただきました。

D72d1e309eeaebbbcd5711d407c02b51?s=128

Recruit Communications

May 29, 2019
Tweet

Transcript

  1. σʔλ෼ੳج൫$SPJT

  2. ΞδΣϯμ

  3. 1. Crois 2. Crois 3.  4.   5.

     15
  4. $SPJTͷ֓ཁ

  5. 17 

  6.    o D n T mf , ,

    ,, , T D T m e c a T T D r k k 18
  7. @7 8-,. •5C3B <'$%* +4 /  !& " 9A

    •>1!) :62?; # $()/ •&'$' =0  * & +49A 19
  8. Crois  • " •  •   

     •!#API 20
  9. 22 

  10.   

  11. = =  =  

  12. = =  =   

  13. (&=  =  %$"=    #'!

  14. $SPJTͷ࢓૊Έ

  15. 32  

  16.   •     YAML tasks: -

    name: task-1 module: sleep params: hoge: fuga - name: task-2 module: sleep 33
  17.    AWS BatchStep Functions •AWS Batch • 

    •Step Functions •  34
  18.  SFN • 11SFN StateMachine    tasks: -

    name: task-1 module: sleep = 35
  19. SFN = Lambda Lambda Lambda 36

  20. SFN =  AWS Batch    submit-job Lambda

    Lambda Lambda 37
  21. SFN =  AWS Batch  !   !

     !   describe-job Lambda Lambda Lambda 38
  22. SFN  =   AWS Batch " $ 

    #!  SFN $ Lambda Lambda Lambda 39
  23.   •  •SFNParallel  tasks: - name: task-1

    module: sleep - parallel: - tasks: - name: task-3 module: sleep - tasks: - name: task-4 module: sleep - tasks: - name: task-5 module: sleep = 40
  24. SFN(Parallel)  3 Lambda 41

  25. Lambda SFN (Parallel)    flatten 42

  26. AWS Batch  module  agent- downloader Instance mount agent

    Module run.sh execute data- bucket download upload secrets Agent     download Agent project-role 43
  27. SFN Batch9, %(" •&$1+5E ? •'#( - • 500DF &$1++

    67   !%(" •&$ACAPI> 8  •DynamoDB.4@2;  •AWS Batch))B3/0  •Fargate= :<* 44
  28. 46     

  29.  •Crois " $ • KMSIAM  •  

    Decrypt! project-role api project-key  #%! other-role #%! 47
  30.  #% •"S3(. !" ! $ •S3 21&), -' •

    ! IAM! +* 01  agent data- bucket download upload project-key project-role Container 01/, 48
  31.  • ./%"+& •(- #' • IAM!  *, 

    api + project-key agent project-role Container $ *,)& 49
  32. 50 

  33. API09>=2  •AWS Batch •RegisterJobDefinition@B) ( "5- D% • &!%*

    •Step Functions •4/8:API'!&$!?  • #!A3  •IAM RoleKMS Key Policy •Role61,2030;< KMS Key PolicyC.Role*  • &!%+7 51
  34. SFN  &*!+' •SFN Parallel $%   &*! +')

    •Parallel"  &*!#  (+  52 Lambda   
  35. SFN"& '# •S3 ! 53 Lambda Lambda   S3%)

    $   "& '#(
  36. ։ൃϑϩʔͷ޻෉

  37.  • = In,Out '&Docker  •#" ECR#" •ECR push

      ! %$ api #" module- registry 55
  38. D<5103? >6 •)*-510/%   " /!#IT' +/$C • @>6

    ;E: •GitHub, Docker, =79A  OK •CI(Drone)84 C GF2  •1,(#,1)*-51&. B 56
  39.  (& •1   ! • ")'*  •

    Drone(CI))'* • /%   build, push# • git diff+ • ,  .- module- registry Drone ghe hook (&$ push push git diff & docker build 57
  40. ར༻ࣄྫ

  41. BS@Q2A=)-!$'* •CroisP1;4 9MBS@Q2A= )-!$'* •Web(-<1LG • 7O8#6K> • BS@Q+#/ ,%0

    • @QFILN= • @QD+#/H ER •CroisAPIC? BS@Q<1: 3CroisJ  • -&."50'- C? 59
  42. 2:#,' •< 2:*2:# '  •Crois( 4  • &<2:

    %+ • $;"8)!70 0  • 2: ' • -.< 53 •69$/12769 60
  43. ىͬͨ͜มԽ

  44. 2-1.0 •$"+5')#%&!( Crois2-  •Crois 4    

    /* •Crois 4   3, 62
  45.  .# •Crois *&,    ) • +(/'!

    "$ %- 63
  46. ͋Γ͕ͱ͏͍͟͝·ͨ͠ We are hiring https://www.rco.recruit.co.jp/career/engineer/

  47. +-4' •& •AWS Batch, Step Functions6.0 E/@1"!# 8A •https://www.rco.recruit.co.jp/career/e ngineer/blog/detail_of_job_scheduler_

    on_aws/ • < *:4' •)CD 3?7= B> 2,(;95 $% %1)C# Crois •http://hatenanews.com/articles/2019/ 03/04/103000 65