データ分析基盤Crois

Machine Learning Casual Talks #10 での発表資料です。

リクルートの横断組織であるリクルートコミュニケーションズで開発しているデータ分析基盤のCroisについて紹介させていただきました。

Transcript

1. σʔλ෼ੳج൫$SPJT
   

   View Slide

2. ΞδΣϯμ
   

   View Slide

3. 1. Crois
   2. Crois
   3.
   
   4.
   5.
   15
   

   View Slide

4. $SPJTͷ֓ཁ
   

   View Slide

5. View Slide

6. 
   
   o
   D
   n
   T
   mf
   , ,
   ,, , T
   D T
   m
   e
   c a T
   T
   D
   r
   k
   k
   18
   

   View Slide

7. @7
   8-,.
   •5C3B <'$%*
   +4
   /
   
   !& "
   9A
   •>1!) :62?; #
   $()/
   •&'$'
   =0
   * &
   +49A
   19
   

   View Slide

8. Crois
   •"
   •
   
   •
   
   •!#API
   20
   

   View Slide

9. View Slide

10. 
    
    

    View Slide

11. =
    =
    =
    
    

    View Slide

12. =
    =
    =
    
    
    

    View Slide

13. (&=
    =
    %$"=
    
    
    
    #'!
    

    View Slide

14. $SPJTͷ࢓૊Έ
    

    View Slide

15. View Slide

16. 
    
    •
    YAML
    tasks:
    - name: task-1
    module: sleep
    params:
    hoge: fuga
    - name: task-2
    module: sleep
    33
    

    View Slide

17. 
    AWS BatchStep Functions
    •AWS Batch
    •
    
    •Step Functions
    •
    34
    

    View Slide

18. SFN
    •11SFN
    StateMachine
    
    
    tasks:
    - name: task-1
    module: sleep
    =
    35
    

    View Slide

19. SFN
    = Lambda
    Lambda
    Lambda
    36
    

    View Slide

20. SFN
    =
    
    AWS Batch
    
    submit-job
    Lambda
    Lambda
    Lambda
    37
    

    View Slide

21. SFN
    =
    
    AWS Batch
    !
    !
    
    !
    describe-job
    Lambda
    Lambda
    Lambda
    38
    

    View Slide

22. SFN
    
    =
    
    
    AWS Batch
    " $
    
    #! SFN $
    Lambda
    Lambda
    Lambda
    39
    

    View Slide

23. 
    
    •
    
    •SFNParallel
    tasks:
    - name: task-1
    module: sleep
    - parallel:
    - tasks:
    - name: task-3
    module: sleep
    - tasks:
    - name: task-4
    module: sleep
    - tasks:
    - name: task-5
    module: sleep
    =
    40
    

    View Slide

24. SFN(Parallel)
    
    3
    Lambda
    41
    

    View Slide

25. Lambda
    SFN (Parallel)
    
    
    flatten
    42
    

    View Slide

26. AWS Batch
    
    module
    
    agent-
    downloader
    Instance
    mount
    agent
    Module
    run.sh
    execute
    data-
    bucket
    download
    upload
    secrets
    Agent
    download
    Agent
    project-role
    43
    

    View Slide

27. SFNBatch9,
    %("
    •&$1+5E
    ?
    •'#( -
    • 500DF&$1++
    67
    !%("
    •&$ACAPI>8
    
    •DynamoDB.4@2;
    •AWS Batch))B3/0
    
    •Fargate= :<*
    44
    

    View Slide

28. View Slide

29. 
    •Crois
    "
    $
    • KMSIAM
    
    •
    Decrypt!
    project-role
    api
    project-key
    
    #%!
    other-role
    #%!
    47
    

    View Slide

30. 
    #%
    •"S3(.
    !"
    ! $
    •S3
    21&), -'
    • !
    IAM! +*01
    agent
    data-
    bucket
    download
    upload project-key
    project-role
    Container
    01/,
    48
    

    View Slide

31. 
    •
    ./%"+&
    •(- #'
    •IAM! *,
    api
    + project-key
    agent
    project-role
    Container
    $
    *,)&
    49
    

    View Slide

32. View Slide

33. API09>=2
    •AWS Batch
    •RegisterJobDefinition@B)(
    "5-
    D%
    • &!%*
    •Step Functions
    •4/8:API'!&$!?
    
    • #!A3
    
    •IAM RoleKMS Key Policy
    •Role61,2030;<KMS Key
    PolicyC.Role*
    • &!%+7
    51
    

    View Slide

34. SFN &*!+'
    •SFN Parallel $%
    
    &*! +')
    •Parallel"
    &*!#
    (+
    52
    Lambda
    
    

    View Slide

35. SFN"& '#
    •S3
    
    !
    53
    Lambda
    Lambda
    
    
    S3%)
    $
    
    "& '#(
    

    View Slide

36. ։ൃϑϩʔͷ޻෉
    

    View Slide

37. 
    • = In,Out '&Docker
    
    •#" ECR#"
    •ECRpush
    ! %$
    api
    #"
    module-
    registry
    55
    

    View Slide

38. D<5103? >6
    •)*-510/%
    
    " /!#IT'
    +/$C
    •@>6;E:
    •GitHub, Docker, =79A OK
    •CI(Drone)84CGF2
    
    •1,(#,1)*-51&.
    B
    56
    

    View Slide

39. (&
    •1 !
    • ")'*
    • Drone(CI))'*
    • /%
    build, push#
    • git diff+
    • , .-
    module-
    registry
    Drone
    ghe
    hook
    (&$
    push push
    git diff & docker build
    57
    

    View Slide

40. ར༻ࣄྫ
    

    View Slide

41. BS@Q2A=)-!$'*
    •CroisP1;4
    9MBS@Q2A=
    )-!$'*
    •Web(-<1LG
    • 7O8#6K>
    • BS@Q+#/ ,%0
    • @QFILN=
    • @QD+#/H
    ER
    •CroisAPIC? BS@Q<1:
    3CroisJ
    • -&."50'-C?
    59
    

    View Slide

42. 2:#,'
    •< 2:*2:#
    '
    •Crois( 4
    • &<2: %+
    • $;"8)!70
    0
    • 2: '
    • -.<
    53
    •69$/12769
    60
    

    View Slide

43. ىͬͨ͜มԽ
    

    View Slide

44. 2-1.0
    •$"+5')#%&!(
    Crois2-
    •Crois 4
    
    /*
    •Crois 4
    3,
    62
    

    View Slide

45. 
    .#
    •Crois
    *&,
    
    )
    •+(/'!
    "$ %-
    63
    

    View Slide

46. ͋Γ͕ͱ͏͍͟͝·ͨ͠
    We are hiring
    https://www.rco.recruit.co.jp/career/engineer/
    

    View Slide

47. +-4'
    •&
    •AWS Batch, Step Functions6.0
    E/@1"!# 8A
    •https://www.rco.recruit.co.jp/career/e
    ngineer/blog/detail_of_job_scheduler_
    on_aws/
    • <
    *:4'
    •)CD3?7= B>
    2,(;95 $%
    %1)C#
    Crois
    •http://hatenanews.com/articles/2019/
    03/04/103000
    65
    

    View Slide