Rayleigh: 分散処理フレームワークRayの管理プラットフォーム / Rayleigh: A Management Platform for Ray Clusters

Machine Learning Platform 部では、多種多様なサービスに対して、機械学習を活用した課題解決を提供しています。近年では、機械学習モデルの大規模化・高度化に伴い、Ray を用いた分散処理に取り組んできました。その中で、Ray の実運用を支える基盤として Rayleigh を開発し、開発スピードの向上と高いセキュリティレベルの両立を実現しています。
本発表では、私たちがなぜ Ray を選択したのか、Rayleigh がどのような課題を解決しているのかといった観点から、実際の取り組みについてご紹介します。

Transcript

1. 3BZMFJHI
   "
   .BOBHFNFOU
   1MBUGPSN
   GPS
   3BZ
   $MVTUFST ,B[VLJ
   :PTIJEB .BDIJOF
   -FBSOJOH
   1MBUGPSN
   %FQBSUNFOU
2. Software Engineer @ Machine Learning Platform Department Kazuki Yoshida
3.  0VS
   5FBN  #BDLHSPVOE
   BOE
   $IBMMFOHFT  3BZ
   BOE
   3BZMFJHI  1SFTFOU  'VUVSF
   "HFOEB
4.  0VS
   5FBN  #BDLHSPVOE
   BOE
   $IBMMFOHFT  3BZ
   BOE
   3BZMFJHI  1SFTFOU  'VUVSF
   "HFOEB
5. .BDIJOF
   -FBSOJOH
   1MBUGPSN
   %FQBSUNFOU • "QQSPY
   
   QFPQMF
   UPUBM • *OEFQFOEFOU
   GSPN
   TFSWJDFEFW
   EFQUT • 1SPWJEF
   .-
   TPMVUJPOT GPS
   NVMUJQMF
   TFSWJDFT .BDIJOF
   -FBSOJOH 4FSWFSTJEF *OGSBTUSVDUVSF
6. %BUB
   BOE
   0VUQVU
   4DBMF FY-*/&
   .-
   TPMVUJPOT
   TVQQPSUFE
   CZ
   .BDIJOF
   -FBSOJOH
   1MBUGPSN
   %FQBSUNFOU 4VQQPSUT  "OBMZ[F
   -PHT
   PG . 3VOT  4FSWJDFT +PCT
   1FS
   %BZ (MPCBM
   6TFST
7. 1SPEVDUT
   GSPN
   .BDIJOF
   -FBSOJOH
   1MBUGPSN .BDIJOF
   -FBSOJOH 1MBUGPSN .-
   "1* -JGGZ 0'4 ʜ 4NBSU
   $IBOOFM /&84 45*$,&34 .64*$ "% 0OMJOF
   'FBUVSF
   4UPSF
   0'4 %BUB
   TUPSF
   PSJFOUFE
   UP
   IJHI UISPVHIQVU
   
   MPXMBUFODZ
   SBOEPN
   BDDFTT -JGGZ "
   VOJGJFE
   "#
   UFTUJOH
   QMBUGPSN
   UIBU
   JOUFHSBUFT
   UIPTF
   QSFWJPVTMZ
   PQFSBUFE
   CZ
   -*/&
   BOE
   :BIPP
   +BQBO .-
   "1* "VUP.-MJLF
   1MBUGPSN
8.  0VS
   5FBN  #BDLHSPVOE
   BOE
   $IBMMFOHFT  3BZ
   BOE
   3BZMFJHI  1SFTFOU  'VUVSF
   "HFOEB
9. .-
   4ZTUFN
   0WFSWJFX /FXT 4UJDLFST 0GGJDJBM "DDPVOU 3FSBOLFS 3FUSJFWFS 'FBUVSF
   &YUSBDUJPO 6TFS 'FBUVSFT *UFN 'FBUVSFT ・・・ "// *OEFY 'FBUVSFT 3BUJOH "QQ
   -PH 6TFS
   -PH &5- 1SFQSPDFTT 4FSWJDFT $MJFOU
   "QQ
10. $IBMMFOHFT
    JO
     $IBMMFOHFT $PME
    4UBSU
    1SPCMFN %BUB
    BOE
    .PEFMT 'SBHNFOUFE
    "DSPTT
    4FSWJDFT /FBS
    3FBMUJNF
    3FDPNNFOEBUJPO -JNJUFE
    *OQVU
    *OGPSNBUJPO
11. $IBMMFOHFT
    JO
     $IBMMFOHFT .VMUJ.PEBM .VMUJ%PNBJO 3PCVTU 'PVOEBUJPO
    .PEFM
    4PMVUJPOT $PME
    4UBSU
    1SPCMFN %BUB
    BOE
    .PEFMT 'SBHNFOUFE
    "DSPTT
    4FSWJDFT /FBS
    3FBMUJNF
    3FDPNNFOEBUJPO -JNJUFE
    *OQVU
    *OGPSNBUJPO
12. $IBMMFOHFT
    JO
     $PME
    4UBSU
    1SPCMFN %BUB
    BOE
    .PEFMT 'SBHNFOUFE
    "DSPTT
    4FSWJDFT /FBS
    3FBMUJNF
    3FDPNNFOEBUJPO -JNJUFE
    *OQVU
    *OGPSNBUJPO $IBMMFOHFT .VMUJ.PEBM .VMUJ%PNBJO 3PCVTU 'PVOEBUJPO
    .PEFM
    4PMVUJPOT .FEJVN
    
    -BSHF .PEFMT 4UBUFPGUIFBSU 5FDIOPMPHZ 3FRVJSFNFOUT .-1
    
    
    #&35
    (159
     %JTUSJCVUFE
    5SBJOJOH .FNPSZ
    0QUJNJ[BUJPO
13. 3BUJOH "QQ
    -PH .-
    4ZTUFN
    0WFSWJFX 4FSWJDFT $MJFOU
    "QQ 6TFS
    -PH /FXT 4UJDLFST 6TFS 'FBUVSFT *UFN 'FBUVSFT ・・・ "// *OEFY 0GGJDJBM "DDPVOU 3FSBOLFS 3FUSJFWFS 'FBUVSF
    &YUSBDUJPO 'FBUVSFT &5- 1SFQSPDFTT
14. %JTUSJCVUFE
    $PNQVUJOH
    -JCSBSJFT • *OIPVTF
    1ZUIPO
    MJCSBSZ • 1FSGPSNT
    EJTUSJCVUFE
    DPNQVUJOH • .BZ
    IBWF
    MJNJUFE
    DPNQBUJCJMJUZ
    XJUI
    TPNF
    UIJSEQBSUZ
    MJCSBSJFT (IFF ˞ *U`T
    TUJMM
    HPJOH
    TUSPOH ※1. https://linedevday.linecorp.com/2020/en/sessions/9750/
15. %JTUSJCVUFE
    $PNQVUJOH
    -JCSBSJFT • "O
    PQFOTPVSDF
    GSBNFXPSL
    GPS
    FBTZ
    EJTUSJCVUFE
    DPNQVUJOH • 1PXFST
    GBTU
    TDBMBCMF
    .-
    BOE
    EBUB
    QSPDFTTJOH • 4JNQMF
    DPEF
    FGGJDJFOU
    FYFDVUJPO • 8PSLT
    XFMM
    XJUI
    UIJSEQBSUZ
    MJCSBSJFT
    FH
    %FFQ4QFFE -FU`T
    HJWF
    JU
    B
    USZ 3BZ
16. 3BUJOH "QQ
    -PH .-
    4ZTUFN
    0WFSWJFX 4FSWJDFT $MJFOU
    "QQ 6TFS
    -PH /FXT 4UJDLFST 6TFS 'FBUVSFT *UFN 'FBUVSFT ・・・ "// *OEFY 0GGJDJBM "DDPVOU 3FSBOLFS 3FUSJFWFS 'FBUVSF
    &YUSBDUJPO 'FBUVSFT &5- 1SFQSPDFTT
17. 3BUJOH "QQ
    -PH .-
    4ZTUFN
    0WFSWJFX 4FSWJDFT $MJFOU
    "QQ 6TFS
    -PH /FXT 4UJDLFST 6TFS 'FBUVSFT *UFN 'FBUVSFT ・・・ "// *OEFY 0GGJDJBM "DDPVOU 3FSBOLFS 3FUSJFWFS 'FBUVSF
    &YUSBDUJPO 'FBUVSFT &5- 1SFQSPDFTT
18.  0VS
    5FBN  #BDLHSPVOE
    BOE
    $IBMMFOHFT  3BZ
    BOE
    3BZMFJHI  1SFTFOU  'VUVSF
    "HFOEB
19. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ 1SPKFDU
    /BNFTQBDF $PNQBOZXJEF
    ,VCFSOFUFT 3BZ
    $MVTUFS )FBE 8PSLFS 8PSLFS ,VCF3BZ 3BZ$MVTUFS
    $3
20. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ 1SPKFDU
    /BNFTQBDF $PNQBOZXJEF
    ,VCFSOFUFT 3BZ
    $MVTUFS )FBE 8PSLFS 8PSLFS ⓪ 4FUVQ
    LVCFDUM ① 1PSU
    GPSXBSE
    UP
    UIF
    IFBE
    OPEF ② 4VCNJU
    KPCT
    "DDFTT
    UP
    3BZ
    %BTICPBSE
21. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ $PNQBOZXJEF
    ,VCFSOFUFT 3BZ
    $MVTUFS )FBE 8PSLFS 8PSLFS 1SPKFDU
    /BNFTQBDF
22. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ $PNQBOZXJEF
    ,VCFSOFUFT 3BZ
    $MVTUFS )FBE 8PSLFS 8PSLFS %FW
    4QFFE 1SPKFDU
    /BNFTQBDF
23. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ $PNQBOZXJEF
    ,VCFSOFUFT )%'4 4UPSBHF
    "DDFTT 3BZ
    $MVTUFS )FBE 8PSLFS 8PSLFS %FW
    4QFFE 1SPKFDU
    /BNFTQBDF
24. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ $PNQBOZXJEF
    ,VCFSOFUFT )%'4 -PH
    4UPSBHF 4UPSBHF
    "DDFTT -PH
    1FSTJTUFODF 3BZ
    $MVTUFS )FBE 8PSLFS 8PSLFS %FW
    4QFFE 1SPKFDU
    /BNFTQBDF
25. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ $PNQBOZXJEF
    ,VCFSOFUFT )%'4 -PH
    4UPSBHF 4UPSBHF
    "DDFTT -PH
    1FSTJTUFODF 0CTFSWBCJMJUZ 3BZ
    $MVTUFS )FBE 8PSLFS 8PSLFS %FW
    4QFFE 1SPKFDU
    /BNFTQBDF
26. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ 1SPKFDU
    /BNFTQBDF $PNQBOZXJEF
    ,VCFSOFUFT 3BZ
    $MVTUFS 3BZ
    $MVTUFS 1FSTPOB M 4FDSFUT 1FSTPOB M 4FDSFUT
27. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ $PNQBOZXJEF
    ,VCFSOFUFT 3BZ
    $MVTUFS 3BZ
    $MVTUFS 1FSTPOB M 4FDSFUT 1FSTPOB M 4FDSFUT 1SPKFDU
    /BNFTQBDF
28. 7FSEB*6
    ,VCFSOFUFT &BSMZ
    5SJBMT
    XJUI
    3BZ $PNQBOZXJEF
    ,VCFSOFUFT 3BZ
    $MVTUFS 3BZ
    $MVTUFS 1FSTPOB M 4FDSFUT 1FSTPOB M 4FDSFUT 1SPKFDU
    /BNFTQBDF 3FTPVSDF
    2VPUB
29. 7FSEB*6
    ,VCFSOFUFT 3BZ
    JT
    QPXFSGVM
    3BZ
30. 7FSEB*6
    ,VCFSOFUFT 3BZ
    JT
    QPXFSGVM
    Š FWFO
    NPSF
    DBQBCMF
    XJUI
    3BZMFJHI 3BZ
    3BZMFJHI
31. 3BZMFJHI
    "
    .BOBHFNFOU
    1MBUGPSN
    GPS
    3BZ
    $MVTUFST 3BZ
    $MVTUFS 3BZMFJHI
    1SPYZ 3BZMFJHI
    4FSWFS $SFBUF %FMFUF 4VCNJU
    B
    KPC "DDFTT
    UP
    %BTICPBSE
32. $-*
    
    4%, 3BZMFJHI
    4ZTUFN
    "SDIJUFDUVSF 3BZ$MVTUFS $3 4FSWFS Y%4
    4FSWFS 1SPYZ 3BZMFJHI $SFBUF %FMFUF /FUXPSL
    1PMJDZ )FBE 8PSLF S 8PSLF S 3BZ
    $MVTUFS ,VCF3BZ 5BSHFU
    /BNFTQBDF
33. $-*
    
    4%, 3BZMFJHI
    4ZTUFN
    "SDIJUFDUVSF )FBE 8PSLF S 8PSLF S 3BZ
    $MVTUFS 3BZ$MVTUFS $3 4FSWFS /FUXPSL
    1PMJDZ Y%4
    4FSWFS 1SPYZ %BTICPBSE 4VCNJU
    KPCT 3BZMFJHI "VUI[
    
    3PVUJOH
34. %FQMPZ
    3BZ
    $MVTUFST
    "OZXIFSF PO 7FSE B FY-*/&
    $PSQPSBUJPO FY:BIPP
    +BQBO
    $PSQPSBUJPO -:
    $PSQPSBUJPO
    PO *6 PO "$1 PO 'MBWB $VSSFOUMZ 4VQQPSUFE ,VCFSOFUFT
    4FSWJDFT
    PO
    *O)PVTF
    1SJWBUF
    $MPVE
35. %FQMPZ
    3BZ
    $MVTUFST
    CZ
    "OZPOF 'PS
    3BZ
    #FHJOOFST
36. %FQMPZ
    3BZ
    $MVTUFST
    CZ
    "OZPOF 'PS
    3BZ
    #FHJOOFST
37. %FQMPZ
    3BZ
    $MVTUFST
    CZ
    "OZPOF 'PS
    3BZ
    #FHJOOFST
38. %FQMPZ
    3BZ
    $MVTUFST
    CZ
    "OZPOF 'PS
    3BZ
    "EWBODFE
    6TFST
39. 4FDVSF
    JOUFSBDUJPO
    XJUI
    3BZ
    +PCT
    "1* $PNQBUJCMF
    XJUI
    3BZ
    PGGJDJBM
    $-*
40. 4FDVSF
    "DDFTT
    UP
    3BZ
    %BTICPBSE
41. 4VQQPSU
    UIF
    XIPMF
    EFW
    QSPDFTT 1SPEVDUJPO 4DIFEVMFE
    +PC +PC
    4VCNJUUF S 3BZ
    $MVTUFS 3BZ+PC
    $3 ,VCF3BZ %FWFMPQ 3BZ$MVTUFS
    $3 3FTFBSDI 3BZ
    $MVTUFS ,VCF3BZ
42. $-*
    
    4%, 3BZMFJHI
    4ZTUFN
    "SDIJUFDUVSF )FBE 8PSLF S 8PSLF S 3BZ
    $MVTUFS 3BZ+PC $3 4FSWFS 4VCNJUUF S 4VQQPSU 3BZ+PC
43. $-*
    
    4%, 3BZMFJHI
    4ZTUFN
    "SDIJUFDUVSF )FBE 8PSLF S 8PSLF S 3BZ
    $MVTUFS 3BZ+PC $3 4FSWFS 4VCNJUUF S -PH
    TUPSBHF "QQMJDBUJPO
    -PH
     3BZ
    4ZTUFN
    -PH
    1FSTJTUFODF 4JEFDB S 4JEFDB S 4JEFDB S 4JEFDB S
44.  0VS
    5FBN  #BDLHSPVOE
    BOE
    $IBMMFOHFT  3BZ
    BOE
    3BZMFJHI  1SFTFOU  'VUVSF
    "HFOEB
45. 1SFTFOU
    3BZMFJHI
    JT
    TUJMM
    JO
    UIF
    USJBM
    QIBTF 3BZ
    
    3BZMFJHI
    
    
    &OWJSPONFOU l7FDUPS
    'PSHFz
    1SPKFDU .BDIJOF
    -FBSOJOH
    &OHJOFFST 3BZMFJHI
    "ENJOT 
    6TFST 
    3BZ
    $MVTUFST
46. 7FDUPS
    'PSHF
    1SPKFDU • "
    QSPKFDU
    UP
    EFWFMPQ
    B
    DPNNPO
    SPCVTU
    NPEFM
    GPS
    HFOFSBUJOH
    GFBUVSFT
    GPS
    SFDPNNFOEBUJPO
    TZTUFNT *UFN 6TFS
    -PH 'FBUVSF 3BZ
    
    %FFQ4QFFE .PEFM https://tech-verse.lycorp.co.jp/2025/ja/session/1128/
47. 7PJDF
    PG
    UIF
    $VTUPNFS Things that are great about using Rayleigh ・MLEs can easily set up a distributed training environment in no time. This is a significant advantage. ・MLEs can train models in the same environment, ensuring reproducibility from an infrastructure standpoint.
48. "DDPNQMJTINFOUT
    PG
    UIF
    7FDUPS
    'PSHF
    1SPKFDU     
49.  0VS
    5FBN  #BDLHSPVOE
    BOE
    $IBMMFOHFT  3BZ
    BOE
    3BZMFJHI  1SFTFOU  'VUVSF "HFOEB
50. #FDPNF
    B
    1BSU
    PG
    UIF
    #VJMEJOH
    #MPDLT 7FSEB *6 "$1 'MBWB ,VCF3BZ 3BZMFJHI ,VCF3BZ 3BZ .-
    "1* 7FDUPS
    'PSHF :BNMB .- GPS
    -*/&
    /&84 .-
    GPS
    -*/&
    45*$,&34
    ・・・ ・・・ "QQT
    GPS
    4FSWJDF .-
    1SPEVDUT %JTUSJCVUFE
    1SPDFTTJOH
    &OHJOF $PNQVUJOH （1SJWBUF
    $MPVE） 3BZ
    PO -BLFUBIPF (IFF
51. *EFBT
    GPS
    3FGBDUPSJOH 3BZ
    $MVTUFS 4FSWFS 3&45 Y%4
    4FSWFS H31$ QSPYZ $-* 3%# W 3BZ
    $MVTUFS 3BZ
    $MVTUFS 4FSWFS H31$ QSPYZ $-* 3%# W 3BZ
    $MVTUFS $POGJHT
52.  0VS
    .BDIJOF
    -FBSOJOH
    1MBUGPSN
    %FQBSUNFOU
    QSPWJEFT
    .-
    TPMVUJPOT
    GPS
    B
    XJEF
    SBOHF
    PG
    TFSWJDFT
    BOE
    JT
    DPOUJOVPVTMZ
    XPSLJOH
    UP
    JNQSPWF
    UIFN $PODMVTJPO  *O
    UIJT
    QSPKFDU
    XF
    BEPQUFE
    3BZ
    UP
    FGGJDJFOUMZ
    FYQFSJNFOU
    XJUI
    NVMUJ NPEFM
    TFUVQT
    BOE
    SFMBUJWFMZ
    MBSHF
    NPEFMT  5P
    FOIBODF
    CPUI
    EFWFMPQFS
    QSPEVDUJWJUZ
    BOE
    TFDVSJUZ
    XIFO
    VTJOH
    3BZ
    JO
    PVS
    FOWJSPONFOU
    XF
    EFWFMPQFE
    3BZMFJHI
    XIJDI
    JO
    UVSO
    DPOUSJCVUFE
    UP
    QSPKFDUT
    MJLF
    7FDUPS
    'PSHF
53. 5IBOL
    ZPV