推薦システムと、RayやDeepSpeedを用いたユーザー・アイテムの次世代の埋め込み表現 / Recommender System and Next-Gen. Embedding Representations of Users and Items Using Ray and DeepSpeed

*OUSPEVDUJPOUPUIF7FDUPS'PSHF1SPKFDU 3FDPNNFOEFS4ZTUFNBOE/FYU(FO&NCFEEJOH3FQSFTFOUBUJPOTPG6TFST BOE*UFNT6TJOH3BZBOE%FFQ4QFFE -:$PSQPSBUJPO %BUB4DJFODF(SPVQ ,BUTVZB*JEB

8IPBSFXF $VSSFOUSFDTZTGPS-*/&/&84 7FDUPS'PSHF /FXWFDUPSGFBUVSFT 7FDUPS'PSHFNPEFMBSDIJUFDUVSF
7FDUPS'PSHF#FODINBSL 4VNNBSZBOEGVUVSFQMBOT "HFOEB Ref: Rayleigh: A Management Platform for Ray clusters https://tech-verse.lycorp.co.jp/2025/ja/session/1022/

• +PJOUUFBNPG.-UFBNTGSPN-*/&BOE:BIPP+BQBO • 0OFPGNJTTJPOTJTUPEFMJWFSDPNQBOZXJEFVTFSJUFNWFDUPSTGPS .-UBTLT • .-FOHJOFFSTXJUIFYQFSJFODFEXJUISFDPNNFOEFSTZTUFNT • *BMTPXPSLPO-*/&/&84SFDPNNFOEBUJPO 8IPBSFXF
*OUSPEVDUJPOPG7FDUPS'PSHF1SPKFDU

$VSSFOUSFDTZTGPS-*/&/&84 $VSSFOUSFDTZTGPS-*/&/&84 • 8IPBSFXF • $VSSFOUSFDTZTGPS-*/&/&84 • 7FDUPS'PSHF /FXWFDUPSGFBUVSFT •
7FDUPS'PSHFNPEFMBSDIJUFDUVSF • 7FDUPS'PSHF#FODINBSL • 4VNNBSZBOEGVUVSFQMBOT

• 3FDPNNFOEBUJPOJTCBTFEPOVTFSMPHBOEJUFN • 6TFSMPHEBUBBOEJUFNEBUBBSFQSPDFTTFEUISPVHITFWFSBMTUBHFT 3FDPNNFOEBUJPOQJQFMJOF $VSSFOUSFDTZTGPS-*/&/&84 6TFSMPH EBUB *UFNEBUB Feature
extraction Retrieval model Ranker model raw features raw + processed features item candidates end users Heuristic post-process Vector Forge focuses this area

'FBUVSFFYUSBDUJPO $VSSFOUSFDTZTGPS-*/&/&84 6TFSMPH EBUB *UFNEBUB Feature extraction Retrieval model Ranker
model end users Heuristic post-process

• 0GUFODPOTVNJOHSBXGFBUVSFTEJSFDUMZJTJOFGGJDJFOU • 3BXGFBUVSFTDBOCFIJHIEJNFOTJPOBMBOEDPNQMFY • /POTUSVDUVSFE JNBHFUFYUBOEMBSHFOVNCFSPGJUFNT • 8FQSPEVDFVTFSJUFNFNCFEEJOH WFDUPS
'FBUVSFFYUSBDUJPO $VSSFOUSFDTZTGPS-*/&/&84 6TFSMPH EBUB *UFNEBUB Feature extraction user_1 : [-6.7, 4.9, ..., 3.8] user_2 : [ 4.8, 9.9, ..., -9.7] user_3 : [-8.4, -3.8, ..., -4.7] user_4 : [ 7.5, 8. , ..., 3.4] item_1 : [-9.2, 9.8, ..., 2.9] item_2 : [-3.6, 9.2, ..., -1.8] item_3 : [-4.8, -7.6, ..., 6.4]

• 8FIBWFGFBUVSFFYUSBDUJPONPEFMTUIBUBSFVTFEGPSWBSJPVTUBTLT JODMVEJOH-*/&/&84SFDPNNFOEBUJPO 'FBUVSFFYUSBDUJPO $VSSFOUSFDTZTGPS-*/&/&84 6TFS*UFN EBUBGPS-*/& (JGU -*/&(JGU user_1
: [-6.7, 4.9, ..., 3.8] ... 6TFS*UFN EBUBGPS-*/& 4UJDLFS 6TFS EFNPHSBQIZ -*/&4UJDLFS user_1 : [ 4.8, 9.9, ..., -9.7] ... 6TFS EFNPHSBQIZ user_1 : [-8.4, -3.8, ..., -4.7] ... Each feature extraction model produces a different vector for each user. Training frequency Every week Prediction frequency Every day Feature extraction models

3FUSJFWBMNPEFM $VSSFOUSFDTZTGPS-*/&/&84 6TFSMPH EBUB *UFNEBUB Feature extraction Retrieval model Ranker
model end users Heuristic post-process

3FUSJFWBMNPEFM $VSSFOUSFDTZTGPS-*/&/&84 6TFSMPH EBUBGPS-*/& /&84 *UFNEBUBGPS -*/&/&84 • 'PS-*/&/&84 XFVTF*%CBTFEFNCFEEJOHUXPUPXFS%//
BSDIJUFDUVSFXJUIBEEFEVTFSWFDUPSGFBUVSFT CVUXJUIPVUJUFNWFDUPS GFBUVSFT User tower Item tower List of viewed items Target items User vector Item vector Contrastive loss 6TFSWFDUPSGPS -*/&0GGJDJBM "DDPVOU 6TFSWFDUPSGPS -*/&"ET User/item vectors from retrieval model are different from user/item vectors from feature extraction Item embedding User features

$PMEJUFNQSPCMFN $VSSFOUSFDTZTGPS-*/&/&84 • 8FXBOUUPSFDPNNFOEMBUFTUOFXTBSUJDMFT CVUOFXBSUJDMFTIBWF UPPGFXVTFSJOUFSBDUJPOT • $#$'"/FVSBM.VMUJWJFX$POUFOUUP$PMMBCPSBUJWF'JMUFSJOH.PEFM GPS$PNQMFUFMZ$PME*UFN3FDPNNFOEBUJPOT CVU
*UFNWFDUPSGSPN $'NPEFM *UFNEBUB UFYUEBUB Text BERT Item text data Predicted item vector Text BERT model produces item vector from text content 1 https://arxiv.org/pdf/1611.00384

• *OSFUSJFWBMTUBHF DBOEJEBUFJUFNTGSPN*%CBTFENPEFMBOE$#$' NPEFMBSFNJYFEBOETFOEUPUIFSBOLFSTUBHF • *%CBTFENPEFMJTGBDUCBTFE*UJTHPPEBUPMEJUFNT • $#$'NPEFMVTFTUFYUEBUB*UJTHPPEBUOFXJUFNT .JYJOH*%CBTFEBOE$#$' $VSSFOUSFDTZTGPS-*/&/&84
ID-based model CB2CF model 6TFSWFDUPS *%CBTFE *UFNWFDUPS *%CBTFE *UFNWFDUPS $#$' "//MPPLVQ *%CBTFE "//MPPLVQ $#$' Ranker .JYFE DBOEJEBUFT

7FDUPS'PSHF /FXWFDUPSGFBUVSFT 7FDUPS'PSHF /FXWFDUPSGFBUVSFT • 8IPBSFXF • $VSSFOUSFDTZTGPS-*/&/&84 • 7FDUPS'PSHF
/FXWFDUPSGFBUVSFT • 7FDUPS'PSHFNPEFMBSDIJUFDUVSF • 7FDUPS'PSHF#FODINBSL • 4VNNBSZBOEGVUVSFQMBOT

8IZOFXQSPKFDU 7FDUPS'PSHF /FXWFDUPSGFBUVSFT Want to reduce training cost Want to
use various data Modernize ML framework

8IZOFXQSPKFDU 7FDUPS'PSHF /FXWFDUPSGFBUVSFT Want to reduce training cost

• 'PSBDDVSBUFSFDPNNFOEBUJPO *%CBTFENPEFMOFFETGSFRVFOU USBJOJOH • $#$'OFFETUSBJOJOHBGUFS*%CBTFE • $#$'JUFNWFDUPSTEFQFOETPO*%CBTFEJUFNWFDUPST • #&35
$#$' JTTMPXFSUIBO*%CBTFE.-1NJYFS *%CBTFE $#$'JTFYQFOTJWF 7FDUPS'PSHF /FXWFDUPSGFBUVSFT ID-based model CB2CF model 6TFSWFDUPS *%CBTFE *UFNWFDUPS *%CBTFE *UFNWFDUPS $#$' 6TFSMPH EBUB Expensive to train ID-based → CB2CF training dependency Needs frequent training Updated frequently *UFNEBUB

3FWFSTFUIFEFQFOEFODZ 7FDUPS'PSHF /FXWFDUPSGFBUVSFT Content model User model 6TFSWFDUPS *UFNWFDUPS 6TFSMPH
EBUB Content model inherently needs less frequent training because meanings of text/image don’t change overnight. *UFNEBUB ID-based model CB2CF model 6TFSWFDUPS *UFNWFDUPS *%CBTFE *UFNWFDUPS $#$' 6TFSMPH EBUB ID-based → CB2CF training dependency *UFNEBUB As is New: Vector Forge

8IZOFXQSPKFDU 7FDUPS'PSHF /FXWFDUPSGFBUVSFT Want to use various data

• $#$'BOE*%CBTFEXPSLPOBTJOHMFWFDUPSTQBDF • %JGGJDVMUUPVTFPUIFSLJOEPGWFDUPST 4-.--.#&35$-*1 FUD • &BTJFSJGEFQFOEFODZJTSFWFSTFE 8BOUUPVTFWBSJPVTEBUB 7FDUPS'PSHF
/FXWFDUPSGFBUVSFT single vector space text vector space vision vector space 6TFSNPEFM *%CBTFE *UFNNPEFM *%CBTFE *UFNNPEFM 4-.--.#&3 5 *UFNNPEFM $-*1 6TFSNPEFM 7FDUPS 'PSHF *UFNNPEFM $#$'

8IZOFXQSPKFDU 7FDUPS'PSHF /FXWFDUPSGFBUVSFT Modernize ML framework

• 8FVTFEUPVTFJOIPVTF.-GSBNFXPSL • *UXPSLTXFMMXJUIUSBEJUJPOBMCBUDI%//USBJOJOHQSFEJDUJPO • 4JNQMFSVOUJNFQSFQSPDFTTJOHJTQPTTJCMF • 0MEGSBNFXPSLXPO`UHPBXBZTPPO CVUXFUSZTPNFUIJOHOFX
.PEFSOJ[F.-GSBNFXPSL 7FDUPS'PSHF /FXWFDUPSGFBUVSFT 1 https://linedevday.linecorp.com/2020/en/sessions/9750/

• 3BZ %FFQ4QFFE • 4VQQPSUTEJTUSJCVUFEUSBJOJOHOBUJWFMZ • 4VQQPSUTMJCSBSJFTGPS4-.--.USBJOJOHMJLF%FFQ4QFFE • $PNQMJDBUFEQSFQSPDFTTJOHDBOCFXSJUUFOFBTJMZ •
4QMJUUJOHUJNFTFSJFTJOUPJOQVUUBSHFUQBJS • 6OTUSVDUVSFEUFYUQSPDFTTJOH • -.NBTLJOHPG#&35 .PEFSOJ[F.-GSBNFXPSL 7FDUPS'PSHF /FXWFDUPSGFBUVSFT

7FDUPS'PSHFNPEFMBSDIJUFDUVSF 7FDUPS'PSHFNPEFMBSDIJUFDUVSF • 8IPBSFXF • $VSSFOUSFDTZTGPS-*/&/&84 • 7FDUPS'PSHF /FXWFDUPSGFBUVSFT •

• 8FUSBJO*%CBTFE $#$'PODFXJUIMPOHFSVTFSDPOUFYU • (FUBMMVTFSWFDUPSTGSPN*%CBTFENPEFM • (FUBMMJUFNWFDUPSTGSPN$#$'NPEFM • 6TFSNPEFM EPFTO`UVTFJUFNWFDUPSTGSPN*%CBTFENPEFM
-FTTGSFRVFOU$#$'USBJOJOH 7FDUPS'PSHFNPEFMBSDIJUFDUVSF ID-based model CB2CF model VTFSWFDUPS *%CBTFE *UFNWFDUPS *%CBTFE *UFNWFDUPS $#$' User model

• 8IBUJT6*&$' 6TFSBOE*UFN&NCFEEJOHNPEFMTUP$PMMBCPSBUJWF'JMUFSJOH • *OQVUJTBTFRVFODFPGJUFNWFDUPST PVUQVUJTBVTFSWFDUPS 6*&$'VTFSNPEFM 7FDUPS'PSHFNPEFMBSDIJUFDUVSF Transformer encoder
I1 I2 I3 I4 I5 I6 O1 O2 O3 O4 O5 O6 Pooling layer 6TFSWFDUPS *%CBTFE *UFNWFDUPS $#$' U Regression loss

• <4"43FD> 4FMG"UUFOUJWF4FRVFOUJBM3FDPNNFOEBUJPO • 4"43FD JTBDBVTBMNPEFM MJLF(15 0OFPVUQVUGPSBTUFQ • <#&353FD>
4FRVFOUJBM3FDPNNFOEBUJPOXJUI#JEJSFDUJPOBM&ODPEFS3FQSFTFOUBUJPOTGSPN5SBOTGPSNFS • #&353FD JTBNBTLFENPEFM0OFPVUQVUGPSBTUFQ • <1JOOFS'PSNFS>4FRVFODF.PEFMJOHGPS6TFS3FQSFTFOUBUJPOBU1JOUFSFTU • 1JOOFS'PSNFS VTFTMBSHFXJOEPX EBZT BTUBSHFU • <0VSNFUIPE6*&$'> • 6*&$'JTCJEJSFDUJPOBM0OFPVUQVUGPSBTFRVFODF • 6*&$'JTOPUFOEUPFOE 3FMBUFEXPSLT 7FDUPS'PSHFNPEFMBSDIJUFDUVSF 1 https://arxiv.org/pdf/1808.09781 2 https://arxiv.org/pdf/1904.06690 3 https://arxiv.org/pdf/2205.04507

• 7FSZDPNQMJDBUFEQSFQSPDFTTJOH4QBSL • $PNQMJDBUFEQSFQSPDFTTJOHˠ 3BZ • -BSHFSNPEFMTJ[Fˠ %FFQ4QFFE • 5SBJOFEPOOPEFTXJUI"
1$*FPSOPEFTXJUI" 4.9 4QBSL 3BZBOE%FFQ4QFFE 7FDUPS'PSHFNPEFMBSDIJUFDUVSF 6TFSMPH EBUB *UFNEBUB Apache Spark 1 Ray 2 DeepSpeed 3 Feature extraction Last-mile Preprocessing Distributed training Trained model 1 https://spark.apache.org/ 2 https://github.com/ray-project/ray 3 https://github.com/deepspeedai/DeepSpeed

%FFQ4QFFEQFSGPSNBODF 7FDUPS'PSHFNPEFMBSDIJUFDUVSF Model Baseline 80GB PCIe x 2 Zero Stage
2 80GB PCIe x 2 Zero Stage 3 80GB PCIe x 2 Baseline 40GB SXM4 x 8 Zero Stage 2 40GB SXM4 x 8 Zero Stage 3 40GB SXM4 x 8 GPTX Neo 1B 12.49s/it OOM 3.10s/it 3.09s/it GPTX Neo 3B 27.70s/it OOM 6.62s/it 6.36s/it GPTX Neo 7B OOM SEGV SEGV OOM 13.80s/it 13.62s/it Models are trained with contrastive task

7FDUPS'PSHF#FODINBSL 7FDUPS'PSHF#FODINBSL • 8IPBSFXF • $VSSFOUSFDTZTGPS-*/&/&84 • 7FDUPS'PSHF /FXWFDUPSGFBUVSFT •

• 8FXBOUUPFWBMVBUFPVSWFDUPSTXJUIEPXOTUSFBNUBTLT • 3FQSPEVDJCMFFWBMVBUJPOXJUIUSBJOUFTUEBUBTFU • 5SBJOUFTUEBUBTFU&5- 7FDUPS'PSHF#FODINBSL 7FDUPS'PSHF#FODINBSL 6TFSMPH EBUB
*UFNEBUB Train/test ETL #FODINBSL USBJOEBUB #FODINBSL UFTUEBUB #FODINBSL USBJOEBUB 'PS7FDUPS'PSHFUSBJOUFTU • -*/&/&84SFUSJFWBMJJ • -*/&(*'5 +1 • -*/&"ET 'PSFWBMUBTLUSBJOUFTU • -*/&/&84SBOLFS • :BIPP+"1"/4IPQQJOHSFDPNNFOEBUJPO • 6TFSEFNPHSBQIZQSFEJDUJPO BHF HFOEFS 4QMJUEBUF FH

• 1SFEJDUJPOGSPNEPXOTUSFBNUBTLTBSFFWBMVBUFE • &WBMVBUPSTPVUQVUNFUSJDT #FODINBSLFWBMVBUPST 7FDUPS'PSHF#FODINBSL Vector Forge model #FODINBSLUSBJO
EBUB 6TFSWFDUPS *UFNWFDUPS Downstream tasks 5BTLQSFEJDUJPO #FODINBSLUFTU EBUB Benchmark Evaluators • OEDH • FOUSPQZ • BDDVSBDZ • G@NBDSP •

&WBMVBUJPO PGQFSGPSNBODF 7FDUPS'PSHF#FODINBSL

&WBMVBUJPOPGBHJOHEFHSBEBUJPO 7FDUPS'PSHF#FODINBSL 0MEWFSTJPOPG6*&$'XBTVTFEGPSUIJTDPNQBSJTPO 0 0.005 0.01 0.015 0.02 0.025 20230624
20230924 20231224 20240324 20240624 CB2CF full ndcg@5 UIECF full ndcg@5 Trained with old data but less degradation

• -*/&/&84"#UFTUJTQMBOOFEJO+VO -*/&/&84"# 7FDUPS'PSHF#FODINBSL

• 6*&$'XBTFWBMVBUFEXJUI7FDUPS'PSHF#FODINBSL • 5SBJOJOHEBUB • 6TFSMPHPG-*/&/&84 /PUZFUNVMUJEPNBJO • 5FYUBOEJNBHFPGJUFNT .VMUJNPEBM
• 5FTUEBUB • -*/&/&84SFDPNNFOEBUJPO • "OEPUIFSEPXOTUSFBNUBTLT .VMUJEPNBJO • #VUNPEFMTXJUIJNBHFEBUBJTOPUTUVEJFENVDI /PUZFUNVMUJNPEBM • "#UFTUXJUI-*/&/&84JTPOHPJOH 8IFSFXFBSF 7FDUPS'PSHF#FODINBSL

• /FXNPEFMGPSGFBUVSFWFDUPSTCZ7FDUPS'PSHF • 6TFDPOUFOUGFBUVSFT UFYUWJTJPO GPSJUFNWFDUPST • 3FEVDFEUSBJOJOHDPTUCZSFWFSTFEEFQFOEFODZ • 'VUVSFQMBOT
• #FUUFSNPEFMGPSNVMUJEPNBJOSFDPNNFOEBUJPO • 'VSUIFSSFEVDFUIFDPTUPGNPEFMUSBJOJOH • &WBMVBUFNVMUJNPEBMNVMUJNPEBMJUFNNPEFM 4VNNBSZBOEGVUVSFQMBOT 4VNNBSZBOEGVUVSFQMBOT

推薦システムと、RayやDeepSpeedを用いたユーザー・アイテムの次世代の埋め込み表現 /...

推薦システムと、RayやDeepSpeedを用いたユーザー・アイテムの次世代の埋め込み表現 / Recommender System and Next-Gen. Embedding Representations of Users and Items Using Ray and DeepSpeed

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Featured

Transcript