Upgrade to Pro — share decks privately, control downloads, hide ads and more …

推薦システムと、RayやDeepSpeedを用いたユーザー・アイテムの次世代の埋め込み表現 /...

推薦システムと、RayやDeepSpeedを用いたユーザー・アイテムの次世代の埋め込み表現 / Recommender System and Next-Gen. Embedding Representations of Users and Items Using Ray and DeepSpeed

「Vector Forge」はベース技術としてコンテンツモデルを利用した高効率で性能の経時劣化を抑えたモデルです。LINE Newsのレコメンデーションでの経験をベースとしてそれを改善、サービス横断させました。コンテンツモデルとシーケンスレコメンドモデルの連携の重要性やモデルの評価システムの構築に関する話です。

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Transcript

  1.  8IPBSFXF  $VSSFOUSFDTZTGPS-*/&/&84  7FDUPS'PSHF /FXWFDUPSGFBUVSFT  7FDUPS'PSHFNPEFMBSDIJUFDUVSF 

    7FDUPS'PSHF#FODINBSL  4VNNBSZBOEGVUVSFQMBOT "HFOEB Ref: Rayleigh: A Management Platform for Ray clusters https://tech-verse.lycorp.co.jp/2025/ja/session/1022/
  2. • 3FDPNNFOEBUJPOJTCBTFEPOVTFSMPHBOEJUFN • 6TFSMPHEBUBBOEJUFNEBUBBSFQSPDFTTFEUISPVHITFWFSBMTUBHFT 3FDPNNFOEBUJPOQJQFMJOF $VSSFOUSFDTZTGPS-*/&/&84 6TFSMPH EBUB *UFNEBUB Feature

    extraction Retrieval model Ranker model raw features raw + processed features item candidates end users Heuristic post-process Vector Forge focuses this area
  3. • 0GUFODPOTVNJOHSBXGFBUVSFTEJSFDUMZJTJOFGGJDJFOU • 3BXGFBUVSFTDBOCFIJHIEJNFOTJPOBMBOEDPNQMFY • /POTUSVDUVSFE JNBHFUFYUBOEMBSHFOVNCFSPGJUFNT • 8FQSPEVDFVTFSJUFNFNCFEEJOH WFDUPS

    'FBUVSFFYUSBDUJPO $VSSFOUSFDTZTGPS-*/&/&84 6TFSMPH EBUB *UFNEBUB Feature extraction user_1 : [-6.7, 4.9, ..., 3.8] user_2 : [ 4.8, 9.9, ..., -9.7] user_3 : [-8.4, -3.8, ..., -4.7] user_4 : [ 7.5, 8. , ..., 3.4] item_1 : [-9.2, 9.8, ..., 2.9] item_2 : [-3.6, 9.2, ..., -1.8] item_3 : [-4.8, -7.6, ..., 6.4]
  4. • 8FIBWFGFBUVSFFYUSBDUJPONPEFMTUIBUBSFVTFEGPSWBSJPVTUBTLT JODMVEJOH-*/&/&84SFDPNNFOEBUJPO 'FBUVSFFYUSBDUJPO $VSSFOUSFDTZTGPS-*/&/&84 6TFS*UFN EBUBGPS-*/& (JGU -*/&(JGU user_1

    : [-6.7, 4.9, ..., 3.8] ... 6TFS*UFN EBUBGPS-*/& 4UJDLFS 6TFS EFNPHSBQIZ -*/&4UJDLFS user_1 : [ 4.8, 9.9, ..., -9.7] ... 6TFS EFNPHSBQIZ user_1 : [-8.4, -3.8, ..., -4.7] ... Each feature extraction model produces a different vector for each user. Training frequency Every week Prediction frequency Every day Feature extraction models
  5. 3FUSJFWBMNPEFM $VSSFOUSFDTZTGPS-*/&/&84 6TFSMPH EBUBGPS-*/& /&84 *UFNEBUBGPS -*/&/&84 • 'PS-*/&/&84 XFVTF*%CBTFEFNCFEEJOHUXPUPXFS%//

    BSDIJUFDUVSFXJUIBEEFEVTFSWFDUPSGFBUVSFT CVUXJUIPVUJUFNWFDUPS GFBUVSFT User tower Item tower List of viewed items Target items User vector Item vector Contrastive loss 6TFSWFDUPSGPS -*/&0GGJDJBM "DDPVOU 6TFSWFDUPSGPS -*/&"ET User/item vectors from retrieval model are different from user/item vectors from feature extraction Item embedding User features
  6. $PMEJUFNQSPCMFN $VSSFOUSFDTZTGPS-*/&/&84 • 8FXBOUUPSFDPNNFOEMBUFTUOFXTBSUJDMFT CVUOFXBSUJDMFTIBWF UPPGFXVTFSJOUFSBDUJPOT • $#$'"/FVSBM.VMUJWJFX$POUFOUUP$PMMBCPSBUJWF'JMUFSJOH.PEFM GPS$PNQMFUFMZ$PME*UFN3FDPNNFOEBUJPOT CVU

    *UFNWFDUPSGSPN $'NPEFM *UFNEBUB UFYUEBUB Text BERT Item text data Predicted item vector Text BERT model produces item vector from text content 1 https://arxiv.org/pdf/1611.00384
  7. 7FDUPS'PSHF /FXWFDUPSGFBUVSFT 7FDUPS'PSHF /FXWFDUPSGFBUVSFT • 8IPBSFXF • $VSSFOUSFDTZTGPS-*/&/&84 • 7FDUPS'PSHF

    /FXWFDUPSGFBUVSFT • 7FDUPS'PSHFNPEFMBSDIJUFDUVSF • 7FDUPS'PSHF#FODINBSL • 4VNNBSZBOEGVUVSFQMBOT
  8. • 'PSBDDVSBUFSFDPNNFOEBUJPO *%CBTFENPEFMOFFETGSFRVFOU USBJOJOH • $#$'OFFETUSBJOJOHBGUFS*%CBTFE • $#$'JUFNWFDUPSTEFQFOETPO*%CBTFEJUFNWFDUPST • #&35

    $#$' JTTMPXFSUIBO*%CBTFE.-1NJYFS *%CBTFE $#$'JTFYQFOTJWF 7FDUPS'PSHF /FXWFDUPSGFBUVSFT ID-based model CB2CF model 6TFSWFDUPS *%CBTFE *UFNWFDUPS *%CBTFE *UFNWFDUPS $#$' 6TFSMPH EBUB Expensive to train ID-based → CB2CF training dependency Needs frequent training Updated frequently *UFNEBUB
  9. 3FWFSTFUIFEFQFOEFODZ 7FDUPS'PSHF /FXWFDUPSGFBUVSFT Content model User model 6TFSWFDUPS *UFNWFDUPS 6TFSMPH

    EBUB Content model inherently needs less frequent training because meanings of text/image don’t change overnight. *UFNEBUB ID-based model CB2CF model 6TFSWFDUPS *UFNWFDUPS *%CBTFE *UFNWFDUPS $#$' 6TFSMPH EBUB ID-based → CB2CF training dependency *UFNEBUB As is New: Vector Forge
  10. • $#$'BOE*%CBTFEXPSLPOBTJOHMFWFDUPSTQBDF • %JGGJDVMUUPVTFPUIFSLJOEPGWFDUPST 4-.--.#&35$-*1 FUD • &BTJFSJGEFQFOEFODZJTSFWFSTFE 8BOUUPVTFWBSJPVTEBUB 7FDUPS'PSHF

    /FXWFDUPSGFBUVSFT single vector space text vector space vision vector space 6TFSNPEFM *%CBTFE *UFNNPEFM *%CBTFE *UFNNPEFM 4-.--.#&3 5 *UFNNPEFM $-*1 6TFSNPEFM 7FDUPS 'PSHF *UFNNPEFM $#$'
  11. • 3BZ %FFQ4QFFE • 4VQQPSUTEJTUSJCVUFEUSBJOJOHOBUJWFMZ • 4VQQPSUTMJCSBSJFTGPS4-.--.USBJOJOHMJLF%FFQ4QFFE • $PNQMJDBUFEQSFQSPDFTTJOHDBOCFXSJUUFOFBTJMZ •

    4QMJUUJOHUJNFTFSJFTJOUPJOQVUUBSHFUQBJS • 6OTUSVDUVSFEUFYUQSPDFTTJOH • -.NBTLJOHPG#&35 .PEFSOJ[F.-GSBNFXPSL 7FDUPS'PSHF /FXWFDUPSGFBUVSFT
  12. • 8FUSBJO*%CBTFE $#$'PODFXJUIMPOHFSVTFSDPOUFYU • (FUBMMVTFSWFDUPSTGSPN*%CBTFENPEFM • (FUBMMJUFNWFDUPSTGSPN$#$'NPEFM • 6TFSNPEFM EPFTO`UVTFJUFNWFDUPSTGSPN*%CBTFENPEFM

    -FTTGSFRVFOU$#$'USBJOJOH 7FDUPS'PSHFNPEFMBSDIJUFDUVSF ID-based model CB2CF model VTFSWFDUPS *%CBTFE *UFNWFDUPS *%CBTFE *UFNWFDUPS $#$' User model
  13. • <4"43FD> 4FMG"UUFOUJWF4FRVFOUJBM3FDPNNFOEBUJPO • 4"43FD JTBDBVTBMNPEFM MJLF(15 0OFPVUQVUGPSBTUFQ • <#&353FD>

    4FRVFOUJBM3FDPNNFOEBUJPOXJUI#JEJSFDUJPOBM&ODPEFS3FQSFTFOUBUJPOTGSPN5SBOTGPSNFS • #&353FD JTBNBTLFENPEFM0OFPVUQVUGPSBTUFQ • <1JOOFS'PSNFS>4FRVFODF.PEFMJOHGPS6TFS3FQSFTFOUBUJPOBU1JOUFSFTU • 1JOOFS'PSNFS VTFTMBSHFXJOEPX EBZT BTUBSHFU • <0VSNFUIPE6*&$'> • 6*&$'JTCJEJSFDUJPOBM0OFPVUQVUGPSBTFRVFODF • 6*&$'JTOPUFOEUPFOE 3FMBUFEXPSLT 7FDUPS'PSHFNPEFMBSDIJUFDUVSF 1 https://arxiv.org/pdf/1808.09781 2 https://arxiv.org/pdf/1904.06690 3 https://arxiv.org/pdf/2205.04507
  14. • 7FSZDPNQMJDBUFEQSFQSPDFTTJOH4QBSL • $PNQMJDBUFEQSFQSPDFTTJOHˠ 3BZ • -BSHFSNPEFMTJ[Fˠ %FFQ4QFFE • 5SBJOFEPOOPEFTXJUI"

    1$*FPSOPEFTXJUI" 4.9 4QBSL 3BZBOE%FFQ4QFFE 7FDUPS'PSHFNPEFMBSDIJUFDUVSF 6TFSMPH EBUB *UFNEBUB Apache Spark 1 Ray 2 DeepSpeed 3 Feature extraction Last-mile Preprocessing Distributed training Trained model 1 https://spark.apache.org/ 2 https://github.com/ray-project/ray 3 https://github.com/deepspeedai/DeepSpeed
  15. %FFQ4QFFEQFSGPSNBODF 7FDUPS'PSHFNPEFMBSDIJUFDUVSF Model Baseline 80GB PCIe x 2 Zero Stage

    2 80GB PCIe x 2 Zero Stage 3 80GB PCIe x 2 Baseline 40GB SXM4 x 8 Zero Stage 2 40GB SXM4 x 8 Zero Stage 3 40GB SXM4 x 8 GPTX Neo 1B 12.49s/it OOM 3.10s/it 3.09s/it GPTX Neo 3B 27.70s/it OOM 6.62s/it 6.36s/it GPTX Neo 7B OOM SEGV SEGV OOM 13.80s/it 13.62s/it Models are trained with contrastive task
  16. • 8FXBOUUPFWBMVBUFPVSWFDUPSTXJUIEPXOTUSFBNUBTLT • 3FQSPEVDJCMFFWBMVBUJPOXJUIUSBJOUFTUEBUBTFU • 5SBJOUFTUEBUBTFU&5- 7FDUPS'PSHF#FODINBSL 7FDUPS'PSHF#FODINBSL 6TFSMPH EBUB

    *UFNEBUB Train/test ETL #FODINBSL USBJOEBUB #FODINBSL UFTUEBUB #FODINBSL USBJOEBUB 'PS7FDUPS'PSHFUSBJOUFTU • -*/&/&84SFUSJFWBMJJ • -*/&(*'5 +1 • -*/&"ET  'PSFWBMUBTLUSBJOUFTU • -*/&/&84SBOLFS • :BIPP+"1"/4IPQQJOHSFDPNNFOEBUJPO • 6TFSEFNPHSBQIZQSFEJDUJPO BHF  HFOEFS   4QMJUEBUF FH
  17. • 1SFEJDUJPOGSPNEPXOTUSFBNUBTLTBSFFWBMVBUFE • &WBMVBUPSTPVUQVUNFUSJDT #FODINBSLFWBMVBUPST 7FDUPS'PSHF#FODINBSL Vector Forge model #FODINBSLUSBJO

    EBUB 6TFSWFDUPS *UFNWFDUPS Downstream tasks 5BTLQSFEJDUJPO #FODINBSLUFTU EBUB Benchmark Evaluators • OEDH • FOUSPQZ • BDDVSBDZ • G@NBDSP • 
  18. &WBMVBUJPOPGBHJOHEFHSBEBUJPO 7FDUPS'PSHF#FODINBSL 0MEWFSTJPOPG6*&$'XBTVTFEGPSUIJTDPNQBSJTPO 0 0.005 0.01 0.015 0.02 0.025 20230624

    20230924 20231224 20240324 20240624 CB2CF full ndcg@5 UIECF full ndcg@5 Trained with old data but less degradation
  19. • 6*&$'XBTFWBMVBUFEXJUI7FDUPS'PSHF#FODINBSL • 5SBJOJOHEBUB • 6TFSMPHPG-*/&/&84 /PUZFUNVMUJEPNBJO • 5FYUBOEJNBHFPGJUFNT .VMUJNPEBM

    • 5FTUEBUB • -*/&/&84SFDPNNFOEBUJPO • "OEPUIFSEPXOTUSFBNUBTLT .VMUJEPNBJO • #VUNPEFMTXJUIJNBHFEBUBJTOPUTUVEJFENVDI /PUZFUNVMUJNPEBM • "#UFTUXJUI-*/&/&84JTPOHPJOH 8IFSFXFBSF 7FDUPS'PSHF#FODINBSL
  20. • /FXNPEFMGPSGFBUVSFWFDUPSTCZ7FDUPS'PSHF • 6TFDPOUFOUGFBUVSFT UFYUWJTJPO GPSJUFNWFDUPST • 3FEVDFEUSBJOJOHDPTUCZSFWFSTFEEFQFOEFODZ • 'VUVSFQMBOT

    • #FUUFSNPEFMGPSNVMUJEPNBJOSFDPNNFOEBUJPO • 'VSUIFSSFEVDFUIFDPTUPGNPEFMUSBJOJOH • &WBMVBUFNVMUJNPEBMNVMUJNPEBMJUFNNPEFM 4VNNBSZBOEGVUVSFQMBOT 4VNNBSZBOEGVUVSFQMBOT