Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling and Potential Uses for the HyperCLOVA Japanese Base Model

Scaling and Potential Uses for the HyperCLOVA Japanese Base Model

Toshinori Sato (LINE / NLP Development Team / Engineering Manager)
Takato Yamazaki (LINE / NLP Development Team / Software Engineer)

https://tech-verse.me/ja/sessions/52
https://tech-verse.me/en/sessions/52
https://tech-verse.me/ko/sessions/52

Tech-Verse2022
PRO

November 17, 2022
Tweet

More Decks by Tech-Verse2022

Other Decks in Technology

Transcript

  1. View Slide

  2. Toshinori Sato
    (@overlast)
    - Natural Language Processing
    - Information Retrieval
    Principal Software Engineer / Manager
    - Japanese Corpus / Evaluation / Application
    - mecab-ipadic-NEologd
    HyperCLOVA
    OSS: Main Contributor of NEologd
    ࢁ࡚ ఱ Takato Yamazaki
    Takato Yamazaki
    (@y_takaten)
    - Dialog Systems
    - HyperCLOVA Applications
    Software Engineer
    - Aizawa Lab. / University of Tokyo
    - “Phrase-Level Action Reinforcement Learning for Neural
    Dialog Response Generation”, Findings of ACL 2021
    - Young Researcher Award for Excellent Research at
    11th Dialogue System Symposium
    MS Degree in Computer Science

    View Slide

  3. At #linedevday 2021, we presented
    JP 39B model

    View Slide

  4. 82B

    View Slide

  5. Agenda
    Realizing Applications with Prompting
    Performance Evaluation of 82B JP Model
    Developing Applications with HyperCLOVA
    Deploying Foundation Models in Real World

    View Slide

  6. Agenda
    Realizing Applications with Prompting
    Performance Evaluation of 82B JP Model
    Developing Applications with HyperCLOVA
    Deploying Foundation Models in Real World

    View Slide

  7. Meta-learning to treat target task information as text input and
    predict the next word from that input
    Pros
    - Highly economical in terms of time
    - High accuracy can be achieved without model tuning if
    the appropriate information and example (few-shot) can
    be described
    Cons
    - Requires a considerably larger number of parameters than
    when using a tunable mode
    - Requires a linguistic background and extensive social
    experience
    Two Major Ways of Controlling Output with Foundation Model
    Prompting
    Fine-tuning
    Meta-learning to treat target task information as text
    input and predict the next word from that input
    Pros
    - Highly economical in terms of money
    - High accuracy can be achieved if sufficient time is
    available for model training and parameter
    exploration
    Cons
    - Need to have the right amount of properly labeled
    teacher data for each task
    - Need to create a model for each task

    View Slide

  8. Q: But how do we use HyperCLOVA?
    A: Prompt Programming

    View Slide

  9. View Slide

  10. https://huggingface.co/spaces/stabilityai/stable-diffusion

    View Slide

  11. ;
    5FDI7FSTFJTBOPOMJOFUFDIOJDBMDPOGFSFODF
    XIJDIXJMMCFIFMEBTBKPJOUQSPKFDUCFUXFFO-*/&BOE
    :BIPP+"1"/PO/PWFNCFSUIBOEUI"EJWFSTF
    BSSBZPGNFNCFSTGSPNUIFUXPIPTUDPNQBOJFTBOE
    PUIFSQBSUJDJQBUJOHPSHBOJ[BUJPOTXJMMTIBSFUIFJS
    DVUUJOHFEHFDIBMMFOHFTBOEBDDVNVMBUFELOPXMFEHF
    1SPNQUBTBRVFSZTUSJOHPG4UBCMF%JGGVTJPO
    “Prompting” is becoming very common w/ Stable Diffusion

    View Slide

  12. “Prompting” is becoming very common w/ Stable Diffusion
    Generate
    ;
    5FDI7FSTFJTBOPOMJOFUFDIOJDBMDPOGFSFODF
    XIJDIXJMMCFIFMEBTBKPJOUQSPKFDUCFUXFFO-*/&BOE
    :BIPP+"1"/PO/PWFNCFSUIBOEUI"EJWFSTF
    BSSBZPGNFNCFSTGSPNUIFUXPIPTUDPNQBOJFTBOE
    PUIFSQBSUJDJQBUJOHPSHBOJ[BUJPOTXJMMTIBSFUIFJS
    DVUUJOHFEHFDIBMMFOHFTBOEBDDVNVMBUFELOPXMFEHF
    1SPNQUBTBRVFSZTUSJOHPG4UBCMF%JGGVTJPO

    View Slide

  13. ;
    5FDI7FSTFJTBOPOMJOFUFDIOJDBMDPOGFSFODF
    XIJDIXJMMCFIFMEBTBKPJOUQSPKFDUCFUXFFO-*/&BOE
    :BIPP+"1"/PO/PWFNCFSUIBOEUI"EJWFSTF
    BSSBZPGNFNCFSTGSPNUIFUXPIPTUDPNQBOJFTBOE
    PUIFSQBSUJDJQBUJOHPSHBOJ[BUJPOTXJMMTIBSFUIFJS
    DVUUJOHFEHFDIBMMFOHFTBOEBDDVNVMBUFELOPXMFEHF
    1SPNQUBTBRVFSZTUSJOHPG4UBCMF%JGGVTJPO
    Hmm, this does look like
    a diverse array of Japanese members…
    “Prompting” is becoming very common w/ Stable Diffusion
    Generate

    View Slide

  14. HyperCLOVA is also used with Prompt

    View Slide

  15. "BOE#BSFDIJUDIBUUJOH
    HyperCLOVA is also used with Prompt
    1SPNQUUPHFOFSBUFBSFTQPOTFPGBEJBMPH

    View Slide

  16. "BOE#BSFDIJUDIBUUJOH
    ")J IPXBSFZPV
    #*`NHSFBU8IBUEPZPVEPJOZPVSGSFFUJNF
    ")NN *MJLFUPEPCPXMJOH
    #/JDF8IBU`TZPVSCFTUTDPSF
    ".ZCFTUTDPSFJT*VTVBMMZHFUBSPVOE
    #5IBU`TBXFTPNF*`MMOFWFSCFBCMFUPHFUUIBUTDPSF
    HyperCLOVA is also used with Prompt
    1SPNQUUPHFOFSBUFBSFTQPOTFPGBEJBMPH

    View Slide

  17. ===
    "BOE#BSFDIJUDIBUUJOH
    ")J IPXBSFZPV
    #*`NHSFBU8IBUEPZPVEPJOZPVSGSFFUJNF
    ")NN *MJLFUPEPCPXMJOH
    #/JDF8IBU`TZPVSCFTUTDPSF
    ".ZCFTUTDPSFJT*VTVBMMZHFUBSPVOE
    #5IBU`TBXFTPNF*`MMOFWFSCFBCMFUPHFUUIBUTDPSF
    HyperCLOVA is also used with Prompt
    1SPNQUUPHFOFSBUFBSFTQPOTFPGUIFEJBMPH

    View Slide

  18. ===
    "BOE#BSFDIJUDIBUUJOH
    ")J IPXBSFZPV
    #*`NHSFBU8IBUEPZPVEPJOZPVSGSFFUJNF
    ")NN *MJLFUPEPCPXMJOH
    #/JDF8IBU`TZPVSCFTUTDPSF
    ".ZCFTUTDPSFJT*VTVBMMZHFUBSPVOE
    #5IBU`TBXFTPNF*`MMOFWFSCFBCMFUPHFUUIBUTDPSF
    ")J XIBUEPZPVEPJOZPVSGSFFUJNF
    #*MJLFUPXBUDINPWJFT
    "
    1SPNQU
    HyperCLOVA is also used with Prompt
    1SPNQUUPHFOFSBUFBSFTQPOTFPGUIFEJBMPH

    View Slide

  19. ===
    "BOE#BSFDIJUDIBUUJOH 5BTL%FTDSJQUJPO
    ")J IPXBSFZPV
    #*`NHSFBU8IBUEPZPVEPJOZPVSGSFFUJNF
    ")NN *MJLFUPEPCPXMJOH
    #/JDF8IBU`TZPVSCFTUTDPSF
    ".ZCFTUTDPSFJT*VTVBMMZHFUBSPVOE
    #5IBU`TBXFTPNF*`MMOFWFSCFBCMFUPHFUUIBUTDPSF
    %JBMPH&YBNQMFº/
    ")J XIBUEPZPVEPJOZPVSGSFFUJNF
    #*MJLFUPXBUDINPWJFT
    "
    $VSSFOU%JBMPH
    HyperCLOVA is also used with Prompt
    *OUIJTDBTF
    JU`TDBMMFETIPU
    1SPNQUUPHFOFSBUFBSFTQPOTFPGUIFEJBMPH

    View Slide

  20. ===
    "BOE#BSFDIJUDIBUUJOH 5BTL%FTDSJQUJPO
    ")J IPXBSFZPV
    #*`NHSFBU8IBUEPZPVEPJOZPVSGSFFUJNF
    ")NN *MJLFUPEPCPXMJOH
    #/JDF8IBU`TZPVSCFTUTDPSF
    ".ZCFTUTDPSFJT*VTVBMMZHFUBSPVOE
    #5IBU`TBXFTPNF*`MMOFWFSCFBCMFUPHFUUIBUTDPSF
    %JBMPH&YBNQMFº/
    ")J XIBUEPZPVEPJOZPVSGSFFUJNF
    #*MJLFUPXBUDINPWJFT
    "
    $VSSFOU%JBMPH
    HyperCLOVA is also used with Prompt
    1SPNQUUPHFOFSBUFBSFTQPOTFPGUIFEJBMPH

    View Slide

  21. ===
    "BOE#BSFDIJUDIBUUJOH 5BTL%FTDSJQUJPO
    ")J IPXBSFZPV
    #*`NHSFBU8IBUEPZPVEPJOZPVSGSFFUJNF
    ")NN *MJLFUPEPCPXMJOH
    #/JDF8IBU`TZPVSCFTUTDPSF
    ".ZCFTUTDPSFJT*VTVBMMZHFUBSPVOE
    #5IBU`TBXFTPNF*`MMOFWFSCFBCMFUPHFUUIBUTDPSF
    %JBMPH&YBNQMFº/
    ")J XIBUEPZPVEPJOZPVSGSFFUJNF
    #*MJLFUPXBUDINPWJFT
    "
    $VSSFOU%JBMPH
    HyperCLOVA is also used with Prompt
    Input the prompt and inference
    1SPNQUUPHFOFSBUFBSFTQPOTFPGUIFEJBMPH

    View Slide

  22. 8IBULJOEPGNPWJFT
    EPZPVXBUDI
    ===
    "BOE#BSFDIJUDIBUUJOH 5BTL%FTDSJQUJPO
    ")J IPXBSFZPV
    #*`NHSFBU8IBUEPZPVEPJOZPVSGSFFUJNF
    ")NN *MJLFUPEPCPXMJOH
    #/JDF8IBU`TZPVSCFTUTDPSF
    ".ZCFTUTDPSFJT*VTVBMMZHFUBSPVOE
    #5IBU`TBXFTPNF*`MMOFWFSCFBCMFUPHFUUIBUTDPSF
    %JBMPH&YBNQMFº/
    ")J XIBUEPZPVEPJOZPVSGSFFUJNF
    #*MJLFUPXBUDINPWJFT
    "
    $VSSFOU%JBMPH
    Generate
    Input the prompt and inference
    HyperCLOVA is also used with Prompt
    1SPNQUUPHFOFSBUFBSFTQPOTFPGUIFEJBMPH

    View Slide

  23. 8IBULJOEPGNPWJFT
    EPZPVXBUDI
    ===
    "BOE#BSFDIJUDIBUUJOH 5BTL%FTDSJQUJPO
    ")J IPXBSFZPV
    #*`NHSFBU8IBUEPZPVEPJOZPVSGSFFUJNF
    ")NN *MJLFUPEPCPXMJOH
    #/JDF8IBU`TZPVSCFTUTDPSF
    ".ZCFTUTDPSFJT*VTVBMMZHFUBSPVOE
    #5IBU`TBXFTPNF*`MMOFWFSCFBCMFUPHFUUIBUTDPSF
    %JBMPH&YBNQMFº/
    ")J XIBUEPZPVEPJOZPVSGSFFUJNF
    #*MJLFUPXBUDINPWJFT
    "
    $VSSFOU%JBMPH
    8IBULJOEPGNPWJFTEPZPVXBUDI
    (FOFSBUFEXJUI
    )ZQFS$-07"
    HyperCLOVA is also used with Prompt
    Input the prompt and inference
    Generate
    1SPNQUUPHFOFSBUFBSFTQPOTFPGUIFEJBMPH

    View Slide

  24. That's wonderful!
    What do you think
    is the most
    important for your
    management work?
    OR
    Image Text

    View Slide

  25. Image
    <4IPLP XIPJTBMXBZTDIFFSGVM BTLTRVFTUJPOTBCPVUUIFDMJFOUT
    PDDVQBUJPO>

    "DMJFOUJTBNBOJOIJTMBUFT
    4IPLP RVFTUJPO
    /JDFUPNFFUZPV*BNBUPVSJTUHVJEF8IBUEP
    ZPVEPGPSZPVSMJWJOH
    $VTUPNFS*BNBNBOBHFSJOBO*5DPNQBOZ
    4IPLP RVFTUJPO

    BIJHIMZEFUBJMFEFQJDDJOFNBUJDDPODFQUBSU$(SFOEFS
    EJHJUBMQBJOUJOHBSUXPSLEJFTFMQVOLTUFBNJOHIBMGNBOIBMG
    SPCPU#Z)PLVTBJ,BUTVTIJLB )JSPTIJHF6UBHBXB &JTFO
    ,FJTBJ ,VOJZPTIJ6UBHBXB ,VOJTBEB6UBHBXB 4IVOTIPV
    ,BUTVLBXB 4IJHFOPCV:BOBHBXB USFOEJOHPO"SU4UBUJPO
    TVCUMFNVUFEDJOFNBUJDDPMPST NBEFJO.BZB #MFOEFSBOE
    1IPUPTIPQ PDUBOFSFOEFS FYDFMMFOUDPNQPTJUJPO DJOFNBUJD
    BUNPTQIFSF EZOBNJDESBNBUJDDJOFNBUJDMJHIUJOH QSFDJTF
    DPSSFDUBOBUPNZ BFTUIFUJD WFSZJOTQJSBUJPOBM BSUIPVTF
    Output
    Format
    Text
    Output
    Example
    Prompt
    Example
    Number of people who can agree on an output itself Number of people who receive an output as intended
    Evaluation
    Viewpoint
    Prompt is written differently depending on output modality

    View Slide

  26. Image
    <4IPLP XIPJTBMXBZTDIFFSGVM BTLTRVFTUJPOTBCPVUUIFDMJFOUT
    PDDVQBUJPO>

    "DMJFOUJTBNBOJOIJTMBUFT
    4IPLP RVFTUJPO
    /JDFUPNFFUZPV*BNBUPVSJTUHVJEF8IBUEP
    ZPVEPGPSZPVSMJWJOH
    $VTUPNFS*BNBNBOBHFSJOBO*5DPNQBOZ
    4IPLP RVFTUJPO

    BIJHIMZEFUBJMFEFQJDDJOFNBUJDDPODFQUBSU$(SFOEFS
    EJHJUBMQBJOUJOHBSUXPSLEJFTFMQVOLTUFBNJOHIBMGNBOIBMG
    SPCPU#Z)PLVTBJ,BUTVTIJLB )JSPTIJHF6UBHBXB &JTFO
    ,FJTBJ ,VOJZPTIJ6UBHBXB ,VOJTBEB6UBHBXB 4IVOTIPV
    ,BUTVLBXB 4IJHFOPCV:BOBHBXB USFOEJOHPO"SU4UBUJPO
    TVCUMFNVUFEDJOFNBUJDDPMPST NBEFJO.BZB #MFOEFSBOE
    1IPUPTIPQ PDUBOFSFOEFS FYDFMMFOUDPNQPTJUJPO DJOFNBUJD
    BUNPTQIFSF EZOBNJDESBNBUJDDJOFNBUJDMJHIUJOH QSFDJTF
    DPSSFDUBOBUPNZ BFTUIFUJD WFSZJOTQJSBUJPOBM BSUIPVTF
    Output
    Format
    Text
    Output
    Example
    Prompt
    Example
    Number of people who can agree on an output itself Number of people who receive an output as intended
    Evaluation
    Viewpoint
    Prompt is written differently depending on output modality

    View Slide

  27. Image
    <4IPLP XIPJTBMXBZTDIFFSGVM BTLTRVFTUJPOTBCPVUUIFDMJFOUT
    PDDVQBUJPO>

    "DMJFOUJTBNBOJOIJTMBUFT
    4IPLP RVFTUJPO
    /JDFUPNFFUZPV*BNBUPVSJTUHVJEF8IBUEP
    ZPVEPGPSZPVSMJWJOH
    $VTUPNFS*BNBNBOBHFSJOBO*5DPNQBOZ
    4IPLP RVFTUJPO

    BIJHIMZEFUBJMFEFQJDDJOFNBUJDDPODFQUBSU$(SFOEFS
    EJHJUBMQBJOUJOHBSUXPSLEJFTFMQVOLTUFBNJOHIBMGNBOIBMG
    SPCPU#Z)PLVTBJ,BUTVTIJLB )JSPTIJHF6UBHBXB &JTFO
    ,FJTBJ ,VOJZPTIJ6UBHBXB ,VOJTBEB6UBHBXB 4IVOTIPV
    ,BUTVLBXB 4IJHFOPCV:BOBHBXB USFOEJOHPO"SU4UBUJPO
    TVCUMFNVUFEDJOFNBUJDDPMPST NBEFJO.BZB #MFOEFSBOE
    1IPUPTIPQ PDUBOFSFOEFS FYDFMMFOUDPNQPTJUJPO DJOFNBUJD
    BUNPTQIFSF EZOBNJDESBNBUJDDJOFNBUJDMJHIUJOH QSFDJTF
    DPSSFDUBOBUPNZ BFTUIFUJD WFSZJOTQJSBUJPOBM BSUIPVTF
    Output
    Format
    Text
    Output
    Example
    Prompt
    Example
    Number of people who can agree on an output itself Number of people who receive an output as intended
    Evaluation
    Viewpoint
    Prompt is written differently depending on output modality

    View Slide

  28. Prompt is written differently depending on output modality
    Image
    <4IPLP XIPJTBMXBZTDIFFSGVM BTLTRVFTUJPOTBCPVUUIFDMJFOUT
    PDDVQBUJPO>

    "DMJFOUJTBNBOJOIJTMBUFT
    4IPLP RVFTUJPO
    /JDFUPNFFUZPV*BNBUPVSJTUHVJEF8IBUEP
    ZPVEPGPSZPVSMJWJOH
    $VTUPNFS*BNBNBOBHFSJOBO*5DPNQBOZ
    4IPLP RVFTUJPO

    BIJHIMZEFUBJMFEFQJDDJOFNBUJDDPODFQUBSU$(SFOEFS
    EJHJUBMQBJOUJOHBSUXPSLEJFTFMQVOLTUFBNJOHIBMGNBOIBMG
    SPCPU#Z)PLVTBJ,BUTVTIJLB )JSPTIJHF6UBHBXB &JTFO
    ,FJTBJ ,VOJZPTIJ6UBHBXB ,VOJTBEB6UBHBXB 4IVOTIPV
    ,BUTVLBXB 4IJHFOPCV:BOBHBXB USFOEJOHPO"SU4UBUJPO
    TVCUMFNVUFEDJOFNBUJDDPMPST NBEFJO.BZB #MFOEFSBOE
    1IPUPTIPQ PDUBOFSFOEFS FYDFMMFOUDPNQPTJUJPO DJOFNBUJD
    BUNPTQIFSF EZOBNJDESBNBUJDDJOFNBUJDMJHIUJOH QSFDJTF
    DPSSFDUBOBUPNZ BFTUIFUJD WFSZJOTQJSBUJPOBM BSUIPVTF
    Output
    Format
    Text
    Output
    Example
    Prompt
    Example
    Number of people who can agree on an output itself Number of people who receive an output as intended
    Evaluation
    Viewpoint
    That's wonderful!
    What do you think is
    the most important for
    your management work?

    View Slide

  29. What is better output in response for user's input?
    ===
    "BOE#BSFDIJUDIBUUJOH
    #)FZ XIBU`TVQ
    ")PX`TJUHPJOH "OZUIJOHOFXXJUIZPVMBUFMZ
    #*IFBSEUIBUUIFOFXUFOUIPVTBOEZFOCJMMXJMMIBWF
    4IJCVTBXB&JJDIJPOJU CVUEPZPVLOPXBCPVUIJN
    "
    (FOFSBUFEXJUI)ZQFS$-07"
    3FTQPOTFGSPNVTFS
    • The user's response contains a named
    entity “Shibusawa Eiichi”
    • And we can detect that “Shibusawa
    Eiichi” includes in the entries of
    Wikipedia
    • For simplicity, we would like to make
    use of the WIkipedia text after extracting
    “Shibusawa Eiichi” from the user's
    response
    ?
    "*WFCFFOSFBEJOHBMPUMBUFMZ
    #5IBUTHSFBU8IBUBSFZPVSFBEJOH
    "*WFCFFOSFBEJOHCPPLTCZ4PTFLJ/BUTVNF
    #)FZ *LOPXIJN5IFQFSTPOJOUIF ZFOCJMM SJHIU

    View Slide

  30. A: I've been reading a lot lately.
    B: That's great! What are you reading?
    A: I've been reading books by Soseki Natsume.
    B: Hey, I know him! The person in the 1,000 yen
    bill, right?
    ===
    B: Hey, what’s up!
    A: How’s it going? Anything new with you lately?
    B: I heard that the new ten-thousand yen bill will
    have Shibusawa Eiichi on it, but do you know
    about him?
    A:
    ɹɹɹɹ ?
    Can the training data be accurately retrieved from a model?

    View Slide

  31. A: I've been reading a lot lately.
    B: That's great! What are you reading?
    A: I've been reading books by Soseki Natsume.
    B: Hey, I know him! The person in the 1,000 yen
    bill, right?
    ===
    B: Hey, what’s up!
    A: How’s it going? Anything new with you lately?
    B: I heard that the new ten-thousand yen bill will
    have Shibusawa Eiichi on it, but do you know
    about him?
    A:
    I don’t really know him, who’s that?
    Can the training data be accurately retrieved from a model?
    We should assume that
    an accurate knowledge in the
    foundation model cannot be extracted.
    No !!
    Foundation Model is not
    a search engine !!

    View Slide

  32. HyperCLOVA Prompt chaining can generate complex output
    Knowledge: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    Dialog:
    A: I've been reading a lot lately.
    B: That's great! What are you reading?
    A: I've been reading books by Soseki Natsume.
    B: Hey, I know him! The person in the 1,000 yen
    bill, right?
    ===
    Dialog:
    B: Hey, what’s up!
    A: How’s it going? Anything new with you lately?
    B: I heard that the new ten-thousand yen bill will
    have Shibusawa Eiichi on it, but do you know
    about him?
    A:
    "VUPHFOFSBUFE,OPXMFEHF
    A: I've been reading a lot lately.
    B: That's great! What are you reading?
    A: I've been reading books by Soseki Natsume.
    B: Hey, I know him! The person in the 1,000 yen
    bill, right?
    ===
    B: Hey, what’s up!
    A: How’s it going? Anything new with you lately?
    B: I heard that the new ten-thousand yen bill will
    have Shibusawa Eiichi on it, but do you know
    about him?
    A:
    )BOENBEF,OPXMFEHF
    I don’t really know him, who’s that?
    (Something better with knowledge…)
    Copy
    Copy

    View Slide

  33. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    /BNFE&OUJUZ3FDPHOJUJPO

    View Slide

  34. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    4IJCVTBXB&JJDIJ
    /BNFE&OUJUZ3FDPHOJUJPO

    View Slide

  35. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    /BNFE&OUJUZ3FDPHOJUJPO
    4IJCVTBXB&JJDIJ
    %#4FBSDI FH8JLJQFEJB

    View Slide

  36. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    /BNFE&OUJUZ3FDPHOJUJPO
    4IJCVTBXB&JJDIJ
    %#4FBSDI FH8JLJQFEJB

    Shibusawa Eiichi, 1st Viscount
    Shibusawa (March 16, 1840 –
    November 11, 1931) was a
    Japanese industrialist use
    of double-entry accounting, joint-
    stock corporations and modern
    note-issuing banks.
    He founded the first modern
    bank based on joint stock
    ownership in Japan. The bank
    was aptly named The First
    National Bank (Dai Ichi Kokuritsu
    Ginkō, now merged into Mizuho
    Bank) and had the power to
    issue its own notes. Through this
    bank, he founded hundreds of
    other joint stock corporations in
    Japan. Many of these
    companies still survive …
    Search Result

    View Slide

  37. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    Original: Natsume Sōseki (9 February 1867 – 9
    December 1916), born Natsume Kin'nosuke, was
    a Japanese novelist. He is best known around the
    world for his novels Kokoro, Botchan, I Am a
    Cat, Kusamakura and his unfinished work Light
    and Darkness. He was also a scholar of British
    literature and writer of haiku, kanshi, and fairy
    tales. From 1984 until 2004, his portrait appeared
    on the front of the Japanese 1,000 yen note.
    Summary: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    ===
    Original: Shibusawa Eiichi, 1st Viscount
    Shibusawa (March 16, 1840 – November 11,
    1931) was a Japanese industrialist use of double-
    entry accounting, joint-stock corporations and
    modern note-issuing banks…
    Summary:
    4FBSDI3FTVMUPO
    4IJCVTBXB&JJDIJ
    4VNNBSJ[BUJPO
    /BNFE&OUJUZ3FDPHOJUJPO
    4IJCVTBXB&JJDIJ
    %#4FBSDI FH8JLJQFEJB

    View Slide

  38. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    Original: Natsume Sōseki (9 February 1867 – 9
    December 1916), born Natsume Kin'nosuke, was
    a Japanese novelist. He is best known around the
    world for his novels Kokoro, Botchan, I Am a
    Cat, Kusamakura and his unfinished work Light
    and Darkness. He was also a scholar of British
    literature and writer of haiku, kanshi, and fairy
    tales. From 1984 until 2004, his portrait appeared
    on the front of the Japanese 1,000 yen note.
    Summary: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    ===
    Original: Shibusawa Eiichi, 1st Viscount
    Shibusawa (March 16, 1840 – November 11,
    1931) was a Japanese industrialist use of double-
    entry accounting, joint-stock corporations and
    modern note-issuing banks…
    Summary:
    4VNNBSJ[BUJPO
    /BNFE&OUJUZ3FDPHOJUJPO
    4IJCVTBXB&JJDIJ
    Eiichi Shibusawa was born in
    1840 in Kuroaraijima, Fukaya
    City. During his lifetime, he was
    involved in the establishment of
    approximately 500 companies
    and is known as the "father of
    Japanese capitalism”.
    Summary Result
    4FBSDI3FTVMUPO
    4IJCVTBXB&JJDIJ
    %#4FBSDI FH8JLJQFEJB

    View Slide

  39. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    Original: Natsume Sōseki (9 February 1867 – 9
    December 1916), born Natsume Kin'nosuke, was
    a Japanese novelist. He is best known around the
    world for his novels Kokoro, Botchan, I Am a
    Cat, Kusamakura and his unfinished work Light
    and Darkness. He was also a scholar of British
    literature and writer of haiku, kanshi, and fairy
    tales. From 1984 until 2004, his portrait appeared
    on the front of the Japanese 1,000 yen note.
    Summary: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    ===
    Original: Shibusawa Eiichi, 1st Viscount
    Shibusawa (March 16, 1840 – November 11,
    1931) was a Japanese industrialist use of double-
    entry accounting, joint-stock corporations and
    modern note-issuing banks…
    Summary:
    Knowledge: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    Dialog:
    A: I've been reading a lot lately.
    B: That's great! What are you reading?
    A: I've been reading books by Soseki Natsume.
    B: Hey, I know him! The person in the 1,000 yen
    bill, right?
    ===
    Knowledge: Eiichi Shibusawa was born in 1840
    in Kuroaraijima, Fukaya City. During his lifetime,
    he was involved in the establishment of
    approximately 500 companies and is known as
    the "father of Japanese capitalism”.
    Dialog:
    B: Hey, what’s up!
    A: How’s it going? Anything new with you lately?
    B: I heard that the new ten-thousand yen bill will
    have Shibusawa Eiichi on it, but do you know
    about him?
    A:
    4VNNBSZ3FTVMU
    GSPN)ZQFS$-07"
    4VNNBSJ[BUJPO 3FTQPOTF(FOFSBUJPO
    /BNFE&OUJUZ3FDPHOJUJPO
    4IJCVTBXB&JJDIJ
    4FBSDI3FTVMUPO
    4IJCVTBXB&JJDIJ
    %#4FBSDI FH8JLJQFEJB

    View Slide

  40. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    Original: Natsume Sōseki (9 February 1867 – 9
    December 1916), born Natsume Kin'nosuke, was
    a Japanese novelist. He is best known around the
    world for his novels Kokoro, Botchan, I Am a
    Cat, Kusamakura and his unfinished work Light
    and Darkness. He was also a scholar of British
    literature and writer of haiku, kanshi, and fairy
    tales. From 1984 until 2004, his portrait appeared
    on the front of the Japanese 1,000 yen note.
    Summary: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    ===
    Original: Shibusawa Eiichi, 1st Viscount
    Shibusawa (March 16, 1840 – November 11,
    1931) was a Japanese industrialist use of double-
    entry accounting, joint-stock corporations and
    modern note-issuing banks…
    Summary:
    Knowledge: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    Dialog:
    A: I've been reading a lot lately.
    B: That's great! What are you reading?
    A: I've been reading books by Soseki Natsume.
    B: Hey, I know him! The person in the 1,000 yen
    bill, right?
    ===
    Knowledge: Eiichi Shibusawa was born in 1840
    in Kuroaraijima, Fukaya City. During his lifetime,
    he was involved in the establishment of
    approximately 500 companies and is known as
    the "father of Japanese capitalism”.
    Dialog:
    B: Hey, what’s up!
    A: How’s it going? Anything new with you lately?
    B: I heard that the new ten-thousand yen bill will
    have Shibusawa Eiichi on it, but do you know
    about him?
    A:
    4VNNBSZ3FTVMU
    GSPN)ZQFS$-07"
    4VNNBSJ[BUJPO 3FTQPOTF(FOFSBUJPO
    /BNFE&OUJUZ3FDPHOJUJPO
    4IJCVTBXB&JJDIJ
    4FBSDI3FTVMUPO
    4IJCVTBXB&JJDIJ
    )BOENBEF4VNNBSZ
    %#4FBSDI FH8JLJQFEJB

    View Slide

  41. HyperCLOVA Prompt chaining can generate complex output
    *IFBSEUIBUUIFOFXUFOUIPVTBOE
    ZFOCJMMXJMMIBWF4IJCVTBXB&JJDIJPO
    JU CVUEPZPVLOPXBCPVUIJN
    Original: Natsume Sōseki (9 February 1867 – 9
    December 1916), born Natsume Kin'nosuke, was
    a Japanese novelist. He is best known around the
    world for his novels Kokoro, Botchan, I Am a
    Cat, Kusamakura and his unfinished work Light
    and Darkness. He was also a scholar of British
    literature and writer of haiku, kanshi, and fairy
    tales. From 1984 until 2004, his portrait appeared
    on the front of the Japanese 1,000 yen note.
    Summary: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    ===
    Original: Shibusawa Eiichi, 1st Viscount
    Shibusawa (March 16, 1840 – November 11,
    1931) was a Japanese industrialist use of double-
    entry accounting, joint-stock corporations and
    modern note-issuing banks…
    Summary:
    Knowledge: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    Dialog:
    A: I've been reading a lot lately.
    B: That's great! What are you reading?
    A: I've been reading books by Soseki Natsume.
    B: Hey, I know him! The person in the 1,000 yen
    bill, right?
    ===
    Knowledge: Eiichi Shibusawa was born in 1840
    in Kuroaraijima, Fukaya City. During his lifetime,
    he was involved in the establishment of
    approximately 500 companies and is known as
    the "father of Japanese capitalism”.
    Dialog:
    B: Hey, what’s up!
    A: How’s it going? Anything new with you lately?
    B: I heard that the new ten-thousand yen bill will
    have Shibusawa Eiichi on it, but do you know
    about him?
    A:
    4VNNBSJ[BUJPO 3FTQPOTF(FOFSBUJPO
    /BNFE&OUJUZ3FDPHOJUJPO
    4IJCVTBXB&JJDIJ
    4VNNBSZ3FTVMU
    GSPN)ZQFS$-07"
    4FBSDI3FTVMUPO
    4IJCVTBXB&JJDIJ
    )BOENBEF4VNNBSZ
    %#4FBSDI FH8JLJQFEJB

    View Slide

  42. HyperCLOVA Prompt chaining can generate complex output
    Natsume Sōseki (9 February 1867 – 9
    r 1916), born Natsume Kin'nosuke, was
    se novelist. He is best known around the
    his novels Kokoro, Botchan, I Am a
    makura and his unfinished work Light
    ness. He was also a scholar of British
    and writer of haiku, kanshi, and fairy
    m 1984 until 2004, his portrait appeared
    nt of the Japanese 1,000 yen note.
    y: Soseki Natsume was a Japanese
    His real name is Natsume Kinnosuke.
    it was used on the 1,000 yen bill.
    Shibusawa Eiichi, 1st Viscount
    a (March 16, 1840 – November 11,
    s a Japanese industrialist use of double-
    ounting, joint-stock corporations and
    ote-issuing banks…
    y:
    Knowledge: Soseki Natsume was a Japanese
    novelist. His real name is Natsume Kinnosuke.
    His portrait was used on the 1,000 yen bill.
    Dialog:
    A: I've been reading a lot lately.
    B: That's great! What are you reading?
    A: I've been reading books by Soseki Natsume.
    B: Hey, I know him! The person in the 1,000 yen
    bill, right?
    ===
    Knowledge: Eiichi Shibusawa was born in 1840
    in Kuroaraijima, Fukaya City. During his lifetime,
    he was involved in the establishment of
    approximately 500 companies and is known as
    the "father of Japanese capitalism”.
    Dialog:
    B: Hey, what’s up!
    A: How’s it going? Anything new with you lately?
    B: I heard that the new ten-thousand yen bill will
    have Shibusawa Eiichi on it, but do you know
    about him?
    A:
    4VNNBSJ[BUJPO 3FTQPOTF(FOFSBUJPO
    Yeah, I know him! He’s the one who laid
    the foundation for Japanese capitalism.
    He definitely deserves to be on the bill!
    4VNNBSZ3FTVMU
    GSPN)ZQFS$-07"
    4FBSDI3FTVMUPO
    4IJCVTBXB&JJDIJ
    )BOENBEF4VNNBSZ

    View Slide

  43. Agenda
    Realizing Applications with Prompting
    Performance Evaluation of 82B JP Model
    Developing Applications with HyperCLOVA
    Deploying Foundation Models in Real World

    View Slide

  44. 82B For 82B JP Model, we used…
    • Samples: 10B documents
    • Data size: 1.8TB
    • Token Size: 530B
    We do NOT use any data of our
    LINE Messenger Service
    • All messages on LINE
    • All posts on OpenChat
    Policy for Data Collection
    LINE LM Corpus
    Tokenizer
    Byte-level BPE tokenizer
    Library
    Megatron-LM
    Infrastructure
    NVIDIA Superpod
    128 clustered DGX servers
    1,024 A100 GPUs
    =
    Architecture
    Transformer
    Encoder-Decoder

    View Slide

  45. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/
    HyperCLOVA w/ prompting VS BERT-large
    TASK: RCQA* (answerable ones only)
    • Removed unanswerable questions
    from dataset of the normal RCQA task

    View Slide

  46. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/
    HyperCLOVA w/ prompting VS BERT-large
    - Questionɿதࠃ།Ұͷঁఇɺଇఱ෢޳͕ݐͯͨԦேͷ໊લ͸ԿͰ͠ΐ͏?
    - Contextɿ౰࣌ɺ౜Ԧே͸෢ଇఱʢଇఱ෢޳ʣʹΑΔცୣͰपԦேʹ୅Θ
    ͍ͬͯͨ͜ͱΛ೔ຊଆ͕೺Ѳ͍ͯ͠ͳ͔ͬͨͨΊɺ҄ాਅਓΒ͸ݱ஍Ͱ
    एׯͷࠞཚΛੜͨ͡ɻ·ͨɺ൴Β͕౎ɾ௕҆Ͱݟ࣮ͨࡍͷ౎৓΍཯ྩ੍
    ͷӡ༻࣮ଶ͸ɺ೔ຊࠃ಺Ͱͷ૝૾ͱ͸ࣅͯඇͳΔ΋ͷͰ͋ͬͨɻͨͱ͑
    ͹౻ݪژͰ͸େۃ఼ΛؚΉٶʢ౻ݪٶʣΛ౎৓ͷதԝʹ഑ஔ͍ͯͨ͠
    ͕ɺ௕҆৓Λ͸͡Ίͱ͢Δதࠃͷ౎৓Ͱ͸ଠۃٶΛؚΉߖ৓͸ɺ౎৓ͷ
    ๺୺தԝʹ͋Δͷ͕௨ྫͰ͋ͬͨɻ཯ྩͷӡ༻ܗଶ΋೔ຊͱ͸ҟͳΓɺ
    ཯ྩͷෆඋΛߦ͏֨ࣜͳͲ΋੍ఆ͞Ε͍ͯͨɻେ͖ͳিܸΛड͚ͯؼࠃ
    ͨ҄͠ాਅਓΒ͸ɺ͜ΕΒͷ೔தͷ౎৓΍཯ྩ੍ͷࠩҟΛใࠂ͠ɺͷͪ
    ͷվֵʹੜ͔͞Ε͍ͯ͘ɻ
    Example of an entry of RCQA possible only tasks
    TASK: RCQA* (answerable ones only)
    • Removed unanswerable questions
    from dataset of the normal RCQA task

    View Slide

  47. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/
    HyperCLOVA w/ prompting VS BERT-large
    - Questionɿதࠃ།Ұͷঁఇɺଇఱ෢޳͕ݐͯͨԦேͷ໊લ͸ԿͰ͠ΐ͏?
    - Contextɿ౰࣌ɺ౜Ԧே͸෢ଇఱʢଇఱ෢޳ʣʹΑΔცୣͰपԦேʹ୅Θ
    ͍ͬͯͨ͜ͱΛ೔ຊଆ͕೺Ѳ͍ͯ͠ͳ͔ͬͨͨΊɺ҄ాਅਓΒ͸ݱ஍Ͱ
    एׯͷࠞཚΛੜͨ͡ɻ·ͨɺ൴Β͕౎ɾ௕҆Ͱݟ࣮ͨࡍͷ౎৓΍཯ྩ੍
    ͷӡ༻࣮ଶ͸ɺ೔ຊࠃ಺Ͱͷ૝૾ͱ͸ࣅͯඇͳΔ΋ͷͰ͋ͬͨɻͨͱ͑
    ͹౻ݪژͰ͸େۃ఼ΛؚΉٶʢ౻ݪٶʣΛ౎৓ͷதԝʹ഑ஔ͍ͯͨ͠
    ͕ɺ௕҆৓Λ͸͡Ίͱ͢Δதࠃͷ౎৓Ͱ͸ଠۃٶΛؚΉߖ৓͸ɺ౎৓ͷ
    ๺୺தԝʹ͋Δͷ͕௨ྫͰ͋ͬͨɻ཯ྩͷӡ༻ܗଶ΋೔ຊͱ͸ҟͳΓɺ
    ཯ྩͷෆඋΛߦ͏֨ࣜͳͲ΋੍ఆ͞Ε͍ͯͨɻେ͖ͳিܸΛड͚ͯؼࠃ
    ͨ҄͠ాਅਓΒ͸ɺ͜ΕΒͷ೔தͷ౎৓΍཯ྩ੍ͷࠩҟΛใࠂ͠ɺͷͪ
    ͷվֵʹੜ͔͞Ε͍ͯ͘ɻ
    - Answerɿप
    Example of an entry of RCQA possible only tasks
    TASK: RCQA* (answerable ones only)
    • Removed unanswerable questions
    from dataset of the normal RCQA task

    View Slide

  48. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/
    HyperCLOVA w/ prompting VS BERT-large
    - Questionɿதࠃ།Ұͷঁఇɺଇఱ෢޳͕ݐͯͨԦேͷ໊લ͸ԿͰ͠ΐ͏?
    - Contextɿ౰࣌ɺ౜Ԧே͸෢ଇఱʢଇఱ෢޳ʣʹΑΔცୣͰपԦேʹ୅Θ
    ͍ͬͯͨ͜ͱΛ೔ຊଆ͕೺Ѳ͍ͯ͠ͳ͔ͬͨͨΊɺ҄ాਅਓΒ͸ݱ஍Ͱ
    एׯͷࠞཚΛੜͨ͡ɻ·ͨɺ൴Β͕౎ɾ௕҆Ͱݟ࣮ͨࡍͷ౎৓΍཯ྩ੍
    ͷӡ༻࣮ଶ͸ɺ೔ຊࠃ಺Ͱͷ૝૾ͱ͸ࣅͯඇͳΔ΋ͷͰ͋ͬͨɻͨͱ͑
    ͹౻ݪژͰ͸େۃ఼ΛؚΉٶʢ౻ݪٶʣΛ౎৓ͷதԝʹ഑ஔ͍ͯͨ͠
    ͕ɺ௕҆৓Λ͸͡Ίͱ͢Δதࠃͷ౎৓Ͱ͸ଠۃٶΛؚΉߖ৓͸ɺ౎৓ͷ
    ๺୺தԝʹ͋Δͷ͕௨ྫͰ͋ͬͨɻ཯ྩͷӡ༻ܗଶ΋೔ຊͱ͸ҟͳΓɺ
    ཯ྩͷෆඋΛߦ͏֨ࣜͳͲ΋੍ఆ͞Ε͍ͯͨɻେ͖ͳিܸΛड͚ͯؼࠃ
    ͨ҄͠ాਅਓΒ͸ɺ͜ΕΒͷ೔தͷ౎৓΍཯ྩ੍ͷࠩҟΛใࠂ͠ɺͷͪ
    ͷվֵʹੜ͔͞Ε͍ͯ͘ɻ
    - Answerɿप
    Example of an entry of RCQA possible only tasks
    Create Few-shot with a context, a question
    text and an answer
    • If the correct answer is contained and easily
    extracted from the inference result, we judged it
    is correct
    TASK: RCQA* (answerable ones only)
    • Removed unanswerable questions
    from dataset of the normal RCQA task

    View Slide

  49. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/
    HyperCLOVA w/ prompting VS BERT-large
    - Questionɿதࠃ།Ұͷঁఇɺଇఱ෢޳͕ݐͯͨԦேͷ໊લ͸ԿͰ͠ΐ͏?
    - Contextɿ౰࣌ɺ౜Ԧே͸෢ଇఱʢଇఱ෢޳ʣʹΑΔცୣͰपԦேʹ୅Θ
    ͍ͬͯͨ͜ͱΛ೔ຊଆ͕೺Ѳ͍ͯ͠ͳ͔ͬͨͨΊɺ҄ాਅਓΒ͸ݱ஍Ͱ
    एׯͷࠞཚΛੜͨ͡ɻ·ͨɺ൴Β͕౎ɾ௕҆Ͱݟ࣮ͨࡍͷ౎৓΍཯ྩ੍
    ͷӡ༻࣮ଶ͸ɺ೔ຊࠃ಺Ͱͷ૝૾ͱ͸ࣅͯඇͳΔ΋ͷͰ͋ͬͨɻͨͱ͑
    ͹౻ݪژͰ͸େۃ఼ΛؚΉٶʢ౻ݪٶʣΛ౎৓ͷதԝʹ഑ஔ͍ͯͨ͠
    ͕ɺ௕҆৓Λ͸͡Ίͱ͢Δதࠃͷ౎৓Ͱ͸ଠۃٶΛؚΉߖ৓͸ɺ౎৓ͷ
    ๺୺தԝʹ͋Δͷ͕௨ྫͰ͋ͬͨɻ཯ྩͷӡ༻ܗଶ΋೔ຊͱ͸ҟͳΓɺ
    ཯ྩͷෆඋΛߦ͏֨ࣜͳͲ΋੍ఆ͞Ε͍ͯͨɻେ͖ͳিܸΛड͚ͯؼࠃ
    ͨ҄͠ాਅਓΒ͸ɺ͜ΕΒͷ೔தͷ౎৓΍཯ྩ੍ͷࠩҟΛใࠂ͠ɺͷͪ
    ͷվֵʹੜ͔͞Ε͍ͯ͘ɻ
    - Answerɿप
    Example of an entry of RCQA possible only tasks
    Few-shots were created randomly by extracting a
    context from the RCQA dev-set for each inference
    Create Few-shot with a context, a question
    text and an answer
    • If the correct answer is contained and easily
    extracted from the inference result, we judged it
    is correct
    TASK: RCQA* (answerable ones only)
    • Removed unanswerable questions
    from dataset of the normal RCQA task

    View Slide

  50. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/





    # # # #

    )ZQFS$-07"+1 #&35MBSHF .


    HyperCLOVA w/ prompting VS BERT-large
    Acc.
    Parameters of HyperCLOVA
    Pre-training BERT-large by using
    subset of LINE LM corpus
    HyperCLOVA 6.9B w/ prompting is
    far below Fine-tuned BERT

    View Slide

  51. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/





    # # # #


    )ZQFS$-07"+1 #&35MBSHF .


    HyperCLOVA w/ prompting VS BERT-large
    Acc.
    Parameters of HyperCLOVA
    HyperCLOVA 13B JP cannot
    exceed Fine-tuned BERT-large
    by only prompting

    View Slide

  52. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/





    # # # #



    )ZQFS$-07"+1 #&35MBSHF .


    HyperCLOVA w/ prompting VS BERT-large
    Acc.
    Parameters of HyperCLOVA
    HyperCLOVA 39B JP is near
    BERT-large with fine tuning and
    parameter tuning by prompting

    View Slide

  53. Automated evaluation on RCQA* possible only tasks
    * ղ౴Մೳੑ෇͖ಡղσʔληοτ: http://www.cl.ecei.tohoku.ac.jp/rcqa/





    # # # #




    )ZQFS$-07"+1 #&35MBSHF .


    HyperCLOVA w/ prompting VS BERT-large
    Acc.
    Parameters of HyperCLOVA
    HyperCLOVA JP 82B with
    Prompting over BERT-large with full
    Fine-tuning and Parameter-tuning

    View Slide

  54. Prompt Chaining is Effective for Most of the Applications
    Writing Business Emails
    Writing Business Documents
    Writing Novels
    and more…

    View Slide

  55. Agenda
    Realizing Applications with Prompting
    Performance Evaluation of 82B JP Model
    Developing Applications with HyperCLOVA
    Deploying Foundation Models in Real World

    View Slide

  56. Dialog System

    View Slide

  57. • Many deployed models are rule-based
    • Looks for keywords to determine next response
    • Pros: Fully controllable
    • Cons: Lacks flexibility
    Rule-Based Dialog System
    )J IPX`TJUHPJOH"SFZPVSFBEZGPS5FDI7FSTF
    )PX`TJUHPJOH
    5FDI7FSTFJTBNB[JOH
    *`NHSFBU

    View Slide

  58. • Many deployed models are rule-based
    • Looks for keywords to determine next response
    • Pros: Fully controllable
    • Cons: Lacks flexibility
    Rule-Based Dialog System
    )J IPX`TJUHPJOH"SFZPVSFBEZGPS5FDI7FSTF
    )PX`TJUHPJOH
    5FDI7FSTFJTBNB[JOH
    *`NHSFBU
    Hey, do you like Brown?
    USER

    View Slide

  59. • Many deployed models are rule-based
    • Looks for keywords to determine next response
    • Pros: Fully controllable
    • Cons: Lacks flexibility
    Rule-Based Dialog System
    )J IPX`TJUHPJOH"SFZPVSFBEZGPS5FDI7FSTF
    )PX`TJUHPJOH
    5FDI7FSTFJTBNB[JOH
    *`NHSFBU
    Sorry, I didn’t understand the question.
    BOT
    Hey, do you like Brown?
    USER

    View Slide

  60. • Many deployed models are rule-based
    • Looks for keywords to determine next response
    • Pros: Fully controllable
    • Cons: Lacks flexibility
    Rule-Based Dialog System
    )J IPX`TJUHPJOH"SFZPVSFBEZGPS5FDI7FSTF
    )PX`TJUHPJOH
    5FDI7FSTFJTBNB[JOH
    *`NHSFBU
    Sorry, I didn’t understand the question.
    BOT
    Hey, do you like Brown?
    USER
    Guess what, I just got a LINE GIFT from my friend!
    USER

    View Slide

  61. • Many deployed models are rule-based
    • Looks for keywords to determine next response
    • Pros: Fully controllable
    • Cons: Lacks flexibility
    Rule-Based Dialog System
    )J IPX`TJUHPJOH"SFZPVSFBEZGPS5FDI7FSTF
    )PX`TJUHPJOH
    5FDI7FSTFJTBNB[JOH
    *`NHSFBU
    Sorry, I didn’t understand the question.
    BOT
    Hey, do you like Brown?
    USER
    Guess what, I just got a LINE GIFT from my friend!
    USER
    According to Wikipedia, LINE is a freeware app for instant communications...
    BOT

    View Slide

  62. • Many deployed models are rule-based
    • Looks for keywords to determine next response
    • Pros: Fully controllable
    • Cons: Lacks flexibility
    Rule-Based Dialog System
    )J IPX`TJUHPJOH"SFZPVSFBEZGPS5FDI7FSTF
    )PX`TJUHPJOH
    5FDI7FSTFJTBNB[JOH
    *`NHSFBU
    Sampl
    Sampl
    Sampl Sampl
    )J IPX`TJUHPJOH"SFZPVSFBEZGPS5FDI7FSTF
    *`NHSFBU :FT *`NTPSFBEZGPSUIFBXFTPNFQSFTFOUBUJPOT
    • Trending in dialog research field
    • Uses deep learning technique to generate response
    • Pros: very flexible and responses are interesting
    • Cons: Lacks controllability
    Generation-Based Dialog System

    View Slide

  63. View Slide

  64. Is a single prompt enough to build
    an amazing dialog system?
    %JBMPH
    1SPNQU
    6TFS
    6UUFSBODF
    4ZTUFN
    3FTQPOTF

    View Slide

  65. Probably No…
    A single few-shot prompt was just not enough

    View Slide

  66. Probably No…
    &OEDPOWFSTBUJPO
    WFSZFBSMZ
    A single few-shot prompt was just not enough

    View Slide

  67. Probably No…
    &OEDPOWFSTBUJPO
    WFSZFBSMZ
    #PSJOH
    SFTQPOTFT
    A single few-shot prompt was just not enough

    View Slide

  68. 4ZTUFN
    0VUQVU
    6TFS
    *OQVU
    HyperCLOVA-Based
    Dialog System Architecture

    View Slide

  69. 4ZTUFN
    0VUQVU
    6TFS
    *OQVU
    HyperCLOVA-Based
    Dialog System Architecture
    1SFQSPDFTT
    8IBUDBOXFEPIFSF
    1SPNQU1SPHSBNNJOHBOE
    1BSBNFUFS$POUSPMMJOH
    4VCUBTLTPMWJOHXJUI
    )ZQFS$-07"PS/-15FDI
    *OTUSVDUUIF
    EJSFDUJPOPGPVUQVU
    w 4FBSDIJOHLOPXMFEHFT
    w %FDJEJOHEJBMPHBDU
    w FUD

    View Slide

  70. 4ZTUFN
    0VUQVU
    6TFS
    *OQVU
    HyperCLOVA-Based
    Dialog System Architecture
    1SFQSPDFTT
    8IBUDBOXFEPIFSF
    1SPNQU1SPHSBNNJOHBOE
    1BSBNFUFS$POUSPMMJOH
    4VCUBTLTPMWJOHXJUI
    )ZQFS$-07"PS/-15FDI
    *OTUSVDUUIF
    EJSFDUJPOPGPVUQVU
    w 4FBSDIJOHLOPXMFEHFT
    w %FDJEJOHEJBMPHBDU
    w FUD
    (FOFSBUF
    $PSF0VUQVU

    View Slide

  71. 4ZTUFN
    0VUQVU
    3FHFOFSBUF
    6TFS
    *OQVU
    HyperCLOVA-Based
    Dialog System Architecture
    1SFQSPDFTT
    8IBUDBOXFEPIFSF
    1SPNQU1SPHSBNNJOHBOE
    1BSBNFUFS$POUSPMMJOH
    4VCUBTLTPMWJOHXJUI
    )ZQFS$-07"PS/-15FDI
    *OTUSVDUUIF
    EJSFDUJPOPGPVUQVU
    w 4FBSDIJOHLOPXMFEHFT
    w %FDJEJOHEJBMPHBDU
    w FUD
    1PTUQSPDFTT
    8IBUDBOXFEPIFSF
    'JMUFSJOHX/-15FDI
    "EEJOHBOE&EJUJOHX
    )ZQFS$-07"PS/-15FDI
    w $IFDLJOH)BMMVDJOBUJPO
    w $IFDLJOH&UIJDT
    w FUD
    %FUFDUPSDPSSFDU
    JNQSPQFSPVUQVUT
    (FOFSBUF
    $PSF0VUQVU

    View Slide

  72. HyperCLOVA-Based
    Dialog System Architecture
    %PFTUIF
    LOPXMFEHFCBTF
    IBWFVTFGVMEBUB
    ,OPXMFEHF#BTFE
    3FTQPOTF1SPNQU
    %PFTJUIBWF
    VTFGVMQFSTPOBEBUB
    1FSTPOB$POTJTUFOU
    3FTQPOTF1SPNQU
    (FOFSBM
    3FTQPOTF1SPNQU
    #PSJOH3FTQPOTF
    %FUFDUJPO
    $POWFSTBUJPO&OEJOH
    3FTQPOTF%FUFDUJPO
    .VMUJ5VSO
    3FTQPOTF1SPNQU
    1SPIJCJUFE8PSE
    %FUFDUJPO
    5P3FHFOFSBUJPO
    3FHFOFSBUJPO
    :FT
    :FT
    /P
    /P 1BTTFE
    %FUFDUFE
    1BTTFE
    5P3FHFOFSBUJPO
    %FUFDUFE
    %FUFDUFE
    4ZTUFN
    0VUQVU
    6TFS
    *OQVU

    View Slide

  73. HyperCLOVA-Based
    Dialog System Architecture
    %PFTUIF
    LOPXMFEHFCBTF
    IBWFVTFGVMEBUB
    ,OPXMFEHF#BTFE
    3FTQPOTF1SPNQU
    %PFTJUIBWF
    VTFGVMQFSTPOBEBUB
    1FSTPOB$POTJTUFOU
    3FTQPOTF1SPNQU
    (FOFSBM
    3FTQPOTF1SPNQU
    #PSJOH3FTQPOTF
    %FUFDUJPO
    $POWFSTBUJPO&OEJOH
    3FTQPOTF%FUFDUJPO
    .VMUJ5VSO
    3FTQPOTF1SPNQU
    1SPIJCJUFE8PSE
    %FUFDUJPO
    5P3FHFOFSBUJPO
    3FHFOFSBUJPO
    :FT
    :FT
    /P
    /P 1BTTFE
    %FUFDUFE
    1BTTFE
    5P3FHFOFSBUJPO
    %FUFDUFE
    %FUFDUFE
    &YUFSOBM,OPXMFEHF
    4ZTUFN
    0VUQVU
    6TFS
    *OQVU

    View Slide

  74. HyperCLOVA-Based
    Dialog System Architecture
    %PFTUIF
    LOPXMFEHFCBTF
    IBWFVTFGVMEBUB
    ,OPXMFEHF#BTFE
    3FTQPOTF1SPNQU
    %PFTJUIBWF
    VTFGVMQFSTPOBEBUB
    1FSTPOB$POTJTUFOU
    3FTQPOTF1SPNQU
    (FOFSBM
    3FTQPOTF1SPNQU
    #PSJOH3FTQPOTF
    %FUFDUJPO
    $POWFSTBUJPO&OEJOH
    3FTQPOTF%FUFDUJPO
    .VMUJ5VSO
    3FTQPOTF1SPNQU
    1SPIJCJUFE8PSE
    %FUFDUJPO
    5P3FHFOFSBUJPO
    3FHFOFSBUJPO
    :FT
    :FT
    /P
    /P 1BTTFE
    %FUFDUFE
    1BTTFE
    5P3FHFOFSBUJPO
    %FUFDUFE
    %FUFDUFE
    &YUFSOBM,OPXMFEHF
    4ZTUFN
    0VUQVU
    6TFS
    *OQVU
    *NQSPWJOH$POTJTUFODZ
    w )ZQFS$-07"#BTFE1FSTPOB&YUSBDUJPO4FMFDUJPO6UJMJ[BUJPO ,BXBNPUP

    View Slide

  75. HyperCLOVA-Based
    Dialog System Architecture
    %PFTUIF
    LOPXMFEHFCBTF
    IBWFVTFGVMEBUB
    ,OPXMFEHF#BTFE
    3FTQPOTF1SPNQU
    %PFTJUIBWF
    VTFGVMQFSTPOBEBUB
    1FSTPOB$POTJTUFOU
    3FTQPOTF1SPNQU
    (FOFSBM
    3FTQPOTF1SPNQU
    #PSJOH3FTQPOTF
    %FUFDUJPO
    $POWFSTBUJPO&OEJOH
    3FTQPOTF%FUFDUJPO
    .VMUJ5VSO
    3FTQPOTF1SPNQU
    1SPIJCJUFE8PSE
    %FUFDUJPO
    5P3FHFOFSBUJPO
    3FHFOFSBUJPO
    :FT
    :FT
    /P
    /P 1BTTFE
    %FUFDUFE
    1BTTFE
    5P3FHFOFSBUJPO
    %FUFDUFE
    %FUFDUFE
    &YUFSOBM,OPXMFEHF
    4ZTUFN
    0VUQVU
    6TFS
    *OQVU
    *ODSFBTJOH"NPVOUPG3FTQPOTFT
    *NQSPWJOH$POTJTUFODZ
    w )ZQFS$-07"#BTFE1FSTPOB&YUSBDUJPO4FMFDUJPO6UJMJ[BUJPO ,BXBNPUP

    View Slide

  76. HyperCLOVA-Based
    Dialog System Architecture
    %PFTUIF
    LOPXMFEHFCBTF
    IBWFVTFGVMEBUB
    ,OPXMFEHF#BTFE
    3FTQPOTF1SPNQU
    %PFTJUIBWF
    VTFGVMQFSTPOBEBUB
    1FSTPOB$POTJTUFOU
    3FTQPOTF1SPNQU
    (FOFSBM
    3FTQPOTF1SPNQU
    #PSJOH3FTQPOTF
    %FUFDUJPO
    $POWFSTBUJPO&OEJOH
    3FTQPOTF%FUFDUJPO
    .VMUJ5VSO
    3FTQPOTF1SPNQU
    1SPIJCJUFE8PSE
    %FUFDUJPO
    5P3FHFOFSBUJPO
    3FHFOFSBUJPO
    :FT
    :FT
    /P
    /P 1BTTFE
    %FUFDUFE
    1BTTFE
    5P3FHFOFSBUJPO
    %FUFDUFE
    %FUFDUFE
    &YUFSOBM,OPXMFEHF
    "WPJE&OEJOH
    $POWFSTBUJPO
    4ZTUFN
    0VUQVU
    6TFS
    *OQVU
    *ODSFBTJOH"NPVOUPG3FTQPOTFT
    *NQSPWJOH$POTJTUFODZ
    w )ZQFS$-07"#BTFE1FSTPOB&YUSBDUJPO4FMFDUJPO6UJMJ[BUJPO ,BXBNPUP

    View Slide

  77. With Preprocess and Postprocess
    Natural and engaging responses
    Only Few-Shot Prompt
    Bland and unexciting responses
    Comparing w/ and wo/
    External Modules
    BOT USER
    BOT USER

    View Slide

  78. With Preprocess and Postprocess
    Natural and engaging responses
    Only Few-Shot Prompt
    Bland and unexciting responses
    Comparing w/ and wo/
    External Modules
    BOT USER
    BOT USER

    View Slide

  79. Our system's achievements in dialog competitions
    Dialog System Live Competition 4 (2021)
    Dialogue system researchers gather and do
    live-evaluations of submitted chatbots
    Dialog Robot Competition 2022
    A competition of controlling humanoid robot that can
    communicate and recommend a tourist sight
    inviting boss to a party
    Situation Track
    Situated
    Chat
    TU1MBDF
    X)ZQFS$-07"#
    Open Track
    Open-Domain
    Chitchat
    w/ random topics
    TU1MBDF
    X)ZQFS$-07"#
    recommending a spot
    Main Competition
    Tourist-
    Guide Robot
    TU1MBDF
    X)ZQFS$-07"#

    View Slide

  80. Is that technology only for Dialog Systems?

    View Slide

  81. HyperCLOVA Writer

    View Slide

  82. View Slide

  83. View Slide

  84. Email Generation w/ HyperCLOVA Writer
    Email written by HyperCLOVA Writer
    Rough Plot by Human

    View Slide

  85. Email Generation w/ HyperCLOVA Writer
    Email written by HyperCLOVA Writer
    Rough Plot by Human
    Postprocess: NER Validity Check

    View Slide

  86. HyperCLOVA Writer is Extensible
    NLP APIs

    View Slide

  87. /api/email /api/minute
    HyperCLOVA Writer is Extensible
    NLP APIs
    /api/novel …
    How can HyperCLOVA help you?
    Please tell us at Twitter w/ #HyperCLOVA
    ?????

    View Slide

  88. What’s the goal of HyperCLOVA Writer?
    Human Writing
    Plan / Search / Write

    View Slide

  89. What’s the goal of HyperCLOVA Writer?
    Human Writing
    Plan / Search / Write
    Writing w/ HyperCLOVA
    Plan Generate Fix Tea Time

    View Slide

  90. What’s Next for HyperCLOVA Writer

    View Slide

  91. Agenda
    Realizing Applications with Prompting
    Performance Evaluation of 82B JP Model
    Developing Applications with HyperCLOVA
    Deploying Foundation Models in Real World

    View Slide

  92. 82B
    Are foundation models perfectly
    ready for deployment?

    View Slide

  93. Definitely NOT Yet
    5FDI7FSTFJTBOPOMJOFUFDIOJDBMDPOGFSFODF
    XIJDIXJMMCFIFMEBTBKPJOUQSPKFDUCFUXFFO-*/&BOE
    :BIPP+"1"/PO/PWFNCFSUIBOEUI"EJWFSTF
    BSSBZPGNFNCFSTGSPNUIFUXPIPTUDPNQBOJFTBOE
    PUIFSQBSUJDJQBUJOHPSHBOJ[BUJPOTXJMMTIBSFUIFJS
    DVUUJOHFEHFDIBMMFOHFTBOEBDDVNVMBUFELOPXMFEHF
    Fairness and Ethics of Generated Text
    Accuracy and Reliability of Generated Text
    Dealing with the Cost per Inference by a Model
    There are still many things to think about…

    View Slide

  94. Our First Step Towards Ethical AI
    5FDI7FSTFJTBOPOMJOFUFDIOJDBMDPOGFSFODF
    XIJDIXJMMCFIFMEBTBKPJOUQSPKFDUCFUXFFO-*/&BOE
    :BIPP+"1"/PO/PWFNCFSUIBOEUI"EJWFSTF
    BSSBZPGNFNCFSTGSPNUIFUXPIPTUDPNQBOJFTBOE
    PUIFSQBSUJDJQBUJOHPSHBOJ[BUJPOTXJMMTIBSFUIFJS
    DVUUJOHFEHFDIBMMFOHFTBOEBDDVNVMBUFELOPXMFEHF

    View Slide

  95. Ethics Check in Dialog System
    %PFTUIF
    LOPXMFEHFCBTF
    IBWFVTFGVMEBUB
    ,OPXMFEHF#BTFE
    3FTQPOTF1SPNQU
    %PFTJUIBWF
    VTFGVMQFSTPOBEBUB
    1FSTPOB$POTJTUFOU
    3FTQPOTF1SPNQU
    (FOFSBM
    3FTQPOTF1SPNQU
    #PSJOH3FTQPOTF
    %FUFDUJPO
    $POWFSTBUJPO&OEJOH
    3FTQPOTF%FUFDUJPO
    .VMUJ5VSO
    3FTQPOTF1SPNQU
    1SPIJCJUFE8PSE
    %FUFDUJPO
    5P3FHFOFSBUJPO
    3FHFOFSBUJPO
    :FT
    :FT
    /P
    /P 1BTTFE
    %FUFDUFE
    6TFS
    6UUFSBODF
    4ZTUFN
    3FTQPOTF
    1BTTFE
    5P3FHFOFSBUJPO
    %FUFDUFE
    %FUFDUFE

    View Slide

  96. Ethics Check in Dialog System
    %PFTUIF
    LOPXMFEHFCBTF
    IBWFVTFGVMEBUB
    ,OPXMFEHF#BTFE
    3FTQPOTF1SPNQU
    %PFTJUIBWF
    VTFGVMQFSTPOBEBUB
    1FSTPOB$POTJTUFOU
    3FTQPOTF1SPNQU
    (FOFSBM
    3FTQPOTF1SPNQU
    #PSJOH3FTQPOTF
    %FUFDUJPO
    $POWFSTBUJPO&OEJOH
    3FTQPOTF%FUFDUJPO
    .VMUJ5VSO
    3FTQPOTF1SPNQU
    1SPIJCJUFE8PSE
    %FUFDUJPO
    5P3FHFOFSBUJPO
    3FHFOFSBUJPO
    :FT
    :FT
    /P
    /P 1BTTFE
    %FUFDUFE
    6TFS
    6UUFSBODF
    4ZTUFN
    3FTQPOTF
    1BTTFE
    5P3FHFOFSBUJPO
    %FUFDUFE
    %FUFDUFE
    "WPJE0⒎FOTJWF-BOHVBHFT

    View Slide

  97. Quick Fact Checking to Avoid Hallucination
    LINE and Yahoo! JAPAN will hold the "Tech-
    Verse" conference online (live streaming format)
    on Thursday, November 17 and Friday, November
    18, 2022. Up until now, LINE and Yahoo Japan
    have held annual technology conferences as
    "LINE DEVELOPER DAY" and "Yahoo! JAPAN
    Tech Conference" respectively, but this year, for
    the first time, LINE and Yahoo Japan will hold a
    joint event, "Tech-Verse JAPAN Tech Conference,
    but this year, for the first time, LINE and Yahoo! ...
    Original Sentence
    Summarizer
    The overview of the sessions and speakers are
    available on the official website.
    LINE and Yahoo will jointly hold a technology
    conference.
    The theme of the conference will be "Pioneering
    the Future Society with Technology.”
    The official website, which went live today,
    provides an overview of the 80 sessions and
    speakers, with more details on the panel
    discussions to follow.

    View Slide

  98. Quick Fact Checking to Avoid Hallucination
    LINE and Yahoo! JAPAN will hold the "Tech-
    Verse" conference online (live streaming format)
    on Thursday, November 17 and Friday, November
    18, 2022. Up until now, LINE and Yahoo Japan
    have held annual technology conferences as
    "LINE DEVELOPER DAY" and "Yahoo! JAPAN
    Tech Conference" respectively, but this year, for
    the first time, LINE and Yahoo Japan will hold a
    joint event, "Tech-Verse JAPAN Tech Conference,
    but this year, for the first time, LINE and Yahoo! ...
    Original Sentence
    Summarizer
    The overview of the sessions and speakers are
    available on the official website.
    LINE and Yahoo will jointly hold a technology
    conference.
    The theme of the conference will be "Pioneering
    the Future Society with Technology.”
    The official website, which went live today,
    provides an overview of the 80 sessions and
    speakers, with more details on the panel
    discussions to follow.

    View Slide

  99. Quick Fact Checking to Avoid Hallucination
    LINE and Yahoo! JAPAN will hold the "Tech-
    Verse" conference online (live streaming format)
    on Thursday, November 17 and Friday, November
    18, 2022. Up until now, LINE and Yahoo Japan
    have held annual technology conferences as
    "LINE DEVELOPER DAY" and "Yahoo! JAPAN
    Tech Conference" respectively, but this year, for
    the first time, LINE and Yahoo Japan will hold a
    joint event, "Tech-Verse JAPAN Tech Conference,
    but this year, for the first time, LINE and Yahoo! ...
    Original Sentence
    Summarizer
    The overview of the sessions and speakers are
    available on the official website.
    LINE and Yahoo will jointly hold a technology
    conference.
    The theme of the conference will be "Pioneering
    the Future Society with Technology.”
    The official website, which went live today,
    provides an overview of the 80 sessions and
    speakers, with more details on the panel
    discussions to follow.

    View Slide

  100. Quick Fact Checking to Avoid Hallucination
    LINE and Yahoo! JAPAN will hold the "Tech-
    Verse" conference online (live streaming format)
    on Thursday, November 17 and Friday, November
    18, 2022. Up until now, LINE and Yahoo Japan
    have held annual technology conferences as
    "LINE DEVELOPER DAY" and "Yahoo! JAPAN
    Tech Conference" respectively, but this year, for
    the first time, LINE and Yahoo Japan will hold a
    joint event, "Tech-Verse JAPAN Tech Conference,
    but this year, for the first time, LINE and Yahoo! ...
    Original Sentence
    Summarizer
    The overview of the sessions and speakers are
    available on the official website.
    LINE and Yahoo will jointly hold a technology
    conference.
    The theme of the conference will be "Pioneering
    the Future Society with Technology.”
    The official website, which went live today,
    provides an overview of the 80 sessions and
    speakers, with more details on the panel
    discussions to follow.

    ?

    View Slide

  101. Quick Fact Checking to Avoid Hallucination
    $POWFSUTFOUFODFTUPHSBQIT
    'JOEHSBQINJTNBUDI
    Hallucinated

    View Slide

  102. Human Generating
    Writing w/ Foundation Model
    Planning &
    Planning
    Writing / Searching
    Fixing
    & Fixing
    Generating ?
    Can we actually reduce labor cost by using Foundation Model?

    View Slide

  103. Can we actually reduce labor cost by using Foundation Model?
    A: Depends!

    View Slide

  104. Can we actually reduce labor cost by using Foundation Model?

    View Slide

  105. ! Good Case
    Plan / Write
    Plan Fix
    Gen.
    Can we actually reduce labor cost by using Foundation Model?

    View Slide

  106. ! Good Case
    Plan / Write
    Plan Fix
    Gen.
    " Bad Case 1
    Plan / Write
    Plan Fix
    Gen.
    Can we actually reduce labor cost by using Foundation Model?

    View Slide

  107. ! Good Case
    Plan / Write
    Plan Fix
    Gen.
    " Bad Case 1
    Plan / Write
    Plan Fix
    Gen.
    " Bad Case 2
    Plan / Write
    Plan Fix
    Gen. × 3
    Can we actually reduce labor cost by using Foundation Model?

    View Slide

  108. Utilization of parameter-saving foundation model
    * JGLUE: Japanese General Language Understanding Evaluation - https://github.com/yahoojapan/JGLUE
    HyperCLOVA w/ LoRA tuning VS Waseda RoBERTa large
    {"q_id": 3016,
    "question": "ձࣾͷ࠷ߴ੹೚ऀΛԿͱ͍͏͔ʁ (What
    do you call the chief executive officer of a company?)",
    "choice0": "ࣾ௕ (president)",
    "choice1": "ڭࢣ (teacher)",
    "choice2": "෦௕ (manager)",
    "choice3": "όΠτ (part-time worker)",
    "choice4": "෦Լ (subordinate)",
    "label": 0}
    Example of an entry of JCommonsenseQA tasks
    Verification of LoRA performance of
    HyperCLOVA Japanese 6.7B model
    • Unlike Prompting, a solution is possible in 0-shot
    due to tuning by supervised data
    TASK: JCommonsenseQA
    • Japanese version of CommonsenseQA dataset
    • Five-choice QA questions to assess common
    sense reasoning ability
    This experiment was the result of a
    summer internship 2022 at LINE Corp.
    • More details will be provided at NLP2023

    View Slide

  109. Experimental results in JCommonsenseQA
    HyperCLOVA w/ LoRA tuning VS Waseda RoBERTa large
    Human
    Tohoku BERT
    Large
    Waseda RoBERTa
    Large (s128)
    HyperCLOVA 6.7B
    w/ LoRA
    Accuracy
    0.8 0.85 0.9 0.95 1
    93.6%
    90.7%
    81.6%
    98.8%
    +3.0%
    +14.7%
    -5.5%
    HyperCLOVA JP 6.7B with
    LoRA tuning exceeds
    JGLUE baseline accuracy

    View Slide

  110. Challenges in Multimodal Systems

    View Slide

  111. Tasks that had been solved by CLOVA Assistant
    NLP
    Speech
    Recognition Text
    Speech
    Synthesis
    Text

    View Slide

  112. View Slide

  113. This is one of our “reference” dialog system
    Open-domain
    Dialog System
    (e.g. For Dialog system competition 4
    w/ HyperCLOVA JP 39B)
    Speech
    Recognition
    (LINE’s Speech-to-Text system)
    Speech
    Synthesis
    (LINE’s Text-to-Speech system)
    Text Text
    Voice
    Voice
    Chatting
    Smooth response
    w/ beautiful voice
    A dialogue system beyond the “Uncanny Valley” is imminent
    Wow

    View Slide

  114. Challenge: Active handling of non-textual data in NLP
    NLP
    Speech
    Recognition
    Text
    Speech
    Synthesis
    Text
    Computer
    Vision
    Features
    Features Control
    Control
    Robot / Avatar

    View Slide

  115. Challenge: Active handling of non-textual data in NLP
    NLP
    Speech
    Recognition
    Text
    Speech
    Synthesis
    Text
    Computer
    Vision
    Features
    Features Control
    Control
    Robot / Avatar
    In the future, NLP engineers
    need to acquire features
    from many modalities for NLP
    that other fields do not need.

    View Slide

  116. Challenge: Active handling of non-textual data in NLP
    NLP
    Speech
    Recognition
    Text
    Speech
    Synthesis
    Text
    Computer
    Vision
    Features
    Features Control
    Control
    Robot / Avatar
    We also need to send
    signals to control
    everything that other fields
    are not willing to do.
    In the future, NLP engineers
    need to acquire features
    from many modalities for NLP
    that other fields do not need.

    View Slide

  117. Foundation Model is necessary for Multi-Modal
    - The history of NLP is strongly linked to the
    development of AI-related technologies
    - The widespread use of the Foundation Model has
    made it easier to leverage the output of speech
    and image recognition
    - LINE wants to move in the direction of R&D of
    Multimodal NLP with our own foundation models
    Foundation Model
    DNN
    Traditional only
    ML
    Rule only
    Small LM only
    Text only Multi-Modal

    View Slide

  118. Agenda
    Realizing applications with Prompting
    Performance Evaluation of 82B JP Model
    Developing Applications with HyperCLOVA
    Deploying Foundation Models in Real World
    When will the Japanese version of HyperCLOVA be released?

    View Slide

  119. Agenda
    Realizing applications with Prompting
    Performance Evaluation of 82B JP Model
    Developing Applications with HyperCLOVA
    Deploying Foundation Models in Real World
    When will the Japanese version of HyperCLOVA be released?

    View Slide

  120. CLOVA Studio is already released for KR model

    View Slide

  121. CLOVA Studio for JP model will come at FY2023

    View Slide

  122. Summary
    - A foundation model application is already in a state to produce results that are
    useful enough for humans, and we are working hard to make it available to you!
    - However, it is necessary to prepare many subsystems to compensate for the
    missing functions in the foundation model
    - LINE is R&D these systems with the aim of providing them to you as well
    - LINE has successfully built a Japanese model of HyperCLOVA's 82B. We
    shared evaluation results and know-how for application

    View Slide

  123. Thank you!

    View Slide