FunctionChat-Bench: 도구 연결 대화에서의 언어 모델 생성 능력 평가

--. -BSHF-BOHVBHF.PEFM

--. ঱যݽ؛੉ೡࣻহחѪ਷

पदрࢿ ز੘ࢿ पदр੿ࠁ੽Ӕ पࣁ҅ز੘ࣻ೯ ঱যݽ؛੄ೠ҅

'VODUJPO$BMM

'VODUJPO$BMMӝמ੉ۆ 'VODUJPO$BMMࢿמ਷যڌѱಣоೡө ࢜۽਍'VODUJPO$BMMࢿמಣоߑߨۿࣻ݀ӝ 'VODUJPO$IBU - #FODIࢎਊߨ ೱറѐߊ҅ദ

"* 6TFS tನܰైтࣻبоয٣ঠ u tܻझࠄੑפ׮u

"* 6TFS t૑Әܻझࠄզॿחযٸ u tपदр੿ࠁחઁҕೡࣻহणפ׮u

6TFS "* 5PPM

6TFS "* 5PPM t૑Әܻझࠄզॿחযٸ u

6TFS "* 5PPM t૑Әܻझࠄզॿחযٸ u tHFU$VSSFOU8FBUIFS ܻझࠄ u

6TFS "* 5PPM t૑Әܻझࠄզॿחযٸ u tHFU$VSSFOU8FBUIFS ܻझࠄ u \tXFBUIFSut 4VOOZu^

6TFS "* 5PPM t૑Әܻझࠄզॿחযٸ u tHFU$VSSFOU8FBUIFS ܻझࠄ u \tXFBUIFSut 4VOOZu^
tഅ੤ܻझࠄ਷੉Ҋ ݉णפ׮u

'VODUJPO$BMM ঱যݽ؛੉੗୓੸ਵ۽ࣻ೯ೡࣻহחपࣁ҅ز੘ਸ૑दೞѢաࢎ੹ী೟णೞ૑ঋ਷੿ࠁܳ पदрਵ۽߉ইৢࣻ੓ب۾঱যݽ؛җ"1*١੄৻ࠗبҳܳোѾೞחӝמ ঱যݽ؛ 'VODUJPO$BMM *OQVU NFTTBHFT ࣻߊनݫद૑ղ৉ UPPMT
ഐ୹оמೠೣࣻݾ۾ 0VUQVU ؀ചഋ੗োযݫद૑ ೣࣻഐ୹ݫद૑ +40/0CKFDU 3FDJQJFOU VTFS UPPM 3PMF TZTUFN VTFS BTTJTUBOU UPPM

঱যݽ؛ࢿמಣо

7JFX ݽ؛ѐࢶ ࢿמಣо ੿ഛೞҊ ਬ੄޷ೠ ૓ױ ݽ؛ࢶఖ

IUUQT PQFOBJDPNJOEFYGVODUJPO - DBMMJOH - BOE - PUIFS -
BQJ - VQEBUFT] IUUQT BJHPPHMFEFWHFNJOJ - BQJEPDTGVODUJPO - DBMMJOH] IUUQT HJUIVCDPN.FFU,BJGVODUJPOBSZ 'VODUJPO$BMM૑ਗ੄ഛ੢

'VODUJPO$BMMࢿמಣо IUUQT HPSJMMBDTCFSLFMFZFEVCMPHT@CFSLFMFZ@GVODUJPO@DBMMJOH@MFBEFSCPBSEIUNM] IUUQT HJUIVCDPNNJDSPTPGU5PPM5BML

{ "name": "get_movie_details", "description": "хة, ੢ܰ, ಣ੼ ١ ৔ച੄ ࢚ࣁ
੿ࠁܳ ઁҕ೤פ׮.", "parameters": { "type": "object", "properties": { "title": { "type": "string", "description": "৔ച ઁݾ" }, "year": { "type": "integer", "description": "৔ച ѐࠉ োب" } }, "required": [ "title" ] } } { "name": "get_movie_details", "arguments": "{\"title\": \"بقٜ\", \"year\": 2012}" } ೣࣻഐ୹ݫद૑ࢤࢿ੄੿ഛࢿ ֙ীѐࠉೠ৔ചبقٜ ੿ࠁખঌ۰઴ې

{ "name": "get_movie_details", "arguments": "{\"title\": \"بقٜ\", \"year\": 2012}" } ৔࢚ࢸݺఫझ౟ӝੑ
ೣࣻഐ୹ݫद૑ࢤࢿ੄੿ഛࢿ { "name": "get_movie_details", "description": "хة, ੢ܰ, ಣ੼ ١ ৔ച੄ ࢚ࣁ ੿ࠁܳ ઁҕ೤פ׮.", "parameters": { "type": "object", "properties": { "title": { "type": "string", "description": "৔ച ઁݾ" }, "year": { "type": "integer", "description": "৔ച ѐࠉ োب" } }, "required": [ "title" ] } } ੸੺ೠೣࣻܳࢶఖ೮חо ೣࣻ੉ܴਸ੿ഛೞѱ୹۱೮חо ౵ۄ޷ఠী೧׼ೞח੿ࠁܳ੸੺൤୶୹೮חо ౵ۄ޷ఠ੄пчਸ੿੄ػನݘਵ۽ࢤࢿ೮חо

'VODUJPO$BMM ঱যݽ؛੉੗୓੸ਵ۽ࣻ೯ೡࣻহחपࣁ҅ز੘ਸ૑दೞѢաࢎ੹ী೟णೞ૑ঋ਷੿ࠁܳ पदрਵ۽߉ইৢࣻ੓ب۾঱যݽ؛җ"1*١੄৻ࠗبҳܳোѾೞחӝמ ঱যݽ؛ 'VODUJPO$BMM *OQVU NFTTBHFT ࣻߊनݫद૑ղ৉ UPPMT
ഐ୹оמೠೣࣻݾ۾ 0VUQVU ؀ചഋ੗োযݫद૑ ೣࣻഐ୹ݫद૑ +40/0CKFDU 3FDJQJFOU VTFS UPPM 3PMF TZTUFN VTFS BTTJTUBOU UPPM

*OQVU ಽҊ੗ೠޙઁ

ࢎप੸ੋ׮নೠ*OQVU ಽҊ੗ೠޙઁ

{ "type": "function", "function": { "name": "addMemo", "description": “ݫݽ੢ী ࢜۽਍
ݫݽܳ ୶о", "parameters": { "type": "object", "properties": { "content": { "type": "string", "description": "ݫݽ ղਊ" } }, "required": [ "content" ] } } } { "name": "addMemo", "arguments": "{\"content\": “\஠஠য় ѐߊ੗ ஶಌ۠झ 2024.10.22-24\"}" } ಽҊ੗ೠޙઁ ஠஠য়ѐߊ੗ஶಌ۠झ ੉Ѣݫݽ੢ী੷੢೧઻

ݫݽܳ ୶о", "parameters": { "type": "object", "properties": { "content": { "type": "string", "description": "ݫݽ ղਊ" } }, "required": [ "content" ] } } } ಽҊ੗ೠޙઁ ஠஠য়ѐߊ੗ஶಌ۠झ ੉Ѣݫݽ੢ী੷੢೧઻ ݫݽ੷੢ӝמ਷হणפ׮

ݫݽܳ ୶о", "parameters": { "type": "object", "properties": { "content": { "type": "string", "description": "ݫݽ ղਊ" } }, "required": [ "content" ] } } } ಽҊ੗ೠޙઁ ஠஠য়ѐߊ੗ஶಌ۠झ ੉Ѣݫݽ੢ী੷੢೧઻

ݫݽܳ ୶о", "parameters": { "type": "object", "properties": { "content": { "type": "string", "description": "ݫݽ ղਊ" } }, "required": [ "content" ] } } } ಽҊ੗ೠޙઁ ݫݽ੢ীݫݽೞա੷੢೧઻

ݫݽܳ ୶о", "parameters": { "type": "object", "properties": { "content": { "type": "string", "description": "ݫݽ ղਊ" } }, "required": [ "content" ] } } } ಽҊ੗ೠޙઁ ݫݽ੢ীݫݽೞա੷੢೧઻ ޤۄҊ੷੢ೡөਃ ݫݽղਊਸঌ۰઱ࣁਃ

߮஖݃௼ѐߊҗ੿ بҳࢎਊ؀ചदաܻয়੘ࢿ ӝ؀׹উ ೲਊ׹উ੘ࢿ ಣоझ௼݀౟ѐߊ ಣоೞ۰חמ۱ী؀ೠ੿੄ ಣоਃࣗࣻ݀ ؘ੉ఠࣇҳࢿ߂ನݘ٣੗ੋ ಣоӝળ੘ࢿ झ௏য࢑੿ߑधࣻ݀
Ѿҗܻನ౟ߑधࣻ݀ ׮নೠݽ؛߈ࠂಣо ਬബࢿѨૐ

IUUQT HJUIVCDPNLBLBP'VODUJPO$IBU - #FODI USER USER I walked from Hangeul
Museum today. How long did I walk? USER (Slot Question) AI {"origin": "Hangeul Museum", "destination": "Hannam police precinct", "distance": 3.3, "distance_unit": "km", "time": {"hours": null, "minutes": 52}} TOOL (Answer Completion) The walking distance from Hangeul Museum to Hannam police precinct is approximately 3.3km and takes approximately 52 minutes. AI (Relevance Detection) That sounds like a refreshing walk! It's great to hear that. AI (Relevance Detection) I'm sorry, I can't provide weather forecasts. AI USER I walked to the Hannam police precinct. I walked quite a bit. The weather was nice and I felt really good. What's the weather like tomorrow? I'll try walking tomorrow too. Please tell me the weather forecast. (Tool Call) AI "function": {"name": "getWalkInfo", "arguments": "{\"origin\": \"Hangeul Museum\", \"destination\": \"Hannam police precinct\"}"}} Please let me know your destination to calculate walking distance and walking time.

IUUQT HJUIVCDPNLBLBP'VODUJPO$IBU - #FODI ೣࣻഐ୹ݫद૑ࢤࢿ USER USER I walked from
Hangeul Museum today. How long did I walk? USER (Slot Question) AI {"origin": "Hangeul Museum", "destination": "Hannam police precinct", "distance": 3.3, "distance_unit": "km", "time": {"hours": null, "minutes": 52}} TOOL (Answer Completion) The walking distance from Hangeul Museum to Hannam police precinct is approximately 3.3km and takes approximately 52 minutes. AI (Relevance Detection) That sounds like a refreshing walk! It's great to hear that. AI (Relevance Detection) I'm sorry, I can't provide weather forecasts. AI USER I walked to the Hannam police precinct. I walked quite a bit. The weather was nice and I felt really good. What's the weather like tomorrow? I'll try walking tomorrow too. Please tell me the weather forecast. (Tool Call) AI "function": {"name": "getWalkInfo", "arguments": "{\"origin\": \"Hangeul Museum\", \"destination\": \"Hannam police precinct\"}"}} Please let me know your destination to calculate walking distance and walking time.

IUUQT HJUIVCDPNLBLBP'VODUJPO$IBU - #FODI ؀ചݫद૑ࢤࢿ USER USER I walked from
Hangeul Museum today. How long did I walk? USER (Slot Question) AI {"origin": "Hangeul Museum", "destination": "Hannam police precinct", "distance": 3.3, "distance_unit": "km", "time": {"hours": null, "minutes": 52}} TOOL (Answer Completion) The walking distance from Hangeul Museum to Hannam police precinct is approximately 3.3km and takes approximately 52 minutes. AI (Relevance Detection) That sounds like a refreshing walk! It's great to hear that. AI (Relevance Detection) I'm sorry, I can't provide weather forecasts. AI USER I walked to the Hannam police precinct. I walked quite a bit. The weather was nice and I felt really good. What's the weather like tomorrow? I'll try walking tomorrow too. Please tell me the weather forecast. (Tool Call) AI "function": {"name": "getWalkInfo", "arguments": "{\"origin\": \"Hangeul Museum\", \"destination\": \"Hannam police precinct\"}"}} Please let me know your destination to calculate walking distance and walking time.

Hangeul Museum today. How long did I walk? USER (Slot Question) AI {"origin": "Hangeul Museum", "destination": "Hannam police precinct", "distance": 3.3, "distance_unit": "km", "time": {"hours": null, "minutes": 52}} TOOL (Answer Completion) The walking distance from Hangeul Museum to Hannam police precinct is approximately 3.3km and takes approximately 52 minutes. AI (Relevance Detection) That sounds like a refreshing walk! It's great to hear that. AI (Relevance Detection) I'm sorry, I can't provide weather forecasts. AI USER I walked to the Hannam police precinct. I walked quite a bit. The weather was nice and I felt really good. What's the weather like tomorrow? I'll try walking tomorrow too. Please tell me the weather forecast. (Tool Call) AI "function": {"name": "getWalkInfo", "arguments": "{\"origin\": \"Hangeul Museum\", \"destination\": \"Hannam police precinct\"}"}} Please let me know your destination to calculate walking distance and walking time. 4MPU2VFTUJPO ೣࣻഐ୹ਸਤ೧ԙ೙ਃೠ ੿ࠁо־ۅغ঻ਸٸ ࠗ઒ೠ੿ࠁܳ঳ӝਤ೧ ࢎਊ੗ীѱ૕ޙೞחݫद૑

Hangeul Museum today. How long did I walk? USER (Slot Question) AI {"origin": "Hangeul Museum", "destination": "Hannam police precinct", "distance": 3.3, "distance_unit": "km", "time": {"hours": null, "minutes": 52}} TOOL (Answer Completion) The walking distance from Hangeul Museum to Hannam police precinct is approximately 3.3km and takes approximately 52 minutes. AI (Relevance Detection) That sounds like a refreshing walk! It's great to hear that. AI (Relevance Detection) I'm sorry, I can't provide weather forecasts. AI USER I walked to the Hannam police precinct. I walked quite a bit. The weather was nice and I felt really good. What's the weather like tomorrow? I'll try walking tomorrow too. Please tell me the weather forecast. (Tool Call) AI "function": {"name": "getWalkInfo", "arguments": "{\"origin\": \"Hangeul Museum\", \"destination\": \"Hannam police precinct\"}"}} Please let me know your destination to calculate walking distance and walking time. "OTXFS $PNQMFUJPO ౠ੿ೣࣻܳഐ୹ೠѾҗܳ ࢎਊ੗ীѱ੹׳ೞחݫद૑

Hangeul Museum today. How long did I walk? USER (Slot Question) AI {"origin": "Hangeul Museum", "destination": "Hannam police precinct", "distance": 3.3, "distance_unit": "km", "time": {"hours": null, "minutes": 52}} TOOL (Answer Completion) The walking distance from Hangeul Museum to Hannam police precinct is approximately 3.3km and takes approximately 52 minutes. AI (Relevance Detection) That sounds like a refreshing walk! It's great to hear that. AI (Relevance Detection) I'm sorry, I can't provide weather forecasts. AI USER I walked to the Hannam police precinct. I walked quite a bit. The weather was nice and I felt really good. What's the weather like tomorrow? I'll try walking tomorrow too. Please tell me the weather forecast. (Tool Call) AI "function": {"name": "getWalkInfo", "arguments": "{\"origin\": \"Hangeul Museum\", \"destination\": \"Hannam police precinct\"}"}} Please let me know your destination to calculate walking distance and walking time. 3FMFWBODF %FUFDUJPO بҳࢎਊҗҙ۲੉হח ੑ۱ী؀ೠ਽׹ݫद૑ ഐ୹ೡࣻহחೣࣻ৬ҙ۲ ػਃ୒ী؀ೠѢ੺ݫद૑

1$ژחݽ߄ੌজਸా೧ੌ࢚ࢤഝীࢲօܻࢎਊغחبҳٜਸ߈৔ೞب۾ೣࣻಽҳࢿ զॿ ੌ੿ ਤ஖੿ࠁ ജ੹ ࢚ಿѨ࢝ ৔ച߂਺ঈ Ѥъ١ੌ࢚ࢤഝҗ޻੽ೠ׮নೠبݫੋਸನҚ
঱যݽ؛੄'VODUJPODBMMࢿמਸಣоೞח୐ೠҴযؘ੉ఠ IUUQT HJUIVCDPNLBLBP'VODUJPO$IBU - #FODI

'VODUJPO$IBU - #FODI ଻੼ݽ؛ ಣо؀࢚ݽ؛ ಣоޙ೦ ਽׹ ಣоޙ೦ ਽׹ ଻੼ӝળ
ݽߧ׹উ ଻੼ ୭ઙझ௏য Ѿҗ࠙ࢳܻನ౟

'VODUJPO$IBU - #FODIEBUB 4JOHMFDBMM %JBMPH ಣо೦ݾࣻ ੋುਬഋ ױੌఢߊച
׮઺ఢ؀ച ইਓುਬഋ 5PPM$BMMPOMZ 5PPM$BMM "OTXFS$PNQMFUJPO 4MPU2VFTUJPO 3FMFWBODF%FUFDUJPO ೣࣻݾ۾ ࣻ۝җਬࢎب۽Ӓܛച FYBDU SBOEPN DMPTF SBOEPN DMPTF ࣻ۝_ߧਤ ਬࢎبӏ஗হ਺

ಣо৘द 4JOHMFDBMM FYBDU SBOEPN DMPTF SBOEPN DMPTF "7( HQUP
HQUUVSCP HQUUVSCP %JBMPH 5PPM $BMM "OTXFS $PNQMFUJPO 4MPU 2VFTUJPO 3FMFWBODF %FUFDUJPO "7( HQUP HQUUVSCP HQUUVSCP

ಣо৘द 4JOHMFDBMM FYBDU SBOEPN DMPTF SBOEPN DMPTF "7( HQUP
HQUUVSCP HQUUVSCP 4JOHMFDBMM FYBDU SBOEPN DMPTF SBOEPN DMPTF "7( HFNJOJQSP - QSFWJFX HFNJOJ fl BTI - QSFWJFX HFNJOJQSP

ಣо৘द %JBMPH 5PPM $BMM "OTXFS $PNQMFUJPO 4MPU 2VFTUJPO
3FMFWBODF %FUFDUJPO "7( HQUP HQUUVSCP HQUUVSCP %JBMPH 5PPM $BMM "OTXFS $PNQMFUJPO 4MPU 2VFTUJPO 3FMFWBODF %FUFDUJPO "7( HFNJOJQSP - QSFWJFX HFNJOJ fl BTI - QSFWJFX HFNJOJQSP

&SSPSTJO5PPM$BMM *OQVU VTFSߣߑ੄ࢶޛ਷ޖटղਊ੉ঠ (SPVOE 5SVUI \OBNFHFU@NPWJF@EFUBJMT BSHVNFOUT\=UJUMF==ߣߑ੄ࢶޛ=^^ .PEFM 0VUQVU
\OBNFtHJGU@TFBSDI@QSPEVDU BSHVNFOUT\=RVFSZ==ߣߑ੄ࢶޛ=^^ *OQVU VTFS׮ࢽݺ੉ࢶޛч࢖݅சୌਗਸաׇյԄؘੌੋ ׼঴݃ղݶغח૑ঌ۰઻ (SPVOE 5SVUI \OBNFDBMDVMBUF@CJMM@TQMJU BSHVNFOUT\=CJMM@UPUBM= =OVN@QFPQMF=^^ .PEFM 0VUQVU \OBNFDBMDVMBUF@CJMM@TQMJU BSHVNFOUT\=CJMM@UPUBM= =OVN@QFPQMF=^^

*OQVU VTFS੢ࠁӝݾ۾ݫݽ୶о೧઻ BTTJTUBOUঌѷणפ׮ݫݽղਊਸঌ۰઱ࣁਃ VTFS઱ߑࣁઁ ࣻ࠙௼ܿ ਃѢ౟ ୒ನب BTTJTUBOU\OBNFBEE.FNP BSHVNFOUT
\=UJUMF==੢ࠁӝݾ۾= =DPOUFOU==઱ ߑࣁઁ ࣻ࠙௼ܿ ਃѢ౟ ୒ನب=^^ UPPM\OBNFBEE.FNP DPOUFOU \=TUBUVT==TVDDFTT=^^ (SPVOE 5SVUI ݫݽܳ୶о೮णפ׮ .PEFM 0VUQVU ঌѷणפ׮஫ܽ؊ী୶оೡডࣘ੄ઁݾҗղਊਸঌ ۰઱ࣁਃ *OQVU VTFS૑Ә౸Үզॿযٸ BTTJTUBOU\OBNFJOGPSN8FBUIFS BSHVNFOUT \=MPDBUJPO==౸Ү=^^ UPPM\OBNFJOGPSN8FBUIFS DPOUFOU \=UFNQFSBUVSF= XFBUIFS%FTDSJQUJPO=ҳܴ݆਺=^^ (SPVOE 5SVUI അ੤౸Ү੄ӝৡ਷ب੉Ҋ ҳܴ੉݆णפ׮ .PEFM 0VUQVU ૑Ә౸Үחب੉Ҋ݉਷զॿীਃ &SSPSTJO"OTXFS$PNQMFUJPO

*OQVU VTFSӝୡ؀ࢎਯ੉ޤঠ рױ൤ࢸݺ೧઻ BTTJTUBOUӝୡ؀ࢎਯ #.3 ਷ইޖ۠ഝزਸೞ૑ ঋਸٸ ૊৮੹ೠോध࢚కীࢲ਋ܻރ੉ࢤݺਬ૑৬ ӝࠄन୓ӝמਸਤ೧दрزউࣗݽೞח୭ࣗೠ੄ ীց૑۝ਸ݈೤פ׮
VTFSղӝୡ؀ࢎਯ੉ҾӘ೧ (SPVOE 5SVUI ୓઺җః ա੉ ࢿ߹ਸঌ۰઱दݶӝୡ؀ࢎਯਸ҅࢑ ೡࣻ੓णפ׮ .PEFM 0VUQVU \OBNFDBMDVMBUF#.3 BSHVNFOUT \=XFJHIU= =IFJHIU= =BHF= =HFOEFS==NBMF=^^ *OQVU VTFS֙ਘੌ٣ؘ੉ࢸ੿೧઻ BTTJTUBOUঌѷणפ׮٣ؘ੉੄੉ܴҗ٣ؘ੉׼ੌ զ૞ܳੌ۽ನೣೡ૑ৈࠗܳঌ۰઱ࣁਃ VTFSѾഒ (SPVOE 5SVUI ٣ؘ੉զ૞ܳੌ۽ನೣೡөਃ .PEFM 0VUQVU Ѿഒ٣ؘ੉ࢸ੿৮ܐ೮णפ׮֙ਘੌө૑ ੌթওणפ׮ &SSPSTJO4MPU2VFTUJPO

*OQVU VTFSೖ੗ખ઱ޙ೧઴ې (SPVOE 5SVUI ೖ੗ח઱ޙೡࣻহणפ׮ .PEFM 0VUQVU જইਃযڃઙܨ੄ೖ੗ܳ઱ޙೡөਃ *OQVU VTFSೖ੗ખ઱ޙ೧઴ې
(SPVOE 5SVUI ೖ੗ח઱ޙೡࣻহणפ׮ .PEFM 0VUQVU \OBNFPSEFS@QJ[[B BSHVNFOUT \=TJ[F==NFEJVN= =DSVTU==UIJO= =UPQQJOHT=<=QFQQFSPOJ= =NVTISPPNT= =POJPOT=>^^ &SSPSTJO3FMFWBODF%FUFDUJPO

୶оಣоࣇ ѐ߹੗୓ಣоࣇࢎਊ২࣌ ৔যߡ੹ ળ࠺઺ੋӝמٜ

OJLJZ ੉नࠂ EBOJFMW ੉׮פৃ LPOH ੿ࢿ൞ BOOJF ࢲо਷ LPCJHP Ҋ߽ੌ
JOEFYJ नݺ୍ 'VODUJPO$IBU - #FODIѐߊ੗

ݽٚোѾਸ࢜܂ѱ

ݽٚোѾਸ࢜܂ѱ 'VODUJPO$BMM

FunctionChat-Bench: 도구 연결 대화에서의 언어 모델 생성 능력 평가

FunctionChat-Bench: 도구 연결 대화에서의 언어 모델 생성 능력 평가

More Decks by kakao

Other Decks in Programming

Featured

Transcript