安全なAI利用のためのLLM（大規模言語モデル）の利用と評価 / japanr2025

Transcript

1. ӝੜਅ໵ʢಙౡେֶσβΠϯܕ"*ڭҭݚڀηϯλʔʣ ҆શͳ"*ར༻ͷͨΊͷ
   --.ʢେن໛ݴޠϞσϧʣͷ
   ར༻ͱධՁ 
   +BQBO3
    !V@SJCP 
2. എܠ
   --.ධՁͷඞཁੑ σʔλ४උ Ϟσϧ܇࿅ʢֶशʣ ςετσʔλͰධՁ ਫ਼౓ɾ࠶ݱ཰ͳͲࢉग़ ίʔυͰ࠶ݱՄೳ ϓϩϯϓτઃܭ --.Ͱਪ࿦ ࠾఺ ίʔυͰ࠶ݱՄೳʁ ػցֶशϞσϧͷධՁ --.ͷධՁ ධՁ͢΂͖ͭͷϙΠϯτ ✓
   ͲͷϞσϧ͕ߴੑೳ͔ͩͬͨ
   ✓
   ͳͥͦͷ݁࿦ʹࢸͬͨͷ͔
   ✓
   खॱͱաఔ͕࢒ͤΔ͔ w $IBU(15Ͱ਺ճࢼͯ͠ʮ͍͍ײͩ͡ͳʯ
   w ʮ(15͕ݡ͍ʯͱ͍͏ӟ͚ͩͰϞσϧબ୒
   w ͨ·ͨ·੒ޭͨ͠ϓϩϯϓτͰʮ༏लʯͱ൑அ
3. 6SZV
   4
    
   &WBMVBUJOH
   -BSHF
   -BOHVBHF
   .PEFMT
   GPS
   *6$/
   3FE
   -JTU
   4QFDJFT
   *OGPSNBUJPO
   BS9JW w *6$/ઈ໓ةዧछධՁͷࣄྫ
   w ੜ෺ଟ༷ੑอશͷ෼໺Ͱ--.ͷ׆༻͕ظ଴͞Ε͍ͯΔ͕ɺ ઐ໳త൑அʹ͓͚Δ৴པੑʹ͸ٙ໰͕࢒Δɻ
   w ʢݱߦͷ--.ʹڞ௨ͨ͠ʣͭͷॏେͳ՝୊
   w ஌ࣝͱਪ࿦ͷΪϟοϓ
   ˠࣄ࣮͸஌͍ͬͯΔ͕ɺͦΕΛԠ༻ͨ͠൑அ͸ࠔ೉
   w ಺ࡏ͢ΔόΠΞε
   ˠ੸௣ಈ෺ʢਓؾछʣʹ͸ڧ͘ɺແ੸௣ಈ෺ʹ͸ऑ͍ എܠ
   --.ධՁͷඞཁੑ https://arxiv.org/abs/2510.02830 ٬؍త͔ͭݫີͳධՁϑϨʔϜϫʔΫ͕ෆՄܽ ਖ਼ղ཰ͷဃ཭ ෼ྨֶత஌ࣝ อશঢ়گͷਪ࿦ 94.9% 27.2%
4. w Φʔϓϯιʔε
   ಁ໌ੑͷߴ͍࣮૷
   w ҆શੑࢤ޲
   ҆શੑͱ৴པੑΛ࠷ॏཁࢹ
   w ࠶ݱੑ
   ࠶ݱՄೳͳՊֶతݕূ
   w ॊೈੑͱ֦ுੑ
   0QFO"*
   (PPHMF
   "OUISPQJD
   Y"*
   
   ϩʔΧϧ؀ڥʢ0MMBNBʣ౳ɺଟ༷ͳϞσϧΛ
   ϕϯμʔϩοΫΠϯͳ͠ͰධՁɻ ӳࠃ"*҆શݚڀॴ͕ओಋ ධՁϑϨʔϜϫʔΫʮ*OTQFDU
   "*ʯ https://inspect.aisi.org.uk/ ++
   "MMBJSF 34UVEJP૑ઃऀ ͕
   
   
   
   
   
   ϓϩδΣΫτΛϦʔυ
5. ධՁͷϞδϡʔϧԽ
   5BTL
   %BUBTFU
   4PMWFS
   4DPSFS ධՁϩδοΫΛίʔυͱͯ͠ମܥతʹ؅ཧɺ࠶ར༻͕ՄೳͱͳΔ 5BTL
   ࣮ݧܭը %BUBTFU
   ೖྗσʔλ 4PMWFS
   ճ౴ઓུ 4DPSFS
   ධՁج४ ධՁʹ࢖༻͢Δೖྗσʔλͱ
   ਖ਼ղϥϕϧͷηοτ ϓϩϯϓτΤϯδχΞϦϯάͳͲɺ
   Ϟσϧ͔Βճ౴ΛҾ͖ग़ͨ͢Ίͷઓུ ධՁશମͷϫʔΫϑϩʔΛఆٛ Ϟσϧͷग़ྗΛਖ਼ղͱൺֱ͠ɺ
   είΞΛࢉग़͢ΔͨΊͷධՁج४ Task( dataset=..., solver=chain(...), scorer=..., ) 
6. *OTQFDU
   "*ʹΑΔ*6$/ධՁλεΫͷ࣮૷ 6SZV
   
   ͷͭͷλεΫ΁ͷద༻ྫ λεΫ ໨త ࢖༻ͨ͠
   4PMWFS
   
   4DPSFS
   ͷྫ 
   ෼ྨֶత෼ྨ 
   ϨουϦετΧςΰϦධՁ 
   ஍ཧత෼෍ 
   ڴҖͷಛఆ ਖ਼͍͠෼ྨ܈Λબ୒ͤ͞Δ ͭͷΧςΰϦ͔ΒͭΛಛఆ ࠃ໊ͷϦετΛੜ੒ ͷڴҖΧςΰϦ͔Βෳ਺Λબ୒ https://github.com/uribo/iucn-redlist-evals chain(), optimize_choices()*, system_message(), multiple_choice_with_cache()*, taxon_partial_scorer()* system_message(), generate(), match() system_message(), generate(), geo_distribution_scorer()* system_message(), generate(), threat_assessment_scorer()*
7. *OTQFDU
   "*ʹΑΔ*6$/ධՁλεΫͷ࣮૷ 6SZV
   
   ͷͭͷλεΫ΁ͷద༻ྫ *OQVU 5BSHFU  Aquila chrysaetos https://github.com/uribo/iucn-redlist-evals    1IPUP
   3PDLZ
   $$
   #:
   
   IUUQTDSFBUJWFDPNNPOTPSHMJDFOTFTCZ
   WJB
   8JLJNFEJB
   $PNNPOT B LC $IPJDFT A. Animalia (Kingdom) > Chordata (Phylum) > Aves (Class) > Accipitriformes (Order) > Pandionidae (Family), B. … (Kingdom) > … (Phylum) > … (Class) > … (Order) > Accipitridae (Family), C. … (Kingdom) > … (Phylum) > … (Class) > … (Order) > Cathartidae (Family), D. … (Kingdom) > … (Phylum) > … (Class) > … (Order) > Sagittariidae (Family)”, E. … (Kingdom) > … (Phylum) > … (Class) > … (Order) > Elanidae (Family)" "OTXFS B &WBMVBUF Correct EX, EW, CR, EN, VU, NT, LC, DD NT Incorrect Montenegro; Italy; France; Albania etc., Country list Montenegro; France; Iraq etc., Partial Agriculture & aquaculture; Pollution; Energy production & mining; Transportation & service corridors etc. Threats list None Incorrect 5BTL ʢΠψϫγʣ ܽམɺ৑௕
8. 3൛΋͋ΔϤʂ
   WJUBMTύοέʔδ --.ͱͷର࿩͸FMMNFSύοέʔδΛհͯ͠ߦ͏ https://vitals.tidyverse.org/ library(vitals) library(ellmer) simple_qa <- tibble::tibble( input = c("日本の初代総理大臣は誰か", "Posit（旧RStudio）のチーフサイエンティストは誰か"), target = c("伊藤博文", "Hadley Wickham") ) tsk <- Task$new( dataset = simple_qa, solver = generate(chat_ollama(model = "gpt-oss:20b")), scorer = model_graded_fact() ) tsk$eval() tsk$score()  5BTL
   ࣮ݧܭը %BUBTFU
   ೖྗσʔλ 4PMWFS
   ճ౴ઓུ 4DPSFS
   ධՁج४ ਪ࿦Ϟσϧͷࢦఆ
9. %&.0
10. ධՁͷίʔυԽ ධՁϓϩηεΛίʔυͱͯ͠هड़ɾ؅ཧ͢Δ ͭͷϝϦοτ ✓
    ৽ϞσϧͰͷଈ࠲ͳ࠶ݕূ
    ✓
    ධՁͷಁ໌ੑͱ௥੻Մೳੑ
    ✓
    ίϛϡχςΟͰͷڞ༗ɾվળ w Ͳͷج४Ͱ൑ఆ͔ͨ͠໌֬
    w ݁Ռͷࠜڌ͕௥੻Մೳ
    w ϩάͱͯ͠ه࿥͞ΕΔ ධՁͷಁ໌ੑ w ධՁίʔυΛ(JU)VCͰެ։
    w ϕϯνϚʔΫͱͯ͠ػೳ ίϛϡχςΟͰͷར༻
11. "*ͷԸܙΛ࠷େԽ͠ɺϦεΫΛ࠷খԽ͢Δ ͨΊʹɻ
    ʮ࢖͏ʯ͚ͩͰͳ͘ɺਖ਼͘͠ʮධՁ͢Δʯ ϓϩηε͕ඞਢɻ ͓ΘΓ