հऀɿՃ౻ ↅଠʢژେֶʣ ͱ͘ʹऍ͕ͳ͍ݶΓɼਤදࣄྫհจ͔ΒͷҾ༻Ͱ͢ ˞հऀͷίϝϯτͰ͢ Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, Huan Sun ACL 2023 https://aclanthology.org/2023.acl-long.153/ !ୈճ࠷ઌ/-1ษڧձ
A: The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9. Chain-of-Thought Prompting Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: The answer is 11. Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? A: The answer is 27. Standard Prompting Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11. Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? Model Input Model Output Model Output Model Input େنݴޠϞσϧͷϓϩϯϓτ େنݴޠϞσϧʢ--.ʣͷೖྗʢϓϩϯϓτʣΛ͢Δͱ ৽͍͠λεΫͰߴ͍ੑೳΛୡͰ͖Δɽ • ຊจதֶशʢIn-Context Learning; ICLʣ[Brown+,20] • ࢥߟ࿈ܕʢChain-of-Thought; CoTʣϓϩϯϓτ [Wei+,22] [Wei+,22] 2
A: The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9. Chain-of-Thought Prompting Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: The answer is 11. Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? A: The answer is 27. Standard Prompting Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11. Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? Model Input Model Output Model Output Model Input Chain-of-Thought (CoT) ϓϩϯϓτ େنݴޠϞσϧʢ--.ʣͷೖྗʢϓϩϯϓτʣΛ͢Δͱ ৽͍͠λεΫͰߴ͍ੑೳΛୡͰ͖Δɽ • ຊจதֶशʢIn-Context Learning; ICLʣ[Brown+,20] • ࢥߟ࿈ܕʢChain-of-Thought; CoTʣϓϩϯϓτ [Wei+,22] CoT ϓϩϯϓτΛ༻͍Δͱɺ ੑೳ্͕͢Δ ྫɿࢉज़ਪλεΫ [Cobbe+,21] Accuracy: 15.4 → 48.5 (InstructGPT-175B text-davinci-002 [Ouyang+,22;Brown+,20]) CoT ϓϩϯϓτ͕ߴ͍ੑೳΛ ୡͰ͖Δͷͳ͔ͥʁ [Wei+,22] 3
࣮ݧ̎ cख๏ CoT ϓϩϯϓτΛมߋͯ͠ɼ ύλʔϯͷϓϩϯϓτΛ৽ͨʹ࡞͢Δɽ • No coherence for bridging objects • No relevance for bridging objects • No coherence for language templates • No relevance for language templates • No coherence • No relevance 17
ࢀߟจݙ [Brown+,20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [Wei+,22] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc. [Cobbe+,21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. [Ouyang+,22] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 25
ࢀߟจݙ [Chowdhery+,22] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [Chung+,22] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction- finetuned language models. arXiv preprint arXiv:2210.11416. [Press+,22] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. [Golovneva+,23] Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations. 26
stktu
muraoka7
ya9do
inoichan
kosuken
ahspragu
monochromegane
yosukefk
aya_watanabe
kazutoshishinoda
matsui_528
sms_tech
yoshipon
cassininazir
lynnandtonic
rasmusluckow
paigeccino
zakiyama
soudai
holman
bermonpainter
smashingmag
hannesfritz
jcasabona
yeseniaperezcruz