Foundation Model(基盤モデル) “On the Opportunities and Risks of Foundation Models”, 2021 5 • 2021/8/16初出のホワイト ペーパーで登場した⾔葉 • Stanfordの研究機関の名称にも なっている(⻘枠) (Abstractより抜粋) “AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT- 3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character”
⾃⼰教師あり学習 + ⼤規模Transformer (High-Capacity Networks) 11 1) Causal Language Model 2) Masked Language Model ⼤規模Transformer ⼤規模Transformer Input: Language models determine Output: word probability by analyzing text data Input: Language models determine [mask] [mask] by [mask] text data Input: Language models determine word probability by analyzing text data Original: Language models determine word probability by analyzing text data 最近の成功している⾔語モデルはほぼ上記のような構造を利⽤