PubMedBERT: 生物医学NLPのための事前学習

Transcript

1. PubMedBERT Domain-Specific Pretraining for Biomedical NLP Naoto Usuyama @naotous Microsoft Research, Health Futures Project Hanover, Biomedical NLP
2. Naoto Usuyama @naotous Microsoft Research, Health Futures Past: 東京⼤学情報科学科 医科研ヒトゲノム解析センター
3. お伝えしたいこと ドメイン特化の事前学習モデル Microsoft x ヘルスケア 医師のキャリア先としてのGAFAM
4. Supervised Learning • 学習データを⽤意するのは⼤変 • タスクごとに必要 ⼊⼒ 正解 ラベル モデル
5. 5 https://imgur.com/rwUE5 Everyday PubMed: 4000 new papers Expert can curate 4-5 Information Overload
6. EMR: 60-80% In Unstructured Text 6 Wolters Kluwer: Health Language Blog Curation: 2-3 hours X 1.8M = 4-5 million expert hours
7. Supervised Learning 7 Domain Knowledge Algorithm Application Model Unlabeled Data Labeled Data Public Abundant Bottleneck
8. Self-Supervised Learning 8 Domain Knowledge 事前学習 Fine-Tuning Unlabeled Data Labeled Data Public Abundant Bottleneck Application Model
9. Neural Language Model Pretraining The 2 mutations that were only found in the neuroblastoma resistance screen (G1123S/D) are located in the glycine-rich loop, which is known to be crucial for ATP and ligand binding and are the first mutations described that induce resistance to TAE684, but not to PF02341066 9 Unlabeled text
10. Masked Language Modeling (MLM) The 2 mutations that were only found in the neuroblastoma resistance screen (G1123S/D) are located in the glycine-rich loop, which is known to be crucial for ATP and ligand binding and are the first mutations described that induce resistance to TAE684, but not to PF02341066 10 [MASK] [MASK] [MASK] [MASK] [MASK]
11. 11 Standard PubMedBERT
12. 12 Standard PubMedBERT
13. Biomed-Specific Pretraining 13 lymphoma ® l, ##ym, ##ph, ##oma acetyltransferase ® ace, ##ty, ##lt, ##ran, ##sf, ##eras, ##e Yu et al. “Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing”, Special Issue on Computational Methods for Biomedical Natural Language Processing, ACM Transactions on Computing for Health, 2021. (arxiv) Domain-Specific Vocabulary
14. 14 PubMedBERT outperforms prior language models
15. PubMedBERT on HuggingFace • PubMed BERT (abst) • PubMed BERT (abst + full-text) • ⽉間25万回以上ダウンロード 15
16. Fine-Tuning の注意点 • Hyper-parameters • Learning Rate • Layer re-init. • Layer-wise Learning Rate Decay • Random Seed, GPU が違うだけで結果が⼤きく変わることも • ちゃんと評価する時は平均を取る Tinn et al. “Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing”, in submission, 2021. (arxiv)
17. Why? • 膨⼤な未整理データから有⽤な情報をタイムリーに • PubMed 論⽂の整理 • 電⼦カルテから Real World Evidence • ⼈の⼿間を減らす、⽣産性向上 • 事前学習: 膨⼤な未整理データの活⽤、アノテーションコストを減らす • 膨⼤な計算資源 • PubMedBERT の学習 DGX-2 で約⼀週間
18. http://aka.ms/biomedsearch Wang, Li and Naumann et al. “Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature”, KDD, 2021 (arxiv)
19. Assisted Curation 19 Wong et al. “Breaching the curation bottleneck with human-machine reading symbiosis”, in submission, 2021.
20. Precision Oncology RWE 20 Preston and Wei et al. “Towards Structuring Real-World Data at Scale: Deep Learning for Extracting Key Oncology Information from Clinical Text with Patient-Level Supervision”, 2022. (arxiv)
21. Nuance • 病院向け⾳声認識、8割以上の病院で利⽤ • $20B (約2兆2500億円) • 医療従事者の⽣産性向上、バーンアウト軽減 • RWE データ活⽤
22. お伝えしたいこと ドメインにあった事前学習モデルを使う Microsoft のヘルスケアAI分野での取り組み 医師のキャリア先としての GAFAM Naoto Usuyama @naotous
23. Microsoft Research, Health Futures • https://aka.ms/health_futures
24. Biomedical NLP Group • Project Hanover • Biomedical NLP Group • BLURB benchmark