Training Data Extraction From Pre-trained Language Models: A Survey

Shotaro Ishihara (2023). Training Data Extraction From Pre-trained Language Models: A Survey. Proceedings of Third Workshop on Trustworthy Natural Language Processing.
https://arxiv.org/abs/2305.16157
https://trustnlpworkshop.github.io/

Transcript

1. [1] Nicholas Carlini et al. Extracting training data from large language models. In USENIX Security 21. [2] Nicholas Carlini et al. Quantifying memorization across neural language models. In ICLR 2023. Training Data Extraction From Pre-trained Language Models: A Survey Overview • This study is the first to provide a comprehensive survey of training data extraction from Pre-trained Language Models (PLMs). • Our review covers more than 100 key papers in fields NLP and security: ◦ Preliminary knowledge ◦ Taxonomy of memorization, attacks, defenses, and empirical findings ◦ Future research directions Shotaro Ishihara (Nikkei Inc. shotaro.ishihara@nex.nikkei.com ) arXiv preprint: https://arxiv.org/abs/2305.16157 Attacks, defenses, and findings • The attack is consist of candidate generation and membership inference. • The pioneering work has identified that personal information can be extracted from pre-trained GPT-2 models [1]. • Experiments show that memorization is related to model size, prompt length and duplications in the training data [2]. • Defenses include pre-processing, training and post-processing. Definition of memorization With the advent of approximate memorization, the concern became similar to a famous issue called model inversion attack. Future research directions • Is memorization always evil? ◦ memorization vs association ◦ memorization vs performance • Toward broader research fields ◦ model inversion attacks ◦ plagiarism detection ◦ image similarity • Evaluation schema ◦ benchmark dataset ◦ evaluation metrics • Model variation ◦ masked language models