[ACL 2026 Demo] Fast-MIA: Efficient and Scalable Membership Inference for LLMs

https://aclanthology.org/2026.acl-demo.9/
https://github.com/Nikkei/fast-mia

Transcript

1. Hiromu Takahashi and Shotaro Ishihara ACL 2026 System Demonstrations Fast-MIA: Efficient and Scalable Membership Inference for LLMs
2. uv run --with vllm python main.py \ --config config/sample.yaml 1. High-throughput batch inference using vLLM (about 5 times faster individually) 2. Cross-method caching architecture (Reduce the total processing time for benchmarking multiple methods) https://github.com/Nikkei/fast-mia Fast-MIA: Efficient and Scalable 2 LLM LOSS vLLM backend batch inference Shared Cache Reuse across methods PPL/zlib Min-K% Prob DC-PDD Lowercase PAC ReCaLL Con-ReCall SaMIA ……
3. Membership Inference Attack (MIA) on LLMs 3 LLM Is this text included? Text Pre-training Data • Calculate the log-likelihood, etc. • Various methods have been proposed.
4. Challenges in MIA on LLMs 4 LLM Is this text included? Text Pre-training Data • Calculate the log-likelihood, etc. • Various methods have been proposed. 1. Growing computational demands for individual MIA methods. 2. Redundant computation across methods for benchmarking.
5. We introduce Fast-MIA 5 1. Growing computational demands for individual MIA methods. 2. Redundant computation across methods for benchmarking. LLM LOSS vLLM backend batch inference Shared Cache Reuse across methods PPL/zlib Min-K% Prob DC-PDD Lowercase PAC ReCaLL Con-ReCall SaMIA …… 1. High-throughput batch inference using vLLM. 2. Cross-method caching architecture.
6. uv run --with vllm python main.py \ --config config/sample.yaml How to Use: https://github.com/Nikkei/fast-mia 6 model: model_id: "huggyllama/llama-30b" data: data_path: "swj0419/WikiMIA" format: "huggingface" text_length: 32 methods: - type: "loss"
7. AUC Reproducibility and Speed 7 Left: Fast-MIA Right: Transformers-based implementations
8. Inference time (the number of inferences) The cache is working 8
9. uv run --with vllm python main.py \ --config config/sample.yaml 1. High-throughput batch inference using vLLM (about 5 times faster individually) 2. Cross-method caching architecture (Reduce the total processing time for benchmarking multiple methods) https://github.com/Nikkei/fast-mia Contributions Welcome 9 LLM LOSS vLLM backend batch inference Shared Cache Reuse across methods PPL/zlib Min-K% Prob DC-PDD Lowercase PAC ReCaLL Con-ReCall SaMIA ……