Share
Embed
Anomaly Detection for a Water Treatment System Using Unsupervised Machine Learning
Yoriyuki Yamagata
May 28, 2018
Transcript
20 slides
-
Anomaly Detection for a Water Treatment System Using Unsupervised Machine
Learning Nov 18, 2017 DMCIS Jun Inoue¹, Yoriyuki Yamagata¹, Yuqi Chen², Christopher M. Poskitt² and Jun Sun² 1. National Institute of Advanced Industrial Science and Technology (AIST), Japan 2. Singapore University of Technology and Design (SUTD), Singapore -
SWaT test bed in SUTD • Scaled down but fully
operational water treatment system • For security research on social infrastructures and CPSs -
Dataset • 26 actuators, each has 3 positions and 25
sensors • Data for normal behaviors and behaviors under attacks • Subject to 36 network attacks to all subsystems P1-P6 • Each network attack changes transmitted sensor values and actuator commands in different ways -
Application of DNN and SVM • Unsupervised machine learning •
DNN and SVM are applied to normal data to learn normal behavior • Attack data are used for hyper-parameter tuning and evaluation (only) -
LSTM a₁ a₁ a₁ a₁ a₁ a₁ v₁ μ₁ σ₁
a₁ v₁ v₂ v₃ v₄ v₅ a₁ a₁ O DNN • Novel DNN architecture • Feed-forward layers on LSTM • Compute outlier factors -
Training error and F1 score • Training error decreases smoothly
• F1 score does not improve along with epochs • This may suggest a problem in the neural architecture -
One-class SVM (RBF kernel) • Create fixed size vectors by
sliding windows of width 4 • Learn the characteristics of data using normal data • Hyper-parameter tuning using attack data • Logarithmic grid search • Randomized search -
Performance (overall) Method Precision Recall F1 DNN 0.98295 0.67847 0.80281
SVM 0.92500 0.69901 0.79628 .)% 0.84767 0.64473 0.73240 Range 0.12829 0.93803 0.22571 Trivial 0.11980 1.00000 0.21397 -
Attack Classification A. Override control B. Unnatural sensor values C.
Gradual change of sensor values -
Override control ID DNN SVM MHD 1 0.00000 0.00000 0.00000
2 0.00000 0.00000 0.11061 4 0.00000 0.03571 0.00000 13 0.00000 0.00000 0.00000 14 0.00000 0.00000 0.00000 17 0.00000 0.00000 0.01813 21(#) 0.00000 0.01667 0.00000 22(#) 0.99792 1.00000 0.00000 23(#) 0.87639 0.87500 0.97561 24 0.00000 0.00000 0.00000 -
Override control *% DNN SVM MHD 0.00000 0.00909
0.00000 0.00000 0.00000 0.03391 0.00000 0.00000 0.00000 0.87639 0.93570 0.93443 0.00000 0.00000 0.00000 0.00000 0.00333 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.03119 1.00000 1.00000 0.15565 -
Unnatural sensor values *% DNN SVM MHD 0.71667 0.72083
0.00000 0.00000 0.88800 0.00000 0.92708 0.88810 0.49170 1.00000 0.43333 0.06832 0.97833 1.00000 0.99822 0.12333 0.13000 0.18919 0.84524 0.84762 0.94177 0.00000 0.01667 0.00000 0.99792 1.00000 0.00000 0.87500 0.87639 0.97561 0.00000 0.00909 0.00000 -
Unnatural sensor values *% DNN SVM MHD 0.00000
0.00000 0.03391 0.00000 0.00000 0.00000 0.00000 0.00333 0.00000 0.00000 0.00000 0.00000 0.00000 0.90455 0.00000 0.00000 0.00000 0.11036 0.00000 0.11852 0.00000 1.00000 1.00000 0.15565 0.92333 0.92667 0.50178 0.94048 0.00000 0.00000 0.93333 0.92667 0.84512 -
Gradual change of sensor values *% %// 47. .)%
0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.35679 0.00000 -
Observation • Unnatural sensor value(s) is easiest to detect •
Attacks which Override control and changes sensor values gradually are difficult to detect -
Conclusion • DNN and SVM have almost same performance •
SVM is more sensitive, but prone to generate false positives • Recall rate widely varies across types of attacks -
Threats to validity • We use only SWaT dataset •
Attacks are all artificial • Recall and precision are measured by numbers of log entries • Duration of attacks can affect the outcome -
Future works • Improve recall rates • DNN : better
neural architecture • SVM : feature engineering • Comparison to other methods • Experiment using different systems than SWaT