Messages Exploratory Attack: Spam Disguise in Practice Question Given a spam, how do you disguise it to evade from being detected? How? • Create dummy@gmail.com • Generate disguised spams and send to dummy@gmail.com • Select the most desired modiﬁcation from the inbox.
Messages Exploratory Attack: Spam Disguise in Practice Question Given a spam, how do you disguise it to evade from being detected? How? • Create dummy@gmail.com • Generate disguised spams and send to dummy@gmail.com • Select the most desired modiﬁcation from the inbox. Questions: • What is the “most desired” mail? • How to generate eﬃciently?
Messages Training Data from Online Services Users may vary in expertise, dedication and motivation What if haters dominate? Are they going to subvert the learning algorithm? How to recover the unbiased labels/ratings?
Messages Modeling the Adversary Adversarial settings The adversary manipulates instances to mislead the decision of the classiﬁer in their favor. Exploratory attack • in the test phrase; • disguise a malicious instance to evade from being detected; • e.g. disguise a spam, mutate a virus.
Messages Modeling the Adversary Adversarial settings The adversary manipulates instances to mislead the decision of the classiﬁer in their favor. Exploratory attack • in the test phrase; • disguise a malicious instance to evade from being detected; • e.g. disguise a spam, mutate a virus. Causative attack • in the training phrase; • manipulate the training set to subvert the learning process; • e.g. poisoning the spam ﬁlter, unfair rating on SNS.
Messages Why Adversarial Learning is Interesting? (1) Because social network and crowdsourcing platform (e.g. Amazon mechanical turk) are popular. (2) Know your enemies and yourself, you will not be imperiled in a hundred battles. –Sun Tzu, The Art of War, 544 BC
Messages Why Adversarial Learning is Interesting? (1) Because social network and crowdsourcing platform (e.g. Amazon mechanical turk) are popular. (2) Know your enemies and yourself, you will not be imperiled in a hundred battles. –Sun Tzu, The Art of War, 544 BC Secure learning The ultimate goal is to develop robust learning algorithms, which are resilient to the adversarial noise.
Messages Binary Classiﬁcation Formalize the problem in math Term Notation Real world Input space X ⊆ RD All possible mails Response space Y := {−1, 1} All possible labels Instance x ∈ X, i.e. a D-dimensional vector A mail
Messages Binary Classiﬁcation Formalize the problem in math Term Notation Real world Input space X ⊆ RD All possible mails Response space Y := {−1, 1} All possible labels Instance x ∈ X, i.e. a D-dimensional vector A mail Hypothesis space H All possible ﬁlters Classiﬁer f : X → Y, f ∈ H A ﬁlter
Messages Binary Classiﬁcation Formalize the problem in math Term Notation Real world Input space X ⊆ RD All possible mails Response space Y := {−1, 1} All possible labels Instance x ∈ X, i.e. a D-dimensional vector A mail Hypothesis space H All possible ﬁlters Classiﬁer f : X → Y, f ∈ H A ﬁlter Positive set X+ := {x ∈ X | f(x) = +1} All possible spams Negative set X− := {x ∈ X | f(x) = −1} All possible legit mails
Messages Binary Classiﬁcation Formalize the problem in math Term Notation Real world Input space X ⊆ RD All possible mails Response space Y := {−1, 1} All possible labels Instance x ∈ X, i.e. a D-dimensional vector A mail Hypothesis space H All possible ﬁlters Classiﬁer f : X → Y, f ∈ H A ﬁlter Positive set X+ := {x ∈ X | f(x) = +1} All possible spams Negative set X− := {x ∈ X | f(x) = −1} All possible legit mails Loss function V : Y × Y → R0+ Cost of misclassiﬁcation
Messages Training the Classiﬁer Solving an optimization problem Classiﬁcation Given a training set S := {(xi, yi ) | xi ∈ X, yi ∈ Y}n i=1 . Find the classiﬁer fS ∈ H that performs best on some test set T. Solving an optimization problem: fS := arg min f γ n i=1 V (yi, f(xi )) + f 2 H , where γ ∈ R0+ is a ﬁxed parameter for quantifying the trade oﬀ.
Messages Spam Disguise Problem formulation Disguise a spam from being detected by a ﬁlter. Be eﬃcient. Problem Formulation Given • a trained classiﬁer f; • a positive (malicious) instance xA ∈ X+; • a random negative (benign) instance x− ∈ X−. Find an instance x∗ ∈ X− f such that • x∗ should be similar to xA; • issuing as few queries to f as possible. Han, Thomas, Claudia. Evasion Attack of Multi-Class Linear Classiﬁers PAKDD 2012 16 of 46
Messages Assumptions Assumption Real world Know the dimension of X Know how many features Attack a ﬁxed f Spam ﬁlter is not updated Observe f(x) by a membership query Observe the label of a sent mail Design a cost function Know the cost of misclassiﬁcation Han, Thomas, Claudia. Evasion Attack of Multi-Class Linear Classiﬁers PAKDD 2012 17 of 46
Messages Exploratory Attack as ℓp -norm Minimization Exploratory Attack Given xA, f, and a cost function g : X × X → R0+ , solve min x g(x, xA) subject to x ∈ X−, where X− is speciﬁed by the membership oracle f. For example, g(x, xA) := x − xA ℓ1 Han, Thomas, Claudia. Evasion Attack of Multi-Class Linear Classiﬁers PAKDD 2012 18 of 46
Messages Illustration of the Problem g(x) := x − xA ℓ1 X+ f X− f xA x− Han, Thomas, Claudia. Evasion Attack of Multi-Class Linear Classiﬁers PAKDD 2012 19 of 46
Messages Illustration of the Problem g(x) := x − xA ℓ1 X+ f X− f xA x− ⊗ x∗ Han, Thomas, Claudia. Evasion Attack of Multi-Class Linear Classiﬁers PAKDD 2012 19 of 46
Messages Face Camouﬂage Considering a suspect tries to disguise herself as innocent. Han, Thomas, Claudia. Evasion Attack of Multi-Class Linear Classiﬁers PAKDD 2012 20 of 46
Messages Label Flips Attack Given a training set, the adversary contaminates the training data through ﬂipping labels. Han, Huang, Claudia. Adversarial Label Flips Attack on Support Vector Machines ECAI 2012 21 of 46
Messages Adversarial Label Flips Attack Adversarial Label Flip Attack Find a combination of label ﬂips under a given budget so that a classiﬁer trained on such data will have maximal classiﬁcation error on some test data. Han, Huang, Claudia. Adversarial Label Flips Attack on Support Vector Machines ECAI 2012 22 of 46
Messages Adversarial Label Flips Attack Adversarial Label Flip Attack Find a combination of label ﬂips under a given budget so that a classiﬁer trained on such data will have maximal classiﬁcation error on some test data. Training set: S := {(xi, yi ) | xi ∈ X, yi ∈ Y}n i=1 ; Indicator: zi ∈ {0: normal, 1: ﬂipped}, i = 1, . . . , n; Flipping cost: ci ∈ R0+, i = 1, . . . , n; Han, Huang, Claudia. Adversarial Label Flips Attack on Support Vector Machines ECAI 2012 22 of 46
Messages Adversarial Label Flips Attack Adversarial Label Flip Attack Find a combination of label ﬂips under a given budget so that a classiﬁer trained on such data will have maximal classiﬁcation error on some test data. Training set: S := {(xi, yi ) | xi ∈ X, yi ∈ Y}n i=1 ; Indicator: zi ∈ {0: normal, 1: ﬂipped}, i = 1, . . . , n; Flipping cost: ci ∈ R0+, i = 1, . . . , n; Tainted label: y′ i := yi (1 − 2zi ); Tainted training set: S′ := {(xi, y′ i )}. Han, Huang, Claudia. Adversarial Label Flips Attack on Support Vector Machines ECAI 2012 22 of 46
Messages A Bilevel Formulation Finding the optimal label ﬂips Given S, a test set T and a budget C, solve max z (x,y)∈T V (y, fS′ (x)) , s.t. fS′ ∈ arg min f γ n i=1 V y′ i , f(xi ) + f 2 H , n i=1 cizi ≤ C, zi ∈ {0, 1}. Han, Huang, Claudia. Adversarial Label Flips Attack on Support Vector Machines ECAI 2012 23 of 46
Messages A Bilevel Formulation Finding the optimal label ﬂips Given S, a test set T and a budget C, solve max z (x,y)∈T V (y, fS′ (x)) , s.t. fS′ ∈ arg min f γ n i=1 V y′ i , f(xi ) + f 2 H , n i=1 cizi ≤ C, zi ∈ {0, 1}. Defender Han, Huang, Claudia. Adversarial Label Flips Attack on Support Vector Machines ECAI 2012
Messages A Bilevel Formulation Finding the optimal label ﬂips Given S, a test set T and a budget C, solve max z (x,y)∈T V (y, fS′ (x)) , s.t. fS′ ∈ arg min f γ n i=1 V y′ i , f(xi ) + f 2 H , n i=1 cizi ≤ C, zi ∈ {0, 1}. Defender Attacker Han, Huang, Claudia. Adversarial Label Flips Attack on Support Vector Machines ECAI 2012 23 of 46
Messages Subjective opinions from crowds Learning objective assessment from subjective opinions Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 27 of 46
Messages Subjective opinions from crowds Learning objective assessment from subjective opinions Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 27 of 46
Messages Subjective opinions from crowds Learning objective assessment from subjective opinions Fair rating (Groundtruth)? Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 27 of 46
Messages Subjective opinions from crowds Learning objective assessment from subjective opinions Fair rating (Groundtruth)? What’s wrong with “majority vote” and “take average”? They completely ignore the individual expertise and may fail in the settings with non-Gaussian or adversarial noise! Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 27 of 46
Messages Unreliable readings from sensors 237◦ 229◦ 240◦ 236◦ −13◦ Groundtruth? Questions 1. How to integrate readings from multiple sensors? 2. How accurate is each sensor? Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 28 of 46
Messages Learning from Multiple Observers Problems • How to learn a regression function to predict the ground truth precluding the prior knowledge of observers? • How to estimate the expertise of each observer without knowing the ground truth? Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 29 of 46
Messages Intuition behind Leveraging the neighborhood information Instance space X x2 x1 x3 Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 30 of 46
Messages Intuition behind Leveraging the neighborhood information Instance space X x2 x1 x3 f(x1 ) f(x2 ) f(x3 ) Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 30 of 46
Messages Intuition behind Leveraging the neighborhood information Instance space X x2 x1 x3 f(x1 ) f(x2 ) f(x3 ) Groundtruth space Z z1 z2 z3 (Latent) Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 30 of 46
Messages Nonparametric probabilistic model xn yn,m zn M N p(Y, Z, X) = p(Z | X)p(Y | Z, X)p(X). Gaussian process: a less-parametric approach for modeling a function. Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 31 of 46
Messages Nonparametric probabilistic model xn yn,m zn M N p(Y, Z, X) = p(Z | X)p(Y | Z, X)p(X). Gaussian process: a less-parametric approach for modeling a function. Maximizing the posterior, which gives log p(Z, Θ | Y, X) = log p(Y | Z, X, Θ)+log p(Z | X, Θ)+constant. Deriving the gradient w.r.t. z, κ, φ, η, respectively. Feed the gradients to L-BFGS method for ﬁnding the stationary point. Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 31 of 46
Messages 1-D example Groundtruth function: f(t) = 10 sin(6t) sin( t 2 ), Randomly sample responses at t ∈ [0, 6] from four sensors. 0 1 2 3 4 5 6 0 0.5 1 0 0.5 1 0 0.5 1 Ground truth Ob.1 resp. Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 32 of 46
Messages 1-D example Groundtruth function: f(t) = 10 sin(6t) sin( t 2 ), Randomly sample responses at t ∈ [0, 6] from four sensors. 0 1 2 3 4 5 6 0 0.5 1 0 0.5 1 0 0.5 1 Ground truth Ob.1 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.2 resp. Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 32 of 46
Messages 1-D example Groundtruth function: f(t) = 10 sin(6t) sin( t 2 ), Randomly sample responses at t ∈ [0, 6] from four sensors. 0 1 2 3 4 5 6 0 0.5 1 0 0.5 1 0 0.5 1 Ground truth Ob.1 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.2 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.3 resp. Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 32 of 46
Messages 1-D example Groundtruth function: f(t) = 10 sin(6t) sin( t 2 ), Randomly sample responses at t ∈ [0, 6] from four sensors. 0 1 2 3 4 5 6 0 0.5 1 0 0.5 1 0 0.5 1 Ground truth Ob.1 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.2 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.3 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.4 resp. Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 32 of 46
Messages 1-D example Groundtruth function: f(t) = 10 sin(6t) sin( t 2 ), Randomly sample responses at t ∈ [0, 6] from four sensors. 0 1 2 3 4 5 6 0 0.5 1 0 0.5 1 0 0.5 1 Ground truth Ob.1 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.2 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.3 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.4 resp. Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 32 of 46
Messages 1-D example What do we know? Only the readings from each sensor 0 1 2 3 4 5 6 0 0.5 1 Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 33 of 46
Messages 1-D example What do we want to know? 1. Groundtruth function, i.e. f(t). 2. Response function of each sensor. 0 1 2 3 4 5 6 0 0.5 1 0 0.5 1 0 0.5 1 Ground truth Ob.1 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.2 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.4 resp. 0 0.5 1 0 0.5 1 Ground truth Ob.3 resp. Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 33 of 46
Messages TUM1002 photo rating data set Contributed by Huang http://ml.sec.in.tum.de/opars Huang, Han, Claudia. OPARS: Objective Photo Aesthetics Ranking System (demo paper). ECIR 2013 Han, Huang, Claudia. Learning from Multiple Observers with Unknown Expertise PAKDD 2013 Meyyar. Leveraging the Wisdom of Crowds for Reputation Management Master’s thesis 35 of 46