CONTROLLING THE FALSE DISCOVERY RATE Under the direction of Christian P. Robert Speaker: WANG Bing 25/11/2013

CONTROLLING THE FALSE DISCOVERY RATE  The common approach to the multiplicity problem calls for controlling the family wise error rate (FWER). This approach, though, has faults, and we point out a few. A different approach to problems of multiple significance testing is presented. It calls for controlling the expected proportion of falsely rejected hypotheses-the false discovery rate. This error rate is equivalent to the FWER when all hypotheses are true but is smaller otherwise. Therefore, in problems where the control of the false discovery rate rather than that of the FWER is desired, there is potential for a gain in power. A simple sequential Bonferroni-type procedure is proved to control the false discovery rate for independent test statistics, and a simulation study shows that the gain in power is substantial. The use of the new procedure and the appropriateness of the criterion are illustrated with examples.

CONTENTS  Introduction  Authors  Abbreviations  Development of the relevant domain  Development  Definition of False Discovery Rate  Two properties of false discovery rate  False Discovery Rate Controlling Procedure  Example of False Discovery Rate Controlling Procedure  Conclusion

AUTHORS  Yoav Benjamini:  His work combines theoretical research in statistical methodology with applied research that involves complex problems with massive data. The methodological work is on selective and simultaneous inference (multiple-comparisons), and centers on the “False Discovery Rate” (FDR) criterion, as well as on general methods for data analysis, data mining and data visualization.  Yosef Hochberg  Ph.D., professor in the School of Mathematical Sciences at Tel Aviv University, has taught several courses in multiple comparisons. He has published an impressive number of articles and technical reports on various statistical methods.

ABBREVIATIONS  FWER: familywise error rate  MCPs: multiple-comparison procedures  PCER: per comparison error rate  FDR: false discovery rate---controlling the expected proportion of falsely rejected hypotheses

DEVELOPMENT OF THE RELEVANT DOMAIN (RELEVANT SCIENTIST AND THEIR CONTRIBUTIONS )  Connections have been made between the FDR and Bayesian approaches (including empirical Bayes methods),  Storey, John D. (2003). "The positive false discovery rate: A Bayesian interpretation and the q-value"  Generalizing the confidence interval into the False coverage statement rate (FCR)  Benjamini Y, Yekutieli Y (2005). "False discovery rate controlling confidence intervals for selected parameters".  Thresholding wavelets coefficients and model selection  Donoho D, Jin J; Jin (2006). "Asymptotic minimaxity of false discovery rate thresholding for sparse exponential data“

STRUCTURE OF THIS STUDY PAPER  A formal definition of the FDR.  Some examples where the control of the FDR is desirable  A simple Bonferroni-type FDR controlling procedure.  A simulation study of the power of the procedure.

FALSE DISCOVERY RATE  m: the problem of testing null hypotheses  m0 : true null hypotheses  m-m0 : not true null hypotheses  R: number of hypotheses rejected, an observable random variable  U,V,S and T: unobservable random variables  PCER=E(V/m)  FWER=P(V≥1)

DEFINITION OF FALSE DISCOVERY RATE  Proportion of the rejected null hypotheses which are erroneously rejected: Q=V/(V+S)  We define the FDR to be the expectation of Q : e Q     ( ) / ( ) / e Q E Q E V V S E V R    

TWO PROPERTIES OF FALSE DISCOVERY RATE:  If all null hypotheses are true, the FDR is equivalent to the FWER:  In this case s=0 and v=r , so if v=0 then Q=0,and if v>0 then Q=1,leading to .  When , the FDR is smaller than or equal to the FWER:  In this case, if v>0 then , leading to Taking expectations on both sides we obtain and the two can be quite different. p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed. It is corresponding to the true null hypotheses and is U(0,1) independent random variables.     1 e P V E Q Q    0 m m  / 1 v r    1 V X Q     1 e P V Q       ( ) / ( ) / e Q E Q E V V S E V R    

EXAMPLES 3 examples show the relevance of FDR control in some typical situations:  multiple-comparison problem involves an overall decision  Control of the probability of any error is unnecessarily stringent, as a small proportion of errors will not change the overall validity of the conclusion.  multiple separate decisions without an overall decision being required  Two treatments are compared in multiple subgroups, and separate recommendations on the preferred treatments must be made for all subgroups  multiple potential effects are screened to weed out the null effects  one example is screening of various chemicals for potential drug development.

FALSE DISCOVERY RATE CONTROLLING PROCEDURE  The procedure :  Consider testing based on the corresponding p-values, is in ordered.  Define the Bonferroni-type multiple-testing procedure:  Let k be the largest i for that then reject all , i=1,2,…,k q*: maximizes the number of rejections 1 2 , ,..., m H H H   i P   * i i P q m  ( ) i H

FALSE DISCOVERY RATE CONTROLLING PROCEDURE  Theorem 1 For independent test statistics and for any configuration of false null hypotheses, the above procedure controls the FDR at .  Remark. The independence of the test statistics corresponding to the false null hypotheses is not needed for the proof of the theorem * q

FALSE DISCOVERY RATE CONTROLLING PROCEDURE  For any independent p-values corresponding to true null hypotheses, and for any values that the p-values corresponding to false null hypotheses can take,  FDR controlling procedure: 0 0 m m   1 0 m m m   0 1 * 0 1 1 ( ) ( | ,..., ) m m m m E Q E Q P p P p q m        * * 0 m E Q q q m   Lemma.

FALSE DISCOVERY RATE CONTROLLING PROCEDURE  Define the Hochberg’s procedure: Let k be the largest i for that then reject all , i=1,2,…,k  Remark : note the relationship between Hochberg’s procedure and the FDR controlling procedure when q* is chosen to equal α.   * 1 i i P q m i      i H

EXAMPLE OF FALSE DISCOVERY RATE CONTROLLING PROCEDURE  Neuhaus et al.(1992) investigated the effects of a new front- loaded administration of rt-PA versus those obtained with a standard regimen of APSAC, in a randomized multicentre trial in 421 patients with acute myocardial infarction.  rt-PA: Thrombolysis with recombinant tissue-type plasminogen activator  APSAC: anisoylated plasminogen streptokinase activator

EXAMPLE OF FALSE DISCOVERY RATE CONTROLLING PROCEDURE Four families of hypotheses can be identified in the study: 1. Base-line comparisons(11 hypotheses), where the problem is of showing equivalence 2. Patency of infarct-related artery (8 hypotheses) 3. Reocclusion rates of patent infarct-related artery (6 hypotheses) 4. Cardiac and other events after the start of thrombolytic treatment (15 hypotheses)

EXAMPLE OF FALSE DISCOVERY RATE CONTROLLING PROCEDURE  The statement about the mortality is based on a p- value of 0.0095.  The ordered s for the 15 comparisons made are: 0.0001, 0.0004, 0.0019, 0.0095, 0.0201, 0.0278, 0.0298, 0.0344, 0.0459, 0.3240, 0.4262, 0.5719, 0.6528, 0.7590, 1.000 .   i p

EXAMPLE OF FALSE DISCOVERY RATE CONTROLLING PROCEDURE  Controlling the FWER at 0.05, the Bonferroni approach, using 0.05/15=0.0033, rejects the 3 hypotheses corresponding to the smallest p-value. (0.0001, 0.0004, 0.0019, correspond to reduced allergic reaction, and to two different aspects of bleeding).  Using Hochberg’s procedure leaves us with the same 3 hypotheses rejected.

EXAMPLE OF FALSE DISCOVERY RATE CONTROLLING PROCEDURE  Using the FDR controlling procedure with comparing sequentially each with 0.05i/15, starting with . The first p-value to satisfy the constraint is as thus we reject the 4 hypotheses having p-value which are less than or equal to 0.013.  we may support now with appropriate confidence the statements about mortality decrease, of which we did not have sufficiently strong evidence before. * 0.05 q    i p   15 p   4 p   4 4 0.0095 0.05 0.013 15 p    

ANOTHER LOOK AT FDR CONTROLLING PROCEFURE Theorem 2 Choose α that maximizes the number of rejections at this level, r(α) Subject to the constraint αm/ r(α)≤q* (1) Proof: for each α, if P(i) ≤ α ≤ P(i+1) ,then r(α)=i. Furthermore, as the ratio on the left-hand side of constraint (1) increases in α over the range on which r(α) is constant, it is enough to investigate αs which are equal to one of the P(i) s. this α= P(k) satisfies the constraint because α/ r(α)= P(k) /k ≤ q*/m. By considering the largest potential αs first, the procedure yields the α with the largest r(α) satisfying the constraint.

POWER COMPARISONS The setting  Using a large simulation study, the family of hypotheses is the expectations of m independent normally distribution random variables being equal to 0.  Each individual hypotheses is tested by z-test, and the test statistics are independent. We use .  The configurations of the hypotheses involve m=4, 8, 16, 32, 64. And the number of truly null hypotheses being 3m/4, m/2, m/4, 0  The non-zero expectations were divided into 4 groups and placed at L/4, L/2, L3/4, and L in the following ways: (a) Linearly Decreasing (D) number of hypotheses of away from 0 in each group (b) Equally (E) number of hypotheses in each group (c) Linearly Increasing (I) number of hypotheses away from 0 in each group  These expectations were fixed (per configuration) throughout the experiment.  The variance of all variables was set to 1, and L was chosen at two levels 5 and 10 * 0.05 q   

THE ESTIMATES OF THE AVERAGE POWER  Simulation-based estimates of the average power: the proportion of the false null hypotheses which are correctly rejected.  Comparing the three methods 1. FDR controlling procedure : 2. Hochberg’s : - - - - - - 3. The Bonferroni-type : ……..

D E I m power

RESULT  The power of all the methods decreases when the number of hypotheses tested increases-this is the cost of multiplicity control.  The power is smallest for the D-configuration, where the non- null hypotheses are closer to the null, and is largest for I.  The power of the FDR controlling method is uniformly larger than that of the other methods.  The advantage increases in m. Therefore, the loss of power as m increases is relatively small for the FDR controlling method in the E- and I-configurations.  The advantage in some situations is extremely large.  Hochberg’s method offers a more powerful alternative to the Bonferroni method.

CONCLUSION  The new approach calls for the control of the FDR instead, and thereby also the control of the FWER in the weak sense. In many applications this is the desirable control against errors originating from multiplicity.  this paper focused on presenting and motivating the controlling the FDR, and it can be developed into a simple and powerful procedure. Thus the cost paid for the control of multiplicity need not be large.

WANG BING Thanks so much for your attention!