Simultaneous Testing of Multiple Hypotheses Using Generalized p-values


Kam-Wah Tsui and Shijie Tang
University of Wisconsin, Madison

Abstract

            For problems of simultaneously testing many hypotheses, Benjamini and Hochberg (1995) propose a procedure that guarantees the False Discovery Rate (FDR) to be less than or equal to a prefixed value. Here, FDR is the expected value of the ratio of the number of incorrectly rejected hypotheses and the total number of rejected hypotheses with the ratio defined to be zero if no hypothesis is rejected. Lehmann and Romano (2005) give a comprehensive overview of the development of the research in this area of multiple testing. To our knowledge, the research in this area assumes that a usual p-value is available for each hypothesis. However, in circumstances when nuisance parameters are present, usual p-values that are free of nuisance parameters may not exist. Tsui and Weerahandi (1989) introduce the generalized p-value method to handle testing problems where nuisance parameters are present. In this paper, we describe a multiple testing procedures using generalized p-values with the FDR not exceeding a prefixed value. Using an upper bound result from Tang and Tsui (2007) for the probability of type I error using a generalized p-value for the Behrens-Fisher problem, we develop a procedure to control FDR for the problem of simultaneously testing many Behrens-Fisher problems. Finally, we use a data set from a microarray experiment to illustrate our multiple testing procedure based on generalized p-values.