Suppose we want to compare the results of two microarray experiments, each of them reporting for the same set of n genes a measure of differential expression on a probability scale (for example, p value; Table 1).

We rank the genes according to the recorded probability measures. For each cut-off q,(0 ≤ q ≤ 1), we obtain the number of differentially expressed genes for each of the two lists as O₁₊(q) and O₊₁(q) and the number O₁₁(q) of differentially expressed genes in common between the two experiments (Table 2). The threshold q is a continuous variable but, in practice, we consider a discretization of q. In the present paper, we specify a vector q = (q₀ = 0, q₁ = 0.001. ..., q, ..., q_k = 1), formed by K = 101 elements, but other discretizations can be used without loss of generality. For a threshold q, under the hypothesis of independence of the contrasts investigated by the two experiments, the number of genes in common by chance is calculated as:

O 1 + ( q ) × O + 1 ( q ) n MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaadaWcaaqaaiaad+eadaWgaaWcbaGaaGymaiabgUcaRaqabaGccaGGOaGaamyCaiaacMcacqGHxdaTcaWGpbWaaSbaaSqaaiabgUcaRiaaigdaaeqaaOGaaiikaiaadghacaGGPaaabaGaamOBaaaaaaa@4033@

In the 2 × 2 Table, where the marginal frequencies O₁₊(q), O₊₁(q) and the total number of genes n are assumed fixed quantities, given q, the only random variable is O₁₁(q).

The conditional distribution of O₁₁(q) is hypergeometric 7:

O₁₁(q) ~ Hyper(O₁₊(q), O₊₁(q), n).

We then calculate the statistic T(q) as the observed to expected ratio:

T ( q ) = O 11 ( q ) O 1 + ( q ) × O + 1 ( q ) n . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaacaWGubGaaiikaiaadghacaGGPaGaeyypa0ZaaSaaaeaacaWGpbWaaSbaaSqaaiaaigdacaaIXaaabeaakiaacIcacaWGXbGaaiykaaqaamaalaaabaGaam4tamaaBaaaleaacaaIXaGaey4kaScabeaakiaacIcacaWGXbGaaiykaiabgEna0kaad+eadaWgaaWcbaGaey4kaSIaaGymaaqabaGccaGGOaGaamyCaiaacMcaaeaacaWGUbaaaaaacaGGUaaaaa@49F2@

In other words, T(q) quantifies the strength of association between lists at cut-off q in terms of ratio of observed to expected. The denominator is a fixed quantity, so the distribution of T(q) is also proportional to a hypergeometric distribution:

T_q ∝ Hyper(O₁₊(q), O₊₁(q), n)

with mean and variance:

E(T(q)|O₁₊(q), O₊₁(q), n) = 1

V a r ( T ( q ) | O 1 + ( q ) , O + 1 ( q ) , n ) = ( 1 − O 1 + ( q ) n ) × ( n − O + 1 ( q ) n − 1 ) . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaacaWGwbGaamyyaiaadkhacaGGOaGaamivaiaacIcacaWGXbGaaiykaiaacYhacaWGpbWaaSbaaSqaaiaaigdacqGHRaWkaeqaaOGaaiikaiaadghacaGGPaGaaiilaiaad+eadaWgaaWcbaGaey4kaSIaaGymaaqabaGccaGGOaGaamyCaiaacMcacaGGSaGaamOBaiaacMcacqGH9aqpdaqadaqaaiaaigdacqGHsisldaWcaaqaaiaad+eadaWgaaWcbaGaaGymaiabgUcaRaqabaGccaGGOaGaamyCaiaacMcaaeaacaWGUbaaaaGaayjkaiaawMcaaiabgEna0oaabmaabaWaaSaaaeaacaWGUbGaeyOeI0Iaam4tamaaBaaaleaacqGHRaWkcaaIXaaabeaakiaacIcacaWGXbGaaiykaaqaaiaad6gacqGHsislcaaIXaaaaaGaayjkaiaawMcaaiaac6caaaa@5FA8@

Throughout, we use the symbol | to denote conditioning, thus E(T(q)|O₁₊(q), O₊₁(q), n) indicates the conditional expectation of T(q) given O₁₊(q), O₊₁(q) and n.

As a first step, we focus attention on the ordinal statistic T(q_max) ≡ max_qT(q), which represents the maximal deviation from the null model of independence between the two experiments, or equivalently the largest relative increase of the number of genes in common. This maximum value is associated with a threshold q_max on the probability measure and with a number O₁₁(q_max) of genes in common, which can be selected for further investigations and mined for relevant biological pathways.

The exact distribution of T(q_max) is not easily obtained, since the series of 2 × 2 tables are not independent. We thus suggest performing a Monte Carlo permutation test of T(q) under the null hypothesis of independence between the two experiments. To be precise, the probability measures of one list are randomly permuted S times, while those of the other list are kept fixed, leading to S values of the statistic T^S(q_max), which represent the null distribution of T(q_max). From these, a Monte Carlo p value for the observed value of T(q_max) can be computed and the choice of S adapted to the required degree of precision.

For extreme values of the threshold q (q ≅ 0), O₁₊(q) and O₊₁(q) can be very small. In this case, the denominator of T(q) assumes values smaller than 1 and T(q) explodes, leading to unreliable estimates of the ratio. In addition, the hypergeometric sampling model specified for T(q_max) in our previous procedure does not take into account the uncertainty of the margins of the table (since they are all considered fixed).

To address these issues and to improve our statistical procedure, we thus propose to consider a joint model of the experiments, which also treats O₁₊(q) and O₊₁(q) as random variables, releasing the conditioning. Furthermore, we specify this in a Bayesian framework, where the underlying probabilities,

θ i ( q ) , 1 ≤ i ≤ 4 , ∑ i = 1 4 θ i ( q ) = 1 , MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaaiiGacqWF4oqCdaWgaaWcbaGaamyAaaqabaGccaGGOaGaamyCaiaacMcacaGGSaGaaGymaiabgsMiJkaadMgacqGHKjYOcaaI0aGaaiilamaaqahabaGae8hUde3aaSbaaSqaaiaadMgaaeqaaOGaaiikaiaadghacaGGPaaaleaacaWGPbGaeyypa0JaaGymaaqaaiaaisdaa0GaeyyeIuoakiabg2da9iaaigdacaGGSaaaaa@4CDC@

for the four cells in the 2 × 2 contingency table (indexes from left to right) are given a prior distribution. In this way, we account for the variability in O₁₊(q) and O₊₁(q) and smooth the ratio T(q) for extreme, small values of q.

Starting from Table 2, we model the observed frequencies as arising from a multinomial distribution:

M u l t i ( O | θ , n ) α θ 1 ( q ) O 11 ( q ) × θ 2 ( q ) [ O 1 + ( q ) − O 11 ( q ) ] × θ 3 ( q ) [ O + 1 ( q ) − O 11 ( q ) ] × θ 4 ( q ) [ n − O 1 + ( q ) − O + 1 ( q ) + O 11 ( q ) ] MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaacaWGnbGaamyDaiaadYgacaWG0bGaamyAaiaacIcaieqacaWFpbGaaiiFaGGaciab+H7aXjaacYcacaWGUbGaaiykaiab+f7aHjab+H7aXnaaBaaaleaacaaIXaaabeaakiaacIcacaWGXbGaaiykamaaCaaaleqabaGaam4tamaaBaaameaacaaIXaGaaGymaaqabaWccaGGOaGaamyCaiaacMcaaaGccqGHxdaTcqGF4oqCdaWgaaWcbaGaaGOmaaqabaGccaGGOaGaamyCaiaacMcadaahaaWcbeqaaiaacUfacaWGpbWaaSbaaWqaaiaaigdacqGHRaWkaeqaaSGaaiikaiaadghacaGGPaGaeyOeI0Iaam4tamaaBaaameaacaaIXaGaaGymaaqabaWccaGGOaGaamyCaiaacMcacaGGDbaaaOGaey41aqRae4hUde3aaSbaaSqaaiaaiodaaeqaaOGaaiikaiaadghacaGGPaWaaWbaaSqabeaacaGGBbGaam4tamaaBaaameaacqGHRaWkcaaIXaaabeaaliaacIcacaWGXbGaaiykaiabgkHiTiaad+eadaWgaaadbaGaaGymaiaaigdaaeqaaSGaaiikaiaadghacaGGPaGaaiyxaaaakiabgEna0kab+H7aXnaaBaaaleaacaaI0aaabeaakiaacIcacaWGXbGaaiykamaaCaaaleqabaGaai4waiaad6gacqGHsislcaWGpbWaaSbaaWqaaiaaigdacqGHRaWkaeqaaSGaaiikaiaadghacaGGPaGaeyOeI0Iaam4tamaaBaaameaacqGHRaWkcaaIXaaabeaaliaacIcacaWGXbGaaiykaiabgUcaRiaad+eadaWgaaadbaGaaGymaiaaigdaaeqaaSGaaiikaiaadghacaGGPaGaaiyxaaaaaaa@8CC6@

Since we are in a Bayesian framework, we need to specify a prior distribution for all the parameters. The vector of parameters θ(q) is modeled as arising from a Dirichlet distribution 8:

θ(q) ~ Dir(a, a, a, a), a = 0.05,

which ensures the constraint ∑i=14θi(q)=1 MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaadaaeWaqaaGGaciab=H7aXnaaBaaaleaacaWGPbaabeaakiaacIcacaWGXbGaaiykaiabg2da9iaaigdaaSqaaiaadMgacqGH9aqpcaaIXaaabaGaaGinaaqdcqGHris5aaaa@3F8D@.

The derived quantity of interest is, as before, the ratio of the probability that a differentially expressed gene is truly common for both experiments, to the probability that a gene is included in the common list by chance:

R ( q ) = θ 1 ( q ) ( θ 1 ( q ) + θ 2 ( q ) ) × ( θ 1 ( q ) + θ 3 ( q ) ) . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaacaWGsbGaaiikaiaadghacaGGPaGaeyypa0ZaaSaaaeaacqaH4oqCdaWgaaWcbaGaaGymaaqabaGccaGGOaGaamyCaiaacMcaaeaacaGGOaGaeqiUde3aaSbaaSqaaiaaigdaaeqaaOGaaiikaiaadghacaGGPaGaey4kaSIaeqiUde3aaSbaaSqaaiaaikdaaeqaaOGaaiikaiaadghacaGGPaGaaiykaiabgEna0kaacIcacqaH4oqCdaWgaaWcbaGaaGymaaqabaGccaGGOaGaamyCaiaacMcacqGHRaWkcqaH4oqCdaWgaaWcbaGaaG4maaqabaGccaGGOaGaamyCaiaacMcacaGGPaaaaiaac6caaaa@5779@

The Dirichlet prior is conjugate for the multinomial likelihood 8 and the posterior distribution of θ(q)|O, n is again a Dirichlet distribution, given by:

θ | O , n ~ D i r ( O 11 ( q ) + a , [ O 1 + ( q ) − O 11 ( q ) ] + a , [ O + 1 ( q ) − O 11 ( q ) ] + a , [ n − O 1 + ( q ) − O + 1 ( q ) + O 11 ( q ) ] + a ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaacqaH4oqCcaGG8bacbeGaa83taiaacYcacaWGUbGaaiOFaiaadseacaWGPbGaamOCaiaacIcacaWGpbWaaSbaaSqaaiaaigdacaaIXaaabeaakiaacIcacaWGXbGaaiykaiabgUcaRiaadggacaGGSaGaai4waiaad+eadaWgaaWcbaGaaGymaiabgUcaRaqabaGccaGGOaGaamyCaiaacMcacqGHsislcaWGpbWaaSbaaSqaaiaaigdacaaIXaaabeaakiaacIcacaWGXbGaaiykaiaac2facqGHRaWkcaWGHbGaaiilaiaacUfacaWGpbWaaSbaaSqaaiabgUcaRiaaigdaaeqaaOGaaiikaiaadghacaGGPaGaeyOeI0Iaam4tamaaBaaaleaacaaIXaGaaGymaaqabaGccaGGOaGaamyCaiaacMcacaGGDbGaey4kaSIaamyyaiaacYcacaGGBbGaamOBaiabgkHiTiaad+eadaWgaaWcbaGaaGymaiabgUcaRaqabaGccaGGOaGaamyCaiaacMcacqGHsislcaWGpbWaaSbaaSqaaiabgUcaRiaaigdaaeqaaOGaaiikaiaadghacaGGPaGaey4kaSIaam4tamaaBaaaleaacaaIXaGaaGymaaqabaGccaGGOaGaamyCaiaacMcacaGGDbGaey4kaSIaamyyaiaacMcaaaa@7876@

This distribution is easily sampled from using standard algorithms. Note that the prior weights a = 0.05 can be interpreted as the number of hypothetical counts in each cell observed prior to the investigation. Further, it can be shown that the variance of the vector of probabilities in the Dirichlet distribution increases as the prior weights tend to zero. Thus, our choice of value of 0.05 for the prior weights allows both high variability and a small influence of the prior specification on the posterior distribution of θ(q). The posterior distribution of R(q)|O, n can be easily derived from that of θ(q) using for example a sample of values of θ(q), generated from the posterior distribution (equation 5). In particular, from a sample of values of R(q)|O, n, the 95% two sided credibility interval, CI₉₅(q), can be easily computed, for each R(q).

In the Bayesian context, several decision rules can be envisaged to choose the threshold corresponding to the common list showing a clear evidence of association between experiments. The general principle is as follows: first, select a ratio R(q) according to a decision rule; second, consider the threshold q corresponding to the selected ratio; and third, return the list O₁₁(q), that is, the intersection of the lists for the threshold q. Figure 1 (right) shows a typical plot of R(q) and its credibility interval as a function of q in case of associated experiments (a different shape for R(q) is presented in Additional data file 1). As the p value increases, the ratio R(q) decreases and the associated list of common genes O₁₁(q) becomes larger (the number of genes in common for each ratio is indicated on the right axis of the plot). We need a rule to select a threshold on the p value and the corresponding list of genes in common. To this purpose we now discuss two decision rules.

Under the null model of no association between the experiments, Median(R(q)|H₀) = 1, so we consider R(q) as indicating departure from independence if its credibility interval does not contain 1.

As an extension of T(q_max) we thus propose to consider the maximum of Median(R(q)|O, n) only for the subset of credibility intervals that do not include 1 and define:

q_max = argmax{Median(R(q)|O, n) over the set of values of q for which CI₉₅(q) excludes 1}.

In other words, q_max is defined to be the threshold associated with the maximum of the ratio, which we denote R(q_max). If all credibility intervals contain 1, the maximum of R(q) can still be computed, but we do not associate it with a list since there is no departure from independence that could be considered significant.

Note that in the Bayesian context many R(q) can have a CI that excludes 1 and they all represent a significant deviation from the independence. An advantage of the maximum statistic is that it returns a list of interesting features with few false positives (FP), as will be shown later in the simulations. On the other hand, this list is usually rather small and in cases where the level of noise is substantial it excludes a large number of true positives (TP), for which the evidence is less strong.

We next consider an alternative to the max ratio: the largest threshold q for which the ratio R(q) ≥ 2. It is the largest threshold where the number of genes called in common at least doubles the number of genes in common under independence:

q₂ = max{over the set of values of q for which Median(R(q)|O, n) ≥ 2 and CI₉₅(q) excludes 1}.

Using this rule provides a fair balance between specificity and sensitivity as we will show later. Indeed, it is expected that when going beyond this point to larger values of q, the marginal benefit of adding a few more true positives and of reducing the false negatives (FN) to the list will be outweighed by the expected larger number of false positives that would also be added. By our simulations we show indeed that this rule is close to giving the minimal global error (FP + FN).

Figure 2 (top) plots the false discovery rate:

FDR = FP(q)/O₁₁(q)

and false non-discovery rate:

FNR = FN(q)/(n - O₁₁(q))

for 50 simulations carried out as described in Materials and methods, for scenario I structure A. It is clear that R(q_max) has the smallest FDR. On the other hand, q₂ corresponds to the intersection between FDR and FNR. Moreover, in Figure 2 (bottom) we show that the same threshold minimizes the global misclassification error as the sum of false positives and false negatives. Note that if we considered the minimum significant ratio, defined as the minimum of the R(q) over the set of credibility intervals excluding 1, FDR would increase dramatically and the FNR would decrease only marginally with respect to R(q_max) and R(q₂). As expected, the global misclassification error would also be much larger, making this rule inappropriate.

When there are no ratios R(q) equal or greater than 2 (which can happen in the case of large noise or when there is only a small proportion of genes in common), this rule does not apply and we recommend using the rule corresponding to R(q_max).

Our computations have been implemented in the statistical programming language R 9. The R package for simulating the data, for the two tests and for visualizing the results is called BGcom and is available on our project BGX website 10.

Besides assessing the operating characteristics of our proposed rules, we also applied the method proposed by Hwang et al. implemented in Matlab 11. Note that their aim is to integrate p values from different experiments in a meta-analysis and they present three statistics to do so: Fisher's weighted F, Mudholkar-George's weighted T and Liptak-Stouffer's weighted Z. We report Fisher's weighted F (the default statistic in the Matlab function), defined as:

F g = − 2 ∑ k = 1 2 w k l n ( p g k ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaacaWGgbWaaSbaaSqaaiaadEgaaeqaaOGaeyypa0JaeyOeI0IaaGOmamaaqadabaGaam4DamaaBaaaleaacaWGRbaabeaaieGakiaa=XgacaWFUbGaaiikaiaadchadaWgaaWcbaGaam4zaiaadUgaaeqaaOGaaiykaaWcbaGaam4Aaiabg2da9iaaigdaaeaacaaIYaaaniabggHiLdaaaa@45A1@

where w_k is the weight for the k^th experiment and p_gk is the p value for the gene g in the experiment k. F_g will be a new global p value that integrates those weights from different experiments. The authors also present several rules to select differentially expressed genes from F_g, the simplest one using a fixed threshold on the p values equal to 0.05, and others that minimize the number of false positives and false negatives, in a parametric or non-parametric framework. We follow the authors' suggestion and use the non-parametric rule. For more details on the method, see 2.

The behavior of T(q) and of the credibility intervals CI₉₅(q) for a typical simulation are displayed in Figure 1 (associated experiments) and Figure 3 (independent experiments). When the two experiments are not associated (the number of simulated genes in common is equal to 0), the plot of T(q) for different cut-offs q is, as expected, a horizontal line of height 1, with evidence of noise for small p values. In the same Figure, one sees that all the credibility intervals derived by the Bayesian procedure include the value 1 and have decreasing width as q gets larger, as expected.

In the case of two independent experiments we never declare any gene to be in common in any of the 50 simulations, so our procedure has no error. On the other hand, Hwang et al.'s method picks up 320 genes on average (Table 3, independence case), which are all false positives.

When there is a positive association between the two experiments, T(q) can assume two shapes: it can decrease monotonically as the p values increase (Figure 1), or reach a peak and then decrease (Additional data file 1) as the p values increase. The Bayesian estimates exhibit a similar shape, but since in this approach the variability of the denominator of T(q) is modeled, the resulting ratio estimates are smoothed.

We see that our proposed method gives a sensible and interpretable procedure, with a pattern that is easily distinguishable from that of the no association case. This is confirmed by the results given in Table 4.