We first performed an extensive series of simulations, carefully designed to systematically assess the ability of each method to detect gene categories with varying expression pattern deviations (details in Materials and methods). In short, we simulated the global gene population by drawing 10,000 gene scores from a standard normal distribution. To simulate gene categories with known deviations, we used a mixture model 4 in which a proportion of the genes are given scores from a modulated normal distribution whereas the remaining genes scores follow a standard normal distribution like the population (Figure 1). Four parameters control the types of categories modeled: the number of genes in the category (N); the proportion of modulated genes (π); and the mean and standard deviation of the modulated gene scores (μ and σ). By varying these parameters, we could artificially recreate gene categories with a broad range of score-distribution dissimilarities.

To achieve near-exhaustive testing, we selected 1,800 parameter configurations from wide and relevant intervals, and determined the method powers for each one (see Materials and methods). Hence, for each method, we obtain an 1,800-dimensional performance profile, or detection spectrum, indicating the category types that can be detected.

We determined the detection spectra for the EDF-based methods and, for completeness, a discrete method with six thresholds for calling genes differentially expressed (D1 to D6; see Materials and methods). As evident in Figure 2 and Additional data file 1, the detection power varied considerably between methods and between category types: First, all methods worked well for detecting high-proportion-high-effects categories (Figure 2, upper right pie charts), but failed for low-proportion-low-effects categories (Figure 2, lower left pie charts). Second, all methods performed uniformly better in large than in small categories, owing to the fact that larger categories allow for detection of subtler deviations. Third, much more interestingly, substantial performance differences were observed for low to intermediate modulation effect sizes or proportions. In particular, some methods were better suited for detecting low-proportion-high-effects categories (Figure 2, lower right pie charts) whereas others were more apt for detecting high-proportion-low-effects categories (Figure 2, upper left pie charts). For low-proportion-high-effects categories, ZC and Zhang K (ZK) yielded the best results, followed by Zhang A (ZA) and Anderson-Darling (AD) (see Materials and methods). The discrete method also worked well, but exhibited strong threshold dependency. For high-proportion-low-effects categories, KS, the Cramér-von Mises (CM) statistic, and AD yielded the best results for narrow (σ = 0.1), intermediate (σ = 0.5) and diffuse (σ = 1.0) effect spreads, respectively. This is consistent with the fact that narrow effects spreads cause discrepancies near the center of the category gene score distribution (KS optimal), whereas intermediate and diffuse spreads lead to dissimilarities which, to greater extents, engage the tails (AD better suited). Fourth, we estimated the coverages of the detection spectra by computing overall (average) powers. The highest values were observed for AD, ZA, ZC, and ZK, implying that these methods are able to detect a broader range of categories than discrete and previous threshold-free (KS-based) methods (Additional data file 2). Taken together, these simulations clearly show that different ontological analysis methods focus on different types of categories, and provide an exact map of the method performances under varying circumstances.

To gain an overview of the mutual method relationships, we next quantified the method-method agreements, that is, the expected concordances between results, by computing the Spearman and Jaccard metrics (see Materials and methods) for all pairs of category-detection statistics. Interestingly, multidimensional scaling (MDS) of the resulting similarity matrices (Additional data file 3) showed that, property-wise, the methods represent a continuum ranging from the high-proportion-low-effects-focused (that is, center oriented) KS and CM at the one extreme to the low-proportion-high-effects-focused (that is, tail oriented) discrete method on the other with the remaining methods in between (Figure 3). In particular, we observe that ZC is the closest threshold-free approximation to the discrete method, and, hence, should be regarded as an appealing replacement for that method. Furthermore, we note that ZK is only slightly less tail oriented than ZC, and that the pairs ZA versus AD, and KS versus CM yield similar results. Moreover, because MDS captures the largest variability in the data, Figure 3 shows that a major determining factor of detection spectrum diversity lies in the methods' preferences for detecting high-proportion-low-effects or low-proportion-high-effects categories. In conclusion, the agreement data summarize the method relationships, and provide directions for the choice of method in practice.

We proceeded to apply all methods to real microarray data, starting with a dataset from a recent study of ours (P.H., B.N., A Andersson, C Lassen, U Gullberg, and T.F., unpublished work). The aim of this study was to map the transcriptional response of cells to the activity of the fusion oncogene BCR/ABL1, associated with chronic myeloid leukemia (CML), in a reverse way by blocking the activity of the fusion protein using the tyrosine-kinase inhibitory drug imatinib mesylate 9. Essentially, expression profiles of imatinib-treated and non-treated CML cell lines were acquired, and gene scores quantifying the imatinib response were computed (see Materials and methods).

As shown in Table 1, ontological analysis of the gene scores computed from the data from the imatinib experiment confirmed that the choice of method strongly influences the results when applied to real data. The overlaps between sets of detected categories approximately followed the simulations (Table 1), as did the category rankings (data not shown). Consistent with the overall power simulations, the threshold-free methods detected more categories than the discrete method, whereas the difference between the threshold-free methods were less pronounced (Figure 4). Investigating the putative biological relevance of the detected categories (Table 1), we noted several gene categories previously implicated in BCR/ABL1-mediated leukemogenesis or in the effects of imatinib. For example, consistent with data in the literature, significant enrichments of overexpression were observed in the categories 'heme biosynthesis', and enrichments of underexpression in the 'interferon-gamma signaling pathway', the 'MAPKKK cascade', and in categories related to apoptosis regulation. Also identified was the 'EGF receptor pathway', individual members of which are known to become phosphorylated/activated by BCR/ABL1. Taken together, these findings support the validity of the ontological analysis methodology. Finally, to point at important connections between gene score distributions and detection spectra, and to exemplify the utility of the indicator functions, we selected four illustrative categories, which are discussed in Figure 5.

To verify the generality of our results, we applied all methods to 25 other differential expression comparisons (Additional data file 4) based on seven publicly available microarray datasets (see Materials and methods). While a detailed description of the vast amount of resulting data is beyond the scope of this paper, we point out the following recurrent observations. First, as in Table 1, the choice of method strongly impacted on the results in accordance with the simulated method-method agreements. Second, the numbers of detected categories also approximately followed the simulated method relationships. In broad outline, the threshold-free methods detected many more categories than the discrete methods, because of the noticeable difference in overall power between these two groups. In contrast, the differences within the threshold-free group were less pronounced and more variable, which is explained by the fact that these methods have more similar overall powers, implying that the number of detected categories will, to a greater extent, be determined by the match between the detection spectrum and the set of deviating categories that are actually present in the data. Taken together, these findings further underscore the fact that different methods focus on different category types, and, hence, that it is important to be aware of this in practice.

The total time required for analyzing the 25 studies (all methods and ontologies) was 32 seconds (C++ implementation; 2 GHz Core2Duo PC), illustrating the benefit of the fast significance computation scheme (see Materials and methods).

Let F_N(x) and F(x) denote the empirical (cumulative) distribution functions for the gene-specific differential expression scores x₁,...,x_N for an N-gene category and the scores x'₁,...,x'_M for an M-gene reference gene population, respectively. We consider six EDF statistics to measure discrepancy between F_N(x) and F(x), that is, to detect gene categories with deviating gene-score distributions. A technicality that arises is that, normally when using EDF statistics, F(x) is specified by a continuous function. This is obviously not the case here as F(x) is an EDF, jumping by 1/M at each x'_i. However, we note that, in this application, this issue can be ignored because M is large (on the order of 5,000 to 40,000), making F(x) sufficiently smooth to be regarded as continuous.

First, we consider the Kolmogorov-Smirnov (KS) statistic 1011, which is, without doubt, the best-known EDF statistic and, as stated above, has been used previously for ontological analysis. The KS statistic is the largest distance between F(x) and F_N(x)

D = x ∈ R sup ⁡ | F N ( x ) − F ( x ) | = i = 1... N max ⁡ max ⁡ { i N − y i , y i − i − 1 N } , MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakqaaeeqaaiaadseacqGH9aqpdaqhaaWcbaGaamiEaiabgIGiolaadkfaaeaaciGGZbGaaiyDaiaacchaaaGccaGG8bGaamOramaaBaaaleaacaWGobaabeaakiaacIcacaWG4bGaaiykaiabgkHiTiaadAeacaGGOaGaamiEaiaacMcacaGG8baabaGaeyypa0Zaa0baaSqaaiaadMgacqGH9aqpcaaIXaGaaiOlaiaac6cacaGGUaGaamOtaaqaaiGac2gacaGGHbGaaiiEaaaakiGac2gacaGGHbGaaiiEaiaacUhadaWcaaqaaiaadMgaaeaacaWGobaaaiabgkHiTiaadMhadaWgaaWcbaGaamyAaaqabaGccaGGSaGaamyEamaaBaaaleaacaWGPbaabeaakiabgkHiTmaalaaabaGaamyAaiabgkHiTiaaigdaaeaacaWGobaaaiaac2hacaGGSaaaaaa@60F8@

where y_i = F(x_i). While intuitively straightforward and capable of detecting discrepancies near the center of the distribution, KS fails to notice subtle discrepancies in the tails as well as small but consistent deviations. Second and third, we use two members of the Cramér-von Mises family of quadratic EDF statistics, defined by

N ∫ − ∞ ∞ ( F N ( x ) − F ( x ) ) 2 ψ ( x ) d F ( x ) , MathType@MTEF@5@5@+=feaafiart1ev1aqatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakeaacaWGobWaa8qmaeaacaGGOaGaamOramaaBaaaleaacaWGobaabeaakiaacIcacaWG4bGaaiykaiabgkHiTiaadAeacaGGOaGaamiEaiaacMcacaGGPaWaaWbaaSqabeaacaaIYaaaaaqaaGGaciab=jHiTGGaaiab+5HiLcqaaiab+5HiLcqdcqGHRiI8aOGaeqiYdKNaaiikaiaadIhacaGGPaGaamizaiaadAeacaGGOaGaamiEaiaacMcacaGGSaaaaa@4D4F@

where ψ(x) is a suitable weight function. We consider ψ(x) = 1, which generates the Cramér-von Mises (CM) statistic itself 12

W 2 = N ∫ − ∞ ∞ ( F N ( x ) − F ( x ) ) 2 d F ( x ) = 1 12 N + ∑ i = 1 N ( y i − 2 i − 1 2 N ) 2 , MathType@MTEF@5@5@+=feaafiart1ev1aqatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakqaaeeqaaiaadEfadaahaaWcbeqaaiaaikdaaaGccqGH9aqpcaWGobWaa8qmaeaacaGGOaGaamOramaaBaaaleaacaWGobaabeaakiaacIcacaWG4bGaaiykaiabgkHiTiaadAeacaGGOaGaamiEaiaacMcacaGGPaWaaWbaaSqabeaacaaIYaaaaaqaaiabgkHiTiabg6HiLcqaaiabg6HiLcqdcqGHRiI8aOGaamizaiaadAeacaGGOaGaamiEaiaacMcaaeaacqGH9aqpdaWcaaqaaiaaigdaaeaacaaIXaGaaGOmaiaad6eaaaGaey4kaSYaaabCaeaacaGGOaGaamyEamaaBaaaleaacaWGPbaabeaakiabgkHiTmaalaaabaGaaGOmaiaadMgacqGHsislcaaIXaaabaGaaGOmaiaad6eaaaaaleaacaWGPbGaeyypa0JaaGymaaqaaiaad6eaa0GaeyyeIuoakiaacMcadaahaaWcbeqaaiaaikdaaaGccaGGSaaaaaa@6122@

which is able to integrate small consistent deviations, regarded as more powerful than KS, but is still not optimal for detecting discrepancies in the tails. Therefore, we also consider ψ(x) = F(x)^-1(1-F(x))^-1 which gives more weight to the tails and generates the Anderson-Darling (AD) statistic 13

A 2 = N ∫ − ∞ ∞ ( F N ( x ) − F ( x ) ) 2 F ( x ) ( 1 − F ( x ) ) d F = − N − 1 N ∑ i = 1 N ( 2 i − 1 ) ( ln ⁡ y i + ln ⁡ ( 1 − y N + 1 − i ) ) . MathType@MTEF@5@5@+=feaafiart1ev1aqatCvAUfeBSjuyZL2yd9gzLbvyNv2Caerbhv2BYDwAHbqedmvETj2BSbqee0evGueE0jxyaibaiKI8=vI8tuQ8FMI8Gi=hEeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciGacaGaaeqabaqadeqadaaakqaaeeqaaiaadgeadaahaaWcbeqaaiaaikdaaaGccqGH9aqpcaWGobWaa8qmaeaadaWcaaqaaiaacIcacaWGgbWaaSbaaSqaaiaad6eaaeqaaOGaaiikaiaadIhacaGGPaGaeyOeI0IaamOraiaacIcacaWG4bGaaiykaiaacMcadaahaaWcbeqaaiaaikdaaaaakeaacaWGgbGaaiikaiaadIhacaGGPaGaaiikaiaaigdacqGHsislcaWGgbGaaiikaiaadIhacaGGPaGaaiykaaaaaSqaaiabgkHiTiabg6HiLcqaaiabg6HiLcqdcqGHRiI8aOGaamizaiaadAeaaeaacqGH9aqpcqGHsislcaWGobGaeyOeI0YaaSaaaeaacaaIXaaabaGaamOtaaaadaaeWbqaaiaacIcacaaIYaGaamyAaiabgkHiTiaaigdacaGGPaGaaiikaiGacYgacaGGUbGaamyEamaaBaaaleaacaWGPbaabeaakiabgUcaRiGacYgacaGGUbGaaiikaiaaigdacqGHsislcaWG5bWaaSbaaSqaaiaad6eacqGHRaWkcaaIXaGaeyOeI0IaamyAaaqabaGccaGGPaGaaiykaiaac6caaSqaaiaadMgacqGH9aqpcaaIXaaabaGaamOtaaqdcqGHris5aaaaaa@736E@

Fourth, fifth, and sixth, we include three EDF statistics recently derived by Zhang 14. These are denoted Zhang A (ZA), Zhang C (ZC) and Zhang K (ZK) to reflect theoretical relationships with AD, CM, and KS. However, simulations have shown that the Zhang statistics are sometimes substantially more powerful 14. The derivations of the Zhang statistics are beyond the scope of this paper, but can be found in the original work. The computing formulas are:

Z A = − ∑ i = 1 N ( ln ⁡ y i N − i + 1 / 2 + ln ⁡ ( 1 − y i ) i − 1 / 2 ) , Z C = ∑ i = 1 N ln ⁡ 2 ( y i − 1 − 1 ( N − 1 / 2 ) / ( i − 3 / 4 ) − 1 ) , Z K = i = 1... N max ⁡ ( ( i − 1 2 ) ln ⁡ ( i − 1 / 2 N y i ) + ( N − i + 1 2 ) ln ⁡ ( N − i + 1 / 2 N ( 1 − y