We assume that the number of reads in sample j that are assigned to gene i can be modeled by a negative binomial (NB) distribution,

K i j ~ NB ( μ i j , σ i j 2 ) ,

which has two parameters, the mean μ_ij and the variance σ i j 2 . The read counts K_ij are non-negative integers. The probabilities of the distribution are given in Supplementary Note A. (All Supplementary Notes are in Additional file 1.) The NB distribution is commonly used to model count data when overdispersion is present 12 .

In practice, we do not know the parameters μ_ij and σ i j 2 , and we need to estimate them from the data. Typically, the number of replicates is small, and further modelling assumptions need to be made in order to obtain useful estimates. In this paper, we develop a method that is based on the following three assumptions.

First, the mean parameter μ_ij , that is, the expectation value of the observed counts for gene i in sample j, is the product of a condition-dependent per-gene value q _{i, ρ(j)} (where ρ(j) is the experimental condition of sample j) and a size factor s _j ,

μ i j = q i , ρ ( j ) S j .

q _i,ρ(j) is proportional to the expectation value of the true (but unknown) concentration of fragments from gene i under condition ρ(j). The size factor s _j represents the coverage, or sampling depth, of library j, and we will use the term common scale for quantities, such as q _{i, ρ(j)}, that are adjusted for coverage by dividing by s _j .

Second, the variance σ i j 2 is the sum of a shot noise term and a raw variance term,

σ i j 2 = μ i j ︸ shot     noise + s j 2 v i , ρ ( j ) ︸ raw     variance .

Third, we assume that the per-gene raw variance parameter v _{i, ρ} is a smooth function of q _i , ρ,

v i , ρ ( j ) = v ρ ( q i , ρ ( j ) ) .

This assumption is needed because the number of replicates is typically too low to get a precise estimate of the variance for gene i from just the data available for this gene. This assumption allows us to pool the data from genes with similar expression strength for the purpose of variance estimation.

The decomposition of the variance in Equation (3) is motivated by the following hierarchical model: We assume that the actual concentration of fragments from gene i in sample j is proportional to a random variable R_ij , such that the rate that fragments from gene i are sequenced is s_jr_ij . For each gene i and all samples j of condition ρ, the R_ij are i.i.d. with mean q_iρ and variance v_iρ . Thus, the count value K_ij , conditioned on R_ij = r_ij , is Poisson distributed with rate s_jr_ij . The marginal distribution of K_ij - when allowing for variation in R_ij - has the mean μ_ij and (according to the law of total variance) the variance given in Equation (3). Furthermore, if the higher moments of R_ij are modeled according to a gamma distribution, the marginal distribution of K_ij is NB (see, for example, 12 , Section 4.2.2).

We now describe how the model can be fitted to data. The data are an n × m table of counts, k_ij , where i = 1,..., n indexes the genes, and j = 1,..., m indexes the samples. The model has three sets of parameters:

(i) m size factors s_j ; the expectation values of all counts from sample j are proportional to s_j .

(ii) for each experimental condition ρ, n expression strength parameters q_iρ ; they reflect the expected abundance of fragments from gene i under condition ρ, that is, expectation values of counts for gene i are proportional to q_iρ .

(iii) The smooth functions v_ρ : ℝ⁺ → ℝ⁺; for each condition ρ, v_ρ models the dependence of the raw variance v_iρ on the expected mean q_iρ .

The purpose of the size factors s_j is to render counts from different samples, which may have been sequenced to different depths, comparable. Hence, the ratios ( E K_ij )/( E K_ij' ) of expected counts for the same gene i in different samples j and j' should be equal to the size ratio s_j /s_j' if gene i is not differentially expressed or samples j and j' are replicates. The total number of reads, Σ _i k_ij , may seem to be a good measure of sequencing depth and hence a reasonable choice for s_j . Experience with real data, however, shows this not always to be the case, because a few highly and differentially expressed genes may have strong influence on the total read count, causing the ratio of total read counts not to be a good estimate for the ratio of expected counts.

Hence, to estimate the size factors, we take the median of the ratios of observed counts. Generalizing the procedure just outlined to the case of more than two samples, we use:

s ^ j = median i k i j ( ∏ v = 1 m k i v ) 1 / m .

The denominator of this expression can be interpreted as a pseudo-reference sample obtained by taking the geometric mean across samples. Thus, each size factor estimate s ^ j is computed as the median of the ratios of the j-th sample's counts to those of the pseudo-reference. (Note: While this manuscript was under review, Robinson and Oshlack 13 suggested a similar method.)

To estimate q_iρ , we use the average of the counts from the samples j corresponding to condition ρ, transformed to the common scale:

q ^ i ρ = 1 m ρ ∑ j : ρ ( j ) = ρ k i j s ^ j ,

where m_ρ is the number of replicates of condition ρ and the sum runs over these replicates. the functions v_ρ , we first calculate sample variances on the common scale

w i ρ = 1 m ρ − 1 ∑ j : ρ ( j ) = ρ ( k i j s ^ j − q ^ i ρ ) 2

and define

z i ρ = q ^ i ρ m ρ ∑ j : ρ ( j ) = ρ 1 s ^ j .

In Supplementary Note B in Additional file 1 we show that w_iρ - z_iρ is an unbiased estimator for the raw variance parameter v_iρ of Equation (3).

However, for small numbers of replicates, m_ρ , as is typically the case in applications, the values w_iρ are highly variable, and w_iρ - z_iρ would not be a useful variance estimator for statistical inference. Instead, we use local regression 14 on the graph ( q ^ i ρ , w i ρ ) to obtain a smooth function w_ρ (q), with

v ^ ρ ( q ^ i ρ ) = w ρ ( q ^ i ρ ) − z i ρ

as our estimate for the raw variance.

Some attention is needed to avoid estimation biases in the local regression. w_iρ is a sum of squared random variables, and the residuals w i ρ − w ( q ^ i ρ ) are skewed. Following References 15 , Chapter 8 and 14 , Section 9.1.2, we use a generalized linear model of the gamma family for the local regression, using the implementation in the locfit package 16 .

We present results based on the following data sets:

We start by demonstrating the variance estimation. Figure 1a shows the sample variances w _iρ (Equation (7)) plotted against the means q ^ i ρ (Equation (6)) for condition A in the fly RNA-Seq data. Also shown is the local regression fit w_ρ (q) and the shot noise s ^ j q ^ i ρ . In Figure 1b, we plotted the squared coefficient of variation (SCV), that is the ratio of the variance to the mean squared. In this plot, the distance between the orange and the purple line is the SCV of the noise due to biological sampling (cf. Equation (3)).

The many data points in Figure 1b that lie far above the fitted orange curve may let the fit of the local regression appear poor. However, a strong skew of the residual distribution is to be expected. See Supplementary Note E in Additional file 1 for details and a discussion of diagnostics suitable to verify the fit.

In order to verify that DESeq maintains control of type-I error, we contrasted one of the replicates for condition A in the fly data against the other one, using for both samples the variance function estimated from the two replicates. Figure 2 shows the empirical cumulative distribution functions (ECDFs) of the P values obtained from this comparison. To control type-I error, the proportion of P values below a threshold α has to be ≤ α, that is, the ECDF curve (blue line) should not get above the diagonal (gray line). As the figure indicates, type-I error is controlled by edgeR and DESeq, but not by a Poisson-based χ ² test. The latter underestimates the variability of the data and would thus make many false positive rejections. In addition to this evaluation on real data, we also verified DESeq's type-I error control on simulated data that were generated from the error model described above; see Supplementary Note G in Additional file 1. Next, we contrasted the two A samples against the two B samples. Using the procedure described in the previous Section, we computed a P value for each gene. Figure 3 shows the obtained fold changes and P values. 12% of the P values were below 5%. Adjustment for multiple-testing with the procedure of Benjamini and Hochberg 21 yielded significant differential expression at false discovery rate (FDR) of 10% for 864 genes (of 17,605). These are marked in red in the figure. Figure 3 demonstrates how the ability to detect differential expression depends on overall counts. Specifically, the strong shot noise for low counts causes the testing procedure to call only very high fold changes significant. It can also be seen that, for counts below approximately 100, even a small increase in count levels reduces the impact of shot noise and hence the fold-change requirement, while at higher counts, when shot noise becomes unimportant (cf. Figure 1b), the fold-change cut-off depends only weakly on count level. These plots are helpful to guide experiment design: For weakly expressed genes, in the region where shot noise is important, power can be increased by deeper sequencing, while for the higher-count regime, increased power can only be achieved with further biological replicates.

We also analyzed the data with edgeR (version 1.6.0; 8 10 11 ). We ran edgeR with four different settings, namely in common-dispersion and in tagwise-dispersion mode, and either using the size factors as estimated by DESeq or taking the total numbers of sequenced reads. The results did not depend much on these choices, and here we report the results for tag-wise dispersion mode with DESeq-estimated size factors. (The R code required to reproduce all analyses, figures and numbers reported in this article is provided in Additional file 2; in addition, this supplement provides the results for the other settings of edgeR. The raw data can be found in Additional file 3.)

Going back to Figure 1 we see that edgeR's single-value dispersion estimate of the variance is lower than that of DESeq for weakly expressed genes and higher for strongly expressed genes. As a consequence, as we have seen in Figure 2 edgeR is anti-conservative for lowly expressed genes. However, it compensates for this by being more conservative with strongly expressed genes, so that, on average, type-I error control is maintained.

Nevertheless, in a test between different conditions, this behavior can result in a bias in the list of discoveries; for the present data, as Figure 4 shows, weakly expressed genes seem to be overrepresented, while very few genes with high average level are called differentially expressed by edgeR. While overall the sensitivity of both methods seemed comparable (DESeq reported 864 hits, edgeR 1, 127 hits), DESeq produced results which were more balanced over the dynamic range.

Similar results were obtained with the neural stem cell data, a data set with a different biological background and different noise characteristics (see Supplementary Note F in Additional file 1). The flexibility of the variance estimation scheme presented in this work appears to offer real advantages over the existing methods across a range of applications.

DESeq allows analysis of experiments with no biological replicates in one or even both of the conditions. While one may not want to draw strong conclusions from such an analysis, it may still be useful for exploration and hypothesis generation.

If replicates are available only for one of the conditions, one might choose to assume that the variance-mean dependence estimated from the data for that condition holds as well for the unreplicated one.

If neither condition has replicates, one can still perform an analysis based on the assumption that for most genes, there is no true differential expression, and that a valid mean-variance relationship can be estimated from treating the two samples as if they were replicates. A minority of differentially abundant genes will act as outliers; however, they will not have a severe impact on the gamma-family GLM fit, as the gamma distribution for low values of the shape parameter has a heavy right-hand tail. Some overestimation of the variance may be expected, which will make that approach conservative.

We performed such an analysis with the fly RNA-Seq and the neural cell Tag-Seq data, by restricting both data sets to only two samples, one from each condition. For the neural cell data, the estimated variance function was, as expected, somewhat above the two functions estimated from the GNS and NS replicates.

Using it to test for differential expression still found 269 hits at FDR = 10%, of which 202 were among the 612 hits from the more reliable analysis with all available samples. In the case of the fly RNA-Seq data, however, only 90 of the 862 hits (11%) were recovered (with two new hits). These observations are explained by the fact that in the neural cell data, the variability between replicates was not much smaller than between conditions, making the latter a usable surrogate for the former. On the other hand, for the fly data, the variability between replicates was much smaller than between the conditions, indicating that the replication provided important and otherwise not available information on the experimental variation in the data (see also next Section).

Given a variance-mean dependence, a variance-stabilizing transformation (VST) is a monotonous mapping such that for the transformed values, the variance is (approximately) independent of the mean. Using the variance-mean dependence w(q) estimated by DESeq, a VST is given by

τ ( κ ) = ∫ κ d q w ( q ) .

Applying the transformation τ to the common-scale count data, k_ij /s_j , yields values whose variances are approximately the same throughout the dynamic range. One application of VST is sample clustering, as in Figure 5; such an approach is more straightforward than, say, defining a suitable distance metric on the untransformed count data, whose choice is not obvious, and may not be easy to combine with available clustering or classification algorithms (which tend to be designed for variables with similar distributional properties).

DESeq can also be used to analyze comparative ChIP-Seq assays. Kasowski et al. 20 analyzed transcription factor binding for HapMap individuals and counted for each sample how many reads mapped to pre-determined binding regions. We considered two individuals from their data set, HapMap IDs GM12878 and GM12891, for both of which at least four replicates had been done, and tested for differential occupation of the regions. The upper left two panels of Figure 6 which show comparisons within the same individual, indicate that type-I error was controlled by DESeq. No region was significant at 10% FDR using Benjamini-Hochberg adjustment. Differential occupation was found, however, when contrasting the two individuals, with 4,460 of 19,028 regions significant when only two replicates each were used and 8,442 when four replicates were used (upper right two panels).

Using an alternative approach, Kasowski et al. fitted generalized linear models (GLMs) of the Poisson family. This (lower row of Figure 6) resulted in an enrichment of small P values even for comparisons within the same individual, indicating that the variance was underestimated by the Poisson GLM, and literal use of the P values would lead to anti-conservative (overly optimistic) bias. Kasowski et al. addressed this and adjusted for the bias by using additional criteria for calling differential occupation.