Prior to applying ICA, we normalize the expression matrices X to contain log ratios x_ij = log₂(R_ij/G_ij) of red and green intensities and we remove any samples that are closely approximated as linear combinations of other samples. We find as many independent components as samples in the input dataset, that is, M = N (see Discussion). The algorithms we use for ICA are described in Methods.

Based on our model, each component is a putative genomic expression program of an independent biological process. Our hypothesis is that genes showing relatively high or low expression levels within the component are the most important for the process. First, for each independent component, we sort genes by the loads within the component. Then we create two clusters for each component: one cluster containing C% of all genes with larger loads, and one cluster containing C% of genes with smaller loads.

Cluster _i,1 = {gene j | y_ij = (C% × K)^th largest load in y_i}

Cluster _i,2 = {gene j | y_ij = (C% × K)^th smallest load in y_i}     (3)

In Equation 3, y_i is the i^th independent component, a vector of length K; and C is an adjustable coefficient.

For each cluster, we measure the enrichment with genes of known functional annotations.

In our datasets we measured the biological significance of each cluster as follows. For datasets 1-4, we used the Gene Ontology (GO) 43 and the Kyoto Encyclopedia of Genes and Genomes (KEGG) 44 annotation databases. We combined all annotations in 502 gene categories for yeast, and 996 categories for C. elegans (see Methods). For dataset 5, we used the seven categories of tissues annotated by Hsiao et al. 45. We matched each ICA cluster with every category and calculated the p value, that is, the chance probability of the observed intersection between the cluster and the category (see Methods for details). We ignored categories with p values greater than 10^-7. Assuming that there are at most 1,000 functional categories and roughly 500 ICA clusters, any p value larger than 1/(500 × 1,000) = 2 × 10^-6 is not significant.

We applied ICA on the following five expression datasets (Table 1): dataset 1, budding yeast during cell cycle and CLB2/CLN3 overactive strain 46, consisting of spotted array measurements of 4,579 genes in 22 experimental conditions; dataset 2, budding yeast during cell cycle 47 consisting of Affymetrix oligonucleotide array measurements of 6,616 genes in synchronized cell cultures at 17 time points; dataset 3, yeast in various stressful conditions 48 consisting of spotted array measurements of 6,152 genes in 173 experimental conditions that include temperature shocks, hyper- and hypoosmotic shocks, exposure to various agents such as peroxide, menadione, diamide, dithiothreitol, amino acid starvation, nitrogen source depletion and progression into stationary phase; dataset 4, C. elegans in various conditions 8 consisting of spotted array measurements of 11,917 genes in 179 experimental conditions and 17,817 genes in 374 experimental conditions that include growth conditions, developmental stages and a variety of mutants; and dataset 5, normal human tissue 45 consisting of Affymetrix oligonucleotide array measurements of 7,070 genes in 59 samples of 19 kinds of tissues. We used KNNimpute 49 to fill in missing values. For each dataset, first we decomposed the expression matrix into independent components using ICA, and then we performed clustering of genes based on the decomposition.

We evaluated the performance of ICA in finding components that result in gene clusters with biologically coherent annotations, and compared our results with the performance of other methods that were used to analyze the same datasets. In particular, we compared with the following methods: PCA, which Alter et al. 18 applied to the analysis of the yeast cell cycle data (dataset 1) and Misra et al. 19 applied to the analysis of human tissue data (dataset 5); k-means clustering, which Tavazoie et al. 10 applied to the yeast cell cycle data (dataset 2); the Plaid model 14 applied to the dataset of yeast cells under stressful conditions (dataset 3); and the topographical map-based method (topomap) that Kim et al. 8 applied to the C. elegans data (dataset 4). In all comparisons we applied the natural-gradient maximum-likelihood estimation (NMLE) ICA algorithm 2829 for linear ICA, and a kernel-based nonlinear BSS algorithm 34 for nonlinear ICA. The single parameter in our method was the coefficient C in Equation 3, with a default C = 7.5%.

Detailed results and gene lists for all the clusters that we obtained with our methods are provided in the web supplements in 50.

Alter et al. 18 introduced the use of PCA in microarray analysis. They decomposed a matrix X of N experiments × K genes into the product X = U Σ V^T of a N × L orthogonal matrix U, a diagonal matrix Σ, and a K × L orthogonal matrix V, where L = rank(X). The columns of U are called the eigengenes, and the columns of V are called the eigenarrays. Both eigenarrays and eigengenes are uncorrelated. Alter et al. 18 hypothesized that each eigengene represents a transcriptional regulator and the corresponding eigenarray represents the expression pattern in samples where the regulator is overactive or underactive.

ICA expresses X as a product X = AS (Equations 1 and 2), where S is an L × K matrix whose rows are statistically-independent profiles of gene expression. The main mathematical difference between ICA and PCA is that PCA finds L uncorrelated expression profiles, whereas ICA finds L statistically-independent expression profiles. Statistical independence is a stronger condition than uncorrelatedness. The two mathematical conditions are equivalent for Gaussian random variables, such as random noise, but different for non-Gaussian variables. We hypothesized that biological processes have highly non-Gaussian distributions, and therefore will be best separated by ICA. To test this hypothesis we compared ICA with PCA on datasets 1 and 5, which Alter et al. 18 and Misra et al. 19, respectively, analyzed with PCA.

Alter et al. 18 preprocessed dataset 1 with normalization and degenerate subspace rotation, and subsequently applied PCA to recover 22 eigengenes and 22 eigenarrays. The expression matrix they used consists of ratios x_ij = (R_ij/G_ij) between the red and green intensities. Since a logarithm transformation is the most commonly used method for variance normalization 51, we used data processed to contain log-ratios x_ij = log₂(R_ij/G_ij) between red and green intensities obtained from 52. We applied ICA to the microarray expression matrix X without any preprocessing, and found 22 independent components. We compared the biological coherence of 44 clusters consisting of genes with significantly high or low expression levels within the independent components, with clusters similarly obtained from the principal components of Alter et al. 18. (We used the most favorable clustering coefficient C for each of the principal components and independent components, see Methods); C was fixed to 17.5 for ICA but it was varied from five to 45 with an interval of 2.5 for PCA and the result for C = 37.5 (best) is illustrated in Figure 2a, while three of others are illustrated in Figure 2b. For each cluster, we calculated p values with every functional category from GO and KEGG, and retained functional categories with p value < 10^-7. This resulted in 13 functional categories covered only with PCA clusters, 27 only with ICA clusters, and 33 with both. Categories covered by either method but not both, typically had high p values (low significance). For functional categories detected by either ICA or PCA clusters, we made a scatter plot to compare the negative log of the best p values of each category (Figure 2a). In the majority of the functional categories ICA produced significantly lower p values than PCA did. For instance, among the functional categories with p value < 10^-7, ICA outperformed PCA in 28 out of 33 cases, with a median difference of 7.3 in -log₁₀ (p value) in the 33 cases. In Figure 2a, about a half of the functional categories (13 out of 28) represented around the diagonal or under the diagonal have close connection (parent or child) within the GO tree with another category for which ICA has much smaller p value than PCA. This means that if we look at a group of similar functional categories instead of a single category, most of the groups have considerably smaller p values with ICA than with PCA. We listed the five most significant ICA clusters based on the smallest p value of functional categories within the clusters in the web supplement 50. Cluster 13 is driven from the seventh independent component contained 915 genes that are annotated in KEGG, of which 96 are annotated as 'ribosome'-related (out of 111 total 'ribosome'-related genes in KEGG). The same cluster is highly enriched with genes annotated in GO as 'protein biosynthesis', 'structural constituent of ribosome' and 'cytosolic ribosome'. A plausible hypothesis is that the corresponding independent component represents the expression program of a biological mechanism related to protein synthesis.

We also applied ICA to another yeast cell cycle dataset using a different synchronization method produced by Spellman et al. 46 and to which PCA is applied by Alter et al. 18. For this dataset, ICA outperformed PCA in finding significant clusters (data shown in the web supplement 50).

We also applied nonlinear ICA to the same dataset. First, we mapped the input data from the 22-dimensional input space to a 30-dimensional feature space (see Methods). We found 30 independent components in the feature space and produced 60 clusters from these components. We compared the biological coherence of nonlinear ICA clusters to linear ICA clusters and to PCA clusters (Figure 2c,d). Overall, nonlinear ICA performed significantly better than the other methods. The five most significant clusters are shown in the web supplement 50. Similarly to linear ICA, the most significant nonlinear ICA cluster was enriched with genes annotated as 'protein biosynthesis', 'structural constituent of ribosome', 'cytosolic ribosome' and 'ribosome' with the smallest p value being 10^-61 for 'ribosome' compared to the p value of 10^-51 for the corresponding ICA cluster.

Misra et al. 19 applied PCA to dataset 5 of 7,070 genes in 19 kinds of human normal tissue (containing 59 microarray experiments) produced by Hsiao et al. 45 available at 53. The dataset they used contains 40 experiments; 19 additional microarray experiments have been performed subsequently by Hsiao et al. 45. After applying PCA and a filtering method, Misra et al. 19 obtained 425 genes upon which they reapplied PCA and plotted a scatter plot with loadings (expression levels) of these genes in the two most dominant principal components (eigenarrays). By visual inspection they observed three linear clusters on the resulting two-dimensional plot, enriched for liver-specific, brain-specific and muscle-specific genes, respectively (no p values were provided), as annotated by Hsiao et al. 45. We removed three experiments that made the expression matrix X to be nearly singular, and applied ICA on the remaining 56 experiments, resulting in 56 independent components. We generated 112 clusters using our default clustering parameter (C = 7.5%), and measured the enrichment of each of the seven tissue-specific categories annotated by Hsiao et al. 45 within each cluster. The three most significant independent components were enriched for liver-specific, muscle-specific and vulva-specific genes with p values of 10^-133, 10^-127 and 10^-101, respectively. The fourth most significant cluster was brain-specific (p value = 10^-86). In the ICA liver cluster, 214 genes were liver-specific (out of a total of 293), as compared with the 23 liver-specific genes identified by Misra et al. 19. The ICA muscle cluster of 258 genes contains 211 muscle-specific genes compared to 19 muscle-specific genes identified by Misra et al. 19. The ICA brain cluster consisting of 277 genes contains 258 brain-specific genes compared to 19 brain-specific genes identified by Misra et al. 19. We generated a three-dimensional scatter plot of the coefficients of all genes annotated by Hsiao et al. 45 on the three most significant ICA components (Figure 3). We observe that the liver-specific, muscle-specific and vulva-specific genes are strongly biased to lie on the x-, y- and z-axes of the plot, respectively.

We applied nonlinear ICA to this dataset (dataset 5) and the four most significant clusters from nonlinear ICA with Gaussian radial basis function (RBF) kernel were muscle-specific, liver-specific, vulva-specific and brain-specific with p values of 10^-157, 10^-125, 10^-112 and 10^-70, respectively.

Tavazoie et al. 10 applied k-means clustering to the yeast cell cycle data generated by Cho et al. 47 (dataset 2) and available at 54. First they excluded two experiments due to less efficient labeling of the mRNA during chip hybridization, and then selected 3,000 genes that exhibited the greatest variation across the 15 remaining experiments. They generated 30 clusters with k-means clustering, after normalizing the variance of the expression of each gene across the 15 experiments. We used the same expression dataset and normalized the variance in the same manner, but we did not remove the two problematic experiments. Instead, we removed one experiment that made the input matrix nearly singular, which destabilizes ICA algorithms. We obtained 16 independent components, and constructed 32 clusters with our default clustering parameter (C = 7.5%). We collected functional categories detected with a p value < 10^-7 by ICA clusters only (4), or k-means clusters only (16), or both (44). Categories covered by either method but not both typically had high p values. For functional categories detected by both ICA and k-means clusters, we made a scatter plot to compare the negative log of the best p values of the two approaches (Figure 4a). In the majority of the functional categories ICA produced significantly lower p values. Among the functional categories with p value < 10^-7 (or 10^-10), ICA outperformed k-means clustering in 30 out of 44 (27 out of 30) cases, with a median difference of 6.1 (8.9) in - log₁₀ (p value). The seven most significant clusters are shown in Table 2. In Figure 4a, several functional categories are represented around the diagonal. Some of them have close connections within the GO tree with other categories for which ICA has much smaller p-values than PCA. This means that if we look at a group of similar functional categories instead of a single category, most of the groups would have smaller p values with ICA than with PCA. When adjusting the parameter C in our method (Equation 3) from four to 14, we found similar results, with ICA still significantly outperforming k-means clustering (results shown in web supplement 50).

To understand whether ICA clusters typically outperform k-means clusters because of larger overlaps with the GO category, or because of fewer genes outside the GO category, we defined two quantities: True Positive (TP) and Sensitivity (SN). They are determined as: TP = k/n and SN = k/f, where k is the number of genes that are shared by the functional category, the cluster n is the number of genes within the cluster that are in any functional category and f is the number of genes within the functional category that appear in the microarray dataset. For all functional categories appeared in Figure 4a, we compared TP and SN of ICA clusters with those of k-means clusters in Figure 4b and 4c, respectively. From Figure 4b and 4c, we see that ICA-based clusters usually cover more of the functional category (more sensitive), while they are comparable with k-means clusters in the percentage of the cluster's genes contained in the functional category (equally specific). We also applied nonlinear ICA to the same dataset. We first mapped the input data from the 16-dimensional input space to a 20-dimensional feature space (see Methods), found 20 independent components in the feature space and produced 40 clusters from these components. Comparison of the biological coherence of nonlinear ICA clusters to ICA clusters and to k-means clusters (Figure 4d,e) showed that overall nonlinear ICA performed significantly better than the other methods. The seven most significant nonlinear ICA clusters are shown in our web supplement 50.

Lazzeroni and Owen 14 proposed the Plaid model for microarray analysis. The Plaid model takes the input expression data in the form of a matrix X_ij (where i ranges over N samples and j ranges over K genes). It linearly decomposes X into component matrices, namely layers, each containing non-zero values only for subsets of genes and samples in the input X that are considered to be member genes and samples of that layer. Genes that show a similar expression pattern through a set of samples, together with those samples, are assigned to be members of that layer. Each gene is assigned a load value representing the activity level of the gene in that layer. We downloaded the Plaid software from 55, and applied it to the analysis of yeast stress data of 6,152 genes in 173 experiments (dataset 3) obtained by Gasch et al. 48 available at 56. We imputed dataset 3 after eliminating 868 environmental stress response (ESR) genes defined by Gasch et al. 48 - because clustering of the ESR genes is trivial - and obtained 173 layers. To check the biological coherence of each layer, we grouped genes showing significant activity level in each layer into clusters. For each layer, we grouped the top C% of up-regulated/down-regulated genes into a cluster. The value of C was varied from 2.5 to 42.5 with an interval of five. The setting that maximized the average p value of the functional categories was C = 32.5, with p value of <10^-20. (We used the most favorable clustering coefficient C for the Plaid model, see Methods.)

We applied ICA to the dataset (5,284 genes, 173 experiments), after we had also eliminated the 868 ESR genes that are easy to cluster. We found 173 independent components, constructed 346 clusters by using our default clustering parameters (C = 7.5, in Equation 3), and performed the same p value comparison of statistical significance with the Plaid model (Figure 5). Figure 5a compared ICA with the Plaid model when C is the optimal value (C = 32.5), and Figure 5b compared ICA with the Plaid model with C from 2.5 to 45. In Figure 5a, when C = 32.5, in the 56 functional categories detected by both the Plaid model and ICA with p value <10^-7, the ICA clusters had smaller p values for 51 out of 56 functional categories. We list the five most significant clusters from our model in Table 3. In Table 3, clusters are characterized by functional categories related to various kinds of processes for synthesis of ATP (that is, energy metabolism), whereas clusters in Table 2 are characterized by biological events occurring during the cell cycle, most of which are catabolic processes consuming ATP. This result is consistent with the fact that the many cellular stresses induce ATP depletion, which induces a drop in the ATP:AMP ratio and leads to expression of genes associated with energy metabolism 5758. We also applied our approach to the dataset without removing ESR genes and the results were significantly better (see our webpage at 50).

Kim et al. 8 assembled a large and diverse dataset of 553 C. elegans microarray experiments produced by 30 laboratories (available at 59). This dataset contains experiments from many different conditions, as well as several experiments on mutant worms. Of the total, 179 of the experiments contain 11,917 gene measurements, while 374 of the experiments contain 17,817 gene measurements. Kim et al. 8 clustered the genes with a versatile topographical map (topomap) visualization approach that they developed for analyzing this dataset. Their approach resembles two-dimensional hierarchical clustering, and is designed to work well with large collections of highly diverse microarray measurements. Using their method, they found by visual inspection 44 clusters (the mounts) that show significant biological coherence.

The ICA method is sensitive to large amounts of missing values, while methods for imputing missing values are also not appropriate in such cases. We applied ICA to the 250 experiments that had missing values for < 7,000 out of the 17,661 genes, removed four experiments that make the expression matrix to be nearly singular, and generated 492 clusters by using our default parameters. In total, 333 GO and KEGG categories were detected by both ICA and topomap clusters with p values <10^-7 (Figure 6). Categories covered by either method, but not both, typically had high p values. We observe that the two methods perform very similarly, with most categories having roughly the same p value in the ICA and in the topomap clusters. The topomap clustering approach performs slightly better in a larger fraction of the categories. Still, we consider this performance a confirmation that ICA is a widely applicable method that requires minimal training, as in this case the missing values and high diversity of the data make clustering especially challenging.

We also carried out a comparison of the TP and SN quantities. For all functional categories that appeared in Figure 6a, we compared TP and SN of ICA clusters with those of topomap-driven clusters in Figure 6b and 6c, respectively. Again, typically, ICA clusters cover more genes from the functional category than the corresponding topomap clusters.

We tested six linear ICA methods: Natural Gradient Maximum Likelihood Estimation (NMLE) 2829; Joint Approximate Diagonalization of Eigenmatrices (JADE) 30; Fast Fixed Point ICA with three decorrelation and nonlinearity approaches (different measures of non-Gaussianity: FP, FPsym and FPsymth) 31; and Extended Information Maximization (ExtIM) 32. We also tested two variations of nonlinear ICA: Gaussian radial basis function (RBF) kernel (NICAgauss) and polynomial kernel (NICApoly). For each dataset, we compared the biological coherence of clusters generated by each method. Among the six linear ICA algorithms, NMLE performed well in all datasets. Among both linear and nonlinear methods, the Gaussian kernel nonlinear ICA method was the best in datasets 1 and 2, the polynomial kernel nonlinear ICA method was best in dataset 5. NMLE, FPsymth and ExtM were best in the large datasets, 3 and 4. In Figure 7, we compare the NMLE method with three other ICA methods. We show the remaining comparisons in our web supplement 50. Overall, the linear ICA algorithms consistently performed well in all datasets. The nonlinear ICA algorithms performed best in the small datasets, but were unstable in the two largest datasets.

The Extended Infomax ICA algorithm 32 can automatically determine whether the distribution of each source signal is super-Gaussian, with a sharp peak at the mean and long tails (such as the Laplace distribution), or sub-Gaussian, with a small peak at the mean and short tails (such as the uniform distribution). Interestingly, the application of Infomax ICA to all the expression datasets uncovered no source signal with sub-Gaussian distribution. A likely explanation is that the microarray expression datasets are mixtures of super-Gaussian sources rather than of sub-Gaussian sources. This finding is consistent with the following intuition: underlying biological processes are super-Gaussian, because they affect sharply the relevant genes, typically a fraction of all genes (long tails in the distribution), and leave the majority of genes relatively unaffected (sharp peak at the mean of the distribution).

There have been several empirical comparisons of ICA algorithms using various real datasets 2760. Even though many of the ICA algorithms have close theoretical connections, they often reveal different independent components in real world problems. The reasons for such discrepancies are usually deviations between the assumed ICA model and the underlying behavior of real data. Discrepancies can often result from noise or wrong estimation of the source distributions. Such factors affect the convergence of each ICA algorithm differently, and therefore it is useful to apply several different ICA algorithms 27. In our case, overall, the different ICA algorithms perform similarly. NMLE, ExtIM, and FPsymth algorithms yielded similar results except in dataset 2 where NMLE performed best. Interestingly, dataset 2 is the only one in this comparison where the data comes from oligonucleotide microarrays (Affymetrix), where the distribution is highly unbalanced and required application of variance normalization (see Methods).

The yeast cell-cycle dataset in 46 was preprocessed to contain log-ratios x_ij = log₂(R_ij/G_ij) between red and green intensities. ICA was applied on the 22 experiments with 4,579 genes that were analyzed by Alter et al. 18.

Variance normalization was applied to the yeast cell-cycle dataset in 47 for the 3,000 most variant genes (as in 10). The 17th experiment, which made the expression matrix close to singular, was removed.

The yeast stress dataset in 48 was preprocessed to contain log-ratios x_ij = log₂(R_ij/G_ij) between red and green intensities. As in Segal et al. 15, we eliminated 868 environmental stress response (ESR) genes defined by Gasch et al. 48 for which clustering is trivial and applied ICA to the remaining genes and experiments.

Experiments in the C. elegans dataset 8 that contained more than 7,000 missing values were discarded. The 250 remaining experiments were used, containing expression levels for 17,817 genes preprocessed to be log-ratios x_ij = log₂(R_ij/G_ij) between red and green intensities.

The expression levels of each gene in the human normal tissue 45 were normalized across the 59 experiments, and the logarithms of the resulting values were taken. Experiments 57, 58 and 59 were removed because they made the expression matrix nearly singular.

Denoting by x[i] the i^th column of X, the kernel method proposed by Harmeling et al. 34 constructs an L-dimensional feature space where a dot product of two mapped inputs Φ(x[i]) and Φ(x[j]) in the feature space is determined by a kernel function k(.,.) in the input space (Equation 11).

k (x[i], x[j]) = Φ(x[i])·Φ(x[j])     1 ≤ i, j ≤ K     (11)

Define L points v₁,..., v_L in the input space so that their images Φ_V = [Φ(v₁),...,Φ(v_L)] form a basis in the feature space. Denoting by Φ_X = [Φ(x[1]),...,Φ(x[K])], since Φ_V is a basis of a feature space ([∈R^L), span(Φ_V) = span(Φ_X) and rank(Φ_V) = L hold [34]. Then, orthonormal basis is defined as follows:

Here, multiplying (Φ_V^T Φ_V)^-1/2 with Φ_V leads to orthonormal matrix and it does not change the column space of Φ_V, that is, rank(Φ_V (Φ_V^T Φ_V)^-1/2) is still L. The L points v₁,..., v_L can be selected from x[1],...x[K] if they fulfill rank(Φ_V) ≈ L, which implies that Φ_V^T Φ_V is full rank 34. To determine the best basis of the feature space, we randomly sample L input vectors from x[1],..., x[K], obtain v₁,..., v_L, and calculate the conditional number of Φ_V^T Φ. We repeat this random sampling 1,000 times and choose the input vectors, v = {v₁, ..., v_L}, that results in the smallest conditional number, so that Φ_V^T Φ_V becomes close to full rank. From Equation 11, we can use a kernel function k(.,.) when calculating Φ_V^T Φ_V (Equation 13).

We map all the vectors x[1],..., x[K] to the feature space with Ξ as a basis. For an input vector x[k], the coordinates of mapped point Φ(x[i]), denoted by Ψ(x[i]), are calculated by a kernel function k(.,.).

An alternative way to find an orthonormal basis of the feature space is to use kernel principal component analysis 61. First, calculate L eigenvectors with Φ_X = [Φ(x[1]),...,Φ(x[K])] such that (Φ_X^T Φ_X/K) E_L = E_L Λ holds. Then, determine the basis in the feature space, as Ξ: = Φ_XE_L (K Λ)^-1/2 and map the input vectors x[1],..., x[K] into the feature space defined by these basis as:

We tried this alternative method, but the results were generally worse that with random sampling.

We linearly decompose the mapped data Ψ = [Ψ(x[1]),..., Ψ(x[K])] ∈R^{L × K} into statistically independent components using NMLE.

The requirements for a valid kernel function that specifies the feature space are described by Muller et al. 37. We choose a Gaussian radial basis function (RBF) kernel described as k(x,y) = exp(-|x - y|²) and a polynomial kernel of degree 2 described as k(x,y) = (x^Ty+1)². We refer to nonlinear ICA with Gaussian RBF kernel as NICAgauss, and with polynomial kernel as NICApoly.