Urgent need for minimum standards for reproducible functional enrichment analysis

So our preprint is online called “Guidelines for reliable and reproducible functional enrichment analysis” so I thought I’d give you an overview.

Enrichment analysis is widely used for exploration and interpretation of omics data, but I’ve noticed sloppy work is becoming more common. Over the past few years I’ve been asked to review several manuscripts where the enrichment analysis was poorly conducted and reported. Examples of this include lack of FDR control, incorrect background gene list specification and lack of essential methodological details. In the article we show how common these problems are and what the impact could be on the results.


In this article we did two screens of articles from PubMed Central. Firstly selected 200 articles with terms in the abstract related to enrichment analysis. These were examined with regards to a checklist around reporting and methodological issues, and were cross-checked for accuracy. As some articles describe >1 analysis, the total number of analyses was 235.




They did a variety of different types of omics analysis, but the most common were gene expression analysis




we show that these methodological problems are very common. For example out of 235 analyses in our sample, only 119 (53%) specified conducting p-value correction for multiple testing! 





In terms of tools, over-representation analysis including DAVID, Ingenuity and PANTHER were the most popular, followed by GSEA





Over-representation relies on an accurate background gene list, but we found this was correctly described in only 8/197 cases (4.1%)!!





There were 15 analyses that lacked even the most basic methodological detail, like which tool was used to calculate enrichment. The excerpt below is an example basically saying that KEGG pathway analysis was conducted but not stating how it was done. The article linked is the original KEGG paper from 2000 (PMID: 10592173) which doesn't describe any statistical enrichment analysis.




After examining another 1395 articles, we looked for trends between rigour/reporting and publication metrics. There wasn’t any association between out analysis scores and metrics like Scimago Journal Rank and number of citations accrued. This shows this isn’t a problem only with lower ranked journals. 




This is a massive wake-up call that serious methodological issues are widespread with enrichment analysis.


We also conducted some benchmarking analysis to understand how much these errors can impact results. We began by using differential RNA-seq expression data, we show that when used properly, over-representation (ORA) and GSEA like methods (functional class scoring; FCS) give similar results. In this example we used the clusterProfiler package for ORA and mitch package for FCS; more details in the article.




When an inappropriate background gene list is used in this case the whole genome (instead of only the genes detected in the assay), the results are vastly different! Likely these extra sets are false positives.



We also observed that most over-representation analyses performed enrichment of up and down genes combined into one list. Our own testing shows that this approach has much lower power to discover differentially regulated gene sets as compared to analysing up and down separately. In fact the number of differentially regulated gene sets was down by ~90% using the combined approach.



To combat these problems we provide a checklist of minimum standards for enrichment analysis.



The manuscript also contains some recommendations if you want to take your analysis from these minimum standards up to gold standard.


Hopefully this is of use to you. Happy to receive your feedback


Big shout out to Sameer Jadaan, Kaushalya Perera and Tanuveer Kaur for massive help in screening these articles. Also big thanks to Nick Wong, Antony Kasi and Anup Shah for comments on the draft.


If you are interested in exploring the data collected and code used, our R scripts are deposited here: https://github.com/markziemann/SurveyEnrichmentMethods 


Popular posts from this blog

Data analysis step 8: Pathway analysis with GSEA

Image
In our RNA-seq series so far we've performed differential analysis and generated some pretty graphs, showing thousands of differentially expressed genes after azacitidine treatment. In order to understand the biology underlying the differential gene expression profile, we need to perform pathway analysis. We use Gene Set Enrichment Analysis ( GSEA ) because it can detect pathway changes more sensitively and robustly than some methods. A 2013 paper compared a bunch of gene set analyses software with microarrays and is worth a look. Generate a rank file The rank file is a list of detected genes and a rank metric score. At the top of the list are genes with the "strongest" up-regulation, at the bottom of the list are the genes with the "strongest" down-regulation and the genes not changing are in the middle. The metric score I like to use is the sign of the fold change multiplied by the inverse of the p-value, although there may be better methods out there
Read more

Two subtle problems with over-representation analysis

Image
Over-representation analysis is a helpful and really frequently used technique for understanding trends from omics data and gene lists more generally. Just to demonstrate this, I tabulated the number of citations of some of the most popular web tools and packages, which reaches a massive 191k citations! But if you have used some of these tools, you will notice that they yield subtly different results. We were curiuous about that and ran a whole bunch of investigations into the internal workings of these tools, in particular clusterProfiler. We identified two subtle problems with clusterProfiler, which we unpack in detail in our new publication , but here I will give you a quick overview. Problem #1: The background problem The first one we call the “background problem,” because it involves the software eliminating large numbers of genes from the background list if they are not annotated as belonging to any category. This results in removing a large number of unannotated genes from the b
Read more

Uploading data to GEO - which method is faster?

If you have had to upload omics data to GEO before, you'll know it's a bit of a hassle and takes a long time. There are a few methods suggested by the GEO team if you are using the Unix command line: Using 'ncftp' ncftp set passive on set so-bufsize 33554432 open  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/your @mail.com_ yourfolder put -R Folder_with_submission_files Using 'lftp' lftp  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/ your @mail.com _ yourfolder mirror -R Folder_with_submission_files Using 'sftp' (expect slower transfer speeds since this method encrypts on-the-fly) sftp  geoftp @s ftp-private.ncbi.nlm.nih.gov password: yourpasscode cd uploads/ your @mail.com _ yourfolder mkdir new_geo_submission cd new_geo_submission put file_name Using 'ncftpput' (transfers from the command-line without entering an interactive shell) Usage example: ncftpput -F -R -z -u  geoftp  -p "yourpasscode"
Read more