DEE2 project update Jan 2020

Happy New Year! Here I'll go through a few updates on the DEE2 project. In case you don't know about it you can read the paper here.

1) Data processing

One of the most obvious updates since the paper came out was that the number of datasets hosted by DEE2 has increased substantially from 581,094 to 805,385 runs. This was achieved by using a combination of minicluster at Deakin Uni, several Nectar Coud servers and the Massive HPC located at Monash Uni. The queue is still growing, so I'll need to further expand compute resources in future.

2) Getting a new source of metadata

DEE2 was reliant on the SRADBV2 package for SRA metadata. This was really critical to provide a list of accession numbers for the database to process as well as sample information so that end users can search for data sets of interest. Unfortunately, SRADBV2 became deprecated only a few months after the paper was published. This was possibly due to the expanding size of the SRA metadata making it infeasible to be distributed as a single file. OmicIDXR is slated to be a replacement source of metadata, however I have doubts to its sustainability over the next few years. Therefore I have taken the approach to obtain metadata directly from the SRA website through the Run Selector app. This has the drawback that it needs to be done "manually" and has a batch limit of about 40,000 runs but as it is based on the SRA website, it is likely to be a better long term solution. It is also relatively straightforward to download rmetadata for recently added studies, for example every month. After incorporating new metadata from SRA last week, the queue has grown substantially.

3) The problem of studies missing runs

In the next few weeks I will be working to improve some of the backend scripts. One of the drawbacks of DEE2 at the moment is the fact that for some of the projects, some datasets/runs are missing. I will be working toward ensuring that only wholly completed studies are added to the dataset.

4) Increasing queue efficiency

Related to that is the proposed changes to data processing queue management. The queue should prioritise these missing datasets. As the webserver relays the data sets from nodes to the central data server, it can remove those datasets from the queue which have already passed basic checks.

5) Making the R package more efficient

There are also some improvements that can be made to the R interface, the getDEE2 package. In particular, the handling of the metadata can be improved by only downloading one time. Perhaps searching metadata can be made better with a search engine type function. I'm also considering adding the package to Bioconductor.

6) Website improvements

Other improvements on the horizon include facilitating quicker metadata searches with the DEE2 webpage with elastic search or similar serch engine. I have tried this before but could not pull it off. If you have any good resources for how to implement search engine for a tsv file please share in the comments.

7) Call for compute resources

Finally, if you are a frequent DEE2 user or have used the bulk datasets already or would like to contribute to the project, we are calling for contributors to volunteer their server or HPC compute resources. There is a docker based pipeline for servers/cloud and singularity for HPC. Docker nodes can be initiated using the command:
docker run mziemann/tallyup hsapiens
Substituting you favourite organism. If you would like to process a specific dataset you can provide the SRA run accessions like this
docker run mziemann/tallyup hsapiens SRR123,SRR124,SRR125
You will be able to access the data immediately from the docker container, as well as contributing them to the public database.
Although I will be applying for grants to expand DEE2, your contribution will help a lot.

Popular posts from this blog

Data analysis step 8: Pathway analysis with GSEA

Image
In our RNA-seq series so far we've performed differential analysis and generated some pretty graphs, showing thousands of differentially expressed genes after azacitidine treatment. In order to understand the biology underlying the differential gene expression profile, we need to perform pathway analysis. We use Gene Set Enrichment Analysis ( GSEA ) because it can detect pathway changes more sensitively and robustly than some methods. A 2013 paper compared a bunch of gene set analyses software with microarrays and is worth a look. Generate a rank file The rank file is a list of detected genes and a rank metric score. At the top of the list are genes with the "strongest" up-regulation, at the bottom of the list are the genes with the "strongest" down-regulation and the genes not changing are in the middle. The metric score I like to use is the sign of the fold change multiplied by the inverse of the p-value, although there may be better methods out there
Read more

Two subtle problems with over-representation analysis

Image
Over-representation analysis is a helpful and really frequently used technique for understanding trends from omics data and gene lists more generally. Just to demonstrate this, I tabulated the number of citations of some of the most popular web tools and packages, which reaches a massive 191k citations! But if you have used some of these tools, you will notice that they yield subtly different results. We were curiuous about that and ran a whole bunch of investigations into the internal workings of these tools, in particular clusterProfiler. We identified two subtle problems with clusterProfiler, which we unpack in detail in our new publication , but here I will give you a quick overview. Problem #1: The background problem The first one we call the “background problem,” because it involves the software eliminating large numbers of genes from the background list if they are not annotated as belonging to any category. This results in removing a large number of unannotated genes from the b
Read more

Uploading data to GEO - which method is faster?

If you have had to upload omics data to GEO before, you'll know it's a bit of a hassle and takes a long time. There are a few methods suggested by the GEO team if you are using the Unix command line: Using 'ncftp' ncftp set passive on set so-bufsize 33554432 open  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/your @mail.com_ yourfolder put -R Folder_with_submission_files Using 'lftp' lftp  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/ your @mail.com _ yourfolder mirror -R Folder_with_submission_files Using 'sftp' (expect slower transfer speeds since this method encrypts on-the-fly) sftp  geoftp @s ftp-private.ncbi.nlm.nih.gov password: yourpasscode cd uploads/ your @mail.com _ yourfolder mkdir new_geo_submission cd new_geo_submission put file_name Using 'ncftpput' (transfers from the command-line without entering an interactive shell) Usage example: ncftpput -F -R -z -u  geoftp  -p "yourpasscode"
Read more