Update on DEE2 project for Jan 2018

Today I'd like to share some updates on the DEE2 project, which I wrote about  in an earlier post. The  project code can be viewed on github here.

Pipeline and images

As the pipeline was recently finalised, I was able to roll out the working docker image. To facilitate users without root access, this image was ported to singularity. This took a lot of effort and some expertise from our local HPC team to get things working (many thanks to the Massive/M3 team). The singularity image is available from the webserver (link) and instructions for running it are available on github here. I have started testing a heavyweight singularity image, which includes the genome indexes, which will be more efficient for running jobs with large genomes and will make it available once testing is complete.

Queue management

It may sound simple to write a script to determine which datasets have been completed and add new datasets to the queue but when taking about tens of thousands of datasets it's quite a challenge. Several updates have recently gone into the R script that checks incoming data and manages the queue. Initially I wanted this script to also perform matrix construction, but I am finding R to be too slow an memory hungry for this purpose so I'm reverting to python or shell tools to do that. The queue script will be run every 6 to 12 hours, updating the queue on the webserver. Queue management on the webserver is also updated, so that the number of times a certain dataset is allocated is recorded, so that allocations are spread more evenly. This should result in more efficient allocations.

Data throughput

Since the start of the year, I've focused compute power on getting 95% of the Arabidopsis datasets completed which I think we've now achieved. These datasets were processed by our Nectar cloud servers as well as a home server. On the 26th Jan, we got the singularity image working on the M3 cluster, focusing on D. rerio with 4-6 individual servers, as you can see in the graph below, it is progressing rapidly. 


But we are FAR from finished, as you can see from the size of the queue below, mouse and human datasets dwarf the others. Which is why we're investigating all avenues to add processing power. This will include spot instances on AWS and preemptive Google cloud, and there will be step-by-step guides for new users who want to use cloud computing to process their RNA-seq data. We are also keen to gain an allocation at the NCI supercomputer. 

Data provision and webserver

I mentioned that the way the bulk matrices are generated is changing, indeed I found this to be a bottleneck and unfeasible for more than a few thousand datasets. Using the current R script, matrix generation of the Arabidopsis data crashed the 128GB server in multi-core mode but took >3 days in single core mode so for the moment it will be done using unix tools like split and paste (about 7 minutes with my prototype script). Matrix generation will be run daily but might have to be less frequent as more data is processed. The current matrices can be found at the bulk data page here.

Currently the search function is not operation so the data is only available from the bulk data page. I will be working on developing the user interface over the next 2 months, which I think will resemble DEE v1 but with some modifications. Firstly I will have another crack at implementing a search engine to identify datasets of interest from the metadata. And secondly I will be making cosmetic changes to the site to make it more attractive. Currently the webserver is (here) but will be mapped to a new domain once the data processing is near completion. 

How you can use DEE2 in your work

There are a number of ways you can use DEE2 already. You can download and play with the bulk data matrices located here. If SRA/GEO datasets of interest are missing - as is likely the case at this early stage in the data processing phase, then you can use the docker image on your server or spin up a cloud server and follow the steps outlined in the README.md. One of the nicest features of the image is that it can process your own fastq files. This is perfect when comparing your own RNA-seq data in fastq format to some public data. Finally the singularity image will be of use for users without sudo or root access, for instance HPC users. 

We fully welcome contributions to compute time so if you have an idle machine consider leaving it to process data using this command, replacing <species> with your favourite model organism: 

nohup docker run mziemann/tallyup <species> & tailf nohup.out

or

nohup bash -c 'while true ; do docker run mziemann/tallyup <species> & tailf nohup.out ; done'

Also contributions to code base are welcome as are your suggestions and comments below.
Location: Melbourne VIC, Australia

Popular posts from this blog

Data analysis step 8: Pathway analysis with GSEA

Image
In our RNA-seq series so far we've performed differential analysis and generated some pretty graphs, showing thousands of differentially expressed genes after azacitidine treatment. In order to understand the biology underlying the differential gene expression profile, we need to perform pathway analysis. We use Gene Set Enrichment Analysis ( GSEA ) because it can detect pathway changes more sensitively and robustly than some methods. A 2013 paper compared a bunch of gene set analyses software with microarrays and is worth a look. Generate a rank file The rank file is a list of detected genes and a rank metric score. At the top of the list are genes with the "strongest" up-regulation, at the bottom of the list are the genes with the "strongest" down-regulation and the genes not changing are in the middle. The metric score I like to use is the sign of the fold change multiplied by the inverse of the p-value, although there may be better methods out there
Read more

Two subtle problems with over-representation analysis

Image
Over-representation analysis is a helpful and really frequently used technique for understanding trends from omics data and gene lists more generally. Just to demonstrate this, I tabulated the number of citations of some of the most popular web tools and packages, which reaches a massive 191k citations! But if you have used some of these tools, you will notice that they yield subtly different results. We were curiuous about that and ran a whole bunch of investigations into the internal workings of these tools, in particular clusterProfiler. We identified two subtle problems with clusterProfiler, which we unpack in detail in our new publication , but here I will give you a quick overview. Problem #1: The background problem The first one we call the “background problem,” because it involves the software eliminating large numbers of genes from the background list if they are not annotated as belonging to any category. This results in removing a large number of unannotated genes from the b
Read more

Uploading data to GEO - which method is faster?

If you have had to upload omics data to GEO before, you'll know it's a bit of a hassle and takes a long time. There are a few methods suggested by the GEO team if you are using the Unix command line: Using 'ncftp' ncftp set passive on set so-bufsize 33554432 open  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/your @mail.com_ yourfolder put -R Folder_with_submission_files Using 'lftp' lftp  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/ your @mail.com _ yourfolder mirror -R Folder_with_submission_files Using 'sftp' (expect slower transfer speeds since this method encrypts on-the-fly) sftp  geoftp @s ftp-private.ncbi.nlm.nih.gov password: yourpasscode cd uploads/ your @mail.com _ yourfolder mkdir new_geo_submission cd new_geo_submission put file_name Using 'ncftpput' (transfers from the command-line without entering an interactive shell) Usage example: ncftpput -F -R -z -u  geoftp  -p "yourpasscode"
Read more