SRA toolkit tips and workarounds

The Short Read Archive (SRA) is the main repository for next generation sequencing (NGS) raw data. Considering the sheer rate at which NGS is generated (and accelerating), the team at NCBI should be congratulated for providing this service to the scientific community. Take a look at the growth of SRA:
Growth of SRA data (http://www.ncbi.nlm.nih.gov/Traces/sra/i/g.png)

SRA however doesn't provide directly the fastq files that we commonly work with, they prefer the .sra archive that require specialised software (sra-toolkit) to extract. Sra-toolkit has been described as buggy and painful; and I've had my frustrations with it. In this post, I'll share some of my best tips sra-toolkit tips that I've found.

Get the right version of the software and configure it

When downloading, make sure you download the newest version from the NCBI website (link). Don't download it from GitHub or from Ubuntu software centre (or apt-get), as it will probably be an older version. In the binary directory (looks like /path/to/sratoolkit.2.4.3-ubuntu64/bin) there will be a file called sratoolkit.jar. In linux use "java -jar sratoolkit" to open the graphical interface. in the preferences menu, enable the local repository and select a path for it. By doing this, you can then use sra-toolkit to "stream" fastq data (see below).

EDIT: if you are seeing an error like this one:

/data/app/sratoolkit.2.4.3-ubuntu64/bin/fastq-dump --split-files -A ERR366438

2015-02-15T21:47:01 fastq-dump.2.4.3 err: binary large object corrupt while reading binary large object within virtual database module - failed ERR366438

=============================================================
An error occurred during processing.
A report was generated into the file '/data/home/ncbi_error_report.xml'.
If the problem persists, you may consider sending the file
to 'sra@ncbi.nlm.nih.gov' for assistance.
=============================================================
Then grab the new sra-toolkit version 2.4.4 which seems to fix problems with SRA archives using reference based compression (when submitters provide data in aligned bam format).


Try streaming the data

You can convert sra to fastq on-the-fly by doing either of the following:

fastq-dump -A SRR1722641 -O SRR1722641.fastq

 fastq-dump -A SRR900186 -Z > SRR900186.fastq

Streaming paired-end data could be problematic. Use the following to save forward and reverse reads as separate files. Thanks to the folks at Biostars for this idea.

SRR=SRR1041311 ; fastq-dump -X 10 --split-files -I -Z $SRR \
| tee >(grep '@.*\.1\s' -A3 --no-group-separator \
> ${SRR}_1.fastq) >(grep '@.*\.2\s' -A3 --no-group-separator \
> ${SRR}_2.fastq) >/dev/null

Use download accelerator

The SRA team actually recommend using Aspera connect to speed up the download of SRA files. If the stream isn't working for you, give Aspera a try using this script. If you struggle to get Aspera configured, you can try a download accelerator such as axel or aria2c. Here's an example with axel.

axel -n5 ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByExp/sra/SRX/SRX709/SRX709649/SRR1585277/SRR1585277.sra

After downloading the SRA archive, dump the fastq:

fastq-dump -A SRR900186.sra -Z --split-files

Via the browser

Here are two useful approaches suggested by SeqAnswers. You can download each fastq.gz file individually from your web-browser (not command line interface) replacing the digits after SSR in this link:

http://www.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?cmd=dload&run_list=SRR515925&format=fastq

or batch download with a link like:

http://www.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?cmd=dload&run_list=SRR294514,SRR352727,SRR364895&format=fastq

Alternatively find your study accession number (ie. SRP013698) and go to the SRA run selector:

http://trace.ncbi.nlm.nih.gov/Traces/study/?go=home

Search with your SRP number, then click on the "Run" link. Click on the "Reads" tab, then click "Filtered Download", change the format to "FASTQ" and hit "Download".

SRA mirrors

Most of the data on SRA is mirrorred at ENA or DNAnexus.

You can download the compressed fastq files from ENA for forward and reverse reads

ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR504/SRR504687/SRR504687_1.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR504/SRR504687/SRR504687_2.fastq.gz

You can download the SRA archive from DNAnexus too.

ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR504/SRR504687/SRR504687.sra

Stream directly into your analysis pipeline

You can send the data straight through your QC and alignment pipeline without saving intermediate files. Here is an example using SRA toolkit for Olego alignment:

fastq-dump -A SRR764858 -Z \
| fastq_quality_trimmer -l 25 -t 20 -Q33 \
| olego -t 8  Athaliana.TAIR10.23.dna_sm.genome.fa - \
| samtools view -uSh - \
| samtools sort - SRR764858_sra.sort

And another using curl from EBI:

curl ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR764/SRR764858/SRR764858.fastq.gz \
| pigz -d | fastq_quality_trimmer -t 20 -l 25 -Q33 \
| olego -t 8 Athaliana.TAIR10.23.dna_sm.genome.fa - \
| samtools view -uSh - \
| samtools sort - SRR764858_ebi.sort

Dump color-space sequence

Occasionally you'll come across data in color-space format. After downloading the SRA archive do the following.

abi-dump -A SRR1657115.sra

That will dump the sequence in fasta format (SRR1657115.sra.csfasta) and the quality string (SRR1657115.sra.qual) in separate files. Then I use solid-trimmer.py to do quality trimming. Here's an example:

solid-trimmer.py -c SRR1657115.sra.csfast -q SRR1657115.sra.qual --moving-average 7:12 --min-read-length 25 > SRR1657115.fasta
Happy data mining!

Location: Saint Kilda VIC 3182, Australia

Popular posts from this blog

Data analysis step 8: Pathway analysis with GSEA

Image
In our RNA-seq series so far we've performed differential analysis and generated some pretty graphs, showing thousands of differentially expressed genes after azacitidine treatment. In order to understand the biology underlying the differential gene expression profile, we need to perform pathway analysis. We use Gene Set Enrichment Analysis ( GSEA ) because it can detect pathway changes more sensitively and robustly than some methods. A 2013 paper compared a bunch of gene set analyses software with microarrays and is worth a look. Generate a rank file The rank file is a list of detected genes and a rank metric score. At the top of the list are genes with the "strongest" up-regulation, at the bottom of the list are the genes with the "strongest" down-regulation and the genes not changing are in the middle. The metric score I like to use is the sign of the fold change multiplied by the inverse of the p-value, although there may be better methods out there
Read more

Two subtle problems with over-representation analysis

Image
Over-representation analysis is a helpful and really frequently used technique for understanding trends from omics data and gene lists more generally. Just to demonstrate this, I tabulated the number of citations of some of the most popular web tools and packages, which reaches a massive 191k citations! But if you have used some of these tools, you will notice that they yield subtly different results. We were curiuous about that and ran a whole bunch of investigations into the internal workings of these tools, in particular clusterProfiler. We identified two subtle problems with clusterProfiler, which we unpack in detail in our new publication , but here I will give you a quick overview. Problem #1: The background problem The first one we call the “background problem,” because it involves the software eliminating large numbers of genes from the background list if they are not annotated as belonging to any category. This results in removing a large number of unannotated genes from the b
Read more

Uploading data to GEO - which method is faster?

If you have had to upload omics data to GEO before, you'll know it's a bit of a hassle and takes a long time. There are a few methods suggested by the GEO team if you are using the Unix command line: Using 'ncftp' ncftp set passive on set so-bufsize 33554432 open  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/your @mail.com_ yourfolder put -R Folder_with_submission_files Using 'lftp' lftp  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/ your @mail.com _ yourfolder mirror -R Folder_with_submission_files Using 'sftp' (expect slower transfer speeds since this method encrypts on-the-fly) sftp  geoftp @s ftp-private.ncbi.nlm.nih.gov password: yourpasscode cd uploads/ your @mail.com _ yourfolder mkdir new_geo_submission cd new_geo_submission put file_name Using 'ncftpput' (transfers from the command-line without entering an interactive shell) Usage example: ncftpput -F -R -z -u  geoftp  -p "yourpasscode"
Read more