Posts

Mitch gets upgraded - now with gene set networks

Image
Interpreting pathway enrichment analysis results is a big challenge. There may be hundreds of statistically significant pathways from an analysis and getting to a shortlist of key mechanisms to follow up with experiments and describe in a publication is difficult. I recently got a request from a collaborator to come up with a way to visualise the key networks. I groaned... because network analysis in bioinformatics is sometimes characterised by showing hairballs of hundreds/thousands of meaningless interactions. Mostly it is done poorly and the charts themselves do not have any explanatory function and mostly appear to be decorative. After seeing a few well done examples such as Figure 2 from Chappel et al (PMID: 32138627), I thought this could be something we include as a common step in the mitch workflow. For those of you unaware, mitch is the R/bioconductor package that Dr Antony Kaspi and I published in 2020 (PMID: 32600408) with the main focus being on multi-dimensional enrichmen...
Read more

Installing pip based tools for bioinformatics

Loads of bioinformatics and data science tools are available through the `pip` package manager/installer. In days gone by, it was possible to type somethinig like pip install mytool --user to get a tool installed, but due to changes this isn't possible anymore. When you try a command like that on Ubuntu 24, you will get an error and suggestion to use a virtual environment (or "venv"). While there is a lot of in depth documentation on using venv, I couldn't find a quick start guide specifically foir bioinformaticians. Quickly, a virtual environment is an isolated environment which should help by stabilising the base python version and the package version, so there isn't a chance of problems from version mismatch. So here is a quick guide to install the MultiQC tool, used extensively in genome sequence analysis. Install multiqc via pip The first step is to create a folder to put your venvs (you will likely have several if you do a lot of analysis). mkdir ~/venv Next...
Read more

Bioinformatics data processing power depends on CPU L3 cache A LOT!

When speccing a CPU for a new bioinformatics computer we tend to focus on threads and frequency, but do you look at the cache? Well you definitely should. On January 30, I started jobs on two servers. The first one has 2x Xeon E5-2680 v3 (48 threads total) and the second has 1x Xeon E5-2667 v3 (16 threads). The work we are doing is processing raw RNA-seq data using a pipeline of Skewer+STAR+Kallisto. Server 1 has many more threads, so I ran two parallel jobs of 12 threads each, while on Server 2 I'm running one job using 8 cores. The cluster nodes are attached to the same network storage, so I/O capacity is the same. Since Jan 30, these two machines have collectively processed 4670 datasets and the difference between these two machines was a big surprise. Not only did Server 2 process more datasets, when normalised by number of threads, Server 2 processed 4.1 times more datasets!  Server 1 Server 2 CPU 2x Xeon E5-2680 v3 1x Xeon E5-2667 v3 Cores 24 16 Threads 48 16 Frequency ...
Location: Melbourne VIC, Australia
Read more

DEE2 2025 updates: growth and development

Image
DEE2 is a database service I co-founded in 2015 aiming to provide uniformly processed gene expression profiles for each and every RNA-seq dataset in NCBI's Sequence Read Archive. After some major revisions, we published the database/service journal article in 2019. Over time, DEE2 has grown dramatically, with the number of datasets and total number of counted reads, as seen in the tables below. Here, I'll walk you  through the  growth of the database over time and the new features we've added. Growth of DEE2 DEE2  continues to ingest new metadata from NCBI SRA and GEO, and the growth of this metadata set has caused us a lot of issues over time. In the early years of DEE2 we used SRAdb, and then it became too large for its design, so we sought different solutions. Currently we are using pySRA and only fetching quarterly. The size of each request has been a challenge, with human and mouse requests of annual  dumps exceeding the 64 GB RAM of our backend workstation! S...
Location: Melbourne VIC, Australia
Read more

Two subtle problems with over-representation analysis

Image
Over-representation analysis is a helpful and really frequently used technique for understanding trends from omics data and gene lists more generally. Just to demonstrate this, I tabulated the number of citations of some of the most popular web tools and packages, which reaches a massive 191k citations! But if you have used some of these tools, you will notice that they yield subtly different results. We were curiuous about that and ran a whole bunch of investigations into the internal workings of these tools, in particular clusterProfiler. We identified two subtle problems with clusterProfiler, which we unpack in detail in our new publication , but here I will give you a quick overview. Problem #1: The background problem The first one we call the “background problem,” because it involves the software eliminating large numbers of genes from the background list if they are not annotated as belonging to any category. This results in removing a large number of unannotated genes from the b...
Location: Melbourne VIC, Australia
Read more

Screen for mycoplasma contamination in DNA-seq and RNA-seq data: Part 2

Image
If you're culturing cells, avoiding contamination is an extremely important priority. Mycoplasma are common contaminant that can change the behaviour of cells and don't cause any obvious morphological changes, so routine testing is key. As bioinformaticians we should be incorporating myco screening into all our workflows, but the available tools haven't been that great. Even the shell script I wrote in a previous blog post isn't that great. It suffers from a few problems in miscounting reads, usability and leaving many files in the working directory. So it is high time for a refresh. With this refresh, the idea was to make a command line tool that folks could easily incorporate into their workflows, would process single- and paired-end data, be quick and be concise in its output. The result is `contam`. To give you an idea of what it can do, here is the help page: contam is a script for the quantification of contaminant sequences in Illumina short read sequence data i...
Location: Melbourne VIC, Australia
Read more

Need to build and run a Docker image but don't have root access? No problem with Apptainer!

Image
My group is moving towards a working paradigm where each project has a Docker image. This helps keep control of the environments for each project, as you know that operating systems and programming languages like R and Python undergo regular upgrades, which are mostly undesirable for long running data analytics projects. For example in the field of biomedical genomics, some projects can take up to 10 years between initiation and final publication, so having a stable and portable computing environment is crucial. While this solution is fantastic if you have a workstation where you have free reign to use root/sudo permissions, many data analysts are restricted to using shared computing resources without root. There are some ways to mitigate these restrictions. One approach is to make a Docker image on another computer and use Singularity  or Udocker  to pull and convert it to be run by non-root user. But sometimes, users don't have access to another computer to build these image...
Location: Melbourne VIC, Australia
Read more