Posts

Need to build and run a Docker image but don't have root access? No problem with Apptainer!

Image
My group is moving towards a working paradigm where each project has a Docker image. This helps keep control of the environments for each project, as you know that operating systems and programming languages like R and Python undergo regular upgrades, which are mostly undesirable for long running data analytics projects. For example in the field of biomedical genomics, some projects can take up to 10 years between initiation and final publication, so having a stable and portable computing environment is crucial. While this solution is fantastic if you have a workstation where you have free reign to use root/sudo permissions, many data analysts are restricted to using shared computing resources without root. There are some ways to mitigate these restrictions. One approach is to make a Docker image on another computer and use Singularity  or Udocker  to pull and convert it to be run by non-root user. But sometimes, users don't have access to another computer to build these images. Op

Weird multi-threaded behaviour of R/Bioconductor under Docker

Image
As I was running some R code under Docker code recently, I noticed that processes that should be single threaded, were using all available threads. And this behaviour was different between R on a native Linux machine as compared to Docker.  A search of the forums found that this is due to the configuraiton of the BLAS system dependancy on those Docker images, which is set to use all available threads for matrix operations. This configuration sounds like a good idea at first because dedicating more threads to a problem should speed up the execution. But realise that parallel processing incurs some overhead to coordinate the sub tasks and communicate the data to/from daughter threads. This means that you rarely achieve linear speedup the more threads you add. Typically what happens is that parallelisation has a sweetspot where the first 5-10 threads provide some speed-up, but beyond that there is either no improvement in speed or that adding additional threads actually makes the code sl

Stress test RAM annually

Image
TLDR: System memory can go bad. Use `memtester` on your Linux system annually to spot any problems early. Now the long story... In bioinformatics, we process a lot of data and conduct a lot of analysis. We use a range of devices from laptops to desktop workstations and remote servers and cloud. One particular desktop workstation of mine has been showing intermittent freezing and other problems, so I spent a bit of time trying to diagnose the issue. It is based on the AMD Threadripper 2990WX 32 Core CPU with 8x 16GB DDR4 modules, and we have been using it to process thousands of RNA sequencing datasets for the DEE2 project. It has been working at maximum capacity for about 6 years. Symptoms it had were sudden shutdowns and freezing. I checked the CPU temperatures (using the `stress` command) and it was high, in the 90 °C. range which was odd given it was water cooled with an all in one system. I removed the block to inspect the thermal paste, and I found that the block did not appear t

Pathway analysis - speed of FGSEA versus Mitch

Image
FGSEA is among the most used pathway enrichment tools due to its speed and straightforward design. While doing a comparison of different enrichment tools recently, I compared it against a tool called "mitch" and I noticed something interesting. Mitch was actually faster than fgsea. This is strange, because my own tests from 2020 published in BMC Bioinformatics showed that fgsea was >10x faster (fgsea version 1.11.1) (Fig 1). Fig 1. Tests conducted in 2020 show FGSEA is 10x faster than mitch. As emphasised in our 2020 paper, speed wasn't the main priority for our software. We were focused on enabling multi-dimensional analysis. But given this curious observation, I did a systematic test of speed of fgsea and mitch using a real dataset to try and understand whether fgsea underwent some code changes that made it slower. The codes for my work are on GitHub , and I used the bioconductor docker images  to repeat this across Bioconductor versions from 3-11 onwards. Before 3.1

Adding a Docker image to your R bioinformatics project to improve reproducibility

Image
I have gotten into the habit of making Docker images for each of my projects, which is helpful if these projects are large, and span over multiple years. Long-running projects are a headache for bioinformaticians because software like R has significant updates each year and we want to keep our machines running the most up to date software with recent bug fixes, features and new packages.  Docker images also help in making the research more reproducible for others who are interested in taking a deeper dive into the research materials. This helps auditability and could even help in improving research quality through better transparency. That said, still it is a bit tricky to achieve, so I thought I would write walk through on how I do it with an R workflow. Step 0: install docker on a linux system On Ubuntu you can use: sudo apt install docker.io Other OSs will vary. Step 1: Write your R Markdown script You probably have a workflow that you've been working on which works for your cur

The power of enrichment analysis for epigenetics: Application to schizophrenia

Image
As regular readers of GenomeSpot will know, we have adopted the mitch package to enable enrichment analysis of methylation array data, as shown in our recent pre-print . You might be thinking that enrichment analysis of methylation data has existed for a long time, so why do we need a new approach? Well the answer to that is the current methods suffer one of the critical problems: (1) they use over-representation analysis (ORA), which has very low sensitivity, or (2) they do not distinguish between up- and down- directions of methylation change.  (1) is a problem because ORA relies on binary classification of probes or genes into differential or not, whereas in reality probes and genes are spread on a distribution of differential-ness. This makes ORA easy to compute, but leads to low sensitivity. In serious cases, the (arbitrary) threshold used for selection of differential probes means that very few probes are present in the foreground. (2) is a problem because gene and pathways are t

Yes, mitch can be used for pathway analysis of Methylation array data

Image
In 2020, Dr Antony Kaspi and I published a method called "mitch" [1] which is like GSEA, but was specifically designed for multi-contrast analysis, and based on rank-MANOVA statistics inspired by a 2012 paper by Cox and Mann. Mitch worked well for various types of omics data downstream of commonly used differential abundance tools like DESeq2, edgeR, DiffBind, etc, but we didn't consider at the time how mitch could be applied to microarray data.  You might think that microarrays are outdated, but they are still used extensively for epigenome-wide association studies (EWASs), which are frequently used to understand disease processes and to identify biomarkers of disease. To demonstrate, there are 1536 publicly available methylation array studies on NCBI GEO, and probably many more thousands that are restricted access. The tools available for pathway analysis of methylation array data are a bit limited. There's an over-representation method that take into consideration