Stress test RAM annually

TLDR: System memory can go bad. Use `memtester` on your Linux system annually to spot any problems early.

Now the long story...

In bioinformatics, we process a lot of data and conduct a lot of analysis. We use a range of devices from laptops to desktop workstations and remote servers and cloud. One particular desktop workstation of mine has been showing intermittent freezing and other problems, so I spent a bit of time trying to diagnose the issue. It is based on the AMD Threadripper 2990WX 32 Core CPU with 8x 16GB DDR4 modules, and we have been using it to process thousands of RNA sequencing datasets for the DEE2 project. It has been working at maximum capacity for about 6 years. Symptoms it had were sudden shutdowns and freezing.

I checked the CPU temperatures (using the `stress` command) and it was high, in the 90 °C. range which was odd given it was water cooled with an all in one system. I removed the block to inspect the thermal paste, and I found that the block did not appear to cover the entire surface of the CPU, only a circle in the middle. I found another cooler with the proper shaped block, the Silverstone IceGem 280, and it reduced the temperatures a lot and it no longer had any sudden shutdowns. But the freezing symptom continued. 

I considered it could be a memory issue so I wanted to run a memory test. On this motherboard, there is no option to run a memory test before loading the OS, so I installed the `memtester` utility, which was much more effective than my custom script.

`sudo apt install memtester`

First step is to check the available RAM.

`free -h`

The system has 128 GB, with 120 free so I tested 115 GB.

`memtester 115G 5`

The number at the end refers to the number of times the test should be repeated. My hunch is that it isn't the same addresses being tested, but a different addresses.

Here's how this system looks.


The test gave some errors, so I processed to test each stick separately. The results were strange. Three of the eight sticks showed memory errors. Then I put the good five sticks in and did a test of the 70 GB overnight and it still gave errors. Frustrating!

But after some research I read that some BIOS settings might cause such problems and it is worth putting the settings back to default. I didn't set this machine up myself, so I went into the BIOS and there were some CPU settings that were different to the default - probably an attempt at overclocking by the person setting up the system. I put them back to default and repeated the test. Suddenly, most of the memory problems went away! Turns out only one of the sticks was bad. After removing that one, the system passed all memtests and has been rock solid since. 

So the take home messages here are:

  • Don't ignore it when a computer system freezes or shuts down randomly, there is something wrong!
  • Check the CPU tempueratures under load, there may be a problem with CPU cooling like mine.
  • Run memtester of all system memory annually, and whenever the memory configuration is changed. If there are problems, return the BIOS settings to default.
  • If it still gives you problems, then check each stick separately. You may be able to return the defective modules to the manufacturer for replacement.
  • If you want to have good uptime and longevity of the system, keep BIOS settings default and don't overclock CPU or RAM.

Location: Geelong VIC 3220, Australia

Popular posts from this blog

Data analysis step 8: Pathway analysis with GSEA

Image
In our RNA-seq series so far we've performed differential analysis and generated some pretty graphs, showing thousands of differentially expressed genes after azacitidine treatment. In order to understand the biology underlying the differential gene expression profile, we need to perform pathway analysis. We use Gene Set Enrichment Analysis ( GSEA ) because it can detect pathway changes more sensitively and robustly than some methods. A 2013 paper compared a bunch of gene set analyses software with microarrays and is worth a look. Generate a rank file The rank file is a list of detected genes and a rank metric score. At the top of the list are genes with the "strongest" up-regulation, at the bottom of the list are the genes with the "strongest" down-regulation and the genes not changing are in the middle. The metric score I like to use is the sign of the fold change multiplied by the inverse of the p-value, although there may be better methods out there
Read more

Uploading data to GEO - which method is faster?

If you have had to upload omics data to GEO before, you'll know it's a bit of a hassle and takes a long time. There are a few methods suggested by the GEO team if you are using the Unix command line: Using 'ncftp' ncftp set passive on set so-bufsize 33554432 open  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/your @mail.com_ yourfolder put -R Folder_with_submission_files Using 'lftp' lftp  ftp://geoftp:yourpasscode@ftp-private.ncbi.nlm.nih.gov cd uploads/ your @mail.com _ yourfolder mirror -R Folder_with_submission_files Using 'sftp' (expect slower transfer speeds since this method encrypts on-the-fly) sftp  geoftp @s ftp-private.ncbi.nlm.nih.gov password: yourpasscode cd uploads/ your @mail.com _ yourfolder mkdir new_geo_submission cd new_geo_submission put file_name Using 'ncftpput' (transfers from the command-line without entering an interactive shell) Usage example: ncftpput -F -R -z -u  geoftp  -p "yourpasscode"
Read more

Generating a custom gmt file for gene set analysis

Image
Pathway and gene set analysis is a common procedure for interpretation of RNA-seq or other genome-wide expression assays. Most of the time, we use GSEA to tell us whether our gene sets of interest are up- or down-regulated. We can use gene sets from KEGG , Reactome , GO ,  MSigDB  and other sources, but you can also generate your own gene sets. The format used for GSEA is gmt . I'm going to take you through two examples of generating custom gene sets: Generate gene sets from published data sets using GEO2R Let's say you're interested in the transcription factor STAT1. I found a dataset in GEO called "Knockdown of STAT1 in SCC61 tumor xenografts leads to alterations in the expression of energy metabolic pathways", which has a paper in BMC Med.  Most uploaded array data sets can be reanalysed with GEO2R, which runs the array analysis tool Limma but this is embedded in the webpage and has a GUI which makes it very accessible for biologists. Click this link
Read more