"Dev" and "Prod" for bioinformatics
I’ve been thinking a lot about best practices lately. I even co-wrote a best practices article just last month. As I have been working with students and colleagues and reflecting on my own practices I have come to the conclusion that us researchers need to align our work more closely towards software developers rather than other researchers.
In software development there is a strong differentiation between “development” and “production” work environments. Production is the live app after release to consumers, it is critical that the app functions as expaected, is reliable, useful and with a good user experience. This is after all, where these tech companies make their money. Any downtime is going to be embarrassing and will cost the company money and customers.
The development environment on the other hand is the place where software developers can experiment with creating new features, prototype, and refine ideas. As things are built, the software code contains lots of bugs, and this nascent codebase undergoes progressive improvement.
In order to minimise issues in production, newly developed code is extensively tested in a “staging” environment, which is identical to the production environment, except that it is not released to consumers.
As bioinformaticians, I think we should have a stronger mindset around “dev”, “staging” and “prod”. So what is “prod” for a bioinformatician? I say it is any analysis work that we present for publication (preprint, peer-reviewed article, conference, etc). Just like the software developers, our reputation depends on the reliability of the workflow we use, so it better be thoroughly tested, documented and use environment management so that it yields consistent and reliable/reproducible results.
As for the “dev” environment, such considerations are less important. In the dev mindset we are focused on creating new features and data visualisations that we don’t worry too much about whether it is reproducible or not. I sometimes see practitioners using Rstudio interactively executing commands out of sequence in the Rmd scripts to generate charts for presentation. Such work could be problematic due to the effect of code order execution, as well as the presence of objects in the environment that maybe don’t belong. My advice in such cases is for practitioners to not rely on interactive execution much at all. Sure, if you’re actively developing a chart or other analysis task, feel free to use interactive execution, but when it comes to presenting this work to other people, only show results that have come from a successfully completed R Markdown script (Jupyter for Python folks). The reason for this is two-fold. First, when knitting an Rmd file, it begins with a clean environment - no loaded packages and no objects in the environment. Secondly, knitting forces code to be executed in a set sequence so it should be reproducible.
So how do we know our work is ready for “prod” (publication)? It needs to be proven to be reproducible. That is, it should be able to be re-run on another computer system, yielding the same result. This could be considered “staging”. But do you go to the effort of doing this for your research projects? Have you asked your colleagues to reproduce your entire workflow to ensure that documentation is up to scratch? Maybe you should. Remember your reputation as a good analyst rests on how reliable your work is.
As most research computer systems have different setups, from processor architecture, to operating systems, and low level dependencies this process is made a lot easier using containers such as Docker. Documentation is also important so that folks reading through your work can understand the thought process (including your future self 6 months from now). Code testing is also extremely important. In its simplest form, this could be ensuring that the R Markdown script completes without errors. Code testing probably needs its own blog post.
As I reflect on my own work and those around me I see the tendency to rely too much on preliminary “dev” results. As computational researchers, we need to treat the work we present to others more like the mission critical “prod” like our software developer cousins.
Related reading
Development, Staging and Production - IT PRo
The Five Pillars of Computational Reprducibility: Bioinformatics and Beyond
In software development there is a strong differentiation between “development” and “production” work environments. Production is the live app after release to consumers, it is critical that the app functions as expaected, is reliable, useful and with a good user experience. This is after all, where these tech companies make their money. Any downtime is going to be embarrassing and will cost the company money and customers.
The development environment on the other hand is the place where software developers can experiment with creating new features, prototype, and refine ideas. As things are built, the software code contains lots of bugs, and this nascent codebase undergoes progressive improvement.
In order to minimise issues in production, newly developed code is extensively tested in a “staging” environment, which is identical to the production environment, except that it is not released to consumers.
As bioinformaticians, I think we should have a stronger mindset around “dev”, “staging” and “prod”. So what is “prod” for a bioinformatician? I say it is any analysis work that we present for publication (preprint, peer-reviewed article, conference, etc). Just like the software developers, our reputation depends on the reliability of the workflow we use, so it better be thoroughly tested, documented and use environment management so that it yields consistent and reliable/reproducible results.
As for the “dev” environment, such considerations are less important. In the dev mindset we are focused on creating new features and data visualisations that we don’t worry too much about whether it is reproducible or not. I sometimes see practitioners using Rstudio interactively executing commands out of sequence in the Rmd scripts to generate charts for presentation. Such work could be problematic due to the effect of code order execution, as well as the presence of objects in the environment that maybe don’t belong. My advice in such cases is for practitioners to not rely on interactive execution much at all. Sure, if you’re actively developing a chart or other analysis task, feel free to use interactive execution, but when it comes to presenting this work to other people, only show results that have come from a successfully completed R Markdown script (Jupyter for Python folks). The reason for this is two-fold. First, when knitting an Rmd file, it begins with a clean environment - no loaded packages and no objects in the environment. Secondly, knitting forces code to be executed in a set sequence so it should be reproducible.
So how do we know our work is ready for “prod” (publication)? It needs to be proven to be reproducible. That is, it should be able to be re-run on another computer system, yielding the same result. This could be considered “staging”. But do you go to the effort of doing this for your research projects? Have you asked your colleagues to reproduce your entire workflow to ensure that documentation is up to scratch? Maybe you should. Remember your reputation as a good analyst rests on how reliable your work is.
As most research computer systems have different setups, from processor architecture, to operating systems, and low level dependencies this process is made a lot easier using containers such as Docker. Documentation is also important so that folks reading through your work can understand the thought process (including your future self 6 months from now). Code testing is also extremely important. In its simplest form, this could be ensuring that the R Markdown script completes without errors. Code testing probably needs its own blog post.
As I reflect on my own work and those around me I see the tendency to rely too much on preliminary “dev” results. As computational researchers, we need to treat the work we present to others more like the mission critical “prod” like our software developer cousins.
Related reading
Development, Staging and Production - IT PRo
The Five Pillars of Computational Reprducibility: Bioinformatics and Beyond
