Good laboratory practices (GLP) in the field of bioinformatics

Image generated by imgflip.com

Hi & good day everyone !

I wish someone had told me, guided me and reminded me earlier to adhere to the good laboratory practices of doing bioinformatics work 2 years ago, so that I wouldn’t encounter so many errors, which ended up dragging my research timeline.

Generally, we are often taught and reminded to comply with all the good laboratory practices (GLP) when we are conducting our wet lab experiments. For instance, we have to wear proper personal protective equipment (PPE) to protect ourselves and avoid sample contamination. We have to label all the samples, reagents, and solutions that we use clearly, so that other shared lab users are well-informed to avoid unnecessary cross-sample contamination and disputes. Also, we have to regularly calibrate the instrument to ensure accurate and reliable measurement. In addition, we have to always clean the used instruments and working surfaces before and after use. We have to document and record the use of instruments in a logbook to ensure user traceability. We have to follow the respective lab’s standard operating procedures (SOPs) as always. We have to replicate our experiments to ensure results accuracy and reproducibility. We have to practise safe disposal of chemical and biological wastes into the correct biohazard bins to ensure the safety of other shared lab users. And the list goes on and on !

These are what we as biologists have often been taught and reminded of ever since we were doing our experiments back in graduate school. However, when it comes to a dry lab setting, what are the good laboratory practices (GLP) that we have to follow as bioinformaticians or computational biologists? Have you ever wondered about this question before? I bet this question often gets underemphasized in the field of biology.

Below are the lists of good bioinformatics laboratory practices that I think every single bioinformatician should know beforehand before starting to code and run their analysis:

1. Use the right tool for the job. As there are dozens of bioinformatics tools out there that perform the similar functions, it’s highly recommended and advisable to look for the latest publication to replicate the most widely used bioinformatics pipeline/workflow for your specific field of study.
2. Performing a thorough bioinformatics literature review is crucial. Use bioinformatics tools that are most widely used and cited as oftentimes these tools are user-friendly to generate quick, accurate results.
3. Become the master of conda user. Master the use of conda, mamba, miniforge, mambaforge & know their differences early.
4. Think carefully and thoroughly about the consequences before you try deleting the files. Make sure they are no longer needed for downstream analysis for instance those temporary files generated during the run.
5. Compress the .fastq files whenever possible to save disk space. Also, convert the .sam files to .bam files whenever possible via samtools view to save disk space. In addition, to save disk space, remember to delete the unused, large temporary files generated during Trinity run as well. Cultivate a good habit to regularly spring-clean your unused and unnecessary files and directories to make it clean, tidy and well-organized as always. These are a few tips and tricks to help you save disk space, especially when working on a shared HPC cluster.
6. Always backup your data after running each and every analysis into a large-storage hard disk drive (HDD) or solid state drive (SSD) to avoid potential data loss. Bear in mind that backing up data to a Network Attached Storage (NAS) device after running each analysis is a good practice to ensure data safety and facilitate quick recovery. This approach, often called Continuous Data Protection (CDP) will help to minimize data loss and enable you to roll back to any point before a failure or attack. NAS devices provide a dedicated storage solution, offering redundancy and protection against data loss from hardware failures or accidental deletions.
7. Always run bioinformatics analysis as a non-root user by running the switch user (su) command, especially when working on a shared multi-users high-performance computing (HPC) cluster. Also, it’s best to create an isolated conda environment for each bioinformatics tool and install bioinformatics tools via conda whenever possible. Installing tools in isolated conda/mamba environments is always the best practice. This is to avoid breaking the overall global system utilities and avoid encountering dependency conflicts issue. Be extra careful when you are operating within the base environment as a root user, especially when you are working on a shared multi-user HPC cluster. Be careful to use and run any sudo commands. As you might easily break the global HPC system if you’re not careful enough when running commands. Never run the command, sudo apt update && sudo apt upgrade -y, in the root environment if you are working on a multiple user HPC cluster.
8. Master the other ways to install bioinformatics tools, which include using the commands, wget, git clone and make, although I previously said installing tools in isolated conda/mamba environments is always the best practice. This is because although bioconda packages are stable, and tested releases of tools, updates to the bioconda package by tools’ developers (to debug due to problems reported by the other tool users in the GitHub Issues section) may sometimes lag behind the latest release. Not all bioinformatics tools are maintained in the Anaconda packages. Bioinformatic tools are updated frequently to fix bugs, improve performance and add new features. When a newer version of the updated bioinformatics tool gets released in its official GitHub repository, these updates may not be immediately reflected in the official Anaconda repositories. If you wish to use the latest tool’s updated features and bug fixes, it’s best to install the bioinformatics tools via wget or git clone commands. Direct downloads (wget), GitHub cloning (git clone) and source code compilation (make) give you the latest, official release of the bioinformatics tool, which include the new features, bug fixes, and improvements, as soon as they are made available by the tool’s developers.
9. Source code is the human-readable code written by developers. Compiling is the process of converting this source code into a binary, a machine-readable file that the computer can execute. The make command automates the process of compiling the source code.
10. Avoid running too many heavy computational analysis at the same time (in parallel) as excessive usage and consumption of threads beyond critical limit can heat up computer and cause to crash/shut down suddenly. When you’re working on a shared HPC cluster, it’s always a good practice to leave some threads and memory reserved to maintain system responsiveness, prevent systems instability, and to prevent sudden memory usage spikes, in order to reduce the risk of encountering OutofMemory errors and reserve some computational resources for use by other systems’ tasks. For instance, if your HPC cluster has 48 threads, then you can specify the use of a maximum 16 threads for each analysis that you run. This is a good practice as you leave the remaining 32 threads for systems stability and for other shared HPC users to run their own analysis respectively. You can run multiple bioinformatics analyses in parallel, but always monitor system load by running the commands, either htop or top to ensure you don’t oversubscribe the CPU and memory.
11. Bear in mind that not all the bioinformatics’s tools and softwares can efficiently leverage parallel processing. As a result, whenever available, multi-threading or parallelization solutions were used to maximise computing performance and speed up the run. For instance, corset, which is a tool used to deduplicate the transcriptome assembly, does not provide any multi-threading option for users to parallelize the run.
12. Bear in mind that using more threads will lead to faster parallel processing of data, but high memory usage, whereas using fewer number of threads lead to slower processing of data, but lower memory usage. Using all the available computational threads and memory will just lead to inefficiency (it will fight for space and tool processing) as it will just make your system to appear unstable and crash easily (it defeats the purpose as it doesn’t speed up the process at all). If you specify the use of the maximum number of threads, the CPU may get overloaded, causing delays due to thread scheduling. Some threads will be forced to wait because the CPU has a limited number of cores. The system may slow down for other tasks or even freeze. If you specify the use of maximum memory, your bioinformatics pipeline may crash if additional memory is needed. The system might start using SWAP memory (which is stored on the hard drive and is much slower than RAM). Excessive swapping of memory will happen when RAM is full and spills to disk, thus slowing everything down. If all the computational memory is used, your computer may freeze, thus forcing you to have a hard reboot (restart) of your HPC system. Using fewer number of threads can help reduce CPU contention and RAM pressure, where this means less simultaneous memory allocation per thread, thus avoiding the “malloc(): out of memory” error. Bear in mind that a slower and stable run is better than a crash. Carefully monitor jobs if your machine was close to maxing RAM.
13. Master the use of tmux, nohup and screen commands to run bioinformatics analysis in the background. This is especially useful when working with shared multi-users high performance computing (HPC) cluster. Learn how to run the command, ps aux | grep <tool’s name that you wrote in the command line>, for example, ps aux | grep busco, to list out all the currently running BUSCO processes in the background to double check whether the process is still running in the background.
14. Learn how to kill the running process in the background after running the commands with nohup/screen/tmux flags. Running the command, kill <process ID>, is always the safest way to kill a running bioinformatics process as you only target to kill one exact process, and there will be no unintentional casualties, rather than running the command, pkill -9 -f busco (might unintentionally kill the shared services altogether if they happen to reference to busco). After running the command, kill <process ID>, please kindly wait for at least 5 minutes for the process to killed completely, before you run the command, ps aux | grep <tool’s name that you wrote in the command line>, again to double check whether the process has been killed successfully.
15. Remember to carefully monitor your running bioinformatics analysis/pipeline in the background to check the usage of computational resources such as the random access memory (RAM) and central processing unit (CPU) to avoid running into OutofMemory error. You can run the command, free -h, to check the current memory usage condition. In addition, you can run the command, df -h, to check the total GB/TB storage that is currently available for the mounted directory that you’re using right now.
16. Make sure to double check that all the required dependencies of bioinformatics tools have been successfully installed before you run any analysis. Otherwise, you are going to run into error or your process will terminate halfway abruptly.
17. Always run bioinformatics analysis by using the toy dataset first whenever possible (if available and deposited in the GitHub repository of the tool). Look for any test_demo directory being uploaded in the bioinformatics GitHub repository and use the deposited toy dataset. This will help you to easily troubleshoot the error/debug that you encounter later on (if you encounter any error) to check whether the error arises due to your system or due to installation. You can view the results of the test demonstration to help with troubleshooting & debugging.
18. Always read the bioinformatics tool documentation as some bioinformatics tools require you to write a bash script like running the Longstitch pipeline, where users cannot straight away run through one long command line. Always check how to run the bioinformatics tool beforehand, whether running Perl script, R script, python script, bash script or command prompt. Bear in mind that it’s best to always refer to the latest installation guide of the tools in the official documentation every time when you want to use the tool.
19. GitHub issues section, Stack Overflow, Biostar, Bioinformatics Stack Exchange, Biocontainers, SEQanswers are your go-to sections whenever you encounter any error while trying to run the bioinformatics tool/pipeline. They are your life saviours. This is because oftentimes, the errors that you’re encountering have been experienced by others previously and potentially solved by other experts via the conversation guide. Trust me, they are indeed your life saviours and you will thank them profusely deep down in your heart. Once you’ve trained yourself hard enough and become the tools’ expert one day, don’t forget to help others in return.
20. Always double check the analysis results that you obtain to make biological sense of the results before further using it as input for downstream analysis. This is to avoid encountering the garbage in, garbage out problem. Always check the quality of the obtained data at each stage of the pipeline before proceeding further.
21. Dependencies are like the tools/packages/libraries required by the bioinformatics software to run and function properly. Ensure the versions of the installed dependencies are compatible with the version of the bioinformatics tools that you’re using. Otherwise, you need to update the dependency, but be extra careful when running the “sudo update” as a root user, especially when using a shared HPC cluster, as it might cause dependency conflicts later.
22. Always try to install the required Perl modules via Anaconda first. If the Perl modules are not available within conda, then install Comprehensive Perl Archive Network (CPAN) via conda-forge inside an already created conda environment, followed by installing the required Perl module. This is because using Anaconda to install Perl modules is the safest and most reproducible approach as it keeps your system clean, avoids permission issues and ensure compatibility.
23. Always ensure that your written script is executable before running it. When you write the script on a DOS/Windows system and later transfer the script to Unix/Linux system, remember to run the dos2unix command to convert the script format from the DOS/Windows format to Unix/Linux format. After that, run the chmod +x commands to add the executable permission to the file to make sure the file is executable.
24. Always name your file and working directory properly to make sure they are human-readable, machine-readable and sortable. Follow ISO 8601 for date. Use consistent prefixes. To make sure the files are machine-readable, it’s always best practice to name the file and directory with underscores or hyphen in between. Don’t name the file, like: “latest_final_corrected_DEG_masterlist.tsv” or “final_final_thesis.docx”.
25. Know how to use the export command to set and export the PATH environment variables in your terminal to run tools and executables more efficiently. By running the command, export PATH=$PATH:/path/to/your/tool, you temporarily add the directory containing the tools’ executables file into the PATH (as the PATH variable is meant to store the directories that hold executable programs.), allowing you to execute programs from any directory without typing the full path of the program executable every time when you want to run the tool/script. If necessary, you can run the command, source ~/.bashrc, to apply changes permanently to the terminal to make the environment variable available to all terminal sessions if you wish to maintain a well-documented and reproducible computing environment.
26. It’s best practice to run the commands, sudo apt update and sudo apt upgrade to make sure that your system is up-to-date before you install any tools. But again, be extra careful when you are running the sudo commands as a root user within the base environment on a shared HPC cluster.
27. Always choose the installation guide that matches with your computer’s specifications.
28. It’s always best to run the docker container as a non-root user by specifying the -user flag and the -sudo flag before pulling and running the docker container to avoid affecting the global HPC cluster root system.
29. Learn how to use docker and singularity as it really simplifies the installation of bioinformatics tools a lot and save you tons of time.
30. Warnings are just as bad as errors. Ignoring warnings won’t make them disappear. If something feels off, think thoroughly before you use it and don’t justify the off thingy without any thinking. Please walk away early from them (either warning or error) to avoid future heartbreak.
31. Remember to document your written code and scripts within a logbook step by step (either making a logbook website/recording in a notebook) at your convenience for others to be able to reproduce your bioinformatics pipeline results with the previously specified parameters. When you document your code inside your logbook website, it’s best to include comments for each & every single code. Trust me, you will thank yourself later when there’s a need for you to refer back later, either to rerun the command & script or to answer reviewer’s questions.
32. Document all the previous errors that you encountered if possible and list out the possible solutions that you used to solve them. Trust me, this will definitely save you ample time to troubleshoot the same error that you are going to encounter again when you run the tool in the future for your next publication.
33. Double check & verify the path location/working directory of files before you execute the command line every single time to avoid encountering error such as “no such file/directory”.
34. Only update conda in a non-root environment (if conda or python has a broken installation, updating it within the root environment could cause dependency conflicts, thus potentially leading to system instability). Updating Conda in the base environment as a root user could cause dependency conflicts and potentially break the conda installation.
35. Master R and make use of its R packages with pre-defined/built-in functions to perform your bioinformatics analysis. Remember to run the command, install.packages(), to install R packages from the Comprehensive R Archive Network (CRAN) & run the command, BiocManager::install(), to install R Bioconductor packages (Bioconductor is must-know open-source software project built for building dozens of useful bioinformatics R packages).
36. Stop being scared to break things, read errors like clues, not failures, ask why before copying anything.
37. Brace yourself to test your bioinformatics pipeline with different tools and different parameters for each tool and be patient to iterate through multiple runs until you successfully get the desired results. Good things take time. For instance, sometimes you might need to run multiple genome assemblers or transcriptome assemblers and then compare the BUSCO completeness score of the assemblies to choose the most complete version of the assembly.
38. Learn how to run GNU Parallel instead of running xargs to maximize the use of all the CPUs or threads to parallelize tasks. GNU Parallel is generally considered more powerful and versatile than xargs, though xargs can be faster for very short-running jobs due to lower overhead.
39. It’s poor practice to save project-specific files in the softwares’ bin directories as it is going to clutter the software installation. It will cause permission issues and it will be difficult for you to track which config belongs to which project, and the future software updates might also possibly overwrite and delete your file. Always group software installations and project-specific-files by directories properly.
40. Remember to disable your HPC cluster system from sleeping when your background job is running. Otherwise, the background job will get terminated when the system sleeps. This can be achieved by disabling sleep settings or using a mechanism to keep the system awake during the job’s execution. You might also need to adjust scheduling to avoid overlaps with system maintenance windows that could trigger sleep.
41. Leverage the use of AI tools for instance ChatGPT, Perplexity, Claude, GitHub Copilot when you are coding. I personally love to use GitHub Copilot and Perplexity as they help me to debug much faster and way more accurate. GitHub Copilot is a must-go-to platform when you want to try vibe coding.
42. Learn how to use checkpoint if your bioinformatics tool offers options to users to run analysis with specific checkpoints. This is to prevent you from running the analysis all over again from scratch if your run fails halfway or your run gets disturbed/terminated. Checkpointing allows you to save the state of a bioinformatics analysis at specific points during its execution, enabling you to resume from the last saved checkpoint if the analysis is interrupted. This is particularly useful for long-running or complex analyses where failure can be costly in terms of time and resources. However, checkpointing can significantly consume the disk space and increase the runtime, hence, use when it is necessary. For instance, the SPAdes run allows users to run the SPAdes assembly process with the — checkpoints option and it also offers the — restart-from and — continue options to users to restart the run or continue the run from a specific last-saved checkpoint.
43. Learn how to use the flag, “ — tmp” inside the command line to direct temp files to a specific directory with a larger disk space if needed.
44. Be aware that if you are installing lots of bioinformatics tools via conda within conda environments, the computer disk usage can grow very quickly, especially if you create many conda environments and you don’t clear the package cache.
45. Use Google drive to transfer directories of files to the remote HPC system or to the NAS device for backup and use. While working on a remote HPC cluster, learn how to use the free file sync (https://freefilesync.org/manual.php) to transfer your files as it will save tons of time and ease your file transfer process.
46. You can try to experiment with different bioinformatics tools performing the same analysis to conduct a methodological comparison study to showcase & publish the respective strength & weaknesses of different tools.
47. Read & understand the terms and usage conditions of bioinformatics tools to save you from facing lots of trouble during the peer review process later.
48. Set a password to lock and encrypt your data directory if you are not comfortable with other shared users accessing your data when you are working on a shared multi-user HPC cluster.
49. Install mailutils by running the command, sudo apt-get install mailutils, to notify you through email when your task is done.
50. Don’t delete failed results instantly, perhaps you can use them for comparison and publish them in the Journal of Negative Results (if possible).
51. Always use the latest stable CUDA Toolkit that is compatible with your NVIDIA drivers while running CUDA-enabled GPU-accelerated bioinformatics tools.
52. Learn how to work and submit jobs on a remote HPC cluster using job schedulers such as SLURM, PBS, or LSF.
53. Download BLUF.Ai chrome extension and use it to summarize research articles, webpages and YouTube videos.
54. Be a Git control master. Master the use of Git and GitHub to push your project code to your created GitHub profile account to showcase and share your bioinformatics research output, which is a plus/bonus for your working resume and CV. Also, via git pushing, you can easily collaborate with your team members to work on new features and revise your project code to perfection before officially releasing it to the public for use and reference. This is because Git is a distributed version control system (VCS) that can be used as a memory card to easily save all the changes that you’ve made to your code for your project and roll back/rewind to the previous code changes whenever it is necessary.
55. Try to be a pipeline builder. Learn how to construct a reproducible workflow using the Galaxy Europe web server. Master the use of Snakemake or Nextflow to construct a reproducible pipeline/workflow that you can reuse for multiple research studies for others to easily reproduce your research output and to save ample time, rather than running command by command in a tedious way.
56. Bear in mind that Microsoft Windows has a MAX_PATH limit of ~256 characters. If the length of the path and filename combined is too long (exceed ~256 characters), you will still be able to see the path/files via Windows/File Explorer, but you cannot delete/move/rename these directories/files. Hence, always write a short, concise filename and file directory when you are running your bioinformatics analysis, especially when you are working on your Windows RStudio environment.
57. In R and RStudio, when you close your session, the previously loaded R packages are unloaded. Hence, the next time you open RStudio, you need to rerun the library() command to load the R packages again.
58. Run wget command to download online resources for instance large FASTA format of NCBI NR database files, instead of downloading the FASTA file via web browser.
59. When you encounter version conflicts of R packages, you can try to remove all of the R packages by running the Rcode, remove.packages() first, followed by restarting the R session, and installing the R packages by using the Rcode, install.packages() once again.
60. Always be as resourceful as possible. Be aware and learn how to utilize the already constructed bioinformatics web servers (without requiring any coding or programming task) to analyze your bioinformatics data. For instance, using Venny to perform Venn comparison analysis, using blast2go to perform GO annotation, using UniProt and BioMart to map between gene IDs, protein IDs, KEGG IDs, Reactome IDs, COG IDs while trying to annotate the genes, and the list goes on and on ! Take initiative to polish your bioinformatics skills via attending bioinformatics workshops, Udemy courses, Coursera courses, Udacity courses whenever you are free to become proficient in understanding and writing workable bash scripts, Python scripts and Rmarkdown/Rscripts to streamline your bioinformatics analysis.

All in all, above are all the tips and tricks to become a full-fledged, professional bioinformatician that I gathered and integrated from my past experiences of working on my master in silico project. I sincerely hope that you find this post and tips and tricks helpful and resourceful to guide you through streamlining and accelerating your bioinformatics analysis. Rest assured that equipping yourself with all of the aforementioned bioinformatics GLP is crucial to avoid your HPC clusters from breaking down & crashing, thus extending their lives even longer. Bear in mind that the best bioinformatician don’t just write scripts and code. They ask sharp biological research questions as they know the biology, which is the WHY behind the research motive. Also, they should be able to know how to spot the hidden signals quickly and tell a clear, biologically meaningful story after spending weeks or even months of analyzing the data.

If you would ask me what makes bioinformaticians feel happy and fulfilling, I can confidently answer that for me, happiness is when my code runs without errors just like when I’m chilling with my family💯I bet this is most common abundant source of happiness of many bioinformaticians out there as well ! Happy coding and digging for biological insights from the data ! All the very best to those bioinformatics warriors out there ! The future depends on you !

Thank you.