Software Engineering Principles I Learned Building a Genomics Tool From Scratch

For many parts of my bioinformatics career, my relationship with code has been temporary. I wrote scripts to solve immediate problems: extract this, filter that, generate a plot, move on. These scripts may be labelled as “fragile” or “disposable” because they sometimes depended on implicit assumptions about file locations, environment setup, and the fact that I was the one running them. And yeah, they would very likely crash in the wild.

Building a **variant calling pipeline in 2022** was the first time that changed. Slightly because I wasn’t emotionally invested. The second time around, I actually named the tool and made it installable, configurable, and predictable. It needed to behave consistently regardless of who ran it or where it ran. Most importantly, it needed to produce results that one could interpret with confidence. And at the point where I started worrying about installation and use, I actively started letting my brain think the words “software engineering” and “software development”.

Now, traditionally, the words biomedicine and software engineering together may seem out of place, which is understandable, but these days, the world is proudly interdisciplinary. So, this is a reflective piece on the engineering process behind building a bioinformatics software.

Problem definition

The core problem was clear:

Given a single cancer genome sample with identified somatic variants, can we identify those associated with methylation changes and report their locations, magnitudes, and directions?

The goal: to provide a structured way to prioritise candidate loci for further investigation.

I adopted what I believe is called the agile principle, where you build incrementally. I started with single-nucleotide variants (SNVs) because they’re the simplest kinds of variants, structurally or mathematically speaking. It’s one letter change in one location, period. In fact, I was prepared to leave the tool at that, but thankfully, with encouragements and not-so-subtle hints from my supervisor, I extended it to support large structural variants (SVs), which are more difficult to handle. So, technically speaking, the version that worked on SNVs was the minimum viable product.

I also started with SNVs from a single chromosome, because, again, it’s easier to handle 1 than all 23 human chromosomes at once. Plus, if you can get a script/tool to work on a defined feature of one chromosome, you can certainly extend it to the others. In hindsight, I should’ve probably started with a smaller and shorter chromosome, maybe chr22 instead of chr1, because it’d expectably have fewer variants. That might have helped with development speed.

Designing the interface with argument parsing

My early career scripts took positional/hardcoded arguments, rather than something more structured, and that’s when I’m not providing file paths directly within. For Unduliner, an engineering decision I had to confront was how users would interact with the tool, which, at the time of writing, is controlled by 2 required arguments (17 optional) that govern aspects of input data, filtering, and output behaviour. So, to manage this complexity, I used Python’s argparse library to implement structured argument parsing.

Every argument represents an explicit decision about what flexibility is allowed, what assumptions are safe, and what behaviour should be default. Required arguments enforce essential dependencies, while optional arguments provide controlled flexibility. I think defaults add transparency, informing users about parameters that influence their final outputs. Importantly, it aids informative failure modes, such that if an input is invalid or missing, the software should fail immediately and explain why. I believe this fits into the “fail fast” philosophy that preaches detecting errors quickly rather than continuing with faulty data silently.

Designing this interface solidified the knowledge I gained in argument parsing during the first year of my PhD.

Error handling

Speaking of failing fast, genomic data is inherently messy, and software that interacts with it must be designed with that reality in mind. Early versions of my tool crashed repeatedly and sometimes failed silently during the variant and sequence read processing stage. After manual investigation, I found that variant separation in SNVs was not progressing as neatly as intended, and later on, for SVs, some spanned genomic regions so large that my processing technique would require terabytes of memory.

So, I addressed the initial issue by changing my approach, where instead of trying to group reads by all variants at once for each chromosome, I grouped reads by each variant individually. It takes longer, but it’s neater and more accurate. To address the SV issue, I redesigned the system to detect these memory-intensive variants, skip them, and log exactly what happened and why. This approach, known as graceful degradation, ensures that software continues to operate even when individual inputs are impractical to process. It also preserves transparency, which is critical in scientific workflows. Silent failures are dangerous because they create false confidence. Explicitly logged limitations, on the other hand, allow users to interpret results appropriately.

Logging and provenance

In the spirit of explicitness, I used structured logging to record input files, processing summaries, and any skipped variants. These logs document exactly how results were generated because if you cannot trace a result back to its inputs and parameters, then you can’t meaningfully evaluate or trust that result.

For example, in an ideal world, if someone uses unduliner with an input variant file containing 10000 variants, they would expect the output file to have 10001 lines, with the extra line being the table header. However, that’s not always the case because sometimes, variants in the VCF are not represented in the corresponding BAM (sequence read) file. So, instead of wondering what the heck happened, users get a supplementary output file that basically says, “I couldn’t process these variants because no reads support them.”

Modularity and abstraction

This is something I did right from the start, before I even knew what the internal skeleton of the tool would look like. I created functions for every little step because I had a strong feeling it wouldn’t be so little in the big picture. So, read extraction, variant processing, and output generation each had their designated units. This made debugging very easy, and I could trace errors right to the source without having to strain through several lines of interconnected gibberish.

This modular design also made the software easier to extend to SVs because all I had to do was duplicate the function for SNVs and adjust where necessary to fit the SV structure. In the context of software engineering, this also aligns with the “don’t repeat yourself” principle, because repetition creates opportunities for inconsistency and bugs.

Testing to prove that the software “works”

This helps to demonstrate that the software worked under defined conditions. Even better, if you supply the exact test data and expected results, and have someone else try it out. I did multiple levels of testing, like checking what happens when the file is a structural variant but the appropriate argument is missing (it obviously crashes) and ensuring that new changes/fixes did not break existing functionality. It was also through testing that I discovered the memory issue I mentioned earlier with some structural variants, because my development data was not as complex as the test data.

Testing does not eliminate bugs. A user might still run into an issue for many reasons, but it does guarantee that the software works with expected data.

Dependency management, packaging, and deployment

The major step in transitioning from script to software was making the tool installable. This required structuring the project as a Python package to allow for pip installation, defining dependencies explicitly, and configuring installation using modern packaging standards. This ensures that the software can be installed and run consistently across environments.

I also implemented automated deployment using GitHub Actions, enabling continuous integration and continuous deployment. This basically means the system automatically builds and publishes new versions of my software when I make certain changes and tag them as an upgrade. It ensures that updates are distributed reliably.

Shoutout to the Python packing user guide (linked above) and all the connected resources and tutorials cos what would I do without wonderful documentation?

Future work includes Conda packaging, which is standard in bioinformatics due to system-level dependencies. For some reason, I’m stalling/procrastinating on that one.

Documentation

My favourite part. Documentation is the interface between your intent and the user’s understanding. Because this is a CLI tool, I prioritised a clear README with installation instructions, usage examples, argument explanations, and expected output.

If users cannot figure out how to use your software independently, the software is incomplete.

Conclusion

Engineering is about building things that work or breaking things down to figure out what makes them work and then rebuilding them. Either way, it involves a process, some forms of measurements/calculations, and objective decision-making. Just as important is expecting failures that could have you rethinking your entire design and handling them explicitly, testing assumptions, and documenting the process as a whole.

P.S. It’s fascinating how normal, random, and/or expected things have jargony niche names… fail fast, graceful degradation, etc. Thanks, ChatGPT, for the names.