From Terminal Scripts to the Cloud: Designing an Automated AMR Detection Platform with Nextflow and AWS

With a background in biotechnology, I was accustomed to the ‘black box’ of genomic workflows. While I was already writing custom scripts to handle RNA-seq and ChIP-seq data, I was essentially building linear tools rather than integrated systems. My focus was on the biological output.

It wasn’t until my thesis when I began using Nextflow to orchestrate massive datasets that I transitioned from writing isolated bioinformatics scripts to architecting scalable, end-to-end pipelines. That experience kicked off an intensive learning journey. To build the kind of tool I envisioned, I had to step far outside my initial comfort zone, diving deep into cloud infrastructure (AWS), Infrastructure-as-Code (Terraform) and frontend web development (Streamlit). My goal shifted from simply writing a bioinformatics script to engineering a complete, scalable, full-stack application.

That drive led me to develop NextAMR: a cloud-native, end-to-end platform deployed on AWS for bacterial genome assembly and Antimicrobial Resistance (AMR) detection.

In this post, I want to share that journey. I’ll walk through why I chose my specific tech stack, the architectural roadblocks I slammed into and how I navigated my way out of them to build a scalable platform.

1. Choosing the Stack: The Foundation of NextAMR

I structured NextAMR around three critical functional layers:

• The Executor: Nextflow (DSL2): Nextflow was my natural starting point after my thesis. I designed the main workflow to route data into three distinct processing lanes (short-read, long-read, or hybrid). The main script acts as a traffic controller, delegating the biological heavy lifting to independent DSL2 modules for trimming, assembly and AMR detection. I also wired Nextflow’s -resume feature directly into a toggle on the Streamlit UI. If a massive run fails halfway through, researchers can just check a box to instantly restart from the last successful checkpoint, saving hours of compute time.
• The Orchestrator: Terraform (Infrastructure as Code): I chose AWS because its Batch ecosystem offers better Nextflow integration, allowing the pipeline to scale instantly from zero to 256 vCPUs. While I started in the console to learn the “what,” I moved to Terraform to enable one-command reproducibility. Despite the HCL (HashiCorp Configuration Language) learning curve, managing the entire environment from VPCs to IAM via code ensures the infrastructure is auditable and 100% reproducible. This design enforces “Least Privilege” security with granular job roles instead of broad admin keys. Furthermore, opting for EC2 Spot Instances over On-Demand compute slashed our processing costs by up to 70–90%.
• The UI: Streamlit: I chose Streamlit because it enables seamless integration with the AWS SDK (Boto3) for cloud logic and Plotly for visualizing complex genomic metrics. Its reactive execution model allowed me to build, test, and iterate on the UI in sync with the backend development, ensuring users get a clean interface to upload data and immediately view condensed reports of raw FastP metrics, assembly stats like L50/N50 and filtered AMR results.

Figure 1: High-level NextAMR execution lifecycle and event-driven cloud architecture.

2. The Biological Architecture: From Reads to Resistance

I designed NextAMR to be a modular pipeline that natively supports short-read (Illumina), long-read (Oxford Nanopore/PacBio) and hybrid sequencing inputs. Here is exactly why I chose these specific tools for the job:

Phase 1: Preprocessing & Quality Control

Garbage in, garbage out. The first step is stripping adapters and filtering low-quality reads so the assemblers don’t choke.

• FastP (Short-Read): I chose FastP because it is an ultra-fast, all-in-one preprocessor that handles quality profiling, adapter trimming and read filtering in a single pass, keeping compute time exceptionally low.
• Filtlong (Long-Read): Long-read data is notoriously noisy. I implemented Filtlong because it acts as a smart filter that scores reads based on both quality and length, allowing the pipeline to discard the “junk” reads while preserving the longest, highest-quality fragments for the assembler.

Phase 2: Assembly & Polishing

This is the most compute-heavy phase, where the pipeline reconstructs the bacterial genome.

• Unicycler (Short-Read & Hybrid): Unicycler is the gold standard for bacterial assembly. I chose it for the hybrid lane because it uses the short reads to build an accurate foundational graph (via SPAdes) and then uses the long reads to bridge the gaps, reliably producing complete, circular genomes and plasmids.
• Flye (Long-Read Only): When only long reads are available, Flye is my go-to. It uses a unique repeat-graph algorithm that is highly resilient to the uneven coverage and higher error rates typical of raw Oxford Nanopore data.
• Medaka (Polishing): Because long-read assemblers often leave behind structural errors (like homopolymer deletions), I added Medaka. It uses a neural network trained specifically on Oxford Nanopore data to polish the draft assembly and fix those final base-level errors.

Phase 3: Hybrid Polishing & Genome Standardization

A raw assembly isn’t a finished genome. For hybrid pipelines, I built a rigorous polishing phase to ensure absolute base-level accuracy and structural standardization.

• BWA-MEM & Polypolish / Pypolca (Short-Read Polishing): Long reads give you the structure, but short reads give you the accuracy. I use BWA-MEM to map the high-quality Illumina reads back to the draft assembly. Then, the pipeline runs Polypolish and Pypolca to aggressively target and correct any remaining indels or SNPs that Medaka missed.
• Dnaapler (Standardization): A complete circular genome isn’t truly “finished” until it’s standardized. I integrated Dnaapler to automatically reorient the circular chromosomes and plasmids to start at the dnaA replication origin. This ensures the output is perfectly prepped for comparative genomics and NCBI database submission.

Phase 4: Annotation & AMR Profiling

Once the genome is assembled, polished and rotated, the pipeline translates the raw DNA sequence into actionable biological insights.

• Bakta (Annotation): I chose Bakta because it is incredibly fast and standardizes the output. It comprehensively annotates genes, tRNAs and features, providing a clean database for the final step.
• AMRFinderPlus (AMR Detection): I integrated AMRFinderPlus because it goes beyond simple gene detection. It accurately identifies complex point mutations, providing researchers with the most rigorous and authoritative resistance profile possible.

Figure 2: The three main sequencing lanes of the NextAMR pipeline (Short-Read, Long-Read and Hybrid).

3. Optimizing the Pipeline: Data Validation, Testing and Results Visualization

Building a robust bioinformatics pipeline extends far beyond simply chaining tools together. To transform NextAMR into a production-ready application, I focused on three critical areas: preventing failures, optimizing costs and delivering immediate, actionable insights to the user.

Preventing Costly Failures with Pre-Flight Validation

One of the most frustrating and expensive experiences in cloud computing is waiting hours for a job to provision, only for it to fail instantly due to a simple input error. To mitigate this, NextAMR incorporates a pre-flight validation step. Before any compute is spun up on AWS Batch, the application automatically verifies the user’s input TSV against the files present in the designated S3 bucket. This ensures that any mistyped filenames or missing datasets are caught immediately, saving valuable compute time and cost.

Cost-Effective CI/CD and “Smoke Testing”

To maintain stability and prevent expensive cloud failures on every code change, I implemented a lean Continuous Integration/Continuous Deployment (CI/CD) strategy using GitHub Actions. Every code push triggers a Nextflow “smoke test” against a minimal dataset, comprising one hybrid, one short-read, and one long-read sample. This approach offers significant value:

• It’s just enough data to rapidly verify routing logic and ensure Docker containers can be successfully pulled.
• It quickly tests the cloud provisioning setup in a matter of minutes.

This crucial safety net has already proven its worth, catching file-path bugs early that would have otherwise led to costly crashes on a full-scale production instance.

Interactive Dashboards for Actionable Results

Once a run is complete, NextAMR transforms into an intuitive dashboard, replacing raw data dumps with clear, actionable insights. Users can immediately view condensed reports including critical QC metrics like FastP outputs, assembly statistics such as L50/N50 and filtered Antimicrobial Resistance (AMR) results. Crucially, the system automates threat classification, dynamically flagging Beta-lactamases (e.g.,bla, ndm) as “CRITICAL” and difficult-to-treat genes (like mcr, vanA) as “HIGH RISK.” This focused presentation empowers researchers to quickly assess threats rather than sifting through extensive raw lists.

Figure 3: Streamlit dashboard displaying raw read metrics and assembly quality statistics (N50 & GC) for E. coli samples.

Figure 4: Full Antimicrobial Resistance (AMR) report matrix for E. coli.

Figure 5: Live query search bar isolating beta-lactamase threats within the processed sample data.

4. Roadblocks

The journey wasn’t perfectly smooth. Building this pipeline forced me to learn hard lessons about cloud architecture limitations.

The EC2 vs. Fargate Battle

My initial goal was aggressive cost-efficiency: routing lightweight tasks to serverless Fargate and heavy tasks to EC2 Spot Instances. However, mixing these within a single AWS Batch environment proved to be an exercise in frustration due to sparse documentation and cryptic networking errors.

• The Pivot: I realized I was overcomplicating the architecture for marginal gains. I pivoted to a robust, EC2-only setup, using Terraform Launch Templates to provision Spot Instances with 200GB of high-performance gp3 storage.

The 10GB Upload Bottleneck

Streamlit’s default file upload limit is 200MB. I initially overrode this to 10GB, but quickly realized that pushing massive genomic files through a browser is architecturally risky due to timeouts and RAM limitations.

• The Pivot: I re-engineered the ingestion strategy. The app now generates S3 pre-signed URLs, allowing users to facilitate direct, multipart uploads from their browser straight to S3, bypassing the Streamlit server entirely.

Data Persistence

To keep costs lean, I implemented a dual-bucket strategy:

• Database Bucket: Stores persistent reference data (like Bakta). I used the prevent_destroy = true lifecycle rule to ensure this data is never accidentally deleted.
• Compute Bucket: Handles intermediate files with an S3 Lifecycle Policy that automatically deletes data every 7 days to prevent “cloud bloat”

What’s Next: The Journey Continues

A pipeline is never truly “finished.” Before starting this project, my bioinformatics work lived inside terminal windows running linear scripts that worked perfectly on my machine. Stepping into full-stack architecture forced me to stop thinking just about the biological outputs and start thinking about the system holding them up. Figuring out how to orchestrate AWS Batch dynamically, tie it to a responsive frontend, and lock down security via Terraform was an intensive learning curve, but it completely changed how I think about delivering scientific software.

My focus now, is shifting from simply getting the system to work to scaling it into a generalized, multi-omic platform. To expand this ecosystem my upcoming roadmap centers on four concrete engineering evolutions

• Expanding to a Multi-Pipeline Ecosystem: I am planning to transition the backend infrastructure into a centralized hub capable of spinning up distinct, modular pipelines depending on the user’s needs. Beyond bacterial genomics and AMR tracking, this expansion will introduce dedicated lanes for metagenomics (WGS) and transcriptomics.
• Preempting Heavy Data Constraints: As the platform scales into resource-intensive domains like metagenomics where raw abundance and taxonomic profiling matrices can easily freeze a standard web browser my goal is to implement server-side aggregation. By using Nextflow to pre-compute summaries and top-hit profiles before saving them to S3, the Streamlit interface will read only highly condensed, optimized payloads. Combined with lazy loading and row virtualization, the UI will remain lightweight and responsive.
• Refactoring to nf-core Guidelines: I plan to adapt our existing DSL2 modules to match nf-core template and linting guidelines. This will standardize our configuration structures, making the underlying pipeline easily shareable and ready for broader community adoption and open-source contribution.
• Granular Cost Visualization & Cloud Monitoring: Because expanding into a multi-pipeline hub means a major increase in infrastructure utilization, transparency is paramount. I am currently designing dynamic cost-tracking modules to parse AWS billing metrics per run, allowing users to visualize the exact compute cost of each isolated execution alongside centralized AWS CloudWatch monitoring for real-time cluster health.

NextAMR started as a massive personal challenge to bridge my background in biotechnology with cloud systems engineering. The core infrastructure is built, the loop is closed, and now it’s time to see how far this platform can scale.

See More

📂 Interactive System Design: NextAMR Cloud Architecture Canvas on

Let’s Connect!

GitHub: NaouelEldjouher/NextAMR 📫 LinkedIn: Naouel Eldjouher