Closing the Loop on Storage, and a Lesson in Why DB-Agnostic Code Stays DB-Agnostic

*Week 1 | Week 2 | Week 3* Week Four closed out a storage decision that had been open since Week One, pushed the local-deploy work further along, and produced a small but genuinely instructive lesson in why “it works on my database” is never the bar for a change that touches shared infrastructure. It also took me into openmrs-core for the first time this GSoC, with two separate contributions, one of which needed a second pass after a maintainer caught something I'd missed.

Merged: PR #13 — SeaweedFS Replaces MinIO

TRUNK-6627 is merged. This closes out a thread that started all the way back in Week One, when the Bitnami MinIO chart was discovered to be archived and carrying an unpatched CVE.

OpenMRS clusters built from this Helm chart now run on an actively maintained, Apache-2.0-licensed, S3-compatible storage backend instead of a frozen one. A small milestone, but a meaningful one — it removes a piece of long-term security debt from the project.

In Progress: PR #14 — One-Command make deploy

TRUNK-6650 kept moving this week. The work here has stayed iterative rather than landing in one shot, moving forward again each time I caught and closed a small gap (a stale .tgz chart archive issue, missing README prerequisites, a spinner for long-running bootstrap/teardown operations). It's the kind of PR that's earned its seven-plus commits by actually getting more correct with each one, rather than just growing.

This PR also turned out to be the source of an interesting downstream discovery — see below.

openmrs-core PR #6193 — Verifying Database Authentication in startup.sh

While working through PR #14’s Liquibase/Galera timing fix at the Helm chart level, it became clear the same underlying problem existed one layer down, inside openmrs-core itself: startup.sh waits for the database's TCP port to open via wait-for-it.sh, but never confirms the database is actually ready to accept authenticated queries. On a Galera cluster, the TCP port can open several seconds before InnoDB recovery finishes — exactly the gap that caused the pod-hang issue from Week Three.

I opened TRUNK-6650 (core) to fix this at the source: a retrying mariadb -e "SELECT 1" credential check, inserted after the TCP check and before Tomcat starts.

This is where the review got genuinely useful. Maintainer @dkayiwa caught two real problems with the first version:

The dev image had no MariaDB client. startup-dev.sh sources the same startup.sh, but the dev image's Dockerfile only installs tar gzip git curl — no mariadb-client. Against a healthy database, every single check attempt failed with a command-not-found error, the retries were exhausted, and the dev container failed to start entirely. This is the default local development path, so the fix as originally written would have broken docker-compose dev environments everywhere.

The check assumed MySQL/MariaDB unconditionally. startup.sh is meant to be database-agnostic — OpenMRS also supports PostgreSQL as a runtime backend, with its own Liquibase changesets and its own default port. Hardcoding a mariadb client call meant that any PostgreSQL deployment would fail the handshake outright and exit after the full 60-second retry window, even against a perfectly healthy database.

Both fixes were straightforward once named: add mariadb-client to the dev image's package list, and guard the new check so it only runs when OMRS_DB is mysql or mariadb.

It’s a good reminder that “DB-agnostic” isn’t just a comment in the code — it’s a property that has to be tested against every backend the system actually supports, not just the one in front of you.

openmrs-core PR #6210 — Fixing CI for Every Fork Contributor

TRUNK-6670 is the small CI fix that got split out of PR #6193 on @dkayiwa’s suggestion. The OWASP dependency-check workflow always passed --nvdApiKey ${{ secrets.NVD_API_KEY }}, but GitHub Actions doesn't expose repository secrets to workflows triggered from forks. That meant the argument silently evaluated to an empty string, the CLI treated it as a missing required value, and crashed — failing CI on every single fork pull request, regardless of what the contributor had actually changed.

The fix makes --nvdApiKey conditional on the secret being present, falling back to the public, rate-limited NVD API when it isn't. Quality gates passed clean, and it's a fix that benefits every external contributor to openmrs-core, not just my own work.

Subtask Planning: Multi-Tenant Deployment

The other major piece of this week was less visible in any single diff: breaking down TRUNK-6483 — Support deploying backend and frontend only for additional tenant into a full set of subtasks, reviewed with my mentor and marked ready for work.

This ticket is its own meaningful arc within the larger horizontal scaling project. The idea is to let a single cluster host an additional tenant — a second implementation — by deploying only a lightweight backend and frontend, while reusing the shared infrastructure (MariaDB, Elasticsearch, SeaweedFS) that the primary chart already manages. That “only deploy the app layer, share everything else” framing is what makes multi-tenancy on this architecture actually affordable.

The subtask breakdown ended up being:

• TRUNK-6652 — Create Skeleton openmrs-tenant Helm Chart. A new chart with just a backend StatefulSet and frontend Deployment, deliberately bundling zero infrastructure subcharts. All infra endpoints come in as configuration values from outside.
• TRUNK-6653 — Database Bootstrapping via Helm Pre-Install Job. An idempotent pre-install/pre-upgrade Job that creates the tenant’s database, user, and grants on the shared MariaDB instance automatically, rather than requiring manual setup per tenant.
• TRUNK-6654 — Traffic Routing via HTTPRoute Resources. Wiring tenant traffic into the same Traefik Gateway API setup from PR #12, so each tenant gets routed correctly without its own gateway.
• TRUNK-6655 — Horizontal Scaling with Sticky Sessions and Clustered Cache. This is the one I’m most looking forward to. When replicaCount > 1 for a tenant, three mechanisms kick in together: the TraefikService sticky-session pattern from PR #12 (scoped per tenant this time), an Infinispan/JGroups clustered Hibernate L2 cache using Kubernetes DNS discovery, and HorizontalPodAutoscalers for both backend and frontend. This is genuinely the heart of the whole GSoC project — multi-tenant scaling is the proving ground for it.
• TRUNK-6656 — Elasticsearch Integration with Per-Tenant Index Prefix. Tenants share the cluster’s Elasticsearch instance but write to indices prefixed per tenant (e.g. coast_patient, coast_concept), so search data never leaks across tenants.
• TRUNK-6657 — SeaweedFS Integration for Shared Object Storage. Tenants use the same shared SeaweedFS cluster from PR #13 for file attachments, so files survive pod restarts and are reachable from any replica.
• TRUNK-6658 — Repeatable Tenant Deployment. Originally scoped as a Terraform module wrapping the openmrs-tenant chart, Rafal raised a sharper idea mid-week: instead of Terraform, build an OpenMRSTenant custom resource into the openmrs-operator chart from PR #14, so creating a tenant is just "apply a CRD instance and let the operator handle the rest." I agreed — it's a more lightweight model, and it slots naturally into infrastructure I'd already built. This subtask's scope is being revised accordingly.
• TRUNK-6659 — Primary Chart Hardening for Custom Database Names. A small but necessary fix: the backend chart’s ConfigMap currently only reads the database name from mariadb.auth.database, which breaks when mariadb.enabled: false — exactly the case for a tenant pointing at the shared primary database with its own schema name.
• TRUNK-6660 — Local Dev Kind Support for Multi-Tenant Testing. A Kind-specific values override and documented workflow so multi-tenant setups can be tested entirely on a laptop, building on the make deploy foundation from PR #14.

Having this scoped and agreed now means Week Five can move straight into building TRUNK-6652, rather than spending time re-deriving the plan.

Reflecting on the Week

If Week Three’s lesson was about compounding failure modes in a single environment, Week Four’s lesson was about compounding assumptions across environments. The same root cause — a TCP check standing in for a readiness check — showed up at two different layers of the stack (Helm chart and startup.sh), and the fix at the lower layer needed to survive contact with a dev image and a second database engine I hadn't been testing against.

Looking Ahead to Week Five

• Land PR #14 (make deploy) and PR #6193 (startup.sh DB-auth check) after the requested fixes
• Merge PR #6210 (CI fix) — already approved, pending final merge
• Start TRUNK-6652 — the skeleton openmrs-tenant Helm chart, the first subtask in the multi-tenant deployment work
• Think through the OpenMRSTenant CRD design for TRUNK-6658, now that the direction has shifted away from Terraform

Thank you to my mentor Rafal Korytkowski for closing out the storage decision, and to @dkayiwa for a review on PR #6193 that caught two real production-breaking issues before they shipped.

Onward. 🚀