The Reverse Hydra: surviving backend outages with STIP

In the financial sector the stand-in-processing (STIP) is often used by card companies to make the system work while there is an outage of the backend system.

The outage can be caused by maintenance, or someone just shipped their code late on a Friday. Sometimes these outages can take a lot of time to fix, and your company’s reputation is on the line. What can you do so your customers aren’t stuck trying to buy something at the store because the card terminal gives them an error?

I was lucky enough to get the task of designing and implementing a STIP system for an API that was highly critical for card movements in one country, and it has been working seamlessly since we implemented it.

First their were key factors that needed to be accomplished

1. The costumer should not notice an outage
2. Our end users should not need to route their traffic
3. The backend system should be easy to maintain
4. Easy to switch on and off

So lets start creating something I like to call a “REVERSE HYDRA”. An API with one contract or HEAD and multiple bodies.

Reverse Hydra architecture: one gRPC contract routed to regular or STIP bodies

Pre-requirements for this example

• Docker
• Dotnet
• Helm
• git

So lets start by setting up a local kubernetes cluster via docker desktop. Just go to.

Setting -> Kubernetes -> Enable kubernetes

Then create your cluster

When the setup has finished you should see a node called docker-desktop

>> kubectl get nodes
NAME             STATUS   ROLES           AGE   VERSION
docker-desktop   Ready    control-plane   14m   v1.32.2

Install helm and add ingress to the newly created kubernetes cluster.

# Install Helm if not installed
brew install helm          # macOS
# or
choco install kubernetes-helm # Windows

# Add the NGINX ingress controller repo
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Add ingress to the kubernetes cluster
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace

# Check if it is up and running
kubectl get pods -n ingress-nginx

Now lets get the API I have created for this demo

git clone https://github.com/<your-handle>/medium_stip_grpc_kubernetes.git
cd medium_stip_grpc_kubernetes

Inside you’ll find a small gRPC API (DebitCardApi) with a single contract — DebitCardService.CreatePayment — and two implementations behind it: the regular one that would talk to the backend, and the stip one that stands in. Same contract, different body. That's our Reverse Hydra.

The secret sauce is in the dependency injection logic where we check if the environment variable “STIP_Enabled” is set. So when we start the pods we it checks what role it has and injects the right datalayer and businesslogic while still respecting all interfaces.

program.cs code from the repo that has the secret sauce.

Scripts come in two flavors so this works on any machine (at least mine) :

• macOS / Linux: scripts/bash/*.sh
• Windows: scripts/powershell/*.ps1

Build the image

The scripts tag the image with the current git commit, so Kubernetes always picks up your latest build instead of a stale cache.

# macOS / Linux
./scripts/bash/build-image.sh

# Windows
.\scripts\powershell\build-image.ps1

On Docker Desktop’s Kubernetes the cluster shares your local Docker images, so there’s nothing to “load.” (On kind or minikube, run the matching `load-image-` script.)*

Deploy both bodies

# macOS / Linux
./scripts/bash/deploy.sh

# Windows
.\scripts\powershell\deploy.ps1

This installs one Helm release into the debit-card-api namespace: both deployments, the router Service, the ingress (debit-card-api.local), and a self-signed TLS cert. activeVariant starts as regular. Check it:

kubectl get pods -n debit-card-api -L debit-card-api/variant

NAME                                      READY   STATUS    VARIANT
debit-card-api-regular-...                1/1     Running   regular
debit-card-api-stip-...                   1/1     Running   stip

Both bodies are alive; the router just decides who gets the traffic.

(Optional) Point a hostname at your cluster.

Add one line to your hosts file (/etc/hosts on macOS/Linux, C:\Windows\System32\drivers\etc\hosts on Windows) so the contract address resolves locally — Docker Desktop's load balancer publishes it on

127.0.0.1  debit-card-api.local

The call-api script connects to localhost directly and just sends debit-card-api.local as the request authority, so you don't need this to follow along. But if you want to call the API with your own gRPC client (or grpcurl) using the hostname, add this so it resolves:

The moment of truth — switch the body under live traffic

Open two terminals side by side.

Left — keep calling the API (this is your stream of customers tapping cards). It loops until you press Ctrl+C and prints which body answered:

# macOS / Linux
./scripts/bash/call-api.sh

# Windows
.\scripts\powershell\call-api.ps1

--> https://debit-card-api.local  CreatePayment  every 1000ms - press Ctrl+C to stop
[   1] OK   servedBy=regular  message="Payment created"
[   2] OK   servedBy=regular  message="Payment created"
[   3] OK   servedBy=regular  message="Payment created"

Right — the backend just went down. Flip to STIP with one command:

# macOS / Linux
./scripts/bash/switch.sh stip

# Windows
.\scripts\powershell\switch.ps1 stip

Watch the left terminal keep running and quietly change body:

[   8] OK   servedBy=regular  message="Payment created"
[   9] OK   servedBy=stip     message="STIP payment created"
[  10] OK   servedBy=stip     message="STIP payment created"

No dropped calls. No client reconfiguration. The customer at the terminal never noticed. Flip back the same way when the backend recovers:

./scripts/bash/switch.sh regular

🔍 How one command moves live traffic

The trick is a router Service whose selector is templated:

*# charts/debit-card-api/templates/service.yaml selector: app.kubernetes.io/name: debit-card-api debit-card-api/variant: {{ .Values.activeVariant }} # regular | stip*

Both bodies are always running as separate Deployments, labelled variant: regular and variant: stip. The Service points at whichever label matches activeVariant.

When you run switch.sh stip, all it does is:

*helm upgrade --reuse-values --set activeVariant=stip*

That changes one line — the Service selector. Kubernetes immediately updates the Service’s EndpointSlice to the stip pod’s IP, and the nginx ingress re-resolves its upstream to that pod. New gRPC calls land on the stip body within a second or two.

Crucially, nothing is torn down. The regular pods keep running, so any call already in flight finishes against the body it started on instead of erroring out — exactly what you want when a real payment is mid-authorisation. No client ever reconnects, because the address (debit-card-api.local) and the gRPC contract never change. One head, swappable bodies.

Verify it yourself at any time:

*kubectl get svc debit-card-api -n debit-card-api \ -o jsonpath='{.spec.selector.debit-card-api/variant}'*

I’ve wanted to write this post for a long time, and I hope it was worth your time. I’ve accrued a lot of fintech wisdom over the past few years and I’m keen to share more of it — so expect follow-ups.

Thanks for reading.