How to build a Scalable Vending System for your Bank App or Fintech Platform

Introduction

I have been working in the fintech sector as a product manager for the last 5 years, and while I have solved many problems and created system solutions to some underlying problems, the issue of vending is quite a challenging and delicate one at that. For instance, a bank I once worked with had a vending system that allowed for a 48 million naira month-on-month loss before I came. Investigating the situation made me see the gap, and this birthed this article to help other product managers working in fintech as well.

Problem Statement

One of the most overlooked yet critical features in digital banking and fintech platforms is airtime, data and bill payment vending, this is because of how quickly it can generate revenue at low cost. When it works, customers barely notice. When it fails, trust is lost and revenue takes a hit.

As a product manager at a digital bank, I encountered a recurring challenge: unreliable airtime vendors.

A traditional vending system looks simple, like the one below;

1. The customer initiates an airtime purchase.
2. The system debits the customer.
3. The airtime request is sent to a single vendor.
4. If the vendor fails, the customer is notified of the failure.

Manual Vending Flow

This system had a major flaw — no fallback mechanism. If the vendor was down (even intermittently), we lost the transaction, and more importantly, we lost the customer’s trust.

The Problem in Numbers

I have observed the following:

• Vendor downtimes occur every day.
• Telco downtime constitutes to the vendor's downtime too.
• For every 10 failed transactions, 6 customers never retried.
• Hourly revenue loss from failed airtime vending averaged ₦500,000, depending on the customer size, but assuming the total number of customers is greater than 400,000.
• CS tickets related to airtime issues made up 20% of support volume.

Hence, there is a need for a resilient, scalable and intelligent vending logic — one that prioritises successful delivery, even in unstable conditions.

Solution Consideration

To design a system that works, which I will soon describe, you need three key elements.

1. Reliable vendors: You must test and ensure your vendor’s uptime is up to 90%, at worst 80%. Anything below is a no-no!
2. Multiple vendors: Relying on one or two vendors is not a good thing to have. From experience, you need 4 reliable vendors at best or more if you have the capacity.
3. Transient Wallet Account: You need a temporary account that can help you hold funds you debit from the customers before settling the vendors.

Technical Considerations (with Product Rationale)

1. Transient GL Account

• Keeps funds safe until success is confirmed.
• Prevents financial misstatements or user refund disputes.

2. Retry Engine

• Smart queueing avoids bottlenecks and respects vendor rate limits.
• Timeout thresholds (e.g., 5 seconds) prevent slow vendor drag.

3. Vendor Switch Logic

• Vendor health metrics are stored in Redis or a similar cache.
• Automatically disable failing vendors and prioritise better ones.

Example cache structure:

jsonCopyEdit

{
  "vendor_A_failure_count": 3,
  "vendor_A_last_failure": "2025-05-13T11:15:00Z"
}

Once a vendor hits 3 failures in a short time window, it is marked temporarily unhealthy and excluded from upcoming transactions.

Proposed Solution

The proposed solution is a multi-retry system that allows for retrying every vendor until a successful vending is reached.

Here’s how I approached the redesign of the airtime vending system to achieve that.

1. Request & Debit Logic:

We begin the flow by separating financial movement from service fulfilment.

• When a customer initiates a transaction, we debit them immediately.

However, the funds are not committed to a final vendor account. Instead, they’re temporarily held in a transient GL (General Ledger).

Why does this matter?

This protects against vendor-level failure and allows us to safely retry or refund without affecting reconciliation or triggering support issues.

2. Vendor 1 routing

Once the customer has been successfully debited, the vending request is routed to the first vendor for processing.

If the first vendor fails, we trigger the multi-vendor retry engine.

How the Multi-Vendor Retry Engine Works

My userstory process flow

Completion of my user story process flow

Process flow explanation

1. If the Default Vendor is unable to vend, this triggers the multi-retry system.
2. Vending request is routed to Vendor 2 for processing
3. If vendor 2 processes the vending request successfully, the value is credited to the customer, and a success notification is sent to the customer.
4. A debit from the GL account is triggered, and a credit to the Vendor 2 income account is enacted after a value credit to the customer.
5. If Vendor 2's vending processing fails, the request is routed to Vendor 3 for processing.
6. If Vendor 3 processes it successfully, the customer is credited for the value, and a success notification is sent to the customer.
7. A debit from the GL account is triggered, and a credit to the Vendor 3 income account is enacted after a value credit to the customer.
8. If vendor 3 vending fails also, vendor 4 and so on are triggered with the same process repeat.
9. If all vendors fail, a refund is triggered — all within milliseconds, powered by asynchronous background services and message queues.

Benefits of this Retry method:

• Ensures that no customer is punished for bad vendor performance.
• Customers experience fewer failures, which directly translates to higher NPS and lower churn.
• Technical teams get fewer support tickets and have no need for manual refunds.
• No need for manual switching of vendors, also.

3. Vendor Health Tracking:

Instead of waiting for a full outage or monitoring dashboards, the system becomes proactive.

• Every vendor failure is counted.
• If 3 consecutive failures happen within a short window (e.g., 5 minutes), the system automatically stops using that vendor.
• Subsequent transactions skip that vendor and start with a healthier one.

Example Flow:

• Customer 1: Vendor A fails → fallback to Vendor B
• Customer 2: Vendor A fails → fallback to Vendor C
• Customer 3: Vendor A fails → fallback to B
• Customer 4: Vendor A is now blocked → start directly with Vendor B

Why this matters:

• You avoid repeated customer pain.
• The product dynamically adjusts to real-time vendor health, which is crucial at scale.
• Builds trust — no user wants to be the 4th failed transaction. In other words, no user wants to wait more than 1 minute to get a value or a refund.

4. Success Handling

Once a vendor finally delivers the airtime:

• The system confirms success (e.g., via webhook or delivery receipt).
• Then it moves the money from the transient GL to the vendor’s GL for final reconciliation.
• A successful response is returned to the customer.

Why this matters:

• Ensures data integrity and accurate financial reporting.
• Delivers instant gratification to the user, which is critical in micro-transactions like airtime.

Edge Cases to Consider

There are quite a so technical edges that you need to consider,r and the dynamics might be different for you.

1. Package issues:

For instance, some vendors’ payment packages may be missing in other vendors, hence, a retry approach may be tricky to implement.

Example:

Vendor 1 has 20 GB of MTN data for N200, Vendor 2 and 3 do not have that package.

What do you do?

A simple solution is to allow to vend to only that vendor for such packages. Or you can look into the pros and cons to design your own system- better still, you can send me an email (on olalemis@gmail.com) so we can look into designing a perfect solution together.

2. Vendor Success but No Delivery:

Wait for vendor confirmation (e.g., webhook or status query) before committing the funds in the GL.

3. Retry Flooding:

Space out retries or stagger vendors to avoid rate limits and ensure fair load distribution.

4. Reversed Vendor Priority:

Over time, allow the system to reorder vendor priority based on success rate, latency, or telco compatibility.

5. Delayed Vendor Response:

Use a message broker (like RabbitMQ, Kafka, or AWS SQS) to decouple request handling from UI and timeout-sensitive operations.

6. Vendor Response format:

Most vendors you will be using would not have the same response format. The best way to resolve this is to set your own format and inform these vendors to configure their own response to meet your format. This is the easier, but in a case where they can’t help, your engineers can harmonise the response in a database using 1–1 or 1–2 or even 1–3 mapping in the database.

Benefits of This Flow

1. Increased Success Rate: Fallback logic avoids unnecessary failures.
2. Improved Customer Trust: Minimises the chances of failed transactions and refund delays.
3. Lower Support Tickets: Reduces operational overhead, allowing support teams to focus on more complex issues.
4. Revenue Assurance: No revenue loss due to vendor downtime or manual interventions.

Final Word to Fintech PMs

The invisible infrastructure behind a “Buy Airtime”, “Buy Data”, or “Buy Bill Payment” button matters more than you think.

Designing smart retry logic and vendor redundancy doesn’t just save your team support headaches — it directly improves your customer retention, transactional success rate, and revenue reliability.

This kind of thinking isn’t just technical — it’s deeply product-driven.

You have got the power to revolutionize and build product that works.

Reach out to me on- olalemis@gmail.com

Click this link to buy me a coffee- https://buymeacoffee.com/solomonolalemi