Developer’s Journey of Donasi Otomatis (DONAT) V1 & V2

In the middle of Q3 2022, our team (backend user platform squad) just finished Kurban 2022 project. We plan to spend another 2 sprints to fully migrated everything from PHP to Go since this was one of our major milestone in 2022.

Little did we know there will be another project discussed in the background from as early as May 2022 that we need to develop. A big one indeed, the one that really shape the rest of 2022 and the whole year of 2023. It is called Donasi otomATis aka Donat.

A proof of history and time in our notion that the development or research related technical complexity for DONAT in mid Q3.

Before we begin, let me tell you a fun fact about donasi otomatis and why it is called donat.

the term DONAT was coined because I hate spelling “donasi otomatis” repeatedly when explaining technical term or communicating with someone else so I said “why didn’t we just make the term simpler in bahasa?” so “DONAT” was born then and let’s call it DONAT from here.

There are multiple version of DONAT but in this post I will only explain about the V1 and V2. I no longer involved in the V3 onwards since the other team takeover the project. This blog post describe the journey of how this feature being researched and implemented and what challenges we faced after the release.

Initial Requirement

These are the requirements that I can remember of for the initial version

• System must create donation based on the time (hour based) and amount the user’s choose.
• Only supported payment method is our own wallet, Kado (Kantong Donasimu)
• Not every hour is allowed to be picked. At first, we didn’t let user to pick between 23:00–01:00.
• There is a minimum amount Rp. 1.000 and the number format must be in thousand, similar to how user donate normally in web/app.
• For the first iteration, only app user will have this feature. There will be no update schedule so user can only opt-in and out on first release.
• We have to release this at the end of 2022 Q3, so we have only 2 months to ship this product (backend, mobile, qa)

The Beginning (V1)

Technically, my initial thought based on the requirement was that the first component or the trigger should definitely be a cron which will read schedule from DB. The rules is very simple

If time.Now() == user choice of schedule, then create the donation.

oversimplified diagram flow

The trigger seems to be clear enough. The next thing is to figure out how to create donation from within the system. For this part, we have to figure out how the existing system behave during donation creation with Kado.

donation creation from UI

Above diagram is the very simple flow process

• Donation : process donation creation data (donation, campaign, user)
• Payment : process payment data.
• Wallet : process wallet / kantong donasimu (balance and transaction)

This is how the flow from app / web + additional process will be involved such as checking the user is non login or login, checking the campaign is live or not and user will still need decide how much money they want to donate along the payment method choice.

If you are kitabisa users, you should already familiar with this flow 😃

However for DONAT, since it will be the system who will create the donation automatically, we as developers need less thing to check. (Yay for us I guess)

• DONAT only available for app user (login only) and support only KADO for this version. Hence, no need to worry about the non login user and payment method choice (at this time!).
• For every donation created by DONAT, it will donate to random campaign from list of campaigns populated by our “campaign program” team. The list of campaign updated every week. This allow us to skip choosing and checking the campaign is live or not (it will always be live and valid)

All seems to be clear enough for the development to start.

Trade off — Performance vs Speed of Development

The duration to do development work was less than 3 sprints (~1.5 months, include testing), so we have to cut corner. There were some concerns regarding performance since we reused a lot of existing donation creation logic + we added some stuff that we haven’t measured precisely.

In the end, since this is just the first iteration, we discussed with the products and decided that the product is “good enough”. If performance issues occurred, we will handle it in the next iteration.

Release of V1

At 24th Oktober 2022 on version 4.49.0, DONAT V1 released to Android & IOS and in the first week after that, we started to notice a problem (as always 😁)

automatic donation for daily

In the first week, the growth was higher than we expected. We expect the growth to be 50 users per day but it is actually ~250 users per day (yay for our marketing and business team I guess)

To make it worse, we never thought that majority of user will register DONAT at the same time between 4–6 AM. This is because a lot of user of Kitabisa are moslem and majority of them create DONAT which relate with the Subuh prayer, Sedekah Subuh program.

O(n) will never be good enough

In the end, we started to trace how long does it take to create 1 DONAT. We take a look at our observability platform, mainly Grafana Loki for logging. Turns out for 1 donation, it takes around 600ms end to end (from creation to send the notification). If there are 7000 users, it will take 4200 seconds which will take almost 1.5 hours.

Number of donat scheduled per 15 November 2022

600 ms isn’t too bad for normal donation creation. The problem was that the user expect the donation to be created between 4:30 AM to 5:30 AM. With current situation, the system will finish the donation process after 6:00 AM because DONAT process occurred sequentially.

There were a lot of user complaints regarding to this issue at that time. Our problem didn’t stop at that. If the cron that handle 5:00 AM still running until 6:00 AM, the DONAT scheduled at 6:00 won’t be able to run because we didn’t allow concurrent process to prevent double donation created. We also have weird bug such as panic occurred at the end of the process which make the process stuck (and require manual restart).

This is when we realized the V2 will have to start as soon as possible.

Async Version (V2)

During the research of V1, I proposed 2 approaches (the one that make sense given the amount of resource we had at that time)

• Cron which will call the donation creation service layer (the one we take for V1)
• Cron and event based process. For every schedule, system will process the donation creation asynchronously. There are complexity vs performance trade off again here but having to scale the consumer and not relying on single cron are the key factor why we decide this should be the V2 approach.

The V2 basically divide the process into 2 core processes

• Cron get the data. If run time match user schedule, publish the event to to the queue so DONAT can be created for that user.
• The consumer for this queue will start the chain to process the DONAT. Once completed, it will publish the event to another queue that will notify the user that their donation has been created automatically

oversimplification of V2 process

It is better but still not enough

Once we develop this, we load test it and it was able to achieve 100k donation creation in an hour with 10 consumers.

We tested with 100k because at this point before releasing the V2, we were processing 32k DONAT schedule for schedule 4AM & 5AM. Theoretically, if we run at 10 consumers, DONAT creation will finish in less than 30 minutes.

Number of Consumer result in faster processing, theoritically..

Fundamental Challenge of DONAT Creation

From performance perspective, you might think

why 10 consumers can only consume a total message of 25 messages per second?

It is a valid question with several explanations.

1. The consumer can scale but the overall infrastructure like DB cannot be scaled horizontally. We use CloudSQL from GCP and we cannot make it scale just between 4 AM to 6 AM. Sometimes we receive the alert that the CPU spiked through 80%++ in those peak hours because a lot of consumers bombard the DB by querying and processing update simultaneously.
2. Another fundamental problem regarding donation creation is that for getting the id of the donation, we rely on the auto increment of the database. In other words, we have to insert commit the transaction so the donation id can be used by further transaction in downstream process. At that time, we haven’t found a way to pre-allocate the donation id and do bulk insert.
3. The same case with deducting wallet. We cannot deduct your wallet before the process began. We have to check whether user balance is sufficient or not and it must be processed at that specific time before their DONAT creation begin (otherwise user might complaint why their balance missing). Checking user balance is sufficient or not actually waste a lot of time because a lot of user actually don’t have enough balance. Checking one user from API takes around 60 ms. If 30k user don’t have balance, we waste around half an hour just to check user that didn’t have enough balance.

Changing the above processes are possible but it will require non trivial effort plus refactor in several places. We need to deliver another feature for DONAT V2 so it was not like we are only dealing with performance issue.

So, how is DONAT right now?

DONAT is doing quite okay right now. The numbers of active users for DONAT already reached more than 100k.

In 2023 we released Donat Pool of Fund (PoF) which basically campaign that never ends and always accept donation. Couple months ago, we also released DONAT PWA so you can also create DONAT through web browser. We have also enhanced the DONAT engine by tuning the DB pool which increase the speed of consumer by ~33%.

There will be many new and exciting improvement to DONAT in the future so please keep using the Kitabisa APP on Play Store and Apple Store as well as PWA on Kitabisa.com 😃