Rehearsing for Perfection: The importance of production dry run

Introduction

Games24x7 products have seen tremendous growth of users on its platform across multiple games such as RummyCircle and My11Circle. Just to put this in perspective, My11Circle saw almost 100% growth in terms of liquidity on a year to year basis. Some of our microservices saw almost 300% growth in terms of requests per second. What this entails for us is to have a very robust load testing framework for each of the microservices.

In this post, we will discuss why we started conducting load testing in a production environment, which we internally refer to as “Production Dry Run.”

The production dry run activity has become a critical step in our software development journey, especially for events like the IPL and the World Cup. These events are high-profile, with millions of viewers and users accessing the system simultaneously, and any failure or glitch can lead to bad user experience. In this blog, we will discuss some of our learnings while doing load testing and the importance of production dry run activity and how it helped us ensure smooth and successful events in the recent past.

Load Testing @ Games24x7

Until now, at Games24x7, we have been conducting load testing in a separate AWS environment, which we internally refer to as the Performance Testing (PT) environment.We prefer Locust as the load testing framework primarily for the following reasons:

• Python: Locust tests scripts are written in Python. Python is a concise language which is easy to learn and understand.
• Distributed: Locust is designed to be distributed. The kind of loads we run are not possible from a single machine and hence its distributed nature allows us to generate load over multiple machines.
• Event Based: Locust is event based using gevent. Being event based ensures that our load generation machines are fully utilized since most of our load consists of I/O (Rest API and database access).

There are locust scripts for each of our services, which tests the core services by simulating the desired amount of traffic through the locust load test machines. These scripts are triggered through Jenkins pipeline which generate required prerequisite data, serve as mock for any dependent services and also launch the load testing machines in the PT environment.

Need for Production Dry Run

Despite performing these tests successfully in PT environment, we have seen outages in production happening because of following reasons:

• Tests in PT environments are confined to stress testing few components in isolation where other components are bypassed.
• Many times, even if other components are not bypassed, they are mocked. This results in differences in terms of infra, data, configurations, etc
• There are cost / execution challenges to replicate a full production like environment in PT environment in order to perform the stress test.
• We want to make sure that our infrastructure hosting the big events is ready & can bear the load, Basically avoiding any risk during live matches, this is very similar to the prod sanity exercise that we do for any new feature release.

That is when we decided to conduct a production dry run going forward prior to any major events on our platform.

A production dry run is a process that involves running the load in a production environment, which is identical to the production environment. The simulation includes all the software, hardware, and network components that are involved in running the system during the event. The purpose of this activity is to identify and fix any issues or bugs that may occur during the event.The production dry run is usually conducted a few weeks before the event to ensure that the system is stable and ready to handle the expected load. This activity involves a series of tests that are performed to evaluate the performance, functionality, and scalability of the system.

Dry Run Prerequisites

Generating significant load in the production environment is not as straightforward as doing it in a non-production environment as any outage / service degradation can cause the production infrastructure to go down which will end up impacting real end users.

In addition to that, if business and product teams are dependent upon the metrics, reports for making critical product decisions, it is imperative to ensure that the production dry run activity doesn’t impact those reports.

In short, for a dry run, it is important to work with business stakeholders, business intelligence teams to devise a plan which will ensure that the activity doesn’t impact the “Business as usual (BAU)”.

Some of the key decisions we took as part of these prerequisites:

• DryRun user attribute and dry run users: Since we wanted to perform the end to end user scenarios during our test run, we introduced dummy dry run users in our system which were tagged with “dry_run” attribute.
• Filter dry run transactions during report generations: Since we didn’t want any of the business reports to get tainted due to these test runs, it was ensured that these reports filtered any transactions generated by dry run users.
• Custom Visibility: For a product like ours, where a lot of users participate together in a contest, it is imperative that dry run users are not getting mixed with real users. We introduced the concept of “custom visibility” where any entity can be tagged to a cohort of users, and it remains visible to only those users.
• Feature flags: Feature flag is not a new concept in software development. However, we extended it to include all major asynchronous flows in our systems to have the ability to exclude processing of dry run users and their transactions. This is because these flows are sometimes time intensive and resource intensive. Any unforeseen issue or delay in these flows can quickly impact the real users. We wanted to make sure we have the necessary controls to filter out these dry run transactions in case of any issue.

Production Dry Run CheckList

Here are some key steps involved in the production dry run activity:

• Define the scope: The first step is to define the scope of the production dry run. This includes identifying the key features and functionalities of the software, the expected traffic load, and the types of scenarios that will be tested during the dry run. For example, login / registration, contest joins and settlement are some of the examples.
• Plan the dry run: Based on the scope, a detailed plan is created, including the timeline, resources, and tasks involved in the dry run. This plan should be communicated to all stakeholders, including the development team, operations team.
• Prepare the infrastructure: The next step is to prepare the infrastructure for the dry run. The infrastructure should be designed to handle peak loads and unexpected failures.

1. Hardware and Network Infrastructure: The first step is to ensure that the hardware infrastructure is set up correctly. This includes servers, storage, and networking equipment. Ensure that the hardware is powerful enough to handle the expected load and that the network infrastructure is robust enough to handle the data traffic.
2. Middleware Infrastructure: The next step is to set up the middleware infrastructure required for the load to run. This includes the operating system, database servers, application servers, web servers, and other tools that are required to run the load.
3. Load Testing Infrastructure: To simulate real-world conditions, it’s important to set up load testing infrastructure. This includes tools that can generate load on the system, simulating a large number of concurrent users or requests.
4. Performance Testing Infrastructure: Set up performance testing infrastructure to test the system’s performance under different scenarios. This includes load testing, stress testing, and capacity testing to ensure that the system performs as expected under different conditions.

• Conduct the dry run: Once the infrastructure is ready, the dry run can be conducted. The testing should be performed under realistic conditions to simulate the actual event. The team should monitor the performance of the system, identify any issues, and take corrective actions as necessary.
• Monitoring: To monitor the systems performance during the production dry run, it’s essential to set up monitoring and logging infrastructure. This infrastructure should include tools that can monitor the system’s performance, such as CPU and memory usage, network traffic, and response times.
• Analyze the results: After the dry run, the team should analyze the results and identify any issues or areas for improvement. This includes reviewing the system logs,identifying any bottlenecks, and measuring the system’s performance against the expected metrics.
• Implement fixes and improvements: Based on the results of the dry run, the team should implement fixes and improvements to the system. This may include upgrading the hardware, optimizing the code, or improving the infrastructure.
• Repeat the dry run: Once the fixes and improvements are implemented, the dry run should be repeated to ensure that the system is functioning correctly. This may be repeated several times until the team is confident that the system is ready for the actual event.

Impact of our first Dry Run

We conducted our dry run a couple of weeks before the IPL and we could immediately feel the impact of it.

• Identifying potential issues early: By testing the systems production environment, we identified a few issues which were not covered in testing(PT) environments. This enables us to address these issues before the feature is released to the public, reducing the risk of bad user experience occurring during the event.
• Reusable Framework: Once the dry run users are created and all our systems can identify and filter these transactions in reports, it leaves us with a powerful reusable framework to perform dry run of further scenarios in production.
• Improving the user experience: A production dry run helped us ensure that the system is stable and user-friendly. This improved the overall user experience, reducing frustration and ensuring that users can easily access the platform and perform the desired actions.
• Saving time and resources: By identifying and addressing potential issues early, a production dry run helped us save time and resources. It prevented the need for emergency fixes or updates during the event, allowing us to focus on providing support and enhancing the user experience.
• Reducing Downtime: Downtime during a live event can be catastrophic, leading to loss of revenue, user dissatisfaction, and brand damage. The production dry run activity helped us ensure that the systems are up and running smoothly during the live event, reducing the chances of downtime.

Conclusion

To sum it up, a production dry run is a vital part of the software development process, particularly when it comes to major events like the IPL or World Cup. It involves testing the system in a production environment, without actually launching it to the public, to identify any potential issues or bugs that could impact its performance or functionality.Ultimately, investing time and resources in a production dry run can help ensure that the system performs as expected during the event, reducing the risk of major problems occurring and providing a better user experience for all involved.

About the Author

Sanket Dharia is working as SDET3 at Games24x7. He has around 13 years of experience. He has experience with manual and automation testing with strong knowledge on QA practices and methodologies including, Web services testing, Data migration testing, System testing, functional automation etc…