Caching Architecture for Distributed Applications

to read in Turkish

The caching systems temporarily store data that is needed to respond to user requests faster, improve application performance, prevent bottlenecks, and reduce the load on systems.

1. The Need for Distributed Caching

The application we are developing has many different modules. Each of these modules uses its own database and the backend services of each module can be deployed separately if desired. This approach can be thought of as microservices and micro databases.

When it comes to distributed applications, different tools and technologies such as Event/Message Bus, and Rest/RPC calls are used for communication between services. A distributed application generates more network traffic than a monolithic structure, and the application's performance (throughput) is negatively affected by this. In our application, we have designed a distributed caching architecture to increase the response time of services, minimize the need for communication between services, and most importantly, reduce the load on databases.

2. Redis as a Caching Solution

Redis is an open-source, in-memory key-value NoSQL database. Redis is a strong alternative for caching because it can provide high availability and performance under load. Depending on your project and needs, you can also look at alternative products like Memcached and Hazelcast. The main factors that influenced our decision to choose Redis are:

Know-how about the product Pub/Sub capability Sharding, persistence, and replication support You can take a look here for a detailed comparison between Memcached, Hazelcast, and Redis.

3. Architecture

We have designed three basic caching methods:

• In-Memory Caching (Main Memory)
• Server Side Distributed Caching (Redis)
• Server Side Browser Client Caching (IndexedDB)

Client Side Cache on network structures such as CDN, API Gateway, and LB outside of these three are not covered in the architecture we will discuss.

We built our distributed cache on Redis. Using the Redis Pub/Sub structure, we ensured synchronization between the in-memory cached data. For the web application developed with ReactJS, we developed a caching structure on the client side using IndexedDB. To be able to clear (invalidate) the client-side cache, we provided a real-time event stream via SignalR.

We use a hybrid approach for caching and cache invalidation using LRU and LFU (Least Frequently Used) algorithmic approaches (which can be changed according to the needs of the services). These algorithms are called cache replacement policies.

LRU is a fast algorithm aiming to remove the oldest element in the cache when a new addition is made. LFU is more effective than the LRU algorithm, but relatively slower, as it removes the least used data from the cache when the capacity limit is reached.

3.1 In-Memory Caching

In-memory caching is one of the fastest and easiest caching methods. In this structure, applications reserve part of the available RAM of the server, they are running on as temporary storage space for caching. The application keeps the data it frequently accesses in this reserved space on RAM. Although RAM provides high performance and availability in terms of structure, its storage capacity is limited.

Figure 1: Server-Side Memory Caching

The above figure shows the in-memory caching flow on the backend services.

• (1, 2) The user makes a service request.
• (3) The backend service checks if the desired data is in RAM.
• (4) If the data is not in the cache, it is retrieved from the relevant database and stored in the cache.
• The relevant service uses the same data from the cache when it needs it.

The data is stored as Key-Value during storage. For us, the Key is the hash of the user’s service request body or data created according to a specific template by the developer. Value is the data needed from the database itself.

For the in-memory caching implementation, you can take a look at the in-memory package of the open-source EasyCache project.

3.2 Distributed Caching

Although in-memory caching is fast and easy, it is not sufficient for distributed applications on its own. In a distributed structure, a central caching solution is needed that provides high availability and speed between servers, has a larger capacity for caching, and can synchronize processing power and application servers.

We built our distributed caching and cache invalidation architecture on Redis. Most of our backend services are developed with .Net Core. We used StackExchange.Redis package as a Redis client. StackExchange.Redis is more advantageous than alternative libraries such as ServiceStack.Redis in terms of performance and connection management advantages and the absence of any connection limit.

Figure 2: Distributed Caching with Redis Cluster

The above figure shows our distributed caching architecture. We mentioned that our backend services check the in-memory cache before going to the database when they need data. If the relevant data is not in the server’s cache, the request is not sent directly to the database. It is checked whether the data is in the distributed cache, i.e. in Redis. This process is as follows.

• (4) A Key is generated from the incoming user request and the presence of the data in the server cache is checked.
• (5) It is checked whether there is an equivalent of this Key on Redis. If it exists on Redis, this data is retrieved and the service request is returned using this data.
• (3) If there is no data on Redis, it is retrieved from the relevant database server.
• (4) The data retrieved from the database is stored asynchronously in Redis and in memory in RAM.

3.3 Client-Side Caching

In-memory caching alone is not enough for service providers. To improve user experience, reduce data loading times, and speed up page load times, non-critical data can be stored on the client side.

When users first open our web applications, many data, such as profile information, configuration definitions, internationalization (i18n), and localization (l10n), are loaded. Even in the intranet, there is inevitably a waiting time due to the connection speed. To prevent this waiting time and reduce the number of client-side service requests, we have designed a caching structure on the browser using IndexedDB. This allowed us to reduce the load on the services.

Figure 3: Client Side (Browser) Caching

The caching structure on the client side is shown in the figure above. When the response to a client-side service request is provided by the server-side cache (In-Memory, Redis), a property (e.g. isCachable) is added to the response to indicate that it was obtained from the cache. The ReactJS application uses this property to determine which data to cache on the browser.

• (1) The user makes an API request.
• (2) A KEY is generated from the service request and the existence of the data in IndexDB is checked.
• (3) If the data is not in IndexedDB, the application makes a Rest request.
• If the backend service has added the data to the cache, a flag and version information are added to the response.
• The client application checks the flag in the service response and, if the data needs to be cached, creates a KEY and stores the data in IndexedDB.

4 Cache Invalidation

“There are only two hard things in Computer Science: cache invalidation and naming things.”

— Phil Karlton

Cache invalidation is the process of making the cache’s data consistent with the source data when the latter changes.

In this section, we will discuss the process of cleaning the cache on the server side and client side.

When there is only one application server (instance), it is easy to clean the RAM cache of the application, as it is sufficient to delete the data from the main memory using the Key. When the application runs on multiple servers (distributed application), it is complicated to clean the cache of the servers in a coordinated manner and to ensure that other services are aware of the deletion process. We use Redis’ Pub/Sub ability for cache synchronization between application servers. Redis uses the channel structure for Pub/Sub so that an invalidation operation is transmitted to all applications via Redis.

Figure 4: Client & Server Cache Invalidation

In the above figure, the payment service’s first instance is cleaning its cache, and the other instance running the same application must be aware that this data has been deleted. In the event that any of the services want to remove or change the data in the cache (In-Memory or Redis), the flow is as follows:

• (1) The payment service (Instance 1) cleans the cache on the server it runs on.
• (2) The payment service (Instance 1) sends a DEL command to Redis to clean the distributed cache.
• (3) The data is deleted on Redis, and the Del command performed is sent to all services as an event through the KeySpace Notification channel (Publisher).
• (4) All services listen to the KeySpaceNotification channel (subscriber) and process DEL, SET events, and update their caches. The events contain the changing Key and the operation (SET, DEL).
• (5) The BFF service sends an event to the SignalR Hub between the client application. This event contains a list of keys that need to be deleted or updated on the client side.
• (6) The client revokes its cache.

The above flow works successfully when the client application is open, but cleaning the cache when the application is not available is a challenging process. For this scenario, the server sends a signal to the client through an external service such as Firebase Cloud Messaging (FCM) or Apple Push Notification Service (APNS). When the client receives this signal, it connects to the application and receives the updated keys.

Figure 5: Cache Versioning and Version Validation

In the client-side cache, the cache clean-up process is the same as the server-side, but the event is triggered by the client application. In other words, if the client application determines that the cache needs to be cleaned up, it sends an event to the server through SignalR. When the server receives this event, it performs the same flow as above.

References

• https://en.wikipedia.org/wiki/Cache_replacement_policies
• https://en.wikipedia.org/wiki/Cache_invalidation
• https://docs.microsoft.com/en-US/aspnet/core/performance/caching/distributed
• https://redis.io/topics/client-side-caching

Share your thoughts on this topic with us and leave your questions in the comments section — I’m here and ready to answer!