Chapter 1: Design Data Intensive Applications (WIP)

Reliable, Scalable & Maintainable Applications

Common functionalities done by application related to data

1. Store data for later use (database)

2. Read data fast (cache)

3. Search & filter data (search indexes)

4. Send async message to other process (stream processing)

5. Periodically accumulate large amount of data (batch processing)

Is this that simple? NO, fucking NO

we use different databases for different situations, there are different caching mechanisms, several ways to apply indexes and so on… We should know what tool & concept to use in what situation. This is only possible once you have complete knowledge. This series tries to explain everything in a easy to understand way.

Lets Design a simple we application

This simple web application, is using different data systems, applied different strategies to fetch, store, modify data.

When we develop an application, we have combined different data tools together to meet our different needs

Is knowing how to combine different data tools enough? NO, fucking NO

Now, ask yourself these questions?

1. Are we able to give data to user all the time?
2. What if any Data System goes down?
3. What if user data does not fit in our data systems?
4. What if the communication with data goes down?
5. What if a data system is down & user sends us info to store in it?
6. How to improve performance?
7. How to provide consistency? even when part of system is degraded
8. How to handle sudden increase in load?

All these questions can be answered, if we give attention to 3 most important words

1. Reliability, 2. Scalability & 3. Maintainability

Reliability: The system should continue to work correctly, even in the face of adversity

Scalability: As the system grows (in volume, traffic & complexity), there should be reasonable ways of dealing with that growth

Maintainability: Over time, many different people will work on the system & they should be able to work on it productively

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Completed? No fucking NO

What do we mean by Reliability in software development?

• The application performs the function that the user expected
• It can tolerate the user making mistakes or using the softwre in unexpected ways
• Its performance is good enough for the required use case under expected load & data volume
• The system prevents any unauthorized access & abuse

In short, Reliability means, “Continuing to work correctly, even when things go wrong”

Things that can go wrong are called faults. Systems that anticipate faults & can cope with them are called fault-tolerent systems.

In order to improve our confidence in our system, we have to continuously test by injecting faults into our systems. For example: suddenly stopping an instance, passing wrong data, etc.

Though we must try to cure a fault, in some situations we should prevent fault rather than trying to cure a fault. For example: We try to prevent our system is comprimized & a hacker getting all our sensitive info.

But first we will look into how to cure various faults in our system so we can call our system is reliable:

1. Hardware Faults, 2. Software Faults, 3. Human Errors

Hardware Faults: Ram breaking down, database stopped working, suddenly power goes off, internet cable breaks, etc. These all are hardware faults.

Well, you can use reduendency to avoid these faults, like maintaining power generators as backup, redundent disks, 2–3 different internet modes.

Mostly when one goes down, we can quickly add other with minimum down time.

But if you want zero downtime, you can take software help for using redundency.

For example: if your master database goes down, automatically software should bring slave database & use it as master. Here software helps to eliminate downtime.

Software Faults: Software faults are more dangerous than hardware faults.

Atleast we can predict Hardware faults. But these software faults are unpredictable. We don’t know when a small bug present in our code gets executed and suddenly brings the whole system down.

There is no proper direct solution for software faults. We can only reduce them by properly writing tests, carefully thinking about the corner cases, process isolation, allowing processes to crash & restart, monitoring. The system should constantly check itself & raise alert when a deviation is found

Human Errors: Someone said, humans are the most dangerous species on the planet. To be honest, they are also the dangerous part in causing faults in system

Humans are unreliable. They are the weakest part in building a reliable system.

What to do now? shall we replace Humans with AI? No fucking No

You can reduce human errors by adding monitoring, fast rollbacks, testing thoroughly using unit-integration-manual-automated tests. Using sandbox environments where developers can explore & experiment safely without affecting real data of real users. And throgh proper trainings

Why do we need Scalability?

Why do we need Maintainability?

Do you think the work of software developer is completed once he complete writing code? No, fucking NO

It is more than initial development. We have to :

• Fix Bugs
• Keeping system operational
• Investigate failures
• Adapt it to new platforms
• Modify code for new use cases
• Add new features

Being a software developer it is our responsibiltyto develop a software in a way that reduces pain during maintenance. For this we have three design principles for software systems:

1. Observability, 2. Simplicity, 3. Evolvability

1. Operability: Make life easy for operations

Operations team is responsible for keeping the software system running smoothly. A good operation team is responsible for following:

• Monitor health, and restore if something goes wrong
• Tracking systems for problems such as system failures or reduced performance
• Keep software, platform & dependencies UpToDate
• Add security patches
• Keep track on dependency of one system on other & taking care of change in one system should not affect other
• Predicting future problems & solving them before they occur
• Establishing good practices & tools for deployment, configuration management
• Moving one application from one platform to another
• Maintain security of system, esp. when config changes are made
• Keeping production env stable
• Making proper documentations about system, so when one employee leave, the understanding of system should be still present for new employee

Good operability means making routine tasks easy, so team can focus their efforts on high value activities.

Data systems can do many things making routine tasks easy:

• Good monitoring for checking runtime behavior & system internals
• Support for automation & integration with standard tools
• Avoiding dependency on individual machines
• Good documentation
• Give default behavior, & option to override defaults
• Self-healing + manual control