Kaizen - Continuous Improvement + Observability

Achieving Continuous Feedback in Software Development

In the software development industry, we use the phrase "Continuous XYZ" in different contexts, such as continuous integration, delivery, deployment, and more. In this article, I will focus on Continuous Feedback and its conjunction with Observability and the Kaizen concept.

🟧 Not a Medium member? Read this article for free.

· What is Kaizen? · Iterative approach in Software development · From Kaizen to Continuous Feedback ∘ Absence of Kaizen in observability ∘ What is continuous feedback? ∘ A practical example of Continuous Feedback ∘ Digma: A continuous feedback tool · Final Thoughts

What is Kaizen?

**Kaizen is a Japanese philosophy that is applicable to different aspects of our lives. To put it simply, Kaizen means “change for better” in the Japanese language and is often used as a synonym for Continuous Improvement**.

Kaizen is a culture that can help us boost our productivity and quality by improving functions and processes gradually in small steps instead of in a big bang. Kaizen has some key principles:

• Engage everyone in the process.
• Focus on small changes.
• Continuous learning and improvement.

Although the Kaizen technique started to be applied in the manufacturing industry, it can be effectively applied to non-manufacturing industries such as software development.

The PDCA cycles

Iterative approach in Software development

For years, the software industry has used an iterative approach. After discarding the waterfall method in the Software Development Process, Agile approaches replaced it with flexible and rapid iterations.

Waterfall vs Agile

We accept that changes in requirements are a fact in software development.

Using an iterative approach, we can implement changes early and frequently. Then, by getting feedback from the users (stakeholders), we can improve our product quickly and achieve high-quality software.

By profing the value of iterative software development, nowadays, we use this concept in every step of the development, not only to manage the process. For example, Continuous Integration (CI) is another proven practice based on the iterative concepts that enables us to integrate code changes frequently by automating the build and test processes.

From Kaizen to Continuous Feedback

In the previous section we saw that Kaizen or continuous improvement is an inseparable part of the software production process. Whether we know it or not, nowadays in the modern software production practices we use the Kaizen concept in each steps.

Absence of Kaizen in observability

Observability is an important part of software systems, especially these days, with the increasing popularity of microservices. We can get valuable feedback about system behavior at runtime, but traditionally, in the software development process, we postpone getting this kind of feedback until deployment time (at least in the first environment, such as test or staging).

If we can find a way to use Observability data as an “early” feedback source in our development process, then we will implement Kaizen (continuous improvement) in our development process.

Development Process without Continuous Feedback

What is continuous feedback?

By fusing Observability concepts and tools with the Kaizen philosophy, we can gain several benefits in our software development process. As I mentioned before, since Observability data is collected at application runtime and is in the category of runtime analysis approaches, we can find many issues and problems early in software development that static analysis tools can not. This practice is called **Continuous Feedback**.

You can read the Continuous Feedback Manifesto from this link to learn more about it. But In a nutshell:

Continuous Feedback is a new development practice that completes the DevOps loop, enabling developers to make shorter feedback iterations.

Continuous Feedback

A practical example of Continuous Feedback

Imagine we are implementing a database-centric application that has many interactions with the database, and we also use **ORM** in our database layer to communicate with the database in object oriented way.

One of the common problems in this kind of application is slow queries, which makes our application slow. “**N+1 query problem” is a typical example of this kind of slow query when we use ORMs in our application. You can read this article to learn more about “[What is the N+1 query problem and how to detect it?](https://digma.ai/n1-query-problem-and-how-to-detect-it/)**”:

[embed]What is the N+1 query problem and how to detect it? What is the N+1 query problem? How to detect and mitigate its effects on application performance: using a practical…digma.ai

Continuous Feedback will help us detect and mitigate this kind of slow query problem during development instead of waiting for a production issue.

Development Process with Continuous Feedback

As you can see in the image above, using Observability as a feedback source during development will make our development process more agile and shorten the feedback loop.

Digma: A continuous feedback tool

If you are interested in the Continuous Feedback practice so far, the good news is that **Digma** is the first tool that allows us to implement this concept in our software development process.

**Digma** uses a bunch of tools and standards in the observability world, such as OpenTelemetry, Jaeger, and more, to implement a pipeline for Continuous Feedback practice.

Continuous Feedback Pipeline in Digma

Digma allows the Continuous Feedback practice to happen automatically. In addition to being able to integrate Digma with CI and our deployment environments (test, stating, or even prod), Digma has an **IDE plugin **that communicates with Digma infrastructure (can run locally or on other environments) and shows us meaningful insights about the code and its runtime behavior which can help us as developers to find issues during the development just inside the IDE without need to deploy the code to any environment.

For example, in the Digma view in IntelliJ IDEA, we can see a lot of insights about our application, including detailed descriptions and direct connection to the line of code.

Digma insight view

Or we can see the recent calls for our application endpoints in the Observability view in IntelliJ IDEA:

Digma insights in Observability view

We can see traces for each call in detail in a new window by clicking on the Trace button:

Digma Tracing view in IntelliJ

There is also a **Udemy course** on continuous feedback concepts and Digma. You can check it out if you are interested in learning more about this topic.

Step up your coding with Continuous Feedback

Final Thoughts

By integrating the principles of Kaizen, Observability, and Continuous Feedback into our software development process, we can create a culture of continuous improvement, early issue detection, and rapid iteration. Using these concepts not only enhances the quality and performance of our software but also cultivates a more agile and responsive development environment.

📕 Read more related articles ⬇️

[embed]List: 🔎 Observability Series | Curated by Saeed Zarinfam | Medium 8 stories · This series of articles is related to the Observability concepts and tools, mainly about OpenTelemetry and…zarinfam.medium.com

🙏 Thanks for reading. You can connect with me on:

**🖊️ Medium | [🐦 Twitter](https://twitter.com/zarinfam)**