Low Latency Java, Part I: Overview

by bless2k@gmail.com / June 20, 2024

Low latency Java refers to the designing and optimizing Java applications to achieve minimal delay in response times, often targeting response times in the sub-millisecond range. Low latency and high performance are important concepts in software engineering that, while similar, have different focuses and approaches.

• Low Latency Java, Part I: Overview
• ***Low Latency Java, Part II: Java CAS (Compare-And-Swap) Explained***

Low Latency

Low latency aims to minimize the delay between making a request and receiving a response (in a exteme way).

• Minimize Response Time
• RealTime approach(in memory data store, IPC, shared memory, etc)
• Reducing delays(such as locks, blockings, gc’s)

High Performance

High performance aims to maximize the overall efficiency and throughput(ex Transactions/Operations/Bytes in a sec).

• Throughput: Maximizing the total amount of work the system can handle over time.
• Resource Utilization: Efficient use of CPU, memory, and other system resources to achieve maximum throughput.
• Scalability: Ability to handle increased load without significant performance degradation.

Simply put, low latency aims to process a single request as quickly as possible, while high performance aims to handle as many requests as possible within the same amount of time.

Building a low latency-oriented service in Java is indeed challenging and has faced a lot of criticism. For example, garbage collection (GC) typically consumes 2 to 4 or more CPU cores and can account for 20% to 40% of performance overhead. Older GC implementations were particularly prone to causing jittering or hiccups in response times due to their ‘stop the world’ nature. Despite these challenges, there has been an increase in the use of Java for building low latency services recently. This is largely due to significant improvements in Java itself, including advancements in garbage collection (GC) and other critical areas.

In this article, we will briefly summarize recent low latency Java technologies, such as pauseless GC, off-heap memory based libraries, and high-performance messaging systems.

1. Low Latency Garbage Collectors

The potential for high-performance low-latency Java VMs was first demonstrated by Azul Systems’ JVM. It almost completely overcame the memory and latency limitations that traditional Java VMs had. Azul’s C4 (Continuously Concurrent Compacting Collector) is renowned for its pauseless GC, which allows concurrent garbage collection throughout the entire gc process, eliminating the notorious ‘stop the world’ gc pauses.

Oracle and the open-source community have developed technologies to address these challenges, notably ZGC and Shenandoah GC. Although overall performance may be slightly reduced, these garbage collectors maintain GC pauses reliably within about 1 millisecond in typical scenarios. ZGC, in particular, is continually being improved by Oracle and is likely to become the default GC for backend services in the future.

• Azul Zing JVM + C4 GC
• Oracle JVM 17 LTS + ZGC / Oracle JVM 21 LTS + Generational ZGC
• OpenJDK + (Generational ZGC or Shenandoah GC)

In terms of performance and stability, Azul’s C4 might receive the highest evaluations, but since it is a commercial product, it is more practical to use OpenJDK with ZGC. For a detailed understanding of the principles and performance of recent garbage collectors (GC), refer to the following articles:

[embed]Java Garbage Collectors, their working and comparisons anmolsehgal.medium.com

• A Performance Comparision of Modern GC’s
• https://www.uber.com/en-KR/blog/jvm-tuning-garbage-collection/

2. Low Latency Messaging Libraries

When discussing messaging systems in general Java applications, it is easy to think of message queues or remote call technologies such as JMS MQ(ActiveMQ, RabitMQ, etc), ZeroMQ, NATS, and RPC. However, when pursuing extreme low latency, libraries that support ultra-fast messaging between threads are used.

A prominent example is the LMAX Disruptor, a high-performance library that uses a ring buffer and boasts performance of millions of operations per second (ops). It is also one of the few libraries that have been backported to other languages, such as C.

[embed]LMAX Disruptor LMAX aims to be the fastest trading platform in the world. Clearly, in order to achieve this we needed to do something…lmax-exchange.github.io

LMAX Disruptor Summary

• A high-performance inter-thread messaging library.
• Uses a ring buffer to pass data between threads, which eliminates the need for traditional locking mechanisms and reduces latency.
• Thruoghput: Capable of handling millions of operations per second (ops), making it suitable for applications requiring extreme performance and low latency.
• Latency: Achieves sub-millisecond latencies, with some configurations reaching as low as microseconds.

Additionally, although it offers slightly lower performance, the Blocking Queue is also a good alternative to the Disruptor due to its ease of use and status as a standard library.

[embed]Guide to java.util.concurrent.BlockingQueue | Baeldung Overview of java.util.concurrent.BlockingQueue with a usage examplewww.baeldung.com

• https://jenkov.com/tutorials/java-util-concurrent/blockingqueue.html

3. Other Low Latency Frameworks/Methods

• OpenHFT Chronicle Queue/Map, etc
• Off Heap(Shared Memory) based frameworks
• CPU Affinity related frameworks
• GraalVM Java Native
• RealTime or low latency OS(Kernel)
• GC, Jitter(“hiccups”) monitoring tools
• Virtual Thread + Channel like frameworks (similar to go func + channel)

Summary

Examining several frameworks related to low latency reveals that they are implemented and operate in a manner quite different from general-purpose frameworks, especially those typically used with Spring Framework. Low latency frameworks are tailored for performance-critical applications where every microsecond counts. They sacrifice some of the ease of use and flexibility found in general-purpose frameworks to achieve their performance goals. This makes them distinct from frameworks like Spring, which are designed to balance performance with a wide range of features and ease of integration.

Even though the language is the same ‘Java’, low latency frameworks and applications differ significantly from typical enterprise Java programs. They require meticulous attention to detail in coding and extensive testing to ensure optimal performance.