Stream Join Functionality for Low Latency Systems

1. Goals

The upstream data is highly normalized which means that different data streams are being produced by different sources which needs to be merged to arrive at derived attributes, which are used in the downstream pipeline.

When the join is being performed, the join keys can be different when the join is being performed between different streams

The join need to be performed between 1 stream which will be the deriving stream or main stream and side stream(s) which can be more than 1 in number. The join key with each of the side stream can be same or different.

The side stream data can be mandatory meaning thereby that if the side stream is not arrived , the output join will not be performed or

The side stream can be optional meaning that even if the side stream has not arrived, the output join data will be sent to downstream channel. Whenever the side stream data arrives, the output join will be again performed and then the downstream channel will receive it as a new event.

The join functionality should be able to generate corresponding events for each event happening at the source irrespective of whether the event is out of order event.

2. Proposed Solution Design

2.1 Solution Design when all joins are mandatory

Figure 1: Proposed Solution Design for performing Join (All Mandatory joins)

Step 1: Branching Out from the Main Stream

· The process begins with Source C as the main stream. Incoming events from Source C are used as the basis for joining with other streams.

Step 2: Joining and State Management

· For each event from Source C, the system attempts to join it with related events from Source A, and B using the specified join Key. The join with each of the sources, source A and source B are done independently and separately.

· The source A and source B being the side sources are maintained in a value store, meaning that only 1 record is maintained for 1 joinKey from each of the sources in the state store and main source is maintained in list store meaning that there will be multiple records of 1 join key in the state Store.

· The time to retain the data in the state Store is specified through the ttl after which the data is cleaned.

· The join result(s) thus obtained are merged using the union ID.

Step 3: Merging join Result using Union ID

· A FlatMapFunction is used to process each join Result, and a state store (ValueState) keeps track of the final join results for each union ID.

· When a new event arrives, the system retrieves the previous state, updates the join result, and checks if all required fields from A, B, C are present.

Step 4: Publishing of final Result

· When all necessary streams have been joined and the result is fully enriched, the final merged event is published to a sink topic.

· If any data conversion is needed before publishing, a MapFunction is used to transform the data.

2.2 Solution Design when all one source is Optional

Step 1: Sources and Keys

· The architecture starts with three sources: Source C (main stream), Source A (mandatory join key: Key1), and Source B (optional join key: Key2).

· Source C branches out to join with both Source A and Source B.

Step 2: Join Operations

· Join 1: Source C and Source A are joined using Key1. The output is stored and can be unioned with other join outputs.

· Join 2: Source C and Source B are joined using Key2 (if available). The output is stored in the Value Store.

Step 3: Union and Output

• The results of the join operations are unioned based on a union ID (e.g., Union Id: X).
• If Join 2 data has not arrived, only the result of Join 1 is sent to the sink topic.
• If Join 2 data arrives, the result is retrieved from the Value Store and sent to the sink topic.

Step 4: Sink Topic

· The final joined and unioned results are published to sink topics for downstream processing.

3. Alternative Options

3.1 Using Multilevel Join, with a single Join Key

Solution Description

· If the join key is the same to join source A, Source B and Source C, then join the data using the key at level 1.

· The joining between source A and Source D, is done using the key 2.

· The output thus generated from the sources at level 1 is joined using the Key 3 to produce the final output.

Limitations of the Solution

· When different join keys are used for the same source, it is difficult to find the right partition of the kafka Topic and will be difficult to implement this solution.

· If the data is late arriving from one of the topic, the whole event needs to wait until the late arriving event has arrived. This will introduce a significant latency in ingestion of the data or otherwise the late arriving events scenario cannot be handled

· The solution will have no option to make any topic as optional topic. The data from each of the event will have to be mandatory.

· Using multi level joins, will increase the latency in the output of the event.

3.2 Using Union Merge

Solution Description

· The keys are extracted from the input source topics

· It is checked if the keys have arrived from all the mandatory topics, then the merge process will be carried out in the convertor function

· The output thus generated will be published

Limitations of the Solution

· When different join keys are used for the same source, it is impossible to perform the join using the above architecture.

· If the data is late arriving from one of the topic, the whole event needs to wait until the late arriving event has arrived. This will introduce a significant latency in ingestion of the data or otherwise the late arriving events scenario cannot be handle

· The solution will have no option to make any topic as optional topic. The data from each of the event will have to be mandatory.

4. Non Functional Requirements

• Performance is optimized by increasing job parallelism and adjusting memory settings as needed.
• The solution can handle out of order events/late arrving events without introducing latency in processing
• The solution is a generic solution which can handle join streams Data durability