Exploring Sequin (Part 1)

Sequin is a tool that we stumbled upon as we wanted to mirror data from our Postgres to Redis. We considered Debezium, but found that Sequin was simpler to setup and easy to use for us in this particular scenario.

Basic Architecture

Fig: Simple Flow Diagram

Source: This is where the data resides and we want to get that data and put it into a sink.

Sink: This is where the data would land into, this may or may not be a 1:1 copy of the data from the source. This can change if we apply changes withing Sequin using functions.

Sequin offers 4 different types of functions to perform SMT (Single Message Transforms) out of the box which fulfills our needs perfectly.

• Routing
• Transform
• Filter
• Enrichment

All of the above functions follow the pattern shown below:

def transform(action, record, changes, metadata) do
 # Your Code goes here
end

The function above is an Elixir function. If Elixir is new to you, there is the official documentation here or one of the free AI models can definitely give you some knowledge about the basics.

In addition, to these function Sequin also offers backfill which can be triggered from the UI or YAML config to fill in older data.

The Sequin UI The UI is pretty good and can do everything we would need for a CDC (Change Data Capture) connector but the idea is to explore a more real life scenario that is easily deployable to a Kubernetes cluster.

Figure 1: Sequin UI

Going through the important bits in the UI.

1. Sinks: This is where one can configure the sinks. All of the sinks available at the time of writing are as follows shown in Figure 2.
2. Databases: This is where you can configure the database connection to Postgres.
3. Functions: This is where one can write the SMT functions for the four different types of transformation listed above.

Figure 2: Sequin Sinks

Regardless of using the UI or a YAML configuration there are some requirements to get it working.

1. Write-Ahead Logging (WAL) level needs to be changed to logical. One can use the following commands to change the WAL level and also verify it.

ALTER SYSTEM SET wal_level = logical;
SHOW wal_level;

2. Creating publication needs to be taken care of using SQL statement if using the UI

CREATE PUBLICATION sequin_pub FOR TABLE users;

3. Creating replication slot also needs to be manual with the following SQL statement when using the UI

SELECT pg_create_logical_replication_slot('sequin_slot', 'pgoutput');

Connecting to Postgres using the UI

Upon clicking Databases and adding a new connection, we are presented with a screen that looks as in Figure 3. We need to fill in the required field and connect to the database and voila, sequin can now connect to Postgres.

Figure 3: Adding a Postgres Connection

Redis String Sink

Using the UI we can create a new Redis String sink which presents a pop up similar to what we see in Figure 4. Here, Sequin fetches the schema metadata and table metadata using the earlier Postgres connection and we configure what events we listen to. Functions can be created also in the UI and be used for different steps. Redis connection info needs to be filled further down in the pop-up which enables the sink.

Figure 4: Redis String Sink Configuration

This concludes the first part of the article where I explain a simple and practical way to get started with Sequin. In the second part of the article I will focus on the infrastructure as code, which is what we will want to use in a production environment in order to automate things and keep minimal changes via the UI.