Redis

Redis is an open-source in-memory data structure store, commonly used as a database, cache and message broker.

Redis stores the data in RAM, which makes it very fast compared to the traditional databases and uses disk only to persist data for durability, not for querying. Redis loads all data into memory on startup. RAM is always the source of truth while running. Periodically, Redis writes the data to disk so it can recover after a crash or a restart.

Redis provides two persistence options: RDB (Snapshotting) and AOF (Append Only File). — Snapshotting is the default mechanism that periodically saves a snapshot of the in-memory data to disk(every few seconds). There may be data loss from the last snapshot. — The Append Only File must be enabled manually. Redis logs every write operation to disk, which minimizes potential data loss. In production environments, both mechanisms are usually enabled.

Redis is a NoSQL database because it: → does not use SQL → has no fixed schema → is non-relational(no joins, tables, FKs) → stores data as key-value pairs → scales horizontally → prioritizes performance over relational constrains

Redis Data Structures

>>> Strings The maximum size of keys and values is 512 MB each.

set <key> <value>
get <key>
del <key1> <key2>
exists <key1> <key2> — returns the number of found keys

// shows all keys
keys *

// shows all keys that match a given pattern
keys h?llo 
keys h*llo 
keys h[ae]llo

// automatically populates the db with the given number of keys
// it's a very useful command for testing
debug populate <noOfKeys>

// renames a key, if the new key already exists, it will be deleted
// the deletion of the newKey may cause a high latency if the value of it is large
rename <key> <newKey>

// this command is preferred over the rename command
// renames a key only if the new key doesn't exist
renamenx <key> <newKey>

// del removes keys synchronously
// unlink removes it asynchronously
// in prod, use unlink instead of del
del <key>
unlink <key> 

// returns the type of its value
type <key>

// sets multiple keys and values, if the key exists, its value is overwritten
mset <key1> <value1> <key2> <value2> 

// gets the values of multiple keys
mget <key1> <key2>

// sets multiple keys and values, if the key exists, its value is NOT overwritten
msetnx <key1> <value1> <key2> <value2>

>>> Hashes A hashes stores field — value pairs under a single key.

Hash values can only be strings. It’s not allowed to store lists, sets or hashes in a value. Redis treats integers as strings with numeric semantic.

Small hashes are very memory efficient and scale well up to thousands of fields. Redis supports hashes with up to 4 billion fields-values pairs, but in practice a hash with millions of fields is unusual and risky.

// the structure of a hash:
key
- field1 -> value1
- field2 -> value2

// creates a hash in Redis
HSET <key1> <field1> <value1> <field2> <value2>

// returns the value of a field and key
hget <key> <field>

// returns all fields of a key
hgetall <key>

>>> Lists In Redis, a list is an ordered sequence of values and the elements are strings. A list can theoretically contain up to 4 billion elements.

// creates a list or if exists, appends the elements to the left side of the list
lpush <key> <value1> <value2>

// creates a list or if exists, appends the elements to the right side of the list
rpush <key> <value1> <value2>

// returns the first n elements of the list
// if n is -1, it returns all of them
lrange <key> 0 <n>

// returns the elements witht the given index
lindex <key> <indexOfElement>

>>> Sets A set stores unordered, unique elements and is optimized for fast membership checks and set operations. It also supports a theoretical maximum of 4 billion elements.

// creates a set or if exists, adds the elements to it
sadd <key> <element1> <element2>

// returns all elements from a set
smemebers <key>

// checks if element exists in the set
sismember <key> <element>

// removes the element from the set
srem <key> <element>

>>>Hyperloglogs Hyperloglog stores sketches which use hashing and probability to summarizes a large dataset and provide the approximate number of unique elements, using very little memory(~12 KB in Redis).

Hyperloglogs are used to count unique things. They are not 100% accurate, but they can count millions of items very efficiently. They don’t store the data itself, only the cardinality(the number of distinct elements in a set).

Use cases include counting unique page views, unique API consumers and distinct Kafka message keys.

pfadd <key> <el1> <el2> <eln>

pfcount <key>

pfmerge <newKey> <existingKey1> <existingKey2>

Redis Keys Expiration

Passive expiration — a key is expired when a client tries to access it and the key is found to have timed out.

Active expiration — Redis runs this process 10 times per second. It tests 20 random keys from the set of keys with an associated expiration time, deletes the expired keys and repeats the process if more than 25% of the sampled keys were expired.

//sets a key to expire in a given number of seconds
set <key> <value> ex <noOfSeconds> 

// checks how many seconds are left before a key expires
// if it returns a negative value, it means that the key is expired
ttl <key>

// expires a key in a given number of seconds
expire <key> <noOfSeconds>

// sets a key to expire in a given number of miliseconds
set <key> <value> px <noOfMiliSeconds>

// checks how many miliseconds are left before a key expires
// if it returns a negative value, it means that the key is expired
pttl <key>

// expires a key in a given number of miliseconds
pexpire <key> <noOfMiliSeconds>

// removes the expiration time associated with a key
persist <key>

Redis Key Spaces

They are implemented as logical databases and allows you to have same key name in multiple keyspaces.

Standalone Redis provides 16 databases by default, numbered from 0 to 15. This is configured by the databases setting: databases 16, which can be modified.

// switches to a keyspace
select <keyspace index>

// shows all keys from a keyspace
keys *

// removes all keys from a keyspace
flushdb

Redis Cluster supports only one keyspace (DB Index 0) and this is a hard limitation.

Redis Key Naming Convention

The recommended format is to concatenate the objects with colon( : ): <entity>:<id>:<attribute>

i.e. given a table order with two columns: id and items, the following keys can be used for the order with id=100: order:100 order:100:items

KESY vs SCAN

The KEYS command may severely impact performance when executed against large database. The recommended alternative is to use the SCAN command instead. SCAN iterates over the keyspace incrementally, returning keys in small batches and avoiding blocking the server.

// returns the first 10 keys by default and the next cursor
scan 0

// returns the next 10 keys 
scan <cursorReturnedByThePreviousScanCommand>

// returns the given number of keys
scan 0 count <noOfKeys>

// returns the keys that match a given pattern
scan 0 MATCH <regexPattern> COUNT <noOfKeys>

Redis Message Broker

Redis acts as a central broker and allows to publish messages to a channel and subscribe to messages from a channel.

// subscribe to a channel to receive the new published events
subscribe <channel>

// publish a messa to a channel
publish <channel> <message>

// subscribe to multiple channels using a regex pattern 
psubscribe <regexPattern>
psubscribe <regexPattern1> <regexPattern2>

// displayes the ctive channels
pubsub channels *

// returns the number of subscribers of a channel
pubsub numsub <channel>

unsubscribe <channel>

punsubscribe <regexPattern>

Redis Bulk Insertion

Pipelining is the standard way to load large volumes of data very fast by sending thousands of commands at once. Redis processes them sequentially and they are not executed in an atomic operation.

cat file_with_redis_commands.csv | redis-cli --pipe

Redis Replication

Redis replication allows Redis to copy data from one primary node (master) to one or more replicas (slaves) for high availability(in Cluster, if a primary node fails, the replica takes over its role), read scalability and crash recovery.

The master can read and write data while replicas are read-only. Replicas automatically receives updates whenever a change is made on the master. Replicas are eventually consistent and writes can be lost if the master crashes before replication occurs. Replication happens after the command is executed on the primary and in parallel with client replies.

When a master and a replica are connected, master_replid is the same on both servers. Offsets are replication position markers that indicate how much data a replica has processed from the primary.

In production environments, masters and replicas are usually configured on separate physical servers.

// create two master servers
redis-server --port 6379 --dbfilename db1.rdb
redis-server --port 6380 --dbfilename db2.rdb

// make the server with port 6380 a replica of the server with port 6379
// firstly, connect to the second server
redis-cli -p 6380

// secondly, run this command
replicaof localhost 6379

// check if the second server became a slave
info 

// provides details about servers, ports and offsets
role

// group commands attomically on the primary node
// it does not wait for the replicas
// replicas receive the commands in the same order, later depending on the network and load
MULTI
set key1 value1
set key2 value2
EXEC

Redis Cluster

Redis cluster is a solution that uses data sharding and replication to enable horizontal scalability(requests spread across nodes) and high availability(as each shard has its own replica). Redis clusters automatically shards data across multiple nodes. Each shard has one master and zero or multiple replicas. The cluster allows multiple masters and ideally the master shards should run on separate physical servers to provide fault tolerance and disaster recovery. If a master fails, one of its replicas is automatically promoted to master. If a replica fails, no other components are affected. If a majority of masters fail, the cluster will stop operating.

As a side note: Data partitioning splits data into smaller logical or physical parts, on one machine or many, to improve performance. Sharding is a form of partitioning where each partition resides on a different node and is used for horizontal scaling. Partitioning can be done horizontally(splitting by rows) or vertically(splitting by columns). Sharding is done horizontally using one of the following strategies: hash based, range based or consistent hashing.

Keys are distributed across nodes using hash slots. Redis uses 16384 hash slots. Every key is mapped to one hash slot and a hash slot can contains many keys. Each slot belongs to one primary node. Multi-key operations work only on the keys that belong to the same slot because that slot is owned by a single node.

With three primary nodes, the hash slots are allocated as follows: M1 → slots 0–5460 M2 → slots 5461–10922 M3 → slots 10923–16383

Redis cluster supports only database 0.

// node1.conf
port 7050
cluster-enabled yes
cluster-config file nodes.conf
cluster-node-timeout 5000
appendonly yes

// starts redis in background
redis-server node.conf &

// creates a cluster with 2 primary nodes and 2 replicas and allocates the slaves automatically
redis-cli --cluster <ip:port> <ip:port> <ip:port> <ip:port> --cluster-replicas 1

// connects to the first redis node, -c = cluster mode
redis-cli -c -p <port>

// creating a key on a server, may redirect us to another server 
// and store the key on that server, since the key may belong to another hash slot
set k1 v1

// useful commands that display information about cluster
redis-cli - cluster check <ip:port>
redis-cli -p <port> cluster nodes 
redis-cli -c -p <port>
// returns the node ID which is a unique identifier
// the IP of a node may change but its ID never
cluster nodes
cluster slots
cluster help
cluster info
cluster myid
cluster replicas
role

// to shutdown a node, we need to connect to it and use the shutdown command
redis-cli -c -p <port>
shutdown

// find the hash of a key
cluster keyslot <key>

// retrieve the keys from a slot
cluster getkeysinslot <slot> <count>

// to shut down a cluster, stop the slaves first and then the masters
// otherwise if the masters are stopped first, Redis will switch make the replicas masters
redis-cli -c -p <port> shutdown

// check the redis processes:
ps -ef | grep redis
redis-cli -p <port>
shutdown

// saves the keys from in memory on the disk
shutdown save

// doesn't save the new keys from in memory on the disk but the old ones remain saved on disk
shutdown nosave

RedisInsight

It’s the official GUI tool for Redis, provided by Redis. It helps users vizualize, interact and debug Redis data. It provides memory analysis for Redis, trace redis command and an intuitive CLI.

RediSearch

It’s a Redis module that adds full text search, indexing and querying capabilities to Redis. It allows creating an index on hashes and JSON data, filtering and sorting the data. RediSearch is fully supported in Redis Clusters, the indexes are sharded and queries are fan out and aggregated.

Note: RediSearch is not built into Redis core because Redis is designed to stay simple, fast and modular, and full-text search doesn’t fit the core Redis philosophy.

RedisJSON

It’s a Redis module that allows users to store, query and update JSON documents in Redis. It turns Redit into a document store similar to MongoDB, but much faster because it is an in-memory store. Documents are stored as binary data in a tree structure, allowing fast access to subfields. Each document is schema free, so it can contain different elements. The JSON properties can be retrieved without fetching the entire object.

In plain Redis, a JSON object can be stored as a serialized value in a string data type, but if you want to update a single property in a string, you must fetch the entire string value, deserialize it, modify the field, reserialize it and store it again which is inefficient.