Scylla to Scylla Data Migration

As a Software Engineer, its likely that you may encounter a situation where you are requested to move your resources from one cloud cluster to another in order to utilize resources more efficiently and save millions of dollars💰 for your organization. 🚀

[embed]

A database and its data migration are examples of such resources; they are essential and must eventually become consistent. In this section, I’ll go over what to do if you need to move data from one Scylla cluster keyspace to another. So, let’s say we have the following Source & Target Cluster Information and assuming that the schema is same in both the clusters and data exists in the target cluster.

Snapshot & Refresh✌️

So, here we’re going to use nodetool utility to address this problem. Its a command line interface for managing a cluster.

— — — — — — —

SOURCE NODE :

1. Tunnel into your source cluster node via command

ssh -i ~/.ssh/source_cluster_id_rsa ubuntu@10.1.2.3

2. Flush your keyspace from the memtable to SSTables on disk.

ubuntu@10.1.2.3:~$ nodetool flush company

3. Now take the snapshot of your keyspace table, here -t flag allows you to create snapshot by any name.

Note : Make sure you take snapshots in all the source nodes and name of each snapshot should be different, so that when you copy the data from source to target cluster, the files are not overridden.

ubuntu@10.1.2.3:~$ nodetool snapshot - table user company -t "snapshot-akshara-apr-29–2024"

ubuntu@10.1.2.3:~$ nodetool listsnapshots

4. Now copy the data from source cluster snapshotdirectory to local machine. The source snapshot follows the directory structure */var/lib/scylla/data/<keyspace>/<table>-<hash1>/snapshots/*

Note : This step has to be executed in all the source nodes before proceeding to the next step involved. The snapshot of all the nodes should

ubuntu@10.1.2.3:~$ exit

home :~$ scp -i ~/.ssh/source_cluster_id_rsa -r ubuntu@10.1.2.3:/var/lib/scylla/data/company/user-7da228d003e811ef8f13059d50409d03/snapshots/snapshot-akshara-apr-29–2024/ /home/akshara/snapshots

— — — — — — —

LOCAL :

5. Finally move all the data of snapshot from local machine to target upload directory as follows

home :~$ scp -i ~/.ssh/target_cluster_id_rsa -r /home/akshara/snapshot/snapshot-akshara-apr-29–2024/* ubuntu@10.2.2.3:/var/lib/scylla/data/company/user-1f659ac003ae11efa6c0715a51fe7f32/upload/

Note : Repeat for all the target nodes before moving to the next step. The path of upload directory will be different from the source directory, hence tunnel into the target cluster & copy the appropriate path.

→ You can combine step 4 & 5 and directly send your snapshot files to upload folder too using the same scp command.

— — — — — — —

DESTINATION NODE :

6. Once you have uploaded the data to appropriate directory there is a chance that the target cluster does not have proper permissions to access the files of this directory. Hence use the below command to give acces.

Note : Repeat for all the target node directories.

ubuntu@10.2.2.3:~$ sudo chown -R scylla:scylla /var/lib/scylla/data/company/user-1f659ac003ae11efa6c0715a51fe7f32/upload/

7. Lastly, execute the below command to refresh your keyspace whithout restarting your scylla nodes & verify the data by querying the db.

Note : *nodetool refresh *has to be performed in all the target nodes, else the data may not be completly accurate. Since each snapshot file holds a different set of data for each scylla node (due to varying replication), hence its adviced to run this in all nodes before you begin the data verification.

*You can check your scylla logs by running the command **journalctl *_COMM=scylla --since="5 minutes ago"**

ubuntu@10.2.2.3:~$ nodetool refresh company user -las
ubuntu@10.2.2.3:~$ cqlsh
~> select * from company.user limit 10;

In case the snapshot size is large or your cluster space is insufficient, you can compress your snapshot folder and repeat the same process. Below are the list of steps needed.

SOURCE NODE : 

1. ssh -i ~/.ssh/source_cluster_id_rsa ubuntu@10.1.2.3

2. nodetool flush company

3. nodetool snapshot --table user company -t "snapshot-akshara-compression-apr-29-2024"

4. cp -r /var/lib/scylla/data/company/user-7da228d003e811ef8f13059d50409d03/snapshots/snapshot-akshara-compression-apr-29-2024/ /home/scyllaadm/

5. cd snapshot-akshara-compression-apr-29-2024/

6. tar -zcvf snapshot.tar.gz * && md5sum snapshot.tar.gz > snapshot_checksum.md5

LOCAL : 

7. scp -i ~/.ssh/source_cluster_id_rsa -r ubuntu@10.1.2.3:/home/scyllaadm/snapshot-akshara-compression-apr-29-2024/snapshot.tar.gz  ubuntu@10.1.2.3:/home/scyllaadm/snapshot-akshara-compression-apr-29-2024/snapshot_checksum.md5 /home/akshara/snapshot

8. scp -i ~/.ssh/target_cluster_id_rsa -r /home/akshara/snapshot/ ubuntu@10.2.2.3:/home/scyllaadm/backup

DESTINATION NODE : 

9. cd backup/snapshot/

10. md5sum -c snapshot_checksum.md5

11. tar -zxvf backup/snapshot/snapshot.tar.gz -C backup/unzip/ 

12. sudo cp -r backup/unzip/* /var/lib/scylla/data/company/user-1f659ac003ae11efa6c0715a51fe7f32/upload/

13. sudo chown -R scylla:scylla /var/lib/scylla/data/company/user-1f659ac003ae11efa6c0715a51fe7f32/upload/

14. nodetool refresh company user -las

Phew!! that was a lot of steps…..

[embed]

Conclusion :

In conclusion, we successfully 🙌 migrated data for a particular timeframe (the point at which the snapshot was taken) to the target cluster. However this process is without a doubt the best technique if we have to do onetime lift & shift, but incase this has to be done regularly until the resources are decommissioned it becomes tedious. Thus Spark Migrator comes to your rescue, you can refer to the detailed document[4] for details and below table for proper ROI.

References :

[1] https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cassandra-to-scylla-migration-process.html

[2] https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/tools/toolsSnapShot.html

[3] https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/tools/toolsBulkloader.html

[4] https://www.scylladb.com/2019/03/12/deep-dive-into-the-scylla-spark-migrator/