Day 05 of MLOps: Data Versioning with DVC, Git, and Amazon S3

Introduction

In traditional software engineering, Git helps us version control source code efficiently.

But in Machine Learning systems, datasets continuously evolve, and Git alone is not designed to handle large-scale data versioning efficiently.

This is where DVC comes into the picture.

DVC helps track datasets, machine learning artefacts, and data pipelines while integrating seamlessly with Git and remote storage solutions like Amazon S3.

In this blog, we will understand why data versioning matters and learn how to use DVC with Git and S3 in a practical workflow.

Objective

• Understand why data versioning is important
• Learn the limitations of Git for datasets
• Understand how DVC works internally
• Configure DVC with Amazon S3
• Push and version datasets using DVC
• Understand how Git, DVC, and S3 work together

What is Data Versioning?

In Machine Learning and Data Engineering, data is one of the most important components because ML models are trained using datasets.

However, datasets keep changing over time:

Datasets continuously evolve as records are added, updated, or removed over time.

Because of these continuous changes, tracking dataset versions becomes very important. This is where Data Versioning comes into the picture.

You can think of it like Git versioning for code, but here we are versioning datasets instead of source code. One of the primary goals of DVC is reproducibility, along with dataset tracking, collaboration, and ML pipeline management.

Why Not Use Git for Datasets?

At first, it may sound simple to store datasets directly in Git, but Git is not designed for handling large datasets.

There are multiple reasons for this:

• Large files increase the repository size very quickly
• Git becomes slower with huge binary files
• Storage cost increases significantly
• Operations like clone, pull, and push become inefficient
• Image, video, and structured datasets are usually very large in size

Because of these limitations, Git is not an ideal solution for storing datasets.

Where Should We Store Datasets?

Instead of Git, datasets are usually stored in object storage services such as:

• Amazon S3
• Azure Blob Storage
• Google Cloud Storage

These storage services are designed to handle large-scale data with:

• high durability
• high availability
• better scalability
• faster transfer rates
• lower latency

Since these services are cloud-based, datasets are accessible from anywhere and are not dependent on local machines.

What is DVC (Data Version Control)?

DVC is a tool that helps us version control datasets and machine learning artifacts.

It works alongside Git and helps track datasets stored in remote storage services like S3 instead of storing the actual data inside Git repositories.

This makes dataset tracking efficient and scalable.

If someone modifies the dataset, DVC helps us identify:

• what changed
• Which dataset version is being used
• whether the dataset is old or updated

How DVC Works

DVC works somewhat similarly to Git.

In Git, we usually do:

• init
• add
• commit
• push

In DVC, the workflow is also similar, but instead of versioning code, we version datasets.

When a dataset is added or updated, DVC generates a unique checksum (hash) for that dataset.

This checksum helps identify:

• the exact dataset version
• whether the dataset changed
• Which dataset belongs to which experiment or model

Checksums help guarantee dataset integrity and uniquely identify dataset states.

Instead of storing the actual dataset in Git, DVC stores lightweight metadata files and configuration files.

DVC creates .dvc files that contain metadata and checksums.

The actual dataset gets stored in remote storage like S3.

Role of Git in DVC

One important thing to understand is that Git still plays a role here.

Git does not store the datasets themselves.

Instead, Git stores:

• DVC configuration files
• metadata
• dataset tracking information
• remote storage configuration

For example, Git can track:

• where datasets are stored
• whether the remote location is S3
• dataset references and versions

So in simple words:

• Git tracks metadata and configuration
• DVC tracks datasets
• S3 stores the actual data

That’s how all three work together.

In this demo, we will be doing DVCS with Git and pushing our datasets to S3 as a remote. Also, we will push multiple datasets to understand how versioning works in DVC and the role of Git in DVC.

HandsOn

As usual, we will activate the Python virtual environment

python3 -m venv .venv 
source .venv/bin/activate
which python3.12

Now, we will create a separate folder for the DVC demo called wine_prediction_dvc_s3_demo

mkdir wine_prediction_dvc_s3_demo

Now, we will initialise DVC in our newly created folder. But it won’t work

As we did not install the DVC utility on our machine, it is also not present in our Python virtual environment. So, we will install the dvc utility using the command below.

python3 -m pip install dvc

Now, initialise the dvc and use the subdir flag if your .git is not present in the current directory.

dvc init --subdir

Here is our dataset, which is in a .csv file, and all the datasets are present in the data folder

Now, we will add the dataset by using the command below and confirm the status.

P.S.: When using dvc add, DVC automatically updates .gitignore

dvc add data/wine_sample_dataset.csv
dvc status

Before pushing the dataset to DVC, we need to create an S3 bucket on AWS. If you are using any other cloud, feel free to do it. Also, make sure to configure AWS credentials on your machine to push the datasets to the S3 bucket.

Now, we will add an S3 bucket as an origin. So, it will push objects/datasets to a specific S3 bucket.

dvc remote add -d mlops-dvcs3 s3://amzn-mlops-dvcs3-demo
dvc remote list

Now, we are ready to push our dataset. But we will get one more error if we directly push the datasets. If you see the snippet below, it is saying No module present named ‘dvc_s3’. To push the dataset to S3, we need to install one more dependency. Use the command below to install the dependency and push the datasets.

python3 -m pip install dvc_s3
dvc push

After pushing the dataset to S3, navigate to the AWS Console and the S3 bucket to check whether the dataset has been pushed or not. In our case, we can see the datasets have been pushed.

You can also download the dataset and view its content. It would be the same as we pushed with the correct checksum, as you can see in the snippet.

Now, let's modify some data, and that will change the checksums and file as well to version the new updated datasets

Now, push the updated datasets.

Here you can see the checksum name is updated, which creates a new folder, and in that folder, it will have a new datasets file.

Again, you can view the content of the pushed datasets in the S3 bucket.

Now, we will do a git commit to push the DVC config that holds the metadata of the DVC

Here is the .dvc/config file thats is holding the metadata of DVC, such as remote URL, autostage, etc.

0

In the data folder on GitHub, you can see the .dvc file followed by dataset.csv that holds the info such as checksum, size of the datasets, hash, and path/filename of the datasets. So both files are very important to track the datasets used to train the model.

Conclusion

In this blog, we explored one of the most important concepts in MLOps: data versioning.

We learned why Git alone is not suitable for large datasets and how DVC helps manage datasets efficiently while keeping Git repositories lightweight.

We also configured Amazon S3 as remote storage and understood how dataset changes create new dataset versions using checksums and metadata tracking.

As we continue this journey, these concepts will become even more important when we start building reproducible ML pipelines and experiment tracking systems.