Day 13: Restoring DVC Data on a Fresh Clone

Welcome back to the 100 Days of MLOps challenge!

In our previous labs, we successfully migrated our heavy datasets from Git to DVC and pushed them to a shared SeaweedFS remote storage bucket. We achieved the holy grail of MLOps: separating code versioning from data versioning.

But infrastructure is only useful if it enables team collaboration. Today, we look at the workflow from the perspective of a new teammate joining xFusionCorp Industries. They have cloned the repository, but their data is missing, and their attempts to pull it are failing. Let’s play the role of the senior engineer and help them diagnose and fix their setup.

The Scenario

A new team member has cloned the fraud-detection repository onto a fresh machine.

If they run ls data/raw/, they will see the transactions.csv.dvc pointer file, but the actual transactions.csv dataset is completely missing from their disk.

The DVC remote is already configured to point at the team’s SeaweedFS bucket (http://localhost:8333), and we know the data exists on the server. However, running dvc pull fails.

Our Mission: Diagnose the configuration, authenticate against SeaweedFS using the required credentials (weedadmin / weedadmin123), and successfully pull the dataset so the new teammate can start working.

Step-by-Step Solution

Step 1: Diagnose the Failure

First, navigate into the cloned project directory.

cd /root/code/fraud-detection/

If you attempt to run dvc pull right now, you will encounter an authorization error (e.g., 403 Forbidden or Credentials not found). DVC knows where to look for the data, but it doesn't have the "keys" to get in.

Let’s inspect the current DVC configuration:

cat .dvc/config

You will see the remote URL and the endpoint, but the access_key_id and secret_access_key are either missing or incorrect.

Step 2: Correct the Authentication Configuration

We need to provide DVC with the correct SeaweedFS S3 credentials. We can use the DVC CLI to securely modify the configuration file for the s3 remote.

Run the following two commands to inject the credentials:

dvc remote modify s3 access_key_id weedadmin
dvc remote modify s3 secret_access_key weedadmin123

(Note for Production: In a real-world enterprise environment, committing secrets to a shared .dvc/config tracked by Git is a security risk. You would typically use the --local flag so credentials are saved in .dvc/config.local, which is git-ignored, or rely on AWS IAM roles/environment variables. For this lab, modifying the main config fulfills the requirement.)

Step 3: Pull the Dataset

With the correct “keys” in hand, we can now ask DVC to fetch our data.

dvc pull

What happens behind the scenes?

• DVC reads the data/raw/transactions.csv.dvc file tracked by Git.
• It extracts the MD5 hash recorded inside that pointer.
• It authenticates with our SeaweedFS s3 remote.
• It locates the exact object matching that hash in the dvc-storage bucket and downloads it into the local .dvc/cache.
• It creates a hardlink (or copy) of the file in data/raw/transactions.csv so you can interact with it normally.

Step 4: Verify the Restoration

Finally, verify that the dataset is actually present and ready for use.

ls -la data/raw/

You should now see the hefty transactions.csv sitting right next to its tiny .dvc pointer. Because DVC checks the hash upon download, you can be 100% confident that this file is byte-for-byte identical to the one the original author pushed to the server.

Why This Matters for MLOps

This lab represents the complete “loop” of data versioning.

When a developer joins a traditional software team, they run git clone and npm install (or pip install). When a data scientist joins a modern MLOps team, they run:

• git clone (to get the code and DVC pointers)
• make setup (to build the environment)
• dvc pull (to fetch the precise data matching that code)

This workflow completely eliminates the “which dataset were you using?” conversation. The code and the data are permanently linked, enabling seamless collaboration across distributed teams.

Stay tuned for the next lab as we dive deeper into automation and CI/CD pipelines!

Tags: #MLOps #DVC #Git #DataEngineering #DevOps #KodeKloud