How to import CSV/TSV to Firestore

Disclaimer: I am a Firestore beginner and I want to share my approach

Problem

My task is to create an event listing page with Firestore. I have a Google sheet file which contains a sample event database.

Approach

After digging in Firebase Admin SDK documentation, I found that I can programmatically upload my JSON file to Firestore. Before I can do that, I need to convert my data file from CSV/TSV to JSON format. Since I only need to upload data, I use Python to make the data cleaning easier.

Step 1: Process data and Convert to JSON file

From my Google sheet file, I downloaded it as TSV file, only because my data values contain commas. Processing with tabs is more straightforward in my case.

I process the file by splitting it into two variables data and headers. The first row is a guide to add event in the Google sheet, that’s why headers start at line[1].

# Read file
with open('./events.tsv', 'r') as inputfile:
    data = inputfile.read().split('\r')
    lines = data[0].split("\n")
    headers = lines[1].split('\t')

Then I process line by line, transforming data value from string to intended data type. More specifically for my database, data type for

• recurring should be boolean ;
• price should be float ;
• any array-like value should be splitted instead of a big string.

# Process data
results = []
for i, line in enumerate(lines):
    if i < 2: continue
    event = create_event_object(line, headers)
    results.append(event)

# Helper function
def create_event_object(current_line, headers):
    values = current_line.split("\t")
    object = {}
    for i, col in enumerate(headers):
        if col == "recurring":
            object[col] = True if values[i] == "TRUE" else False
        elif col == "location_types" or col == "tags":
            object[col] = [x.strip() for x in values[i].split(",")]
        elif col == "adult_price" or col == "student_price":
            object[col] = round(float(values[i]), 2) if values[i] else 0
        else:
            object[col] = values[i]
    json_object = json.dumps(object)
    return json_object

Finally, save it as a JSON file.

# Write to JSON file
with open('events.json', 'w') as outputfile:
    json.dump(results, outputfile, indent=2)

Step 2: Upload to Firestore

Setting up a Firebase project and installing Admin SDK is specified in the documentation. Follow the instructions to how you can save a service account key file (I name it as serviceAccountkey.json), then initialize it in your app.

import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore

# Initialize Firebase app
cred = credentials.Certificate("./serviceAccountKey.json")
firebase_admin.initialize_app(cred)
db = firestore.client()

Then simply insert each value into your collections.

# Upload event data
with open('events.json', 'r') as file:
    event_data = json.load(file)

    for item in event_data:
        event = json.loads(item)
        db.collection("events").document(generate_id(20)).set(event)

In here, I create my own event ID with function generate_id.

import random
import string

# Helper function
def generate_id(length):
    characters = string.ascii_letters + string.digits
    return ''.join(random.choice(characters) for _ in range(length))

Step 3: Check your Firestore console

Make sure data is uploaded correctly.

Results

Voila! You have successfully imported your data from CSV/TSV file to Firestore 🎊