Harnessing the Power of GPT-4 for Data Science: Insights from Analyzing NYC’s Uber Dataset

Introduction

I constantly seek efficient tools and methods to streamline analysis and gain deeper insights into datasets. My recent project involved exploring a dataset of Uber rides in New York City, and for this task, I turned to an unlikely assistant: OpenAI’s GPT-4. In this article, I share my experience using GPT-4’s code interpreter to accelerate and enrich each step of my data science workflow.

The Dataset

The dataset was a straightforward compilation of Uber data, including the dispatching base number, date, active vehicles, and number of trips. Simple yet rich, it offered a window into the ebb and flow of urban mobility.

Initial Data Exploration

Utilizing GPT-4, I began with exploratory data analysis (EDA). The AI helped generate code to load the data, identify missing values, and examine data types. Notably, there were no missing values — a rarity in real-world datasets. The initial insights included the number of active vehicles and trips, both exhibiting wide-ranging distributions.

import pandas as pd

# Load the dataset
file_path = '/mnt/data/Uber-Jan-Feb-FOIL.csv'
uber_data = pd.read_csv(file_path)

# Displaying the first few rows of the dataset to understand its structure
uber_data.head()

Initial Data Exploration

Utilizing GPT-4, I began with exploratory data analysis (EDA). The AI helped generate code to load the data, identify missing values, and examine data types. Notably, there were no missing values — a rarity in real-world datasets. The initial insights included the number of active vehicles and trips, both exhibiting wide-ranging distributions.

The dataset contains the following columns:

dispatching_base_number: The identifier for the Uber base (hub) that dispatched the cars.

date: The date on which the data was recorded.

active_vehicles: The number of active vehicles for the corresponding base on that date.

trips: The number of trips made by vehicles from the corresponding base on that date.

Next steps in the EDA process will include:

· Checking for missing values.

· Examining the data types of each column.

· Getting a statistical summary of the dataset.

· Visualizing the distribution of trips and active vehicles.

· Exploring the time series aspect of the data (as it contains a date field).

Let’s proceed with these EDA steps.​

import matplotlib.pyplot as plt
import seaborn as sns

# Check for missing values and data types
missing_values = uber_data.isnull().sum()
data_types = uber_data.dtypes

# Statistical summary
statistical_summary = uber_data.describe()

# Convert 'date' from string to datetime
uber_data['date'] = pd.to_datetime(uber_data['date'])

# Plotting distributions of 'active_vehicles' and 'trips'
plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
sns.histplot(uber_data['active_vehicles'], kde=True)
plt.title('Distribution of Active Vehicles')

plt.subplot(1, 2, 2)
sns.histplot(uber_data['trips'], kde=True)
plt.title('Distribution of Trips')

plt.tight_layout()
plt.show()

missing_values, data_types, statistical_summary

Exploratory Data Analysis (EDA) Findings

Missing Values and Data Types

There are no missing values in the dataset.

The data types are as follows:

o dispatching_base_number: Object (string)

o date: Initially object, but converted to DateTime.

o active_vehicles: Integer

o trips: Integer

o Statistical Summary

o Active Vehicles:

o Mean: 1307

o Standard Deviation: 1162

o Minimum: 112

o Maximum: 4395

o Trips:

o Mean: 11667

o Standard Deviation: 10648

o Minimum: 629

o Maximum: 45858

Distributions

The distributions of both active_vehicles and trips show a wide range, indicating variability across different days and dispatching bases.

The distributions seem skewed, with more data points on the lower end.

Next Steps

With the EDA complete, we should move on to data preprocessing. This includes:

Data Cleaning: There seem to be no missing or anomalous values at first glance. However, we should further examine the dispatching_base_number for any irregularities.

Feature Engineering: We might consider extracting features from the date field, such as day of the week, month, or day part, which could influence the number of trips or active vehicles.

Data Transformation: The skewed distribution suggests that transforming the trips and active_vehicles columns (e.g., using log transformation) might be beneficial for certain types of analysis or modeling.

Let’s continue with these preprocessing steps. I’ll start by examining the dispatching_base_number for any irregularities and then move on to feature engineering and data transformation.​

Delving Deeper with GPT-4’s Assistance

Feature Engineering and Transformation

GPT-4 suggested enhancing the dataset by extracting day, month, and day of the week from the date, and applying a logarithmic transformation to normalize the skewed distributions of trips and active vehicles. These transformations were crucial for revealing underlying patterns.

import numpy as np

# Apply log transformation
uber_data['log_trips'] = np.log1p(uber_data['trips'])
uber_data['log_active_vehicles'] = np.log1p(uber_data['active_vehicles'])

# Check the results of the transformations
transformed_data_summary = uber_data[['log_trips', 'log_active_vehicles']].describe()

transformed_data_summary, base_number_counts, uber_data.head()

  (      log_trips  log_active_vehicles
 count  354.000000           354.000000
 mean     8.899730             6.771664
 std      1.053232             0.949495
 min      6.445720             4.727388
 25%      7.787174             5.696245
 50%      9.169727             6.982862
 75%      9.526044             7.257002
 max     10.733327             8.388450,
 B02512    59
 B02765    59
 B02764    59
 B02682    59
 B02617    59
 B02598    59
 Name: dispatching_base_number, dtype: int64,
   dispatching_base_number       date  active_vehicles  trips day_of_week  \
 0                  B02512 2015-01-01              190   1132    Thursday   
 1                  B02765 2015-01-01              225   1765    Thursday   
 2                  B02764 2015-01-01             3427  29421    Thursday   
 3                  B02682 2015-01-01              945   7679    Thursday   
 4                  B02617 2015-01-01             1228   9537    Thursday   

    month  day  log_trips  log_active_vehicles  
 0      1    1   7.032624             5.252273  
 1      1    1   7.476472             5.420535  
 2      1    1  10.289498             8.139732  
 3      1    1   8.946375             6.852243  
 4      1    1   9.163039             7.113956  )

Feature Engineering and Transformation

GPT-4 suggested enhancing the dataset by extracting day, month, and day of the week from the date, and applying a logarithmic transformation to normalize the skewed distributions of trips and active vehicles. These transformations were crucial for revealing underlying patterns.

Data Preprocessing Summary

Feature Engineering

Extracted features from the date field:

Day of the Week: Provides insights into weekly patterns.

Month and Day: Useful for identifying any seasonal or date-specific trends.

Data Transformation

Applied a log transformation to both trips and active_vehicles to mitigate the skewness in their distributions.

The transformed features are log_trips and log_active_vehicles.

Dispatching Base Number

The dispatching_base_number field has six unique values, each appearing 59 times. This indicates a balanced representation across different dispatching bases in the dataset.

Next Steps

Exploring Temporal Patterns: Analyzing trends over time (daily, weekly, monthly) using the newly created time-based features.

Clustering and Anomaly Detection: We can explore clustering (e.g., K-means) to find patterns among the bases or days. Anomaly detection can help identify unusual days or base activities.

Data Imputation: Currently, there seems to be no need for imputation as there are no missing values.

Let’s begin with analyzing temporal patterns to gain insights into how trips and active vehicles vary over time. We’ll look at patterns by day of the week, month, and daily trends.​

Visualizing Temporal Patterns: I created visualizations to unearth trends and patterns. For instance, line plots for daily trends and boxplots for weekly and monthly variations provided clear insights into the temporal dynamics of the data.

# Plotting temporal patterns

# Daily patterns in trips and active vehicles
plt.figure(figsize=(15, 6))

plt.subplot(2, 1, 1)
sns.lineplot(x='day', y='log_trips', data=uber_data, ci=None)
plt.title('Daily Trends in Log of Trips')

plt.subplot(2, 1, 2)
sns.lineplot(x='day', y='log_active_vehicles', data=uber_data, ci=None)
plt.title('Daily Trends in Log of Active Vehicles')

plt.tight_layout()
plt.show()

# Weekly and Monthly patterns
plt.figure(figsize=(15, 10))

plt.subplot(2, 2, 1)
sns.boxplot(x='day_of_week', y='log_trips', data=uber_data)
plt.title('Weekly Patterns in Log of Trips')

plt.subplot(2, 2, 2)
sns.boxplot(x='day_of_week', y='log_active_vehicles', data=uber_data)
plt.title('Weekly Patterns in Log of Active Vehicles')

plt.subplot(2, 2, 3)
sns.boxplot(x='month', y='log_trips', data=uber_data)
plt.title('Monthly Patterns in Log of Trips')

plt.subplot(2, 2, 4)
sns.boxplot(x='month', y='log_active_vehicles', data=uber_data)
plt.title('Monthly Patterns in Log of Active Vehicles')

plt.tight_layout()
plt.show()

Temporal Pattern Analysis

Daily Trends

Trips: There appears to be some variability in the number of trips across days of the month. This could be influenced by specific events or weekends.

Active Vehicles: The pattern is less clear for active vehicles, but there seems to be some fluctuation, possibly reflecting different levels of demand.

Weekly Patterns

Trips and Active Vehicles: The boxplots for weekly data show variation in both the number of trips and active vehicles across different days of the week. This indicates a potential dependency of trips and active vehicle counts on the day of the week.

Monthly Patterns

Since the dataset likely covers only a couple of months (given the name of the file and the scope of the data), the monthly boxplots don’t provide a clear seasonal trend. However, they do show some differences within the available months.

Advanced Data Processing

Clustering

I applied K-Means clustering, advised by GPT-4, to segment the data into meaningful groups. The AI-generated plots vividly depicted these clusters, highlighting variations in the number of trips across different days and times.

from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
import matplotlib.dates as mdates

# Selecting features for clustering
features_for_clustering = ['log_trips', 'log_active_vehicles', 'day_of_week', 'day', 'month']
clustering_data = uber_data[features_for_clustering]

# Converting 'day_of_week' to numerical values for clustering
clustering_data['day_of_week'] = clustering_data['day_of_week'].astype('category').cat.codes

# Standardizing the data
scaler = StandardScaler()
clustering_data_scaled = scaler.fit_transform(clustering_data)

# Applying KMeans clustering
kmeans = KMeans(n_clusters=4, random_state=0)
clustering_data['cluster'] = kmeans.fit_predict(clustering_data_scaled)

# Plotting the results of clustering
plt.figure(figsize=(12, 6))
sns.scatterplot(x='day', y='log_trips', hue='cluster', data=clustering_data, palette='viridis')
plt.title('Clustering of Days Based on Log of Trips and Other Features')
plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%d'))
plt.gca().xaxis.set_major_locator(mdates.DayLocator())
plt.show()

clustering_data.groupby('cluster').mean()

Clustering Results We’ve performed K-Means clustering with 4 clusters, based on the log_trips, log_active_vehicles, and the time-based features. Here are some insights: Cluster Characteristics The clusters show different average values for log_trips and log_active_vehicles. Each cluster likely represents different types of days or dispatching conditions. Visualization The scatter plot (clustered by day and log_trips) visually demonstrates these groupings. Different colors represent different clusters.

Anomaly Detection

Another fascinating phase was anomaly detection. GPT-4 assisted in coding a simple yet effective statistical method to flag days with unusually high or low activity. This step was pivotal in identifying outliers that could signify special events or data inconsistencies.

import pandas as pd
import numpy as np

# Reload the dataset since the kernel has been reset
file_path = '/mnt/data/Uber-Jan-Feb-FOIL.csv'
uber_data = pd.read_csv(file_path)
uber_data['date'] = pd.to_datetime(uber_data['date'])

# Reapply log transformation
uber_data['log_trips'] = np.log1p(uber_data['trips'])
uber_data['log_active_vehicles'] = np.log1p(uber_data['active_vehicles'])

# Anomaly Detection based on trips and active vehicles
anomaly_threshold = 2

# Calculating the mean and standard deviation
mean_trips = uber_data['log_trips'].mean()
std_trips = uber_data['log_trips'].std()

mean_active_vehicles = uber_data['log_active_vehicles'].mean()
std_active_vehicles = uber_data['log_active_vehicles'].std()

# Detecting anomalies
uber_data['trips_anomaly'] = ((uber_data['log_trips'] < (mean_trips - anomaly_threshold * std_trips)) |
                              (uber_data['log_trips'] > (mean_trips + anomaly_threshold * std_trips)))

uber_data['active_vehicles_anomaly'] = ((uber_data['log_active_vehicles'] < (mean_active_vehicles - anomaly_threshold * std_active_vehicles)) |
                                        (uber_data['log_active_vehicles'] > (mean_active_vehicles + anomaly_threshold * std_active_vehicles)))

# Summarizing anomalies
anomalies_summary = uber_data[['trips_anomaly', 'active_vehicles_anomaly']].sum()
anomalies_data = uber_data[uber_data['trips_anomaly'] | uber_data['active_vehicles_anomaly']]

anomalies_summary, anomalies_data.head()

(trips_anomaly              3
 active_vehicles_anomaly    1
 dtype: int64,
     dispatching_base_number       date  active_vehicles  trips  log_trips  \
 8                    B02512 2015-01-02              175    875   6.775366   
 18                   B02512 2015-01-04              147    791   6.674561   
 160                  B02512 2015-01-27              112    629   6.445720   

      log_active_vehicles  trips_anomaly  active_vehicles_anomaly  
 8               5.170484           True                    False  
 18              4.997212           True                    False  
 160             4.727388           True                     True  )

AutoML for Model Building:

I have used AutoML tool(Jadbio) for building models

Reflections on Using GPT-4

Accelerated Workflow

The most striking advantage of using GPT-4 was the significant time savings. What typically takes hours in coding and debugging was achieved in minutes.

Enhanced Analysis

GPT-4 didn’t just speed up the process; it also offered new perspectives and analytical techniques I hadn’t considered, enriching the overall analysis.

User Experience

Interacting with GPT-4 was seamless. The AI understood my objectives, provided relevant code, and even suggested alternative approaches. This interaction was not only efficient but also educational, enhancing my understanding of various data science techniques.

Challenges and Future Directions

While GPT-4 was a formidable tool, it wasn’t without limitations. The AI occasionally needed guidance to align with specific data science objectives, and complex, custom analyses still required a hands-on approach. Going forward, I’m excited to see how tools like GPT-4 evolve, becoming even more intuitive and adaptable to the diverse needs of data science.

Conclusion

GPT-4 has proven to be a game-changer in the way I approach data science projects. It’s an excellent tool for rapid prototyping, exploratory analysis, and even some aspects of advanced data processing. As AI tools continue to evolve, I anticipate a future where they play an integral role in every data scientist’s toolkit, transforming how we explore and interpret the vast oceans of data in our digital world.