Uber Fare and Demand Prediction, Data Analysis

Team Members : Abhijith P, Kirpali, Rishabh Kumar Pundhir

Online Taxi Services have become quite popular in recent times because of accessibility, convenience and privacy. With the huge amount of data produced everyday related to the bookings, locations and the payments, the companies use predictive analysis to gather information regarding the prices, the revenues and the demand.

Problem Statement :

This project aims to predict the estimated price of a ride and demand of a uber cab given the factors like distance of the ride, surge multipliers, pick-up and drop location, weather and wind conditions, traffic and time of the commute.

Motivation :

Whenever we booked a cab, we were fascinated to know how actually the process for predicting the fare works with respect to like distance of the ride, surge multipliers, pick-up and drop location, weather and wind conditions, traffic and time of the commute. Also cab booking is a booming industry, we were curious to know the demand of cabs on basis of source and destination locations.

Dataset :

The dataset has been collected from 26–11–2018 to 18–12–2018 from Boston city of Massachusetts in the United States. The dataset contains data of 693071 bookings done with 57 features related to time, cab type, geographical location, temperature, wind and weather conditions.

Data Preprocessing :

1. Replacing Nan values with their mean : The data has been visualized and analyzed to get the information about the features and their types. Invalid data such as zero prices has also been replaced to null values such that they can be replaced with appropriate values. The Missing values have been handled by replacing the null values with the mean.
2. Correlation between the features of the dataset with respect to price: The features correlation with our target variable i.e. ‘price’ were compared using the correlation matrix.

Correlation between various temperature features

Correlation between various weather features

Correlation between surge multiplier, distance and price

3. Mapping source and destination location names to their respective latitude and longitude : Additional features have been added to the dataset by mapping source and destination location names to their respective latitude and longitude for better results of fare and demand prediction.

Exploratory Data Analysis :

1. Plots between the rides count vs days/hours: As we need to find out the demands and price, some bar plots is as shown below for rides count on a particular day/hours.

Bar plot between rides count vs day

Bar plot between rides count vs hours

2. Scatter plot for location with respect to it’s temperature: A scatter plot was made to depict the relation with price as the hue and two dimensions being latitude and longitude of the location.

A 2D scatter plot with latitude and longitude as features

3. A distance plot for price vs distance: To calculate the price with respect to distance a distplot was plotted for determining the influence of distance feature on price.

Distplot for price vs distance

4. Scatter plot for Lyft and Uber with respect to their surge multipliers: As surge multiplier is an important feature for price prediction, so a plot was done between surge multiplier of Lyft and Uber both using a scatter plot.

Scatter plot for Uber and Lyft

Evaluation Metrics:

1. R-squared: The higher the value, the better the model. The values are taken as percentages between 0 to 1.
2. MSE (Mean Square Error): It is the average error rate which is the difference between the original value and predicted value.
3. RMSE (Root Mean Square Error): It is the square root of MSE.

Price Prediction using various predictive models:

1. Linear Regression (Baseline): Linear Regression finds out a linear relationship between explanatory variables and the target variable by giving appropriate weights to each explanatory variable.

Results for Linear Regression

2. Random Forrest Regressor (Baseline): Random Forest regressor is made up of various distinct decision trees which are collectively helpful in predicting the target value.

Results for Random Forest Regressor

3. XGBRegressor : As XGBoost is known for “Extreme Gradient Boosting” and it is an implementation of gradient boosting trees algorithm. XGBoost straight way prunes the nodes of the tree with low similarity score. Results for XGBRegressor is depicted below.

Results for XGBRegressor

4. K- Nearest Neighbors : This is a non-parametric and lazy learning algorithm. The KNN algorithm uses ‘feature similarity’ to predict the values of any new data points. The average of the values is taken to be the final prediction. Here grid search was used to calculate the best value of ‘K’ on which we perform our KNN Regressor model, the best value of ‘K’ was 39 from a list of [1 to 50]. The results after performing model are as seen below. For this ablation analysis was also used to improve our RMSE values, but there were only slight changes in the score values i.e. from 8.51 to 8.47.

Results for K- Nearest Neighbors

5. Neural Network : This model depicts a complex non-linear relationship between dependent and independent variables. Activation function used in this model for the project is ReLu. In this model Neural Network has been implemented using Sequential method API and number of epochs taken were ‘3' for getting better results. For model creation number of hidden layers used were ‘6’ and number of neurons in the input layer were ‘512’. Batch Normalization has also been used as this technique is used for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks.

Results for Neural Network

Demand Prediction using clustering method and other predictive algorithms:

For demand prediction the location features have been converted to the geological coordinates so that the distance and the area clusters could be determined. The regions are divided in terms of longitude and latitude with the help of K-Means Clustering. The count of the cabs booked from each cluster on each datetime value is grouped and we get the demand. Also the datetime is converted to new features i.e. day of week, date, month, year and a new data frame is made. The data has been standardized using MinMaxScaler so that all the features will have the exact same scale.

Models used for demand prediction:

1. Linear Regression : It finds out a linear relationship between explanatory variables and the target variable by giving appropriate weights to each explanatory variable.

Results for Linear Regression

2. Random Forest Regression : It is made up of various distinct decision trees which are collectively helpful in predicting the target value.

Results for Random Forest Regression

3. XGBREGRESSOR : It is an ensemble learning model involving training and combining individual models (known as base learners) to get a single prediction.

Results for XGBREGRESSOR

Analysis :

• If the data has some non-linear dependencies then the neural network would perform better than Linear Regression. Here the RMSE value of the neural network is less than that of Linear Regression, hence there is some non- linearity in the dataset.
• Linear Regression has outperformed KNN may be because the points are centered around the linear line, making linear regression easy to predict.
• For Neural Networks, the data needs to be scaled to a similar range.
• Even though the data is structured, Neural Network performs better than Random Forest may be because Random Forest overfit the data or NN performs better on huge data sets.

Comparison Table for different predictive models for price prediction

Comparison Table for different predictive models for demand prediction

Conclusion :

Comparison between Price Predictive Models-

• RMSE value of XGBoost was lower than RandomForestRegressor. This might be because XGBoost straight way prunes the nodes of the tree with low similarity score. Whereas in RandomForestRegressor, there is a chance of overfitting.
• RMSE value of Neural Networks was lower than XGBoost because Neural Networks performs better when the data is huge/complex.
• RMSE value of Linear Regression depends on the dataset. If there is a linear dependency between independent variables and the dependent variables, then it performs better. Hence the RMSE of Linear Regression was lower than the KNN model.
• In Demand prediction of cabs, XGBRegressor performed better than Linear and Random forest regressors due to the above mentioned reasons

Contributions :

Abhijith P : Studied Various models, implemented baseline and designed presentation by studying various research papers.(https://www.linkedin.com/in/abhijith1605p/)

Kirpali : Implemented baseline, data preparation and designed the report by studying various research papers. (https://www.linkedin.com/in/kirpali-tuli-047294209/)

Rishabh Kumar Pundhir : Implemented baseline , gathered information from various resources and designed presentation by studying various research papers.(https://www.linkedin.com/in/rishabh-pundhir-3ba124138/)

Under the guidance of :

Instructor- https://www.linkedin.com/in/tanmoy-chakraborty-89553324/

Website- *https://www.iiitd.ac.in/tanmoy*

TA’s- Abhishek Madaan, Neha Rana, Tharun Suresh, Anubhav Singh, Navneet Agarwal

TF- Shivani Kumar

References :

1. “Predictive Analysis of Taxi Fare using Machine Learning” — Pallab Banerjee, Biresh Kumar, Amarnath Singh, Priyeta Ranjan, Kunal Soni.
2. “Modeling and Analysis of Uber’s Rider Pricing” — Junzhi Chao.
3. “Data Analysis of Uber and Lyft Cab Services” — Shashank H.
4. “Predicting Short-Term Uber Demand Using Spatio-Temporal Modeling: A New York City Case Study” — Sabiheh Sadat Faghih, Abolfazl Safikhani, Bahman Moghimi, Camille Kamga.
5. “Short‑Term Prediction of Demand for Ride‑Hailing Services: A Deep Learning Approach” — Long Chen, Piyushimita Thakuriah, Konstantinos Ampountolas.