Boston House Price Data Analysis

Author — Xuefeng Bai

The basic idea of the article is to construct a realistic model to precisely predict the price of houses in Boston.It is generally believed that it’s impossible to precisely predict the price of house as it depends on various factors. For this dataset price(MEDV) is the dependent variable and others are independent variables.This notebook use statistical methods like p-value, t-statistics and visualization techniques like histogram, Q-Q plot, scatter plot, box-plot of python’s matplotlib and seaborn library to find which factors will significantly affect Boston home prices. Through the data analysis and test it can be summarized that the linear regression model can effectively predict and analyze the housing price to some extent.

Here, we will be importing Boston house price data from the Kaggle. The Boston housing price dataset consists of the columns such as CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, B and LSTAT and MEDV/PRICE.

During the flow of this notebook, you will see analysis of variables affecting house prices in Boston with the help of Q-Q plot, heatmap, Liner regression, H2O AutoML and SHAP summary.

Q-Q plot to analysize each variable

Looking at the Q-Q plot above, it appears that most of independent variables are roughly following normal distribution, which is a crucial assumption for many statistical methods such as linear regression. However, there are a few variables that deviate from the diagonal line, indicating a departure from normality. In such cases, it may be necessary to transform the data or use non-parametric methods to avoid violating the normality assumption. Additionally, it is important to keep in mind that the Q-Q plot is just one method for assessing normality and should be used in conjunction with other methods such as the Shapiro-Wilk test. Overall, ensuring that the data is normally distributed is an important step in statistical analysis to ensure the validity and accuracy of the results.

The Range of all variables

1. CRIM, ZN,PRICE, B,LSTAT has outliers meaning that A small number of residential areas have higher crime rates，high property of residential land ,high house price and fewer African-Americans and more lower statue property.
2. CHAS is categorical variable only has few outliers at 1 and all others are 0 meaning that most houses in Boston are not along the Charles River.
3. RM has very even ranges and outliers meaning that most houses have only 6 to 7 rooms, but there are also almost equal numbers of houses with more than 8 rooms and less than 5 rooms.
4. RAD and TAX has more Q3 area meaning that the index of accessibility to radial highways and TAX of boston’s house price are generally high.
5. The overall data distribution of DIS and AGE is low meaning that most houses are close to the 5 Boston employment centres and most of the houses were built after 1940.

Co-relation among the variables

1. We can see there are few co-relation among the variables. The co-relation between TAX and RAD being the highest, it may mean that the index of accessibility to radial highways affects the tax of the house.
2. There is a correlation between the number of rooms and the house price.(RM-PRICE 0.7)
3. There is a correlation between the proportion of non-retail business acres per town and the nitric oxides concentration.(INDUS-NOX 0.76).
4. There is a correlation between the proportion of non-retail business acres per town and the TAX.(INDUS-TAX 0.72).
5. There is a correlation between the AGE of house and the nitric oxides concentration.(AGE-NOX 0.73).

Permutation_importance

The results here are the same as results from the above two methods of determining importance by p-value and weights,the most important variables are LSTAT,DIS and RM etc.The least important variables are INDUS and AGE etc.

H2O AutoML

A positive coefficient indicates a positive relationship between the feature and the response, where an increase in the feature corresponds with an increase in the response, while a negative coefficient represents a negative relationship between the feature and the response where an increase in the feature corresponds with a decrease in the response.

Building the Model to Identifing predictor significance(Train ,Validation and Test split)

from sklearn.model_selection import  train_test_split

X = bdata[ ['CRIM', 'ZN', 'INDUS', 'NOX', 'RM',
       'AGE', 'DIS','RAD', 'TAX', 'PTRATIO','B','LSTAT','CHAS']]

y = bdata['PRICE']

#Spliting data into Training 76.5%, Validation set 13.5% and Test set 10%

X_t, X_test, y_t, y_test = train_test_split(X, y, test_size=0.1, random_state=1)

X_train, X_val, y_train, y_val = train_test_split(X_t, y_t, test_size=0.15, random_state=1)

Spliting data into Training 76.5%, Validation set 13.5% and Test set 10%

# Make predictions using the validation set
y_pred = regr.predict(X_val)

# The mean squared error
print('Mean squared error: %.2f'% mean_squared_error(y_val, y_pred))
# The coefficient of determination: 1 is perfect prediction
print('Coefficient of determination: %.2f'% r2_score(y_val, y_pred))
r2_val = r2_score(y_val,y_pred)
print('R^2 score on validation set =',r2_val)

Making predictions using the validation set.

Prediction Accuracy

1. A high weight means that the feature has a greater influence on the prediction result, i.e., the value of the feature plays an important role in the prediction result when making the prediction.
2. The results here are the same as results from the above method of determining importance by p-value,the most important variables are LSTAT,DIS etc.The least important variables are INDUS and AGE etc.

Conclusion

Through the above methods I successfully proved that Boston house prices and RM, LSTAT and other independent variables are related, roughly linearly, and in line with linear regression.Changes in RM and LSTAT have a clear impact on Boston home prices.In conclusion, our analysis using linear regression model has revealed that there exists a significant linear relationship between Boston’s house prices and several independent variables such as the number of rooms, the proportion of non-retail business acres per town, and the percentage of lower status of the population. Moreover, we have identified that the most significant independent variable that impacts house price is the average number of rooms per dwelling. Our findings suggest that understanding the impact of various factors on house prices can help buyers and sellers make informed decisions in the Boston housing market.

Reference

• https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d
• https://medium.com/@kalia_65609/interpreting-an-nlp-model-with-lime-and-shap-834ccfa124e4#:~:text=Unlike%20LIME%2C%20SHAP%20does%20not,to%20calculate%20the%20feature%20importance.

Authors

1. Xuefeng Bai
2. Nik Bear Brown