My First ML Competition

I started my ML Competitions with the titanic dataset from kaggle

Explorative data analysis (EDA) Initially I checked with the columns of the dataset and it’s datatypes, to know what columns are useful for our training and if there’s any chance of getting some new information or new columns from the existing columns. Even tried plotting some columns to check the importance of features or detect any patterns in the dataset.

Now, checked for missing values in the dataset by using isnull() function and found that there were missing values of age, cabin and embarked columns. I Imputed the age column using median to take the missing values as the average of the dataset and embarked column with mode of the data to take more repeated value as the missing value because taking median here doesn’t make any sense and we dropped the cabin column as the percentage of missing values is very high.

Feature Engineering As sex column is non-numeric, I have mapped male as 0 and female as 1 to make the column available for training the model. The embarked column also has 3 entries as Q,S,C so I did one hot encoding for the column which resulted in two boolean columns Embarked_Q and Embarked_S, here we dropped C to remove redundancy, two columns are sufficient.

Decision Tree Classifier Split the dataset into train and test with test size 20%. Then implemented a Decision tree classifier as the first model with default values and checked the accuracy initially I have got score around 0.77, then tried changing the hyper parameters of decision tree classifier like max_depth, min_samples_leaf, min_samples_split. To overcome overfitting the test dataset I have used cross validation with cv=5 which slightly improved the test score by taking the mean of CV scores. Now the next challenge was to find the decision tree classifier model with the best combination of the hyper parameters then tried implementing grid search using the param_grid having all the hyper parameters and tried with different values of the parameters and checked the best scores and best params each time, and finalised on a decision which gave an accuracy score of 0.81 with the params as below. {‘max_depth’: 10, ‘min_samples_leaf’: 5, ‘min_samples_split’: 2}

Random Forest Classifier Similarly I have tried with the Random Forest classifier, initially with the default values of the classifier I have got the accuracy score around 0.81, then to overcome overfitting during parameter tuning I used cross validation scores and tried different model with different parameters like n_estimators, max_depth, min_samples_leaf, etc each time and got the best mean cross validation score around 0.8. Then to test out more values of hyperparams I used Grid Search with param_grid having 4 parameters mentioned above tried different combinations of values for each parameter and compared the best score and best params of all tested and got the best accuracy score of the test dataset approximately 0.83 with the below combination of params. {‘max_depth’: 5, ‘min_samples_leaf’: 5, ‘min_samples_split’: 8 ‘n_estimators’: 100}

What can be Improved? I think the performance can be improved by doing some feature engineering like adding some new columns by extracting it from the existing columns. I have tried tuning the model but never got a good accuracy score like around 0.90.

Conclusion Overall, I enjoyed participating in my first competition, and it was a very valuable learning experience. The quality of the dataset was good, as it was well-structured and provided good features which let me focus on improving my approach instead of wasting time fixing data issues.