Predicting Default Risk (Creditworthiness)

Comments: No comment

Step 1: Business and Data Understanding

  1. What decisions need to be made?

We need to make a decision to determine the creditworthiness of the 500 new applications received

  1. What data is needed to inform those decisions?

Data on past applications received by the bank with an outcome (either creditworthy or not creditworthy). This data includes credits taken by the applicant, account balance, age of the applicant, payment status of previous loans, valuable assets etc

  1. What kind of model (Continuous, Binary, Non-Binary, Time-Series) do we need to use to help make these decisions?

A binary classification model is needed because there are only two options. An applicant is either creditworthy or not

Step 2: Building the Training Set

  1. In your cleanup process, which field(s) did you impute or remove?

To make this decision, I ran the field summary tool:

The following fields were removed or imputed:

  1. The first field I removed was “Duration-in-Current-address”.  This field has 69% of its data missing and is irrelevant to what we are trying to predict
  2. The other field with incomplete data, with 2% missing is ”Age-years”. This is a small number missing and I decided to impute the missing data with the median (33.0) because the data is skewed to the left.This option is best as it’s the closest to reality
  3. “Concurrent-credits” field isn’t missing any data but only has one unique value so is not a good predicting variable therefore I would also be removing this.
  4. “Occupation” also has one unique value therefore I would also drop this field
  5. “Telephone” is not a good predicting variable and should be removed
  6. “Foreign-Worker” has a small number of unique values and is skewed so I removed it as well
  7. “No-of-dependents” does not appear to be a good predicting variable and was also dropped
  8. “Guarantors” has two unique values with one of them dominant and was also dropped

Step 3. Train your Classification Models

Using the Create Samples tool in Alteryx, 70% was chosen to create the Estimation set and 30% to create the Validation set and the Random Seed was set to 1.

The dataset was trained using the following models:

  • Logistic Regression
  • Decision Tree
  • Forest Model
  • Boosted Tree
  1. Which predictor variables are significant or the most important? Please show the p-values or variable importance charts for all of your predictor variables.

Logistic Regression:

The significant predictor variables (with P values less than 0.05) are Account Balance, Payment.status.of.previous.CreditSome Problems, PurposeNew Car, Credit.amount, Length of current employment, Instalment percent based on the report:

Decision Tree:

The top variables used are Account.Balance, Duration.of.Credit.Month,   Value.Savings.Stocks

Forest Model:

The plot below shows the variable importance for the forest model with the top variables being; Credit.Amount, Age.years, Duration.of.Credit.Month, Account.Balance

Boosted Tree

The image below shows the variable importance plot for the boosted tree model with the top variables being; Account.Balance, Credit.Amount, Duration.of.Credit.Month, Payment.Status.of.Previous.Credit

  1. Validate your model against the Validation set. What was the overall percent accuracy? Show the confusion matrix. Is there any bias seen in the model’s predictions?

To validate and get the accuracy of the four models, I used the model comparison tool 

The overall accuracy for the Logistic regression model is 0.7800; the overall accuracy for the Decision Tree is 0.6667; the overall accuracy for the Forest Model is 0.7933; and the overall accuracy for the Boosted Model is 0.7867. 

To find out if the models have a bias, I used the formula: 

NPV=(No. of true negatives)/(Total number of negatives predicted) 

PPV=(No. of true positives)/(Total no. of positives predicted)

For the Boosted model,

PPV = 101/129 = 78% and NPV = 17/21 = 81%. 

The values are close to each other therefore this model has no bias.

For the Decision Tree, 

PPV = 83/111 = 75% and NPV = 17/39 = 44%. 

This model has a bias to Creditworthy.

For the Forest model, 

PPV = 102/130 = 78% and NPV = 17/20 = 85%. 

The values are close to each other therefore this model has no bias.

For the logistic regression, we have

 PPV = 95/118 = 81% and NPV = 22/32 = 69%. 

As the values are not so close, we can say that this model has a bias to Creditworthy.

Step 4. Writeup

Which model did you choose to use? 

I decided to use the Forest Model based on the following reasons:

Overall Accuracy against your Validation set:

Forest Model had the highest overall accuracy of all the four models

Accuracies within “Creditworthy” and “Non-Creditworthy” segments

The forest model has the highest accuracy in the creditworthy and the non-creditworthy segments

ROC graph

Bias in the Confusion Matrices

As seen above, there are two models that have no bias but the forest model has a higher NPV and is a better model

How many individuals are creditworthy?

There are 408 creditworthy individuals and 92 non-creditworthy

Leave a Reply