Overview

Teaching: 20 min
Exercises: 0 min

Questions

• How to train a Machine Learning model using Tree-based model

Objectives

• Learn to use different Tree-based algorithm for Machine Learning training

Supervised Learning training

Train model using Decision Tree

• Tree based learning algorithms are considered to be one of the best and mostly used supervised learning methods.
• Tree based methods empower predictive models with high accuracy, stability and ease of interpretation
• Non-parametric and non-linear relationships
• Types: Categorical and Continuous

Spliting algorithm

• Gini Impurity: (Categorical)
• Chi-Square index (Categorical)
• Cross-Entropy & Information gain (Categorical)
• Reduction Variance (Continuous)

Pros & Cons

Implementation

Here we will use iris data

library(caret)
data(iris)
set.seed(123)
indT <- createDataPartition(y=iris$Species,p=0.6,list=FALSE)
training <- iris[indT,]
testing  <- iris[-indT,]

Next we will train using method="rpart" with gini splitting algorithm:

ModFit_rpart <- train(Species~.,data=training,method="rpart",
                      parms = list(split = "gini"))
# gini can be replaced by chisquare, entropy, information

#fancier plot
library(rattle)
fancyRpartPlot(ModFit_rpart$finalModel)

Apply decision tree model to predict output of testing data

predict_rpart <- predict(ModFit_rpart,testing)
confusionMatrix(predict_rpart, testing$Species)

testing$PredRight <- predict_rpart==testing$Species
ggplot(testing,aes(x=Petal.Width,y=Petal.Length))+
  geom_point(aes(col=PredRight))

Train model using Random Forest

• Random Forest is considered to be a panacea of all data science problems. On a funny note, when you can’t think of any algorithm (irrespective of situation), use random forest!
• Opposite to Decision Tree, Random Forest use bootstrapping technique to grow multiple tree
• Random Forest is a versatile machine learning method capable of performing both regression and classification tasks.
• It is a type of ensemble learning method, where a group of weak models combine to form a powerful model.
• The end output of the model is like a black box and hence should be used judiciously.

  Detail explaination

• If there are M input variables, a number m<M is specified such that at each node, m variables are selected at random out of the M. The best split on these m is used to split the node. The value of m is held constant while we grow the forest.
• Each tree is grown to the largest extent possible and there is no pruning.
• Predict new data by aggregating the predictions of the ntree trees (i.e., majority votes for classification, average for regression).

Pros & Cons of Random Forest

Implementation of Random Forest

ModFit_rf <- train(Species~.,data=training,method="rf",prox=TRUE)

predict_rf <- predict(ModFit_rf,testing)
confusionMatrix(predict_rf, testing$Species)

testing$PredRight <- predict_rf==testing$Species
ggplot(testing,aes(x=Petal.Width,y=Petal.Length))+
  geom_point(aes(col=PredRight))

We can see that Random Forest result has better prediction than Decision Tree

Key Points

• Decision Tree, Random Forest