Decision Tree

Splitting algorithm

• Gini Impurity: (Categorical)
• Chi-Square index (Categorical)
• Cross-Entropy & Information gain (Categorical)
• Reduction Variance (Continuous)

Example on Decision Tree

Assume we have a sample data of 30 students in the training set with three input variables Gender (Boy/ Girl), Class( IX/ X) and Height (5 to 6 ft); one output variable: Play_Cricket (binary)

• 15 out of these 30 play cricket in leisure time (15 Yes and 15 No)
• predict who will play cricket during leisure period?

This is a typical problem for Decision Tree algorithm

Next we gonna use different splitting algorithm to split the nodes:

Gini Impurity

As there are 3 input variables (Gender, Class, Height), the algorithm will be splitting for all variables and calculate the Gini Impurity correspondingly based on the above formulation:

Splitting based on Gender:

• The whole population of splitting is 30, the probability of Yes & No are 50% for the whole sample

• After splitting by Gender, the number of Male students is 20, the number of cricket player is 13, equivalent to 65% male population. Therefore Gini value of Male is:

• the number of Female students is 10, the number of cricket player is 2, equivalent to 20% female population. Therefore Gini value of Female is:

• The Gini Impurity for Gender splitting is:

Similarly, splitting based on Height:

• The Gini Impurity for Height splitting is 0.5

Splitting based on Class:

• The Gini Impurity for Class splitting is 0.49

Base on the 3 Gini Impurity on the splitting, we go with the smallest values, which is splitting by Gender.

The process is continue with the next nodes with other variables.

Chi-Squared

Entropy